Changes to requirements,txt to remove unneeded dependencies (#114)

* Changes to requirements,txt to remove unneeded dependencies Added exclusions to conf.py and setup.py to skip install of some dependencies for ReadTheDocs - these packages will be mocked in their build Bug in azure_auth_core.py - raise from should have been an exception object, not a class Bug in secret_settings.py - potential uninitialized variable Updated test_mordor_driver.py to remove files on completion, Also marked file download tests as CI-only Added pylint and flake8 as pre-commit hooks - should stop fewer errors getting into build Some tidying of code (Sourcery) in azure_auth_core and secret_settings. Updated version to 0.8.7 * Fixing some warnings in the documentation. Fixing other things too.
microsoft · Oct 21, 2020 · db6e6f9 · db6e6f9
1 parent 3c87222
commit db6e6f9
Show file tree

Hide file tree

Showing 15 changed files with 322 additions and 178 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,22 +1,35 @@
 repos:
--   repo: https://github.com/pre-commit/pre-commit-hooks
+  - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v3.2.0
     hooks:
-    -   id: check-yaml
-    -   id: check-json
-    -   id: trailing-whitespace
+      - id: check-yaml
+      - id: check-json
+      - id: trailing-whitespace
         args: [--markdown-linebreak-ext=md]
--   repo: https://github.com/ambv/black
+  - repo: https://github.com/ambv/black
     rev: 20.8b1
     hooks:
-    - id: black
-      language: python
--   repo: local
+      - id: black
+        language: python
+  - repo: https://github.com/pre-commit/mirrors-pylint
+    rev: v2.6.0
     hooks:
-    - id: download_tlds
-      name: download_tlds
-      entry: python -m pre-commit.download_tlds
-      pass_filenames: False
-      language: python
-      types: [python]
-      args: [--tld-seed=./msticpy/sectools/tld_seed.txt]
+      - id: pylint
+        args:
+          - --disable=E0401
+  - repo: https://gitlab.com/pycqa/flake8
+    rev: 3.8.4
+    hooks:
+      - id: flake8
+        args:
+          - --extend-ignore=E0401,E501
+          - --max-line-length=90
+  - repo: local
+    hooks:
+      - id: download_tlds
+        name: download_tlds
+        entry: python -m pre-commit.download_tlds
+        pass_filenames: False
+        language: python
+        types: [python]
+        args: [--tld-seed=./msticpy/sectools/tld_seed.txt]
diff --git a/conda/conda-reqs-pip.txt b/conda/conda-reqs-pip.txt
@@ -1,4 +1,3 @@
-azure-cli-core>=2.5.0
 azure-core>=1.2.2
 azure-identity==1.4.0
 azure-keyvault-secrets>=4.0.0
@@ -9,6 +8,5 @@ azure-mgmt-subscription>=0.2.0
 geoip2>=2.9.0
 ipwhois>=1.1.0
 Kqlmagic>=0.1.106
-msal~=1.0.0
 splunk-sdk>=1.6.0
 tldextract>=2.2.2
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -6,6 +6,14 @@
 # full list see the documentation:
 # http://www.sphinx-doc.org/en/master/config
 
+# type: ignore
+
+# pylint: disable=invalid-name, missing-module-docstring, import-error
+
+# noqa D100
+
+# flake8: noqa
+
 # -- Path setup --------------------------------------------------------------
 
 # If extensions (or modules to document with autodoc) are in another directory,
@@ -194,3 +202,16 @@
 }
 
 autoclass_content = "both"
+
+autoapi_dirs = ["../../msticpy"]
+
+autodoc_mock_imports = [
+    "Kqlmagic",
+    "azure-cli-core",
+    "matplotlib",
+    "statsmodels",
+    "scipy",
+    "sklearn",
+    "splunk-sdk",
+    "seaborn",
+]
diff --git a/docs/source/data_acquisition/AzureData.rst b/docs/source/data_acquisition/AzureData.rst
@@ -61,13 +61,18 @@ See :py:class:`Azure Data <msticpy.data.azure_data.AzureData>`
 In order to connect to the Azure API and retrieve the required data
 we need to instantiate an Azure Data Connector and connect to the API.
 Authentication to the Azure API is handled via the azure_auth package.
-By default this package will attempt to use a prioritized list of authentication 
+By default this package will attempt to use a prioritized list of authentication
 options. Available options are:
+
     * 'env' - This checks for credentials stored as environment variables. If this option is selected valid credentials in msticpyconfig.yaml will be written as environment variable values and used.
-    * 'cli' - This attempts to use credentials generated by logging in via the `Azure CLI <https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli>`__on the host running the notebook kernel.
-    * 'msi' - This attempts to use an `Azure Managed Identity. <https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview>`__
+    * 'cli' - This attempts to use credentials generated by logging in via the
+      `Azure CLI <https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli>`__
+      on the host running the notebook kernel.
+    * 'msi' - This attempts to use an
+      `Azure Managed Identity. <https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview>`__
     * 'interactive' - This prompts the browser to interactively login using the device's browser.
-By default `['env', 'cli', 'msi', 'interactive']` is used but you can provide an alternative 
+
+By default `['env', 'cli', 'msi', 'interactive']` is used but you can provide an alternative
 list to `.connect` via the auth_methods parameter.
 
 .. code:: ipython3

diff --git a/docs/source/data_acquisition/AzureSentinel.rst b/docs/source/data_acquisition/AzureSentinel.rst
@@ -5,7 +5,7 @@ Description
 -----------
 
 This package contains functionality making calls to Azure Sentinel directly.
-These can be used to get data from Azure Sentinel, as well as perform 
+These can be used to get data from Azure Sentinel, as well as perform
 configuration and other actions on the Azure Sentinel Workspace
 
 :py:mod:`Azure Sentinel API documentation<msticpy.data.azure_sentinel>`
@@ -56,13 +56,20 @@ See :py:class:`Azure Sentinel <msticpy.data.azure_data.AzureSentinel>`
 In order to connect to the Azure API and retrieve the required data
 we need to instantiate an Azure Data Connector and connect to the API.
 Authentication to the Azure API is handled via an the azure_auth package.
-By default this package will attempt to use a prioritized list of authentication 
+By default this package will attempt to use a prioritized list of authentication
 options. Available options are:
-    * 'env' - This checks for credentials stored as environment variables. If this option is selected valid credentials in msticpyconfig.yaml will be written as environment variable values and used.
-    * 'cli' - This attempts to use credentials generated by logging in via the `Azure CLI <https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli>`__on the host running the notebook kernel.
-    * 'msi' - This attempts to use an `Azure Managed Identity. <https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview>`__
+
+    * 'env' - This checks for credentials stored as environment variables.
+      If this option is selected valid credentials in msticpyconfig.yaml will be
+      written as environment variable values and used.
+    * 'cli' - This attempts to use credentials generated by logging in via the
+      `Azure CLI <https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli>`__
+      on the host running the notebook kernel.
+    * 'msi' - This attempts to use an
+      `Azure Managed Identity. <https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview>`__
     * 'interactive' - This prompts the browser to interactively login using the device's browser.
-By default `['env', 'cli', 'msi', 'interactive']` is used but you can provide an alternative 
+
+By default `['env', 'cli', 'msi', 'interactive']` is used but you can provide an alternative
 list to `.connect` via the auth_methods parameter.
 
 .. code:: ipython3
@@ -75,7 +82,7 @@ Get Azure Subscription Details
 
 See :py:meth:`get_subscriptions <msticpy.data.azure_data.AzureData.get_subscriptions>`
 
-Details of which Azure subscription to access is required before identifying and connecting 
+Details of which Azure subscription to access is required before identifying and connecting
 to an Azure Sentinel workspace.
 
 AZURE_SENTINEL_CONNECTOR.list_subscriptions() returns a pandas DataFrame
@@ -121,15 +128,16 @@ with details of all the subscriptions within the tenant.
       </tbody>
     </table>
     </div>
+
 |
 
 Get Azure Sentinel Workspaces
 -----------------------------
 
 See :py:meth:`get_sentinel_workspaces <msticpy.data.azure_sentinel.AzureSentinel.get_sentinel_workspaces>`
 
-A list of Azure Sentinel workspaces within a specified subscription. 
-Note that this will only return workspaces that the authenticated account 
+A list of Azure Sentinel workspaces within a specified subscription.
+Note that this will only return workspaces that the authenticated account
 is permitted to view.
 
 .. code:: ipython3
@@ -139,10 +147,10 @@ is permitted to view.
 List Hunting get_hunting_queries
 --------------------------------
 
-Return a dataframe detailing all hunting queries configured in the workspace. This allows for 
-analysis and configuration of hunting queries, as well as the ability to take a 
-hunting query and run it with a QueryProvider. This function requires that you 
-pass it the resource ID string of the Azure Sentinel workspace to get the queries 
+Return a dataframe detailing all hunting queries configured in the workspace. This allows for
+analysis and configuration of hunting queries, as well as the ability to take a
+hunting query and run it with a QueryProvider. This function requires that you
+pass it the resource ID string of the Azure Sentinel workspace to get the queries
 from. This ID can be obtained with `get_sentinel_workspaces`
 
 See :py:meth:`get_hunting_queries <msticpy.data.azure_sentinel.AzureSentinel.get_hunting_queries>`
@@ -151,12 +159,12 @@ See :py:meth:`get_hunting_queries <msticpy.data.azure_sentinel.AzureSentinel.get
 
     azs.get_hunting_queries(res_id = "subscriptionId/3b701f84-d04b-4479-89b1-fa8827eb537e/resourceGroup/SentinelRG/workspaceName/SentinelWorkspace")
 
-List Configured Alert Rules 
+List Configured Alert Rules
 ---------------------------
 
 Return a dataframe detailing all configured alert/analytics rules configured with Azure Sentinel.
-This includes scheduled queries, as well as Fusion based detections. The returned dataframe include 
-details of the rule configuration as well as the query run (where applicable). As with other functions 
+This includes scheduled queries, as well as Fusion based detections. The returned dataframe include
+details of the rule configuration as well as the query run (where applicable). As with other functions
 the resource ID of the workspace to get alerts from is requried.
 
 See :py:meth:`get_alert_rules <msticpy.data.azure_sentinel.AzureSentinel.get_alert_rules>`
@@ -165,12 +173,12 @@ See :py:meth:`get_alert_rules <msticpy.data.azure_sentinel.AzureSentinel.get_ale
 
     azs.get_alert_rules(res_id = "subscriptionId/3b701f84-d04b-4479-89b1-fa8827eb537e/resourceGroup/SentinelRG/workspaceName/SentinelWorkspace")
 
-List Bookmarks 
+List Bookmarks
 --------------
 
-Return a list of all the bookmarks saved in the workspace. This includes details of the bookmark, who 
-created it, when and with what details. It also includes query text that can be executed with a 
-`QueryProvider` in order to get the details of the bookmark's logs. As with other functions the resource 
+Return a list of all the bookmarks saved in the workspace. This includes details of the bookmark, who
+created it, when and with what details. It also includes query text that can be executed with a
+`QueryProvider` in order to get the details of the bookmark's logs. As with other functions the resource
 ID of the workspace to get alerts from is requried.
 
 See :py:meth:`get_bookmarks <msticpy.data.azure_sentinel.AzureSentinel.get_bookmarks>`