Sentinel and Kusto new providers (#656)

* Initial code for az_monitor_driver.py

Added provider_settings.py for global proxy config
Add settings.py as a facade module for common setttings functions
Added lazy_import function and generic functions for implementing dynamic __getattr__ and __dir__ functions for __init__ modules
Added ability to create (partial) WorkspaceConfig from connection string.
Moved MpConfigEdit and MpConfigFile to dynamic imports in config/__init__.py
Changed MSSentinel import in ce_azure_sentinel.py to be imported on demand
Extended _execute_query to extract and supply timespan parameters to driver query functions
Removed deprecated imports or nbtools and sectools from nbinit.py
Importing get_config to msticpy/__init__.py

* Removing use of multiple workspace IDs from az_monitor_driver

* Adding workaround protection for process tree NA values for Bokeh 3.0

Moving bokeh 3.0 requirements back to 2.4.3 for compat with panel.

* Initial code for azure_monitor_driver

* Refactoring methods from QueryProvider into mixin classes

* Azure kusto driver based on azure-data-kusto

* Azure Kusto driver and Azure monitor updates:

- Adding to DataEnvironments and drivers/__init__.py
Adding raw Kusto response test data

* Finished unit tests for test_azure_monitor_driver.py / azure_monitor_driver.py

Implemented driver properties dictionary and use of DriverProps class to normalize naming in multiple drivers.
Added code to allow driver to override environment name for reading queries.

* Added documentation and fixed unit tests.

Updated config documentation
Fixed keyring_client.py test for valid backend
Merged several global settings (proxies, http timeout) into "msticpy" section of config
Added settings editor support for msticpy global settings in ce_msticpy.py
Some Mypy fixes in file_browser.py
Removed some commented-out code from query_source.py
Added filtering for queries to query_source.py and data_providers.py - this allows individual kusto providers
to show only queries that are relevant for the connected cluster. Additional test files and unit tests to support this.
Added consistent support for proxy settings and timeout in azure_kusto_driver.py and azure_monitor_driver.py
Updated setup.py, requirements-all.txt and conda-reqs.txt to add azure-kusto-data and azure-monitor-query packages.
Added alternative to custom_mp_config test utility. This patches get_config calls in specified modules - more complex to set up but does not rely on a lock file, so allows more unit tests to run in parallel.

* Fix for sentinel and azurekusto test failures

* Fixing test and lint failures

* Fixing settings source for test_sentinel_core.py

Change folder name for test_azure_monitor_driver.py (not sure why but seems to have an effect on linux)

* isort fixes in polling_detection.py

pylint check name change in azure_auth_core.py
adding validate_config to settings.py
added ability to get workspace using case-insensitive ws name, ID or key name in wsconfig.py
removed some debugging lines from sentinel_utils.py
suppressed pylint error in data_providers.py
fixed field initializer in query_template.py
adding more logging to azure_monitor_driver.py
adding more logging to azure_kusto_driver.py
removing matplotlib import in nbinit, simplifying config loading code if config is already loaded.
Moving matplotlib to extra in requirements.txt, setup.py
Black re-format of test_periodogram_polling_detector.py
Added tests for WorkspaceConfig to test_wsconfig.py
Fixed test_azure_monitor_driver.py to use consistent settings.
Fixed test_nbinit.py to avoid using default loaded settings.

* Fix for 0.14.0 of statsmodels

conda/conda-reqs.txt

@@ -4,10 +4,12 @@ azure-core>=1.24.0
 azure-mgmt-core>=1.2.1
 azure-identity>=1.10.0
 azure-keyvault-secrets>=4.0.0
+azure-kusto-data>=4.0.0
 azure-mgmt-compute>=4.6.2
 azure-mgmt-keyvault>=2.0.0
 azure-mgmt-network>=2.7.0
 azure-mgmt-resource>=16.1.0
+azure-monitor-query>=1.0.0
 azure-storage-blob>=12.5.0
 beautifulsoup4>=4.0.0
 bokeh>=1.4.0, <=2.4.3

docs/source/DataAcquisition.rst

@@ -17,6 +17,7 @@ Individual Data Environments
    :maxdepth: 2
 
    data_acquisition/DataProv-MSSentinel
+   data_acquisition/DataProv-MSSentinel-New
    data_acquisition/DataProv-MSDefender
    data_acquisition/DataProv-MSGraph
    data_acquisition/DataProv-LocalData
@@ -25,6 +26,7 @@ Individual Data Environments
    data_acquisition/MordorData
    data_acquisition/DataProv-Sumologic
    data_acquisition/DataProv-Kusto
+   data_acquisition/DataProv-Kusto-New
    data_acquisition/DataProv-Cybereason
    data_acquisition/DataProv-OSQuery
 

docs/source/api/msticpy.common.proxy_settings.rst

@@ -0,0 +1,7 @@
+msticpy.common.proxy\_settings module
+=====================================
+
+.. automodule:: msticpy.common.proxy_settings
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/api/msticpy.common.rst

@@ -27,5 +27,7 @@ Submodules
    msticpy.common.exceptions
    msticpy.common.pkg_config
    msticpy.common.provider_settings
+   msticpy.common.proxy_settings
+   msticpy.common.settings
    msticpy.common.timespan
    msticpy.common.wsconfig

docs/source/api/msticpy.common.settings.rst

@@ -0,0 +1,7 @@
+msticpy.common.settings module
+==============================
+
+.. automodule:: msticpy.common.settings
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/api/msticpy.common.utility.rst

@@ -16,3 +16,4 @@ Submodules
    msticpy.common.utility.ipython
    msticpy.common.utility.package
    msticpy.common.utility.types
+   msticpy.common.utility.yaml_loader

docs/source/api/msticpy.common.utility.yaml_loader.rst

@@ -0,0 +1,7 @@
+msticpy.common.utility.yaml\_loader module
+==========================================
+
+.. automodule:: msticpy.common.utility.yaml_loader
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/api/msticpy.config.ce_msticpy.rst

@@ -0,0 +1,7 @@
+msticpy.config.ce\_msticpy module
+=================================
+
+.. automodule:: msticpy.config.ce_msticpy
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/api/msticpy.config.rst

@@ -17,6 +17,7 @@ Submodules
    msticpy.config.ce_common
    msticpy.config.ce_data_providers
    msticpy.config.ce_keyvault
+   msticpy.config.ce_msticpy
    msticpy.config.ce_other_providers
    msticpy.config.ce_provider_base
    msticpy.config.ce_simple_settings

docs/source/api/msticpy.data.core.query_provider_connections_mixin.rst

@@ -0,0 +1,7 @@
+msticpy.data.core.query\_provider\_connections\_mixin module
+============================================================
+
+.. automodule:: msticpy.data.core.query_provider_connections_mixin
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/api/msticpy.data.core.query_provider_utils_mixin.rst

@@ -0,0 +1,7 @@
+msticpy.data.core.query\_provider\_utils\_mixin module
+======================================================
+
+.. automodule:: msticpy.data.core.query_provider_utils_mixin
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/api/msticpy.data.core.query_template.rst

@@ -0,0 +1,7 @@
+msticpy.data.core.query\_template module
+========================================
+
+.. automodule:: msticpy.data.core.query_template
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/api/msticpy.data.core.rst

@@ -17,5 +17,8 @@ Submodules
    msticpy.data.core.param_extractor
    msticpy.data.core.query_container
    msticpy.data.core.query_defns
+   msticpy.data.core.query_provider_connections_mixin
+   msticpy.data.core.query_provider_utils_mixin
    msticpy.data.core.query_source
    msticpy.data.core.query_store
+   msticpy.data.core.query_template

docs/source/api/msticpy.data.drivers.azure_kusto_driver.rst

@@ -0,0 +1,7 @@
+msticpy.data.drivers.azure\_kusto\_driver module
+================================================
+
+.. automodule:: msticpy.data.drivers.azure_kusto_driver
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/api/msticpy.data.drivers.azure_monitor_driver.rst

@@ -0,0 +1,7 @@
+msticpy.data.drivers.azure\_monitor\_driver module
+==================================================
+
+.. automodule:: msticpy.data.drivers.azure_monitor_driver
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/api/msticpy.data.drivers.rst

@@ -12,6 +12,8 @@ Submodules
 .. toctree::
    :maxdepth: 4
 
+   msticpy.data.drivers.azure_kusto_driver
+   msticpy.data.drivers.azure_monitor_driver
    msticpy.data.drivers.cybereason_driver
    msticpy.data.drivers.driver_base
    msticpy.data.drivers.elastic_driver

docs/source/conf.py

@@ -226,6 +226,7 @@
     "azure.identity",
     "azure.keyvault.secrets",
     "azure.keyvault",
+    "azure.kusto.data",
     "azure.mgmt.compute.models",
     "azure.mgmt.compute",
     "azure.mgmt.keyvault.models",
@@ -235,6 +236,7 @@
     "azure.mgmt.resource",
     "azure.mgmt.resourcegraph",
     "azure.mgmt.subscription",
+    "azure.monitor.query",
     "azure.storage.blob",
     "azure.storage",
     "bokeh",
@@ -259,6 +261,7 @@
     "nest_asyncio",
     "networkx",
     "openpyxl",
+    "panel",
     "passivetotal",
     "pygeohash",
     "pygments",