Ianhelle/merge2.0 to main 2022 06 14 #443

ianhelle · 2022-06-15T01:30:42Z

No description provided.

* move query files * Modules moves and import updates * Stub files and restructure fixes * More stubs and Pivots restructure * Updated _init__ files * Merging in updates to Azure auth * Linting fixes * Fixed circular import and test imports * updated failing test * Test fix * Test fix * Making tests more resilient for multiple environments * removing accidental additional parent in cmd_line default path * Added additional stub files * added exports for back compatibility * re-adding httpx changes lost in merge * restructure cyberreason and splunk queries * Fixed incorrect vtlookup * re-adding vt-graph-api fix * Adding back in query regex * Added missing httpx update * Fixed broken test * Fixing incorrect import in test * Updating missed Conda version for respx * Updating API docs * Updated cybereason folder names * Fixed Sentinel APIs * Updated test mocked data to match new API * Renamed data.context_providers to context Renamed analysis.data_processing to data Renamed data.common to data.core Removed some un-needed redirection files Moved some of the context modules (geoip, ip_utils, domain_utils) to data/context folder Added docstrings to redirection files so that they link to right location in read-the-docs Updated docs with new paths Updated notebooks with new module paths Changed RTD to generate a page for each module. Add text to deprecation warning that we'll remove in v2.0.0 * Adding some additional path fixes for tests and linting errors * Added missing changes to test_cybereason_driver Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com>

…331) * Updated names in AzureData to match MicrosoftSentinel Moved list_sentinel_workspaces to AzureData Added connection checks to Sentinel features * Added linting suppression for Mixin errors

* adding devcontainer files (#321) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> # Conflicts: # .devcontainer/devcontainer.json * Syncing recent IPython-related changes (to skip magic creation if not in IPython)

* Keyring refresh changes * Powershell viewer PR * Elastic driver skeleton and changes to allow driver-specific param substitution * Synced updates to nbmagics.py * Cleaning up some import redirections to point to new locations Fixing circular import in vtfile_behavior Adding placeholder class to allow imports to sort of work even if sub-modules fail to import Refactored _value_or_default in query_source to reduce complexity Fixed import errors in elastic_driver.py and splunk_driver.py Fixed import from old location in nbinit Fixing warning in code_view.py Fixed test failure in test_code_view Changing test_timeline.py to use new mp_plot accessor in place of deprecated one. Fixing test failure in test_timeline.py * Supressing bandit false positives

* move query files * Modules moves and import updates * Stub files and restructure fixes * More stubs and Pivots restructure * Updated _init__ files * Merging in updates to Azure auth * Linting fixes * Fixed circular import and test imports * updated failing test * Test fix * Test fix * Making tests more resilient for multiple environments * removing accidental additional parent in cmd_line default path * Added additional stub files * added exports for back compatibility * re-adding httpx changes lost in merge * restructure cyberreason and splunk queries * Fixed incorrect vtlookup * re-adding vt-graph-api fix * Adding back in query regex * Added missing httpx update * Fixed broken test * Fixing incorrect import in test * Updating missed Conda version for respx * Updating API docs * Updated cybereason folder names * Fixed Sentinel APIs * Updated test mocked data to match new API * Renamed data.context_providers to context Renamed analysis.data_processing to data Renamed data.common to data.core Removed some un-needed redirection files Moved some of the context modules (geoip, ip_utils, domain_utils) to data/context folder Added docstrings to redirection files so that they link to right location in read-the-docs Updated docs with new paths Updated notebooks with new module paths Changed RTD to generate a page for each module. Add text to deprecation warning that we'll remove in v2.0.0 * Adding isort to pre-commit.yaml Isorting all of the imports in msticpy, tests and tools Renaming query folders Removing some automatic imports from msticpy.__init__.py Adding search function to find modules to utility.py Fixing old paths in test_timeline.py * Adding triggers for release branches Cherry picked last two commits to petebryan/2020-1-25 * Adding isort to requirements-dev and conda-reqs-dev.txt * Bandit FPs in anomaly sequence modules * Fixed failing clustering notebook * Errors in notebook and keyvault tests Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com>

* move query files * Modules moves and import updates * Stub files and restructure fixes * More stubs and Pivots restructure * Updated _init__ files * Merging in updates to Azure auth * Linting fixes * Fixed circular import and test imports * updated failing test * Test fix * Test fix * Making tests more resilient for multiple environments * removing accidental additional parent in cmd_line default path * Added additional stub files * added exports for back compatibility * re-adding httpx changes lost in merge * restructure cyberreason and splunk queries * Fixed incorrect vtlookup * re-adding vt-graph-api fix * Adding back in query regex * Added missing httpx update * Fixed broken test * Fixing incorrect import in test * Updating missed Conda version for respx * Updating API docs * Updated cybereason folder names * Fixed Sentinel APIs * Updated test mocked data to match new API * Renamed data.context_providers to context Renamed analysis.data_processing to data Renamed data.common to data.core Removed some un-needed redirection files Moved some of the context modules (geoip, ip_utils, domain_utils) to data/context folder Added docstrings to redirection files so that they link to right location in read-the-docs Updated docs with new paths Updated notebooks with new module paths Changed RTD to generate a page for each module. Add text to deprecation warning that we'll remove in v2.0.0 * Adding isort to pre-commit.yaml Isorting all of the imports in msticpy, tests and tools Renaming query folders Removing some automatic imports from msticpy.__init__.py Adding search function to find modules to utility.py Fixing old paths in test_timeline.py * Adding triggers for release branches Cherry picked last two commits to petebryan/2020-1-25 * Adding isort to requirements-dev and conda-reqs-dev.txt * Bandit FPs in anomaly sequence modules * Fixed failing clustering notebook * Errors in notebook and keyvault tests * Fixing test issues in MicrosoftDefender.ipynb and EventClustering.ipynb Bug using wrong httpx code syntax in odata_driver.py Removing auto-load of VTLookup in nbinit Forcing notebook tests to use test msticpyconfig-test.yaml Added missing __init__.py to tests/data/browsers * Removing vtlookup import from sectools init because of circular import error Moving IPStack check for API key to first call (rather than __init__) to avoid error on load. * prospector config changed produces deprecation warning and non-zero exit code. * Fixing misconfigured prospector.yaml * Getting rid of warning from test_nbinit Adding McCabe suppression to ip_utils.py * removing version restriction for prospector in Github actions python-package.yaml Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com>

* Query providers load pivots dynamically when created. Renamed query paths and changed data_providers so that only env-specific queries are loaded. Moving ensure_df_datetimes to common/data_utils.py to avoid circular imports Consolidated data-related pandas accessors into single module. * Fixing circular dependency in iocextract Fixing linting errors in data_providers.py, azure_resource.py, host.py, process.py, pivot_data_queries.py * Adding default timeout values to httpx calls. Changing tor_exit_nodes.py Tor provider to defer download of tor list until first lookup Fixing test for trigger Tor node download before running test. * Re-ordering arguments so doesn't break inheritance and cause pylint warning * Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. (from main branch) Now loads with empty settings rather than throw exception. Added unit test case * Reverting changes to args and adding pylint suppressions # Conflicts: # msticpy/data/uploaders/splunk_uploader.py * Aligning splunk_uploader params with base class

and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352.

* Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters. Fixed query_time widget so that you can reset time range from parameter Added additional unit test for QueryTime setter Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line. Also changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName. Updated TimeSpan class so it has more flexible constructor Added account_id as a parameter for list_aad_signins_by_account query * Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts to defaults. This can cause errors if the defaults are different to user-specified parameters. There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings. * MyPy warning in sentinel_core.py

…e, pkg_config and kusto_driver (#359) Some fixes to Kusto common_imports - now works with Kusto config entry without instance suffix - can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query - added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding in the data_famiies key. Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py. Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings. Reorganized logic for handling parameters and failing on invalid file path in config module.

* hotfix for bug found in testing * Fixed re-auth on query issue in KQL driver * Removing un-needed code * Fixed kql_driver tests * Liniting fixes Co-authored-by: Pete Bryan <pebryan@microsoft.com>

* Fixed minor issues * Fixed additional use case

* Merging changes from main for geoip.py, mp_config_edit, mp_config_file, pkg_config and kusto_driver Some fixes to Kusto common_imports - now works with Kusto config entry without instance suffix - can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query - added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding in the data_famiies key. Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py. Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings. Reorganized logic for handling parameters and failing on invalid file path in config module. * Moving analysis.data to transform folder * Moving data.context to context * Merging changes from main * Move auth and secrets modules to auth folder * Move nbwidgets to new folder Update deprecation warning to v2.0 * Fix to URLs in README.md * Adding init folder - moved: - nbinit.py, user_config.py, pivot.py, azure_ml_tools.py to here - also moved vt_pivot.py and pivot_ti_provider.py to init/pivot_init Renaming datamodel/pivots to datamodel/pivot Moved azure_blob_storage.py to data/storage folder Refactored query_container to data_types.py - to be separate types for queries (query_container.py) and pivots (pivot_container.py) Moved browsers to vis folder * Updating API docs * Final documentation and test fixes * Moved all pivot functions to init folder. Added functionality to pkg_config to delete and translate settings (for AzureSentinel->MSSentinel switch TBD) Added automatic acquisition of globals() in nbinit.py Fixed a couple of bugs in pivot_pipeline.py Removed direct import of pivot into datamodel/pivot and added code to add them dynamically after init.pivot initiialization. Added trap to timeline when supplied with no data. Fixed incorrect escaping in regex in kql_driver.py Notebook updates for errors and invalid links. Added script to run all notebooks for testing Updated API docs * Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x) Other components use the default. # Conflicts: # README.md # docs/source/getting_started/JupyterAndAzureSentinel.rst # msticpy/common/pkg_config.py # msticpy/context/azure/sentinel_analytics.py # msticpy/context/azure/sentinel_bookmarks.py # msticpy/context/azure/sentinel_incidents.py # msticpy/context/azure/sentinel_utils.py # msticpy/context/azure/sentinel_watchlists.py # msticpy/context/tiproviders/http_base.py # msticpy/data/drivers/cybereason_driver.py # msticpy/data/drivers/driver_base.py # msticpy/data/drivers/mordor_driver.py # msticpy/data/drivers/odata_driver.py # msticpy/data/uploaders/loganalytics_uploader.py # msticpy/sectools/domain_utils.py # msticpy/sectools/geoip.py # msticpy/sectools/vtlookup.py * Updated typing rigor for pkg_config::get_http_timeout Fixed bug in test test_pkg_config.py Also fixed bug test_code_view.py

Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default # Conflicts: # tests/data/azure/test_sentinel_search.py

Refactored a bit of code around tilookup and ti_provider_base.py. Added ability to supply "providers" parameter from pivot TI functions. Some fixes from incomplete merge of Sentinel search functions and documentation

Added new add_ips method to take an iterable of IP addresses. Updated all add methods to accept a "layer" parameter Allow IP entities and IP addresses with locations to be supplied (will use GeoLiteLookup) Removed a bit of unused code from geoip.py and avoid looking up non-Public IPs

…_editors and mpconfig_defaults.yaml

Refactoring http_provider.py to allow extending for non-TI purposes. Added http_lookup as generic API caller based on TI pattern. Made preprocess_observable extensible. Fixed some bugs in enable/disable providers in tilookup.py restored OPR lookup_iocs (it was not being called after addition of async). Updated test_tiproviders.py to more maintainable format. Updated test_tiprovider_kql.py to pytest Updated test_ip_utils.py to pytest Fixed/suppressed a bunch of test warnings.

…Accessor-2022-04-30 # Conflicts: # msticpy/context/tiproviders/__init__.py # msticpy/context/tiproviders/ti_provider_base.py

…config Fix to tor_exit_nodes.py - Tor no longer supplying list Changed Tor test to avoid online access - test_tiproviders.py

Some refactoring of kql_base.py to simplify code. Removed useless pylint suppresses from msticpy.context subpackage. Refactored test_tiprovider_kql.py Set mypy.ini to use Python 3.8 rules Removed Py 3.6, 3.7 from setup.cfg

review-notebook-app · 2022-06-15T01:30:49Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

…main-2022-06-14

…thub) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml

…) - odata_driver.py [fix] missing await in url_checker_async.py

* Bump sphinx from 4.3.2 to 4.4.0 (#283) * Bump sphinx from 4.3.2 to 4.4.0 Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.3.2 to 4.4.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](sphinx-doc/sphinx@v4.3.2...v4.4.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating requirements-dev.txt to sync with dependabot updates Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * doc updates (#316) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * adding devcontainer files (#321) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Bump respx from 0.17.1 to 0.19.2 (#314) Bumps [respx](https://github.com/lundberg/respx) from 0.17.1 to 0.19.2. - [Release notes](https://github.com/lundberg/respx/releases) - [Changelog](https://github.com/lundberg/respx/blob/master/CHANGELOG.md) - [Commits](lundberg/respx@0.17.1...0.19.2) --- updated-dependencies: - dependency-name: respx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated Cybereason docs to fit pattern (#324) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Ianhelle/1.6.1 hotfixes 2022 01 27 for 1.6.2 (#317) * Importing a couple of items into init for backward compatibility fixing keyvault authentication error in AML Fixing bug reading None value in mordor_browser * Fixing requirements so that msticpy will still install on Py3.6 Fixing tests for packages to use pkg_resource specifier parsing (which it should have always used) Updating Kqlmagic version to official release. * Adding AzCli URi to exceptions Updating black params in pre-commit and pipelines to remove -t py36 flag * removing unused warnings from import_analyzer.py * Updating to 1.7.0 * Changing magics creation so that they don't get created if not in ipython (#332) Adding import of magics to nbinit and removing from __init__ and Pivot class. Updating docs (including some auto-gen'd) * Removing un-needed config * Redacted sample credentials * Added refresh and delete functions for keyring cached secrets (#336) * Added refresh and delete functions for keyring cached secrets * Black reformatting of secret_settings * Powershell simple de-obfuscator and code viewer. (#335) * Simple code de-obfuscator and display for PowerShell * Setting default style to "default" and making display_html DisplayHandle return optional * pep257 doc string linting errors in code_cleanup and code_view * Adding pygments to requirements (this is already a dependency of other core dependencies so should have no impact on install) * Fixing test failure in test_code_view Also linting errors suppressed from bandit, prospector and pylint * Added Splunk async provider and unit_tests (#337) * Added Splunk async provider and unit_tests * Fixed incorrect property call Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixed incorrect use of httpx.codes enum in odata_driver (#345) * Fixed incorrect use of httpx.codes enum in odata_driver Added httpx timeout to cybereason_driver and http_base * Added additional context for exceptions. Formatting change for http_base.py Moving import of VTFileBehaviour out of try/except block in vtlookupv3.py * Changing default timeout for httpx client to match requests 30sec for connect, 10sec elsewhere, # Conflicts: # msticpy/data/drivers/cybereason_driver.py # msticpy/sectools/tiproviders/http_base.py * Adding timeouts to missing httpx calls * Splitting keyring into its own module so that we can load without this as a dependency # Conflicts: # msticpy/common/secret_settings.py * Needed type hint in exceptions.py * Putting IPStack APIKey check happen when first used (rather than in __init__) so it doesn't throw exception on loading * prospector config changed produces deprecation warning and non-zero exit code. * Fixing misconfigured prospector.yaml * Test fix for test_cybereason_driver copied from v2 branch * removing version restriction for prospector in Github actions python-package.yaml # Conflicts: # .github/workflows/python-package.yml * Bump readthedocs-sphinx-ext from 2.1.4 to 2.1.5 (#339) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.4 to 2.1.5. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](readthedocs/readthedocs-sphinx-ext@2.1.4...2.1.5) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updating version to 1.7.5 (#348) Suppressing FP bandit warning Adding Module-Analysis.ipynb notebook to tools * Ianhelle/mp config edit load fix 2022 03 28 (#352) * Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. Now loads with empty settings rather than throw exception. Added unit test case * Re-ordering arguments so doesn't break inheritance and cause pylint warning (from v2.0 branch) * Reverting changes to args and adding pylint suppressions * Aligning splunk_uploader params with base class * Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters. Fixed query_time widget so that you can reset time range from parameter Added additional unit test for QueryTime setter Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line. Also changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName. Updated TimeSpan class so it has more flexible constructor Added account_id as a parameter for list_aad_signins_by_account query * Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts to defaults. This can cause errors if the defaults are different to user-specified parameters. There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings. * Pebryan/2022 3 29 auth updates (#351) * new msal delegated auth option for graph * Switch to DefaultAzureCredential * renamed MSALAuth * Linting fixes * Add Unit Test and PR changes * Updates to fix tests * Fixed execption error * formatting * Merging in Splunk fixes from #352 * fixed incorrect merge * New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352. * hotfix for bug found in testing * Fixed re-auth on query issue in KQL driver * Removing un-needed code * Fixed kql_driver tests * Liniting fixes Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Bump sphinx from 4.4.0 to 4.5.0 (#350) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.4.0 to 4.5.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](sphinx-doc/sphinx@v4.4.0...v4.5.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixes for GeoLiteLookup and MpConfigEdit (#356) * Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py. Fixed bug where empty/new msticpyconfig.yaml didn't save any settings. * Reorganized logic for handling parameters and failing on invalid file path. * Some fixes to Kusto common_imports (#358) - now works with Kusto config entry without instance suffix - can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query - added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding in the data_famiies key. Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * linting fixes * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * Changing the pattern for httpx timeout to default to Timeout(None). (#378) * Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x) Other components use the default. * Adding case for reading list from yaml instead of tuple - now handles any iterable. * Added unit test and made some fixes to pkg_config.py * Bug in test test_pkg_config.py Also in test_code_view.py * Add Workflow to Tweet (#369) * Add Workflow to Tweet * Update tweet.yml * Update tweet.yml Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing warnings for malformed regexes in kql_driver, test_sentinel_search Re-enabling pytest.skip in test_nbwidgets.py * Fixed minor issues (#371) * Fixed minor issues * Fixed additional use case Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#374) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Missing import in test_nbwidgets * Forgot to add "r" prefix to strings in test_sentinel_search * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#379) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Updating version to 1.8.0 * Fix for MpConfigEdit ValueError Updating version for hotfix * MpConfig edit throws error with invalid file path. (#395) * Updating Dockerfile source to mcr anaconda * Update API version for list_alert_rules To be consistent with the documentation (https://docs.microsoft.com/en-us/rest/api/securityinsights/stable/alert-rules/list) api version should be set to "2021-10-01" * (fix) moving conda-supported files from conda-*pip* files to conda requirements * Update deprecated prospector tool names. * Updated Tweet bot to include more context in the tweets * Updated tweet action to include more detail in the tweets * Updated OData drivers to allow for Delegated auth settings to be passed when connecting. Includes the ability to use Delegated Auth as well as the method. Added documentation on how to use the feature. * Fixed linting issues in odata_driver * Updated requirement for azure-identity to 1.10.0 * Microsoft mandatory file (#407) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> * Bump readthedocs-sphinx-ext from 2.1.5 to 2.1.6 (#400) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.5 to 2.1.6. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](https://github.com/readthedocs/readthedocs-sphinx-ext/commits) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated default config file to include username for MDE and Graph. Fixed string formatting in security_graph_driver as per PR comments. * Suppressed exception logger message from msal_extensions in kql_driver.py (#411) Removed auto-load of VT Pivots - causes an exception when vt SDK is not installed - in nbinit.py Added check for null config values in user_config.py Added requirement for typing-extensions 4.2.0 (required by bokeh) Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * Updating version to 1.8.2 * Replace MSAL auth plaintext file cache with memory cache (#413) * Removed plaintext token chace from MSAL auth and replaced it will fall back to in memory caching. * Adding in catch of additional execptions in msal_auth * Removed := to retain 3.6 support in main Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Removing some files from merge errors * Merge remote-tracking branch 'origin/main' into ianhelle/merge2.0_to_main-2022-06-14 * [fix] removing deprecated pylint warnings from python-package.yml (github) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml * [fix] CodeQL fixes for incorrect regex (all but one were in test code) - odata_driver.py [fix] missing await in url_checker_async.py * [fix] Adding updated ContiLeaksAnalysis notebook * [fix] Trying different suppressions for credscan Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ashwin Patil <ashwin-patil@users.noreply.github.com> Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: FlorianBracq <97248273+FlorianBracq@users.noreply.github.com> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>

…/msticpy into release/msticpy-v2.0.0

Adding cryptograph to docs/requirements.txt

Adding intersphinx to conf.py

…2022-06-14

… 22.04

* Bump sphinx from 4.3.2 to 4.4.0 (#283) * Bump sphinx from 4.3.2 to 4.4.0 Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.3.2 to 4.4.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](sphinx-doc/sphinx@v4.3.2...v4.4.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating requirements-dev.txt to sync with dependabot updates Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * doc updates (#316) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * adding devcontainer files (#321) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Bump respx from 0.17.1 to 0.19.2 (#314) Bumps [respx](https://github.com/lundberg/respx) from 0.17.1 to 0.19.2. - [Release notes](https://github.com/lundberg/respx/releases) - [Changelog](https://github.com/lundberg/respx/blob/master/CHANGELOG.md) - [Commits](lundberg/respx@0.17.1...0.19.2) --- updated-dependencies: - dependency-name: respx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated Cybereason docs to fit pattern (#324) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Ianhelle/1.6.1 hotfixes 2022 01 27 for 1.6.2 (#317) * Importing a couple of items into init for backward compatibility fixing keyvault authentication error in AML Fixing bug reading None value in mordor_browser * Fixing requirements so that msticpy will still install on Py3.6 Fixing tests for packages to use pkg_resource specifier parsing (which it should have always used) Updating Kqlmagic version to official release. * Adding AzCli URi to exceptions Updating black params in pre-commit and pipelines to remove -t py36 flag * removing unused warnings from import_analyzer.py * Updating to 1.7.0 * Changing magics creation so that they don't get created if not in ipython (#332) Adding import of magics to nbinit and removing from __init__ and Pivot class. Updating docs (including some auto-gen'd) * Removing un-needed config * Redacted sample credentials * Added refresh and delete functions for keyring cached secrets (#336) * Added refresh and delete functions for keyring cached secrets * Black reformatting of secret_settings * Powershell simple de-obfuscator and code viewer. (#335) * Simple code de-obfuscator and display for PowerShell * Setting default style to "default" and making display_html DisplayHandle return optional * pep257 doc string linting errors in code_cleanup and code_view * Adding pygments to requirements (this is already a dependency of other core dependencies so should have no impact on install) * Fixing test failure in test_code_view Also linting errors suppressed from bandit, prospector and pylint * Added Splunk async provider and unit_tests (#337) * Added Splunk async provider and unit_tests * Fixed incorrect property call Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixed incorrect use of httpx.codes enum in odata_driver (#345) * Fixed incorrect use of httpx.codes enum in odata_driver Added httpx timeout to cybereason_driver and http_base * Added additional context for exceptions. Formatting change for http_base.py Moving import of VTFileBehaviour out of try/except block in vtlookupv3.py * Changing default timeout for httpx client to match requests 30sec for connect, 10sec elsewhere, # Conflicts: # msticpy/data/drivers/cybereason_driver.py # msticpy/sectools/tiproviders/http_base.py * Adding timeouts to missing httpx calls * Splitting keyring into its own module so that we can load without this as a dependency # Conflicts: # msticpy/common/secret_settings.py * Needed type hint in exceptions.py * Putting IPStack APIKey check happen when first used (rather than in __init__) so it doesn't throw exception on loading * prospector config changed produces deprecation warning and non-zero exit code. * Fixing misconfigured prospector.yaml * Test fix for test_cybereason_driver copied from v2 branch * removing version restriction for prospector in Github actions python-package.yaml # Conflicts: # .github/workflows/python-package.yml * Bump readthedocs-sphinx-ext from 2.1.4 to 2.1.5 (#339) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.4 to 2.1.5. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](readthedocs/readthedocs-sphinx-ext@2.1.4...2.1.5) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updating version to 1.7.5 (#348) Suppressing FP bandit warning Adding Module-Analysis.ipynb notebook to tools * Ianhelle/mp config edit load fix 2022 03 28 (#352) * Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. Now loads with empty settings rather than throw exception. Added unit test case * Re-ordering arguments so doesn't break inheritance and cause pylint warning (from v2.0 branch) * Reverting changes to args and adding pylint suppressions * Aligning splunk_uploader params with base class * Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters. Fixed query_time widget so that you can reset time range from parameter Added additional unit test for QueryTime setter Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line. Also changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName. Updated TimeSpan class so it has more flexible constructor Added account_id as a parameter for list_aad_signins_by_account query * Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts to defaults. This can cause errors if the defaults are different to user-specified parameters. There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings. * Pebryan/2022 3 29 auth updates (#351) * new msal delegated auth option for graph * Switch to DefaultAzureCredential * renamed MSALAuth * Linting fixes * Add Unit Test and PR changes * Updates to fix tests * Fixed execption error * formatting * Merging in Splunk fixes from #352 * fixed incorrect merge * New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352. * hotfix for bug found in testing * Fixed re-auth on query issue in KQL driver * Removing un-needed code * Fixed kql_driver tests * Liniting fixes Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Bump sphinx from 4.4.0 to 4.5.0 (#350) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.4.0 to 4.5.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](sphinx-doc/sphinx@v4.4.0...v4.5.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixes for GeoLiteLookup and MpConfigEdit (#356) * Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py. Fixed bug where empty/new msticpyconfig.yaml didn't save any settings. * Reorganized logic for handling parameters and failing on invalid file path. * Some fixes to Kusto common_imports (#358) - now works with Kusto config entry without instance suffix - can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query - added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding in the data_famiies key. Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * linting fixes * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * Changing the pattern for httpx timeout to default to Timeout(None). (#378) * Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x) Other components use the default. * Adding case for reading list from yaml instead of tuple - now handles any iterable. * Added unit test and made some fixes to pkg_config.py * Bug in test test_pkg_config.py Also in test_code_view.py * Add Workflow to Tweet (#369) * Add Workflow to Tweet * Update tweet.yml * Update tweet.yml Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing warnings for malformed regexes in kql_driver, test_sentinel_search Re-enabling pytest.skip in test_nbwidgets.py * Fixed minor issues (#371) * Fixed minor issues * Fixed additional use case Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#374) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Missing import in test_nbwidgets * Forgot to add "r" prefix to strings in test_sentinel_search * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#379) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Updating version to 1.8.0 * Fix for MpConfigEdit ValueError Updating version for hotfix * MpConfig edit throws error with invalid file path. (#395) * Updating Dockerfile source to mcr anaconda * Update API version for list_alert_rules To be consistent with the documentation (https://docs.microsoft.com/en-us/rest/api/securityinsights/stable/alert-rules/list) api version should be set to "2021-10-01" * (fix) moving conda-supported files from conda-*pip* files to conda requirements * Update deprecated prospector tool names. * Updated Tweet bot to include more context in the tweets * Updated tweet action to include more detail in the tweets * Updated OData drivers to allow for Delegated auth settings to be passed when connecting. Includes the ability to use Delegated Auth as well as the method. Added documentation on how to use the feature. * Fixed linting issues in odata_driver * Updated requirement for azure-identity to 1.10.0 * Microsoft mandatory file (#407) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> * Bump readthedocs-sphinx-ext from 2.1.5 to 2.1.6 (#400) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.5 to 2.1.6. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](https://github.com/readthedocs/readthedocs-sphinx-ext/commits) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated default config file to include username for MDE and Graph. Fixed string formatting in security_graph_driver as per PR comments. * Suppressed exception logger message from msal_extensions in kql_driver.py (#411) Removed auto-load of VT Pivots - causes an exception when vt SDK is not installed - in nbinit.py Added check for null config values in user_config.py Added requirement for typing-extensions 4.2.0 (required by bokeh) Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * Updating version to 1.8.2 * Replace MSAL auth plaintext file cache with memory cache (#413) * Removed plaintext token chace from MSAL auth and replaced it will fall back to in memory caching. * Adding in catch of additional execptions in msal_auth * Removed := to retain 3.6 support in main Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Fix for list_hunting_queries function Fix for list_hunting_queries function, referred to alert_rules api, which does not contain such. Rather going towards savedSearches endpoint. It could also be pointed out somewhere that this is for custom queries only, i.e. Provider="Custom Queries" * Update calls to credential.modern.get_token Tenant_id should only be used when it is defined. * Adding ContiLeaks Analysis (#428) Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * [update] Changed data_providers to only add pivots on connect(), adding instance property [update] exposing driver instance property in driver_base, cybereason_driver, kql_driver [update] Updated PivotFunctions.ipynb and PivotFunctions.rst with new behavior [update] Added new SingletonClass to types, rename previous class to SingletonArgsClass in types.py [update] Exposing workspace instance name in wsconfig.py [update] Updating geoip.py to use renamed SingletonArgsClass [update] Adding short name to multiple MDE queries [update] Added process query using only file_hash parameter for pivot query [update] Adding replaceable table parameter to kql_mdatp_user.yaml queries [update] pivots() and get_pivot_list() now supports search string and returns sorted list [update] Adding "pivot" attribute to msticpy after loading pivot [update] Added doc string to txt2df magic in nbmagics.py [update] Pivot is now a singleton, rationalized query time setting, removed adding data provider queries at load [update] Changed clipboard/function text to match usage with imported entities in pivot_browser.py [update] Added use of "explode" in list_to_rows in pivot_pd_accessor.py [update] Importing vt_pivot into pivot_core/__init__.py [update] Changed to support multiple provider instances, removed shortcut query functions, renamed some tables, pivot data queries now use central Pivot.timespan by default [update] Removed provider-specific and IPv4/v6 specific functions - huge simplification in pivot_ti_provider.py [update] Updated and rationalized Pivot tests for new behavior. add test_vt_pivot.py [fix] fixed proper reporting of pivot functions in pivot_container.py [fix] removing deprecated PyLint warning suppression from account.py and process.py [fix] popping extra ioc_type from params in ti_provider_base.py [fix] Fixing warnings for Pylint 2.14.0 - removing deprecated warning types in .pylintrc [fix] Removing duplicate syslog_utils.py (from graphs_plot branch) [fix] Pylint warning in vtlookup.py (from graphs_plot branch) [fix] Pylint warning in entity_graph_tools.py (from graphs_plot branch) [fix] Removing unsupport Pylint warning type from account.py, process.py and base64unpack.py [fix] Updating docs for removed syslog_utils.py in msticpy.transform.rst and msticpy.transform.syslog_utils.rst [fix] Incorrect heading underlining in SettingsEditor.rst * [update] Updated PivotFunctions-Introduction notebook for new behavior [update] Added references to notebooks in PivotFunctions.rst [update] auto-update to DataQueries.rst [fix] formatting error in Installing.rst [update] removing shortcut functions from VT pivots * [fix] moving pivot tests to tests/init folder * [fix] Correcting doc strings in time series functions and accessors * [fix] minor fixes in FoliumMap and PivotFunctions notebooks * Removing some files from merge errors * Merge remote-tracking branch 'origin/main' into ianhelle/merge2.0_to_main-2022-06-14 * [fix] removing deprecated pylint warnings from python-package.yml (github) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml * [fix] removing deprecated pylint warnings from python-package.yml (github) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml * [fix] CodeQL fixes for incorrect regex (all but one were in test code) - odata_driver.py [fix] missing await in url_checker_async.py * [fix] CodeQL fixes for incorrect regex (all but one were in test code) - odata_driver.py [fix] missing await in url_checker_async.py * [fix] Adding updated ContiLeaksAnalysis notebook * [fix] Adding updated ContiLeaksAnalysis notebook * [fix] Trying different suppressions for credscan * [fix] adding back intersphinx and updating RTD build to Py 3.9 Ubuntu 22.04 * [fix] typing-extensions exception added to import_analyzer.py * [fix] avoid trying to add Pivot functions if VTLookupV3 can't be initialized - in vt_pivot.py Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ashwin Patil <ashwin-patil@users.noreply.github.com> Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: FlorianBracq <97248273+FlorianBracq@users.noreply.github.com> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: pensivepaddle <104833713+pensivepaddle@users.noreply.github.com> Co-authored-by: Thomas Roccia <thomas.roccia@gmail.com>

* Bump sphinx from 4.3.2 to 4.4.0 (#283) * Bump sphinx from 4.3.2 to 4.4.0 Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.3.2 to 4.4.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](sphinx-doc/sphinx@v4.3.2...v4.4.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating requirements-dev.txt to sync with dependabot updates Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * doc updates (#316) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * adding devcontainer files (#321) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Bump respx from 0.17.1 to 0.19.2 (#314) Bumps [respx](https://github.com/lundberg/respx) from 0.17.1 to 0.19.2. - [Release notes](https://github.com/lundberg/respx/releases) - [Changelog](https://github.com/lundberg/respx/blob/master/CHANGELOG.md) - [Commits](lundberg/respx@0.17.1...0.19.2) --- updated-dependencies: - dependency-name: respx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated Cybereason docs to fit pattern (#324) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Ianhelle/1.6.1 hotfixes 2022 01 27 for 1.6.2 (#317) * Importing a couple of items into init for backward compatibility fixing keyvault authentication error in AML Fixing bug reading None value in mordor_browser * Fixing requirements so that msticpy will still install on Py3.6 Fixing tests for packages to use pkg_resource specifier parsing (which it should have always used) Updating Kqlmagic version to official release. * Adding AzCli URi to exceptions Updating black params in pre-commit and pipelines to remove -t py36 flag * removing unused warnings from import_analyzer.py * Updating to 1.7.0 * Changing magics creation so that they don't get created if not in ipython (#332) Adding import of magics to nbinit and removing from __init__ and Pivot class. Updating docs (including some auto-gen'd) * Removing un-needed config * Redacted sample credentials * Added refresh and delete functions for keyring cached secrets (#336) * Added refresh and delete functions for keyring cached secrets * Black reformatting of secret_settings * Powershell simple de-obfuscator and code viewer. (#335) * Simple code de-obfuscator and display for PowerShell * Setting default style to "default" and making display_html DisplayHandle return optional * pep257 doc string linting errors in code_cleanup and code_view * Adding pygments to requirements (this is already a dependency of other core dependencies so should have no impact on install) * Fixing test failure in test_code_view Also linting errors suppressed from bandit, prospector and pylint * Added Splunk async provider and unit_tests (#337) * Added Splunk async provider and unit_tests * Fixed incorrect property call Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixed incorrect use of httpx.codes enum in odata_driver (#345) * Fixed incorrect use of httpx.codes enum in odata_driver Added httpx timeout to cybereason_driver and http_base * Added additional context for exceptions. Formatting change for http_base.py Moving import of VTFileBehaviour out of try/except block in vtlookupv3.py * Changing default timeout for httpx client to match requests 30sec for connect, 10sec elsewhere, # Conflicts: # msticpy/data/drivers/cybereason_driver.py # msticpy/sectools/tiproviders/http_base.py * Adding timeouts to missing httpx calls * Splitting keyring into its own module so that we can load without this as a dependency # Conflicts: # msticpy/common/secret_settings.py * Needed type hint in exceptions.py * Putting IPStack APIKey check happen when first used (rather than in __init__) so it doesn't throw exception on loading * prospector config changed produces deprecation warning and non-zero exit code. * Fixing misconfigured prospector.yaml * Test fix for test_cybereason_driver copied from v2 branch * removing version restriction for prospector in Github actions python-package.yaml # Conflicts: # .github/workflows/python-package.yml * Bump readthedocs-sphinx-ext from 2.1.4 to 2.1.5 (#339) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.4 to 2.1.5. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](readthedocs/readthedocs-sphinx-ext@2.1.4...2.1.5) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updating version to 1.7.5 (#348) Suppressing FP bandit warning Adding Module-Analysis.ipynb notebook to tools * Ianhelle/mp config edit load fix 2022 03 28 (#352) * Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. Now loads with empty settings rather than throw exception. Added unit test case * Re-ordering arguments so doesn't break inheritance and cause pylint warning (from v2.0 branch) * Reverting changes to args and adding pylint suppressions * Aligning splunk_uploader params with base class * Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters. Fixed query_time widget so that you can reset time range from parameter Added additional unit test for QueryTime setter Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line. Also changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName. Updated TimeSpan class so it has more flexible constructor Added account_id as a parameter for list_aad_signins_by_account query * Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts to defaults. This can cause errors if the defaults are different to user-specified parameters. There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings. * Pebryan/2022 3 29 auth updates (#351) * new msal delegated auth option for graph * Switch to DefaultAzureCredential * renamed MSALAuth * Linting fixes * Add Unit Test and PR changes * Updates to fix tests * Fixed execption error * formatting * Merging in Splunk fixes from #352 * fixed incorrect merge * New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352. * hotfix for bug found in testing * Fixed re-auth on query issue in KQL driver * Removing un-needed code * Fixed kql_driver tests * Liniting fixes Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Bump sphinx from 4.4.0 to 4.5.0 (#350) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.4.0 to 4.5.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](sphinx-doc/sphinx@v4.4.0...v4.5.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixes for GeoLiteLookup and MpConfigEdit (#356) * Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py. Fixed bug where empty/new msticpyconfig.yaml didn't save any settings. * Reorganized logic for handling parameters and failing on invalid file path. * Some fixes to Kusto common_imports (#358) - now works with Kusto config entry without instance suffix - can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query - added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding in the data_famiies key. Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * linting fixes * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * Changing the pattern for httpx timeout to default to Timeout(None). (#378) * Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x) Other components use the default. * Adding case for reading list from yaml instead of tuple - now handles any iterable. * Added unit test and made some fixes to pkg_config.py * Bug in test test_pkg_config.py Also in test_code_view.py * Add Workflow to Tweet (#369) * Add Workflow to Tweet * Update tweet.yml * Update tweet.yml Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing warnings for malformed regexes in kql_driver, test_sentinel_search Re-enabling pytest.skip in test_nbwidgets.py * Fixed minor issues (#371) * Fixed minor issues * Fixed additional use case Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#374) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Missing import in test_nbwidgets * Forgot to add "r" prefix to strings in test_sentinel_search * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#379) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Updating version to 1.8.0 * Fix for MpConfigEdit ValueError Updating version for hotfix * MpConfig edit throws error with invalid file path. (#395) * Updating Dockerfile source to mcr anaconda * Update API version for list_alert_rules To be consistent with the documentation (https://docs.microsoft.com/en-us/rest/api/securityinsights/stable/alert-rules/list) api version should be set to "2021-10-01" * (fix) moving conda-supported files from conda-*pip* files to conda requirements * Update deprecated prospector tool names. * Updated Tweet bot to include more context in the tweets * Updated tweet action to include more detail in the tweets * Updated OData drivers to allow for Delegated auth settings to be passed when connecting. Includes the ability to use Delegated Auth as well as the method. Added documentation on how to use the feature. * Fixed linting issues in odata_driver * Updated requirement for azure-identity to 1.10.0 * Microsoft mandatory file (#407) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> * Bump readthedocs-sphinx-ext from 2.1.5 to 2.1.6 (#400) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.5 to 2.1.6. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](https://github.com/readthedocs/readthedocs-sphinx-ext/commits) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated default config file to include username for MDE and Graph. Fixed string formatting in security_graph_driver as per PR comments. * Suppressed exception logger message from msal_extensions in kql_driver.py (#411) Removed auto-load of VT Pivots - causes an exception when vt SDK is not installed - in nbinit.py Added check for null config values in user_config.py Added requirement for typing-extensions 4.2.0 (required by bokeh) Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * Updating version to 1.8.2 * Replace MSAL auth plaintext file cache with memory cache (#413) * Removed plaintext token chace from MSAL auth and replaced it will fall back to in memory caching. * Adding in catch of additional execptions in msal_auth * Removed := to retain 3.6 support in main Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Fix for list_hunting_queries function Fix for list_hunting_queries function, referred to alert_rules api, which does not contain such. Rather going towards savedSearches endpoint. It could also be pointed out somewhere that this is for custom queries only, i.e. Provider="Custom Queries" * [update] Adding folium maps documentation - Updating Folium document FoliumMap.rst [fix] Fixing error in geoip if list contains non-string elements (e.g. nans) in geoip.py [fix] Fixing doc and exception wording errors in foliummap.py * Update calls to credential.modern.get_token Tenant_id should only be used when it is defined. * Adding ContiLeaks Analysis (#428) Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * Removing some files from merge errors * [fix] removing deprecated pylint warnings from python-package.yml (github) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml * [fix] CodeQL fixes for incorrect regex (all but one were in test code) - odata_driver.py [fix] missing await in url_checker_async.py * [fix] Adding updated ContiLeaksAnalysis notebook * [fix] Trying different suppressions for credscan * [fix] adding back intersphinx and updating RTD build to Py 3.9 Ubuntu 22.04 * [fix] typing-extensions exception added to import_analyzer.py Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ashwin Patil <ashwin-patil@users.noreply.github.com> Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: FlorianBracq <97248273+FlorianBracq@users.noreply.github.com> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: pensivepaddle <104833713+pensivepaddle@users.noreply.github.com> Co-authored-by: Thomas Roccia <thomas.roccia@gmail.com>

* Bump sphinx from 4.3.2 to 4.4.0 (#283) * Bump sphinx from 4.3.2 to 4.4.0 Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.3.2 to 4.4.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](sphinx-doc/sphinx@v4.3.2...v4.4.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating requirements-dev.txt to sync with dependabot updates Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * doc updates (#316) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * adding devcontainer files (#321) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Bump respx from 0.17.1 to 0.19.2 (#314) Bumps [respx](https://github.com/lundberg/respx) from 0.17.1 to 0.19.2. - [Release notes](https://github.com/lundberg/respx/releases) - [Changelog](https://github.com/lundberg/respx/blob/master/CHANGELOG.md) - [Commits](lundberg/respx@0.17.1...0.19.2) --- updated-dependencies: - dependency-name: respx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated Cybereason docs to fit pattern (#324) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Ianhelle/1.6.1 hotfixes 2022 01 27 for 1.6.2 (#317) * Importing a couple of items into init for backward compatibility fixing keyvault authentication error in AML Fixing bug reading None value in mordor_browser * Fixing requirements so that msticpy will still install on Py3.6 Fixing tests for packages to use pkg_resource specifier parsing (which it should have always used) Updating Kqlmagic version to official release. * Adding AzCli URi to exceptions Updating black params in pre-commit and pipelines to remove -t py36 flag * removing unused warnings from import_analyzer.py * Updating to 1.7.0 * Changing magics creation so that they don't get created if not in ipython (#332) Adding import of magics to nbinit and removing from __init__ and Pivot class. Updating docs (including some auto-gen'd) * Removing un-needed config * Redacted sample credentials * Added refresh and delete functions for keyring cached secrets (#336) * Added refresh and delete functions for keyring cached secrets * Black reformatting of secret_settings * Powershell simple de-obfuscator and code viewer. (#335) * Simple code de-obfuscator and display for PowerShell * Setting default style to "default" and making display_html DisplayHandle return optional * pep257 doc string linting errors in code_cleanup and code_view * Adding pygments to requirements (this is already a dependency of other core dependencies so should have no impact on install) * Fixing test failure in test_code_view Also linting errors suppressed from bandit, prospector and pylint * Added Splunk async provider and unit_tests (#337) * Added Splunk async provider and unit_tests * Fixed incorrect property call Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixed incorrect use of httpx.codes enum in odata_driver (#345) * Fixed incorrect use of httpx.codes enum in odata_driver Added httpx timeout to cybereason_driver and http_base * Added additional context for exceptions. Formatting change for http_base.py Moving import of VTFileBehaviour out of try/except block in vtlookupv3.py * Changing default timeout for httpx client to match requests 30sec for connect, 10sec elsewhere, # Conflicts: # msticpy/data/drivers/cybereason_driver.py # msticpy/sectools/tiproviders/http_base.py * Adding timeouts to missing httpx calls * Splitting keyring into its own module so that we can load without this as a dependency # Conflicts: # msticpy/common/secret_settings.py * Needed type hint in exceptions.py * Putting IPStack APIKey check happen when first used (rather than in __init__) so it doesn't throw exception on loading * prospector config changed produces deprecation warning and non-zero exit code. * Fixing misconfigured prospector.yaml * Test fix for test_cybereason_driver copied from v2 branch * removing version restriction for prospector in Github actions python-package.yaml # Conflicts: # .github/workflows/python-package.yml * Bump readthedocs-sphinx-ext from 2.1.4 to 2.1.5 (#339) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.4 to 2.1.5. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](readthedocs/readthedocs-sphinx-ext@2.1.4...2.1.5) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updating version to 1.7.5 (#348) Suppressing FP bandit warning Adding Module-Analysis.ipynb notebook to tools * Ianhelle/mp config edit load fix 2022 03 28 (#352) * Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. Now loads with empty settings rather than throw exception. Added unit test case * Re-ordering arguments so doesn't break inheritance and cause pylint warning (from v2.0 branch) * Reverting changes to args and adding pylint suppressions * Aligning splunk_uploader params with base class * Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters. Fixed query_time widget so that you can reset time range from parameter Added additional unit test for QueryTime setter Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line. Also changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName. Updated TimeSpan class so it has more flexible constructor Added account_id as a parameter for list_aad_signins_by_account query * Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts to defaults. This can cause errors if the defaults are different to user-specified parameters. There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings. * Pebryan/2022 3 29 auth updates (#351) * new msal delegated auth option for graph * Switch to DefaultAzureCredential * renamed MSALAuth * Linting fixes * Add Unit Test and PR changes * Updates to fix tests * Fixed execption error * formatting * Merging in Splunk fixes from #352 * fixed incorrect merge * New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352. * hotfix for bug found in testing * Fixed re-auth on query issue in KQL driver * Removing un-needed code * Fixed kql_driver tests * Liniting fixes Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Bump sphinx from 4.4.0 to 4.5.0 (#350) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.4.0 to 4.5.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](sphinx-doc/sphinx@v4.4.0...v4.5.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixes for GeoLiteLookup and MpConfigEdit (#356) * Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py. Fixed bug where empty/new msticpyconfig.yaml didn't save any settings. * Reorganized logic for handling parameters and failing on invalid file path. * Some fixes to Kusto common_imports (#358) - now works with Kusto config entry without instance suffix - can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query - added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding in the data_famiies key. Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * linting fixes * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * Changing the pattern for httpx timeout to default to Timeout(None). (#378) * Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x) Other components use the default. * Adding case for reading list from yaml instead of tuple - now handles any iterable. * Added unit test and made some fixes to pkg_config.py * Bug in test test_pkg_config.py Also in test_code_view.py * Add Workflow to Tweet (#369) * Add Workflow to Tweet * Update tweet.yml * Update tweet.yml Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing warnings for malformed regexes in kql_driver, test_sentinel_search Re-enabling pytest.skip in test_nbwidgets.py * Fixed minor issues (#371) * Fixed minor issues * Fixed additional use case Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#374) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Missing import in test_nbwidgets * Forgot to add "r" prefix to strings in test_sentinel_search * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#379) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Updating version to 1.8.0 * Fix for MpConfigEdit ValueError Updating version for hotfix * MpConfig edit throws error with invalid file path. (#395) * Updating Dockerfile source to mcr anaconda * Update API version for list_alert_rules To be consistent with the documentation (https://docs.microsoft.com/en-us/rest/api/securityinsights/stable/alert-rules/list) api version should be set to "2021-10-01" * (fix) moving conda-supported files from conda-*pip* files to conda requirements * Update deprecated prospector tool names. * Updated Tweet bot to include more context in the tweets * Updated tweet action to include more detail in the tweets * Updated OData drivers to allow for Delegated auth settings to be passed when connecting. Includes the ability to use Delegated Auth as well as the method. Added documentation on how to use the feature. * Fixed linting issues in odata_driver * Updated requirement for azure-identity to 1.10.0 * Microsoft mandatory file (#407) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> * Bump readthedocs-sphinx-ext from 2.1.5 to 2.1.6 (#400) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.5 to 2.1.6. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](https://github.com/readthedocs/readthedocs-sphinx-ext/commits) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated default config file to include username for MDE and Graph. Fixed string formatting in security_graph_driver as per PR comments. * Suppressed exception logger message from msal_extensions in kql_driver.py (#411) Removed auto-load of VT Pivots - causes an exception when vt SDK is not installed - in nbinit.py Added check for null config values in user_config.py Added requirement for typing-extensions 4.2.0 (required by bokeh) Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * Updating version to 1.8.2 * Replace MSAL auth plaintext file cache with memory cache (#413) * Removed plaintext token chace from MSAL auth and replaced it will fall back to in memory caching. * Adding in catch of additional execptions in msal_auth * Removed := to retain 3.6 support in main Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Fix for list_hunting_queries function Fix for list_hunting_queries function, referred to alert_rules api, which does not contain such. Rather going towards savedSearches endpoint. It could also be pointed out somewhere that this is for custom queries only, i.e. Provider="Custom Queries" * Update calls to credential.modern.get_token Tenant_id should only be used when it is defined. * Adding ContiLeaks Analysis (#428) Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * [update] Changed data_providers to only add pivots on connect(), adding instance property [update] exposing driver instance property in driver_base, cybereason_driver, kql_driver [update] Updated PivotFunctions.ipynb and PivotFunctions.rst with new behavior [update] Added new SingletonClass to types, rename previous class to SingletonArgsClass in types.py [update] Exposing workspace instance name in wsconfig.py [update] Updating geoip.py to use renamed SingletonArgsClass [update] Adding short name to multiple MDE queries [update] Added process query using only file_hash parameter for pivot query [update] Adding replaceable table parameter to kql_mdatp_user.yaml queries [update] pivots() and get_pivot_list() now supports search string and returns sorted list [update] Adding "pivot" attribute to msticpy after loading pivot [update] Added doc string to txt2df magic in nbmagics.py [update] Pivot is now a singleton, rationalized query time setting, removed adding data provider queries at load [update] Changed clipboard/function text to match usage with imported entities in pivot_browser.py [update] Added use of "explode" in list_to_rows in pivot_pd_accessor.py [update] Importing vt_pivot into pivot_core/__init__.py [update] Changed to support multiple provider instances, removed shortcut query functions, renamed some tables, pivot data queries now use central Pivot.timespan by default [update] Removed provider-specific and IPv4/v6 specific functions - huge simplification in pivot_ti_provider.py [update] Updated and rationalized Pivot tests for new behavior. add test_vt_pivot.py [fix] fixed proper reporting of pivot functions in pivot_container.py [fix] removing deprecated PyLint warning suppression from account.py and process.py [fix] popping extra ioc_type from params in ti_provider_base.py [fix] Fixing warnings for Pylint 2.14.0 - removing deprecated warning types in .pylintrc [fix] Removing duplicate syslog_utils.py (from graphs_plot branch) [fix] Pylint warning in vtlookup.py (from graphs_plot branch) [fix] Pylint warning in entity_graph_tools.py (from graphs_plot branch) [fix] Removing unsupport Pylint warning type from account.py, process.py and base64unpack.py [fix] Updating docs for removed syslog_utils.py in msticpy.transform.rst and msticpy.transform.syslog_utils.rst [fix] Incorrect heading underlining in SettingsEditor.rst * [update] Updated PivotFunctions-Introduction notebook for new behavior [update] Added references to notebooks in PivotFunctions.rst [update] auto-update to DataQueries.rst [fix] formatting error in Installing.rst [update] removing shortcut functions from VT pivots * [fix] moving pivot tests to tests/init folder * [fix] Correcting doc strings in time series functions and accessors * [fix] minor fixes in FoliumMap and PivotFunctions notebooks * [update] What's New in MSTICPy 2.0 notebook [update] Timeline - refactored timeline and timeline_values into separate modules: timeline.py, timeline_values.py and timeline_common.py - implemented PlotParams parameter handline for timeline and timeline_values [update] Added deprecation warnings to old PD accessors [update] Added data masking method - mask - to mp_pandas_accessors.py plus a few miscellaneous linting and sourcery fixes. [update] new API docs generated [fix] replace references to mp_timeline accessor with mp_plot in notebooks and RST files * Removing some files from merge errors * Merge remote-tracking branch 'origin/main' into ianhelle/merge2.0_to_main-2022-06-14 * [fix] removing deprecated pylint warnings from python-package.yml (github) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml * [fix] removing deprecated pylint warnings from python-package.yml (github) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml * [fix] CodeQL fixes for incorrect regex (all but one were in test code) - odata_driver.py [fix] missing await in url_checker_async.py * [fix] CodeQL fixes for incorrect regex (all but one were in test code) - odata_driver.py [fix] missing await in url_checker_async.py * [fix] Adding updated ContiLeaksAnalysis notebook * [fix] Adding updated ContiLeaksAnalysis notebook * [fix] Trying different suppressions for credscan * [fix] adding back intersphinx and updating RTD build to Py 3.9 Ubuntu 22.04 * [fix] typing-extensions exception added to import_analyzer.py * [fix] avoid trying to add Pivot functions if VTLookupV3 can't be initialized - in vt_pivot.py Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ashwin Patil <ashwin-patil@users.noreply.github.com> Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: FlorianBracq <97248273+FlorianBracq@users.noreply.github.com> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: pensivepaddle <104833713+pensivepaddle@users.noreply.github.com> Co-authored-by: Thomas Roccia <thomas.roccia@gmail.com>

…helle/merge2.0_to_main-2022-06-14

petebryan and others added 30 commits February 15, 2022 20:24

Adding triggers for release branches

0f072f7

Minor quality of life improvements to AzureData & MicrosoftSentinel (#…

f7923e5

…331) * Updated names in AzureData to match MicrosoftSentinel Moved list_sentinel_workspaces to AzureData Added connection checks to Sentinel features * Added linting suppression for Mixin errors

Sync changes to main into v2 branch (#330)

3f3140a

* adding devcontainer files (#321) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> # Conflicts: # .devcontainer/devcontainer.json * Syncing recent IPython-related changes (to skip magic creation if not in IPython)

New MSAL delegated auth methods added

f3e82d9

and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352.

moved list_sentinel_workspaces to AzureData

aa14784

Pebryan/2022 4 14 auth merge (#368)

380a005

* hotfix for bug found in testing * Fixed re-auth on query issue in KQL driver * Removing un-needed code * Fixed kql_driver tests * Liniting fixes Co-authored-by: Pete Bryan <pebryan@microsoft.com>

Fixed minor issues (#372)

2f90ab8

* Fixed minor issues * Fixed additional use case

Async TI lookup for lookup_iocs.

ab3464e

Refactored a bit of code around tilookup and ti_provider_base.py. Added ability to supply "providers" parameter from pivot TI functions. Some fixes from incomplete merge of Sentinel search functions and documentation

Added pd accessor for time series functions.

f26fc61

Lost the sentinel_search module in the merge

9507b0a

Making TI Providers load dyamically

fb71833

Fix to tests for two cases caused by setup config changes - test_item…

577850d

…_editors and mpconfig_defaults.yaml

Bad name in timeseries functions. analysis/timeseries.py

e27dc6c

Some mypy errors

4d7acb7

Merge branch 'ianhelle/TIAsyncLookup-2022-04-27' into ianhelle/Folium…

b530bf8

…Accessor-2022-04-30 # Conflicts: # msticpy/context/tiproviders/__init__.py # msticpy/context/tiproviders/ti_provider_base.py

Changing pytest fixture scope in some tests to prevent locking of mp_…

e4a7fa4

…config Fix to tor_exit_nodes.py - Tor no longer supplying list Changed Tor test to avoid online access - test_tiproviders.py

Working async tilookup.py

5dab3f3

Some refactoring of kql_base.py to simplify code. Removed useless pylint suppresses from msticpy.context subpackage. Refactored test_tiprovider_kql.py Set mypy.ini to use Python 3.8 rules Removed Py 3.6, 3.7 from setup.cfg

Updating mypy.ini to py 3.8

b3d831d

Removed bogus sentinel_search from merged changed

91a6a09

ianhelle added 3 commits June 14, 2022 18:36

Merge remote-tracking branch 'origin/main' into ianhelle/merge2.0_to_…

ce6d845

…main-2022-06-14

[fix] removing deprecated pylint warnings from python-package.yml (gi…

cf08046

…thub) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml

[fix] CodeQL fixes for incorrect regex (all but one were in test code…

40049b4

…) - odata_driver.py [fix] missing await in url_checker_async.py

ianhelle mentioned this pull request Jun 15, 2022

Adding ContiLeaks Analysis (#428) #442

Closed

ianhelle and others added 23 commits June 15, 2022 15:03

[fix] Adding updated ContiLeaksAnalysis notebook

f0494e0

[fix] Trying different suppressions for credscan

a74f264

Changing some params in .readthedocs.yaml

31b1598

Merge branch 'release/msticpy-v2.0.0' of https://github.com/microsoft…

ef1d66b

…/msticpy into release/msticpy-v2.0.0

Adding jinja<3.1.0 to docs/requirements.txt

458a98f

Removing msticpy requirements from .readthedocs.yaml

7f6ac61

Adding cryptograph to docs/requirements.txt

remove path: . from .readthedocs.yaml

423f446

Removing install key from .readthedocs.yaml

8afbbf1

Updating docs/requirements.txt

2e8e64a

Adding intersphinx to conf.py

Documentation updates to sphinx files

7a7512c

[fix] Revert to Py 3.7 build with typing-extensions (#448)

25e91ef

[fix] Adding updated sphinx packages to requirements.txt

bf1885b

[fix] wrong path in .readthedocs.yaml

ada5124

Merge branch 'release/msticpy-v2.0.0' into ianhelle/merge2.0_to_main-…

c5f6e57

…2022-06-14

Update RTD Python and Linux versions

4d59db1

[fix] adding back intersphinx and updating RTD build to Py 3.9 Ubuntu…

2afe34b

… 22.04

[fix] typing-extensions exception added to import_analyzer.py

1a425c5

[fix] if AuthKey or ApiID is None (#449)

efa9976

Merge remote-tracking branch 'origin/release/msticpy-v2.0.0' into ian…

7525b52

…helle/merge2.0_to_main-2022-06-14

ianhelle merged commit 633d451 into main Jun 18, 2022

ianhelle deleted the ianhelle/merge2.0_to_main-2022-06-14 branch June 18, 2022 00:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ianhelle/merge2.0 to main 2022 06 14 #443

Ianhelle/merge2.0 to main 2022 06 14 #443

ianhelle commented Jun 15, 2022

review-notebook-app bot commented Jun 15, 2022

Ianhelle/merge2.0 to main 2022 06 14 #443

Ianhelle/merge2.0 to main 2022 06 14 #443

Conversation

ianhelle commented Jun 15, 2022

review-notebook-app bot commented Jun 15, 2022