Ianhelle/geoip init fix 2022 05 27 #423
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…in geoip.py Correcting docstring in foliummap.py
…oip classes. Fixing check for unset path in geoip.py Update unit tests in test_pkg_config.py for new initialization of geoip classes. Removing duplicated test_pkg_config.py from tests/config Moving test_wsconfig.py back to tests\common Changing test_sentinel_core.py to avoid actual authentication attempt (copied from ianhelle/sentinel-workspace-lookup-2022-05-19)
Adding httpx to Sphinx requirements.txt
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
ianhelle
added a commit
that referenced
this pull request
Jun 18, 2022
* Pebryan/2022 1 25 restructure (#325) * move query files * Modules moves and import updates * Stub files and restructure fixes * More stubs and Pivots restructure * Updated _init__ files * Merging in updates to Azure auth * Linting fixes * Fixed circular import and test imports * updated failing test * Test fix * Test fix * Making tests more resilient for multiple environments * removing accidental additional parent in cmd_line default path * Added additional stub files * added exports for back compatibility * re-adding httpx changes lost in merge * restructure cyberreason and splunk queries * Fixed incorrect vtlookup * re-adding vt-graph-api fix * Adding back in query regex * Added missing httpx update * Fixed broken test * Fixing incorrect import in test * Updating missed Conda version for respx * Updating API docs * Updated cybereason folder names * Fixed Sentinel APIs * Updated test mocked data to match new API * Renamed data.context_providers to context Renamed analysis.data_processing to data Renamed data.common to data.core Removed some un-needed redirection files Moved some of the context modules (geoip, ip_utils, domain_utils) to data/context folder Added docstrings to redirection files so that they link to right location in read-the-docs Updated docs with new paths Updated notebooks with new module paths Changed RTD to generate a page for each module. Add text to deprecation warning that we'll remove in v2.0.0 * Adding some additional path fixes for tests and linting errors * Added missing changes to test_cybereason_driver Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Adding triggers for release branches * Minor quality of life improvements to AzureData & MicrosoftSentinel (#331) * Updated names in AzureData to match MicrosoftSentinel Moved list_sentinel_workspaces to AzureData Added connection checks to Sentinel features * Added linting suppression for Mixin errors * Sync changes to main into v2 branch (#330) * adding devcontainer files (#321) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> # Conflicts: # .devcontainer/devcontainer.json * Syncing recent IPython-related changes (to skip magic creation if not in IPython) * Ianhelle/msticpy v2.0.0 merge updates 2022 03 14 (#338) * Keyring refresh changes * Powershell viewer PR * Elastic driver skeleton and changes to allow driver-specific param substitution * Synced updates to nbmagics.py * Cleaning up some import redirections to point to new locations Fixing circular import in vtfile_behavior Adding placeholder class to allow imports to sort of work even if sub-modules fail to import Refactored _value_or_default in query_source to reduce complexity Fixed import errors in elastic_driver.py and splunk_driver.py Fixed import from old location in nbinit Fixing warning in code_view.py Fixed test failure in test_code_view Changing test_timeline.py to use new mp_plot accessor in place of deprecated one. Fixing test failure in test_timeline.py * Supressing bandit false positives * Ianhelle/implement isort 2022 02 15 (#327) * move query files * Modules moves and import updates * Stub files and restructure fixes * More stubs and Pivots restructure * Updated _init__ files * Merging in updates to Azure auth * Linting fixes * Fixed circular import and test imports * updated failing test * Test fix * Test fix * Making tests more resilient for multiple environments * removing accidental additional parent in cmd_line default path * Added additional stub files * added exports for back compatibility * re-adding httpx changes lost in merge * restructure cyberreason and splunk queries * Fixed incorrect vtlookup * re-adding vt-graph-api fix * Adding back in query regex * Added missing httpx update * Fixed broken test * Fixing incorrect import in test * Updating missed Conda version for respx * Updating API docs * Updated cybereason folder names * Fixed Sentinel APIs * Updated test mocked data to match new API * Renamed data.context_providers to context Renamed analysis.data_processing to data Renamed data.common to data.core Removed some un-needed redirection files Moved some of the context modules (geoip, ip_utils, domain_utils) to data/context folder Added docstrings to redirection files so that they link to right location in read-the-docs Updated docs with new paths Updated notebooks with new module paths Changed RTD to generate a page for each module. Add text to deprecation warning that we'll remove in v2.0.0 * Adding isort to pre-commit.yaml Isorting all of the imports in msticpy, tests and tools Renaming query folders Removing some automatic imports from msticpy.__init__.py Adding search function to find modules to utility.py Fixing old paths in test_timeline.py * Adding triggers for release branches Cherry picked last two commits to petebryan/2020-1-25 * Adding isort to requirements-dev and conda-reqs-dev.txt * Bandit FPs in anomaly sequence modules * Fixed failing clustering notebook * Errors in notebook and keyvault tests Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * Ianhelle/implement isort branch post-fixes 2022 03 21 (#346) * move query files * Modules moves and import updates * Stub files and restructure fixes * More stubs and Pivots restructure * Updated _init__ files * Merging in updates to Azure auth * Linting fixes * Fixed circular import and test imports * updated failing test * Test fix * Test fix * Making tests more resilient for multiple environments * removing accidental additional parent in cmd_line default path * Added additional stub files * added exports for back compatibility * re-adding httpx changes lost in merge * restructure cyberreason and splunk queries * Fixed incorrect vtlookup * re-adding vt-graph-api fix * Adding back in query regex * Added missing httpx update * Fixed broken test * Fixing incorrect import in test * Updating missed Conda version for respx * Updating API docs * Updated cybereason folder names * Fixed Sentinel APIs * Updated test mocked data to match new API * Renamed data.context_providers to context Renamed analysis.data_processing to data Renamed data.common to data.core Removed some un-needed redirection files Moved some of the context modules (geoip, ip_utils, domain_utils) to data/context folder Added docstrings to redirection files so that they link to right location in read-the-docs Updated docs with new paths Updated notebooks with new module paths Changed RTD to generate a page for each module. Add text to deprecation warning that we'll remove in v2.0.0 * Adding isort to pre-commit.yaml Isorting all of the imports in msticpy, tests and tools Renaming query folders Removing some automatic imports from msticpy.__init__.py Adding search function to find modules to utility.py Fixing old paths in test_timeline.py * Adding triggers for release branches Cherry picked last two commits to petebryan/2020-1-25 * Adding isort to requirements-dev and conda-reqs-dev.txt * Bandit FPs in anomaly sequence modules * Fixed failing clustering notebook * Errors in notebook and keyvault tests * Fixing test issues in MicrosoftDefender.ipynb and EventClustering.ipynb Bug using wrong httpx code syntax in odata_driver.py Removing auto-load of VTLookup in nbinit Forcing notebook tests to use test msticpyconfig-test.yaml Added missing __init__.py to tests/data/browsers * Removing vtlookup import from sectools init because of circular import error Moving IPStack check for API key to first call (rather than __init__) to avoid error on load. * prospector config changed produces deprecation warning and non-zero exit code. * Fixing misconfigured prospector.yaml * Getting rid of warning from test_nbinit Adding McCabe suppression to ip_utils.py * removing version restriction for prospector in Github actions python-package.yaml Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * Ianhelle/pivot dataprov selfload 2022 03 15 (#343) * Query providers load pivots dynamically when created. Renamed query paths and changed data_providers so that only env-specific queries are loaded. Moving ensure_df_datetimes to common/data_utils.py to avoid circular imports Consolidated data-related pandas accessors into single module. * Fixing circular dependency in iocextract Fixing linting errors in data_providers.py, azure_resource.py, host.py, process.py, pivot_data_queries.py * Adding default timeout values to httpx calls. Changing tor_exit_nodes.py Tor provider to defer download of tor list until first lookup Fixing test for trigger Tor node download before running test. * Re-ordering arguments so doesn't break inheritance and cause pylint warning * Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. (from main branch) Now loads with empty settings rather than throw exception. Added unit test case * Reverting changes to args and adding pylint suppressions # Conflicts: # msticpy/data/uploaders/splunk_uploader.py * Aligning splunk_uploader params with base class * New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352. * moved list_sentinel_workspaces to AzureData * Ianhelle/main mergeback 2022 04 05 (#355) * Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters. Fixed query_time widget so that you can reset time range from parameter Added additional unit test for QueryTime setter Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line. Also changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName. Updated TimeSpan class so it has more flexible constructor Added account_id as a parameter for list_aad_signins_by_account query * Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts to defaults. This can cause errors if the defaults are different to user-specified parameters. There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings. * MyPy warning in sentinel_core.py * Merging changes from main for geoip.py, mp_config_edit, mp_config_file, pkg_config and kusto_driver (#359) Some fixes to Kusto common_imports - now works with Kusto config entry without instance suffix - can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query - added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding in the data_famiies key. Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py. Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings. Reorganized logic for handling parameters and failing on invalid file path in config module. * Pebryan/2022 4 14 auth merge (#368) * hotfix for bug found in testing * Fixed re-auth on query issue in KQL driver * Removing un-needed code * Fixed kql_driver tests * Liniting fixes Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Fixed minor issues (#372) * Fixed minor issues * Fixed additional use case * Ianhelle/v2 reorg directories 2 2022 04 12 (#377) * Merging changes from main for geoip.py, mp_config_edit, mp_config_file, pkg_config and kusto_driver Some fixes to Kusto common_imports - now works with Kusto config entry without instance suffix - can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query - added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding in the data_famiies key. Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py. Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings. Reorganized logic for handling parameters and failing on invalid file path in config module. * Moving analysis.data to transform folder * Moving data.context to context * Merging changes from main * Move auth and secrets modules to auth folder * Move nbwidgets to new folder Update deprecation warning to v2.0 * Fix to URLs in README.md * Adding init folder - moved: - nbinit.py, user_config.py, pivot.py, azure_ml_tools.py to here - also moved vt_pivot.py and pivot_ti_provider.py to init/pivot_init Renaming datamodel/pivots to datamodel/pivot Moved azure_blob_storage.py to data/storage folder Refactored query_container to data_types.py - to be separate types for queries (query_container.py) and pivots (pivot_container.py) Moved browsers to vis folder * Updating API docs * Final documentation and test fixes * Moved all pivot functions to init folder. Added functionality to pkg_config to delete and translate settings (for AzureSentinel->MSSentinel switch TBD) Added automatic acquisition of globals() in nbinit.py Fixed a couple of bugs in pivot_pipeline.py Removed direct import of pivot into datamodel/pivot and added code to add them dynamically after init.pivot initiialization. Added trap to timeline when supplied with no data. Fixed incorrect escaping in regex in kql_driver.py Notebook updates for errors and invalid links. Added script to run all notebooks for testing Updated API docs * Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x) Other components use the default. # Conflicts: # README.md # docs/source/getting_started/JupyterAndAzureSentinel.rst # msticpy/common/pkg_config.py # msticpy/context/azure/sentinel_analytics.py # msticpy/context/azure/sentinel_bookmarks.py # msticpy/context/azure/sentinel_incidents.py # msticpy/context/azure/sentinel_utils.py # msticpy/context/azure/sentinel_watchlists.py # msticpy/context/tiproviders/http_base.py # msticpy/data/drivers/cybereason_driver.py # msticpy/data/drivers/driver_base.py # msticpy/data/drivers/mordor_driver.py # msticpy/data/drivers/odata_driver.py # msticpy/data/uploaders/loganalytics_uploader.py # msticpy/sectools/domain_utils.py # msticpy/sectools/geoip.py # msticpy/sectools/vtlookup.py * Updated typing rigor for pkg_config::get_http_timeout Fixed bug in test test_pkg_config.py Also fixed bug test_code_view.py * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default # Conflicts: # tests/data/azure/test_sentinel_search.py * Async TI lookup for lookup_iocs. Refactored a bit of code around tilookup and ti_provider_base.py. Added ability to supply "providers" parameter from pivot TI functions. Some fixes from incomplete merge of Sentinel search functions and documentation * Added pd accessor for time series functions. * Lost the sentinel_search module in the merge * Added new folium plot_map function to foliummap.py Added new add_ips method to take an iterable of IP addresses. Updated all add methods to accept a "layer" parameter Allow IP entities and IP addresses with locations to be supplied (will use GeoLiteLookup) Removed a bit of unused code from geoip.py and avoid looking up non-Public IPs * Making TI Providers load dyamically * Fix to tests for two cases caused by setup config changes - test_item_editors and mpconfig_defaults.yaml * Bad name in timeseries functions. analysis/timeseries.py * TIproviders are now imported dynamically based on config settings. Refactoring http_provider.py to allow extending for non-TI purposes. Added http_lookup as generic API caller based on TI pattern. Made preprocess_observable extensible. Fixed some bugs in enable/disable providers in tilookup.py restored OPR lookup_iocs (it was not being called after addition of async). Updated test_tiproviders.py to more maintainable format. Updated test_tiprovider_kql.py to pytest Updated test_ip_utils.py to pytest Fixed/suppressed a bunch of test warnings. * Some mypy errors * Changing pytest fixture scope in some tests to prevent locking of mp_config Fix to tor_exit_nodes.py - Tor no longer supplying list Changed Tor test to avoid online access - test_tiproviders.py * Working async tilookup.py Some refactoring of kql_base.py to simplify code. Removed useless pylint suppresses from msticpy.context subpackage. Refactored test_tiprovider_kql.py Set mypy.ini to use Python 3.8 rules Removed Py 3.6, 3.7 from setup.cfg * Updating mypy.ini to py 3.8 * Removed bogus sentinel_search from merged changed * Removing erroneous .values attrib for ndarray * Removing erroneous .values attrib for ndarray Fixing utils tests * Failed test due to generating legit domain names. * Fixing broken test in test_tiproviders.py * Ianhelle/mpconfigedit fix from main 2022 05 22 (#396) * Fix for MpConfigEdit ValueError Updating version for hotfix # Conflicts: # msticpy/_version.py * MpConfig edit throws error with invalid file path. * Updating Dockerfile source to mcr anaconda * Updated propspector tool names due to deprecation. * Removed un-used import from ip_utils. * Updated OData Drivers to support delegated auth. Includes the ability to set auth options. Inlcudes documentaiton updates. * Updated azure-identity requirement to 1.10.0 * Updated tweet action to include more context in the tweets (#406) Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Add Device Code fallback option for when interactive auth isn't avaliable. (#401) * Added function to azure_auth to fallback to device code auth if needed In addition updated KQL driver and AzureSentinel to use this fallback of needed. This supports cases where interactive auth not avaliable i.e. AML. * Updated prospsector tool naming to new formats Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Suppressed exception logger message from msal_extensions in kql_driver.py Removed auto-load of VT Pivots - causes an exception when vt SDK is not installed - in nbinit.py Added check for null config values in user_config.py Added requirement for typing-extensions 4.2.0 (required by bokeh) Moved conda-sourced packages to main conda requirements files. * Typo in comment in kql_driver * Added Username fields to default config for MDE and Graph. Updated formatting in security_graph_driver as per PR comments. * Spurious bandit SQL injection warning suppressed in azure_ml_tools * Temp commit for working notebook * Removed plaintext token chace from MSAL auth and replaced it with (#414) fall back to in memory caching Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Ianhelle/kql nbinit fixes merge2.0 2022 05 18 (#412) * Suppressed exception logger message from msal_extensions in kql_driver.py Removed auto-load of VT Pivots - causes an exception when vt SDK is not installed - in nbinit.py Added check for null config values in user_config.py Added requirement for typing-extensions 4.2.0 (required by bokeh) Moved conda-sourced packages to main conda requirements files. * Typo in comment in kql_driver * Spurious bandit SQL injection warning suppressed in azure_ml_tools * Fixing incorrect version of azure-identity in conda-reqs.txt * Removing redundant packages from conda-reqs-pip.txt Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * Sentinel workspaces mixin class added to MicrosoftSentinel - sentinel_workspaces.py and sentinel_core.py Added workspace lookup functionality to MpConfigEdit and MpConfigFile - mp_config_file.py and ce_azure_sentinel.py Added documentation in SentinelWorkspaces.rst and SettingsEditor.rst Add Resource Graph queries for Sentinel in sentinel_resources.yaml Moved AML-specific code from nbinit.py to azure_ml_tools.py - changed default search paths - for msticpyconfig to start from "." instead of ".." - for config.json to start from aml user folder Unit test in test_sentinel_workspaces.py Moved azure and sentinel unit tests to tests\context\azure Moved test_azure_blob_storage.py to tests\data\storage Moved test_azuredata.py to tests\context\azure Changed test_azuresent_connect_fail to use Mocks - otherwise tries live connect with a *really* long timeout Added a couple of other unit tests to test_sentinel_core.py Fixed logic in OPR test in test_tiproviders.py Fixed test logic error in test_nbinit.py::test_check_config Regenerated API docs causing a few unrelated changes. * Update version to 2.0.0-pre2 * Removing WorkspaceId.ipynb test notebook * Ianhelle/geoip init fix 2022 05 27 (#421) * Removing get_provider_settings from initialization of geo ip classes in geoip.py Correcting docstring in foliummap.py * Fixing error in GeoIPLookups notebook due to new initialization of geoip classes. Fixing check for unset path in geoip.py Update unit tests in test_pkg_config.py for new initialization of geoip classes. Removing duplicated test_pkg_config.py from tests/config Moving test_wsconfig.py back to tests\common Changing test_sentinel_core.py to avoid actual authentication attempt (copied from ianhelle/sentinel-workspace-lookup-2022-05-19) * Ianhelle/geoip init fix 2022 05 27 (#422) * Removing get_provider_settings from initialization of geo ip classes in geoip.py Correcting docstring in foliummap.py * Fixing error in GeoIPLookups notebook due to new initialization of geoip classes. Fixing check for unset path in geoip.py Update unit tests in test_pkg_config.py for new initialization of geoip classes. Removing duplicated test_pkg_config.py from tests/config Moving test_wsconfig.py back to tests\common Changing test_sentinel_core.py to avoid actual authentication attempt (copied from ianhelle/sentinel-workspace-lookup-2022-05-19) * Documentation fixes for V2.0.0 * Ianhelle/geoip init fix 2022 05 27 (#423) * Removing get_provider_settings from initialization of geo ip classes in geoip.py Correcting docstring in foliummap.py * Fixing error in GeoIPLookups notebook due to new initialization of geoip classes. Fixing check for unset path in geoip.py Update unit tests in test_pkg_config.py for new initialization of geoip classes. Removing duplicated test_pkg_config.py from tests/config Moving test_wsconfig.py back to tests\common Changing test_sentinel_core.py to avoid actual authentication attempt (copied from ianhelle/sentinel-workspace-lookup-2022-05-19) * Documentation fixes for V2.0.0 * Updating Sphinx conf.py to add more mocked packages Adding httpx to Sphinx requirements.txt * Fixing typo in sphinx requirements https => httpx * Removing msticpy from sphinx requirements.txt * Fixing Sphinx imports and mocks in docs/requirements.txt, and conf.py (#424) Removing unneeded test package respx from requirements.txt and requirements-all.txt * Removing Sphinx version constrain in docs/requirements. * Removing respx from conda package requirements. * Networkx graphs from dataframe (#427) * DataFrame to network graph and plot * Added test_network.py unit test for transform/nework * Updating Observation class with new properties in observationlist.py * Fixed attribute - when it is a list, is converted to string representation - in network.py De-duplicating node attributes when same attrib is specified for source and target in network_plot.py * [fix] Minor fixes from testing for network.py and network_plot.py [update] Unit tests for network.py and network_plot.py - test_network.py and test_network_plot.py * [update] Added user-supplied layout for network_plot.py * [fix] for kwargs of network_plot.py added unit test for layouts * [fix] pylint and prospector errors in mp_pandas_plot [fix] adding docstring explictly to mp_pandas_plot.py and timeseries.py [fix] pylint warning in entity_graph_tools.py [fix] test broken in test_observationlist.py [fix] pylint warning in vtlookup.py * [fix] Pylint, mypy fixes for observationlist.py, network_plot.py, test_observationlist.py [fix] Added additional import libs to skip for mypy [fix] Removed duplicate syslog_utils.py from transform folder [fix] Removed duplicate code from nbdisplay.py (now in network_plot.py [todo] Consolidate entity graph in network_plot.py and entity_graph_tools.py [update] Adding API docs for changes * [fix] Fixing warnings for Pylint 2.14.0 - removing deprecated warning types in .pylintrc [fix] Removing unsupport Pylint warning type from account.py, process.py and base64unpack.py * Ianhelle/msticpy __init__ imports and Quickstart doc (#435) * [update] Dynamic imports for msticpy __init__.py [update] Added auto imports of entities in init_notebook Added test case [fix] Fix to output in plain text when not in notebook in geoip * [fix] corrected module name in __init__.py [fix] corrected potential None assignment to text widgets in mp_config_file * [update] Added QuickStart brief into to MSTICPy [update] Updated JupyterAndAzureSentinel to remove unnecessary details (covered elsewhere) and bring up to date. [fix] Corrected a few things in Installing.rst, PackageSummary.rst, GeoIPLookups.rst, Visualization.rst and SettingsEditor.rst [fix] Updated index pages GettingStarted.rst. [fix] Fixed bug of duplicate parameter [fix] Fixing wording and examples in docstring in __init__.py [update] Clarifying docstring for connect function. Adding "workspace" parameter. * [fix] Fixing warnings for Pylint 2.14.0 - removing deprecated warning types in .pylintrc [fix] Removing duplicate syslog_utils.py (from graphs_plot branch) [fix] Pylint warning in vtlookup.py (from graphs_plot branch) [fix] Pylint warning in entity_graph_tools.py (from graphs_plot branch) [fix] Removing unsupport Pylint warning type from account.py, process.py and base64unpack.py [fix] Updating docs for removed syslog_utils.py in msticpy.transform.rst and msticpy.transform.syslog_utils.rst * Updating docs\requirements.txt Adding typing-extensions since RTD builds using Python 3.7, where typing.Literal is not available. * Create .readthedocs.yaml Need to force Python 3.8+ because RTD default is Python 3.7, which doesn't understand typing.Literal and some other 3.8+ syntax * Delete misplace readthedocs.yaml * Update .readthedocs.yaml Updating Python version and switching to new RTD yaml format * [fix] Updating readthedocs yaml and docs/requirements.txt * Removing some files from merge errors * [fix] removing deprecated pylint warnings from python-package.yml (github) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml * [fix] CodeQL fixes for incorrect regex (all but one were in test code) - odata_driver.py [fix] missing await in url_checker_async.py * [fix] Adding updated ContiLeaksAnalysis notebook * [fix] Trying different suppressions for credscan * Ianhelle/main updates to msticpy v2.0.0 2022 06 14 (#444) * Bump sphinx from 4.3.2 to 4.4.0 (#283) * Bump sphinx from 4.3.2 to 4.4.0 Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.3.2 to 4.4.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.3.2...v4.4.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating requirements-dev.txt to sync with dependabot updates Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * doc updates (#316) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * adding devcontainer files (#321) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Bump respx from 0.17.1 to 0.19.2 (#314) Bumps [respx](https://github.com/lundberg/respx) from 0.17.1 to 0.19.2. - [Release notes](https://github.com/lundberg/respx/releases) - [Changelog](https://github.com/lundberg/respx/blob/master/CHANGELOG.md) - [Commits](https://github.com/lundberg/respx/compare/0.17.1...0.19.2) --- updated-dependencies: - dependency-name: respx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated Cybereason docs to fit pattern (#324) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Ianhelle/1.6.1 hotfixes 2022 01 27 for 1.6.2 (#317) * Importing a couple of items into init for backward compatibility fixing keyvault authentication error in AML Fixing bug reading None value in mordor_browser * Fixing requirements so that msticpy will still install on Py3.6 Fixing tests for packages to use pkg_resource specifier parsing (which it should have always used) Updating Kqlmagic version to official release. * Adding AzCli URi to exceptions Updating black params in pre-commit and pipelines to remove -t py36 flag * removing unused warnings from import_analyzer.py * Updating to 1.7.0 * Changing magics creation so that they don't get created if not in ipython (#332) Adding import of magics to nbinit and removing from __init__ and Pivot class. Updating docs (including some auto-gen'd) * Removing un-needed config * Redacted sample credentials * Added refresh and delete functions for keyring cached secrets (#336) * Added refresh and delete functions for keyring cached secrets * Black reformatting of secret_settings * Powershell simple de-obfuscator and code viewer. (#335) * Simple code de-obfuscator and display for PowerShell * Setting default style to "default" and making display_html DisplayHandle return optional * pep257 doc string linting errors in code_cleanup and code_view * Adding pygments to requirements (this is already a dependency of other core dependencies so should have no impact on install) * Fixing test failure in test_code_view Also linting errors suppressed from bandit, prospector and pylint * Added Splunk async provider and unit_tests (#337) * Added Splunk async provider and unit_tests * Fixed incorrect property call Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixed incorrect use of httpx.codes enum in odata_driver (#345) * Fixed incorrect use of httpx.codes enum in odata_driver Added httpx timeout to cybereason_driver and http_base * Added additional context for exceptions. Formatting change for http_base.py Moving import of VTFileBehaviour out of try/except block in vtlookupv3.py * Changing default timeout for httpx client to match requests 30sec for connect, 10sec elsewhere, # Conflicts: # msticpy/data/drivers/cybereason_driver.py # msticpy/sectools/tiproviders/http_base.py * Adding timeouts to missing httpx calls * Splitting keyring into its own module so that we can load without this as a dependency # Conflicts: # msticpy/common/secret_settings.py * Needed type hint in exceptions.py * Putting IPStack APIKey check happen when first used (rather than in __init__) so it doesn't throw exception on loading * prospector config changed produces deprecation warning and non-zero exit code. * Fixing misconfigured prospector.yaml * Test fix for test_cybereason_driver copied from v2 branch * removing version restriction for prospector in Github actions python-package.yaml # Conflicts: # .github/workflows/python-package.yml * Bump readthedocs-sphinx-ext from 2.1.4 to 2.1.5 (#339) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.4 to 2.1.5. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](https://github.com/readthedocs/readthedocs-sphinx-ext/compare/2.1.4...2.1.5) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updating version to 1.7.5 (#348) Suppressing FP bandit warning Adding Module-Analysis.ipynb notebook to tools * Ianhelle/mp config edit load fix 2022 03 28 (#352) * Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. Now loads with empty settings rather than throw exception. Added unit test case * Re-ordering arguments so doesn't break inheritance and cause pylint warning (from v2.0 branch) * Reverting changes to args and adding pylint suppressions * Aligning splunk_uploader params with base class * Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters. Fixed query_time widget so that you can reset time range from parameter Added additional unit test for QueryTime setter Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line. Also changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName. Updated TimeSpan class so it has more flexible constructor Added account_id as a parameter for list_aad_signins_by_account query * Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts to defaults. This can cause errors if the defaults are different to user-specified parameters. There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings. * Pebryan/2022 3 29 auth updates (#351) * new msal delegated auth option for graph * Switch to DefaultAzureCredential * renamed MSALAuth * Linting fixes * Add Unit Test and PR changes * Updates to fix tests * Fixed execption error * formatting * Merging in Splunk fixes from #352 * fixed incorrect merge * New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352. * hotfix for bug found in testing * Fixed re-auth on query issue in KQL driver * Removing un-needed code * Fixed kql_driver tests * Liniting fixes Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Bump sphinx from 4.4.0 to 4.5.0 (#350) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.4.0 to 4.5.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.4.0...v4.5.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixes for GeoLiteLookup and MpConfigEdit (#356) * Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py. Fixed bug where empty/new msticpyconfig.yaml didn't save any settings. * Reorganized logic for handling parameters and failing on invalid file path. * Some fixes to Kusto common_imports (#358) - now works with Kusto config entry without instance suffix - can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query - added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding in the data_famiies key. Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * linting fixes * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * Changing the pattern for httpx timeout to default to Timeout(None). (#378) * Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x) Other components use the default. * Adding case for reading list from yaml instead of tuple - now handles any iterable. * Added unit test and made some fixes to pkg_config.py * Bug in test test_pkg_config.py Also in test_code_view.py * Add Workflow to Tweet (#369) * Add Workflow to Tweet * Update tweet.yml * Update tweet.yml Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing warnings for malformed regexes in kql_driver, test_sentinel_search Re-enabling pytest.skip in test_nbwidgets.py * Fixed minor issues (#371) * Fixed minor issues * Fixed additional use case Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#374) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Missing import in test_nbwidgets * Forgot to add "r" prefix to strings in test_sentinel_search * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#379) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Updating version to 1.8.0 * Fix for MpConfigEdit ValueError Updating version for hotfix * MpConfig edit throws error with invalid file path. (#395) * Updating Dockerfile source to mcr anaconda * Update API version for list_alert_rules To be consistent with the documentation (https://docs.microsoft.com/en-us/rest/api/securityinsights/stable/alert-rules/list) api version should be set to "2021-10-01" * (fix) moving conda-supported files from conda-*pip* files to conda requirements * Update deprecated prospector tool names. * Updated Tweet bot to include more context in the tweets * Updated tweet action to include more detail in the tweets * Updated OData drivers to allow for Delegated auth settings to be passed when connecting. Includes the ability to use Delegated Auth as well as the method. Added documentation on how to use the feature. * Fixed linting issues in odata_driver * Updated requirement for azure-identity to 1.10.0 * Microsoft mandatory file (#407) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> * Bump readthedocs-sphinx-ext from 2.1.5 to 2.1.6 (#400) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.5 to 2.1.6. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](https://github.com/readthedocs/readthedocs-sphinx-ext/commits) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated default config file to include username for MDE and Graph. Fixed string formatting in security_graph_driver as per PR comments. * Suppressed exception logger message from msal_extensions in kql_driver.py (#411) Removed auto-load of VT Pivots - causes an exception when vt SDK is not installed - in nbinit.py Added check for null config values in user_config.py Added requirement for typing-extensions 4.2.0 (required by bokeh) Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * Updating version to 1.8.2 * Replace MSAL auth plaintext file cache with memory cache (#413) * Removed plaintext token chace from MSAL auth and replaced it will fall back to in memory caching. * Adding in catch of additional execptions in msal_auth * Removed := to retain 3.6 support in main Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Removing some files from merge errors * Merge remote-tracking branch 'origin/main' into ianhelle/merge2.0_to_main-2022-06-14 * [fix] removing deprecated pylint warnings from python-package.yml (github) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml * [fix] CodeQL fixes for incorrect regex (all but one were in test code) - odata_driver.py [fix] missing await in url_checker_async.py * [fix] Adding updated ContiLeaksAnalysis notebook * [fix] Trying different suppressions for credscan Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ashwin Patil <ashwin-patil@users.noreply.github.com> Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: FlorianBracq <97248273+FlorianBracq@users.noreply.github.com> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> * Changing some params in .readthedocs.yaml * Adding jinja<3.1.0 to docs/requirements.txt * Removing msticpy requirements from .readthedocs.yaml Adding cryptograph to docs/requirements.txt * remove path: . from .readthedocs.yaml * Removing install key from .readthedocs.yaml * Updating docs/requirements.txt Adding intersphinx to conf.py * Documentation updates to sphinx files * [fix] Revert to Py 3.7 build with typing-extensions (#448) * [fix] Adding updated sphinx packages to requirements.txt * [fix] wrong path in .readthedocs.yaml * Update RTD Python and Linux versions * [fix] adding back intersphinx and updating RTD build to Py 3.9 Ubuntu 22.04 * [fix] typing-extensions exception added to import_analyzer.py * [fix] if AuthKey or ApiID is None (#449) * Ianhelle/query pivot naming 2022 06 06 (#437) * Bump sphinx from 4.3.2 to 4.4.0 (#283) * Bump sphinx from 4.3.2 to 4.4.0 Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.3.2 to 4.4.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.3.2...v4.4.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating requirements-dev.txt to sync with dependabot updates Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * doc updates (#316) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * adding devcontainer files (#321) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Bump respx from 0.17.1 to 0.19.2 (#314) Bumps [respx](https://github.com/lundberg/respx) from 0.17.1 to 0.19.2. - [Release notes](https://github.com/lundberg/respx/releases) - [Changelog](https://github.com/lundberg/respx/blob/master/CHANGELOG.md) - [Commits](https://github.com/lundberg/respx/compare/0.17.1...0.19.2) --- updated-dependencies: - dependency-name: respx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated Cybereason docs to fit pattern (#324) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Ianhelle/1.6.1 hotfixes 2022 01 27 for 1.6.2 (#317) * Importing a couple of items into init for backward compatibility fixing keyvault authentication error in AML Fixing bug reading None value in mordor_browser * Fixing requirements so that msticpy will still install on Py3.6 Fixing tests for packages to use pkg_resource specifier parsing (which it should have always used) Updating Kqlmagic version to official release. * Adding AzCli URi to exceptions Updating black params in pre-commit and pipelines to remove -t py36 flag * removing unused warnings from import_analyzer.py * Updating to 1.7.0 * Changing magics creation so that they don't get created if not in ipython (#332) Adding import of magics to nbinit and removing from __init__ and Pivot class. Updating docs (including some auto-gen'd) * Removing un-needed config * Redacted sample credentials * Added refresh and delete functions for keyring cached secrets (#336) * Added refresh and delete functions for keyring cached secrets * Black reformatting of secret_settings * Powershell simple de-obfuscator and code viewer. (#335) * Simple code de-obfuscator and display for PowerShell * Setting default style to "default" and making display_html DisplayHandle return optional * pep257 doc string linting errors in code_cleanup and code_view * Adding pygments to requirements (this is already a dependency of other core dependencies so should have no impact on install) * Fixing test failure in test_code_view Also linting errors suppressed from bandit, prospector and pylint * Added Splunk async provider and unit_tests (#337) * Added Splunk async provider and unit_tests * Fixed incorrect property call Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixed incorrect use of httpx.codes enum in odata_driver (#345) * Fixed incorrect use of httpx.codes enum in odata_driver Added httpx timeout to cybereason_driver and http_base * Added additional context for exceptions. Formatting change for http_base.py Moving import of VTFileBehaviour out of try/except block in vtlookupv3.py * Changing default timeout for httpx client to match requests 30sec for connect, 10sec elsewhere, # Conflicts: # msticpy/data/drivers/cybereason_driver.py # msticpy/sectools/tiproviders/http_base.py * Adding timeouts to missing httpx calls * Splitting keyring into its own module so that we can load without this as a dependency # Conflicts: # msticpy/common/secret_settings.py * Needed type hint in exceptions.py * Putting IPStack APIKey check happen when first used (rather than in __init__) so it doesn't throw exception on loading * prospector config changed produces deprecation warning and non-zero exit code. * Fixing misconfigured prospector.yaml * Test fix for test_cybereason_driver copied from v2 branch * removing version restriction for prospector in Github actions python-package.yaml # Conflicts: # .github/workflows/python-package.yml * Bump readthedocs-sphinx-ext from 2.1.4 to 2.1.5 (#339) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.4 to 2.1.5. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](https://github.com/readthedocs/readthedocs-sphinx-ext/compare/2.1.4...2.1.5) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updating version to 1.7.5 (#348) Suppressing FP bandit warning Adding Module-Analysis.ipynb notebook to tools * Ianhelle/mp config edit load fix 2022 03 28 (#352) * Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. Now loads with empty settings rather than throw exception. Added unit test case * Re-ordering arguments so doesn't break inheritance and cause pylint warning (from v2.0 branch) * Reverting changes to args and adding pylint suppressions * Aligning splunk_uploader params with base class * Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters. Fixed query_time widget so that you can reset time range from parameter Added additional unit test for QueryTime setter Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line. Also changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName. Updated TimeSpan class so it has more flexible constructor Added account_id as a parameter for list_aad_signins_by_account query * Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts to defaults. This can cause errors if the defaults are different to user-specified parameters. There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings. * Pebryan/2022 3 29 auth updates (#351) * new msal delegated auth option for graph * Switch to DefaultAzureCredential * renamed MSALAuth * Linting fixes * Add Unit Test and PR changes * Updates to fix tests * Fixed execption error * formatting * Merging in Splunk fixes from #352 * fixed incorrect merge * New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352. * hotfix for bug found in testing * Fixed re-auth on query issue in KQL driver * Removing un-needed code * Fixed kql_driver tests * Liniting fixes Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Bump sphinx from 4.4.0 to 4.5.0 (#350) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.4.0 to 4.5.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.4.0...v4.5.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixes for GeoLiteLookup and MpConfigEdit (#356) * Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py. Fixed bug where empty/new msticpyconfig.yaml didn't save any settings. * Reorganized logic for handling parameters and failing on invalid file path. * Some fixes to Kusto common_imports (#358) - now works with Kusto config entry without instance suffix - can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query - added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding in the data_famiies key. Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * linting fixes * Added new Sentinel Search Features: Create a new search Check the status of a search Delete a completed search Includes docs and unit tests Updated WorkspaceConfig: If one workspace in config but not called default it is still used by default * Changing the pattern for httpx timeout to default to Timeout(None). (#378) * Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x) Other components use the default. * Adding case for reading list from yaml instead of tuple - now handles any iterable. * Added unit test and made some fixes to pkg_config.py * Bug in test test_pkg_config.py Also in test_code_view.py * Add Workflow to Tweet (#369) * Add Workflow to Tweet * Update tweet.yml * Update tweet.yml Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing warnings for malformed regexes in kql_driver, test_sentinel_search Re-enabling pytest.skip in test_nbwidgets.py * Fixed minor issues (#371) * Fixed minor issues * Fixed additional use case Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#374) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Missing import in test_nbwidgets * Forgot to add "r" prefix to strings in test_sentinel_search * Fixing bug in local_data_driver.py if CSV with no TimeGenrated field (#379) Adding new query for logon attempts for IP address. Fix bug in select_alert if time column is supplied in list of columns to display. Adding back pytest skip for widgets notebook test. * Updating version to 1.8.0 * Fix for MpConfigEdit ValueError Updating version for hotfix * MpConfig edit throws error with invalid file path. (#395) * Updating Dockerfile source to mcr anaconda * Update API version for list_alert_rules To be consistent with the documentation (https://docs.microsoft.com/en-us/rest/api/securityinsights/stable/alert-rules/list) api version should be set to "2021-10-01" * (fix) moving conda-supported files from conda-*pip* files to conda requirements * Update deprecated prospector tool names. * Updated Tweet bot to include more context in the tweets * Updated tweet action to include more detail in the tweets * Updated OData drivers to allow for Delegated auth settings to be passed when connecting. Includes the ability to use Delegated Auth as well as the method. Added documentation on how to use the feature. * Fixed linting issues in odata_driver * Updated requirement for azure-identity to 1.10.0 * Microsoft mandatory file (#407) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> * Bump readthedocs-sphinx-ext from 2.1.5 to 2.1.6 (#400) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.1.5 to 2.1.6. - [Release notes](https://github.com/readthedocs/readthedocs-sphinx-ext/releases) - [Commits](https://github.com/readthedocs/readthedocs-sphinx-ext/commits) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Updated default config file to include username for MDE and Graph. Fixed string formatting in security_graph_driver as per PR comments. * Suppressed exception logger message from msal_extensions in kql_driver.py (#411) Removed auto-load of VT Pivots - causes an exception when vt SDK is not installed - in nbinit.py Added check for null config values in user_config.py Added requirement for typing-extensions 4.2.0 (required by bokeh) Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * Updating version to 1.8.2 * Replace MSAL auth plaintext file cache with memory cache (#413) * Removed plaintext token chace from MSAL auth and replaced it will fall back to in memory caching. * Adding in catch of additional execptions in msal_auth * Removed := to retain 3.6 support in main Co-authored-by: Pete Bryan <pebryan@microsoft.com> * Fix for list_hunting_queries function Fix for list_hunting_queries function, referred to alert_rules api, which does not contain such. Rather going towards savedSearches endpoint. It could also be pointed out somewhere that this is for custom queries only, i.e. Provider="Custom Queries" * Update calls to credential.modern.get_token Tenant_id should only be used when it is defined. * Adding ContiLeaks Analysis (#428) Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * [update] Changed data_providers to only add pivots on connect(), adding instance property [update] exposing driver instance property in driver_base, cybereason_driver, kql_driver [update] Updated PivotFunctions.ipynb and PivotFunctions.rst with new behavior [update] Added new SingletonClass to types, rename previous class to SingletonArgsClass in types.py [update] Exposing workspace instance name in wsconfig.py [update] Updating geoip.py to use renamed SingletonArgsClass [update] Adding short name to multiple MDE queries [update] Added process query using only file_hash parameter for pivot query [update] Adding replaceable table parameter to kql_mdatp_user.yaml queries [update] pivots() and get_pivot_list() now supports search string and returns sorted list [update] Adding "pivot" attribute to msticpy after loading pivot [update] Added doc string to txt2df magic in nbmagics.py [update] Pivot is now a singleton, rationalized query time setting, removed adding data provider queries at load [update] Changed clipboard/function text to match usage with imported entities in pivot_browser.py [update] Added use of "explode" in list_to_rows in pivot_pd_accessor.py [update] Importing vt_pivot into pivot_core/__init__.py [update] Changed to support multiple provider instances, removed shortcut query functions, renamed some tables, pivot data queries now use central Pivot.timespan by default [update] Removed provider-specific and IPv4/v6 specific functions - huge simplification in pivot_ti_provider.py [update] Updated and rationalized Pivot tests for new behavior. add test_vt_pivot.py [fix] fixed proper reporting of pivot functions in pivot_container.py [fix] removing deprecated PyLint warning suppression from account.py and process.py [fix] popping extra ioc_type from params in ti_provider_base.py [fix] Fixing warnings for Pylint 2.14.0 - removing deprecated warning types in .pylintrc [fix] Removing duplicate syslog_utils.py (from graphs_plot branch) [fix] Pylint warning in vtlookup.py (from graphs_plot branch) [fix] Pylint warning in entity_graph_tools.py (from graphs_plot branch) [fix] Removing unsupport Pylint warning type from account.py, process.py and base64unpack.py [fix] Updating docs for removed syslog_utils.py in msticpy.transform.rst and msticpy.transform.syslog_utils.rst [fix] Incorrect heading underlining in SettingsEditor.rst * [update] Updated PivotFunctions-Introduction notebook for new behavior [update] Added references to notebooks in PivotFunctions.rst [update] auto-update to DataQueries.rst [fix] formatting error in Installing.rst [update] removing shortcut functions from VT pivots * [fix] moving pivot tests to tests/init folder * [fix] Correcting doc strings in time series functions and accessors * [fix] minor fixes in FoliumMap and PivotFunctions notebooks * Removing some files from merge errors * Merge remote-tracking branch 'origin/main' into ianhelle/merge2.0_to_main-2022-06-14 * [fix] removing deprecated pylint warnings from python-package.yml (github) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml * [fix] removing deprecated pylint warnings from python-package.yml (github) azure-pipelines.yml, and riskiq.py [fix] adding required sphinx packages to azure-pipelines.yml * [fix] CodeQL fixes for incorrect regex (all but one were in test code) - odata_driver.py [fix] missing await in url_checker_async.py * [fix] CodeQL fixes for incorrect regex (all but one were in test code) - odata_driver.py [fix] missing await in url_checker_async.py * [fix] Adding updated ContiLeaksAnalysis notebook * [fix] Adding updated ContiLeaksAnalysis notebook * [fix] Trying different suppressions for credscan * [fix] adding back intersphinx and updating RTD build to Py 3.9 Ubuntu 22.04 * [fix] typing-extensions exception added to import_analyzer.py * [fix] avoid trying to add Pivot functions if VTLookupV3 can't be initialized - in vt_pivot.py Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ashwin Patil <ashwin-patil@users.noreply.github.com> Co-authored-by: Pete Bryan <pebryan@microsoft.com> Co-authored-by: FlorianBracq <97248273+FlorianBracq@users.noreply.github.com> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: pensivepaddle <104833713+pensivepaddle@users.noreply.github.com> Co-authored-by: Thomas Roccia <thomas.roccia@gmail.com> * Ianhelle/folium update docs 2022 05 29 (#438) * Bump sphinx from 4.3.2 to 4.4.0 (#283) * Bump sphinx from 4.3.2 to 4.4.0 Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.3.2 to 4.4.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.3.2...v4.4.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating requirements-dev.txt to sync with dependabot updates Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> * doc updates (#316) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * adding devcontainer files (#321) Co-authored-by: Ian Hellen <ianhelle@microsoft.com> * Bump respx from 0.17.1 to 0.19.2 (#314) Bumps [respx](https://github.com/lundberg/respx) from 0.17.1 to 0.19.2. - [Release notes](https://github.com/lundberg/respx/releases) - [Changelog](https://github.com/lundberg/respx/blob/master/CHANGELOG.md) - [Commits](https://github.com/lundberg/respx/compare/0.17.1...0.19.2) --- updated-dependencies: - dependency-name: respx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: d…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.