Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Ianhelle/settings mgmt 2021 02 02 (#136)
* Typo in opening sentence * Adding hash_account as separate item type to data_obfus.py Making hash_ip more flexible - ignoring things like localhost Updating documentation, tests and mapping file. Correcting typo in timeline.py. * Adding missed documentation for hash_account * Initial code for Mordor driver and browser * Mordor data provider and browser. unit tests and documentation * Fixing some linting errors. * Fixed a couple of broken tests because of data providers API change. * Replacing custom json reader with pd.read_json() Added ability to set query defaults (like cache directory) from provider. Fixed a bug in path construction for download file. Clarified the description of the search functionality and corrected Mitre Attack => ATT&CK Add URL for Mitre Updated notebook and doc to reflect these changes. * Fixing lint/formatting errors in vtlookupv3. Some other random black reformatting Added test_mordor_browser.py for notebook test. * Updated formatting for new black version * Updating pre-commit version * Bug fix and nasty workaround for old test setup removed in pkg_config.py * Update MordorData.rst doc with better intro section * Splitting entities into separate modules * Moved entities to datamodel package and initial refactoring for pivoting * Renaming files to lowercase phase 1 * Renaming entities phase 2 * Start of pivot main library * Commit to re-merge with master * Code complete - still docs to do. * Added test case and fix for couple of misc methods in Pivot and Entity * Phase 1 code complete with docs. * Fixing the credscan suppression for test_splunk_uploader * Adding pre-release version, removing old config file. * Initial dependency separation * Implemented extras for msticpy install. Refactored a few classes to make it easier to import and use modules if only partial msticpy install. Installing Main one is data_providers - dynamically loading drivers. Also eventcluster and auditdextract. Moved latter two into analysis folder. Remove unneeded code from keyvault_client.py since Pete's code eliminated the need for them. Made AzureSentinel and MDE the preferred names for LogAnalytics and MDE drivers. Fixed up several unit tests to handle partial installs and still produce results (most should be skipped now instead of erroring). Fixed a random bugs (like GeoIP Maxmind download) Fixed pivot_register_reader to skip classes that cannot be instantiated (e.g. IPStack if user doesn't have API key) Added documentation to Installing.rst Fixed some problems and renamed module locations in notebooks and RST docs. * Additions/corrections to Installing.rst * Somehow these two data files were changed. * Bandit exception to except: pass * Correction to FoliumMap.ipynb * Removing dropna from read_csv in FoliumMap.ipynb * Adding requirements-all and pre-commit hook to generate this file * Adding vt, vt_graph to Sphinx mock list * Added pivot_browser UI - pivot_browser.py Added ability to read pipeline definitions from yaml files - pivot_pipeline.py Adding pivot.tee_exec pipeline function - in pivot_pd_accessor.py Add ability to add arbitrary/ad hoc functions as pivots - in pivot.py Exposing get_timespan function in Pivot class as public function - in pivot.py. Added Dns entity to several pivot functions - mp_pivot_reg.yaml * Fixing some queries for more consistency. Pivot data query functions now prefixed with table name. Added ability for pivot functions to return raw output. Added pyperclip to pkg dependencies exceptions. * Some corrections to documentation in AzureSentinel and DataAcquisition docs. Added lru_cache for geoip lookups. * User environment configuration for notebooks. Added minimal output from nbinit to show imported modules (I'd noticed some examples of people import stuff that had already been imported) * Fixing mordor tests and updating azure-mgmt-monitor version in setup.py extras * Some fixes and changes to the UserDefaults feature - esp the format of the config settings. Also - some fixes to tests for test_pkg_imports and import_analyzer.py - fix to config2kv.py to correct some problems, Also added a function to retrieve and show current KV secrets - fix for ipwidgets warning about deprecated on_submit() method - multiple fixes for typos and duplicate section names in: DataProviders.rst, UploadData.rst, PivotFunctions.rst - added SplunkProvider.rst doc for Splunk provider - fixed issue in nbinit.py where extra_imports were being lost. - fix for QueryTime in nbwidgets.py - exception if user types invalid value into date field. - fixed several issues in test_mp_release.cmd with messed up folders/current folder. * MSTICPY config settings management Two main classed - MpConfigFile (to manage settings file and do a few utility things) - MpConfigEdit (to edit settings for mp config sections) Still to add docs/notebook * PR updates adding comments, some grammer fixes and obfuscation of names. * PR updates adding comments, some grammar fixes and obfuscation of names. * Some fixes and changes to the UserDefaults feature - esp the format of the config settings. Also - some fixes to tests for test_pkg_imports and import_analyzer.py - fix to config2kv.py to correct some problems, Also added a function to retrieve and show current KV secrets - fix for ipwidgets warning about deprecated on_submit() method - multiple fixes for typos and duplicate section names in: DataProviders.rst, UploadData.rst, PivotFunctions.rst - added SplunkProvider.rst doc for Splunk provider - fixed issue in nbinit.py where extra_imports were being lost. - fix for QueryTime in nbwidgets.py - exception if user types invalid value into date field. - fixed several issues in test_mp_release.cmd with messed up folders/current folder. MSTICPY config settings management Two main classed - MpConfigFile (to manage settings file and do a few utility things) - MpConfigEdit (to edit settings for mp config sections) Still to add docs/notebook Additional tests, start of validation checks Added validation step (none-blocking) to forms Fixing some settings validation issues Fixed default values being overwritten for new items Adding more tests and fixes. Added check_version.py and added call to this from nbinit.py Added Mordor and LocalData as configurable providers in settings. mordor_driver, local_data_driver and azure_auth now check settings for defaults. Add list() type to mpconfig_defaults.yaml Settings documentation and notebook. Also updating README.md and PackageSummary.rst with something more contemporary. * Some tests failing after merge. Fixed URL in README.md * Merge tag 'v0.9.0' into ianhelle/MP-Pivot-Phase2-2021-01-04 Fixing some test and linting errors after merge. Removing lru_cache from ip_lookup in geoip.py * test_file_browsert test failing because it was trying to change into parent folder and parent folder doesn't exist in CI test environment * Add joins for pivot data queries in pivot_data_queries.py Add "print" query debug parameter in data_providers.py Add find_entity function in entities __init__.py Add alias "pivots" for get_pivot_list in entity.py Add ability to set timespan more flexibly. Calling set_timespan no longer resets the timespan. Add PivotBrowser method to Pivot class - in pivot.py Add missing entity list box in pivot_browser.py. Switched engine to "Python" for pd.read_csv in pivot_magic_core.py to handle more formatting types. Add positional params to pipeline step and cleaned up code in pivot_pipeline.py Updated PivotFunctions.rst and PivotFunctions.ipynb for new functionality. More tests for test_pivot.py (timespan) New tests for PivotBrowser - test_pivot_browser.py Enable and fix tests for pivot data query joins in test_pivot_data_queries_run.py Add test for positional params in test_pivot_pipeline.py * Suppressing expected user warnings in tests. Fixing a bug with the "print_query" debug option being called from TIProviders/kql_base.py. Cleaning up mordor data file cleanup in test_mordor_driver.py. Adding an optimistic random delay to geoip.py to avoid instances in different processes trying to download the same file simultaneously. Really only an issue in multi-processing distributed tests. * Fixing test error in test_user_config.py McCabe complexity warning in config2kv.py * Updating version * Bandit warning on use of random.randint() Updating version * Removing fake secret from MPSettingsEditor.ipynb Moving list definition for mypy in local_data_driver.py Black reformatting test_user_config.py * Failing test and linter warnings * Adding notice and badge to Readme * Adding documentation diagrams * Updates from PR. Also fixing a bug and merge conflict in mp_config_file.py where I was passing the whole URL as the secret name. Also put a catch for this in keyvault_client.py.
- Loading branch information
