Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Ianhelle/mp pivot phase3 2021 02 22 (#140)
* Typo in opening sentence * Adding hash_account as separate item type to data_obfus.py Making hash_ip more flexible - ignoring things like localhost Updating documentation, tests and mapping file. Correcting typo in timeline.py. * Adding missed documentation for hash_account * Initial code for Mordor driver and browser * Mordor data provider and browser. unit tests and documentation * Fixing some linting errors. * Fixed a couple of broken tests because of data providers API change. * Replacing custom json reader with pd.read_json() Added ability to set query defaults (like cache directory) from provider. Fixed a bug in path construction for download file. Clarified the description of the search functionality and corrected Mitre Attack => ATT&CK Add URL for Mitre Updated notebook and doc to reflect these changes. * Fixing lint/formatting errors in vtlookupv3. Some other random black reformatting Added test_mordor_browser.py for notebook test. * Updated formatting for new black version * Updating pre-commit version * Bug fix and nasty workaround for old test setup removed in pkg_config.py * Update MordorData.rst doc with better intro section * Splitting entities into separate modules * Moved entities to datamodel package and initial refactoring for pivoting * Renaming files to lowercase phase 1 * Renaming entities phase 2 * Start of pivot main library * Commit to re-merge with master * Code complete - still docs to do. * Added test case and fix for couple of misc methods in Pivot and Entity * Phase 1 code complete with docs. * Fixing the credscan suppression for test_splunk_uploader * Adding pre-release version, removing old config file. * Initial dependency separation * Implemented extras for msticpy install. Refactored a few classes to make it easier to import and use modules if only partial msticpy install. Installing Main one is data_providers - dynamically loading drivers. Also eventcluster and auditdextract. Moved latter two into analysis folder. Remove unneeded code from keyvault_client.py since Pete's code eliminated the need for them. Made AzureSentinel and MDE the preferred names for LogAnalytics and MDE drivers. Fixed up several unit tests to handle partial installs and still produce results (most should be skipped now instead of erroring). Fixed a random bugs (like GeoIP Maxmind download) Fixed pivot_register_reader to skip classes that cannot be instantiated (e.g. IPStack if user doesn't have API key) Added documentation to Installing.rst Fixed some problems and renamed module locations in notebooks and RST docs. * Additions/corrections to Installing.rst * Somehow these two data files were changed. * Bandit exception to except: pass * Correction to FoliumMap.ipynb * Removing dropna from read_csv in FoliumMap.ipynb * Adding requirements-all and pre-commit hook to generate this file * Adding vt, vt_graph to Sphinx mock list * Added pivot_browser UI - pivot_browser.py Added ability to read pipeline definitions from yaml files - pivot_pipeline.py Adding pivot.tee_exec pipeline function - in pivot_pd_accessor.py Add ability to add arbitrary/ad hoc functions as pivots - in pivot.py Exposing get_timespan function in Pivot class as public function - in pivot.py. Added Dns entity to several pivot functions - mp_pivot_reg.yaml * Fixing some queries for more consistency. Pivot data query functions now prefixed with table name. Added ability for pivot functions to return raw output. Added pyperclip to pkg dependencies exceptions. * Some corrections to documentation in AzureSentinel and DataAcquisition docs. Added lru_cache for geoip lookups. * Fixing mordor tests and updating azure-mgmt-monitor version in setup.py extras * PR updates adding comments, some grammer fixes and obfuscation of names. * PR updates adding comments, some grammar fixes and obfuscation of names. * Merge tag 'v0.9.0' into ianhelle/MP-Pivot-Phase2-2021-01-04 Fixing some test and linting errors after merge. Removing lru_cache from ip_lookup in geoip.py * Add joins for pivot data queries in pivot_data_queries.py Add "print" query debug parameter in data_providers.py Add find_entity function in entities __init__.py Add alias "pivots" for get_pivot_list in entity.py Add ability to set timespan more flexibly. Calling set_timespan no longer resets the timespan. Add PivotBrowser method to Pivot class - in pivot.py Add missing entity list box in pivot_browser.py. Switched engine to "Python" for pd.read_csv in pivot_magic_core.py to handle more formatting types. Add positional params to pipeline step and cleaned up code in pivot_pipeline.py Updated PivotFunctions.rst and PivotFunctions.ipynb for new functionality. More tests for test_pivot.py (timespan) New tests for PivotBrowser - test_pivot_browser.py Enable and fix tests for pivot data query joins in test_pivot_data_queries_run.py Add test for positional params in test_pivot_pipeline.py * Suppressing expected user warnings in tests. Fixing a bug with the "print_query" debug option being called from TIProviders/kql_base.py. Cleaning up mordor data file cleanup in test_mordor_driver.py. Adding an optimistic random delay to geoip.py to avoid instances in different processes trying to download the same file simultaneously. Really only an issue in multi-processing distributed tests. * Bandit warning on use of random.randint() Updating version * Black version mismatch * Adding notice and badge to Readme * Small changes from PR review * Adding column filtering accessor for pipeline * tilookup functions to allow enabling/disabling individual providers Fixed a problem with logon failure query parameters. * Minor corrections to queries in kql_sent_az_network and kql_sent_azure * Adding shortcut functions to entities * Pivot phase 3 code complete with better joins, more pd accessor functions and short/friendly aliases for pivot functions (especially queries) * Updated documentation. Added remove_pivot_funcs method to pivot.pivot.py + unit test in test_pivot Extracted dup code from pivot_data_queries to pivot_register. Fixing test in test_pivot_pd_accessor for renamed parameter. Updating notebooks for some pivot function renaming. * Spelling errors in SettingsEditor.rst * Prospector doc string warnings fixed * Add list_to_rows to pivot_pd_accessor.py * Add list_to_rows and parse_json to pivot_pd_accessor.py Adding str and repr to Edge in entity_graph.py Fixing path handling in file.py Adding (not yet used) graph_property.py to handle automatic creation of edges Add better timestamp conversion in vtlookup3.py. Remove unnecessary list comprehension in test_security_alert.py Add tests for list_to_rows and parse_json in pivot_pd_accessor.py - test_pivot_pd_accessor.py * Merge PivotFunctions-Introduction notebook. * Added missing entities: iot_device, mail_cluster, mail_message, mailbox, submission_mail Added check for attribute name similarity in query_container Fixed some errors in entities with description_str and identity fields Add missing FullName property to Host Fixed exception in IpAddress Removing duplicate mp_config file Fixing some test errors in test_mordor_driver (due to test file race conditions) * Updating blog_articles with recent publications Updating version to 1.1.0 Capturing output from load_user_defaults in nbinit.py Correcting problem with passing namespace argument to notebooklets in user_config.py Fixes to analyze_imports.py and import_analyzer.py that prevented use from commandline and use with other packages. * Fix to list_to_rows pd accessor in pivot_pd_accessor. * Changed timeline.py so that it doesn't error when no source_columns are supplied * Fixing bug in calculating period in timespan.py Adding find_anomaly_periods function and shorter alias for timeseries_anomalies_stl in timeseries.py Cosmetic fixes in eventcluster.py * Fixing an issue in ti_browser if passed an empty dataframe. Updating requirements files for updated versions of lxml and aiohttp for security issues.
- Loading branch information
