Ianhelle/rel1.0.0 cleanup 2021 03 12 (#147)

* Cleanup and bug fixes for v1.0 - dependencies. - Updated msticpy notebooks - Updated setup.py, requirements.txt and conda-reqs with new/non-conflicting version - Fix to test_nbinit.py and import_analyzer.py - Added test script for dependencies/extras - Updating version to pre3/rc3 - Fix to README - Updated notebooksamples.rst with latest notebooks.notebooksamples - Removed random unicode char in PivotFunctions.rst - Added Releases.rst - pointer to GitHub release page. * Fixing test error in test_nbinit.py Some fixes to notebooks (mainly getting rid of uneeded Seaborn refs) * Doc string error - prospector - in test_mp_extras.py * More updates for release: - Added links to medium articles in ReadtheDocs - Fixed QueryProviderDocumenter.ipynb - Generated new DataQueries.rst doc - Added KqlmagicCustom[jupyter-basic] back to core components - Updating docs for this in Installing.rst - Updated test_pkg_imports.py to account for this. - Setting KQLMAGIC_EXTRAS_REQUIRE env var in package __init__.py to prevent warnings in Kqlmagic - Minor clarification in MPSettingsEditor.ipynb - Reordered params in wsconfig.py WorkspaceConfig so that you can supply the workspace name as single positional parameter - Change kql_driver.py so that you can supply a WorkspaceConfig instance as the "connection_str" - Change to pivot_register to return single "raw" result if it is a list of one item - Add Pivot and entities as auto-imported items - * Add create static method to entity to instantiate entity from dict or pd.Series Fix bug in ip_utils.py convert to entities * Fix to provider name handling in user_config.py * Adding more queries for notebooklets - VMComputer and DNSEvents. Fixing wording in SelectAlert widget Fixing potential None value error in GetText and GetEnvironmentText Fixing a few bugs and tidying in convert_to_ip_entities in ip_utils.py Random linting errors. Fixing issue #146 - Error is thrown when AzureSentinel config is not present in msticpyconfig.yaml file Added extra tests for nbinit.py in test_nbinit.py * Fixing infinite recursion issue in process_tree_utils.py Issue #148 * Adding file lock to unit_test_lib::custom_mp_config - because settings are global, multiple test processes can change the settings on each other. Minor updates to test_nbinit.py Fix in pivot_register.py - _iterate_func did not pass through **kwargs to function. Also added a few comments to explain what's going on Bug in security_alert_graph - if NTDomain attribute is None Added local function cache to better handle repeated IPs Missing update to all_ips perf optimizatio nin convert_to_ip_entities Adding filelock to dev requirements-dev.txt * Corrected pip extras syntax in Installing.rst Invalid return type in pkg_config.py:: validate_config Added prompt_for_ws function to wsconfig.py. Also added more detail to warnings on how to fix things. Replace matplotlib draw_entity_alert_graph with Bokeh version in nbdisplay.py nbinit.py: - Added resource URLs to warnings - added KQLMAGIC_CONFIGURATION to enable trying AzureCLI SSO by default. - added pandas config to return schema with dataframe html to render using native nteract data browser. * Adding markdown to requirements-dev.txt * black formatting of security_alert * Adding beautifulsoup4 and markdown to dev/test requirements in requirements-dev.txt and conda-reqs-dev.txt Adding time unit control to QueryTime widget to allow interactive setting of day/week/month. Also increased the max range for these units - in nbwidgets.py * Fixing test error - seems like subtle change or bug in pandas groupby behavior - in sessionize.py Changing Development Status classifier and adding some extra keywords in setup.py Adding more verbose output to try to catch spurious errors in test_nbinit.py * Workarounds for some test errors in test_nbinit.py and test_user_config.py Updating requirements-dev.txt with pip-compatible versions. Adding same updates to conda-reqs-dev.txt and conda-reqs-dev-pip.txt
microsoft · Apr 3, 2021 · 675042c · 675042c
1 parent 707a201
commit 675042c
Show file tree

Hide file tree

Showing 50 changed files with 7,736 additions and 6,281 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -11,12 +11,15 @@ repos:
     hooks:
       - id: black
         language: python
+        args:
+          - -t
+          - py36
   - repo: https://github.com/pre-commit/mirrors-pylint
     rev: v2.6.0
     hooks:
       - id: pylint
         args:
-          - --disable=E0401
+          - --disable=E0401,W0511
           - --ignore-patterns=test_
   - repo: https://gitlab.com/pycqa/flake8
     rev: 3.8.4

diff --git a/conda/conda-reqs-dev-pip.txt b/conda/conda-reqs-dev-pip.txt
@@ -1,5 +1,4 @@
-flake8-mypy>=17.8.0
-mypy-extensions>=0.4.1
-prospector>=1.1.7
-pyroma>=2.5
-pytest-check>=0.3.9
+
+prospector>=1.3.1
+pyroma>=3.1
+pytest-check>==1.0.1
diff --git a/conda/conda-reqs-dev.txt b/conda/conda-reqs-dev.txt
@@ -1,21 +1,24 @@
 aiohttp>=3.0.0
-bandit>=1.6.2
-black>=19.3b0
-coverage>=4.5.4
-flake8>=3.7.8
+bandit>=1.7.0
+beautifulsoup4
+black>=20.8b1
+coverage>=5.5
+filelock>=3.0.0
+flake8>=3.8.4
+markdown>=3.3.4
 mccabe>=0.6.1
-mypy>=0.720
-nbdime>=1.1.0
-pep8-naming>=0.4.1
+mypy>=0.821
+nbdime>=2.1.0
+pep8-naming>=0.10.0
 pep8>=1.7.1
 pipreqs>=0.4.9
-pycodestyle>=2.5.0
-pydocstyle>=3.0.0
-pyflakes>=2.1.1
-pylint>=2.3.1
-pytest-cov>=2.7.1
+pycodestyle>=2.6.0
+pydocstyle>=6.0.0
+pyflakes>=2.2.0
+pylint>=2.5.3
+pytest-cov>=2.11.1
 pytest>=5.0.1
-responses>=0.10.8
-sphinx_rtd_theme
+responses>=0.13.2
+sphinx_rtd_theme>=0.5.1
 sphinx>=2.1.2
 virtualenv
diff --git a/docs/notebooks/MPSettingsEditor.ipynb b/docs/notebooks/MPSettingsEditor.ipynb
diff --git a/docs/source/blog_articles.rst b/docs/source/blog_articles.rst
@@ -0,0 +1,30 @@
+Blog articles on our Medium account
+===================================
+
+
+`MSTICPy v1.0 pre-release <https://msticpy.medium.com/msticpy-1-0-pre-release-6d6edc5df79c>`__
+March 17, 2021
+
+`Pivot Functions 0.9.0 release <https://msticpy.medium.com/msticpy-0-9-0-pivot-functions-2be851ae2001?source=friends_link&sk=b3ba3a1096c694854a11c2bbdae5333e>`__
+Feb 22, 2021
+
+`MSTICPy 0.8.8 release <https://msticpy.medium.com/msticpy-0-8-8-release-5e8fe28a77d6?source=friends_link&sk=4b3682409a3b266cde7e4d805e35b406>`__
+Oct 28, 2020
+
+`MSTICPy 0.8.0 release <https://msticpy.medium.com/msticpy-0-8-0-release-5e7a94e0f2f1?source=friends_link&sk=125ba48ad84f1ed462b92c22f66612d8>`__
+Sep 21, 2020
+
+`MSTIC Notebooklets <https://msticpy.medium.com/announcing-mstic-notebooklets-d32479bd07f?source=friends_link&sk=6cf84354153dcf86498bac84412788b0>`__
+Aug 17, 2020
+
+`MSTICPy 0.7.1 release <https://msticpy.medium.com/msticpy-0-7-0-1-release-758c5cbbf06d?source=friends_link&sk=a77c6479783e79439d6b2acfbf07ecf9>`__
+Aug 14, 2020
+
+`MSTICPy 0.6.1 release <https://msticpy.medium.com/msticpy-0-6-0-1-release-4b12e76099a7?source=friends_link&sk=5bfca0ae257d19800c1cad4d71cceced>`__
+Jul 2, 2020
+
+`MSTICPy 0.5.1 release <https://msticpy.medium.com/msticpy-0-5-1-release-107f531a738f?source=friends_link&sk=10d584982ae261b4cc090d72bf43939d>`__
+May 29, 2020
+
+`MSTICPy 0.5.0 release <https://msticpy.medium.com/msticpy-0-5-0-released-a1ebfc362a1?source=friends_link&sk=66640f711c88311bf737e031368d936d>`__
+May 14, 2020
diff --git a/docs/source/data_acquisition/DataQueries.rst b/docs/source/data_acquisition/DataQueries.rst
diff --git a/docs/source/getting_started/Installing.rst b/docs/source/getting_started/Installing.rst
@@ -67,7 +67,7 @@ Selective Installation - using "extras"
 pip supports specification of an additional parameter sequence
 known as extras. The syntax for this is:
 
-``pip install package_name[extra1, extra2...]``
+``pip install package_name[extra1,extra2,...]``
 
 As of version 0.9.0 *MSTICPy* has its dependencies split into
 extras. This allows you to install only the packages that you
@@ -78,8 +78,9 @@ that you do not need.
    installed - only the external libraries on which certain
    functions inside *MSTICPy* need to work.
 
-.. warning:: the core install no longer includes **Azure Sentinel** or
-   **Azure** libraries such as Kqlmagic. If you are an Azure Sentinel
+.. warning:: the core install no longer includes the
+   **Azure** libraries and only a limited install of the Kqlmagic package
+   used for most Azure Sentinel data queries. If you are an Azure Sentinel
    user, you should always install with the "azsentinel" extra.
 
 Extras in *MSTICPy*
@@ -92,6 +93,7 @@ The extras available in *MSTICPy* are described in the following table:
 |                  |                                    | (increment)  | (full)       |
 +==================+====================================+==============+==============+
 | [none]           | - Most functionality (approx 75%)  |       --     |   1m:13s     |
+|                  | - Kqlmagic Jupyter basic           |              |              |
 +------------------+------------------------------------+--------------+--------------+
 | keyvault         | - Key Vault and keyring storage of |       5s     |   1m:18s     |
 |                  |   settings secrets                 |              |              |
@@ -104,7 +106,7 @@ The extras available in *MSTICPy* are described in the following table:
 |                  | - Also includes "keyvault"         |              |              |
 +------------------+------------------------------------+--------------+--------------+
 | kql              | - Azure Sentinel data queries      |   2m:07s     |   3m:20s     |
-|                  | - Kqlmagic                         |              |              |
+|                  | - Kqlmagic Jupyter extended        |              |              |
 +------------------+------------------------------------+--------------+--------------+
 | azsentinel       | - Combination of core install      |   3m:48s     |   5m:00s     |
 |                  |   plus "azure", "keyvault" and     |              |              |
@@ -139,19 +141,23 @@ installed: jupyter, pandas and matplotlib.
 
 If you do not specify an "extra" in your pip install command, the base
 dependencies for *MSTICPy* will be installed. This has a lot of functionality
-such as networking, pivoting, visualization but excludes dependencies
+such as networking, pivoting, visualization but excludes most dependencies
 that are specific to a particular data environment like Azure Sentinel or
 Splunk.
 
 Some of the extras, like "all" and "azsentinel" are combinations of
 other options collected together as a convenience. You can also specify
 multiple extras during install, separating them with commas.
 
-.. note:: Since Azure Sentinel is
-
 .. code:: bash
 
-    pip install msticpy[azure, kql]
+    pip install msticpy[azure,kql]
+
+.. warning:: when specifying multiple extras, do not leave spaces between
+   the options - just separate with commas.
+
+Missing "extra" exceptions
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 If you try to use functionality for a component that needs a dependency
 that you have not installed you will usually get an informative

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -70,6 +70,7 @@ Contents
    Visualization
    msticpyAPI
    notebooksamples
+   blog_articles
    Releases
    contributing
    license

diff --git a/msticpy/__init__.py b/msticpy/__init__.py
@@ -28,6 +28,8 @@
 
 """
 
+import os
+
 # flake8: noqa: F403
 from .nbtools.nbinit import init_notebook, current_providers
 from .common import pkg_config as settings
@@ -40,3 +42,6 @@
 
 __version__ = VERSION
 __author__ = "Ian Hellen, Pete Bryan, Ashwin Patil"
+
+if not os.environ.get("KQLMAGIC_EXTRAS_REQUIRES"):
+    os.environ["KQLMAGIC_EXTRAS_REQUIRES"] = "jupyter-basic"
diff --git a/msticpy/_version.py b/msticpy/_version.py
@@ -1,2 +1,2 @@
 """Version file."""
-VERSION = "1.0.0.pre3"
+VERSION = "1.0.0rc4"
diff --git a/msticpy/analysis/anomalous_sequence/sessionize.py b/msticpy/analysis/anomalous_sequence/sessionize.py
@@ -73,7 +73,7 @@ def sessionize_data(
     # aggregate by the session_ind column
     agg_df = (
         df_with_sesind.sort_values(["session_ind", time_col])
-        .groupby(["session_ind"] + user_identifier_cols)
+        .groupby(["session_ind"] + user_identifier_cols, as_index=False)
         .agg({time_col: ["min", "max"], event_col: list})
         .reset_index()
     )
@@ -93,6 +93,8 @@ def sessionize_data(
     agg_df["number_events"] = agg_df["{}_list".format(event_col)].apply(len)
 
     agg_df = agg_df.drop("session_ind", axis=1)
+    if "index" in agg_df.columns:
+        agg_df = agg_df.drop("index", axis=1)
 
     # replace dummy_str with nan values
     for col in user_identifier_cols:

diff --git a/msticpy/common/check_version.py b/msticpy/common/check_version.py
@@ -35,7 +35,7 @@ def check_version():
         latest_version,
     )
     if installed_version < latest_version:
-        print(f"A newer version of MSTICPy - {latest_version} is available.")
+        print(f"A newer version of msticpy - {latest_version} is available.")
         print("Upgrade with pip install --upgrade msticpy")
     else:
         print("Latest version is installed.")
diff --git a/msticpy/common/pkg_config.py b/msticpy/common/pkg_config.py
@@ -293,7 +293,7 @@ def validate_config(mp_config: Dict[str, Any] = None, config_file: str = None):
     _print_validation_report(mp_errors, mp_warn)
     if mp_errors or mp_warn:
         return mp_errors, mp_warn
-    return None
+    return [], []
 
 
 def _print_validation_report(mp_errors, mp_warn):

diff --git a/msticpy/common/utility.py b/msticpy/common/utility.py
@@ -12,6 +12,7 @@
 import sys
 import uuid
 import warnings
+from enum import Enum
 from pathlib import Path
 from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Union
 
@@ -257,6 +258,13 @@ def check_and_install_missing_packages(
 
     """
     missing_packages = []
+    if isinstance(required_packages, str):
+        if "," in required_packages:
+            required_packages = [
+                req.strip() for req in required_packages.split(",") if req.strip()
+            ]
+        else:
+            required_packages = [required_packages]
     # Check package requirements against installed set
     for req in required_packages:
         pkg_req = pkg_resources.Requirement.parse(req)
@@ -573,3 +581,48 @@ def valid_pyname(identifier: str) -> str:
     if identifier[0].isdigit():
         identifier = f"n_{identifier}"
     return identifier
+
+
+def enum_parse(enum_cls: type, value: str) -> Optional[Enum]:
+    """Try to parse a string value to an Enum member."""
+    if not issubclass(enum_cls, Enum):
+        raise TypeError("Can only be used with classes derived from enum.Enum.")
+    if value in enum_cls.__members__:
+        return enum_cls.__members__[value]
+    val_lc = value.casefold()
+    val_map = {name.casefold(): name for name in enum_cls.__members__}
+    if val_lc in val_map:
+        return enum_cls.__members__[val_map[val_lc]]
+    return None
+
+
+def arg_to_list(arg: Union[str, List[str]], delims=",; ") -> List[str]:
+    """
+    Convert an optional list/str/str with delims into a list.
+
+    Parameters
+    ----------
+    arg : Union[str, List[str]]
+        A string, delimited string or list
+    delims : str, optional
+        The default delimiters to use, by default ",; "
+
+    Returns
+    -------
+    List[str]
+        List of string components
+
+    Raises
+    ------
+    TypeError
+        If `arg` is not a string or list
+
+    """
+    if isinstance(arg, list):
+        return arg
+    if isinstance(arg, str):
+        for char in delims:
+            if char in arg:
+                return [item.strip() for item in arg.split(char)]
+        return [arg]
+    raise TypeError("`arg` must be a string or a list.")