Ianhelle/msticpy1 0 0rc4 2021 04 05 (#150)

* Changed URL formatting so exception URLs use about="_blank" to open in new tab - in exceptions.py. Also switched to using a list for output instead of concatenated string in MsticpyUserError::_repr_html_()

Sourcery change in utility.py
Tidied up formatting of text and updated exception URL in wsconfig.py
nbinit.py:
- Add about="_blank" to URLs in warnings,
- Tidied up some text
- Changed red error to orange warning
- Trapped seaborn import error if not installed
- Print out list of imported packages by default
- Filtering warnings from call to WorkspaceConfig.py

* Added additional extra for KqlmagicCustom to install pyperclip

Bump version to rc5
az_connect defaults to all auth_methods if None supplied in params or config - azure_auth.py
azure_auth_core.py - added default_auth_methods() function
keyvault_settings.py - default to using all auth_methods
provider_settings.py - do not instantiate SecretSettings unless KeyVault config has settings
check_and_install_missing_packages - run "python -m pip" rather than just pip
wsconfig.py:
- Changed wording of wanings/error messages
- Does not now raise an exception if no suitable config found
- Does not raise a Python warning if we rely on searching to find a config file.
Changing default entity graph plot size in draw_entity_alert_graph. plot_entity_graph shows the plot by default unless hide=True parameter - nbdisplay.py
nbinit.py
- Changed all output so that it is HTML text vs. print
- Captured output from called functions that print output
- Minor rewording and changing header size of titles.
security_alert_graph.py - check if File entity has FullPath attribute before accessing it.
geoip.py - updating wording in exceptions if no API keyvault_settings
tilookup.py - added raise Config exception with help URIs if no providers are enabled - in lookup_ioc, lookup_iocs
Added moz_sql_parser back to mypy.ini
Changed test_pkg_config.py so that it gives clearer message when test fails - added "KqlmagicCustom[jupyter-extended]" to conda exceptions
Updated test_pkg_config.py to account for WorkspaceConfig no longer produces and error.

* Missed capturing warning output from WorkspaceConfig in nbinit.py

* If any warning from WorkspaceConfig, print this out - nbinit.py

* If running in IPython environment check_and_install_missing_packages() will using "pip" magic rather than subprocess in utility.py

Suppressing warnings in Kqlmagic load in kql_driver.py
File entity "FullPath" generation now handles None values for directory and separator more gracefully in file.py
Changed color attribute extraction so that it defaults to a color even if the node has no color attribute - in nbdisplay.py
Handle inter-entity references in child entities in security_alert.py
Handle no "Name" attribute in account entities.
Added additional test cases for nested entity references in test_security_alert.py

* Linux bug - Error loading secretsettings in env that doesn't support Keyring. Fixed this to test Keyring before trying to load. Also fixed logic to honor settings (subject to above check) - previously the param defaulted to True so would also load even if settings were = False.

* Minor Fixes

* Updating version to 1.0.0

Fixing test_security_alert.py unit test
Adding small test for KeyringClient - test_provider_secrets.py

* roll back DF boolean changes.

* Adding MC0001 Mcabe suppression to nbinit.py # 400

Fixing typo in DF name in syslog_utils.py

* Handling cases where empty data set is passed to timeline functions.

Fixing occasional race condition in testing with KeyringClient.is_keyring_available

* Updating KqlmagicCustom version

Sorcery fixes in syslog_utils.py

Co-authored-by: Pete Bryan <peter.bryan@microsoft.com>

conda/conda-reqs-pip.txt

@@ -10,7 +10,7 @@ azure-storage-blob>=12.5.0
 geoip2>=2.9.0
 html5lib
 ipwhois>=1.1.0
-KqlmagicCustom[jupyter-extended]>=0.1.114.dev25
+KqlmagicCustom[jupyter-basic,auth_code_clipboard]>=0.1.114.dev26
 moz_sql_parser>=4.5.0,<=4.11.21016
 splunk-sdk>=1.6.0
 tldextract>=2.2.2

msticpy/_version.py

@@ -1,2 +1,2 @@
 """Version file."""
-VERSION = "1.0.0rc4"
+VERSION = "1.0.0"

msticpy/common/azure_auth.py

@@ -11,23 +11,23 @@
 from azure.mgmt.subscription import SubscriptionClient
 
 from .._version import VERSION
-from .azure_auth_core import AzCredentials, az_connect_core
+from .azure_auth_core import AzCredentials, az_connect_core, default_auth_methods
 from .provider_settings import get_provider_settings
 
 __version__ = VERSION
 __author__ = "Pete Bryan"
 
 
 def az_connect(
-    auth_methods: List = None,
+    auth_methods: List[str] = None,
     silent: bool = False,
 ) -> AzCredentials:
     """
     Connect to Azure SDK/API.
 
     Parameters
     ----------
-    auth_methods : List, optional
+    auth_methods : List[str], optional
         List of authentication methods to try
         Possible options are:
         - "env" - to get authentication details from environment varibales
@@ -55,10 +55,11 @@ def az_connect(
     # If using env options try to load from msticpy
     data_provs = get_provider_settings(config_section="DataProviders")
     az_cli_config = data_provs.get("AzureCLI")
+    auth_methods = auth_methods or default_auth_methods()
     if az_cli_config and az_cli_config.args:
-        if auth_methods is None and "auth_methods" in az_cli_config.args:
+        if "auth_methods" in az_cli_config.args:
             auth_methods = az_cli_config.args.get("auth_methods")
-        if auth_methods is not None and "env" in auth_methods:
+        if isinstance(auth_methods, list) and "env" in auth_methods:
             os.environ["AZURE_CLIENT_ID"] = az_cli_config.args.get("clientId") or ""
             os.environ["AZURE_TENANT_ID"] = az_cli_config.args.get("tenantId") or ""
             os.environ["AZURE_CLIENT_SECRET"] = (

msticpy/common/azure_auth_core.py

@@ -31,8 +31,8 @@
 _AUTH_OPTIONS = {
     "env": EnvironmentCredential(),
     "cli": AzureCliCredential(),
-    "interactive": InteractiveBrowserCredential(),
     "msi": ManagedIdentityCredential(),
+    "interactive": InteractiveBrowserCredential(),
 }
 
 
@@ -83,7 +83,7 @@ def az_connect_core(
 
     """
     if not auth_methods:
-        auth_methods = ["env", "cli", "msi", "interactive"]
+        auth_methods = default_auth_methods()
     try:
         auths = [_AUTH_OPTIONS[meth] for meth in auth_methods]
     except KeyError as err:
@@ -134,3 +134,8 @@ def _filter_all_warnings(record) -> bool:
         if ".get_token" in message:
             return not message
     return True
+
+
+def default_auth_methods() -> List[str]:
+    """Get the default (all) authentication options."""
+    return list(_AUTH_OPTIONS)

msticpy/common/exceptions.py

@@ -7,8 +7,8 @@
 from typing import List, Tuple, Union
 
 from IPython.display import display
-from IPython import get_ipython
 
+from .utility import is_ipython
 from .._version import VERSION
 
 __version__ = VERSION
@@ -94,8 +94,7 @@ def __init__(
         title = kwargs.pop("title", "we've hit an error while running")
         self._output.append((f"{self.__class__.__name__} - {title}", "title"))
 
-        for arg in args:
-            self._output.append(arg)
+        self._output.extend(args)
 
         self._output.append("\nFor more help on fixing this error see:")
         if not help_uri:
@@ -137,25 +136,27 @@ def _repr_html_(self):
         </style>
         """
         div_tmplt = "<div class='solid'>{content}</div>"
-        content = ""
+        about_blank = "target='_blank' rel='noopener noreferrer'"
+        content = []
         for line in self._output:
             if isinstance(line, tuple):
                 l_content, l_type = line
                 if l_type == "title":
-                    content += f"<h3><p class='title'>{l_content}</p></h3>"
+                    content.append(f"<h3><p class='title'>{l_content}</p></h3>")
                 elif l_type == "uri":
                     if isinstance(l_content, tuple):
                         name, uri = l_content
                     else:
                         name = uri = l_content
-                    content += (
-                        f"<ul class='circle'><li><a href='{uri}'>{name}</a></li></ul>"
+                    content.append(
+                        f"<ul class='circle'><li><a href='{uri}' {about_blank}>"
+                        f"{name}</a></li></ul>"
                     )
             else:
                 text_line = line.replace("\n", "<br>")
-                content += f"{text_line}<br>"
+                content.append(f"{text_line}<br>")
 
-        return "".join((ex_style, div_tmplt.format(content=content)))
+        return "".join((ex_style, div_tmplt.format(content="".join(content))))
 
     def _display_txt_exception(self):
         """Display text-only version of the exception text."""
@@ -201,10 +202,7 @@ def __init__(
             + " the MSTICPYCONFIG environment variable.",
             "Or ensure that a copy of this file is in the current directory.",
         ]
-        if args:
-            add_args = [*args, *mp_loc_mssg]
-        else:
-            add_args = [def_mssg, *mp_loc_mssg]
+        add_args = [*args, *mp_loc_mssg] if args else [def_mssg, *mp_loc_mssg]
         if help_uri:
             uri: Union[Tuple[str, str], str] = help_uri
             add_uris = {"basehelp_uri": self.DEF_HELP_URI}
@@ -388,17 +386,3 @@ class MsticpyAzureConnectionError(MsticpyUserError):
         "https://msticpy.readthedocs.io/en/latest/data_acquisition/AzureData.html"
         + "#instantiating-and-connecting-with-an-azure-data-connector",
     )
-
-
-def is_ipython() -> bool:
-    """
-    Return True if running in IPython environment.
-
-    Returns
-    -------
-    bool
-        True if running in IPython environment,
-        otherwise False
-
-    """
-    return bool(get_ipython())

msticpy/common/keyvault_settings.py

@@ -10,6 +10,7 @@
 
 from .._version import VERSION
 from . import pkg_config as config
+from .azure_auth_core import default_auth_methods
 from .exceptions import MsticpyKeyVaultConfigError
 from .utility import export
 
@@ -86,7 +87,9 @@ def __init__(self):
             az_cli = config.get_config("DataProviders.AzureCLI")
         except KeyError:
             az_cli = {}
-        self.auth_methods = az_cli.get("Args", {}).get("auth_methods")
+        self.auth_methods = az_cli.get("Args", {}).get(
+            "auth_methods", default_auth_methods()
+        )
 
         if "authority_uri" in self:
             rev_lookup = {uri.casefold(): code for code, uri in self.AAD_AUTHORITIES}

msticpy/common/provider_settings.py

@@ -56,7 +56,7 @@ class ProviderSettings:
 
 
 _SECRETS_CLIENT: Any = None
-if "KeyVault" in config.settings and _SECRETS_ENABLED:
+if "KeyVault" in config.settings and config.settings["KeyVault"] and _SECRETS_ENABLED:
     _SECRETS_CLIENT = SecretsClient()
 
 

msticpy/common/secret_settings.py

@@ -4,12 +4,13 @@
 # license information.
 # --------------------------------------------------------------------------
 """Settings provider for secrets."""
+import random
 import re
 from functools import partial
 from typing import Any, Callable, Dict, Optional, Set, Tuple
 
 import keyring
-from keyring.errors import KeyringError, KeyringLocked
+from keyring.errors import KeyringError, KeyringLocked, NoKeyringError
 
 from .._version import VERSION
 from . import pkg_config as config
@@ -33,7 +34,7 @@ def __init__(self, name: str = "key-cache", debug: bool = False):
         Parameters
         ----------
         name : str, optional
-            Name of the credential group, by default "system"
+            Name of the credential group, by default "key-cache"
         debug : bool, optional
             Output debug info, by default False
 
@@ -96,12 +97,37 @@ def set_secret(self, secret_name: str, secret_value: Any):
         self._secret_names.add(secret_name)
         keyring.set_password(self.keyring, secret_name, secret_value)
 
+    @staticmethod
+    def is_keyring_available() -> bool:
+        """
+        Test if valid keyring backend is available.
+
+        Returns
+        -------
+        bool
+            True if Keyring has a usable backend, False if not.
+
+        """
+        char_list = list("abcdefghijklm1234567890")
+        random.shuffle(char_list)
+        test_value = "".join(char_list)
+        try:
+            keyring.set_password("test", test_value, test_value)
+            # If no exception clear the test key
+            try:
+                keyring.delete_password("test", test_value)
+            except keyring.errors.PasswordDeleteError:
+                pass
+            return True
+        except NoKeyringError:
+            return False
+
 
 @export
 class SecretsClient:
     """Secrets client - manages keyvault and keyring secrets."""
 
-    def __init__(self, tenant_id: str = None, use_keyring: bool = True):
+    def __init__(self, tenant_id: str = None, use_keyring: bool = False):
         """
         Initialize SecretsClient instance.
 
@@ -110,7 +136,7 @@ def __init__(self, tenant_id: str = None, use_keyring: bool = True):
         tenant_id : str, optional
             TenantID, by default None
         use_keyring : bool, optional
-            If True use keyring to cache secrets, by default True
+            If True use keyring to cache secrets, by default False
 
         Raises
         ------
@@ -134,6 +160,7 @@ def __init__(self, tenant_id: str = None, use_keyring: bool = True):
         self.kv_secret_vault: Dict[str, str] = {}
         self.kv_vaults: Dict[str, BHKeyVaultClient] = {}
         self._use_keyring = use_keyring or self._kv_settings.get("UseKeyring", False)
+        self._use_keyring = self._use_keyring and KeyringClient.is_keyring_available()
         if self._use_keyring:
             self._keyring_client = KeyringClient("Providers")
 

msticpy/common/utility.py

@@ -229,7 +229,7 @@ def resolve_pkg_path(part_path: str):
 def check_and_install_missing_packages(
     required_packages: List[str],
     force_notebook: bool = False,
-    user: bool = True,
+    user: bool = False,
     upgrade: bool = False,
 ) -> bool:
     """
@@ -246,7 +246,7 @@ def check_and_install_missing_packages(
         by default False (autodetect)
     user : bool, optional
         Boolean value to toggle user flag while installing pip packages,
-        by default True
+        by default False
     upgrade: bool, option
         If true supply `--upgrade` flag to pip to install the latest
         version (applies to all package in `required_packages`)
@@ -285,25 +285,28 @@ def check_and_install_missing_packages(
     else:
         pkgbar = tqdm(missing_packages, desc="Installing...", unit="bytes")
 
-    pkg_command = ["pip", "install"]
+    pkg_command = ["install"] if is_ipython() else ["python", "-m", "pip", "install"]
     if user:
         pkg_command.append("--user")
     if upgrade:
         pkg_command.append("--upgrade")
     pkg_success = True
     for package in pkgbar:
-        try:
-            subprocess.run(  # nosec
-                pkg_command + [package],
-                check=True,
-                stdout=subprocess.PIPE,
-                stderr=subprocess.PIPE,
-            )
-        except subprocess.CalledProcessError as proc_err:
-            print(f"An Error has occured while installing {package}.")
-            print(f"Output: {str(proc_err.stdout)}")
-            print(f"Errs: {str(proc_err.stderr)}")
-            pkg_success = False
+        if is_ipython():
+            get_ipython().run_line_magic("pip", " ".join(pkg_command + [package]))
+        else:
+            try:
+                subprocess.run(  # nosec
+                    pkg_command + [package],
+                    check=True,
+                    stdout=subprocess.PIPE,
+                    stderr=subprocess.PIPE,
+                )
+            except subprocess.CalledProcessError as proc_err:
+                print(f"An Error has occured while installing {package}.")
+                print(f"Output: {str(proc_err.stdout)}")
+                print(f"Errs: {str(proc_err.stderr)}")
+                pkg_success = False
         print(f"{package} installed.")
 
     return pkg_success
@@ -351,7 +354,7 @@ def md(
         else:
             style_str = _F_STYLES.get(styles, "")
     if isinstance(styles, list):
-        style_str = ";".join([_F_STYLES.get(style, "") for style in styles])
+        style_str = ";".join(_F_STYLES.get(style, "") for style in styles)
     content = HTML(f"<p style='{style_str}'>{string}</p>")
 
     if isinstance(disp_id, bool) and disp_id: