Iputils (#34)

* ip utils and heartbeat queries

* fix pylint and import errors

* fix pylint errors and kql heartbeat queries

* missing project columns in heartbeat queries

* adding KQL time series queries

* fix yaml parsing error in timeseries kql

* add scoreanomolies query

* fixes in time series kql queries

* refactor timeseries kql queries

* changes to mv-expand in kql query

* replace queryproject values

* Miscellaneous fixes from notebook testing:

- Query templates
- Doc updates (new doc page on msticpyconfig.yaml)
- Changed param_extractor to always prefer supplied params over defaults
- Several linter/mypy errors
- wsconfig throws meaningful error if config values are not found
- tilookup fix - exception thrown if an empty IoCs list sent to it
- geoip - fixed multiple problems with the DF lookup version of the API

* pkg and whois function addition

* Typo in wsconfig.py

* logic change to check for missing packages

* added tqdm dependency

* fix black formatting and add ipwhois dependency

* pylint warning fix

* fixing more pylint warnings

* user option for missing package installation

* docstring update

* Updated Pandas requirement

* Updates to version and requirements/setup.py

* Merge fixes

* Linting warnings in ip_utils

* ip_utils linting fixes (post-black)

docs/source/TIProviders.rst

@@ -54,6 +54,9 @@ multiple observables. Processing may require a an API key and processing
 performance may be limited to a specific number of requests per minute
 for the account type that you have.
 
+:py:mod:`TILookup API documentation<msticpy.sectools.tilookup>`
+
+
 
 Constructor
 ~~~~~~~~~~~

docs/source/msticpy.sectools.rst

@@ -61,7 +61,7 @@ msticpy.sectools.outliers module
     :show-inheritance:
 
 msticpy.sectools.syslog_utils module
-----------------------------------
+------------------------------------
 
 .. automodule:: msticpy.sectools.syslog_utils
     :members:

docs/source/msticpyconfig.rst

@@ -0,0 +1,108 @@
+
+msticpy Package Configuration
+=============================
+
+Some elements of msticpy require configuration parameters. An
+example is the Threat Intelligence providers. Values for these
+and other parameters can be set in the `msticpyconfig.yaml` file.
+
+The package has a default configuration file, which is stored in the
+package directory. You should not need to edit this file directly.
+Instead you can create a custom file with your own parameters - these
+settings will combine with or override the settings in the default file.
+
+By default, the custom `msticpyconfig.yaml` is read from the current
+directory. You can specify an explicit location using an environment
+variable ``MSTICPYCONFIG``.
+
+Configuration sections
+----------------------
+
+AzureSentinel
+~~~~~~~~~~~~~
+Here you can specify your default workspace and tenant IDs and add additional
+workspaces if needed.
+
+QueryDefinitions
+~~~~~~~~~~~~~~~~
+This allows you to specify paths to additional yaml query template files.
+
+TIProviders
+~~~~~~~~~~~
+This allows you to configure which providers are run by default and to
+supply any authorization keys needed to access the service.
+
+Comment configuration file sample
+---------------------------------
+
+
+.. code:: yaml
+
+    AzureSentinel:
+      Workspaces:
+        # Workspace used if you don't explicitly name a workspace when creating WorkspaceConfig
+        # Specifying values here overrides config.json settings unless you explictly load
+        # WorkspaceConfig with config_file parameter (WorkspaceConfig(config_file="../config.json")
+        Default:
+          WorkspaceId: "d973e3d2-28e6-458e-b2cf-d38876fb1ba4"
+          TenantId: "4cdf87a8-f0fc-40bb-9d85-68bcf4ac8e61"
+        # To use these launch with an explicit name - WorkspaceConfig(workspace_name="Workspace2")
+        Workspace2:
+          WorkspaceId: "c88dd3c2-d657-4eb3-b913-58d58d811a41"
+          TenantId: "f1f64e65-ff7c-4d71-ad5b-091b6ab39d51"
+        Workspace3:
+          WorkspaceId: "17e64332-19c9-472e-afd7-3629f299300c"
+          TenantId: "4ea41beb-4546-4fba-890b-55553ce6003a"
+    QueryDefinitions:
+      # Add paths to folders containing custom query definitions here
+      Custom:
+        - /var/global-queries
+        - /home/myuser/queries
+        - c:/users/myuser/documents
+    TIProviders:
+      # If a provider has Primary: True it will be run by default on IoC lookups
+      # Secondary providers can be
+      OTX:
+        Args:
+          AuthKey: "4ea41beb-4546-4fba-890b-55553ce6003a"
+        Primary: True
+        Provider: "OTX" # WARNING - Do not change Provider values!
+      VirusTotal:
+        Args:
+          AuthKey: "4ea41beb-4546-4fba-890b-55553ce6003a"
+        Primary: False
+        Provider: "VirusTotal"
+      XForce:
+        # You can store items in an environment variable using this syntax
+        Args:
+          ApiID:
+            EnvironmentVar: "XFORCE_ID"
+          AuthKey:
+            EnvironmentVar: "XFORCE_KEY"
+        Primary: True
+        Provider: "XForce"
+      AzureSentinel:
+        # Note this can be a different workspace/tenant from your main workspace
+        # This only controls where the Azure Sentinel TI provider looks for the
+        # ThreatIndicator table.
+        Args:
+          WorkspaceID: "c88dd3c2-d657-4eb3-b913-58d58d811a41"
+          TenantID: "f1f64e65-ff7c-4d71-ad5b-091b6ab39d51"
+        Primary: True
+        Provider: "AzSTI"
+      OpenPageRank:
+        Args:
+          AuthKey: "c88dd3c2-d657-4eb3-b913-58d58d811a41"
+        Primary: False
+        Provider: "OPR"
+      TorExitNodes:
+        Primary: True
+        Provider: "Tor"
+
+
+See also
+--------
+
+:doc:`The Threat Intelligence Providers documention <TIProviders>`
+
+:py:mod:`msticpy.nbtools.wsconfig`

docs/source/otherdocs.rst

@@ -14,3 +14,5 @@ Other Documents
 
     EventTimeline.rst
 
+    msticpyconfig.rst
+

msticpy/_version.py

@@ -1,2 +1,2 @@
 """Version file."""
-VERSION = "0.2.6"
+VERSION = "0.2.7"

msticpy/data/param_extractor.py

@@ -4,7 +4,7 @@
 # license information.
 # --------------------------------------------------------------------------
 """Parameter extractor helper functions for use with IPython/Juptyer queries."""
-from typing import List, Dict, Tuple, Any
+from typing import List, Dict, Tuple, Any, Iterable
 
 from .query_store import QuerySource
 from ..nbtools.utility import export
@@ -42,54 +42,57 @@ def extract_query_params(
         List of any missing parameters
 
     """
-    # get the required parameters for this query and build a dictionary
+    # get the parameters for this query
+    all_param_names = query_source.params.keys()
+
+    # required_params are those that don't have defaults set in the query
+    # template. Build a dictionary to hold the values. This will contain
+    # at least the required params plus any that are extracted from args and
+    # kwargs and have been added dynamically.
     req_param_names = query_source.required_params.keys()
     req_params: Dict[str, Any] = {param: None for param in req_param_names}
 
-    # Iterate through required parameters. If any are set in the supplied
-    # provider objects, assign them to our output dictionary
-    query_providers = [prov for prov in args if hasattr(prov, "query_params")]
-    for provider in query_providers:
-        for param in req_param_names:
-            if param in provider.query_params:
-                req_params[param] = provider.query_params[param]
+    # try to retrieve any parameters as attributes of the args objects
+    _get_object_params(args, all_param_names, req_params)
 
-    # If any custom parameters have been supplied add these
-    # overriding any parameters from the QueryParamProviders
+    # If any kwargs parameters have been supplied, add these.
+    # These any parameters obtained from _get_object_params
     if kwargs:
         req_params.update(kwargs)
 
-    # If we have missing parameters try to retrieve them
-    # as attributes of the object
+    # Get the names of any params that were required but we didn't
+    # find a value for
     missing_params = [p_name for p_name, p_value in req_params.items() if not p_value]
-    if missing_params:
-        _get_missing_params(args, missing_params, req_params)
-
     return req_params, missing_params
 
 
-def _get_missing_params(
-    args: Tuple[Any, ...], missing_params: List[str], req_params: Dict[str, Any]
+def _get_object_params(
+    args: Tuple[Any, ...], param_names: Iterable[str], req_params: Dict[str, Any]
 ):
     """
-    Get missing params from arguments.
+    Get params from attributes of arg objects.
 
     Parameters
     ----------
     args : Tuple[Any]
-        Args list from calling funtion
-    missing_params : List[str]
-        The list of missing parameters to get
+        Args list from calling function
+    param_names : Iterable[str]
+        The list of parameter names to look for
     req_params : Dict[str, str]
         Dictionary of required parameters
 
     """
-    for arg_object in [obj for obj in args if not isinstance(obj, QueryParamProvider)]:
-        for m_param in missing_params:
-            if isinstance(arg_object, dict) and m_param in arg_object:
-                req_params[m_param] = arg_object.get(m_param, None)
-            elif hasattr(arg_object, m_param):
-                req_params[m_param] = getattr(arg_object, m_param)
-        missing_params = [
+    for arg_object in args:
+        if isinstance(arg_object, QueryParamProvider):
+            for param in param_names:
+                if param in arg_object.query_params:
+                    req_params[param] = arg_object.query_params[param]
+        else:
+            for param in param_names:
+                if isinstance(arg_object, dict) and param in arg_object:
+                    req_params[param] = arg_object.get(param, None)
+                elif hasattr(arg_object, param):
+                    req_params[param] = getattr(arg_object, param)
+        param_names = [
             p_name for p_name, p_value in req_params.items() if p_value is not None
         ]

msticpy/data/queries/kql_sent_alert.yaml

@@ -123,7 +123,7 @@ sources:
             and (Entities has src_accountname or Entities has src_acct
             or ExtendedProperties has src_accountname
             or ExtendedProperties has src_acct), true, false)
-        | extend proc_match = iif(isnotempty(src_acct)
+        | extend proc_match = iif(isnotempty(src_proc)
             and (Entities has src_procname or Entities has src_proc
             or ExtendedProperties has src_procname
             or ExtendedProperties has src_proc), true, false)
@@ -161,11 +161,11 @@ sources:
         | where TimeGenerated <= datetime({end})
         | project SystemAlertId, ExtendedProperties, Entities
         | extend source_ips_str =
-            extract("\\"Source IPs\\": \\"([^\\"]+)\\"", 1, ExtendedProperties)
+            extract("\"Source IPs\": \"([^\"]+)\"", 1, ExtendedProperties)
         | extend source_ips_1 =
             iif(isnotempty(source_ips_str), split(source_ips_str, ","), dynamic([]))
         | extend source_ips_2 =
-            extract_all("\\"Address\\": \\"([^\\"]+)\\"", dynamic([1]), Entities)
+            extract_all("\"Address\": \"([^\"]+)\"", dynamic([1]), Entities)
         | mvexpand alert_ip_1 =
             source_ips_1 to typeof(string), alert_ip_2 = source_ips_2 to typeof(string)
         | where isnotempty(alert_ip_1) or isnotempty(alert_ip_2)

msticpy/data/queries/kql_sent_azure.yaml

@@ -29,7 +29,7 @@ sources:
         let accountName = "{account_name}";
         let account = case(
             accountName has "@", tostring(split(accountName, "@")[0]),
-            accountName has "\\\\", tostring(split(accountName, "\\\\")[1]),
+            accountName has "\\", tostring(split(accountName, "\\")[1]),
             accountName
         );
         {table}
@@ -45,6 +45,24 @@ sources:
       account_name:
         description: The account name to find
         type: str
+  list_aad_signins_for_ip:
+    description: Lists Azure AD Signins for an IP Address
+    metadata:
+    args:
+      query: '
+        {table}
+        | where IPAddress in ({ip_address_list})
+        | where TimeGenerated >= datetime({start})
+        | where TimeGenerated <= datetime({end})
+        {add_query_items}'
+    parameters:
+      table:
+        description: Table name
+        type: str
+        default: SigninLogs
+      ip_address_list:
+        description: The IP Address or list of Addresses
+        type: list
   list_azure_activity_for_account:
     description: Lists Azure Activity for Account
     metadata:
@@ -53,7 +71,7 @@ sources:
         let accountName = "{account_name}";
         let account = case(
             accountName has "@", tostring(split(accountName, "@")[0]),
-            accountName has "\\\\", tostring(split(accountName, "\\\\")[1]),
+            accountName has "\\", tostring(split(accountName, "\\")[1]),
             accountName
         );
         {table}
@@ -75,7 +93,7 @@ sources:
     args:
       query: '
         {table}
-        | where CallerIpAddress in ({ip_addresses})
+        | where CallerIpAddress in ({ip_address_list})
         | where TimeGenerated >= datetime({start})
         | where TimeGenerated <= datetime({end})
         {add_query_items}'
@@ -84,7 +102,7 @@ sources:
         description: Table name
         type: str
         default: AzureActivity
-      ip_addresses:
+      ip_address_list:
         description: The IP Address or list of Addresses
         type: list
   list_azure_activity_for_resource:

msticpy/data/queries/kql_sent_azuresentinel.yaml

@@ -55,7 +55,7 @@ sources:
         {query_project}
         | where TimeGenerated >= datetime({start})
         | where TimeGenerated <= datetime({end})
-        | where QueryResultRow has {entity_id}
+        | where QueryResultRow has "{entity_id}"
         | extend QryResults = todynamic(QueryResultRow)
         | extend Computer = QryResults["Computer"]
         | extend Account = QryResults["Account"]