tutorials-and-examples/training-notebooks/Training - MSTICPy Training 1221.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# MSTICPy - Microsoft Threat Intelligence Center Jupyter & Python Security Tools\n",
        "\n",
        "msticpy is a library for InfoSec investigation and hunting in Jupyter Notebooks. It includes functionality to:\n",
        "- query log data from multiple sources\n",
        "- enrich the data with Threat Intelligence, geolocations and Azure resource data\n",
        "- extract Indicators of Activity (IoA) from logs and unpack encoded data\n",
        "- perform sophisticated analysis such as anomalous session detection and time series decomposition\n",
        "- visualize data using interactive timelines, process trees and multi-dimensional Morph Charts\n",
        "\n",
        "It also includes some time-saving notebook tools such as widgets to set query time boundaries, select and display items from lists, and configure the notebook environment.\n",
        "\n",
        "Source Code: https://github.com/microsoft/msticpy\n",
        "Python Package: https://pypi.org/project/msticpy/#:~:text=Microsoft%20Threat%20Intelligence%20Python%20Security%20Tools.%20msticpy%20is,functionality%20to%3A%20query%20log%20data%20from%20multiple%20sources\n",
        "Docs: https://msticpy.readthedocs.io/en/latest/"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Why use MSTICPy?\n",
        "\n",
        "Libraries such as MSTICPy include a wide range of functionality that you might want to use in a notebook, and make them avaliable in a easy to access way. This saves you significant time in writing code, identifying how specific APIs work, and coverting data so that it works between functions/services. \n",
        "Whilst there are other libraries that can do *some* of what MSTICPy does, MSTICPy provides all of these features in one place, with a integrated datamodel and configuration."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"color: Black; background-color: Red; border: solid; padding: 5pt;\"><b>\r\n",
        "Note:</b> This notebook has deliberate errors in it for the purpose of teaching how to troubleshoot them. Executing the notebook as is will fail.\r\n",
        "</div>"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Installing and importing pacakges in Python\n",
        "\n",
        "To use any library in Python you first need to install the pacakge and import it.\n",
        "There are several ways to do this depending on how you want to access the library, however the simplest and easiest is using pip. [Pip](https://pypi.org/project/pip/) is the pacakge installer for Python and makes finding and installing Python pacakges simple.\n",
        "You can use pip to install packages via the command line, or if you are using a notebook, directly in a notebook cell. Azure ML compute come with Pip installed already but if you are running your notebook elsewhere you may need to install pip first.\n",
        "\n",
        "To do this we need to use `%pip` followed by install and the pacakge name. e.g.:\n",
        "`%pip install requests`\n",
        "\n",
        "<div style=\"color: Black; background-color: Khaki; border: solid; padding: 5pt;\"><b>\n",
        "Note:</b> `%pip` is whats called a magic function in Jupyter. This tells the notebook to use pip to install the package in the notebooks compute environment.\n",
        "</div>"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "%pip install requests"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "%pip install requests==2.2"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you have a package installed but you want to update it to the latest version you can add the `--upgrade` parameter"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "%pip install requests --upgrade"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"color: Black; background-color: Khaki; border: solid; padding: 5pt;\"><b>\n",
        "Note:</b> Once you have installed a pacakge its a good idea to restart the kernel, this will ensure that when you import the package you will be using the latest version.\n",
        "</div>\n",
        "\n",
        "<div style=\"color: Black; background-color: skyblue; border: solid; padding: 5pt;\"><b>\n",
        "Note:</b> During installation of pacakges you may see some warnings related to pacakge dependency, this is due to the fact that some packages have requirements on other pacakges being installed and something these requirements can clash (i.e. package 1 requires package A version 1.1 but pacakge 2 also requires package A but version 1.2). Often these warnings do not cause significant issues so attempt to run the notebook and see if it can execute correctly.\n",
        "</div>\n",
        "\n",
        "![Example error message](1.png)\n",
        "\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Importing\n",
        "\n",
        "Once a package has been installed you need to import some or all of it.\n",
        "\n",
        "This is done with the `import` statement.\n",
        "\n",
        "Generally there are 2 ways to import things in Python:\n",
        "- `import <package>` - this imports everything in the pacakge\n",
        "- `from <package> import <item>` - this imports a specific item from the package\n",
        "\n",
        "You can also import pacakages and rename them for ease when calling them later:\n",
        "`import <pacakage> as <alias>`\n",
        "e.g. `import pandas as pd`"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1634829700253
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import xyz"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%pip list"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Some packages do not use the same name for installation and import. You many need to check package documentation to ensure you are improting correctly."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%pip install scikit-learn"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import sklearn"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634829618093
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Installing and Importing MSTICPy\n",
        "Now that we have seen the fundamentals of installing and importing lets install and import MSTICPy:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Install the latest version of MSTICPy\n",
        "%pip install msticpy --upgrade"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1634829744932
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Don't forget to restart that kernel!\r\n",
        "\r\n",
        "No we could import MSTICPy as a whole with `import msticpy` however its a big pacakge with a lot of features, so to make it easier we have a function called `nbinit` that conducts a number of checks to make sure the environment is good, handles key imports and set up for us."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from msticpy.nbtools import nbinit\r\n",
        "nbinit.init_notebook(\r\n",
        "    namespace=globals()\r\n",
        ")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1634829888354
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"color: Black; background-color: Green; color: white; padding: 5pt;\"><b>\r\n",
        "Great! we are now ready to get going.</b>\r\n",
        "</div>\r\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## MSTICPy's config file\r\n",
        "\r\n",
        "MSTICPy can handle connections to a variety of data sources and services, including Azure Sentinel.\r\n",
        "\r\n",
        "To make it easier to manage and re-use the configuration and credentials fo these things MSTICPy has its own config file that holds these items - `msticpyconfig.yaml`\r\n",
        "\r\n",
        "When you launched this notebook from Azure Sentinel it copied a basic configuration file - `config.json` -\r\n",
        "to your workspace folder.<br>\r\n",
        "You should be able to see this file in the file browser to the left.<br>\r\n",
        "This file contains details about your Azure Sentinel workspace but has\r\n",
        "no configuration settings for other external services that we need.\r\n",
        "\r\n",
        "If you didn't have a `msticpyconfig.yaml` file in your workspace folder (which is likely\r\n",
        "if this is your first use of notebooks), the `init_notebook` function should have created\r\n",
        "one for you and populated it\r\n",
        "with the Azure Sentinel workspace data taken from your config.json.\r\n",
        "\r\n",
        "<p style=\"border: solid; padding: 5pt; color: white; background-color: DarkOliveGreen\"><b>Tip:</b>\r\n",
        "If you do not see a \"msticpyconfig.yaml\" file in your user folder, click the refresh button<br>\r\n",
        "at the top of the file browser.\r\n",
        "</p>\r\n",
        "\r\n",
        "We can check this now by opening the settings editor and view the settings.\r\n",
        "\r\n",
        "<div style=\"color: Black; background-color: Khaki; border: solid; padding: 5pt;\"><b>\r\n",
        "You should not have to change anything here unless you need to add\r\n",
        "one or more additional workspaces.</b></div>\r\n",
        "<p/>\r\n",
        "\r\n",
        "When you have verified that this looks OK. Click **Save Settings**\r\n",
        "\r\n"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from msticpy.config import MpConfigEdit\r\n",
        "import os\r\n",
        "\r\n",
        "mp_conf = \"msticpyconfig.yaml\"\r\n",
        "\r\n",
        "# check if MSTICPYCONFIG is already an env variable\r\n",
        "mp_env = os.environ.get(\"MSTICPYCONFIG\")\r\n",
        "mp_conf = mp_env if mp_env and Path(mp_env).is_file() else mp_conf\r\n",
        "\r\n",
        "if not Path(mp_conf).is_file():\r\n",
        "    print(\r\n",
        "        \"No msticpyconfig.yaml was found!\",\r\n",
        "        \"Please check that there is a config.json file in your workspace folder.\",\r\n",
        "        \"If this is not there, go back to the Azure Sentinel portal and launch\",\r\n",
        "        \"this notebook from there.\",\r\n",
        "        sep=\"\\n\"\r\n",
        "    )\r\n",
        "else:\r\n",
        "    mpedit = MpConfigEdit(mp_conf)\r\n",
        "    mpedit.set_tab(\"AzureSentinel\")\r\n",
        "    display(mpedit)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634830729428
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We are going to use [VirusTotal](https://www.virustotal.com) (VT) as an example of a popular threat intelligence source.\r\n",
        "To use VirusTotal threat intel lookups you will need a VirusTotal account and API key.\r\n",
        "\r\n",
        "You can sign up for a free account at the\r\n",
        "[VirusTotal getting started page](https://developers.virustotal.com/v3.0/reference#getting-started) website.\r\n",
        "\r\n",
        "If you are already a VirusTotal user, you can, of course, use your existing key.\r\n",
        "\r\n",
        "<p style=\"border: solid; padding: 5pt; color: black; background-color: Khaki\">\r\n",
        "<b>Warning</b> If you are using a VT enterprise key we do not recommend storing this\r\n",
        "in the msticpyconfig.yaml file.<br>\r\n",
        "MSTICPy supports storage of secrets in\r\n",
        "Azure Key Vault. You can read more about this\r\n",
        "<a href=https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html#specifying-secrets-as-key-vault-secrets >in the MSTICPY docs</a><br>\r\n",
        "For the moment, you can sign up for a free acount, until you can take the time to\r\n",
        "set up Key Vault storage.\r\n",
        "</p>\r\n",
        "\r\n",
        "\r\n",
        "As well as VirusTotal, we also support a range\r\n",
        "of other threat intelligence providers: https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html\r\n",
        "<br><br>\r\n",
        "\r\n",
        "To add the VirusTotal details, run the following cell.\r\n",
        "\r\n",
        "1. Select \"VirusTotal\" from the **Add prov** drop down\r\n",
        "2. Click the **Add** button\r\n",
        "3. In the left-side Details panel select **Text** as the Storage option.\r\n",
        "4. Paste the API key in the **Value** text box.\r\n",
        "5. Click the **Update** button to confirm your changes.\r\n",
        "\r\n",
        "Your changes are not yet saved to your configuration file. To\r\n",
        "do this, click on the **Save Settings** button at the bottom of the dialog.\r\n",
        "\r\n",
        "If you are unclear about what anything in the configuration editor means, use the **Help** drop-down. This\r\n",
        "has instructions and links to more detailed documentation."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "mpedit.set_tab(\"TI Providers\")\r\n",
        "mpedit"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634593482751
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Our notebooks commonly use IP geo-location information. \r\n",
        "In order to enable this we are going to set up [MaxMind GeoLite2](https://www.maxmind.com)\r\n",
        "to provide geolocation lookup services for IP addresses.\r\n",
        "\r\n",
        "GeoLite2 uses a downloaded database which requires an account key to download.\r\n",
        "You can sign up for a free account and a license key at \r\n",
        "[The Maxmind signup page - https://www.maxmind.com/en/geolite2/signup](https://www.maxmind.com/en/geolite2/signup).\r\n",
        "<br>\r\n",
        "\r\n",
        "<details>\r\n",
        "    <summary>Using IPStack as an alernative to GeoLite2...</summary>\r\n",
        "    <p>\r\n",
        "    For more details see the\r\n",
        "    <a href=https://msticpy.readthedocs.io/en/latest/data_acquisition/GeoIPLookups.html >\r\n",
        "    MSTICPy GeoIP Providers documentation</a>\r\n",
        "    </p>\r\n",
        "</details>\r\n",
        "<br>\r\n",
        "\r\n",
        "Once, you have an account, run the following cell to add the Maxmind GeopIP Lite details to your configuration.\r\n",
        "\r\n",
        "The procedure is similar to the one we used for VirusTotal:\r\n",
        "\r\n",
        "1. Select the \"GeoIPLite\" provider from the **Add prov** drop-down\r\n",
        "2. Click **Add**\r\n",
        "3. Select **Text** Storage and paste the license (API/Auth) key into the text box\r\n",
        "4. Click **Update**\r\n",
        "5. Click **Save Settings** to write your settings to your configuration.\r\n"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "mpedit.set_tab(\"GeoIP Providers\")\r\n",
        "mpedit"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634606266533
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Validate your settings\r\n",
        "\r\n",
        "- click on the **Validate settings** button.\r\n",
        "\r\n",
        "You may see some warnings about missing sections but not about the Azure Sentinel, TIProviders or GeoIP Providers settings.\r\n",
        "\r\n",
        "Click on the **Close** button to hide the validation output.\r\n",
        "\r\n",
        "If you need to make any changes as a result of the Validation,\r\n",
        "remember to save your changes by clicking the **Save File** button."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "msticpy.settings.refresh_config()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634830804320
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Getting Data From Azure Sentinel\r\n",
        "\r\n",
        "Now that the setup is out the way we want to focus on "
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!az login"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": true
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Querying data from Azure Sentinel is handled by MSTICPy's `QueryProvider`. The first step is to initalize a QueryProvider and tell it we want to use the Azure Sentinel Query provider.\r\n",
        "\r\n",
        "The other thing we want to provide the QueryProvider with is some details of the workspace we want to connect to. We *could* do this manually, but its much easier to get details from the configuration we set up earlier. We can do this with `WorkspaceConfig`"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from msticpy.nbtools import nbinit\r\n",
        "nbinit.init_notebook(namespace=globals())\r\n",
        "\r\n",
        "qry_prov=QueryProvider(\"AzureSentinel\")\r\n",
        "ws_config = WorkspaceConfig(workspace=\"CyberSecDemo\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634830639779
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "What `WorkspaceConfig` is doing for is is creating the connection string used by the `QueryProvider`:"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ws_config.code_connect_str"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634830406771
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Once set up we can tell the `QueryProvider` to `connect` which will kick off the authentication process. There are a number of ways that we can handle that authentication."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#qry_prov.connect(ws_config)\r\n",
        "qry_prov.connect(ws_config, mp_az_auth=\"cli\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634830816257
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that we are connected to Azure Sentinel we can start to look at running some queries to get some data.\r\n",
        "\r\n",
        "MSTICPy comes with a number of built in Azure Sentinel queries to get some common datasets into the Notebook. \r\n",
        "\r\n",
        "You can see a list of the avaliable queries with: `.list_queries`"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "qry_prov.list_queries()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634830909783
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "However this output only has some use. To make these in-built queries more accesible and findable there is a query browser which makes searching for, and learning about, these queries much easier."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "qry_prov.browse_queries()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634830923519
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that we have found a query that we want to run we simply pass its name to the `QueryProvider` and that in turn returns to results of the query in a Pandas DataFrame.\r\n",
        "\r\n",
        "In addition to the stock query we can customize certain elements of the query."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#qry_prov.Azure.list_all_signins_geo()\r\n",
        "#qry_prov.SecurityAlert.list_alerts('?')\r\n",
        "qry_prov.SecurityAlert.list_alerts(add_query_items=\"| take 10\")\r\n",
        "#qry_prov.SecurityAlert.list_alerts(add_query_items=\"take 10\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831224552
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We also don't need to use the built-in queries. We can write our own queries and have then executed using `.exec_query` "
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"SecurityAlert | take 10\"\r\n",
        "#qry_prov.exec_query(query)\r\n",
        "alert_df = qry_prov.exec_query(query)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831422008
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alert_df"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831438128
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Working with the data\r\n",
        "\r\n",
        "Data returned by the `QueryProvider` comes back in a Pandas DataFrame. This provides us with a powerful and flexible way to access our data.\r\n",
        "\r\n",
        "One of the core things we want to do is look at specific rows in our table. Each table has an index that can be used to call a row using `.loc`, alternatively we can return a row by its position in the table with `.iloc`"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alert_df.loc[1]"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831424809
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can also choose just to return specific columns by providing a list of them to the DataFrame:"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alert_df.iloc[:5][[\"AlertName\", \"AlertSeverity\", \"Description\"]]"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831513056
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can also do things such as search for rows with specific data."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alert_df[alert_df[\"AlertName\"].str.contains(\"credential theft\")]"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831533555
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Pandas also has some features to allow you to visualize the data you have:"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alert_df[\"AlertSeverity\"].value_counts().plot(kind='pie')"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831562854
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alert_df[\"AlertSeverity\"].value_counts().plot(kind='bar')"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831568127
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "There are many, many more features in Pandas. When starting with MSTICPy its a good idea to spend some time learning about the power of Pandas - https://pandas.pydata.org/docs/"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Enriching data using external data sources\r\n",
        "\r\n",
        "One of the powerful elements of Notebooks is you can combine data from Azure Sentinel with data from other sources. One of the most common sources of this data in security is Threat Intelligence (TI) data. MSTICPy has a support for a number of Threat Intelligence data sources including:\r\n",
        "- VirtusTotal\r\n",
        "- GreyNoise\r\n",
        "- AlienVault OTX\r\n",
        "- IBM XForce\r\n",
        "- Azure Sentinel TI data\r\n",
        "- OPR (for PageRank details)\r\n",
        "- ToR ExitNode information."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"SigninLogs | sample 100\"\r\n",
        "signin_df = qry_prov.exec_query(query)\r\n",
        "signin_df.head()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831830947
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The first step in using these TI sources is to create a `TILookup` object. This is can then be used to perform lookups.\r\n",
        "\r\n",
        "Lookups can be done against individual items via `.lookup_ioc` or against multiple items with `.lookup_iocs`."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ti = TILookup()\r\n",
        "ti.lookup_iocs(signin_df, obs_col=\"IPAddress\", providers=[\"GreyNoise\"])"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831838524
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ti_hits = ti.lookup_iocs(signin_df, obs_col=\"IPAddress\",providers=[\"GreyNoise\"])\r\n",
        "ti_hits[ti_hits[\"Result\"]==True]"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831844217
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "signin_df.set_index('IPAddress').join(ti_hits[ti_hits[\"Result\"]==True].set_index('Ioc'), rsuffix=\"_\", how=\"inner\")[[\"TimeGenerated\", \"UserPrincipalName\"]]"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831932461
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "vt_df = ti.lookup_iocs(signin_df[\"IPAddress\"].unique()[:4], providers=[\"VirusTotal\"])\r\n",
        "vt_df"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831962774
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ti.browse_results(vt_df)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634831995346
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ti.browse_results(ti.result_to_df(ti.lookup_ioc(\"87.97.178.92\")))"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832039142
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Azure API access\r\n",
        "\r\n",
        "MSTICPy also has integration with a range of Azure APIs that can be used to retrieve additional informaiton or perform actions."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from msticpy.data.azure_sentinel import AzureSentinel\r\n",
        "\r\n",
        "azs = AzureSentinel()\r\n",
        "azs.connect()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832128744
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "subs = azs.get_subscriptions()\r\n",
        "subs.head()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832148561
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "azs.get_subscription_info(subs.iloc[0][\"Subscription ID\"])"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832152738
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "azs.get_incident(incident_id = \"7a4f5e0e-c202-4298-8cb6-e1278500fbc7\", sub_id = \"d1d8779d-38d7-4f06-91db-9cbc8de0176f\", res_grp= \"soc\", ws_name=\"cybersecuritysoc\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832176430
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Visualizations with MSTICPy  \r\n",
        "\r\n",
        "The ability to create complex, interactive visualizations is one of the key benefits of Notebooks. Creating these visulizations from scratch can be quite complex and involve a lot of code. \r\n",
        "\r\n",
        "To make the process easier MSTICPy contains a number of common visualization that can quickly and easily be called with minimal code."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Timelines\r\n",
        "\r\n",
        "Understanding when events occured and in what order is key component of many security investigations. MSTICPy has the ability to plot various types of timelines."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "user_df = qry_prov.Azure.list_aad_signins_for_account(account_name=\"pdemo@seccxpninja.onmicrosoft.com\")\r\n",
        "#timeline.display_timeline(user_df)\r\n",
        "timeline.display_timeline(user_df, source_columns=[\"UserPrincipalName\", \"ResultType\"])"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832373712
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "user_df.columns"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832354206
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "timeline.display_timeline(user_df, source_columns=[\"UserPrincipalName\", \"ResultDescription\"])   \r\n"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634674845002
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ref_time = user_df[\"TimeGenerated\"].iloc[5]\r\n",
        "timeline.display_timeline(user_df, source_columns=[\"UserPrincipalName\", \"ResultDescription\"], group_by=\"ResultType\", ref_time=ref_time)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832394540
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alert_df = qry_prov.SecurityAlert.list_alerts(add_query_items=\"| take 10\")\r\n",
        "alert_df"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832504018
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "timeline_duration.display_timeline_duration(alert_df, group_by=\"AlertName\", time_column=\"StartTimeUtc\", end_time_column=\"EndTimeUtc\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832533401
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#alert_df.mp_plot.timeline()\r\n",
        "alert_df.mp_plot.timeline(group_by=\"Severity\", source_columns=[\"AlertName\", \"TimeGenerated\"])"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832602846
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "MSTICPY also includes a number of interactive widgets that make it easier for users to interact with notebooks."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "network_vendor_data_q = \"CommonSecurityLog | summarize by DeviceVendor\"\r\n",
        "network_vendor_data = qry_prov.exec_query(network_vendor_data_q)\r\n",
        "network_selector = nbwidgets.SelectItem(\r\n",
        "    item_list=network_vendor_data[\"DeviceVendor\"].to_list(),\r\n",
        "    description='Select an vendor',\r\n",
        "    action=print,\r\n",
        "    auto_display=True\r\n",
        ");\r\n"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832653113
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "network_data_q = f\"\"\"CommonSecurityLog \r\n",
        "    | where DeviceVendor == '{network_selector.value}'\r\n",
        "    | take 50\"\"\"\r\n",
        "network_data = qry_prov.exec_query(network_data_q)\r\n",
        "network_data.head()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832769050
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The Matrix Plot graph in MSTICPy allows you to plot the interactions between two elements in your data."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "network_data.mp_plot.matrix(x=\"SourceIP\", y=\"DestinationIP\", title=\"IP Interaction\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832799239
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "q_times = nbwidgets.QueryTime(units='day', max_before=20, before=5, max_after=1)\r\n",
        "q_times.display()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832849660
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(123)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832948332
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "security_alerts = qry_prov.SecurityAlert.list_alerts(add_query_items=\"| take 10\")\r\n",
        "alert_select = nbwidgets.SelectAlert(alerts=security_alerts, action=nbdisplay.display_alert)\r\n",
        "display(Markdown('### Alert selector with action=DisplayAlert'))\r\n",
        "display(HTML(\"<b> Alert selector with action=DisplayAlert </b>\"))\r\n",
        "alert_select.display()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1634832963750
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## What to do next:\r\n",
        "\r\n",
        "Run the Getting Started Notebook in Azure Sentinel\r\n",
        "    - This will help you get your config set up\r\n",
        "\r\n",
        "\t\r\n",
        "Try the MSTICPy Lab – https://aka.ms/msticpy-demo \r\n",
        "\r\n",
        "Go and read the docs – https://msticpy.readthedocs.io/en/latest/GettingStarted.html \r\n",
        "\r\n",
        "Learn more about Pandas - https://pandas.pydata.org/docs/ \r\n",
        "\r\n",
        "Check out our other notebooks for ideas! - https://github.com/Azure/Azure-Sentinel-Notebooks \r\n",
        "\r\n",
        "\r\n"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    }
  ],
  "metadata": {
    "interpreter": {
      "hash": "70673dbb0e081aa3831869ef2042eea6ce08ea300fe5aabb70c669fac0ce2a09"
    },
    "kernelspec": {
      "name": "python3",
      "language": "python",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.9",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "orig_nbformat": 4,
    "kernel_info": {
      "name": "python3"
    },
    "microsoft": {
      "host": {
        "AzureML": {
          "notebookHasBeenCompleted": true
        }
      }
    },
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}