demos/ai/cyber/redteam-umap-gtc-gpu.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "beb5e3e3-f8cd-40ed-bc63-8a862000f192",
   "metadata": {},
   "source": [
    "# Analyzing Network Identity Data and Red Team Response with Graphistry AutoML + UMAP\n",
    "\n",
    "We find a simple model that when clustered in a 2d plane via UMAP allows fast identification of anomalous \n",
    "computer to computer connections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9de6fd3-b87b-4dc4-8d1c-b8f3feceb5e6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ! pip install graphistry[ai] \n",
    "! pip install --user --no-deps git+https://github.com/graphistry/pygraphistry.git@cudf-alex3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0215906c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from joblib import load, dump\n",
    "from collections import Counter\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pylab as plt\n",
    "\n",
    "import graphistry\n",
    "from graphistry.features import topic_model, search_model, ModelDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b34bebd-c91d-49fe-82c9-ec1c83a4a6c1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "graphistry.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e1747b9-c903-4398-9aa0-b52b69fce021",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(137)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d2669fd-6164-4376-81bd-79c6c6f4112f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "RENDER = True # set to True to render Graphistry UI inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59e1cc0b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "graphistry.register(api=3, protocol=\"https\", server=\"hub.graphistry.com\", username = '..',\n",
    "                    #os.environ['USERNAME'], \n",
    "                    password='..'\n",
    "                    #os.environ['GRAPHISTRY_PASSWORD']\n",
    "                   )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "877b4e50-8fa8-4663-bba0-91b661fc735f",
   "metadata": {},
   "source": [
    "Alert on & visualize anomalous identity events\n",
    "\n",
    "Demo dataset: 1.6B windows events over 58 days => logins by 12K user over 14K systems\n",
    "adapt to any identity system with logins. Here we subsample down to a small set of 50k events to prove out the pipeline. \n",
    "\n",
    "* => Can we identify accounts & computers acting anomalously? Resources being oddly accessed?\n",
    "* => Can we spot the red team?\n",
    "* => Operations: Identity incident alerting + identity data investigations\n",
    "\n",
    "Community/contact for help handling bigger-than-memory & additional features\n",
    "\n",
    "Runs on both CPU + multi-GPU\n",
    "Tools: PyGraphistry[AI], DGL + PyTorch, and NVIDIA RAPIDS / umap-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe6e61b0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# data source citation\n",
    "# \"\"\"A. D. Kent, \"Cybersecurity Data Sources for Dynamic Network Research,\"\n",
    "# in Dynamic Networks in Cybersecurity, 2015.\n",
    "\n",
    "# @InProceedings{akent-2015-enterprise-data,\n",
    "#    author = {Alexander D. Kent},\n",
    "#    title = {{Cybersecurity Data Sources for Dynamic Network Research}},\n",
    "#    year = 2015,\n",
    "#    booktitle = {Dynamic Networks in Cybersecurity},\n",
    "#    month =        jun,\n",
    "#    publisher = {Imperial College Press}\n",
    "# }\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "554c0d85-1c8a-47f0-87ec-1629d7f7ba3b",
   "metadata": {},
   "source": [
    "# Get the Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efe68cf8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# small sample (get almost equivalent results without overheating computer over the 1.6B events in the full dataset)\n",
    "df = pd.read_csv('https://gist.githubusercontent.com/silkspace/c7b50d0c03dc59f63c48d68d696958ff/raw/31d918267f86f8252d42d2e9597ba6fc03fcdac2/redteam_50k.csv', index_col=0)\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93bab916-a6c1-4a63-95de-2e8d2a72d8a6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-20T17:41:26.708147Z",
     "iopub.status.busy": "2023-03-20T17:41:26.707740Z",
     "iopub.status.idle": "2023-03-20T17:41:26.711459Z",
     "shell.execute_reply": "2023-03-20T17:41:26.710695Z",
     "shell.execute_reply.started": "2023-03-20T17:41:26.708118Z"
    }
   },
   "source": [
    "# Graphistry UMAP in a single line of code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29e99375-5b24-4760-b5ed-909f51949f7f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# umap pipeline in one line\n",
    "g = graphistry.nodes(df.sample(1000)).umap(engine='umap_learn')\n",
    "g.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03610297",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(df.shape) # -> 50+k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66c5126e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# here are the post-facto red team events\n",
    "red_team = pd.read_csv('https://gist.githubusercontent.com/silkspace/5cf5a94b9ac4b4ffe38904f20d93edb1/raw/888dabd86f88ea747cf9ff5f6c44725e21536465/redteam_labels.csv', index_col=0)\n",
    "red_team['feats2'] = red_team.feats  # since red team data didn't include metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c6615aa",
   "metadata": {},
   "source": [
    "# Modeling\n",
    "\n",
    "Make sure you `mkdir(data)` or change path below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3641d3b5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "process = True  \n",
    "# makes a combined feature we can use for topic modeling!\n",
    "if process:\n",
    "    # we create two types of models\n",
    "    df['feats'] = df.src_computer + ' ' + df.dst_computer + ' ' + df.auth_type + ' ' + df.logontype\n",
    "    # and one of just computer to computer \n",
    "    df['feats2'] = df.src_computer + ' ' + df.dst_computer\n",
    "    ndf = df.drop_duplicates(subset=['feats'])\n",
    "    ndf.to_parquet('auth-feats-one-column.parquet')\n",
    "else:\n",
    "    ndf = pd.read_parquet('auth-feats-one-column.parquet')\n",
    "    \n",
    "print(ndf.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32d1755d",
   "metadata": {},
   "source": [
    "## Red Team Data \n",
    "Add it to the front of the DataFrame so we can keep track of it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d67c86b8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# make a subsampled dataframe with the anom red-team data at top...so we can keep track.\n",
    "# we don't need the full `df`, only the unique entries of 'feats' in `ndf` for \n",
    "# fitting a model (in a few cells below)\n",
    "\n",
    "tdf = pd.concat([red_team.reset_index(), ndf.reset_index()])\n",
    "tdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f62b7b5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# add a fidicial index used later\n",
    "tdf['node'] = range(len(tdf))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ffd6aac",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# total number of red team events\n",
    "tdf.RED.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4264d547-b4a9-49d1-bc68-894f1e839c38",
   "metadata": {},
   "source": [
    "## Enrichment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72c53f98",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_confidences_per_cluster(g, col='RED', verbose=False):\n",
    "    \"\"\"\n",
    "        From DBSCAN clusters, will assess how many Red Team events exist,\n",
    "        assessing confidence.\n",
    "        \n",
    "    \"\"\"\n",
    "    resses = []\n",
    "    df = g._nodes\n",
    "    labels = df._dbscan\n",
    "    cnt = Counter(labels)\n",
    "    for clust, count in cnt.most_common():\n",
    "        res = df[df._dbscan==clust]\n",
    "        n = res.shape[0]\n",
    "        n_reds = res[col].sum()\n",
    "        resses.append([clust, n_reds/n, n_reds, n])\n",
    "        if n_reds>0 and verbose:\n",
    "            print('-'*20)\n",
    "            print(f'cluster: {clust}\\n   red {100*n_reds/n:.2f}% or {n_reds} out of {count}')\n",
    "    conf_dict = {k[0]: k[1] for k in resses}\n",
    "    confidence = [conf_dict[k] for k in df._dbscan.values]\n",
    "    # enrichment\n",
    "    g._nodes['confidence'] = confidence\n",
    "    conf_df = pd.DataFrame(resses, columns=['_dbscan', 'confidence', 'n_red', 'total_in_cluster'])\n",
    "    conf_df = conf_df.sort_values(by='confidence', ascending=False)\n",
    "    return g, conf_df\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a3da6e3-b280-4c69-b0e0-4a92d9aac231",
   "metadata": {},
   "source": [
    "# The Full UMAP Pipelines\n",
    "Fit a model on 'feats' column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "504781dc-9fbe-467c-9b4d-2e907133cfb7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# this is a convienence method for setting parameters in `g.featurize()/umap()` -- just a verbose dictionary\n",
    "cyber_model = ModelDict('A topic model for computer to computer', **topic_model)\n",
    "\n",
    "# umap_params_gpu = {'n_components': 2, \n",
    "#                    'n_neighbors': 20,\n",
    "#                    'min_dist': 0.1, \n",
    "#                    'spread': 1, \n",
    "#                    'local_connectivity': 1, \n",
    "#                    'repulsion_strength': 2, \n",
    "#                    'negative_sample_rate': 5}\n",
    "#cyber_model.update(umap_params_gpu)\n",
    "\n",
    "cyber_model.update(dict(n_topics=32, X=['feats2']))  # name the column to featurize, which we lumped into `feats2`\n",
    "\n",
    "cyber_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5474ef79-b2dd-4299-bee7-e12d94c79613",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# if you stop processing during execution, sometimes calling this will unblock you on subsequent calls should it give an error.\n",
    "#g.reset_caches()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6909cc90",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "process = True  # set to false after it's run for ease of speed\n",
    "if process:\n",
    "    # ##################################\n",
    "    g = graphistry.nodes(tdf, 'node')  # two lines does the heavy lifting\n",
    "    # gpu version, will detect gpu and run\n",
    "    #g5 = g.umap(engine='auto', **cyber_model, verbose=True).dbscan(min_dist=1, verbose=True)\n",
    "    \n",
    "    # cpu version\n",
    "    g5 = g.umap(engine='umap_learn', **cyber_model, verbose=True).dbscan(min_dist=0.1, verbose=True)\n",
    "    # #########################\n",
    "    \n",
    "    g5, cluster_confidences = get_confidences_per_cluster(g5, verbose=True)\n",
    "    g5.save_search_instance('auth-feat-topic.search')\n",
    "else:\n",
    "    g = graphistry.bind()\n",
    "    g5 = g.load_search_instance('auth-feat-topic.search')\n",
    "    g5, cluster_confidences = get_confidences_per_cluster(g5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01632281-2ace-4917-9932-86b507b3d9e3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# nodes dataframe is now enriched with _dbscan label\n",
    "g5._nodes._dbscan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c1ba011-2aaa-4663-a319-4478502b1b8e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# the UMAP coordinates\n",
    "g5._node_embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54c13cba-bc36-4d49-8e7a-7dc05b27610a",
   "metadata": {},
   "source": [
    "## Plot Graph\n",
    "Color by `confidence` and hover over `red` team histogram to see where events occur. Alternatively, color by `cluster` assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "279fef41",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "g5.name('auth test').plot(render=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79ece955",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# see how the model has organized features\n",
    "X = g5._node_features\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87b32e09-3ca4-49de-b8c3-2b40ffa2b01d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "x = g5.get_matrix(['interactive', 'c17', 'microsoft'])\n",
    "x.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "632d6d0f-8212-4f4a-a920-7600d7456351",
   "metadata": {},
   "source": [
    "## Predict | Online Mode\n",
    "\n",
    "Once a model is fit, predict on new batches as we demonstrate here\n",
    "\n",
    "There are three main methods\n",
    "\n",
    "`g.transform` and `g.transform_umap` and if dbscan has been run, `g.transform_dbscan` \n",
    "\n",
    "see help(*) on each to learn more\n",
    "\n",
    "One may save the model as above, load it, and wrap in a FastAPI endpoint, etc, to serve in production pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b44d418",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# first sample a batch from the normal data (auth=df)\n",
    "sdf = df.sample(200)\n",
    "emb_normal, xp_normal, _ = g5.transform_umap(sdf, None, kind='nodes', return_graph=False)\n",
    "# then transform all the red team data\n",
    "emb_red, xp_red, _ = g5.transform_umap(red_team, None, kind='nodes', return_graph=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe3e5058-6ac6-4d1a-a368-66ecd5dd703b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "emb_red"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2b6c471-338a-40d6-92a8-03c2505c433f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# transform_dbscan will predict on new data (here just red_team to prove it works)\n",
    "g7 = g5.transform_dbscan(red_team, None, kind='nodes', return_graph=True, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad82c787-c246-440d-9ed6-97ddc2805491",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "_, ccdf = get_confidences_per_cluster(g7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e0760fe-40c0-45b9-a787-d4f98d557c24",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f'total confidence across clusters {ccdf.confidence.mean()*100}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2911840d-ffd7-4815-97fd-53bc43cbc522",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "g7.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ace3e171-2e33-435e-82d7-7158d7931d14",
   "metadata": {},
   "source": [
    "# We can simulate how a batch of new data would behave"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03672813-db4e-4d0c-a5f5-598ab165986c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cpu version\n",
    "plt.figure(figsize=(10,7))\n",
    "plt.scatter(g5._node_embedding.x, g5._node_embedding.y, c='b', s=60, alpha=0.5)  # the totality of the fit data\n",
    "plt.scatter(emb_normal.x, emb_normal.y, c='g') # batch of new data\n",
    "plt.scatter(emb_red.x, emb_red.y, c='r') # red labels to show good cluster seperation\n",
    "plt.scatter(emb_normal.x, emb_normal.y, c='g') # batch of new data, to see if they occlude "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a8d5aa9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# gpu version\n",
    "# scatter to see how well it does.\n",
    "plt.figure(figsize=(10,7))\n",
    "plt.scatter(g5._node_embedding.x.to_numpy(), g5._node_embedding.y.to_numpy() , c='b', s=60, alpha=0.5)  # the totality of the fit data\n",
    "plt.scatter(emb_normal.x.to_numpy(), emb_normal.y.to_numpy(), c='g') # batch of new data\n",
    "plt.scatter(emb_red.x.to_numpy(), emb_red.y.to_numpy(), c='r') # red labels to show good cluster seperation\n",
    "plt.scatter(emb_normal.x.to_numpy(), emb_normal.y.to_numpy(), c='g') # batch of new data, to see if they occlude "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b53dd8ed-39b2-4000-9ec7-139d1e2a6a85",
   "metadata": {},
   "source": [
    "## 96% Reduction in Alerts\n",
    "\n",
    "This indicates a huge reduction in the search space needed.\n",
    "\n",
    "Since we have clear cluster assignments along with (post facto) confidences of known anomalous activity, we can reduce the search space on new events (gotten via Kafka, Splunk, etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14d207db-9a58-45a3-9876-058632389f17",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# percent of RED team labels we get with 10% confidence or above\n",
    "p = cluster_confidences[cluster_confidences.confidence>0.1].n_red.sum()/cluster_confidences[cluster_confidences.confidence>0.1].total_in_cluster.sum()\n",
    "print(f'{100*p:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "755a3f27-935d-4ba8-96cb-cbff11fdf00e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# number of data points *not* to consider (and it's more if we look at df proper!)\n",
    "cluster_confidences[cluster_confidences.confidence<0.1].total_in_cluster.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fd1cc50-0900-4694-8400-c426e314ec2e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "p = cluster_confidences[cluster_confidences.confidence<0.1].total_in_cluster.sum()/cluster_confidences.total_in_cluster.sum()\n",
    "print(f'Alert Reduction {100*p:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ee508a5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,7))\n",
    "plt.plot(np.cumsum([k[2] for k in cluster_confidences.values]))\n",
    "plt.xlabel('Anomolous Cluster Number')  # shows that we can ignore first clusters (containing most of the alerts)\n",
    "plt.ylabel('Number of Identified Red Team Events')\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f168ac8-2324-4f47-b0d7-e4a0b041624f",
   "metadata": {},
   "source": [
    "## Supervised UMAP\n",
    "Here we use the RED team label to help supervise the UMAP fit. \n",
    "This might be useful once teams have actually identified RED team events \n",
    "and want to help separate clusters. \n",
    "While separation is better, the unsupervised version does well without."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34ad4768-58e5-493e-a5e8-6f4748168e05",
   "metadata": {},
   "outputs": [],
   "source": [
    "# g.reset_caches()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0c6a16d-a899-43b6-a7ba-75b45f855a78",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "process = True\n",
    "if process:\n",
    "    # ##################################  # an example of setting features explicitly, could use ModelDict \n",
    "    g = graphistry.nodes(tdf, 'node')\n",
    "    g6 = g.umap(X=['feats'], y =['RED'], \n",
    "                min_words=100000, # set high to bypass sbert encoding\n",
    "                cardinality_threshold=2, # set low to force topic modeling\n",
    "                n_topics=32,\n",
    "                spread=1,\n",
    "                use_scaler_target=None,  # keep labels unscaled\n",
    "                dbscan=True, engine='umap_learn')  # add dbscan here\n",
    "    # ##################################\n",
    "    \n",
    "    g6, cluster_confidences6  = get_confidences_per_cluster(g6, verbose=True)\n",
    "    g6.save_search_instance('auth-feat-supervised-topic.search')\n",
    "else:\n",
    "    g = graphistry.bind()\n",
    "    g6 = g.load_search_instance('auth-feat-supervised-topic.search')\n",
    "    g6, cluster_confidences6  = get_confidences_per_cluster(g6)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a98ef657-5307-41d9-ae31-79c1794b3728",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "g6.get_matrix(target=True).astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cc72ab4-c0da-4541-b32b-aa771d6e510f",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Plot\n",
    "Color by `confidence` and hover over `red` team histogram to see where events occur. Alternatively, color by `_dbscan` assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16e09a7d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "g6.name('auth topic with supervised umap').plot(render=RENDER)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88169a53",
   "metadata": {},
   "source": [
    "## A model of Computer-Computer and metadata features\n",
    "Here we include `auth_type` and `logontype` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1731ae44-57e0-4c3e-bad0-ac486bba589c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tdf['feats']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35b03bc4-915b-431b-ada5-d8281a4ece6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "process = True\n",
    "if process:\n",
    "    # #####################################\n",
    "    g = graphistry.nodes(tdf, 'node')\n",
    "    g7 = g.umap(X=['feats'], #y =['RED'], \n",
    "                min_words=100000, \n",
    "                cardinality_threshold=2, \n",
    "                n_topics=32,\n",
    "                use_scaler=None,\n",
    "                use_scaler_target=None, \n",
    "                spread=1,\n",
    "                dbscan=True, engine='auto')  # add dbscan here\n",
    "    # ###################################\n",
    "    g7, cluster_confidences7  = get_confidences_per_cluster(g7)\n",
    "    #g7.save_search_instance('auth-just-ip-topic.search')\n",
    "else:\n",
    "    g7 = graphistry.bind().load_search_instance('auth-just-ip-topic.search')\n",
    "    g7, cluster_confidences7  = get_confidences_per_cluster(g7)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f291e227-ae14-4205-96dd-3c1de29d12e6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cluster_confidences7"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "836883cb-bc66-4a40-9ca8-f01fd38b6f2a",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Plot\n",
    "Color by `confidence` and hover over `red` team histogram to see where events occur. Alternatively, color by `cluster` assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1e586a3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "g7.name('auth topic ips-ips only, no supervision').plot(render=RENDER)\n",
    "# very similar to graph with metadata included, showing that ip-ip is strong indicator of phenomenon"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cf68ed4",
   "metadata": {},
   "source": [
    "# Conditional Probability\n",
    "Let's see if conditiona probability of computer to computer connections can give us good histograms to tease out red team nodes? This is to baseline the above UMAP models, and we find in retrospect, UMAP wins. \n",
    "\n",
    "The conditional graph is however useful to see aggregate behavior, and coloring by 'red' team shows topology of Infection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d6f58dd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "g = graphistry.edges(tdf, \"src_computer\", \"dst_computer\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3b44db2-b34e-4398-8c5a-7a10bbe5d681",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "x='dst_computer'\n",
    "given='src_computer'\n",
    "cg = g.conditional_graph(x, given, kind='edges')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b2af6a2-4f10-4707-beb8-4f3447d3e3b8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# the new edge dataframe assess conditiona prob of computer-to-computer connection\n",
    "cprob = cg._edges\n",
    "cprob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5258aee1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# enrich the edges dataframe with the redteam data\n",
    "# since cprobs lost those labels during the function call\n",
    "indx = cprob.src_computer.isin(red_team.src_computer) & cprob.dst_computer.isin(red_team.dst_computer)\n",
    "cprob.loc[indx, 'red'] = 1\n",
    "cprob.loc[~indx, 'red'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ff921fc-3ecd-4404-acd7-8db943a4ebcc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cprob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4b10152-cac9-4497-b016-dd67b54cdcf2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# add edges back to graphistry instance\n",
    "cg._edges = cprob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b3af1cd-6423-4484-8b99-81fad821f118",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# full condprob graph\n",
    "cg.plot(render=RENDER)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42fb3dff",
   "metadata": {},
   "source": [
    "## Learning\n",
    "The conditional graph shows that most of the edge probabilities are between 4e-7 and 0.03, whose bucket contains most of the events. Thus the chances of finding the red team edges are ~ 1e-4 -- slim indeed. UMAP wins."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d2cd536",
   "metadata": {},
   "source": [
    "Likewise the transpose conditional is even worse \n",
    "with prob_detection ~ 6e-5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0cbef82-421d-489e-8666-84d412cae5a9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}