demos/demos_databases_apis/splunk/splunk_demo_public.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eL9SNNiNv9Me"
   },
   "source": [
    "# Splunk<> Graphistry\n",
    "\n",
    "Graphistry brings modern visual analytics to event data in Splunk. The full platform is intended for enterprise teams, while this tutorials shares visibility techniques for researchers and hunters.\n",
    "\n",
    "To use:\n",
    "* Read along, start the prebuilt visualizations by clicking on them\n",
    "* Plug in your Graphistry API Key & Splunk credentials to use for yourself\n",
    "\n",
    "Further reading:\n",
    "* UI Guide: https://labs.graphistry.com/graphistry/ui.html\n",
    "* Python client tutorials & demos: https://github.com/graphistry/pygraphistry \n",
    "* Graphistry API Key: https://www.graphistry.com/api-request \n",
    "* DoD / VAST challenges: https://www.cs.umd.edu/hcil/varepository/benchmarks.php \n",
    "\n",
    "\n",
    "## 0. Configure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "a_7QzizSwEeb"
   },
   "outputs": [],
   "source": [
    "\n",
    "#splunk\n",
    "SPLUNK = {\n",
    "    'host': 'MY.SPLUNK.com',\n",
    "    'scheme': 'https',\n",
    "    'port': 8089,\n",
    "    'username': 'MY_SPLUNK_USER',\n",
    "    'password': 'MY_SPLUNK_PWD'   \n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NDVzk3wczY4C"
   },
   "source": [
    "## 1. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "vwMRhMi2vep2"
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ALj6vlm1z69z"
   },
   "source": [
    "### Graphistry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "colab_type": "code",
    "id": "Yy92ctrqvati",
    "outputId": "115eda49-2bf2-45d3-e425-182e459524f9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: graphistry in /usr/local/lib/python2.7/dist-packages (0.9.56)\n",
      "Requirement already satisfied: pandas>=0.17.0 in /usr/local/lib/python2.7/dist-packages (from graphistry) (0.22.0)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python2.7/dist-packages (from graphistry) (1.14.6)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python2.7/dist-packages (from graphistry) (2.18.4)\n",
      "Requirement already satisfied: future>=0.15.0 in /usr/local/lib/python2.7/dist-packages (from graphistry) (0.16.0)\n",
      "Requirement already satisfied: protobuf>=2.6.0 in /usr/local/lib/python2.7/dist-packages (from graphistry) (3.6.1)\n",
      "Requirement already satisfied: pytz>=2011k in /usr/local/lib/python2.7/dist-packages (from pandas>=0.17.0->graphistry) (2018.5)\n",
      "Requirement already satisfied: python-dateutil in /usr/local/lib/python2.7/dist-packages (from pandas>=0.17.0->graphistry) (2.5.3)\n",
      "Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests->graphistry) (2.6)\n",
      "Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests->graphistry) (1.22)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests->graphistry) (2018.8.24)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests->graphistry) (3.0.4)\n",
      "Requirement already satisfied: six>=1.9 in /usr/local/lib/python2.7/dist-packages (from protobuf>=2.6.0->graphistry) (1.11.0)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python2.7/dist-packages (from protobuf>=2.6.0->graphistry) (39.1.0)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "u'0.9.56'"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!pip install graphistry\n",
    "\n",
    "import graphistry\n",
    "graphistry.__version__\n",
    "\n",
    "# To specify Graphistry account & server, use:\n",
    "# graphistry.register(api=3, username='...', password='...', protocol='https', server='hub.graphistry.com')\n",
    "# For more options, see https://github.com/graphistry/pygraphistry#configure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "32fiCtkKz95Z"
   },
   "source": [
    "### Splunk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 188
    },
    "colab_type": "code",
    "id": "XPK5n5Yrvjb5",
    "outputId": "04e436c6-5a8b-4148-cd31-874421e6967e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting splunk-sdk\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d4/bb/408c504f4307fcf4a89909cc85bc912d8529c9ca88200682f94a31a06186/splunk-sdk-1.6.5.tar.gz (103kB)\n",
      "\u001b[K    100% |████████████████████████████████| 112kB 2.6MB/s \n",
      "\u001b[?25hBuilding wheels for collected packages: splunk-sdk\n",
      "  Running setup.py bdist_wheel for splunk-sdk ... \u001b[?25l-\b \bdone\n",
      "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/87/83/8f/5f78fbc79322715add8f39ba8adc97511f27297852eb4dc270\n",
      "Successfully built splunk-sdk\n",
      "Installing collected packages: splunk-sdk\n",
      "Successfully installed splunk-sdk-1.6.5\n"
     ]
    }
   ],
   "source": [
    "!pip install splunk-sdk\n",
    "\n",
    "import splunklib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "Jym3LNOSxtr-"
   },
   "outputs": [],
   "source": [
    "#Connect to Splunk. Replace settings with your own setup.\n",
    "import splunklib.client as client\n",
    "import splunklib.results as results\n",
    "\n",
    "service = client.connect(**SPLUNK)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "a3eoM0S8yEEd"
   },
   "outputs": [],
   "source": [
    "def extend(o, override):\n",
    "  for k in override.keys():\n",
    "    o[k] = override[k]\n",
    "  return o\n",
    "\n",
    "STEP = 10000;                       \n",
    "def splunkToPandas(qry, overrides={}):\n",
    "    kwargs_blockingsearch = extend({\n",
    "        \"count\": 0,\n",
    "        \"earliest_time\": \"2010-01-24T07:20:38.000-05:00\",\n",
    "        \"latest_time\": \"now\",\n",
    "        \"search_mode\": \"normal\",\n",
    "        \"exec_mode\": \"blocking\"\n",
    "    }, overrides)\n",
    "    job = service.jobs.create(qry, **kwargs_blockingsearch)\n",
    "\n",
    "    print \"Search results:\\n\"\n",
    "    resultCount = job[\"resultCount\"]\n",
    "    offset = 0;                         \n",
    "\n",
    "    print 'results', resultCount\n",
    "    out = None\n",
    "    while (offset < int(resultCount)):\n",
    "        print \"fetching:\", offset, '-', offset + STEP\n",
    "        kwargs_paginate = extend(kwargs_blockingsearch,\n",
    "                                 {\"count\": STEP,\n",
    "                                  \"offset\": offset})\n",
    "\n",
    "        # Get the search results and display them\n",
    "        blocksearch_results = job.results(**kwargs_paginate)\n",
    "        reader = results.ResultsReader(blocksearch_results)\n",
    "        lst = [x for x in reader]\n",
    "        df2 = pd.DataFrame(lst)    \n",
    "        out = df2 if type(out) == type(None) else pd.concat([out, df2], ignore_index=True)\n",
    "        offset += STEP\n",
    "    return out        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LdDScZ56aEdL"
   },
   "source": [
    "## 2. Get data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 313
    },
    "colab_type": "code",
    "id": "B51tGgcE0L1e",
    "outputId": "c00b4e9f-974e-4d95-972e-58fddd9f729a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search results:\n",
      "\n",
      "results 5035\n",
      "fetching: 0 - 10000\n",
      "CPU times: user 4.95 s, sys: 13.3 ms, total: 4.96 s\n",
      "Wall time: 7.92 s\n",
      "results 5035\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dest_ip</th>\n",
       "      <th>src_ip</th>\n",
       "      <th>protocol</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4324</th>\n",
       "      <td>10.138.235.111</td>\n",
       "      <td>172.30.0.4</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2806</th>\n",
       "      <td>10.0.3.5</td>\n",
       "      <td>10.12.15.152</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2630</th>\n",
       "      <td>10.0.4.5</td>\n",
       "      <td>10.12.15.152</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>10.0.4.7</td>\n",
       "      <td>10.6.6.7</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>866</th>\n",
       "      <td>10.0.2.8</td>\n",
       "      <td>10.17.15.10</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519787</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             dest_ip        src_ip protocol        time\n",
       "4324  10.138.235.111    172.30.0.4      TCP  1505519752\n",
       "2806        10.0.3.5  10.12.15.152      TCP  1505519767\n",
       "2630        10.0.4.5  10.12.15.152      TCP  1505519769\n",
       "20          10.0.4.7      10.6.6.7      TCP  1505519795\n",
       "866         10.0.2.8   10.17.15.10      TCP  1505519787"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = 'search index=\"vast\" srcip=* destip=* | rename destip -> dest_ip, srcip -> src_ip | fields dest_ip _time src_ip protocol | eval time=_time | fields - _* '\n",
    "%time df = splunkToPandas(query, {\"sample_ratio\": 1000})\n",
    "\n",
    "#df = splunkToPandasAll('search index=\"vast\" | head 10')\n",
    "#df = pd.concat([ splunkToPandas('search index=\"vast\" | head 10'), splunkToPandas('search index=\"vast\" | head 10') ], ignore_index=True)\n",
    "\n",
    "\n",
    "print 'results', len(df)\n",
    "\n",
    "df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5QIdlCuAaI40"
   },
   "source": [
    "## 3. Visualize!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jE-o1wB9UZk3"
   },
   "source": [
    "### A) Simple IP<>IP: 1326 nodes, 253K edges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 543
    },
    "colab_type": "code",
    "id": "fGMMmvTW1TmJ",
    "outputId": "16c58120-e3e0-4abd-c1b7-7f8d990d6298"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <iframe id=\"16f3cbf6-d4fd-4b82-9cb6-ac2934508c79\" src=\"https://labs.graphistry.com/graph/graph.html?dataset=s3://graphistry.data/pygraphistry/bae362ecdd30a3958010a2007a415926/dataset.json&type=jsonMeta&viztoken=d6528fc5135ae708c7b605c58671f5ab9e209cf5&usertag=a338be27-pygraphistry-0.9.56&splashAfter=1539471234&info=true\"\n",
       "                    allowfullscreen=\"true\" webkitallowfullscreen=\"true\" mozallowfullscreen=\"true\"\n",
       "                    oallowfullscreen=\"true\" msallowfullscreen=\"true\"\n",
       "                    style=\"width:100%; height:500px; border: 1px solid #DDD\">\n",
       "            </iframe>\n",
       "        \n",
       "            <script>\n",
       "                $(\"#16f3cbf6-d4fd-4b82-9cb6-ac2934508c79\").bind('mousewheel', function(e) {\n",
       "                e.preventDefault();\n",
       "                });\n",
       "            </script>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graphistry.bind(source='src_ip', destination='dest_ip').edges(df).plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2LwJRfZOYIxq"
   },
   "source": [
    "### B) IP<>IP + srcip<>protocol: 1328 nodes, 506K edges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 225
    },
    "colab_type": "code",
    "id": "01xjex0k2jhh",
    "outputId": "fbb6dd6c-5781-4c1d-b46c-7e3d79f8fc3c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dest_ip</th>\n",
       "      <th>src_ip</th>\n",
       "      <th>protocol</th>\n",
       "      <th>time</th>\n",
       "      <th>src</th>\n",
       "      <th>dst</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6889</th>\n",
       "      <td>10.0.3.5</td>\n",
       "      <td>10.13.77.49</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519777</td>\n",
       "      <td>10.13.77.49</td>\n",
       "      <td>TCP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3440</th>\n",
       "      <td>10.0.2.6</td>\n",
       "      <td>10.12.15.152</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519761</td>\n",
       "      <td>10.12.15.152</td>\n",
       "      <td>10.0.2.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6396</th>\n",
       "      <td>10.0.4.5</td>\n",
       "      <td>10.138.235.111</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519782</td>\n",
       "      <td>10.138.235.111</td>\n",
       "      <td>TCP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1394</th>\n",
       "      <td>10.0.4.5</td>\n",
       "      <td>10.138.235.111</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519782</td>\n",
       "      <td>10.138.235.111</td>\n",
       "      <td>10.0.4.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5975</th>\n",
       "      <td>10.0.2.7</td>\n",
       "      <td>10.17.15.10</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519786</td>\n",
       "      <td>10.17.15.10</td>\n",
       "      <td>TCP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8683</th>\n",
       "      <td>10.0.2.4</td>\n",
       "      <td>10.12.15.152</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1505519759</td>\n",
       "      <td>10.12.15.152</td>\n",
       "      <td>TCP</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       dest_ip          src_ip protocol        time             src       dst\n",
       "6889  10.0.3.5     10.13.77.49      TCP  1505519777     10.13.77.49       TCP\n",
       "3440  10.0.2.6    10.12.15.152      TCP  1505519761    10.12.15.152  10.0.2.6\n",
       "6396  10.0.4.5  10.138.235.111      TCP  1505519782  10.138.235.111       TCP\n",
       "1394  10.0.4.5  10.138.235.111      TCP  1505519782  10.138.235.111  10.0.4.5\n",
       "5975  10.0.2.7     10.17.15.10      TCP  1505519786     10.17.15.10       TCP\n",
       "8683  10.0.2.4    10.12.15.152      TCP  1505519759    10.12.15.152       TCP"
      ]
     },
     "execution_count": 15,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def make_edges(df, src, dst):\n",
    "  out = df.copy()\n",
    "  out['src'] = df[src]\n",
    "  out['dst'] = df[dst]\n",
    "  return out\n",
    "\n",
    "\n",
    "\n",
    "ip2ip = make_edges(df, 'src_ip', 'dest_ip')\n",
    "srcip2protocol = make_edges(df, 'src_ip', 'protocol')\n",
    "\n",
    "combined = pd.concat([ip2ip, srcip2protocol], ignore_index=True)\n",
    "combined.sample(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 543
    },
    "colab_type": "code",
    "id": "y2uUZeUnSTAU",
    "outputId": "fa66f6fa-4dbc-4deb-9f2a-29fb59dc3c3e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <iframe id=\"fde3d481-6228-4a96-8e24-7af79fad3a3d\" src=\"https://labs.graphistry.com/graph/graph.html?dataset=s3://graphistry.data/pygraphistry/2c5e30d9c6cab49b542c256a24b2fd54/dataset.json&type=jsonMeta&viztoken=647e8add9aa31810f16c30610418b92783e5ba72&usertag=a338be27-pygraphistry-0.9.56&splashAfter=1539471262&info=true\"\n",
       "                    allowfullscreen=\"true\" webkitallowfullscreen=\"true\" mozallowfullscreen=\"true\"\n",
       "                    oallowfullscreen=\"true\" msallowfullscreen=\"true\"\n",
       "                    style=\"width:100%; height:500px; border: 1px solid #DDD\">\n",
       "            </iframe>\n",
       "        \n",
       "            <script>\n",
       "                $(\"#fde3d481-6228-4a96-8e24-7af79fad3a3d\").bind('mousewheel', function(e) {\n",
       "                e.preventDefault();\n",
       "                });\n",
       "            </script>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graphistry.bind(source='src', destination='dst').edges(combined).plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "73boIhlNZeZB"
   },
   "source": [
    "## 3. All<>All via Hypergraph: 254K nodes, 760K edges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 610
    },
    "colab_type": "code",
    "id": "hNffLQmrZkL1",
    "outputId": "7852e9eb-2ccf-4fd3-e4b5-b4d69c1ea1ae"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('# links', 15105)\n",
      "('# event entities', 5035)\n",
      "('# attrib entities', 170)\n",
      "['entities', 'nodes', 'edges', 'events', 'graph']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "            <iframe id=\"a9e33097-ce0d-4239-a494-d46359ec82ed\" src=\"https://labs.graphistry.com/graph/graph.html?dataset=s3://graphistry.data/pygraphistry/d22cce1d6fdde273d3ab8da56fa01499/dataset.json&type=jsonMeta&viztoken=968f654a3f57eaab46ecdacfa755c59ed81f31e6&usertag=a338be27-pygraphistry-0.9.56&splashAfter=1539471276&info=true\"\n",
       "                    allowfullscreen=\"true\" webkitallowfullscreen=\"true\" mozallowfullscreen=\"true\"\n",
       "                    oallowfullscreen=\"true\" msallowfullscreen=\"true\"\n",
       "                    style=\"width:100%; height:500px; border: 1px solid #DDD\">\n",
       "            </iframe>\n",
       "        \n",
       "            <script>\n",
       "                $(\"#a9e33097-ce0d-4239-a494-d46359ec82ed\").bind('mousewheel', function(e) {\n",
       "                e.preventDefault();\n",
       "                });\n",
       "            </script>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 17,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hg = graphistry.hypergraph(df, entity_types=[ 'src_ip', 'dest_ip', 'protocol'] )\n",
    "print hg.keys()\n",
    "hg['graph'].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "CdEObsxYZpah"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SzUQLcNA7Dnz"
   },
   "source": [
    "## Node Colors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "id": "3ikSjuf57FLt",
    "outputId": "18f06889-9ca1-457f-8e0a-ef6f8280506e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>orig_col</th>\n",
       "      <th>color</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4383</th>\n",
       "      <td>172.30.0.3</td>\n",
       "      <td>src_ip</td>\n",
       "      <td>90005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9403</th>\n",
       "      <td>10.0.0.42</td>\n",
       "      <td>dest_ip</td>\n",
       "      <td>46005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4206</th>\n",
       "      <td>172.30.0.4</td>\n",
       "      <td>src_ip</td>\n",
       "      <td>90005</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              id orig_col  color\n",
       "4383  172.30.0.3   src_ip  90005\n",
       "9403   10.0.0.42  dest_ip  46005\n",
       "4206  172.30.0.4   src_ip  90005"
      ]
     },
     "execution_count": 18,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "nodes = pd.concat([ \n",
    "    df[['src_ip']].rename(columns={'src_ip': 'id'}).assign(orig_col='src_ip'), \n",
    "    df[['dest_ip']].rename(columns={'dest_ip': 'id'}).assign(orig_col='dest_ip') ], \n",
    "    ignore_index=True).drop_duplicates(['id'])\n",
    "\n",
    "#see https://labs.graphistry.com/docs/docs/palette.html\n",
    "col2color = { \n",
    "    \"src_ip\": 90005,\n",
    "    \"dest_ip\": 46005   \n",
    "}\n",
    "\n",
    "nodes_with_color = nodes.assign(color=nodes.apply(lambda row: col2color[ row['orig_col'] ], axis=1))\n",
    "\n",
    "nodes_with_color.sample(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 543
    },
    "colab_type": "code",
    "id": "8iELxWid7tNO",
    "outputId": "525ab5fe-8df3-4bfe-a264-aad2f212fa99"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <iframe id=\"ae244a20-1215-4afa-a965-d06101112e5b\" src=\"https://labs.graphistry.com/graph/graph.html?dataset=s3://graphistry.data/pygraphistry/e70c56304c57f9180d257f1d231c6219/dataset.json&type=jsonMeta&viztoken=6bc2b146d63f01dfeaa26fbbdcce5f91d7219711&usertag=a338be27-pygraphistry-0.9.56&splashAfter=1539471303&info=true\"\n",
       "                    allowfullscreen=\"true\" webkitallowfullscreen=\"true\" mozallowfullscreen=\"true\"\n",
       "                    oallowfullscreen=\"true\" msallowfullscreen=\"true\"\n",
       "                    style=\"width:100%; height:500px; border: 1px solid #DDD\">\n",
       "            </iframe>\n",
       "        \n",
       "            <script>\n",
       "                $(\"#ae244a20-1215-4afa-a965-d06101112e5b\").bind('mousewheel', function(e) {\n",
       "                e.preventDefault();\n",
       "                });\n",
       "            </script>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 19,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graphistry.bind(source='src_ip', destination='dest_ip').edges(df).nodes(nodes_with_color).bind(node='id', point_color='color').plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "61BQS0qC9Wlm"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "splunk_demo_public",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}