demos/demos_databases_apis/gpu_rapids/part_ii_gpu_cudf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visual GPU Log Analytics Part II: GPU dataframes with RAPIDS Python cudf bindings\n",
    "\n",
    "Graphistry is great -- Graphistry and RAPIDS/BlazingDB is better!\n",
    "\n",
    "This tutorial series visually analyzes Zeek/Bro network connection logs using different compute engines:\n",
    "\n",
    "* Part I: [CPU Baseline in Python Pandas](./part_i_cpu_pandas.ipynb)\n",
    "* Part II: [GPU Dataframe with RAPIDS Python cudf bindings](./part_ii_gpu_cudf.ipynb)\n",
    "* Part III: GPU SQL - deprecated as Dask-SQL replaced BlazingSQL in the RAPIDS ecosystem\n",
    "* Part IV: [GPU ML with RAPIDS cuML UMAP and PyGraphistry](./part_iv_gpu_cuml.ipynb)\n",
    "* [Graphistry cuGraph bindings](./cugraph.ipynb)\n",
    "\n",
    "**Part II Contents:**\n",
    "\n",
    "Time using GPU-based RAPIDS Python cudf bindings and Graphistry for a full ETL & visual analysis flow:\n",
    "\n",
    "1. Load data\n",
    "2. Analyze data\n",
    "3. Visualize data\n",
    "\n",
    "_**TIP**_: If you get out of memory errors, you usually must restart the kernel & refresh the page\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#!pip install graphistry -q\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import cudf\n",
    "\n",
    "import graphistry\n",
    "graphistry.__version__\n",
    "\n",
    "# To specify Graphistry account & server, use:\n",
    "# graphistry.register(api=3, username='...', password='...', protocol='https', server='hub.graphistry.com')\n",
    "\n",
    "# For more options, see https://github.com/graphistry/pygraphistry#configure\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tue Nov 22 06:20:14 2022       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 470.141.03   Driver Version: 470.141.03   CUDA Version: 11.4     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla T4            Off  | 00000000:00:1E.0 Off |                    0 |\n",
      "| N/A   38C    P0    27W /  70W |   8037MiB / 15109MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "# check nvidia config \n",
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100  523M  100  523M    0     0  31.5M      0  0:00:16  0:00:16 --:--:-- 30.0M\n",
      "CPU times: user 240 ms, sys: 105 ms, total: 345 ms\n",
      "Wall time: 17.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# download data \n",
    "#!if [ ! -f conn.log ]; then \\\n",
    "#    curl https://www.secrepo.com/maccdc2012/conn.log.gz | gzip -d > conn.log; \\\n",
    "#fi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1331901000.000000\tCCUIP21wTjqkj8ZqX5\t192.168.202.79\t50463\t192.168.229.251\t80\ttcp\t-\t-\t-\t-\tSH\t-\t0\tFa\t1\t52\t1\t52\t(empty)\n",
      "1331901000.000000\tCsssjd3tX0yOTPDpng\t192.168.202.79\t46117\t192.168.229.254\t443\ttcp\t-\t-\t-\t-\tSF\t-\t0\tdDafFr\t3\t382\t9\t994\t(empty)\n",
      "1331901000.000000\tCHEt7z3AzG4gyCNgci\t192.168.202.79\t50465\t192.168.229.251\t80\ttcp\thttp\t0.010000\t166\t214\tSF\t-\t0\tShADfFa\t4\t382\t3\t382\t(empty)\n"
     ]
    }
   ],
   "source": [
    "!head -n 3 conn.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# OPTIONAL: For slow or limited devices, work on a subset:\n",
    "LIMIT = 12000000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.02 s, sys: 676 ms, total: 2.69 s\n",
      "Wall time: 2.69 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "cdf = cudf.read_csv(\"./conn.log\", sep=\"\\t\", header=None, \n",
    "                   names=[\"time\", \"uid\", \"id.orig_h\", \"id.orig_p\", \"id.resp_h\", \"id.resp_p\", \"proto\", \"service\",\n",
    "                        \"duration\", \"orig_bytes\", \"resp_bytes\", \"conn_state\", \"local_orig\", \"missed_bytes\",\n",
    "                        \"history\", \"orig_pkts\", \"orig_ip_bytes\", \"resp_pkts\", \"resp_ip_bytes\", \"tunnel_parents\"],\n",
    "                   dtype=['date', 'str', 'str', 'int', 'str', 'int', 'str', 'str',\n",
    "                         'int', 'int', 'int', 'str', 'str', 'int',\n",
    "                         'str', 'int', 'int', 'int', 'int', 'str'],\n",
    "                   na_values=['-'], index_col=False, nrows=LIMIT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#fillna\n",
    "for c in cdf.columns:\n",
    "    if c in ['uid', 'id.orig_h', 'id.resp_h', 'proto', 'service', 'conn_state', 'history', 'tunnel_parents', 'local_orig']:\n",
    "        continue\n",
    "    cdf[c] = cdf[c].fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# rows 12000000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>uid</th>\n",
       "      <th>id.orig_h</th>\n",
       "      <th>id.orig_p</th>\n",
       "      <th>id.resp_h</th>\n",
       "      <th>id.resp_p</th>\n",
       "      <th>proto</th>\n",
       "      <th>service</th>\n",
       "      <th>duration</th>\n",
       "      <th>orig_bytes</th>\n",
       "      <th>resp_bytes</th>\n",
       "      <th>conn_state</th>\n",
       "      <th>local_orig</th>\n",
       "      <th>missed_bytes</th>\n",
       "      <th>history</th>\n",
       "      <th>orig_pkts</th>\n",
       "      <th>orig_ip_bytes</th>\n",
       "      <th>resp_pkts</th>\n",
       "      <th>resp_ip_bytes</th>\n",
       "      <th>tunnel_parents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1753-11-29 22:43:41.128654848</td>\n",
       "      <td>CCUIP21wTjqkj8ZqX5</td>\n",
       "      <td>192.168.202.79</td>\n",
       "      <td>50463</td>\n",
       "      <td>192.168.229.251</td>\n",
       "      <td>80</td>\n",
       "      <td>tcp</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>SH</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>Fa</td>\n",
       "      <td>1</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>52</td>\n",
       "      <td>(empty)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1753-09-09 22:43:41.128654848</td>\n",
       "      <td>Csssjd3tX0yOTPDpng</td>\n",
       "      <td>192.168.202.79</td>\n",
       "      <td>46117</td>\n",
       "      <td>192.168.229.254</td>\n",
       "      <td>443</td>\n",
       "      <td>tcp</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>SF</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>dDafFr</td>\n",
       "      <td>3</td>\n",
       "      <td>382</td>\n",
       "      <td>9</td>\n",
       "      <td>994</td>\n",
       "      <td>(empty)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1753-10-06 22:43:41.128654848</td>\n",
       "      <td>CHEt7z3AzG4gyCNgci</td>\n",
       "      <td>192.168.202.79</td>\n",
       "      <td>50465</td>\n",
       "      <td>192.168.229.251</td>\n",
       "      <td>80</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>0</td>\n",
       "      <td>166</td>\n",
       "      <td>214</td>\n",
       "      <td>SF</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>ShADfFa</td>\n",
       "      <td>4</td>\n",
       "      <td>382</td>\n",
       "      <td>3</td>\n",
       "      <td>382</td>\n",
       "      <td>(empty)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           time                 uid       id.orig_h  \\\n",
       "0 1753-11-29 22:43:41.128654848  CCUIP21wTjqkj8ZqX5  192.168.202.79   \n",
       "1 1753-09-09 22:43:41.128654848  Csssjd3tX0yOTPDpng  192.168.202.79   \n",
       "2 1753-10-06 22:43:41.128654848  CHEt7z3AzG4gyCNgci  192.168.202.79   \n",
       "\n",
       "   id.orig_p        id.resp_h  id.resp_p proto service  duration  orig_bytes  \\\n",
       "0      50463  192.168.229.251         80   tcp    <NA>         0           0   \n",
       "1      46117  192.168.229.254        443   tcp    <NA>         0           0   \n",
       "2      50465  192.168.229.251         80   tcp    http         0         166   \n",
       "\n",
       "   resp_bytes conn_state local_orig  missed_bytes  history  orig_pkts  \\\n",
       "0           0         SH       <NA>             0       Fa          1   \n",
       "1           0         SF       <NA>             0   dDafFr          3   \n",
       "2         214         SF       <NA>             0  ShADfFa          4   \n",
       "\n",
       "   orig_ip_bytes  resp_pkts  resp_ip_bytes tunnel_parents  \n",
       "0             52          1             52        (empty)  \n",
       "1            382          9            994        (empty)  \n",
       "2            382          3            382        (empty)  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('# rows', len(cdf))\n",
    "cdf.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Analyze Data\n",
    "\n",
    "* Summarize network activities between every communicating src/dst IP,  split by connection state\n",
    "* RAPIDS currently fails when exceeding GPU memory, so limit workload size as needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.19 s, sys: 68.2 ms, total: 1.26 s\n",
      "Wall time: 1.26 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "cdf_summary=(cdf\n",
    "    .pipe(lambda df: df.assign(sum_bytes=df.orig_bytes + df.resp_bytes))\n",
    "    .groupby(['id.orig_h', 'id.resp_h', 'conn_state'])\n",
    "    .agg({\n",
    "        'time': ['min', 'max', 'count'],\n",
    "        'id.resp_p':  ['count'],\n",
    "        'uid': ['count'],\n",
    "        'duration':   ['min', 'max', 'mean'],\n",
    "        'orig_bytes': ['min', 'max', 'sum', 'mean'],\n",
    "        'resp_bytes': ['min', 'max', 'sum', 'mean'],\n",
    "        'sum_bytes':  ['min', 'max', 'sum', 'mean']\n",
    "}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# rows 50140\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">time</th>\n",
       "      <th>id.resp_p</th>\n",
       "      <th>uid</th>\n",
       "      <th colspan=\"3\" halign=\"left\">duration</th>\n",
       "      <th colspan=\"4\" halign=\"left\">orig_bytes</th>\n",
       "      <th colspan=\"4\" halign=\"left\">resp_bytes</th>\n",
       "      <th colspan=\"4\" halign=\"left\">sum_bytes</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>count</th>\n",
       "      <th>count</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>mean</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id.orig_h</th>\n",
       "      <th>id.resp_h</th>\n",
       "      <th>conn_state</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0.0.0</th>\n",
       "      <th>255.255.255.255</th>\n",
       "      <th>S0</th>\n",
       "      <td>1753-12-19 22:43:41.128654848</td>\n",
       "      <td>1774-05-02 22:43:41.128654848</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0</td>\n",
       "      <td>142</td>\n",
       "      <td>17.620690</td>\n",
       "      <td>0</td>\n",
       "      <td>9323</td>\n",
       "      <td>74099</td>\n",
       "      <td>851.712644</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>9323</td>\n",
       "      <td>74099</td>\n",
       "      <td>851.712644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">10.10.10.10</th>\n",
       "      <th>10.255.255.255</th>\n",
       "      <th>S0</th>\n",
       "      <td>1755-03-06 22:43:41.128654848</td>\n",
       "      <td>1769-12-03 22:43:41.128654848</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>52</td>\n",
       "      <td>5.296296</td>\n",
       "      <td>0</td>\n",
       "      <td>1062</td>\n",
       "      <td>4558</td>\n",
       "      <td>168.814815</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1062</td>\n",
       "      <td>4558</td>\n",
       "      <td>168.814815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192.168.202.78</th>\n",
       "      <th>OTH</th>\n",
       "      <td>1753-11-09 22:43:41.128654848</td>\n",
       "      <td>1773-12-17 22:43:41.128654848</td>\n",
       "      <td>34</td>\n",
       "      <td>34</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>95</td>\n",
       "      <td>10.352941</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                time  \\\n",
       "                                                                 min   \n",
       "id.orig_h   id.resp_h       conn_state                                 \n",
       "0.0.0.0     255.255.255.255 S0         1753-12-19 22:43:41.128654848   \n",
       "10.10.10.10 10.255.255.255  S0         1755-03-06 22:43:41.128654848   \n",
       "            192.168.202.78  OTH        1753-11-09 22:43:41.128654848   \n",
       "\n",
       "                                                                            \\\n",
       "                                                                 max count   \n",
       "id.orig_h   id.resp_h       conn_state                                       \n",
       "0.0.0.0     255.255.255.255 S0         1774-05-02 22:43:41.128654848    87   \n",
       "10.10.10.10 10.255.255.255  S0         1769-12-03 22:43:41.128654848    27   \n",
       "            192.168.202.78  OTH        1773-12-17 22:43:41.128654848    34   \n",
       "\n",
       "                                       id.resp_p   uid duration       \\\n",
       "                                           count count      min  max   \n",
       "id.orig_h   id.resp_h       conn_state                                 \n",
       "0.0.0.0     255.255.255.255 S0                87    87        0  142   \n",
       "10.10.10.10 10.255.255.255  S0                27    27        0   52   \n",
       "            192.168.202.78  OTH               34    34        0   95   \n",
       "\n",
       "                                                  orig_bytes               \\\n",
       "                                             mean        min   max    sum   \n",
       "id.orig_h   id.resp_h       conn_state                                      \n",
       "0.0.0.0     255.255.255.255 S0          17.620690          0  9323  74099   \n",
       "10.10.10.10 10.255.255.255  S0           5.296296          0  1062   4558   \n",
       "            192.168.202.78  OTH         10.352941          0     0      0   \n",
       "\n",
       "                                                   resp_bytes               \\\n",
       "                                              mean        min max sum mean   \n",
       "id.orig_h   id.resp_h       conn_state                                       \n",
       "0.0.0.0     255.255.255.255 S0          851.712644          0   0   0  0.0   \n",
       "10.10.10.10 10.255.255.255  S0          168.814815          0   0   0  0.0   \n",
       "            192.168.202.78  OTH           0.000000          0   0   0  0.0   \n",
       "\n",
       "                                       sum_bytes                           \n",
       "                                             min   max    sum        mean  \n",
       "id.orig_h   id.resp_h       conn_state                                     \n",
       "0.0.0.0     255.255.255.255 S0                 0  9323  74099  851.712644  \n",
       "10.10.10.10 10.255.255.255  S0                 0  1062   4558  168.814815  \n",
       "            192.168.202.78  OTH                0     0      0    0.000000  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('# rows', len(cdf_summary))\n",
    "cdf_summary.head(3).to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Visualize data\n",
    "\n",
    "* Nodes: \n",
    "  * IPs\n",
    "  * Bigger when more sessions (split by connection state) involving them\n",
    "* Edges:\n",
    "  * src_ip -> dest_ip, split by connection state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['id.orig_h', 'id.resp_h', 'conn_state', 'timemin', 'timemax',\n",
       "       'timecount', 'id.resp_pcount', 'uidcount', 'durationmin', 'durationmax',\n",
       "       'durationmean', 'orig_bytesmin', 'orig_bytesmax', 'orig_bytessum',\n",
       "       'orig_bytesmean', 'resp_bytesmin', 'resp_bytesmax', 'resp_bytessum',\n",
       "       'resp_bytesmean', 'sum_bytesmin', 'sum_bytesmax', 'sum_bytessum',\n",
       "       'sum_bytesmean'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# flatten multi-index\n",
    "cdfs2 = cdf_summary.copy(deep=False).reset_index()\n",
    "cdfs2.columns = [''.join(c) for c in cdfs2.columns]\n",
    "cdfs2.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# links 50140\n",
      "# events 50140\n",
      "# attrib entities 5048\n"
     ]
    }
   ],
   "source": [
    "hg = graphistry.hypergraph(\n",
    "    cdfs2,\n",
    "    ['id.orig_h', 'id.resp_h'],\n",
    "    direct=True,\n",
    "    opts={\n",
    "        'CATEGORIES': {\n",
    "            'ip': ['id.orig_h', 'id.resp_h']\n",
    "        }\n",
    "    }, engine='cudf') # or use default of 'pandas' with cdfs2.to_pandas() above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <iframe id=\"12552dd8-bbef-4321-8042-bff41773d6c0\" src=\"http://18.116.65.247//graph/graph.html?dataset=9a722f4723ed4f5ca389b882306c3bdb&type=arrow&viztoken=049fd740-b86a-477c-b5a2-f10d15415d14&usertag=e771a43f-pygraphistry-0.27.3&splashAfter=1669098036&info=true\"\n",
       "                    allowfullscreen=\"true\" webkitallowfullscreen=\"true\" mozallowfullscreen=\"true\"\n",
       "                    oallowfullscreen=\"true\" msallowfullscreen=\"true\"\n",
       "                    style=\"width:100%; height:500px; border: 1px solid #DDD; overflow: hidden\"\n",
       "                    \n",
       "            >\n",
       "            </iframe>\n",
       "        \n",
       "            <script>\n",
       "                try {\n",
       "                  $(\"#12552dd8-bbef-4321-8042-bff41773d6c0\").bind('mousewheel', function(e) { e.preventDefault(); });\n",
       "                } catch (e) { console.error('exn catching scroll', e); }\n",
       "            </script>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hg['graph'].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "* Part I: [CPU Baseline in Python Pandas](./part_i_cpu_pandas.ipynb)\n",
    "* Part II: [GPU Dataframe with RAPIDS Python cudf bindings](./part_ii_gpu_cudf.ipynb)\n",
    "* Part III: GPU SQL - deprecated as Dask-SQL replaced BlazingSQL in the RAPIDS ecosystem\n",
    "* Part IV: [GPU ML with RAPIDS cuML UMAP and PyGraphistry](./part_iv_gpu_cuml.ipynb)\n",
    "* [Graphistry cuGraph bindings](./cugraph.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}