3_Probability.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Probability\n",
    "\n",
    "**Start downloading this file and place it in example_pcaps: https://drive.google.com/file/d/1Lr1dleCbZcQWfHoW_u6Q2uZFte17Y2Z_/view?usp=sharing. You'll need it later!**\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Today, we're going to explore probability. The concept of probability is a powerful tool that lets us answer interesting questions about our data, and it serves as the foundation of a commonly used machine learning technique for classification We'll also be building a Naive Bayes classifier from scratch, so you'll get hands-on experience coding a machine learning classifier from scratch!\n",
    "\n",
    "Let's start with some simple probability examples on the board. Let's see how much you can recall from lecture!\n",
    "\n",
    "Say I have a bucket with 10 blue balls and 20 red balls. If I choose a ball at random from the bucket, what is the probability that I choose a red ball? That is, we want to calculate:\n",
    "\n",
    "$P($red ball$)\\ =\\ ??$\n",
    "\n",
    "This is equal to the fraction of red balls over the total number of balls.\n",
    "\n",
    "$P($red ball$)\\ =\\ \\frac{\\text{# of red balls}}{\\text{# of total balls}}\\ =\\ \\frac{20}{30}\\ =\\ \\frac{2}{3}$\n",
    "\n",
    "Similarly, the chance of picking a blue ball is:\n",
    "\n",
    "$P($blue ball$)\\ =\\ \\frac{\\text{# of blue balls}}{\\text{# of total balls}}\\ =\\ \\frac{10}{30}\\ =\\ \\frac{1}{3}$\n",
    "\n",
    "Now, let's say we want to find the probability of picking a red ball out of the bucket, **AND THEN** picking a blue ball out of the bucket. When we want to find the probability of two events both occurring, we multiply their probabilities together. The resulting probability is:\n",
    "\n",
    "$P(\\text{red ball})*P(\\text{blue ball}\\ |\\ \\text{red ball missing})$\n",
    "\n",
    "Here, we introduce the concept of conditional probability. $P(\\text{blue ball}\\ |\\ \\text{red ball missing})$ represents the probability that a blue ball is pulled from the bucket, **given** that a red ball has already been taken out.\n",
    "\n",
    "Are these two events independent? Does pulling a red ball affect the result of the probability of pulling a red ball followed by a blue ball? If it had no effect, the overall probability would be equivalent to:\n",
    "\n",
    "$P(\\text{red ball})*P(\\text{blue ball})$\n",
    "\n",
    "But it's not! By removing a red ball, there are now fewer overall balls to choose from, which changes the resulting probability. The full probability is therefore calculated as:\n",
    "\n",
    "$P(\\text{red ball})*P(\\text{blue ball}\\ |\\ \\text{red ball missing})\\ =\\ \\frac{20}{30}*\\frac{10}{29}\\ =\\ \\frac{20}{87}$\n",
    "\n",
    "Now that you've had a chance to review, let's dive into the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Probability Analysis on Network Traffic\n",
    "\n",
    "### Probability of a TCP Packet\n",
    "\n",
    "Let's compute the probability that a packet from our capture was a TCP packet:\n",
    "\n",
    "$P(\\text{TCP Packet})\\ =\\ \\frac{\\text{# of TCP packets}}{\\text{# of total packets}}$\n",
    "\n",
    "We'll start by loading some captured data into Python, and filtering out packets that don't have a DNS query field or a DNS response field. You'll need to fill in the blanks with the correct information. For tcp_packets, there are three options for each blank: \"data\", \"protocol\", or \"TCP\". Consult yesterday's lab if you need!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from data_collection.parse_pcap import pcap_to_pandas\n",
    "from utils import *\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime, timezone\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from math import log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# call our helper \"pcap_to_pandas\" function, and pass in the argument \"example_pcaps/tplink_switch.pcap\"\n",
    "data = pcap_to_pandas('example_pcaps/tplink_switch.pcap') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a quick look at the data.  How many packets are there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(207, 18)\n"
     ]
    }
   ],
   "source": [
    "num_total_packets = len(data) # number of total packets\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>dns_query</th>\n",
       "      <th>dns_resp</th>\n",
       "      <th>ip_dst</th>\n",
       "      <th>ip_dst_int</th>\n",
       "      <th>ip_src</th>\n",
       "      <th>ip_src_int</th>\n",
       "      <th>is_dns</th>\n",
       "      <th>length</th>\n",
       "      <th>mac_dst</th>\n",
       "      <th>mac_dst_int</th>\n",
       "      <th>mac_src</th>\n",
       "      <th>mac_src_int</th>\n",
       "      <th>port_dst</th>\n",
       "      <th>port_src</th>\n",
       "      <th>protocol</th>\n",
       "      <th>time</th>\n",
       "      <th>time_normed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-12-07 14:11:24.315450</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>113</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-12-07 14:11:24.415702</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>113</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>0.100252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-12-07 14:11:25.240070</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>135</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>0.924620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-12-07 14:11:25.241256</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>169</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>0.925806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-12-07 14:11:25.242138</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>135</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>0.926688</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    datetime dns_query dns_resp ip_dst  ip_dst_int ip_src  \\\n",
       "0 2017-12-07 14:11:24.315450      None     None   None         NaN   None   \n",
       "1 2017-12-07 14:11:24.415702      None     None   None         NaN   None   \n",
       "2 2017-12-07 14:11:25.240070      None     None   None         NaN   None   \n",
       "3 2017-12-07 14:11:25.241256      None     None   None         NaN   None   \n",
       "4 2017-12-07 14:11:25.242138      None     None   None         NaN   None   \n",
       "\n",
       "   ip_src_int  is_dns  length            mac_dst      mac_dst_int  \\\n",
       "0         NaN   False     113  50:c7:bf:09:f3:4c   88818833814348   \n",
       "1         NaN   False     113  50:c7:bf:09:f3:4c   88818833814348   \n",
       "2         NaN   False     135  b8:27:eb:2d:24:15  202481588839445   \n",
       "3         NaN   False     169  50:c7:bf:09:f3:4c   88818833814348   \n",
       "4         NaN   False     135  b8:27:eb:2d:24:15  202481588839445   \n",
       "\n",
       "             mac_src      mac_src_int  port_dst  port_src protocol  \\\n",
       "0  b8:27:eb:2d:24:15  202481588839445       NaN       NaN     None   \n",
       "1  b8:27:eb:2d:24:15  202481588839445       NaN       NaN     None   \n",
       "2  50:c7:bf:09:f3:4c   88818833814348       NaN       NaN     None   \n",
       "3  b8:27:eb:2d:24:15  202481588839445       NaN       NaN     None   \n",
       "4  50:c7:bf:09:f3:4c   88818833814348       NaN       NaN     None   \n",
       "\n",
       "           time  time_normed  \n",
       "0  1.512677e+09     0.000000  \n",
       "1  1.512677e+09     0.100252  \n",
       "2  1.512677e+09     0.924620  \n",
       "3  1.512677e+09     0.925806  \n",
       "4  1.512677e+09     0.926688  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Probability: TCP Packets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>dns_query</th>\n",
       "      <th>dns_resp</th>\n",
       "      <th>ip_dst</th>\n",
       "      <th>ip_dst_int</th>\n",
       "      <th>ip_src</th>\n",
       "      <th>ip_src_int</th>\n",
       "      <th>is_dns</th>\n",
       "      <th>length</th>\n",
       "      <th>mac_dst</th>\n",
       "      <th>mac_dst_int</th>\n",
       "      <th>mac_src</th>\n",
       "      <th>mac_src_int</th>\n",
       "      <th>port_dst</th>\n",
       "      <th>port_src</th>\n",
       "      <th>protocol</th>\n",
       "      <th>time</th>\n",
       "      <th>time_normed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>2017-12-07 14:11:31.780800</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>34.195.88.49</td>\n",
       "      <td>5.832274e+08</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>74</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>50443.0</td>\n",
       "      <td>55594.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>7.465350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>2017-12-07 14:11:31.790298</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>34.195.88.49</td>\n",
       "      <td>5.832274e+08</td>\n",
       "      <td>False</td>\n",
       "      <td>74</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>55594.0</td>\n",
       "      <td>50443.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>7.474848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>2017-12-07 14:11:31.791840</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>34.195.88.49</td>\n",
       "      <td>5.832274e+08</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>66</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>50443.0</td>\n",
       "      <td>55594.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>7.476390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>2017-12-07 14:11:32.208559</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>34.195.88.49</td>\n",
       "      <td>5.832274e+08</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>281</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>50443.0</td>\n",
       "      <td>55594.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>7.893109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>2017-12-07 14:11:32.216527</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>34.195.88.49</td>\n",
       "      <td>5.832274e+08</td>\n",
       "      <td>False</td>\n",
       "      <td>66</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>55594.0</td>\n",
       "      <td>50443.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>7.901077</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     datetime dns_query dns_resp        ip_dst    ip_dst_int  \\\n",
       "18 2017-12-07 14:11:31.780800      None     None  34.195.88.49  5.832274e+08   \n",
       "19 2017-12-07 14:11:31.790298      None     None   172.24.1.81  2.887254e+09   \n",
       "20 2017-12-07 14:11:31.791840      None     None  34.195.88.49  5.832274e+08   \n",
       "23 2017-12-07 14:11:32.208559      None     None  34.195.88.49  5.832274e+08   \n",
       "24 2017-12-07 14:11:32.216527      None     None   172.24.1.81  2.887254e+09   \n",
       "\n",
       "          ip_src    ip_src_int  is_dns  length            mac_dst  \\\n",
       "18   172.24.1.81  2.887254e+09   False      74  b8:27:eb:2d:24:15   \n",
       "19  34.195.88.49  5.832274e+08   False      74  50:c7:bf:09:f3:4c   \n",
       "20   172.24.1.81  2.887254e+09   False      66  b8:27:eb:2d:24:15   \n",
       "23   172.24.1.81  2.887254e+09   False     281  b8:27:eb:2d:24:15   \n",
       "24  34.195.88.49  5.832274e+08   False      66  50:c7:bf:09:f3:4c   \n",
       "\n",
       "        mac_dst_int            mac_src      mac_src_int  port_dst  port_src  \\\n",
       "18  202481588839445  50:c7:bf:09:f3:4c   88818833814348   50443.0   55594.0   \n",
       "19   88818833814348  b8:27:eb:2d:24:15  202481588839445   55594.0   50443.0   \n",
       "20  202481588839445  50:c7:bf:09:f3:4c   88818833814348   50443.0   55594.0   \n",
       "23  202481588839445  50:c7:bf:09:f3:4c   88818833814348   50443.0   55594.0   \n",
       "24   88818833814348  b8:27:eb:2d:24:15  202481588839445   55594.0   50443.0   \n",
       "\n",
       "   protocol          time  time_normed  \n",
       "18      TCP  1.512677e+09     7.465350  \n",
       "19      TCP  1.512677e+09     7.474848  \n",
       "20      TCP  1.512677e+09     7.476390  \n",
       "23      TCP  1.512677e+09     7.893109  \n",
       "24      TCP  1.512677e+09     7.901077  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# packets with the protocol column equal to \"TCP\"\n",
    "tcp_packets = data.loc[data['protocol'] == 'TCP'] \n",
    "tcp_packets.head(n=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(142, 18)\n"
     ]
    }
   ],
   "source": [
    "print(tcp_packets.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(142, 18)\n",
      "0.6859903381642513\n"
     ]
    }
   ],
   "source": [
    "# len gives the number of packets in some data\n",
    "# number of TCP packets\n",
    "num_tcp_packets = len(tcp_packets)\n",
    "print(tcp_packets.shape)\n",
    "tcp_probability = num_tcp_packets / num_total_packets # probability that a packet is a TCP packet\n",
    "\n",
    "print(tcp_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Probability: UDP Packets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32, 18)\n",
      "0.15458937198067632\n"
     ]
    }
   ],
   "source": [
    "# len gives the number of packets in some data\n",
    "# number of TCP packets\n",
    "udp_packets = data.loc[data['protocol'] == 'UDP']\n",
    "num_udp_packets = len(udp_packets) \n",
    "print(udp_packets.shape)\n",
    "\n",
    "udp_probability = num_udp_packets / num_total_packets # probability that a packet is a UDP packet\n",
    "\n",
    "print(udp_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conditional Probability: DNS Packets Given Port 53\n",
    "\n",
    "A DNS packet is a DNS query **OR** a DNS response. DNS traffic is typically sent or received on port 53. Let's compute the probability that a packet is a DNS packet, given that the source port or destination port is 53. \n",
    "\n",
    "We are calculating:\n",
    "\n",
    "$P($DNS Query $\\cup$ DNS Response | Source Port == 53 $\\cup$ Dst Port == 53$)$\n",
    "\n",
    "The $\\cup$ means \"union\".\n",
    "\n",
    "The probability can be calculated as:\n",
    "\n",
    "$P(\\text{DNS Query} \\cup \\text{DNS Response}\\ |\\ \\text{Source Port == 53} \\cup \\text{Dst Port == 53})\\ =\\ \\frac{\\text{# of packets with a DNS query or DNS response field}}{\\text{# of packets with a SRC port or DST port 53}}$\n",
    "\n",
    "According to conditional probability, rather than dividing by the total number of packets, we divide by only the # of packets that satisfy the condition that the SRC or DST port is equal to 53.\n",
    "\n",
    "For dns_queries and dns_responses, there are three options for each blank: \"data\", \"dns_query\", or \"dns_resp\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>dns_query</th>\n",
       "      <th>dns_resp</th>\n",
       "      <th>ip_dst</th>\n",
       "      <th>ip_dst_int</th>\n",
       "      <th>ip_src</th>\n",
       "      <th>ip_src_int</th>\n",
       "      <th>is_dns</th>\n",
       "      <th>length</th>\n",
       "      <th>mac_dst</th>\n",
       "      <th>mac_dst_int</th>\n",
       "      <th>mac_src</th>\n",
       "      <th>mac_src_int</th>\n",
       "      <th>port_dst</th>\n",
       "      <th>port_src</th>\n",
       "      <th>protocol</th>\n",
       "      <th>time</th>\n",
       "      <th>time_normed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>2017-12-07 14:11:31.775682</td>\n",
       "      <td>b'devs.tplinkcloud.com.'</td>\n",
       "      <td>b'devs.tplinkcloud.com.'</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>172.24.1.1</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>True</td>\n",
       "      <td>533</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>43866.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>UDP</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>7.460232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>2017-12-07 14:11:31.885528</td>\n",
       "      <td>b's1a.time.edu.cn.'</td>\n",
       "      <td>b's1a.time.edu.cn.'</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>172.24.1.1</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>True</td>\n",
       "      <td>121</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>32835.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>UDP</td>\n",
       "      <td>1.512677e+09</td>\n",
       "      <td>7.570078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>2017-12-07 14:11:47.952146</td>\n",
       "      <td>b's1b.time.edu.cn.'</td>\n",
       "      <td>b's1b.time.edu.cn.'</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>172.24.1.1</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>True</td>\n",
       "      <td>91</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>39900.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>UDP</td>\n",
       "      <td>1.512678e+09</td>\n",
       "      <td>23.636696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>2017-12-07 14:12:04.028974</td>\n",
       "      <td>b'0.cn.pool.ntp.org.'</td>\n",
       "      <td>b'0.cn.pool.ntp.org.'</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>172.24.1.1</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>True</td>\n",
       "      <td>141</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>55754.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>UDP</td>\n",
       "      <td>1.512678e+09</td>\n",
       "      <td>39.713524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>2017-12-07 14:16:20.105092</td>\n",
       "      <td>b'fr.pool.ntp.org.'</td>\n",
       "      <td>b'fr.pool.ntp.org.'</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>172.24.1.1</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>True</td>\n",
       "      <td>513</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>34673.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>UDP</td>\n",
       "      <td>1.512678e+09</td>\n",
       "      <td>295.789642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>2017-12-07 14:16:20.364034</td>\n",
       "      <td>b'devs.tplinkcloud.com.'</td>\n",
       "      <td>b'devs.tplinkcloud.com.'</td>\n",
       "      <td>172.24.1.81</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>172.24.1.1</td>\n",
       "      <td>2.887254e+09</td>\n",
       "      <td>True</td>\n",
       "      <td>533</td>\n",
       "      <td>50:c7:bf:09:f3:4c</td>\n",
       "      <td>88818833814348</td>\n",
       "      <td>b8:27:eb:2d:24:15</td>\n",
       "      <td>202481588839445</td>\n",
       "      <td>59227.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>UDP</td>\n",
       "      <td>1.512678e+09</td>\n",
       "      <td>296.048584</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      datetime                 dns_query  \\\n",
       "17  2017-12-07 14:11:31.775682  b'devs.tplinkcloud.com.'   \n",
       "21  2017-12-07 14:11:31.885528       b's1a.time.edu.cn.'   \n",
       "58  2017-12-07 14:11:47.952146       b's1b.time.edu.cn.'   \n",
       "70  2017-12-07 14:12:04.028974     b'0.cn.pool.ntp.org.'   \n",
       "119 2017-12-07 14:16:20.105092       b'fr.pool.ntp.org.'   \n",
       "123 2017-12-07 14:16:20.364034  b'devs.tplinkcloud.com.'   \n",
       "\n",
       "                     dns_resp       ip_dst    ip_dst_int      ip_src  \\\n",
       "17   b'devs.tplinkcloud.com.'  172.24.1.81  2.887254e+09  172.24.1.1   \n",
       "21        b's1a.time.edu.cn.'  172.24.1.81  2.887254e+09  172.24.1.1   \n",
       "58        b's1b.time.edu.cn.'  172.24.1.81  2.887254e+09  172.24.1.1   \n",
       "70      b'0.cn.pool.ntp.org.'  172.24.1.81  2.887254e+09  172.24.1.1   \n",
       "119       b'fr.pool.ntp.org.'  172.24.1.81  2.887254e+09  172.24.1.1   \n",
       "123  b'devs.tplinkcloud.com.'  172.24.1.81  2.887254e+09  172.24.1.1   \n",
       "\n",
       "       ip_src_int  is_dns  length            mac_dst     mac_dst_int  \\\n",
       "17   2.887254e+09    True     533  50:c7:bf:09:f3:4c  88818833814348   \n",
       "21   2.887254e+09    True     121  50:c7:bf:09:f3:4c  88818833814348   \n",
       "58   2.887254e+09    True      91  50:c7:bf:09:f3:4c  88818833814348   \n",
       "70   2.887254e+09    True     141  50:c7:bf:09:f3:4c  88818833814348   \n",
       "119  2.887254e+09    True     513  50:c7:bf:09:f3:4c  88818833814348   \n",
       "123  2.887254e+09    True     533  50:c7:bf:09:f3:4c  88818833814348   \n",
       "\n",
       "               mac_src      mac_src_int  port_dst  port_src protocol  \\\n",
       "17   b8:27:eb:2d:24:15  202481588839445   43866.0      53.0      UDP   \n",
       "21   b8:27:eb:2d:24:15  202481588839445   32835.0      53.0      UDP   \n",
       "58   b8:27:eb:2d:24:15  202481588839445   39900.0      53.0      UDP   \n",
       "70   b8:27:eb:2d:24:15  202481588839445   55754.0      53.0      UDP   \n",
       "119  b8:27:eb:2d:24:15  202481588839445   34673.0      53.0      UDP   \n",
       "123  b8:27:eb:2d:24:15  202481588839445   59227.0      53.0      UDP   \n",
       "\n",
       "             time  time_normed  \n",
       "17   1.512677e+09     7.460232  \n",
       "21   1.512677e+09     7.570078  \n",
       "58   1.512678e+09    23.636696  \n",
       "70   1.512678e+09    39.713524  \n",
       "119  1.512678e+09   295.789642  \n",
       "123  1.512678e+09   296.048584  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# packets with a DNS query column that isn't None\n",
    "dns_queries = data.loc[data[\"dns_query\"].notnull()] # This produces garbage.\n",
    "dns_queries = dns_queries.loc[dns_queries[\"port_dst\"] == 53]\n",
    "\n",
    "# packets with a DNS response column that isn't Nonedns_queries = dns_queries.loc[dns_queries[\"port_dst\"] == 53]\n",
    "dns_responses = data.loc[data[\"dns_resp\"].notnull()] # This produces garbage.\n",
    "dns_responses = dns_responses.loc[dns_responses[\"port_src\"] == 53]\n",
    "\n",
    "dns_responses.head(n=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should expect one response for each query. Let's check that assumption."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(6, 18)\n",
      "(6, 18)\n"
     ]
    }
   ],
   "source": [
    "print(dns_queries.shape)\n",
    "print(dns_responses.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now see how many packets have either a source port of 53 or a destination port of 53."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(6, 18)\n",
      "(6, 18)\n"
     ]
    }
   ],
   "source": [
    "src_port_53 = data.loc[data[\"port_src\"] == 53]\n",
    "dst_port_53 = data.loc[data[\"port_dst\"] == 53]\n",
    "print(src_port_53.shape)\n",
    "print(dst_port_53.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12\n",
      "12\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "num_dns_queries = len(dns_queries)\n",
    "num_dns_responses = len(dns_responses)\n",
    "\n",
    "num_dns_total = num_dns_queries + num_dns_responses\n",
    "\n",
    "# check the total\n",
    "print(num_dns_total)\n",
    "\n",
    "num_port_53 = len(src_port_53) + len(dst_port_53)\n",
    "\n",
    "# should be the same\n",
    "print(num_port_53)\n",
    "\n",
    "# probability that a packet is a DNS packet, given that at least one port is 53\n",
    "dns_probability = (num_dns_queries + num_dns_responses) / num_port_53 \n",
    "print(dns_probability) # Should be 1 (100%)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should expect an answer of 100%. If you got over 100% instead, your probability is likely overcounting some packets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Probability that a DNS Response is Longer than the Mean Packet Length\n",
    "\n",
    "Now let's answer the following questions about our packets: \n",
    "What is the probability that a given DNS response has a length longer than the average length of all packets?\n",
    "\n",
    "$P($Length > Mean Length of **All** Packets | DNS Response$)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "238.58937198067633\n"
     ]
    }
   ],
   "source": [
    "# the mean length of all packets\n",
    "mean_length = data['length'].mean() \n",
    "print(mean_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "322.0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the mean length of DNS responses\n",
    "dns_responses['length'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n",
      "0.5\n"
     ]
    }
   ],
   "source": [
    "# number of DNS packets with a length longer than mean_length\n",
    "longer_than_mean = dns_responses[dns_responses[\"length\"] > int(mean_length)] \n",
    "num_longer = len(longer_than_mean)\n",
    "print(num_longer)\n",
    "\n",
    "prob_longer = num_longer / len(dns_responses)\n",
    "print(prob_longer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Homework Exercises\n",
    "\n",
    "(Challenge!) Find the probability that a DNS request is immediately followed by a DNS response in the packet trace. This will give us an idea of how fast DNS responses are received, relative to other network traffic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Naïve Bayes Classifier\n",
    "\n",
    "Now we're going to use the naïve Bayes algorithm to predict which task a user is most likely doing given a particular packet. While there are existing python functions for performing a naive Bayes classification, we already know everything we need to do it ourselves!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the Packets and the Labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first need to label the data with what activity was happening at the time each packet is received.\n",
    "\n",
    "First, download the ross.pcap file at https://drive.google.com/file/d/1Lr1dleCbZcQWfHoW_u6Q2uZFte17Y2Z_/view?usp=sharing. Place it in the example_pcaps folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>activity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-07-30 14:51:41.327734</td>\n",
       "      <td>WEB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-07-30 14:54:12.815653</td>\n",
       "      <td>AUDIO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-07-30 14:56:09.083618</td>\n",
       "      <td>VIDEO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-07-30 14:58:24.929799</td>\n",
       "      <td>WEB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-07-30 14:58:33.808876</td>\n",
       "      <td>GAMING</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         time activity\n",
       "0  2018-07-30 14:51:41.327734      WEB\n",
       "1  2018-07-30 14:54:12.815653    AUDIO\n",
       "2  2018-07-30 14:56:09.083618    VIDEO\n",
       "3  2018-07-30 14:58:24.929799      WEB\n",
       "4  2018-07-30 14:58:33.808876   GAMING"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the data, may take a few minutes.\n",
    "data = pcap_to_pandas(\"example_pcaps/ross.pcap\")\n",
    "labels = pd.read_csv('example_pcaps/ross_labels.txt', header=None, names=[\"time\", \"activity\"])\n",
    "labels.head(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's add a UNIX timestamp (remember, this is measured in seconds since the epoch) to the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>activity</th>\n",
       "      <th>datetime</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-07-30 14:51:41.327734</td>\n",
       "      <td>WEB</td>\n",
       "      <td>2018-07-30 14:51:41.327734</td>\n",
       "      <td>1.532980e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-07-30 14:54:12.815653</td>\n",
       "      <td>AUDIO</td>\n",
       "      <td>2018-07-30 14:54:12.815653</td>\n",
       "      <td>1.532980e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-07-30 14:56:09.083618</td>\n",
       "      <td>VIDEO</td>\n",
       "      <td>2018-07-30 14:56:09.083618</td>\n",
       "      <td>1.532981e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-07-30 14:58:24.929799</td>\n",
       "      <td>WEB</td>\n",
       "      <td>2018-07-30 14:58:24.929799</td>\n",
       "      <td>1.532981e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-07-30 14:58:33.808876</td>\n",
       "      <td>GAMING</td>\n",
       "      <td>2018-07-30 14:58:33.808876</td>\n",
       "      <td>1.532981e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2018-07-30 14:00:20.571626</td>\n",
       "      <td>INACTIVE</td>\n",
       "      <td>2018-07-30 14:00:20.571626</td>\n",
       "      <td>1.532977e+09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         time   activity                   datetime  \\\n",
       "0  2018-07-30 14:51:41.327734        WEB 2018-07-30 14:51:41.327734   \n",
       "1  2018-07-30 14:54:12.815653      AUDIO 2018-07-30 14:54:12.815653   \n",
       "2  2018-07-30 14:56:09.083618      VIDEO 2018-07-30 14:56:09.083618   \n",
       "3  2018-07-30 14:58:24.929799        WEB 2018-07-30 14:58:24.929799   \n",
       "4  2018-07-30 14:58:33.808876     GAMING 2018-07-30 14:58:33.808876   \n",
       "5  2018-07-30 14:00:20.571626   INACTIVE 2018-07-30 14:00:20.571626   \n",
       "\n",
       "      timestamp  \n",
       "0  1.532980e+09  \n",
       "1  1.532980e+09  \n",
       "2  1.532981e+09  \n",
       "3  1.532981e+09  \n",
       "4  1.532981e+09  \n",
       "5  1.532977e+09  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dateutil import tz\n",
    "\n",
    "def convert_to_datetime(time):\n",
    "    return datetime.strptime(time, '%Y-%m-%d %H:%M:%S.%f')\n",
    "    \n",
    "labels['datetime'] = labels['time'].apply(convert_to_datetime)\n",
    "\n",
    "# Force GMT -0400\n",
    "tzlocal = tz.gettz('CDT')\n",
    "#tzlocal = datetime.now().astimezone().tzinfo\n",
    "labels['timestamp'] = labels['datetime'].apply(lambda dt: dt.replace(tzinfo=tzlocal).timestamp())\n",
    "\n",
    "labels.head(n=50)\n",
    "#print(tzlocal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assign an activity label to each row in the packet trace.\n",
    "\n",
    "Next, we're going to use the activity log to label the the packet trace. \n",
    "\n",
    "We use a **label encoder** to assign an integer label for each activity in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(6, 5)\n",
      "4\n",
      "<class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "label_encoder = LabelEncoder()\n",
    "labels['label'] = label_encoder.fit_transform(labels['activity'])\n",
    "\n",
    "print(labels.shape)\n",
    "num_labels = max(labels['label'])\n",
    "print(num_labels)\n",
    "\n",
    "labels.head(n=6)\n",
    "print(type(labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.532980e+09\n",
       "1    1.532980e+09\n",
       "2    1.532980e+09\n",
       "3    1.532980e+09\n",
       "4    1.532980e+09\n",
       "5    1.532980e+09\n",
       "6    1.532980e+09\n",
       "7    1.532980e+09\n",
       "8    1.532980e+09\n",
       "9    1.532980e+09\n",
       "Name: time, dtype: float64"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for index, row in labels.sort_values(by=['timestamp']).iterrows():\n",
    "    # bug in assigning labels of activities to the packet data\n",
    "    data.loc[data['time'] >= row['timestamp'], 'label'] = row['label']\n",
    "    \n",
    "\n",
    "#data.loc[data['label' == 2]]\n",
    "data[data['label'] != 2].head(n=20)\n",
    "data['time'].head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0          WEB\n",
      "1        AUDIO\n",
      "2        VIDEO\n",
      "3          WEB\n",
      "4       GAMING\n",
      "5     INACTIVE\n",
      "Name: activity, dtype: object 0   2018-07-30 14:51:41.327734\n",
      "1   2018-07-30 14:54:12.815653\n",
      "2   2018-07-30 14:56:09.083618\n",
      "3   2018-07-30 14:58:24.929799\n",
      "4   2018-07-30 14:58:33.808876\n",
      "5   2018-07-30 14:00:20.571626\n",
      "Name: datetime, dtype: datetime64[ns]\n",
      "4\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>dns_query</th>\n",
       "      <th>dns_resp</th>\n",
       "      <th>ip_dst</th>\n",
       "      <th>ip_dst_int</th>\n",
       "      <th>ip_src</th>\n",
       "      <th>ip_src_int</th>\n",
       "      <th>is_dns</th>\n",
       "      <th>length</th>\n",
       "      <th>mac_dst</th>\n",
       "      <th>mac_dst_int</th>\n",
       "      <th>mac_src</th>\n",
       "      <th>mac_src_int</th>\n",
       "      <th>port_dst</th>\n",
       "      <th>port_src</th>\n",
       "      <th>protocol</th>\n",
       "      <th>time</th>\n",
       "      <th>time_normed</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-07-30 14:51:40.670566</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>255.255.255.255</td>\n",
       "      <td>4.294967e+09</td>\n",
       "      <td>128.112.93.99</td>\n",
       "      <td>2.154848e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>184</td>\n",
       "      <td>ff:ff:ff:ff:ff:ff</td>\n",
       "      <td>281474976710655</td>\n",
       "      <td>0c:4d:e9:b0:8e:4b</td>\n",
       "      <td>13528772677195</td>\n",
       "      <td>17500.0</td>\n",
       "      <td>17500.0</td>\n",
       "      <td>UDP</td>\n",
       "      <td>1.532980e+09</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-07-30 14:51:40.670856</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>128.112.93.255</td>\n",
       "      <td>2.154848e+09</td>\n",
       "      <td>128.112.93.99</td>\n",
       "      <td>2.154848e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>184</td>\n",
       "      <td>ff:ff:ff:ff:ff:ff</td>\n",
       "      <td>281474976710655</td>\n",
       "      <td>0c:4d:e9:b0:8e:4b</td>\n",
       "      <td>13528772677195</td>\n",
       "      <td>17500.0</td>\n",
       "      <td>17500.0</td>\n",
       "      <td>UDP</td>\n",
       "      <td>1.532980e+09</td>\n",
       "      <td>0.000290</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-07-30 14:51:41.370868</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>162.222.44.11</td>\n",
       "      <td>2.732469e+09</td>\n",
       "      <td>128.112.92.150</td>\n",
       "      <td>2.154847e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>82</td>\n",
       "      <td>04:09:73:5f:c9:00</td>\n",
       "      <td>4438636873984</td>\n",
       "      <td>a8:60:b6:01:d0:a9</td>\n",
       "      <td>185133323899049</td>\n",
       "      <td>4282.0</td>\n",
       "      <td>56524.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.532980e+09</td>\n",
       "      <td>0.700302</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-07-30 14:51:41.370965</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>162.222.44.11</td>\n",
       "      <td>2.732469e+09</td>\n",
       "      <td>128.112.92.150</td>\n",
       "      <td>2.154847e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>1514</td>\n",
       "      <td>04:09:73:5f:c9:00</td>\n",
       "      <td>4438636873984</td>\n",
       "      <td>a8:60:b6:01:d0:a9</td>\n",
       "      <td>185133323899049</td>\n",
       "      <td>4282.0</td>\n",
       "      <td>56524.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.532980e+09</td>\n",
       "      <td>0.700399</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-07-30 14:51:41.370966</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>162.222.44.11</td>\n",
       "      <td>2.732469e+09</td>\n",
       "      <td>128.112.92.150</td>\n",
       "      <td>2.154847e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>1514</td>\n",
       "      <td>04:09:73:5f:c9:00</td>\n",
       "      <td>4438636873984</td>\n",
       "      <td>a8:60:b6:01:d0:a9</td>\n",
       "      <td>185133323899049</td>\n",
       "      <td>4282.0</td>\n",
       "      <td>56524.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.532980e+09</td>\n",
       "      <td>0.700400</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2018-07-30 14:51:41.370966</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>162.222.44.11</td>\n",
       "      <td>2.732469e+09</td>\n",
       "      <td>128.112.92.150</td>\n",
       "      <td>2.154847e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>1514</td>\n",
       "      <td>04:09:73:5f:c9:00</td>\n",
       "      <td>4438636873984</td>\n",
       "      <td>a8:60:b6:01:d0:a9</td>\n",
       "      <td>185133323899049</td>\n",
       "      <td>4282.0</td>\n",
       "      <td>56524.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.532980e+09</td>\n",
       "      <td>0.700400</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2018-07-30 14:51:41.370967</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>162.222.44.11</td>\n",
       "      <td>2.732469e+09</td>\n",
       "      <td>128.112.92.150</td>\n",
       "      <td>2.154847e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>1514</td>\n",
       "      <td>04:09:73:5f:c9:00</td>\n",
       "      <td>4438636873984</td>\n",
       "      <td>a8:60:b6:01:d0:a9</td>\n",
       "      <td>185133323899049</td>\n",
       "      <td>4282.0</td>\n",
       "      <td>56524.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.532980e+09</td>\n",
       "      <td>0.700401</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2018-07-30 14:51:41.370973</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>162.222.44.11</td>\n",
       "      <td>2.732469e+09</td>\n",
       "      <td>128.112.92.150</td>\n",
       "      <td>2.154847e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>1514</td>\n",
       "      <td>04:09:73:5f:c9:00</td>\n",
       "      <td>4438636873984</td>\n",
       "      <td>a8:60:b6:01:d0:a9</td>\n",
       "      <td>185133323899049</td>\n",
       "      <td>4282.0</td>\n",
       "      <td>56524.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.532980e+09</td>\n",
       "      <td>0.700407</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2018-07-30 14:51:41.370974</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>162.222.44.11</td>\n",
       "      <td>2.732469e+09</td>\n",
       "      <td>128.112.92.150</td>\n",
       "      <td>2.154847e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>1514</td>\n",
       "      <td>04:09:73:5f:c9:00</td>\n",
       "      <td>4438636873984</td>\n",
       "      <td>a8:60:b6:01:d0:a9</td>\n",
       "      <td>185133323899049</td>\n",
       "      <td>4282.0</td>\n",
       "      <td>56524.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.532980e+09</td>\n",
       "      <td>0.700408</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2018-07-30 14:51:41.370975</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>162.222.44.11</td>\n",
       "      <td>2.732469e+09</td>\n",
       "      <td>128.112.92.150</td>\n",
       "      <td>2.154847e+09</td>\n",
       "      <td>False</td>\n",
       "      <td>1514</td>\n",
       "      <td>04:09:73:5f:c9:00</td>\n",
       "      <td>4438636873984</td>\n",
       "      <td>a8:60:b6:01:d0:a9</td>\n",
       "      <td>185133323899049</td>\n",
       "      <td>4282.0</td>\n",
       "      <td>56524.0</td>\n",
       "      <td>TCP</td>\n",
       "      <td>1.532980e+09</td>\n",
       "      <td>0.700409</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    datetime dns_query dns_resp           ip_dst  \\\n",
       "0 2018-07-30 14:51:40.670566      None     None  255.255.255.255   \n",
       "1 2018-07-30 14:51:40.670856      None     None   128.112.93.255   \n",
       "2 2018-07-30 14:51:41.370868      None     None    162.222.44.11   \n",
       "3 2018-07-30 14:51:41.370965      None     None    162.222.44.11   \n",
       "4 2018-07-30 14:51:41.370966      None     None    162.222.44.11   \n",
       "5 2018-07-30 14:51:41.370966      None     None    162.222.44.11   \n",
       "6 2018-07-30 14:51:41.370967      None     None    162.222.44.11   \n",
       "7 2018-07-30 14:51:41.370973      None     None    162.222.44.11   \n",
       "8 2018-07-30 14:51:41.370974      None     None    162.222.44.11   \n",
       "9 2018-07-30 14:51:41.370975      None     None    162.222.44.11   \n",
       "\n",
       "     ip_dst_int          ip_src    ip_src_int  is_dns  length  \\\n",
       "0  4.294967e+09   128.112.93.99  2.154848e+09   False     184   \n",
       "1  2.154848e+09   128.112.93.99  2.154848e+09   False     184   \n",
       "2  2.732469e+09  128.112.92.150  2.154847e+09   False      82   \n",
       "3  2.732469e+09  128.112.92.150  2.154847e+09   False    1514   \n",
       "4  2.732469e+09  128.112.92.150  2.154847e+09   False    1514   \n",
       "5  2.732469e+09  128.112.92.150  2.154847e+09   False    1514   \n",
       "6  2.732469e+09  128.112.92.150  2.154847e+09   False    1514   \n",
       "7  2.732469e+09  128.112.92.150  2.154847e+09   False    1514   \n",
       "8  2.732469e+09  128.112.92.150  2.154847e+09   False    1514   \n",
       "9  2.732469e+09  128.112.92.150  2.154847e+09   False    1514   \n",
       "\n",
       "             mac_dst      mac_dst_int            mac_src      mac_src_int  \\\n",
       "0  ff:ff:ff:ff:ff:ff  281474976710655  0c:4d:e9:b0:8e:4b   13528772677195   \n",
       "1  ff:ff:ff:ff:ff:ff  281474976710655  0c:4d:e9:b0:8e:4b   13528772677195   \n",
       "2  04:09:73:5f:c9:00    4438636873984  a8:60:b6:01:d0:a9  185133323899049   \n",
       "3  04:09:73:5f:c9:00    4438636873984  a8:60:b6:01:d0:a9  185133323899049   \n",
       "4  04:09:73:5f:c9:00    4438636873984  a8:60:b6:01:d0:a9  185133323899049   \n",
       "5  04:09:73:5f:c9:00    4438636873984  a8:60:b6:01:d0:a9  185133323899049   \n",
       "6  04:09:73:5f:c9:00    4438636873984  a8:60:b6:01:d0:a9  185133323899049   \n",
       "7  04:09:73:5f:c9:00    4438636873984  a8:60:b6:01:d0:a9  185133323899049   \n",
       "8  04:09:73:5f:c9:00    4438636873984  a8:60:b6:01:d0:a9  185133323899049   \n",
       "9  04:09:73:5f:c9:00    4438636873984  a8:60:b6:01:d0:a9  185133323899049   \n",
       "\n",
       "   port_dst  port_src protocol          time  time_normed  label  \n",
       "0   17500.0   17500.0      UDP  1.532980e+09     0.000000    2.0  \n",
       "1   17500.0   17500.0      UDP  1.532980e+09     0.000290    2.0  \n",
       "2    4282.0   56524.0      TCP  1.532980e+09     0.700302    4.0  \n",
       "3    4282.0   56524.0      TCP  1.532980e+09     0.700399    4.0  \n",
       "4    4282.0   56524.0      TCP  1.532980e+09     0.700400    4.0  \n",
       "5    4282.0   56524.0      TCP  1.532980e+09     0.700400    4.0  \n",
       "6    4282.0   56524.0      TCP  1.532980e+09     0.700401    4.0  \n",
       "7    4282.0   56524.0      TCP  1.532980e+09     0.700407    4.0  \n",
       "8    4282.0   56524.0      TCP  1.532980e+09     0.700408    4.0  \n",
       "9    4282.0   56524.0      TCP  1.532980e+09     0.700409    4.0  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(labels['activity'],labels['datetime'])\n",
    "print(num_labels)\n",
    "data.head(n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create training and testing sets\n",
    "\n",
    "Finally, we're going to take 20% of the data set and reserve it as test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "151503\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0     2.0\n",
       "1     2.0\n",
       "2     4.0\n",
       "3     4.0\n",
       "6     4.0\n",
       "7     4.0\n",
       "8     4.0\n",
       "10    4.0\n",
       "11    4.0\n",
       "13    4.0\n",
       "Name: label, dtype: float64"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "msk = np.random.rand(len(data)) < 0.8\n",
    "train = data[msk]\n",
    "test = data[~msk]\n",
    "\n",
    "training_size = len(train)\n",
    "print(training_size)\n",
    "\n",
    "train['label'].head(n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification\n",
    "\n",
    "The simplest statistic we need to compute is the probability that each label occurs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 0.11236741186643169\n",
      "1 0.29738685042540414\n",
      "2 1.3201058724909737e-05\n",
      "3 0.2916113872332561\n",
      "4 0.2986211494161832\n"
     ]
    }
   ],
   "source": [
    "label_probs = np.zeros(num_labels + 1)\n",
    "\n",
    "for i in range(num_labels + 1):\n",
    "    label_probs[i] = len(train[train['label'] == i]) / training_size\n",
    "    print(i, label_probs[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we are going to go through the training set and tally up what values appear in different fields and how often they appear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "ip_table = {} # The table stores tuples of the form (label, value)\n",
    "ip_list = [] # The list stores every value we've seen (uniquely).\n",
    "\n",
    "dns_table = {}\n",
    "dns_list = []\n",
    "\n",
    "port_table = {}\n",
    "port_list = []\n",
    "\n",
    "protocol_table = {}\n",
    "protocol_list = []\n",
    "\n",
    "################\n",
    "# Function: update_table\n",
    "# input: table, list, label, value\n",
    "# output: \n",
    "# 1. update table to increment value for corresponding (label,value) key\n",
    "# 2. add value to list of unique values (activities)\n",
    "\n",
    "def update_table(table, lst, label, value):\n",
    "    if value is not None:\n",
    "        table[(label, value)] = table.get((label, value), 0) + 1\n",
    "    if value is not None and value not in lst:\n",
    "        lst.append(value)\n",
    "\n",
    "##################\n",
    "        \n",
    "        \n",
    "for index, row in train.iterrows():\n",
    "    # for each row in the training set:\n",
    "    # - count unique source and destination IPs and ports, protocols, DNS queries, and DNS responses\n",
    "    update_table(ip_table, ip_list, row['label'], row['ip_src'])\n",
    "    update_table(ip_table, ip_list, row['label'], row['ip_dst'])\n",
    "    update_table(port_table, port_list, row['label'], row['port_src'])\n",
    "    update_table(port_table, port_list, row['label'], row['port_dst'])\n",
    "    update_table(protocol_table, protocol_list, row['label'], row['protocol'])\n",
    "    update_table(dns_table, dns_list, row['label'], row['dns_query'])\n",
    "    update_table(dns_table, dns_list, row['label'], row['dns_resp'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we use these tallies to compute the logarithm of the event probabilites. \n",
    "\n",
    "Typically, we prefer to work with log probabilities because many of these events have very small chances of happening. When multiplied together the resulting joint probability often end up inconveniently small for computers to work with. \n",
    "\n",
    "Taking logarithms will not change what we are trying to do conceptually, but improves the numerical properties of the algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_log_probs(table, lst, smoothing):\n",
    "    log_probs = {}\n",
    "    \n",
    "    # iterate through all of the labels\n",
    "    for l in range(num_labels + 1):\n",
    "        total = sum([table.get((l, val), 0) for val in lst])\n",
    "        \n",
    "        # lst has the unique values for the keys in the table\n",
    "        for val in lst:\n",
    "            if (l, val) in table:\n",
    "                log_probs[(l, val)] = log(table[(l, val)] + smoothing) - log(total + smoothing * (len(lst) + 1))\n",
    "        \n",
    "        log_probs[(l, '<UNK>')] = log(smoothing) - log(total + smoothing * (len(lst) + 1))\n",
    "    \n",
    "    return log_probs\n",
    "\n",
    "ip_log_prob = compute_log_probs(ip_table, ip_list, 1e-5)\n",
    "dns_log_prob = compute_log_probs(dns_table, dns_list, 1e-5)\n",
    "port_log_prob = compute_log_probs(port_table, port_list, 1e-5)\n",
    "protocol_log_prob = compute_log_probs(protocol_table, protocol_list, 1e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the first 5 elements to get an idea of what we just did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[((0, '128.112.93.99'), -8.824558273367144), ((0, '255.255.255.255'), -7.4895586803171526), ((0, '128.112.93.255'), -7.175901263162451), ((0, '128.112.92.150'), -0.6965061868680138), ((0, '162.222.44.11'), -1.0909642712218623)]\n"
     ]
    }
   ],
   "source": [
    "from itertools import islice\n",
    "\n",
    "def take(n, iterable):\n",
    "    \"Return first n items of the iterable as a list\"\n",
    "    return list(islice(iterable, n))\n",
    "\n",
    "top = take(5,ip_log_prob.items())\n",
    "print(top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each entry shows the log probability of an (activity, value) pair occurring together, for each combination of activity and value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we are ready to create the classifier. When presented with a new row of data, we simply sum all the relevant log probabilities for each class and report the class with the highest log probability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.7176041722101005\n"
     ]
    }
   ],
   "source": [
    "##################\n",
    "# Function: return log probability for a particular value\n",
    "\n",
    "def get_log_prob(table, val, label):\n",
    "    return table.get((label, val), table[(label, '<UNK>')])\n",
    "\n",
    "\n",
    "##################\n",
    "# Function: Naïve Bayes Classifier\n",
    "\n",
    "def classify(row):\n",
    "    best_label = -1\n",
    "    best_label_score = float('-Inf')\n",
    "\n",
    "    # For each possible activity ...\n",
    "    for l in range(num_labels + 1):\n",
    "        score = log(label_probs[l])\n",
    "        \n",
    "        # probability of seeing this src IP, given activity (label) 'l'\n",
    "        # P(IP src == i | activity = <l>)\n",
    "        \n",
    "        score = score + get_log_prob(ip_log_prob, row['ip_src'], l)\n",
    "        \n",
    "        # probability of seeing this dst IP, given activity (label) 'l'\n",
    "        # P(IP src == i | activity = <l>)\n",
    "        score = score + get_log_prob(ip_log_prob, row['ip_dst'], l)\n",
    "        \n",
    "        if row['is_dns']:\n",
    "            if row['dns_query'] is not None:\n",
    "                score = score + get_log_prob(dns_log_prob, row['dns_query'], l)\n",
    "            \n",
    "            if row['dns_resp'] is not None:\n",
    "                score = score + get_log_prob(dns_log_prob, row['dns_resp'], l)\n",
    "        \n",
    "        score = score + get_log_prob(port_log_prob, row['port_src'], l)\n",
    "        score = score + get_log_prob(port_log_prob, row['port_dst'], l)\n",
    "        \n",
    "        score = score + get_log_prob(protocol_log_prob, row['protocol'], l)\n",
    "        \n",
    "        # At the end score is a large joint probability:\n",
    "        # P(IP src = i AND \n",
    "        #   IP dst = i AND ... | activity = <l>)\n",
    "        \n",
    "        \n",
    "        # Is the score we have the largest we have so far?\n",
    "        # If so, this is the most likely label\n",
    "        \n",
    "        if score > best_label_score:\n",
    "            best_label = l\n",
    "            best_label_score = score\n",
    "    \n",
    "    return best_label\n",
    "\n",
    "correct = 0\n",
    "\n",
    "############\n",
    "# Important: Use the test set!\n",
    "\n",
    "for index, row in test.iterrows():\n",
    "    if classify(row) == row['label']:\n",
    "        correct = correct + 1\n",
    "\n",
    "print('Accuracy: {}'.format(correct / len(test)))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}