370_Homework_4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SI 370: Homework 4: Malicious and Benign Websites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This homework assignment uses the dataset  from the \"Malicious and Benign Websites\" Kaggle project at https://www.kaggle.com/xwolf12/malicious-and-benign-websites/home (you'll need to go to that URL to get a full description of the data).\n",
    "\n",
    "From the overview:\n",
    "\n",
    "> The project consisted to evaluate different classification models to predict malicious and benign websites, based on application layer and network characteristics. The data were obtained by using different verified sources of benign and malicious URL's, in a low interactive client honeypot to isolate network traffic. We used additional tools to get other information, such as, server country with Whois."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, you will perform three tasks, each worth 10 points (specific point allocations are provided for each task).  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's load the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('data/malicious_and_benign.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>URL</th>\n",
       "      <th>URL_LENGTH</th>\n",
       "      <th>NUMBER_SPECIAL_CHARACTERS</th>\n",
       "      <th>CHARSET</th>\n",
       "      <th>SERVER</th>\n",
       "      <th>CONTENT_LENGTH</th>\n",
       "      <th>WHOIS_COUNTRY</th>\n",
       "      <th>WHOIS_STATEPRO</th>\n",
       "      <th>WHOIS_REGDATE</th>\n",
       "      <th>WHOIS_UPDATED_DATE</th>\n",
       "      <th>...</th>\n",
       "      <th>DIST_REMOTE_TCP_PORT</th>\n",
       "      <th>REMOTE_IPS</th>\n",
       "      <th>APP_BYTES</th>\n",
       "      <th>SOURCE_APP_PACKETS</th>\n",
       "      <th>REMOTE_APP_PACKETS</th>\n",
       "      <th>SOURCE_APP_BYTES</th>\n",
       "      <th>REMOTE_APP_BYTES</th>\n",
       "      <th>APP_PACKETS</th>\n",
       "      <th>DNS_QUERY_TIMES</th>\n",
       "      <th>Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M0_109</td>\n",
       "      <td>16</td>\n",
       "      <td>7</td>\n",
       "      <td>iso-8859-1</td>\n",
       "      <td>nginx</td>\n",
       "      <td>263.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>10/10/2015 18:21</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>700</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>1153</td>\n",
       "      <td>832</td>\n",
       "      <td>9</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B0_2314</td>\n",
       "      <td>16</td>\n",
       "      <td>6</td>\n",
       "      <td>UTF-8</td>\n",
       "      <td>Apache/2.4.10</td>\n",
       "      <td>15087.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>1230</td>\n",
       "      <td>17</td>\n",
       "      <td>19</td>\n",
       "      <td>1265</td>\n",
       "      <td>1230</td>\n",
       "      <td>17</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B0_911</td>\n",
       "      <td>16</td>\n",
       "      <td>6</td>\n",
       "      <td>us-ascii</td>\n",
       "      <td>Microsoft-HTTPAPI/2.0</td>\n",
       "      <td>324.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       URL  URL_LENGTH  NUMBER_SPECIAL_CHARACTERS     CHARSET  \\\n",
       "0   M0_109          16                          7  iso-8859-1   \n",
       "1  B0_2314          16                          6       UTF-8   \n",
       "2   B0_911          16                          6    us-ascii   \n",
       "\n",
       "                  SERVER  CONTENT_LENGTH WHOIS_COUNTRY WHOIS_STATEPRO  \\\n",
       "0                  nginx           263.0          None           None   \n",
       "1          Apache/2.4.10         15087.0          None           None   \n",
       "2  Microsoft-HTTPAPI/2.0           324.0          None           None   \n",
       "\n",
       "      WHOIS_REGDATE WHOIS_UPDATED_DATE  ...   DIST_REMOTE_TCP_PORT  \\\n",
       "0  10/10/2015 18:21               None  ...                      0   \n",
       "1              None               None  ...                      7   \n",
       "2              None               None  ...                      0   \n",
       "\n",
       "   REMOTE_IPS  APP_BYTES  SOURCE_APP_PACKETS  REMOTE_APP_PACKETS  \\\n",
       "0           2        700                   9                  10   \n",
       "1           4       1230                  17                  19   \n",
       "2           0          0                   0                   0   \n",
       "\n",
       "   SOURCE_APP_BYTES  REMOTE_APP_BYTES  APP_PACKETS  DNS_QUERY_TIMES  Type  \n",
       "0              1153               832            9              2.0     1  \n",
       "1              1265              1230           17              0.0     0  \n",
       "2                 0                 0            0              0.0     0  \n",
       "\n",
       "[3 rows x 21 columns]"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color = 'red'> Q1: k-means clustering </font>\n",
    "#### Perform a K-means clustering of the websites in the dataset (7 points).\n",
    "#### Examine the silhouette scores for different values of K, and report the optimal value of K (3 points).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color='red'>Q2: Random forest classification </font>\n",
    "*NOTE*: Use the ```malicious_and_benign_train.csv``` and ```malicious_and_benign_test.csv``` for your training and testing datasets, respectively.\n",
    "#### Create a random forest classifier (3 points).\n",
    "#### Report the accuracy of your resulting classifier (2 points).\n",
    "#### Report the most important feature (2 points).\n",
    "#### Try tuning the hyperparameters to see if you can improve the accuracy (3 points).\n",
    "##### *BONUS*: The top 5 scores in the class will receive a 5-point bonus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color='red'>Q3: Dimension reduction</font>\n",
    "#### Use principal components analysis to reduce the original dataset to fewer dimensions (remember to scale the data) (6 points).\n",
    "#### Describe the first dimension in terms of the original variables (2 points).\n",
    "#### Report the optimal number of reduced dimensions based on a scree plot (2 points)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}