notebooks/flair_log_parsing/FLAIR_DNS_Log_Parsing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DNS Log Parsing with FLAIR\n",
    "\n",
    "## Authors\n",
    " - Gorkem Batmaz (NVIDIA) [gbatmaz@nvidia.com]\n",
    "\n",
    "## Development Notes\n",
    "* Developed using: RAPIDS v0.10.0 \n",
    "* Last tested using: RAPIDS v0.10.0 on Nov 4, 2019\n",
    "\n",
    "## Table of Contents\n",
    "* Introduction\n",
    "* Log Parsing on a Clean Dataset from a Single Source\n",
    "* Log Parsing on a Corrupted Dataset from a Single Source\n",
    "* Conclusion\n",
    "\n",
    "## Introduction\n",
    "Log parsing is a complex and highly manual process. In addition to cyBERT, we present an alternate technique that utilizes a combination of character and word embeddings using [FLAIR](https://github.com/zalandoresearch/flair) combined with [RAPIDS](https://rapids.ai). The long term goal of this work is to parse unstructured and nonstandard log types using a probabilistic method so it can also perform on previously unseen log types. Phase 1 demonstrates this works on a single dataset. Tags in the raw logs might vary depending on the sensor, thus we only use the values to predict their tags. For Phase 1, we assume that we know where a log ends and the key/value pair values. This allows ut to reframe the log parsing problem as an entity recognition challenge. Phase 2 proves this can also work with corrupt and messy data in the same dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Phase 1: Using a Clean dataset from A Single Source"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing of the DNS Logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cudf, io, requests\n",
    "from io import StringIO\n",
    "import nvstrings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "data1 = cudf.read_csv('query_output1545120200000_1545163200000.tab', sep='\\t',nrows=500000, quoting=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['time',\n",
       " 'uuid',\n",
       " 'hostname',\n",
       " 'flow_id',\n",
       " 'bytes',\n",
       " 'bytes_in',\n",
       " 'bytes_out',\n",
       " 'dest_ip',\n",
       " 'dest_mac',\n",
       " 'dest_port',\n",
       " 'endtime',\n",
       " 'message_type',\n",
       " 'name',\n",
       " 'protocol_stack',\n",
       " 'query',\n",
       " 'query_type',\n",
       " 'reply_code',\n",
       " 'reply_code_id',\n",
       " 'response_time',\n",
       " 'src_ip',\n",
       " 'src_mac',\n",
       " 'src_port',\n",
       " 'ttl',\n",
       " 'time_taken',\n",
       " 'transaction_id',\n",
       " 'transport',\n",
       " 'insert_date',\n",
       " 'id']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(data1) # Listing the column names of the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "data1['hostname'] = data1['hostname'].replace(' ', '')#Remove spaces so that each field can be treated as a single word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check for and Eliminate `null` Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id\n"
     ]
    }
   ],
   "source": [
    "for i in list(data1):\n",
    "    if len(data1)==data1[i].isna().sum():\n",
    "        print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `id` column happens to be the last column, we will change it to a dot to indicate the end of each log so it works seamlessly with the NLP framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "data1.drop_column('id')\n",
    "data1['id']='.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "data1[\"\"]=\"\" # add an empty column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change the Data Type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changing the data type to `string` so that it can be manipulated later. Then the columns are pivoted in to single column so that the output can be directly used for training in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in list(data1):\n",
    "    data1[i]=data1[i].astype(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pivoting the columns\n",
    "data1=data1.reset_index().melt(id_vars=['index']).sort_values('index').drop(columns=['index'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>variable</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>time</td>\n",
       "      <td>1545142695971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>500000</th>\n",
       "      <td>uuid</td>\n",
       "      <td>8aca9eb6-b16e-463a-b188-3bf029a669fc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000000</th>\n",
       "      <td>hostname</td>\n",
       "      <td>elb-agent.agent.datadoghq.com;e.gtld-servers.n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1500000</th>\n",
       "      <td>flow_id</td>\n",
       "      <td>958d6f44-902d-4dcc-aad7-9373679832a9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000000</th>\n",
       "      <td>bytes</td>\n",
       "      <td>556</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         variable                                              value\n",
       "0            time                                      1545142695971\n",
       "500000       uuid               8aca9eb6-b16e-463a-b188-3bf029a669fc\n",
       "1000000  hostname  elb-agent.agent.datadoghq.com;e.gtld-servers.n...\n",
       "1500000   flow_id               958d6f44-902d-4dcc-aad7-9373679832a9\n",
       "2000000     bytes                                                556"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create the columns that the FLAIR framework will use for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "data1['ner']=data1['variable']\n",
    "data1['pos']=data1['ner']\n",
    "data1.drop_column('variable') #Create two tag columns for POS and NER options\n",
    "\n",
    "for i in list(data1):\n",
    "    data1[i]=data1[i].astype(str)#change type so categories become strings too\n",
    "    \n",
    "dftrain=data1[0:1000000]#remember to increase the size back.\n",
    "dftest=data1[1000000:1100000]\n",
    "dfdev=data1[1100000:1200000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>value</th>\n",
       "      <th>ner</th>\n",
       "      <th>pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1545142695971</td>\n",
       "      <td>time</td>\n",
       "      <td>time</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>500000</th>\n",
       "      <td>8aca9eb6-b16e-463a-b188-3bf029a669fc</td>\n",
       "      <td>uuid</td>\n",
       "      <td>uuid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000000</th>\n",
       "      <td>elb-agent.agent.datadoghq.com;e.gtld-servers.n...</td>\n",
       "      <td>hostname</td>\n",
       "      <td>hostname</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1500000</th>\n",
       "      <td>958d6f44-902d-4dcc-aad7-9373679832a9</td>\n",
       "      <td>flow_id</td>\n",
       "      <td>flow_id</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000000</th>\n",
       "      <td>556</td>\n",
       "      <td>bytes</td>\n",
       "      <td>bytes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     value       ner       pos\n",
       "0                                            1545142695971      time      time\n",
       "500000                8aca9eb6-b16e-463a-b188-3bf029a669fc      uuid      uuid\n",
       "1000000  elb-agent.agent.datadoghq.com;e.gtld-servers.n...  hostname  hostname\n",
       "1500000               958d6f44-902d-4dcc-aad7-9373679832a9   flow_id   flow_id\n",
       "2000000                                                556     bytes     bytes"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Training, Validation, and Test Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FLAIR Framework already has an ingest method for NER problems. To make it readable for the next step, the dataframe will be written into a CSV and then the quotes that are added by the `to_csv` function are removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "dftrain.to_csv(\"rapids_train.txt\",sep='\\t',header=False,index=False)\n",
    "dftest.to_csv(\"rapids_test.txt\",sep='\\t',header=False,index=False)\n",
    "dfdev.to_csv(\"rapids_dev.txt\",sep='\\t',header=False,index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('rapids_dev.txt', 'r') as f, open('rapids_dev_.txt', 'w') as fo:\n",
    "    for line in f:\n",
    "        fo.write(line.replace('\"', ''))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('rapids_test.txt', 'r') as f, open('rapids_test_.txt', 'w') as fo:\n",
    "    for line in f:\n",
    "        fo.write(line.replace('\"', ''))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('rapids_train.txt', 'r') as f, open('rapids_train_.txt', 'w') as fo:\n",
    "    for line in f:\n",
    "        fo.write(line.replace('\"', ''))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training of the Model and Inference Against the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-11-04 10:36:09,995 Reading data from .\n",
      "2019-11-04 10:36:09,995 Train: rapids_train_.txt\n",
      "2019-11-04 10:36:09,996 Dev: rapids_dev_.txt\n",
      "2019-11-04 10:36:09,996 Test: rapids_test_.txt\n",
      "[b'<unk>', b'O', b'time', b'uuid', b'hostname', b'flow_id', b'bytes', b'bytes_in', b'bytes_out', b'dest_ip', b'dest_mac', b'dest_port', b'endtime', b'message_type', b'name', b'protocol_stack', b'query', b'query_type', b'reply_code', b'reply_code_id', b'response_time', b'src_ip', b'src_mac', b'src_port', b'ttl', b'time_taken', b'transaction_id', b'transport', b'insert_date', b'id', b'mini._sftp-ssh._tcp.local', b'[14:10:9f:dd:22:9d]._workstation._tcp.local;kshook-mlt.local;kshook-mlt.local;kshook-mlt._ssh._tcp.local;kshook-mlt._sftp-ssh._tcp.local;kshook-mlt._companion-link._tcp.local;kshook-mlt', b'<START>', b'<STOP>']\n",
      "lentgh of train 34483\n",
      "2019-11-04 10:36:36,437 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:36:36,438 Model: \"SequenceTagger(\n",
      "  (embeddings): StackedEmbeddings(\n",
      "    (list_embedding_0): FlairEmbeddings(\n",
      "      (lm): LanguageModel(\n",
      "        (drop): Dropout(p=0.5, inplace=False)\n",
      "        (encoder): Embedding(300, 100)\n",
      "        (rnn): LSTM(100, 2048)\n",
      "        (decoder): Linear(in_features=2048, out_features=300, bias=True)\n",
      "      )\n",
      "    )\n",
      "    (list_embedding_1): FlairEmbeddings(\n",
      "      (lm): LanguageModel(\n",
      "        (drop): Dropout(p=0.5, inplace=False)\n",
      "        (encoder): Embedding(300, 100)\n",
      "        (rnn): LSTM(100, 2048)\n",
      "        (decoder): Linear(in_features=2048, out_features=300, bias=True)\n",
      "      )\n",
      "    )\n",
      "  )\n",
      "  (word_dropout): WordDropout(p=0.05)\n",
      "  (locked_dropout): LockedDropout(p=0.5)\n",
      "  (embedding2nn): Linear(in_features=4096, out_features=4096, bias=True)\n",
      "  (rnn): LSTM(4096, 256, bidirectional=True)\n",
      "  (linear): Linear(in_features=512, out_features=34, bias=True)\n",
      ")\"\n",
      "2019-11-04 10:36:36,438 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:36:36,438 Corpus: \"Corpus: 34483 train + 3449 dev + 3450 test sentences\"\n",
      "2019-11-04 10:36:36,439 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:36:36,440 Parameters:\n",
      "2019-11-04 10:36:36,440  - learning_rate: \"0.3\"\n",
      "2019-11-04 10:36:36,440  - mini_batch_size: \"128\"\n",
      "2019-11-04 10:36:36,441  - patience: \"3\"\n",
      "2019-11-04 10:36:36,441  - anneal_factor: \"0.5\"\n",
      "2019-11-04 10:36:36,442  - max_epochs: \"4\"\n",
      "2019-11-04 10:36:36,442  - shuffle: \"True\"\n",
      "2019-11-04 10:36:36,442  - train_with_dev: \"False\"\n",
      "2019-11-04 10:36:36,443 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:36:36,443 Model training base path: \"resources/taggers/example-ner\"\n",
      "2019-11-04 10:36:36,444 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:36:36,444 Device: cuda:0\n",
      "2019-11-04 10:36:36,444 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:36:36,445 Embeddings storage mode: cpu\n",
      "2019-11-04 10:36:36,448 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:36:39,192 epoch 1 - iter 0/270 - loss 119.57215118 - samples/sec: 1263.94\n",
      "2019-11-04 10:37:39,502 epoch 1 - iter 27/270 - loss 63.58949791 - samples/sec: 57.46\n",
      "2019-11-04 10:38:36,132 epoch 1 - iter 54/270 - loss 39.41765953 - samples/sec: 61.19\n",
      "2019-11-04 10:39:38,016 epoch 1 - iter 81/270 - loss 28.00740742 - samples/sec: 55.99\n",
      "2019-11-04 10:40:39,998 epoch 1 - iter 108/270 - loss 21.49619452 - samples/sec: 55.91\n",
      "2019-11-04 10:41:41,844 epoch 1 - iter 135/270 - loss 17.36262155 - samples/sec: 56.03\n",
      "2019-11-04 10:42:44,123 epoch 1 - iter 162/270 - loss 14.55895582 - samples/sec: 55.63\n",
      "2019-11-04 10:43:52,094 epoch 1 - iter 189/270 - loss 12.50056805 - samples/sec: 50.96\n",
      "2019-11-04 10:44:51,840 epoch 1 - iter 216/270 - loss 10.95644337 - samples/sec: 58.00\n",
      "2019-11-04 10:45:57,453 epoch 1 - iter 243/270 - loss 9.75081264 - samples/sec: 52.80\n",
      "2019-11-04 10:46:54,723 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:46:54,725 EPOCH 1 done: loss 8.8158 - lr 0.3000\n",
      "2019-11-04 10:48:15,724 DEV : loss 0.0025359306018799543 - score 1.0\n",
      "2019-11-04 10:48:16,012 BAD EPOCHS (no improvement): 0\n",
      "2019-11-04 10:48:16,640 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:48:16,966 epoch 2 - iter 0/270 - loss 0.01773477 - samples/sec: 10800.94\n",
      "2019-11-04 10:48:25,708 epoch 2 - iter 27/270 - loss 0.02330188 - samples/sec: 402.95\n",
      "2019-11-04 10:48:34,408 epoch 2 - iter 54/270 - loss 0.02084385 - samples/sec: 404.58\n",
      "2019-11-04 10:48:43,165 epoch 2 - iter 81/270 - loss 0.02144139 - samples/sec: 401.83\n",
      "2019-11-04 10:48:51,880 epoch 2 - iter 108/270 - loss 0.02069665 - samples/sec: 403.77\n",
      "2019-11-04 10:49:00,531 epoch 2 - iter 135/270 - loss 0.01907291 - samples/sec: 406.99\n",
      "2019-11-04 10:49:09,233 epoch 2 - iter 162/270 - loss 0.01866662 - samples/sec: 404.35\n",
      "2019-11-04 10:49:17,831 epoch 2 - iter 189/270 - loss 0.01936224 - samples/sec: 409.41\n",
      "2019-11-04 10:49:26,527 epoch 2 - iter 216/270 - loss 0.02045359 - samples/sec: 404.64\n",
      "2019-11-04 10:49:35,309 epoch 2 - iter 243/270 - loss 0.01940588 - samples/sec: 400.63\n",
      "2019-11-04 10:49:43,666 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:49:43,667 EPOCH 2 done: loss 0.0217 - lr 0.3000\n",
      "2019-11-04 10:50:14,634 DEV : loss 0.0009210893185809255 - score 1.0\n",
      "2019-11-04 10:50:14,897 BAD EPOCHS (no improvement): 1\n",
      "2019-11-04 10:50:15,547 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:50:15,873 epoch 3 - iter 0/270 - loss 0.02637887 - samples/sec: 10752.09\n",
      "2019-11-04 10:50:24,661 epoch 3 - iter 27/270 - loss 0.01520973 - samples/sec: 400.16\n",
      "2019-11-04 10:50:33,476 epoch 3 - iter 54/270 - loss 0.01216725 - samples/sec: 399.17\n",
      "2019-11-04 10:50:42,214 epoch 3 - iter 81/270 - loss 0.01338935 - samples/sec: 403.55\n",
      "2019-11-04 10:50:50,980 epoch 3 - iter 108/270 - loss 0.01219468 - samples/sec: 402.09\n",
      "2019-11-04 10:50:59,739 epoch 3 - iter 135/270 - loss 0.01244372 - samples/sec: 402.02\n",
      "2019-11-04 10:51:08,439 epoch 3 - iter 162/270 - loss 0.01330024 - samples/sec: 404.71\n",
      "2019-11-04 10:51:17,143 epoch 3 - iter 189/270 - loss 0.01390909 - samples/sec: 404.31\n",
      "2019-11-04 10:51:26,114 epoch 3 - iter 216/270 - loss 0.01379610 - samples/sec: 392.04\n",
      "2019-11-04 10:51:34,861 epoch 3 - iter 243/270 - loss 0.01303860 - samples/sec: 402.39\n",
      "2019-11-04 10:51:43,231 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:51:43,232 EPOCH 3 done: loss 0.0122 - lr 0.3000\n",
      "2019-11-04 10:52:13,968 DEV : loss 0.00016586198762524873 - score 1.0\n",
      "2019-11-04 10:52:14,236 BAD EPOCHS (no improvement): 2\n",
      "2019-11-04 10:52:14,868 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:52:15,199 epoch 4 - iter 0/270 - loss 0.00816083 - samples/sec: 10599.16\n",
      "2019-11-04 10:52:25,896 epoch 4 - iter 27/270 - loss 0.00536717 - samples/sec: 398.04\n",
      "2019-11-04 10:52:34,668 epoch 4 - iter 54/270 - loss 0.00502122 - samples/sec: 401.39\n",
      "2019-11-04 10:52:43,470 epoch 4 - iter 81/270 - loss 0.00464402 - samples/sec: 400.05\n",
      "2019-11-04 10:52:52,205 epoch 4 - iter 108/270 - loss 0.00666049 - samples/sec: 403.14\n",
      "2019-11-04 10:53:00,915 epoch 4 - iter 135/270 - loss 0.00735177 - samples/sec: 404.32\n",
      "2019-11-04 10:53:09,640 epoch 4 - iter 162/270 - loss 0.00842327 - samples/sec: 403.43\n",
      "2019-11-04 10:53:18,403 epoch 4 - iter 189/270 - loss 0.00789958 - samples/sec: 401.94\n",
      "2019-11-04 10:53:27,119 epoch 4 - iter 216/270 - loss 0.00758327 - samples/sec: 403.70\n",
      "2019-11-04 10:53:35,954 epoch 4 - iter 243/270 - loss 0.00842111 - samples/sec: 398.22\n",
      "2019-11-04 10:53:44,257 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:53:44,258 EPOCH 4 done: loss 0.0079 - lr 0.3000\n",
      "2019-11-04 10:54:15,213 DEV : loss 8.405170956393704e-05 - score 1.0\n",
      "2019-11-04 10:54:15,495 BAD EPOCHS (no improvement): 3\n",
      "2019-11-04 10:54:16,717 ----------------------------------------------------------------------------------------------------\n",
      "2019-11-04 10:54:16,718 Testing using best model ...\n",
      "2019-11-04 10:54:16,719 loading file resources/taggers/example-ner/best-model.pt\n",
      "2019-11-04 10:55:36,525 1.0\t1.0\t1.0\n",
      "2019-11-04 10:55:36,526 \n",
      "MICRO_AVG: acc 1.0 - f1-score 1.0\n",
      "MACRO_AVG: acc 1.0 - f1-score 0.9999857142857144\n",
      "bytes      tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "bytes_in   tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "bytes_out  tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "dest_ip    tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "dest_mac   tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "dest_port  tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "endtime    tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "flow_id    tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "hostname   tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "id         tp: 3449 - fp: 0 - fn: 0 - tn: 3449 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "insert_date tp: 3449 - fp: 0 - fn: 0 - tn: 3449 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "message_type tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "name       tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "protocol_stack tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "query      tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "query_type tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "reply_code tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "reply_code_id tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "response_time tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "src_ip     tp: 3448 - fp: 1 - fn: 0 - tn: 3448 - precision: 0.9997 - recall: 1.0000 - accuracy: 0.9997 - f1-score: 0.9998\n",
      "src_mac    tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "src_port   tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "time       tp: 3448 - fp: 0 - fn: 1 - tn: 3448 - precision: 1.0000 - recall: 0.9997 - accuracy: 0.9997 - f1-score: 0.9998\n",
      "time_taken tp: 3449 - fp: 0 - fn: 0 - tn: 3449 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "transaction_id tp: 3449 - fp: 0 - fn: 0 - tn: 3449 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "transport  tp: 3449 - fp: 0 - fn: 0 - tn: 3449 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "ttl        tp: 3449 - fp: 0 - fn: 0 - tn: 3449 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "uuid       tp: 3448 - fp: 0 - fn: 0 - tn: 3448 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "2019-11-04 10:55:36,527 ----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'test_score': 1.0,\n",
       " 'dev_score_history': [1.0, 1.0, 1.0, 1.0],\n",
       " 'train_loss_history': [8.815754337360461,\n",
       "  0.02166381318949991,\n",
       "  0.012212355707392649,\n",
       "  0.007855682107792408],\n",
       " 'dev_loss_history': [tensor(0.0025, device='cuda:0'),\n",
       "  tensor(0.0009, device='cuda:0'),\n",
       "  tensor(0.0002, device='cuda:0'),\n",
       "  tensor(8.4052e-05, device='cuda:0')]}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Original of this part is at https://github.com/zalandoresearch/flair/blob/master/resources/docs/TUTORIAL_7_TRAINING_A_MODEL.md#training-a-sequence-labeling-model\n",
    "import torch\n",
    "import os\n",
    "from flair.data import Corpus\n",
    "from flair.datasets import ColumnCorpus\n",
    "from flair.embeddings import TokenEmbeddings, WordEmbeddings, StackedEmbeddings, CharacterEmbeddings, FlairEmbeddings\n",
    "from typing import List\n",
    "\n",
    "# define columns\n",
    "\n",
    "columns = {0: 'text', 1: 'pos', 2: 'ner'}\n",
    "\n",
    "# this is the folder in which train, test and dev files reside\n",
    "data_folder = '.'\n",
    "\n",
    "# 1. init a corpus using column format, data folder and the names of the train, dev and test files\n",
    "corpus: Corpus = ColumnCorpus(data_folder, columns,\n",
    "                              train_file='rapids_train_.txt',\n",
    "                              test_file='rapids_test_.txt',\n",
    "                              dev_file='rapids_dev_.txt')\n",
    "# 2. what tag do we want to predict?\n",
    "tag_type = 'ner'\n",
    "\n",
    "# 3. make the tag dictionary from the corpus\n",
    "tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)\n",
    "print(tag_dictionary.idx2item)\n",
    "\n",
    "# 4. initialize embeddings\n",
    "embedding_types: List[TokenEmbeddings] = [\n",
    "\n",
    "    FlairEmbeddings('multi-forward'),\n",
    "    FlairEmbeddings('multi-backward'),\n",
    "]\n",
    "\n",
    "embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=embedding_types)\n",
    "\n",
    "# 5. initialize sequence tagger\n",
    "from flair.models import SequenceTagger\n",
    "\n",
    "tagger: SequenceTagger = SequenceTagger(hidden_size=256,\n",
    "                                        embeddings=embeddings,\n",
    "                                        tag_dictionary=tag_dictionary,\n",
    "                                        tag_type=tag_type,\n",
    "                                        use_crf=True)\n",
    "\n",
    "# 6. initialize trainer\n",
    "from flair.trainers import ModelTrainer\n",
    "\n",
    "\n",
    "trainer: ModelTrainer = ModelTrainer(tagger, corpus)\n",
    "print(\"lentgh of train\",len(corpus.train))\n",
    "#print(corpus.dev[0])\n",
    "# 7. start training and run test after each epoch\n",
    "trainer.train('resources/taggers/example-ner',\n",
    "              learning_rate=0.3,\n",
    "              mini_batch_size=128,\n",
    "              max_epochs=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Phase 2: Data Perturbation to Test Against Corrupted and Missing Values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this phase, we investigate if performance deteroiates when values in the fields get corrupted. We iterate the preprocessing corruption process to increase the level of corruption in the test datset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare the New Test Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "import random\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#load the data into a dataframe\n",
    "df1 = pd.read_csv('query_output1545120200000_1545163200000.tab', sep='\\t',low_memory=False,nrows=100000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Function for Character Insertion Corruption\n",
    "\n",
    "Here we create a function to insert a character into a string to break the format and the originality of each field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def insert_str(string, str_to_insert, totalrandom, index):\n",
    "    return string[:index] + (totalrandom)*(str_to_insert) + string[index:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We decide how many random characters we need to insert into each field and how much of the data should be excluded from the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "missing_proportion = 1\n",
    "totalrandom = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1[\"id\"].fillna( '.', inplace = True)\n",
    "df1.rename(columns={'id':'.'}, inplace=True)\n",
    "df2=df1.dropna()\n",
    "df2=df2.astype(str)\n",
    "for i in list(df2):\n",
    "    df2[i] = df2[i].str.replace(' ', '') #remove spaces to be able to treat each field as one word. \n",
    "df2[\"\"]=\"\" # add an empty column\n",
    "df2=df2.stack()\n",
    "df2 = df2.to_frame().reset_index() #change the dataframe from multilevel to single level\n",
    "df2=df2.drop(['level_0'], axis=1)\n",
    "df2.columns=['pos','text']\n",
    "df2 = df2[['text','pos']]\n",
    "df2['ner']=df2['pos'] #Create two tag columns for POS and NER options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each log ends with a `.` and not to add random characters in to the empty lines and to the `.` fields we run the commands below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(7)\n",
    "\n",
    "n_drop = missing_proportion * round((len(df2)) / 100)\n",
    "\n",
    "drop_indices = np.random.choice(df2.index, n_drop, replace=False)\n",
    "\n",
    "dataindex=range(1,28)\n",
    "counter = 0\n",
    "for i in range(1000007:1050032):\n",
    "\n",
    "    counter += 1\n",
    "    if counter in dataindex:\n",
    "        \n",
    "        df2['text'][i] = insert_str((df2['text'][i]), str(random.choice(string.ascii_letters)), totalrandom,\n",
    "                                random.randint(0, len(df2['text'][i])))\n",
    "    elif counter == 29:\n",
    "        counter = 0\n",
    "df2 = df2.drop(drop_indices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "dftest=df2[1000007:1050032]\n",
    "dftest.to_csv(\"perturbed_test_data.txt\",index=False,sep='\\t',header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The content of the test dataset has been changed and standard formats (e.g., IP addresses) are broken. Categorical columns have added spurious categories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training of the Model and Inference Against the Model Using Corrupted Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-10-11 20:41:52,335 Reading data from .\n",
      "2019-10-11 20:41:52,336 Train: train.txt\n",
      "2019-10-11 20:41:52,346 Dev: val.txt\n",
      "2019-10-11 20:41:52,347 Test: perturbed_test_data.txt\n",
      "[b'<unk>', b'O', b'time', b'uuid', b'hostname', b'flow_id', b'bytes', b'bytes_in', b'bytes_out', b'dest_ip', b'dest_mac', b'dest_port', b'endtime', b'message_type', b'name', b'protocol_stack', b'query', b'query_type', b'reply_code', b'reply_code_id', b'response_time', b'src_ip', b'src_mac', b'src_port', b'ttl', b'time_taken', b'transaction_id', b'transport', b'insert_date', b'.', b'<START>', b'<STOP>']\n",
      "lentgh of train 34483\n",
      "Sentence: \"544 34 510 172.16.136.26 AC:16:2D:88:D8:38 53 2018-12-18T12:48:25.298790Z QUERY;RESPONSE dns.msftncsi.com;com;com;com;com;com;com;com;com;com;com;com;com;com;a.gtld-servers.net;b.gtld-servers.net;c.gtld-servers.net;d.gtld-servers.net;e.gtld-servers.net;f.gtld-servers.net;g.gtld-servers.net;h.gtld-servers.net;i.gtld-servers.net;j.gtld-servers.net;k.gtld-servers.net;l.gtld-servers.net;m.gtld-servers.net;a.gtld-servers.net ip:udp:dns dns.msftncsi.com A NoError 0 99 172.17.169.202 00:DE:FB:5A:4A:C1 37086 9;115731;115731;115731;115731;115731;115731;115731;115731;115731;115731;115731;115731;115731;103858;103858;103858;103858;103858;119705;103858;67311;103858;103858;103858;148764;103858;110657 99 2 udp 2018-12-18 .\" - 24 Tokens\n",
      "2019-10-11 20:42:14,284 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 20:42:14,285 Model: \"SequenceTagger(\n",
      "  (embeddings): StackedEmbeddings(\n",
      "    (list_embedding_0): FlairEmbeddings(\n",
      "      (lm): LanguageModel(\n",
      "        (drop): Dropout(p=0.5, inplace=False)\n",
      "        (encoder): Embedding(300, 100)\n",
      "        (rnn): LSTM(100, 2048)\n",
      "        (decoder): Linear(in_features=2048, out_features=300, bias=True)\n",
      "      )\n",
      "    )\n",
      "    (list_embedding_1): FlairEmbeddings(\n",
      "      (lm): LanguageModel(\n",
      "        (drop): Dropout(p=0.5, inplace=False)\n",
      "        (encoder): Embedding(300, 100)\n",
      "        (rnn): LSTM(100, 2048)\n",
      "        (decoder): Linear(in_features=2048, out_features=300, bias=True)\n",
      "      )\n",
      "    )\n",
      "  )\n",
      "  (word_dropout): WordDropout(p=0.05)\n",
      "  (locked_dropout): LockedDropout(p=0.5)\n",
      "  (embedding2nn): Linear(in_features=4096, out_features=4096, bias=True)\n",
      "  (rnn): LSTM(4096, 256, bidirectional=True)\n",
      "  (linear): Linear(in_features=512, out_features=32, bias=True)\n",
      ")\"\n",
      "2019-10-11 20:42:14,285 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 20:42:14,286 Corpus: \"Corpus: 34483 train + 17242 dev + 1730 test sentences\"\n",
      "2019-10-11 20:42:14,286 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 20:42:14,287 Parameters:\n",
      "2019-10-11 20:42:14,287  - learning_rate: \"0.3\"\n",
      "2019-10-11 20:42:14,288  - mini_batch_size: \"128\"\n",
      "2019-10-11 20:42:14,288  - patience: \"3\"\n",
      "2019-10-11 20:42:14,289  - anneal_factor: \"0.5\"\n",
      "2019-10-11 20:42:14,289  - max_epochs: \"3\"\n",
      "2019-10-11 20:42:14,290  - shuffle: \"True\"\n",
      "2019-10-11 20:42:14,290  - train_with_dev: \"False\"\n",
      "2019-10-11 20:42:14,290 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 20:42:14,291 Model training base path: \"resources/taggers/example-ner\"\n",
      "2019-10-11 20:42:14,292 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 20:42:14,292 Device: cuda:0\n",
      "2019-10-11 20:42:14,292 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 20:42:14,293 Embeddings storage mode: cpu\n",
      "2019-10-11 20:42:14,297 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 20:42:15,947 epoch 1 - iter 0/270 - loss 102.64485168 - samples/sec: 2099.05\n",
      "2019-10-11 20:43:16,836 epoch 1 - iter 27/270 - loss 48.46231859 - samples/sec: 56.94\n",
      "2019-10-11 20:44:21,882 epoch 1 - iter 54/270 - loss 28.41668063 - samples/sec: 53.28\n",
      "2019-10-11 20:45:41,436 epoch 1 - iter 81/270 - loss 19.88595992 - samples/sec: 43.54\n",
      "2019-10-11 20:46:41,407 epoch 1 - iter 108/270 - loss 15.19021626 - samples/sec: 57.80\n",
      "2019-10-11 20:47:39,378 epoch 1 - iter 135/270 - loss 12.21623010 - samples/sec: 59.80\n",
      "2019-10-11 20:48:42,698 epoch 1 - iter 162/270 - loss 10.21142120 - samples/sec: 54.73\n",
      "2019-10-11 20:49:46,336 epoch 1 - iter 189/270 - loss 8.77801154 - samples/sec: 54.46\n",
      "2019-10-11 20:50:50,238 epoch 1 - iter 216/270 - loss 7.69013954 - samples/sec: 54.24\n",
      "2019-10-11 20:51:54,002 epoch 1 - iter 243/270 - loss 6.84324121 - samples/sec: 54.35\n",
      "2019-10-11 20:52:53,851 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 20:52:53,852 EPOCH 1 done: loss 6.1863 - lr 0.3000\n",
      "2019-10-11 20:59:46,332 DEV : loss 0.001958210952579975 - score 1.0\n",
      "2019-10-11 20:59:47,876 BAD EPOCHS (no improvement): 0\n",
      "2019-10-11 20:59:48,490 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 20:59:48,787 epoch 2 - iter 0/270 - loss 0.01560211 - samples/sec: 11817.32\n",
      "2019-10-11 20:59:56,786 epoch 2 - iter 27/270 - loss 0.01746165 - samples/sec: 441.36\n",
      "2019-10-11 21:00:04,692 epoch 2 - iter 54/270 - loss 0.01412502 - samples/sec: 447.41\n",
      "2019-10-11 21:00:12,459 epoch 2 - iter 81/270 - loss 0.01613659 - samples/sec: 455.21\n",
      "2019-10-11 21:00:20,165 epoch 2 - iter 108/270 - loss 0.01605078 - samples/sec: 459.20\n",
      "2019-10-11 21:00:28,000 epoch 2 - iter 135/270 - loss 0.01503408 - samples/sec: 451.49\n",
      "2019-10-11 21:00:35,700 epoch 2 - iter 162/270 - loss 0.01448011 - samples/sec: 459.22\n",
      "2019-10-11 21:00:43,619 epoch 2 - iter 189/270 - loss 0.01434995 - samples/sec: 446.44\n",
      "2019-10-11 21:00:51,452 epoch 2 - iter 216/270 - loss 0.01382386 - samples/sec: 451.11\n",
      "2019-10-11 21:00:59,283 epoch 2 - iter 243/270 - loss 0.01310806 - samples/sec: 451.00\n",
      "2019-10-11 21:01:06,716 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 21:01:06,717 EPOCH 2 done: loss 0.0129 - lr 0.3000\n",
      "2019-10-11 21:03:36,953 DEV : loss 0.0007582574617117643 - score 1.0\n",
      "2019-10-11 21:03:38,444 BAD EPOCHS (no improvement): 1\n",
      "2019-10-11 21:03:39,084 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 21:03:39,388 epoch 3 - iter 0/270 - loss 0.00281262 - samples/sec: 11539.99\n",
      "2019-10-11 21:03:47,413 epoch 3 - iter 27/270 - loss 0.00842747 - samples/sec: 440.13\n",
      "2019-10-11 21:03:55,173 epoch 3 - iter 54/270 - loss 0.01388396 - samples/sec: 455.84\n",
      "2019-10-11 21:04:02,924 epoch 3 - iter 81/270 - loss 0.01239082 - samples/sec: 456.32\n",
      "2019-10-11 21:04:10,704 epoch 3 - iter 108/270 - loss 0.01099977 - samples/sec: 454.45\n",
      "2019-10-11 21:04:18,408 epoch 3 - iter 135/270 - loss 0.01033568 - samples/sec: 459.38\n",
      "2019-10-11 21:04:26,138 epoch 3 - iter 162/270 - loss 0.00943592 - samples/sec: 456.72\n",
      "2019-10-11 21:04:33,942 epoch 3 - iter 189/270 - loss 0.00901383 - samples/sec: 453.29\n",
      "2019-10-11 21:04:41,736 epoch 3 - iter 216/270 - loss 0.00864582 - samples/sec: 453.73\n",
      "2019-10-11 21:04:49,516 epoch 3 - iter 243/270 - loss 0.00832354 - samples/sec: 454.07\n",
      "2019-10-11 21:04:56,816 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 21:04:56,817 EPOCH 3 done: loss 0.0082 - lr 0.3000\n",
      "2019-10-11 21:07:22,736 DEV : loss 0.0004057696496602148 - score 1.0\n",
      "2019-10-11 21:07:24,230 BAD EPOCHS (no improvement): 2\n",
      "2019-10-11 21:07:25,404 ----------------------------------------------------------------------------------------------------\n",
      "2019-10-11 21:07:25,405 Testing using best model ...\n",
      "2019-10-11 21:07:25,406 loading file resources/taggers/example-ner/best-model.pt\n",
      "2019-10-11 21:08:04,542 0.9986\t0.9986\t0.9986\n",
      "2019-10-11 21:08:04,543 \n",
      "MICRO_AVG: acc 0.9971 - f1-score 0.9986\n",
      "MACRO_AVG: acc 0.9971 - f1-score 0.9985392857142857\n",
      ".          tp: 1728 - fp: 0 - fn: 0 - tn: 1728 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "bytes      tp: 1728 - fp: 15 - fn: 0 - tn: 1728 - precision: 0.9914 - recall: 1.0000 - accuracy: 0.9914 - f1-score: 0.9957\n",
      "bytes_in   tp: 1710 - fp: 0 - fn: 16 - tn: 1710 - precision: 1.0000 - recall: 0.9907 - accuracy: 0.9907 - f1-score: 0.9953\n",
      "bytes_out  tp: 1716 - fp: 1 - fn: 0 - tn: 1716 - precision: 0.9994 - recall: 1.0000 - accuracy: 0.9994 - f1-score: 0.9997\n",
      "dest_ip    tp: 1725 - fp: 0 - fn: 0 - tn: 1725 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "dest_mac   tp: 1721 - fp: 0 - fn: 0 - tn: 1721 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "dest_port  tp: 1718 - fp: 0 - fn: 0 - tn: 1718 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "endtime    tp: 1726 - fp: 0 - fn: 0 - tn: 1726 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "flow_id    tp: 1726 - fp: 0 - fn: 1 - tn: 1726 - precision: 1.0000 - recall: 0.9994 - accuracy: 0.9994 - f1-score: 0.9997\n",
      "hostname   tp: 1726 - fp: 0 - fn: 0 - tn: 1726 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "insert_date tp: 1732 - fp: 0 - fn: 0 - tn: 1732 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "message_type tp: 1734 - fp: 0 - fn: 0 - tn: 1734 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "name       tp: 1735 - fp: 0 - fn: 0 - tn: 1735 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "protocol_stack tp: 1727 - fp: 0 - fn: 0 - tn: 1727 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "query      tp: 1725 - fp: 0 - fn: 0 - tn: 1725 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "query_type tp: 1729 - fp: 0 - fn: 4 - tn: 1729 - precision: 1.0000 - recall: 0.9977 - accuracy: 0.9977 - f1-score: 0.9988\n",
      "reply_code tp: 1724 - fp: 4 - fn: 0 - tn: 1724 - precision: 0.9977 - recall: 1.0000 - accuracy: 0.9977 - f1-score: 0.9988\n",
      "reply_code_id tp: 1722 - fp: 21 - fn: 0 - tn: 1722 - precision: 0.9880 - recall: 1.0000 - accuracy: 0.9880 - f1-score: 0.9940\n",
      "response_time tp: 1690 - fp: 2 - fn: 21 - tn: 1690 - precision: 0.9988 - recall: 0.9877 - accuracy: 0.9866 - f1-score: 0.9932\n",
      "src_ip     tp: 1719 - fp: 1 - fn: 2 - tn: 1719 - precision: 0.9994 - recall: 0.9988 - accuracy: 0.9983 - f1-score: 0.9991\n",
      "src_mac    tp: 1724 - fp: 0 - fn: 0 - tn: 1724 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "src_port   tp: 1720 - fp: 0 - fn: 2 - tn: 1720 - precision: 1.0000 - recall: 0.9988 - accuracy: 0.9988 - f1-score: 0.9994\n",
      "time       tp: 1723 - fp: 0 - fn: 0 - tn: 1723 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "time_taken tp: 1708 - fp: 13 - fn: 11 - tn: 1708 - precision: 0.9924 - recall: 0.9936 - accuracy: 0.9861 - f1-score: 0.9930\n",
      "transaction_id tp: 1726 - fp: 11 - fn: 0 - tn: 1726 - precision: 0.9937 - recall: 1.0000 - accuracy: 0.9937 - f1-score: 0.9968\n",
      "transport  tp: 1724 - fp: 0 - fn: 0 - tn: 1724 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - f1-score: 1.0000\n",
      "ttl        tp: 1711 - fp: 1 - fn: 13 - tn: 1711 - precision: 0.9994 - recall: 0.9925 - accuracy: 0.9919 - f1-score: 0.9959\n",
      "uuid       tp: 1729 - fp: 1 - fn: 0 - tn: 1729 - precision: 0.9994 - recall: 1.0000 - accuracy: 0.9994 - f1-score: 0.9997\n",
      "2019-10-11 21:08:04,543 ----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'test_score': 0.9986,\n",
       " 'dev_score_history': [1.0, 1.0, 1.0],\n",
       " 'train_loss_history': [6.186323038999129,\n",
       "  0.012852874613815436,\n",
       "  0.008184892177599034],\n",
       " 'dev_loss_history': [tensor(0.0020, device='cuda:0'),\n",
       "  tensor(0.0008, device='cuda:0'),\n",
       "  tensor(0.0004, device='cuda:0')]}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Original of this part is at https://github.com/zalandoresearch/flair/blob/master/resources/docs/TUTORIAL_7_TRAINING_A_MODEL.md#training-a-sequence-labeling-model\n",
    "import torch\n",
    "import os\n",
    "from flair.data import Corpus\n",
    "from flair.datasets import ColumnCorpus\n",
    "from flair.embeddings import TokenEmbeddings, WordEmbeddings, StackedEmbeddings, CharacterEmbeddings, FlairEmbeddings\n",
    "from typing import List\n",
    "\n",
    "# define columns\n",
    "\n",
    "columns = {0: 'text', 1: 'pos', 2: 'ner'}\n",
    "\n",
    "# this is the folder in which train, test and dev files reside\n",
    "data_folder = '.'\n",
    "\n",
    "# init a corpus using column format, data folder and the names of the train, dev and test files\n",
    "corpus: Corpus = ColumnCorpus(data_folder, columns,\n",
    "                              train_file='_train.txt',\n",
    "                              test_file='perturbed_test_data.txt',\n",
    "                              dev_file='_val.txt')\n",
    "# 2. what tag do we want to predict?\n",
    "tag_type = 'ner'\n",
    "\n",
    "# 3. make the tag dictionary from the corpus\n",
    "tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)\n",
    "print(tag_dictionary.idx2item)\n",
    "\n",
    "# 4. initialize embeddings\n",
    "embedding_types: List[TokenEmbeddings] = [\n",
    "\n",
    "    FlairEmbeddings('multi-forward'),\n",
    "    FlairEmbeddings('multi-backward'),\n",
    "]\n",
    "\n",
    "embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=embedding_types)\n",
    "\n",
    "# 5. initialize sequence tagger\n",
    "from flair.models import SequenceTagger\n",
    "\n",
    "tagger: SequenceTagger = SequenceTagger(hidden_size=256,\n",
    "                                        embeddings=embeddings,\n",
    "                                        tag_dictionary=tag_dictionary,\n",
    "                                        tag_type=tag_type,\n",
    "                                        use_crf=True)\n",
    "\n",
    "# 6. initialize trainer\n",
    "from flair.trainers import ModelTrainer\n",
    "\n",
    "\n",
    "trainer: ModelTrainer = ModelTrainer(tagger, corpus)\n",
    "print(\"lentgh of train\",len(corpus.train))\n",
    "print(corpus.dev[0])\n",
    "# 7. start training and run test after each epoch\n",
    "trainer.train('resources/taggers/example-ner',\n",
    "              learning_rate=0.3,\n",
    "              mini_batch_size=128,\n",
    "              max_epochs=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After 3 epochs, the F1 score of the corrupted test set is 0.9986. This test dataset has one randomcharacter inserted in each field and 1% of the fields in the dataset are missing values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first part of the notebook proves that this set of DNS logs can be parsed probabilistically with high accuracy. In pertubing the data, we show that the impact of corrupted data on model performance is minimal. In typical real-life DNS logs, less than 1% of the logs might be corrupted. The experiment shown here goes well beyond 1% corruption to show the effects at extreme levels."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}