notebooks/MLPipelinePreparation2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Disaster messages - Part 2 - Deep learning\n",
    "\n",
    "In an earlier __[project](https://github.com/pranath/disaster_response/blob/master/notebooks/MLPipelinePreparation.ipynb)__ I explored several conventional machine learning models in order to predict one of several categories for messages from disasters\n",
    "\n",
    "In this project I will try to predict categories for the same messages but using one of the most advanced deep learning models for text classification available using the fastai library.\n",
    "\n",
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n",
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import precision_score, recall_score, accuracy_score, f1_score\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tags</th>\n",
       "      <th>message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>related</td>\n",
       "      <td>Weather update - a cold front from Cuba that c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>related;aid_related;other_aid;weather_related;...</td>\n",
       "      <td>Is the Hurricane over or is it not over</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>related</td>\n",
       "      <td>Looking for someone but no name</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>related;request;aid_related;medical_products;o...</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>related</td>\n",
       "      <td>says: west side of Haiti, rest of the country ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                tags  \\\n",
       "0                                            related   \n",
       "1  related;aid_related;other_aid;weather_related;...   \n",
       "2                                            related   \n",
       "3  related;request;aid_related;medical_products;o...   \n",
       "4                                            related   \n",
       "\n",
       "                                             message  \n",
       "0  Weather update - a cold front from Cuba that c...  \n",
       "1            Is the Hurricane over or is it not over  \n",
       "2                    Looking for someone but no name  \n",
       "3  UN reports Leogane 80-90 destroyed. Only Hospi...  \n",
       "4  says: west side of Haiti, rest of the country ...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load data\n",
    "path = Path('data')\n",
    "df = pd.read_csv(path/'messages_disaster_clean.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26215, 2)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 26215 entries, 0 to 26214\n",
      "Data columns (total 2 columns):\n",
      "tags       20093 non-null object\n",
      "message    26215 non-null object\n",
      "dtypes: object(2)\n",
      "memory usage: 409.7+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create language model\n",
    "\n",
    "For our deep learning classifier we are first going to create a language model. Our language model 'AWD_LSTM' will be a Recurrent neural network (RNN), that is implemented as a long short-term memory network (LSTM).\n",
    "\n",
    "The AWD_LSTM architecture is inspired by this __[paper](https://arxiv.org/abs/1708.02182)__.\n",
    "\n",
    "This model has been trained on several million wikipedia articles, and is able to predict the next word in a sequence given some starting words i.e. it is a pre-trained model.\n",
    "\n",
    "We will then use the first half of this pre-trained AWD_LSTM model, fine tune it on our dataset of disaster messages, and then chop of the first part of it to use it as the feature detector/first half of our classification model.\n",
    "\n",
    "We would hope then this gives our model a good understanding of text in general first, before being optimised for our own dataset in order to give it the best chance to classify our disaster messages correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set batch size\n",
    "bs = 48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create data object for our language model using whole dataset \n",
    "data_lm = (TextList.from_df(df, path, cols='message') \n",
    "                .random_split_by_pct(0.1) # Use most of dataset to train this langauge model\n",
    "                .label_for_lm() # LM creates its own labels from the text\n",
    "                .databunch(bs=bs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save for quick load when returning\n",
    "data_lm.save('data_lm.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load language model data\n",
    "data_lm = load_data(path, 'data_lm.pkl', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>haiti , rest of the country today and tonight xxbos xxmaj storm at xxunk heart of jesus xxbos xxmaj please , we need tents and water . xxmaj we are in xxmaj silo , xxmaj thank you ! xxbos i would like to receive the messages , thank you xxbos i am in xxmaj croix - des - xxmaj bouquets . xxmaj we have health issues . xxmaj they (</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>, i found a large building on lamentin 54 that could be an hospital . .. please confirm . xxbos xxmaj asking for xxmaj water , xxmaj food and xxmaj tents xxbos xxmaj we need medicine for xxmaj hypertension and xxunk / pads and medicine for xxunk baby too . xxbos xxmaj need tents , water and food . xxmaj children are sick with fever . xxbos i am without</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>that . .. xxbos .. . zone xxmaj croix de xxmaj bouquets , i 'm asking please what i can do to get help especially a place to sleep with my children because my place . ... xxbos xxmaj hi , we are on xxmaj rte 9 , right after the xxmaj minustah xxmaj base , we have no water and food . xxmaj the way they distributed the supplies</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>to xxmaj digicel xxmaj organisation of young victims sou ray ( name of the main road closest to the sea in xxmaj carrefour . xxmaj we are a bunch of people who are victims of the catastrophe of xxmaj tuesday xxmaj january 12 xxbos xxmaj message important asking whether xxmaj haiti is under occupation xxbos i am from xxmaj port - au - xxmaj prince in the west . xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>all in my area . xxbos xxmaj the hunger is killing us do something for us in xxmaj l'acul , xxmaj leogane xxbos you will never get hurt or xxunk in distribution of food in cite soleil xxbos xxmaj my family is in xxmaj delmas 14 and needs some aid . xxmaj they need food , water , and shelter . xxmaj my cousin has an infant who needs baby</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Have a quick look at how our messages have been automagically pre-processed / tokenised\n",
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create pre-trained language model AWD_LSTM with our data\n",
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhcZd3/8fc3+9rsbdM2oRvdS2mbslOoKFIUERUEBBGXiqI+yOOOP/VxV1ARUBB4QIUHRFEUAVkEKktLS0MXSvd9TZM0+77dvz9mUkKapGk7Z85M5vO6rrly5pwzcz4nbeY759zn3Lc55xARkdgV53cAERHxlwqBiEiMUyEQEYlxKgQiIjFOhUBEJMYl+B3gaOXn57uxY8f6HUNEJKqUlpZWOucK+loWdYVg7NixrFixwu8YIiJRxcx29rfM01NDZrbDzN40s1Vm1u+nt5nNM7NOM/uIl3lERORw4TgiWOCcq+xvoZnFAz8DnglDFhER6SUSGou/CPwVKPc7iIhILPK6EDjgWTMrNbNFvRea2WjgEuCugd7EzBaZ2QozW1FRUeFRVBGR2OR1ITjTOTcHWAhcb2bzey2/Ffi6c65zoDdxzt3tnCtxzpUUFPTZ6C0iIsfI0zYC59y+4M9yM3sMOAV4qccqJcCfzAwgH7jQzDqcc3/3MpeIiLzNs0JgZulAnHOuPjh9PvD9nus458b1WP/3wBMqAiIi4eXlqaERwCtmthpYDjzpnHvazK4zs+s83K6IyJBz67838fJmb9pIPTsicM5tA2b1Mb/PhmHn3Ce8yiIiEs26uhy3Pb+Z6xdM5OwTQ99OGgmXj4qIyADqWtrpcpCdluTJ+6sQiIhEuOqmdgBy0hI9eX8VAhGRCFfV2AZATrqOCEREYlJNU7AQ6NSQiEhs6j41lKtCICISm6qDp4ay09VGICISk6qb2kiIMzKTvbniX4VARCTCVTe1kZ2WRLA7npBTIRARiXDVje2eXToKKgQiIhGvuqnNs0tHQYVARCTiVTe16YhARCSWVTe1k6sjAhGR2OSco7qxzbN+hkCFQEQkojW0dtDR5XRqSEQkVtUc6nBORwQiIjHpUIdzKgQiIrGpusnbnkdBhUBEJKIdKgRqIxARiU3VjWojEBGJaTVNbcQZDEvVEYGISEyqamojKzWR+DhvOpwDFQIRkYhW3dTuaUMxqBCIiES06sY2T9sHQIVARCSiVTd52wU1qBCIiES0miYdEYiIxLSqRm/HIgAVAhGRiNXc1klrR5eOCEREYlVVGO4qBhUCEZGIVR3scM7LsQhAhUBEJGJ1d0Ht5ehkoEIgIhKxdGpIRCTG1YShC2pQIRARiVjdg9Jke9jhHHhcCMxsh5m9aWarzGxFH8svNrM13cvN7Cwv84iIRJOapnYyUxJIiPf2O3uCp+8esMA5V9nPsueBx51zzsxOAv4MTAlDJhGRiFfd1OZ5QzGEpxD0yznX0ONpOuD8yiIiEmmqGts8v3QUvG8jcMCzZlZqZov6WsHMLjGzDcCTwCf7WWdR8NTRioqKCg/jiohEjpqmdnI9vmIIvC8EZzrn5gALgevNbH7vFZxzjznnpgAfBH7Q15s45+52zpU450oKCgq8TSwiEiGqwtAFNXhcCJxz+4I/y4HHgFMGWPclYIKZ5XuZSUQkWtQ0ed/hHHhYCMws3cwyu6eB84G1vdaZaGYWnJ4DJAEHvcokIhItWjs6aWzr9PxmMvC2sXgE8Fjwcz4BeMg597SZXQfgnLsL+DDwcTNrB5qBjzrn1GAsIjGvu3uJcDQWe1YInHPbgFl9zL+rx/TPgJ95lUFEJFpVB+8qDsflo7qzWEQkAh26q3gIXDUkIiLHIFw9j4IKgYhIRKo+1POoCoGISEyq1qkhEZHYVt3UTnpSPMkJ8Z5vS4VARCQCVYepnyFQIRARiUjh6nkUVAhERCJSdVN7WNoHQIVARCQiVdS3kp+RHJZtqRCIiESYri7HgboWCrNSwrI9FQIRkQhT2dhKR5dTIRARiVVltS0AjBimQiAiEpO6C0FhVmpYtqdCICISYcrqAoVgpE4NiYjEpv21LSTGG3m6j0BEJDaV1bYwYlgKcXEWlu2pEIiIRJj9tc1hu2IIVAhERCJOWW0LI8PUUAwqBCIiEcU5x/7a8N1MBioEIiIRpaapndaOLkaG6R4CUCEQEYko+2vDe+koqBCIiESUA2G+hwBUCEREIsr+Q3cVqxCIiMSkstpm4gwKwtQFNagQiIhElP21LQzPTCEhPnwfzyoEIiIRpKyuJaztA6BCICISUcJ9DwGoEIiIRJTAXcUqBCIiMam+pZ2G1g4dEYiIxKqyQzeTha+fIVAhEBGJGIfuKg5j9xKgQiAiEjG6RyYL96mhBC/f3Mx2APVAJ9DhnCvptfxjwNeDTxuAzznnVnuZSUQkUnWfGho+LHw3k4HHhSBogXOusp9l24FznHPVZrYQuBs4NQyZREQizv7aFvIzkkhOiA/rdsNRCPrlnFvS4+lrwBi/soiI+K2stjnsl46C920EDnjWzErNbNER1v0U8C+P84iIRKz9tS2MHBbeK4bA+yOCM51z+8xsOPCcmW1wzr3UeyUzW0CgEJzV15sEi8gigOLiYi/zioj4pqyuhXljc8O+XU+PCJxz+4I/y4HHgFN6r2NmJwH3Ahc75w728z53O+dKnHMlBQUFXkYWEfFFc1snNU3tQ+vUkJmlm1lm9zRwPrC21zrFwN+Aq51zm7zKIiIS6fy6dBS8PTU0AnjMzLq385Bz7mkzuw7AOXcX8B0gD/htcL3DLjEVEYkF+2ubgfDfTAYeFgLn3DZgVh/z7+ox/Wng015lEBGJFn4MUdlNdxZHuC3l9Vxw60t87/G3ONjQ6nccEfGIH4PWd1MhiGA7DzZy5T3L2FfTzAOv7eScmxdz+/ObaWrr8DuaiIRYWW0LWamJpCWF//YuX28ok/7tq2nmynuW0d7ZxaOfO4M4M37+9AZ+8dwm/rB0B/NPLGB2cTazi3OYMjKz32Hturoce2ua2VBWz/r9dWwsq+fcyQVcWlIU3h0SkQHtq2n2paEYVAj61NHZxTX3L2dHZRPvnT6ShTNHMrc4h6b2ThZvLOfptWW8sqWSCQUZLJwxkgtmjGRMTlrItl9R38pV9y6jrrmdhz5zGpNGZAJw98dLKN1ZxT0vbeelzZX8beVeABLijLSkeNKTE0hNiifejIbWDhpaOmho68C5t997WEoCz64rY/qoLKaNGhayzCJyfHZXNVOUG7rPkaNhruenRBQoKSlxK1as8HQbtzyzkTte3MIp43JZtbuGto4u8jOSqGvpODQ9/8QCNpTVs25/HQCzxmTx9YVTOGNC/lFvr7PLselAPSt31bByVzUvb66ktrmdBz51CiX93FzinGNPdTNv7KpmY1k9TW2dNLZ20NTWSWeXIyMlgcyUBDKTExg+LIWphcOYMjKTto4uzr/1JfLSk/jHF84Me58mInI45xzTv/sMl88r5jsXTfNkG2ZW2t9VmToi6GXJlkp+s3gLl84dw82XzqKhtYMXNpTz/PoD5KUnc8GMkcw9IYf4OANgR2UjT79VxkPLdnHlPcu4+rQT+MbCKaQnH/lXu7+2mYeX7eLh13dTUR9oCM5NT+Lkomw+d+6EfosAgJlRlJt21N8g0pPh5x85iWvvf51fPruJb1449aheLyKhd7Cxjaa2Topyw9+9BMRQIXhjVzU3P72ReWNzmDcul9nFOWT0+rA+2NDKDY+sYlx+Ov9z8XQAMpIT+MCsUXxg1qg+33dsfjrXnTOBa04fy83PbOT+JdtZvKmcH18yk9PH5x127r66sY1l2w/y95X7eG79AbqcY8Hk4Vw0q5C5xbkU5aYSvKfCMwsmD+fKU4u5++VtvGvKcE4dn+fp9kRkYLurmgAoCuEp5qMxqEJgZhOAPc65VjM7FzgJ+KNzrsbLcKHU2NpBfWs7d7y4ha4XID7OmDFqGPMnFXDOpAJmFWXz1UfXUNPUzv3XzjvqlvvUpHi+c9E0Lpgxkq89upqr/3c5yQlxTBqRydTCTNKSEli2vYoNZXU4BzlpiXzm7PF87NRiX84L3nThVF7dUsl//2U1f/vcGRRkJg+qAHV2uUNHQyISGrurAzeTFedFcBuBma0CSoCxwDPA48Bk59yFnqbrw/G2ETS0drByVzWvb69iydaDrNxdQ2eXIzUxnub2Tr530TQ+cea448rY3NbJv9buZ92+OtaX1bF+fz0NrR3MLc7h9Al5nD4hj1ljsklK8Pfq3dKdVVx611K6HKQlxTMmJ5UxOWmckJfG+Px0xuVnkJeRxJo9NSzbXsXy7VUcqGvhtPF5nD9tBO+eNoLCHmOrdv9f8vqIRmSo+c2LW7j5mY2s+/57Pbt8dKA2gsEWgjecc3PM7KtAi3PudjNb6ZybHeqwRxLqxuLa5naWbKlk8cYKhqUm8K0Lp4b8g8w5R5cjIr9Jr9pdQ+nOavZUN7G3upnd1c3sqGykub3zHevlpSdxyrhcRmal8J+NFWyrbAQC/aK0dXTR3N5Jc3snM0ZlcevlJzOhIMOP3RGJSt/46xr+vb6cFd9+t2fbCEVjcbuZXQFcA1wUnJcYinB+y0pNZOHMQhbOLPRsG2ZGfOTVAABOLsrm5KLsd8xzzlFe38q2ikbK61uYPmoYEwoy3i6QF8GW8gaeXVfG1vJGUpPiSE2MJyE+jkde381Ft7/Cjy6ZwSWzNc6QyGDsqmryraEYBl8IrgWuA37knNtuZuOAB72LJX4yM0YMS2HEAJ1fTRyewcThEw+bf83pY/nSn1by5UdWs3TrQb570fRBXUElEst2VzcxuyjHt+0P6i/UObcO+BKAmeUAmc65n3oZTKLTyKwUHvr0qfz6+c3c8eIW/rFqH6eOz+PcSQXMn1RAYryxv7aFstoWqpvamD+pQKeRJKZ1dHaxr6aFi2f501AMg79qaDHwgeD6q4AKM/uPc+5GD7NJlEqIj+O/z5/Mu6YM5/HV+/jPpgq+/8S6ftefP6mAa88YyzmTCoiLwHYUES/tr22hs8tFxamhLOdcnZl9GrjfOfddM1vjZTCJfrOLc5hdHDjc3XWwiSVbK0mIj2PksBRGZqWQnBDHYyv38uBrO7n2968zJieVeWNzmVqYybTCLMYVpJPQozBkpSaSkqg7oWVoOXQPgU/dS8DgC0GCmRUClwE3eZhHhqjivDSK8w4fb/pL553IdedM4Om3yvjHyr0s3XqQx4J9KPWWmZLA58+dyLVnjlVBkCFjl883k8HgC8H3Cdw/8Kpz7nUzGw9s9i6WxJKkhLh33L1d1djG+v117KpqOtRhnsPxwvpyfvb0Bh5YuoOvXjCZi2eN1qkkiXq7q5uIjzPfeh4FdTonUWbJ1kp+/NR61u6tY0xOKudOLuCcScM5Y0Kerk6SqPTFh1eyencNL31tgafbOe77CMxsDHA7cCbggFeA/3LO7QlZSpFBOGNCPo9ffxZPvLmfx1ft429v7OXB13aRGG/MHJ3FrOB9EbOLcsLSb5PI8drt8z0EMPhTQ/cDDwGXBp9fFZz3Hi9CiQwkLs4OnUpq6+hixc4q/rOpgjd2VvPw8l3c/+oOIHCvQ/d6Y/PT/Q0t0o/dVU2cP32ErxkGWwgKnHP393j+ezO7wYtAIkcjKSGOMybkHxoHor2zi00H6indWc2Ta/bzy+c28cvnNnHSmCxOGx/o42lWURajs3W0IP5rbO3gYGNbSAe2OhaDLQSVZnYV8HDw+RXAQW8iiRy7xPg4po/KYvqoLD5++lj21TTzxJp9/GttGb9/dQdtnV1AoI+kH35wBudN9febmMS2PcFeR/28dBQGXwg+CdwB/IpAG8ESAt1OiES0UdmpLJo/gUXzJ9DW0cXGsnpW7anh4WW7+NQfVvDps8bxtQum+N4TrMSm7ktHi6OhEDjndhG4s/iQ4KmhW70IJeKFpIQ4Zo7JYuaYLC6dO4afPLWee1/Zzus7qvj5R2aRkZJAc1snLe2d1LW0U9XYxsGGNg42tjFpRAYXzijU5aoSUm8PSBMdjcV9uREVAolSKYnx/M/FMzh9Qh5fe3QN7731pSO+ZsborXxr4VTOmHj041KL9GVXVRPpSfHkpif5muN4CoG+GknUu2BGITPHZPPC+gMkJcSRkhhPamI8GSkJ5KUnk5eRRFZqIk+s2cctz2ziynuXce7kAi6cUUh2WiI56UnkpCWSl55MVmqijhjkqOypbqIoN833CxeOpxBE151oIv0YnZ3K1aePHXCdS2aPYeGMQv64dAd3vLCFxRsrDlsnIc7Iy0giPyOZjOQEMpITSE9OIDstkYnDMzhxeCaTR2b6/u1PIseuqiaKc/2/tHnAQmBm9fT9gW+Avye1RMIsJTGeRfMncM0ZYymva6WmqZ3qpjaqmwJtCZUNrVTUt1LV2EZ9awdldS00tHYEnrd0HHqf/IxkJo/MYPKIYUwemUFRThqpSfGkJsWTlpjAqOwUEuLVeD3UOefYXdXMWRML/I4ycCFwzmWGK4hItEhOiKcoN42i3MGt3z3i28ayejYdqD/08+Hluw4bEhRgfEE6t10+mxmjs0KcXCLJwcY2mts7fb+rGI7v1JCIDELPEd/mT3r7219Xl2NXVRP7a1to6eikua2TqsY2bn9hMx/67RK+vnAKnzxzrO/nj8UbkXLpKKgQiPgmLs4Ym59+WPcXF84s5GuPruEHT6zj5c0VfP7ciYwvSCcvPUlFYQiJhHEIuqkQiESY3PQk7vn4XB58bSc/eHL9oYbpzJQExhdkcNr4XM6bMoI5xdlqS4hiuyNgHIJuKgQiEcjMuPr0sbx3xkje2lfH9opGdhxsZGNZPfe9sp3f/Wcb2WmJnDUxn7F56QwflszwzGSKc9OZWpipI4cosKW8gcKsFFKT/B9kydNCYGY7gHqgE+jo3Re2mU0h0IvpHOAm59wtXuYRiTbDM1MYPjmFBZPfnlff0s7Lmyt5fn05S7dW8q+1ZXR2vX1x36yibBadPZ73Th+hI4YItqGsnikjI+N6nHAcESxwzlX2s6wK+BLwwTDkEBkSMlMSuXBmIRfOLASgs8tR1dhGeX0LpTurue+V7Vz/0BuMyUnlilOKmTk6iykjMynITNaRQoRo7+xia0UD504e7ncUwOdTQ865cqDczN7nZw6RaBYfZxRkJlOQmcz0UVl87NQTeG7dAe55eRs3P7Px0Ho5aYmUjM3lQ7NH866pw0lO8P+URKzaVtFIe6eLmSMCBzxrZg74nXPubo+3JxLz4uOMC2aM5IIZI6lqbGNDWR0by+rZsL+exZvKeW7dAbJSE7loViGfOXs8J+T5f2drrNlQVgfAlMLYKARnOuf2mdlw4Dkz2+CcO3LvXr2Y2SJgEUBxcXGoM4oMWbnpSe8YuKezy/Hqlkr++sYeHi3dw4sbKnjqv84mKzXR56SxZUNZPQlxxvj8DL+jAOBpS5Jzbl/wZznwGHDKMb7P3c65EudcSUGB/7dji0Sr+Dhj/qQCfn35bB7+zGkcqGvhW397E+fUdVg4bSyrZ0JBRsSMg+FZCjNLN7PM7mngfGCtV9sTkaMzuziHG8+fxJNv7ueR13f7HSembCyrj5jTQuDtEcEI4BUzWw0sB550zj1tZteZ2XUAZjbSzPYQGNvg22a2x8yGeZhJRHq4bv4EzpqYz/f++RabD9T7HScm1Da3s7emmckR0lAMHhYC59w259ys4GO6c+5Hwfl3OefuCk6XOefGOOeGOeeyg9N1XmUSkXeKizN+edks0pMS+OLDK2npoxM8Ca1NwYIbKVcMgcdtBCIS+YYPS+GWy2axoaxe7QVhsKGsuxBEzskPFQIRYcHk4Xz53ZP428q9/PK5TX7HGdI2ltWRmZJAYVaK31EOUV9DIgLAl86byL6aZm5/YQuFWalceaou1fbChv2BriUi6S5vHRGICBDo6O6Hl8zg3MkFfPvvb/L8+gN+RxpynHNsPFAfUaeFQIVARHpIjI/jN1fOYfqoLL7w0ErW7q31O9KQsq+2hfqWjoi6YghUCESkl/TkBO77xDxy0hL57AOlVDe2+R1pyNiwP9i1hAqBiES6gsxk7rxqLhX1rfzXI6ve0c21HLvuK4YmqRCISDSYVZTN9z4wnZc2VfDr5zf7HWdI2FhWz+jsVIalRFbfTioEItKvK04p4iNzx3Db85t5YYMaj4/XhrK6iDstBCoEIjIAM+OHH5zBtMJh3PCnVeyobPQ7UtRq6+hiW0VjRPUx1E2FQEQGlJIYz++unkt8nPHJP7xObVO735Gi0taKBjq6HJMj7NJRUCEQkUEoyk3jrqvmsruqic8/VEp7Z5ffkaLO+gi9YghUCERkkE4dn8ePL5nJq1sO8t3H31KfREdp5a4aMpITmFAQGYPR9KQuJkRk0C4tKWJbZSN3Lt7KxIIMPnnWOL8jRY3SndXMLs4mPi5yupbopiMCETkqXz1/Mu+dPoIfPrnu0OkOGVhDawcbyuqYU5zjd5Q+qRCIyFGJizN+9uGTGJaayA+fXKdTRIOwencNXQ7mnqBCICJDRHZaEjecdyKvbjnI8+vL/Y4T8VbsqMYMTi7O9jtKn1QIROSYfOy0E5hQkM6Pn1pPW4euIhpI6a5qJo/IjLg7irupEIjIMUmMj+Pb75vGtspGHnhtp99xIlZXl2PlzmrmROhpIVAhEJHjcO7kAs4+MZ9f/3uTeintx+byBupbOyhRIRCRocjM+Pb7ptHQ2sGt/9YQl31ZsbMKiNyGYlAhEJHjNHlkJh879QQeeG0ny7Yd9DtOxCndWU1+RhLFuWl+R+mXCoGIHLdvLJxCcW4aX35klfoi6uWNndXMKc6JqDGKe1MhEJHjlp6cwK8vn015fSvf+vuburcgqLKhlR0HmyL6tBCoEIhIiMwqyubG8yfx5Jr9/KV0j99xIsIbO6uByG4fABUCEQmhz86fwGnjc/ne42+xXWMXULqzmqT4OGaMzvI7yoBUCEQkZOLjjF999GQS4+P4yl9Wx/wpotKd1cwYPYyUxHi/owxIhUBEQqowK5VvLpxC6c5qnnqzzO84vmnt6GTN3tqIPy0EKgQi4oFLS4qYPCKTnz29gdaOTr/j+GLt3lraOrpUCEQkNsXHGd9631R2VTXxwNLY7H5iyZaDmMEp4/L8jnJEKgQi4olzJgW6n7j9hS3UNMVe9xOvbKlk+qhh5KYn+R3liFQIRMQzN71vKvUt7dzxwha/o4RVU1sHb+yq5syJ+X5HGRQVAhHxzJSRw7h0bhF/WLqDXQeb/I4TNsu3V9He6ThLhQDMbIeZvWlmq8xsRR/LzcxuM7MtZrbGzOZ4mUdEwu/G8yeREBfHdx5fGzOXk766pZKk+DhKTsj1O8qghOOIYIFz7mTnXEkfyxYCJwYfi4A7w5BHRMJoxLAUvn7BZBZvrOD3S3b4HScsXtlykLkn5JCaFNn3D3Tz+9TQxcAfXcBrQLaZFfqcSURC7JozxnLelOH85KkNQ37A+8qGVtbvr+OsE6PjtBB4Xwgc8KyZlZrZoj6WjwZ293i+JzjvHcxskZmtMLMVFRUVHkUVEa+YGT//yElkpyXyxYdX0tw2dO8tWLo10BV3tDQUg/eF4Ezn3BwCp4CuN7P5vZb31S/rYScRnXN3O+dKnHMlBQUFXuQUEY/lZSTzy8tOZmtFAz94cp3fcTzz6pZKMlMSmBnh/Qv15GkhcM7tC/4sBx4DTum1yh6gqMfzMcA+LzOJiH/OOjGfRfPH89CyXTy9dr/fcULOOcfLmys5Y0Ie8XGRO/5Ab54VAjNLN7PM7mngfGBtr9UeBz4evHroNKDWOTf0/neIyCH//Z7JnDQmi689uoY91UPrktJdVU3srWmOmstGu3l5RDACeMXMVgPLgSedc0+b2XVmdl1wnaeAbcAW4B7g8x7mEZEIkJQQx+1XzKbLwZceXkl7Z5ffkULmlS2VQHS1DwAkePXGzrltwKw+5t/VY9oB13uVQUQi0wl56fzkQzP54sMr+dVzm/jaBVP8jhQSr26pZFRWCuPy0/2OclT8vnxURGLURbNGcfm8Iu78z1Ze3hz9VwN2dTmWbD3ImRPzI3p84r6oEIiIb7570XQmFmTw5UdWU9nQ6nec47Jk60Fqmto5Z3L0XdmoQiAivklNiueOK+dQ09TG7c9v9jvOcXlkxW6yUhN599QRfkc5aioEIuKrySMz+fCcMTy8fDf7a5v9jnNMapraeOatMj548qiIH5ayLyoEIuK7L7xrIl3O8dsXt/od5Zj8feVe2jq6uGxe0ZFXjkAqBCLiu6LcNC4tKeKR13ezrya6jgqcczyyYg8zRg9j+qjouZu4JxUCEYkIX3jXRByO37wYXYPYrN1bx/r9dXy0JDqPBkCFQEQixOjsVC4rKeLPK3ZH1R3Hj6zYRXJCHB84+bD+MqOGCoGIRIzrF0zEMH4TJW0FLe2d/GPVPhbOGElWaqLfcY6ZCoGIRIxR2al8dF4Rf1mxmw1lkT9uwdNry6hv6YjaRuJuKgQiElG+dN6J5KYn8dkHSqltavc7zoAeeX03xblpnDYuz+8ox0WFQEQiSkFmMndeNYd9Nc3c8MhKuroic5zj9fvrWLrtIB+dV0RcFHU53RcVAhGJOHNPyOU7F03nxY0V3PrvTX7H6dO9L28nLSmej51a7HeU46ZCICIR6apTi7l07hhue2ELz75V5necdyirbeHx1Xu5rKSI7LQkv+McNxUCEYlIZsYPPjiDk8ZkceOfV7OjstHvSIfcv2Q7nV2OT501zu8oIaFCICIRKyUxnjuvmkt8nHH9Q2/Q0u7/oPf1Le089NouFs4spCg3ze84IaFCICIRbXR2Kr+4dBZv7avjR0+u9zsOj7y+m/rWDhadPd7vKCGjQiAiEe/d00awaP54HnhtJ0+s2edbjvbOLu57ZTunjstlVlG2bzlCTYVARKLCV987mTnF2Xzjr2/61l7w1Jv72VfbwqL5Q+doAFQIRCRKJMbHcfuVc0iIN657sJSaprawbr+stoVbnt3IhIJ0FkweHtZte02FQESixujsVO64Yg7bKhq5+n+XU9scnjuPy+tbuPLe16hubOeWSwDlWAYAAAtvSURBVGdF/Q1kvakQiEhUOevEfH539Vw2lNXx8fuWU9fibTE42NDKVfcuo6y2hfuvncfs4hxPt+cHFQIRiToLpgzntx+by1t7a/nEfctpaO3wZDs1TW1c9b/L2XmwiXuvKWHe2FxPtuM3FQIRiUrvmTaC26+Yzeo9tVx+91K2VjSE9P0r6lu54p5lbC1v4J6Pl3DGhPyQvn8kUSEQkai1cGYhd189lz3Vzbz/tlf4v2U7ce74O6nbW9PMZb9byo7KRu69poT5kwpCkDZyqRCISFQ7b+oInrlhPiVjc7jpsbV85o+lVNS3HvP7ba1o4NI7l1DZ0MqDnz5lyBcBUCEQkSFgxLAU/nDtKfy/90/jpU0VLLhlMXe8sJnmtqPrkuKNXdVcdtdS2jq7+NOi05h7wtBsE+hNhUBEhoS4OONTZ43jXzeczRkT8rjl2U0suGUxf16xm47OrgFf29rRyc3PbOAjdy4hNSmeP3/2dKaPygpTcv9ZKM6nhVNJSYlbsWKF3zFEJMIt317Fj55az+rdNeSmJ3HhzJFcdNIo5o3Nfcd9AOv21XHjn1exoayey0rG8O33T2NYSvSOP9wfMyt1zpX0uUyFQESGKuccL2wo57GVe/n3+gO0tHeRn5FEWlICbR1dtHd2Ud3URl5GMj/90EzOmzrC78ieGagQJIQ7jIhIuJgZ500dwXlTR9DY2sG/1x/gPxsr6HKOpIQ4khLiyE1L4tozx5GTHv0DzBwrFQIRiQnpyQlcfPJoLj55tN9RIo4ai0VEYpznhcDM4s1spZk90ceyE8zseTNbY2aLzWyM13lEROSdwnFE8F9Af8MK3QL80Tl3EvB94CdhyCMiIj14WgiC3/DfB9zbzyrTgOeD0y8CF3uZR0REDuf1EcGtwNeA/u7mWA18ODh9CZBpZnm9VzKzRWa2wsxWVFRUeJNURCRGeVYIzOz9QLlzrnSA1b4CnGNmK4FzgL3AYf3JOufuds6VOOdKCgqGfr8fIiLh5OXlo2cCHzCzC4EUYJiZPeicu6p7BefcPuBDAGaWAXzYOVfrYSYREenFsyMC59w3nXNjnHNjgcuBF3oWAQAzyzez7gzfBO7zKo+IiPQt7DeUmdn3gRXOuceBc4GfmJkDXgKuP9LrS0tLK81sZx+LsoDeRxM95/Ve3v28r3XygcrB7M8gcwxm+ZHy937e17TyR0Z+OPZ9OFL+gdYZKG/v50Mxf8/pSMg/UM6ez8P1GXRCv69wzg2JB3D3QPN6L+9+3tc6BApVyHIMZvmR8g+0P733Rfn9zX88+3Ck/EezD7GWPxT/h0KZf6CcA/zePf8b6OsxlO4s/ucR5vVe/s9BrBOqHINZfqT8vZ/3Na38Qz//QOsMlLf386GYf7DbH0go8/eeFymfQYeJut5Hw8HMVrh+eumLBsrvv2jfB+X3V7jzD6UjglC62+8Ax0n5/Rft+6D8/gprfh0RiIjEOB0RiIjEOBUCEZEYN+QLgZndZ2blZrb2GF4718zeNLMtZnabmVmPZV80s41m9paZ/Ty0qd+RIeT5zex7ZrbXzFYFHxeGPvmhDJ78/oPLv2JmzszyQ5f4sAxe/P5/EOx6fZWZPWtmo0Kf/FAGL/LfbGYbgvvwmJllhz75O3J4sQ+XBv92u8ws5I2yx5O5n/e7xsw2Bx/X9Jg/4N/IoB3rtarR8gDmA3OAtcfw2uXA6YAB/wIWBucvAP4NJAefD4+y/N8DvhKtv//gsiLgGWAnkB9N+YFhPdb5EnBXlOU/H0gITv8M+Fm0/R8CpgKTgcVASaRkDuYZ22teLrAt+DMnOJ0z0P4d7WPIHxE4514CqnrOM7MJZva0mZWa2ctmNqX368yskMAf7FIX+I3/EfhgcPHngJ8651qD2yiPsvxh42H+XxHo2dbTqx28yO+cq+uxajoe7oNH+Z91znV3Dvka4OmAUh7tw3rn3MZIy9yP9wLPOeeqnHPVwHPABaH8Gx/yhaAfdwNfdM7NJdAD6m/7WGc0sKfH8z3BeQCTgLPNbJmZ/cfM5nma9nDHmx/gC8FD+/vMLMe7qH06rvxm9gFgr3NutddB+3Hcv38z+5GZ7QY+BnzHw6x9CcX/n26fJPBNNNxCuQ/hMpjMfRkN7O7xvHs/QrZ/MTd4vQV6OT0D+EuP02nJfa3ax7zub24JBA7RTgPmAX82s/HBquypEOW/E/hB8PkPgF8Q+IP23PHmN7M04CYCpyfCLkS/f5xzNwE3mdk3gS8A3w1x1D6FKn/wvW4i0G38/4Uy45GEch/CZaDMZnYtgZEcASYCT5lZG7DdOXcJ/e9HyPYv5goBgaOgGufcyT1nmlk80D12wuMEPix7HvKOAfYFp/cAfwt+8C83sy4CnUSFY9Sc487vnDvQ43X3AIeNJ+2h480/ARgHrA7+QY0B3jCzU5xzZR5nh9D8/+npIeBJwlQICFH+YIPl+4HzwvEFqJdQ/xuEQ5+ZAZxz9wP3A5jZYuATzrkdPVbZQ6CDzm5jCLQl7CFU+xfqRpJIfABj6dFoAywBLg1OGzCrn9e9TuBbf3dDzIXB+dcB3w9OTyJw2GZRlL+wxzpfBv4UTb//XuvswMPGYo9+/yf2WOeLwKNRlv8CYB1Q4GXucPwfwqPG4mPNTP+NxdsJnIXICU7nDmb/Bp01XP+Qfj2Ah4H9QDuBCvopAt8onyYwVOY64Dv9vLYEWAtsBe7g7Tuxk4AHg8veAN4VZfkfAN4E1hD45lQYTfl7rbMDb68a8uL3/9fg/DUEOggbHWX5txD48rMq+PDsqicP9+GS4Hu1AgeAZyIhM30UguD8TwZ/71uAa4/mb2QwD3UxISIS42L1qiEREQlSIRARiXEqBCIiMU6FQEQkxqkQiIjEOBUCGRLMrCHM27vXzKaF6L06LdAT6Voz++eRevM0s2wz+3woti0CGqFMhggza3DOZYTw/RLc2x2reapndjP7A7DJOfejAdYfCzzhnJsRjnwy9OmIQIYsMysws7+a2evBx5nB+aeY2RIzWxn8OTk4/xNm9hcz+yfwrJmda2aLzexRC/S//3/d/b0H55cEpxuCncitNrPXzGxEcP6E4PPXzez7gzxqWcrbnetlmNnzZvaGBfqcvzi4zk+BCcGjiJuD6341uJ01ZvY/Ifw1SgxQIZCh7NfAr5xz84APA/cG528A5jvnZhPo+fPHPV5zOnCNc+5dweezgRuAacB44Mw+tpMOvOacmwW8BHymx/Z/Hdz+EfuACfaVcx6Bu70BWoBLnHNzCIyB8YtgIfoGsNU5d7Jz7qtmdj5wInAKcDIw18zmH2l7It1isdM5iR3vBqb16O1xmJllAlnAH8zsRAK9NSb2eM1zzrme/cgvd87tATCzVQT6j3ml13baeLvjvlLgPcHp03m7f/iHgFv6yZna471LCfQ3D4H+Y34c/FDvInCkMKKP158ffKwMPs8gUBhe6md7Iu+gQiBDWRxwunOuuedMM7sdeNE5d0nwfPviHosbe71Ha4/pTvr+m2l3bze29bfOQJqdcyebWRaBgnI9cBuBsQoKgLnOuXYz2wGk9PF6A37inPvdUW5XBNCpIRnaniXQ1z8AZtbdBXAWsDc4/QkPt/8agVNSAJcfaWXnXC2BoSu/YmaJBHKWB4vAAuCE4Kr1QGaPlz4DfDLY5z1mNtrMhodoHyQGqBDIUJFmZnt6PG4k8KFaEmxAXUeg+3CAnwM/MbNXgXgPM90A3Ghmy4FCoPZIL3DOrSTQO+XlBAZ8KTGzFQSODjYE1zkIvBq83PRm59yzBE49LTWzN4FHeWehEBmQLh8V8UhwNLVm55wzs8uBK5xzFx/pdSLhpjYCEe/MBe4IXulTQ5iGAxU5WjoiEBGJcWojEBGJcSoEIiIxToVARCTGqRCIiMQ4FQIRkRj3/wEjEwrt0ArngQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Find optimal learning rates\n",
    "learn_lm.lr_find()\n",
    "learn_lm.recorder.plot(skip_end=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.947898</td>\n",
       "      <td>3.717568</td>\n",
       "      <td>0.325702</td>\n",
       "      <td>00:38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fit the last layers\n",
    "learn_lm.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save language model for quicker start when returning\n",
    "learn_lm.save('fit_head')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LanguageLearner(data=TextLMDataBunch;\n",
       "\n",
       "Train: LabelList (23594 items)\n",
       "x: LMTextList\n",
       "xxbos xxmaj weather update - a cold front from xxmaj cuba that could pass over xxmaj haiti,xxbos xxmaj is the xxmaj hurricane over or is it not over,xxbos xxmaj looking for someone but no name,xxbos xxup un reports xxmaj leogane 80 - 90 destroyed . xxmaj only xxmaj hospital xxmaj st. xxmaj croix functioning . xxmaj needs supplies desperately .,xxbos says : west side of xxmaj haiti , rest of the country today and tonight\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: data;\n",
       "\n",
       "Valid: LabelList (2621 items)\n",
       "x: LMTextList\n",
       "xxbos xxmaj back in xxmaj santiago from traveling the s of xxmaj chile in 2 days . xxunk traveled & visited 5 cities giving supplies .. xxmaj it 's amazing what you see !,xxbos willing to help in any way i can ; i do n't own a car but can drive and pack and xxunk and carry items . xxmaj can help with info , preparing food , etc .,xxbos xxmaj it was the only available but rather difficult option in the wake of harsh weather conditions including snow , heavy rains and landslides .,xxbos xxmaj mali was xxunk into chaos in 2012 when the xxup mnla launched an offensive in the northern desert helped by xxmaj islamist militants linked to xxmaj al - xxmaj qaeda who then cast aside the xxmaj tuareg rebels and took the region for themselves .,xxbos xxmaj extreme water shortage caused by two to three years of failed rains has killed huge numbers of farm animals and severely reduced crop production .\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: data;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): AWD_LSTM(\n",
       "    (encoder): Embedding(11536, 400, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(11536, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(400, 1152, batch_first=True)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(1152, 1152, batch_first=True)\n",
       "      )\n",
       "      (2): WeightDropout(\n",
       "        (module): LSTM(1152, 400, batch_first=True)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=11536, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7f7c1813fef0>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
       "learn: ...\n",
       "alpha: 2.0\n",
       "beta: 1.0], layer_groups=[Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): Embedding(11536, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(11536, 400, padding_idx=1)\n",
       "  )\n",
       "  (2): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=11536, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")], add_time=True, silent=False)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load language model from this point\n",
    "learn_lm.load('fit_head')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.503204</td>\n",
       "      <td>3.571204</td>\n",
       "      <td>0.341560</td>\n",
       "      <td>00:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.378392</td>\n",
       "      <td>3.490188</td>\n",
       "      <td>0.349952</td>\n",
       "      <td>00:51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.097170</td>\n",
       "      <td>3.456255</td>\n",
       "      <td>0.357571</td>\n",
       "      <td>00:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.781008</td>\n",
       "      <td>3.477144</td>\n",
       "      <td>0.357262</td>\n",
       "      <td>00:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.577574</td>\n",
       "      <td>3.506318</td>\n",
       "      <td>0.356274</td>\n",
       "      <td>00:52</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Unfreeze other layers and train some more\n",
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(5, 1e-3, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy here is a measure of how well the model predicts the next word of all of the words in our messages, at around 35% accuracy our model is doing quite well!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXgV1fnA8e+bfd8TCJAQwg4BAkQEZUcRrUVr3VDbYq24tFWrVam1dWlrtVpFqz+tG1RrpSqKu7hBQZQlYQm7bAHCngAhZF/O74+ZQBKSEJabucv7eZ77ZO7MuXPfySTzzjln5owYY1BKKeW7/JwOQCmllLM0ESillI/TRKCUUj5OE4FSSvk4TQRKKeXjApwO4GQlJCSYtLQ0p8NQSimPkpOTU2CMSWxqmcclgrS0NLKzs50OQymlPIqIbGtumTYNKaWUj9NEoJRSPk4TgVJK+TiP6yNQSnmPqqoq8vPzKS8vdzoUrxESEkKnTp0IDAxs9Wc0ESilHJOfn09kZCRpaWmIiNPheDxjDIWFheTn59OlS5dWf06bhpRSjikvLyc+Pl6TwBkiIsTHx590DUsTgVLKUZoEzqxT+X36TCLYsKeYx+es52BJpdOhKKWUW/GZRLC1oITn5m5m56Eyp0NRSrmJwsJCMjMzyczMpH379nTs2PHo+8rK1p00Xn/99WzYsMHFkbqWz3QWJ0QEAXBAawRKKVt8fDwrVqwA4MEHHyQiIoLf/va3DcoYYzDG4OfX9Hnz9OnTXR6nq/lMjSAu3EoEuw6V8b/v91NVU+twREopd7Vp0yYyMjK4+eabGTRoELt372bKlClkZWXRt29fHn744aNlhw8fzooVK6iuriYmJoapU6cyYMAAhg0bxr59+xzcitbzmRpBfEQwAFPfXQXAU1cN4EcDOzkZklKqnoc+XMPaXYfP6Dr7dIjigR/2PaXPrl27lunTp/PCCy8A8OijjxIXF0d1dTVjxozh8ssvp0+fPg0+U1RUxKhRo3j00Ue58847efXVV5k6deppb4er+UyNICqkYc7bvK/EoUiUUp6ga9eunHXWWUffv/nmmwwaNIhBgwaxbt061q5de9xnQkNDufDCCwEYPHgweXl5bRXuafGZGkHjS6q2FmgiUMqdnOqZu6uEh4cfnd64cSNPP/00S5YsISYmhuuuu67Ja/WDgoKOTvv7+1NdXd0msZ4un6kRNLZ5/xGnQ1BKeYjDhw8TGRlJVFQUu3fvZs6cOU6HdEb5VCL45LYRR6fX7ynWDmOlVKsMGjSIPn36kJGRwY033si5557rdEhnlBhjnI7hpGRlZZnTeTDNnDV7eHnBFpbmHeRPl/TlJ8PSzlxwSqmTsm7dOnr37u10GF6nqd+riOQYY7KaKu9TNQKAC/q25/UbzgZgi/YTKKWU7yUCgJBAfwCmL8yjuLyKrQUlFB6pcDgqpZRyhk8mAoCbRqYD0O/BzxnzxDx+8soSPK2ZTCmlzgSfTQR3je959G5jgLW7D3P/7NVU2x3IOw+Vcd3Li/nt2yvZXljqVJhKKeVyPnMfQWNBAX58O3Us/n6CAEP/+jVvLN7OG4u3H1f2nZx8Hr+8P1dkpbR9oEop5WI+WyMAq68g0N+PAH8/Pv/NyAbLUuPCmHxOGj8e1Ik+yVHcP3s1u3TkUqWUF3JZIhCREBFZIiIrRWSNiDzUTLkrRWStXeY/rornROLCg3jpp1l8evsIFt83jvn3jOHBiX35+5UD+OdPBlNrDOc8+jV3vrWC8qoap8JUSp1Bo0ePPu7msGnTpnHrrbc2+5mIiAgAdu3axeWXX97sek90mfu0adMoLT3W7HzRRRdx6NCh1oZ+RrmyRlABjDXGDAAygQkiMrR+ARHpDvwOONcY0xe4w4XxnND5fdrROzmKdlEhDeanxIUxffIQ2keF8O6ynfT6w2eMenwuMxZuBWDJ1gPkH9R+BKU8zaRJk5g5c2aDeTNnzmTSpEkn/GyHDh145513Tvm7GyeCTz75hJiYmFNe3+lwWSIwlrpxHALtV+PLcm4EnjPGHLQ/47Zjtg7vnsCi+8bx2/E9SI0Lo/BIJQ9+uJa0qR9z5T+/Y/hjc/lw5S698kgpD3L55Zfz0UcfUVFhXT6el5fHrl27yMzMZNy4cQwaNIh+/frx/vvvH/fZvLw8MjIyACgrK+Pqq6+mf//+XHXVVZSVHWtGvuWWW44OX/3AAw8A8Mwzz7Br1y7GjBnDmDFjAEhLS6OgoACAJ598koyMDDIyMpg2bdrR7+vduzc33ngjffv2Zfz48Q2+53S4tLNYRPyBHKAb1gF/caMiPexyCwF/4EFjzGdNrGcKMAUgNTXVlSGf0K/GdudXY7tTWlnNFS98xxp72NyI4AB+/eZy/vVtHn+4uA8DUpzJ7Ep5rE+nwp5VZ3ad7fvBhY82uzg+Pp4hQ4bw2WefcckllzBz5kyuuuoqQkNDee+994iKiqKgoIChQ4cyceLEZp8H/PzzzxMWFkZubi65ubkMGjTo6LK//OUvxMXFUVNTw7hx48jNzeW2227jySefZO7cuSQkJDRYV05ODtOnT2fx4sUYYzj77LMZNWoUsbGxbNy4kTfffJOXXnqJK6+8klmzZnHddded9q/JpYnAGFMDZIpIDPCeiGQYY1Y3+v7uwGigE7DALnOo0XpeBF4Ea4gJV8bcWmFBAXx82wgqqmsIDvCnptbwdvYO/vzxOi55biF3X9ATgLG9kliVX0R8RBDdkiLoHB9+gjUrpdpSXfNQXSJ49dVXMcZw3333MX/+fPz8/Ni5cyd79+6lffv2Ta5j/vz53HbbbQD079+f/v37H1321ltv8eKLL1JdXc3u3btZu3Ztg+WNffPNN/zoRz86OvrpZZddxoIFC5g4cSJdunQhMzMTOLPDXLfJ5aPGmEMiMg+YANRPBPnAImNMFbBVRDZgJYalbRHXmRAcYN2l7O8nXD0klXO7JfCz6Ut4fI71DNO6n3WevjqTSzI7tnmcSrm9Fs7cXenSSy/lzjvvZNmyZZSVlTFo0CBmzJjB/v37ycnJITAwkLS0tCaHna6vqdrC1q1beeKJJ1i6dCmxsbFMnjz5hOtpqXk5ODj46LS/v/8Zaxpy5VVDiXZNABEJBc4D1jcqNhsYY5dJwGoq2uKqmNpCSlwYr/18CEPT4xjXK4mpF/biplHpTBmZjr+fcPc7udz8eg6b9ukw2Eq5g4iICEaPHs3Pf/7zo53ERUVFJCUlERgYyNy5c9m2bVuL6xg5ciRvvPEGAKtXryY3Nxewhq8ODw8nOjqavXv38umnnx79TGRkJMXFxU2ua/bs2ZSWllJSUsJ7773HiBEjjit3JrmyRpAM/MvuJ/AD3jLGfCQiDwPZxpgPgDnAeBFZC9QAdxtjCl0YU5voFBvGzCnDjpt/08h0/vzxOt5bvpOv1u9l/Z8uxN+v6TbHMyI/B/IWQOow6JAJAcEn/oxSPmjSpElcdtllR68guvbaa/nhD39IVlYWmZmZ9OrVq8XP33LLLVx//fX079+fzMxMhgwZAsCAAQMYOHAgffv2JT09vcHw1VOmTOHCCy8kOTmZuXPnHp0/aNAgJk+efHQdv/jFLxg4cKBLn3bmc8NQu4M/zF7N64u2MaZnIq9OPqvZDqjTtuBJ+Mq+fcM/GDoOgpSzrcSQMgTC4lzzvUq1kg5D7RonOwy1zw4x4aQ//rAPeYUlzN2wn/+bt5lbR3d1TTIYcScMvA52LIbti6zXd8/CQutyNBJ7HUsMqWdDbBdwVVJSSrktrRE4xBjDT19dwoKNBcSHB5EYGcyz1wyiW1KEa7+4shR2LYPt38H2xbBjCVQUWcsi2tmJYaj1at8f/ANdG4/yaVojcA2tEXgIEeGpqzLJ+vOXFJZUUlhSyfin/sd/bhxKt6QIEiJc1J4fFAZpw60XQG0t7F93LDFsXwTrPrCWBYZBx8HHagydhkBIlGviUj7LGOO65lEfdCon91ojcAO5+YfYuPcIf3x/NSWV1jhGgzvH8sQVA+iS4MB9B0U7YcciOzF8B3tXg6kF8YOkvlZSSB1m1R5idERWdeq2bt1KZGQk8fHxmgzOAGMMhYWFFBcX06VLlwbLWqoRaCJwIyt3HOKxz9bz7eZjF07d/4Pe3DC8i7P/JBXFkL/UbkpaBDuWQpX9mM+oTscSQ+pQSOoDfv7Oxao8SlVVFfn5+Se8tl61XkhICJ06dSIwsGGzriYCD1NaWU1ufhFXv7gIgJBAP649uzO/GNGF5OhQh6MDaqqtWsL2RXbNYREU77aWBUdBp6xjNYZOWRCkd1Mr5TRNBB6qtLKaJ+Z8z6v2KKd1Zt0yjMGd3ejST2Pg0PaGiWHfOsCA+ENy/2OJIXUoRDZ9m75SynU0EXi4orIqtheWMuPbPGYtywfg3gm9GN+3HV0TXXyV0akqO2g1IdUlhp05UG1X/2PT6iWGYZDQA/x8+hlJSrmcJgIvsmlfMde8tJh9xdawuTeNTOeu8T0JCvBjX3E5H6zYxYCUGM5Kc6MaA0B1JexeeSwxbF8EpdaQu4TEWDWFusTQYSAEhrS8PqXUSdFE4GUKjlTw7eZC3snJZ/73+5ssc/25aQxNj+eCvm7aDGMMFG62E4N96WrhRmuZf5CVDFLqXZ0UHu9svEo5xRjrgo3SAggMh8h2p7QaTQRe7KPcXTz5xffkFZTQr1MMHaJD+HT1nqPLQwP9GdQ5hnO6JvDLMd0cjLQVSgoa3gW9aznUVlnLEnrUuwt6KMSl613QyjMZA5VHrL/30kIo2W9Nl+xv+L60wJ5fADVWCwDD74TzHjilr9VE4APq35RTW2vI3VnE3z/fwIKNBUfLvHXTMIZ0cbMmo5ZUlVnJoC4x7FgM5fajKsITGyaG9v0hIMjZeJXvqiyxD+D2gby0oN4Bvf7B3j7AVzdzuWxgGIQlQHjdKxHC4o9Nt+8P7TNOKURNBD7sf9/v59VvtvLd5kI6xYYy5zcjCfT30I7Z2loo2FAvMSyCg3nWsoBQ+y7ooZA8ACKTrSp0RDsddVWdvMrSegf0gnpn7AWNDvb2q7qZ5wIEhB47qIfZB/Pw+HrT9ZcluPRSa00Eii/W7uXG17J55Ef9uOZsZx/3eUYV72mYGHbngqlpWCY0FiLa24mhiZ8R7azp4EhntkG5XlXZ8WflTTXH1C2rKm16PQEhLZ+xNz7YB4W7TROmjjWkOK93ElmdY5n25ff8aGBHQoO85O7fyPbQ91LrBVBxBAo3wZG9VpJo/LNwoTVdU3n8uuo64k6UNMLi3Oaf2ycZYx3YSwubOENv3Dxjv+ruhG/MP7jhWXlCD3s6vt4Ze72DfVCEV+57TQQ+QkS498JeXPHCd/z547X8+dIM7xzbJTjCeghPS4yx7nNoLlkc2WvVLI58YXXqNeYXeKwW0VLSCE8Ef/0XO051JVQchvIi61VxGMoPNzPdTJna6qbX7R/U8Iw9vpv9Pr7pM/bgSK88sJ8s/Sv1IWelWY/PfGPxdmLDgrhrfA/vTAYnImKd1YfFQdIJhkCuONIwQTROGge3Wpe/lh1o6ousg1FLyaLup6fcN1Fbc+ygfPRA3Xi6qJn59nRzHaX1BUVCSLQ12m1wlPU7SuhhTdfND407/ow9OEoP7KdAE4GPefKqTH4+YynPzt1Enw5RXNQv2emQ3FtwhPWK79pyuerKRoliDxTvbThv7xo4su/4PgywDm4R7SEiyWruimhn/2yUNE7nQGeMdXVLkwfwohYO7PWmK49/xu5xAkKPHcBDoq1XdEq9A3t0w4N84+ngSB24sI1pZ7EPqqyuZehfv+JASSW3ju7KPRNafh6rOoNqa6y27QZNUXVJY4+VKOqWNXXmHBBar7+iXtIICG7mAH6o3vThppNQfX4B9gG57qBc72De1EG78QE/OEov43VTetWQOs7cDfu4fvpSAD69fQS9k/WBM27FGOuA3lI/RrGdOOqeMAf1zqobH8ybmo4+fn5gmDateClNBKpJew+X84NnviEmLJD3f3ku4cHaUuiRKkutq6CCo3TwPtWslhKB/tX4sHZRITx55QA27TtC3wfm8OL8zewp0geEeJygMAiN0SSgTpn+5fi4kT0S+eUYqyP0kU/WM/SvX3Hfe6tYtKXwBJ9USnkLTQSKuy/oxdLfn0dGR6uf4D+Lt3P1i4t4fM56hyNTSrUF7SNQx8nNP8TEZxcefZ+ZEsOkISlcmZXim/cdKOUFtI9AnZT+nWJY/ofz6Z5kPf1sxY5D3DtrFf9etM3hyJRSruCyy0REJASYDwTb3/OOMabJgbRF5HLgbeAsY4ye7ruB2PAgvrhzFOVVNeRsO8g/vt7IAx+sITEymAkZehOaUt7ElTWCCmCsMWYAkAlMEJGhjQuJSCRwG7DYhbGoUxQS6M+53RJ4dfJZdE2M4OZ/L+ORT9ZRcKSCNbuKKCqtcjpEpdRpclmNwFidD3UjdgXar6Y6JP4E/A34ratiUacvLCiAqRf24oZ/ZfPi/C28OH8LAHHhQbw6+Sz++sk61u4+zPXnpHHbuO4EnOYzD5ZtP8jz8zaT0SGanu0jtBailAu5tLNYRPyBHKAb8Jwx5t5GywcC9xtjfiwi84DfNtU0JCJTgCkAqampg7dt07ZqpxSVVvH45+vZebCMuRuafl4ywJSR6Uyd0As/v2OdyzW1Bn8/obK6lqKyKhIjjz0wpu7vcPaKnczK2cmKHYc4UnFshMn0hHAmZLTn7gt6aoe1UqfA8TuLRSQGeA/4tTFmtT3PD/gamGyMyWspEdSnVw25ly/X7iU3/xDDuibQJSGce2blMv97K0F0jg9jVI9E/nBxH9bsOswNM5bSKTaUVTuLEBEuGdCBtIRwvt9bzEe5uwkL8qe0sgYROL93O341thslFTV8u7mA/y7dwb7iCjJTYiivqqHgSAXj+7bnhuFd6JoY4fBvQSn353gisIN4ACgxxjxhv48GNnOs+ag9cACY2FIy0ETg/vYUlfPIJ+v4YOWuZstEBgdQXNFwTPlxvZIYmBrDL0akExLYcPTJ2lrDda8s5tvNx9/odmlmByZmdqC4vJph6fEkRXnIkM5KtSFHEoGIJAJVxphDIhIKfA48Zoz5qJny89AagVcprazmgffX8HZOPiGBfnzwq+HU1BrSE8MJDvCnqqaW/yzezuieicRHBBNxgrGOKqtrKS63Oqdjw4JYt+cw0xfm8U5OfoNyj17Wj6uHeNHjOJU6A5xKBP2BfwH+WFcnvWWMeVhEHgayjTEfNCo/D00EXun7vcVUVteS0THaJesvKqviv0u3s+NAGf9evI32USEsvHdsg/4JpXydI88sNsbkAgObmP/HZsqPdlUsylk92rn2ofDRoYFMGWmNl3RWlzhue3M5i7YWck7XBJd+r1LeQu8sVl7l/N7tCA7w44u1e50ORSmPoYlAeZXQIH/OTo9nwcYCp0NRymNoIlBeZ2T3BDbtO8KuQ2VOh6KUR9BEoLzOiO6JAHyjtQKlWkUTgfI6PdpFkBQZzPyNzd/5rJQ6RhOB8joiwvDuCXyzqYCaWs963oZSTtBEoLzSyO6JHCqtYs2uIqdDUcrtaSJQXuncbtY9BHr1kFInpolAeaXEyGD6JEexQPsJlDohTQTKa43okUDOtoOUNBrcTinVkCYC5bXG9WpHVY3hs9V7nA5FKbemiUB5rbPSYukcH8bsFTudDkUpt6aJQHktEeH83u1YvOUApZXaPKRUczQRKK82tncSlTW1zGr0zAKl1DGaCJRXG5Yez8DUGP788ToKjlQ4HY5SbkkTgfJqIsLNo7pSUV173JPMlFIWTQTK613Qtz19O0Tx9bp9ToeilFvSRKB8wrheSWRvO8Ch0kqnQ1HK7WgiUD5hbO921Br4SmsFSh1HE4HyCf07RpMaF8bbOTucDkUpt6OJQPkEPz/hqrNSWLTlAFv2H3E6HKXciiYC5TOuyOpEgJ8wc6nWCpSqTxOB8hlJkSFckNGeNxZtY3+x3lOgVB1NBMqn3HV+Dyqqa/n75xucDkUpt6GJQPmU9MQIJp+Txn+zd7B8+0Gnw1HKLWgiUD7njvN7EBEcwOvfbXM6FKXcgssSgYiEiMgSEVkpImtE5KEmytwpImtFJFdEvhKRzq6KR6k6EcEBXNw/mU9X7+GIPrRGKZfWCCqAscaYAUAmMEFEhjYqsxzIMsb0B94B/ubCeJQ66seDOlFWVcOnq3Y7HYpSjnNZIjCWugu2A+2XaVRmrjGm1H67COjkqniUqm9w51jS4sN45ZutVFTXOB2OUo5yaR+BiPiLyApgH/CFMWZxC8VvAD5tZj1TRCRbRLL379eHkavTJyL87qLerN9TzB9nr8EYc+IPKeWlXJoIjDE1xphMrDP9ISKS0VQ5EbkOyAIeb2Y9LxpjsowxWYmJia4LWPmU8X3acVVWCv/N3sH/zdusyUD5rIC2+BJjzCERmQdMAFbXXyYi5wG/B0YZY/QuH9VmRIS/XtaP0qoaHp+zgSB/P24cme50WEq1uVbVCESkq4gE29OjReQ2EYk5wWcS68qISChwHrC+UZmBwD+BicYYHRZStTk/P+HpqzI5r3cSf/lkHTMWbnU6JKXaXGubhmYBNSLSDXgF6AL85wSfSQbmikgusBSrj+AjEXlYRCbaZR4HIoC3RWSFiHxw8pug1Onx8xP+dGkGMWGBPPjhWpbmHXA6JKXalLSmXVRElhljBonI3UC5MeYfIrLcGDPQ9SE2lJWVZbKzs9v6a5UPKKusYcgjX5IWH87sX56Lv584HZJSZ4yI5Bhjsppa1toaQZWITAJ+Bnxkzws8E8Ep5S5Cg/y5d0IvVu0s4oJp83lx/mbyCkqoqql1OjSlXKq1ieB6YBjwF2PMVhHpAvzbdWEp5Yxrz07lmUkDCQvy55FP1jP6iXmc++jX2lykvFqrmoYafEAkFkgxxuS6JqSWadOQaivfbi7g/tmrySsoIS48iM9/M4q48CCnw1LqlJx205CIzBORKBGJA1YC00XkyTMZpFLu5pyuCXx912g+/PVwisqquPblxew4UHriDyrlYVrbNBRtjDkMXAZMN8YMxrocVCmv17dDNNcMSWXd7sOM+Ntcfv3mcmpq9eYz5T1ae0NZgIgkA1di3fyllE956JIMLujbnq/W7+OVb7aSGhfKb87rQYC/juSuPF9rE8HDwBxgoTFmqYikAxtdF5ZS7uecbgkM6xrPtsJSnpu7ma/W7ePVyWfRISbU6dCUOi0n3VnsNO0sVk6rrqnlsc/W89KCrQT4Cff/oDfXDe2stQPl1s5EZ3EnEXlPRPaJyF4RmSUiOmS08kkB/n7cO6EXt47uSlpCOA9+uJb73ltFUVmV06EpdUpaewozHfgA6AB0BD605ynlkwL8/bhnQi++vHMU15ydylvZ+Qx46HMe+2w9JRXV7C4qY8nWA8zbsI/K6lp2HCjFGHNKI5xWVusNbcq1WjvExAp7OOkW57UFbRpS7qa8qoYHP1jDe8t3UtHCQbtHuwjyCkoJDvQjKTKYMT2TmJjZgQc+WEPhkUou6pfMBX3bsXHvEd7K3sHa3YcprbQemjO4cywXZrTnJ8M6Exzg31abprxIS01DrU0EXwIzgDftWZOA640x485UkK2liUC5q/KqGv71bR65+UUUV1Qzvk87VuUXMXvFTrokhLN+TzEAXRPD2by/5ITrG5IWR2lVNUVlVeQfLMMYiA0L5J4Jvbj6rBQA1u0u5vVFefywfwfO6Zbg0u1Tnu1MJIJU4FmsYSYM8C1wmzFm+5kMtDU0EShPU11TS4C/39Gb0VLiwigqq2LGwjw27T/CPRf0pF1UCP/4eiOb9h0hwN+P28d1o1tS5NF1GGN46suNPPNV0xfrpSeE8/VvR7fF5igPddqJoJmV3mGMmXZakZ0CTQTKl+UVlPDc3E18mLsLgPKqWoakxbEk7wA5959HfESwwxEqd+WqRLDdGJN6WpGdAk0ESjWUs+0gP37+W/7v2kFc1C/Z6XCUmzoTw1A3ud7T+KxS6gzp3yma8CB/Fm4qcDoU5aFOJxF41p1oSnmpQH8/hqbHayJQp6zFRCAixSJyuIlXMdY9BUopNzC8ewJ5haU6Oqo6JS0mAmNMpDEmqolXpDGmteMUKaVcbLh96ajWCtSp0MFRlPIC3ZIiaBcVzDeaCNQp0ESglBcQEc7tlsC3mwup1WclqJOkiUApLzG0SzwHSirZUnDE6VCUh9FEoJSXyEqLBWDJ1oMOR6I8jSYCpbxEl4RwkiKDWbhZ+wnUydFEoJSXEBFGdE/km40FOnS1OimaCJTyIhcPSKaorIr3luc7HYryIC5LBCISIiJLRGSliKwRkYeaKBMsIv8VkU0islhE0lwVj1K+YHSPRPp1jObZuZuortFagWodV9YIKoCxxpgBQCYwQUSGNipzA3DQGNMNeAp4zIXxKOX1RISbR3Vlx4EyluQdcDoc5SFclgiMpe46tkD71fgC50uAf9nT7wDjREQHs1PqNIzplUhwgB+fr9nrdCjKQ7i0j0BE/EVkBbAP+MIYs7hRkY7ADgBjTDVQBMQ3sZ4pIpItItn79+93ZchKebywoABG9Ujk09W7T+kZycr3uDQRGGNq7OcadwKGiEhGoyJNnf0f95drjHnRGJNljMlKTEx0RahKeZXRPZPYe7iCrQUnfiSmUm1y1ZAx5hAwD5jQaFE+kAIgIgFANKANm0qdprPT4wBYtEX/ndSJufKqoUQRibGnQ4HzgPWNin0A/Myevhz42mhdVqnTlp4QTmpcGJ+u3u10KMoDuLJGkAzMFZFcYClWH8FHIvKwiEy0y7wCxIvIJuBOYKoL41HKZ4gIEwd0YOGmAvYXVzgdjnJzLnumgDEmFxjYxPw/1psuB65wVQxK+bJLMjvw7NxNfJy7i8nndnE6HOXG9M5ipbxU93aR9E6O4v2Vu5wORbk5TQRKebFLMjuwfPsh1uwqcjoU5cY0ESjlxa4Y3Im48CDumLmC0spqp8NRbkoTgVJeLD4imCeu6M/GfUeYuWSH0+EoN6WJQCkvN7ZXO4akxfHKN1up0oHoVBM0ESjlA24alc7OQ2XMXr7T6VCUG9JEoJQPGNsriYyOUfzj601aK/plY28AABMqSURBVFDH0USglA8QEX5zXg+2Hyjl+XmbnQ5HuRlNBEr5iLG9khjRPYFXvtnKgZJKp8NRbkQTgVI+QkT4/Q96c7i8ikc+Wed0OMqNaCJQyof0ah/Fz8/twrvL8nWIanWUJgKlfMxNo9IJ9Pfj+XmbnA5FuQlNBEr5mKTIECYNSWXWsp3M/16f+Kc0ESjlk+4a34OU2FAe+nANldV6Oamv00SglA+KDAnkN+f3YPP+Er7bUuh0OMphmgiU8lEX9G1PSKAf7y/fqQ+593GaCJTyUSGB/lwzpDPvLt/JT19dQm2tJgNfpYlAKR/2u4t6MfmcNBZsLOATfb6xz9JEoJQPC/T34w8X96FbUgS3z1zBnf9dQVFZldNhqTamiUApH+fvJ/xj0kDOSovl3eU7GfDQ5zz4wRpKKvRBNr5CE4FSit7JUbx541D+fsUAggL8mPFtHvfMynU6LNVGNBEopQBrLKIfD+7E4t+N46y0WD7O3c3gP33BzCXbnQ5NuViA0wEopdxLbHgQ//xJFv+cv5lFWw4w9d1VLNxcSIfoEI5UVDN3/T4qawxZnWP55ZhuhAb50y0pwumw1WkQT7t+OCsry2RnZzsdhlI+oaqmlite+I4VOw4BIAJNHTJeuG4QEzKS2zg6dTJEJMcYk9XkMk0ESqmWGGPYXVROSUU1qfFhTPtyI9eenUpufhEb9hTz9FcbiQgOIDMlhlpj+OWYbgxLj8fPT5wOXdWjiUAp5TKb9h1hymvZbKk3rHW3pAiuykrhcHkVXRMjmJDRnv3FFXSKDUVEE4QTHEkEIpICvAa0B2qBF40xTzcqEw38G0jF6q94whgzvaX1aiJQyv1U19Syfk8xh8uqmL1iJ29l5zdZbkhaHNOuzqRDTGgbR6icSgTJQLIxZpmIRAI5wKXGmLX1ytwHRBtj7hWRRGAD0N4Y0+xz9DQRKOXeyqtq+Gz1HpIig8krLCUsyJ+5G/YRHhzAOzn5VFbX8vy1g7iwn/YptKWWEoHLrhoyxuwGdtvTxSKyDugIrK1fDIgUq64YARwA9C4WpTxYSKA/lw7sCMA53ax5de+vzErh0ucWcssby8hMiWH65LOIDQ9yKlRla5P7CEQkDRgILG606FmgN7ALWAXcbow5bnB0EZkiItkikr1/vz5IQylPlZkSwzf3jmFMz0RW7DjEwD99wfSFWxsMeFdbaziidzW3KZd3FotIBPA/4C/GmHcbLbscOBe4E+gKfAEMMMYcbm592jSklHf45RvL+HiVNdBdYmQwf7i4D6O6J3L9jCUs236Ivh2i6Nk+kp8M7UxkSABFZVX0SY4mJNBPO5xPgWNXDYlIIPARMMcY82QTyz8GHjXGLLDffw1MNcYsaW6dmgiU8g7lVTU89cX37DxUxke5rR/5ND0hnMzUGO6+oCfJ0drp3FqO9BHY7f6vAOuaSgK27cA4YIGItAN6AltcFZNSyn2EBPrzu4t6A/DwJZX84+uNrNxxiNvP60Hv5EjeX76LrLRYXvlmKzW1hoSIYHYdKmPNrsO8u2wn2wpLmXXLOQ5vhXdw5VVDw4EFWG3/de3+92FdKoox5gUR6QDMAJIBwaod/Lul9WqNQCnfVl5Vw+0zlzNnzV6+nTpWL0VtJaeuGvoG6+DeUpldwHhXxaCU8j4hgf7cd1Fv5m7Yz+NzNvDUVZlOh+TxdPRRpZTH6RwfzpVZnfhk1W52HSpzOhyPp4lAKeWRpozoSq0x/OPrTU6H4vE0ESilPFJqfBjXDEnlzSXb2bL/iNPheDRNBEopj3XL6G6IwOzlO50OxaNpIlBKeaz20SGM6J7Im0t3UFRW5XQ4HksTgVLKo911fg8OlFRy7zu5VNUcN0KNagVNBEopjzYgJYY7xnXnszV7uPblxQ3GLVKto4lAKeXxfj2uO7eO7sqSrQdYmnfA6XA8jiYCpZRX+PXY7kQEB/DoZ+t19NKTpIlAKeUVQoP8efzy/qzccYgHP1jjdDgeRROBUsprXNgvmcnndOGdnHy9t+AkaCJQSnmVm0enE+AnzFy6w+lQPIYmAqWUV0mKDOG83u14O1vvLWgtTQRKKa9zy+iuHC6v5vfvrcLVT2H0BpoIlFJeZ0BKDLeP685HubtZu7vZJ98qmyYCpZRXmjQklQA/4W+fbdBawQloIlBKeaXEyGDuGt+T/32/n9krdFC6lmgiUEp5rSkj0+nfKZqps1bx3eZCp8NxW5oIlFJey99PePlnWUSGBHLT69kUl+tVRE3RRKCU8mpJkSG89NPBHC6v5tVv8pwOxy257OH1SinlLgamxnJRv/Y89eX3PDt3I/HhwZRX1zC2VxJXDE7h7C5x+PmJ02E6RhOBUsonPHllJlU1y1m3+zD5B60H3r+7bCfvLttJh+gQJg1J5caR6YQE+jscadsTT7usKisry2RnZzsdhlLKg+0uKiMqJJCcbQf5bksh//zfZmoNjOyRSG2t4fw+7Zg4oAOx4UFOh3rGiEiOMSaryWWaCJRSCv7y8VpeWrC1wbwBKTG8+JPBtIsKcSiqM6elRKBNQ0opBdw1vichgf5kdIymusawYW8xryzYwqQXF/Hfm4aRGBl83GeMMew9XEFQgB9xHlx70BqBUko1Y/GWQq55eTE1tYZzu8VzxeAUDIbXvtvG3qJy9hVXUG0/GvNHAzsycUAHhnWNd8t+BkeahkQkBXgNaA/UAi8aY55uotxoYBoQCBQYY0a1tF5NBEqptrRs+0Fm5eTzdnY+lTW1xy2PCQukb4colm07RFlVDX4CNwzvwq/Gdic6NNCBiJvmVCJIBpKNMctEJBLIAS41xqytVyYG+BaYYIzZLiJJxph9La1XE4FSygmHy6t4a+kOeraPpF/HaA6XVRMa5H+0yajgSAV/fH81Ow6UsWpnEdecncojP+rncNTHuEVnsYi8DzxrjPmi3rxbgQ7GmPtbux5NBEopd3fXWyuZtSyfPslRjOqZSEpsGO2igsnoGO1Yx7PjncUikgYMBBY3WtQDCBSReUAk8LQx5rUmPj8FmAKQmprqylCVUuq0PXxJX/IKS8jZdrDBMNgBfsIjl/XjyqwUB6M7nstrBCISAfwP+Isx5t1Gy54FsoBxQCjwHfADY8z3za1PawRKKU9QUlHNhyt3cbC0iorqGsqravly3V427TvC5HPSeOCHfaioruXfi7axp6iceyb0IijAdaP+OFYjEJFAYBbwRuMkYMvH6iAuAUpEZD4wAGg2ESillCcIDw7g6iENWzDuOK87t89czoxv81i7+zABfsK39qio7y7fyR8u7s0lAzpSYwyllTVEhwZijOGdnHwiQwKZkNHeJbG6srNYgH8BB4wxdzRTpjfwLHABEAQsAa42xqxubr1aI1BKebLaWsP/zdvEM19torKmlikj0+nbIYrn521m/Z5iggL8qKyuJT48iD9c3Idnvt7Ilv0lALx+wxBGdE88pe916qqh4cACYBXW5aMA9wGpAMaYF+xydwPX22VeNsZMa2m9mgiUUt5g16EyKqpr6ZIQDlgJ4v2VO5mxMI/N+0s4UlF9tOzkc9IwxnDn+J6nfEmqW1w1dKZoIlBK+YJthSVMX5jHjSPT6RgTetrrc/yqIaWUUienc3w4D07s2ybfpQ+mUUopH6eJQCmlfJwmAqWU8nGaCJRSysdpIlBKKR+niUAppXycJgKllPJxmgiUUsrHedydxSKyH9h2ih9PAArOYDjuxFu3TbfLs+h2ua/OxpgmByryuERwOkQku7lbrD2dt26bbpdn0e3yTNo0pJRSPk4TgVJK+ThfSwQvOh2AC3nrtul2eRbdLg/kU30ESimljudrNQKllFKNaCJQSikf5zOJQEQmiMgGEdkkIlOdjudkiEiKiMwVkXUiskZEbrfnx4nIFyKy0f4Za88XEXnG3tZcERnk7Ba0TET8RWS5iHxkv+8iIovt7fqviATZ84Pt95vs5WlOxt0SEYkRkXdEZL2934Z5w/4Skd/Yf4OrReRNEQnx1P0lIq+KyD4RWV1v3knvIxH5mV1+o4j8zIltOV0+kQhExB94DrgQ6ANMEpE+zkZ1UqqBu4wxvYGhwC/t+KcCXxljugNf2e/B2s7u9msK8Hzbh3xSbgfW1Xv/GPCUvV0HgRvs+TcAB40x3YCn7HLu6mngM2NML2AA1vZ59P4SkY7AbUCWMSYD8AeuxnP31wxgQqN5J7WPRCQOeAA4GxgCPFCXPDyKMcbrX8AwYE69978Dfud0XKexPe8D5wMbgGR7XjKwwZ7+JzCpXvmj5dztBXTC+ocbC3wECNYdnAGN9x0wBxhmTwfY5cTpbWhim6KArY1j8/T9BXQEdgBx9u//I+ACT95fQBqw+lT3ETAJ+Ge9+Q3KecrLJ2oEHPsDrpNvz/M4dvV6ILAYaGeM2Q1g/0yyi3nS9k4D7gFq7ffxwCFjTLX9vn7sR7fLXl5kl3c36cB+YLrd5PWyiITj4fvLGLMTeALYDuzG+v3n4Pn7q76T3Ucese9OxFcSgTQxz+OumxWRCGAWcIcx5nBLRZuY53bbKyIXA/uMMTn1ZzdR1LRimTsJAAYBzxtjBgIlHGtiaIpHbJfd5HEJ0AXoAIRjNZk05mn7qzWa2xav2EZfSQT5QEq9952AXQ7FckpEJBArCbxhjHnXnr1XRJLt5cnAPnu+p2zvucBEEckDZmI1D00DYkQkwC5TP/aj22UvjwYOtGXArZQP5BtjFtvv38FKDJ6+v84Dthpj9htjqoB3gXPw/P1V38nuI0/Zdy3ylUSwFOhuX90QhNXB9YHDMbWaiAjwCrDOGPNkvUUfAHVXKfwMq++gbv5P7SsdhgJFddVdd2KM+Z0xppMxJg1rn3xtjLkWmAtcbhdrvF1123u5Xd7tzr6MMXuAHSLS0541DliLh+8vrCahoSISZv9N1m2XR++vRk52H80BxotIrF1jGm/P8yxOd1K01Qu4CPge2Az83ul4TjL24VjVzVxghf26CKu99Stgo/0zzi4vWFdJbQZWYV3l4fh2nGAbRwMf2dPpwBJgE/A2EGzPD7Hfb7KXpzsddwvbkwlk2/tsNhDrDfsLeAhYD6wGXgeCPXV/AW9i9XVUYZ3Z33Aq+wj4ub2Nm4Drnd6uU3npEBNKKeXjfKVpSCmlVDM0ESillI/TRKCUUj5OE4FSSvk4TQRKKeXjNBEotyMiNSKyQkRWisgyETnnBOVjROTWVqx3noh47QPIT4WIzBCRy09cUnkzTQTKHZUZYzKNMQOwBgj86wnKxwAnTAROqXfXrVJuSROBcndRWEMbIyIRIvKVXUtYJSKX2GUeBbratYjH7bL32GVWisij9dZ3hYgsEZHvRWSEXdZfRB4XkaX2WPM32fOTRWS+vd7VdeXrE5E8EXnMXucSEelmz58hIk+KyFzgMXuc+9n2+heJSP962zTdjjVXRH5szx8vIt/Z2/q2Pc4UIvKoiKy1yz5hz7vCjm+liMw/wTaJiDxrr+Njjg2qpnyZ03e06UtfjV9ADdbd0+uxRqwcbM8PAKLs6QSsOzmF44cSvhD4Fgiz39fdHToP+Ls9fRHwpT09Bbjfng7GuiO4C3AX9l3oWGPvRzYRa169Mj/l2N3RM7CGafa33/8DeMCeHgussKcfA6bVW1+svW3zgXB73r3AH7GGf97AsWeNx9g/VwEdG81rbpsuA76wt6cDcAi43Ol9ri9nX1plVe6ozBiTCSAiw4DXRCQD66D/iIiMxBq2uiPQronPnwdMN8aUAhhj6g90VjdgXw5WAgFrfJj+9drKo7EeQLIUeNUe8G+2MWZFM/G+We/nU/Xmv22MqbGnhwM/tuP5WkTiRSTajvXqug8YYw7ao7L2ARZaQ/oQBHwHHAbKgZfts/mP7I8tBGaIyFv1tq+5bRoJvGnHtUtEvm5mm5QP0USg3Jox5jsRSQASsc7iE7FqCFX2qKUhTXxMaH4o4Ar7Zw3H/v4F+LUx5rjBwuyk8wPgdRF53BjzWlNhNjNd0iimpj7XVKwCfGGMmdREPEOwBnu7GvgVMNYYc7OInG3HuUJEMpvbJhG5qInvUz5O+wiUWxORXljNGIVYZ7X77CQwBuhsFysGIut97HPg5yISZq8j7gRfMwe4xT7zR0R6iEi4iHS2v+8lrNFfm3uW8FX1fn7XTJn5wLX2+kcDBcZ6psTnWAf0uu2NBRYB59brbwizY4oAoo0xnwB3YA1sh4h0NcYsNsb8EespYCnNbZMdx9V2H0IyMOYEvxvlA7RGoNxRqIjUNcMI8DNjTI2IvAF8KCLZHOtDwBhTKCILxXoI+afGmLvts+JsEakEPgHua+H7XsZqJlomVlvMfuBSrBFR7xaRKuAIVh9AU4JFZDHWidVxZ/G2B7GeWJYLlHJsqOM/A8/ZsdcADxlj3hWRycCbIhJsl7sfK+G9LyIh9u/lN/ayx0Wkuz3vK2Al1qinTW3Te1h9FKuwRuP9Xwu/F+UjdPRRpU6D3TyVZYwpcDoWpU6VNg0ppZSP0xqBUkr5OK0RKKWUj9NEoJRSPk4TgVJK+ThNBEop5eM0ESillI/7fy7Vmze/tRM5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train v validation loss\n",
    "learn_lm.recorder.plot_losses()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save our optimised model\n",
    "learn_lm.save('fine_tuned')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the first half of our model, the encoder - to use with our classifier\n",
    "learn_lm.save_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The encoder is the first part of the language model, this is the part concerned with understanding the text so far (as opposed to the last part which aims to predict the next word). \n",
    "\n",
    "This is why we save the encoder, as it could most likely be useful for our classifier.\n",
    "\n",
    "## Create classifier model\n",
    "\n",
    "We will now create our message clasifier, making use of the encoder from our language model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set batch size\n",
    "bs = 24"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define precison for our purposes\n",
    "def precision(log_preds, targs, thresh=0.5, epsilon=1e-8):\n",
    "    pred_pos = (log_preds > thresh).float()\n",
    "    tpos = torch.mul((targs == pred_pos).float(), targs.float())\n",
    "    return (tpos.sum()/(pred_pos.sum() + epsilon))#.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define recall for our purposes\n",
    "def recall(log_preds, targs, thresh=0.5, epsilon=1e-8):\n",
    "    pred_pos = (log_preds > thresh).float()\n",
    "    tpos = torch.mul((targs == pred_pos).float(), targs.float())\n",
    "    return (tpos.sum()/(targs.sum() + epsilon))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split data into train and validation\n",
    "X_train, X_test, y_train, y_test = train_test_split(df['message'], df['tags'], random_state=49)\n",
    "df_trn = pd.concat([X_train, y_train], axis=1)\n",
    "df_val = pd.concat([X_test, y_test], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 19661 entries, 18830 to 426\n",
      "Data columns (total 2 columns):\n",
      "message    19661 non-null object\n",
      "tags       15088 non-null object\n",
      "dtypes: object(2)\n",
      "memory usage: 460.8+ KB\n"
     ]
    }
   ],
   "source": [
    "# Check for missing values\n",
    "df_trn.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 6554 entries, 7603 to 21079\n",
      "Data columns (total 2 columns):\n",
      "message    6554 non-null object\n",
      "tags       5005 non-null object\n",
      "dtypes: object(2)\n",
      "memory usage: 153.6+ KB\n"
     ]
    }
   ],
   "source": [
    "# Check for missing values\n",
    "df_val.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop missing values from train and validation sets\n",
    "df_trn = df_trn.dropna()\n",
    "df_val = df_val.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create data object for our classifier with specific train & validation sets\n",
    "data_clas = TextClasDataBunch.from_df(path, train_df=df_trn, valid_df=df_val, \n",
    "                                  vocab=data_lm.vocab, # Use same vocab from our language model\n",
    "                                  text_cols='message', \n",
    "                                  label_cols='tags',\n",
    "                                  label_delim=';',\n",
    "                                  bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save our data object\n",
    "data_clas.save('tmp_clas')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RNNLearner(data=TextClasDataBunch;\n",
       "\n",
       "Train: LabelList (15088 items)\n",
       "x: TextList\n",
       "xxbos xxmaj photos on xxmaj china xxmaj news xxmaj service showed people in xxmaj fujian 's xxmaj xxunk county wearing xxunk and xxunk deep in water through flooded streets .,xxbos xxup ocha says a series of short - term measures need to be taken to improve conditions for the farmers , including the repair and maintenance of xxunk plants , the installation of manual pumps and increased capacity for stocking and treating water .,xxbos xxmaj seasonal downpours cause dozens of landslides and flash floods each year in xxmaj indonesia , where millions of people live in mountainous regions and near fertile flood plains close to rivers .,xxbos xxmaj in xxmaj xxunk , about 8 kms from xxmaj xxunk , at least 20 people drowned including a xxunk as a rescue boat capsized due to uncontrolled crowd and over loading .,xxbos xxmaj authorities have also rushed 180 medical teams comprising more than 1,000 doctors amid fears that crowded and unsanitary conditions could lead to outbreaks of cholera and diarrhoea .\n",
       "y: MultiCategoryList\n",
       "related;aid_related;search_and_rescue;weather_related;floods,related;aid_related;water,related;weather_related;floods;storm;earthquake,related;aid_related;medical_help;missing_people;death;transport;weather_related;floods,related;aid_related;medical_help\n",
       "Path: data;\n",
       "\n",
       "Valid: LabelList (5005 items)\n",
       "x: TextList\n",
       "xxbos we are xxunk by mouskitous , what can we do in that situation .,xxbos xxmaj niger state radio said that the xxmaj moroccan xxunk came to xxmaj niger to express his full solidarity with the country where three million people are suffering from hunger .,xxbos xxmaj xxunk have xxunk attacked xxunk in all three countries after being taken in by a variety of conspiracy xxunk , often xxunk the outbreak as a plot by the xxmaj west to murder xxmaj africans and harvest their organs .,xxbos xxmaj sandy is getting me xxunk shall be a special # xxmaj sandy show . xxmaj who 's xxunk .. the power is out # sandy,xxbos xxmaj only needs toddler clothes and food . xxmaj did n't fill out xxunk\n",
       "y: MultiCategoryList\n",
       "related;request;aid_related;other_aid,related;aid_related;food,related;aid_related;military,related;weather_related;storm;direct_report,related;request;aid_related;clothing\n",
       "Path: data;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): MultiBatchEncoder(\n",
       "    (module): AWD_LSTM(\n",
       "      (encoder): Embedding(11536, 400, padding_idx=1)\n",
       "      (encoder_dp): EmbeddingDropout(\n",
       "        (emb): Embedding(11536, 400, padding_idx=1)\n",
       "      )\n",
       "      (rnns): ModuleList(\n",
       "        (0): WeightDropout(\n",
       "          (module): LSTM(400, 1152, batch_first=True)\n",
       "        )\n",
       "        (1): WeightDropout(\n",
       "          (module): LSTM(1152, 1152, batch_first=True)\n",
       "        )\n",
       "        (2): WeightDropout(\n",
       "          (module): LSTM(1152, 400, batch_first=True)\n",
       "        )\n",
       "      )\n",
       "      (input_dp): RNNDropout()\n",
       "      (hidden_dps): ModuleList(\n",
       "        (0): RNNDropout()\n",
       "        (1): RNNDropout()\n",
       "        (2): RNNDropout()\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (1): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.2, inplace=False)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace=True)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1, inplace=False)\n",
       "      (6): Linear(in_features=50, out_features=35, bias=True)\n",
       "    )\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[<function accuracy_thresh at 0x7f1c5d7dbb90>, <function precision at 0x7f1c59623200>, <function recall at 0x7f1c59623320>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
       "learn: ...\n",
       "alpha: 2.0\n",
       "beta: 1.0], layer_groups=[Sequential(\n",
       "  (0): Embedding(11536, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(11536, 400, padding_idx=1)\n",
       "  )\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.2, inplace=False)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace=True)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1, inplace=False)\n",
       "      (6): Linear(in_features=50, out_features=35, bias=True)\n",
       "    )\n",
       "  )\n",
       ")], add_time=True, silent=False)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create our classifier model\n",
    "learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)\n",
    "# Define metrics for training\n",
    "learn.metrics = [accuracy_thresh, precision, recall]\n",
    "# Load in the encoder of our language model\n",
    "learn.load_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy_thresh</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.201225</td>\n",
       "      <td>0.180335</td>\n",
       "      <td>0.934117</td>\n",
       "      <td>0.888722</td>\n",
       "      <td>0.492152</td>\n",
       "      <td>00:24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Lets freeze the model so we start training the classifier on only the last layers\n",
    "learn.freeze()\n",
    "# Train the model\n",
    "learn.fit_one_cycle(1, 3e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model at this point\n",
    "learn.save('first_factors')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RNNLearner(data=TextClasDataBunch;\n",
       "\n",
       "Train: LabelList (15088 items)\n",
       "x: TextList\n",
       "xxbos xxmaj photos on xxmaj china xxmaj news xxmaj service showed people in xxmaj fujian 's xxmaj xxunk county wearing xxunk and xxunk deep in water through flooded streets .,xxbos xxup ocha says a series of short - term measures need to be taken to improve conditions for the farmers , including the repair and maintenance of xxunk plants , the installation of manual pumps and increased capacity for stocking and treating water .,xxbos xxmaj seasonal downpours cause dozens of landslides and flash floods each year in xxmaj indonesia , where millions of people live in mountainous regions and near fertile flood plains close to rivers .,xxbos xxmaj in xxmaj xxunk , about 8 kms from xxmaj xxunk , at least 20 people drowned including a xxunk as a rescue boat capsized due to uncontrolled crowd and over loading .,xxbos xxmaj authorities have also rushed 180 medical teams comprising more than 1,000 doctors amid fears that crowded and unsanitary conditions could lead to outbreaks of cholera and diarrhoea .\n",
       "y: MultiCategoryList\n",
       "related;aid_related;search_and_rescue;weather_related;floods,related;aid_related;water,related;weather_related;floods;storm;earthquake,related;aid_related;medical_help;missing_people;death;transport;weather_related;floods,related;aid_related;medical_help\n",
       "Path: data;\n",
       "\n",
       "Valid: LabelList (5005 items)\n",
       "x: TextList\n",
       "xxbos we are xxunk by mouskitous , what can we do in that situation .,xxbos xxmaj niger state radio said that the xxmaj moroccan xxunk came to xxmaj niger to express his full solidarity with the country where three million people are suffering from hunger .,xxbos xxmaj xxunk have xxunk attacked xxunk in all three countries after being taken in by a variety of conspiracy xxunk , often xxunk the outbreak as a plot by the xxmaj west to murder xxmaj africans and harvest their organs .,xxbos xxmaj sandy is getting me xxunk shall be a special # xxmaj sandy show . xxmaj who 's xxunk .. the power is out # sandy,xxbos xxmaj only needs toddler clothes and food . xxmaj did n't fill out xxunk\n",
       "y: MultiCategoryList\n",
       "related;request;aid_related;other_aid,related;aid_related;food,related;aid_related;military,related;weather_related;storm;direct_report,related;request;aid_related;clothing\n",
       "Path: data;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): MultiBatchEncoder(\n",
       "    (module): AWD_LSTM(\n",
       "      (encoder): Embedding(11536, 400, padding_idx=1)\n",
       "      (encoder_dp): EmbeddingDropout(\n",
       "        (emb): Embedding(11536, 400, padding_idx=1)\n",
       "      )\n",
       "      (rnns): ModuleList(\n",
       "        (0): WeightDropout(\n",
       "          (module): LSTM(400, 1152, batch_first=True)\n",
       "        )\n",
       "        (1): WeightDropout(\n",
       "          (module): LSTM(1152, 1152, batch_first=True)\n",
       "        )\n",
       "        (2): WeightDropout(\n",
       "          (module): LSTM(1152, 400, batch_first=True)\n",
       "        )\n",
       "      )\n",
       "      (input_dp): RNNDropout()\n",
       "      (hidden_dps): ModuleList(\n",
       "        (0): RNNDropout()\n",
       "        (1): RNNDropout()\n",
       "        (2): RNNDropout()\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (1): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.2, inplace=False)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace=True)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1, inplace=False)\n",
       "      (6): Linear(in_features=50, out_features=35, bias=True)\n",
       "    )\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[<function accuracy_thresh at 0x7f1c5d7dbb90>, <function precision at 0x7f1c59623200>, <function recall at 0x7f1c59623320>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
       "learn: ...\n",
       "alpha: 2.0\n",
       "beta: 1.0], layer_groups=[Sequential(\n",
       "  (0): Embedding(11536, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(11536, 400, padding_idx=1)\n",
       "  )\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.2, inplace=False)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace=True)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1, inplace=False)\n",
       "      (6): Linear(in_features=50, out_features=35, bias=True)\n",
       "    )\n",
       "  )\n",
       ")], add_time=True, silent=False)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load model at this point\n",
    "learn.load('first_factors')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy_thresh</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.186708</td>\n",
       "      <td>0.170866</td>\n",
       "      <td>0.939718</td>\n",
       "      <td>0.879542</td>\n",
       "      <td>0.563686</td>\n",
       "      <td>00:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.175495</td>\n",
       "      <td>0.161384</td>\n",
       "      <td>0.943126</td>\n",
       "      <td>0.891711</td>\n",
       "      <td>0.582238</td>\n",
       "      <td>00:40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Unfreeze last 2 layers, not whole model (works better than total unfreeze for text models)\n",
    "learn.freeze_to(-2)\n",
    "# Train some more\n",
    "learn.fit_one_cycle(2, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7), wd=0.1)\n",
    "# Note 2.6**4! This seems to work well from text classification RNN models from experimentation by JH\n",
    "# Related: Some have used random forrest models to find most optimal hyper params like lr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model at this point\n",
    "learn.save('second_factors')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RNNLearner(data=TextClasDataBunch;\n",
       "\n",
       "Train: LabelList (15088 items)\n",
       "x: TextList\n",
       "xxbos xxmaj photos on xxmaj china xxmaj news xxmaj service showed people in xxmaj fujian 's xxmaj xxunk county wearing xxunk and xxunk deep in water through flooded streets .,xxbos xxup ocha says a series of short - term measures need to be taken to improve conditions for the farmers , including the repair and maintenance of xxunk plants , the installation of manual pumps and increased capacity for stocking and treating water .,xxbos xxmaj seasonal downpours cause dozens of landslides and flash floods each year in xxmaj indonesia , where millions of people live in mountainous regions and near fertile flood plains close to rivers .,xxbos xxmaj in xxmaj xxunk , about 8 kms from xxmaj xxunk , at least 20 people drowned including a xxunk as a rescue boat capsized due to uncontrolled crowd and over loading .,xxbos xxmaj authorities have also rushed 180 medical teams comprising more than 1,000 doctors amid fears that crowded and unsanitary conditions could lead to outbreaks of cholera and diarrhoea .\n",
       "y: MultiCategoryList\n",
       "related;aid_related;search_and_rescue;weather_related;floods,related;aid_related;water,related;weather_related;floods;storm;earthquake,related;aid_related;medical_help;missing_people;death;transport;weather_related;floods,related;aid_related;medical_help\n",
       "Path: data;\n",
       "\n",
       "Valid: LabelList (5005 items)\n",
       "x: TextList\n",
       "xxbos we are xxunk by mouskitous , what can we do in that situation .,xxbos xxmaj niger state radio said that the xxmaj moroccan xxunk came to xxmaj niger to express his full solidarity with the country where three million people are suffering from hunger .,xxbos xxmaj xxunk have xxunk attacked xxunk in all three countries after being taken in by a variety of conspiracy xxunk , often xxunk the outbreak as a plot by the xxmaj west to murder xxmaj africans and harvest their organs .,xxbos xxmaj sandy is getting me xxunk shall be a special # xxmaj sandy show . xxmaj who 's xxunk .. the power is out # sandy,xxbos xxmaj only needs toddler clothes and food . xxmaj did n't fill out xxunk\n",
       "y: MultiCategoryList\n",
       "related;request;aid_related;other_aid,related;aid_related;food,related;aid_related;military,related;weather_related;storm;direct_report,related;request;aid_related;clothing\n",
       "Path: data;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): MultiBatchEncoder(\n",
       "    (module): AWD_LSTM(\n",
       "      (encoder): Embedding(11536, 400, padding_idx=1)\n",
       "      (encoder_dp): EmbeddingDropout(\n",
       "        (emb): Embedding(11536, 400, padding_idx=1)\n",
       "      )\n",
       "      (rnns): ModuleList(\n",
       "        (0): WeightDropout(\n",
       "          (module): LSTM(400, 1152, batch_first=True)\n",
       "        )\n",
       "        (1): WeightDropout(\n",
       "          (module): LSTM(1152, 1152, batch_first=True)\n",
       "        )\n",
       "        (2): WeightDropout(\n",
       "          (module): LSTM(1152, 400, batch_first=True)\n",
       "        )\n",
       "      )\n",
       "      (input_dp): RNNDropout()\n",
       "      (hidden_dps): ModuleList(\n",
       "        (0): RNNDropout()\n",
       "        (1): RNNDropout()\n",
       "        (2): RNNDropout()\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (1): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.2, inplace=False)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace=True)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1, inplace=False)\n",
       "      (6): Linear(in_features=50, out_features=35, bias=True)\n",
       "    )\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[<function accuracy_thresh at 0x7f1c5d7dbb90>, <function precision at 0x7f1c59623200>, <function recall at 0x7f1c59623320>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
       "learn: ...\n",
       "alpha: 2.0\n",
       "beta: 1.0], layer_groups=[Sequential(\n",
       "  (0): Embedding(11536, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(11536, 400, padding_idx=1)\n",
       "  )\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.2, inplace=False)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace=True)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1, inplace=False)\n",
       "      (6): Linear(in_features=50, out_features=35, bias=True)\n",
       "    )\n",
       "  )\n",
       ")], add_time=True, silent=False)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load model at this point\n",
    "learn.load('second_factors')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy_thresh</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.162541</td>\n",
       "      <td>0.156966</td>\n",
       "      <td>0.945878</td>\n",
       "      <td>0.874520</td>\n",
       "      <td>0.632789</td>\n",
       "      <td>01:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.164536</td>\n",
       "      <td>0.155622</td>\n",
       "      <td>0.946152</td>\n",
       "      <td>0.881230</td>\n",
       "      <td>0.622061</td>\n",
       "      <td>01:11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Unfreeze whole model\n",
    "learn.unfreeze()\n",
    "# Train some more\n",
    "learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7), wd=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict on the validation dataset\n",
    "y_pred, y_true =  learn.get_preds()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7499250374812594"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get overall F1 score for predicting all classes\n",
    "f1_score(y_true, y_pred>0.35, average='micro')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So in my earlier study our best overall F1 score was 0.41, 0.75 is a significant improvement!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get numeric values\n",
    "y_true = y_true.numpy()\n",
    "scores = y_pred.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((5005, 35), (5005, 35))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check dimensions\n",
    "scores.shape, y_true.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                        precision    recall  f1-score   support\n",
      "\n",
      "           aid_centers       0.20      0.04      0.07        76\n",
      "           aid_related       0.69      0.90      0.78      2704\n",
      "             buildings       0.68      0.58      0.62       327\n",
      "              clothing       0.49      0.44      0.47        79\n",
      "                  cold       0.73      0.40      0.52       148\n",
      "                 death       0.76      0.52      0.62       304\n",
      "         direct_report       0.64      0.74      0.68      1251\n",
      "            earthquake       0.84      0.82      0.83       569\n",
      "           electricity       0.62      0.50      0.55       131\n",
      "                  fire       0.67      0.06      0.12        63\n",
      "                floods       0.71      0.63      0.67       550\n",
      "                  food       0.78      0.86      0.82       711\n",
      "             hospitals       0.50      0.07      0.13        67\n",
      "infrastructure_related       0.43      0.18      0.26       428\n",
      "          medical_help       0.59      0.44      0.51       508\n",
      "      medical_products       0.58      0.42      0.49       315\n",
      "              military       0.63      0.59      0.61       216\n",
      "        missing_people       0.12      0.01      0.03        69\n",
      "                 money       0.55      0.35      0.42       150\n",
      "                 offer       0.00      0.00      0.00        24\n",
      "             other_aid       0.48      0.40      0.43       865\n",
      "  other_infrastructure       0.39      0.14      0.20       281\n",
      "         other_weather       0.44      0.06      0.11       342\n",
      "              refugees       0.53      0.29      0.38       229\n",
      "               related       1.00      1.00      1.00      5005\n",
      "               request       0.70      0.79      0.74      1095\n",
      "     search_and_rescue       0.67      0.07      0.12       175\n",
      "              security       0.00      0.00      0.00       109\n",
      "               shelter       0.70      0.69      0.69       580\n",
      "                 shops       0.00      0.00      0.00        40\n",
      "                 storm       0.67      0.71      0.69       601\n",
      "                 tools       0.00      0.00      0.00        26\n",
      "             transport       0.65      0.30      0.41       306\n",
      "                 water       0.78      0.76      0.77       408\n",
      "       weather_related       0.83      0.83      0.83      1805\n",
      "\n",
      "             micro avg       0.77      0.73      0.75     20557\n",
      "             macro avg       0.54      0.42      0.45     20557\n",
      "          weighted avg       0.74      0.73      0.72     20557\n",
      "           samples avg       0.79      0.82      0.75     20557\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "# Print metrics for different classes predicted\n",
    "metrics = classification_report(y_true, scores>0.35, target_names=data_clas.valid_ds.classes)\n",
    "print(metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the earlier study we looked at the top 12 predicted classes for our best model (Gradient Boosting), lets compare the results for these with our new deep learning model.\n",
    "\n",
    "\n",
    "| Category | Gradient Boosting - F1 | AWD_LSTM - F1|\n",
    "| --- | --- | --- |\n",
    "|earthquake | 0.79 | 0.83 |\n",
    "|food|0.76|0.82|\n",
    "|weather_related|0.71|0.83|\n",
    "|water|0.68|0.77|\n",
    "|floods|0.66|0.67|\n",
    "|shelter|0.64|0.69|\n",
    "|aid_related|0.64|0.78|\n",
    "|storm|0.61|0.69|\n",
    "|clothing|0.60|0.47|\n",
    "|cold|0.59|0.52|\n",
    "|request|0.55|0.74|\n",
    "|death|0.54|0.62|\n",
    "|direct_report|0.51|0.68|\n",
    "\n",
    "We can see the AWD_LSTM has a better score in most of these categories, as well as better scores overall on average.\n",
    "\n",
    "## Conclusion \n",
    "\n",
    "The deep learning model classifier AWD_LSTM seems to have made better predictions overall than the best of our previous model."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}