nbs/dl1/lesson3-imdb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IMDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.set_device(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's download the dataset we are going to study. The [dataset](http://ai.stanford.edu/~amaas/data/sentiment/) has been curated by Andrew Maas et al. and contains a total of 100,000 reviews on IMDB. 25,000 of them are labelled as positive and negative for training, another 25,000 are labelled for testing (in both cases they are highly polarized). The remaning 50,000 is an additional unlabelled data (but we will find a use for it nonetheless).\n",
    "\n",
    "We'll begin with a sample we've prepared for you, so that things run quickly before going over the full dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jhoward/.fastai/data/imdb_sample/texts.csv'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb_sample/models'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb_sample/export.pkl')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It only contains one csv file, let's have a look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>negative</td>\n",
       "      <td>Un-bleeping-believable! Meg Ryan doesn't even ...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>positive</td>\n",
       "      <td>This is a extremely well-made film. The acting...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>negative</td>\n",
       "      <td>Every once in a long while a movie will come a...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>positive</td>\n",
       "      <td>Name just says it all. I watched this movie wi...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>negative</td>\n",
       "      <td>This movie succeeds at being one of the most u...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      label                                               text  is_valid\n",
       "0  negative  Un-bleeping-believable! Meg Ryan doesn't even ...     False\n",
       "1  positive  This is a extremely well-made film. The acting...     False\n",
       "2  negative  Every once in a long while a movie will come a...     False\n",
       "3  positive  Name just says it all. I watched this movie wi...     False\n",
       "4  negative  This movie succeeds at being one of the most u...     False"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(path/'texts.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is a extremely well-made film. The acting, script and camera-work are all first-rate. The music is good, too, though it is mostly early in the film, when things are still relatively cheery. There are no really superstars in the cast, though several faces will be familiar. The entire cast does an excellent job with the script.<br /><br />But it is hard to watch, because there is no good end to a situation like the one presented. It is now fashionable to blame the British for setting Hindus and Muslims against each other, and then cruelly separating them into two countries. There is some merit in this view, but it\\'s also true that no one forced Hindus and Muslims in the region to mistreat each other as they did around the time of partition. It seems more likely that the British simply saw the tensions between the religions and were clever enough to exploit them to their own ends.<br /><br />The result is that there is much cruelty and inhumanity in the situation and this is very unpleasant to remember and to see on the screen. But it is never painted as a black-and-white case. There is baseness and nobility on both sides, and also the hope for change in the younger generation.<br /><br />There is redemption of a sort, in the end, when Puro has to make a hard choice between a man who has ruined her life, but also truly loved her, and her family which has disowned her, then later come looking for her. But by that point, she has no option that is without great pain for her.<br /><br />This film carries the message that both Muslims and Hindus have their grave faults, and also that both can be dignified and caring people. The reality of partition makes that realisation all the more wrenching, since there can never be real reconciliation across the India/Pakistan border. In that sense, it is similar to \"Mr & Mrs Iyer\".<br /><br />In the end, we were glad to have seen the film, even though the resolution was heartbreaking. If the UK and US could deal with their own histories of racism with this kind of frankness, they would certainly be better off.'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It contains one line per review, with the label ('negative' or 'positive'), the text and a flag to determine if it should be part of the validation set or the training set. If we ignore this flag, we can create a DataBunch containing this data in one line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = TextDataBunch.from_csv(path, 'texts.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By executing this line a process was launched that took a bit of time. Let's dig a bit into it. Images could be fed (almost) directly into a model because they're just a big array of pixel values that are floats between 0 and 1. A text is composed of words, and we can't apply mathematical functions to them directly. We first have to convert them to numbers. This is done in two differents steps: tokenization and numericalization. A `TextDataBunch` does all of that behind the scenes for you.\n",
    "\n",
    "Before we delve into the explanations, let's take the time to save the things that were calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next time we launch this notebook, we can skip the cell above that took a bit of time (and that will take a lot more when you get to the full dataset) and load those results like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = TextDataBunch.load(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step of processing we make texts go through is to split the raw sentences into words, or more exactly tokens. The easiest way to do this would be to split the string on spaces, but we can be smarter:\n",
    "\n",
    "- we need to take care of punctuation\n",
    "- some words are contractions of two different words, like isn't or don't\n",
    "- we may need to clean some parts of our texts, if there's HTML code for instance\n",
    "\n",
    "To see what the tokenizer had done behind the scenes, let's have a look at a few texts in a batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='90%'>  <col width='10%'>  <tr>\n",
       "    <th>text</th>\n",
       "    <th>target</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxfld 1 xxmaj raising xxmaj victor xxmaj vargas : a xxmaj review \\n\\n xxmaj you know , xxmaj raising xxmaj victor xxmaj vargas is like sticking your hands into a big , xxunk bowl of xxunk . xxmaj it 's warm and gooey , but you 're not sure if it feels right . xxmaj try as i might , no matter how warm and gooey xxmaj raising xxmaj</th>\n",
       "    <th>negative</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxfld 1 xxmaj many xxunk that this is n't just a classic due to the fact that it 's the first xxup 3d game , or even the first xxunk - up . xxmaj it 's also one of the first xxunk games , one of the xxunk definitely the first ) truly claustrophobic games , and just a pretty well - xxunk gaming experience in general . xxmaj</th>\n",
       "    <th>positive</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxfld 1 i really wanted to love this show . i truly , honestly did . \\n\\n xxmaj for the first time , gay viewers get their own version of the \" xxmaj the xxmaj bachelor \" . xxmaj with the help of his obligatory \" hag \" xxmaj xxunk , xxmaj james , a good looking , well - to - do thirty - something has the chance</th>\n",
       "    <th>negative</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxfld 1 \\n\\n i 'm sure things did n't exactly go the same way in the real life of xxmaj homer xxmaj hickam as they did in the film adaptation of his book , xxmaj rocket xxmaj boys , but the movie \" xxmaj october xxmaj sky \" ( an xxunk of the book 's title ) is good enough to stand alone . i have not read xxmaj</th>\n",
       "    <th>positive</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxfld 1 xxmaj to review this movie , i without any doubt would have to quote that memorable scene in xxmaj tarantino 's \" xxmaj pulp xxmaj fiction \" ( xxunk ) when xxmaj jules and xxmaj vincent are talking about xxmaj mia xxmaj wallace and what she does for a living . xxmaj jules tells xxmaj vincent that the \" xxmaj only thing she did worthwhile was pilot</th>\n",
       "    <th>negative</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = TextClasDataBunch.load(path)\n",
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The texts are truncated at 100 tokens for more readability. We can see that it did more than just split on space and punctuation symbols: \n",
    "- the \"'s\" are grouped together in one token\n",
    "- the contractions are separated like his: \"did\", \"n't\"\n",
    "- content has been cleaned for any HTML symbol and lower cased\n",
    "- there are several special tokens (all those that begin by xx), to replace unkown tokens (see below) or to introduce different text fields (here we only have one)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numericalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have extracted tokens from our texts, we convert to integers by creating a list of all the words used. We only keep the ones that appear at list twice with a maximum vocabulary size of 60,000 (by default) and replace the ones that don't make the cut by the unknown token `UNK`.\n",
    "\n",
    "The correspondance from ids tokens is stored in the `vocab` attribute of our datasets, in a dictionary called `itos` (for int to string)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxfld',\n",
       " 'xxmaj',\n",
       " 'xxup',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " 'the',\n",
       " ',']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.vocab.itos[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if we look at what a what's in our datasets, we'll see the tokenized text as a representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text xxbos xxfld 1 xxmaj matthew mcconaughey is a mysterious man waiting for xxmaj agent xxmaj wesley xxmaj doyle ( xxmaj powers xxmaj boothe ) in his xxup fbi office . xxmaj he claims to have information about a serial killer chased by xxup fbi . xxmaj when xxmaj agent xxmaj doyle arrives in the office , he tells him that the serial killer is indeed his dead brother . xxmaj agent xxmaj doyle xxunk some evidence , and the man tells the story of his life , since his childhood . xxmaj they were a simple family of three : his widow father xxmaj xxunk ( xxmaj bill xxmaj paxton ) , his brother and himself . xxmaj one night , his father xxunk the two brothers and tells them that an angel of xxmaj god had just visited him and assigned his family to destroy demons . xxmaj what happens next is one of the most scary movie i have ever seen . \n",
       "\n",
       " i watched this movie four months ago on xxup vhs , and yesterday i watched again , now on xxup dvd . xxmaj although being a low - budget movie , the screenplay is sharp , with no flaw . xxmaj the cast is outstanding , but i would like to highlight the performance of xxmaj matt xxunk as the young xxmaj xxunk . xxmaj it is a very difficult and complex role to be performed by a young teenager . xxmaj the direction of xxmaj bill xxmaj paxton is remarkable . xxmaj there is no explicit violence in this horror movie . a great debut behind the camera . i regret the xxmaj xxunk title of this movie : ' a xxmaj xxunk do xxmaj xxunk ' ( xxmaj the xxmaj devil 's xxmaj hand ' ) . xxmaj if at least it were ' xxmaj the xxmaj god 's xxmaj hand ' , it might be acceptable . xxmaj but calling this movie as ' the devil 's hand ' is indeed ridiculous . xxmaj xxunk xxmaj xxunk , the screenwriter , did not deserve such a lack of respect from the xxmaj xxunk xxunk . xxmaj this film is highly recommended . xxmaj my vote is xxunk . \n",
       "\n",
       " xxmaj title ( xxmaj xxunk ) : \" a xxmaj xxunk do xxmaj xxunk \" ( \" xxmaj the xxmaj devil 's xxmaj hand \" )"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.train_ds[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the underlying data is all numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,    3,   45,    4, 3229, 3805,   15,   12, 1232,  132])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.train_ds[0][0].data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### With the data block API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the data block API with NLP and have a lot more flexibility than what the default factory methods offer. In the previous example for instance, the data was randomly split between train and validation instead of reading the third column of the csv.\n",
    "\n",
    "With the data block API though, we have to manually call the tokenize and numericalize steps. This allows more flexibility, and if you're not using the defaults from fastai, the variaous arguments to pass will appear in the step they're revelant, so it'll be more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = (TextList.from_csv(path, 'texts.csv', cols='text')\n",
    "                .split_from_df(col=2)\n",
    "                .label_from_df(cols=0)\n",
    "                .databunch())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that language models can use a lot of GPU, so you may need to decrease batchsize here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs=48"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's grab the full dataset for what follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jhoward/.fastai/data/imdb/imdb.vocab'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/models'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/tmp_lm'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/train'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/test'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/README'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/tmp_clas')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB)\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jhoward/.fastai/data/imdb/train/pos'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/train/unsup'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/train/unsupBow.feat'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/train/labeledBow.feat'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/train/neg')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(path/'train').ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reviews are in a training and test set following an imagenet structure. The only difference is that there is an `unsup` folder in `train` that contains the unlabelled data.\n",
    "\n",
    "We're not going to train a model that classifies the reviews from scratch. Like in computer vision, we'll use a model pretrained on a bigger dataset (a cleaned subset of wikipeia called [wikitext-103](https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset)). That model has been trained to guess what the next word, its input being all the previous words. It has a recurrent structure and a hidden state that is updated each time it sees a new word. This hidden state thus contains information about the sentence up to that point.\n",
    "\n",
    "We are going to use that 'knowledge' of the English language to build our classifier, but first, like for computer vision, we need to fine-tune the pretrained model to our particular dataset. Because the English of the reviex lefts by people on IMDB isn't the same as the English of wikipedia, we'll need to adjust a little bit the parameters of our model. Plus there might be some words extremely common in that dataset that were barely present in wikipedia, and therefore might no be part of the vocabulary the model was trained on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is where the unlabelled data is going to be useful to us, as we can use it to fine-tune our model. Let's create our data object with the data block API (next line takes a few minutes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = (TextList.from_folder(path)\n",
    "           #Inputs: all the text files in path\n",
    "            .filter_by_folder(include=['train', 'test']) \n",
    "           #We may have other temp folders that contain text files so we only keep what's in train and test\n",
    "            .random_split_by_pct(0.1)\n",
    "           #We randomly split and keep 10% (10,000 reviews) for validation\n",
    "            .label_for_lm()           \n",
    "           #We want to do a language model so we label accordingly\n",
    "            .databunch(bs=bs))\n",
    "data_lm.save('tmp_lm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to use a special kind of `TextDataBunch` for the language model, that ignores the labels (that's why we put 0 everywhere), will shuffle the texts at each epoch before concatenating them all together (only for training, we don't shuffle for the validation set) and will send batches that read that text in order with targets that are the next word in the sentence.\n",
    "\n",
    "The line before being a bit long, we want to load quickly the final ids by using the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = TextLMDataBunch.load(path, 'tmp_lm', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='5%'>  <col width='95%'>  <tr>\n",
       "    <th>idx</th>\n",
       "    <th>text</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>xxbos xxmaj in a xxmaj woman xxmaj under the xxmaj influence xxmaj mabel goes crazy , but i can see why she does go crazy . xxmaj if i lived the kind of life she lived with the family she has i would go crazy too . xxmaj everyone in her family is off their rocker and not completely with it . xxmaj she is constantly surrounded by people yelling</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>, fresh from success as xxmaj elliot 's mom in \" xxup e.t - xxmaj the xxmaj extra xxmaj terrestrial \" ) is a mother whose marriage to husband xxmaj vic ( xxmaj daniel xxmaj hugh - xxmaj kelly ) is hanging by a thread . xxmaj she 's been having an affair with a local worker , and is now dwelling on whether or not to leave her husband</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>. xxbos xxmaj the reason why this movie sucks , have these people even read a bible ? xxmaj everything in the movie was about moses , xxmaj god was staying out of it . xxup that xxup didn't xxup happen ! xxmaj god directed everything , he told them where to go and what to do . xxmaj also the people wandered for 40 years xxup after they arrived</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>to sleep . xxbos a little bit of a let down . xxmaj personal opinion , this was a movie with much more potential that was never allowed to reach that . xxmaj sean xxmaj patrick xxmaj flanery has good moments in the movie , but as a whole his performance does n't come close to some of his other movies like xxmaj powder and xxmaj boondock xxmaj saints .</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>purity that it will remind you of the work of some of the great masters of the past . xxmaj the regard for its audience is something that we 're not used to anymore . i do n't know if we ever were . xxmaj riveting , moving , without concessions and xxmaj benicio del xxmaj toro is just extraordinary . xxmaj we can see his soul , we can</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then put this in a learner object very easily with a model loaded with the pretrained weights. They'll be downloaded the first time you'll execute the following line and stored in `~/.fastai/models/` (or elsewhere if you specified different paths in your config file)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_lm, pretrained_model=URLs.WT103_1, drop_mult=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ4AAAEKCAYAAAAiizNaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl4VdXVwOHfysyQCRIwJJAAMipzFBRFnHBAEaROFUWr0sFa69BaP621VqtVa61atVZFcZ6r4gBURUVBCDMyEyCEKQkEyEDm9f1xT+ASbiCQO2e9z3Mf7t1nn3PW5kJW9jn77C2qijHGGOMvEYEOwBhjTMtiiccYY4xfWeIxxhjjV5Z4jDHG+JUlHmOMMX5liccYY4xfWeIxxhjjV5Z4jDHG+JUlHmOMMX4VFegA/CElJUWzsrICHYYxxoSU+fPnF6lqqreP2yIST1ZWFjk5OYEOwxhjQoqIbPTFce1SmzHGGL+yxGOMMcavLPEYY4zxK0s8xhhj/MoSjzHGGL+yxGOMMcavLPEYY4zxK0s8xhgThlZtK+GxGaspLKkMdCgHscRjjDFhaEn+Lp74Yg3lVTWBDuUglniMMSYMFZVWAZDSNjbAkRzMEo8xxoShwpJKWsdE0iY2+GZGs8RjjDFhqKi0Mih7O2CJxxhjwlJhSSWp8ZZ4jDHG+ImrxxMT6DA8ssRjjDFhqKjUejzGGGP8pLq2juLyarvHY4wxxj92OEOpW2SPR0Q2iMhSEVkkIgctASouT4jIWhFZIiKDnfKBIjJbRH50yi9z2+clEVnvHHORiAz0ZRuMMSbU1M9WEKw9Hn8M8D5dVYsa2XYe0MN5DQWecf4sB65W1TUi0gmYLyLTVHWXs9/vVPVdXwdujDGhqKjUlXiCtccT6CeLLgKmqKoCc0QkSUTSVHV1fQVV3SIiBUAqsKuxAxljjHEprE88Qdrj8fU9HgWmi8h8EZnkYXs6sMntc75Tto+InAjEAOvcih9wLsH9Q0SC82/WGGMCJNgvtfk68QxX1cG4LqndKCIjGmwXD/vovo0iacArwLWqWucU3wn0Bk4A2gF3eDqxiEwSkRwRySksLGxmM4wxJnQUlVbSNjaKVjGRgQ7FI58mHlXd4vxZAHwAnNigSj7Q2e1zBrAFQEQSgE+Au1V1jtsxt6pLJTDZwzHr6z2nqtmqmp2amuqtJhljTNArLAneh0fBh4lHRNqISHz9e2AUsKxBtY+Aq53RbcOA3aq6VURicCWqKar6ToPjpjl/CjDWwzGNMaZFC+aHR8G3gws6Ah+48gNRwOuq+rmI/AJAVZ8FPgXOB9biGsl2rbPvpcAIoL2IXOOUXaOqi4DXRCQV12W6RcAvfNgGY4wJOYUllfTsGB/oMBrls8SjqrnAAA/lz7q9V+BGD3VeBV5t5LhneDFMY4wJO0WlVQw/Nnh7PDZzgTHGhJHKmlp27w3e6XLAEo8xxoSVYJ8uByzxGGNMWKmftcB6PMYYY/xi/8OjLXA4tTHGGP8L9nnawBKPMcaElWCfLgcs8RhjTFgpKq0iPi6KuOjgnC4HLPEYY0xYKSytDNpZqetZ4jHGmDBSWFJJShDf3wFLPMYYE1aKrMdjjDHGn4J9ZmqwxGOMMWGjorqWkoqaoB5KDZZ4jDEmbITCrAVgiccYY8JGUQjM0waWeIwxJmwUhcDDo2CJxxhjwkZhCEyXA5Z4jDEmbNT3eNrbqDZjjDH+UFhaSWKraGKjgne6HLDEY4wxYaOoNPif4QEfJx4R2SAiS0VkkYjkeNguIvKEiKwVkSUiMtht20QRWeO8JrqVD3GOudbZV3zZBmOMCRVFJVVBP7AA/NPjOV1VB6pqtodt5wE9nNck4BkAEWkH/AkYCpwI/ElEkp19nnHq1u93rm/DN8aY0FBYWhn0Awsg8JfaLgKmqMscIElE0oBzgBmqulNVi4EZwLnOtgRVna2qCkwBxgYsemOMCSJFJZXW4wEUmC4i80Vkkoft6cAmt8/5TtmhyvM9lB9ERCaJSI6I5BQWFjajCcYYE/wqqmspqQz+6XLA94lnuKoOxnVJ7UYRGdFgu6f7M3oU5QcXqj6nqtmqmp2amnokMRtjTMipX3k02GemBh8nHlXd4vxZAHyA636Nu3ygs9vnDGDLYcozPJQbY0yLFioPj4IPE4+ItBGR+Pr3wChgWYNqHwFXO6PbhgG7VXUrMA0YJSLJzqCCUcA0Z1uJiAxzRrNdDXzoqzYYY0yoCJXpcgCifHjsjsAHzmjnKOB1Vf1cRH4BoKrPAp8C5wNrgXLgWmfbThH5CzDPOdZ9qrrTef9L4CWgFfCZ8zLGmBYtVCYIBR8mHlXNBQZ4KH/W7b0CNzay/4vAix7Kc4DjvRepMcaEvoKSCgDatWnhD5AaY4zxj8KSStq1iSEmKvh/rAd/hMYYYw6roKSSDiFwmQ0s8RhjTFgoKAmNWQvAEo8xxoSFwj0VdIiPC3QYTWKJxxhjQpyqUlhaSYcE6/EYY4zxg+Lyaqpr1e7xGGOM8Y/6odR2j8cYY4xfFOxxzVpg93iMMcb4RUFJfeKxHo8xxhg/qL/UZoMLjDHG+EXBnkraxkbROsaX0296jyUeY4wJcYWloTNrAVjiMcaYkFe4J3RmLQBLPMYYE/IKSirokBAaI9rAEo8xxoS8UJogFCzxGGNMSCutrKG8qtYutRljjPGPgj3OUGpLPMYYY/xh/8Ojdo9nHxGJFJGFIjLVw7ZMEflCRJaIyEwRyXDKTxeRRW6vChEZ62x7SUTWu20b6Os2GGNMsNqXeELk4VEAfzxtdDOwAkjwsO1RYIqqviwiZwAPAlep6lfAQAARaQesBaa77fc7VX3Xt2EbY0zws0ttDTg9mNHA841U6Qt84bz/CrjIQ52fAJ+parn3IzTGmNBWWFJJTFQEia2iAx1Kk/n6UtvjwO+Buka2LwbGO+/HAfEi0r5BncuBNxqUPeBcnvuHiIROmjfGGC8rLKkktW0sIhLoUJrMZ4lHRC4AClR1/iGq3Q6cJiILgdOAzUCN2zHSgH7ANLd97gR6AycA7YA7Gjn/JBHJEZGcwsLCZrXFGGOCVUFJ6Kw8Ws+XPZ7hwBgR2QC8CZwhIq+6V1DVLap6saoOAu5yyna7VbkU+EBVq9322aoulcBk4ERPJ1fV51Q1W1WzU1NTvdowY4wJFgUlFSF1fwd8mHhU9U5VzVDVLFyXy75U1QnudUQkRUTqY7gTeLHBYa6gwWU2pxeEuPqVY4FlPgjfGGNCQkFJaM3TBgF4jkdE7hORMc7HkcAqEVkNdAQecKuXBXQGvm5wiNdEZCmwFEgB7vdxyMYYE5Qqa2rZVV4dUs/wgH+GU6OqM4GZzvt73MrfBTwOi1bVDUC6h/IzfBGjMcaEmsIQW3m0ns1cYIwxISoUHx4FSzzGGBOyCvaE3nQ5YInHGGNCVmFJ6M1aAJZ4jDEmZBWUVBIh0L6tJR5jjDF+ULCnkvZtY4mMCJ1ZC8ASjzHGhKzC0tBaebSeJR5jjAlRBSUVIffwKFjiMcaYkFWwx3o8xhhj/KS2TikqrQy5odRgiccYY0LSjrJK6jT0Hh4FSzzGGBOS9j88aonHGGOMH9TP05Zql9qMMcb4Q0GIzloAlniMMSYk1V9qs+HUxhhj/GJ7SQWJraKJi44MdChHzBJPM7w4az3DH/qSvVW1gQ7FGNPCbC7eS0Zyq0CHcVQs8RylbbsreHT6Kjbv2sv05ds81snbUc5l/57NtB89bzfGmKOVH+6JR0S6i0is836kiPxGRJJ8G1pwe/jzldTUKqnxsby3YLPHOi9+t54f1u/k56/M554Pl1FRbT0jY0zzqaqTeFoHOpSj0tQez3tArYgcC7wAdAVe91lUQW5BXjHvL9zM9ad25YoTOjNrTSHb91QcUKe8qob35uczul8a15/SlSmzNzL2X9+xtqAkQFEbY8LFzrIq9lbXhnePB6hT1RpgHPC4qt4CpDVlRxGJFJGFIjLVw7ZMEflCRJaIyEwRyXDbVisii5zXR27lXUXkBxFZIyJviUhME9vgFXV1yp8/+pGOCbHcePqxjBucQZ3Cfxce2Ov5aNEWSipruHZ4Fndf0JfJ15xAYUklFz75HW/MzUNVj+r8awtKWJhX7I2mGGNCVH7xXoCw7/FUi8gVwESgPoFEN3Hfm4EVjWx7FJiiqv2B+4AH3bbtVdWBzmuMW/nfgH+oag+gGLiuiXF4xbsL8lmcv5s/nNebNrFRdE1pw+AuSby3IH9fMlFVXpmzkd7HxDMkMxmA03t34LObT2VIZjJ3vr+USa/MZ2dZ1RGdu7KmlqtfmMtP//MDm3aWe71txpjQsD/xhHeP51rgJOABVV0vIl2BVw+3k9ODGQ0830iVvsAXzvuvgIsOczwBzgDedYpeBsYeNnovKamo5uHPVzGoSxIXDUjfV37x4AxWby/lxy17AFi0aRc/btnDlcMycYXs0iEhjik/O5G7R/fh61WFnPP4N3yzurDJ53/9hzy27K6gpq6OP33041H3mowxoS2/2PWLZ3o4Jx5VXa6qv1HVN0QkGYhX1YeasOvjwO+Buka2LwbGO+/HAfEi0t75HCciOSIyR0Tqk0t7YJdz2Q8gH9ifAdyIyCRn/5zCwqb/cG+MqnL/1BUUlVZy74XHEeG24t+F/TsRExnBewvyAXh1Th5tYiIZN+jg0CIihOtP7cZ/bxxOUqtorn5xLm/MzTvs+csqa/jXV2s5qVt77ji3N1+uLODzZTZazpiWKL94L4mtokmIa+qFp+DS1FFtM0UkQUTa4UoWk0XkscPscwFQoKrzD1HtduA0EVkInAZsBuqTShdVzQZ+CjwuIt0BT+u7evy1X1WfU9VsVc1OTU09ZPua4tHpq3grZxO/GtmdAZ0PHNCX2Dqas/p24KNFWygsqWTqki2MHZRO29ioRo/Xt1MCH990Cqccm8J9Hy9nQ1HZIc//0vcbKCqt4vZzenHNyVn0TUvg3o9/pKSiutltM8aElvzi8pC9zAZNv9SWqKp7gIuByao6BDjrMPsMB8aIyAbgTeAMETng8pyqblHVi1V1EHCXU7a7fpvzZy4wExgEFAFJIlL/Ez0D2NLENhy1/3yTy7++WscVJ3bmd+f08ljn4kEZ7Cir4pa3FlFZU8eEYZmHPW5cdCSPXNKfqEjh9ncWU1vn+dLZ7vJqnv16HWf16cCQzGSiIiP468X9KCip5O/TVzerbcaY0BPKz/BA0xNPlIikAZeyf3DBIanqnaqaoapZwOXAl6o6wb2OiKSISH0MdwIvOuXJbs8NpeBKYsvVdVPjK+Anzj4TgQ+b2Iaj8va8TTzw6QpG90vj/rH9Drhn4+60Xqm0axPDrLVFZGcm0yctoUnHT0tsxZ/HHEfOxmJenLXeY51/f7OOkooabhu1P+kN7JzE1cMyeXn2BhZv2nXE7TLGhKZQf4YHmp547gOmAetUdZ6IdAPWHM0JReQ+EakfpTYSWCUiq4GOwANOeR8gR0QW40o0D6nqcmfbHcCtIrIW1z2fF44mjqb4fNlW/vD+Ek7tkcJjlw0gMsJz0gGIjoxgzIBOAE3q7bgbNyids/t25JHpqw56zqegpILJ321gzIBOByWz287pRWrbWO756McjOp8xJnSF+jM8AI3fhHCjqu8A77h9zmX/oICm7D8T1+UyVPUet/J32T9Czb3+90C/Ro6VC5zY1HMfLVXlnZx8BnZO4t9XDSE26vAT8U0a0Y3Y6AjO63fMEZ1LRPjruH6M+sfX3Pb2Yt6YNIz1RWWs3FrCh4u3UFVbxy1n9zxov4S4aK47pSsPfraSgj0VdEgIvXU5jDFHZlOIP8MDTUw8zrDoJ3Fd8lJgFnCzqub7MLaAEhGenjCYiuo6Wsc06a+JTkmtuPO8Pkd1vtT4WO4f248bX1/AcX+aRv1I6ZioCG4641i6prTxuN+wbq5BgHPW79zX4zLGhK/6odRh3+MBJuOaIucS5/MEp+xsXwQVLGKjIpvU0/GW0f3T2LKrD7v2VtH7mAT6pMWT1b4NUZGNXxE9rlMCbWOj+CF3hyUeY1qA+odHQ/UZHmh64klV1clun18Skd/6IqCW7oYR3Y6oflRkBEMyk/lh/U4fRWSMCSb5xeUh/QwPNH1wQZGITHDmXYsUkQnADl8GZppuaLd2rC0opai0MtChGGN8LNSHUkPTE8/PcA2l3gZsxTWc+VpfBWWOTP19nrnW6zEm7LWYxKOqeao6RlVTVbWDqo7F9TCpCQL90hNpHRPJD7nWCTUmnLme4SkP6RFt0LwVSG/1WhSmWaLtPo8xLcKOsioqqutaRo+nEY0/TWn8bmjXdqzcVkLxES61YIwJHaG+Dk+95iQem5M/iAx17vNYr8eY8FX/DE/ndmHc4xGREhHZ4+FVAthDI0Gkf0YisVER/LDe7vMYE672PcOTFNqJ55DP8ahqvL8CMc0TGxXJ4C7J/JBrPR5jwlV+cTlJraOJD+FneKB5l9pMkBnarR0rtu1hd7mt0WNMOAqHodRgiSesDO3aHlWYt8F6PcaEo/zivWQkhfbAArDEE1YGdUkiJtLu8xgTjvY/w2M9HhNE4qIjGdg5yUa2GROGwuUZHrDEE3aGdW/Pss27Wb295PCVjTEhI1ye4QFLPGHn6pMySWodw21vL6a6ti7Q4RhjvGTfOjwh/gwPWOIJOyltY/nruONZunk3T3+17rD1VZUNRWXUWJIyJqiFyzM80PT1eI6aiEQCOcBmVb2gwbZM4EUgFdgJTFDVfBEZCDwDJAC1wAOq+pazz0vAacBu5zDXqOoiX7cjlJx7fBpjB3biyS/XcGafDhyfnnjA9ro6ZeGmXXy2dCufLdvG5l17SU9qxcSTM7nshC4kttr/jEBxWRUL8opZV1jKxh3l5O0sZ9POciIihK7t25DZvg1dU1pzfHoiAzsnIWIzKRnjC+HyDA/4IfEANwMrcCWRhh4FpqjqyyJyBvAgcBVQDlytqmtEpBMwX0SmqeouZ7/fqeq7fog9ZP15zPHMzt3BrW8v4uObTiE2KpKCkgrenLuJN+fmsWV3BdGRwqk9UvnZKV2ZsXwbf/10JY//bw0XD06ntg5yNuxkTUHpvmMmtY6mS7vWHJeeSG2tsmFHGd+v28He6lrA9ZvY6P5pjO6XRt9OCWzcUc7aglLWFpQQGRHB9ad2JfoQq6kaYxq3cUd4jGgDHyceEckARgMP4Hk2677ALc77r4D/Aqjq6voKqrpFRApw9Yp2HXQE41Fi62geGt+fayfP4//eX0ZNXR2fLt1Kda1yao8Ubj+nF2f17bhvFcPrTunKss27mfzdBt6at4m4qEiGZCUzdlA6QzKT6ZOWcEBPqJ6qsn1PJd+vK2Lqkq1M/m49z32Tiwhog9n8Vm8v4e+XDCAiwv+9oo8Wb2FR3i7uHt0nIOc3prnWF5UxuEtyoMPwCl/3eB4Hfg80NvXOYmA88E9gHBAvIu1Vdd+DKCJyIhADuN+weEBE7gG+AP6gqrb0pgen9+rA5Sd05s15m4iPjWLCsEyuGpZJt9S2Husfn57I3y8dwP1jjycmKoLIJvyAFhGOSYzj4sEZXDw4g93l1Uxbvo28HeV0S21Djw7xdEttw+Tv1vPo9NW0axPD3aP7HNEluYrqWiZ/t4Gtu/eiCurMTzuwczIXDkgjNiqy0X1VlX9/k8tDn60EXJMrXju8a5POW1NbR5T10EwQqKiuZfOuvfxkSEagQ/EKnyUeEbkAKFDV+SIyspFqtwNPicg1wDfAZqDG7RhpwCvARFWtv/t9J66VUGOA54A7gPs8nH8SMAmgS5cuXmhRaLp3zHGc0bsDw49NoU1s077uVjGN/yA/nMTW0Vya3fmg8htPP5YdZVW8MGs97dvG8KuRxzbpeMu37OG3by1k9fZSklpHI7iSXU1tHa/OyePhz1dy7fCu/HRol4N6ZHV1yv2frODF79Zz4YBO7NlbzcOfr+LM3h3p0v7QQ1IX5BUz8YW53HNhXy7x0B5j/GnDjjJUoWtKm0CH4hW+7PEMB8aIyPlAHJAgIq+q6oT6Cqq6BWclUxFpC4xX1d3O5wTgE+BuVZ3jts9W522liEzGlbwOoqrP4UpMZGdnt9glHOKiIxl13DGBDgMR4Y+j+7KzrIqHP19Fu9YxXH5i478Q1NUpz8/K5dFpq0lsHc1L157AyF4d9m1XVb5dU8Rz3+Tyt89X8tSXazirb0d6doynZ8d4uqe24fH/reGjxVu4dngWfxzdl217Khj1j2+4470lvH7D0EZ7XXV1yp8/Xk5JZQ3/98FSuqW2YUhmO6//nRjTVLmFZQB0b+RqRajxWeJR1Ttx9U5wejy3uycdpzwF2On0Zu7ENcINEYkBPsA18OCdBvukqepWcf3UGAss81UbjHdFRAiP/GQAu8qr+cP7S3l65jqGdm3HiV3bMbBzEjvKqti4o4wNO8qZk7uDhXm7GNW3Iw+N70+7NjEHHEtEGNEzlRE9U/lxy25e+HY9c3J38OGiLQfUu+Pc3vzitG6ICJ2SWnHX6D7c+f5SXp+bx5VDMz3G+fGSLSzetIs/XtCXKbM38PNXFvDRr4fTKQyGsZrQtL7IlXisx3OUROQ+IEdVPwJGAg+KiOK61HajU+1SYATQ3rkMB/uHTb8mIqm4VkBdBPzCj+GbZoqJiuDZCUN4c14es9ftYMaK7bwzP/+AOlERQmb71jw8vj+XZGcc9n7QcZ0SeeyygQCUVFSzpqCUNdtLSE9qzSk9Ug6oe/kJnZm6ZAsPfrqSkb06HPRMREV1LX/7bCXHpydw7clZjOiRwrinv2fSKzm88/OTm3UZ0pijta6wlGMS4pp8uTzYiTYcehSGsrOzNScnJ9BhGA/q6pQ1BaUs27yb1PhYstq3oVNSnE9v6m/aWc45j3/DkMxknp+YfcDghKe+XMOj01fz5qRhDHNWdf1ixXaun5LD6H5pPHnFIHtWyfjd2H99R6voSN6YNMyv5xWR+aqa7e3j2pAdE1AREUKvY+IZPySDET1T6dK+tc9HknVu15q7Rvfh2zVFjH5iFgvyigEoKKng6ZnrOOe4jvuSDsCZfTryu3N6MXXJVm57ZzFllTWNHdoYr1NVcgtL6ZYaHpfZIACX2owJBlcOzSQ9qRX/9/5Sxj/zPT8b3pXisiqqa+u487w+B9X/5Wndqaqp459frGHRpl3866eD6ZPm6ZloY7xrZ1kVeypqGn0MIhRZj8e0WCN7dWDaLSO4cmgXXpi1nvcXbmbiSVlkebiBKyL89qyevHb9UEorarjoX9/xypyNtIRL1Sawcp2BBeHU47HEY1q0+Lho7h/bjzduGMYVJ3bhpjN7HLL+yd1T+PTmUzmpW3v++N9lTHjhB5Zt3n3IfYxpjtxC17RV3cJkRBtY4jEGgJO6t+fBi/t5nBaooZS2sUy+5gT+POY4lm/ZwwVPzuKWtxbtm7beGG/KLSojJjIiLNbhqWf3eIw5ChERwsSTsxg7KJ1nv17Hi7PW88nSrVzQL43TeqVyyrEptG8bG+gwTRjILSwjs33rJk1hFSos8RjTDImtornj3N5cNSyTJ79cw+fLtvH+ws2IwPGdEhk/OJ2rTsoKqx8axr9yC0s5tkP4DCwAu9RmjFd0SmrFgxf3J+fus/nwxuHcelZPIiKEez9ezsVPf8eKrXsCHaIJQTW1deTtLKdriiUeY0wjIiOEAZ2TuOnMHvz3VyfzxBWDyC/ey4VPzuLv01dRWVMb6BBNCMkv3kt1rYbViDawxGOMz4gIYwZ04n+3nsaYAZ148su1/OSZ2ZRX2QOopmlyi1wj2rpb4jHGHInkNjE8dtlAnrlyMD9u2c3v3lliz/+YJqmflbqbXWozxhyN8/qlcce5vflk6Vb+9dXaQIdjQkBuURlJraNJbjA7e6izxGOMH00a0Y2xAzvx6PTVzFi+PdDhmCCXW1gaVg+O1rPEY4wfiQgPje9Pv/REbnlrEWu2lwQ6JBPEcgvLwmqOtnqWeIzxs7joSJ67eghx0ZHcMCWH3XurAx2SCUKllTUUlFSG3Yg2sMRjTECkJbbi2QmDyS/ey21vL6auzgYbmAOtD9OBBWCJx5iAyc5qx/+d34f/rdjOv7/JDXQ4JsjUD6W2Ho8xxquuHZ7FBf3TeGTaSr5fVxTocEwQyS0sI0Igs334TA5azxKPMQEkIvxtfH+6pbblN28sZNvuikCHZIJEblEZGcmtD1iaPVz4PPGISKSILBSRqR62ZYrIFyKyRERmikiG27aJIrLGeU10Kx8iIktFZK2IPCEiNvuiCWltYqN4dsJgyqtq+cWr89ngLPzVUF2dsnp7id/uB1VU17JxRxmrtvnvnGa/NdtLwvIyG/hnduqbgRWAp3WCHwWmqOrLInIG8CBwlYi0A/4EZAMKzBeRj1S1GHgGmATMAT4FzgU+830zjPGdYzvE89ilA7j5zUWc+djXXDSwEzeefizdU9tSWFLJ2zmbeHNeHpt27uXiwek8PL4/UZHN/71x6pItvDVvEzW1Sq0qdXVKWVUt2/dUsLOsal+9tMQ4zjs+jfP7HcPgLslEeJhtu7q2jg8WbuaV2RtpFRNJz45t6dUxnp4d4xnUJZmYKLvA0lSllTWs3l7COccdE+hQfMKnicfpwYwGHgBu9VClL3CL8/4r4L/O+3OAGaq60znODOBcEZkJJKjqbKd8CjAWSzwmDJx7fBrf3pHMf77J5dU5efx34WYGdUlm8aZd1NQpw7q149Qeqbz+Qx6VNXU8ftlAopuRfL5dU8jNby4iPakVHRNiiRAhNjqCxFbRDOqSRKfEOI5JbEVdnTJ9+XZe/WEjL363ng7xsQw/NoVh3doxrFt70hJb8cHCfJ76ai2bdu6l9zHxxNRF8OHCLZRUuualG5CRyH8mZtMhPs5bf11hbfGmXdQpDM5MDnQoPuHrHs/jwO+B+Ea2LwbGA/8ExgHxItIeSAc2udXLd8rSnfcNyw8iIpNw9Yzo0qXL0bfAGD/qEB/HXaP78vPTuvP8t+v5YsV2rjk5iyuGdqG78yBhVvvW/PXTlVTV1PHUTwftuwegquTtLKd921jaxh76v/baglJ+9doCenRoy7u/PPmw9S89oTMlFdWtD9r0AAAVkElEQVR8ubKA6cu38+2aQj5YuBmA2KgIKmvq6J+RyL0XHscZvTsgIqgq2/ZUMGtNEfd8+CPj/vU9L1yTTe9jPF38MO4WbCwGYGDnpABH4hs+SzwicgFQoKrzRWRkI9VuB54SkWuAb4DNQA3g6b6NHqL84ELV54DnALKzs+0CtQkpKW1j+cN5vfnDeb0P2jZpRHfioiO558Mfuf7lHE7IaseCvGIW5u1i995qoiKEIZnJjOiZymk9U+mblnDApbHisique3kesVERPD8x+7BJp158XDQXDUznooHpqCrrCsuYk7uD5Vv3cHafjozslYr7LVcRIS2xFZdkd6ZPWgLXvTyPnzwzmyd/OojTe3Vo/l9SGFuQV0zPjm2btBR7KPJlj2c4MEZEzgfigAQReVVVJ9RXUNUtwMUAItIWGK+qu0UkHxjpdqwMYCauHk5Gg/ItPmyDMUHp6pOyiI2K4A/vL+XbNUX06NCWc487hv6dE9m0cy/frC7kkWmreGTaKlLaxnJG71TO7NORoV3b8fNX57N1dwVv3DCMjOSjG6orIhzboW2TV8Y8Pj2RD288hetensd1L83jlrN6csOIbsRFh9+Ireaqq1MWbtrFuWF6fwdA/DE9u9PjuV1VL2hQngLsVNU6EXkAqFXVe5zBBfOBwU7VBcAQVd0pIvOAm4AfcA0ueFJVPz3U+bOzszUnJ8e7jTImCGzfU0FcdKTH34wLSir4dnURX60q4OvVhZRU1CACqvDPywdy0UCPV6l9qqyyht+/u4RPlm4lPakVvz+3Fxf27+RxsEJLtbaglLMe+5qHx/fn0hM6BzQWEZmvqtnePq4/RrUdQETuA3JU9SNcvZoHRURxXWq7EcBJMH8B5jm73Vc/0AD4JfAS0ArXoAIbWGBarI4Jjd+s7xAfx/ghGYwfkkF1bR05G4r5cuV2Mtu3CUjSAdfQ8X9dOZgr1xbxwKcruPnNRbw4az33jjmOQV3C80b6kVqQ57q/MzgzPO/vgJ96PIFmPR5jgk9dnfLBws08Mm0VO8uqePKng8J2+PCRuPP9JXyyZCuL7hkV8J6gr3o8NrDeGBMQERHC+CEZfP7bU+nbKYFfvbaA9+bnH37HMLdg4y4GZ3p+VipcWOIxxgRUUusYXrt+KMO6teO2dxbz0nfrAx1SwOypqGZ1QQmDw/yyoyUeY0zAtYmN4oWJJzCqb0fu/Xg5/5ixmpZwG6ChRXm7UMUSjzHG+ENcdCRPXzmYnwzJ4J9frOHXry+kvKom0GH51YK8YkRgQOfEQIfiU34f1WaMMY2JiozgkZ/0p2fHtjz02UrWFZbyn6uz6dwu/JYG8GRB3i56dYwnPi48HxytZz0eY0xQEREmjejO5GtPZMuuvYx5ahbfrw3/tYrq6pSFecVhOz+bO0s8xpigdFrPVD769SmktI1l4uS5+55vCVdrC0spqagJ+/s7YInHGBPEslLa8M4vTiItsRW/fHU+hSWVgQ7JZ+onBh3cJXwfHK1niccYE9SSWsfw7IQh7N5bza9fX0B1bV2gQ/KJBXnFJLeOpmtKeC7+5s4SjzEm6PXtlMCDF/fjh/U7eeizlYEOxyfmbyxmUJfkA2b4DleWeIwxIWHcoAyuOTmLF2at58NFmwMdjldt2lnOusIyTu7ePtCh+IUlHmNMyLhrdB9OzGrHHe8t4YsV2wMdjtdMX+5qy9l9OwY4Ev+wxGOMCRnRkRE8PWEwPTrEc8OUHJ7/NjcsZjiYsXwbPTu2JbN9+N/fAUs8xpgQk9I2lrd/fhKj+h7D/Z+s4P8+WBbSAw52lVcxb0Nxi+ntgCUeY0wIahXjml7nVyO788bcPCa+OJdNO8sDHdZR+XJlAbV1ytl9W86SEJZ4jDEhKSJC+P25vXn0kgHM31jMGX+fyd3/Xcr2PRWBDu2IzFi+nQ7xsfRPD+/52dxZ4jHGhLSfDMng69+dzqXZnXlz7iZGPPwVD3yynIrq2kCHdlgV1bV8vbqQs/p2DOv1dxqyxGOMCXnHJMbxwLh+fHnbSEb3T+M/367n8f+tCXRYhzV73Q7Kq2pb1P0d8EPiEZFIEVkoIlM9bOsiIl8525eIyPlO+ZUissjtVSciA51tM0Vkldu2Dr5ugzEmNHRp35rHLh3IxYPSmfzderbs2hvokA5p+vLttImJbDHP79TzR4/nZmBFI9vuBt5W1UHA5cDTAKr6mqoOVNWBwFXABlVd5LbflfXbVbXAl8EbY0LPraN6ogqPzVgd6FAaVVen/G/Fdkb26kBsVGSgw/ErnyYeEckARgPPN1JFgQTnfSKwxUOdK4A3vB+dMSZcZSS3ZuLJmby3IJ+V2/YEOhyPFuXvorCkssVdZgPf93geB34PNDbI/l5ggojkA58CN3mocxkHJ57JzmW2P0pLmNjIGHPEbjz9WOJjo/hbkM7tNmP5diIjhNN7tby7BT5LPCJyAVCgqvMPUe0K4CVVzQDOB14RkX0xichQoFxVl7ntc6Wq9gNOdV5XNXL+SSKSIyI5hYWFzW2OMSbEJLWO4cbTj+WrVYV8vy64FpJTVWYs387Qru1IbB3eq4164ssez3BgjIhsAN4EzhCRVxvUuQ54G0BVZwNxQIrb9stp0NtR1c3OnyXA68CJnk6uqs+paraqZqempja/NcaYkDPx5Cw6Jcbx0GcrqasLnql1vlpVwNqCUkb3Twt0KAHhs8SjqneqaoaqZuFKIF+q6oQG1fKAMwFEpA+uxFPofI4ALsGVtHDKokQkxXkfDVwALMMYYzyIi47k1lG9WJK/mw8WBseM1lU1ddw/dQXdUtpwyZDOgQ4nIPz+HI+I3CciY5yPtwE3iMhiXD2ba3T/jH8jgHxVzXXbPRaYJiJLgEXAZuA/fgrdGBOCxg1KZ0hmMn/66EfWF5UFOhymzN5AblEZd1/Qh5iolvkopYTDzK6Hk52drTk5OYEOwxgTIJt37WX0E9/SKbEV7//qZOKiAzN8eUdpJSMfncmgLsm8fO0JQb/om4jMV9Vsbx+3ZaZbY0yLkp7Uir9fMoDlW/dw/yfLAxbHYzNWU15Vyx9H9wn6pONLlniMMS3CmX068vMR3Xh1Th5Tl3h6ZNC3Vmzdwxtz87hqWCY9Osb7/fzBxBKPMabFuP2cXgzuksQf3lvq1/s9dXXKX6YuJ6FVNL89q4ffzhusLPEYY1qM6MgInvrpYKIihRtfW+CXGayLy6q4fkoO36/bwW1n9ySpdYzPzxnsLPEYY1qUTkmteOxS1/2e+6b69n7P/I3FjH7iW2atKeLPY45jwrBMn54vVFjiMca0OGf07sgvTuvO6z/k8eEi7z/fo6r855tcLvv3bCIjhXd/eRITT85q0QMK3FniMca0SLeP6skJWcnc+f5S1haUevXYT365lgc+XcHZfTsy9aZT6Z+R5NXjhzpLPMaYFikqMoInrxhMXHQkN762gL1V3rnf8+nSrTw2YzUXD0rn6SsHk9iq5c3FdjiWeIwxLdYxiXE8ftlAVheU8KvX5lNcVtWs4y3N382tby9iSGYyD47vZ5fWGmGJxxjToo3omcp9Y45j1toizvvnt3y/9uhmst6+p4Lrp8yjfZtY/n3VkBa3uNuRsMRjjGnxrjopi/d/OZzWMZFc+cIPPPjZCqpqGltG7GB7q2qZNCWHkooanp+YTUrbWB9GG/qiAh2AMcYEg34ZiUz9zSn8Zepy/v11Lu/Nz+e4Ton0Tounb1oCQzKTyUhufdB+hSWV3DAlhyWbd/PcVdn0SUvwcHTjzhKPMcY4WsdE8eDF/TmrT0c+WbqVFVtL+H5dEdW1SlSEcPVJWdx8Zo99i7et2lbCz16ax46ySp65ckiLXMb6aFjiMcaYBs7s05Ez+7iSSFVNHesKS3llzkYmf7+e/y7azG2jetIpsRU3vbGQ1jGRvPPzk+mXkRjgqEOHLYtgjDFN9OOW3fz54+XMXb8TgD5pCbx4TTZpia0CHJlv+GpZBOvxGGNMEx3XKZG3Jg3j06XbWJhXzC1n96RNrP0YPVL2N2aMMUdARBjdP43R/dMCHUrIsuHUxhhj/MoSjzHGGL/yeeIRkUgRWSgiUz1s6yIiXznbl4jI+U55lojsFZFFzutZt32GiMhSEVkrIk+IzUlhjDEhxR/3eG4GVgCenqq6G3hbVZ8Rkb7Ap0CWs22dqg70sM8zwCRgjlP/XOAzbwdtjDHGN3za4xGRDGA08HwjVZT9CSkROORC6CKSBiSo6mx1jQOfAoz1UrjGGGP8wNeX2h4Hfg80NunRvcAEEcnH1Xu5yW1bV+cS3NcicqpTlg7ku9XJd8oOIiKTRCRHRHIKCwub0wZjjDFe5LPEIyIXAAWqOv8Q1a4AXlLVDOB84BURiQC2Al1UdRBwK/C6iCQAnu7neHwCVlWfU9VsVc1OTU1tVluMMcZ4jy/v8QwHxjgDBuKABBF5VVUnuNW5Dtc9GlR1tojEASmqWgBUOuXzRWQd0BNXDyfDbf8MDnN5zhhjTHDxy5Q5IjISuF1VL2hQ/hnwlqq+JCJ9gC9wXTpLAXaqaq2IdAO+Bfqp6k4RmYfrktwPuC7PPamqnx7m/IXAxgbFicDuw5S5fz7c+xTg6BbyaDyeptY50rY0/Fz/3r2sOe1pTlsa22bfjX03RxJnU+oFU1sOFWdT6vjq31kbVfX+JSNV9fkLGAlMdd7fB4xx3vcFvgMWA4uAUU75eOBHp3wBcKHbsbKBZcA64Cmc5HkUMT13uDL3z4d7D+Q08+/ooHiaWudI23KINriXHXV7mtMW+27su/FmW7z53di/M++9/DJljqrOBGY67+9xK1+O65Jcw/rvAe81cqwc4HgvhPVxE8o+PsL33o6nqXWOtC0NP3/cSJ2j1Zy2NLbNvhvvCKfvpqnH8NZ3Y//OvKRFzE7tDyKSoz6YxTVQwqk94dQWCK/2WFuCly/bY1PmeM9zgQ7Ay8KpPeHUFgiv9lhbgpfP2mM9HmOMMX5lPR5jjDF+ZYnHAxF5UUQKRGTZUezb6CSmInKTiKwSkR9F5GHvRt1oPF5vi4jcKyKb3SZxPd/7kTcak0++G2f77SKiIpLivYgPGY8vvpu/OBPuLhKR6SLSyfuRNxqTL9rziIisdNr0gYgkeT9yj/H4oi2XOP/360TE5/eCmtOGRo43UUTWOK+JbuVHPnGzr4bLhfILGAEMBpYdxb5zgZNwzbLwGXCeU3468D8g1vncIYTbci+u57LC4rtxtnUGpuF63islVNuCay7D+jq/AZ4N5e8GGAVEOe//BvwthNvSB+iFa4RvdrC2wYkvq0FZOyDX+TPZeZ98qPYe6mU9Hg9U9Rtgp3uZiHQXkc9FZL6IfCsivRvuJ4eexPSXwEOqWj8jQ4FvW+Hio7YEjA/b8w9c8wr67aanL9qiqnvcqrYh9NszXVVrnKpzOHDmEp/xUVtWqOoqf8TvnO+o2tCIc4AZqrpTVYuBGcC5R/tzwhJP0z0H3KSqQ4Dbgac91DnUJKY9gVNF5AdxTXx6gk+jPbTmtgXg187ljxdFJNl3oTZJs9ojImOAzaq62NeBNkGzvxsReUBENgFXAvcQWN74t1bvZwR2CRRvtiVQmtIGT9KBTW6f69t1VO31ywOkoU5E2gInA++4Xb6M9VTVQ1n9b5xRuLqow4ATgLdFpJvzW4LfeKktzwB/cT7/Bfg7rh8Kftfc9ohIa+AuXJd0AspL3w2qehdwl4jcCfwa+JOXQ20Sb7XHOdZdQA3wmjdjbCpvtiVQDtUGEbkW19ppAMcCn4pIFbBeVcfReLuOqr2WeJomAtilDRamE5FIoH727Y9w/UBubBLTfOB9J9HMFZE6XHMh+XvNhma3RVW3u+33H+Cg1WX9qLnt6Q50BRY7/xkzgAUicqKqbvNx7A1549+Zu9eBTwhQ4sFL7XFuZF8AnOnvX9TcePu7CQSPbQBQ1cnAZAARmQlco6ob3Krk45r6rF4GrntBRzdxs69vcIXqC9dKqMvcPn8PXOK8F2BAI/vNw9Wrqb/Rdr5T/gvgPud9T1zd1qOaZy4I2pLmVucW4M1Q/m4a1NmAnwYX+Oi76eFW5ybg3VD+bnDNXr8cSPVnO3z57ww/DS442jbQ+OCC9biu2iQ779s1pb0e4/L3lxkKL+ANXGsCVePK6Nfh+q34c1wTly4H7mlkX4+TmAIxwKvOtgXAGSHclleApcASXL/lpfmjLb5qT4M6G/DfqDZffDfvOeVLcM27lR7K3w2wFtcvaYucl19G6fmoLeOcY1UC24FpwdgGPCQep/xnzvexFrj2cO091MtmLjDGGONXNqrNGGOMX1niMcYY41eWeIwxxviVJR5jjDF+ZYnHGGOMX1niMS2SiJT6+XzPi0hfLx2rVlyzTy8TkY8PN2OziCSJyK+8cW5jvMGGU5sWSURKVbWtF48Xpfsns/Qp99hF5GVgtao+cIj6WcBUVT3eH/EZczjW4zHGISKpIvKeiMxzXsOd8hNF5HsRWej82cspv0ZE3hGRj4HpIjJSRGaKyLviWkPmtfq1SZzybOd9qTOR52IRmSMiHZ3y7s7neSJyXxN7ZbPZP9lpWxH5QkQWiGt9lIucOg8B3Z1e0iNO3d8551kiIn/24l+jMYdliceY/f4J/ENVTwDGA8875SuBEao6CNdsz3912+ckYKKqnuF8HgT8FugLdAOGezhPG2COqg4AvgFucDv/P53zH3a+K2eesDNxzR4BUAGMU9XBuNZ/+ruT+P4ArFPVgar6OxEZBfQATgQGAkNEZMThzmeMt9gkocbsdxbQ123m3gQRiQcSgZdFpAeumXej3faZoarua57MVdV8ABFZhGuurFkNzlPF/olV5wNnO+9PYv9aJq8DjzYSZyu3Y8/HtTYKuObK+quTROpw9YQ6eth/lPNa6HxuiysRfdPI+YzxKks8xuwXAZykqnvdC0XkSeArVR3n3C+Z6ba5rMExKt3e1+L5/1i17r+52lidQ9mrqgNFJBFXArsReALX+jupwBBVrRaRDUCch/0FeFBV/32E5zXGK+xSmzH7Tce1fg0AIlI/fXwisNl5f40Pzz8H1yU+gMsPV1lVd+Na3vp2EYnGFWeBk3ROBzKdqiVAvNuu04CfOeuzICLpItLBS20w5rAs8ZiWqrWI5Lu9bsX1QzzbueG+HNdSFgAPAw+KyHdApA9j+i1wq4jMBdKA3YfbQVUX4ppp+HJci6Rli0gOrt7PSqfODuA7Z/j1I6o6HdelvNkishR4lwMTkzE+ZcOpjQkSzmqoe1VVReRy4ApVvehw+xkTauwejzHBYwjwlDMSbRcBWk7cGF+zHo8xxhi/sns8xhhj/MoSjzHGGL+yxGOMMcavLPEYY4zxK0s8xhhj/MoSjzHGGL/6f/TxyKRIEKP+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_end=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 21:15 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>4.206495</th>\n",
       "    <th>4.067840</th>\n",
       "    <th>0.292376</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('fit_head')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('fit_head');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To complete the fine-tuning, we can then unfeeze and launch a new training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 3:59:03 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>3.949835</th>\n",
       "    <th>3.892163</th>\n",
       "    <th>0.309857</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>3.863976</th>\n",
       "    <th>3.823716</th>\n",
       "    <th>0.319508</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>3.827637</th>\n",
       "    <th>3.776987</th>\n",
       "    <th>0.325163</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>3.782025</th>\n",
       "    <th>3.738658</th>\n",
       "    <th>0.329756</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>3.704519</th>\n",
       "    <th>3.705750</th>\n",
       "    <th>0.333159</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>3.658038</th>\n",
       "    <th>3.682820</th>\n",
       "    <th>0.335856</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>3.608049</th>\n",
       "    <th>3.662626</th>\n",
       "    <th>0.338366</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>3.552923</th>\n",
       "    <th>3.648938</th>\n",
       "    <th>0.340119</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>3.518708</th>\n",
       "    <th>3.642814</th>\n",
       "    <th>0.340864</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>3.480421</th>\n",
       "    <th>3.641818</th>\n",
       "    <th>0.340905</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(10, 1e-3, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('fine_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How good is our model? Well let's try to see what it predicts after a few given words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('fine_tuned');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = \"i liked this movie because\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i liked this movie because it was clearly a movie . xxmaj so i gave it a 2 out of 10 . xxmaj so , just say something . xxbos xxmaj this is a really stunning picture , light years off of the late xxmaj\n",
      "i liked this movie because it would be a good one for those who like deep psychological and drama and you can go see this movie if you like a little slow motion and some magic should n't be there . i would give it\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to save the model but also it's encoder, the part that's responsible for creating and updating the hidden state. For the next part, we don't care about the part that tries to guess the next word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll create a new data object that only grabs the labelled data and keeps those labels. Again, this line takes a bit of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.IMDB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = (TextList.from_folder(path, vocab=data_lm.vocab)\n",
    "             #grab all the text files in path\n",
    "             .split_by_folder(valid='test')\n",
    "             #split by train and valid folder (that only keeps 'train' and 'test' so no need to filter)\n",
    "             .label_from_folder(classes=['neg', 'pos'])\n",
    "             #label them all with their folders\n",
    "             .databunch(bs=bs))\n",
    "\n",
    "data_clas.save('tmp_clas')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = TextClasDataBunch.load(path, 'tmp_clas', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='90%'>  <col width='10%'>  <tr>\n",
       "    <th>text</th>\n",
       "    <th>target</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules</th>\n",
       "    <th>pos</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxmaj some have praised xxunk :- xxmaj the xxmaj lost xxmaj xxunk as a xxmaj disney adventure for adults . i do n't think so -- at least not for thinking adults . \\n\\n xxmaj this script suggests a beginning as a live - action movie , that struck someone as the type of crap you can not sell to adults anymore . xxmaj the \" crack staff \"</th>\n",
       "    <th>neg</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxunk ) is the developing world 's answer to xxmaj silence of the xxmaj lambs . xxmaj where ` xxmaj silence ' terrorized our peace of mind , ` xxmaj citizen ' exhausts and saddens us instead . xxmaj this dramatization of the xxmaj chikatilo case translates rather well , thanks to a xxmaj westernized friendship between two xxmaj rostov cops who become equals . \\n\\n citizenx may also</th>\n",
       "    <th>pos</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos 8 xxmaj simple xxmaj rules for xxmaj dating xxmaj my xxmaj teenage xxmaj daughter had an auspicious start . xxmaj the supremely - talented xxmaj tom xxmaj shadyac was involved in the project . xxmaj this meant that the comedy would be nothing less of spectacular , and that 's exactly what happened : the show remains one of the freshest , funniest , wittiest shows made in a</th>\n",
       "    <th>pos</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxmaj the vigilante has long held a fascination for audiences , inasmuch as it evokes a sense of swift , sure justice ; good triumphs over evil and the bad guy gets his deserts . xxmaj it is , in fact , one of the things that has made the character of xxmaj dirty xxmaj harry xxmaj callahan ( as played by xxmaj clint xxmaj eastwood ) so popular</th>\n",
       "    <th>pos</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_clas.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then create a model to classify those reviews and load the encoder we saved before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = text_classifier_learner(data_clas, drop_mult=0.5)\n",
    "learn.load_encoder('fine_tuned_enc')\n",
    "learn.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gc.collect();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl4VdW5+PHvm4mEQMKQACFhCPMkBIigooi2KlRlqNWC9SqO17b2Xu11vPqzLQ7V2l6HVq3zVBUVJ+qEEygiCEEhzFOYwpQAmSBz8v7+ODt4CBkO5OxzTpL38zzn8Zy11977XR6SN3utvdcSVcUYY4w5UWHBDsAYY0zzZonEGGNMk1giMcYY0ySWSIwxxjSJJRJjjDFNYonEGGNMk1giMcYY0ySWSIwxxjSJJRJjjDFNEhHsAAIhISFBe/fuHewwjDGmWVm+fPl+VU1srF6rSCS9e/cmIyMj2GEYY0yzIiLbfalnXVvGGGOaxBKJMcaYJrFEYowxpkkskRhjjGkSSyTGGGOaxBKJMcaYJrFEYowxpkkskTTg87X7eH3pjmCH0ShVZf76HDKz84MdijGmFWoVDySeqNnLdrJwUy5jUjvRN7FdsMM5hqqyYEMuf/tsA6t3FRITGc6/rhnD6F6dgh2aMaYVsSuSBtw/bRhtIsK4dU4mVdUa7HCOsnjLAX7+5Ldc+eIyCkoquHfqMLrFR3PlC8tYt6cw2OEZY1oRSyQN6BIXzR8nD2X59jxe/HZbsMMBICv3ENe8lMGMZ5awt6CU+6edxJf/M4HLTunFK1ePoW1UBJc/v5TtBw4HO1RjTCthiaQR00Ym85NBXXho3nq27g/eL+eCkgr+OHcN5z78NUuyDnDrxIHMv3kCl47tSWS452tM6diWV64eQ2VVNZc99x37CkuDFq8xpvWwRNIIEeH+n59EVHgYt85ZSbUfuriqq5WFm3L5v083UFha4VP9619ZzsuLt3HJyT2Yf/MEfjOhH9GR4cfU7d+1PS9eOYaDh8q59Jkl5FgyMca4zBKJD7rGRXP3hUNZtq1pXVx7C0r5+xebGP/QfP7juaU89uVmrnkxg9KKqgb3e3XpDhZnHeC+aSdx/7STSGzfpsH6I3p04PmZJ7OnoJQZlkyMMS5zNZGIyEQR2SAim0Xk9jq2PywiK5zXRhHJ99pW5bVtrld5qoh8JyKbROQNEYlysw01Lhrl6eJ64JP1rN97/IPZX67fx/i/zOdvn22kZ6e2PDo9jb9dPIJl2w/ym1e/p6Kqus79dh4s5s8freOM/glMP7mHz+cb26czL145xpKJMcZ1riUSEQkHHgcmAUOAGSIyxLuOqt6kqmmqmgb8HXjHa3NJzTZVnexV/iDwsKr2B/KAq91qgzcR4cFfDCcuOpLfvfYDJeUNX0V4W7Ezn9+++gMDurVjwc0TeO3aU5iSlsxFo1O4Z8owvlyfw81vHdttVl2t3DonkzARHrhoOCJyXDGPSe10JJlMt2RijHGJm1ckY4DNqpqlquXAbGBKA/VnAK83dEDx/CY9G5jjFL0ETPVDrD5JaNeGh385gk05h7j3w7U+7bN1/2GuenEZCe2jeGHmGHonxB61/bJTenHLeQN5f8Vu7nxvFdl5xUe2veZ0ad15/mCSO8ScUMw1yWRvQSlXvbSM4vLKEzqOMcbUx81Ekgzs9Pqc7ZQdQ0R6AanAl17F0SKSISJLRKQmWXQG8lW15rdhvcd0yxn9E/nP8X149bsdfLJ6T4N1c4vKuOL5pQC8fNXYesc2fjOhL/85vg+vL93J6Q/O54y/fMktb608oS6tuoxJ7cQ/Lh3J2t2F3PTGCr/cMGCMMTXcTCR19cPU9xtsOjBHVb37i3qqajpwKfCIiPQ9nmOKyHVOIsrIzc09nrgb9T/nDmR4Sjy3vX30FYS3LbmHmPnCUnKLynjuinRSa12J1IqVO342mE9uPIO7LxjCoG5xzFuzl/CwE+vSqsvZg7py5/lDmLdmHw99uqHJxzPGmBpuTpGSDXj/KZ0C7K6n7nTgt94Fqrrb+W+WiCwARgJvAx1EJMK5Kqn3mKr6NPA0QHp6ul//BI+KCOPR6SO54LGFnPvw10w/uSdXn5FKcocY8ovLefSLTbyyeDvRkeE88atRjOzZ0afjDuoWx6BucVx1eipV1UpZZRVto/z3FV01rjdbcg/x5IIt9EmI5eL0pl3pGGMMgKi6080hIhHARuAnwC5gGXCpqq6pVW8gMA9IVScYEekIFKtqmYgkAIuBKaq6VkTeAt5W1dki8k8gU1WfaCiW9PR0zcjI8HcT2bSviCcXbGHuyt0o8JNBXVi67SCFJRVMH9OT358zgIR2Dd+qG2gVVdXMfGEpS7ceZFhyPMVlVRwqq6SqWvnHpSNJ723zdBljPERkudMz1HA9txKJE8TPgEeAcOB5Vb1PRGYBGao616nzRyBaVW/32u804CmgGk/32yOq+pyzrQ+egftOwA/AZapa1lAcbiWSGrvyS3j+m628lbGTk1Liuev8IQxOinPtfE1VUFzBne+tIr+4gtg24cRGRbBw836S4qN57zfjCAtreleaMab5C4lEEircTiQtwZzl2dz81koemzGSySO6BzscY0wI8DWR2JPtBvDMKTY4KY6H5q2nrNL3Z2SMMcbWIzEAhIcJd0waxOXPL+VfS3Zw9empddarqlaWbj3I0q0HySsup6CkgoKSCpLio7nr/CHERB07/5cxpmWzRGKOGD8gkTP6J/D3Lzfxi9EpxMdEAp4FtFbtKmDuit38O3M3+wo9Q1LtoyPo0DaSuOhI5m/IYeO+Ip6beTJx0ZHBbIYxJsAskZij3DZxEBf+4xueWLCZKSOS+SBzNx+u2sP2A8VEhgsTBnZhSlp3fjKo61FXHx9k7uamN1Yw/aklvHz1mJC7W80Y4x4bbDfH+P0bK3jnh12Ap8vrtL6dOf+kJCYNSyK+bf1XGws25HD9v5bTPT6GV64Ze8LTuhhjQoPdteXFEsnx2VdYygMfr2d0r45MGtaNzsdxdZGx7SBXvrgMFC49pSdXj0ulS1y0i9EaY9xiicSLJZLA2pxziEc+38hHq/YQERbGz0clc80ZfejXpV2wQzPGHAdLJF4skQTH9gOHeWZhFm9lZFNWWc1pfTvzH6f04qdDuh5ZHtgYE7oskXixRBJc+w+V8caynbz23Q525ZfQpX0brj49lStO613ncsHGmNBgicSLJZLQUFWtLNiQw4vfbmPhpv2kdIzh9kmDOP+kJL/McGyM8S9LJF4skYSebzbt594P17J+bxGjenbg56NS6JMQS++EWLrFRbP1wGG+yzrId1sP8MOOfIpKKyivrKa8qhoR4YKTkrjq9FSGJcfXe4784nL++VUW3209QIeYSDrFtqFTbCQDurbn/OFJfp1Z2ZiWyBKJF0skoamqWpmzfCd/+3QjOUU/zrsZHiZUOYtvJbZvw8m9O9IpNoo2EeFERYSRX1zB3BW7OFxexZjUTvzHKb0YlhxPSscYIsPDKC6v5IVF2/jnV1s4VFZJeq+OlFZUc/BwOQcOl1FaUU376AguGpXCZaf0spsAjKmHJRIvlkhCW3W1sq+olK25h9l64DA7DhbTu3MsY1M7kZoQW2e3V0FJBW9l7OSFRdvYlV8CeBJQcocYDpdVcuBwOT8d3JWbzxvAoG4/zsSsqizfnscrS7bz0ao9VFQp4/p15prT+3DmgESb+dgYL5ZIvFgiabkqq6pZmZ3P1v3FbD9wmG0HiqmsquaaM/owulfDC4rV3ATw8uJt7Csso29iLFef3odpI5NtzjBjsERyFEskpiHlldV8tGoPzyzMYs3uQtq1iWDisG5MG5nMKX06E25XKaaVskTixRKJ8YWqZ2bjOcuz+WT1XorKKuka14aTe3eid2fPjQB9EmNJS+lgXWCmVbBE4sUSiTlepRVVfLEuh3+v3M26vYVk55UcuQHgP07pxT1ThwU5QmPc52sisfsfjalDdGQ45w9P4vzhSYBnrfvsvBKeXZjFK0u2c9agRM4e1DXIURoTGmyeCmN8EBkeRmpCLHdfOIRB3dpz65xM9h8qa3xHY1oBSyTGHIc2EeE8Mj2NwpJKbn97Fd5dw0WlFXy+dh/lldVBjNCYwLOuLWOO06Bucdw6cSD3friON5bt5IwBiby4aCuzl+6kqKySC4Yn8ej0kXa3l2k1XE0kIjIReBQIB55V1QdqbX8YOMv52BbooqodRCQNeBKIA6qA+1T1DWefF4EzgQJnv5mqusLNdhhT21XjUpm/IYe75645Mgj/s5OS6B4fzVNfZxEXE8l9U4fZHGKmVXAtkYhIOPA4cA6QDSwTkbmquramjqre5FX/d8BI52MxcLmqbhKR7sByEZmnqvnO9ltUdY5bsRvTmLAw4a8Xj+C3r37PqJ4dufL01CMrQoaFCU8u2EJ8TCS3TRwU5EiNcZ+bVyRjgM2qmgUgIrOBKcDaeurPAP4AoKobawpVdbeI5ACJQH49+xoTcEnxMbzzm3HHlN963kAKSyqOJJPrz+wbhOiMCRw3B9uTgZ1en7OdsmOISC8gFfiyjm1jgChgi1fxfSKSKSIPi0id68CKyHUikiEiGbm5uSfaBmOOm4gwa8owLhzRnQc+Xs+/V+4OdkjGuMrNRFJX53B9Tz9OB+aoatVRBxBJAl4BrlTVmlth7gAGAScDnYDb6jqgqj6tqumqmp6YmHgi8RtzwsLDhL9dPIL0Xh25dU4m6/cWBjskY1zjZiLJBnp4fU4B6vvTbDrwuneBiMQBHwJ3qeqSmnJV3aMeZcALeLrQjAk5URFhPPGrUbSPjuC6l5eTX1we7JCMcYWbiWQZ0F9EUkUkCk+ymFu7kogMBDoCi73KooB3gZdV9a1a9ZOc/wowFVjtWguMaaIucdE8edlo9hSU8F+zVxy5w8uYlsS1RKKqlcANwDxgHfCmqq4RkVkiMtmr6gxgth496dclwHhgpoiscF5pzrZXRWQVsApIAO51qw3G+MPoXh350+RhfL0xl79+uiHY4RjjdzZpozEBcsc7mby+dCdvXHcKY/t0DnY4xjTK10kbbYoUYwLk/10whOQOMdz13mqbRsW0KJZIjAmQtlERzJoylE05h3j2m6xgh2OM31giMSaAfjK4K+cN7cpjX2xi58HiYIdjjF9YIjEmwP44eSjhIvy/91fTGsYoTctnicSYAEuKj+GmcwawYEMun6zeG+xwjGkySyTGBMHM03ozOCmOW+ZkctucTOZvyLEBeNNs2XokxgRBRLjnqfdHP9/Ih6v28EbGTtpHR3DFqb25+byBwQ7PmONiicSYIElNiOWR6SMpq6xi0eb9zF66k3/M38yw5DgmDksKdnjG+My6towJsjYR4Zw9qCtP/GoUw5LjuOu9NTYvl2lWLJEYEyIiwsN48KLh5BeXc++H64IdjjE+s0RiTAgZ2j2e68/sy5zl2Xy10dbRMc2DJRJjQswNZ/ejb2Is//vOKg6VVQY7HGMaZYnEmBATHRnOX34xnN0FJdz34Vp7aNGEPEskxoSg0b06cf2ZfXl96U4e+XxTsMMxpkF2+68xIeqWcweyv6iMR7/YRLs2EVw7vk+wQzKmTpZIjAlRYWHCAxcNp7i8ivs+WkdsmwguHdsz2GEZcwxLJMaEsPAw4eFfpnG4vJI731tFbJtwpqQlBzssY45iYyTGhLioiDD+edloxvTuxO/fXMlna/cFOyRjjmKJxJhmIDoynGevSGdY9zh++9r3LNq8P9ghGXOEJRJjmon20ZG8eOUYUjvHcu3LGXy/Iy/YIRkDuJxIRGSiiGwQkc0icnsd2x8WkRXOa6OI5Httu0JENjmvK7zKR4vIKueYj4mIuNkGY0JJx9goXrl6DF3at2Hm80tZt6cw2CEZ414iEZFw4HFgEjAEmCEiQ7zrqOpNqpqmqmnA34F3nH07AX8AxgJjgD+ISEdntyeB64D+zmuiW20wJhR1iYvmX9eMJToynNvfzrQHFk3QuXlFMgbYrKpZqloOzAamNFB/BvC68/484DNVPaiqecBnwEQRSQLiVHWxen56XgamutcEY0JTSse23HLeQFZmF9gqiybo3EwkycBOr8/ZTtkxRKQXkAp82ci+yc57X455nYhkiEhGbq5Nfmdanp+PSqF/l3Y89OkGKqtsdUUTPG4mkrrGLuq7Bp8OzFHVqkb29fmYqvq0qqaranpiYmKjwRrT3ISHCTefN5Cs3MPMWZ7d+A7GuMTNRJIN9PD6nALsrqfudH7s1mpo32znvS/HNKbFO3dIV9J6dOCRzzdRWlHV+A7GuMDNRLIM6C8iqSIShSdZzK1dSUQGAh2BxV7F84BzRaSjM8h+LjBPVfcARSJyinO31uXA+y62wZiQJiLcNnEQewtLeXnxtmCHY1op1xKJqlYCN+BJCuuAN1V1jYjMEpHJXlVnALPV69YTVT0I3IMnGS0DZjllAL8GngU2A1uAj91qgzHNwal9O3PmgEQen7+FgpKKYIdjWiFpDbcOpqena0ZGRrDDMMY1q3cVcMHfv+H6M/ty+6RBwQ7HtBAislxV0xurZ0+2G9MCDEuO5+cjk3l+0VZ2HiwOdjimlbFEYkwLccvEgYQJPPDx+mCHYloZSyTGtBBJ8TFcf2ZfPly1h2XbDja+gzF+YonEmBbkuvF96BYXzax/r6W6uuWPf5rQYInEmBakbVQEt04cyKpdBbz7w65gh2OCqLSiisLSwNzFZ4nEmBZmaloyI1Li+cu89RSXVwY7HBMkX67PYfgfPw3IDNGWSIxpYcLChLsvHMK+wjKeXLAl2OGYIMnO89y9171DjOvnskRiTAs0ulcnJo/ozlNfZ9ntwK3UrrwS2kdHEB8T6fq5LJEY00Ld8bNBhItw34frgh2KCYLsvBJSOrYNyLkskRjTQiXFx/CbCX35ZM1evrU13ludXfklJAegWwsskRjTol07vg8pHWP407/X2polrYiqOlcklkiMMU0UHRnOXecPZsO+Il79bkewwzEBUlBSwaGySkskxhj/OG9oN8b168z/fbaRvMPlwQ7HBEB2XgmAJRJjjH+ICHdfMJTC0gqeX7Q12OGYAKhJJMkdbLDdGOMnA7u157wh3Xjp220UBehpZxM8u/LtisQY44JfT+hLYWklry+1sZKWLjuvmLZR4XRo6/4zJOBjIhGRviLSxnk/QUT+S0Q6uBuaMcafRvTowLh+nXl24VbKKm1995Zsl3PHlmdFcvf5ekXyNlAlIv2A54BU4DXXojLGuOLXZ/Yjp6iMd763CR1bsuy8wD1DAr4nkmpnDfZpwCOqehOQ5F5Yxhg3jOvXmeEp8Tz11RaqbJr5FmtXfuCeagffE0mFiMwArgA+cMoC0/lmjPEbEeHXZ/Zl24FiPl69J9jhGBcUllZQUFIRsIF28D2RXAmcCtynqltFJBX4V2M7ichEEdkgIptF5PZ66lwiImtFZI2IvOaUnSUiK7xepSIy1dn2oohs9dqW5mMbjDF4nivpkxjLE/O3oGpXJS3Nrppbf0MtkajqWlX9L1V9XUQ6Au1V9YGG9hGRcOBxYBIwBJghIkNq1ekP3AGMU9WhwI3O+earapqqpgFnA8XAp1673lKzXVVX+NZUYwx4ppm//sy+rN1TyHPf2HMlLc2uIw8jhljXlogsEJE4EekErAReEJH/a2S3McBmVc1S1XJgNjClVp1rgcdVNQ9AVXPqOM4vgI9V1ebCNsZPfjEqhfOGduX+j9Yxf31dP3amuapZhyQUB9vjVbUQ+DnwgqqOBn7ayD7JwE6vz9lOmbcBwAARWSQiS0RkYh3HmQ68XqvsPhHJFJGHa25LNsb4LixMePiXaQzqFsfvXv+BjfuKgh2S8ZNd+SW0iQgjoV1UwM7payKJEJEk4BJ+HGxvTF03MNfukI0A+gMTgBnAs97PpzjnPAmY57XPHcAg4GSgE3BbnScXuU5EMkQkIzc318eQjWk92kZF8NzMdGKiwrn6pWUcOFQW7JCMH2TnlZAcwGdIwPdEMgvPL/MtqrpMRPoAmxrZJxvo4fU5BdhdR533VbVCVbcCG/AklhqXAO+q6pE5HVR1j3qUAS/g6UI7hqo+rarpqpqemJjoQxONaX2S4mN45vJ0cgrLuP5fy+2W4BYg0Lf+gu+D7W+p6nBV/bXzOUtVL2pkt2VAfxFJFZEoPF1Uc2vVeQ84C0BEEvB0dWV5bZ9BrW4t5yoF8aTbqcBqX9pgjKlbWo8O3DN1GMu25fHpmr3BDsc0UaAfRgTfB9tTRORdEckRkX0i8raIpDS0j/MA4w14rmTWAW+q6hoRmSUik51q84ADIrIWmI/nbqwDzjl747mi+arWoV8VkVXAKiABuNeXNhhj6nfRqBR6dIqxu7iaueLySg4eLg/oMyTgGaPwxQt4pkS52Pl8mVN2TkM7qepHwEe1yu72eq/A751X7X23cezgPKp6to8xG2N8FB4mzDwtlXs+WMuKnfmk9bCp9JqjXQFeh6SGr2Mkiar6gqpWOq8XARt4MKYFuSQ9hXZtIuyqpBkL9IJWNXxNJPtF5DIRCXdelwEH3AzMGBNY7aMj+eXJPfho1R52O+tZmOYlOz/wDyOC74nkKjx3UO0F9uB5SPBKt4IyxgTHzNN6o6q8tHhbsEMxJyA7r5io8DAS2wX28Tpf79raoaqTVTVRVbuo6lQ8DycaY1qQHp3aMnFYN17/bgeHyyqDHY45TrvySujeIZqwsMA9QwJNWyHxmAFyY0zzd/XpqRSWVvL299nBDsUcp5qHEQOtKYkksCnPGBMQo3p2ZESPDjz3zVYqqqqDHY45DrvyS0jpENjxEWhaIrFHYI1pgUSE353Vj+0Hinlx0bZgh2N8VFpRRW5RWehdkYhIkYgU1vEqAroHKEZjTID9ZHAXzh7UhYc/38ieAruDqznYlR+cW3+hkUSiqu1VNa6OV3tV9fVhRmNMMyMi/PHCoVRVK/d+sC7Y4RgfvP/DLgAGdG0f8HM3pWvLGNOC9ezclhvO6seHq/bw1UabQTuUrdldwBMLtjBtZDLDkuMDfn5LJMaYel13Zh9SE2L5w/urKa2oCnY4pg4VVdXc/FYmHdpG8YcLhzS+gwsskRhj6tUmIpw/TR7KtgPFPP11VuM7mIB7Yv4W1u0p5P5pw+jQNnCLWXmzRGKMadD4AYmcf1ISj8/fbAPvIWbdnkL+MX8Tk0d059yh3YIWhyUSY0yjbp80CAUemrch2KEYR2VVNbfMWUl8TCR/nDw0qLFYIjHGNKpHp7ZcOa437/6wi9W7CoIdjgFW7Mxn9a5Cbp80mE6xwenSqmGJxBjjk99M6EeHmEju+3AdnqWETDB9vyMPgAkDg7+ihyUSY4xP4mMiufGnA1icdYAv1+cEO5xWb/n2PHp1bktCgGf6rYslEmOMzy4d25M+CbHc/9E6Km0erqBRVZZvz2dUz47BDgWwRGKMOQ6R4WHcPmkQW3IP8/qyncEOp9XKzith/6EyRvWyRGKMaYbOGdKVsamdeOiT9UfmdzKBtXy7Z3xkdGu4IhGRiSKyQUQ2i8jt9dS5RETWisgaEXnNq7xKRFY4r7le5aki8p2IbBKRN0QkuLcrGNPKiAh/+cVwqhVunP0DVdU28B5o3+/IIzYqnIHdAj+vVl1cSyQiEg48DkwChgAzRGRIrTr9gTuAcao6FLjRa3OJqqY5r8le5Q8CD6tqfyAPuNqtNhhj6tarcyz3TB3Ksm15/OPLzcEOp9VZvj2PtJ4dCA/wSoj1cfOKZAywWVWzVLUcmA1MqVXnWuBxVc0DUNUGbwUREQHOBuY4RS8BU/0atTHGJ9NGpjA1rTuPfrGRjG0Hgx1Oq3G4rJL1e4tCZqAd3E0kyYD3aFy2U+ZtADBARBaJyBIRmei1LVpEMpzymmTRGchX1ZrFpOs6pjEmQO6ZOoyUjm3579krKCipCHY4rcLK7HyqqjVkBtrB3URS1zVX7c7UCKA/MAGYATwrIh2cbT1VNR24FHhERPr6eEzPyUWucxJRRm6uTYFtjBvaR0fy6PQ09hWW8rvXf6C4vLLxnUyT/LAjH4BRPVpHIskGenh9TgF211HnfVWtUNWtwAY8iQVV3e38NwtYAIwE9gMdRCSigWPi7Pe0qqaranpiYvCf/DSmpRrZsyP3TRvGN5tyufSZ7zh4uDzYIbVoy7fn0a9LO+LbRgY7lCPcTCTLgP7OXVZRwHRgbq067wFnAYhIAp6uriwR6SgibbzKxwFr1TMvw3zgF87+VwDvu9gGY4wPfnlyT568bDTr9hRy0ZPfsvNgMQBrdxdy13urSJv1Kb96dgk/ONN6mBOjqny/Iy9kbvut4dpyuapaKSI3APOAcOB5VV0jIrOADFWd62w7V0TWAlXALap6QEROA54SkWo8ye4BVV3rHPo2YLaI3Av8ADznVhuMMb47b2g3Xr1mLFe/lMG0J74lpWMMK3bm0yYijJ8M7sJ3WQeZ9sS3nDukKzefN5Cu7aPZV1TK3oJSikorGT8ggfbRofNXdijK2n+Y/OIKRvXq0HjlAJLWMPlaenq6ZmRkBDsMY1qFzTlFzHxhGW0iwrh0bC8uGpVMh7ZRHC6r5PlvtvL011kUlR07lpLQLoobfzqA6Sf3ICLcnpWuy5sZO7l1Tiaf/348/bq4/wyJiCx3xqobrmeJxBjjb9XVSlg9zzjkHS5nzvJsRKBrXDRd46KprK7mkc82sXTbQfp1acedPxvMWYO6BDjq0HfHO5l8mLmHFXefW+//X3/yNZG41rVljGm9Gvol1zE2imvH9zmm/NQ+nfl07T4e+Hg9V764jL/PGMmFI7q7GWazs3x7HqN6dQxIEjkedv1ojAkJIsJ5Q7sx78bxpPXowF3vrWZfYWmwwwoZxeWVbMo5xMgQuu23hiUSY0xIiYoI4/8uGUFZZRW3zsm0RbQcewtKUYUenWKCHcoxLJEYY0JOn8R2/O/PBvPVxlxeW7oj2OGEhJyiMgC6tI8OciTHskRijAlJl43txRn9E7j3g3Vs23842OEE3ZFEEhf8FRFrs0RijAlJYWGe6eojwoX/eWsl1a18uvocZ7yoq12RGGOM75LiY7jr/MEs357HkqwDwQ6mDesdAAAUA0lEQVQnqHKLyoiKCCMuJvRutrVEYowJaZNHJBMbFc7clXVOq9dq7CsspUv7NnhW0wgtlkiMMSEtJiqcc4Z05ePVeymvrA52OEGTU1RGl/ahNz4ClkiMMc3A5LTuFJRUsHBT610SwpNIQm98BCyRGGOagdP7JdKhbWSr7t7KKSylawjesQWWSIwxzUBURBiThiXx2dp9lJRXBTucgCutqKKwtJIucXZFYowxJ2zyiO4Ul1fx+bp9wQ4l4HIKPc+QJNoYiTHGnLgxqZ3oGtemVXZv5RR5niGxwXZjjGmC8DDhguHd+WpDLgUlFcEOJ6BCeXoUsERijGlGJo/oTnlVNfNW7w12KAF15Kl2G2w3xpimGZ4ST6/ObVtd91ZOURkRYULHtlHBDqVOlkiMMc2GiDA1LZlFW/azeEvrmTIlp6iMxPZtQm5BqxqWSIwxzcp14/uQ2jmWm95YQd7h8mCHExA106OEKlcTiYhMFJENIrJZRG6vp84lIrJWRNaIyGtOWZqILHbKMkXkl171XxSRrSKywnmludkGY0xoiW0TwWMzRnLwcDm3zFl51MJXBw+Xc+3LGYy9/3OueH4pD36yng8yd3OwmSec3KIyEkN0oB1cXLNdRMKBx4FzgGxgmYjMVdW1XnX6A3cA41Q1T0S6OJuKgctVdZOIdAeWi8g8Vc13tt+iqnPcit0YE9qGJcdz+6RBzPpgLS99u42Z41L5YUcev331e/YfLufcIV3ZknuYRV9nUVmtJLSL4snLRnNy707BDv2E5BSVMbpX6C2xW8PN+YjHAJtVNQtARGYDU4C1XnWuBR5X1TwAVc1x/ruxpoKq7haRHCARyMcYY4Arx/Xmm837uf+j9ewrKuPZhVl0jYvmnV+fxrDkeADKKqtYlV3ALXMyufSZJfxp8jAuHdszyJEfn/LKag4eLg/ZW3/B3a6tZGCn1+dsp8zbAGCAiCwSkSUiMrH2QURkDBAFbPEqvs/p8npYREK349AY4xoR4aFfDKdD20ieXLCF8f0T+fB3ZxxJIgBtIsJJ792J9347jtP6JvC/767izndXNatZhPcfCt2VEWu4eUVS1+0FtZc4iwD6AxOAFGChiAyr6cISkSTgFeAKVa355u8A9uJJLk8DtwGzjjm5yHXAdQA9ezavv0CMMb7p3K4NL101htW7CrhoVEq9dzXFx0Ty/MyT+cu89Tz1VRardxfy6C/T6J0QG+CIj9+PDyOGbiJx84okG+jh9TkFqH3zdzbwvqpWqOpWYAOexIKIxAEfAnep6pKaHVR1j3qUAS/g6UI7hqo+rarpqpqemJjot0YZY0LL4KQ4Lk7v0eitseFhwh2TBvP4paPYmnuInz22kDeX7TxqsD4U7SusmR6ldXZtLQP6i0iqiEQB04G5teq8B5wFICIJeLq6spz67wIvq+pb3js4VymIZ5mwqcBqF9tgjGlhzh+exCc3jmd4Sjy3vp3Jr//1/ZHuo1BUc0USqk+1g4uJRFUrgRuAecA64E1VXSMis0RkslNtHnBARNYC8/HcjXUAuAQYD8ys4zbfV0VkFbAKSADudasNxpiWqXuHGF675hTumDSIL9bv44wH5/Pnj9dxIAQTSm5hKWHi6cYLVRLql3X+kJ6erhkZGcEOwxgTgrbkHuLvX2zi/ZW7iYkM5/JTe3PtGakh84v79rcz+WJ9Dsvu/GnAzy0iy1U1vbF69mS7MaZV65vYjkemj+Szm87knCFdeerrLZz+4Hzu/2gduUXBv0IJ5bXaa1giMcYYoF+XdjzqJJSJw7rx7MIszvjLl9zzwVqKSoM3bX2oT48ClkiMMeYo/bq04+FfpvHF/0zgguHdefHbbVz6zHdBGz/JKSqja4gusVvDEokxxtQhNSGWv148gmcuH83GfUVc8tRidueXBDSGqmrlwCHr2jLGmGbt7EFdeeXqseQUlnHxPxeTlXsoYOc+cKiMaoVEuyIxxpjmbUxqJ16/7hRKK6q4+J+L2VMQmCuT5vBUO1giMcYYnwxLjmf2dadQUFLBswu3BuScOUU1T7VbIjHGmBahf9f2XDA8idlLd1BQ4v6dXPsKa55qt64tY4xpMa45ow+Hy6uYvXSH6+fKcRJJQog8HFkfSyTGGHMchiXHc1rfzrywaJvr09HnFJXSKTaKqIjQ/lUd2tEZY0wIunZ8H/YWlvLhqtoTmvtXc3iqHSyRGGPMcZswIJH+Xdrx9NdbXZ2GPqeojERLJMYY0/KICNee0Yd1ewpZtPmAa+fJLSwN+YF2sERijDEnZMrI7iS0a8PTC7NcOf7eglL2FJbSu3NbV47vT5ZIjDHmBLSJCGfmab34emMuy7Yd9PvxP8jcjSqcP7y734/tb5ZIjDHmBF05LpWUjjHcNieT0ooqvx77vRW7GJEST2ozWFfeEokxxpyg2DYR/PnnJ5G1/zCPfL7Jb8fdnHOI1bsKmZyW7LdjuskSiTHGNMEZ/RO5JD2FZxZmsSq74Lj2/deS7byw6NjpVt5fsYswgQtHJPkrTFdZIjHGmCa68/whdI6N4pY5K4/rIcUnF2zhng/Wkpmdf6RMVXl/xW7G9UugS/vQv2MLLJEYY0yTxcdEct+0k1i/t4h/frXFp332HypjV34J1Qr/++4qKqs8Cej7HfnsOFjMlGbSrQWWSIwxxi/OGdKVC0d05x9fbubg4fJG69dchVx+ai9W7yrkpcXbAZi7YhdtIsI4b2hXV+P1J1cTiYhMFJENIrJZRG6vp84lIrJWRNaIyGte5VeIyCbndYVX+WgRWeUc8zERETfbYIwxvvrP8X0or6rm0zV7G62bmV2ACNw6cRBnDUzkb59uYOfBYj7I3MNPB3elfXRkACL2D9cSiYiEA48Dk4AhwAwRGVKrTn/gDmCcqg4FbnTKOwF/AMYCY4A/iEhHZ7cngeuA/s5rolttMMaY4zG0exw9O7Xlw1V7Gq2bmV1Av8R2tGsTwawpw6hW5VfPfseBw+VMSQv9Z0e8uXlFMgbYrKpZqloOzAam1KpzLfC4quYBqGqOU34e8JmqHnS2fQZMFJEkIE5VF6tngpuXgakutsEYY3wmIpw/PIlvtxxosHtLVcnMzmd4SgcAenRqy00/HcCOg8XEx0QyYWCXQIXsF24mkmRgp9fnbKfM2wBggIgsEpElIjKxkX2TnfcNHRMAEblORDJEJCM3N7cJzTDGGN+df1ISVdXaYPfW7oJS9h8qZ3hK/JGyq05PZUxqJ341tmfITxtfW4SLx65r7KL2NJkReLqnJgApwEIRGdbAvr4c01Oo+jTwNEB6erp703MaY4wX7+6t6WN61llnlTPQ7p1IIsPDePM/Tw1IjP7mZtrLBnp4fU4Bak/enw28r6oVqroV2IAnsdS3b7bzvqFjGmNM0PjSvbUyu4CIMGFwUlyAo3OHm4lkGdBfRFJFJAqYDsytVec94CwAEUnA09WVBcwDzhWRjs4g+7nAPFXdAxSJyCnO3VqXA++72AZjjDlujXVvZWbnMyipPdGR4QGOzB2uJRJVrQRuwJMU1gFvquoaEZklIpOdavOAAyKyFpgP3KKqB1T1IHAPnmS0DJjllAH8GngW2AxsAT52qw3GGHMiGrp7q7paycwuODLQ3hK4OUaCqn4EfFSr7G6v9wr83nnV3vd54Pk6yjOAYX4P1hhj/KSme+vpr7PIO1xOx9ioI9u2HyymqLSS4cnxDRyheWletwYYY0wzUdO9Na9W91bmkYH2lnNFYonEGGNcUF/31sqdBURHhjGga7sgReZ/lkiMMcYFIsJFo1JYuGk/73z/4+Nvmdn5DO0eT0R4y/n123JaYowxIeY3Z/XltL6duf3tVWRsO0hlVTWrdxcc9fxIS2CJxBhjXBIZHsYTvxpFcscYrntlOfM35FJaUc2IFjQ+ApZIjDHGVR3aRvHcFelUVSu/fe17AE6yKxJjjDHHo09iO5781Siqq5X2bSJI7Rwb7JD8ytXnSIwxxnic1i+Bv88YyYHD5YSFtaxllCyRGGNMgEw6KSnYIbjCuraMMcY0iSUSY4wxTWKJxBhjTJNYIjHGGNMklkiMMcY0iSUSY4wxTWKJxBhjTJNYIjHGGNMk4lmksGUTkVxge63ieKCgkTLvz429TwD2n0B4dcXhax1/tMG7rLm34UTjbyg+X+o0FG9Dn/3576ih+BrbHio/Cw3F2Nh2+1lw52ehl6omNrqHqrbKF/B0Y2Xenxt7D2T4Kw5f6/ijDbXKmnUbTjR+f7fB18/+/HfkSxvc/A5aQhvsZ8H3NtR+teaurX/7UPbv43zvrzh8reOPNjQ1fl+P0Zra4Otnf/478uUYbn4HvpzfF8FsQ6j9O6qrLFTacJRW0bUVCCKSoarpwY6jKZp7G5p7/GBtCBXNvQ2Bjr81X5H429PBDsAPmnsbmnv8YG0IFc29DQGN365IjDHGNIldkRhjjGkSSyR1EJHnRSRHRFafwL6jRWSViGwWkcdERLy2/U5ENojIGhH5i3+jPioGv8cvIn8UkV0issJ5/cz/kR8VhyvfgbP9ZhFREUnwX8R1xuHG93CPiGQ638GnItLd/5EfFYcbbXhIRNY77XhXRFxbwNyl+C92foarRcS1cYimxF7P8a4QkU3O6wqv8gZ/XnxyoreIteQXMB4YBaw+gX2XAqcCAnwMTHLKzwI+B9o4n7s0s/j/CNzcnL8DZ1sPYB6e54oSmlsbgDivOv8F/LMZtuFcIMJ5/yDwYDOLfzAwEFgApIda7E5cvWuVdQKynP92dN53bKidx/OyK5I6qOrXwEHvMhHpKyKfiMhyEVkoIoNq7yciSXh+0Ber5xt6GZjqbP418ICqljnnyGlm8QeUi214GLgVcH1w0I02qGqhV9VYXG6HS234VFUrnapLgJRmFv86Vd3gVsxNjb0e5wGfqepBVc0DPgMm+utn3hKJ754Gfqeqo4GbgSfqqJMMZHt9znbKAAYAZ4jIdyLylYic7Gq0x2pq/AA3ON0Rz4tIR/dCrVeT2iAik4FdqrrS7UAb0OTvQUTuE5GdwK+Au12MtT7++LdU4yo8fwUHkj/jDzRfYq9LMrDT63NNe/zSTluz3Qci0g44DXjLq/uwTV1V6yir+YsxAs8l5SnAycCbItLH+SvAVX6K/0ngHufzPcDf8PwSCIimtkFE2gJ34ulWCQo/fQ+o6p3AnSJyB3AD8Ac/h1ovf7XBOdadQCXwqj9jbIg/4w+0hmIXkSuB/3bK+gEfiUg5sFVVp1F/e/zSTkskvgkD8lU1zbtQRMKB5c7HuXh+2XpfpqcAu5332cA7TuJYKiLVeObDyXUzcEeT41fVfV77PQN84GbAdWhqG/oCqcBK54cwBfheRMao6l6XY6/hj39H3l4DPiSAiQQ/tcEZ7L0A+Ekg/pjy4u/vIJDqjB1AVV8AXgAQkQXATFXd5lUlG5jg9TkFz1hKNv5op1sDRc39BfTGa5AL+Ba42HkvwIh69luG56qjZuDqZ0759cAs5/0APJeZ0oziT/KqcxMwu7l9B7XqbMPlwXaXvof+XnV+B8xphm2YCKwFEt2O3c1/R7g82H6isVP/YPtWPL0iHZ33nXxpp09xBuKLbG4v4HVgD1CBJ2Nfjeev2U+Alc4Pwd317JsOrAa2AP/gx4c+o4B/Odu+B85uZvG/AqwCMvH8xZbkVvxutaFWnW24f9eWG9/D2055Jp45kZKbYRs24/lDaoXzcu3OM5fin+YcqwzYB8wLpdipI5E45Vc5/+83A1cez89LYy97st0YY0yT2F1bxhhjmsQSiTHGmCaxRGKMMaZJLJEYY4xpEkskxhhjmsQSiWmVRORQgM/3rIgM8dOxqsQz++9qEfl3Y7PnikgHEfmNP85tTF3s9l/TKonIIVVt58fjReiPExG6yjt2EXkJ2Kiq9zVQvzfwgaoOC0R8pvWxKxJjHCKSKCJvi8gy5zXOKR8jIt+KyA/Ofwc65TNF5C0R+TfwqYhMEJEFIjJHPOttvFqztoNTnu68P+RMvLhSRJaISFenvK/zeZmIzPLxqmkxP05K2U5EvhCR78WzvsQUp84DQF/nKuYhp+4tznkyReRPfvzfaFohSyTG/OhR4GFVPRm4CHjWKV8PjFfVkXhm273fa59TgStU9Wzn80jgRmAI0AcYV8d5YoElqjoC+Bq41uv8jzrnb3S+I2d+qJ/gmWkAoBSYpqqj8Kx/8zcnkd0ObFHVNFW9RUTOBfoDY4A0YLSIjG/sfMbUxyZtNOZHPwWGeM2sGici7YF44CUR6Y9nZtRIr30+U1XvNSOWqmo2gIiswDNX0je1zlPOj5NeLgfOcd6fyo9rQbwG/LWeOGO8jr0cz9oS4Jkr6X4nKVTjuVLpWsf+5zqvH5zP7fAklq/rOZ8xDbJEYsyPwoBTVbXEu1BE/g7MV9VpznjDAq/Nh2sdo8zrfRV1/4xV6I+Dk/XVaUiJqqaJSDyehPRb4DE865MkAqNVtUJEtgHRdewvwJ9V9anjPK8xdbKuLWN+9Cme9T0AEJGa6brjgV3O+5kunn8Jni41gOmNVVbVAjzL7d4sIpF44sxxkshZQC+nahHQ3mvXecBVzvoWiEiyiHTxUxtMK2SJxLRWbUUk2+v1ezy/lNOdAei1eKb+B/gL8GcRWQSEuxjTjcDvRWQpkAQUNLaDqv6AZybY6XgWiEoXkQw8VyfrnToHgEXO7cIPqeqneLrOFovIKmAORycaY46L3f5rTIhwVnEsUVUVkenADFWd0th+xgSbjZEYEzpGA/9w7rTKJ4BLGRvTFHZFYowxpklsjMQYY0yTWCIxxhjTJJZIjDHGNIklEmOMMU1iicQYY0yTWCIxxhjTJP8fZs2+Ia2PoFMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total time: 02:46\n",
      "epoch  train_loss  valid_loss  accuracy\n",
      "1      0.294225    0.210385    0.918960  (02:46)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 2e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 04:03 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.507567</th>\n",
       "    <th>0.344481</th>\n",
       "    <th>0.855000</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 2e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('first')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('first');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total time: 03:03\n",
      "epoch  train_loss  valid_loss  accuracy\n",
      "1      0.268781    0.180993    0.930760  (03:03)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "learn.freeze_to(-2)\n",
    "learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('second')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('second');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total time: 04:06\n",
      "epoch  train_loss  valid_loss  accuracy\n",
      "1      0.211133    0.161494    0.941280  (04:06)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "learn.freeze_to(-3)\n",
    "learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('third')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('third');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total time: 10:01\n",
      "epoch  train_loss  valid_loss  accuracy\n",
      "1      0.188145    0.155038    0.942480  (05:00)\n",
      "2      0.159475    0.153531    0.944040  (05:01)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('pos', tensor(0), tensor([[9.9999e-01, 1.1991e-05]]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"I really loved that movie, it was awesome!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}