4. NLP/4.4 RNN with fastai2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center> Deep learning for text with Fast.ai v2\n",
    "\n",
    "### Installation\n",
    "```\n",
    "sudo pip install -U fastai2\n",
    "```\n",
    "\n",
    "### Reference notebook\n",
    "https://github.com/fastai/fastai2/blob/master/nbs/course/lesson3-imdb.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai2.text.all import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##### DOWNLOAD DATASET. ONLY IN COLAB !!!\n",
    "#!wget -P ./data \"https://raw.githubusercontent.com/SaturdaysAI-Murcia/machine-learning/master/4. NLP/data/movieReviews_trainLabel.tsv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11311_4</th>\n",
       "      <td>0</td>\n",
       "      <td>I vaguely remember Ben from my Sci-Fi fandom days of the '60s, I was doing several interviews &amp; bios of obscure actors/actresses, most notably Ben, actress Fay Spain, and Jody Fair, who played Angela in 1961's The Young Savages. Ben was one of the people at a low-key Sci-Fi con in Chicago, about 1970, when I had a nice chat with him and his \\career\\\" and life. All these were published in some now-long-forgotten fanzine of the day. Wish I still had copies of those interviews, but time marches on, and any of those people surely wouldn't' remember me at all so many years later. Ben was a real...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10118_7</th>\n",
       "      <td>1</td>\n",
       "      <td>I really like Miikes movies about Yakuza, this one I saw about 2 years ago and it really fu**ed my head. Never before seen such a sick and twisted thing. The Story is good and the actors do their thing very well. I haven't seen the UK or Japan version, but I have to say that I believe that the German DVD is a bit censored. If you haven't seen the movie already and live in Germany maybe you better look out for a DVD from the Nederlands or Austria. The I-ON DVD contains a lot of very hard and nasty scenes, but at the showdown I felt that something was missing, about one or two very short sce...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6764_4</th>\n",
       "      <td>0</td>\n",
       "      <td>NATURAL BORN KILLERS (1994)&lt;br /&gt;&lt;br /&gt;Cinema Cut: R &lt;br /&gt;&lt;br /&gt;Director's Cut: NC-17&lt;br /&gt;&lt;br /&gt;It's an unusual Oliver Stone picture, but when I read he was on drugs during the filming, I needed no further explanation. 'Natural Born Killers' is a risky, mad, all out film-making that we do not get very often; strange, psychotic, artistic pictures.&lt;br /&gt;&lt;br /&gt;'Natural Born Killers' is basically the story of how two mass killers were popularised and glorified by the media; there is a great scene where an interviewer questions some teenagers about Mickey and Mallory, and the teenager says 'M...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9728_3</th>\n",
       "      <td>0</td>\n",
       "      <td>Tobe Hooper has made great movies so I was certain this couldn't be BAD. I didn't read any reviews and tried to watch this unintentionally humorous film. At times this made me laugh, sometimes I almost fell asleep, sometimes made me almost CRY for Hooper.&lt;br /&gt;&lt;br /&gt;I rated this 3/10 because its 1990 \\horror\\\"-movie and many interesting or funny things happened there. Throughout the movie I was thinking something like \\\"they simply CAN'T add more things in this movie...\\\" .. but they did.&lt;br /&gt;&lt;br /&gt;Some tell this is some sort of Firestarter clone but truly isn't. It's based on that idea b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3052_1</th>\n",
       "      <td>0</td>\n",
       "      <td>This Santa movie starts off strange and I think Santa might be a pedo. Instead of the usual elf toy makers, this Santa has apparently kidnapped kids from all across the globe and makes them sing a bit like characters from \\It's a Small World\\\"! I guess there are no child labor laws on the weird astral plane on which he lives (it's apparently NOT the North Pole and not on Earth)!! None of these kids seem very happy and I kept wanting to see commandos break in and rescue the tykes, though I guess for some of the third world kids, these working conditions were perhaps an improvement over loca...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         label  \\\n",
       "id               \n",
       "11311_4      0   \n",
       "10118_7      1   \n",
       "6764_4       0   \n",
       "9728_3       0   \n",
       "3052_1       0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            text  \n",
       "id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \n",
       "11311_4  I vaguely remember Ben from my Sci-Fi fandom days of the '60s, I was doing several interviews & bios of obscure actors/actresses, most notably Ben, actress Fay Spain, and Jody Fair, who played Angela in 1961's The Young Savages. Ben was one of the people at a low-key Sci-Fi con in Chicago, about 1970, when I had a nice chat with him and his \\career\\\" and life. All these were published in some now-long-forgotten fanzine of the day. Wish I still had copies of those interviews, but time marches on, and any of those people surely wouldn't' remember me at all so many years later. Ben was a real...  \n",
       "10118_7  I really like Miikes movies about Yakuza, this one I saw about 2 years ago and it really fu**ed my head. Never before seen such a sick and twisted thing. The Story is good and the actors do their thing very well. I haven't seen the UK or Japan version, but I have to say that I believe that the German DVD is a bit censored. If you haven't seen the movie already and live in Germany maybe you better look out for a DVD from the Nederlands or Austria. The I-ON DVD contains a lot of very hard and nasty scenes, but at the showdown I felt that something was missing, about one or two very short sce...  \n",
       "6764_4   NATURAL BORN KILLERS (1994)<br /><br />Cinema Cut: R <br /><br />Director's Cut: NC-17<br /><br />It's an unusual Oliver Stone picture, but when I read he was on drugs during the filming, I needed no further explanation. 'Natural Born Killers' is a risky, mad, all out film-making that we do not get very often; strange, psychotic, artistic pictures.<br /><br />'Natural Born Killers' is basically the story of how two mass killers were popularised and glorified by the media; there is a great scene where an interviewer questions some teenagers about Mickey and Mallory, and the teenager says 'M...  \n",
       "9728_3   Tobe Hooper has made great movies so I was certain this couldn't be BAD. I didn't read any reviews and tried to watch this unintentionally humorous film. At times this made me laugh, sometimes I almost fell asleep, sometimes made me almost CRY for Hooper.<br /><br />I rated this 3/10 because its 1990 \\horror\\\"-movie and many interesting or funny things happened there. Throughout the movie I was thinking something like \\\"they simply CAN'T add more things in this movie...\\\" .. but they did.<br /><br />Some tell this is some sort of Firestarter clone but truly isn't. It's based on that idea b...  \n",
       "3052_1   This Santa movie starts off strange and I think Santa might be a pedo. Instead of the usual elf toy makers, this Santa has apparently kidnapped kids from all across the globe and makes them sing a bit like characters from \\It's a Small World\\\"! I guess there are no child labor laws on the weird astral plane on which he lives (it's apparently NOT the North Pole and not on Earth)!! None of these kids seem very happy and I kept wanting to see commandos break in and rescue the tykes, though I guess for some of the third world kids, these working conditions were perhaps an improvement over loca...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data/movieReviews_trainLabel.tsv\", sep=\"\\t\", index_col=\"id\")\n",
    "df.rename(columns={\"review\": \"text\", \"sentiment\": \"label\"}, inplace=True)\n",
    "\n",
    "# Only top 1000\n",
    "df = df.sample(1000, random_state=0)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I vaguely remember Ben from my Sci-Fi fandom days of the \\'60s, I was doing several interviews & bios of obscure actors/actresses, most notably Ben, actress Fay Spain, and Jody Fair, who played Angela in 1961\\'s The Young Savages. Ben was one of the people at a low-key Sci-Fi con in Chicago, about 1970, when I had a nice chat with him and his \\\\career\\\\\" and life. All these were published in some now-long-forgotten fanzine of the day. Wish I still had copies of those interviews, but time marches on, and any of those people surely wouldn\\'t\\' remember me at all so many years later. Ben was a really nice fellow, ekeing out a living (The cons of those days didn\\'t even pay their guest, unless, of course they were big-name stars, and even then the pay was a couple hundred dollars, at most! Good to know Ben\\'s still alive & kicking! How \\'bout a remake of Creature, but 50 years older! Ugly then, uglier now!\"'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenization\n",
    "\n",
    "The first step of processing we make the texts go through is to split the raw sentences into words, or more exactly tokens. The easiest way to do this would be to split the string on spaces, but we can be smarter:\n",
    "\n",
    "- we need to take care of punctuation\n",
    "- some words are contractions of two different words, like isn't or don't\n",
    "- we may need to clean some parts of our texts, if there's HTML code for instance\n",
    "\n",
    "The texts are truncated at 100 tokens for more readability. We can see that it did more than just split on space and punctuation symbols:\n",
    "\n",
    "- the \"'s\" are grouped together in one token\n",
    "- the contractions are separated like this: \"did\", \"n't\"\n",
    "- content has been cleaned for any HTML symbol and lower cased\n",
    "- there are several special tokens (all those that begin by xx), to replace unknown tokens (see below) or to introduce different text fields (here we only have one).\n",
    "\n",
    "# Numericalization\n",
    "Once we have extracted tokens from our texts, we convert to integers by creating a list of all the words used. We only keep the ones that appear at least twice with a maximum vocabulary size of 60,000 (by default) and replace the ones that don't make the cut by the unknown token UNK.\n",
    "\n",
    "This is done automatically behind the scenes if we use a facotry method of TextDataLoaders.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dbunch_lm = TextDataLoaders.from_df(df, text_col='text', label_col='label', is_lm=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if we look at what a what's in our datasets, we'll see the numericalized text as a representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorText([   2,    8, 5953,    8,  268,    8, 1203,   18,    8,   87,    8,  272,\n",
       "            8, 3506,  353,  176,   44,   14,    9,  136,  368,  135,   46,   47,\n",
       "          300,   11,    8,   38,   16,  550,   27,    8,    0,    8, 4025,   10,\n",
       "           13,  156,   49,   61,    0,    0,   10,   49,  116, 1275,   15,  892,\n",
       "           12, 1927,   28,   40,  272, 3506,   11,   13,   96, 2710,    8,  918,\n",
       "           29,  107,   12,    8, 2403,    8,    0,   49,  309,   40, 1119,  487,\n",
       "           11,    8,    0,    0,   16, 1247,   27,    9, 1408,    8,    0,    8,\n",
       "         4025,   12,    8, 1311,    0,   16,   98,   27,    9,  316,   11,    8,\n",
       "          273,  160,   29,    9, 1576,  368,   11]),)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dbunch_lm.train_ds[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxeos',\n",
       " 'xxfld',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " 'xxup',\n",
       " 'xxmaj',\n",
       " 'the',\n",
       " ',',\n",
       " '.',\n",
       " 'and',\n",
       " 'a',\n",
       " 'of',\n",
       " 'to',\n",
       " 'is',\n",
       " 'it',\n",
       " 'in',\n",
       " 'i']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dbunch_lm.vocab[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos xxmaj i 've always thought that most huge box - office xxunk usually have something to recommend them , but after the remake of xxmaj around the xxmaj world in 80 xxmaj days and xxmaj xxunk , xxmaj i 'm beginning to doubt it . xxmaj for those not familiar , it 's based on a xxunk show about a family of astronauts who use state of the art xxunk ,</td>\n",
       "      <td>xxmaj i 've always thought that most huge box - office xxunk usually have something to recommend them , but after the remake of xxmaj around the xxmaj world in 80 xxmaj days and xxmaj xxunk , xxmaj i 'm beginning to doubt it . xxmaj for those not familiar , it 's based on a xxunk show about a family of astronauts who use state of the art xxunk , xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>xxunk , mafia score - xxunk and xxmaj world xxmaj war xxup ii battles , xxmaj xxunk could be xxunk on to give us two solid hours of entertainment on xxmaj saturday night . xxmaj he worked with the cream of the male actors of his day : xxmaj xxunk , xxmaj xxunk , xxmaj xxunk , xxmaj delon , xxmaj xxunk , xxmaj anthony xxmaj quinn . i … xxunk xxmaj</td>\n",
       "      <td>, mafia score - xxunk and xxmaj world xxmaj war xxup ii battles , xxmaj xxunk could be xxunk on to give us two solid hours of entertainment on xxmaj saturday night . xxmaj he worked with the cream of the male actors of his day : xxmaj xxunk , xxmaj xxunk , xxmaj xxunk , xxmaj delon , xxmaj xxunk , xxmaj anthony xxmaj quinn . i … xxunk xxmaj xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>has got to be the xxup dumbest movie xxmaj i 've ever seen . xxmaj we watched it in english class … and this movie made xxup absolutely no sense . i would never , xxup ever watch this movie again … and my sympathy to those who have ever xxup paid to see it . xxbos xxmaj samuel xxmaj fuller is hardly one of xxmaj america 's great directors . xxmaj</td>\n",
       "      <td>got to be the xxup dumbest movie xxmaj i 've ever seen . xxmaj we watched it in english class … and this movie made xxup absolutely no sense . i would never , xxup ever watch this movie again … and my sympathy to those who have ever xxup paid to see it . xxbos xxmaj samuel xxmaj fuller is hardly one of xxmaj america 's great directors . xxmaj i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>allowed to have his character slowly xxunk into despair -- someone has to , or the xxmaj xxunk would be too detached , and no one wants to see that . xxmaj except the monster that has at the time of this writing become the news - media . xxmaj they 'll always eat train xxunk up and feed the xxunk xxunk to the xxunk public . \" xxbos xxmaj this very</td>\n",
       "      <td>to have his character slowly xxunk into despair -- someone has to , or the xxmaj xxunk would be too detached , and no one wants to see that . xxmaj except the monster that has at the time of this writing become the news - media . xxmaj they 'll always eat train xxunk up and feed the xxunk xxunk to the xxunk public . \" xxbos xxmaj this very strange</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>of xxunk , only to be captured and xxunk by a couple of violent hillbillies , is surely the worst nightmare of xxunk % of the world 's population . xxmaj it would have been easy for xxmaj deliverance to slip into exploitation xxunk , but xxmaj john xxmaj xxunk has cleverly avoided the xxunk to go down such a route and has made a film that explores , questions and challenges</td>\n",
       "      <td>xxunk , only to be captured and xxunk by a couple of violent hillbillies , is surely the worst nightmare of xxunk % of the world 's population . xxmaj it would have been easy for xxmaj deliverance to slip into exploitation xxunk , but xxmaj john xxmaj xxunk has cleverly avoided the xxunk to go down such a route and has made a film that explores , questions and challenges the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>- looking back - xxunk since \" on xxmaj her xxmaj xxunk 's xxmaj secret xxmaj service \" . ( * * ) \" xxbos xxmaj edward xxmaj xxunk 's xxmaj tattooed xxmaj stranger is supposed to play like a crime thriller with a little film noir mixed in for flavor . xxmaj instead , it 's a poorly acted , witless look into low budget and uninspired film making . xxmaj</td>\n",
       "      <td>looking back - xxunk since \" on xxmaj her xxmaj xxunk 's xxmaj secret xxmaj service \" . ( * * ) \" xxbos xxmaj edward xxmaj xxunk 's xxmaj tattooed xxmaj stranger is supposed to play like a crime thriller with a little film noir mixed in for flavor . xxmaj instead , it 's a poorly acted , witless look into low budget and uninspired film making . xxmaj the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>fade - ins - fade - outs are done because this was originally made for xxup tv , but that 's really no excuse . \\n\\n xxmaj overall , the movie is just garbage . xxmaj i 'm a sensitive guy , i cried during two episodes of the xxmaj simpsons . i never cried during this crap , not even close . xxmaj really , this movie is not worth your</td>\n",
       "      <td>- ins - fade - outs are done because this was originally made for xxup tv , but that 's really no excuse . \\n\\n xxmaj overall , the movie is just garbage . xxmaj i 'm a sensitive guy , i cried during two episodes of the xxmaj simpsons . i never cried during this crap , not even close . xxmaj really , this movie is not worth your time</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>xxunk . xxmaj some of the special effects may seem outdated compared to modern standards . xxmaj this minor flaw is easily ignored . xxmaj there is so much to discover in this story . xxmaj the romantic relation between the two main characters is so beautiful that it hurts . xxmaj the visuals are beautiful too . xxmaj the action is great which is no surprise , it is xxunk from</td>\n",
       "      <td>. xxmaj some of the special effects may seem outdated compared to modern standards . xxmaj this minor flaw is easily ignored . xxmaj there is so much to discover in this story . xxmaj the romantic relation between the two main characters is so beautiful that it hurts . xxmaj the visuals are beautiful too . xxmaj the action is great which is no surprise , it is xxunk from xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>xxunk you to xxunk , to make friends . \\n\\n xxmaj for those around at the time , ' loulou ' also provides an extra bonus : its perfectly captured mood of 1980 . xxbos xxmaj the film is xxunk enjoyable with a great cast , and excellent direction by xxmaj james xxmaj xxunk . xxmaj the movie is entertaining with a very charismatic performance from xxmaj xxunk xxmaj xxunk and everyone</td>\n",
       "      <td>you to xxunk , to make friends . \\n\\n xxmaj for those around at the time , ' loulou ' also provides an extra bonus : its perfectly captured mood of 1980 . xxbos xxmaj the film is xxunk enjoyable with a great cast , and excellent direction by xxmaj james xxmaj xxunk . xxmaj the movie is entertaining with a very charismatic performance from xxmaj xxunk xxmaj xxunk and everyone is</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dbunch_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## With the data block API\n",
    "We can use the data block API with NLP and have a lot more flexibility than what the default factory methods offer. In the previous example for instance, the data was randomly split between train and validation instead of reading the third column of the csv.\n",
    "\n",
    "With the data block API though, we have to manually call the tokenize and numericalize steps. This allows more flexibility, and if you're not using the defaults from fastai, the various arguments to pass will appear in the step they're revelant, so it'll be more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "imdb_lm = DataBlock(blocks=(TextBlock.from_df('text', is_lm=True),),\n",
    "                    get_x=ColReader('text'),\n",
    "                    splitter=RandomSplitter())\n",
    "\n",
    "dbunch_lm = imdb_lm.dataloaders(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language model (Predict next word): [AWD LSTM](https://arxiv.org/pdf/1708.02182.pdf)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reviews are in a training and test set following an imagenet structure. The only difference is that there is an unsup folder on top of train and test that contains the unlabelled data.\n",
    "\n",
    "We're not going to train a model that classifies the reviews from scratch. Like in computer vision, we'll use a model pretrained on a bigger dataset (a cleaned subset of wikipedia called wikitext-103). That model has been trained to guess what the next word is, its input being all the previous words. It has a recurrent structure and a hidden state that is updated each time it sees a new word. This hidden state thus contains information about the sentence up to that point.\n",
    "\n",
    "We are going to use that 'knowledge' of the English language to build our classifier, but first, like for computer vision, we need to fine-tune the pretrained model to our particular dataset. Because the English of the reviews left by people on IMDB isn't the same as the English of wikipedia, we'll need to adjust the parameters of our model by a little bit. Plus there might be some words that would be extremely common in the reviews dataset but would be barely present in wikipedia, and therefore might not be part of the vocabulary the model was trained on.\n",
    "\n",
    "This is where the unlabelled data is going to be useful to us, as we can use it to fine-tune our model. Let's create our data object with the data block API (next line takes a few minutes the first minute you run it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos xxmaj this movie was awful . i had a very difficult time watching this all the way through . i did n't get the point of the movie . xxmaj what was the point of this movie ? xxmaj the soundtrack was bad , acting was bad and the story xxunk . xxmaj the two main characters in the movie were very boring and their dialog was uninteresting . xxmaj there</td>\n",
       "      <td>xxmaj this movie was awful . i had a very difficult time watching this all the way through . i did n't get the point of the movie . xxmaj what was the point of this movie ? xxmaj the soundtrack was bad , acting was bad and the story xxunk . xxmaj the two main characters in the movie were very boring and their dialog was uninteresting . xxmaj there was</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>xxmaj patrick bring together the surviving members of the xxmaj bombshells . xxmaj we do n't know if enough of them can be found . xxmaj the xxunk more often than not turn into off - key shambles . xxmaj while they do this , we share xxmaj elizabeth 's flashbacks of what life was like when she and xxmaj patrick were young in war - time xxmaj london , playing in</td>\n",
       "      <td>patrick bring together the surviving members of the xxmaj bombshells . xxmaj we do n't know if enough of them can be found . xxmaj the xxunk more often than not turn into off - key shambles . xxmaj while they do this , we share xxmaj elizabeth 's flashbacks of what life was like when she and xxmaj patrick were young in war - time xxmaj london , playing in the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>to what xxmaj i 've seen him do . xxmaj oh , and the father was n't half - bad to my memory , but he was in the movie for such a small amount of time i can hardly remember . \\n\\n xxmaj the story itself was not very good . xxmaj more xxunk than you can imagine . xxmaj predictable story ( until the ending , which i barely understood</td>\n",
       "      <td>what xxmaj i 've seen him do . xxmaj oh , and the father was n't half - bad to my memory , but he was in the movie for such a small amount of time i can hardly remember . \\n\\n xxmaj the story itself was not very good . xxmaj more xxunk than you can imagine . xxmaj predictable story ( until the ending , which i barely understood )</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>orson xxmaj welles : xxmaj the one - man xxmaj band ( 1995 ) ) , where he seems to be just reading to the camera and believing that he 's xxunk , xxunk dramatic . xxmaj i 'm also someone who almost never xxunk about accents , but somehow xxmaj welles manages to make his xxmaj irish accent sound affected and pretentious to me , too . \\n\\n xxmaj as for</td>\n",
       "      <td>xxmaj welles : xxmaj the one - man xxmaj band ( 1995 ) ) , where he seems to be just reading to the camera and believing that he 's xxunk , xxunk dramatic . xxmaj i 'm also someone who almost never xxunk about accents , but somehow xxmaj welles manages to make his xxmaj irish accent sound affected and pretentious to me , too . \\n\\n xxmaj as for the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>and never came back even once as a guest ) and xxmaj hoss had died and xxmaj little xxmaj joe had left after his wife ( in the series ) had been killed by xxunk . \\n\\n xxmaj overall , the era from 1959 - 1965 is the best of this series . xxmaj once xxmaj adam left , it slowly declined . xxmaj most of the shows before 1970 are pretty</td>\n",
       "      <td>never came back even once as a guest ) and xxmaj hoss had died and xxmaj little xxmaj joe had left after his wife ( in the series ) had been killed by xxunk . \\n\\n xxmaj overall , the era from 1959 - 1965 is the best of this series . xxmaj once xxmaj adam left , it slowly declined . xxmaj most of the shows before 1970 are pretty good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>- and xxup fun - way possible . \\n\\n xxmaj so yes , this movie is n't an \\ epic , life - affirming xxunk / 11 picture with obligatory social commentary \" effort . xxmaj the pacing is uneven , some of the scenes could have been cut and not all the actors tow the line . xxmaj but car movies rarely come better than this . xxmaj so if you</td>\n",
       "      <td>and xxup fun - way possible . \\n\\n xxmaj so yes , this movie is n't an \\ epic , life - affirming xxunk / 11 picture with obligatory social commentary \" effort . xxmaj the pacing is uneven , some of the scenes could have been cut and not all the actors tow the line . xxmaj but car movies rarely come better than this . xxmaj so if you hate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>earlier . xxmaj he does n't create the atmosphere as well in this film as he did in the earlier one ; but the surreal aspects of the story come off well , and the mystery is always kept up which stops the film from becoming boring . xxmaj the film stars xxmaj rosemary xxmaj xxunk , who provides eye candy throughout and also delivers a good performance . xxmaj most of</td>\n",
       "      <td>. xxmaj he does n't create the atmosphere as well in this film as he did in the earlier one ; but the surreal aspects of the story come off well , and the mystery is always kept up which stops the film from becoming boring . xxmaj the film stars xxmaj rosemary xxmaj xxunk , who provides eye candy throughout and also delivers a good performance . xxmaj most of the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>had the courage to include them in a screenplay . xxmaj and what about the return of xxmaj james to the hospital to bring the files of the dead xxunk ? xxmaj and the cast , composed of ham actors and actresses ? xxmaj honestly , i do not know what or who is the worst in this film : the screenplay , the director or the cast . xxmaj the correct</td>\n",
       "      <td>the courage to include them in a screenplay . xxmaj and what about the return of xxmaj james to the hospital to bring the files of the dead xxunk ? xxmaj and the cast , composed of ham actors and actresses ? xxmaj honestly , i do not know what or who is the worst in this film : the screenplay , the director or the cast . xxmaj the correct answer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>scary obsession for the xxunk bombshell xxmaj angela . xxmaj ethan xxunk xxunk the aid of xxmaj david and his friends who have been xxunk the school for their entire run with blackmail to help win xxmaj angela . i do n't like to give spoilers out , but for a piece of crap like this i can make an exception . xxmaj angela falls for xxmaj david , xxmaj ethan intentionally</td>\n",
       "      <td>obsession for the xxunk bombshell xxmaj angela . xxmaj ethan xxunk xxunk the aid of xxmaj david and his friends who have been xxunk the school for their entire run with blackmail to help win xxmaj angela . i do n't like to give spoilers out , but for a piece of crap like this i can make an exception . xxmaj angela falls for xxmaj david , xxmaj ethan intentionally xxunk</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dbunch_lm.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6760"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dbunch_lm.vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Langauge model: Train only head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(dbunch_lm, AWD_LSTM, drop_mult=0.3, metrics=[accuracy, Perplexity()])#.to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(0.012022644281387329, 0.010964781977236271)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEOCAYAAACaQSCZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl0nPV97/H3V6N9tyzJu7yAMYSUVZhQNwRnISQhkK0tJGmBLL7kZmmT2zTpaU96m9y06U17moU2rksg5CQkTQikQAiBk5RLAoFYZjEGDBhjY1k2WqxlJI2kkeZ7/5hH9lhoGdl6ZpE+r3PmaOb3/H7zfDW25+vf8jw/c3dERERmUpDtAEREJD8oYYiISFqUMEREJC1KGCIikhYlDBERSYsShoiIpEUJQ0RE0qKEISIiaVHCEBGRtChhiIhIWgqzHcBcqq+v9zVr1mQ7DBGRvLFjx45Od29Ip+68Shhr1qyhpaUl22GIiOQNM9ufbl0NSYmISFqUMEREJC2hJgwzqzWz28xst5k9a2YXTTh+iZn1mtkTweMLKccuM7PnzGyPmX0+zDhFRGRmYc9hfB24193fZ2bFQPkkdX7t7penFphZBPhX4C1AK7DdzO5092dCjldERKYQWg/DzKqBi4FvA7j7iLv3pNl8I7DH3fe6+wjwQ+DKcCIVEZF0hDkktQ7oAG42s8fN7EYzq5ik3kVm9qSZ/dzMzgzKVgAHUuq0BmUiIpIlYSaMQuA84Fvufi4wAEyci3gMWO3uZwPfBH4alNsk7zfpXrJmtsXMWsyspaOjY24iFxHJE0+39fLg85n57gszYbQCre7+aPD6NpIJ5Ch373P3/uD5PUCRmdUHbVelVF0JtE12Enff5u7N7t7c0JDWtSciIvPG9x7Zz2d+9GRGzhVawnD3w8ABM9sQFL0JOG7S2syWmpkFzzcG8XQB24H1ZrY2mCy/CrgzrFhFRPJVz2CcReVFGTlX2KukPgl8P/jS3wtcZ2bXA7j7VuB9wMfMbBSIAVe5uwOjZvYJ4BdABLjJ3Z8OOVYRkbzTPThC7XxIGO7+BNA8oXhryvEbgBumaHsPcE940YmI5L+ewTir6ia7YmHu6UpvEZE8lskhKSUMEZE8lhySKs7IuZQwRETy1FB8jOHRRMbmMJQwRETyVPfgCAC1ZephiIjINLoH4gCawxARken1xIIehuYwRERkOj2DyR6G5jBERGRa4wljkXoYIiIynaOT3uphiIjIdHoGRygtKqC0KJKR8ylhiIjkqeRV3pkZjgIlDBGRvNU9GKemLDPDUaCEISKSt3oGR9TDEBGRmfXE4hmb8AYlDBGRvNWTwRsPghKGiEhecveM3tocQk4YZlZrZreZ2W4ze9bMLppw/ANmtjN4PGxmZ6cc22dmT5nZE2bWEmacIiL5pn94lNGEZ3RIKuwtWr8O3Ovu7wu2aZ24LdRLwBvcvdvM3gZsAy5MOb7Z3TtDjlFEJO8cuy1I5oakQksYZlYNXAxcC+DuI8BIah13fzjl5SPAyrDiERGZT44mjHmyrHYd0AHcbGaPm9mNZlYxTf0PAz9Pee3AfWa2w8y2TNXIzLaYWYuZtXR0dMxN5CIiOW78tiCLKubHpHchcB7wLXc/FxgAPj9ZRTPbTDJhfC6leJO7nwe8Dfi4mV08WVt33+buze7e3NDQMKe/gIhIruqJZXYvDAg3YbQCre7+aPD6NpIJ5DhmdhZwI3Clu3eNl7t7W/CzHbgD2BhirCIieaUn6GHUZGi3PQgxYbj7YeCAmW0Iit4EPJNax8yagNuBP3H351PKK8ysavw5cCmwK6xYRUTyzfhue/NpldQnge8HK6T2AteZ2fUA7r4V+AKwGPg3MwMYdfdmYAlwR1BWCNzq7veGHKuISN7oiY1QWVJIUSRzl9OFmjDc/QmgeULx1pTjHwE+Mkm7vcDZE8tFRCSpZzCztwUBXektIpKXMn3jQVDCEBHJS93qYYiISDoyfeNBUMIQEclLPbF4Rq/yBiUMEZG8M5ZwemOZvVMtKGGIiOSd6FAc98zeeBCUMERE8k73YOYv2gMlDBGRvHP0xoPqYYiIyHR6gx5GjXoYIiIyHfUwREQkLeObJ2mVlIiITKtncAQzqCpVwhARkWl0D8apKSsiUmAZPa8ShohInsnGVd6ghCEikneycR8pCDlhmFmtmd1mZrvN7Fkzu2jCcTOzb5jZHjPbaWbnpRy7xsxeCB7XhBmniEg+6R4cyfiEN4S/497XgXvd/X3BrnvlE46/DVgfPC4EvgVcaGZ1wN+S3HzJgR1mdqe7d4ccr4hIzjvSP8JpS6oyft7QehhmVg1cDHwbwN1H3L1nQrUrge960iNArZktA94K3O/uR4IkcT9wWVixiojkC3eno3+YxqrSjJ87zCGpdUAHcLOZPW5mN5pZxYQ6K4ADKa9bg7KpykVEFrSewTjxMaexqiTj5w4zYRQC5wHfcvdzgQHg8xPqTLYmzKcpfxUz22JmLWbW0tHRcTLxiojkvPboMAAN8yxhtAKt7v5o8Po2kglkYp1VKa9XAm3TlL+Ku29z92Z3b25oaJiTwEVEclVHkDDmVQ/D3Q8DB8xsQ1D0JuCZCdXuBP40WC31OqDX3Q8BvwAuNbNFZrYIuDQoExFZ0NqjQ0B2ehhhr5L6JPD9YIXUXuA6M7sewN23AvcAbwf2AIPAdcGxI2b2JWB78D5fdPcjIccqIpLzjvYwqjM/6R1qwnD3J0gujU21NeW4Ax+fou1NwE3hRScikn/ao8OUFUWoKI5k/Ny60ltEJI90RIdprC7BLLP3kQIlDBGRvNIeHaKhMvPzF6CEISKSV8Z7GNmghCEikkc6osPqYYiIyPSG4mP0DY1mZUktKGGIiOSNYxftZX5JLShhiIjkjWzeFgSUMERE8kaHEoaIiKSjI7gtSDbuIwVKGCIieaMjOkyBwWKtkhIRkem0R4epqyghUpD5q7xBCUNEJG90RIezNhwFShgiInmjPTqctQlvUMIQEckb6mGIiMiMEgmnsz+7PYxQ98Mws31AFBgDRt29ecLxzwIfSInlDKAh2EBp2rYiIgtJ9+AIownPag8j7B33ADa7e+dkB9z9q8BXAczsncCnJ+ysN2VbEZGF5NhV3tm5LQjk1pDU1cAPsh2EiEguOrY16/ydw3DgPjPbYWZbpqpkZuXAZcBPZttWRGQhOHpbkCxdtAfhD0ltcvc2M2sE7jez3e7+4CT13gk8NGE4Kq22QTLZAtDU1BTG7yAiknXZvvEghNzDcPe24Gc7cAewcYqqVzFhOCrdtu6+zd2b3b25oaFhrkIXEckpHdFhKoojVJRkYup5cqElDDOrMLOq8efApcCuSerVAG8A/mu2bUVEFor26BCN1dmb8IZwh6SWAHeY2fh5bnX3e83segB33xrUezdwn7sPzNQ2xFhFRHJaNrdmHRdawnD3vcDZk5RvnfD6O8B30mkrIrJQdUSHOWNZdVZjyKVltSIiMoWOLN9HCpQwRERyXmxkjOjwqBKGiIhM71BvDIBlNdmd9FbCEBHJcYd6k1uzLqspy2ocShgiIjnuYE+yh7GiVglDRESm0dYTwwyW1GgOQ0REpnGoZ4j6yhJKCiNZjUMJQ0Qkx7X1xlie5eEoUMIQEcl5B3tirKjN7gopUMIQEclp7s6hnqGsr5ACJQwRkZzWMxgnFh/TkJSIiEzv2JJaDUmJiMg0cuWiPUgzYZjZKWZWEjy/xMw+ZWa14YYmIiJtQQ8jn4akfgKMmdmpwLeBtcCtoUUlIiJAMmEUFxawuKI426GknTAS7j5KcrOjr7n7p4FlMzUys31m9pSZPWFmLZMcv8TMeoPjT5jZF1KOXWZmz5nZHjP7fLq/kIjIfNLWO8SymlIKCizboaS9gVLczK4GrgHeGZQVpdl2s7t3TnP81+5+eWqBmUWAfwXeArQC283sTnd/Js1ziojMC209MZbnwPwFpN/DuA64CPiyu79kZmuB74UXFhuBPe6+191HgB8CV4Z4PhGRnNTWkxtXeUOaCcPdn3H3T7n7D8xsEVDl7l9Jpylwn5ntMLMtU9S5yMyeNLOfm9mZQdkK4EBKndagTERkwRgdS/BK3xDLc2BJLaQ5JGVmDwBXBPWfADrM7P+5+2dmaLrJ3dvMrBG438x2u/uDKccfA1a7e7+ZvR34KbAemGywzqeIbQuwBaCpqSmdX0dEJC+8Eh0m4bmxQgrSH5Kqcfc+4D3Aze5+PvDmmRq5e1vwsx24g+RQU+rxPnfvD57fAxSZWT3JHsWqlKorgbYpzrHN3ZvdvbmhoSHNX0dEJPfl0pJaSD9hFJrZMuCPgLvTaWBmFWZWNf4cuBTYNaHOUjOz4PnGIJ4uYDuw3szWmlkxcBVwZ5qxiojMC0cTRpa3Zh2X7iqpLwK/AB5y9+1mtg54YYY2S4A7gnxQCNzq7vea2fUA7r4VeB/wMTMbBWLAVe7uwKiZfSI4ZwS4yd2fnuXvJiKS19p6gqu8c6SHkVbCcPcfAz9Oeb0XeO8MbfYCZ09SvjXl+Q3ADVO0vwe4J534RETmo7aeGDVlRVSWpPt/+3Cle2uQlWZ2h5m1m9krZvYTM1sZdnAiIgvZod4Yy3JkOArSn8O4meQcwnKSy1vvCspERCQkB3uGWJEjw1GQfsJocPeb3X00eHwH0JIkEZEQ5dJFe5B+wug0sw+aWSR4fJDkaiYREQnBwPAovbE4y3Lkoj1IP2F8iOSS2sPAIZKrm64LKygRkYXuUO/4xkl51sNw95fd/Qp3b3D3Rnd/F8mL+EREJAQHe3Jn46RxJ7Pj3ky3BRERkRP08pFBAJrqyrMcyTEnkzCyf3N2EZF5an/nAKVFBTRWlWQ7lKNOJmFMejNAERE5efu6BlldV5ETGyeNm/byQTOLMnliMCB3BtZEROaZ/V0DrK2vyHYYx5k2Ybh7VaYCERGRpETC2X9kkM2nN2Y7lOOczJCUiIiE4HDfECOjCVYvzp0Jb1DCEBHJOfu6BgBYszi3hqSUMEREcszLXckltephiIjItPZ1DVIUsZy6aA+UMEREcs7+rgFW1ZUTyaEltZD+jnsnxMz2AVFgDBh19+YJxz8AfC542Q98zN2fTKetiMh8ta9rMOfmLyDkhBHY7O6dUxx7CXiDu3eb2duAbcCFabYVEZl33J39XQO8bl1dtkN5lazu++fuD6e8fATQLn4isqB19A8zODKWkz2MsOcwHLjPzHaY2ZYZ6n4Y+PkJthURmRf25+gKKQi/h7HJ3dvMrBG438x2u/uDEyuZ2WaSCeMPTqDtFmALQFNTUzi/hYhIhuzrzM1rMCDkHoa7twU/24E7gI0T65jZWcCNwJXu3jWbtsHxbe7e7O7NDQ3aNVZE8tv+rkEiBcaKRbm1pBZCTBhmVmFmVePPgUuBXRPqNAG3A3/i7s/Ppq2IyHy0r2uAlYvKKIrk3lUPYQ5JLQHuMLPx89zq7vea2fUA7r4V+AKwGPi3oN748tlJ24YYq4hITtjfNcjqHByOghAThrvvBc6epHxryvOPAB9Jt62IyHzm7uzrGuDcptpshzKp3OvziIgsUN2DcaJDoznbw1DCEBHJEcfuUpt7S2pBCUNEJGccu0utehgiIjKNlzoHMINVdbm3pBaUMEREcsZzh6OsriunpDCS7VAmpYQhIpIjdrX1cuaKmmyHMSUlDBGRHNA7GKe1O8aZy6uzHcqUlDBERHLA04d6AThzuXoYIiIyjWfa+gDUwxARkentOtjL0upS6itLsh3KlJQwRERywNNtfTnduwAlDBGRrIuNjPFiR39Or5ACJQwRkazbfbiPhOf2/AUoYYiIZN2uPJjwBiUMEZGse6atl9ryIlbU5uYtQcaFmjDMbJ+ZPWVmT5hZyyTHzcy+YWZ7zGynmZ2XcuwaM3sheFwTZpwiItk0PuEdbBqXs8LccW/cZnfvnOLY24D1weNC4FvAhWZWB/wt0Aw4sMPM7nT37gzEKyKSMfGxBLsPRbl205pshzKjbA9JXQl815MeAWrNbBnwVuB+dz8SJIn7gcuyGaiISBj2tPczMpbI+fkLCD9hOHCfme0wsy2THF8BHEh53RqUTVUuIjKvPH10wju3l9RC+ENSm9y9zcwagfvNbLe7P5hyfLIBO5+m/FWCRLQFoKmp6WTjFRHJqKfbeikrirC2Pjc3TUoVag/D3duCn+3AHcDGCVVagVUpr1cCbdOUT3aObe7e7O7NDQ0NcxW6iEhG7DrYyxnLqogU5PaEN4SYMMyswsyqxp8DlwK7JlS7E/jTYLXU64Bedz8E/AK41MwWmdmioO0vwopVRCQbhkfH2Nnay3lNi7IdSlrCHJJaAtwRLBMrBG5193vN7HoAd98K3AO8HdgDDALXBceOmNmXgO3Be33R3Y+EGKuISMbtOtjH8GiC5jV12Q4lLaElDHffC5w9SfnWlOcOfHyK9jcBN4UVn4hItm3fl/x/cPOa/OhhZHtZrYjIgtWy7wjrGipy+pbmqZQwRESyIJFwtu/r5oLV+TEcBUoYIiJZsaejn95YPG+Go0AJQ0QkK8bnLzauVQ9DRESmsf2lIzRUldBUV57tUNKmhCEikgXb93VzwZpFOX+H2lRKGCIiGdbWE+NgT4wL8uT6i3FKGCIiGTY+f6GEISIi02rZ101FcYTTl1ZlO5RZUcIQEcmw7fuOcN7qRRRG8usrOL+iFRHJc+3RIZ57JcrGPBuOAiUMEZGMumfnIdzhstcuzXYos6aEISKSQXfvPMTpS6tYvyS/5i9ACUNEJGPaemK07O/mnWcvz3YoJ0QJQ0QkQ3628xAAl5+1LMuRnJiw9/TGzCJAC3DQ3S+fcOxfgM3By3Kg0d1rg2NjwFPBsZfd/YqwYxURCdNdO9s4a2UNqxfn/v7dkwk9YQB/BjwLVE884O6fHn9uZp8Ezk05HHP3c8IP72gseXWJvojkl32dA+xs7eWv335GtkM5YaEOSZnZSuAdwI1pVL8a+EGY8UxmKD7G27/+a7Y9uDfTpxaRBeRnTyWHo96Rp8NREP4cxteAvwQS01Uys9XAWuBXKcWlZtZiZo+Y2bvCCrC0KMJYwvnNns6wTiEiwl1PttG8ehHLa8uyHcoJCy1hmNnlQLu770ij+lXAbe4+llLW5O7NwPuBr5nZKVOcZ0uQWFo6OjpOKNZNp9bzu5eOMBQfm7myiMgsPf9KlN2Ho3k72T0uzB7GJuAKM9sH/BB4o5l9b4q6VzFhOMrd24Kfe4EHOH5+I7XeNndvdvfmhoaGEwr0D9YvZng0wWP7u0+ovYjIdG7b0UphgfGOs/JzOe240BKGu/+Vu6909zUkE8Kv3P2DE+uZ2QZgEfDblLJFZlYSPK8nmXyeCSvWjWsXU1hgGpYSkTkXH0tw+2OtvPH0RhqqSrIdzknJxCqp45jZF4EWd78zKLoa+KG7e0q1M4B/N7MEyaT2FXcPLWFUlhRyblMtDylhiMgc+9Xudjr7R/jjC1ZlO5STlpGE4e4PkBxWwt2/MOHY/56k/sPA72UgtKM2nVrP13/5Ar2DcWrKizJ5ahGZx37ccoDGqhLecNqJDZnnEl3pHdh0aj3u8Nu96mWIyNxo7xviv5/r4L3nr8y7W5lPJv9/gzlyzqpaKoojmscQkTnzk8cOMpZw/vD8ldkOZU4oYQSKIgVcuG4xD+3pynYoIjIPuDs/bjnAxjV1rGuozHY4c0IJI8WmU+t5qXOA1u7BbIciInmuZX83ezsH+MPm+dG7ACWM4/zBqfUAPKxehoicBHfnm7/aQ1VpYV7fCmQiJYwUpy2ppL6yhHufPkwi4TM3EBGZxH8/186Dz3fw528+jfLijF+9EBoljBRmxtUbV/Gr3e189Lst9Mbi2Q5JRPLMyGiCL939LKc0VPCnF63Odjhzav6kvjnymbecRn1lCV+6+xmuvOE3fOW9Z+EOh/tixEYSvOvc5fPqfwwiMrdueXgfL3UOcPN1F1A0D5bSptI33wRmxjW/v4Yzl1fzse8/xlXbHjnu+O2PtXLTdRdQXaqL+0TkeJ39w3zjly+weUMDmzc0ZjucOaeEMYXmNXXc86nX8/CLndRXlrCsppSnDvbyv370JB+88VFuuW4jiyqKQ48jkXDM0OZOIjnO3fn7nz1LLD7G31z+mmyHEwoljGk0VJVw5Tkrjr5e11BJZUnh0Z7HDe8/l/VLqubsfO5Oe3SY5w5H+d1LR3j0pS6eONADJO93VVlaSH1lCStqy1ixqIzVdRWc0lDBKY2VLK4oVlIRyaLvPLyP2x8/yKfetJ5T5sl1FxPZ8ff8y2/Nzc3e0tIS+nke2tPJR7/bwuDIGKcvreLys5axvLaMvR0D7O3s58jACNWlRdSWF1FeXEhn/zDt0WE6+4dZWl3KGcuqOX1pFU5y28Z9XQO81DnI/q4BBkeSe3JECozXrqjhgtWLKCosoH9olOhQnPboMAd7YrT1xIiPHfuzqy0vYn1jJac2VnHakkrOWFbNa5ZXz3robCzhPHuoj31dA7T1xGjrGaJvKE58zImPJvfBqikroraiiNqyYooiRmGBEYkUUBwxiiIFFEUKcGB0LMHomOM4pUURSgojlBdHWFJdytKaUqpLC5XkZF548PkOrr35d7zpjCX8+wfPp6Agf/5em9mOYO+hmesqYZyY9ugQ9+w8xN07D9ES7KMRKTCa6sqpryymLzZKbyzOwMgoiyuKaawqpa6imEO9MZ57JcpQPPnlW1hgrKorZ83ictbUV7C2voJ19ZWc01RLZcnUHcCxhNPWE2Nv5wAvtvezp6OfF16J8vwr/cet7lpVV8aaxRUsqyllaU0ZjVUlLK4opq6imIqSQnoG43QPjnC4d4jf7TvCo3u76BsaPdq+ojhCbXkxJYXJRJBwpycWp2dw5LiEdSLKiiIsKi+iuqyIqtJCSosiFJgRKUg+iiMFFEaSP8uKI5QVRSgtigCQcGcs4cTiY0SHRukfGiXhTm15EbXlxSwqL2Z5bSkrastYXltGVWkh5cWFRPLoH7Lkhxc7+nnXvz7EitoyfvKx36dimn+3uUgJI8MO9w7RPzxKU105xYUzr4oYSzj7uwYoMGPForI5XUnh7nREh3nmUF/y0dbHge4Yh3tjtEeHme6Pu6munN8/ZTEXnbKYDUurWFZTNmUvwD35ZR0fS35xj44lGE048bEEI6MJzKCwIPmFb2YMxccYio8xODLGK31DHO4d4lDvEL2xOH2xOH1DcYZHEyQSzpg7o2N+3PuNtx0OejkFlkzQpUURqoLhOsPojcXpiY0cTcgTFUcKWFxZzNKaUpbXlFFXUUxxkAyLIob7sWQ0ODLGwPAoAyOjyeRWUcziimJqyoupLi2ksiSZ5EZSYuzqH6FrYJiu/hHMjLKiCGXFBcRGErRHh2iPDjMcH6OhqpSlNSXUVZTg7ske3FiCokgBpUUFlBVFKC8ppKqkkKrSQsygMzpCR/8wfUPxo7218uIINWVF1FeWUFdRTKTAiI0kP+viwgKa6sqp03BlaA71xrh62yNEh0b5r09sYuWi8myHNGtKGDKp+FiCIwMjdPWPcGRghIGRUWrLiqgLehyLK3N/c5d0FwEMDI/S1hOjtSfGoZ4hBoZHicXHGBgZpTM6wuG+ZHlPLM7IaPILP55IUGBGgUGBGeXFESpKCqkoLmQwPkr3QJz+4dFpzwvJRFYXLIgYGhljMD5GSWEBjVUlNFaVUlJUQEd0mFf6hugejGOWvJdZUYERTzgjo5Mnu3HFkQJGxqavk6qypJA19eWc1ljFaUur2LCkiqbF5ayoLTvaY5PZe7lrkPff+Ag9g3Fu+dBGzl+9KNshnRAlDJGQDMXH6IvFiQ6PEh0aPfo/+ZLCAkoKI8keSFlR2mPYiYS/qu5YwhkKklt0KPlIuNNQWUJ9ZQllxRESCWd4NMHAyCg9g8f+E5BwjvZQYvEx9ncl58b2dg7wwiv9HO4bOu5cyUUUpSyrSQ7draor45SGSk5prGRZdWlejcVn0p72KB+48VGGRxN890MbOWtlbbZDOmGzSRihD7aZWQRoAQ66++UTjl0LfBU4GBTd4O43BseuAf4mKP8/7n5L2LGKzKQ0mEeZqxX2k30hRwos2bMpKaRxikV4BQWWnNcpjlBfWcKpaQbUOxjnhfYoB7oHaT0So7U7RltvjBfaozz4QsfRRReQTDwrF5XTVJd8nNpYyYalVZy2pIqasoV5HVIi4dy1s42/u+sZCsz4zy0XsWHp3K2UzHWZmJ35M+BZoHqK4//p7p9ILTCzOuBvgWbAgR1mdqe7d4caqcg8V1NeRPOaOprX1L3qmLvT2T/Cix39vNjRz0sdAxzoHuTlIzEe3dvFQEoyWVVXxtkrazl7ZS1nrazhzBU10y7SmA9a9h3hSz97licP9HDm8mq+efW58+a25ekK9U/YzFYC7wC+DHxmFk3fCtzv7keC97kfuAz4wZwHKSJAcl6ooaqEhqoSXrdu8XHH3J2DPTGefyXKs4ei7DrYy+Mv93D3zkNBW1i7uIIzV9TwmmBJ92uWVdNQlfvzYtNxdx5+sYv/+PVeHniugyXVJfzTH57Ne85dsSCH68L+L8HXgL8EpuuzvdfMLgaeBz7t7geAFcCBlDqtQdmrmNkWYAtAU1PTXMQsIhOYGSsXlbNyUTlvPH3J0fKO6DC7Dvay62AvTx3s5bH93dz1ZNvR43UVxZy2pJLTllRxxrJqXru8htOWVlJSmNuT7X1Dce596jC3/HYfT7f1UV9ZwmffuoHrNq1Z0PeSC+03N7PLgXZ332Fml0xR7S7gB+4+bGbXA7cAbwQmS92Tzs67+zZgGyQnvU86cBFJW0NVCZtPb2Tz6ccmUXoH40eXdb/wSpTnXoly+2MH6R/eDySvPVpbX8G6hgrWNVSyvrGS166o4ZSGyqxeJ9PVP8zDL3bxs52H+NVz7YyMJji1sZJ/fO/vceU5K7SijHB7GJuAK8zs7UApUG1m33P3D45XcPfUnYr+A/jH4HkrcEnKsZXAAyHGKiJzpKa8iIuC63nGJRLOge5Bnm7r4+m2Xp5/pZ897f388tl2RoO9Z8qKIpy+rIpTGypZ21DBuvoKltaUsbiimIaqkjmMJziHAAAJAElEQVT9wnZ3WrtjPPZyN4+/3MMje7vYfTgKJFeOfeDCJq48ZwVnr6zRNSwpMrKsNuhh/MUkq6SWufuh4Pm7gc+5++uCSe8dwHlB1ceA88fnNKaiZbUi+SU+lmBvxwBPt/Wy62AymeztHKAjOvyquvWVxZzaWMn6xipWLy5ncWXyiv76yhJWLSqnpnz6lVut3YP85oVOfr2nk0f3dtHZPwIkE1XzmkW8bl0yyZ21oobCeXZb8unk1LLaiczsi0CLu98JfMrMrgBGgSPAtQDufsTMvgRsD5p9caZkISL5pyhSwIalVWxYWsV7zjtWHh2Ks69zkPboEF39ySvcX+4a5IX2KD994iDRoVdfQFldWsiqunKqS4soD5Yc9w+Pcrh3iMN9Q/QMJm+Zs6S6hIvXN3D+mkWcs6qWDUuqFlSCOBm6cE9E8oq70xcb5cjgCEcGhumIDnPgSIwD3YMcODJI//AoA8NjxOJjVJREWBrc7HJdfSWvX1/PqY2VGmZKkdM9DBGRk2Fm1JQXUVNexNr6imyHs6CoHyYiImlRwhARkbQoYYiISFqUMEREJC1KGCIikhYlDBERSYsShoiIpEUJQ0RE0jKvrvQ2sw5gf/CyBuid5vnEn/VA5yxOl/qe6RybWJbN+E4mxunK9BnqMzzZ+KaLabK4Jitb6J/hdPFNFtdq4EPufteMkbj7vHwA26Z7PsnPlhN9/3SOTSzLZnwnE+MMseoz1Gd4UvFNF5M+w5OPb6bPcKbHfB6SumuG5xN/nsz7p3NsYlk245vqeDoxzlQ2G/oMF/ZnONWxqWKaKh59htOXzeYznNa8GpI6GWbW4mnegCsbcj0+yP0Ycz0+yP0Ycz0+yP0Ycz2+6cznHsZsbct2ADPI9fgg92PM9fgg92PM9fgg92PM9fimpB6GiIikRT0MERFJixKGiIikRQlDRETSooSRBjN7vZltNbMbzezhbMczkZkVmNmXzeybZnZNtuOZyMwuMbNfB5/hJdmOZypmVmFmO8zs8mzHMpGZnRF8freZ2ceyHc9kzOxdZvYfZvZfZnZptuOZyMzWmdm3zey2bMeSKvh7d0vw2X0g2/FMZ94nDDO7yczazWzXhPLLzOw5M9tjZp+f7j3c/dfufj1wN3BLrsUHXAmsAOJAaw7G50A/UDrX8c1hjACfA36Ui/G5+7PB38E/AuZ8SeYcxfhTd/8ocC3wxzkY3153//BcxjWVWcb7HuC24LO7IhPxnbDZXHGYjw/gYuA8YFdKWQR4EVgHFANPAq8Bfo9kUkh9NKa0+xFQnWvxAZ8H/kfQ9rYcjK8gaLcE+H4u/hkDbwauIvlld3muxRe0uQJ4GHh/Ln6GKe3+GTgvh+Ob038jcxDvXwHnBHVuDTu2k3kUMs+5+4NmtmZC8UZgj7vvBTCzHwJXuvs/AJMOR5hZE9Dr7n25Fp+ZtQIjwcuxXIsvRTdQMpfxzVWMZrYZqCD5DzhmZve4eyJX4gve507gTjP7GXDrXMQ2lzGamQFfAX7u7o/lWnyZNJt4Sfa6VwJPkOOjPvM+YUxhBXAg5XUrcOEMbT4M3BxaRMebbXy3A980s9cDD4YZWGBW8ZnZe4C3ArXADeGGdtSsYnT3vwYws2uBzrlKFtOY7Wd4CcmhixLgnlAjO2a2fw8/SbKnVmNmp7r71jCDY/af4WLgy8C5ZvZXQWLJpKni/QZwg5m9gxO/fUhGLNSEYZOUTXsFo7v/bUixTGZW8bn7IMmElimzje92kkktk2b9Zwzg7t+Z+1AmNdvP8AHggbCCmcJsY/wGyS+/TJltfF3A9eGFM6NJ43X3AeC6TAdzInK6+xOiVmBVyuuVQFuWYpmM4jt5uR5jrscHuR9jrsc3Ub7F+yoLNWFsB9ab2VozKyY52XlnlmNKpfhOXq7HmOvxQe7HmOvxTZRv8b5atmfdw34APwAOcWzJ6YeD8rcDz5NctfDXii8/48uHGHM9vnyIMdfjy/d4033o5oMiIpKWhTokJSIis6SEISIiaVHCEBGRtChhiIhIWpQwREQkLUoYIiKSFiUMmdfMrD/D57vRzF4zR+81ZmZPmNkuM7vLzGpnqF9rZv9zLs4tMhldhyHzmpn1u3vlHL5fobuPztX7zXCuo7Gb2S3A8+7+5WnqrwHudvfXZiI+WXjUw5AFx8wazOwnZrY9eGwKyjea2cNm9njwc0NQfq2Z/djM7gLus+QOgg9Ycve73Wb2/eDW3gTlzcHzfkvuhPikmT1iZkuC8lOC19vN7Itp9oJ+S/Jup5hZpZn90sweM7OnzOzKoM5XgFOCXslXg7qfDc6z08z+bg4/RlmAlDBkIfo68C/ufgHwXuDGoHw3cLG7nwt8Afj7lDYXAde4+xuD1+cCf05y/4x1wKZJzlMBPOLuZ5O87fxHU87/9eD8M958zswiwJs4dt+hIeDd7n4esBn45yBhfR540d3PcffPWnKb1PUk92E4BzjfzC6e6XwiU1motzeXhe3NwGuCTgFAtZlVATXALWa2nuRtsotS2tzv7kdSXv/O3VsBzOwJYA3wmwnnGSG52xvADuAtwfOLgHcFz28F/mmKOMtS3nsHcH9QbsDfB1/+CZI9jyWTtL80eDwevK4kmUAysWeKzENKGLIQFQAXuXsstdDMvgn8t7u/O5gPeCDl8MCE9xhOeT7G5P+W4n5sknCqOtOJufs5ZlZDMvF8nOR+Ex8AGoDz3T1uZvtI7pc+kQH/4O7/PsvzikxKQ1KyEN0HfGL8hZmdEzytAQ4Gz68N8fyPkBwKg+Qtrqfl7r3Ap4C/MLMiknG2B8liM7A6qBoFqlKa/gL4kJmNT5yvMLPGOfodZAFSwpD5rtzMWlMenyH55dscTAQ/w7Fd2P4v8A9m9hAQCTGmPwc+Y2a/A5YBvTM1cPfHgSdJJpjvk4y/hWRvY3dQpwt4KFiG+1V3v4/kkNdvzewp4DaOTygis6JltSIZZmblJIeb3MyuAq529ytnaieSbZrDEMm884EbgpVNPcCHshyPSFrUwxARkbRoDkNERNKihCEiImlRwhARkbQoYYiISFqUMEREJC1KGCIikpb/Dxc1dhTMiun3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.355417</td>\n",
       "      <td>01:54</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-15-c34b2e785634>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_one_cycle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2e-2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmoms\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0.8\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0.7\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0.8\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/fastai2/callback/schedule.py\u001b[0m in \u001b[0;36mfit_one_cycle\u001b[0;34m(self, n_epoch, lr_max, div, div_final, pct_start, wd, moms, cbs, reset_opt)\u001b[0m\n\u001b[1;32m     88\u001b[0m     scheds = {'lr': combined_cos(pct_start, lr_max/div, lr_max, lr_max/div_final),\n\u001b[1;32m     89\u001b[0m               'mom': combined_cos(pct_start, *(self.moms if moms is None else moms))}\n\u001b[0;32m---> 90\u001b[0;31m     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn_epoch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcbs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mParamScheduler\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mscheds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mL\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcbs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreset_opt\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreset_opt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwd\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwd\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     91\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     92\u001b[0m \u001b[0;31m# Cell\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/fastai2/learner.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, n_epoch, lr, wd, cbs, reset_opt)\u001b[0m\n\u001b[1;32m    293\u001b[0m                     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    294\u001b[0m                         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mepoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mepoch\u001b[0m\u001b[0;34m;\u001b[0m          \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'begin_epoch'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 295\u001b[0;31m                         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_epoch_train\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    296\u001b[0m                         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_epoch_validate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    297\u001b[0m                     \u001b[0;32mexcept\u001b[0m \u001b[0mCancelEpochException\u001b[0m\u001b[0;34m:\u001b[0m   \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_cancel_epoch'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/fastai2/learner.py\u001b[0m in \u001b[0;36m_do_epoch_train\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    268\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    269\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdls\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m;\u001b[0m                        \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'begin_train'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 270\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mall_batches\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    271\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mCancelTrainException\u001b[0m\u001b[0;34m:\u001b[0m                         \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_cancel_train'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    272\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m                                             \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_train'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/fastai2/learner.py\u001b[0m in \u001b[0;36mall_batches\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    246\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mall_batches\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    247\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_iter\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 248\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mo\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mone_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    249\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    250\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mone_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/fastai2/learner.py\u001b[0m in \u001b[0;36mone_batch\u001b[0;34m(self, i, b)\u001b[0m\n\u001b[1;32m    252\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    253\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_split\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m;\u001b[0m                                  \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'begin_batch'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 254\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mxb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m;\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_pred'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    255\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0myb\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    256\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpred\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0myb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m;\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_loss'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    539\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    540\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 541\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    542\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    543\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m     90\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 92\u001b[0;31m             \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     93\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     94\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    539\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    540\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 541\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    542\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    543\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/fastai2/text/models/awdlstm.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, inp, from_embeds)\u001b[0m\n\u001b[1;32m    100\u001b[0m         \u001b[0mnew_hidden\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mrnn\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mhid_dp\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrnns\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhidden_dps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 102\u001b[0;31m             \u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnew_h\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrnn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhidden\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0ml\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    103\u001b[0m             \u001b[0mnew_hidden\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnew_h\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    104\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0ml\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_layers\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhid_dp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    539\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    540\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 541\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    542\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    543\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/fastai2/text/models/awdlstm.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m     49\u001b[0m             \u001b[0;31m#To avoid the warning that comes because the weights aren't flattened.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     50\u001b[0m             \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msimplefilter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"ignore\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 51\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     52\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input, hx)\u001b[0m\n\u001b[1;32m    562\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward_packed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    563\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 564\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    565\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    566\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py\u001b[0m in \u001b[0;36mforward_tensor\u001b[0;34m(self, input, hx)\u001b[0m\n\u001b[1;32m    541\u001b[0m         \u001b[0munsorted_indices\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    542\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 543\u001b[0;31m         \u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhidden\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_sizes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_batch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msorted_indices\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    544\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    545\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpermute_hidden\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0munsorted_indices\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py\u001b[0m in \u001b[0;36mforward_impl\u001b[0;34m(self, input, hx, batch_sizes, max_batch_size, sorted_indices)\u001b[0m\n\u001b[1;32m    524\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mbatch_sizes\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    525\u001b[0m             result = _VF.lstm(input, hx, self._get_flat_weights(), self.bias, self.num_layers,\n\u001b[0;32m--> 526\u001b[0;31m                               self.dropout, self.training, self.bidirectional, self.batch_first)\n\u001b[0m\u001b[1;32m    527\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    528\u001b[0m             result = _VF.lstm(input, batch_sizes, hx, self._get_flat_weights(), self.bias,\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 2e-2, moms=(0.8,0.7,0.8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('fit_head')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Langauge model: Unfreeze and train all layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('fit_head');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(10, 2e-3, moms=(0.8,0.7,0.8))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}