nbs/39_tutorial.transformers.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "#| eval: false\n",
    "! [ -e /content ] && pip install -Uqq fastai  # upgrade fastai on colab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transformers\n",
    "\n",
    "> An example of how to incorporate the transfomers library from HuggingFace with fastai"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "skip_exec: true\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we will see how we can use the fastai library to fine-tune a pretrained transformer model from the [transformers library](https://github.com/huggingface/transformers) by HuggingFace. We will use the mid-level API to gather the data. Even if this tutorial is self contained, it might help to check the [imagenette tutorial](http://docs.fast.ai/tutorial.imagenette.html) to have a second look on the mid-level API (with a gentle introduction using the higher level APIs) in computer vision."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing a transformers pretrained model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First things first, we will need to install the transformers library. If you haven't done it yet, install the library:\n",
    "\n",
    "```\n",
    "!pip install -Uq transformers\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then let's import what will need: we will fine-tune the GPT2 pretrained model and fine-tune on wikitext-2 here. For this, we need the `GPT2LMHeadModel` (since we want a language model) and the `GPT2Tokenizer` to prepare the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import GPT2LMHeadModel, GPT2TokenizerFast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use several versions of this GPT2 model, look at the [transformers documentation](https://huggingface.co/transformers/pretrained_models.html) for more details. Here we will use the basic version (that already takes a lot of space in memory!) You can change the model used by changing the content of `pretrained_weights` (if it's not a GPT2 model, you'll need to change the classes used for the model and the tokenizer of course)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a1b530b2b41c4b05b2cea7030ffadde2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.04M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1dc01424a2a74940a6ad335042c59e0f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ade814d6b9b84fe19318b644c4171fc2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ea74d74d63fa42d7954db7920d0eb675",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/665 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0044b6ddc22d4496a31d7863d966651b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/548M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pretrained_weights = 'gpt2'\n",
    "tokenizer = GPT2TokenizerFast.from_pretrained(pretrained_weights)\n",
    "model = GPT2LMHeadModel.from_pretrained(pretrained_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we move on to the fine-tuning part, let's have a look at this `tokenizer` and this `model`. The tokenizers in HuggingFace usually do the tokenization and the numericalization in one step (we ignore the padding warning for now):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1212, 318, 281, 1672, 286, 2420, 11, 290]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ids = tokenizer.encode('This is an example of text, and')\n",
    "ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like fastai `Transform`s, the tokenizer has a `decode` method to give you back a text from ids:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is an example of text, and'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model can be used to generate predictions (it is pretrained). It has a `generate` method that expects a batch of prompt, so we feed it our ids and add one batch dimension (there is a padding warning we can ignore as well):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    }
   ],
   "source": [
    "t = torch.LongTensor(ids)[None]\n",
    "preds = model.generate(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predictions, by default, are of length 20:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1, 20]),\n",
       " tensor([1212,  318,  281, 1672,  286, 2420,   11,  290,  340,  338,  407,  257,\n",
       "          922,  530,   13,  198,  198,  464,  717, 1517]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds.shape,preds[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the decode method (that prefers a numpy array to a tensor):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"This is an example of text, and it's not a good one.\\n\\nThe first thing\""
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(preds[0].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bridging the gap with fastai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how we can use fastai to fine-tune this model on wikitext-2, using all the training utilities (learning rate finder, 1cycle policy etc...). First, we import all the text utilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text.all import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we download the dataset (if not present), it comes as two csv files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#2) [Path('/home/jhoward/.fastai/data/wikitext-2/test.csv'),Path('/home/jhoward/.fastai/data/wikitext-2/train.csv')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.WIKITEXT_TINY)\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at what those csv files look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the &lt;unk&gt; season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\n = Big Boy ( song ) = \\n \\n \" Big Boy \" &lt;unk&gt; \" I 'm A Big Boy Now \" was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including \" Big Boy \" . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \\n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\n = The Remix ( Lady Gaga album ) = \\n \\n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and &lt;unk&gt; composit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\n = New Year 's Eve ( Up All Night ) = \\n \\n \" New Year 's Eve \" is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica &lt;unk&gt; and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \\n During Reagan ( Christina Applegate ) and Chris 's ( Will &lt;unk&gt; ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\n = Geopyxis carbonaria = \\n \\n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family &lt;unk&gt; . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf &lt;unk&gt; cup , &lt;unk&gt; &lt;unk&gt; cup , or pixie cup . The small , &lt;unk&gt; @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers ....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         0\n",
       "0   \\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...\n",
       "1   \\n = Big Boy ( song ) = \\n \\n \" Big Boy \" <unk> \" I 'm A Big Boy Now \" was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including \" Big Boy \" . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \\n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re...\n",
       "2   \\n = The Remix ( Lady Gaga album ) = \\n \\n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit...\n",
       "3   \\n = New Year 's Eve ( Up All Night ) = \\n \\n \" New Year 's Eve \" is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \\n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch...\n",
       "4   \\n = Geopyxis carbonaria = \\n \\n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers ...."
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = pd.read_csv(path/'train.csv', header=None)\n",
    "df_valid = pd.read_csv(path/'test.csv', header=None)\n",
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We gather all texts in one numpy array (since it will be easier to use this way with fastai):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_texts = np.concatenate([df_train[0].values, df_valid[0].values])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To process this data to train a model, we need to build a `Transform` that will be applied lazily. In this case we could do the pre-processing once and for all and only use the transform for decoding (we will see how just after), but the fast tokenizer from HuggingFace is, as its name indicates, fast, so it doesn't really impact performance to do it this way.\n",
    "\n",
    "In a fastai `Transform` you can define:\n",
    "\n",
    "- an <code>encodes</code> method that is applied when you call the transform (a bit like the `forward` method in a `nn.Module`)\n",
    "- a <code>decodes</code> method that is applied when you call the `decode` method of the transform, if you need to decode anything for showing purposes (like converting ids to a text here)\n",
    "- a <code>setups</code> method that sets some inner state of the `Transform` (not needed here so we skip it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformersTokenizer(Transform):\n",
    "    def __init__(self, tokenizer): self.tokenizer = tokenizer\n",
    "    def encodes(self, x): \n",
    "        toks = self.tokenizer.tokenize(x)\n",
    "        return tensor(self.tokenizer.convert_tokens_to_ids(toks))\n",
    "    def decodes(self, x): return TitledStr(self.tokenizer.decode(x.cpu().numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two comments on the code above:\n",
    "\n",
    "- in <code>encodes</code> we don't use the `tokenizer.encode` method since it does some additional preprocessing for the model after tokenizing and numericalizing (the part throwing a warning before). Here we don't need any post-processing so it's fine to skip it.\n",
    "- in <code>decodes</code> we return a `TitledStr` object and not just a plain string. That's a fastai class that adds a `show` method to the string, which will allow us to use all the fastai show methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then group your data with this `Transform` using a `TfmdLists`. It has an s in its name because it contains the training and validation set. We indicate the indices of the training set and the validation set with `splits` (here all the first indices until `len(df_train)` and then all the remaining indices):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4576 > 1024). Running this sequence through the model will result in indexing errors\n"
     ]
    }
   ],
   "source": [
    "splits = [range_of(df_train), list(range(len(df_train), len(all_texts)))]\n",
    "tls = TfmdLists(all_texts, TransformersTokenizer(tokenizer), splits=splits, dl_type=LMDataLoader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We specify `dl_type=LMDataLoader` for when we will convert this `TfmdLists` to `DataLoaders`: we will use an `LMDataLoader` since we have a language modeling problem, not the usual fastai `TfmdDL`.\n",
    "\n",
    "In a `TfmdLists` you can access the elements of the training or validation set quite easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([220, 198, 796,  ..., 198, 220, 198]),\n",
       " tensor([220, 198, 796,  ..., 198, 220, 198]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tls.train[0],tls.valid[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They look the same but only because they begin and end the same way. We can see the shapes are different:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([4576]), torch.Size([1485]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tls.tfms(tls.train.items[0]).shape, tls.tfms(tls.valid.items[0]).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can have a look at both decodes using `show_at`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \n",
      " = 2013 – 14 York City F.C. season = \n",
      " \n",
      " The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club, a professional football club based in York, North Yorkshire, England. Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two. The season ran from 1 July 2013 to 30 June 2014. \n",
      " Nigel Worthington, starting his first full season as York manager, made eight permanent summer signings. By the turn of the year York were only above the relegation zone on goal difference, before a 17 @-@ match unbeaten run saw the team finish in seventh @-@ place in the 24 @-@ team 2013 – 14 Football League Two. This meant York qualified for the play @-@ offs, and they were eliminated in the semi @-@ final by Fleetwood Town. York were knocked out of the 2013 – 14 FA Cup, Football League Cup and Football League Trophy in their opening round matches. \n",
      " 35 players made at least one appearance in nationally organised first @-@ team competition, and there were 12 different <unk>. Defender Ben Davies missed only five of the fifty @-@ two competitive matches played over the season. Wes Fletcher finished as leading scorer with 13 goals, of which 10 came in league competition and three came in the FA Cup. The winner of the <unk> of the Year award, voted for by the club's supporters, was <unk> Oyebanjo. \n",
      " \n",
      " = = Background and pre @-@ season = = \n",
      " \n",
      " The 2012 – 13 season was York City's first season back in the Football League, having won the Conference Premier play @-@ offs in 2011 – 12 after <unk> years in the Football Conference. Manager Gary Mills was sacked in March 2013 following an 11 @-@ match run without a victory, and was replaced by former Northern Ireland manager Nigel Worthington. Despite being in the relegation zone with three matches remaining, Worthington led the team to safety from relegation after a 1 – 0 win away to Dagenham & Redbridge on the final day of the season. York finished the season in 17th @-@ place in the 2012 – 13 League Two table. \n",
      " Following the previous season's conclusion Lee <unk>, Jon <unk>, Chris <unk>, Ben Everson, Scott Kerr, David <unk>, Patrick <unk>, Michael Potts, Jamie Reed and Jason Walker were released by York, while <unk> Blair departed for Fleetwood Town. David McGurk, <unk> Oyebanjo, Danny Parslow, Tom Platt and Chris Smith signed new contracts with the club. New players signed ahead of the start of the season were goalkeeper Chris <unk> on a season @-@ long loan from Blackpool, defender Ben Davies on loan from Preston North End, midfielders Craig Clay from Chesterfield and Lewis Montrose from Gillingham, winger <unk> Puri from St <unk> and strikers Ryan Bowman from Hereford United, Richard Cresswell from Sheffield United, Wes Fletcher from Burnley and Ryan Jarvis from Torquay United. Defender Mike Atkinson and striker Chris Dickinson entered the first @-@ team squad from the youth team after agreeing professional contracts. \n",
      " York retained the previous season's home and away kits. The home kit comprised red shirts with white sleeves, light blue shorts and white socks. The away kit included light blue shirts with white sleeves, white shorts and light blue socks. <unk> Health continued as shirt sponsors for the second successive season. \n",
      " \n",
      " = = Review = = \n",
      " \n",
      " \n",
      " = = = August = = = \n",
      " \n",
      " York began the season with a 1 – 0 home win over the previous season's play @-@ off finalists, Northampton Town, with <unk> Jarvis scoring the winning goal in the 90th @-@ minute. However, defeat came in York's match against Championship side Burnley in the first round of the League Cup, going down 4 – 0 at home. The team endured their first league defeat of the season in the following game after being beaten 2 – 0 away by Dagenham & Redbridge, the home team scoring in each half. York then held Hartlepool United to a 0 – 0 home draw, before being beaten 3 – 2 away by Bristol Rovers, in which Jarvis scored twice before John @-@ Joe O 'Toole scored the winning goal for the home team in the 67th @-@ minute. Two signings were made shortly before the transfer deadline ; defender George Taft was signed on a one @-@ month loan from Leicester City, while Middlesbrough midfielder Ryan Brobbel joined on a one @-@ month loan. <unk> John <unk>, who had been told he had no future with the club, departed after signing for FC Halifax Town. Jarvis gave York the lead away at Exeter City before Alan <unk> scored in each half to see the home team win 2 – 1. \n",
      " \n",
      " = = = September = = = \n",
      " \n",
      " York suffered their first home league defeat of the season after AFC Wimbledon won 2 – 0, with Michael Smith scoring in each half. Former Ipswich Town midfielder Josh Carson, who had a spell on loan with York the previous season, signed a contract until the end of 2013 – 14 and Sheffield United midfielder Elliott <unk> signed on a one @-@ month loan. Brobbel opened the scoring in the second minute of his home debut against Mansfield Town, although the away team went on to score twice to win 2 – 1. York's run of four defeats ended following a 1 – 1 draw away to Wycombe Wanderers, in which McGurk gave York the lead before the home team levelled through Dean Morgan. Taft was sent back to Leicester after he fell behind McGurk, Parslow and Smith in the pecking order for a central defensive berth. York achieved their first win since the opening day of the season after beating Portsmouth 4 – 2 at home, with Fletcher ( 2 ), Montrose and Jarvis scoring. \n",
      " \n",
      " = = = October = = = \n",
      " \n",
      " Defender Luke O 'Neill was signed from Burnley on a 28 @-@ day emergency loan. He made his debut in York's 3 – 0 win away at Torquay, which was the team's first successive win of the season. York were knocked out of the Football League Trophy in the second round after being beaten 3 – 0 at home by League One team Rotherham United, before their winning streak in the league was ended with a 3 – 0 defeat away to Newport County. York drew 2 – 2 away to Chesterfield, having taken a two @-@ goal lead through O 'Neill and Jarvis, before the home team fought back through Armand <unk> and Jay O <unk>. The team then hosted Fleetwood Town, and the visitors won 2 – 0 with goals scored in each half by Gareth Evans and <unk> Matt. Scunthorpe United were beaten 4 – 1 at home to end York's three @-@ match run without a win, with all the team's goals coming in the first half from Carson, Fletcher and Brobbel ( 2 ). \n",
      " \n",
      " = = = November = = = \n",
      " \n",
      " Bowman scored his first goals for York away to Cheltenham Town, as York twice fought back from behind to draw 2 – 2. York drew 3 – 3 away to Bristol Rovers to earn a first round replay in the FA Cup, taking the lead through Jarvis before Eliot Richards equalised for the home team. Carson scored a 30 yard volley to put York back in the lead, and after Bristol Rovers goals from Matt <unk> and Chris <unk>, Fletcher scored an 86th @-@ minute equaliser for York. Bowman scored with a header from an O 'Neill cross to open the scoring at home to Plymouth Argyle, which was the first goal the visitors had conceded in 500 minutes of action. However, Plymouth equalised 11 minutes later through <unk> <unk> and the match finished a 1 – 1 draw. York were knocked out of the FA Cup after losing 3 – 2 at home to Bristol Rovers in a first round replay ; the visitors were 3 – 0 up by 50 @-@ minutes before Fletcher pulled two back for York with a penalty and a long @-@ range strike. \n",
      " Defender Keith Lowe, of Cheltenham, and goalkeeper Nick Pope, of Charlton Athletic, were signed on loan until January 2014. They both played in York's first league defeat in four weeks, 2 – 1 away, to Southend United. <unk> <unk> gave Southend the lead early into the match and Bowman equalised for York with a low strike during the second half, before Luke Prosser scored the winning goal for the home side in stoppage time. With Pope preferred in goal, <unk> returned to Blackpool on his own accord, although his loan agreement would stay in place until January 2014. York then drew 0 – 0 away to Morecambe. After Pope was recalled from his loan by Charlton, York signed Wolverhampton Wanderers goalkeeper Aaron McCarey on loan until January 2014. McCarey kept a clean sheet in York's 0 – 0 home draw with Rochdale. \n",
      " \n",
      " = = = December = = = \n",
      " \n",
      " Cresswell retired from playing as a result of an eye complaint and a knee injury. York drew 1 – 1 away to Burton Albion, with an own goal scored by Shane <unk> @-@ <unk> giving York the lead in the 64th @-@ minute before the home team equalised eight minutes later through Billy <unk>. Atkinson was released after failing to force himself into the first team and signed for Scarborough Athletic, with whom he had been on loan. York drew 0 – 0 at home with second @-@ placed Oxford United, in which Carson came closest to scoring with a volley that <unk> across the face of the goal. This was followed by another draw after the match away to Accrington Stanley finished 1 – 1, with the home team <unk> 10 minutes after a Fletcher penalty had given York the lead in the 35th @-@ minute. Striker <unk> McDonald, who had been released by Peterborough United, was signed on a contract until the end of the season. York's last match of 2013 was a 2 – 1 defeat away at Bury, a result that ended York's run of consecutive draws at five. The home team were 2 – 0 up by the 19th @-@ minute, before Michael Coulson scored York's goal in the 73rd @-@ minute. This result meant York would begin 2014 in 22nd @-@ position in the table, only out of the relegation zone on goal difference. \n",
      " \n",
      " = = = January = = = \n",
      " \n",
      " Jarvis scored the only goal in York's first win since October 2013, a 1 – 0 home victory over Morecambe on New Year's Day. McCarey was recalled by Wolverhampton Wanderers due to an injury to one of their <unk>, while O 'Neill was recalled by Burnley to take part in their FA Cup match. York achieved back @-@ to @-@ back wins for the first time since October 2013 after Dagenham & Redbridge were beaten 3 – 1 at home, with Bowman opening the scoring in the second half before Fletcher scored twice. Adam Reed, who had a spell on loan with York in the previous season, was signed on a contract until the end of the season after parting company with Burton. Davies'loan was extended, while Brobbel and <unk> returned to their parent clubs. Cheltenham club captain Russell Penn, a midfielder, was signed on a two @-@ and @-@ a @-@ half @-@ year contract for an undisclosed fee. Lowe was subsequently signed permanently from Cheltenham on a two @-@ and @-@ a @-@ half @-@ year contract for an undisclosed fee. Having been allowed to leave the club on a free transfer, Ashley Chambers signed for Conference Premier club Cambridge United. \n",
      " York achieved three successive wins for the first time in 2013 – 14 after beating Northampton 2 – 0 away, with Bowman and Fletcher scoring in three @-@ second half minutes. Defender John McCombe was signed on a two @-@ and @-@ a @-@ half @-@ year contract following his release from Mansfield, before Clay and Jamal <unk> left York by mutual consent. Pope returned to York on loan from Charlton for the remainder of the season. York's run of wins ended with a 0 – 0 draw at home to Bristol Rovers, before their first defeat of the year came after losing 2 – 0 away to Hartlepool. Preston winger Will Hayhurst, a Republic of Ireland under @-@ 21 international, was signed on a one @-@ month loan. York fell to a successive defeat for the first time since September 2013 after being beaten 2 – 0 at home by Chesterfield. Shortly after the match, Smith left the club by mutual consent to pursue first @-@ team football. \n",
      " \n",
      " = = = February = = = \n",
      " \n",
      " Fletcher scored a 90th @-@ minute winner for York away to Fleetwood in a 2 – 1 win, a result that ended Fleetwood's five @-@ match unbeaten run. York then drew 0 – 0 at home to fellow mid @-@ table team Cheltenham, before beating Plymouth 4 – 0 away with goals from Fletcher, McCombe ( 2 ) and Carson as the team achieved successive away wins for the first time in 2013 – 14. York went without scoring for a fourth consecutive home match after drawing 0 – 0 with Southend. Having worn the <unk> since an injury to McGurk, Penn was appointed captain for the rest of the season, a position that had earlier been held by Smith and Parslow. \n",
      " \n",
      " = = = March = = = \n",
      " \n",
      " York achieved their first home win in five matches after beating Exeter 2 – 1, with first half goals scored by McCombe and Coulson. Hayhurst's loan was extended to the end of the season, having impressed in his six appearances for the club. Coulson scored again with the only goal, a 41st @-@ minute header, in York's 1 – 0 away win over AFC Wimbledon. Bowman scored the only goal with a 32nd @-@ minute penalty as York won 1 – 0 away against Mansfield, in which Fletcher missed the opportunity to extend the lead when his stoppage time penalty was saved by Alan Marriott. York moved one place outside the play @-@ offs with a 2 – 0 home win over Wycombe, courtesy of a second Bowman penalty in as many matches and a Carson goal from the edge of the penalty area. Coulson scored York's only goal in a 1 – 0 away win over struggling Portsmouth with a low volley in the fifth @-@ minute ; this result meant York moved into the play @-@ offs in seventh @-@ place with eight fixtures remaining. \n",
      " Striker Calvin Andrew, who had been released by Mansfield in January 2014, was signed on a contract for the remainder of the season. He made his debut as a substitute in York's 1 – 0 home win over bottom of the table Torquay, in which Hayhurst scored the only goal in the 11th @-@ minute with an 18 yard shot that <unk> off Aaron <unk>. Middlesbrough winger Brobbel rejoined on loan until the end of the season, following an injury to Carson. York's run of successive wins ended on six matches after a 0 – 0 home draw with Burton, and this result saw York drop out of the play @-@ offs in eighth @-@ place. With the team recording six wins and one draw in March 2014, including six clean sheets, Worthington was named League Two Manager of the Month. \n",
      " \n",
      " = = = April = = = \n",
      " \n",
      " Pope made a number of saves as York held league leaders Rochdale to a 0 – 0 away draw, with a point being enough to lift the team back into seventh @-@ place. York were prevented from equalling a club record of eight consecutive clean sheets when Accrington scored a stoppage time equaliser in a 1 – 1 home draw, in which York had taken earlier taken the lead with a Coulson penalty. A 1 – 0 win away win over Oxford, which was decided by a second half Coulson penalty, resulted in York moving one place above their opponents and back into seventh @-@ place. York consolidated their place in a play @-@ off position after beating Bury 1 – 0 at home with a fifth @-@ minute goal scored by Lowe from a Hayhurst corner. The result meant York opened up a five @-@ point lead over eighth @-@ placed Oxford with two fixtures remaining. A place in the League Two play @-@ offs was secured following a 1 – 0 win over Newport at home, in which Coulson scored the only goal in the 77th @-@ minute with a 25 yard free kick. Pope earned a nomination for League Two Player of the Month for April 2014, having conceded only one goal in five matches in that period. \n",
      " \n",
      " = = = May = = = \n",
      " \n",
      " The league season concluded with an away match against divisional runners @-@ up Scunthorpe ; having gone two goals down York fought back to draw 2 – 2 with goals scored by Brobbel and Andrew. This result meant York finished the season in seventh @-@ place in League Two, and would thus play fourth @-@ placed Fleetwood in the play @-@ off semi @-@ final on the back of a 17 @-@ match unbeaten run. York lost 1 – 0 to Fleetwood in the first leg at <unk> Crescent ; the goal came from former York player <unk> Blair in the 50th @-@ minute, who scored from close range after Antoni <unk>'s shot was blocked on the line. A 0 – 0 draw away to Fleetwood in the second leg meant York were eliminated 1 – 0 on aggregate, ending the prospect of a second promotion in three seasons. At an awards night held at York Racecourse, Oyebanjo was voted <unk> of the Year for 2013 – 14. \n",
      " \n",
      " = = Summary and aftermath = = \n",
      " \n",
      " York mostly occupied the bottom half of the table before the turn of the year, and dropped as low as 23rd in September 2013. During February 2014 the team broke into the top half of the table and with one match left were in sixth @-@ place. York's defensive record was the third best in League Two with 41 goals conceded, bettered only by Southend ( 39 ) and Chesterfield ( 40 ). Davies made the highest number of appearances over the season, appearing in 47 of York's 52 matches. Fletcher was York's top scorer in the league and in all competitions, with 10 league goals and 13 in total. He was the only player to reach double figures, and was followed by Jarvis with nine goals. \n",
      " After the season ended York released Tom Allan, Andrew, Dickinson, McDonald, Puri and Reed, while McGurk retired from professional football. Bowman and Oyebanjo left to sign for Torquay and Crawley Town respectively while Coulson signed a new contract with the club. York's summer signings included goalkeeper Jason <unk> from Tranmere Rovers, defenders <unk> <unk> from Dagenham, Marvin McCoy from Wycombe and Dave Winfield from Shrewsbury Town, midfielders <unk> <unk> from Mansfield, Anthony <unk> from Southend and Luke <unk> from Shrewsbury and striker Jake Hyde from <unk>. \n",
      " \n",
      " = = Match details = = \n",
      " \n",
      " League positions are sourced by <unk>, while the remaining information is referenced individually. \n",
      " \n",
      " = = = Football League Two = = = \n",
      " \n",
      " \n",
      " = = = League table ( part ) = = = \n",
      " \n",
      " \n",
      " = = = FA Cup = = = \n",
      " \n",
      " \n",
      " = = = League Cup = = = \n",
      " \n",
      " \n",
      " = = = Football League Trophy = = = \n",
      " \n",
      " \n",
      " = = = Football League Two play @-@ offs = = = \n",
      " \n",
      " \n",
      " = = <unk> = = \n",
      " \n",
      " \n",
      " = = = In = = = \n",
      " \n",
      " <unk> around club names denote the player's contract with that club had expired before he joined York. \n",
      " \n",
      " = = = Out = = = \n",
      " \n",
      " <unk> around club names denote the player joined that club after his York contract expired. \n",
      " \n",
      " = = = Loan in = = = \n",
      " \n",
      " \n",
      " = = = Loan out = = = \n",
      " \n",
      " \n",
      " = = Appearances and goals = = \n",
      " \n",
      " Source : \n",
      " Numbers in parentheses denote appearances as substitute. \n",
      " Players with names struck through and marked left the club during the playing season. \n",
      " Players with names in italics and marked * were on loan from another club for the whole of their season with York. \n",
      " Players listed with no appearances have been in the <unk> squad but only as unused <unk>. \n",
      " Key to positions : <unk> – <unk> ; <unk> – Defender ; <unk> – <unk> ; <unk> – Forward \n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "show_at(tls.train, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \n",
      " = Tropical Storm <unk> ( 2008 ) = \n",
      " \n",
      " Tropical Storm <unk> was the tenth tropical storm of the 2008 Atlantic hurricane season. <unk> developed out of a strong tropical wave which moved off the African coast on August 31. The wave quickly became organized and was declared Tropical Depression Ten while located 170 mi ( 270 km ) to the south @-@ southeast of the Cape Verde Islands on September 2. The depression was quickly upgraded to Tropical Storm <unk> around noon the same day. Over the next several days, <unk> moved in a general west @-@ northwest direction and reached its peak intensity early on September 3. Strong wind shear, some due to the outflow of Hurricane Ike, and dry air caused the storm to weaken. On September 6, the combination of wind shear, dry air, and cooling waters caused <unk> to weaken into a tropical depression. <unk> deteriorated into a remnant low shortly after as convection continued to dissipate around the storm. The low ultimately dissipated while located 520 mi ( 835 km ) east of <unk> on September 10. However, the remnant moisture led to minor flooding on the island of St. Croix. \n",
      " \n",
      " = = Meteorological history = = \n",
      " \n",
      " Tropical Storm <unk> formed as a tropical wave that emerged off the west coast of Africa near the end of August 2008. It tracked south of Cape Verde and slowly developed, and on September 2 the disturbance became Tropical Depression Ten while located south @-@ southeast of the Cape Verde islands. As the depression became more organized, an eye @-@ like feature developed in the upper levels of the system. The depression was upgraded to Tropical Storm <unk> six hours after forming. <unk> was located in an area which was supportive for rapid intensification but was not forecast to intensify quickly. \n",
      " <unk> continued to intensify throughout the afternoon as the storm became more symmetrical. However, due to the location of the storm, there was a lack of accurate wind speed readings, and the National Hurricane Center was uncertain of its actual intensity. Despite the lack of wind shear around the storm, the center became slightly exposed and ceased further intensification. The storm was also heading into an area where shear was <unk> to significantly increase due to an upper @-@ level trough diving southward. Despite convection being partially removed from the center of <unk>, the storm intensified slightly in the early morning hours on September 3 as thunderstorm activity to the south of the center became more organized. The intensification was forecast to be short in duration as the trough to the north was deepening, causing the wind shear to the west to become stronger. \n",
      " <unk> reached its peak intensity of 65 mph ( 100 km / h ) around 8 a.m. ( <unk> ) as it continued to become more organized. However, there were indications that it had already begun to weaken. <unk> towards the north was becoming restricted and arc clouds began emanating from the storm, a sign that dry air was entering the system. During the afternoon hours, the structure of <unk> began to rapidly deteriorate as strong wind shear and dry air took their toll. By the late night, the center was almost completely exposed and only a band of convection persisted near the center. \n",
      " Despite continuing effects from the strong wind shear, a large, deep burst of convection formed in the northern <unk> of <unk>. The center was found to have shifted towards the new convection leading to an increase in intensity. The forecast showed a slight decrease in wind shear as <unk> continued westward and no change in intensity over the 5 @-@ day forecast was predicted. However, the convection decreased once more and the low became completely exposed by the late morning hours and <unk> weakened again. By the afternoon, the center of <unk> was only a <unk> of clouds, devoid of convection. During the overnight hours on September 4 into the morning of September 5, convection associated with <unk> began to <unk> somewhat, mostly to the north of the circulation, due to the strong <unk> wind shear. By mid @-@ morning, <unk> re @-@ intensified slightly due to the redevelopment of some convection. However, the redevelopment was short lived and wind shear again took its toll on <unk> by late morning. The convection around the system became <unk> from the center and <unk> weakened slightly. \n",
      " The weakening trend continued through the afternoon as the storm was being affected by strong <unk> shear. <unk> became almost fully devoid of any convection by mid @-@ afternoon and the storm weakened to 40 mph ( 65 km / h ), barely holding on to tropical storm status. <unk> regained a small amount of convection in the late night hours, but not enough to still be classified a tropical storm. Due to the lack of convection, <unk> was downgraded to a Tropical Depression at <unk> ( <unk> ) with winds of 35 mph ( 55 km / h ). Since there was no convection around the system, it would have normally been classified a remnant low but, due to the possibility of the storm <unk> over the next several days, it was considered a tropical depression. The next morning, <unk> was downgraded to a remnant low as strong wind shear and dry air caused the demise of the storm. No redevelopment was expected with <unk> as it began to move over colder waters and remain under strong wind shear until it dissipated. \n",
      " However, the remnant low associated with <unk> began to show signs of redevelopment during the afternoon on September 7. <unk> around the system increased significantly and the low was no longer exposed. On September 8, wind shear took over the system again. <unk> around the remnant low was torn away and the low was exposed once more. The National Hurricane Center did not state the chance of regeneration once the low became exposed. Finally, on September 9, wind shear and dry air led to the remnants of <unk> deteriorating into an open wave. However, on September 10, the remnants of <unk> redeveloped and global models picked up on the reformed system. Once more, the chance of regeneration was possible as the remnants of <unk> headed towards the Bahamas. However, on September 14, dry air and wind shear caused the remnants to dissipate entirely. \n",
      " \n",
      " = = Impact = = \n",
      " \n",
      " As <unk> passed to the south of the Cape Verde islands on September 2, outer rain bands produced minor rainfall, totaling around 0 @.@ 55 inches ( 14 mm ). There were no reports of damage or flooding from the rain and overall effects were minor. \n",
      " Several days after the low dissipated, the remnant moisture from <unk> brought showers and thunderstorms to St. Croix where up to 1 in ( 25 @.@ 4 mm ) of rain fell. The heavy rains led to minor street flooding and some urban flooding. No known damage was caused by the flood. \n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "show_at(tls.valid, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fastai library expects the data to be assembled in a `DataLoaders` object (something that has a training and validation dataloader). We can get one by using the `dataloaders` method. We just have to specify a batch size and a sequence length. We'll train with sequences of size 256 (GPT2 used sequence length 1024, but not everyone has enough GPU RAM for that):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs,sl = 4,256\n",
    "dls = tls.dataloaders(bs=bs, seq_len=sl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that you may have to reduce the batch size depending on your GPU RAM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fastai, as soon as we have a `DataLoaders`, we can use `show_batch` to have a look at the data (here texts for inputs, and the same text shifted by one token to the right for validation):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n = Jacqueline Fernandez = \\n \\n Jacqueline Fernandez ( born 11 August 1985 ) is a Sri Lankan actress, former model, and the winner of the 2006 Miss Universe Sri Lanka pageant. As Miss Universe Sri Lanka she represented her country at the 2006 world Miss Universe pageant. She graduated with a degree in mass communication from the University of Sydney, and worked as a television reporter in Sri Lanka. \\n While on a modelling assignment in India in 2009, Fernandez successfully auditioned for &lt;unk&gt; &lt;unk&gt;'s fantasy drama &lt;unk&gt;, which marked her acting debut. Fernandez'breakthrough role was in &lt;unk&gt; &lt;unk&gt;'s psychological thriller Murder 2 ( 2011 ), her first commercial success. This was followed by glamorous roles in the ensemble @-@ comedy Housefull 2 ( 2012 ) and its sequel Housefull 3, and the action thriller Race 2 ( 2013 ), all of which were box @-@ office</td>\n",
       "      <td>\\n = Jacqueline Fernandez = \\n \\n Jacqueline Fernandez ( born 11 August 1985 ) is a Sri Lankan actress, former model, and the winner of the 2006 Miss Universe Sri Lanka pageant. As Miss Universe Sri Lanka she represented her country at the 2006 world Miss Universe pageant. She graduated with a degree in mass communication from the University of Sydney, and worked as a television reporter in Sri Lanka. \\n While on a modelling assignment in India in 2009, Fernandez successfully auditioned for &lt;unk&gt; &lt;unk&gt;'s fantasy drama &lt;unk&gt;, which marked her acting debut. Fernandez'breakthrough role was in &lt;unk&gt; &lt;unk&gt;'s psychological thriller Murder 2 ( 2011 ), her first commercial success. This was followed by glamorous roles in the ensemble @-@ comedy Housefull 2 ( 2012 ) and its sequel Housefull 3, and the action thriller Race 2 ( 2013 ), all of which were box @-@ office successes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>small farms in between small residential subdivisions. In the community of Freeland, M @-@ 47 runs near the &lt;unk&gt; International Airport off Freeland Road. North of town, M @-@ 47 leaves Midland Road and becomes a freeway near &lt;unk&gt; Park. The freeway section of M @-@ 47 runs through rural farm land. There is a diamond interchange with &lt;unk&gt; Road before the terminal interchange at US 10. \\n As part of its maintenance duties, the Michigan Department of Transportation ( MDOT ) tracks the volume of traffic on the highways it maintains. This number is expressed in terms of annual average daily traffic ( AADT ), a calculation of the average traffic for a segment of roadway on any average day of the year. In 2009, the department measured a peak of 19 @,@ &lt;unk&gt; vehicles daily on the stretch north of &lt;unk&gt; Road. The section south of the</td>\n",
       "      <td>farms in between small residential subdivisions. In the community of Freeland, M @-@ 47 runs near the &lt;unk&gt; International Airport off Freeland Road. North of town, M @-@ 47 leaves Midland Road and becomes a freeway near &lt;unk&gt; Park. The freeway section of M @-@ 47 runs through rural farm land. There is a diamond interchange with &lt;unk&gt; Road before the terminal interchange at US 10. \\n As part of its maintenance duties, the Michigan Department of Transportation ( MDOT ) tracks the volume of traffic on the highways it maintains. This number is expressed in terms of annual average daily traffic ( AADT ), a calculation of the average traffic for a segment of roadway on any average day of the year. In 2009, the department measured a peak of 19 @,@ &lt;unk&gt; vehicles daily on the stretch north of &lt;unk&gt; Road. The section south of the US</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(max_n=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to gather the data is to preprocess the texts once and for all and only use the transform to decode the tensors to texts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='662' class='' max='662' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [662/662 00:12<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def tokenize(text):\n",
    "    toks = tokenizer.tokenize(text)\n",
    "    return tensor(tokenizer.convert_tokens_to_ids(toks))\n",
    "\n",
    "tokenized = [tokenize(t) for t in progress_bar(all_texts)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we change the previous `Tokenizer` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformersTokenizer(Transform):\n",
    "    def __init__(self, tokenizer): self.tokenizer = tokenizer\n",
    "    def encodes(self, x): \n",
    "        return x if isinstance(x, Tensor) else tokenize(x)\n",
    "        \n",
    "    def decodes(self, x): return TitledStr(self.tokenizer.decode(x.cpu().numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the <code>encodes</code> method, we still account for the case where we get something that's not already tokenized, just in case we were to build a dataset with new texts using this transform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tls = TfmdLists(tokenized, TransformersTokenizer(tokenizer), splits=splits, dl_type=LMDataLoader)\n",
    "dls = tls.dataloaders(bs=bs, seq_len=sl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can check it still works properly for showing purposes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n = Otra Nota = \\n \\n Otra Nota ( English : Another Note ) is the debut album by American singer Marc Anthony that was released on January 26, 1993, by RMM Records. Produced by Sergio George, it was the first album by Anthony to record in salsa after starting his career as a freestyle musician. Recording of the album began after Anthony asked RMM president Ralph Mercado to record Juan Gabriel's \" Hasta Que Te Conocí \" in salsa after hearing it on the radio during a taxi ride. Recorded on a low budget, the album peaked at No. 2 on the Billboard Tropical Albums chart and reached No. 30 on the Billboard Top Latin Albums chart. \\n The album was well received by critics who complimented George's production and Anthony's youthful voice. Anthony received two awards for \" Best New Artists \" at the Billboard Latin</td>\n",
       "      <td>\\n = Otra Nota = \\n \\n Otra Nota ( English : Another Note ) is the debut album by American singer Marc Anthony that was released on January 26, 1993, by RMM Records. Produced by Sergio George, it was the first album by Anthony to record in salsa after starting his career as a freestyle musician. Recording of the album began after Anthony asked RMM president Ralph Mercado to record Juan Gabriel's \" Hasta Que Te Conocí \" in salsa after hearing it on the radio during a taxi ride. Recorded on a low budget, the album peaked at No. 2 on the Billboard Tropical Albums chart and reached No. 30 on the Billboard Top Latin Albums chart. \\n The album was well received by critics who complimented George's production and Anthony's youthful voice. Anthony received two awards for \" Best New Artists \" at the Billboard Latin Music</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>reactions and prejudices \", which leaves no room for any further interest. Donoghue complained that Lessing has not made up her mind on whether her characters are \" the salt of the earth or its &lt;unk&gt; \". In a review in the Chicago Tribune, Kuehn felt that the work has little impact and is not memorable. He said Lessing's real interest is character development, but complained that the characters are \" trivial or two @-@ dimensional or crippled by self @-@ &lt;unk&gt; \". \\n The Good Terrorist was shortlisted for the 1985 Booker Prize, and in 1986 won the &lt;unk&gt; Prize and the &lt;unk&gt; Smith Literary Award. In 2007 Lessing was awarded the Nobel Prize in Literature for being \" part of both the history of literature and living literature \". In the award ceremony speech by Swedish writer Per &lt;unk&gt;, The Good Terrorist was cited as \" an</td>\n",
       "      <td>and prejudices \", which leaves no room for any further interest. Donoghue complained that Lessing has not made up her mind on whether her characters are \" the salt of the earth or its &lt;unk&gt; \". In a review in the Chicago Tribune, Kuehn felt that the work has little impact and is not memorable. He said Lessing's real interest is character development, but complained that the characters are \" trivial or two @-@ dimensional or crippled by self @-@ &lt;unk&gt; \". \\n The Good Terrorist was shortlisted for the 1985 Booker Prize, and in 1986 won the &lt;unk&gt; Prize and the &lt;unk&gt; Smith Literary Award. In 2007 Lessing was awarded the Nobel Prize in Literature for being \" part of both the history of literature and living literature \". In the award ceremony speech by Swedish writer Per &lt;unk&gt;, The Good Terrorist was cited as \" an in</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(max_n=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fine-tuning the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HuggingFace model will return a tuple in outputs, with the actual predictions and some additional activations (should we want to use them in some regularization scheme). To work inside the fastai training loop, we will need to drop those using a `Callback`: we use those to alter the behavior of the training loop.\n",
    "\n",
    "Here we need to write the event `after_pred` and replace `self.learn.pred` (which contains the predictions that will be passed to the loss function) by just its first element. In callbacks, there is a shortcut that lets you access any of the underlying `Learner` attributes so we can write `self.pred[0]` instead of `self.learn.pred[0]`. That shortcut only works for read access, not write, so we have to write `self.learn.pred` on the right side (otherwise we would set a `pred` attribute in the `Callback`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DropOutput(Callback):\n",
    "    def after_pred(self): self.learn.pred = self.pred[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course we could make this a bit more complex and add some penalty to the loss using the other part of the tuple of predictions, like the `RNNRegularizer`.\n",
    "\n",
    "Now, we are ready to create our `Learner`, which is a fastai object grouping data, model and loss function and handles model training or inference. Since we are in a language model setting, we pass perplexity as a metric, and we need to use the callback we just defined. Lastly, we use mixed precision to save every bit of memory we can (and if you have a modern GPU, it will also make training faster):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), cbs=[DropOutput], metrics=Perplexity()).to_fp16()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check how good the model is without any fine-tuning step (spoiler alert, it's pretty good!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(#2) [3.2537169456481934,25.88637924194336]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.validate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This lists the validation loss and metrics (so 26.6 as perplexity is kind of amazing).\n",
    "\n",
    "Now that we have a `Learner` we can use all the fastai training loop capabilities: learning rate finder, training with 1cycle etc... "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.017378008365631102, lr_steep=0.14454397559165955)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEKCAYAAAAVaT4rAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfU0lEQVR4nO3deZxcZZ3v8c+vqrp675ClgyyGJGyKOGytbIow4Iw6KIjLdcEBBxO8LqPOSx196R3Gmesdx5fj9eowakBBR8QFYWREQUVZFAWbVSAESIISlnQngXT1Uvvv/nFOd5q20+nudJ1TVef7fr3q1VWnTtXze7q6f+c5z3nqeczdERGR5EjFHYCIiERLiV9EJGGU+EVEEkaJX0QkYZT4RUQSRolfRCRhMnEHMBvLli3zlStXxh2GiEhDufPOO7e5e+/U7Q2R+FeuXEl/f3/cYYiINBQz+8N029XVIyKSMEr8IiIJo8QvIpIwSvwiIgmjxC8ikjBK/CIiCaPELyJSh3aOlrjhgacZzBUW/L2V+EVE6tCjgzku/M87eeDJnQv+3kr8IiJ1aChfBqC7rWXB31uJX0SkDuXCxN/TtvATLCjxi4jUoWG1+EVEkiWXLwHQ3UgtfjP7upkNmNn9k7a9ycweMLOqmfXVqmwRkUaXy5dJp4yObHrB37uWLf7LgVdN2XY/cA5wSw3LFRFpeLl8ia7WDGa24O9ds2mZ3f0WM1s5Zdt6oCYVERFpJrl8uSbdPFDHffxmttbM+s2sf3BwMO5wREQiNZQv1+TCLtRx4nf3de7e5+59vb1/soCMiEhTy+VLyWvxi4gkWS5frskYflDiFxGpS7lCqfG6eszsSuA3wOFmtsXMLjCz15vZFuBE4Dozu6FW5YuINLJaXtyt5aiet+7mqWtqVaaISDNw92SO6hERSaqxUoVK1Ruvq0dEROYnNzFPj1r8IiKJsGueHrX4RUQSYUgtfhGRZJno6mlV4hcRSQR19YiIJIwu7oqIJEwtF2EBJX4RkboznC9jBp1ZJX4RkUQYypfpas2QStVm7RIlfhGROhPMzFmbC7ugxC8iUndqORc/KPGLiNSdWk7QBkr8IiJ1p5Zz8YMSv4hI3VGLX0QkYRo28ZvZ181swMzun7RtiZn9zMweCX8urlX5IiKNKFiEpXG7ei4HXjVl28eAG939UODG8LGIiIQK5Sqlijdmi9/dbwF2TNl8FvCN8P43gLNrVb6ISCMaqvEEbRB9H/++7v4UQPhz+e52NLO1ZtZvZv2Dg4ORBSgiEqfxCdp6GrHFv7fcfZ2797l7X29vb9zhiIhEotYzc0L0iX+rme0HEP4ciLh8EZG6Vuu5+CH6xH8tcF54/zzghxGXLyJS1xq6xW9mVwK/AQ43sy1mdgHwGeCVZvYI8MrwsYiIhMZb/F01WnYRoGbv7O5v3c1Tp9eqTBGRRrerxd88XT0iIjKDoTDx17LFr8QvIlJHcvkSXa0Z0jVahAWU+EVE6kqt5+kBJX4RkboyrMQvIpIstZ6LH5T4RUTqirp6REQSJkj8avGLiCRGrRdaByV+EZG6MqSuHhGR5CiUKxTLVXrU1SMikgxRTNAGSvwiInVDiV9EJGEm5uJvVVePiEgiqMUvIpIwUay+BUr8IiJ1Y6iZW/xm9gEzu9/MHjCzD8YRg4hIvRnv6mm64ZxmdiSwBngpcBRwppkdGnUcIiL1Zryrp7M1XdNy4mjxvxD4rbuPunsZuBl4fQxxiIjUlVy+TEc2TSZd29QcR+K/HzjFzJaaWQfwGuD5U3cys7Vm1m9m/YODg5EHKSIStSjm6YEYEr+7rwf+FfgZcD1wL1CeZr917t7n7n29vb0RRykiEr0oZuaEmC7uuvvX3P1Ydz8F2AE8EkccIiL1JIq5+AFqX8I0zGy5uw+Y2QrgHODEOOIQEaknuUKZRe21b/HHkviBH5jZUqAEvNfdn4kpDhGRupHLlzhwcXvNy4kl8bv7y+MoV0SknuXyZXqa8eKuiIhMLxjV06QXd0VE5LlKlSr5UpXuVrX4RUQSIaqZOUGJX0SkLkQ1Myco8YuI1IXxFn+XWvwiIsmgrh4RkYQZ7+qp9ZTMoMQvIlIXhgthV49G9YiIJIO6ekREEmaixa/ELyKSDEP5EtlMitZMbVffAiV+EZG6MJwvR/KtXVDiFxGpC8OFaObiByV+EZG6kMuXI+nfByV+EZG6EHT11H4MP8SU+M3sQ2b2gJndb2ZXmllbHHGIiNSLoXypeVv8ZnYA8LdAn7sfCaSBt0Qdh4hIPUlCH38GaDezDNABPBlTHCIidSHXzKN63P0J4HPAH4GngJ3u/tOo4xARqRfuHrb4m7SP38wWA2cBq4D9gU4zO3ea/daaWb+Z9Q8ODkYdpohIZMZKFSpVb94+fuAMYLO7D7p7CbgaOGnqTu6+zt373L2vt7c38iBFRKIyHOE8PRBP4v8jcIKZdZiZAacD62OIQ0SkLgzlo5uZE+Lp478duAq4C/h9GMO6qOMQEakX4xO0RdXij6aUKdz9IuCiOMoWEak3Ua63C/rmrohI7IabvatHRESeK8pFWECJX0QkdrnxPv56mqvHzDrNLBXeP8zMXmdm0UQoItLkxvv4620c/y1AWzjPzo3AO4HLaxWUiEiSDOfLdGTTpFMWSXmzTfzm7qPAOcCX3P31wBG1C0tEJDly+egmaIM5JH4zOxF4O3BduC2WoaAiIs1muFCObEQPzD7xfxD4OHCNuz9gZquBX9YsKhGRBBnKlyIbww+zbLW7+83AzQDhRd5t7v63tQxMRCQpopyLH2Y/qufbZtZjZp3Ag8AGM/tIbUMTEUmG4Trt4z/C3YeAs4EfAyuAd9QqKBGRJMnl67OPvyUct3828MNwOmWvWVQiIgkS5SIsMPvE/1XgMaATuMXMDgKGahWUiEhSVKoe+aie2V7c/SLwxUmb/mBmp9UmJBGR5BgpRjtPD8z+4u4iM/v8+FKIZvZvBK1/ERHZC1FP0Aaz7+r5OpAD3hzehoDLahWUiEhS7Fp2sc7G8QMHu/sbJj3+lJndM58Czexw4LuTNq0G/sHdvzCf9xMRaWQTE7TVWx8/MGZmL3P3XwGY2cnA2HwKdPcNwNHh+6SBJ4Br5vNeIiKNbnxK5qhm5oTZJ/53A980s0Xh42eA8xag/NOBje7+hwV4LxGRhjPex99Tb4nf3e8FjjKznvDxkJl9ELhvL8t/C3DlXr6HiEjD2rXsYv2N4weChB9+gxfg7/amYDPLAq8Dvr+b59eOjyIaHBzcm6JEROrWroXW629Uz3T2dsWAVwN3ufvW6Z5093Xu3ufufb29vXtZlIhIfRoulEkZdGTTkZW5N4l/b6dseCvq5hGRhBufp8csmtW3YA99/GaWY/oEb0D7fAs1sw7glcCF830PEZFmEKy+Fe0S5jMmfnfvrkWh4TKOS2vx3iIijSSXL0Xavw9719UjIiJ7KeoJ2kCJX0QkVlEvtA5K/CIisRoulOmKuI9fiV9EJEZq8YuIJEwuX6JbffwiIslQLFcplKtq8YuIJMXw+MycavGLiCRDHIuwgBK/iEhshsYXYVFXj4hIMox39aiPX0QkISYWWo9wLn5Q4hcRic1wQV09IiKJMtHiV+IXEUmGXF7DOUVEEiWXL5NNp2hriW71LVDiFxGJzXChFHn/PsSU+M1sHzO7ysweMrP1ZnZiHHGIiMQpjgnaYA8rcNXQ/wOud/c3mlkW6IgpDhGR2Azno1+EBWJI/GbWA5wCnA/g7kWgGHUcIiJxi6vFH0dXz2pgELjMzO42s0vNrHPqTma21sz6zax/cHAw+ihFRGosVyjTFfGXtyCexJ8BjgW+7O7HACPAx6bu5O7r3L3P3ft6e3ujjlFEpOZy+RI9CWnxbwG2uPvt4eOrCA4EIiKJEiy7mIDE7+5PA4+b2eHhptOBB6OOQ0QkTu6euFE97weuCEf0bALeGVMcIiKxyJeqVKoeSx9/LInf3e8B+uIoW0SkHuTCufiTMqpHRCTxfnL/0wC8cL/uyMtW4hcRiVixXOWrN2/kJSsXc9xBSyIvX4lfRCRi/3X3Ezy5M897TjsklvKV+EVEIlSpOl++eSNHHtDDqYfF8x0lJX4RkQj9+PdPsXnbCO899RDMLJYYlPhFRCLi7lz8y0c5uLeTv3zR82KLQ4lfRCQiN64f4KGnc7zn1ENIpeJp7YMSv4hIJNydf//loxy4uJ3XHb1/rLEo8YuIROC2jdu55/FnufAVB9OSjjf1KvGLiETg4l8+yvLuVt503IFxh6LELyJSa3f98Rlu27idNS9fHfnC6tNR4hcRqbGLf/Eo+3S08LbjV8QdCqDELyJSUw8+OcSNDw3wNyevojOG9XWno8QvIlJD/3HTo3S1ZjjvxJVxhzJBiV9EpEY2DQ5z3e+f4twTDmJRR/Tz7u+OEr+ISI185eaNZNMpLnjZqrhDeY5YOpzM7DEgB1SAsrtrURYRaSobB4e5+q4nOPeEg+jtbo07nOeI80rDae6+LcbyRURq5p9/9CDtLWne9+fxTL08E3X1iIgssF88tJWbNgzygTMOZVlXfbX2Ib7E78BPzexOM1s73Q5mttbM+s2sf3BwMOLwRETmp1iu8s8/Ws/q3k7+uo5G8kwWV+I/2d2PBV4NvNfMTpm6g7uvc/c+d+/r7Y1nsQIRkbm67Neb2bxthH848wiymfrsVIklKnd/Mvw5AFwDvDSOOEREFtLAUJ4v3vgIp79gOacevjzucHYr8sRvZp1m1j1+H/gL4P6o4xARWWj/ev0GipUqnzzziLhDmVEco3r2Ba4JlxzLAN929+tjiENEZME8+OQQP7hrCxeesppVyzrjDmdGkSd+d98EHBV1uSIitfT5n22guy3De06tv+GbU9XnlQcRkQZy1x+f4efrB7jwlNV1NTXD7ijxi4jspc/dsIGlnVneeXJ9Tc2wO0r8IiJ74dePbuO2jdt572mH1M20y3uixC8iMk/uzmdv2MB+i9rqZpGV2VDiFxGZp5+vH+Dex5/lA6cfWhdLKs6WEr+IyDw8M1LkMz9Zz6plnbyhDhZQn4vG6JASEakjA7k877j0Dh5/Zoyvn/cSWtKN1YZW4hcRmYMnnh3j7Zf8lq1DBS47/yWcfMiyuEOaMyV+EZFZ2rxthHMvvZ2hfIlvveulHHfQkrhDmhclfhGRWXhka463XXo7lapz5ZoTOPKARXGHNG9K/CIie/DAkzt5x9fuIJMyvrv2BA7dtzvukPZKY12REBGJ2L2PP8tb1/2WtkyK7114YsMnfVCLX0Rkt/of28H5l/2OJZ1ZrnjX8Tx/SUfcIS0IJX4RkWn8dtN2/uby3/G8njauWHM8+y1qjzukBaPELyIyxa8f3cYF3/gdBy7u4Ntrjmd5d1vcIS0oJX4RkUlueXiQNd/sZ+XSTq5YczzLulrjDmnBxXZx18zSZna3mf0orhhERCa7cf1W3vXNflYt6+TbTZr0Id4W/weA9UBPjDGIiFAoV/jcDRu45NbNvGj/Hr51wfEs7szGHVbNxNLiN7MDgb8CLo2jfBGRcQ9vzXH2xbdxya2befvxK7jq3Sc1ddKH+Fr8XwA+Cux2QKyZrQXWAqxY0TjzXItIY6hWnW/85jE+85OH6GrNcOlf93HGEfvGHVYkIk/8ZnYmMODud5rZqbvbz93XAesA+vr6PJroRCQJHts2wkd/cB93bN7BaYf38tk3HkVvd3P2508njhb/ycDrzOw1QBvQY2bfcvdzY4hFRBKkWnUuv+0xPnvDQ7SkU3z2jX/Gm447EDOLO7RIRZ743f3jwMcBwhb/h5X0RaTWiuUq77niTn6+foDTDu/l/5zz4qb6UtZcaBy/iDS9StX50Hfv4efrB7jotUdw/kkrE9fKnyzWxO/uNwE3xRmDiDS3atX52A/u47rfP8UnXvNC3nnyqrhDip1m5xSRpuXu/NOPHuT7d27hA6cfyppTVscdUl1ITFfP0zvz/OCuLfxh+wgrl3WyelkXq3s7OWhpB62ZdNzhicgCK1eqfPrH67n8tse44GWr+OAZh8YdUt1o6sRfqlT5xUMDfPd3j3PThgGqDks7s2wfKU7sk04ZBy3t4LDl3Ry2bxdLOrPky1XypQpjpQppM7raMnS1BreethYWd7awT0eWxR1Z2lqee9JULFcZKVYYK5YZK1ap+HNHora3pOluywTvmc2QSiW3n1GkVgaG8rzvyru5Y/MOzj9pJZ/8qxcmuk9/qqZO/H9/1X1cffcTLO9u5d2vOJg39z2flcs6yeVLbN42wqbBETYODvPw1hwbtub46YNPU52Up7OZFNWqU67W7msEXa0ZutvGby30tGVY1N5CT3sLPW0t7NvTyvMWtbPfojaWd7dSKFfZOVZiKF9iOF/Ggcl/zuMHrUJ44BouVBgplBkplIMDWcrIpIxMOkU2naKtJU1bS4r2ljTplFEYP+gVK+TLFYrlanCrVDGMnvbg4NfT3kJ32/j94Gd7Nk0mlZooIx0e1FIpI2WQNqMlnaIlk6IlHTxXrULFnUrVaW9Jk82o91H2zu2btvO+K+8mly/x+TcfxTnHHhh3SHXH3Ov/u1F9fX3e398/59fd/cdn2D5c5NTDe8mk95xQ8qUKo8UKbS0p2jJpUinD3SmUq+TyZYYLZXaOlXh2tMizoyWeGS1SLFcnXu9ANp2iI5umozVDe0uazKQWveOMFavk8iWGC2WG8mVy+RK55/wsM5QvMTRWYihfprIAB53ObJrO1gzt2TSVqlOuOOVqkNDzYWKfzCw4M2lrSZNNp8hmgpu7T8SXL1V3U9re6W7NsLgzy+LOLJ3Z4EDQEsbQ0RLUo7M1+NmWCZ5vzaRoDc+kFndk2ae9hX06Wuhua5k4+Ejz2zFS5JJbN7Hulk2sWNLBl889lhc8L9lTgZnZne7eN3V7U7f4j1mxeE77t4XJbjIzm9ge9Tf7qlVn20iBp57N89TOPIPDhYmuop62FrpaM0w9e23NBK341rAV3zmL7qRK1SmUK5TKTls2OBPY02lxoVxhaCw4YA3lywyNlRgtVqh6cIZUqVYpVxwH8F2t+nKlSqniFCvBgSNlRjoV/BwrVtgxWuSZkSLbR4qMFYOzlUJ4xjFWrDAcnr3M9njYHZ5R9bS3hAeNzMSBsK0lRWsmPXGg7wkPGIvag668nvB1PW0tOhOpY9uGC1xy6yb+8zd/YKxU4eyjD+BTZ72InraWuEOrW02d+BtdKmUs725jeXcbRz2/duWkU0ZHNgNzmJeqNZOmtzv6gyEwcRaWLwVdUYVylUK5ws6xMjvHxs/Gxs+aSgyNBWcpI+EZ25PPjjFaKJMvVymUKuTL1T2eWXVm0xywuJ0VSzo4cHEHBy5uZ2lXliWdrSzpyLK4s4Ulndng9yg1t324wK2PbOOmDQNc/8DTFMtVXnvU/rzvtEOaYk3cWtNfqTScyWdhC6VYrjKUL/HsaGni4DFx0BgrsWO0yJZnxnh8xyi/2bidkWJl2vdpa0mFB4IsSzqz7NORZUlHC4s6snSHZxxdbcFZx/j1leCsI01r2KXWmknRnk1rtBkwWizz5LNjPLZtlMe2j/DY9hF+v2Un9z2xEw8Ha5x11AFc+IrVrO7tijvchqHEL0JwIX9ZV+usFt5wd3aOBWcVO0aCrqkdI0V2jIY/w9szo0Ue3zHKjpEiQ/nynGNqzaToaQ+6nronjSzrnPiZpiMb3M9mUhjB9RnDyKQtPIiE10nCi+3plJFK2cSF/fGuwY5smvaWdGSjzEaLZbbligwO5xkYKjCQK7B1KM/WoQIDuTxP78zz9FCe3JTf26L2Fg7bt4sPnXEYrzislxcfsEgj4+ZBiV9kjsyMfTqC1vyqZZ2zek2l6owWy4wUgusUo8Uy+VKVsVKFfHgbHz1VKFUZLe660L9zrDQxuGDrUJ7h8P5IsbIgF/8n68gGB5PWcORVSzpFJh3cT9mu0VrjZybZTHBNCILBDVWHqnswcKBUoVAKuuGCegb1HSmUGZ3mjCmTMpZ3t9Lb08bq3k5OOngp+y5qY/9F7Ry0tINVyzrZp6O558mPihK/SATSKaO7LRhptFDGr3UMF8qUKlXcg+Tr4YX0Qrk6kXjLVZ8YmlyZnJjD6xyjxcrE90+GCxVKlerErVgOL9ZXnao7pYozXCizfTg4UI2PCktZcFA0gjOo8a6sfTqy7NeSpj3s3urIplnW1Upvd3Bb1pVl3542lnRk1XqPiBK/SIOqxbUOSQaNURMRSRglfhGRhFHiFxFJGCV+EZGEiTzxm1mbmd1hZvea2QNm9qmoYxARSbI4RvUUgD9392EzawF+ZWY/cfffxhCLiEjixLHYugPD4cOW8Fb/U4SKiDSJWPr4zSxtZvcAA8DP3P32afZZa2b9ZtY/ODgYeYwiIs0q1vn4zWwf4Brg/e5+/wz77QQembRpEbBzmseTt4/fXwZsm2eIU8uZyz7Tbd9d3Lu7P3nbfOsRdx0m36/Xz2I2dar3Okx9HOdnMdPzM8U80+NG+nua+rhW/9szxTjuIHfv/ZOt7h7rDbgI+PAe9lk3m8eTt0/a1r8Xsa2b7z7TbZ+pHruJffK2edUj7jo0wmcxmzrVex3q6bOY6fnZ/i/v7vffLJ/FQvxvz7Ye093iGNXTG7b0MbN24AzgoT287L9n+fi/Z9hnPmbzHrvbZ7rtM9VjuvvNUIfZxrAntazHbOpU73WY+jjOz2Km52f7vzz1cSP9PU19XKu/p3m/T+RdPWb2Z8A3gDTBNYbvufs/1bC8fp9m6bFG0wz1UB3qRzPUoxnqAPHUI45RPfcBx0RY5LoIy6qlZqiH6lA/mqEezVAHiKEeDbHYuoiILBxN2SAikjBK/CIiCaPELyKSMIlO/Gb2cjP7ipldama3xR3PfJhZysw+bWZfMrPz4o5nvszsVDO7Nfw8To07nvkys04zu9PMzow7lvkwsxeGn8FVZvY/445nvszsbDO7xMx+aGZ/EXc882Fmq83sa2Z21UK/d8MmfjP7upkNmNn9U7a/ysw2mNmjZvaxmd7D3W9193cDPyIYYhqphagDcBZwAFACttQq1pksUD3G53BqI4Z6LFAdAP4e+F5topzZAv1PrA//J94MxDJUcoHq8V/uvgY4H/gfNQx3WgtUh03ufkFNApzvN8bivgGnAMcC90/algY2AquBLHAvcATwYoLkPvm2fNLrvgf0NGIdgI8BF4avvapRPwsgFb5uX+CKBq3DGcBbCJLNmY1Yh/A1rwNuA97WqH9Pk173b8CxDV6HBf+/btjF1t39FjNbOWXzS4FH3X0TgJl9BzjL3f8FmPbU28xWADvdfaiW8U5nIepgZluAYviwUsNwd2uhPovQM0BrTQKdwQJ9FqcBnQT/zGNm9mN3r9Y28l0W6nNw92uBa83sOuDbNQx5Wgv0WRjwGeAn7n5XjUP+Ewv8P7HgGjbx78YBwOOTHm8Bjt/Day4ALqtZRHM31zpcDXzJzF4O3FLLwOZoTvUws3OAvwT2Af69ppHN3pzq4O6fADCz84FtUSb9Gcz1czgVOIfg4PvjWgY2R3P9v3g/wRnYIjM7xN2/UsvgZmmun8VS4NPAMWb28fAAsSCaLfHbNNtm/Iaau19Uo1jma051cPdRgoNXvZlrPa4mOIjVkzn/PQG4++ULH8q8zfVzuAm4qVbB7IW51uOLwBdrF868zLUO24F31yKQhr24uxtbgOdPenwg8GRMscxXM9QBmqMeqkP9aIZ61E0dmi3x/w441MxWmVmW4ELbtTHHNFfNUAdojnqoDvWjGepRP3WI46r9Al01vxJ4il3DGC8It78GeJjg6vkn4o6z2evQLPVQHern1gz1qPc6aJI2EZGEabauHhER2QMlfhGRhFHiFxFJGCV+EZGEUeIXEUkYJX4RkYRR4peGZWbDEZe3IGs2hGsP7DSzu83sITP73Cxec7aZHbEQ5Yso8YuEzGzGuavc/aQFLO5Wdz8GOAY408xO3sP+ZxPM+imy15ptkjZJODM7GLgY6AVGgTXu/pCZvRb4JME86NuBt7v7VjP7R2B/YCWwzcweBlYQzJm+AviCBxN+YWbD7t4VzmD5j8A24EjgTuBcd3czew3w+fC5u4DV7r7bKXfdfczM7iGYuREzWwOsDeN8FHgHcDTBHPmvMLNPAm8IX/4n9Zzv702SRS1+aTbrgPe7+3HAh4H/CLf/CjghbGV/B/jopNccRzAv+tvCxy8gmCL6pcBFZtYyTTnHAB8kaIWvBk42szbgq8Cr3f1lBEl5Rma2GDiUXVNqX+3uL3H3o4D1BF/1v41gTpePuPvR7r5xhnqK7JFa/NI0zKwLOAn4frAOB7BrUZcDge+a2X4ErenNk156rbuPTXp8nbsXgIKZDRCsCjZ1Ocg73H1LWO49BGcMw8Amdx9/7ysJWu/TebmZ3QccDnzG3Z8Otx9pZv+bYF2CLuCGOdZTZI+U+KWZpIBn3f3oaZ77EvB5d792UlfNuJEp+xYm3a8w/f/JdPtMN9/67tzq7mea2WHAr8zsGne/B7gcONvd7w0XdDl1mtfOVE+RPVJXjzQND5bP3Gxmb4Jg+T0zOyp8ehHwRHj/vBqF8BCwetKSe3tc5NvdHwb+hWCRdoBu4Kmwe+ntk3bNhc/tqZ4ie6TEL42sw8y2TLr9HUGyvMDM7gUeAM4K9/1Hgq6RWwkuvC64sLvoPcD1ZvYrYCuwcxYv/QpwipmtAv4XcDvwM4IDybjvAB8Jh4AezO7rKbJHmpZZZAGZWZe7D4eLfV8MPOLu/zfuuEQmU4tfZGGtCS/2PkDQvfTVeMMR+VNq8YuIJIxa/CIiCaPELyKSMEr8IiIJo8QvIpIwSvwiIgmjxC8ikjD/H3mDTySDnCXwAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The learning rate finder curve suggests picking something between 1e-4 and 1e-3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.986238</td>\n",
       "      <td>2.721945</td>\n",
       "      <td>15.209874</td>\n",
       "      <td>04:56</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now with just one epoch of fine-tuning and not much regularization, our model did not really improve since it was already amazing. To have a look at some generated texts, let's take a prompt that looks like a wikipedia article:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n = Tropical Storm &lt;unk&gt; ( 2008 ) = \\n \\n Tropical Storm &lt;unk&gt; was the tenth tropical storm of the 2008 Atlantic hurricane season . &lt;unk&gt; developed out of a strong tropical wave which moved off the African coast on August 31 . The wave quickly became organized and was declared Tropical Depression Ten while located 170 mi ( 270 km ) to the south @-@ southeast of the Cape Verde Islands on September 2 . The depression was quickly upgraded to Tropical Storm &lt;unk&gt; around noon the same day . Over the next several days , &lt;unk&gt; moved in a general west @-@ northwest direction and reached its peak...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         0\n",
       "0   \\n = Tropical Storm <unk> ( 2008 ) = \\n \\n Tropical Storm <unk> was the tenth tropical storm of the 2008 Atlantic hurricane season . <unk> developed out of a strong tropical wave which moved off the African coast on August 31 . The wave quickly became organized and was declared Tropical Depression Ten while located 170 mi ( 270 km ) to the south @-@ southeast of the Cape Verde Islands on September 2 . The depression was quickly upgraded to Tropical Storm <unk> around noon the same day . Over the next several days , <unk> moved in a general west @-@ northwest direction and reached its peak..."
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_valid.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Article seems to begin with new line and the title between = signs, so we will mimic that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = \"\\n = Unicorn = \\n \\n A unicorn is a magical creature with a rainbow tail and a horn\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The prompt needs to be tokenized and numericalized, so we use the same function as before to do this, before we use the `generate` method of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 21])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt_ids = tokenizer.encode(prompt)\n",
    "inp = tensor(prompt_ids)[None].cuda()\n",
    "inp.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    }
   ],
   "source": [
    "preds = learn.model.generate(inp, max_length=40, num_beams=5, temperature=1.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n = Unicorn = \\n \\n A unicorn is a magical creature with a rainbow tail and a horn @-@ shaped head. It is a member of the <unk> family of <unk'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(preds[0].cpu().numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### fin -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}