Machine_Human_Text_Detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detecting machine VS human-generated Wikipedia articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Creating the dataset\n",
    "\n",
    "The idea is to get a small subset of articles from the [WikiText-2](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) dataset, and run the first couple of sentences of each post through a [gpt2-large](https://huggingface.co/transformers/v2.1.1/pretrained_models.html) pre-trained language model to generate fake content. This would allow us to have the same number (~300) of human-curated and machine-faked articles.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#!pip install transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#!pip install torchtext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import argparse\n",
    "import numpy as np\n",
    "import torch\n",
    "from os import path\n",
    "from pathlib import Path\n",
    "import os\n",
    "import random\n",
    "from shutil import copyfile\n",
    "\n",
    "from transformers import (\n",
    "    GPT2LMHeadModel,\n",
    "    GPT2Tokenizer\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading a subset of the `wikitext-2` dataset into a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(662, 2)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wikitext = pd.read_csv('/home/ec2-user/.fastai/data/wikitext-2/test.csv', header=None)\n",
    "wikitext = wikitext.append(pd.read_csv('/home/ec2-user/.fastai/data/wikitext-2/train.csv', header=None))\n",
    "wikitext.reset_index(inplace=True)\n",
    "\n",
    "wikitext.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>\\n = Tropical Storm &lt;unk&gt; ( 2008 ) = \\n \\n Tropical Storm &lt;unk&gt; was the tenth tropical storm of the 2008 Atlantic hurricane season . &lt;unk&gt; developed out of a strong tropical wave which moved off the African coast on August 31 . The wave quickly became organized and was declared Tropical Depression Ten while located 170 mi ( 270 km ) to the south @-@ southeast of the Cape Verde Islands on September 2 . The depression was quickly upgraded to Tropical Storm &lt;unk&gt; around noon the same day . Over the next several days , &lt;unk&gt; moved in a general west @-@ northwest direction and reached its peak...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>\\n = Calvin &lt;unk&gt; = \\n \\n Calvin &lt;unk&gt; ( born November 2 , 1984 ) is a Canadian football running back for the Edmonton &lt;unk&gt; of the Canadian Football League ( &lt;unk&gt; ) . He played as a &lt;unk&gt; until 2014 , when he became the starting fullback for the &lt;unk&gt; . &lt;unk&gt; is known for being able to fill many roles at his position , with &lt;unk&gt; &lt;unk&gt; Chris Schultz noting in 2010 that he is a \" multi @-@ purpose running back who catches the ball extremely well , blocks well and runs well \" . He is a champion of the &lt;unk&gt; Grey Cup . \\n Prior to being drafted by the Edmonton &lt;unk&gt; in the fourth round of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>\\n = The Boat Race 2008 = \\n \\n The &lt;unk&gt; Boat Race took place on 29 March 2008 . Held annually , the event is a side @-@ by @-@ side rowing race between crews from the Universities of Oxford and Cambridge along the River Thames . Oxford won the race . Oxford 's crew featured the oldest competitor in Boat Race history . The race took place in very difficult weather conditions – strong winds and heavy rain – resulting in the slowest winning time in over sixty years . Oxford won by six lengths , the largest margin of victory since the 2004 race . \\n Oxford 's Isis beat Cambridge 's Goldie i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>\\n = Angel of Death ( Slayer song ) = \\n \\n \" Angel of Death \" is the opening track on the American thrash metal band Slayer 's 1986 album Reign in Blood . The lyrics and music were written by Slayer guitarist , Jeff Hanneman and are based on Nazi physician &lt;unk&gt; &lt;unk&gt; , who conducted human experiments at the Auschwitz concentration camp during World War II . \" Angel of Death \" led to the band facing accusations of Nazi &lt;unk&gt; and racism throughout their career . \\n Despite the controversy surrounding the song and its contribution to the delay in the release of Reign in Blood , \" Angel of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>\\n = 2011 – 12 Michigan Wolverines men 's basketball team = \\n \\n The 2011 – 12 Michigan Wolverines men 's basketball team represented the University of Michigan during the 2011 – 12 NCAA Division I men 's basketball season . The team played its home games in Ann Arbor , Michigan at &lt;unk&gt; Center for the &lt;unk&gt; consecutive year . It had a seating capacity of 12 @,@ &lt;unk&gt; . It was also the team 's &lt;unk&gt; straight season as a member of the Big Ten Conference . Fifth @-@ year head coach John &lt;unk&gt; led the team , alongside All @-@ Big Ten players Trey Burke , Tim &lt;unk&gt; , Jr. and Zack Novak . Bur...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index  \\\n",
       "0      0   \n",
       "1      1   \n",
       "2      2   \n",
       "3      3   \n",
       "4      4   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         0  \n",
       "0   \\n = Tropical Storm <unk> ( 2008 ) = \\n \\n Tropical Storm <unk> was the tenth tropical storm of the 2008 Atlantic hurricane season . <unk> developed out of a strong tropical wave which moved off the African coast on August 31 . The wave quickly became organized and was declared Tropical Depression Ten while located 170 mi ( 270 km ) to the south @-@ southeast of the Cape Verde Islands on September 2 . The depression was quickly upgraded to Tropical Storm <unk> around noon the same day . Over the next several days , <unk> moved in a general west @-@ northwest direction and reached its peak...  \n",
       "1   \\n = Calvin <unk> = \\n \\n Calvin <unk> ( born November 2 , 1984 ) is a Canadian football running back for the Edmonton <unk> of the Canadian Football League ( <unk> ) . He played as a <unk> until 2014 , when he became the starting fullback for the <unk> . <unk> is known for being able to fill many roles at his position , with <unk> <unk> Chris Schultz noting in 2010 that he is a \" multi @-@ purpose running back who catches the ball extremely well , blocks well and runs well \" . He is a champion of the <unk> Grey Cup . \\n Prior to being drafted by the Edmonton <unk> in the fourth round of ...  \n",
       "2   \\n = The Boat Race 2008 = \\n \\n The <unk> Boat Race took place on 29 March 2008 . Held annually , the event is a side @-@ by @-@ side rowing race between crews from the Universities of Oxford and Cambridge along the River Thames . Oxford won the race . Oxford 's crew featured the oldest competitor in Boat Race history . The race took place in very difficult weather conditions – strong winds and heavy rain – resulting in the slowest winning time in over sixty years . Oxford won by six lengths , the largest margin of victory since the 2004 race . \\n Oxford 's Isis beat Cambridge 's Goldie i...  \n",
       "3   \\n = Angel of Death ( Slayer song ) = \\n \\n \" Angel of Death \" is the opening track on the American thrash metal band Slayer 's 1986 album Reign in Blood . The lyrics and music were written by Slayer guitarist , Jeff Hanneman and are based on Nazi physician <unk> <unk> , who conducted human experiments at the Auschwitz concentration camp during World War II . \" Angel of Death \" led to the band facing accusations of Nazi <unk> and racism throughout their career . \\n Despite the controversy surrounding the song and its contribution to the delay in the release of Reign in Blood , \" Angel of ...  \n",
       "4   \\n = 2011 – 12 Michigan Wolverines men 's basketball team = \\n \\n The 2011 – 12 Michigan Wolverines men 's basketball team represented the University of Michigan during the 2011 – 12 NCAA Division I men 's basketball season . The team played its home games in Ann Arbor , Michigan at <unk> Center for the <unk> consecutive year . It had a seating capacity of 12 @,@ <unk> . It was also the team 's <unk> straight season as a member of the Big Ten Conference . Fifth @-@ year head coach John <unk> led the team , alongside All @-@ Big Ten players Trey Burke , Tim <unk> , Jr. and Zack Novak . Bur...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wikitext.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up the Huggingface pipeline for the `gpt2-large` LM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "DEVICE = 'cpu'\n",
    "LENGTH = 900\n",
    "TEMPERATURE = 1.0\n",
    "K = 0\n",
    "P = 0.9\n",
    "REPETITION_PENALTY = 1.0\n",
    "NUM_RETURN_SEQUENCES = 1\n",
    "STOP_TOKEN = None\n",
    "\n",
    "def save_text(o, path):\n",
    "    with open(path, \"w\") as output_file:\n",
    "        output_file.write(o)\n",
    "        \n",
    "def generate_text(index):\n",
    "    doc = wikitext.loc[index, 0]\n",
    "    dot = doc[400:len(doc)].find(\".\")\n",
    "    prompt_text = doc[:(400+dot+1)]\n",
    "    \n",
    "    encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors=\"pt\")\n",
    "    encoded_prompt = encoded_prompt.to(DEVICE)\n",
    "\n",
    "    output_sequences = model.generate(\n",
    "        input_ids=encoded_prompt,\n",
    "        max_length=LENGTH + len(encoded_prompt[0]),\n",
    "        temperature=TEMPERATURE,\n",
    "        top_k=K,\n",
    "        top_p=P,\n",
    "        repetition_penalty=REPETITION_PENALTY,\n",
    "        do_sample=True,\n",
    "        num_return_sequences=NUM_RETURN_SEQUENCES\n",
    "    )\n",
    "\n",
    "    generated_sequence = output_sequences[0].tolist()\n",
    "    # Decode text\n",
    "    text = tokenizer.decode(generated_sequence, clean_up_tokenization_spaces=True)\n",
    "    # Remove all text after the stop token\n",
    "    text = text[: text.find(STOP_TOKEN) if STOP_TOKEN else None]\n",
    "\n",
    "    # Add the prompt at the beginning of the sequence. Remove the excess text that was used for pre-processing\n",
    "    total_sequence = (prompt_text + text[len(tokenizer.decode(encoded_prompt[0], clean_up_tokenization_spaces=True)) :])\n",
    "    total_sequence = total_sequence[:(total_sequence.rfind(\".\")+1)]\n",
    "    \n",
    "    save_text(total_sequence, \n",
    "              f\"/home/ec2-user/SageMaker/course-v4/nbs/wikitext/machine/{index}.txt\")\n",
    "    save_text(wikitext.loc[index, 0][: (wikitext.loc[index, 0][:len(total_sequence)].rfind(\".\")+1)], \n",
    "              f\"/home/ec2-user/SageMaker/course-v4/nbs/wikitext/human/{index}.txt\")\n",
    "    print(index, len(generated_sequence))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the LM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "model, tokenizer = GPT2LMHeadModel, GPT2Tokenizer\n",
    "\n",
    "# RUN THE FOLLOWING IF YOU EXECUTE THIS CELL FOR THE FIRST TIME.\n",
    "# THIS IS TO DOWNLOAD THE PRE-TRAINED WEIGHTS FROM HUGGINGFACE\n",
    "# tokenizer = tokenizer.from_pretrained('gpt2-large')\n",
    "# model = model.from_pretrained('gpt2-large')\n",
    "\n",
    "mydir = \"./gpt2/\"\n",
    "\n",
    "if path.isdir(mydir):\n",
    "    tokenizer = tokenizer.from_pretrained(mydir)\n",
    "    model = model.from_pretrained(mydir)\n",
    "else:\n",
    "    tokenizer.save_pretrained(mydir)\n",
    "    model.save_pretrained(mydir)\n",
    "\n",
    "model.to(DEVICE);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feeding the LM with the first 2 sentences of a real article and it generate fake text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n",
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 989\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 902\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 1004\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 1002\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 1009\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n",
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "35 725\n"
     ]
    }
   ],
   "source": [
    "for i in wikitext.index:\n",
    "    try: \n",
    "        generate_text(i)\n",
    "    except: \n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating an appropriate folder structure to be able to build a DL dataset in a more convenient way\n",
    "\n",
    "```\n",
    "+ wikifake_tiny\n",
    "+-- train\n",
    "    +-- human\n",
    "    +-- machine\n",
    "+-- valid\n",
    "    +-- human\n",
    "    +-- machine\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "models\ttrain  valid\r\n"
     ]
    }
   ],
   "source": [
    "!ls ./wikifake_tiny/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "human  machine\r\n"
     ]
    }
   ],
   "source": [
    "!ls ./wikifake_tiny/train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = Path(\"./wikitext/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(67, 268)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.seed(0)\n",
    "machine = os.listdir(path/\"machine\")\n",
    "n = int(len(machine) * 0.2)\n",
    "valid = random.sample(machine, n)\n",
    "train = [t for t in machine if t not in valid]\n",
    "\n",
    "assert (len(train)+len(valid))==len(machine)\n",
    "\n",
    "d = {'train': train, 'valid': valid}\n",
    "len(d['valid']), len(d['train'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "670"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(len(d['valid']) + len(d['train'])) * 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "src = \"./wikitext/\"\n",
    "dst = \"./wikifake_tiny/\"\n",
    "\n",
    "for j in ['machine', 'human']:\n",
    "    for k in ['valid', 'train']:\n",
    "        for t in d[k]:\n",
    "            copyfile(os.path.join(src, j, t), os.path.join(dst, k, j, t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's check an example from `human` and..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \n",
      " = Jin – Song Wars = \n",
      " \n",
      " The Jin – Song Wars were a series of conflicts between the Jurchen Jin dynasty ( 1115 – 1234 ) and Han Chinese Song dynasty ( 960 – 1279 ) . In 1115 , the Jurchens rebelled against their overlords , the <unk> Liao dynasty ( 907 – 1125 ) , and declared the formation of the Jin . <unk> with the Song against their common enemy the Liao , the Jin promised to return to the Song the territories in northern China that had fallen under Liao control since 938 . The Jurchens ' quick defeat of the Liao combined with Song military failures made the Jin reluctant to cede these territories . After a series of failed negotiations that <unk> both sides , the Jurchens attacked the Song in November 1125 , dispatching one army towards Taiyuan and the other towards Kaifeng , the Song capital . \n",
      " <unk> by the news of an invasion , the Song general stationed in Taiyuan retreated from the city , which was besieged and later captured . As the second Jin army approached the capital , Emperor Huizong of the Song abdicated and fled south . A new emperor , Qinzong , was enthroned . The Jurchens began a siege against Kaifeng in 1126 , but Qinzong negotiated for their retreat from the capital after he agreed to pay a large annual indemnity . Qinzong reneged on the deal and ordered Song forces to defend the prefectures instead of <unk> the capital . The Jin resumed their war against the Song and again besieged Kaifeng in 1127 . The Chinese emperor was captured in an event known as the <unk> Incident , the capital was looted , and the Song lost northern China to the Jin . Remnants of the Song retreated to southern China and , after brief stays in several temporary capitals , eventually relocated to Hangzhou . The retreat of the Song court marked the end of the Northern Song era and the beginning of the Southern Song . \n",
      " The Jurchens tried to conquer southern China in the <unk> , but they were bogged down by a pro @-@ Song insurgency in the north and a counteroffensive by the Song generals Yue Fei , Han Shizhong , and others . The generals regained some territories but retreated on the orders of the Southern Song emperor , who supported a peaceful resolution to the war . The Treaty of Shaoxing in 1142 settled the boundary between the two empires along the Huai River , but conflicts between the two dynasties continued until the fall of the Jin in 1234 . A campaign against the Song by the fourth Jin emperor , Wanyan Liang ( the Prince of Hailing ) , was unsuccessful . He lost the Battle of Caishi ( 1161 ) and was later assassinated by his own disaffected officers . An invasion of the Jin motivated by Song <unk> ( 1206 – 1208 ) was also unsuccessful . A decade later , the Jin launched an abortive military campaign against the Song in 1217 to compensate for the territory that they had lost to the invading Mongols . The Song formed an alliance with the Mongols in 1233 , and in the following year jointly captured Caizhou , the last refuge of the Jin emperor . The Jin dynasty collapsed that year in 1234 . After the demise of the Jin , the Song dynasty itself became a target of the Mongols , and fell in 1279 . \n",
      " The wars engendered an era of technological , cultural , and demographic changes in China . Battles between the Song and Jin brought about the introduction of various gunpowder weapons . The siege of De 'an in 1132 was the first recorded appearance of the fire lance , an early ancestor of firearms . There were also reports of battles fought with primitive gunpowder bombs like the incendiary huopao or the exploding <unk> , incendiary arrows , and other related weapons . In northern China , the Jurchen tribes were the ruling minority of an empire that was predominantly inhabited by former subjects of the Northern Song . Jurchen migrants settled in the conquered territories and assimilated with the local culture . The Jin government instituted a centralized imperial bureaucracy modeled on previous Chinese dynasties , basing their legitimacy on Confucian philosophy .\n"
     ]
    }
   ],
   "source": [
    "f = Path(\"./wikifake_tiny/valid/human/104.txt\").read(); print(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ... the same from `machine`, e.g. the LM's output triggered by the first 3 sentences from the previous article\n",
    "\n",
    "> = Jin – Song Wars = The Jin – Song Wars were a series of conflicts between the Jurchen Jin dynasty ( 1115 – 1234 ) and Han Chinese Song dynasty ( 960 – 1279 ) . In 1115 , the Jurchens rebelled against their overlords , the <unk> Liao dynasty ( 907 – 1125 ) , and declared the formation of the Jin . <unk> with the Song against their common enemy the Liao , the Jin promised to return to the Song the territories in northern China that had fallen under Liao control since 938 ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \n",
      " = Jin – Song Wars = \n",
      " \n",
      " The Jin – Song Wars were a series of conflicts between the Jurchen Jin dynasty ( 1115 – 1234 ) and Han Chinese Song dynasty ( 960 – 1279 ) . In 1115 , the Jurchens rebelled against their overlords , the <unk> Liao dynasty ( 907 – 1125 ) , and declared the formation of the Jin . <unk> with the Song against their common enemy the Liao , the Jin promised to return to the Song the territories in northern China that had fallen under Liao control since 938 . They raised a number of brigades and raised fierce banners in honour of the imprisoned Sun emperor of Han. However, the Song conquests in China slowed dramatically during the first years of the Song's reign. But the Jurchen government backed off by official withdrawals of troops and the exportation of many wealth goods, the Han Rebellion, a religious uprising, began in 1118. All the provinces of China became subservient to the Song, from Jin to Shanghai. Then the Song retook most of the lands in the south, the Han flourished, and had a large colonial empire (116 南内, in 382 : 12) with many cities under its control in the north. The treaty of Bao Bao in 1165 ended the Song's rule in China and restored power to the Jurchen for an indefinite period. Some historians, although they prefer to call the Jurchen the <unsung> who did not deserve the name, argue that their own failure in unity against the Liao as well as the reverses in Han Chinese self-government to the south made the Jurchen all but useless, and that no one would ever dare to call the Jurchen by its correct name. Even in the Han period, the Jurchen continued to enjoy a large amount of autonomy and regional autonomy. However, the appearance of the Miao-Shan Empire (936 団, of Lesser Peking) and the Jurchen Crown Prince Kuo Yu-fang in the 960s raised worries among the Jurchen that there was a real threat from the Miao-Shan (Fujian) (四) and other traditional Chinese imperial powers. Jintao of Huo Wudi (905 – 960 径朝) came to power in Huo Wudi, but the Jurchen were persuaded by such emperor Heng Xizhi (927 – 974 径大大) to form the Yuan (何) who had been kept on the retreat by Emperor Taizong (899 – 907 径天) and his ministers. The Jurchen government was by now partly dependent on the Yuan and part of its revenue was sent to the Qing to repay imperial loans to the Jurchen government. As a result, the Qing official Qing Cun Yu-li (927 – 992 径入正) formed a General Court of Jurchen, which brought many western emperor including Cao Cao (871 – 938 径王) (see below) with a detachment of agents to the province and commanded by them to protect their courts from those of the Jurchen king and the Jurchen Yuan, to prevent any penetration of the T'ang imperial court. Yet, the resulting difficulty had a stabilizing effect on the Jurchen court and greatly facilitated a peaceful consolidation of the state under the Yuan.\n",
      "The Jurchen government recovered within a couple of years after this. The Jurchen dynasty was not completely wiped out in the two wars against Song-Liao in 1179 and 1181, although the world famous Bao Bao Treaty of 1187 witnessed the control of the Jurchen lands being restored by their new emperor.\n",
      "The world known as the  Great Song Dynasties, are the great dynasties of the Jin Song Dynasty (南内) of the Second Han Dynasty (221 – 220 富山, 1234 – 1368 ). These dynasties were divided into several tribes and settled in small kingdoms in northern China and the far south of China. Although some of the Irregular tribes from Asia had escaped from the chieftains of the Jurchen and had taken up their places among the Jurchen, the few who had lived in Europe in the time of the Wu emperors were still called after the older Jurchen name (Sihan in Chinese). The following scholars have grouped the parts of the Jurchen government, the internal arrangements, and the internal political circumstances of the Jurchen and the Peking Jurchen authorities to the following four literary periods (generally translated as the \"Six Symbols or Fundamental System or Substance of the Jurchen Kings\").\n"
     ]
    }
   ],
   "source": [
    "f = Path(\"./wikifake_tiny/valid/machine/104.txt\").read(); print(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Training a text classifier to detect `human` VS `machine` articles\n",
    "\n",
    "This section (in large part) runs the code from [this NLP fastbook notebook](https://github.com/fastai/fastbook/blob/master/clean/10_nlp.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import *\n",
    "from fastai2.text.all import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#670) [Path('wikifake_tiny/train/machine/299.txt'),Path('wikifake_tiny/train/machine/260.txt'),Path('wikifake_tiny/train/machine/166.txt'),Path('wikifake_tiny/train/machine/11.txt'),Path('wikifake_tiny/train/machine/111.txt'),Path('wikifake_tiny/train/machine/317.txt'),Path('wikifake_tiny/train/machine/44.txt'),Path('wikifake_tiny/train/machine/278.txt'),Path('wikifake_tiny/train/machine/206.txt'),Path('wikifake_tiny/train/machine/19.txt')...]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = Path(\"./wikifake_tiny/\")\n",
    "files = get_text_files(path, folders = ['train', 'valid'])\n",
    "files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' \\n = Hugh Walpole = \\n \\n Sir Hugh Seymour Walpole , CBE ( 13 March 1884 – 1 June 1941 ) was an English novelist . He was the son of an Anglican clergyman , intended for a career in the church but drawn instead to writing . Among those who encouraged him were the authors Henry James and Arnold Bennett . His skill at scene @-@ setting and vivid plots , as well as his high profile as a lecturer , brought him a large readership in the United Kingdom and North America . He died in 1944. His 1929 short story, \"The Invisible Man\" from his collection The Rothed Committee, remains a popular one-liner, and is quoted frequently by modern authors.'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt = files[0].open().read(); txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizing text and converting tokens into numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(#132) [' \\n ','=','Hugh','Walpole','=','\\n \\n ','Sir','Hugh','Seymour','Walpole',',','CBE','(','13','March','1884','–','1','June','1941',')','was','an','English','novelist','.','He','was','the','son','of','an','Anglican','clergyman',',','intended','for','a','career','in','the','church','but','drawn','instead','to','writing','.','Among','those'...]\n"
     ]
    }
   ],
   "source": [
    "spacy = WordTokenizer()\n",
    "toks = first(spacy([txt]))\n",
    "print(coll_repr(toks, 50))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(#161) ['xxbos','=','xxmaj','hugh','xxmaj','walpole','=','\\n \\n ','xxmaj','sir','xxmaj','hugh','xxmaj','seymour','xxmaj','walpole',',','xxup','cbe','(','13','xxmaj','march','1884','–','1','xxmaj','june','1941',')','was','an','xxmaj','english','novelist','.','xxmaj','he','was','the','son','of','an','xxmaj','anglican','clergyman',',','intended','for','a'...]\n"
     ]
    }
   ],
   "source": [
    "tkn = Tokenizer(spacy)\n",
    "print(coll_repr(tkn(txt), 50))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "txts = L(o.open().read() for o in files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#670) [(#161) ['xxbos','=','xxmaj','hugh','xxmaj','walpole','=','\\n \\n ','xxmaj','sir'...],(#966) ['xxbos','=','xxmaj','june','1941','uprising','in','eastern','xxmaj','herzegovina'...],(#165) ['xxbos','=','xxmaj','tower','xxmaj','building','of','the','xxmaj','little'...],(#1025) ['xxbos','=','xxmaj','the','xxunk','xxmaj','blues','=','\\n \\n ','\"'...],(#346) ['xxbos','=','xxmaj','robbie','xxmaj','fowler','=','\\n \\n ','xxmaj','robert'...],(#783) ['xxbos','=','xxmaj','astraeus','hygrometricus','=','\\n \\n ','xxmaj','astraeus','hygrometricus'...],(#540) ['xxbos','=','xxmaj','frank','xxunk','=','\\n \\n ','xxmaj','air','xxmaj'...],(#1089) ['xxbos','=','xxmaj','first','xxmaj','battle','of','xxmaj','maryang','xxmaj'...],(#182) ['xxbos','=','xxmaj','soviet','cruiser','xxmaj','krasnyi','xxmaj','kavkaz','='...],(#913) ['xxbos','=','1985','xxunk','assassination','plot','=','\\n \\n ','xxmaj','the'...]...]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toks_all = txts.map(tkn)\n",
    "toks_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'(#9840) [\\'xxunk\\',\\'xxpad\\',\\'xxbos\\',\\'xxeos\\',\\'xxfld\\',\\'xxrep\\',\\'xxwrep\\',\\'xxup\\',\\'xxmaj\\',\\'the\\',\\',\\',\\'.\\',\\'of\\',\\'and\\',\\'in\\',\\'a\\',\\'to\\',\\'was\\',\\'\"\\',\\'=\\',\\'(\\',\\')\\',\\'-\\',\\'is\\',\\'on\\',\\'as\\',\\'by\\',\\'for\\',\\'that\\',\\'with\\',\"\\'s\",\\'it\\',\\'at\\',\\'from\\',\\'\\\\n\\',\\'he\\',\\'\\\\n \\\\n \\',\\'his\\',\\'an\\',\\':\\',\\'were\\',\\'which\\',\\'this\\',\\'\\\\n \\',\\'first\\',\\'be\\',\\'are\\',\\'had\\',\"\\'\",\\'not\\'...]'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num = Numericalize()\n",
    "num.setup(toks_all)\n",
    "coll_repr(num.vocab,50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([   2,   19,    8, 2393,    8, 5894,   19,   36,    8, 1968,    8, 2393,    8, 4720,    8, 5894,   10,    7, 6725,   20])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num.encodes(tkn(txt))[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxbos = xxmaj hugh xxmaj walpole = \\n \\n  xxmaj sir xxmaj hugh xxmaj seymour xxmaj walpole , xxup cbe ('"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join(tkn(txt)[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxbos = xxmaj hugh xxmaj walpole = \\n \\n  xxmaj sir xxmaj hugh xxmaj seymour xxmaj walpole , xxup cbe ('"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join(num.decodes(num.encodes(tkn(txt)))[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxbos = xxmaj hugh xxmaj walpole = \\n \\n  xxmaj sir xxmaj hugh xxmaj seymour xxmaj walpole , xxup cbe ('"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([num.vocab[o] for o in num.encodes(tkn(txt))[:20]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.a) Fine tuning a language model before moving to the actual classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls_lm = DataBlock(\n",
    "    blocks=TextBlock.from_folder(path, is_lm=True),\n",
    "    get_items=get_text_files, \n",
    "    splitter=RandomSplitter(0.1)\n",
    ").dataloaders(path, path=path, bs=128, seq_len=80)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([128, 80]), torch.Size([128, 80]), 40)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x, y = first(dls_lm.train)\n",
    "x.shape, y.shape, len(dls_lm.train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos = xxmaj burning of women in xxmaj england = \\n▁\\n▁ xxmaj in xxmaj england , burning was a legal punishment inflicted on women found guilty of high treason , petty treason and heresy . xxmaj over a period of several centuries , female xxunk were publicly burnt at the stake , sometimes alive , for a range of activities including coining and xxunk . \\n▁ xxmaj while men guilty of heresy were also burned at the stake , those</td>\n",
       "      <td>= xxmaj burning of women in xxmaj england = \\n▁\\n▁ xxmaj in xxmaj england , burning was a legal punishment inflicted on women found guilty of high treason , petty treason and heresy . xxmaj over a period of several centuries , female xxunk were publicly burnt at the stake , sometimes alive , for a range of activities including coining and xxunk . \\n▁ xxmaj while men guilty of heresy were also burned at the stake , those who</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>jonathan xxmaj xxunk and xxmaj david xxmaj xxunk and directed by xxmaj nick xxmaj xxunk , the episode premiered on xxmaj the xxup cw on xxmaj october 31 , 2006 . \\n▁ xxmaj the series depicts the adventures of xxmaj veronica xxmaj mars ( xxmaj kristen xxmaj bell ) as she deals with life as a college student while xxunk as a private detective . xxmaj in this episode , xxmaj veronica xxunk when two xxunk men xxunk a xxmaj</td>\n",
       "      <td>xxmaj xxunk and xxmaj david xxmaj xxunk and directed by xxmaj nick xxmaj xxunk , the episode premiered on xxmaj the xxup cw on xxmaj october 31 , 2006 . \\n▁ xxmaj the series depicts the adventures of xxmaj veronica xxmaj mars ( xxmaj kristen xxmaj bell ) as she deals with life as a college student while xxunk as a private detective . xxmaj in this episode , xxmaj veronica xxunk when two xxunk men xxunk a xxmaj xxunk</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls_lm.show_batch(max_n=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(\n",
    "    dls_lm, AWD_LSTM, drop_mult=0.3, \n",
    "    metrics=[accuracy, Perplexity()]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.179878</td>\n",
       "      <td>3.069241</td>\n",
       "      <td>0.415890</td>\n",
       "      <td>21.525564</td>\n",
       "      <td>00:50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 2e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.741649</td>\n",
       "      <td>3.016871</td>\n",
       "      <td>0.419821</td>\n",
       "      <td>20.427279</td>\n",
       "      <td>01:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.631415</td>\n",
       "      <td>2.976815</td>\n",
       "      <td>0.427663</td>\n",
       "      <td>19.625206</td>\n",
       "      <td>01:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.373135</td>\n",
       "      <td>2.961053</td>\n",
       "      <td>0.433551</td>\n",
       "      <td>19.318310</td>\n",
       "      <td>01:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.117501</td>\n",
       "      <td>2.979997</td>\n",
       "      <td>0.436646</td>\n",
       "      <td>19.687752</td>\n",
       "      <td>01:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.910905</td>\n",
       "      <td>3.010571</td>\n",
       "      <td>0.435255</td>\n",
       "      <td>20.298992</td>\n",
       "      <td>01:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(5, 2e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save_encoder('finetuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.b) Training the classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls_clas = DataBlock(\n",
    "    blocks=(TextBlock.from_folder(path, vocab=dls_lm.vocab),CategoryBlock),\n",
    "    get_y = parent_label,\n",
    "    get_items=get_text_files,\n",
    "    splitter=GrandparentSplitter()\n",
    ").dataloaders(path, path=path, bs=64, seq_len=72)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exploring the DataBlock via `DataBlock.summary()`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting-up type transforms pipelines\n",
      "Collecting items from wikifake_tiny\n",
      "Found 670 items\n",
      "2 datasets of sizes 536,134\n",
      "Setting up Pipeline: Tokenizer -> Numericalize\n",
      "Setting up Pipeline: parent_label -> Categorize\n",
      "\n",
      "Building one sample\n",
      "  Pipeline: Tokenizer -> Numericalize\n",
      "    starting from\n",
      "      wikifake_tiny/train/machine/299.txt\n",
      "    applying Tokenizer gives\n",
      "      (#161) ['xxbos','=','xxmaj','hugh','xxmaj','walpole','=','\\n▁\\n▁','xxmaj','sir'...]\n",
      "    applying Numericalize gives\n",
      "      TensorText of size 161\n",
      "  Pipeline: parent_label -> Categorize\n",
      "    starting from\n",
      "      wikifake_tiny/train/machine/299.txt\n",
      "    applying parent_label gives\n",
      "      machine\n",
      "    applying Categorize gives\n",
      "      TensorCategory(1)\n",
      "\n",
      "Final sample: (TensorText([   2,   19,    8, 2393,    8, 5894,   19,   36,    8, 1968,    8, 2393,    8, 4720,    8, 5894,   10,    7, 6725,   20,  347,    8,  330, 4721,   65,   71,    8,  181, 1027,   21,   17,   38,\n",
      "           8,  187, 7940,   11,    8,   35,   17,    9,  615,   12,   38,    8, 6726, 7941,   10, 1150,   27,   15,  305,   14,    9,  295,   50, 1811,  677,   16,  693,   11,    8,  535,  256,   73,\n",
      "        3186,  136,   40,    9, 2394,    8,  628,    8,  774,   13,    8, 5895,    8, 1305,   11,    8,   37,    0,   32, 2258,   22, 2653,   13, 6727, 6728,   10,   25,  140,   25,   37,  157, 3631,\n",
      "          25,   15,    0,   10,  603,  136,   15,  245,    0,   14,    9,    8,  101,    8,  661,   13,    8,   97,    8,  383,   11,    8,   35,  503,   14, 1092,   11,    8,   37, 2818,  370,  268,\n",
      "          10,   18,    9,    8,    0,    8,  266,   18,   33,   37,  971,    8,    9,    8,    0,    8, 1471,   10, 1472,   15,  488,   51,   22,    0,   10,   13,   23,    0, 2518,   26,  734, 2394,\n",
      "          11]), TensorCategory(1))\n",
      "\n",
      "\n",
      "Setting up after_item: Pipeline: ToTensor\n",
      "Setting up before_batch: Pipeline: partial\n",
      "Setting up after_batch: Pipeline: \n",
      "\n",
      "Building one batch\n",
      "Applying item_tfms to the first sample:\n",
      "  Pipeline: ToTensor\n",
      "    starting from\n",
      "      (TensorText of size 161, TensorCategory(1))\n",
      "    applying ToTensor gives\n",
      "      (TensorText of size 161, TensorCategory(1))\n",
      "\n",
      "Adding the next 3 samples\n",
      "\n",
      "Applying before_batch to the list of samples\n",
      "  Pipeline: partial\n",
      "    starting from\n",
      "      [(TensorText of size 161, TensorCategory(1)), (TensorText of size 966, TensorCategory(1)), (TensorText of size 165, TensorCategory(1)), (TensorText of size 1025, TensorCategory(1))]\n",
      "    applying partial gives\n",
      "      [(TensorText of size 1025, TensorCategory(1)), (TensorText of size 1025, TensorCategory(1)), (TensorText of size 1025, TensorCategory(1)), (TensorText of size 1025, TensorCategory(1))]\n",
      "\n",
      "Collating items in a batch\n",
      "\n",
      "No batch_tfms to apply\n"
     ]
    }
   ],
   "source": [
    "DataBlock(\n",
    "    blocks=(TextBlock.from_folder(path, vocab=dls_lm.vocab),CategoryBlock),\n",
    "    get_y = parent_label,\n",
    "    get_items=get_text_files,\n",
    "    splitter=GrandparentSplitter()\n",
    ").summary(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos = xxmaj the xxmaj remix ( xxmaj lady xxmaj gaga album ) = \\n▁\\n▁ xxmaj the xxmaj remix is a remix album by xxmaj american recording artist xxmaj lady xxmaj gaga . xxmaj released in xxmaj japan on xxmaj march 3 , 2010 , it contains remixes of the songs from her first studio album , xxmaj the xxmaj fame ( 2008 ) , and her third extended play , xxmaj the xxmaj fame xxmaj monster ( 2009 ) . a revised version of the track list was prepared for release in additional markets , beginning with xxmaj mexico on xxmaj may 3 , 2010 . \\n xxmaj the xxmaj remix is a xxmaj remix album by xxmaj americ...</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>xxbos = 2007 xxmaj hawaii xxmaj bowl = \\n▁\\n▁ xxmaj the 2007 xxunk xxmaj hawaii xxmaj bowl was a post - season college football bowl game between the xxmaj boise xxmaj state xxmaj university xxmaj broncos from the xxmaj western xxmaj athletic xxmaj conference ( xxup wac ) and the xxmaj east xxmaj carolina xxmaj university xxmaj pirates from xxmaj conference xxup usa ( xxup c - usa ) at the xxunk xxmaj stadium in xxmaj honolulu , xxmaj hawaiʻi on xxmaj december 23 , 2007 . xxmaj the game was the final competition of the 2007 football season for each team and resulted in a 41 – 38 xxmaj east...</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>xxbos = xxmaj battle of xxmaj sullivan 's xxmaj island = \\n▁\\n▁ xxmaj the xxmaj battle of xxmaj sullivan 's xxmaj island or the xxmaj battle of xxmaj fort xxmaj sullivan was fought on xxmaj june 28 , 1776 , during the xxmaj american xxmaj revolutionary xxmaj war . xxmaj it took place near xxmaj charleston , xxmaj south xxmaj carolina , during the first xxmaj british attempt to capture the city from xxmaj american rebels . xxmaj it is also sometimes referred to as the xxmaj first xxmaj siege of xxmaj charleston , owing to a more successful xxmaj british siege in xxunk . xxmaj this event was...</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>xxbos = xxmaj world xxmaj war i xxmaj memorial ( xxmaj east xxmaj providence , xxmaj rhode xxmaj island ) = \\n▁\\n▁ xxmaj the xxmaj world xxmaj war i xxmaj memorial is a bronze sculpture by xxmaj pietro xxmaj montana and is located at the intersection of xxmaj taunton xxmaj avenue , xxunk xxmaj avenue , and xxmaj john xxmaj street in xxmaj east xxmaj providence , xxmaj rhode xxmaj island , xxmaj united xxmaj states . xxmaj the sculpture is modeled on xxmaj charles xxmaj atlas and depicts a dynamically posed soldier standing on a granite base . xxmaj montana 's original design was modified b...</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>xxbos = xxmaj battle of xxmaj hubbardton = \\n▁\\n▁ xxmaj the xxmaj battle of xxmaj hubbardton was an engagement in the xxmaj xxunk campaign of the xxmaj american xxmaj revolutionary xxmaj war fought in the village of xxmaj hubbardton , xxmaj vermont . xxmaj vermont was then a disputed territory sometimes called the xxmaj new xxmaj hampshire xxmaj grants , claimed by xxmaj new xxmaj york , xxmaj new xxmaj hampshire , and the newly organized and not yet recognized but de facto independent government of xxmaj vermont . xxmaj on the morning of xxmaj july 7 , 1 xxrep 3 7 , xxmaj british forces ,...</td>\n",
       "      <td>human</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>xxbos = xxmaj battle of xxmaj hubbardton = \\n▁\\n▁ xxmaj the xxmaj battle of xxmaj hubbardton was an engagement in the xxmaj xxunk campaign of the xxmaj american xxmaj revolutionary xxmaj war fought in the village of xxmaj hubbardton , xxmaj vermont . xxmaj vermont was then a disputed territory sometimes called the xxmaj new xxmaj hampshire xxmaj grants , claimed by xxmaj new xxmaj york , xxmaj new xxmaj hampshire , and the newly organized and not yet recognized but de facto independent government of xxmaj vermont . xxmaj on the morning of xxmaj july 7 , 1 xxrep 3 7 , xxmaj british forces ,...</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls_clas.show_batch(max_n=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = text_classifier_learner(dls_clas, AWD_LSTM, drop_mult=0.5, metrics=accuracy)\n",
    "learn = learn.load_encoder('finetuned')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.06309573650360108, lr_steep=0.007585775572806597)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEOCAYAAAB4nTvgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4VGXax/HvnUmjJBBIQkmhhhKKlEgRpVgQQQE7uO6i69rRVVdXfHWVxXXtrl0XG+oqLIKr6KLAoogKKEF6Dz3UUEJPMpl53j9mwCEkmYTkzJmZ3J/rmouZU2Z+hDi3z3nKEWMMSimlVHki7A6glFIq+GmxUEop5ZcWC6WUUn5psVBKKeWXFgullFJ+abFQSinllxYLpZRSfmmxUEop5ZdlxUJE3hWRPSKyooz9IiIvi0iOiCwTkW4++1wissT7mGZVRqWUUhVjZctiAjConP2XABnexy3AGz77jhtjungfQ62LqJRSqiIirXpjY8xcEWleziHDgA+MZ72RBSJSX0SaGGN2nsnnJSYmmubNy/s4pZRSJS1atGivMSbJ33GWFYsKSAG2+bzO9W7bCcSKSDZQDDxljPmstDcQkVvwtEpIT08nOzvb2sRKKRVmRGRLRY6zs4NbStl2YlXDdGNMFnAd8KKItCrtDYwx440xWcaYrKQkv4VRKaXUGbKzWOQCaT6vU4EdAMaYE39uBOYAXQMdTiml1K/sLBbTgN95R0X1Ag4aY3aKSIKIxACISCLQB1hlY06llKrxLOuzEJGJQH8gUURygceAKABjzJvAdGAwkAMcA270ntoe+KeIuPEUs6eMMVoslFLKRlaOhhrpZ78B7ixl+zygk1W5lFJKVZ7O4FZKKeWXFotK2Lb/GHuPFNodQymlAk6LRQVt23+MIS9/z5ipy+yOopRSAafFogKKit2MnriYQwXF/LRpP2638X+SUkqFES0WFfD012tYui2fQR0ac7igmHV7DtsdSSmlAkqLhR8zV+7inR82Map3Mx4a3A6A7M0HbE6llFKBpcWiHOt2H+b+T5bSMSWe/xvSnvQGtUmsG8OiLVoslFI1ixaLMizacoCr35xPbJSD167rRkykAxHh7OYJLNy83+54ZcrZc5gt+47aHUMpFWa0WJTi27V7+M3bC0ioHcXU28+hWcM6J/d1b5ZA7oHj7D5UYGPC0xU4XTz11RoufvF7LnphLq/PyaHY5T653+U2/LL1gA79VUqdETuXKA9Ks1bt5vZ/LaJdkzgm3NiDxLoxp+zPat4A8PRbDOncxI6Ip3C63GRvPsDD/1nOxr1HuSYrlcMFxTzz9VpmrNjFfQPbsmDjPj5bvJ2dBwuoHe3g1r6tuLlvC2pH6z+/Uqpi9NuihH8t2EKT+rFMvLkXcbFRp+3v0DSe2KgIsrfst6VYOF1u/vndBr5asYvdhwrZd7QQYyA1oRb/uqkn52YkYozhv8t38ujnKxn17s84IoS+GYn8aWBbvlmzm3/8bx0f/bSF+y5qw1XdU4l0aANTKVU+LRYl5B44Rsem9UotFABRjgi6pNW3ZURUzp7D3Dd5KctyD9KzRQM6Z9YjOS6WlPq1GNK5CXViPP+cIsKlnZvSq2VD5m3YR6+WDUiOiwXgqu6pLNqynyf+u5oxny5n/NyN3DewDYM7NiEiorRbjCillBaLUxhj2J5/nAFtk8s9LqtZA974bgNHC4tPfkFbnWvCvM08+dUa6kQ7eOM33bikk/9WTWLdGIae1fS07d2bNWDq7ecwa9Vunpu5ltEfL6ZD0w2MuaQd52XoTaSUUqfTYuFj/9EiCpxuUhJqlXtc9+YJuL41LN2WzzmtE8s91hiDy22qdKnn9TkbeHbGWi5ol8yTV3Y62UqoChFhYIfGXNC+EV8s3cHzs9by23d+pl+bJP5vcHvaNo4r9bxVOw7xy9YDbMg7wsa8oxQWu7i9f2v6tdEio1Q402LhI/fAcQBSE2qXe1y39AREIHvLgVKLxdcrdvHMjDUcOu7k0PFinG43b17fnYs7NK50pg8XbOHZGWu5vGsKz199VrVfKnJECMO7pnBJp8Z8OH8LL89ezyUvzWVQx8b0zUiiT+tEGsXH8tWKnbw/bzO/bM0HoHa0g5ZJdTh43Mmod39mQNskHh6SSevkutWa7wRjDLsPFZIUF4NDL5cpFXBaLHxsz/cUi5T65bcs6tWKom2jOLJLmZxX4HTx6OcrqBMTycAOjYmPjWL68p28MHMdF7VvVKkv+8+XbOfRz1dwYftknrmqs6V9CjGRDv5wXkuu6p7Kq9/k8PnSHUxfvguA2KgICpxuWiTW4dFLMxnUsTFN6sUiIhQWu/hgnqfIXPziXHq3bEhW8wTObt6ALmn1q3SZ7lCBk8Vb8/l2zR5mr9nNtv3Hadsojkcuba+Xy5QKMC0WPnIPHAPwexkKPPMtPl+yg2KX+5RLTJ8symXP4UI+vrbLyVZH28Z1ufffS5m1evcprYsf1u/lvslLaNcknt4tG9K7VUNcbjerdhxixfZDTP0llx7NG/Dqdd2ICtCIpfq1o3nk0kweHtKeDXlH+DFnH+t2H+aizEb0zUg6rWDFRDq4uW9LruiWwvi5G/l+/V5emr0eYyDKIXRvlsB5GUn0a5NEh6bxiJRd8HYdLGDSwq38sjWf9bsPs/OgZy5LbFQE57ZOZMTZ6fx74TZ++46nJfO73s0RgWKXIdIh9GmdGLCfk1I1jXhuWBf6srKyTHZ2dpXe47HPV/Dp4u0sH3ux32Nnr97NTe9nc//ANow+PwPwDGvt/+wcGsXHMPX2c05+MRa73FzwwnfExUbyxehzERHyjxUx8B9ziYwQ6sREsn7PkVPeP6F2FL1bNeTpKzuXOTIrWB0qcPLLlgPM37iP79ftZdXOQwC0axzHDec0Z1iXFGpFOwBPS2zptnw+WLCFr1fswm0M7RvH07ZxHBmN6pLZJJ5eLRsSG+U5vrDYxfvzNvPK7BwOFxaf8rkdmsbzzFWd6dC0XmD/wkqFMBFZZIzJ8nucFotf/eH9heQeOM7X9/T1e6wxhrsnLWH68p1Mua03XdMTmJy9jT9PWcZ7N5zNgHanjqj698KtPDh1+cl9d01czFfLd/LZnX3omFKPvMOF/LxpP9GREXRoGn/yMk842HO4gNmr9/DB/C2s3nmIerWiaNOoLtv2H2f34QKMgfjYSK49O43f9mpOesPy+4wA8o8VsW73ESIdQmSEsGnvUR7/cjX5x4q4vX8rftu7GcZ4Zq7XinKQUCc6AH9TpUKPFoszMOjFuaQm1ObtUX5/bgAcPO5k8Evf44gQvrjrXIa/9iO1ox18ede5p33RFxW7GfDcHJLjY7ixTwvunrj4lFZJTWCM4edN+/lwwRb2HCoktUEt0hvUpmVSXS5sn1zlGeX5x4p4/MvVTP0l95TtEQKXd03lngszSGvgvxApVZPYXixE5F3gUmCPMaZjKfsFeAkYDBwDbjDG/OLdNwp4xHvo34wx7/v7vKoWC2MMncfO5MruqYwd2qHC5/28aT8jxs+nZVJdcvYc4c3ruzGoY+lzID5csIW/fLaCmMgIMpvG88mtvXX2tAV+2ujpZ4mI8LQ61u46wr9+2oIxht/0bMYd/VuRHF/14cdKhYOKFgsrO7gnAK8CH5Sx/xIgw/voCbwB9BSRBsBjQBZggEUiMs0YY+mU6UPHizlcWExqBTq3ffVo0YA7B7TmlW9yyEiuy8DMsofHXt09lVdmr+dwQTH/uKaLFgqL9GzZkJ4tG56y7ea+LXh59no+XLCFj3/eyoiz07i1Xyu/I9+UUh6WFQtjzFwRaV7OIcOAD4ynabNAROqLSBOgPzDLGLMfQERmAYOAiVZlBcjN946EOoMvj7svyOBIYTGDO5W/ZEZslIMJN/agyOWmeWKdMo9T1a9JvVo8eUVnbu3bije/28DEn7fy8U9b6dWyIQVOFwePOylyubmzf2uuOTvN7rhKBR07h86mANt8Xud6t5W13VLbvRPyKjJstqQoRwSPXVaxS1eZTeMr/f6q+jRPrMNTV3bm7gsyGD93Iws37ycuNpKWSXXYdaiQP09dxqZ9R3lgYFtdK0spH3YWi9L+SzTlbD/9DURuAW4BSE9Pr1KYis7eVuGhaf1ap/VNFbvcPDptJW/M2cDWfcd4/pqzTg7ZVaqms/OieS7g295PBXaUs/00xpjxxpgsY0xWUlLVZvRuzz/uGWJZO7TmNKjqE+mI4InhHXl4cHumr9jJ5a/PY+m2fLtjKRUU7CwW04DfiUcv4KAxZicwAxgoIgkikgAM9G6zVO6BY6Qm1AqbuQ3qzIgIN/dtyTujsth/tJDhr//IY5+v4FCB0+5oStnKsstQIjIRT2d1oojk4hnhFAVgjHkTmI5n2GwOnqGzN3r37ReRx4GF3rcad6Kz20rb84+fUX+FCk/nt2vE/+5rwPMz1/H+/M1MX7GL3/RM56ruqXqpUtVIOinPq8u4mQzp1IQnLu9UjalUOFiyLZ/nZ67lh5y9AJzbOpG7zs+gR4sGNidTquoqOs9CB/oDRwuLyT/m1P9jVKXqklafD2/qydwHBvDHCzLI2XOEkW8t4MP5mwmX/9lSyh8tFvgsTa6XoVQ50hrU5p4L2zDz3r70b5PEXz5fyf/9ZwVFxW67oyllOS0W/Lo0eWVnb6uaKS42ivG/y+KO/q2Y+PNWrn/7J/YcKrA7llKW0mLBrxPyUnXpB1VBjgjhz4Pa8dKILizffpDBL//A/A377I6llGW0WOCZkBftiCCxbozdUVSIGdYlhc9H96FerUh+8/YCXvs2B7db+zFU+NFiAeR6h83q8g7qTLRpFMe00ecypHNTnp2xlr9+sdLuSEpVO72tKp7LULr6qKqKOjGRvDyiC8lxMbzzwyYym8Zz7dlVW4JGqWCiLQs8l6G0c1tVlYjw0CXtOC8jkb98tpJftlq6qr5SAVXji0WB08XeI4XaslDVItIRwSsju9K4Xiy3fbiI3TpKSoWJGl8sDhcU06tlA9o2jrM7igoT9WtH89bvsjhSWMxt/1qk8zBUWKjxxSIpLoZJt/RmYIey73CnVGW1bRzHc1efxeKt+Tw3c63dcZSqshpfLJSyyuBOTfhtr2aMn7uRb9bstjuOUlWixUIpCz08pD3tm8Tzp8lL2XnwuN1xlDpjWiyUslBslIPXrutKYbGbuycuxunS/gsVmrRYKGWxlkl1+fvlnVi4+QDDXv2R5bkH7Y6kwkhhsYviAPxPiBYLpQJgeNcU3ry+G3uPFDLstR/425erOFZUbHcsFQZu+WARV7453/LP0WKhVIAM6tiEWff1Y0SPdN7+YRPDX/tR+zFUlTldbqId1i9VpMVCqQCqVyuKv1/eiQ9v6sGO/AKufH0eOXuO2B1LhTCny02Uw/qvci0WStngvIwkJt3SiyKXm6vfnMdiXRpEnaGiYi0WSoW1jin1mHLbOcTFRnHdWz+xYKPeD0NVXpHLhH6xEJFBIrJWRHJEZEwp+5uJyGwRWSYic0Qk1WefS0SWeB/TrMyplF2aJ9Zhyu29SUmoxY3vLeQnLRiqkpwuNzGRIVwsRMQBvAZcAmQCI0Uks8RhzwEfGGM6A+OAJ332HTfGdPE+hlqVUym7JcfF8vHNPWlaP5YbJyxk4eb9dkdSIcTTZxHaHdw9gBxjzEZjTBEwCRhW4phMYLb3+bel7FeqRkiOi2Xizb1oXC+WG979WZc3VxUWDn0WKcA2n9e53m2+lgJXep9fDsSJSEPv61gRyRaRBSIy3MKcSgWF5PhYJt3ci8S4GG77cBF5hwvtjqRCgNPlJiqUL0MBpbWLSt6c+H6gn4gsBvoB24ETM5XSjTFZwHXAiyLS6rQPELnFW1Cy8/LyqjG6UvZIjo/ln7/tzqECJ3+ctBiX3s9b+VFU7CY6xFsWuUCaz+tUYIfvAcaYHcaYK4wxXYGHvdsOntjn/XMjMAfoWvIDjDHjjTFZxpispKQkS/4SSgVau8bxPD6sI/M27OPF/62zO44Kck6XITrEWxYLgQwRaSEi0cAI4JRRTSKSKCInMjwEvOvdniAiMSeOAfoAqyzMqlRQuTorjWuz0njlmxy+XbvH7jgqiBWFege3MaYYGA3MAFYDk40xK0VknIicGN3UH1grIuuARsAT3u3tgWwRWYqn4/spY4wWC1Wj/HVYB9o3iefWDxbxh/cXMmVRLgePOe2OpYKIy21wuQMzzyLSyjc3xkwHppfY9qjP8ynAlFLOmwd0sjKbUsEuNsrBezeczfi5G/l6xU7+t3oPUQ7hpRFdGdypid3xVBA4seR9qF+GUkpVUeN6sTx6WSY/jjmfz+7sQ7vG8Tzy2Qr2Hy2yO5oKAieLRYh3cCulqomI0CWtPs9fcxaHC5z87Uu9Kqs8I6GAkJ9noZSqZm0axXF7v1Z8ung7363T4eI1ndPlGVqtxUIpdZo7z29Nq6Q6/N+nyzlaqDdQqsm0z0IpVaaYSAdPXdmZ7fnHeX6mzsOoyYpcJy5DhfDQWaWUdc5u3oDre6UzYd4mlmzLtzuOsol2cCul/HpwUDuS42IZM3XZyY5OVbNoB7dSyq+42CgeH96RNbsO88/vNtgdR9ngRMsi1BcSVEpZ7KLMRgzp3IRXvskhZ89hu+OoACsq9oyGCsRlKEtncCulrDf2sg78sH4vf56yjDv6t8ZtDG4DXdPr0yg+1u54ykK/joayvoNbi4VSIS4pLobHLsvkvslL+cMH2Se3d0yJZ9qd5xIRYf0XibJHIPsstFgoFQau6JZK1/QEjhQUIwILNu7jb/9dzfQVO7m0c1O74ymLnOyz0GKhlKqoFol1Tj5v3ySeydnbeH7mOi7u0DggXyYq8Ip0Up5SqiocEcL9A9uyae9RpizKtTuOssiJ5T50noVS6oxdlNmIrun1eel/6ylwuuyOoyyg8yyUUlUmIvz54nbsOlTAB/M32x1HWcCpy30opapD71YN6dsmidfnbCD/mN4DI9zoQoJKqWozZlA7Dh138tzMtXZHUdWsKICjobRYKBXmMpvG87vezfnop60sy9VFB8OJ9lkoparVfQPb0LBODH/5bAUut7E7jqomTpcbR4TgCMDES0uLhYgMEpG1IpIjImNK2d9MRGaLyDIRmSMiqT77RonIeu9jlJU5lQp38bFRPDykHUtzD/LvhdvsjqOqidNlAjJsFiwsFiLiAF4DLgEygZEiklnisOeAD4wxnYFxwJPecxsAjwE9gR7AYyKSYFVWpWqC4V1S6NmiAU9/vYZ9RwrtjqOqQVGxOyAjocDalkUPIMcYs9EYUwRMAoaVOCYTmO19/q3P/ouBWcaY/caYA8AsYJCFWZUKeyLC48M7crSwmEEvfc/4uRv0tqwhzulyB2QkFFhbLFIA3/Zurnebr6XAld7nlwNxItKwgucqpSqpTaM4Jt3SizaN6vL36Wvo8/Q3TPhxk92x1BnytCxCv1iU1jYq2bN2P9BPRBYD/YDtQHEFz0VEbhGRbBHJzsvLq2pepWqErOYN+OgPvfjPHefQoWk8Y79YxYrtB+2Opc6A0xUexSIXSPN5nQrs8D3AGLPDGHOFMaYr8LB328GKnOs9drwxJssYk5WUlFTd+ZUKa13TE3jj+u7Urx2lczBClNNlwuIy1EIgQ0RaiEg0MAKY5nuAiCSKyIkMDwHvep/PAAaKSIK3Y3ugd5tSqhrFx0Zxe79WzFmbx8+b9tsdR1VSUTi0LIwxxcBoPF/yq4HJxpiVIjJORIZ6D+sPrBWRdUAj4AnvufuBx/EUnIXAOO82pVQ1+13v5iTHxfDsjDUYo3MwQklRsZvoAI2GsvR+FsaY6cD0Etse9Xk+BZhSxrnv8mtLQyllkVrRDu66IIO/fLaC79bl0b9tst2RVAWFS5+FUipEXJuVRlqDWjw7Yy1uneEdMsJl6KxSKkRER0Zw74VtWLnjENNX7LQ7jqqgIpfRloVSKrCGdUmhTaO6PD9z3cmlr1VwC5d5FkqpEOKIEB64uB2b9h7lk2y9FWso8FyGCv3lPpRSIebC9sl0b5bAS7PXcbxIb8Ua7Jwud3AtJCgirUQkxvu8v4jcLSL1rY2mlAo0EeHBQe3YfaiQCfM22x1H+eEMwstQUwGXiLQG3gFaAB9blkopZZseLRowoG0Sb8zJ4eAxp91xVDmKXG6igmw0lNs7ye5y4EVjzL1AE+tiKaXs9OdB7ThcWMxfv1jJgaN67+5g5ZmUF1zFwikiI4FRwJfebVHWRFJK2a19k3hu6tOCTxdvp/dTs3nks+Vs2nvU7liqhGBcG+pGoDfwhDFmk4i0AP5lXSyllN0euTSTGff0ZehZTZm8MJeB//hO7+EdZDwzuINoNJQxZpUx5m5jzETvwn5xxpinLM6mlLJZ28ZxPHPVWXz/4ADiYqN45mtdnTZYuN2GYneQTcrz3h873nu706XAeyLygrXRlFLBolF8LHcOaM0POXv5Yf1eu+MoPJ3bQHAVC6CeMeYQcAXwnjGmO3ChdbGUUsHm+l7ppNSvxdNf6+q0weDELPtg6+COFJEmwDX82sGtlKpBYiId3HtRG5ZvP8hXK3bZHafGc7o8BTvYOrjH4bkvxQZjzEIRaQmsty6WUioYXd7Vs37UczPWUqzrR9nKGYyXoYwxnxhjOhtjbve+3miMudLaaEqpYHNi/aiNe4/y5FdryD+mczDsUlR8olgE0WgoEUkVkf+IyB4R2S0iU0Uk1epwSqngc2H7ZC7t3IR3fthErydnM2bqMtbtPmx3rBrnRAd3sF2Geg/P/bObAinAF95tSqkaRkR49bpufPXH8xjeJYXPlmznsld+YHv+cbuj1SjB2sGdZIx5zxhT7H1MAJIszKWUCnLtm8Tz1JWdmXVvP9zG8MacHLsj1SjOYk8Hd1D1WQB7ReR6EXF4H9cD+6wMppQKDWkNanNV9zQmL8xl50FtXQRKkcuzhHywLST4ezzDZncBO4Gr8CwBopRS3NG/FW5jeHPOBruj1BhFJ1sWQdTBbYzZaowZaoxJMsYkG2OG45mgVy4RGSQia0UkR0TGlLI/XUS+FZHFIrJMRAZ7tzcXkeMissT7eLPSfzOlVMB4WhepTFy4jd2HCuyOUyOc6LOICbKWRWnuK2+niDiA14BLgExgpIhkljjsEWCyMaYrMAJ43WffBmNMF+/jtirkVEoFwB39W+NyG978TlsXgRCU8yzK4K/t0wPI8c7JKAImAcNKHGOAeO/zesCOKuRRStkovWFtruiawsc/bWWPti4sF0rFwt/iMCnANp/Xud5tvsYC14tILjAduMtnXwvv5anvROS8KuRUSgXI6PM9rYsHpy7D5db1o6xUWBxExUJEDovIoVIeh/HMuSj39FK2lfztGQlMMMakAoOBD0UkAk8nerr38tR9wMciEl/iXETkFhHJFpHsvLw8P3GUUlZr1rAOY4d24Nu1eTzx39V2xwlrJ9aGClSfRWR5O40xcVV471wgzed1KqdfZroJGOT9rPkiEgskGmP2AIXe7YtEZAPQBsgukW88MB4gKytL/zdGqSBwfa9mbMg7wrs/bqJlUh2u79XM7khhKZQuQ/mzEMgQkRYiEo2nA3taiWO2AhcAiEh7IBbIE5Ekbwc53kULM4CNFmZVSlWjR4Zkcn67ZB6btpK567TVb4Vfi0UQDZ09E8aYYmA0ntVqV+MZ9bRSRMaJyFDvYX8CbhaRpcBE4AbjWSi/L7DMu30KcJsxZr9VWZVS1csRIbw8sisZyXW54b2feejT5drpXc1OLiQYDJehqsoYMx1Px7Xvtkd9nq8C+pRy3lRgqpXZlFLWqhsTycSbe/HS7PX8a8EWPl+ynZvPa8no81sH7NJJOCsK0rWhlFKq0hLqRDN2aAf+d18/BrRL5qXZ63lh1jq7Y4WFYF0bSimlzljzxDq8dl03Rpydxj+/28CiLQfsjhTynC43jgjBERHifRZKKVXSw0Pa06ReLe7/ZCnHi1x2xwlpRS53wDq3QYuFUiqA4mKjePbqzmzae5Snv15jd5yQVlTsDmjfjxYLpVRAndMqkRvOac6EeZuZt2Gv3XFCltPlDtiEPNBioZSywYOD2tEisQ4PfbqcAqdejjoTTpe2LJRSYa5WtIO/De/Iln3HdJXaM6SXoZRSNUKf1olc2rkJr8/ZwJZ9R+2OE3KcLqMd3EqpmuGRIZlERQhjp63Es3iDqqgil5voSEfAPk+LhVLKNo3rxXLvRW34dm0eM1fttjtOSHG63ERry0IpVVOMOqc5bRvFMe6LVRwrKrY7TsjQDm6lVI0S5Yjg8eEd2Z5/nFe/ybE7TsjQDm6lVI3To0UDruiWwlvfbyRnzxG744SEIpchWudZKKVqmocuaU+tKAePfr5CO7srwKktC6VUTZQUF8MDg9oxb8M+pi0teVNNVZLT5SY6Uju4lVI10HU90umcWo+//Xc1hwqcdscJakXawa2UqqkcEcLjwzqy90ghr2lnd7n0MpRSqkY7K60+w85qygfzt7D3SKHdcYKWdnArpWq80ednUFDs4q3vN9odJWh5JuVpsVBK1WCtk+tyWeemfDh/C/uPFtkdJyh55lmESQe3iAwSkbUikiMiY0rZny4i34rIYhFZJiKDffY95D1vrYhcbGVOpVTwufuC1hx3auuiLGEzg1tEHMBrwCVAJjBSRDJLHPYIMNkY0xUYAbzuPTfT+7oDMAh43ft+SqkaonVyHEM6NeGDeZs5oK2LU7jdhmJ3+PRZ9AByjDEbjTFFwCRgWIljDBDvfV4PODG4ehgwyRhTaIzZBOR4308pVYPcfUEGR4tcvPHdBgqL9SZJJzjdboCAtiwiLXzvFGCbz+tcoGeJY8YCM0XkLqAOcKHPuQtKnJtiTUylVLBq08jTuhg/dyPj524kOS6G5ol1+OvQDrRvEu//DcJUUbGnWIRLB3dpPS8l5/CPBCYYY1KBwcCHIhJRwXMRkVtEJFtEsvPy8qocWCkVfJ69ujPPX30W917Yhv5tk8jZc4Q/TV5KscttdzTbOF2er8NAdnBb2bLIBdJ8Xqfy62WmE27C0yeBMWa+iMQCiRU8F2PMeGA8QFZWli4mo1QYqh0dyZXdU0++nr58J3d89AsfzN++xNzlAAASYklEQVTC789tYWMy+zi9hTJcbn60EMgQkRYiEo2nw3paiWO2AhcAiEh7IBbI8x43QkRiRKQFkAH8bGFWpVSIuKRjY/q1SeKFWevYfajA7ji2OHEZKiyGzhpjioHRwAxgNZ5RTytFZJyIDPUe9ifgZhFZCkwEbjAeK4HJwCrga+BOY4z2bimlEBHGDeuA0+Vm3Jer7I5ji19bFuHRwY0xZjowvcS2R32erwL6lHHuE8ATVuZTSoWmZg3rcOeA1rwwax3XZuXRt02S3ZECqsgV+NFQOoNbKRWSbu3XkpaJdRgzdRl7Dtesy1HOYk8XbbiMhlJKKcvERDp4eWRX9h8r4tYPF1HgrDlXqk+2LMJkUp5SSlmqY0o9Xry2C4u35vPg1GU15g57TlcYdXArpVQgDOrYhAcubsvnS3bw2rc14x4YdkzKs7SDWymlAuGO/q1Yv/swz81cx/o9R7h/YFvSGtS2O5ZlnDZ0cGuxUEqFPBHh6as606R+Ld79YRPTl+/k+l7N+OMFGdSvHW13vGpnx9BZvQyllAoLMZEOHhzUju8eGMCV3VJ5f95mRr23MCyXBSk6udyHFgullDojjevF8tSVnXlpRFeWbsvnze822B2p2oXbQoJKKWWby85qymVnNeWl2etZueOg3XGq1ck+i0gdDaWUUlU2bmgH6teO5r5/Lw2r+2Gc7LPQloVSSlVdQp1onrmyM2t3H+bF/623O061ObmQoHZwK6VU9RjQLpkRZ6fx5ncbmPDjJrvjVIsiG1oWOnRWKRX2xg7twP6jRYz9YhV7DhfywMVtEQnc9f7qdmJtKB0NpZRS1Sg2ysEb13fnup7pvD5nA/d/suzkdf9Q5HS5cUQIjojwuFOeUkoFDUeE8MTwjiTHxfDi/9YTFxvJ2KEd7I51Rpwud0DXhQItFkqpGkREuOfCNuQfczJh3mYGdmjEOa0S7Y5VaUUud0AvQYFehlJK1UAPDmpHi8Q6/HnKMo4UFtsdp9KKit0B7dwGLRZKqRqoVrSD567uzI784/x9+mq741Sa0+UO6LpQoMVCKVVDdW/WgJvPa8nHP21l7ro8u+NUitNl9DKUUkoFyr0XtSEjuS4PTg2ty1FFNnRwW1osRGSQiKwVkRwRGVPK/n+IyBLvY52I5Pvsc/nsm2ZlTqVUzRQb5eDpqzqz61ABL8xcZ3ecCisqDnwHt2WjoUTEAbwGXATkAgtFZJoxZtWJY4wx9/ocfxfQ1ectjhtjuliVTymlALqlJ3B9z2ZMmLeJy7um0Cm1nt2R/Aq3PoseQI4xZqMxpgiYBAwr5/iRwEQL8yilVKkeGNSWhnVjGPPpspC4/4XTFV6joVKAbT6vc73bTiMizYAWwDc+m2NFJFtEFojIcOtiKqVquvjYKMZe1oGVOw4xYd5mu+P45SwOrw7u0npfTBnHjgCmGGN81xBON8ZkAdcBL4pIq9M+QOQWb0HJzssLrdEMSqngMrhTY85vl8wLs9axPf+43XHKVehyB3TFWbC2WOQCaT6vU4EdZRw7ghKXoIwxO7x/bgTmcGp/xoljxhtjsowxWUlJSdWRWSlVQ4kIfx3aAZfb8NyMtXbHKZez2E10GI2GWghkiEgLEYnGUxBOG9UkIm2BBGC+z7YEEYnxPk8E+gCrSp6rlFLVKa1BbW7s04LPlmxnxfbgvbteWHVwG2OKgdHADGA1MNkYs1JExonIUJ9DRwKTjDG+l6jaA9kishT4FnjKdxSVUkpZ5fb+rahXK4qnv15jd5QyOW1YG8rShQSNMdOB6SW2PVri9dhSzpsHdLIym1JKlaZerSjuOj+Dx79cxdx1efRtE3yXuO2YZ6EzuJVSqoTre6WT1qAWT361Bpe7rHE59inS5T6UUsp+MZEOHri4Hat3HuI/i7fbHec0TpebmHDps1BKqVB2aacmnJVWn0c/X8GCjfvKPXbLvqNM+HFTwFohdtz8SIuFUkqVIiJCeOu33WlavxY3vPcz368vey7X41+uZuwXq/jT5CUBmQFuRwe3FgullCpDcnwsk27pRfOGdbjp/Wy+WbP7tGN25B/nmzW7adsojs+W7OC+yUstLRhut9ElypVSKtgk1o1h4s29aNsojls/XMSqHYdO2T9p4TYM8PaoLB4c1I5pS3dwz7+ta2E43Z73DZt5FkopFS4S6kTzwe97UDcmknFfruTEtLBil5t/L9xKvzZJpDWoze39W/HQJe34ctlORoxfwLb9x6o9i9Pl+exwWkhQKaXCRkKdaO4b2JYFG/fz9YpdAMxes4fdhwq5rkf6yeNu7deKl0Z0Ye2uw1zy0vdMWZTLqXOOq8ZZ7GlZaAe3UkoFqZFnp9GucRxPTF9NgdPFRz9tpXF8LOe3Sz7luGFdUvjqnvPIbBrP/Z8s5YEpy6qtYBR5L2+F00KCSikVViIdETx6aSa5B44zdtpK5q7LY0SPNCJLuSSUmlCbiTf34o7+rZiyKJd3f9xcLRmOem//qh3cSikVxM5pncigDo2ZtHAbjghhxNnpZR7riBAeuLgtF2U24snpq1m89UCVPnvfkUJGf7yYKIfQoWl8ld6rsrRYKKVUJT08pD3RkRFc0C6ZxvViyz1WRHjuqrNoXC+W0R8vJv9Y0Rl95q6DBVzzz/ls3HuEt0edTYemgb39q1Rnx4udsrKyTHZ2tt0xlFI1xModB2kUH0ti3ZgKHb90Wz5XvTmPfm2SeOt3WYiU30G9fvdhNu87RoHTxXGni1e+Wc+Bo07eGZVFz5YNq+OvAICILPLeaK5clq46q5RS4aqy/2d/Vlp9Hh7cnrFfrOKTRblck5VW5rFHC4u57NUfKHD+OlcjoXYUH/2hJ2el1T/jzFWhxUIppQJk1DnN+XzpDp6dsZYhnZpQJ6b0r+AfcvZS4HTzzFWd6ZJWn9hIB0lxMdSKdgQ48a+0z0IppQJERPjLpZnkHS7kjTkbyjzu2zV7iIuJ5PKuKbRpFEd6w9q2FgrQYqGUUgHVLT2BYV2a8tb3G8k9cPoMb2MM36zZQ982SQEfHlue4EmilFI1xIOD2iECT3+99rR9K3ccYs/hQgaUmOhnNy0WSikVYE3r1+KW81ryxdIdLNqy/5R936zZgwj0bxtct3PVYqGUUja4tV8rGsXHMHbaqlNumvTNmj10Tq1f4SG5gaLFQimlbFAnJpKHh2SyfPtBPv5pCwB7jxSyNDefC4LsEhRYXCxEZJCIrBWRHBEZU8r+f4jIEu9jnYjk++wbJSLrvY9RVuZUSik7XNa5CX1aN+SZGWvJO1zInLV5GMNpCxMGA8vmWYiIA3gNuAjIBRaKyDRjzKoTxxhj7vU5/i6gq/d5A+AxIAswwCLvuVVbWEUppYKIiDBuWEcGvTiXJ6evprDYTXJcTMDXfaoIK1sWPYAcY8xGY0wRMAkYVs7xI4GJ3ucXA7OMMfu9BWIWMMjCrEopZYtWSXW5tW8rPl28nVmrd3N+u2S/S4HYwcpikQJs83md6912GhFpBrQAvqnMuSJyi4hki0h2Xl7ZN1NXSqlgdueA1qQm1KKo2B10Q2ZPsLJYlFYay1q1cAQwxRjjqsy5xpjxxpgsY0xWUlJwDTNTSqmKqhXt4JkrO9OrZQPOy0i0O06prFwbKhfwXSkrFdhRxrEjgDtLnNu/xLlzqjGbUkoFlXNaJ3JO6+AsFGBty2IhkCEiLUQkGk9BmFbyIBFpCyQA8302zwAGikiCiCQAA73blFJK2cCyloUxplhERuP5kncA7xpjVorIOCDbGHOicIwEJhmfG2sYY/aLyON4Cg7AOGPMqdMclVJKBYze/EgppWqwit78SGdwK6WU8kuLhVJKKb+0WCillPJLi4VSSim/tFgopZTyK2xGQ4lIHrDF+7IecLCCz0/8mQjsreTH+r5fRfb521ZePqtyVjRrea+DOWsw/fuHUlb9XQ18Vrv+/TOMMfX8voMxJuwewPiKPvf5M7sqn1ORff62+clnSc6KZi3vdTBnDaZ//1DKqr+r+rta8hGul6G+qMRz321V+ZyK7PO3rbx8VuUsa3/JbeW9DuaswfTvX9r2YM2qv6v+hfvv6inC5jJUVYlItqnAxBS7hUpO0KxWCZWsoZITNGtFhGvL4kyMtztABYVKTtCsVgmVrKGSEzSrX9qyUEop5Ze2LJRSSvmlxUIppZRfWiyUUkr5pcXCDxE5T0TeFJG3RWSe3XnKIyIRIvKEiLwiIqPszlMeEekvIt97f7b97c7jj4jUEZFFInKp3VnKIiLtvT/PKSJyu915yiMiw0XkLRH5XEQG2p2nPCLSUkTeEZEpdmcpyft7+b73Z/kbKz8rrIuFiLwrIntEZEWJ7YNEZK2I5IjImPLewxjzvTHmNuBL4P1gzgoMA1IAJ55b0wZzVgMcAWJDICvAg8Bka1JW2+/qau/v6jWAZUMrqynrZ8aYm4EbgGuDPOtGY8xNVmUsqZKZrwCmeH+WQy0NVtmZgKH0APoC3YAVPtscwAagJRANLAUygU54CoLvI9nnvMlAfDBnBcYAt3rPnRLkWSO85zUCPgryrBfiuS3wDcClwZrTe85QYB5wXTD/TH3Oex7oFiJZLftvqgqZHwK6eI/52Mpclt1WNRgYY+aKSPMSm3sAOcaYjQAiMgkYZox5Eij1EoOIpAMHjTGHgjmriOQCRd6XrmDO6uMAEGNFTqi2n+sAoA6e/ziPi8h0Y4w72HJ632caME1E/gt8XJ0ZqzOriAjwFPCVMeYXK3JWV9ZAq0xmPK3yVGAJFl8pCutiUYYUYJvP61ygp59zbgLesyxR2Sqb9VPgFRE5D5hrZbBSVCqriFwBXAzUB161NtppKpXVGPMwgIjcAOyt7kJRjsr+TPvjuSwRA0y3NNnpKvu7eheeFls9EWltjHnTynAlVPbn2hB4AugqIg95i0qglZX5ZeBVERlC1ZYD8asmFgspZVu5MxONMY9ZlMWfSmU1xhzDU9jsUNmsn+Ipbnao9O8AgDFmQvVHKVdlf6ZzgDlWhfGjsllfxvNFZ4fKZt0H3GZdnAopNbMx5ihwYyAChHUHdxlygTSf16nADpuy+KNZrREqWUMlJ2hWq9meuSYWi4VAhoi0EJFoPB2X02zOVBbNao1QyRoqOUGzWs3+zIHo3bfrAUwEdvLrUNKbvNsHA+vwjC542O6cmlWzhkpOzVpzM+tCgkoppfyqiZehlFJKVZIWC6WUUn5psVBKKeWXFgullFJ+abFQSinllxYLpZRSfmmxUGFNRI4E+PPeFpHManovl4gsEZEVIvKFiNT3c3x9EbmjOj5bqZJ0noUKayJyxBhTtxrfL9IYU1xd7+fns05mF5H3gXXGmCfKOb458KUxpmMg8qmaRVsWqsYRkSQRmSoiC72PPt7tPURknogs9v7Z1rv9BhH5RES+AGaK5y5/c8RzR7o1IvKRd8ltvNuzvM+PiOfOhUtFZIGINPJub+V9vVBExlWw9TMfz8qjiEhdEZktIr+IyHIRGeY95imglbc18qz32Ae8n7NMRP5ajT9GVcNosVA10UvAP4wxZwNXAm97t68B+hpjugKPAn/3Oac3MMoYc773dVfgHjz3uGgJ9Cnlc+oAC4wxZ+FZMv5mn89/yfv5fheDExEHcAG/rgVUAFxujOkGDACe9xarMcAGY0wXY8wD4rldaQaeeyF0AbqLSF9/n6dUaWriEuVKXQhkehsDAPEiEgfUA94XkQw8S1ZH+Zwzyxiz3+f1z8aYXAARWQI0B34o8TlFeO62BrAIuMj7vDcw3Pv8Y+C5MnLW8nnvRcAs73YB/u794nfjaXE0KuX8gd7HYu/runiKR6DvdaLCgBYLVRNFAL2NMcd9N4rIK8C3xpjLvdf/5/jsPlriPQp9nrso/b8lp/m1U7CsY8pz3BjTRUTq4Sk6d+K5B8RvgCSguzHGKSKb8dzLvCQBnjTG/LOSn6vUafQylKqJZgKjT7wQkS7ep/WA7d7nN1j4+QvwXP4Cz1LT5TLGHATuBu4XkSg8Ofd4C8UAoJn30MNAnM+pM4Dfi8iJTvIUEUmupr+DqmG0WKhwV1tEcn0e9+H54s3ydvqu4te7oD0DPCkiPwIOCzPdA9wnIj8DTYCD/k4wxiwGluIpLh/hyZ+Np5WxxnvMPuBH71DbZ40xM/Fc5povIsuBKZxaTJSqMB06q1SAiUhtPJeYjIiMAEYaY4b5O08pO2mfhVKB1x141TuCKR/4vc15lPJLWxZKKaX80j4LpZRSfmmxUEop5ZcWC6WUUn5psVBKKeWXFgullFJ+abFQSinl1/8DheIwsKPvGmIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.717503</td>\n",
       "      <td>0.731348</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>00:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.693786</td>\n",
       "      <td>0.567347</td>\n",
       "      <td>0.664179</td>\n",
       "      <td>00:14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(2, 0.003)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-instantiating `text_classifier_learner` is not needed after performing `lr_find()`. fastai takes care of resetting the state of the network and the optimizer before movin on.\n",
    "\n",
    "I still do that to apply `to_fp16()`, which dramatically speeds up training. \n",
    "Apparently `to_fp16()` and `lr_find()` don't get along very well. `fit_one_cycle` fails when running on a `learner.to_fp16()` object after going through `lr_find()`. I will soon open a Github issue around that. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.016289</td>\n",
       "      <td>0.728307</td>\n",
       "      <td>0.522388</td>\n",
       "      <td>00:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.833513</td>\n",
       "      <td>0.495025</td>\n",
       "      <td>0.791045</td>\n",
       "      <td>00:20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = text_classifier_learner(dls_clas, AWD_LSTM, drop_mult=0.5, metrics=accuracy).to_fp16()\n",
    "learn = learn.load_encoder('finetuned')\n",
    "learn.fit_one_cycle(2, 1.4e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.545216</td>\n",
       "      <td>0.400901</td>\n",
       "      <td>0.791045</td>\n",
       "      <td>00:26</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-2)\n",
    "learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.407080</td>\n",
       "      <td>0.309903</td>\n",
       "      <td>0.902985</td>\n",
       "      <td>00:49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-3)\n",
    "learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.330615</td>\n",
       "      <td>0.281932</td>\n",
       "      <td>0.917910</td>\n",
       "      <td>00:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.382046</td>\n",
       "      <td>0.256846</td>\n",
       "      <td>0.902985</td>\n",
       "      <td>00:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.346230</td>\n",
       "      <td>0.249746</td>\n",
       "      <td>0.910448</td>\n",
       "      <td>00:55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.312762</td>\n",
       "      <td>0.247457</td>\n",
       "      <td>0.895522</td>\n",
       "      <td>00:56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.325363</td>\n",
       "      <td>0.267269</td>\n",
       "      <td>0.895522</td>\n",
       "      <td>00:41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.312981</td>\n",
       "      <td>0.289107</td>\n",
       "      <td>0.888060</td>\n",
       "      <td>00:56</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(6, slice(1e-3/(2.6**4),1e-3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.export()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#1) [Path('wikifake_tiny/export.pkl')]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Path(\"./wikifake_tiny/\").ls(file_exts='.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Article's actual class: machine\n",
      "Prediction: machine; Probability: 0.9916\n"
     ]
    }
   ],
   "source": [
    "idx = 0\n",
    "classes = learn.dls.vocab[1]\n",
    "pred,pred_idx,probs = learn.predict(dls_clas.valid_ds[idx][0])\n",
    "actual = int(dls_clas.valid_ds[idx][1])\n",
    "\n",
    "print(f\"Article's actual class: {classes[actual]}\\nPrediction: {classes[pred_idx]}; Probability: {probs[pred_idx]:.04f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **`category`**: actual\n",
    "* **`category_`**: prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>category_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos = xxmaj canning xxmaj dam = \\n▁\\n▁ xxmaj the xxmaj canning xxmaj dam and reservoir provide a major fresh water resource for the city of xxmaj perth , xxmaj western xxmaj australia . xxmaj the dam is situated on the xxmaj darling xxunk and is an xxunk of the xxmaj canning xxmaj river . xxmaj it is noted for its innovative structural and hydraulic design that was considered to be at the forefront of concrete gravity dam design at the time of construction . xxmaj the xxmaj canning xxmaj dam was xxmaj perth 's primary water supply up until the 1960s when other sources of fresh water were...</td>\n",
       "      <td>machine</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>xxbos = xxmaj otra xxmaj nota = \\n▁\\n▁ xxmaj otra xxmaj nota ( xxmaj english : xxmaj another xxmaj note ) is the debut album by xxmaj american singer xxmaj marc xxmaj anthony that was released on xxmaj january 26 , 1993 , by xxup rmm xxmaj records . xxmaj produced by xxmaj sergio xxmaj george , it was the first album by xxmaj anthony to record in salsa after starting his career as a freestyle musician . xxmaj recording of the album began after xxmaj anthony asked xxup rmm president xxmaj ralph xxmaj mercado to record xxmaj juan xxmaj gabriel 's \" xxmaj hasta xxmaj que xxmaj te xxmaj conocí...</td>\n",
       "      <td>human</td>\n",
       "      <td>human</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>xxbos = xxmaj cambodian xxmaj campaign = \\n▁\\n▁ xxmaj the xxmaj cambodian xxmaj campaign ( also known as the xxmaj cambodian xxunk and the xxmaj cambodian xxmaj invasion ) was a series of military operations conducted in eastern xxmaj cambodia during 1970 by the xxmaj united xxmaj states and the xxmaj republic of xxmaj vietnam ( xxmaj south xxmaj vietnam ) ( xxunk ) during the xxmaj vietnam xxmaj war . xxmaj these xxunk were a result of the policy of xxmaj president xxmaj richard xxmaj nixon . a total of 13 major operations were conducted by the xxmaj army of the xxmaj republic of xxmaj vi...</td>\n",
       "      <td>machine</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>xxbos = xxmaj scientology in xxmaj germany = \\n▁\\n▁ xxmaj the xxmaj church of xxmaj scientology has been present in xxmaj germany since 1970 . xxmaj german authorities estimate that there are 4 @,@ xxrep 3 0 active xxmaj scientologists in xxmaj germany today ; the xxmaj church of xxmaj scientology gives a membership figure of around 12 @,@ xxrep 3 0 . xxmaj the xxmaj church of xxmaj scientology has encountered particular xxunk from the xxmaj german press and government and occupies a xxunk legal , social and cultural position in xxmaj germany . xxmaj the xxmaj church of xxmaj scientology w...</td>\n",
       "      <td>machine</td>\n",
       "      <td>human</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>xxbos = xxmaj far xxmaj away xxmaj places ( xxmaj mad xxmaj men ) = \\n▁\\n▁ \" xxmaj far xxmaj away xxmaj places \" is the sixth episode of the fifth season of the xxmaj american television drama series xxmaj mad xxmaj men and the xxunk episode of the series overall . xxmaj it was written by series creator and executive producer xxmaj matthew xxmaj xxunk and writer xxunk xxunk , and directed by xxmaj scott xxunk . xxmaj it originally aired on xxup amc in the xxmaj united xxmaj states on xxmaj april 22 , 2012 . \\n▁ xxmaj the episode takes place almost entirely over a single day , telling three...</td>\n",
       "      <td>machine</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>xxbos = xxmaj andrew xxmaj johnston ( singer ) = \\n▁\\n▁ xxmaj andrew xxmaj johnston ( born 23 xxmaj september 1994 ) is a xxmaj british singer who rose to fame when he appeared as a boy xxunk on the second series of the xxup uk television talent show xxmaj britain 's xxmaj got xxmaj talent in 2008 . xxmaj although he did not win the competition , he received a contract to record with xxmaj xxunk xxmaj music , a label owned by the xxmaj britain 's xxmaj got xxmaj talent judge xxmaj simon xxmaj xxunk . xxmaj johnston 's debut album , xxmaj one xxmaj voice , was released in xxmaj september of...</td>\n",
       "      <td>machine</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>xxbos = xxmaj silver xxmaj bullet ( roller coaster ) = \\n▁\\n▁ xxmaj silver xxmaj bullet is a western - themed steel inverted roller coaster designed by xxmaj bolliger &amp; xxmaj mabillard located at xxmaj knott 's xxmaj berry xxmaj farm , an amusement park in xxmaj buena xxmaj park , xxmaj california . xxmaj the $ 16 million roller coaster was announced on xxmaj december 1 , 2003 and opened on xxmaj december 7 , 2004 . a first xxunk auction was also held where people would bid on seats to be the first riders . a contract was awarded to xxmaj bolliger &amp; xxmaj mabillard and the coaster was at c...</td>\n",
       "      <td>machine</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>xxbos = xxmaj otra xxmaj nota = \\n▁\\n▁ xxmaj otra xxmaj nota ( xxmaj english : xxmaj another xxmaj note ) is the debut album by xxmaj american singer xxmaj marc xxmaj anthony that was released on xxmaj january 26 , 1993 , by xxup rmm xxmaj records . xxmaj produced by xxmaj sergio xxmaj george , it was the first album by xxmaj anthony to record in salsa after starting his career as a freestyle musician . xxmaj recording of the album began after xxmaj anthony asked xxup rmm president xxmaj ralph xxmaj mercado to record xxmaj juan xxmaj gabriel 's \" xxmaj hasta xxmaj que xxmaj te xxmaj conocí...</td>\n",
       "      <td>machine</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>xxbos = xxup uss xxmaj breese ( xxup xxunk ) = \\n▁\\n▁ xxup uss xxmaj breese ( xxup dd – 122 ) was a xxmaj wickes class destroyer in the xxmaj united xxmaj states xxmaj navy during xxmaj world xxmaj war i , and later redesignated , xxup dm-18 in xxmaj world xxmaj war xxup ii . xxmaj she was the only ship named for xxmaj captain xxunk xxmaj breese . \\n▁ xxmaj commissioned as a destroyer in 1919 , she undertook a number of patrol and training duties along the xxmaj east xxmaj coast of the xxmaj united xxmaj states until being decommissioned in 1922 . xxmaj she was presented to the government ...</td>\n",
       "      <td>machine</td>\n",
       "      <td>machine</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.show_results()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "fastai"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}