IMDB.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "IMDB.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q6fzjAGfj1o8",
        "colab_type": "text"
      },
      "source": [
        "**Initialization**\n",
        "* Setting up the Fastai Environment. I am using Colab for the Project so that the process of getting ready for the Fastai Environment might be different in other platforms."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RuTMfec1js3J",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Setting up the Fastai Environment.\n",
        "# !pip install -Uqq fastbook\n",
        "# import fastbook\n",
        "# fastbook.setup_book()"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dyyLc4x0mRSU",
        "colab_type": "text"
      },
      "source": [
        "* I prefer to use these 3 lines of code on top of my Notebook. It helps while reloading the Notebook. The third line of code helps to make plots within the Notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EFE-efhVm_rc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Initialization\n",
        "%reload_ext autoreload\n",
        "%autoreload 2\n",
        "%matplotlib inline"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ddh6Wbsl_7i",
        "colab_type": "text"
      },
      "source": [
        "**Libraries and Dependencies**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r6iJeGx0lD75",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Downloading and Importing the Libraries and Dependencies.\n",
        "from fastbook import *                                        # Importing all the Libraries and Dependencies.\n",
        "from fastai.text.all import *\n",
        "from IPython.display import display                           # Assist in Displaying.\n",
        "from IPython.display import HTML"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bl7D7dv-ny3D",
        "colab_type": "text"
      },
      "source": [
        "**Getting the Data**\n",
        "* Fastai has a number of [Dataset](https://course.fast.ai/datasets) which makes easy to download and to use. I will be using the [IMDB Dataset](https://course.fast.ai/datasets) for this Project available in Fastai. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i26oEe-iszCo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "7f676d7c-e10c-47d5-868c-ec95c0900ad7"
      },
      "source": [
        "# Downloading and accessing the IMDB Dataset.\n",
        "path = untar_data(URLs.IMDB)                                 # Downloads the IMDB Dataset."
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IN9X7bR5t_yo",
        "colab_type": "text"
      },
      "source": [
        "* Now, I will use get text files function to grab all the text files in a path obtained above. In Fastai, the optional parameter folders can be passed to restrict the search to a particular list of sub folders."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vzdd7L4Nt-UT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "6c7237cc-da78-4c5c-abfa-cc7cfdfb7949"
      },
      "source": [
        "# Getting all the Text Files.\n",
        "files = get_text_files(path, folders=[\"train\", \"test\", \"unsup\"])\n",
        "\n",
        "# Inspecting the files.\n",
        "text = files[0].open().read()                                       # It opens only the first document of the text.\n",
        "text[:100]                                                          # Printing the first 100 words of the text."
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Fuckland is an interesting film. I personally love the Dogma movement. I wish it had lasted longer. '"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YbYvTIPawsxH",
        "colab_type": "text"
      },
      "source": [
        "**Word Tokenization**\n",
        "* I will use Fastai Tokenizer for the process of Word Tokenization. Then, I will use Fastai coll_repr function to display the results. It displays the first n items of the collection. The collections of text documents should be wrap into list. The tokens starting with xx are the special tokens which is not a common word prefix in English."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wzmmn66Xv0Fv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "fa0094fe-8849-4d9f-b4f2-6e6ba98d0503"
      },
      "source": [
        "# Word Tokenization\n",
        "spacy = WordTokenizer()                                             # Instantiating the Tokenizer.\n",
        "# tokens = first(spacy([text]))                                     # First refers to every element.\n",
        "tokens = Tokenizer(spacy)                                           # Fastai Tokenizer.\n",
        "display(coll_repr(tokens(text), 30))                                # Printing the first 30 items from tokens."
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"(#424) ['xxbos','xxmaj','fuckland','is','an','interesting','film','.','i','personally','love','the','xxmaj','dogma','movement','.','i','wish','it','had','lasted','longer','.','xxmaj','it','seems','to','have','already','died'...]\""
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ecAj3tY3Aiq",
        "colab_type": "text"
      },
      "source": [
        "**Subword Tokenization**\n",
        "* In Chinese and Japanese languages there are no spaces in the sentences. Similarly Turkish Languages add many subwords together without spaces creating very long words. In such problems the Subword Tokenization plays the key role."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qcRJHX8Kyj3o",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "c218d83b-13b9-40ed-a3ae-0aaae942b815"
      },
      "source": [
        "# Subword Tokenization.\n",
        "texts = L(x.open().read() for x in files[:2000])                    # First 2000 movie reviews.\n",
        "\n",
        "def subword(sz):\n",
        "  sp = SubwordTokenizer(vocab_sz=sz)\n",
        "  sp.setup(texts)\n",
        "  return \" \".join(first(sp([text]))[:40])\n",
        "\n",
        "# Implementing the Subword.\n",
        "subword(1000)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'▁F uck land ▁is ▁an ▁interesting ▁film . ▁I ▁person ally ▁love ▁the ▁Do g ma ▁mo ve ment . ▁I ▁w ish ▁it ▁had ▁last ed ▁long er . ▁It ▁seem s ▁to ▁have ▁already ▁di ed . ▁Ma'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q-_3_gQMVGa4",
        "colab_type": "text"
      },
      "source": [
        "**Numericalization**\n",
        "* Numericalization is the process of mapping tokens to integers. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nvf8o3ODTq1O",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "1981e3c5-1810-46e4-faf0-dcb39428288d"
      },
      "source": [
        "# Numericalization\n",
        "token = tokens(text)\n",
        "token200 = texts[:200].map(tokens)\n",
        "display(token200[0])                                                             # Inspecting the first token.\n",
        "\n",
        "num = Numericalize()                                                             # Instantiating Numericalization.\n",
        "num.setup(token200)                                                              # Numericalizing first 200 tokens.\n",
        "print(coll_repr(num.vocab, 30))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "(#424) ['xxbos','xxmaj','fuckland','is','an','interesting','film','.','i','personally'...]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "(#2208) ['xxunk','xxpad','xxbos','xxeos','xxfld','xxrep','xxwrep','xxup','xxmaj','the',',','.','and','a','of','to','is','it','in','i','\"','that',\"'s\",'this','-','as','\\n\\n','with','was','for'...]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "377iyUNRWfux",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "d33ab925-f4bb-4898-f199-3e8073719d64"
      },
      "source": [
        "# Preparing LMDataLoader.\n",
        "nums200 = token200.map(num)                                                       # Applying Numericalization.\n",
        "dl = LMDataLoader(nums200)                                                        # Preparing LMDataLoader.\n",
        "\n",
        "# Inspecting the LMDataLoader.\n",
        "X, y = first(dl)                                                                  # First refers to every elements.\n",
        "display(f\"Shape of X is {X.shape}\")\n",
        "display(f\"Shape of y is {y.shape}\")"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Shape of X is torch.Size([64, 72])'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Shape of y is torch.Size([64, 72])'"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6X9yR1NjZ81V",
        "colab_type": "text"
      },
      "source": [
        "### **Training the Text Classifier**\n",
        "* Assembling the Data for Training. There are two steps for training the state of art Text classifier using Transfer Learning. First the model should be fine tuned on IMDB reviews corpus on Wikipedia. Then the model can be used to train the classifier."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "izUr8F0BbRbG",
        "colab_type": "text"
      },
      "source": [
        "**Language Model using DataBlock**\n",
        "* Fastai handles Tokenization and Numericalization automatically when TextBlock is passed to the DataBlock. All the arguments that can be passed to Tokenize and Numericalize can also be passed to the TextBlock."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tj8tP5eEZxYI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 247
        },
        "outputId": "5553f206-ba6b-401e-e57f-5153877ff597"
      },
      "source": [
        "# Preparing the Language Model using DataBlock.\n",
        "get_imdb = partial(get_text_files, folders=[\"train\", \"test\", \"unsup\"])\n",
        "\n",
        "# Preparing DataBlock.\n",
        "dls_lm = DataBlock(\n",
        "    blocks = TextBlock.from_folder(path, is_lm=True),\n",
        "    get_items=get_imdb, splitter=RandomSplitter(0.1)\n",
        ").dataloaders(path, path=path, bs=129, seq_len=80)\n",
        "\n",
        "# Inspecting the DataBlock.\n",
        "dls_lm.show_batch(max_n=2)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>text_</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>xxbos xxmaj how hard is it to write a watchable film with xxmaj vince xxmaj vaughn , xxmaj paul xxmaj giamatti and xxmaj kevin xxmaj spacey ? xxmaj apparently xxup very difficult for the writers here . \\n\\n i still have no idea how xxmaj santa is younger and looks 20 years older than xxmaj vince ( who plays the xxup big brother ) . i must have missed that part of the story but in reality , it really</td>\n",
              "      <td>xxmaj how hard is it to write a watchable film with xxmaj vince xxmaj vaughn , xxmaj paul xxmaj giamatti and xxmaj kevin xxmaj spacey ? xxmaj apparently xxup very difficult for the writers here . \\n\\n i still have no idea how xxmaj santa is younger and looks 20 years older than xxmaj vince ( who plays the xxup big brother ) . i must have missed that part of the story but in reality , it really did</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>is xxmaj shelley xxmaj duvall , her scene of finding xxmaj jack 's rant xxmaj all xxmaj work ▁ is incredible , that 's a look of horror and you can see that fear in her face after realizing her husband is mad . xxmaj also another incredible scene is when xxmaj jack sees a ghost woman in the bathtub , it 's honestly one of the most terrifying scenes in horror cinema . xxmaj the reason this film is</td>\n",
              "      <td>xxmaj shelley xxmaj duvall , her scene of finding xxmaj jack 's rant xxmaj all xxmaj work ▁ is incredible , that 's a look of horror and you can see that fear in her face after realizing her husband is mad . xxmaj also another incredible scene is when xxmaj jack sees a ghost woman in the bathtub , it 's honestly one of the most terrifying scenes in horror cinema . xxmaj the reason this film is so</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P9OWcU-EPSu3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "c17af7eb-fe1b-42e3-aa62-ed3f5f208adc"
      },
      "source": [
        "# Preparing the Language Model.\n",
        "learn = language_model_learner(\n",
        "    dls_lm, AWD_LSTM, drop_mult=0.3,\n",
        "    metrics=[accuracy, Perplexity()]\n",
        ").to_fp16()\n",
        "\n",
        "# Training the Model\n",
        "learn.fit_one_cycle(1, 2e-2)                                  # Training the Model for one Epoch."
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>perplexity</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>4.128379</td>\n",
              "      <td>3.920033</td>\n",
              "      <td>0.299411</td>\n",
              "      <td>50.402081</td>\n",
              "      <td>21:41</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gsnbdG-NR_ag",
        "colab_type": "text"
      },
      "source": [
        "* The Perplexity metric used here is often used in Natural Language Processing for Language Models. It is the exponential of the loss function cross entropy. I have also included accuracy as the metric for the Model Evaluation in predicting the next word. Here, the loss function is cross entropy loss."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gbrsj_pEY8xt",
        "colab_type": "text"
      },
      "source": [
        "**Saving and Loading Models**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OnIMnL7hR-Yj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "883e0c46-3e66-46e1-8b0f-288ef6145efa"
      },
      "source": [
        "# Saving the Model trained above.\n",
        "learn.save(\"firstmodel\")\n",
        "\n",
        "# Loading the Model save by the lines of code defined above.\n",
        "learn.load(\"firstmodel\")"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<fastai.text.learner.LMLearner at 0x7fd353fb26d8>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oJO7qSq_Zmdz",
        "colab_type": "text"
      },
      "source": [
        "**Preparing the Model**\n",
        "* Tuning the Final Model after unfreezing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jh7WLljZR1k3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "outputId": "0cbab9cd-0618-48ab-cdc8-618f454e5543"
      },
      "source": [
        "# Preparing the Final Model.\n",
        "learn.unfreeze()                                                    # Unfreezing the Model.\n",
        "learn.fit_one_cycle(6, 2e-3)                                        # Training the Model for 6 Epochs."
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>perplexity</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>3.866808</td>\n",
              "      <td>3.774143</td>\n",
              "      <td>0.317484</td>\n",
              "      <td>43.560181</td>\n",
              "      <td>25:10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>3.783684</td>\n",
              "      <td>3.693521</td>\n",
              "      <td>0.327096</td>\n",
              "      <td>40.186096</td>\n",
              "      <td>25:07</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>3.705884</td>\n",
              "      <td>3.639193</td>\n",
              "      <td>0.333255</td>\n",
              "      <td>38.061111</td>\n",
              "      <td>25:08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>3.620303</td>\n",
              "      <td>3.601449</td>\n",
              "      <td>0.337785</td>\n",
              "      <td>36.651314</td>\n",
              "      <td>25:00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>3.516824</td>\n",
              "      <td>3.582811</td>\n",
              "      <td>0.340341</td>\n",
              "      <td>35.974510</td>\n",
              "      <td>25:16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>3.499033</td>\n",
              "      <td>3.582374</td>\n",
              "      <td>0.340723</td>\n",
              "      <td>35.958801</td>\n",
              "      <td>25:24</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1cSxGQ6Pofu1",
        "colab_type": "text"
      },
      "source": [
        "* Now, I will save the Model except the final layer that converts activations to probabilities of picking each token in vocabulary. The Model which doesnot include final layer is called Encoder. I will save with save encoder. The Model obtained above is Fine Tuned."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5aD0aXX7a7Zu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Saving the Final Model.\n",
        "learn.save_encoder(\"finetuned\")"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x4zzygVOqEnn",
        "colab_type": "text"
      },
      "source": [
        "**Text Generation**\n",
        "* Before moving to fine tuning the Classifier, I will use the Model to generate the random reviews. Since, it is trained to guess the next word of the sentence, I can use the Model to write the new reviews."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t8tquEbqvYDa",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        },
        "outputId": "7a62f962-91d3-4315-b85a-5de8bf432bc0"
      },
      "source": [
        "# Text Generation with Final Model.\n",
        "TEXT = \"I am bored with the movie because\"                                          # Example of Negative sentiment reviews.\n",
        "N_words = 50                                                                        # Number of words in each sentences.\n",
        "N_sents = 3                                                                         # Number of sentences.\n",
        "\n",
        "# Making predictions of the Next word:\n",
        "preds = [learn.predict(TEXT, N_words, temperature=0.75)\n",
        "         for _ in range(N_sents)]\n",
        "\n",
        "# Inspecting the result.\n",
        "print(\"\\n\".join(preds))"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "i am bored with the movie because I 'm a movie goer and i have never seen a movie like this as I 've ever seen . However , i was delighted with the quality of the photography , especially the sound and the overall quality of the movie . The movie was\n",
            "i am bored with the movie because you will be sour at times . The plot is very thin . The acting is also very bad . The story is not that bad . It is based on true events and is not worth the time to spend . The movie was\n",
            "i am bored with the movie because I 'm a Executive Producer . Even with performances of Brad Pitt and Natasha Henstridge , this movie lacks all the dramatic , suspense and emotional power . The movie is basically about Karen ( karen Sillas ) ,\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xy-HQsWLy8JQ",
        "colab_type": "text"
      },
      "source": [
        "**Creating the Classifier Data Loaders**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IiVabUCpzOIX",
        "colab_type": "text"
      },
      "source": [
        "* The Language Model prepared earlier predicts the next word of the Document so it doesn't need any external labels. However, the Classifier predicts external label. In the case of IMDB, it's the sentiment of the Document."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5freGGdnzErE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 380
        },
        "outputId": "f67f17ee-3467-44d5-f22d-c784d94a94a7"
      },
      "source": [
        "# Preparing the TextBlock and DataBlock of the Classifiers.\n",
        "dls_clas = DataBlock(\n",
        "    blocks = (TextBlock.from_folder(path, vocab=dls_lm.vocab), CategoryBlock),\n",
        "    get_y = parent_label, \n",
        "    get_items = partial(get_text_files, folders=[\"train\", \"test\"]),\n",
        "    splitter = GrandparentSplitter(valid_name=\"test\")\n",
        ").dataloaders(path, path=path, bs=128, seq_len=72)\n",
        "\n",
        "# Inspecting the DataBlock.\n",
        "dls_clas.show_batch(max_n=3)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules of the match , both opponents have to go through tables in order to get the win . xxmaj benoit and xxmaj guerrero heated up early on by taking turns hammering first xxmaj spike and then xxmaj bubba xxmaj ray . a xxmaj german xxunk by xxmaj benoit to xxmaj bubba took the wind out of the xxmaj dudley brother . xxmaj spike tried to help his brother , but the referee restrained him while xxmaj benoit and xxmaj guerrero</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad xxpad</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zqiXC_2t9f2F",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "45eb9738-ecee-458e-85c2-50eff661e5a3"
      },
      "source": [
        "# Creating the Model to classify Texts.\n",
        "learn = text_classifier_learner(dls_clas, AWD_LSTM, drop_mult=0.5,\n",
        "                                metrics=accuracy).to_fp16()\n",
        "\n",
        "# Loading the Encoder.\n",
        "learn.load_encoder(\"finetuned\")"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<fastai.text.learner.TextLearner at 0x7fd08153c400>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v2wQNFqgWO5p",
        "colab_type": "text"
      },
      "source": [
        "**Fine Tuning the Classifier**\n",
        "* The last step is to train with Discriminative learning rates and gradually unfreezing. In Natural Language Processing, unfreezing a few layers at a time makes a real difference."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lvowtae8XANO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "5ba47ea2-9789-4293-a4b8-8ff47eb878c7"
      },
      "source": [
        "# Training only one Epoch.\n",
        "learn.fit_one_cycle(1, 2e-2)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.350891</td>\n",
              "      <td>0.192791</td>\n",
              "      <td>0.924720</td>\n",
              "      <td>01:09</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FtYYK0ZHXVdX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "997a0053-3645-4f52-b1b4-2ebcaf3379bc"
      },
      "source": [
        "# Training only one epoch and unfreezing a bit more. \n",
        "learn.freeze_to(-2)                                             # Unfreezing a bit more.\n",
        "learn.fit_one_cycle(1, slice(1e-2/(2.6**4), 1e-2))              # Training one epoch."
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.259890</td>\n",
              "      <td>0.172957</td>\n",
              "      <td>0.933640</td>\n",
              "      <td>01:15</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k3R8-JDpY5Vd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "5bbaa296-2a52-49e9-ae73-2359dc43da9a"
      },
      "source": [
        "# Training the Model after unfreezing a bit more.\n",
        "learn.freeze_to(-3)                                             # Unfreezing a bit more.\n",
        "learn.fit_one_cycle(1, slice(5e-3/(2.6**4), 5e-3))              # Training the Model."
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.210548</td>\n",
              "      <td>0.159870</td>\n",
              "      <td>0.939880</td>\n",
              "      <td>01:34</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P41d1tqDZWg9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        },
        "outputId": "243ca9ef-1543-48c9-9c28-be6329e3575f"
      },
      "source": [
        "# Training the Model after unfreezing the whole Model.\n",
        "learn.unfreeze()                                                # Unfreezing the Model.\n",
        "learn.fit_one_cycle(3, slice(1e-3/(2.6**4), 1e-3))              # Training five Epochs.                                  "
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.182021</td>\n",
              "      <td>0.153751</td>\n",
              "      <td>0.942840</td>\n",
              "      <td>01:54</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.164785</td>\n",
              "      <td>0.154277</td>\n",
              "      <td>0.943320</td>\n",
              "      <td>01:55</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.150096</td>\n",
              "      <td>0.156386</td>\n",
              "      <td>0.944280</td>\n",
              "      <td>01:55</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}