Norm_Stabilizer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Norm-Stabilizer.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "GNy7WHZ3WiAQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from fastai.text import * \n",
        "from fastai import * "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KZnoWhVWZOYT",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "**[1][Regularizing RNNs by Stabilizing Activations](https://arxiv.org/abs/1511.08400)**\n",
        " - by David Krueger, Roland Memisevic\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Y4L-39aVr8M",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "9eedd554-e5e0-4898-fe18-1ca1169ceaac"
      },
      "source": [
        "\n",
        "bs=48\n",
        "path = untar_data(URLs.IMDB)\n",
        "path.ls()\n",
        "data_lm = (TextList.from_folder(path)\n",
        "           #Inputs: all the text files in path\n",
        "            .filter_by_folder(include=['train', 'test', 'unsup']) \n",
        "           #We may have other temp folders that contain text files so we only keep what's in train and test\n",
        "            .split_by_rand_pct(0.1)\n",
        "           #We randomly split and keep 10% (10,000 reviews) for validation\n",
        "            .label_for_lm()           \n",
        "           #We want to do a language model so we label accordingly\n",
        "            .databunch(bs=bs))\n",
        "data_lm.save('data_lm.pkl')\n",
        "\n",
        "data_lm = load_data(path, 'data_lm.pkl', bs=bs)\n",
        "data_lm.show_batch()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>idx</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>takes the original point of it away and just renders it nearly unwatchable . ( one joke about xxmaj garth 's eyes xxunk the audience was good ... the rest i can hardly remember ... ) it feels like a poor quality \" xxmaj knowing xxmaj me , xxmaj knowing xxmaj you . \" i hope this improves as the series goes on , otherwise i shall be seriously disappointed</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>just started walking out during the course of the screening . \\n \\n  xxmaj there 's no way to actually summarize the story into something coherent , so i wo nt even try . xxmaj every cliché ever conceived is in there , and in all the wrong places . i 'm sorry to say this is one of the worst movies i have ever seen in my entire</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>\\n \\n  i must say that , when i first saw this thing on cable , i could n't stop laughing . xxmaj no xxup scifi film had prepared me for this ; it was so incredibly pathetic i could n't believe it ! xxmaj the actors are all a bunch of stereotypical - genre characters . xxmaj but since they 're not so famous ( to say the</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>the movie . xxmaj in the film , the characters and set were just some of the let downs that occurred in the movie . xxmaj the director left out so many important and interesting aspects of the book that made it one of the best literary works ever made . \\n \\n  xxmaj of course any book is better than the movie but these were n't even comparable</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>the xxmaj berlin xxmaj film xxmaj festival and had medium - high expectations . - xxmaj the director is xxmaj bertrand xxmaj tavernier and i was familiar with some of his previous work in xxmaj french cinema . i actually enjoyed some of his earlier movies . -the cast : xxmaj tommy xxmaj lee xxmaj jones , xxmaj john xxmaj goodman , xxmaj peter xxmaj xxunk , xxmaj mary xxmaj</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7obInUTIWspL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)\n",
        "# learn.lr_find()\n",
        "# learn.recorder.plot(skip_end=15)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ws0aGAbKW3HF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn.save_encoder('fine_tuned_enc')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o8xMj8o9XDo1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "outputId": "b4978a9b-9d6c-42f3-94ed-1c46062c2f54"
      },
      "source": [
        "path = untar_data(URLs.IMDB)\n",
        "data_clas = (TextList.from_folder(path, vocab=data_lm.vocab)\n",
        "             #grab all the text files in path\n",
        "             .split_by_folder(valid='test')\n",
        "             #split by train and valid folder (that only keeps 'train' and 'test' so no need to filter)\n",
        "             .label_from_folder(classes=['neg', 'pos'])\n",
        "             #label them all with their folders\n",
        "             .databunch(bs=bs))\n",
        "\n",
        "data_clas.save('data_clas.pkl')\n",
        "data_clas = load_data(path, 'data_clas.pkl', bs=bs)\n",
        "data_clas.show_batch()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj by 1987 xxmaj hong xxmaj kong had given the world such films as xxmaj sammo xxmaj hung 's ` xxmaj encounters of the xxmaj spooky xxmaj kind ' xxmaj chow xxmaj yun xxmaj fat in xxmaj john xxmaj woo 's iconic ` a xxmaj better xxmaj tomorrow ' , ` xxmaj zu xxmaj warriors ' and the classic ` xxmaj mr xxmaj vampire ' . xxmaj jackie xxmaj</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj god ! xxmaj zorro has been the the subject of about as many movies as xxmaj tarzan , and probably had about as many actors in the title role . \\n \\n  xxmaj this xxmaj serial is one of my own personal favourites , and as previously stated , it is one of the xxmaj top 5 xxmaj sound xxmaj serials . xxmaj oddly enough , this</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos \" a xxmaj damsel in xxmaj distress \" is definitely not one of xxmaj fred xxmaj astaire 's better musicals . xxmaj but even xxmaj astaire 's bad films always had some good moments . \\n \\n  xxmaj in \" xxmaj damsel , \" xxmaj astaire is xxmaj jerry xxmaj halliday , an xxmaj american musical star who is in xxmaj london on a personal appearance tour .</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos xxup warning : xxup possible xxup spoilers ( but not really - keep reading ) . xxmaj ahhh , there are so many reasons to become utterly addicted to this spoof gem that i wo n't have room to list them all . xxmaj the opening credits set the playful scene with kitsch late 1950s cartoon stills ; an enchanting xxmaj xxunk ' xxmaj prez ' xxmaj prado mambo</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pkH0vcPwcSxk",
        "colab_type": "text"
      },
      "source": [
        "First using fastai's basic methodology we've created a language model fine tuned on the Wikipedia text. After that we've established a baseline using the fastai's fit_one_cycle policy. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mZ7dISKjXM2E",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 197
        },
        "outputId": "8b0e411d-c957-47e2-f1a7-1d046c3ec228"
      },
      "source": [
        "learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)\n",
        "learn.load_encoder('fine_tuned_enc')\n",
        "# learn.lr_find()\n",
        "# learn.recorder.plot()\n",
        "learn.fit_one_cycle(5, 2e-2, moms=(0.8,0.7))"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.471733</td>\n",
              "      <td>0.457118</td>\n",
              "      <td>0.777040</td>\n",
              "      <td>10:06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.467318</td>\n",
              "      <td>0.395980</td>\n",
              "      <td>0.822320</td>\n",
              "      <td>10:52</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.451140</td>\n",
              "      <td>0.377374</td>\n",
              "      <td>0.832560</td>\n",
              "      <td>09:49</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.449924</td>\n",
              "      <td>0.371972</td>\n",
              "      <td>0.836440</td>\n",
              "      <td>10:39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.438636</td>\n",
              "      <td>0.360387</td>\n",
              "      <td>0.843320</td>\n",
              "      <td>11:00</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YuA-ERRodyJK",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "#### **BASELINE** ####\n",
        "The baseline I have established does not include fine tuning the language model on the IMDB data. I simply loaded the encoder without fine tuning the language model at all. \n",
        "\n",
        "\n",
        "#### **fastai's existing fine tuning techniques** ####\n",
        "fastai already uses a couple of techniques for stabilizing RNNs. \n",
        "Those are defined [here](https://github.com/fastai/fastai/blob/master/fastai/callbacks/rnn.py). \n",
        "\n",
        "fastai documentation for the fine tuning techniques for `fastai.text` can  be found [here](https://docs.fast.ai/callbacks.rnn.html): \n",
        "\n",
        "These techniques are adopted from this paper:\n",
        "\n",
        "**[2][Regularizing and Optimizing LSTM Language Models](https://arxiv.org/abs/1708.02182)**\n",
        " - Stephen Merity, Nitish Shirish Keskar, Richard Socher\n",
        "\n",
        "**[2]** Introduces a technique called `Temporal Activation Regularization (TAR)`\n",
        "\n",
        " The TAR technique is essentially adding the following term to the loss (thus it exists in the callback `backward_begin`): \n",
        "\n",
        " `extra_loss_regularization_term = β L2(ht − ht+1)`\n",
        "\n",
        " Where `ht` is the hidden state at timestep `t`\n",
        "\n",
        "For those who are confused by the fancy terms (as I was) here is the definition if L2: https://en.wikipedia.org/wiki/Norm_(mathematics)#p-norm\n",
        "\n",
        " [2] also states:\n",
        " \n",
        "`As in Merity et al. (2017),\n",
        "the AR and TAR loss are only applied to the output of the\n",
        "final RNN layer as opposed to being applied to all layers.`\n",
        "\n",
        "The Norm Stabilizer paper introduces a technique similar to TAR from the paper above. So it implement this paper we will have to override the existing callback. \n",
        "\n",
        "Below is the callback implemented by fastai for reference. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hmIvYNIOdxj-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class RNNTrainer(LearnerCallback):\n",
        "    \"`Callback` that regroups lr adjustment to seq_len, AR and TAR.\"\n",
        "    def __init__(self, learn:Learner, alpha:float=0., beta:float=0.):\n",
        "        super().__init__(learn)\n",
        "        self.not_min += ['raw_out', 'out']\n",
        "        self.alpha,self.beta = alpha,beta\n",
        "        \n",
        "    def on_epoch_begin(self, **kwargs):\n",
        "        \"Reset the hidden state of the model.\"\n",
        "        self.learn.model.reset()\n",
        "\n",
        "    def on_loss_begin(self, last_output:Tuple[Tensor,Tensor,Tensor], **kwargs):\n",
        "        \"Save the extra outputs for later and only returns the true output.\"\n",
        "        self.raw_out,self.out = last_output[1],last_output[2]\n",
        "        return {'last_output': last_output[0]}\n",
        "\n",
        "    def on_backward_begin(self, last_loss:Rank0Tensor, last_input:Tensor, **kwargs):\n",
        "        \"Apply AR and TAR to `last_loss`.\"\n",
        "        \n",
        "        #AR \n",
        "        if self.alpha != 0.:  last_loss += self.alpha * self.out[-1].float().pow(2).mean()\n",
        "\n",
        "\n",
        "        # TAR\n",
        "        if self.beta != 0.:\n",
        "            h = self.raw_out[-1]\n",
        "            if len(h)>1: last_loss += self.beta * (h[:,1:] - h[:,:-1]).float().pow(2).mean() #L2\n",
        "\n",
        "\n",
        "        return {'last_loss': last_loss}\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3iIotWW7iiT6",
        "colab_type": "text"
      },
      "source": [
        "####**Modifications**####\n",
        "\n",
        "We are going to have to replace the existing callback with a callback which also adds the norm-stabilizer term. \n",
        "\n",
        "####**Norm Stabilizer**####\n",
        "The term reccomended by [1] is as follows: \n",
        "`(β/T)* SIGMA[Across T](L2(ht) - L2(ht-1))^2`\n",
        "\n",
        "I have modified the RNNTrainer to add the norm-stabilizer below. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0IXEHsIGcPDJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def apply_ar(alpha,out): return alpha * out[-1].float().pow(2).mean()\n",
        "\n",
        "def apply_tar(beta,h): return beta * (h[:,1:] - h[:,:-1]).float().pow(2).mean()\n",
        "\n",
        "def apply_normstable(beta,h): return beta/h.shape[1] *  (h[:,1:].pow(2).sum().sqrt() - h[:,:-1].pow(2).sum().sqrt()).float().pow(2)\n",
        "\n",
        "class RNNTrainerNorm(LearnerCallback):\n",
        "    def __init__(self, learn:Learner, alpha:float=2., beta:float=1., beta_norm:float=50.):\n",
        "        super().__init__(learn)\n",
        "        self.not_min += ['raw_out', 'out']\n",
        "        self.alpha,self.beta,self.beta_norm = alpha,beta,beta_norm\n",
        "        \n",
        "    def on_epoch_begin(self, **kwargs):\n",
        "        \"Reset the hidden state of the model.\"\n",
        "        self.learn.model.reset()\n",
        "\n",
        "    def on_loss_begin(self, last_output:Tuple[Tensor,Tensor,Tensor], **kwargs):\n",
        "        \"Save the extra outputs for later and only returns the true output.\"\n",
        "        self.raw_out,self.out = last_output[1],last_output[2]\n",
        "        return {'last_output': last_output[0]}\n",
        "\n",
        "    def on_backward_begin(self, last_loss:Rank0Tensor, last_input:Tensor, **kwargs):\n",
        "        \"Apply AR and TAR to `last_loss`.\"\n",
        "        #AR and TAR\n",
        "        if self.alpha != 0.:  last_loss += apply_ar(self.alpha,self.out)\n",
        "        if self.beta != 0.:\n",
        "            h = self.raw_out[-1]\n",
        "            if len(h)>1: last_loss += apply_tar(self.beta,h)\n",
        "\n",
        "        if self.beta_norm != 0.:\n",
        "            h = self.raw_out[-1]\n",
        "            if len(h)>1: last_loss += apply_normstable(self.beta_norm,h)        \n",
        "        \n",
        "        return {'last_loss': last_loss}\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1lTxtXtjFScW",
        "colab_type": "text"
      },
      "source": [
        "Here are the observations of a small experiment I did. \n",
        "[1] and [2] both provide terms which are supposed to stabilize the LSTMs in the `learn.model`. In Jeremy's lectures he points out that when using the `fit_one_cycle` training policy we only train the last layer. \n",
        "\n",
        "`learn.model` is made of a `MultiBatchEncoder` with an `AWD_LSTM` inside it and a `PoolingLinearClassifier`. \n",
        "\n",
        "I have shown below that the `PoolingLinearClassifier` layer has `requires_grad` set to `True`. So we have a term which is supposed to be affecting the RNNs but since `requires_grad` is set to `False` that can't happen. We should be setting `freeze_to(-2)`\n",
        "\n",
        "This unfreezes only the last `LSTM` (for which we have added the regularization term) and should give us better results. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tQRtI5hBGBUz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 692
        },
        "outputId": "01bbb631-b49b-4cdc-9edb-84f3b936a740"
      },
      "source": [
        "learn.model"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SequentialRNN(\n",
              "  (0): MultiBatchEncoder(\n",
              "    (module): AWD_LSTM(\n",
              "      (encoder): Embedding(60000, 400, padding_idx=1)\n",
              "      (encoder_dp): EmbeddingDropout(\n",
              "        (emb): Embedding(60000, 400, padding_idx=1)\n",
              "      )\n",
              "      (rnns): ModuleList(\n",
              "        (0): WeightDropout(\n",
              "          (module): LSTM(400, 1152, batch_first=True)\n",
              "        )\n",
              "        (1): WeightDropout(\n",
              "          (module): LSTM(1152, 1152, batch_first=True)\n",
              "        )\n",
              "        (2): WeightDropout(\n",
              "          (module): LSTM(1152, 400, batch_first=True)\n",
              "        )\n",
              "      )\n",
              "      (input_dp): RNNDropout()\n",
              "      (hidden_dps): ModuleList(\n",
              "        (0): RNNDropout()\n",
              "        (1): RNNDropout()\n",
              "        (2): RNNDropout()\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (1): PoolingLinearClassifier(\n",
              "    (layers): Sequential(\n",
              "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (1): Dropout(p=0.2, inplace=False)\n",
              "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
              "      (3): ReLU(inplace=True)\n",
              "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (5): Dropout(p=0.1, inplace=False)\n",
              "      (6): Linear(in_features=50, out_features=2, bias=True)\n",
              "    )\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FnPZK-4dq7hZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def has_params(m:nn.Module)->bool:\n",
        "    \"Check if `m` has at least one parameter\"\n",
        "    return len(list(m.parameters())) > 0\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cOKNPqLUlxLv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "modules = [m for m in flatten_model(learn.model) if has_params(m)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p08raE6Ulyk-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 230
        },
        "outputId": "b1e83e15-c2ab-433d-a09f-a39b4cd89d15"
      },
      "source": [
        "modules"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Embedding(60000, 400, padding_idx=1),\n",
              " Embedding(60000, 400, padding_idx=1),\n",
              " LSTM(400, 1152, batch_first=True),\n",
              " ParameterModule(),\n",
              " LSTM(1152, 1152, batch_first=True),\n",
              " ParameterModule(),\n",
              " LSTM(1152, 400, batch_first=True),\n",
              " ParameterModule(),\n",
              " BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),\n",
              " Linear(in_features=1200, out_features=50, bias=True),\n",
              " BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),\n",
              " Linear(in_features=50, out_features=2, bias=True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BOpnhoozrrVJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def requires_grad_bool(m:nn.Module)->Optional[bool]:\n",
        "    ps = list(m.parameters())\n",
        "    return ps[0].requires_grad\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U-tqrb0OrE0V",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 230
        },
        "outputId": "0b6cd267-26cb-4644-f095-e2d7849381e3"
      },
      "source": [
        "for it in modules:\n",
        "  print(requires_grad_bool(it),it)"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "False Embedding(60000, 400, padding_idx=1)\n",
            "False Embedding(60000, 400, padding_idx=1)\n",
            "False LSTM(400, 1152, batch_first=True)\n",
            "False ParameterModule()\n",
            "False LSTM(1152, 1152, batch_first=True)\n",
            "False ParameterModule()\n",
            "False LSTM(1152, 400, batch_first=True)\n",
            "False ParameterModule()\n",
            "True BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "True Linear(in_features=1200, out_features=50, bias=True)\n",
            "True BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "True Linear(in_features=50, out_features=2, bias=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GezBt-PhsLdR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn.freeze_to(-2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ujwl9Y5BsNCP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "modules = [m for m in flatten_model(learn.model) if has_params(m)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VUJYuTv6wELd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 230
        },
        "outputId": "842ff7dd-1a9a-4db0-8e71-a8a3522c53be"
      },
      "source": [
        "for it in modules:\n",
        "  print(requires_grad_bool(it),it)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "False Embedding(60000, 400, padding_idx=1)\n",
            "False Embedding(60000, 400, padding_idx=1)\n",
            "False LSTM(400, 1152, batch_first=True)\n",
            "False ParameterModule()\n",
            "False LSTM(1152, 1152, batch_first=True)\n",
            "False ParameterModule()\n",
            "True LSTM(1152, 400, batch_first=True)\n",
            "True ParameterModule()\n",
            "True BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "True Linear(in_features=1200, out_features=50, bias=True)\n",
            "True BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "True Linear(in_features=50, out_features=2, bias=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I2_JzEmjwMUt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "outputId": "48f63cd1-1023-4bb0-a4c9-0caed6c7d56d"
      },
      "source": [
        "learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)\n",
        "learn.load_encoder('fine_tuned_enc')\n",
        "learn.freeze_to(-2)\n",
        "learn.lr_find()\n",
        "learn.recorder.plot()"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xUVfr48c+TTkJNoUgIhE4EaQEF\nGzYEC1YU1LWXr4q7uuqurvtzLeuuddeylrV3EcGCigKrYgUl1NBLaAkQQhJKEkh9fn/MRYcwCQnk\nzmQmz/v1mhdzzz333mdCkif3nHPPEVXFGGOMqS4s0AEYY4xpnCxBGGOM8ckShDHGGJ8sQRhjjPHJ\nEoQxxhifIgIdQENJTEzULl26BDoMY4wJKvPmzduuqkm+9oVMgujSpQsZGRmBDsMYY4KKiGyoaZ81\nMRljjPHJEoQxxhifLEEYY4zxyRKEMcYYnyxBGGOM8ckShDHGGJ8sQRhjjPHJ1QQhIqNEZKWIrBGR\nu3zsTxGRb0RkgYgsFpEznPLTRGSeiGQ6/57sZpyhYsbSrWzesSfQYRhjQoRrCUJEwoFngdFAGjBe\nRNKqVfsrMElVBwLjgOec8u3A2araD7gCeMutOEPFyq27uf6teVz9+lz2llcGOhxjTAhw8w5iKLBG\nVbNUtQyYCJxTrY4CLZ33rYDNAKq6QFU3O+VLgWYiEu1irEHvnZ83EBEmrNi6m8enrzzs8+0tr6S4\ntKIBIjPGBCs3p9roCGzy2s4Gjq5W5z5ghojcAsQBp/o4zwXAfFUtrb5DRK4HrgdISUlpgJCDU3Fp\nBR/Oz+Hs/kfQPDqCl39Yx0m923Js98RDOl95ZRUXvziHJTk7OSq5FcO7JTC8WyJDU+OJDLduK2Oa\nikD/tI8HXlfVZOAM4C0R+TUmETkSeAS4wdfBqvqiqqaranpSks+5ppqEjxfmUFRawWXHpPCXM/rQ\nNSmO2yctYmdJ+SGd77lv1rJo0w4uGNSRMBFe+DaLS1/+mT9NXtzAkRtjGjM3E0QO0MlrO9kp83YN\nMAlAVWcDMUAigIgkAx8Bl6vqWhfjDGqqyttzNtKnQ0sGpbShWVQ4T108kO1FpdzzcSb1XXN8Sc5O\nnvl6NecOOIJHL+zPlBuHs+hvI7lyeBc+WpBDZvZOlz6JMaaxcTNBzAV6iEiqiETh6YSeWq3ORuAU\nABHpgydB5IlIa+Bz4C5V/dHFGIPe/I07WL5lF5cdk4KIANAvuRW3ndaTzxZv4cn/ra5zkiitqOSP\nkxYSHxfF/WP6/lrePDqC20f2pE1sJI9OX+HK5zDGND6uJQhVrQAmANOB5XhGKy0VkQdEZIxT7Xbg\nOhFZBLwHXKme32YTgO7AvSKy0Hm1dSvWYPbOnA00j47g3AEd9yv/vxO7ccGgZJ76ajW3vb+wTiOb\n/j1zNatyi3jkgqNoFRu5374WMZHcfFJ3vl+9nR/XbG/Qz2CMaZykvk0QjVV6ero2tfUgCorLOOYf\nX3HxkE48eG7fA/arKs/NWstj01eS3rkN//3dYBKa+x4MNm9DIWNf+ImxgzvxyIVH+ayzt7ySU574\nloTmUXxy87G/3rEYY4KXiMxT1XRf+wLdSW3qIb+olK9X5DInK5/M7J288kMWZZVVXHZMZ5/1RYSb\nT+rOs5cMIjNnJ+c+9yMb8osPqLd5xx5ufHseR7Ruxl/P6lPj9WMiw7n11B4szt7JF0u2NtjnMsY0\nTiGzolyoyy8q5exnfmDzzr37lQ/tEk+v9i1qPfbMozpwROsYrn59Lhf/dw7vXnc0XZOaA1BUWsHV\nr8+lpKySt645mhYxkbWe6/xBybz0fRaPT1/JyLR2RNiwV2NCljUxBYGKyioue+Vn5m/cwdPjBtKy\nWQQlpZWUlFeS3rkNR7RuVqfzLN+yi0tf/pmIMOHd646hS0Is172ZwXert/PqlUM4sWfdhgrPXJbL\ndW9mMKJXEpce3ZkRvZLs+QhjglRtTUyWIILA3z9bxss/rOPxsf25cHDyYZ1rVe5uLnnpZwCGd0tg\n6qLN/OO8flxydN0fNFRVnvpqNW/P2cD2ojISm0dz7oAjGJDSmi4JcaQmxhEXbTenxgQDSxCNRHll\nFY9+uYKBKW0YdWR7wsIO3sn7ycIc/jBxIVcM68z95xzYEX0o1mwr4pKX5rBtdynXn9CVv5xRc79D\nbcorq5i1Mo8PMjbx9YptVFT99r2UmhjHS5cPpnvb2pu/jDGBZQmikfh6RS5Xv+6JsXf7Ftx2Wk9G\nprWrcTTQss27OP/5H+nXsRXvXndMgzbjbMwv4fs1eYwfklKnRHUwe8oqWZ9fzPrtxWRtL+bVH9bR\nKjaST24+9qD9GsaYwLFRTI3E54u30jImgifG9qe0ooob3prHmP/8SHZhyQF1d+8t56Z35tEyJpJn\nLx3U4G38KQmxXHp05wZJDgDNosLp06Elo/t14OaTuvOfSwaxIb+E2yctoqqq5j9CCorL+NsnS3jp\nu6wGicMY03AsQfhJWUUVM5dt5bS09lwwOJmZt53A42P7sz6/mCtfm8uOkrJf66oqd03JZFPhHv5z\nySDatogJYOSHZli3BO4e3ZsZy3J5/tsDZ0qpqlIm/rKRk5+YxRuzN/CPL5azaNOOAERqjKmJJQg/\n+WntdnbtreCMfu0BiAgP48LBybx0eTob80u4/s15vz7t/NacDXyeuYU7RvZiaGp8IMM+LNccl8qY\n/kfw+IyVzFi6lbV5Rfy0Zjsfzs/mwhd+4q4PM+nZtgVTbhxGUvNo/vJRJhWVVYEO2xjjsKEmfjIt\ncwstoiM4rsf+U3Af0zWBJy7qzy3vLeCPkxZy/QndePCzZZzcuy03nNA1QNE2DBHh4Qv6sSrXs5iR\nt4S4KB4f258LBnVERLj37DQmvLuAN2dv4OrjUgMUsTHGmyUIPyivrGLGslxOTWtHdET4AfvP7n8E\nubv28vfPl/O/5dto2yKGJ8b2b7D+gUCKjYrgzauH8nnmFtrERtG2ZTTtW8bQsU2z/b4WZ/brwAc9\ns3lixkrO6NeB9q2Cr1nNmFBjTUx+MCcrnx0l5Yzu277GOtccl8q1x6UiwDOXDKRNXJT/AnRZ25Yx\nXHVsKucO7Mjwbol0TWp+QKIUER4450gqqpQHPlsaoEiNMd4sQVRTVlHFJwtzDnld58079rC9aP/F\n76ZlbiEuKpwTanlSWUT461lpLLx3JINS2hzStYNd54Q4bjm5O9Myt/LPacv5ZGEO8zYUsHXnXipr\nGQlljHGHNTFV89L3WTw2fSV/OKUHt53Ws17HqiqXvfwzhSVlvHR5Ould4qmorGL60lxO6dOOmMgD\nm5eqaxZ18Dqh7LoTuvLd6u38t9qw14gwoV3LGDq0iqFty2gqKpW9FVXsLaskJiqcP4/qxZFHtApQ\n1MaEJksQXrYXlfL8rLWECbz8fRaXD+tc4/TYvqzN8zwkFh0RxiUv/8wTY/uTEBdFQXHZr6OXTO2i\nI8KZdMMwSsoq2LxjD5sK95BduIetO/ewZcdetuzcy8qtu4kMDyM6MpxmkWEs27yL8579iXvO7MPl\nwzrbNOTGNBBXE4SIjAKeAsKBl1X14Wr7U4A3gNZOnbtUdZqz7248S5JWAr9X1eluxgrw9Fer2VNe\nyfOXDebGt+fx7DdruffstDofP2vlNgCm3Dic+z9dyi3vLaBrYhyxUeGM6GXrHdVHbFQE3du2qNNU\nHQXFZdzxwSL+NnUpP67ZzqMXHkXr2NDpwzEmUFzrgxCRcOBZYDSQBowXkeq/bf+KZ6W5gXiWJH3O\nOTbN2T4SGAU855zPNWvzinj3542MH9qJ049sz4WDk3l7zgZyduyp8zm+XZVH97bN6duxFW9dczRj\n+h9B1vZiTurdtk7NS+bQxMdF8coV6fz1zD58s3Ib5z33E2UV9jyFMYfLzU7qocAaVc1S1TJgInBO\ntToKtHTetwI2O+/PASaqaqmqrgPWOOdzzSNfrCA6Iow/nOLpd/jDqT1B4MmZq+p0fElZBT9nFTDC\n6YiOiQznyYsH8OTFA7h7dG/X4jYeIsK1x3flqXEDWbe9mK9XbAt0SMYEPTcTREdgk9d2tlPm7T7g\nMhHJBqYBt9TjWETkehHJEJGMvLy8Qw70l3UFzFiWy40jupHUwtPn0LF1M353TGemzM9mzbbdBz3H\n7LX5lFVWcVLv35qSwsKEcwd2JLlN7CHHZupnZFo72raIZvK8TQevbIypVaCHuY4HXlfVZOAM4C0R\nqXNMqvqiqqaranpSUt0Wu/FxDh6atpz2LWO45rj9n1y+aUQ3YqMieHz6we8ivlm5jdiocNK7NM0h\nqo1FRHgY5w3qyDcr89i2e+/BDzDG1MjNBJEDdPLaTnbKvF0DTAJQ1dlADJBYx2MbxPr8EtZvL+b2\nkT0PGGKa0Dyaa49P5culW1m2eVeN51BVZq3MY3i3RJ9PShv/Gju4E5VVyscLXPmWMabJcDNBzAV6\niEiqiETh6XSeWq3ORuAUABHpgydB5Dn1xolItIikAj2AX9wIMjUxjm/vHMH5g3yv1HbV8FRio8J5\n5Yd1NZ5jbV4x2YV7GNHr0O5iTMPq3rY5A1Na80FGNqGy3okxgeBaglDVCmACMB1Yjme00lIReUBE\nxjjVbgeuE5FFwHvAleqxFM+dxTLgS+BmVT20R5vroHVsFOE1zHvUKjaSi9I7MXVRDrm7fDdZ7Bve\nagmi8Rg7uBOrtxWxKHtnoEMxJmi52gehqtNUtaeqdlPVh5yye1V1qvN+maoeq6r9VXWAqs7wOvYh\n57heqvqFm3EezNXHplJRpbzx03qf+/cNb7XO6MbjrP4diIkM44MM66w25lAFupM6KKQkxHJ6Wnve\n+XkjJWUV++0rLt1/eKtpHFrGRDLqyPZMXbT5kOfVMqapswRRR9edkMrOPeVMnpe9X7mv4a2mcRib\n3ondeyuYvnRroEMxJihZgqijwZ3jGZjSmld/WPfrzKLbdu/l1R/X2fDWRmpY1wQ6tm7G+3OtmcmY\nQ2EJoh6uPa4r6/NLmJa5hednreWkx2Yxd30Bt4/sZcNbG6GwMOHyYZ35aW0+s9fmBzocY4KOJYh6\nOP3IdiS3acYt7y3gkS9XMKxbIjNuO5FrbInMRuuK4V3o0CqGf36xnCpbU8KYerEEUQ8R4WH8aVRv\nhnRpw5tXD+XlK9JJTYwLdFimFjGR4dw+sheLs3fyeeaWQIdjTFCRUHmQKD09XTMyMgIdhmmEKquU\nM5/+nuKyCv73xxOtOdAYLyIyT1XTfe2zOwgT8sLDhL+c0YdNBXt4Z87GQIdjTNCwBGGahBN6JnF8\nj0Se/no1O/eUBzocY4KCJQjTZPx5VG927innP1+vDnQoxgQFSxCmyejbsRXjhnTi5R/W2bBXY+rA\nEoRpUv56ZhqpCXHc9v5CCovLAh2OMY2aJQjTpMRFR/D0+IHkF5fypymLbTpwY2phCcI0OX07tuLP\no3ozc1kub/9so5qMqYklCNMkXX1sKif2TOLBz5axYmvNqwUa05RZgjBNUliY8PjY/rSMieTGt+ez\ns8SGvhpTnasJQkRGichKEVkjInf52P9vEVnovFaJyA6vfY+KyFIRWS4iT4uI7yXfjDlESS2ief6y\nQWQXlnDLxAW/ztJrjPFwLUGISDjwLDAaSAPGi0iadx1Vvc1ZSW4A8AzwoXPscOBY4CigLzAEONGt\nWE3TNaRLPA+c05fvVuXxyJcrAh2OMY2Km3cQQ4E1qpqlqmXAROCcWuqPx7MuNYACMUAUEA1EArku\nxmqasPFDU7h8WGde/C6LjxZkH/wAY5oINxNER8B7pZZsp+wAItIZSAW+BlDV2cA3wBbnNV1Vl/s4\n7noRyRCRjLy8vAYO3zQl/++sNI5OjefPUzLJzN4Z6HCMaRQaSyf1OGCyqlYCiEh3oA+QjCepnCwi\nx1c/SFVfVNV0VU1PSrI1oc2hiwwP47lLB5EQF8XvJy44YO1xY5oiNxNEDtDJazvZKfNlHL81LwGc\nB8xR1SJVLQK+AIa5EqUxjoTm0fzrogGszy/mwc+WBTocYwLOzQQxF+ghIqkiEoUnCUytXklEegNt\ngNlexRuBE0UkQkQi8XRQH9DEZExDG9Ytgf87sRvv/bKJL5dsDXQ4xgSUawlCVSuACcB0PL/cJ6nq\nUhF5QETGeFUdB0zU/ec8mAysBTKBRcAiVf3UrViN8XbbqT3p27Eld324mNxdewMdjjEBYyvKGePD\n2rwiznz6e9I7x/Pm1UMJC7PHcExoshXljKmnbknN+X9npfHDmu3MWGZNTaZpsgRhTA0uTu9E2xbR\nTMqwZyNM02QJwpgaRISHccHgZGat3MY264swTZAlCGNqMXZwMlUKHy2oaYS2MaHLEoQxteia1JzB\nndvwwbxsW1zINDmWIIw5iLGDk1mzrYiFm3YcvLIxIcQShDEHceZRHYiJDOODedZZbZoWSxDGHESL\nmEjO6NuBTxdtZm95ZaDDMcZvLEEYUwcXpieze28F05faMxGm6bAEYUwdHJOaQHKbZnxgz0SYJsQS\nhDF1EBYmjB3ciR/WbOfBz5axa6+tYW1CnyUIY+rouhNSGT80hVd/XMfJj89i0txNVNk61iaEWYIw\npo5ioyL45/n9mHrzcaTEx/KnKYsZ99Ic67g2IcsShDH11C+5FVNuHM7D5/fjl3UFtriQCVkRgQ7A\nmGAkIowbmsK67cX897sshnVL4Kyjjgh0WMY0KFfvIERklIisFJE1InKXj/3/FpGFzmuViOzw2pci\nIjNEZLmILBORLm7GasyhuOP0XgxMac1dUzLZkF8c6HCMaVCuJQgRCQeeBUYDacB4EUnzrqOqt6nq\nAFUdADwDfOi1+03gMVXtAwwFtrkVqzGHKjI8jGfGDyRMYMK7CyitsP4IEzrcvIMYCqxR1SxVLQMm\nAufUUn888B6Ak0giVHUmgKoWqWqJi7Eac8iS28Ty2Nj+ZObs5J6PllintQkZbiaIjsAmr+1sp+wA\nItIZSAW+dop6AjtE5EMRWSAijzl3JNWPu15EMkQkIy8vr4HDN6buTj+yPRNO6s7kedmc9cwPzN9Y\nGOiQjDlsjWUU0zhgsqru+9MrAjgeuAMYAnQFrqx+kKq+qKrpqpqelJTkr1iN8emO03vx5tVD2VNW\nyYXP/8RDny9jT5ndTZjg5WaCyAE6eW0nO2W+jMNpXnJkAwud5qkK4GNgkCtRGtOATuiZxJe3Hs/4\noSm89P067pi8KNAhGXPI3BzmOhfoISKpeBLDOOCS6pVEpDfQBphd7djWIpKkqnnAyUCGi7Ea02Ba\nxETy0Hn9aB4dwUvfZ7Fl5x46tGoW6LCMqTfX7iCcv/wnANOB5cAkVV0qIg+IyBivquOAieq1XJfT\n1HQH8JWIZAICvORWrMa44bJjOqPAez9vDHQoxhwSCZVlFNPT0zUjw24yTONy9etzyczZyY9/Ppmo\niMbS5WfMb0Rknqqm+9pn37HGuOh3x3Qmb3cpM5bZOhIm+FiCMMZFJ/RMolN8M96esyHQoRhTb5Yg\njHFReJhw6dGdmZNVwOrc3YEOx5h6sQRhjMsuSu9EVESY3UWYoGMJwhiXxcdFcVa/DkyZn0NxaUWg\nwzGmzixBGOMHlx7TmaLSCj5eWNOzosY0PpYgjPGDQSmt6dWuBVPmZQc6FGPqzBKEMX4gIpw3qCPz\nN+5g/XZbN8IEB0sQxvjJOQOOQARrZjJBwxKEMX7SoVUzhnVN4KMFOYTKDAYmtNUpQYhINxGJdt6P\nEJHfi0hrd0MzJvScO7AjG/JLmL9xx8ErGxNgdb2DmAJUikh34EU803i/61pUxoSo0X3bEx0RxscL\nrJnJNH51TRBVzuys5wHPqOqdQAf3wjImNLWIieS0tHZ8ungzZRVVgQ7HmFrVNUGUi8h44ArgM6cs\n0p2QjAlt5w/qyI6Scr5dZcvkmsatrgniKmAY8JCqrnMWAXrLvbCMCV3H90giIS6KjxbYMxHm8C3Y\nWMgCl9ZAr1OCUNVlqvp7VX1PRNoALVT1EVciMibERYaHcXb/I/jf8m3s3FMe6HBMkPvXzFXc/+ky\nV85d11FMs0SkpYjEA/OBl0TkX3U4bpSIrBSRNSJyl4/9/xaRhc5rlYjsqLa/pYhki8h/6vqBjAkG\n5w7sSFlFFR/Nt7sIc3gKisuIj4ty5dx1bWJqpaq7gPOBN1X1aODU2g4QkXDgWWA0kAaMF5E07zqq\nepuqDlDVAcAzwIfVTvMg8F0dYzQmaPRPbsWQLm14/tu17C2vDHQ4JogVFpfRJjawCSJCRDoAF/Fb\nJ/XBDAXWqGqWqpYBE4Fzaqk/Hnhv34aIDAbaATPqeD1jgoaIcOupPcndVcr7czcFOhwTpFSV/OIy\nEpoHNkE8AEwH1qrqXBHpCqw+yDEdAe/v/Gyn7AAi0hlIBb52tsOAJ4A7aruAiFwvIhkikpGXZyNC\nTHAZ3i2BoV3ieW7WGruLMIdkT3klpRVVgb2DUNUPVPUoVb3R2c5S1QsaMI5xwGRV3fdTchMwTVVr\nbaBV1RdVNV1V05OSkhowHGPcJyLceloPcneVMvGXjYEOxwShguIyAOLj3HnqoK6d1Mki8pGIbHNe\nU0Qk+SCH5eB54nqfZKfMl3F4NS/hGVI7QUTWA48Dl4vIw3WJ1ZhgMqxrAkNT43lulvVFmPorLPaM\ngouPi3bl/HVtYnoNmAoc4bw+dcpqMxfoISKpIhKFJwlMrV5JRHoDbYDZ+8pU9VJVTVHVLniamd5U\n1QNGQRkT7ESE207tybbdpbxndxGmnvKLS4EA30EASar6mqpWOK/XgVrbdJypOSbg6btYDkxS1aUi\n8oCIjPGqOg6YqDa9pWmihnVL4JiunruIIluS1NRDYYmniSnQo5jyReQyEQl3XpcB+Qc7SFWnqWpP\nVe2mqg85Zfeq6lSvOvfVdnegqq+r6oQ6xmlMULpjZC/yi0q5+L+z2bpzb6DDMUGi4NcmpsAmiKvx\nDHHdCmwBLgSudCUiY5qg9C7xvHLlENZvL+acZ39gSc7OQIdkgkBhcRnhYULLmAA2ManqBlUdo6pJ\nqtpWVc8FGnIUkzFN3km92jL5xuGEizD2hdnMXJYb6JBMI5dfXEab2EjCwsSV8x/OinJ/bLAojDEA\n9OnQko8nHEvPds258e151txkauXmU9RweAnCnZRlTBPXtkUMT1zUn4oqZeZyu4swNSsocW8eJji8\nBGGjjoxxSbek5qQmxlkzk6mVmxP1AUTUtlNEduM7EQjQzJWIjDGICKelteO1H9exe285LVzqhDTB\nrbC4jDaBuoNQ1Raq2tLHq4Wq1ppcjDGH59Q+7SivVFt5zvhUVaUUlpQR30j7IIwxLhrcuQ3xcVH8\nz5qZjA+79pZTpe49AwGWIIxptMLDhJN7t+XrFdsor6wKdDimkcn/daI+SxDGNEmnpbVj194K5q4r\nCHQoppEpdBJEwPogjDGBdXyPRKIjwphhzUymml+n+rY+CGOaptioCI7rnsjMZbnYfJbG268JwqXV\n5MAShDGN3mlp7cjZsYflW3YDsGjTDi58/idueW9BgCMzgVRQ4v4dhA1VNaaRO6VPO0Qy+WDeJvbM\nruT9jE2Ei5CxoZDbTu1B16TmgQ7RBEBhcRkxkWE0iwp37Rp2B2FMI5fUIpqBnVrz2o/r+WBeNtcc\nm8rMP55IRJgwce6mg5/AhKSC4nISXFpJbh9XE4SIjBKRlSKyRkQOWPNBRP4tIgud1yoR2eGUDxCR\n2SKyVEQWi8jFbsZpTGN3w4ndOLNfB774w/H89aw0UhPjOC2tHZPnZVNaYUuVNkUFxaW0cWkluX1c\na2ISkXDgWeA0IBuYKyJTVXXZvjqqeptX/VuAgc5mCXC5qq4WkSOAeSIyXVV3uBWvMY3Z6Ue25/Qj\n2+9XNn5oCl8s2cr0pbmM6X9EgCIzgVJQUu7qTK7g7h3EUGCNqmapahkwETinlvrjgfcAVHWVqq52\n3m8GtnGQJU6NaWqO655Ip/hmvPvzhkCHYgKg0OWJ+sDdBNER8G4gzXbKDiAinYFU4Gsf+4YCUcBa\nF2I0JmiFhQnjhqQwJ6uArLyiQIdj/CzYE0R9jAMmq+p+jaki0gF4C7hKVQ+Ya0BErheRDBHJyMuz\nCc1M0zM2PZmIMOG9XzYGOhTjR6UVlewurXB1iCu4myBygE5e28lOmS/jcJqX9hGRlsDnwD2qOsfX\nQar6oqqmq2p6UpK1QJmmp22LGOusboJ2lJQD7k6zAe4miLlADxFJFZEoPElgavVKItIbaAPM9iqL\nAj4C3lTVyS7GaEzQGz80hcKScr5csjXQoRg/KfDDRH3gYoJQ1QpgAjAdWA5MUtWlIvKAiIzxqjoO\nmKj7zyNwEXACcKXXMNgBbsVqTDA7rnsiXRJi+dfMVezeWx7ocIwf+CtBuPoktapOA6ZVK7u32vZ9\nPo57G3jbzdiMCRVhYcJjY/tz8X9n89ePl/DkxQMQsSXjQ1nQ30EYY/xnSJd4bj21J58s3MzkedmB\nDse4rNCZhymYn4MwxvjRzSd155iu8dz7yVLWbLNhr6Fs3x1Em1h3n6S2BGFMiAgPE54aN5BmUeFM\neHc+e8ttVFOoKiguo1WzSCLC3f0VbgnCmBDSrmUMj489ihVbd/PKD+sCHY5xSYEfHpIDSxDGhJyT\ne7djSJc2TF24OdChGJcUlpS53rwEliCMCUln9uvAytzdrNm2O9ChGBfkF5UR7/JU32AJwpiQNLpf\nB0Tg88X28FwoKiwpI97lqb7BEoQxIaldyxiGdI7n80xrZgo1qkphcbnr02yAJQhjQtYZ/dqzKreI\n1bnWzBRKissqKausIsEShDHmUP3azJS5JdChmAZUUOSfh+TAEoQxIWtfM9M0SxAhpaDEP9NsgCUI\nY0LamUd1sGamEFO47ylqSxDGmMMxum97a2YKMfum2bA+CGPMYWnbMoYhXayZKZQU+PEOwtXpvo0x\ngXdmvw78bepSbnx7HkWlFeQXlRERLjwxtj892rUIdHimngpKyogIE1pEu//r2+4gjAlxZx7VgdTE\nOJZt2UVRaQUdWsWQU7iH697MYOceW2Ao2GzesYd2LWP8suaHqylIREYBTwHhwMuq+nC1/f8GTnI2\nY4G2qtra2XcF8Fdn399V9RINl5MAABZ3SURBVA03YzUmVCU2j+abO0bsV5axvoDxL83hDxMX8MoV\nQwgPswWGgkVWXjFdk+L8ci3X7iBEJBx4FhgNpAHjRSTNu46q3qaqA1R1APAM8KFzbDzwN+BoYCjw\nNxFp41asxjQ16V3iuW/MkcxamccTM1YGOhxTR6pKVl4R3ZKa++V6bjYxDQXWqGqWqpYBE4Fzaqk/\nHnjPeX86MFNVC1S1EJgJjHIxVmOanEuP7sz4oSk8N2stny+2TuxgkLurlOKySroF+x0E0BHY5LWd\n7ZQdQEQ6A6nA1/U5VkSuF5EMEcnIy8trkKCNaUruG5PG4M5tuHPyIjbv2BPocMxBrM3zrBQYCncQ\n9TEOmKyq9VoCS1VfVNV0VU1PSkpyKTRjQld0RDhPXjyAiirl8enW1NTY/Zog2gZ/gsgBOnltJztl\nvozjt+al+h5rjDkMneJjuea4VD5ckMPi7B2BDsfUIiuvmLiocNq2cH8tCHA3QcwFeohIqohE4UkC\nU6tXEpHeQBtgtlfxdGCkiLRxOqdHOmXGGBfcNKIbCXFR/P3z5ahqoMMxNVibV0S3ts39MsQVXEwQ\nqloBTMDzi305MElVl4rIAyIyxqvqOGCien1XqmoB8CCeJDMXeMApM8a4oEVMJH8c2ZNf1hUwfWlu\noMMxNVi7zX8jmMDl5yBUdRowrVrZvdW276vh2FeBV10Lzhizn4vTO/H6j+t5+IvlnNy7LVERjaWL\n0gCUlFWweedeuib6ZwQTNJ5OamNMgEWEh3HPmX1Yn1/CC9+uZddee8q6McnKKwb810ENNheTMcbL\niF5tObFnEv+auYp/zVxF69hIUuJjad8yhvi4KNrERZEQF8WpfdrRxY9/yRr/D3EFSxDGmGpeuGww\ns1ZuY1NhCRsLStiQ7/l34aYdFJaUUV6pPPXVal64bDDHdk8MdLhNRlZeMWECnRNi/XZNSxDGmP00\niwpndL8OPvepKhsLSrj+zXlc8eov/PP8foxN7+SzrmlYa/OKSG4TS0xkuN+uaQnCGFNnIkLnhDg+\nuHEYN709nzsnL2ZjQQnpXeKZv6GQ+RsLWbutiH+c348RvdoGOtyQkpVX7LcpNvaxBGGMqbeWMZG8\ndtUQ7vkok2e+XgNAmEDPdi2ICA/j9+8tYOqE46yfooFUVSlZ24sY3i3Br9e1BGGMOSSR4WE8csFR\njO7bgaiIMPp3ak3z6Ag2FZRw9n9+4Ia35vHhTcOJ88PCNqFu88497C2voqsfO6jBhrkaYw6DiHBS\n77Yc2z2R5k4i6BQfyzPjB7J6227+NGWxPZndAH4d4urnJiZLEMaYBnd8jyT+NKo3ny/ewovfZQU6\nnKDn70n69rEEYYxxxQ0ndOXMfh145MsVfJFp600cjrV5RbSMiSAhLsqv17UEYYxxhYjw2NijGNCp\nNb+fuIBvVm4LdEhBKyuv2K+T9O1jCcIY45rYqAheu2ooPdu14P/emsfstfmBDikorfXjMqPeLEEY\nY1zVqlkkb11zNCnxsVzzxlzmbywMdEhBZffecnJ3ldLVzx3UYAnCGOMH8XFRvHPt0SS1iOaq1+ay\nZactb1pX67bvG8FkdxDGmBDVtmUMb1w1lLKKKu6akmnDX+to6eZdQAgmCBEZJSIrRWSNiNxVQ52L\nRGSZiCwVkXe9yh91ypaLyNPi794ZY0yD65IYx91n9ObbVXm8P3dToMNp9LILS3j0yxX0ateCLn6c\npG8f1xKEiIQDzwKjgTRgvIikVavTA7gbOFZVjwRudcqHA8cCRwF9gSHAiW7Faozxn8uO7sywrgn8\n/fPlZBeWBDqcRmtveSU3vj2fikrlhd8NJiLc/w0+bl5xKLBGVbNUtQyYCJxTrc51wLOqWgigqvvG\nwSkQA0QB0UAkYOsgGhMCwsKERy88ClXlz/aktU+qyr2fLCEzZyf/ungAqQGa08rNBNER8L6HzHbK\nvPUEeorIjyIyR0RGAajqbOAbYIvzmq6qy12M1RjjR53iY/nLmX34cU0+b8/ZEOhwAm7Ntt2szt3N\nzpJyVJX3ftnEpIxsbjm5O6eltQtYXIGeRSsC6AGMAJKB70SkH5AI9HHKAGaKyPGq+r33wSJyPXA9\nQEpKir9iNsY0gEuGpvDlkq3cO3Upm3fu5bZTezbJdbBzduzhtH9/x74bqaiIMCoqqzihZxK3ntoz\noLG5mSByAO+VRJKdMm/ZwM+qWg6sE5FV/JYw5qhqEYCIfAEMA/ZLEKr6IvAiQHp6ut2nGhNERIQX\nLhvMA58u4/lZa/l+dR5PXjyQ7n6ebyjQlubsRBXuGNmTmMhw8naXosBNI7oRHhbYsTlupuu5QA8R\nSRWRKGAcMLVanY/xJANEJBFPk1MWsBE4UUQiRCQSTwe1NTEZE2LioiN45MKjeOGyweQU7uGsZ77n\n3Z83Nql+idXbPBPxXXlsKtce35W7z+jDX87oQ+tY/8675ItrCUJVK4AJwHQ8v9wnqepSEXlARMY4\n1aYD+SKyDE+fw52qmg9MBtYCmcAiYJGqfupWrMaYwBrVtz3Tbz2BIV3i+ctHmdw5eTF7yysDHZZf\nrMrdTcfWzX6dLr0xkVDJ1Onp6ZqRkRHoMIwxh6GySnnqf6t4+us19O3YkucvHUyneP+P//enUU9+\nR/tWMbx+1dCAXF9E5qlquq99Ta9HyBjTaIWHCX8c2YuXL09nQ75nZbofVm8PdFiuqaisIiuvmJ7t\nWgQ6FJ8sQRhjGp1T09oxdcJxtG0RzRWv/cI7P4fmUNgNBSWUVVZZgjDGmPpITYxjyo3DOb5HIvd8\ntIS/f7aMyqrQaBLfZ3XubgB6tmucI7csQRhjGq0WMZG8fHk6VwzrzMs/rOOGt+ZRXFoR6LAazKpc\nzwimxjq01xKEMaZRiwgP4/5z+nL/mCP5ekUuZz/zA0tydgY6rAaxKnc3neKbERvV+EYwgSUIY0yQ\nuGJ4F9659hhKyio5/7mfeOWHdagqqsqSnJ08+uUKbngrg8LiskCHWmercnfTs23j7H+AwE+1YYwx\ndTasWwLT/nA8f5q8iAc/W8YXmVvYtruUjQUlvz51fM/HmTx7ySC/r99cX+WVVazbXszJvQM319LB\n2B2EMSaoxMdF8dLl6dw/5kjW55fQJTGORy7ox9x7TuWOkb2YlrmVjxdWn9Wn8Vm/vZjySqVX+8bZ\n/wB2B2GMCUIiwhXDu3DF8C77lV9/Qle+XpHLvR8vZWhqAh1bNwtIfBN/2Uje7lImnNy9xjuZfR3U\nPRpxE5PdQRhjQkZ4mPCviwZQpcrtkxZSFYBhsbNWbuPujzJ5YuYq3qplKvNVubsJk8Y7ggksQRhj\nQkyn+Fj+NuZI5mQV8MoP6/x67ezCEm59fyG92rXg5N5teeDTZczJyvdZd1XublLiY4mJDPdrjPVh\nCcIYE3LGDk5mZFo7HvlyBV8u2eKXa5ZWVHLTO/OprFSev2wwT44bQEpCLDe/M5+cHXsOqL8qdzc9\nGukT1PtYgjDGhBwR4YmL+tO/U2tufncBny92P0nc/+kyFmfv5ImL+pOaGEfLmEheujydsooqbngr\ngz1lv81OW1pRyfr8kkb7BPU+liCMMSGpRUwkb1w9lEEprfn9xAV84tLIpk0FJdw3dSnv/ryR/zux\nGyOPbP/rvm5JzXly3ACWbt613/rb67YXU1mljXYOpn1sFJMxJmQ1j47g9auGcvXrc7nt/YVsLyrj\nwsHJtGoWeVjnVVW+XZXHm7M38M3KbYSJcMGgZO4YeeASoaf0accdI3vx2PSVdG/bnN+f0uPXEUyW\nIIwxJoDioiN47aoh3PDWPB78bBkPf7Gc4d0SGd23PaP7dqBV7IHJYm95JVPmZ3NCj6QD1qPYW17J\nHR8s4rPFW0hsHs2Ek7pzydEpdGhV85Dam0Z0Y+22Iv41cxVdk+JYnbub8DCha1Jcg3/ehuTqgkEi\nMgp4CggHXlbVh33UuQi4D1A8K8dd4pSnAC/jWddagTNUdX1N17IFg4wxtamqUhZm7+DLJVv5YskW\nNhXsoU1sJHeP7sOFg5MJc57EXpKzk9veX8jqbUXERYVz79lpXJTeCRFhe1Ep172ZwYKNO7jz9F5c\nd3xXoiLq1lJfWlHJpS/9TGbOTjonxFJRpXx9+wgXP3Hd1LZgkGsJQkTCgVXAaUA2njWqx6vqMq86\nPYBJwMmqWigibVV1m7NvFvCQqs4UkeZAlaqW1HQ9SxDGmLpSVRZu2sFDny8nY0MhQ7q04f4xfflq\neS5PfbWahOZR3D26D+/P3cTsrHxO7t2Wa49P5c9TFrNtVylPXjyA0f061Pu6+UWlnPvcj2wq2MOo\nI9vzwu8Gu/Dp6idQCWIYcJ+qnu5s3w2gqv/0qvMosEpVX652bBrwoqoeV9frWYIwxtRXVZUyeV42\n//hiOTtKygE4u/8RPHjOkbSOjaKqSnlj9noe/mIFpRVVJDaP5uUr0hnQqfUhX3N17m4ufGE2E07q\nznUndG2gT3LoApUgLgRGqeq1zvbvgKNVdYJXnY/x3GUci6cZ6j5V/VJEzgWuBcqAVOB/wF2qWlnt\nGtcD1wOkpKQM3rAhNFedMsa4q6C4jBe+XctRya0466gjDti/Nq+It+ds4OpjUxtkjey95ZVER4Q1\nigkFa0sQge6kjgB6ACOAZOA7EennlB8PDAQ2Au8DVwKveB+sqi8CL4LnDsJfQRtjQkt8XBR/OaNP\njfu7JTXnb2cf2WDXa8xPT3tz8zmIHDwdzPskO2XesoGpqlququvw3E30cMoXqmqWqlYAHwODXIzV\nGGNMNW4miLlADxFJFZEoYBwwtVqdj/HcPSAiiUBPIMs5trWIJDn1TgaWYYwxxm9cSxDOX/4TgOnA\ncmCSqi4VkQdEZIxTbTqQLyLLgG+AO1U13+lruAP4SkQyAQFecitWY4wxB3L1OQh/slFMxhhTf7V1\nUttcTMYYY3yyBGGMMcYnSxDGGGN8sgRhjDHGp5DppBaRPMDXo9StgJ0HKfPe9vV+37+JwPZDDNFX\nHHXZb/HvX3aon+Fg8ddWp7Z4q28f7L3FX/86B/sequnzNGT8tcV3sP2N/We4s6om+Sj3TFoVyi88\nczrVWua97eu9178ZDRlHXfZb/AeUHdJnOFj89fkM9Y2/If4PLP6ay2r6PA0Zf10+Q7D/DPt6NYUm\npk/rUPbpQd77OkdDxFGX/Ra/f+KvrU5t8Vbfrsv7Q2Hx11xW0+dpyPjrco5Q+BnYT8g0MfmDiGRo\nDeOFg0Gwxw/B/xks/sCy+OunKdxBNKQXAx3AYQr2+CH4P4PFH1gWfz3YHYQxxhif7A7CGGOMT5Yg\njDHG+NRkE4SIvCoi20RkySEcO1hEMkVkjYg8LV7LQonILSKyQkSWOkuqusKN+EXkPhHJEZGFzuuM\nho/81xhc+fo7+28XEXWmkHeNS/8HD4rIYufrP0NEDlzerIG4FP9jzvf/YhH5SEQOfW3Og8fgRvxj\nnZ/dKhFxpTP4cOKu4XxXiMhq53WFV3mtPyd1cqhjaoP9BZyAZxGiJYdw7C/AMXimIf8CGO2Un4Rn\nedRoZ7ttkMV/H3BHsH79nX2d8EwjvwFIDLbPALT0qvN74IUgi38kEOG8fwR4JMji7wP0AmYB6Y0p\nbiemLtXK4vGsoRMPtHHet6ntM9bn1WTvIFT1O6DAu0xEuonIlyIyT0S+F5He1Y8TkQ54fojnqOd/\n4U3gXGf3jcDDqlrqXGNbkMXvNy7G/2/gT4Droy/c+Ayqusurahwufg6X4p+hnrVgAObgWUkymOJf\nrqor3Yr5cOKuwenATFUtUNVCYCYwqqF+zptsgqjBi8AtqjoYz4JFz/mo0xHPkqj7ZDtl4FkR73gR\n+VlEvhWRIa5Ge6DDjR9ggtM88KqItHEvVJ8OK34ROQfIUdVFbgdai8P+PxCRh0RkE3ApcK+LsfrS\nEN9D+1yN5y9Xf2rI+P2pLnH70hHY5LW977M0yGeMqO8BoUpEmgPDgQ+8muqi63maCDy3escAQ4BJ\nItLVyeCuaqD4nwcexPNX64PAE3h+yF13uPGLSCzwFzxNHAHRQP8HqOo9wD0icjeeVRn/1mBB1qKh\n4nfOdQ9QAbzTMNHV6ZoNFr8/1Ra3iFwF/MEp6w5ME5EyYJ2qnud2bJYgfhMG7FDVAd6FIhIOzHM2\np+L5Jep925wM5Djvs4EPnYTwi4hU4ZlcK8/NwB2HHb+q5nod9xLwmZsBV3O48XcDUoFFzg9ZMjBf\nRIaq6laXY9+nIb6HvL0DTMNPCYIGil9ErgTOAk7xxx9HXhr66+8vPuMGUNXXgNcARGQWcKWqrveq\nkgOM8NpOxtNXkUNDfEY3OmGC5QV0waujCPgJGOu8F6B/DcdV7/w5wyn/P+AB531PPLd+EkTxd/Cq\ncxswMZi+/tXqrMflTmqX/g96eNW5BZgcZPGPApYBSW5/7d38HsLFTupDjZuaO6nX4emgbuO8j6/L\nZ6xTnP74T2yML+A9YAtQjucv/2vw/AX6JbDI+Sa/t4Zj04ElwFrgP/z2RHoU8Lazbz5wcpDF/xaQ\nCSzG85dWh2CKv1qd9bg/ismN/4MpTvliPJOrdQyy+Nfg+cNoofNycxSWG/Gf55yrFMgFpjeWuPGR\nIJzyq52v+xrgqvr8nBzsZVNtGGOM8clGMRljjPHJEoQxxhifLEEYY4zxyRKEMcYYnyxBGGOM8ckS\nhAlpIlLk5+u9LCJpDXSuSvHM6rpERD492MyoItJaRG5qiGsbA7ainAlxIlKkqs0b8HwR+ttkdK7y\njl1E3gBWqepDtdTvAnymqn39EZ8JfXYHYZocEUkSkSkiMtd5HeuUDxWR2SKyQER+EpFeTvmVIjJV\nRL4GvhKRESIyS0Qmi2ftg3f2zbXvlKc774ucifcWicgcEWnnlHdztjNF5O91vMuZzW+TEjYXka9E\nZL5zjnOcOg8D3Zy7jsecunc6n3GxiNzfgF9G0wRYgjBN0VPAv1V1CHAB8LJTvgI4XlUH4plF9R9e\nxwwCLlTVE53tgcCtQBrQFTjWx3XigDmq2h/4DrjO6/pPqWo/9p9x0ydnLqFT8DzdDrAXOE9VB+FZ\ng+QJJ0HdBaxV1QGqeqeIjAR6AEOBAcBgETnhYNczZh+brM80RacCaV4zZ7Z0ZtRsBbwhIj3wzGgb\n6XXMTFX1nsP/F1XNBhCRhXjm1vmh2nXK+G3Cw3nAac77Yfw2N/+7wOM1xNnMOXdHYDmeuf7BM7fO\nP5xf9lXO/nY+jh/pvBY4283xJIzvarieMfuxBGGaojDgGFXd610oIv8BvlHV85z2/Fleu4urnaPU\n630lvn+WyvW3Tr6a6tRmj6oOcKYynw7cDDyNZ52IJGCwqpaLyHogxsfxAvxTVf9bz+saA1gTk2ma\nZuCZKRUAEdk3zXIrfpsS+UoXrz8HT9MWwLiDVVbVEjzLj94uIhF44tzmJIeTgM5O1d1AC69DpwNX\nO3dHiEhHEWnbQJ/BNAGWIEyoixWRbK/XH/H8sk13Om6X4ZmmHeBR4J8isgB3765vBf4oIovxLAKz\n82AHqOoCPDO8jsezTkS6iGQCl+PpO0FV84EfnWGxj6nqDDxNWLOdupPZP4EYUysb5mqMnzlNRntU\nVUVkHDBeVc852HHG+Jv1QRjjf4OB/zgjj3bgp2Vdjakvu4Mwxhjjk/VBGGOM8ckShDHGGJ8sQRhj\njPHJEoQxxhifLEEYY4zx6f8DD2fws0QdHe4AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dhNl36NqzLdb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 197
        },
        "outputId": "8b5e7e63-1b81-4647-fb7c-21caeefda047"
      },
      "source": [
        "learn.fit_one_cycle(5, 1e-3, moms=(0.8,0.7))"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.445050</td>\n",
              "      <td>0.361552</td>\n",
              "      <td>0.842000</td>\n",
              "      <td>10:25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.443763</td>\n",
              "      <td>0.360960</td>\n",
              "      <td>0.845600</td>\n",
              "      <td>10:24</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.427118</td>\n",
              "      <td>0.358420</td>\n",
              "      <td>0.843040</td>\n",
              "      <td>10:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.447346</td>\n",
              "      <td>0.355710</td>\n",
              "      <td>0.844480</td>\n",
              "      <td>11:09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.437228</td>\n",
              "      <td>0.361192</td>\n",
              "      <td>0.842680</td>\n",
              "      <td>10:53</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E1j8OCxrGRPk",
        "colab_type": "text"
      },
      "source": [
        "Below are the results for using the Norm-Stabilizer. We are only adding terms to regularize the last LSTM layer which is also the only LSTM layer which is not frozen. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aS-DTVjM0Q1S",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "outputId": "26cde0eb-2e77-4b2d-c861-a02b0d89b914"
      },
      "source": [
        "learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5,)\n",
        "learn = learn.load_encoder('fine_tuned_enc')\n",
        "learn.freeze_to(-2)\n",
        "learn.callbacks =[RNNTrainerNorm(learn,2.,0.,500.)]\n",
        "learn.lr_find()\n",
        "learn.recorder.plot()"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU1fn48c+TnayQjX0Ja9iUJayC\nCFSLVsWlWrBaqVat/brXtvrza+1Xa+vXperXpS1adyl1rzuKQFFZg+yEQAhb2LKxJYGsz++PGXSI\nkwXIzZ1knvfrNS/mnnvuzHOYTJ6cc+49V1QVY4wxprYQtwMwxhgTmCxBGGOM8csShDHGGL8sQRhj\njPHLEoQxxhi/wtwOoKkkJydrjx493A7DGGNalBUrVhSqaoq/fY4mCBGZAjwJhALPq+pDtfY/Dkz0\nbkYDqara1ruvG/A80BVQ4DxV3VbXe/Xo0YPMzMwmb4MxxrRmIrK9rn2OJQgRCQWeAc4G8oDlIvK+\nqm44VkdVb/epfzMw1OclXgEeVNXPRSQWqHEqVmOMMd/n5BzESCBHVXNVtQKYDUytp/504J8AIjIA\nCFPVzwFUtURVyxyM1RhjTC1OJojOwE6f7Txv2feISHcgDZjnLeoLHBCRd0RkpYg84u2R1D7uehHJ\nFJHMgoKCJg7fGGOCW6CcxTQNeEtVq73bYcB44E5gBNATmFH7IFWdqaoZqpqRkuJ3jsUYY8xJcjJB\n7MIzwXxMF2+ZP9PwDi955QGrvMNTVcB7wDBHojTGGOOXkwliOdBHRNJEJAJPEni/diURSQfaAYtr\nHdtWRI51CyYBG2ofa4wxxjmOJQjvX/43AXOALOANVV0vIveLyIU+VacBs9VnWVnvUNOdwBcishYQ\n4DmnYjXGGPN90lqW+87IyFC7DuLUfJ1TCMAZvZNdjsQY01xEZIWqZvjbFyiT1MYBqkpJeVWj6j3/\nZS5X/mMpV/1jKf9ctqMZojPGBLpWs9SGOd6Rimpumb2SrzYX8taNYxjYKcFvveoa5YEPN/DSom1M\nGdiBo1XV3P3OWg4eqeSXE3o1c9TGmEBiPYhWqKiknOnPLWFu1j4iwkK4adZKvz2JIxXV/PK1Fby0\naBu/GJfGsz8dxsyrMjj/tI489MlGHvpkI61lCNIYc+IsQZyi/aUVfLRmDzuLywLil+n2olIu/esi\nsvYc4q8/Hc7Mq4azvaiUe95de1x8ew4e4SczFzM3ax/3XTCA/z5/ACEhQkRYCE9OG8pPR3Xjb//Z\nwq9e/4Y9B4+42CJjjFtsiOkUVFbXcN0rmWRu3w9AalwkGT3accFpnTh3cMdmiaG0vIpN+w6zad9h\nNu49zPurdlOjyqzrRjO8ezsAbv9BXx77fBNjeyXxkxHdWLF9Pze8uoIjFVXMvCqDswe0P+41Q0OE\nP140iM7t2vDE3M0syC7gpkm9uXZcGlHh37ug3RjTStlZTKfgjx9u4PmvtvL78wcQHhZC5rZilm0t\nZu+ho3xw0zgGdfY/7l+fTfsOExcVRseENn7319Qo63YfZEF2AQuy81m18wA13o+wTXgog7sk8NAl\ng+mZEvvtMdU1ytUvLGP5tmJuPKsXz87fQoeEKJ6/OoO+7ePqjWdncRkPfpTFp+v30i0xmsszutC/\nYzz9O8bTMSEKETnhNhpjAkd9ZzFZgjhJH6/dw69e/4arx3Tnf6YO+rb84JFKJj66gN4psfzrhtGN\n+gVaXaPMWb+X57/M5ZsdBwDolRLDuN7JjOmVxKEjVWzce5jsfYfYsPsQ+8sqEYHTOidwZt8UBndO\noF+HOLq2iyYkxP/7FRwu59wnv6SwpJyxvZJ45ophtIuJaHR7v9pcyJ8/yWL97kPflrWLDue+CwZy\n0VC/S2wZY1oASxBNbEtBCVOf/preqbG8ccMYIsKOn8p5fel27nl3Hc9cMYwfnVb/UNN7K3fx2OfZ\n7Cw+QrfEaK4e24OaGuWrnEKWbi3iaKVnlfOo8BD6to+jX/s4xvZO4sw+KSTFRp5Q3GvzDrJ0axEz\nxvYgLPTkpp8OH60ke+9hsvYc4t2Vu1iTd5BXrx3FmF5JJ/V6xhh3WYJoQmUVVVz0zNcUHC7no1vG\n06nt94eCqmuUH/3flxw+WsUXv55Q57j9mrwDXPTM1wzqnMCvzurN2QPaE+rTAyivqmbdrkMkxkTQ\nLTH6uH2B4OCRSi796yIKDpfz7q/GHjesZYxpGexCuSb01wVb2LSvhCenDfWbHMAzyXvfBQPZdeAI\nzy3M9VunsrqGu95eS3JsJK/9YhRTBnX4XgKIDAtlePd2pCXHBFxyAEhoE86LM0YQFiJc89Jy9pdW\nnNDxRyurWZRTyGOfZXPZ3xZx2d8WcbSyuuEDjTHNws5iOgHFpRW88NVWfjS4I2f2rX958TG9kpgy\nsAPPLtjCZRld6ZAQddz+57/cyoY9h/jblcOJjwp3MmxHdU2MZubPhjP9uaXc8NoKfjamOyVHqygp\nr6KsoprQECEyLITIsBAUz6T3tqIydhSVsbWwlIrqGkIE+neMZ/3uQzw1bzO/+WG6280yxmAJgsrq\nGp7/ciuXDOtM+/ioeuv+/T9bOFJZze1n92nUa/+/8/ozLzufu95Zw+OXD/l2UnhbYSlPzN3ElIEd\nmDKowym3wW3DuyfyyI9P49bZq1i2tbjeupFhIXRPiqZbYgxnpacwKi2RjB6JxEeF8+s3VvP3/+Qy\ndUjnBs+uMsY4L+gTxO4DR3hi7iZWbN/Pcz8bXudZR/mHj/Ly4m1MHdKZ3qmN++XVLSmau89N548f\nZXHWowu485y+TB/ZjbvfWUtEWAj/M3VgE7bEXVOHdGZYt3YcrawmJjKMmMgwoiNCqa5RyqtqqKiq\nQVGSYyLrPNPqnh/1Z97Gfdz9zlrevGFMnfWMMc0j6OcguifFcOc5/ZibtY/3V++us96z87dQWa3c\nOrlxvYdjfn5GGh/dMo4BHeO599/rGf/wfBbnFnHXuekN9lhamq6J0fRpH0entm1IaBNOeGgIUeGh\nJLQJJyUuktS4qHp/6SfGRHDPjwawYvt+Zi/fWWc9Y0zzCPoEAXDNuDSGdmvLfe+vJ//w0e/t333g\nCLOW7uDHw7rQIznmhF8/vUM8s64bxbM/HUaICON6JzN9RLemCL3VuXRYZ0b3TOShT7L8fhbGmOZj\nCQLPWUeP/Ph0yiqqufe9dd9bU+mpeTkoys2Te5/0e4gI5w3uyJe/ncjL14y04ZM6iAgPXjyYo5U1\n/O6tNRwsq3Q7JGOCliUIr96psdxxdl/mrN/Hh2v2oKpk7z3M0/M282bmTqaN6EaXdtGn/D4hIRKQ\np6wGkl4psfy/89L5z6YCJj62gDeW76SmpnVcr2NMS2IXyvmoqq7h0r8tZmtBCQnR4ews9qxiOrx7\nO/565TBS41rXnEGgW7/7IPf9ez2Z2/czpGtb/njRoJNa38oYUze7kvoEbN53mGteXk7f1Dgm92/P\n5P6prW4yuSVRVd75Zhd//mQjB8oquP3svvxyQi/rhRnTRCxBmBbvYFkl97y3lg/X7GFEj3b85fIh\ndE089SE/Y4KdLbVhWryE6HCemj6Ux39yOhv3HObcJ7/kwzV1n5ZsjDl1liBMiyEiXDy0Cx/fOp70\nDnHcNGslT8/bHBB38jOmNbIEYVqcronRvH7dKC4e2plHP9vEb99aQ0VVjdthGdPqOJogRGSKiGSL\nSI6I3OVn/+Missr72CQiB2rtjxeRPBF52sk4TcsTGRbKXy4/nVsn9+HNFXnMeHEZB4/YNRPGNCXH\nEoSIhALPAOcCA4DpIjLAt46q3q6qQ1R1CPAU8E6tl3kAWOhUjKZlExFuP7svj112Osu3FXPfv9e5\nHZIxrYqTPYiRQI6q5qpqBTAbmFpP/enAP49tiMhwoD3wmYMxmlbg0uFduHpMDz5Ys4e8/WVuh2NM\nq+FkgugM+K64luct+x4R6Q6kAfO82yHAY8CdDsZnWpFrxqUhwD++2up2KMa0GoEyST0NeEtVj91O\n7FfAx6qaV99BInK9iGSKSGZBQYHjQZrA1altGy48vROzl+3kQNmJ3dnOGOOfkwliF9DVZ7uLt8yf\nafgMLwFjgJtEZBvwKPAzEXmo9kGqOlNVM1Q1IyWl/ju8mdbv+gk9OVJZzWtLtrsdijGtgpMJYjnQ\nR0TSRCQCTxJ4v3YlEUkH2gGLj5Wp6k9VtZuq9sAzzPSKqn7vLChjfKV3iGdC3xReWrTN7m1tTBNw\nLEGoahVwEzAHyALeUNX1InK/iFzoU3UaMFvtaifTBG6Y0JPCkgre+aauzqoxprFsLSbTqqgqFz79\nNaXlVXx+xwRb1M+YBthaTCZoiAg3TOhJbmEpd765mne+yWNrYaktx2HMSQhzOwBjmtqUgR2YOqQT\nn63fy7srPUNNSTERPDV9KGN7J7scnTEthw0xmVarukbZnH+YlTsO8Pf/bKGyWvn8jjOJjrC/i4w5\nxoaYTFAKDRHSO8QzfWQ3Hv7x6ew6cIQn5m52OyxjWgxLECYojExLZNqIrvzjq62s333Q7XCMaREs\nQZigcfe5/WkXHc7d76yluqZ1DK0a4yRLECZoJESHc+/5A1iTd5BXFm9zOxxjAp4lCBNULjy9E+P7\nJPPonGx2HTjidjjGBDRLECaoiAgPXjQYBX79xiobajKmHpYgTNDplhTNHy4YyJLcYp77MtftcIwJ\nWJYgTFC6LKML5w7qwGOfZbM2z85qMsYfSxAmKIkIf75kMEkxkdz6r5WUVVS5HZIxAccShAlabaMj\n+Mvlp7O1sJQ/fpTldjjGBBxLECaoje2dzPXjezJr6Q5W7TzgdjjGBBRLECbo3Ty5D7GRYXZthDG1\nWIIwQS82MoxLhnXmwzV7KC61+1kbc4wlCGOAK0d3p6Kqhjcyd7odijEBwxKEMUDf9nGM7pnI60u3\n28VzxnhZgjDG66rRPdhZfIT/bMp3OxRjAoIlCGO8zhnYnpS4SF5dvN3tUIwJCJYgjPEKDw1h+shu\nLNhUwI6iMrfDMcZ1liCM8XHFyG6EiPD6UutFGGMJwhgfHRKiOGdAe2Yv38miLYVuh2OMqxxNECIy\nRUSyRSRHRO7ys/9xEVnlfWwSkQPe8iEislhE1ovIGhH5iZNxGuPrlsl9iIkI5YrnlnL1C8vYsPuQ\n2yEZ4wpRdeaUPhEJBTYBZwN5wHJguqpuqKP+zcBQVb1GRPoCqqqbRaQTsALor6p1roWQkZGhmZmZ\nTd4OE5yOVlbz6uLtPD0/h0NHK7lyVHceuGiQ22EZ0+REZIWqZvjb52QPYiSQo6q5qloBzAam1lN/\nOvBPAFXdpKqbvc93A/lAioOxGnOcqPBQrjuzJwt/O5FLh3Xh1SXbyS0ocTssY5qVkwmiM+B7WWqe\nt+x7RKQ7kAbM87NvJBABbHEgRmPqldAmnNt+0AeAL7Ls+ggTXAJlknoa8JaqVvsWikhH4FXg56pa\nU/sgEbleRDJFJLOgoKCZQjXBpku7aNI7xPHFxn1uh2LM99z/wQbuenuNI6/tZILYBXT12e7iLfNn\nGt7hpWNEJB74CLhHVZf4O0hVZ6pqhqpmpKTYCJRxzqT0VJZv28/BI5Vuh2LMcVbnHWBHsTPX7TiZ\nIJYDfUQkTUQi8CSB92tXEpF0oB2w2KcsAngXeEVV33IwRmMaZXL/9lTXKAs3WU/VBJaiknKSYyMd\neW3HEoSqVgE3AXOALOANVV0vIveLyIU+VacBs/X406kuB84EZvicBjvEqViNaciQrm1JjIngiywb\nZjKBpbCkgqTYCEdeO8yRV/VS1Y+Bj2uV/b7W9h/8HPca8JqTsRlzIkJDhLP6pTBvYz5V1TWEhQbK\n9J0JZkcrqykpr2p5PQhjWpvJ6e05UFbJSrs1qQkQRd4bXCU71IOwBGFMI43vm0xYiNjpriZgFB4u\nByApxnoQxrgqPiqckWmJzLPTXU2AKCr1JIjkOEsQxrhucv/2bNpXwk6HTis05kQUHvYMMSXF2BCT\nMa6bnJ4KYGczmYBQeKwH4dAktaNnMRnT2vRIjqFnSgyzlu2gorqG5NhIkmIjOb1LAm2jnfkrzpi6\nFB6uICYilDYRoY68viUIY07QFSO78fCcbP708cZvyzolRPHRLeNp51BX3xh/ikrLHZt/AEsQxpyw\nX4zvybXj0igpr6KwpIJN+w5z86yV/PrN1Tz/swxCQsTtEE2QKCqpcGz+AWwOwpiTIiLERYWTlhzD\nDwd24L/P78+8jfk892Wu26GZIFLo4DIbYAnCmCZx1ejunDe4Aw/PyWbF9mK3wzFBwrPMhiUIYwKa\niPDQpafRuW0bbp61kv3eK1yNcUp1jVJcWu7YVdRgCcKYJhMfFc4zVwyjsKSC37+/3u1wTCt3oKyC\nGnXuFFewBGFMkxrcJYEbJvTkg9W7ydpzyO1wTCtWWOK9SM56EMa0HL8Y15O4qDCemLvJ7VBMK1ZU\n4uxFcmAJwpgmlxAdzrXj0pizfh/rdh10OxzTShV8myCsB2FMi3LNuDTirRdhHFRUcmypb+tBGNOi\nxEeFc/2ZPZmblc9qn/tHVNco8zfmU2xnOZlTVFRaTliIEB8V7th7WIIwxiEzzkijbXQ4T8zdhKoy\nd8M+znvyS37+0nL+6/VvqKnRhl/EmDoUHq4gMSbC0Sv3LUEY45DYyDBuOLMX87MLuODpr/jFK5mU\nV1VzxahuLM4tYtayHW6HaFqwolJnr6IGW4vJGEf9bEx3Xvx6K/mHynnw4kFcntGVsBBhe1Epf/44\ni4npqXRu28btME0LVFBS4egprmA9CGMcFRMZxud3TGDhbyfy01HdCQ8N8Vx1fclpKHDX22tQtaEm\nc+KKSspJcbgHYQnCGIcltAknKvz49fq7JkbzuynpfLm5kDdX5LkUmWmpVJXCknLrQRjTWl01ujsj\n0xJ54MMN7Dl4xO1wTAtSVlHN0coax+cgHE0QIjJFRLJFJEdE7vKz/3ERWeV9bBKRAz77rhaRzd7H\n1U7GaYwbQkKEhy89jeoa5ap/LKPQe+GTMQ0p+naZjRaaIEQkFHgGOBcYAEwXkQG+dVT1dlUdoqpD\ngKeAd7zHJgL3AaOAkcB9ItLOqViNcUuP5BhemDGCvP1lXPn8Urs+wjRKc1xFDc72IEYCOaqaq6oV\nwGxgaj31pwP/9D7/IfC5qhar6n7gc2CKg7Ea45rRPZP4x9Uj2FpYypXPL+VAmSUJU7/mWIcJnE0Q\nnYGdPtt53rLvEZHuQBow70SPNaY1OKN3Mn+/ajg5+SX87IVlHDpa6XZIJoA1x0qu0MgEISK9RCTS\n+/wsEblFRNo2YRzTgLdUtfpEDhKR60UkU0QyCwoKmjAcY5rfWf1S+euVw8jac4hrXlxOWUWV2yGZ\nAHWsB5EUExg9iLeBahHpDcwEugKzGjhml7feMV28Zf5M47vhpUYfq6ozVTVDVTNSUlIaCMeYwDe5\nf3uenDaUb3bs5/pXVnC08oT+ZjJBorCknPioMCLCnD0RtbGvXqOqVcDFwFOq+hugYwPHLAf6iEia\niETgSQLv164kIulAO2CxT/Ec4BwRaeednD7HW2ZMq3fe4I48/OPT+SqnkJtmraSyusbtkEyAKSyt\nIDnO2d4DND5BVIrIdOBq4ENvWb1LCHoTyk14frFnAW+o6noRuV9ELvSpOg2YrT6Xk6pqMfAAniSz\nHLjfW2ZMUPjx8C48MHUgc7P2cccbq23i2hynqKScZIeHl6DxazH9HPgl8KCqbhWRNODVhg5S1Y+B\nj2uV/b7W9h/qOPYF4IVGxmdMq3PVmB6UVlTz0CcbmbNuLxPTU7h4aBcmpqcQGRba8AuYVquwpIK+\n7WMdf59GJQhV3QDcAuAd8olT1f91MjBjDPxyQi/G9U7m3ZW7+Peq3cxZv4/Obdvw8a3jSWjj3H0A\nTGArKiknqWeS4+/T2LOYFohIvPcCtm+A50TkL86GZowBGNQ5gXvPH8CSuyfx6GWns+vAERZk57sd\nlnFJZXUN+8sqHT/FFRo/B5GgqoeAS4BXVHUU8APnwjLG1BYWGsIlQzuTFBPBvI2WIILV/lLnbzV6\nTGMTRJiIdAQu57tJamNMMwsJESamp7Igu4AqO7spKDXXMhvQ+ARxP56zkbao6nIR6Qlsdi4sY0xd\nJqWncvBIJd/sONBwZdPqHFuoL2B6EKr6pqqepqo3erdzVfVSZ0Mzxvgzvk8yYSHCFxv3uR2KccGx\nVX+dXskVGj9J3UVE3hWRfO/jbRHp4nRwxpjvi4sKZ1TPRObbPERQ+q4HEThDTC/iuQq6k/fxgbfM\nGOOCSent2bSvhJ3FZW6HYppRdY2St7+MiLAQYiMbexnbyWvsO6Soqm9CeElEbnMiIGNMwyalp/LA\nhxuYtzGfq8f2cDsc46A1eQd4bcl2Nu49zKZ9hzlaWUNacgwi4vh7NzZBFInIlXy3oN50oMiZkIwx\nDUlLjqFncgxfWIJo1d5buYvfvr2GqLAQBndJ4IqR3UnvEMeYXs5fJAeNTxDX4Lnj2+OAAouAGQ7F\nZIxphEnpqbyyeDul5VXENMNwg2k+NTXKo59l8+yCLYxKS+SvVw4nMcb5OYfaGnsW03ZVvVBVU1Q1\nVVUvAuwsJmNcNKl/KhXVNXyVU+h2KKYJlVVUccNrK3h2wRamj+zGq9eOciU5wKndUe6OJovCGHPC\nRvRIJC4yjHlZdjZTa/Laku18vmEff7hgAH+6eJDj93yoz6m8s/MzJMaYOoWHhnBm3xS+2Jhvtyht\nRZZtLaZncgwzzkhrlono+pxKgtCGqxhjnHTVmO4cPFLBjBeWUVJutyht6VSVFdv3M7x7O7dDARpI\nECJyWEQO+XkcxnM9hDHGRaN7JvHU9GGszjvINS/ZfaxbutzCUvaXVbaMBKGqcaoa7+cRp6p22oQx\nAWDKoA488ZMhZG4r5rpXMu0+1i3Yim37Acjo0QIShDGmZbjg9E48etnpLNpSxC3/XOl2OOYkZW4v\npm10OD2Tnb9bXGNYL8CYVuKSYV3I23+Ev3y+iY17D5HeId7tkMwJWrF9P8O7tSMkJDDOAbIehDGt\nyJWjuxMRGsLsZTvdDsWcoP2lFWwpKGV4gAwvgSUIY1qVxJgIzhnYnndX7rK5iBZmxXbP/MPwbpYg\njDEOmT6yGwePVDJn/V63QzEnIHP7fsJDhdO7tnU7lG9ZgjCmlRnTM4luidH8c9kOt0MxJ+Cb7fsZ\n2CmBqPBQt0P5lqMJQkSmiEi2iOSIyF111LlcRDaIyHoRmeVT/rC3LEtE/k/cvqTQmBYiJET4yYiu\nLMktZmthqdvhmEaoqKphdd4BMgLk+odjHEsQIhIKPAOcCwwApovIgFp1+gB3A2eo6kDgNm/5WOAM\n4DRgEDACmOBUrMa0NpcN70JoiDB7ufUiWoJ1uw9SXlUTMNc/HONkD2IkkOO9f3UFMBuYWqvOdcAz\nqrofQFWPrTqmQBQQAUQC4YDdgNeYRkqNj2JSeipvr8ijsrrG7XBMA45dIDcsWHoQQGfA91y7PG+Z\nr75AXxH5WkSWiMgUAFVdDMwH9ngfc1Q1y8FYjWl1po/sSmFJBV9k2d9WgW7F9v10S4wmNS7K7VCO\n4/YkdRjQBzgLz13qnhORtiLSG+gPdMGTVCaJyPjaB4vI9SKSKSKZBQUFzRi2MYHvzD4pdIiP4rUl\nNswUyFSVzO37A27+AZxNELuArj7bXbxlvvKA91W1UlW3ApvwJIyLgSWqWqKqJcAnwJjab6CqM1U1\nQ1UzUlJSHGmEMS1VWGgI145L46ucQhZk2z0jAtWO4jIKS8oD6gK5Y5xMEMuBPiKSJiIRwDTg/Vp1\n3sPTe0BEkvEMOeUCO4AJIhImIuF4JqhtiMmYE3T12B6kJcdw/4cbqKiyuYhAtDS3GICM7okuR/J9\njiUIVa0CbgLm4Pnl/oaqrheR+0XkQm+1OUCRiGzAM+fwG1UtAt4CtgBrgdXAalX9wKlYjWmtIsJC\nuPf8/uQWlPLK4m1uh2P8mLcxn44JUfRtHxgL9PlydLE+Vf0Y+LhW2e99niueW5feUatONXCDk7EZ\nEywmpbfnrH4pPDl3M1OHdCYlLtLtkIxXRZXnnuIXnN7J9bvH+eP2JLUxphnce/4AjlRW8+icbLdD\nMT6WbyumpLyKSempbofilyUIY4JAr5RYZoztwRsrdrI276Db4RiveRvziQgL4YzeSW6H4pclCGOC\nxC0/6ENSTAQPfrzB7VCM1/yN+YzumUR0RGDemscShDFBIj4qnF9O6MWS3GIytxW7HU7Q21ZYSm5h\nKZP6Be4p+pYgjAkiV4zqRmJMBE/Pz3E7lKA3b6Pn2pRJ6e1djqRuliCMCSLREWFcOy6NBdkFrNtl\ncxFump+dT6+UGLolRbsdSp0sQRgTZK4a0524qDCesV6Ea0rLq1iaWxywZy8dYwnCmCATHxXO1WN6\n8On6veTkH3Y7nKD0VU4hFdU1TLQEYYwJNNeMSyMqLJRn529xO5SgcKCsgpoa/XZ7/sZ84iLDGNEj\n8JbX8BWY51YZYxyVGBPBT0d148VF27jtB30Dehy8pVu36yAXP/s1CW3CmdA3lYnpKczPzmd832TC\nQwP7b/TAjs4Y45jrzuxJaIjwwtdb3Q6l1VJVHvhwA3FR4YztlczcrH3cNGsl+w6VM7FfYA8vgfUg\njAla7eOjGNc7mYWb7F4qTpmzfh9LtxbzwEWDuGp0d6qqPfeeXr/7EBcO6eR2eA2yHoQxQWxsryRy\nC0vZc/CI26G0OuVV1fzp4yz6to9l+gjPrXHCQkMY3j2Rn43pQWRYqMsRNswShDFBbGyvZAC+zily\nOZLW5+VF29hRXMZ//2gAYQE+11CXlhm1MaZJpHeIIzEmgkVbCt0OpVUpKinnqS9ymNgvhTP7Bu5S\nGg2xBGFMEAsJEcb0SmJRThGe27OYpvD43E2UVVZzz48GuB3KKbEEYUyQG9srib2HjpJbWOp2KK3C\nul0HmbV0B1eN7k7v1MC7S9yJsARhTJA7wzsPsSjHhplOVVV1DXe9s4ak2EhuP7uv2+GcMksQxgS5\n7knRdEqIYtEWm6g+VS8t2sa6XYf4wwUDSWgT7nY4p8wShDFBTkQY2zuZxblFxy0HYU5M3v4yHvts\nE5PTUzlvcAe3w2kSliCMMStFOwoAABOFSURBVJzRO4kDZZVs2HPI7VBaJFXl9/9ejwjcf9EgRMTt\nkJqEJQhjzLfXQ9jprifno7V7mLcxnzvO7kvntm3cDqfJWIIwxtA+PopeKTF2wdxJqK5R/vhhFoM7\nJzBjbA+3w2lSjiYIEZkiItkikiMid9VR53IR2SAi60Vklk95NxH5TESyvPt7OBmrMcHujN7JLNta\nTEVVjduhtCjLtxWz99BRbpjQs8VeMV0Xx1ojIqHAM8C5wABguogMqFWnD3A3cIaqDgRu89n9CvCI\nqvYHRgL5TsVqjPFcD3GksprVeQfcDqVF+XTdXiLDQlrE6qwnysl0NxLIUdVcVa0AZgNTa9W5DnhG\nVfcDqGo+gDeRhKnq597yElUtczBWY4Le6J5JiMDcrH1uh9Ji1NQon67by5l9U4iJbH2LYzuZIDoD\nO32287xlvvoCfUXkaxFZIiJTfMoPiMg7IrJSRB7x9kiMMQ5pGx3BuYM68PqSHRwoq3A7nBZhVd4B\n9h462mpOa63N7QGzMKAPcBYwHXhORNp6y8cDdwIjgJ7AjNoHi8j1IpIpIpkFBbamvTGn6tbJfSmt\nqOK5L3PdDqVF+HTdXsJDhUnp7d0OxRFOJohdQFef7S7eMl95wPuqWqmqW4FNeBJGHrDKOzxVBbwH\nDKv9Bqo6U1UzVDUjJaXlrphoTKDo1yGO8wZ35KWvt1Fcar2I+qgqn6zbwxm9k1vFVdP+OJkglgN9\nRCRNRCKAacD7teq8h6f3gIgk4xlayvUe21ZEjv3WnwRscDBWY4zXbZP7UFZZbb2IBqzffYidxUc4\nd1DrHF4CBxOE9y//m4A5QBbwhqquF5H7ReRCb7U5QJGIbADmA79R1SJVrcYzvPSFiKwFBHjOqViN\nMd/p0z6O80/rxMuLtlFUUu52OAHrk3V7CA0Rzh7QehOEo9Puqvox8HGtst/7PFfgDu+j9rGfA6c5\nGZ8xxr9bJ/fhozW7mbkwl7vP6+92OAHHM7y0l1FpiSTGRLgdjmNa33lZxphT1js1lgtP78Qri7fT\npV0biksrKSg5SsnRKu78YT+6tIt2O0RXbc4vIbeglJ+3siuna7MEYYzx65bJffho7R7u/fd6ANpF\nh1NSXkW1wlPTh7ocnbs+WbsXEfjhwNY7vASWIIwxdeiZEsvC304EICkmkoiwEB6Zs5Fn5m/hxgm9\nGNAp3uUI3fPp+r0M79aO1Pgot0NxlNvXQRhjAljHhDZ0TGhDRJjnV8X143sRHxXGY59luxyZe/Yc\nPELWnkOcPaB1XvvgyxKEMabREqLD+eVZvfhiYz4rthe7HY4r5m/0XJQ7Mb31rb1UmyUIY8wJmTG2\nB8mxkTz8aTaeExGDy/zsfDq3bUOf1Fi3Q3GcJQhjzAmJjgjj5km9Wbq1mC83B9cNhsqrqvk6p5CJ\n6Smt5q5x9bEEYYw5YdNHdqNLuzY8Mie4ehHLthZTVlHNpCAYXgJLEMaYkxARFsLtP+jL2l0Hue6V\nTAqD5IrreRvziQwLYUzPZLdDaRaWIIwxJ+WSYZ259/wBLNxcyJQnFjJvY+u/j8T8jfmM6ZVEm4jg\nuPuAJQhjzEkREa4dl8YHN40jOTaSa17K5N731lFd0zqHnLYWlrKtqCxohpfAEoQx5hT16xDHv286\ng2vHpfHqku2tdhXYeRs9dz1ujbcWrYslCGPMKYsMC+W/f9Sfcwd14C+fbSJ772G3Q2pyC7Lz6Z0a\nS9fE4FmHyhKEMaZJiAh/vGgQcVFh3PHGKiqqatwOqcmUllexNLc4qIaXwBKEMaYJJcVG8qdLBrN+\n9yGenp/jdjhN5uucQiqqazirX3DdudIShDGmSf1wYAcuGdqZZ+bnsCbvgNvhNIn52fnERoYxokei\n26E0K0sQxpgmd9+FA0mJjeS2f61i36GjbodzSlSVBdkFjOudTHhocP3KDK7WGmOaRUKbcJ6YNoS9\nB49y4dNfsXpny+1J5OSXsOfgUSYE2fASWIIwxjhkdM8k3r5xLGEhIVz+98X8e9Wu4/YfraxuEddM\n/GeTZ/XWM/sGX4KwGwYZYxzTv2M87990Bje+9g23zl7FB6v3cPhoJTuKy9hz8Ci9U2N584YxtAvg\n+zov3FxIr5QYOrdt43Yozc56EMYYRyXFRvLaL0Zx1ejurNt1kKoaZUyvJG48qxc7isv41evfUFkd\nmKfEHq2sZmluUVD2HsB6EMaYZhARFsIDFw3igYsGHVfeOyWWX7+5mvs/2PC9fYFg6dZiyqtqmGAJ\nwhhjmtelw7uQve8wMxfm0q9DHFeO7u52SMdZuKmAiLAQRqUluR2KKxwdYhKRKSKSLSI5InJXHXUu\nF5ENIrJeRGbV2hcvInki8rSTcRpj3PO7KelM7JfCH95fz6ItgXUDov9sKmBUWmLQrN5am2MJQkRC\ngWeAc4EBwHQRGVCrTh/gbuAMVR0I3FbrZR4AFjoVozHGfaEhwpPTh9IjOYZfvJz57VlDbtt94Ag5\n+SWc2Sc4h5fA2R7ESCBHVXNVtQKYDUytVec64BlV3Q+gqvnHdojIcKA98JmDMRpjAkB8VDizfjGK\n7kkxXPvSct5buavhgxy20JuogvH6h2OcTBCdgZ0+23neMl99gb4i8rWILBGRKQAiEgI8BtzpYHzG\nmACSGh/Fv24YzfDu7bjtX6t43uVlwxduLqBDfBR9UmNdjcNNbk9ShwF9gLOALsBCERkMXAl8rKp5\n9d0YXESuB64H6Natm+PBGmOcFR8VzsvXjOT2f63ijx9l8daKPGIjw2gTEUqb8FDG903h8owuRIY5\nOydQVV3Dl5sLOXdQB+r7HdTaOZkgdgFdfba7eMt85QFLVbUS2Coim/AkjDHAeBH5FRALRIhIiaoe\nN9GtqjOBmQAZGRmBf0mmMaZBUeGhPH3FMJ6e51ns70hlNSXlVewsLuOzDft4dn4Ov5rY29FEsTrv\nAIePVgXt9Q/HOJkglgN9RCQNT2KYBlxRq857wHTgRRFJxjPklKuqPz1WQURmABm1k4MxpvUKDRFu\n/UGf48pUla9zinh87ibufW8dz87P4cGLBzEpvX2Tv/8XWfmECIzrndzkr92SODYHoapVwE3AHCAL\neENV14vI/SJyobfaHKBIRDYA84HfqGqRUzEZY1ouEWFcn2Te+uUYXrt2FG2jI7j25UyemZ+DatMN\nICzJLeK5L3OZlJ5K2+jAXQKkOUhT/se6KSMjQzMzM90OwxjTTI5WVvO7t9fw71W7+dHgjjxy2WlE\nR5zaoMiWghIueXYRybERvHPjGSREhzdRtIFLRFaoaoa/fW5PUhtjzEmJCg/liZ8MYUDHeB76dCNb\nCkqYlJ5Km/BQ2kSEEt8mnLP6pZAaF9Wo1ysureCal5YTFiK8OGNkUCSHhliCMMa0WCLCDRN60a9D\nHHe/s5a/L8w9bgnx0BBhYr8Ufjy8K5PSU4kI8z+qfrSymutfyWTPwaP887rRdEuKbq4mBDRLEMaY\nFu+sfqksvnsyABVVNRyprGbPwSO8u3IX73yzi7lZ+aTERfLYZad/78ykQ0cr+a/XvyFz+36evmIo\nw7u3c6MJAcnmIIwxrVpVdQ0LNxfwv59kszn/MHf+sB83TuiFiLCtsJRrX17O9qIyHrx4ED8ZEXzX\nU9kchDEmaIWFhjApvT2j0pL43dtrePjTbNbsPMhlGV349ZurAXj12lGM6RWcK7bWxxKEMSYoxESG\n8dT0oQzp2pY/f7KRT9fvpXdqLP+4OoPuSTFuhxeQLEEYY4KGiPCL8T0Z2CmBuVn7uPUHfYiPsrOV\n6mIJwhgTdMb0SrIhpUawe1IbY4zxyxKEMcYYvyxBGGOM8csShDHGGL8sQRhjjPHLEoQxxhi/LEEY\nY4zxyxKEMcYYv1rNYn0iUgBs97MrATjYQJnvtr/nx/5NBgpPMkR/cTS2TkNtqKs9/uo42Yb69tf3\nf157u6HnbrShKX6OfJ+fbBuc/DmqvV3fdwECsw2NaU+gfZ8bu+3Ud6G7qvq/+baqtuoHMLOhMt9t\nf899/s1syjgaW6ehNtTVnjra4lgb6ttf3/95Yz4Dt9vQFD9HTdEGJ3+OGhm3b1nAtaEx7Qm073Nj\nt5v7u6CqQTHE9EEjyj5o4Lm/12iKOBpbp6E21NWe+uqcjIZeo7799f2f195uzPOTdbJtaIqfo8a8\nf0Oc/Dmqvd2avgu+zwOtDY3dbu7vQusZYmoOIpKpdayb3lJYGwKDtcF9LT1+cL4NwdCDaEoz3Q6g\nCVgbAoO1wX0tPX5wuA3WgzDGGOOX9SCMMcb4ZQnCGGOMX0GbIETkBRHJF5F1J3HscBFZKyI5IvJ/\nIiI++24WkY0isl5EHm7aqL8XR5O3QUT+ICK7RGSV93Fe00d+XByOfA7e/b8WERWR5KaL2G8cTnwO\nD4jIGu9n8JmIdGr6yL+NwYn4H/F+D9aIyLsi0rbpIz8uDifacJn3e1wjIo5NBJ9K7HW83tUistn7\nuNqnvN7vi18new5tS38AZwLDgHUncewyYDQgwCfAud7yicBcINK7ndoC2/AH4M6W/Dl493UF5uC5\neDK5pbUBiPepcwvwtxYW/zlAmPf5/wL/2wI/g/5AP2ABkBFosXvj6lGrLBHI9f7bzvu8XX3trO8R\ntD0IVV0IFPuWiUgvEflURFaIyJcikl77OBHpiOfLu0Q9/+uvABd5d98IPKSq5d73yG+BbWhWDrbh\nceC3gONnYTjRBlU95FM1Bgfb4VD8n6lqlbfqEqCLU/E72IYsVc12Mu5Tib0OPwQ+V9ViVd0PfA5M\nOdnvfNAmiDrMBG5W1eHAncCzfup0BvJ8tvO8ZQB9gfEislRE/iMiIxyN1r9TbQPATd6hgRdEpJ1z\nodbplNogIlOBXaq62ulA63HKn4OIPCgiO4GfAr93MFZ/muLn6Jhr8PzF2tyasg3NrTGx+9MZ2Omz\nfaw9J9XOsEa+aasnIrHAWOBNn6G5yBN8mTA8XbvRwAjgDRHp6c3YjmuiNvwVeADPX6wPAI/h+YI3\ni1Ntg4hEA/8PzxCHK5roc0BV7wHuEZG7gZuA+5osyHo0Vfze17oHqAJeb5roGv2+TdaG5lZf7CLy\nc+BWb1lv4GMRqQC2qurFTR2LJYjvhAAHVHWIb6GIhAIrvJvv4/kF6ttd7gLs8j7PA97xJoRlIlKD\nZzGtAicD93HKbVDVfT7HPQd86GTAfpxqG3oBacBq75erC/CNiIxU1b0Ox35MU/ws+Xod+JhmShA0\nUfwiMgM4H5jcXH8k+Wjqz6A5+Y0dQFVfBF4EEJEFwAxV3eZTZRdwls92FzxzFbs4mXY6NfHSEh5A\nD3wmhoBFwGXe5wKcXsdxtSd7zvOW/xK43/u8L56unrSwNnT0qXM7MLulfQ616mzD4Ulqhz6HPj51\nbgbeamHxTwE2AClO/987/XOEw5PUJxs7dU9Sb8UzQd3O+zyxMe30G1dzfXiB9gD+CewBKvH85X8t\nnr88PwVWe3+4f1/HsRnAOmAL8DTfXZEeAbzm3fcNMKkFtuFVYC2wBs9fWB1bWhtq1dmG82cxOfE5\nvO0tX4NnUbXOLSz+HDx/IK3yPhw7C8vBNlzsfa1yYB8wJ5Bix0+C8JZf4/3/zwF+fiLfl9oPW2rD\nGGOMX3YWkzHGGL8sQRhjjPHLEoQxxhi/LEEYY4zxyxKEMcYYvyxBmFZNREqa+f2eF5EBTfRa1eJZ\nzXWdiHzQ0IqoItJWRH7VFO9tDNgd5UwrJyIlqhrbhK8Xpt8tQuco39hF5GVgk6o+WE/9HsCHqjqo\nOeIzrZ/1IEzQEZEUEXlbRJZ7H2d4y0eKyGIRWSkii0Skn7d8hoi8LyLzgC9E5CwRWSAib4nnngev\nH1tb31ue4X1e4l1wb7WILBGR9t7yXt7ttSLyx0b2chbz3WKEsSLyhYh8432Nqd46DwG9vL2OR7x1\nf+Nt4xoR+Z8m/G80QcAShAlGTwKPq+oI4FLgeW/5RmC8qg7Fs3rqn3yOGQb8WFUneLeHArcBA4Ce\nwBl+3icGWKKqpwMLget83v9JVR3M8Sts+uVdP2gynivbAY4CF6vqMDz3IHnMm6DuArao6hBV/Y2I\nnAP0AUYCQ4DhInJmQ+9nzDG2WJ8JRj8ABvislBnvXUEzAXhZRPrgWc023OeYz1XVd83+ZaqaByAi\nq/CspfNVrfep4LvFDlcAZ3ufj+G7tfhnAY/WEWcb72t3BrLwrO0PnrV0/uT9ZV/j3d/ez/HneB8r\nvduxeBLGwjrez5jjWIIwwSgEGK2qR30LReRpYL6qXuwdz1/gs7u01muU+zyvxv93qVK/m+Srq059\njqjqEO8S5nOA/wL+D8/9IVKA4apaKSLbgCg/xwvwZ1X9+wm+rzGADTGZ4PQZnhVSARCRY8sqJ/Dd\nEsgzHHz/JXiGtgCmNVRZVcvw3Hb01yIShifOfG9ymAh091Y9DMT5HDoHuMbbO0JEOotIahO1wQQB\nSxCmtYsWkTyfxx14ftlmeCduN+BZph3gYeDPIrISZ3vXtwF3iMgaPDd9OdjQAaq6Es/KrtPx3B8i\nQ0TWAj/DM3eCqhYBX3tPi31EVT/DM4S12Fv3LY5PIMbUy05zNaaZeYeMjqiqisg0YLqqTm3oOGOa\nm81BGNP8hgNPe888OkAz3tLVmBNhPQhjjDF+2RyEMcYYvyxBGGOM8csShDHGGL8sQRhjjPHLEoQx\nxhi//j89pDBRYMjIuAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mQwBPnW-VjkY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 197
        },
        "outputId": "b45aaff2-be53-4448-d2ff-f54e25733bdd"
      },
      "source": [
        "learn.fit_one_cycle(5, 1e-3, moms=(0.8,0.7))"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.424120</td>\n",
              "      <td>0.399041</td>\n",
              "      <td>0.815960</td>\n",
              "      <td>12:43</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.362391</td>\n",
              "      <td>0.291880</td>\n",
              "      <td>0.880840</td>\n",
              "      <td>12:27</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.312827</td>\n",
              "      <td>0.259725</td>\n",
              "      <td>0.892560</td>\n",
              "      <td>11:26</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.279835</td>\n",
              "      <td>0.236507</td>\n",
              "      <td>0.904760</td>\n",
              "      <td>12:35</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.271583</td>\n",
              "      <td>0.236181</td>\n",
              "      <td>0.906480</td>\n",
              "      <td>12:50</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j1mPOxdWGjEQ",
        "colab_type": "text"
      },
      "source": [
        "Future Experiments: \n",
        "\n",
        "I think it's worth exploring the effect of un-freezing all the LSTM layers and comparing the effect. \n",
        "\n",
        "Also have to perform a solid grid search to fine tune the beta value. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JxMVwSnFWE3b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}