lm3-portuguese-classifier-TCU-jurisprudencia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (MultiFiT) Portuguese Text Classifier on TCU jurisprudência dataset\n",
    "### MultiFiT configuration\n",
    "- **Architecture 4 QRNN with 1550 hidden parameters by layer, SentencePiece tokenizer (15 000 tokens)**\n",
    "- **Hyperparameters and training method from the MultiFiT paper**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Author: [Pierre Guillou](https://www.linkedin.com/in/pierreguillou)\n",
    "- Date: **edition of October 15, 2019** (initial publication on September 2019)\n",
    "- Post in medium: [link](https://medium.com/@pierre_guillou/nlp-fastai-portuguese-language-model-980c8ec75362)\n",
    "- Ref: [Fastai v1](https://docs.fast.ai/) (Deep Learning library on PyTorch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Warning (15/10/2019)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This notebook is a modified version of the v1 published in September 2019.** Indeed (thanks to [David Vieira](https://medium.com/@davidhsv/ol%C3%A1-pierre-tudo-bom-2bc8ae36dc14)), we noticed that the fine-tuning of the LM and classifier did not use the SentencePiece model and vocab trained for the General Portuguese Language Model ([lm3-portuguese.ipynb](https://github.com/piegu/language-models/blob/master/lm3-portuguese.ipynb)).\n",
    "\n",
    "For example, the code used to create the fine-tuned Portuguese forward LM was : \n",
    "\n",
    "```data_lm = (TextList.from_df(df_trn_val, path, cols=reviews, \n",
    "                            processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_for_lm()           \n",
    "    .databunch(bs=bs, num_workers=1))```\n",
    "    \n",
    "It has been corrected by using the [SPProcessor.load()](https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L481) function:\n",
    "\n",
    "```data_lm = (TextList.from_df(df_trn_val, path, cols=reviews, processor=SPProcessor.load(dest))\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_for_lm()           \n",
    "    .databunch(bs=bs, num_workers=1))```\n",
    "    \n",
    "Therefore, we retrained the fine-tuned Portuguese forward LM and the classifier on TCU jurisprudência dataset and **got better results! :-)** (see the Results paragraph to get all results)\n",
    "\n",
    "- **(fine-tuned) Language Model**   \n",
    "    - forward : (accuracy) **51.56%** instead of 44.66% | (perplexity) 11.38 instead of 15.97\n",
    "    - backward: (accuracy) **52.15%** instead of 44.97% | (perplexity) 12.54 instead of 18.73\n",
    "\n",
    "- **(fine-tuned) Text Classifier**\n",
    "    - **Accuracy** (ensemble) **97.95%** instead of 97.39%\n",
    "    - **f1 score** (ensemble): **0.9795** instead of 0.9737"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview\n",
    "\n",
    "According to this new article \"[MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://arxiv.org/abs/1909.04761)\" (September 10, 2019), the QRNN architecture and the SentencePiece tokenizer give better results than AWD-LSTM and the spaCy tokenizer respectively. \n",
    "\n",
    "Therefore, they have been used in this notebook to **fine-tune a Portuguese Bidirectional Language Model** by Transfer Learning of a Portuguese Bidirectional Language Model (with the QRNN architecture and the SentencePiece tokenizer, too) trained on a Wikipedia corpus of 100 millions tokens ([lm3-portuguese.ipynb](https://github.com/piegu/language-models/blob/master/lm3-portuguese.ipynb)). \n",
    "\n",
    "This Portuguese Bidirectional Language Model has been **fine-tuned on the [tcu_jurisp_reduzido.csv dataset about TCU jurisprudência](https://github.com/fastai-bsb/nlp-tcu-enunciados/blob/master/tcu_jurisp_reduzido.csv?raw=true)\"** and **its encoder part has been transfered to a text classifier which has been finally trained on this corpus**.\n",
    "\n",
    "This process **LM General --> LM fine-tuned --> Classifier fine-tuned** is called [ULMFiT](http://nlp.fast.ai/category/classification.html) but we trained our 3 models with the hyperparameters values and method of the [MultiFiT](https://arxiv.org/abs/1909.04761) paper that are given at the end of the MultiFiT paper."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameters values\n",
    "\n",
    "- Language Model\n",
    "    - (batch size) bs = 50\n",
    "    - (QRNN) 4 QRNN (default: 3) with 1550 hidden parameters each one (default: 1152)\n",
    "    - (SentencePiece) vocab of 15000 tokens\n",
    "    - (dropout) mult_drop = 1.0\n",
    "    - (weight decay) wd = 0.1\n",
    "    - (number of training epochs) 20 epochs\n",
    "    - (learning rate) modified version of 1-cycle learning rate schedule (Smith, 2018) that uses cosine instead of linear annealing, cyclical momentum and discriminative finetuning\n",
    "    - (loss) FlattenedLoss of weighted LabelSmoothingCrossEntropy\n",
    "    \n",
    "\n",
    "- Sentiment Classifier\n",
    "    - (batch size) bs = 18\n",
    "    - (SentencePiece) vocab of 15000 tokens\n",
    "    - (dropout) mult_drop = 0.3\n",
    "    - (weight decay) wd = 0.1\n",
    "    - (number of training epochs) 14 epochs (forward) and 19 epochs (backward)\n",
    "    - (learning rate) modified version of 1-cycle learning rate schedule (Smith, 2018) that uses cosine instead of linear annealing, cyclical momentum and discriminative finetuning\n",
    "    - (loss) FlattenedLoss of weighted LabelSmoothingCrossEntropy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We can conclude that this Bidirectional Portuguese LM model using the MultiFiT configuration is a good model to perform text classification but with about 46 millions of parameters, it is far from being a LM that can gan compete with [GPT-2](https://openai.com/blog/better-language-models/) or [BERT](https://arxiv.org/abs/1810.04805) in NLP tasks like text generation.**\n",
    "    \n",
    "    \n",
    "- **About the data**: the dataset [tcu_jurisp_reduzido.csv](https://github.com/fastai-bsb/nlp-tcu-enunciados/blob/master/tcu_jurisp_reduzido.csv?raw=true) about \"TCU jurisprudência\" is unbalanced. Therefore, we used a weighted loss function (FlattenedLoss of weighted LabelSmoothingCrossEntropy).\n",
    "    - number of texts: 10263\n",
    "        - class 0: 3468 (33.79%)\n",
    "        - class 1: 2723 (26.53%)\n",
    "        - class 2: 2297 (22.38%)\n",
    "        - class 3: 1775 (17.3%)\n",
    "\n",
    "\n",
    "- **(fine-tuned) Language Model**   \n",
    "    - forward : (accuracy) 51.56% | (perplexity) 11.38\n",
    "    - backward: (accuracy) 52.15% | (perplexity) 12.54\n",
    "    \n",
    "\n",
    "- **(fine-tuned) Text Classifier**\n",
    "\n",
    "    - **Accuracy**\n",
    "        - forward : (global) 97.08% | (class 0) 98.49% | (class 1) 98.24% | (class 2) 96.71% | (class 3) 93.40%\n",
    "        - backward: (global) 97.07% | (class 0) 99.10% | (class 1) 97.89% | (class 2) 96.71% | (class 3) 92.89%\n",
    "        - ensemble: (global) **97.95%** | (class 0) **99.40%** | (class 1) **99.30%** | (class 2) **97.18%** | (class 3) **94.42%**\n",
    "\n",
    "    - **f1 score**\n",
    "        - forward: 0.9707\n",
    "        - backward: 0.9708\n",
    "        - ensemble: **0.9795**\n",
    "\n",
    "(neg = negative reviews | pos = positive reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from fastai.callbacks import *\n",
    "\n",
    "import matplotlib.cm as cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\r\n",
      "```text\r\n",
      "=== Software === \r\n",
      "python        : 3.7.4\r\n",
      "fastai        : 1.0.57\r\n",
      "fastprogress  : 0.1.21\r\n",
      "torch         : 1.2.0\r\n",
      "nvidia driver : 410.104\r\n",
      "torch cuda    : 10.0.130 / is available\r\n",
      "torch cudnn   : 7602 / is enabled\r\n",
      "\r\n",
      "=== Hardware === \r\n",
      "nvidia gpus   : 1\r\n",
      "torch devices : 1\r\n",
      "  - gpu0      : 16130MB | Tesla V100-SXM2-16GB\r\n",
      "\r\n",
      "=== Environment === \r\n",
      "platform      : Linux-4.9.0-9-amd64-x86_64-with-debian-9.9\r\n",
      "distro        : #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11)\r\n",
      "conda env     : base\r\n",
      "python        : /opt/anaconda3/bin/python\r\n",
      "sys.path      : /home/jupyter/tutorials/fastai/course-nlp\r\n",
      "/opt/anaconda3/lib/python37.zip\r\n",
      "/opt/anaconda3/lib/python3.7\r\n",
      "/opt/anaconda3/lib/python3.7/lib-dynload\r\n",
      "/opt/anaconda3/lib/python3.7/site-packages\r\n",
      "/opt/anaconda3/lib/python3.7/site-packages/IPython/extensions\r\n",
      "```\r\n",
      "\r\n",
      "Please make sure to include opening/closing ``` when you paste into forums/github to make the reports appear formatted as code sections.\r\n",
      "\r\n",
      "Optional package(s) to enhance the diagnostics can be installed with:\r\n",
      "pip install distro\r\n",
      "Once installed, re-run this utility to get the additional information\r\n"
     ]
    }
   ],
   "source": [
    "!python -m fastai.utils.show_install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bs=48\n",
    "# bs=24\n",
    "bs=50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.set_device(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = Config.data_path()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will create a `{lang}wiki` folder, containing a `{lang}wiki` text file with the wikipedia contents. (For other languages, replace `{lang}` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang = 'pt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = f'{lang}wiki'\n",
    "path = data_path/name\n",
    "path.mkdir(exist_ok=True, parents=True)\n",
    "\n",
    "lm_fns3 = [f'{lang}_wt_sp15_multifit', f'{lang}_wt_vocab_sp15_multifit']\n",
    "lm_fns3_bwd = [f'{lang}_wt_sp15_multifit_bwd', f'{lang}_wt_vocab_sp15_multifit_bwd']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "@np_func\n",
    "def f1(inp,targ): return f1_score(targ, np.argmax(inp, axis=-1), average='weighted')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# source: https://github.com/fastai/fastai/blob/master//fastai/layers.py#L300:7\n",
    "# blog: https://bfarzin.github.io/Label-Smoothing/\n",
    "class WeightedLabelSmoothingCrossEntropy(nn.Module):\n",
    "    def __init__(self, weight, eps:float=0.1, reduction='mean'):\n",
    "        super().__init__()\n",
    "        self.weight,self.eps,self.reduction = weight,eps,reduction\n",
    "        \n",
    "    def forward(self, output, target):\n",
    "        c = output.size()[-1]\n",
    "        log_preds = F.log_softmax(output, dim=-1)\n",
    "        if self.reduction=='sum': loss = -log_preds.sum()\n",
    "        else:\n",
    "            loss = -log_preds.sum(dim=-1)\n",
    "            if self.reduction=='mean':  loss = loss.mean()\n",
    "        return loss*self.eps/c + (1-self.eps) * F.nll_loss(log_preds, target, weight=self.weight, reduction=self.reduction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')  # \"error\", \"ignore\", \"always\", \"default\", \"module\" or \"on"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TCU jurisprudência:\n",
    "- reduzido: https://github.com/fastai-bsb/nlp-tcu-enunciados/blob/master/tcu_jurisp_reduzido.csv\n",
    "- completo: https://github.com/fastai-bsb/nlp-tcu-enunciados/blob/master/tcu_jurisp.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "from converter import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create TCU folder\n",
    "name_data = 'TCU'\n",
    "path_data = data_path/name_data\n",
    "path_data.mkdir(exist_ok=True, parents=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 48 ms, sys: 20 ms, total: 68 ms\n",
      "Wall time: 1.41 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(PosixPath('/home/jupyter/.fastai/data/TCU/tcu_jurisp.csv'),\n",
       " <http.client.HTTPMessage at 0x7f9a1cec6b50>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "# Download each file from url and save it locally under file_name\n",
    "\n",
    "url = 'https://github.com/fastai-bsb/nlp-tcu-enunciados/blob/master/tcu_jurisp_reduzido.csv?raw=true'\n",
    "file_name = 'tcu_jurisp_reduzido.csv'\n",
    "url_file = path_data/file_name\n",
    "urllib.request.urlretrieve(url, url_file)\n",
    "\n",
    "url = 'https://raw.githubusercontent.com/fastai-bsb/nlp-tcu-enunciados/master/tcu_jurisp.csv'\n",
    "file_name = 'tcu_jurisp.csv'\n",
    "url_file = path_data/file_name\n",
    "urllib.request.urlretrieve(url, url_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/TCU/tcu_jurisp_reduzido_preprocessed.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/TCU/tcu_jurisp_reduzido.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/TCU/tcu_jurisp.csv')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_data.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text,labels\r\n",
      "\"A medida cautelar do TCU que determina a suspensão de licitação por falhas no edital não impede o órgão ou a entidade de rever seu ato convocatório, valendo-se do poder de autotutela (art. 49 da Lei 8.666/1993 c/c o art. 9º da Lei 10.520/2002) , com o objetivo de, antecipando-se a eventual deliberação do Tribunal, promover de modo próprio a anulação da licitação e o refazimento do edital, livre dos vícios apontados.\",3\r\n",
      "\"A retenção de recursos pela Administração com vistas ao ressarcimento do prejuízo ou a existência de ação judicial para o reconhecimento do dano ao erário não constituem óbices ao prosseguimento da tomada de contas especial no TCU. Ocorrendo ressarcimento em uma instância, basta que o responsável apresente essa comprovação perante o juízo de execução para evitar o duplo pagamento.</p><p><b>Enunciado </b>\",3\r\n",
      "\"Para fins de admissibilidade de recurso de revisão, considera-se documento novo todo aquele ainda não examinado no processo.\",3\r\n"
     ]
    }
   ],
   "source": [
    "!head -n4 {path_data.ls()[0]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to solve display error of pandas dataframe\n",
    "get_ipython().config.get('IPKernelApp', {})['parent_appname'] = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10263\n",
      "Counter({0: 3468, 1: 2723, 2: 2297, 3: 1775})\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>labels</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>A medida cautelar do TCU que determina a suspe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>A retenção de recursos pela Administração com ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>Para fins de admissibilidade de recurso de rev...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>Fotografias não têm pleno valor probatório, so...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>Não cabe instauração de tomada de contas espec...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  labels                                               text\n",
       "0           0       3  A medida cautelar do TCU que determina a suspe...\n",
       "1           1       3  A retenção de recursos pela Administração com ...\n",
       "2           2       3  Para fins de admissibilidade de recurso de rev...\n",
       "3           3       3  Fotografias não têm pleno valor probatório, so...\n",
       "4           4       3  Não cabe instauração de tomada de contas espec..."
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(path_data/'tcu_jurisp_reduzido.csv', encoding='utf-8')\n",
    "print(len(df))\n",
    "print(Counter(df.labels))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13258\n",
      "Counter({0: 3468, 1: 2723, 2: 2297, 3: 1775, 4: 932, 5: 673, 6: 572, 7: 343, 8: 337, 9: 138})\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>labels</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>O TCU tem competência para fiscalizar a aplica...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>Não compete ao TCU apreciar, para fins de regi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>Compete ao TCU a apreciação da constitucionali...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>É possível a expedição de determinação pelo TC...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>O TCU não tem competência, no âmbito do Progra...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  labels                                               text\n",
       "0           0       6  O TCU tem competência para fiscalizar a aplica...\n",
       "1           1       6  Não compete ao TCU apreciar, para fins de regi...\n",
       "2           2       6  Compete ao TCU a apreciação da constitucionali...\n",
       "3           3       6  É possível a expedição de determinação pelo TC...\n",
       "4           4       6  O TCU não tem competência, no âmbito do Progra..."
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(path_data/'tcu_jurisp.csv', encoding='utf-8')\n",
    "print(len(df))\n",
    "print(Counter(df.labels))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysis (reduzido file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10263\n",
      "Counter({0: 3468, 1: 2723, 2: 2297, 3: 1775})\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>labels</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>A medida cautelar do TCU que determina a suspe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>A retenção de recursos pela Administração com ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>Para fins de admissibilidade de recurso de rev...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>Fotografias não têm pleno valor probatório, so...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>Não cabe instauração de tomada de contas espec...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  labels                                               text\n",
       "0           0       3  A medida cautelar do TCU que determina a suspe...\n",
       "1           1       3  A retenção de recursos pela Administração com ...\n",
       "2           2       3  Para fins de admissibilidade de recurso de rev...\n",
       "3           3       3  Fotografias não têm pleno valor probatório, so...\n",
       "4           4       3  Não cabe instauração de tomada de contas espec..."
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(path_data/'tcu_jurisp_reduzido.csv', encoding='utf-8')\n",
    "print(len(df))\n",
    "print(Counter(df.labels))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# columns names\n",
    "reviews = \"text\"\n",
    "label = \"labels\"\n",
    "\n",
    "# keep columns\n",
    "df2 = df[[reviews,label]].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(orginal csv) number of all reviews: 10263\n",
      "there is no empty review.\n",
      "0 reviews with nan label were deleted\n",
      "\n",
      "number of text of class 0: 3468 (33.79%)\n",
      "number of text of class 1: 2723 (26.53%)\n",
      "number of text of class 2: 2297 (22.38%)\n",
      "number of text of class 3: 1775 (17.3%)\n",
      "\n",
      "(final) number of all texts: 10263\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>A medida cautelar do TCU que determina a suspe...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>A retenção de recursos pela Administração com ...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Para fins de admissibilidade de recurso de rev...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Fotografias não têm pleno valor probatório, so...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Não cabe instauração de tomada de contas espec...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  labels\n",
       "0  A medida cautelar do TCU que determina a suspe...       3\n",
       "1  A retenção de recursos pela Administração com ...       3\n",
       "2  Para fins de admissibilidade de recurso de rev...       3\n",
       "3  Fotografias não têm pleno valor probatório, so...       3\n",
       "4  Não cabe instauração de tomada de contas espec...       3"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of reviews\n",
    "print(f'(orginal csv) number of all reviews: {len(df2)}')\n",
    "\n",
    "# keep not null reviews\n",
    "## delete nan reviews\n",
    "empty_nan = (df2[reviews].isnull()).sum()\n",
    "df2 = df2[df2[reviews].notnull()]\n",
    "## delete empty reviews\n",
    "list_idx_none = []\n",
    "for idxs, row in df2.iterrows():\n",
    "    if row[reviews].strip() == \"\":\n",
    "        df2.drop(idxs, axis=0, inplace=True)\n",
    "        list_idx_none.append(idxs)\n",
    "empty_none = len(list_idx_none)\n",
    "## print results\n",
    "empty = empty_nan+empty_none\n",
    "if empty != 0:\n",
    "    print(f'{empty} empty reviews were deleted')\n",
    "else:\n",
    "    print('there is no empty review.')\n",
    "\n",
    "# # check that there is no twice the same review\n",
    "# # keep the first of unique review_id reviews\n",
    "# same = len(df2) - len(df2[idx].unique())\n",
    "# if same != 0:\n",
    "#     df2.drop_duplicates(subset=[idx], inplace=True)\n",
    "#     print(f'from the {same} identical reviews ids, only the first one has been kept.')\n",
    "# else:\n",
    "#     print('there is no identical review id.')\n",
    "\n",
    "## delete nan labels\n",
    "empty_label_nan = (df2[label].isnull()).sum()\n",
    "df2 = df2[df2[label].notnull()]\n",
    "print(f'{empty_label_nan} reviews with nan label were deleted')\n",
    "\n",
    "# number of reviews by class\n",
    "counter = Counter(df2[label])\n",
    "clas_0, clas_1, clas_2, clas_3 = counter[0], counter[1], counter[2], counter[3]\n",
    "num = len(df2)\n",
    "pc_clas_0, pc_clas_1 = round((clas_0/num)*100,2), round((clas_1/num)*100,2)\n",
    "pc_clas_2, pc_clas_3 = round((clas_2/num)*100,2), round((clas_3/num)*100,2)\n",
    "print(f'\\nnumber of text of class 0: {clas_0} ({pc_clas_0}%)')\n",
    "print(f'number of text of class 1: {clas_1} ({pc_clas_1}%)')\n",
    "print(f'number of text of class 2: {clas_2} ({pc_clas_2}%)')\n",
    "print(f'number of text of class 3: {clas_3} ({pc_clas_3}%)')\n",
    "print(f'\\n(final) number of all texts: {num}')  \n",
    "\n",
    "# convert HTML caracters to normal letters\n",
    "df2[reviews] = df2[reviews].apply(convert)\n",
    "\n",
    "df2.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "number of text of class 0: 3468 (33.79%)\n",
      "number of text of class 1: 2723 (26.53%)\n",
      "number of text of class 2: 2297 (22.38%)\n",
      "number of text of class 3: 1775 (17.3%)\n",
      "\n",
      "(final) number of all texts: 10263\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAARO0lEQVR4nO3dbaic533n8e+vsvNAEtby+iSokliZVn1wCpXDWdkQKKmT2rLzQi40YL9IRHBRCzYk0C1V+sZ5WIMXkhgCqUHF2sgljVc0KRGJWld1HUKgtnWcVRTLitdnHW98KmGdrhwnJtTF3v++OJfoRD4Pcx4006Pr+4Fh7vt/X/fM/x7Eb+5zzT2jVBWSpD78wrgbkCSNjqEvSR0x9CWpI4a+JHXE0Jekjlw27gYWc9VVV9W2bdvG3YYkrStPPvnkP1fVxHzb/l2H/rZt25iamhp3G5K0riT5Pwttc3pHkjpi6EtSRwx9SeqIoS9JHVky9JO8JckTSb6X5GSST7X6l5L8MMnxdtvR6knyhSTTSU4kec/AY+1J8my77bl4hyVJms8wV++8CtxQVa8kuRz4TpK/adv+uKr+6oLxNwPb2+064H7guiRXAncDk0ABTyY5XFUvrcWBSJKWtuSZfs15pa1e3m6L/TTnbuDBtt9jwBVJNgE3AUer6lwL+qPArtW1L0lajqHm9JNsSHIcOMtccD/eNt3TpnDuS/LmVtsMvDCw+0yrLVS/8Ln2JplKMjU7O7vMw5EkLWao0K+q16tqB7AF2JnkN4BPAL8G/GfgSuBP2vDM9xCL1C98rv1VNVlVkxMT836hTJK0Qsv6Rm5V/TjJt4BdVfXZVn41yX8H/ktbnwG2Duy2BTjd6u+7oP6t5bcsqQfb9n1z3C2M1fP3fvCiPO4wV+9MJLmiLb8V+ADwgzZPT5IAtwJPtV0OAx9pV/FcD7xcVWeAh4Ebk2xMshG4sdUkSSMyzJn+JuBgkg3MvUkcqqpvJPmHJBPMTdscB/6wjT8C3AJMAz8DPgpQVeeSfAY41sZ9uqrOrd2hSJKWsmToV9UJ4Np56jcsML6AOxfYdgA4sMweJUlrxG/kSlJHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHVkydBP8pYkTyT5XpKTST7V6lcneTzJs0n+R5I3tfqb2/p0275t4LE+0erPJLnpYh2UJGl+w5zpvwrcUFW/CewAdiW5HvhvwH1VtR14Cbijjb8DeKmqfhm4r40jyTXAbcC7gV3AnyXZsJYHI0la3JKhX3NeaauXt1sBNwB/1eoHgVvb8u62Ttv+/iRp9Yeq6tWq+iEwDexck6OQJA1lqDn9JBuSHAfOAkeB/w38uKpea0NmgM1teTPwAkDb/jLwHwfr8+wz+Fx7k0wlmZqdnV3+EUmSFjRU6FfV61W1A9jC3Nn5r883rN1ngW0L1S98rv1VNVlVkxMTE8O0J0ka0rKu3qmqHwPfAq4HrkhyWdu0BTjdlmeArQBt+38Azg3W59lHkjQCw1y9M5Hkirb8VuADwCngUeD32rA9wNfb8uG2Ttv+D1VVrX5bu7rnamA78MRaHYgkaWmXLT2ETcDBdqXNLwCHquobSZ4GHkryX4H/CTzQxj8A/EWSaebO8G8DqKqTSQ4BTwOvAXdW1etreziSpMUsGfpVdQK4dp76c8xz9U1V/QvwoQUe6x7gnuW3KUlaC34jV5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHhvntHUkrsG3fN8fdwlg9f+8Hx92C5uGZviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdWTJ0E+yNcmjSU4lOZnkY63+yST/lOR4u90ysM8nkkwneSbJTQP1Xa02nWTfxTkkSdJChvkZhteAP6qq7yZ5B/BkkqNt231V9dnBwUmuAW4D3g38IvD3SX6lbf4i8DvADHAsyeGqenotDkSStLQlQ7+qzgBn2vJPk5wCNi+yy27goap6FfhhkmlgZ9s2XVXPASR5qI019CVpRJY1p59kG3At8Hgr3ZXkRJIDSTa22mbghYHdZlptobokaUSGDv0kbwe+Cny8qn4C3A/8ErCDub8EPnd+6Dy71yL1C59nb5KpJFOzs7PDtidJGsJQoZ/kcuYC/8tV9TWAqnqxql6vqv8H/Dn/NoUzA2wd2H0LcHqR+s+pqv1VNVlVkxMTE8s9HknSIoa5eifAA8Cpqvr8QH3TwLDfBZ5qy4eB25K8OcnVwHbgCeAYsD3J1UnexNyHvYfX5jAkScMY5uqd9wIfBr6f5Hir/Slwe5IdzE3RPA/8AUBVnUxyiLkPaF8D7qyq1wGS3AU8DGwADlTVyTU8FknSEoa5euc7zD8ff2SRfe4B7pmnfmSx/SRJF5ffyJWkjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSPDfDlLndq275vjbmGsnr/3g+NuQVpznulLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1ZMnQT7I1yaNJTiU5meRjrX5lkqNJnm33G1s9Sb6QZDrJiSTvGXisPW38s0n2XLzDkiTNZ5gz/deAP6qqXweuB+5Mcg2wD3ikqrYDj7R1gJuB7e22F7gf5t4kgLuB64CdwN3n3ygkSaOxZOhX1Zmq+m5b/ilwCtgM7AYOtmEHgVvb8m7gwZrzGHBFkk3ATcDRqjpXVS8BR4Fda3o0kqRFLWtOP8k24FrgceBdVXUG5t4YgHe2YZuBFwZ2m2m1heoXPsfeJFNJpmZnZ5fTniRpCUOHfpK3A18FPl5VP1ls6Dy1WqT+84Wq/VU1WVWTExMTw7YnSRrCUKGf5HLmAv/LVfW1Vn6xTdvQ7s+2+gywdWD3LcDpReqSpBEZ5uqdAA8Ap6rq8wObDgPnr8DZA3x9oP6RdhXP9cDLbfrnYeDGJBvbB7g3tpokaUSG+Y/R3wt8GPh+kuOt9qfAvcChJHcAPwI+1LYdAW4BpoGfAR8FqKpzST4DHGvjPl1V59bkKCRJQ1ky9KvqO8w/Hw/w/nnGF3DnAo91ADiwnAYlSWvHb+RKUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOjLMN3LXrW37vjnuFsbq+Xs/OO4WJP0745m+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjqyZOgnOZDkbJKnBmqfTPJPSY632y0D2z6RZDrJM0luGqjvarXpJPvW/lAkSUsZ5kz/S8Cueer3VdWOdjsCkOQa4Dbg3W2fP0uyIckG4IvAzcA1wO1trCRphJb8aeWq+naSbUM+3m7goap6FfhhkmlgZ9s2XVXPASR5qI19etkdS5JWbDVz+nclOdGmfza22mbghYExM622UP0NkuxNMpVkanZ2dhXtSZIutNLQvx/4JWAHcAb4XKtnnrG1SP2Nxar9VTVZVZMTExMrbE+SNJ8V/c9ZVfXi+eUkfw58o63OAFsHhm4BTrflheqSpBFZ0Zl+kk0Dq78LnL+y5zBwW5I3J7ka2A48ARwDtie5OsmbmPuw9/DK25YkrcSSZ/pJvgK8D7gqyQxwN/C+JDuYm6J5HvgDgKo6meQQcx/QvgbcWVWvt8e5C3gY2AAcqKqTa340kqRFDXP1zu3zlB9YZPw9wD3z1I8AR5bVnSRpTfmNXEnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOLBn6SQ4kOZvkqYHalUmOJnm23W9s9ST5QpLpJCeSvGdgnz1t/LNJ9lycw5EkLWaYM/0vAbsuqO0DHqmq7cAjbR3gZmB7u+0F7oe5NwngbuA6YCdw9/k3CknS6CwZ+lX1beDcBeXdwMG2fBC4daD+YM15DLgiySbgJuBoVZ2rqpeAo7zxjUSSdJGtdE7/XVV1BqDdv7PVNwMvDIybabWF6m+QZG+SqSRTs7OzK2xPkjSftf4gN/PUapH6G4tV+6tqsqomJyYm1rQ5SerdSkP/xTZtQ7s/2+ozwNaBcVuA04vUJUkjtNLQPwycvwJnD/D1gfpH2lU81wMvt+mfh4Ebk2xsH+De2GqSpBG6bKkBSb4CvA+4KskMc1fh3AscSnIH8CPgQ234EeAWYBr4GfBRgKo6l+QzwLE27tNVdeGHw5Kki2zJ0K+q2xfY9P55xhZw5wKPcwA4sKzuJElrym/kSlJHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHVkVaGf5Pkk309yPMlUq12Z5GiSZ9v9xlZPki8kmU5yIsl71uIAJEnDW4sz/d+uqh1VNdnW9wGPVNV24JG2DnAzsL3d9gL3r8FzS5KW4WJM7+wGDrblg8CtA/UHa85jwBVJNl2E55ckLWC1oV/A3yV5MsneVntXVZ0BaPfvbPXNwAsD+8602s9JsjfJVJKp2dnZVbYnSRp02Sr3f29VnU7yTuBokh8sMjbz1OoNhar9wH6AycnJN2yXJK3cqs70q+p0uz8L/DWwE3jx/LRNuz/bhs8AWwd23wKcXs3zS5KWZ8Whn+RtSd5xfhm4EXgKOAzsacP2AF9vy4eBj7SreK4HXj4/DSRJGo3VTO+8C/jrJOcf5y+r6m+THAMOJbkD+BHwoTb+CHALMA38DPjoKp5bkrQCKw79qnoO+M156v8XeP889QLuXOnzSZJWz2/kSlJHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHVk5KGfZFeSZ5JMJ9k36ueXpJ6NNPSTbAC+CNwMXAPcnuSaUfYgST0b9Zn+TmC6qp6rqn8FHgJ2j7gHSepWqmp0T5b8HrCrqn6/rX8YuK6q7hoYsxfY21Z/FXhmZA2uvauAfx53E+uYr9/q+Pqtznp+/f5TVU3Mt+GyETeSeWo/965TVfuB/aNp5+JKMlVVk+PuY73y9VsdX7/VuVRfv1FP78wAWwfWtwCnR9yDJHVr1KF/DNie5OokbwJuAw6PuAdJ6tZIp3eq6rUkdwEPAxuAA1V1cpQ9jNglMU01Rr5+q+PrtzqX5Os30g9yJUnj5TdyJakjhr4kdcTQX2NJ3pLkiSTfS3IyyafG3dN6kmRrkkeTnGqv38fG3dN6k+RAkrNJnhp3L+vRpf5TMc7pr7EkAd5WVa8kuRz4DvCxqnpszK2tC0k2AZuq6rtJ3gE8CdxaVU+PubV1I8lvAa8AD1bVb4y7n/Wk/VTM/wJ+h7lLzI8Bt19K//48019jNeeVtnp5u/nOOqSqOlNV323LPwVOAZvH29X6UlXfBs6Nu4916pL/qRhD/yJIsiHJceAscLSqHh93T+tRkm3AtYCvn0ZlM/DCwPoMl9hJh6F/EVTV61W1g7lvHO9M4p/Yy5Tk7cBXgY9X1U/G3Y+6seRPxax3hv5FVFU/Br4F7BpzK+tK+yzkq8CXq+pr4+5HXbnkfyrG0F9jSSaSXNGW3wp8APjBeLtaP9oH4Q8Ap6rq8+PuR9255H8qxtBfe5uAR5OcYO4f0NGq+saYe1pP3gt8GLghyfF2u2XcTa0nSb4C/CPwq0lmktwx7p7Wi6p6DTj/UzGngEOX2k/FeMmmJHXEM31J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjry/wFgB763f8TqiQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_trn_val = df2.copy()\n",
    "\n",
    "# number of reviews by class\n",
    "counter = Counter(df_trn_val[label])\n",
    "clas_0, clas_1, clas_2, clas_3 = counter[0], counter[1], counter[2], counter[3]\n",
    "num = len(df_trn_val)\n",
    "pc_clas_0, pc_clas_1 = round((clas_0/num)*100,2), round((clas_1/num)*100,2)\n",
    "pc_clas_2, pc_clas_3 = round((clas_2/num)*100,2), round((clas_3/num)*100,2)\n",
    "print(f'\\nnumber of text of class 0: {clas_0} ({pc_clas_0}%)')\n",
    "print(f'number of text of class 1: {clas_1} ({pc_clas_1}%)')\n",
    "print(f'number of text of class 2: {clas_2} ({pc_clas_2}%)')\n",
    "print(f'number of text of class 3: {clas_3} ({pc_clas_3}%)')\n",
    "print(f'\\n(final) number of all texts: {num}') \n",
    "\n",
    "# plot histogram\n",
    "keys = list(df_trn_val[label].value_counts().keys())\n",
    "values = list(df_trn_val[label].value_counts().array)\n",
    "plt.bar(keys, values[::-1]) \n",
    "plt.xticks(keys, keys[::-1])\n",
    "# print(df_trn_val['label'].value_counts())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>A medida cautelar do TCU que determina a suspe...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>A retenção de recursos pela Administração com ...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Para fins de admissibilidade de recurso de rev...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Fotografias não têm pleno valor probatório, so...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Não cabe instauração de tomada de contas espec...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  labels\n",
       "0  A medida cautelar do TCU que determina a suspe...       3\n",
       "1  A retenção de recursos pela Administração com ...       3\n",
       "2  Para fins de admissibilidade de recurso de rev...       3\n",
       "3  Fotografias não têm pleno valor probatório, so...       3\n",
       "4  Não cabe instauração de tomada de contas espec...       3"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_trn_val.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trn_val.to_csv(path_data/'tcu_jurisp_reduzido_preprocessed.csv', index = None, header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning \"forward LM\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "name_data = 'TCU'\n",
    "path_data = data_path/name_data\n",
    "\n",
    "# Load csv\n",
    "df_trn_val = pd.read_csv(path_data/'tcu_jurisp_reduzido_preprocessed.csv')\n",
    "\n",
    "# columns names\n",
    "reviews = \"text\"\n",
    "label = \"labels\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/ptwiki/corpus2_100/tmp/spm.model'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/corpus2_100/tmp/spm.vocab')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dest = path/'corpus2_100'\n",
    "(dest/'tmp').ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1 s, sys: 156 ms, total: 1.16 s\n",
      "Wall time: 1.67 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = (TextList.from_df(df_trn_val, path, cols=reviews, processor=SPProcessor.load(dest))\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_for_lm()           \n",
    "    .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm.save(f'{path}/{lang}_databunch_lm_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(path, f'{lang}_databunch_lm_tcu_jurisp_reduzido_sp15_multifit_v2', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.14 s, sys: 1.22 s, total: 5.35 s\n",
      "Wall time: 4.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, config=config, pretrained_fnames=lm_fns3, drop_mult=1., \n",
    "                                  metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "46020150"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of model parameters\n",
    "sum([p.numel() for p in learn_lm.model.parameters()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SequentialRNN(\n",
       "  (0): AWD_LSTM(\n",
       "    (encoder): Embedding(15000, 400, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(15000, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): QRNN(\n",
       "        (layers): ModuleList(\n",
       "          (0): QRNNLayer(\n",
       "            (linear): WeightDropout(\n",
       "              (module): Linear(in_features=800, out_features=4650, bias=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "      (1): QRNN(\n",
       "        (layers): ModuleList(\n",
       "          (0): QRNNLayer(\n",
       "            (linear): WeightDropout(\n",
       "              (module): Linear(in_features=1550, out_features=4650, bias=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "      (2): QRNN(\n",
       "        (layers): ModuleList(\n",
       "          (0): QRNNLayer(\n",
       "            (linear): WeightDropout(\n",
       "              (module): Linear(in_features=1550, out_features=4650, bias=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "      (3): QRNN(\n",
       "        (layers): ModuleList(\n",
       "          (0): QRNNLayer(\n",
       "            (linear): WeightDropout(\n",
       "              (module): Linear(in_features=1550, out_features=1200, bias=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "      (3): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=15000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm.model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Change loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.loss_func = FlattenedLoss(LabelSmoothingCrossEntropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of LabelSmoothingCrossEntropy()"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm.loss_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAfyklEQVR4nO3deXBcZ53u8e+vN+2LJcu7HdnOahyyKSROCBUGQgKTCwQuVBi4FQIDNTMMa8G9c4uaYYBi2IYZZpga7uRCEu5luTUhUBACWVhCWBKC48TxksXxEkeyLEu2tm61en3vH30ky45ly7bO6dM+z6eqS92nl/PrlvT02+95+33NOYeIiERHrNoFiIhIsBT8IiIRo+AXEYkYBb+ISMQo+EVEIiZR7QLmYuHCha67u7vaZYiI1JTHH398yDnXdfT2mgj+7u5uNm7cWO0yRERqipm9cKzt6uoREYkYBb+ISMQo+EVEIkbBLyISMQp+EZGIUfCLiESMgl9EJGIU/CIiIbR/dJKvPPAsuwbT8/7YCn4RkRB6cXiCr/3yefpGsvP+2Ap+EZEQSueKADSm5n+CBQW/iEgIZbzgb65T8IuIRMJU8DfVxef9sRX8IiIhlM6VALX4RUQi43CLX8EvIhIJmXyRVCJGMj7/Ma3gFxEJoUyu6Es3Dyj4RURCKZMr0Zia/wO7oOAXEQmltFr8IiLRkskVfTmwCwp+EZFQUvCLiERMpatHffwiIpExkS/R5MM8PaDgFxEJpbS6ekREosM5p3H8IiJRMlkoU3bQqD5+EZFoSPs4JTMo+EVEQmd6gjYd3BURiYa0jzNzgoJfRCR0JvL+zcUPCn4RkdDxc/UtUPCLiISODu6KiETMVIu/UcEvIhIN0y1+jeoREYmGjLfQuvr4RUQiIpMvUpeIkfBhvV1Q8IuIhI6f8/SAj8FvZreb2QEz2zpj29vMbJuZlc2sx699i4jUMj8XYQF/W/x3AjcctW0r8BbgYR/3KyJS09K5kq/B79sjO+ceNrPuo7Y9DWBmfu1WRKTmZXxcfQtC3MdvZu83s41mtnFwcLDa5YiIBCaTL9Lo01BOCHHwO+duc871OOd6urq6ql2OiEhg0rV6cFdERE5N5eBuBLt6RESiasLng7t+Duf8HvAIcJ6Z9ZrZe83sJjPrBTYA95rZ/X7tX0SkFjnnyOT97erxc1TPO2a56od+7VNEpNZlCyXKzr9FWEBdPSIioeL36lug4BcRCZXpCdpSOrgrIhIJGbX4RUSixe/Vt0DBLyISKhN5tfhFRCIl7fXxR3KuHhGRKFIfv4hIxCj4RUQiZnocfxRn5xQRiaJMrkh9MkY85t+6JQp+EZEQSedKvg7lBAW/iEioTOT9XW8XFPwiIqGSyRV97d8HBb+ISKj4vfoWKPhFREIlkyv5uvoWKPhFREKlsuyiWvwiIpGRVh+/iEi0qMUvIhIh5bJjolDydYI2UPCLiIRGtlDC+bzeLij4RURCI4gJ2kDBLyISGkGsvgUKfhGR0JheaF3BLyISDdNTMuvgrohINGQCmIsfFPwiIqGRCWChdVDwi4iERmZ6oXUFv4hIJGTUxy8iEi1BrLcLCn4RkdDI5Io0puLEfFxvFxT8IiKhkQlg2UVQ8IuIhEY6V6Ip5W//Pij4RURCYyKAKZlBwS8iEhppBb+ISLRk8v4vtA4KfhGR0KgstK7gFxGJjPHJglr8IiJR4ZxjZKLAgsak7/tS8IuIhEA6V6RYdixoTPm+L9+C38xuN7MDZrZ1xrYOM3vQzHZ4Pxf4tX8RkVoyMlEAoK3GW/x3Ajccte1vgF84584BfuFdFhGJvOGJPEBtt/idcw8Dh47a/CbgW975bwFv9mv/IiK1ZKrFfyb28S92zvUDeD8XzXZDM3u/mW00s42Dg4OBFSgiUg1TLf72MzD458w5d5tzrsc519PV1VXtckREfDXV4m+v5a6eWQyY2VIA7+eBgPcvIhJK0y3+hjOvxf9j4Bbv/C3AjwLev4hIKI1MFGipT5CI+x/Lfg7n/B7wCHCemfWa2XuBLwDXmdkO4DrvsohI5I1M5APp3wfw7bvBzrl3zHLVa/zap4hIrRqeKAQylBNCfHBXRCRKRrKFQA7sgoJfRCQURibygYzhBwW/iEgoDGfygYzoAQW/iEjVFUtlxiaL6uoREYmKsckiEMx0DaDgFxGpusPTNYSoxW9ma82szjt/rZl9yMza/S1NRCQaRgKcpwfm3uK/GyiZ2dnAN4HVwHd9q0pEJEKGM1Mzc4aoxQ+UnXNF4Cbgq865jwJL/StLRCQ6RrLhDP6Cmb2Dyvw6P/G2BfOZRETkDDfV1RPE6lsw9+C/FdgAfM45t9vMVgPf9q8sEZHoGJ7IE48ZrfW+zaJzhDntxTm3HfgQgLdObotzThOsiYjMg+GJAu0NScwskP3NdVTPQ2bWamYdwGbgDjP7J39LExGJhtGJQmDdPDD3rp4259wY8BbgDufcZcBr/StLRCQ6hifygR3YhbkHf8JbMevtHD64KyIi86AyJXP4WvyfAe4Hdjrn/mhma4Ad/pUlIhIdlUVYgmvxz/Xg7l3AXTMu7wLe6ldRIiJRMuId3A3KXA/urjCzH5rZATMbMLO7zWyF38WJiJzpJgslsoUSC5rC18d/B5WF0pcBy4F7vG0iInIaRiYq39oNap4emHvwdznn7nDOFb3TnUCXj3WJiETC1MycYRzVM2Rm7zKzuHd6F3DQz8JERKJgusUftj5+4D1UhnLuB/qB/0plGgcRETkNIwHPxQ9zDH7n3F7n3Budc13OuUXOuTdT+TKXiIichmGvxb+gKXwt/mP52LxVISISUdOrbzWErMU/i2BmExIROYONZgvUJWI0pOKB7fN0gt/NWxUiIhE1nAl2nh44wTd3zWycYwe8AQ2+VCQiEiHDE4VAx/DDCYLfOdcSVCEiIlFUmacn2OA/na4eERE5TSPZQuBdPQp+EZEqCnpmTlDwi4hUjXOuMjOnunpERKIhnStSLLtAF2EBBb+ISNUcnplTXT0iIpFQjZk5QcEvIlI1w1WYix8U/CIiVTMy3eJX8IuIRIL6+EVEIubwzJxq8YuIRMLBdJ7W+gSJeLBRXJXgN7MPm9lWM9tmZh+pRg0iItXWN5JlWXvw810GHvxmth54H/AK4CLgRjM7J+g6RESqrW84y4oFjYHvtxot/guAR51zE865IvBr4KYq1CEiUjXOOfpGsqxYEIEWP7AVeJWZdZpZI/AGYOXRNzKz95vZRjPbODg4GHiRIiJ+Gs0WSOeK0Qh+59zTwBeBB4H7gM1A8Ri3u8051+Oc6+nq6gq4ShERf/UOZwFYHoU+fgDn3Dedc5c6514FHAJ2VKMOEZFqmQ7+KrT4j7sCl1/MbJFz7oCZrQLeAmyoRh0iItXSN1IJ/moc3K1K8AN3m1knUAA+4JwbrlIdIiJV0TecpSEZD3y6BqhS8DvnrqnGfkVEwqJ3eIIVCxows8D3rW/uiohUQd9Itir9+6DgFxGpit7hbFVG9ICCX0QkcOlckdFsoSoHdkHBLyISuL4qDuUEBb+ISOB6hycAqvKtXVDwi4gEbnoMv/r4RUSioXc4SyoeY2FzXVX2r+AXEQlY33BlKGcsFvwYflDwi4gErnekekM5QcEvIhK4Pu9bu9Wi4BcRCdBkocRQOq8Wv4hIVFRzOuYpCn4RkQBVczrmKQp+EZEAVftbu6DgFxEJVO/wBImYsbilOmP4QcEvIhKovpEsS9rqScSrF78KfhGRAPVVcTrmKQp+EZEA9Q5nq3pgFxT8IiKByRfLDIxPVvXALij4RUQC0z+axbnqTcc8RcEvIhKQXUMZAFZ1qKtHRCQStvWNArBuWWtV61Dwi4gEZEvfKN2djbTWJ6tah4JfRCQgW/vGWL+8rdplKPhFRIIwnMnTN5LlQgW/iEg0bN1X6d9Xi19EJCK2eAd21y9T8IuIRMK2vjFWdjTQ1ljdA7ug4BcRCcSWvtFQ9O+Dgl9ExHejEwX2HprgZSHo5gEFv4iI77Z5B3bV4hcRiYgwjegBBb+IiO+29I2xvL2BjqZUtUsBFPwiIr7b1jfK+uXVnZ9nJgW/iIiPxicL7BrKhGL8/hQFv4iIj7bvGwNg/QoFv4hIJITpG7tTFPwiIj7atm+MJa31dLXUVbuUaQp+EREfbe4dCdWBXahS8JvZR81sm5ltNbPvmVl9kPt3zvHCwQw/3z7Alt5RxicLR1yfL5bpH83SP5pldKJAvlgOsjwROUPsGcqwazDDlWs6q13KERJB79DMlgMfAtY557Jm9p/AzcCd872vn28f4MkXRyg7R9lB2Tl2HkjzxIsjHMrkj7jtwuY62hoSDKXzjGYLL3msRMwAph8rlYjR1VzHotY6uprrWLuomYtWtPHyFe0sbavHzOb76YhIjbl/234Abli/pMqVHCnw4J+x3wYzKwCNwD4/dvLwjkG+84e9xAwMA6sscvya8xdx8ap2zl/SwuB4jt1DE+weSjM+WeSqtXUsbK5jYUuKmBkT+RITuSLZQgkziJlhwGSxzNB4jgPjOXYPZfjlMwcolh0AnU0pulrqaG9M0taQpL0hRXuT97MxSansSOeKZHJFxrIFhtJ5BsdzDKZzAKxb2sr65W2sX95KKh7jUCbPUCbPWLZAzIy6RIxUIoYZTORKpHNFJvJFWuqTLGtvYFl7PQsaUzw3MM7mF0d5qneEA+M5FjSl6GxK0dGUImaQLZTJ5ktMFkrkS2UKpTLFksPhaEjGqU/GaUjGWdJWz9mLmlnb1cyqjkZGswUGx3MMjE2SzhUxMxIxIxYzUnEjlYiRiseJx4x0rsjIROXNNF8q01KXoKU+SXNdglLZMTZZmH4tSuXKG6tzDjOjPhmnPhmjPhmnMRWnKZWgqS5BU12cxpT3M5mgsS5OS32CukT8iN9/oVQmWygR8+pLxSuvWaHkKJbLFEqOgve8C0VHoVymIVnZT0MqTioRwzlHqewoOUciFiMeO/INPVcskclVXkNH5dOkc1CXjNFSl6Q+GVMjIMJ+tnU/Fy5vY8WC6i6ufjRzzgW/U7MPA58DssADzrl3HuM27wfeD7Bq1arLXnjhhWCLPEmThRJP94/xVO8oT/ePcSiTZyRbYHSiwPBEnpGJSvDNFDNorkuwsKXyqaGrpY5iybF13yi9w9mT2n9dIkbuGF1S9ckY65e1say9geGJPIcyeQ6m85jhBWslXFPxyptJMh6bfj4T+RLZfIm+kSzpXPHUXxzADJLx2KzdZg3JOImYYQZmRtk5coXyS16z40klYrTUJSg5x0SudFL3natErPLGm4jHyOZPvI9EzGiqS3hvpJU3sbpknIZkjIZknIZUnEQsRiJmxGNGIm40pbw3x/oErfUJWhuStNZXGhEdTSk6m1PTvycJr/7RLBs+/0s+cf15fODVZ1elBjN73DnXc/T2anT1LADeBKwGRoC7zOxdzrlvz7ydc+424DaAnp6e4N+dTlJ9Ms4lqxZwyaoFx7zeOcdkocxINk/cjOb6ShjM1hocmcizbd8YzjH9z97WkKTsHPlimXyxjAMaU5XWbzxmTBZK7B+dZN9IlqFMnrO7mjl3cTOJ0wwJ5xwHxnM8fyBN7/AEbQ1JFrXWs6iljtaGJK4MJXe4FT1VX6FUrgRWY5KWugSxmJErlkhPFhmfLJKI23Tr/+iW9JRS2U23qjO5Ipl8kUyuxES+WPk0lq9sT+eKjE0WKo8bMxpTCZrrKm9sZeemW/dlB8mYkUxUwnbqzS4Zr1yeLJTI5Etk80XyxTKxmBGzSigXS5Va8sXKG9LUPprrEtQn49NvWgC5Ytl7npVPNJOFEpOFMpOFEtlCiVyhzFA6T7ZQolgqUyw7ymVHvuSmn9vxtDcm6WxK0eTtuyEZp60hyaqORlZ1NLKio4FzFrWEaiRJ1DywbQAIXzcPVKer57XAbufcIICZ/QC4Cvj2ce9V48yMhlSchlTDnG7f3pji6rMXHvO6xlmm+6hPxule2ET3wqZTLfOYzIzFrfUsbj39Y/B1iTh1zXE6m+cWSHEvxBtTiUiFWLFUJpMrMTZZYDRbYGyywFi2wMFMnqHxPEPpHIcyeSbylW7IkWyBXUNp7t3ST6l8uJ20sLmOdctauWBpC2sXNrO6q4nVC5vobEqpC8pnP9vazzleF2nYVCP49wJXmlkjla6e1wAbq1CHSGgl4jHaGmO0NSZZeRL3K5bK9I9O8sLBCZ4bGGd7/xjb941x+84hCqXDbwhtDUnWL68cS7pweRvnL2nlrM5GdSHNk4PpHI/tPlS1Lp4TCTz4nXN/MLPvA5uAIvAEXpeOiJyeRDzGyo5GVnY08spzDn9iLJbK9I1k2T2UYfdQhucGxtnSN8rtv909/YaQiBmrOhs5u6uZdctauXB5G+uXt83LJ72o+fnTA5RdOLt5oEqjepxznwI+VY19i0RRIh7jrM4mzups4trzDm/PFUs8tz/NjgPj7BxMs/NAhucOjPPg0wNMjftY0lrPhrWdbFjbyVVrO0M3QiWM7tu6n5UdDaxbGq4vbk2p1nBOEQmBukScC1e0ceFRE4hlckWe7h9ja98oG18Y5jc7BvnhE30ArOxo4MrVnVy5ppMr13ayvH1ux62iYmyywG+fH+LdV3WH9jiKgl9EXqKpLkFPdwc93R28++rVOOfYcSDN754f4tFdB3nw6QHuerwXgLM6G7lq7UKuPruTq9cuZEFIFhupll89c4BCyXHD+qXVLmVWCn4ROSEz49zFLZy7uIVbr15Nuex4dmCcR3Ye5Pc7h7hn8z6+99he4jHjyjUdvH79Uq5/2ZJIjcSCytDn23+3h+XtDVyysr3a5cyqKl/gOlk9PT1u40YN/BEJq2KpzObeUX71zAF+uqWfXUMZzKDnrAW89oLFXLduMWtCOKxxvt23tZ+/+PYmvvTWl/P2y09mPJY/ZvsCl4JfROaVc47nBtL8dEs/D24fYHt/ZSGSNV1NXP+yJbx+/RIuXN4W2v7vU1UslXndVx8mZsZ9H77mtL84OR9C881dETmzmRnnLWnhvCUtfPS6c+kdnuAXTx/gge37ue3hXXz9oZ0sb2/gxpcv5Z1XnMWqzjNjlNDdm3rZNZjhP/7bZaEI/eNRi19EAjOcyfPg0wPct3U/v35ukLJzvPaCxdx6VTcb1nbW7KeAyUKJa7/8EEvb6/nBX14VmuehFr+IVN2CphRv71nJ23tW0j+a5TuP7uW7j+3lwe0DnL+khT+/Zg1vvGgZqUS4W8xH+9bv97B/bJKv3nxxaEL/eNTiF5GqmiyU+PGT+/jmb3fz7MA4i1rquOWqbt566QqWtIX/W8OHMnle/Y8Pccmqdu689RXVLucIOrgrIqHmnOPhHUN84ze7+M2OIQAu717AjS9fxuvXL2FRCKeOmCyUeOc3/sCWvlF+9IGruSBk39RV8ItIzdg1mOYnT/Vz71P9PDswjhlc3t3Bn164lBvWLwnF/EGlsuMD39nE/dv38+9/dimvvzB8X9hS8ItITdoxMM69W/r56ZZ+nhtIYwZXrO7gLZeu4A0XLqW5LvhDlc45Pn3Pdu78/R7+9sZ1vPeVqwOvYS4U/CJS854/MM5PnurnR0/uY/dQhoZknOtftpj/ctEyXnnOwpcsv+mHctnx9V/v5Mv3P8t7X7mav71xne/7PFUKfhE5Yzjn2LR3hB9s6uWezfsYmyzSUp/gunWLuf5lS7hidQfts61YdBo27R3m0/dsZ/OLI/zpy5fytZsvITbL6nFhoOAXkTNSvljmd88Pce+Wfu7ftp/xycr60Ocububy7g56uhdw6aoFrOpoPKWhls45nj+Q5usP7eQHT/SxqKWO/3HD+dx0yfJQhz4o+EUkAvLFMk/sHeaPew7x2J5hNr0wTDpXeSPobEpx8cp2LljaOv3N4u7Oppd8Z6BQKjM4nmPPwQy/euYAD24fYM/BCVKJGO+7ZjV/de3ZNFXhuMKp0Be4ROSMl0rEuGJNJ1es6QQqI2+eGxjnib0jbNo7zJMvjvDQc4NHrEtcl4jRUp+gqS5BJlfiYCY3vQhNKh5jw9pO/vyaNbxu3eJQDik9FQp+ETljxWPGBUtbuWBpK392xSqgMvZ+12CGZwfG6D2UJZ0vkp4sMj5ZpDEVZ3FrPYtb61naXs/l3R1VGTXktzPvGYmIHEd9Ms66Za2sWxauL1sFqbYmxBARkdOm4BcRiRgFv4hIxCj4RUQiRsEvIhIxCn4RkYhR8IuIRIyCX0QkYmpirh4zGwV2HOOqNmD0ONtOdH7q50Jg6BRKO9b+53L90duPd1l1n7iuE11/KnUfa1uQdc9lm591n6jmudY4W52znZ+5zY+65/o3MpdaZ54P69/2Wc65rpfcyzkX+hNw21y3z9x2ovMzfm6cz7pOtu7jXVbd1al7lm2B1T2XbX7WfaKaT7Xuuf6N+FV31LJktlOtdPXccxLb7zmJ87M97lyd6P5zrft4l1X37Pub6/WnUvdsz+VUnErdc9nmZ91zue+p1F0rfyNHb6uVuudUR0109fjNzDa6Y0xdGnaqO1iqO1i1WHet1FwrLX6/3VbtAk6R6g6W6g5WLdZdEzWrxS8iEjFq8YuIRIyCX0QkYs644Dez283sgJltPYX7XmZmW8zseTP7V5uxMrOZfdDMnjWzbWb2pfmt2p+6zezvzazPzJ70Tm+ohbpnXP9xM3NmtnD+Kp5+bD9e78+a2VPea/2AmS2rgZq/bGbPeHX/0Mza57NmH+t+m/e/WDazeT2Yejr1zvJ4t5jZDu90y4ztx/3799WpjDkN8wl4FXApsPUU7vsYsAEw4GfA673trwZ+DtR5lxfVSN1/D3y81l5v77qVwP3AC8DCWqgbaJ1xmw8B/6sGan4dkPDOfxH4Yo281hcA5wEPAT1hqNerpfuobR3ALu/nAu/8guM9tyBOZ1yL3zn3MHBo5jYzW2tm95nZ42b2GzM7/+j7mdlSKv+4j7jKb+X/AG/2rv5L4AvOuZy3jwM1UrfvfKz7n4H/Dvgy+sCPup1zYzNu2jTftftU8wPOuaJ300eBFfNZs491P+2ce3a+az2demdxPfCgc+6Qc24YeBC4odr/t2dc8M/iNuCDzrnLgI8D/36M2ywHemdc7vW2AZwLXGNmfzCzX5vZ5b5We9jp1g3w197H+NvNbIF/pR7htOo2szcCfc65zX4XepTTfr3N7HNm9iLwTuDvfKx1ynz8jUx5D5WWZxDms+4gzKXeY1kOvDjj8tRzqOpzO+MXWzezZuAq4K4ZXWh1x7rpMbZNtdgSVD6mXQlcDvynma3x3ql9MU91fx34rHf5s8BXqPxz++Z06zazRuCTVLogAjNPrzfOuU8CnzSz/wn8NfCpeS71cCHzVLP3WJ8EisB35rPGY5nPuoNwvHrN7Fbgw962s4Gfmlke2O2cu4nZn0NVn9sZH/xUPtWMOOcunrnRzOLA497FH1MJyZkfc1cA+7zzvcAPvKB/zMzKVCZjGgxz3c65gRn3+9/AT3ysd8rp1r0WWA1s9v7JVgCbzOwVzrn9Ia77aN8F7sXH4GeeavYOON4IvMbPxswM8/1a++2Y9QI45+4A7gAws4eAdzvn9sy4SS9w7YzLK6gcC+ilms8tqIMJQZ6AbmYcmAF+D7zNO2/ARbPc749UWvVTB1ve4G3/C+Az3vlzqXx0sxqoe+mM23wU+H+18HofdZs9+HBw16fX+5wZt/kg8P0aqPkGYDvQ5cdr7PffCD4c3D3Vepn94O5uKj0GC7zzHXN5br7+PoLaUWBPCL4H9AMFKu+q76XSgrwP2Oz9kf/dLPftAbYCO4F/4/A3m1PAt73rNgF/UiN1/19gC/AUlRbU0lqo+6jb7MGfUT1+vN53e9ufojJZ1vIaqPl5Kg2ZJ73TvI5E8rHum7zHygEDwP3VrpdjBL+3/T3e6/w8cOvJ/P37ddKUDSIiEROVUT0iIuJR8IuIRIyCX0QkYhT8IiIRo+AXEYkYBb/UJDNLB7y/b5jZunl6rJJVZvDcamb3nGhGTDNrN7O/mo99i4BW4JIaZWZp51zzPD5ewh2erMxXM2s3s28BzznnPnec23cDP3HOrQ+iPjnzqcUvZwwz6zKzu83sj97pam/7K8zs92b2hPfzPG/7u83sLjO7B3jAzK41s4fM7PtWmaP+O1NzpHvbe7zzaW8yts1m9qiZLfa2r/Uu/9HMPjPHTyWPcHhyumYz+4WZbbLKPO1v8m7zBWCt9ynhy95tP+Ht5ykz+/Q8vowSAQp+OZP8C/DPzrnLgbcC3/C2PwO8yjl3CZUZM/9hxn02ALc45/7Eu3wJ8BFgHbAGuPoY+2kCHnXOXQQ8DLxvxv7/xdv/Cedd8eameQ2Vb1UDTAI3OecupbIGxFe8N56/AXY65y52zn3CzF4HnAO8ArgYuMzMXnWi/YlMicIkbRIdrwXWzZhBsdXMWoA24Ftmdg6VGRCTM+7zoHNu5tzrjznnegHM7Ekqc7b89qj95Dk84d3jwHXe+Q0cnlP9u8A/zlJnw4zHfpzKHO1QmbPlH7wQL1P5JLD4GPd/nXd6wrvcTOWN4OFZ9idyBAW/nEliwAbnXHbmRjP7GvAr59xNXn/5QzOuzhz1GLkZ50sc+3+k4A4fHJvtNseTdc5dbGZtVN5APgD8K5U5/LuAy5xzBTPbA9Qf4/4GfN459x8nuV8RQF09cmZ5gMoc+ACY2dQ0um1An3f+3T7u/1EqXUwAN5/oxs65USpLNH7czJJU6jzghf6rgbO8m44DLTPuej/wHm+eeMxsuZktmqfnIBGg4Jda1WhmvTNOH6MSoj3eAc/tVKbTBvgS8Hkz+x0Q97GmjwAfM7PHgKXA6Inu4Jx7gsqMjzdTWQSlx8w2Umn9P+Pd5iDwO2/455edcw9Q6Up6xMy2AN/nyDcGkePScE6ReeKtHpZ1zjkzuxl4h3PuTSe6n0jQ1McvMn8uA/7NG4kzgs/LXIqcKrX4RUQiRn38IiIRo+AXEYkYBb+ISMQo+EVEIkbBLyISMf8fk5OzTepIk7QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 2e-2\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>23.825766</td>\n",
       "      <td>13.385324</td>\n",
       "      <td>0.969197</td>\n",
       "      <td>0.030803</td>\n",
       "      <td>628923.312500</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>8.161844</td>\n",
       "      <td>6.876748</td>\n",
       "      <td>0.936191</td>\n",
       "      <td>0.063810</td>\n",
       "      <td>571.473938</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(2, lr*10, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned1_tcu_jurisp_reduzido_sp15_multifit_v2')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned1_enc_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>6.500302</td>\n",
       "      <td>6.185975</td>\n",
       "      <td>0.854571</td>\n",
       "      <td>0.145429</td>\n",
       "      <td>262.548920</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>5.490670</td>\n",
       "      <td>5.003970</td>\n",
       "      <td>0.722680</td>\n",
       "      <td>0.277320</td>\n",
       "      <td>68.926476</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>4.755301</td>\n",
       "      <td>4.457559</td>\n",
       "      <td>0.653755</td>\n",
       "      <td>0.346245</td>\n",
       "      <td>36.102261</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4.416079</td>\n",
       "      <td>4.209779</td>\n",
       "      <td>0.621551</td>\n",
       "      <td>0.378449</td>\n",
       "      <td>27.896441</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>4.276728</td>\n",
       "      <td>4.118284</td>\n",
       "      <td>0.608721</td>\n",
       "      <td>0.391279</td>\n",
       "      <td>24.504759</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>4.199481</td>\n",
       "      <td>4.036489</td>\n",
       "      <td>0.597565</td>\n",
       "      <td>0.402435</td>\n",
       "      <td>22.284594</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>4.117595</td>\n",
       "      <td>3.955703</td>\n",
       "      <td>0.584517</td>\n",
       "      <td>0.415483</td>\n",
       "      <td>20.709642</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>4.023516</td>\n",
       "      <td>3.906753</td>\n",
       "      <td>0.574245</td>\n",
       "      <td>0.425755</td>\n",
       "      <td>19.213202</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.942492</td>\n",
       "      <td>3.827708</td>\n",
       "      <td>0.562653</td>\n",
       "      <td>0.437347</td>\n",
       "      <td>17.807081</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.849638</td>\n",
       "      <td>3.768771</td>\n",
       "      <td>0.554177</td>\n",
       "      <td>0.445823</td>\n",
       "      <td>16.522247</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.752683</td>\n",
       "      <td>3.697943</td>\n",
       "      <td>0.540054</td>\n",
       "      <td>0.459946</td>\n",
       "      <td>15.378661</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.659526</td>\n",
       "      <td>3.631438</td>\n",
       "      <td>0.528680</td>\n",
       "      <td>0.471320</td>\n",
       "      <td>14.282874</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.550264</td>\n",
       "      <td>3.568518</td>\n",
       "      <td>0.515796</td>\n",
       "      <td>0.484204</td>\n",
       "      <td>13.360168</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.440210</td>\n",
       "      <td>3.504019</td>\n",
       "      <td>0.503156</td>\n",
       "      <td>0.496843</td>\n",
       "      <td>12.489232</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>3.326972</td>\n",
       "      <td>3.462149</td>\n",
       "      <td>0.493769</td>\n",
       "      <td>0.506231</td>\n",
       "      <td>11.924422</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>3.233320</td>\n",
       "      <td>3.438249</td>\n",
       "      <td>0.488258</td>\n",
       "      <td>0.511741</td>\n",
       "      <td>11.569489</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>3.172660</td>\n",
       "      <td>3.426481</td>\n",
       "      <td>0.484748</td>\n",
       "      <td>0.515252</td>\n",
       "      <td>11.410362</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>3.146528</td>\n",
       "      <td>3.424744</td>\n",
       "      <td>0.484449</td>\n",
       "      <td>0.515551</td>\n",
       "      <td>11.384737</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxddZ3/8df3LsnNvjXN0rRN2kLpFrqkpQqylq2sSrUVUFwG5ic6rqMyMCI4owM6iuI6oAiOHRAKijKIshQBB0pT6EpbuqVNmqbZmn29935/f5ybNmnWlpvkJH0/H4/7uOee5fbzzU3f9+R7vuccY61FRETcyzPaBYiIyMAU1CIiLqegFhFxOQW1iIjLKahFRFzONxxvmpyWbk+fPm043lpEZFzasGFDtbU2s69lwxLU6VmTKC4uHo63FhEZl4wx+/tbNixdH8GQxmaLiETLsAR1Ryg8HG8rInJKGpagbu0IDcfbioickoalj7ozFCYYCuPzalCJiAyus7OTsrIy2traRruUYRcIBMjLy8Pv9w95m2EJagscqG1hWmbicLy9iIwzZWVlJCUlkZ+fjzFmtMsZNtZaampqKCsro6CgYMjbDbrLa4yZaYzZ2O3RYIz54mDbrS+pHXIRInJqa2trIyMjY1yHNIAxhoyMjBP+y2HQoLbW7rTWzrfWzgcWAS3A7wfaxu/18MzmQydUiIic2sZ7SHc5mXaeaCfyRcAea22/4/0A0uNjeHVXNTsqGk64IBER6elEg3oV8OhgK2UkxpAc8PHNp7fREdRQPRFxt7q6On72s5+d8HbLly+nrq5uGCrqachBbYyJAa4Gnuhn+S3GmGJjTHFtTTV3XjWHdftq+fQj66lt7ohWvSIiUddfUIdCAw81fvbZZ0lNTR2uso46kT3qy4G3rLWH+1porX3AWltkrS3KzMxkxaI87r1uHuv21nLVj19jc9nwf+uIiJyM2267jT179jB//nwWL17MBRdcwPXXX8+8efMAuPbaa1m0aBFz5szhgQceOLpdfn4+1dXVlJSUMGvWLG6++WbmzJnDJZdcQmtra9TqO5HheR9lCN0e3a1cPIUzspO5dfVbrPj563zrmjmsWjLlxCoUkVPK3X/axjvl0T2+NTs3mW9eNaff5ffccw9bt25l48aNvPzyy1xxxRVs3br16BC6hx56iPT0dFpbW1m8eDHXXXcdGRkZPd5j165dPProozz44IN85CMf4cknn+TGG2+MSv1D2qM2xsQDFwNPneg/cObkVP70T+dw1rR0bntqC19bs4m2Tp25KCLutWTJkh7jnO+//37OPPNMli5dSmlpKbt27eq1TUFBAfPnzwdg0aJFlJSURK2eIe1RW2tbgIxBV+xHekIMD39yCT984V1+/NJutpU38IsbFzE5Pf5k31JExqmB9nxHSkJCwtHpl19+mRdeeIHXX3+d+Ph4zj///D7HQcfGxh6d9nq9Ue36GLFzvL0ew1cumckvP17EgdoWrvzxa6zdWTlS/7yISL+SkpJobGzsc1l9fT1paWnEx8ezY8cO3njjjRGubhTu8LJsdhbP/NM55KbG8amH13PXH7epK0RERlVGRgZnn302c+fO5atf/WqPZZdddhnBYJDCwkK+8Y1vsHTp0hGvz1gb/WtHFxUV2cFuHNDaEeLbz77Db984QNHUNB7+1BISY4fl0iMi4nLbt29n1qxZo13GiOmrvcaYDdbaor7WH7XL28XFePn3a+fxk+sX8HZpHZ/69Xqa24OjVY6IiGuN+nVIryzM5Ycr51O8v5ZPPryelg6FtYhId6Me1ABXnZnLfSvnU1xSy6cfLtaNB0REunFFUANcM38S962cz7p9Ndz8m2I6dTsvERHARUENTljfe10hr+2u5nt/2Tna5YiIuILrhll8uGgyWw7W88Are5k/OZXl83JGuyQRkVHlqj3qLndcMYuFU1L54mMb2VZeP9rliIj0kpjo3GqwvLycFStW9LnO+eefz2BDlYfClUEd6/Pyq5sWkxzn4/anthAKR3+st4hINOTm5rJmzZph/TdcGdQAaQkxfOPK2Wwqq+fHL/W+AIqISDR9/etf73FN6rvuuou7776biy66iIULFzJv3jyefvrpXtuVlJQwd+5cAFpbW1m1ahWFhYWsXLkyatf7cF0fdXdXn5nL33ZW8aMXd/G+aRmcNe2krwslImPFn2+Dii3Rfc/seXD5PQOusmrVKr74xS9y6623AvD444/z3HPP8aUvfYnk5GSqq6tZunQpV199db/3Pfz5z39OfHw8mzdvZvPmzSxcuDAq5bt2jxqcm0D+27VzmZoez5cf30R9a+dolyQi49SCBQuorKykvLycTZs2kZaWRk5ODrfffjuFhYUsW7aMgwcPcvhwn/dOAeCVV145eg3qwsJCCgsLo1Kbq/eoARJifdy3cj4rfvE6d/9xGz9YOX+0SxKR4TTInu9wWrFiBWvWrKGiooJVq1axevVqqqqq2LBhA36/n/z8/D4vcdrdcNxN3dV71F0WTEnj1vOn89TbB3lpR//fZiIi78WqVat47LHHWLNmDStWrKC+vp6JEyfi9/tZu3Yt+/fvH3D7c889l9WrVwOwdetWNm/eHJW6xkRQA3zuwhnMzEriq09spr5FXSAiEn1z5syhsbGRSZMmkZOTww033EBxcTFFRUWsXr2aM844Y8DtP/OZz9DU1ERhYSHf/e53WbJkSVTqGrXLnA4qHIb6UkibenTWtvJ6rvrxa6xcPIX/+NC891iliLiFLnPq0sucDuqZL8BDl0HnseEtc3JT+OTZBTy2/gBbynQijIicGtwb1IUrobEc3nywx+wvLDuNjIQY7vzjVp0IIyKnBPcGdf45MGMZvPp9aK07Ojs54Of25bN4+0Adv3x17ygWKCLRNBzdsG50Mu0cUlAbY1KNMWuMMTuMMduNMe874X/pZFx0J7TVwf/9uMfsDy6YxMWzs/jhC7uoqB94qIyIuF8gEKCmpmbch7W1lpqaGgKBwAltN9Rx1D8CnrPWrjDGxADxJ1rgSck5E+ZeB2/8DJbcAklZgDNO8c4rZ3PR9//Gvc/t4D6NrRYZ0/Ly8igrK6Oqqmq0Sxl2gUCAvLy8E9pm0KA2xiQD5wKfALDWdgAdJ1HfybngDnjnaXjle3DFfx6dPTk9npvPLeCna/dww1lTKMpPH7GSRCS6/H4/BQUFo12Gaw2l62MaUAX82hjztjHml8aYhONXMsbcYowpNsYUR/VbMWM6LPw4bPg11O7rseizF8wgJyXAnU9v04FFERm3hhLUPmAh8HNr7QKgGbjt+JWstQ9Ya4ustUWZmZnRrfK8r4PHD2u/02N2fIyPO66YxTuHGnhQBxZFZJwaSlCXAWXW2nWR12twgnvkJGXD0s/Alid6XVXrink5LJ+XzX/+ZSebSuv6eQMRkbFr0KC21lYApcaYmZFZFwHvDGtVfTn7CxBIgRe/1WO2MYZ7riskIzGGf/2DxlaLyPgz1HHU/wSsNsZsBuYD3xlk/eiLS4VzvgS7/golf++xqGts9ZaD9TxeXDripYmIDKchBbW1dmOk/7nQWnuttfbIcBfWpyW3QFIOvHg3HDfe8uozc1k0NY0fvbCLjmB4VMoTERkO7j0zsS8x8c6BxdJ18O5zPRYZY/j8RadR0dDGb98Y+FKEIiJjydgKaoAFN0L6dKevOhzqsejc0yZw3umZ3PPcDnZWNI5SgSIi0TX2gtrrhwv/FSrfcUaBdGOM4d7rComP8fK1NZsI68CiiIwDYy+oAWZf65xevvbbEGzvsSg7JcCdkbuX/3lrxSgVKCISPWMzqD0euOibUHcANjzca/E18ydxelYi3/vLDh1YFJExb2wGNcD0CyH/A/C370J7z/5or8fwL5fPoqSmhdXrdGBRRMa2sRvUxsCyu6GlGt74ea/F58/M5OwZGdz/4i7qW3WPRREZu8ZuUAPkLYJZV8Hf74fmmh6LjHH2qutaO/nZ2t2jVKCIyHs3toMa4MJvQGezcyeY48ydlMKKhXn88rV97KlqGoXiRETeu7Ef1JkzYf71sP5BqOt9+vjXLz8Dv9fwk5e0Vy0iY9PYD2qA824DDLx8T69FExJj+eTZBfz+7YOs21vTe1sREZcbH0GdOhmW3Ayb/gcqd/Ra/PkLTyMvLY47/rBVw/VEZMwZH0ENcM6XwZ8AL/1br0VxMV6+dc0cdlc26QYDIjLmjJ+gTsiAsz8PO56BsuJeiy88I4vL52Zz/4u7OFDTMgoFioicnPET1ABLb4X4CfDCXb0ugwrwzavm4PMYvvH01nF/W3oRGT/GV1DHJsJ5X4OSV2HPS70WZ6cE+MolM/nbu1U8vbF8FAoUETlx4yuoARZ9ElKnODcXCPc+cHjT+/NZMCWVu/60jYr6tlEoUETkxIy/oPbFwAX/Coc2wTt/6LXY6zF8b8WZdATD/MNv1tMeDPXxJiIi7jH+ghpg3gqYOAde+ncI9b7Ox4yJidy3cj5bDzZw/4u7RqFAEZGhG59B7fHCRXdC7R54+7/7XOXSOdl8pCiPn67dw4vbD49wgSIiQzc+gxrg9Eth8lJ4+V7o6Hs43reumcvsnGS+tmYzNU3tfa4jIjLahhTUxpgSY8wWY8xGY0zvQcpuZAwsuwuaKuDN/+pzlYDfyw9WnkljW5BP/Fr91SLiTieyR32BtXa+tbZo2KqJtqnvg9Muhdfug9Yjfa5yRnYy37x6NlsO1nP7UxpfLSLuM367PrpcdCe0NcAfbu23C+SGs6by/86bzpNvlfGLv+kUcxFxl6EGtQX+aozZYIy5pa8VjDG3GGOKjTHFVVVV0avwvcqeC5ffCzv/DI9cBU191/a1S2eyfF429z63gx++8K72rEXENYYa1GdbaxcClwOfNcace/wK1toHrLVF1tqizMzMqBb5np31j7Dyt3B4G/xqGVT3HpLn8Rh+/NGFrFiUxw9f2MXXn9xMZ0hX2hOR0TekoLbWlkeeK4HfA0uGs6hhMetK+MQz0N4Ev7oY9v9fr1W8HsN3ryvkH8+bxuPFZXzpdxtp7dABRhEZXYMGtTEmwRiT1DUNXAJsHe7ChkVeEfzDC86Fm35zDWxZ02sVT+QO5l+9dCbPbD7Eed9bq9t4icioGsoedRbwmjFmE/Am8L/W2ueGt6xhlF4An/4r5C2GJz8Nr/6gzyvtffaCGfz0+oW0dYZY+V+v8+a+2lEoVkQEzHAcNCsqKrLFxS4fbh1sd0aCbF0Diz4By78PXl+v1fZUNfGph9ezv6aFDy/K457rCvF6zMjXKyLjmjFmQ3/Dn8f/8Lz++GLhQw/CB74CGx6GR1dCe2Ov1aZnJvLs5z/AhxZM4okNZdz00JuU1urGAyIyck7doAbweJxx1lf9CPashV9fDg29r1OdEOvjByvn8+0PzmVTaR3X/vTvCmsRGTGndlB3WfQJuOFxqN0Hv1zmDOPrww1nTeWJz7yP9mCYTz28noa23lfmExGJNgV1lxnL4FPPOQcWf3Vpn3eIAeeU8wc+voi91c3c+Mt1HG7QzQdEZHgpqLvLnucM30ubCqs/DG/1fYnU90+fwE8+uoAdFY188Kd/5383H2LX4UaNuRaRYdF7mMOpLmUSfPLP8MRN8MfPQd1+uOAO52p83Vw+L4eMxFhue2ozn/2ftwBIjfdz++WzuG5RnkaGiEjUnLrD8wYT6oRnvuTceGDeR+CanzgjRY4TDIV5dXc19S2drF63n/UlR8hNCfD5i07jnNMmkBofQ2Ksvg9FZGADDc9TUA/EWnj1+/DSv0H+B2Dlf0NcWr+rh8OWP2w8yA9f2MWByKgQv9dwy7nT+PLFM7WXLSL9UlC/V5ufgKdvhbQCuOEJpw97AKGw5c19tfz+7TK2lTewrbyBhVNS+eqlZzB3UjJJAf8IFS4iY4WCOhpKXoPHrgfjhcWfhgUfGzSwAax19rK/+fQ2GtqCAKxYlMcdy2eRlhAz3FWLyBihoI6Wqnfhr3fArued19MvgIU3wczl4Bs4dCsb2nh9bw3r9tXy6JsHCPi8/Pu1c7luUd4IFC4ibqegjrb6Mnj7t87wvYYy52p88693QnvCjEE331JWz7ee2cb6kiMsmzWRO66YTcGEhBEoXETcSkE9XMIh58SYDQ/Du89BOAhTz3YCe/bV4I/rd9POUJhfvbaP+55/l/ZgmGWzsli5eDIAU9LjmZmdNEKNEBE3UFCPhMbDsHE1vPUbOLIPAqlQuBIW3QRZc/rd7EBNCw/9fR9PFJfS3O2EmYyEGCalxXHZ3GzmT04lIyGW7OQAyXE+jNHoEZHxRkE9ksJhKHkV3noEtv8JQh0wqcgJ7DkfgtjEPjerb+lk26F6rIX/21NNaW0rm8vqKKnpefGnGJ+HpFgnrGflJJGbEkdHKMwn3p/PmZNTR6KFIjIMFNSjpaUWNj3mhHbVDohJgnnXOV0juQt6ne3Yl9rmDt4+cITy+jZqmtrZfqiBtk7nXo5Vje1UNbVT1dgOwFcuPp2bz51GwO8d1maJSPQpqEebtVD6phPYW5+CYKtzXZGZV0DGDMiYBunTIe7k9oj3VjXxnWd38ML2w8T5vSwuSGdmViJxfi9ej4cZExOZkBjDaVlJdATDJMf5KK9rY+vBeupaOjg9O4kdhxrJTY1jSUE66Ro2KDLiFNRu0lYPW55w+rIPbQa6/fzjJ0DGdCe0M6Yfm06f1m+XSXddJ9k8s+kQje3OmG1j+rzTWL98HsPXLpvJzR+Ypr5wkRGkoHarzjbnwGPNHqjdAzW7oWavM914qOe6STmRAI/sfWfMcII8rQD8gV5vHQpbDNDcEeSd8gZaOkPsqWyiprmDzmCYtIQYzipIJxi2dATDtAfDpCf4efCVfTy3rYKzCtL53IUzmJubohNzREaAgnos6miG2r2R8N4TmY6EeUt1txUNpORBWr5zpmRqfrfpqZA4cUh94V2stfz2jf386MXdVDc5fd9ZybGcnpVEnN/L1Ix4whZS4/yUHmkhxuehurGD/bUtlNW2YAykJ8RgjGFxfhp3XDGblDidMi8ymKgEtTHGCxQDB621Vw60roJ6mLXVdwvvSJDX7Ycj+6Gpoue6vrhjoZ021QnxrunUqRBI7vOfaO0IsXZnJftrWthZ0cCeqmaa24Psr20hFHZ+Z5ICPgxO582MiYkUTEjAYwz1rZ3sr2nm3cNNZCcHmD85lTNykvhw0WRyUwLqUhHpQ7SC+stAEZCsoHaxzlaoO+CE9pGSSICXOK/r9kN7Q8/149K7BXn+senUqZA6uc9Lu4bClrbOEHF+L54Brgi4vqSW7z23kzdLao/OS4z1kZcWR05KgPSEWBraOmkPhvEaOCMnmc5gmKkZ8SQF/Fgsk9PiOXNyKn6v7nEh49t7DmpjTB7wCPBt4MsK6jHKWmg90i3A9/ecrjsA4e73gTRO33j3PfLuz8m54Bl8KGBlQxuHG9rZWHqEHRWNvHWgjuqmdqy1xMf4SAr4aG4P9hoz3sXnMXy4KI8bzprK3EkpUflRiLhNNIJ6DfAfQBLwz30FtTHmFuAWgClTpizav3//eypaRkE45BzE7Nr77nru2kNvOEiPUSoef6R//Pggz4fUKc4oFs/Q9oSttTS2B0mM8bG3uomWjhA+j4edhxt4eWcVf9xUjrVwRnYSF82ayNkzJrC0IIO61k7i/F7iYryR8IfMpN5/BYi43XsKamPMlcBya+2txpjz6Seou9Me9TgV7ID60p4h3v25x0FOwHicrpX4jMij+/Txj8iy2KQ+D34eqGnh8eJS1u6sZFu5032TEOOluSNEUsBHTkqAdw834TFw/syJxPm9JAV8VDS00d4Zxuc1GGO4sjCHKwtziI/RXXfEXd5rUP8H8DEgCASAZOApa+2N/W2joD5FtTc5e99de+FNldBaCy01zlmaLTXHHuFg3+/h8fcO74QJkDLZ2UtPm0p9bA4vl4Z5dXcNk1Lj2F3VxJHmDs4qyOBISwd/3VZBeX0bAb+H9PgYUuNjaOsMUd/aSU1zBwBn5qWwaGo61y2axJxcdafI6Iva8DztUUtUWOsc1OwrwI8+us1vqoS2up7v4Y93gjt1SuTA55Qer0OBNLzHHYC01rK+5Aj/9bc9bC2vp7a5g86QZUl+OhfPzmLupBTm5aWQGOujrTNEVWM7lY3tVDe14zGGGJ+HM/NSiI/x4Y/soYtEy0BBrb//ZOQZA4EU55E+bWjbtDVE9ta7PyLdLqXrnCGL3XhjEo8L7ymY1CksSc5jybXZkFjIkXbDI6+X8OctFXz72e1Ht52UGsfButYBy5mYFMttl5/Bhxbqxg8y/HTCi4wPrXWR/vMDx0awdA/044clgnMp2sQsSJxIW+wEKsIpvFHloz2QiUnMIjN7Mt6UbNImZGOMh4qGNt6taKQjZHl9bw2bSuu4dE4WN5w1lbOmpRPr08Ww5ORpj1rGv7hU55E9r/cya52ukyP7obECmg473SlNh49OByo3kt90mPzObkMEd0eejdc5wzNx4tFgD542kRfivTy3z8s97yRT7c1kwekFLJyaTozPQ35GAtMyE5iSHo8xhlDYcqSlg9rmDoIhS3VTO16PISXOz4yJibrioQxIQS3jnzEQl+Y8BtPedFyQdz1XHJuu2IKvqZLLbIjLACKjAdv2xnBwdwYVNp1DZLDFpnPIZtASyGZnaxLl4QwaiAd69m3Hx3i5b+V8Lp2THe2WyzihoBbpLjbReWRMH3i9cMgJ7oaDzj00G8oJNBwku2o/k5vLsfXv4m+pxEMYQkDkulZBXzytcTkEE3Npj8+mPJzOMyUe/mf1W+wunMMHz19MblbOCV2fRcY/BbXIyfB4ITnHeeQd61bscYviUNDZE68/6NwEuf4gvoaDJDUcdOYdfoXspsMsBCfIdziPdmJpDWQSSszGk5JLnS+TUEIOMWmTyMjNZ39HCodCKTQFDVMzEiitbaHsSCvZKbFcOidbY8THIX2iIsPF63PO3EzJA87qe51gh3M2aOMhqstL2LpjB5Xl+wg0Hya7pZbsytfJNXXEmmOn9s8GzrCGGpKpsGkEbDpJNo3dNoPSl3L52MVLSZs42Tn9Pz5de+fjgEZ9iLjQ4YY2DtW3UVxSS05ygJyYFo4cLqG5qpRcTx2Z1OBvrsDbXEFMy2FSOqvxtNX2fiNvjHMANCnbeSRmH5tOynbCPDFbge4CGvUhMsZkJQfIilwi9qhZMwbeqLOV/Qf2sfr5dZQd2EthahuFya34Wysp6Ggk5fBO/Pte7X3yEIA3hnBiFp6kbEIJWXiSczDJOU6QH31kO2PfFegjTnvUIuOMtZZH3yzlO89up6m956n6c3KTmZxoSA7WcHpCM3HtVfhbDtNae5DEzmqyzBEmUkeO5whJ9HE1Q3/8sT3xo8+R6eTcY/P8cSPU2vFDd3gROQWFwpbOUBivx7CptI5XdlXzzOZyOkNhQiFLdXMH1lq8HsOyWVnkZziHQr0ew7NbDlFaWcMHskPcPD/A4ox2TGPF0f50uqYbDjk3az5eIPVYgCdkOmPcA6kDP/vjT+m9dQW1iPTJWksobPH1cV2U3647wI9eeJfqpg7OyE7iq5fO5MIzJva8xom1zun7PUI8EuQN5c50S41z5mhbPT0uk3s8j3/wQI+JdwLdF3Ce/XGR50C36TjnzkZe/5gKfgW1iJyUYCjMY+tLuf/FXVQ2tjMhMZaclAAtHUGmpMeTFh9DfWsnxjh78Ov21WItpMb7SYnzkxjr4+wZE/jH86YR7/M4p/K31TnB3Xrk2PSgz4OEfF+Mt1uYHxfsvljw+JwvB2/Xs98Zdnl0OvK6a7prPY8vMi/ybLq+5EzkiyHy5dDXdPcvjqPTznwz90MKahE5ee3BEM9sOsTf91Rz8EgrqfF+9lY109QeJC7GS2VDOzkpAeJjfWQmOpeWbWjtpCRy78wl+en87MaFTEg8yZs6hMNOyHe2QmcLBNuOTXe2dntEXgdbj5vffVlk23DQeYQ6nTsbhYPO2PdwZ2Ret+U2FN0faB/M3Q0KahEZHWs2lHH7U1voCIU5PSuR1LgYCiYkMH9KKufMmMCk1LgB773pCuHwseAOd/YMdKzTBXT0meOmOba8v2lrMdlzNDxPREbHikV5zJ+cwo9e3M3h+jbK61t5s6SW3xWXHl0nPsZLQqyP9s4QOSlxzMpJoiMUZtqERDwew6G6VmZmJ3HdwjzSEmJGvhEeD3hiOHotgBGmPWoRGVHWWprag+yrbmbd3loa24M0Rx4doTB1LZ1sK68nbDl6H8y0eD9HWjpJjPVx7ukTuOXc6T3HmI8DOuFFRFzDGENSwE9hXiqFeQOHbWcoTFtniKSAnx0VDfzy1X38dVsFz26pYHpmAstmZZGVHGDupBSykwPkpgZ6jWAZD7RHLSJjSm1zB0+9VcZLOyp5Y28N4W4R5vca8jMSyE2NwwKhcJjOoCUuxkvYWudLItZHwO/l/dMzWJyfzqS0OLwu6CPX8DwRGZdaO0I0tneyqbSeI80d7K1uZndlEwdqmwn4vcR4PXg9hrIjrUxIjCFsobk9yKH6Nlo7nZEc6QkxLMlPZ3J6HAmxPrKTndErwVCYsIWS6mYmJMYwNSOBxQXpJMY6HRHhsI3qQVB1fYjIuBQX4yUuxsvFswMntF17MMTOikbePlDH63tq2F7RwPPbDxMKD7zj6vUYUuP8xPg8HKpvIyngY8bERE6bmEhGYiyxPg8FExJo7Qix/VADB+tayUiIJS7GS3Kcn7MK0slLi6M9GCbO76W+tZO0hBhyUwauX0EtIqecWJ/3aB/5Te/PB5wTdoLhMJUN7bQHQ/g8HkLWMiEhltIjLeyoaGR/TTN7q5pp6Qhy8ews6lo6eXNfLW8f6H2hqzi/l9zUAG/srcVaS3NH/2OxYwbpV1dQi4jg7C17PV4mp8f3WpYSn8LcSSl9bmetpT0YPnp25sEjrfi9HvLS4noc2GxuD/LKu1UcrGslGLYYoKk9SFLAx/6aFr4zQG2DBrUxJgC8gnNnOB+wxlr7zcG2ExE5FRhjetyc+LSspD7XS4j1cfm8nH7f5z0FNdAOXGitbTLG+IHXjDF/tta+MYRtRWPbgEcAAAh1SURBVETkPRo0qK0zLKQp8tIfeUR/qIiIiPRpSCPDjTFeY8xGoBJ43lq7ro91bjHGFBtjiquqqqJdp4jIKWtIQW2tDVlr5wN5wBJjzNw+1nnAWltkrS3KzMyMdp0iIqesEzrX0lpbB7wMXDYs1YiISC+DBrUxJtMYkxqZjgOWATuGuzAREXEMZdRHDvCIMcaLE+yPW2ufGd6yRESky1BGfWwGFoxALSIi0ofxdz1AEZFxRkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i43KBBbYyZbIxZa4zZbozZZoz5wkgUJiIijkHvQg4Ega9Ya98yxiQBG4wxz1tr3xnm2kREhCHsUVtrD1lr34pMNwLbgUnDXZiIiDhOqI/aGJMPLADWDUcxIiLS25CD2hiTCDwJfNFa29DH8luMMcXGmOKqqqpo1igickobUlAbY/w4Ib3aWvtUX+tYax+w1hZZa4syMzOjWaOIyCltKKM+DPArYLu19gfDX5KIiHQ3lD3qs4GPARcaYzZGHsuHuS4REYkYdHietfY1wIxALSIi0gedmSgi4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLjdoUBtjHjLGVBpjto5EQSIi0tNQ9qgfBi4b5jpERKQfgwa1tfYVoHYEahERkT5ErY/aGHOLMabYGFNcVVUVrbcVETnlRS2orbUPWGuLrLVFmZmZ0XpbEZFTnkZ9iIi4nIJaRMTlhjI871HgdWCmMabMGPPp4S9LRES6+AZbwVr70ZEoRERE+qauDxERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxuSEFtTHmMmPMTmPMbmPMbcNdlIiIHDNoUBtjvMBPgcuB2cBHjTGzh7swERFxDGWPegmw21q711rbATwGXDO8ZYmISBffENaZBJR2e10GnHX8SsaYW4BbIi/bjTFb33t5o24CUD3aRUSB2uE+46Utakf0TO1vwVCC2vQxz/aaYe0DwAMAxphia23RkMtzKbXDXcZLO2D8tEXtGBlD6fooAyZ3e50HlA9POSIicryhBPV64DRjTIExJgZYBfxxeMsSEZEug3Z9WGuDxpjPAX8BvMBD1tptg2z2QDSKcwG1w13GSztg/LRF7RgBxtpe3c0iIuIiOjNRRMTlFNQiIi4X1aAea6eaG2NKjDFbjDEbjTHFkXnpxpjnjTG7Is9pkfnGGHN/pG2bjTELR7n2h4wxld3Hq59M7caYmyLr7zLG3OSSdtxljDkY+Vw2GmOWd1v2L5F27DTGXNpt/qj+7hljJhtj1hpjthtjthljvhCZP6Y+kwHaMaY+E2NMwBjzpjFmU6Qdd0fmFxhj1kV+tr+LDJDAGBMbeb07sjx/sPaNKGttVB44Bxr3ANOAGGATMDta7z8cD6AEmHDcvO8Ct0WmbwPujUwvB/6MM658KbBulGs/F1gIbD3Z2oF0YG/kOS0yneaCdtwF/HMf686O/F7FAgWR3zevG373gBxgYWQ6CXg3Uu+Y+kwGaMeY+kwiP9fEyLQfWBf5OT8OrIrM/wXwmcj0rcAvItOrgN8N1L6R/N2y1kZ1j3q8nGp+DfBIZPoR4Npu839jHW8AqcaYnNEoEMBa+wpQe9zsE639UuB5a22ttfYI8Dxw2fBXf0w/7ejPNcBj1tp2a+0+YDfO792o/+5Zaw9Za9+KTDcC23HO6h1Tn8kA7eiPKz+TyM+1KfLSH3lY4EJgTWT+8Z9H1+e0BrjIGGPov30jKppB3dep5gN9wG5ggb8aYzYY5xR4gCxr7SFwfmmBiZH5Y6F9J1q7m9v0uUiXwENd3QWMkXZE/mxegLMXN2Y/k+PaAWPsMzHGeI0xG4FKnC+8PUCdtTbYR01H640srwcycEE7ILpBPaRTzV3mbGvtQpwrA37WGHPuAOuOxfZ16a92t7bp58B0YD5wCPh+ZL7r22GMSQSeBL5orW0YaNU+5rmmLX20Y8x9JtbakLV2Ps7Z1EuAWQPU5Np2QHSDesydam6tLY88VwK/x/kwD3d1aUSeKyOrj4X2nWjtrmyTtfZw5D9ZGHiQY39qurodxhg/TritttY+FZk95j6TvtoxVj8TAGttHfAyTh91qjGm60S/7jUdrTeyPAWnS84V7YhmUI+pU82NMQnGmKSuaeASYCtOzV1H2m8Cno5M/xH4eORo/VKgvutPWhc50dr/AlxijEmL/Cl7SWTeqDqu7/+DOJ8LOO1YFTlCXwCcBryJC373Iv2ZvwK2W2t/0G3RmPpM+mvHWPtMjDGZxpjUyHQcsAynv30tsCKy2vGfR9fntAJ4yTpHE/tr38iK8pHW5ThHifcAd4z0kdETrHUaztHcTcC2rnpx+qVeBHZFntPtsaPIP420bQtQNMr1P4rzJ2gnzrf+p0+mduBTOAdIdgOfdEk7/jtS52ac/yg53da/I9KOncDlbvndA87B+ZN4M7Ax8lg+1j6TAdoxpj4ToBB4O1LvVuDOyPxpOEG7G3gCiI3MD0Re744snzZY+0byoVPIRURcTmcmioi4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJy/x+ULPZY8jAxLQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(18, lr, wd=wd, moms=(0.8,0.7), callbacks=[ShowGraph(learn_lm)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned2_lenerbr_sp15_multifit_v2')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned2_enc_lenerbr_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save best LM learner and its encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned_tcu_jurisp_reduzido_sp15_multifit_v2')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned_enc_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning \"backward LM\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.03 s, sys: 560 ms, total: 1.59 s\n",
      "Wall time: 2.12 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = (TextList.from_df(df_trn_val, path, cols=reviews, processor=SPProcessor.load(dest))\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_for_lm()           \n",
    "    .databunch(bs=bs, num_workers=1, backwards=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm.save(f'{path}/{lang}_databunch_lm_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 84 ms, sys: 12 ms, total: 96 ms\n",
      "Wall time: 94.2 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = load_data(path, f'{lang}_databunch_lm_tcu_jurisp_reduzido_sp15_multifit_bwd_v2', bs=bs, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.32 s, sys: 220 ms, total: 1.54 s\n",
      "Wall time: 2.26 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, config=config, pretrained_fnames=lm_fns3_bwd, drop_mult=1., \n",
    "                                  metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Change loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.loss_func = FlattenedLoss(LabelSmoothingCrossEntropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of LabelSmoothingCrossEntropy()"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm.loss_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXgcd53n8fe3L7VuybYOHyF27NhO4pBM7GRyzJCEcISbABngYXYDYcjDznA/sMssO+wMPAwwWXZ2GJ4ZNgsJ7EIYSEIYwplwhBwkJM5pO4ljO5ct2ZJsnd1q9fnbP7oky45kK7Kqu1r1eT1PP+quru76dkv61K9+VfUrc84hIiLhEal2ASIiUlkKfhGRkFHwi4iEjIJfRCRkFPwiIiETq3YBc7Fs2TK3evXqapchIlJTHnrooYPOuY6jp9dE8K9evZqtW7dWuwwRkZpiZs/PNF1dPSIiIaPgFxEJGQW/iEjIKPhFREJGwS8iEjIKfhGRkFHwi4iEjIJfRCSADoxM8JXbd/LMQGrB31vBLyISQC8MjvPPv9lNz3Bmwd9bwS8iEkDpbAGAxrqFH2BBwS8iEkApL/ibFPwiIuGQVvCLiIRLSl09IiLhMhX8ieiCv7eCX0QkgNLZAsl4hFh04WNawS8iEkCpbNGX/n1Q8IuIBFI6W1Dwi4iESSpb8GXHLij4RUQCScEvIhIy6uoREQmZtFr8IiLhoqN6RERCJpXN01S38CdvgYJfRCRwCsUSE/mSunpERMIinSsC/gzQBgp+EZHA8XNkTvAx+M3sejPrN7Pt06Zda2ZPmdnjZnarmbX5tXwRkVrl50VYwN8W/7eAy4+adgewyTn3cuBp4K99XL6ISE0aq9UWv3PuLmDwqGm3O+cK3sP7gVV+LV9EpFbVcov/eK4Gfj7bk2Z2jZltNbOtAwMDFSxLRKS6Dgf/Ijqc08w+AxSA7842j3PuOufcFufclo6OjsoVJyJSZals+aie5rq4L+/vz3bEMZjZVcAbgcucc67SyxcRCTq/W/wVDX4zuxz4L8DFzrnxSi5bRKRW+Hm9XfD3cM7vAfcBG8xsn5m9H/ga0AzcYWaPmtnX/Vq+iEitSmULxCJGXcyfiPatxe+ce/cMk7/p1/JERBaLyZE5zcyX99eZuyIiAZPycSx+UPCLiASOnxdhAQW/iEjglC+76M8RPaDgFxEJnFS26NsRPaDgFxEJnHS2QHNSwS8iEhrpbIHGhIJfRCQ0Uj5eaB0U/CIigeKc01E9IiJhkskXKTn/hmsABb+ISKBMjtPTpJ27IiLhkM5OXmhdx/GLiITC1JDMOqpHRCQcxib8vd4uKPhFRALF7+vtgoJfRCRQ0jnt3BURCZWpo3rU4hcRCQd19YiIhEzK27nbENfhnCIioZDKFmmqixGJ+HPZRVDwi4gEStrni7CAgl9EJFBSOX9H5gQFv4hIoPg9Mico+EVEAiU14e9FWEDBLyISKKlswdeTt0DBLyISKOmcunpEREIlnS3qqB4RkTDx+3q7oOAXEQmMXKFErlCiSTt3RUTCIV2Byy6Cgl9EJDBSFRigDRT8IiKBMTUWv4JfRCQcKjEkMyj4RUQCoxLX2wUfg9/MrjezfjPbPm3alWa2w8xKZrbFr2WLiNSidLYI1HDwA98CLj9q2nbgbcBdPi5XRKQmHe7q8fcELt9WK865u8xs9VHTngQw8+8CAyIitaoS19uFAPfxm9k1ZrbVzLYODAxUuxwREd+F/nBO59x1zrktzrktHR0d1S5HRMR36WyBuliEeNTfaA5s8IuIhE2qAhdhAQW/iEhgpCswQBv4ezjn94D7gA1mts/M3m9mV5jZPuAC4Kdm9ku/li8iUmtS2WJFgt/Po3rePctTt/q1TBGRWpbK5mny+VBOUFePiEhgpLNF9fGLiIRJzffxi4jISzM6UaDZ57H4QcEvIhIIzjmGx3O0NSR8X5aCX0QkAFLZAoWSo70h7vuyFPwiIgEwPJ4HUItfRCQshsZzALQr+EVEwmHIa/Grq0dEJCSGJ1v8jWrxi4iEwlBaXT0iIqEyOJ7HDFrr1dUjIhIKw+M5WpJxohH/r1Co4BcRCYCh8XxFduyCgl9EJBAqddYuKPhFRAJhaDynFr+ISJgMpfMVOaIHFPwiIoGgrh4RkRDJFUqkc0V19YiIhMXkWbttFThrFxT8IiJVN+gF/xJ19YiIhMNQunIDtIGCX0Sk6qa6eoLU4jeztWZW592/xMw+YmZt/pYmIhIOU0MyNwarxX8LUDSzdcA3gTXAjb5VJSISIpW8CAvMPfhLzrkCcAXwv5xzHweW+1eWiEh4DI/nSMYjJOPRiixvrsGfN7N3A1cBP/GmVWabRERkkSsP0FaZ1j7MPfjfB1wAfME596yZrQG+419ZIiLhUcmzdgFic5nJOfcE8BEAM2sHmp1zX/KzMBGRsKjkkMww96N67jSzFjNbAjwG3GBm/9Pf0kREwmEonQtkV0+rc24UeBtwg3NuM/Aq/8oSEQmPofEcbUFr8QMxM1sO/BmHd+6KiMgJKpUcI5k8Syo0Tg/MPfg/B/wS2OOce9DMTgF2+VeWiEg4jE7kKbnKnbULc9+5exNw07THzwBv96soEZGwmDprN2hdPWa2ysxuNbN+M+szs1vMbNVxXnO9N//2adOWmNkdZrbL+9l+oh9ARKSWVfqsXZh7V88NwI+BFcBK4DZv2rF8C7j8qGmfBn7tnDsV+LX3WEQktA4P0BawFj/Q4Zy7wTlX8G7fAjqO9QLn3F3A4FGT3wJ827v/beCtL6VYEZHF5vCQzMFr8R80sz83s6h3+3Pg0DyW1+Wc2w/g/eycbUYzu8bMtprZ1oGBgXksSkQk+ILc1XM15UM5DwD7gXdQHsbBN86565xzW5xzWzo6jrlxISJSs4bGc0QMmpNzOtZmQcwp+J1zLzjn3uyc63DOdTrn3kr5ZK6Xqs87HwDvZ/883kNEZNEYGs/T1pAgErGKLfNErsD1iXm85seUR/jE+/nvJ7B8EZGaN1zhs3bhxIL/mKsnM/secB+wwcz2mdn7gS8BrzazXcCrvcciIqE1lK7skMwwxxO4ZuGO+aRz757lqctOYJkiIovK0HiOVe0NFV3mMYPfzMaYOeANqPelIhGREBkez3Pmysp29Rwz+J1zzZUqREQkjIbGc7RXcIA2OLE+fhEROQGZXJFsoVRTO3dFROQEVOPkLVDwi4hUzWB6MvjV4hcRCYVhb0jmSo7FDwp+EZGqUVePiEjIDI+rq0dEJFSG1NUjIhIug+kcTXUxErHKRrGCX0SkSnqHM6xoS1Z8uQp+EZEq6R3JsLy18qPfKPhFRKqkd3iCFW0KfhGRUJjIFxlM51iprh4RkXDoHc4AqMUvIhIWvcMTgIJfRCQ0Jlv8KxX8IiLh0DOcwQy6WtTHLyISCr3DGTqb6yp+8hYo+EVEqqJax/CDgl9EpCp6hyeq0r8PCn4RkYpzzlVtuAZQ8IuIVNxgOke2UKrKoZyg4BcRqbhqHsMPCn4RkYrrqeIx/KDgFxGpuGoO1wAKfhGRiusdzpCMRyp+ycVJCn4RkQrrHcmwoq0eM6vK8hX8IiIV1jM8wYoqnbwFCn4RkYrbX8Vj+EHBLyJSUdlCkf6xbNV27IKCX0SkovpGskD1jugBBb+ISEVV+xh+qFLwm9lHzWy7me0ws49VowYRkWqo9jH8UIXgN7NNwAeA84CzgDea2amVrkNEpBomg395a7h27p4G3O+cG3fOFYDfAVdUoQ4RkYrrHcmwrClBMh6tWg3VCP7twCvMbKmZNQCvB046eiYzu8bMtprZ1oGBgYoXKSLih57hiap280AVgt859yTwZeAO4BfAY0Bhhvmuc85tcc5t6ejoqHCVIiL+2D+cqWo3D1Rp565z7pvOuXOcc68ABoFd1ahDRKSSDl+Apbot/lg1Fmpmnc65fjN7GfA24IJq1CEiUkmjmQLpXLGqh3JClYIfuMXMlgJ54K+cc0NVqkNEpGL2DY8DVO0i65OqEvzOuT+txnJFRKrp6b4xANZ1NlW1Dp25KyJSIdt7RqmLRVjb0VjVOhT8IiIVsqN3hNOWtxCLVjd6FfwiIhVQKjl29IxyxoqWapei4BcRqYS9Q+OMZQtsWtla7VIU/CIilbCjdxSATSsU/CIiobC9Z4RYxFjfXd0jekDBLyJSEdt7Rzm1q5m6WPUGZ5uk4BcR8Zlzjh09I2wKwI5dUPCLiPiubzTLoXQuEEf0gIJfRMR323tGAAJxRA8o+EVEfLe9dwQzOG25WvwiIqGwo3eUNcsaaayr1riYR1Lwi4j4rLxjNxjdPKDgFxHx1WA6R+/IBJtWBqObBxT8IiK+2tFb3rF7hlr8IiLhsL2nPFRDUA7lBAW/iIivtveOsKq9nraGRLVLmaLgFxHx0bZ9I4Fq7YOCX0TEN88fSvPC4Djnn7K02qUcQcEvIuKTO3cOAHDphs4qV3IkBb+IiE9+u7OfNcsaWb2sutfYPZqCX0TEB5lckfv2HOKSDR3VLuVFgnH+cAjsH8nwk8f28/Pt+4lFI2xa0coZK1rY0N1MLGoUio5CyQFQF4uUb/EopZIjWygykS8xkS8yli0wNlFgNJMHYGVbPSva6lnRlqQ5Ga/mR/RdseTIF0tkCyUOpbIMjGUZSGUZzxZJJqLUx8u35mSM9oYErQ1xmutiRCI29R7OOVLZAodSOQ6ls6SyRSby5Vu2UCIRjZCMR6iLRYlEjEKxRL7oKJRKGEYsasSjRjwaIektL+ktc0ljgvi0i2g75xjLFugfzdI3OsGBkQkOjE4wkS+ypDHB0qY6ljUmaG9M0NYQp70hQTJe/bHaZWHc/8whsoVS4Lp5YJEHv3OOfLEcFrlCiXyxRCZfZDxXZDxXYCSTZ1dfip19Y+w8MMbYRIGTlzZw8tIGVi9tZGlTgoZEjIZElIZElLpYdCoUMvkie/pT7BlIsWcgzXiucMSy49EIiWiEeDTCs4fSPPjcIM7BppUtOODGB55nIl9a0M+biEVorY/TkozRUh+fCqX6eJS6WIRErFxPPBqhUCqRzhbJ5Atk8yWWNCbobk3S2ZKkq7mOrpYkXS1JljUliEVf2obh2ESepw6MsasvxcFUlkOp8pC0mVyxvPxYhHjUyBcd6WyBdLbARL5IXSxKQ12UxkQMh2NgLMvBVI6BsSzpXAHn5ve9xKNGxIxYxMiXHLnCwn7v07XWx1nalCBXKDEwliU7w7IiBqVZPksiFqElGaOxLkZjIkaz97tsScZprS/f2hrKt9b6OEsaE+WVSGMd9QmtNILktzv7qY9HOW/NkmqX8iKLOvj/24+2890/vHDc+bpbkqzvbuaUjiZeOJTmx4/2MjpROO7rJq1oTdJSf7i17RzkS+UVTb7gaGuI87HL1vOms5ZzSkf5smvFkuPZgyl29aUAiEbKrUmAXKHERL5EtlAkYkZdPErS2wJoTsZoScZoTsZxDnqGM/QOZ+gZzjA0nmM0k2c0U2B0Ik8mV5z6mfVWfPliOfjiUZtaqcWjEXb0jjKQylI8KpEiBi31cZqTMZrr4jQlY+WViLcCiUQot4i9935+MM3ewcwR79GSjLG0qY6GRJRC8XCrPRGL0FgXpSERo60hQbZQZDCdY+/gOAAdzXVsWtnKsqYEzXUxYtEIsaiRiEZY1lRHR3P51pCITm0RjeeKjE3kGR7PT30fhZKj6N2iUWOpF5RLmhK0JGPUxaLUJ6IkopGp2rKFEsVSiVgk4rXyI+Xfa7FEYXLLI19uSGTyRUYzeQbTOW9FlyMRi5Tr8+rsaknS3ZqkuyVJXSzCcCZf3mpJZRkZzzM0nmc4k2NkPM+YtzJMZwuMThTYOzjOaCbPSCZPOlec9e8wEYvQXOetNOpiLG1M0NlSR3dLkuVt9ZzW3czpK1poSCzqf/tAcM7xm6f6uWjd0kBuxS3qv4BXnd7Firb6qU3zWDRCQzxKY12U+kSMproYazsaX3RihXOO4fE8w5k847mCt4VQJJsvMlEokc0XScQirO1omveIe9GIsa6zmXWdzSf0Gbtbk2w+uf2E3mNSseQ4lMpyYHSCPq97om90guHxPKlsgbGJPGMTBVLZwtRKregcscjk92u8fGUb79xyEqctL3djdTYnScS0K+loky31U7te2u8/Xywxkimv2EYyOQbTeQbT5S2qEe/3lM6Wf0cHUzme2ZOifyw71Y0YMTi1s5kzVrawsbuZDd0tnNbdTEdzHWZ2nKXLXO0ZSLNvKMMHL15b7VJmtKiD/9INnfPqXzMz2r2+1zCJRozOlnJ3jwRT3NvaWdZUN+fXlEqOvrEJdvSM8njPCNv2DXP3roP88OGeqXmWNdXx8lWtbFrZypkrW1nf1cSq9gaiEa0M5uPOnf0AgdyxC4s8+EUEIhFjeWs9y1vredXpXVPTB9M5njowylP7x9jRO8q2nmHu3Nk/tf8hEYtwyrJGNnY3c+HaZVy4bimr2huq9Clqy2939k+tPINIwS8SUksaE+VAX7tsatp4rsCT+0fZ059m90CKPf0p7tl9iB892gvAyUsbuGjdMv5k3TIuOGVp6LaK5yKVLfDAs4NcfdGaapcyKwW/iExpSMTYfPISNp98+EgU5xxP96W4d/dB7t19kB8/2suNf3gBM9i0onVqRbBldXsgd2RW2r27D5IvOi4J4GGckxT8InJMZsaG7mY2dDdz9Z+sIV8s8fi+Ye7ZdYh7dx/kG3c/w9d/t4e6WITz1izh4vUdXLy+g3WdTaHcYfyDB/fSkowt2EEXfjA334OjK2jLli1u69at1S5DRGaQ9ro27t51kLt2DbC7v3yI8sq2ei7d2MGrTuvigrVLqYst/q2Bu3cN8B+++QCfft3GQBzRY2YPOee2vGi6gl9EFtK+oXF+9/QAd+4c4J5dB8nkizQmoly8oYPLNy3nlRs7aQrIRccXUqFY4g1fvYfxfIFffeLiQKzoZgv+qnz7ZvZx4C8AB2wD3uecm6hGLSKysFa1N/CePz6Z9/zxyUzky+PV3PFkH3c80cfPth0gEYtw8foO3nDmcl5zRteiOaHs+1v3srNvjH99zzmBCP1jqXiL38xWAvcApzvnMmb2A+BnzrlvzfYatfhFal+p5HjohSF+tm0/P992gAOjEzQmorzuzOW87ZyVnL9m6RHjKtWS0Yk8l1x7J+s6m/j+NecHZt9GoFr83nLrzSwPNAC9VapDRCokEjHOXb2Ec1cv4W/ecDoPPjfIDx/u4afb9nPzQ/tY1V7PO7ecxJVbTqK7tbZOIvzab3YzNJ7js288PTChfyxV6eM3s48CXwAywO3OuffMMM81wDUAL3vZyzY///zzlS1SRCoikyty+xMH+P6De/n9nkNErHzW/TvPPYlLN3YeMeJpEG3vGeGKf7mXt569kmuvPKva5RwhMDt3zawduAV4JzAM3ATc7Jz7zmyvUVePSDg8fyjND7bu5aat++gfy9LRXMc7Nq/iz7acxJqAXcwE4LmDad7x9d+TiEb40YcuorM5WFsqQQr+K4HLnXPv9x7/R+B859xfzvYaBb9IuBSKJX67c4DvP/gCv3mqPIzEOS9r44pzVvGmly9/0cCK1dA/OsE7vn4fYxN5bvrghazrbKp2SS8SpD7+F4DzzayBclfPZYBSXUSmxKIRXn16F68+vYsDIxP86NEefvjwPv7mR9v53G07uGxjF2/fvIpLNnRUpStoJJPnqhse5GAqy40fOD+QoX8s1erj/zvKXT0F4BHgL5xz2dnmV4tfRJxz7Ogd5dZHevj3R3s4mMqxtDHBm89ewZvPWsHZJ7VVZMfq9p4R/uut23hy/yjXv/dc/vTUYI7ACQHq6pkPBb+ITJcvlrjr6QFufmgfv36yn1yxxIrWJK87czmvP7Obs09qX/AhpQfGsnzl9p18f+te2hsSfPFtZ/LaM7oXdBkLTcEvIovSSCbPr57o42fb9nP3roPkiuVLiV6yvoNLN3Zy3poldM7zQjPOOR7fN8JPHu/l3x7YSyZf5L0XrubDl51Ka33wr3Gt4BeRRW8kk+d3Tw/wmyf7uPPpAYbH8wC0NcRZ39XM+q4m1nc1s66ziVM7m1nWlDhihVAqOV4YHOepA6M8sneYn287wAuD48SjxmUbu/jU5RtY21E7/fkKfhEJlWLJ8ejeIbb3jPLUgTGe7hvj6QNjjGUPX087GY9QH4+SjEepi0XoG82SyZevaxyNGBeuXcqbXr6C157RTWtD8Fv4RwvSUT0iIr6LRmzGawv0jWbZ1T/Grr4UvcMZsoUSE971tC9tSrCxu5mN3S2s72qmPhHsMXfmS8EvIqFhZnS3JuluTQb6aBy/BftcaBERWXAKfhGRkFHwi4iEjIJfRCRkFPwiIiGj4BcRCRkFv4hIyCj4RURCpiaGbDCzEWDXDE+1AiPHmHa8+5M/lwEH51HaTMufy/NHTz/WY9V9/LqO9/x86p5pWiXrnss0P+s+Xs1zrXG2Ome7P32aH3XP9W9kLrVOvx/Uv+2TnXMvPlPNORf4G3DdXKdPn3a8+9N+bl3Iul5q3cd6rLqrU/cs0ypW91ym+Vn38Wqeb91z/Rvxq+6wZclst1rp6rntJUy/7SXcn+195+p4r59r3cd6rLpnX95cn59P3bN9lvmYT91zmeZn3XN57XzqrpW/kaOn1Urdc6qjJrp6/GZmW90MI9gFnequLNVdWbVYd63UXCstfr9dV+0C5kl1V5bqrqxarLsmalaLX0QkZNTiFxEJGQW/iEjILLrgN7PrzazfzLbP47WbzWybme02s6/atItxmtmHzWynme0ws39Y2Kr9qdvM/tbMeszsUe/2+lqoe9rznzQzZ2bLFq7iqff24/v+vJk97n3Xt5vZihqo+Voze8qr+1Yza1vImn2s+0rvf7FkZgu6M/VE6p3l/a4ys13e7app04/59++r+RxzGuQb8ArgHGD7PF77AHABYMDPgdd50y8FfgXUeY87a6TuvwU+WWvft/fcScAvgeeBZbVQN9AybZ6PAF+vgZpfA8S8+18Gvlwj3/VpwAbgTmBLEOr1all91LQlwDPez3bvfvuxPlslbouuxe+cuwsYnD7NzNaa2S/M7CEzu9vMNh79OjNbTvkf9z5X/q38X+Ct3tP/CfiScy7rLaO/Rur2nY91/yPwnwFfjj7wo27n3Oi0WRsXunafar7dOTd59fH7gVULWbOPdT/pnNu50LWeSL2zeC1wh3Nu0Dk3BNwBXF7t/9tFF/yzuA74sHNuM/BJ4F9mmGclsG/a433eNID1wJ+a2R/M7Hdmdq6v1R52onUDfMjbjL/ezNr9K/UIJ1S3mb0Z6HHOPeZ3oUc54e/bzL5gZnuB9wCf9bHWSQvxNzLpasotz0pYyLorYS71zmQlsHfa48nPUNXPtugvtm5mTcCFwE3TutDqZpp1hmmTLbYY5c2084FzgR+Y2SnemtoXC1T3vwKf9x5/HvgK5X9u35xo3WbWAHyGchdExSzQ941z7jPAZ8zsr4EPAf99gUs9XMgC1ey912eAAvDdhaxxJgtZdyUcq14zex/wUW/aOuBnZpYDnnXOXcHsn6Gqn23RBz/lrZph59zZ0yeaWRR4yHv4Y8ohOX0zdxXQ693fB/zQC/oHzKxEeTCmgSDX7Zzrm/a6/wP8xMd6J51o3WuBNcBj3j/ZKuBhMzvPOXcgwHUf7Ubgp/gY/CxQzd4OxzcCl/nZmJlmob9rv81YL4Bz7gbgBgAzuxN4r3PuuWmz7AMumfZ4FeV9Afuo5mer1M6ESt6A1UzbMQP8HrjSu2/AWbO87kHKrfrJnS2v96Z/EPicd3895U03q4G6l0+b5+PAv9XC933UPM/hw85dn77vU6fN82Hg5hqo+XLgCaDDj+/Y778RfNi5O996mX3n7rOUewzavftL5vLZfP19VGpBFftA8D1gP5CnvFZ9P+UW5C+Ax7w/8s/O8totwHZgD/A1Dp/ZnAC+4z33MPDKGqn7/wHbgMcpt6CW10LdR83zHP4c1ePH932LN/1xyoNlrayBmndTbsg86t0W9EgkH+u+wnuvLNAH/LLa9TJD8HvTr/a+593A+17K379fNw3ZICISMmE5qkdERDwKfhGRkFHwi4iEjIJfRCRkFPwiIiGj4JeaZGapCi/vG2Z2+gK9V9HKI3huN7Pbjjcippm1mdlfLsSyRUBX4JIaZWYp51zTAr5fzB0erMxX02s3s28DTzvnvnCM+VcDP3HObapEfbL4qcUvi4aZdZjZLWb2oHe7yJt+npn93swe8X5u8Ka/18xuMrPbgNvN7BIzu9PMbrbyGPXfnRwj3Zu+xbuf8gZje8zM7jezLm/6Wu/xg2b2uTluldzH4cHpmszs12b2sJXHaX+LN8+XgLXeVsK13ryf8pbzuJn93QJ+jRICCn5ZTP4J+Efn3LnA24FveNOfAl7hnPsjyiNm/v2011wAXOWce6X3+I+AjwGnA6cAF82wnEbgfufcWcBdwAemLf+fvOUfd9wVb2yayyifVQ0wAVzhnDuH8jUgvuKteD4N7HHOne2c+5SZvQY4FTgPOBvYbGavON7yRCaFYZA2CY9XAadPG0GxxcyagVbg22Z2KuUREOPTXnOHc2762OsPOOf2AZjZo5THbLnnqOXkODzg3UPAq737F3B4TPUbgf8xS5310977IcpjtEN5zJa/90K8RHlLoGuG17/Guz3iPW6ivCK4a5bliRxBwS+LSQS4wDmXmT7RzP4Z+K1z7gqvv/zOaU+nj3qP7LT7RWb+H8m7wzvHZpvnWDLOubPNrJXyCuSvgK9SHsO/A9jsnMub2XNAcobXG/BF59z/fonLFQHU1SOLy+2Ux8AHwMwmh9FtBXq8++/1cfn3U+5iAnjX8WZ2zo1QvkTjJ80sTrnOfi/0LwVO9mYdA5qnvfSXwNXeOPGY2Uoz61ygzyAhoOCXWtVgZvum3T5BOUS3eDs8n6A8nDbAPwBfNLN7gaiPNX0M+ISZPQAsB0aO9wLn3COUR3x8F+WLoGwxs62UW/9PefMcAu71Dv+81jl3O+WupPvMbBtwM0euGESOSYdziiwQ7+phGeecM7N3Ae92zr3leK8TqTT18YssnM3A15QytlgAAAAySURBVLwjcYbx+TKXIvOlFr+ISMioj19EJGQU/CIiIaPgFxEJGQW/iEjIKPhFRELm/wOgUtix3Q9b9gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 2e-2\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>25.742983</td>\n",
       "      <td>12.599795</td>\n",
       "      <td>0.973891</td>\n",
       "      <td>0.026109</td>\n",
       "      <td>274962.812500</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>8.696360</td>\n",
       "      <td>7.004283</td>\n",
       "      <td>0.947224</td>\n",
       "      <td>0.052776</td>\n",
       "      <td>679.854126</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(2, lr*10, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned1_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned1_enc_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>6.780813</td>\n",
       "      <td>6.621964</td>\n",
       "      <td>0.906871</td>\n",
       "      <td>0.093129</td>\n",
       "      <td>438.378723</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>6.005637</td>\n",
       "      <td>5.547102</td>\n",
       "      <td>0.754204</td>\n",
       "      <td>0.245796</td>\n",
       "      <td>127.223434</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>5.041389</td>\n",
       "      <td>4.688619</td>\n",
       "      <td>0.657497</td>\n",
       "      <td>0.342503</td>\n",
       "      <td>48.792183</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4.581665</td>\n",
       "      <td>4.350106</td>\n",
       "      <td>0.612980</td>\n",
       "      <td>0.387020</td>\n",
       "      <td>32.847675</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>4.374249</td>\n",
       "      <td>4.186480</td>\n",
       "      <td>0.594299</td>\n",
       "      <td>0.405701</td>\n",
       "      <td>27.221252</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>4.251293</td>\n",
       "      <td>4.090651</td>\n",
       "      <td>0.578639</td>\n",
       "      <td>0.421361</td>\n",
       "      <td>24.088449</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>4.170560</td>\n",
       "      <td>4.033818</td>\n",
       "      <td>0.570340</td>\n",
       "      <td>0.429660</td>\n",
       "      <td>22.518152</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>4.095521</td>\n",
       "      <td>3.955774</td>\n",
       "      <td>0.559905</td>\n",
       "      <td>0.440095</td>\n",
       "      <td>20.811506</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>4.003619</td>\n",
       "      <td>3.898648</td>\n",
       "      <td>0.549537</td>\n",
       "      <td>0.450463</td>\n",
       "      <td>19.151363</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.920332</td>\n",
       "      <td>3.830895</td>\n",
       "      <td>0.541905</td>\n",
       "      <td>0.458095</td>\n",
       "      <td>18.010420</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.818533</td>\n",
       "      <td>3.758636</td>\n",
       "      <td>0.528082</td>\n",
       "      <td>0.471918</td>\n",
       "      <td>16.681326</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.721527</td>\n",
       "      <td>3.696797</td>\n",
       "      <td>0.518993</td>\n",
       "      <td>0.481007</td>\n",
       "      <td>15.535457</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.627698</td>\n",
       "      <td>3.636295</td>\n",
       "      <td>0.508095</td>\n",
       "      <td>0.491905</td>\n",
       "      <td>14.628825</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.522273</td>\n",
       "      <td>3.582196</td>\n",
       "      <td>0.497810</td>\n",
       "      <td>0.502190</td>\n",
       "      <td>13.698022</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>3.442510</td>\n",
       "      <td>3.540197</td>\n",
       "      <td>0.488354</td>\n",
       "      <td>0.511646</td>\n",
       "      <td>13.101725</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>3.366605</td>\n",
       "      <td>3.512025</td>\n",
       "      <td>0.482422</td>\n",
       "      <td>0.517578</td>\n",
       "      <td>12.719119</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>3.311971</td>\n",
       "      <td>3.501606</td>\n",
       "      <td>0.479374</td>\n",
       "      <td>0.520626</td>\n",
       "      <td>12.563033</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>3.283584</td>\n",
       "      <td>3.500383</td>\n",
       "      <td>0.478476</td>\n",
       "      <td>0.521524</td>\n",
       "      <td>12.537847</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8dd3lkz2PSFhD7LvICIWxN0KWmsrtVhr/XW53G7Xpe1t7fV2sb221tva2tbaavXW22sXRa1L3RfEXQMCsm+yBBISCNkzySzf3x9nAoQkMMFMchLez8djHjlzvmcmn8OEd06+53u+x1hrERER9/L0dQEiInJsCmoREZdTUIuIuJyCWkTE5RTUIiIu50vEm2bl5Noxp4xKxFuLiAxIK1as2G+tLeisLSFBnV80lNLS0kS8tYjIgGSM2dlVW0K6PiIamy0i0mMSEtRR5bSISI9JTFArqUVEekxC+qjV9SEi3REKhSgrKyMYDPZ1KQmXnJzM0KFD8fv9cb8mIUGtI2oR6Y6ysjIyMjIYOXIkxpi+LidhrLUcOHCAsrIySkpK4n6d+qhFpM8Fg0Hy8vIGdEgDGGPIy8vr9l8OiRn1oaQWkW4a6CHd5kT2M0FH1ApqEZGeoiNqETnp1dTU8Lvf/a7br1u4cCE1NTUJqKi9hAR1KBJFNyQQkf6iq6CORCLHfN1TTz1FdnZ2oso6JCGjPixwsClEblpSIt5eRKRH3XjjjWzbto3p06fj9/tJT0+nuLiYVatWsX79ei677DJ2795NMBjkuuuuY8mSJQCMHDmS0tJSGhoaWLBgAfPmzeONN95gyJAhPPbYY6SkpPRIfQkJaoCK2qCCWkS67eYn1rF+b12PvufEwZn84GOTumy/9dZbWbt2LatWrWLZsmVcfPHFrF279tAQuvvuu4/c3Fyam5s57bTTuPzyy8nLy2v3Hlu2bOGvf/0r99xzD1dccQUPP/wwn/3sZ3uk/oRNc1pR15yotxYRSajZs2e3G+f861//mmnTpjFnzhx2797Nli1bOrympKSE6dOnA3DqqaeyY8eOHqsnYUfUK3fWcO74QYl6exEZoI515Ntb0tLSDi0vW7aMF154gTfffJPU1FTOPvvsTsdBBwKBQ8ter5fm5p47WE3IEXVakpdlmysT8dYiIj0uIyOD+vr6Tttqa2vJyckhNTWVjRs38tZbb/VydQk6ok5P9rN2Tx2V9UEKM5IT8S1ERHpMXl4ec+fOZfLkyaSkpDBo0OHegIsuuojf//73TJ06lXHjxjFnzpxer88cbxidMWYc8PcjVo0Cvm+t/VVXr5kyfaZtWPBjPv+REr7/sYk9U6mIDFgbNmxgwoQJfV1Gr+lsf40xK6y1szrb/rhdH9baTdba6dba6cCpQBPw6LFeE/B5WHzacO5/cwdvbT8Qd/EiItJRd/uozwO2WWu7vGVMmxsvGs/w3FSuue8d7lq2jdrm0IlVKCJykutuUC8G/hrPhlmpfh768hnMGZXHz57ZyMI7XuWBt3dyoKGl+1WKiJzE4g5qY0wScCnwUBftS4wxpcaY0qqqKgDy0wPc/4XZ/H3JHHLS/Nz06Fpm/+RFvvznFby2Zb/mrRYRiUN3Rn0sAFZaa/d11mitvRu4G2DWrFntEvj0UXk88fV5rN1Tx+Or97B0RRnPrKugJD+Nq04fzqdOHUZWavx3OxAROZl0J6ivJM5uj84YY5gyNIspQ7P45oXjeHptOf/31i7+658b+Plzm/jEjKF8cd5IRhdmnOi3EBEZkOLq+jDGpAIXAI/0xDdN9nv5xIyhPPyVj/DUtWdy2fQhPLyyjPNvX86//rmU3dVNPfFtREQSJj09HYC9e/eyaNGiTrc5++yzKS0t/dDfK66gttY2WWvzrLW1H/o7HmXi4ExuvXwqb954LtefP4ZXt+zngl++wm9f2kI4Eu3pbyci0qMGDx7M0qVLE/o9EjYpU3flpQe4/vyxvPCNszh3fCE/f24zn7zrDbbs6/yyThGRnvSd73yn3ZzUP/zhD7n55ps577zzmDlzJlOmTOGxxx7r8LodO3YwefJkAJqbm1m8eDFTp07l05/+dI/N95GwSZlO1ODsFH531ak89X45//mPtVz8m9f45gVj+dKZo/B6To57qomc1J6+ESre79n3LJoCC2495iaLFy/m+uuv56tf/SoADz74IM888ww33HADmZmZ7N+/nzlz5nDppZd2ed/Du+66i9TUVNasWcOaNWuYOXNmj5TvuqBus3BKMbNLcrnp0ff56dMbeXZdBT//1DRGFaT3dWkiMgDNmDGDyspK9u7dS1VVFTk5ORQXF3PDDTewfPlyPB4Pe/bsYd++fRQVFXX6HsuXL+faa68FYOrUqUydOrVHanNtUIMzDvv3nz2Vx1fv5fuPrWPBHa/y009O4ZMzh/Z1aSKSKMc58k2kRYsWsXTpUioqKli8eDEPPPAAVVVVrFixAr/fz8iRIzud4vRIibibumv6qDvY+BQ8exPGGD4+fQjP3zCfmcNz+MaDq/nFc5t0olFEetzixYv529/+xtKlS1m0aBG1tbUUFhbi9/t5+eWX2bnz2LNnzJ8/nwceeACAtWvXsmbNmh6py71BXfE+vPlb2PYSAIWZydz/hdksOnUov3lpK199YCXB0LFvPCki0h2TJk2ivr6eIUOGUFxczFVXXUVpaSmzZs3igQceYPz48cd8/Ve+8hUaGhqYOnUqt912G7Nnz+6Ruo47zemJmDVrlv3QYwfDLXDXR8BG4Stvgv/wvNZ/ev0Dbn5yPRdNKuK3n5mpk4wi/ZymOf2Q05z2GV8ALv4FVG+H137Zrun/zS3hpoUTeHptBbf8c0MfFSgi0jvcG9QAo86GKVfAa7fD/q3tmr505ig+P3ck973+AQ+vKOuT8kREeoO7gxrgo7eALwX++Q04qpvmpoUTOG1kDt999H02VejCGJH+LBHdsG50Ivvp/qBOL4TzfwAfvALvt79M0+f18LurTiUz2cd1f3tPJxdF+qnk5GQOHDgw4MPaWsuBAwdITu7evWTdezLxSNEo3HsB1OyEr78LKTntml/eWMnn//QuX5xXwvcu0T0aRfqbUChEWVnZcccoDwTJyckMHToUv7/91M7HOpno6gteDvF44JJfwt1nwYs/hktub9d8zvhCrp4zgntf+4CzxxVw5piCPipURE6E3++npKSkr8twLfd3fbQpngqnfwVK74Oyjkfr/7FwAqML0/nmg6s52NjaBwWKiCRG/wlqgHO+CxnF8OT1EAm3a0pJ8nLH4ukcbGrlxkfWDPi+LhE5efSvoA5kwIKfOVctvvOHDs2TBmfxrQvH8ey6fTxYursPChQR6Xn9K6gBJnwMxnwUXroFajuOn/6XM0fxkVPyuPmJ9ezY39gHBYqI9Kz+F9TGwMLbnEvLn7mxQ7PHY/jFFdPwGsMPHl+nLhAR6ff6X1AD5IyEs74NG56ATc90aC7OSuH6C8byyuYqnlvf6U3TRUT6jXhvbpttjFlqjNlojNlgjDkj0YUd1xlfh4Lx8NS/Q2vHm+Fec8YIxhdl8KMn1tMS1oUwItJ/xXtEfQfwjLV2PDAN6PuZkHxJcPHtULsLlt/Wsdnr4aaLJ7Cnppm/vL2rDwoUEekZxw1qY0wmMB+4F8Ba22qtrUl0YXEZORemfxbe+A1UdvzdMW90Ph85JY87XtyisdUi0m/Fc0Q9CqgC/scY854x5o/GmLQE1xW/C37kDNt78gbnUvMjGGP4wccmUR8M88sXNvdRgSIiH048Qe0DZgJ3WWtnAI1Ah+EWxpglxphSY0xpVVVVD5d5DGl5cMGPYdebsPovHZrHFWWw+LRh/OXtXew60LEvW0TE7eIJ6jKgzFr7duz5Upzgbsdae7e1dpa1dlZBQS/PtTH9Khh+Bjz3PWg80KH52vPG4PMafvH8pt6tS0SkBxw3qK21FcBuY8y42KrzgPUJraq7PB7nxGJLHbzw/Q7NgzKT+eK8Eh5btZfSHdV9UKCIyImLd9THvwEPGGPWANOBnySupBM0aKIzZO+9/4Odb3Ro/to5oxmclcz3HlunO5iLSL8SV1Bba1fFujWmWmsvs9YeTHRhJ+Ssb0PWcHjyGxBuP8ojNcnHf14ykQ3ldTyg4Xoi0o/0zysTu5KUBgv/G6o2wFt3dmheMLmIeaPz+flzm9jf0NIHBYqIdN/ACmqAcRfB+Etg2c/g4M52TcYYfnjpJIKhCD97emMfFSgi0j0DL6jBmQrVeJzLy4+alGl0YTpfmFfCQyvKWLHTnT04IiJHGphBnTUUzvkP2PIsbHyyQ/O1546hKDOZHz6+jkhUs+uJiLsNzKAGOP3LMGgKPP0daKlv15QW8PHdheN5f08t97+xo2/qExGJ08ANaq/PuSFu3V5YdmuH5kunDeaccQXc9uxGXbEoIq42cIMaYNhpcOo18NZdUL6mXZMxhp98cgo+j4fvPba2jwoUETm+gR3UAOf9AFLz4OEvdugCKc5K4frzx/DK5ipe0A0GRMSlBn5Qp+bConvhwFZ4/NoOo0Cujt1g4NsPr6GmSVOhioj7DPygBiiZD+d+D9Y9Am+3v3t5wOfll5+ezsGmVm57VpM2iYj7nBxBDTD3ehi3EJ67CXa93a5pQnEmX5pXwl/e3sUrm3txilYRkTicPEHt8cBldzljrB+6BhraB/I3LxzH6MJ0vrN0DbVNoT4qUkSko5MnqAFSsuGKP0PzQXj4CxA9fNPbZL+X26+YRlVDCzc/sa4PixQRae/kCmqA4qnO3NUfLIeXb2nXNHVoNl87+xQeeW8PD68o66MCRUTaO/mCGmDGVTDzGnj1F7Dp6XZN/3beGOaMyuU//7GW7VUNfVSgiMhhJ2dQAyy4DYqnwSP/CtUfHFrt93r41adnkOTz8PW/vEdjS7gPixQROZmD2p8MV/wvGODBqyHUfKipKCuZXy2ezsaKOr710Gqs1cRNItJ3Tt6gBsgZCZ+8Byreh6e+1a7pnHGF3LhgPE+vreDmJ9YrrEWkz5zcQQ0w9qMw/9+dey2u/N92Tf9y5iiunD2MP72xgzte3NJHBYrIyc4Xz0bGmB1APRABwtbaWYksqted/V0oexf++S0omgqDpwPOxE23XDaF1rDlVy9swe/18LVzRvdxsSJysunOEfU51trpAy6kATxeuPxeSMuHBz/njLNua/IYbls0lYunFvPfz27S/NUi0uvU9dEmLR8+db8zf/WjX4Zo9FCT12O4/YppnDu+kB88vo7Xtuzvw0JF5GQTb1Bb4DljzApjzJJEFtSnhp0GH/0JbH4GXru9XVPA5+XOz8zklII0vvXQaqobNdOeiPSOeIN6rrV2JrAA+JoxZv7RGxhjlhhjSo0xpVVV/Xhio9n/ApMXOVctbnu5XVNKkpc7Fs+guqmVz//pXRo0xlpEekFcQW2t3Rv7Wgk8CszuZJu7rbWzrLWzCgoKerbK3mQMfOwOyB/r3Gygdk+75slDsrjzMzNZu6eWz9zzFgcaWvqoUBE5WRw3qI0xacaYjLZl4EJgYN+7KpDuTN4UbnFm2gu37+a4YOIgfnPlDDaW1/Op379JZV2wjwoVkZNBPEfUg4DXjDGrgXeAf1prn0lsWS5QMBY+/ltn2N5z/9mheeGUYv7vS6ezry7IZ/74NmUHdYNcEUmM4wa1tXa7tXZa7DHJWnvL8V4zYEz6BMz5KrzzB3h/aYfm2SW53PO5WeyrDXLZna9z9/JtNLdGOnkjEZETp+F5x3PBj2DYHOd+i5UbOzR/ZHQ+j35tLtmpSfzkqY1c/OtXWb27pg8KFZGBSkF9PF4/fOp/ICnVmbzpqDuZA4wuTOeZ687k11fOoCUc5Yo/vMntz20iHIl28oYiIt2joI5H5mBYdJ9zJ/PHvuacZDyKz+vh0mmDeezrc5k8JItfv7SVq/74Nm9s2099ULf2EpETZxIxK9ysWbNsaWlpj79vn3v9Dnj++5A5BOZ/C6Z/FnxJHTYLR6Lc+9oH/PalrdS3hEn2e/jkzKFce+4YirKS+6BwEXE7Y8yKrqboUFB31/Zl8NItUPYOZA+H+d+GaYudLpKj1DaFWLn7IM+ureDhlWUk+71cd94YvjivBGNM79cuIq6loO5p1sLWF+Hl/4K970FOCZx9I0z5lDPBUyd27G/kB4+v45XNVUwbmsWVs4dz1rgCirNSerl4EXEjBXWiWOvcc/Hln8C+9yFvjBPYkz4Jno7d/9ZaHlpRxm9e2sLu6mbSkrzcuGA8V84ejs+r0wUiJzMFdaJFo7DxCXj5p1C1AQomwDnfhfEf6zKwN1bU8+Mn1/PGtgOML8rgC/NKuHTaYJL9nR+Ri8jApqDuLdEorHsElt0KB7ZA0RQ45yYYe5Ezh8hRrLU8vbaCW5/eyK7qJoqzkvn3j45j/tgC8tMDfbADItJXFNS9LRKGtUudwD74AQye6QT26PM6Dexo1PKXd3bxoyfX0xp2xl6Pyk/jwklFzBmVy4zhOQR8Hh1tiwxgCuq+EgnB6r/CK/8Ntbtg2Olwzn9AyVmdBnYkalm1+yDLNlWxancNrx5xg4K0JC8zhueQm5bE2EHpnHFKPhOKM0hNiutuaiLicgrqvhZuhff+DMt/DvV7YcRc5wh75NxjvqyiNsjLmyqprGthZ3Uj26sa2d/QQtnBZgA8BiYOzmRCUSafO2MkU4Zm9cbeiEgCKKjdIhSElffDq7+Ahn1OH/bI+U5gDz8DUnPjepv9DS2s3HmQFTsP8s6OajaU1xEMRRmVn8a8MfksOnUoAPXBMFOHZpGR3HGMt4i4i4LabVqbYMWfYNNTsPsdiLQABgZNgpHznCPuEXMhLS+ut6tpauWRlXt4aEUZG8rr2rUleT1cMGkQV8wahgHSAl5GF2aQluTFGIPXowtvRNxAQe1moSDsWQE7X4cdrznBHXa6NiiY4Bxtj5jrBHh64THfylrLlsoGNpTX4fd6aG6N8O6Oah5eWUYo0vnnfOqIHEry0yjJT2Pu6HxK8tPIStERuEhvU1D3J+FW52rHna/Bjtdh11sQanTa8sceDu0RcyGzOK63rKwPsnp3Lcl+D63hKFsqGwiGIqzeXcPWqgYONoZobA1jLfg8hstnDuVLZ5aQFvDh93rIT0/CGEM0avHoCFwkIRTU/VkkBOWrnaPtnbHgbol1b+SOOtxNUjgB8k6BQMYJfZvKuiArdh7k+fX7eHJNOa1HTNFqDKQn+WhsDZOXHiA1ycvkIVnMG53PiNxUJhRn4jEGr9eQHtAoFJEToaAeSKIRqFjjHG3veA12vQHB2sPtaYVOYOeeAnmjYl9PcUI9KS2ub3GgoYV/rNqLARpawjSHIjS3Rgj4PZTXBNm8r57qxlYq6ztO95qa5KUwI0BGsp9zxhdy3Xlj1A8uEgcF9UAWjUDVJudKyAPboHobHNjufG3Y137bjOJOAvwUyC0Bf/cmh4pGLbuqm1i56yA1TSFawlFawhHqg2G2VzVQXhtkY0U9owrSuGTqYM4aW8CMYdmddp1YazWboJz0FNQnq5Z6qN7eMcAPbIOm/UdsaJw5tvNGQfYI56Rlaj6kFUBa29cCSM0Db/xdG0+u2cs9y7fz/p5aorEfs4DPw9hBGXgMpCf7KK8JUnawmaxUPxnJPlrDUaYPyyYzxU9FrXN398KMAFefMYJJgzVOXAauHglqY4wXKAX2WGsvOda2Cup+IFh7RIgfEeY1u50Qj4Y7f11K7uHgTss/IsiPCPS258nZYAy1TSGWba7k3R3VeIxhW1UDwVCU5tYIRVnJDMlOoTkUoak1TFNrhFW7a4hELUOyU/B6DFv2NdAaiTJjeDZnjs5nZH4aYwdlUJgZoDBDN2KQgaGngvobwCwgU0E9wEWjEKyBxv1OaDdWxR5dLDcf7Px9fCmQNdR5ZA+DrGGHn2cNdY7ifR0nn2r7mWzrDqmqb+F/Xv+AV7fsZ93ew0fnHgPnji8k2e9lb00zkajl1BG5+H2GVL+P6cOzyUn14zGGcNTiMTB2UIbmTBFX+tBBbYwZCtwP3AJ8Q0Et7URC0HSgfYg3VELdXqjdDbVlzqOx8qgXGkgf1D6828K8LdhTctrNi1IfDLFjfxPry2vZXtXIP1btIdnvnMBsDkXYsq+BqLVdjhsfmZfKDReM5YKJgzRPirhKTwT1UuCnQAbwrc6C2hizBFgCMHz48FN37tz5oYqWASgUhLo9h4O7dnf7IK8tg3Cw/Wv8qc6Rd3rhUd0rXXe1RKKW+mCI9eV1NASdLpzWSJR9dS388dXtlNcGGVOYztVnjCAnNYnB2SkUpAcOHYmnBby6kYP0ug8V1MaYS4CF1tqvGmPOpougPpKOqOWEWOscmR8d3rVlR3W1VHf+eo+v8xBPzTu0HEkrZHm5h+88U0FlY6TLUiYUZ5KXlkReehKjC9IZlpvK9v2NsXlVIpTXBrl6zgiunD2cJJ9CXT68DxvUPwWuBsJAMpAJPGKt/WxXr1FQS0JFQtBU3UXfeSfPQ00d3sIaD5HUQoLJBTQFCqn359GQVEitL5+ycCYrq5PZG81iS32AqobWQ68rykwmJy2JptYwOw80kZ8e4BMzBnPmmAKG5KSQnxagLhgi4PfoRKd0S48Nz9MRtfRLrY1OYDdUOf3k9eVQXwF15YeX68s7P1L3JhFJL6IluRBvVjGBnCGQUYRNL+L9+jT+8F6QF/f6CNLxpOjFU4u5cOIgslOT2FvTTLLfg8EwJMcZzZKXlkROWhIZAZ/Gkcsxg1pnU2TgS0pzHjkjj71dKOhcJFRf4cwbHgtwb30FqfXlUL0Jdr4CLXUYYCpwJ0AyhJKyaUoeRK0vn1BaEWWRHJ7f6GPp2mzKbS4VNpcGUjv9tqPy07j18qmcOiJHV3FKp3TBi0h3tdTHjsj3OkfidXuc5bq9h5cbqzq8LOJPozW1iKbAIFrTiqjxFVBp8nh4S4QdwXSCgXwmjxvN2MF5zByew7iiDNIDPoX3SUJH1CI9KZDhPPLHdL1NuCUW4nsPPbx1e0mp20NK3V7Y/xbFDRVMsFHOAg71nGyCgxvTqbJZrLXZ7Ceb5qRcWpLzqbLZeDIGEUktZMSIkaTlDMLr9ZKa5MxyWFkf1LDDAUqfqEgi+AJOV8uxulsi4VhXS7kz7rxhH+G6fTSX7ya9YR9ZTZVMCu4gpXUFgcbYsMW286IfQNh6OEAm+20WVTabVpvF0qQ8pk0Yy7hTTiE5p9iZpCu9sMN4dOlfFNQifcXrg6whziPGBwzubNuWBifUGyqxDfuoqSwjWFNBUnMVhfX7KG6qJKV1M77m/fjXhWFd+5dHjZ9Iah6+zEGY9EGxAC9wLjhKK3DCvG1Zoe46CmqR/iCQ7jzyTsEAOZM63ywaifL6+m1s/WA7VeW7sA2VNFWXU2Bqya+tJb+uluFJ2yjyrSKl9QBe23EseQgfjb4c/JkFpGUPcsalp+Y7t4ZLzY+NS29bF7vQyKOx5ImkoBYZQDxeD3OnjGHulMP95wcbWymvDbK+vI5lZTW880E1GyvqMUTJopFZ+SFGBhrJsTVkRapJDx/ENlSSWVXLKU0VDPJuJRA6iKe1ofNvarzOjZnbgrvdcux5clbskX142a9x5vFSUIsMcDmx8doTB2ceukN9Zb3T553i95LeyTju5tYIv3lpC//2xg6aWp2j7gCtDPI1cv4IL/OHGIYFmsjzNJAWOogvWI1p2u9cWbpvvfO1+SBwjFFl3oAT2ClHhHdngX7kuqQ0SEp1phbwpzrzqJ8E3TQaniciXQqGIry1/QDLNlUR8Hmobwnz8sZKymvbz8mS7PcwriiTGcOyGVeUQWNLmG0VNdTXVPGpiemcNTzJmVo3WBN71DqP5iOWDz1i67qaarcd4wR2W3gnpR0O8LblpFTwp7X/6guAx+9MO+Dxgcd7xHJX67p4bmLdPod+YZgTem4yCjU8T0S6L9nv5exxhZw9rvDQOmstm/c1UFkfZOeBJmqbQ5TXNrOtspG/vL2r3f02fR7Dk9tqWDC5iI9Pn8xZ4wpISYpjmllrnUv/jw7zUCO0NjltrY2xr03O+lBz+3V1e2JtzYdfFw0l4p8p4XRELSI9prk1Qk1zKz6Pc/f6SNRyx4tb+MPy7bSGoxgDg7NSmF2SS7Lfg8/jYVBmgKr6FgoyAhRnpVDd2MrO6kZ2VzdTURuMHa1nMGN4DpdMLSYj2X/iBUZCTpiHg85t7KLh2OPI5Xiet60LOb9U2rp4DuVpd5+DOX2JbsUlIn2nPhhi5a4aVuw8yMbyOlaX1RCO2NidfSL4PM7NHY40ujCdEbmp1DSHeL+sltZIlJxUP58+bTjnjCtg2rDsAXUTCN0zUURcqSUcIRSxpPq91DaHqA+GSU/2kZrkbRfC4UiUNXtqufOlrSzbXEUkFurzxxYwLCeFvPQA+elJ5KUFGJ6bysj8VHweD42tYZpaIlgsHmOoC4bYVFHPsNxUZo3IcdVkWApqERkwaptCPLe+gmfXVbB5XwONLWGqm1rpbpSNKUxn4uBMpgzJ4uPTh1CQEcBaSzAUxe81NIUi2CikBrz4e+FGEgpqERnQIlHLwaZW9je0sK2ykV3VTbSGo6QmeUlP9tHY4owgyUj2MXN4Du/uOMgz6yrYVFHHvroWPAbSAj6iUUtja/uLgIyB9CQfKUlOYEdjmZmS5MXnMRRlpTBnVC7Dc1PJSvGTleKnMCOZvPSkdgG/u7qJumCIFTsPUh8MM2N4NhkBP/XBEOnJPqYNy9GoDxEZuLweQ356gPz0AOOLMo+7/ZhBGXzm9OEAbK9q4LFVe6ltDmEMZCb7sdaSGvDh8xjqg2HqgiGaWiIEwxFqmkJkpfiJWks4YtlYUcfyzR1nS0wP+CjICFDbHKIhGG43Gqa7FNQiclIbVZDODReM/VDvURcMsbemmcq6FhpawpTXBtmyr57G1giZsT73vPQAQ7JTmDQ4k9y0JN7aXo3XY0gP+Civbebyn3X9/gpqEZEPKTPZT2aRn/FF8b/mosnxb6yZVEREXE5BLSLicgpqERGXO25QG2OSjTHvGGNWG2PWGWNu7o3CRETEEc/JxBbgXGttgzHGD7xmjHnaWvtWgmsTERHiCDchqIUAAAhTSURBVGrrXBHTNmO4P/bo+atkRESkU3H1URtjvMaYVUAl8Ly19u1OtllijCk1xpRWVXUc/C0iIicmrqC21kastdOBocBsY8zkTra521o7y1o7q6CgoKfrFBE5aXVr1Ie1tgZYBlyUkGpERKSDeEZ9FBhjsmPLKcD5wMZEFyYiIo54Rn0UA/cbY7w4wf6gtfbJxJYlIiJt4hn1sQaY0Qu1iIhIJ3RlooiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nLx3IV8mDHmZWPMBmPMOmPMdb1RmIiIOOK5C3kY+Ka1dqUxJgNYYYx53lq7PsG1iYgIcRxRW2vLrbUrY8v1wAZgSKILExERR7f6qI0xI4EZwNudtC0xxpQaY0qrqqp6pjoREYk/qI0x6cDDwPXW2rqj2621d1trZ1lrZxUUFPRkjSIiJ7W4gtoY48cJ6QestY8ktiQRETlSPKM+DHAvsMFae3viSxIRkSPFc0Q9F7gaONcYsyr2WJjgukREJOa4w/Osta8BphdqERGRTujKRBERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOXiuQv5fcaYSmPM2t4oSERE2ovniPpPwEUJrkNERLpw3KC21i4HqnuhFhER6USP9VEbY5YYY0qNMaVVVVU99bYiIie9Hgtqa+3d1tpZ1tpZBQUFPfW2IiInPY36EBFxOQW1iIjLxTM876/Am8A4Y0yZMeaLiS9LRETa+I63gbX2yt4oREREOqeuDxERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOXiCmpjzEXGmE3GmK3GmBsTXZSIiBwWz13IvcCdwAJgInClMWZiogsTERFHPEfUs4Gt1trt1tpW4G/AxxNbloiItPHFsc0QYPcRz8uA04/eyBizBFgSe9pijFn74cvrc/nA/r4uogdoP9xnoOyL9qPnjOiqIZ6gNp2ssx1WWHs3cDeAMabUWjsr7vJcSvvhLgNlP2Dg7Iv2o3fE0/VRBgw74vlQYG9iyhERkaPFE9TvAmOMMSXGmCRgMfB4YssSEZE2x+36sNaGjTFfB54FvMB91tp1x3nZ3T1RnAtoP9xloOwHDJx90X70AmNth+5mERFxEV2ZKCLicgpqERGX69Gg7m+Xmhtjdhhj3jfGrDLGlMbW5RpjnjfGbIl9zYmtN8aYX8f2bY0xZmYf136fMabyyPHqJ1K7Meaa2PZbjDHXuGQ/fmiM2RP7XFYZYxYe0fbd2H5sMsZ89Ij1ffqzZ4wZZox52RizwRizzhhzXWx9v/pMjrEf/eozMcYkG2PeMcasju3HzbH1JcaYt2P/tn+PDZDAGBOIPd8aax95vP3rVdbaHnngnGjcBowCkoDVwMSeev9EPIAdQP5R624Dbowt3wj8LLa8EHgaZ1z5HODtPq59PjATWHuitQO5wPbY15zYco4L9uOHwLc62XZi7OcqAJTEft68bvjZA4qBmbHlDGBzrN5+9ZkcYz/61WcS+3dNjy37gbdj/84PAotj638PfCW2/FXg97HlxcDfj7V/vfmzZa3t0SPqgXKp+ceB+2PL9wOXHbH+f63jLSDbGFPcFwUCWGuXA9VHre5u7R8FnrfWVltrDwLPAxclvvrDutiPrnwc+Ju1tsVa+wGwFefnrs9/9qy15dbalbHlemADzlW9/eozOcZ+dMWVn0ns37Uh9tQfe1jgXGBpbP3Rn0fb57QUOM8YY+h6/3pVTwZ1Z5eaH+sDdgMLPGeMWWGcS+ABBllry8H5oQUKY+v7w/51t3Y379PXY10C97V1F9BP9iP2Z/MMnKO4fvuZHLUf0M8+E2OM1xizCqjE+YW3Daix1oY7qelQvbH2WiAPF+wH9GxQx3WpucvMtdbOxJkZ8GvGmPnH2LY/7l+brmp36z7dBZwCTAfKgV/E1rt+P4wx6cDDwPXW2rpjbdrJOtfsSyf70e8+E2ttxFo7Hedq6tnAhGPU5Nr9gJ4N6n53qbm1dm/sayXwKM6Hua+tSyP2tTK2eX/Yv+7W7sp9stbui/0niwL3cPhPTVfvhzHGjxNuD1hrH4mt7nefSWf70V8/EwBrbQ2wDKePOtsY03ah35E1Hao31p6F0yXniv3oyaDuV5eaG2PSjDEZbcvAhcBanJrbzrRfAzwWW34c+FzsbP0coLbtT1oX6W7tzwIXGmNyYn/KXhhb16eO6vv/BM7nAs5+LI6doS8BxgDv4IKfvVh/5r3ABmvt7Uc09avPpKv96G+fiTGmwBiTHVtOAc7H6W9/GVgU2+zoz6Ptc1oEvGSds4ld7V/v6uEzrQtxzhJvA27q7TOj3ax1FM7Z3NXAurZ6cfqlXgS2xL7m2sNnke+M7dv7wKw+rv+vOH+ChnB+63/xRGoHvoBzgmQr8HmX7MefY3WuwfmPUnzE9jfF9mMTsMAtP3vAPJw/idcAq2KPhf3tMznGfvSrzwSYCrwXq3ct8P3Y+lE4QbsVeAgIxNYnx55vjbWPOt7+9eZDl5CLiLicrkwUEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOX+P2St4jFkRs8mAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(18, lr, wd=wd, moms=(0.8,0.7), callbacks=[ShowGraph(learn_lm)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned2_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned2_enc_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save best LM learner and its encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned_enc_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning \"forward Classifier\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 80 ms, sys: 32 ms, total: 112 ms\n",
      "Wall time: 678 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = load_data(path, f'{lang}_databunch_lm_tcu_jurisp_reduzido_sp15_multifit_v2', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.27 s, sys: 140 ms, total: 1.41 s\n",
      "Wall time: 2.51 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = (TextList.from_df(df_trn_val, path, vocab=data_lm.vocab, cols=reviews, processor=SPProcessor.load(dest))\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_from_df(cols=label)\n",
    "    .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 188 ms, sys: 28 ms, total: 216 ms\n",
      "Wall time: 240 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas.save(f'{lang}_textlist_class_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get weights to penalize loss function of the majority class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 384 ms, sys: 0 ns, total: 384 ms\n",
      "Wall time: 384 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_tcu_jurisp_reduzido_sp15_multifit_v2', bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9237, 1026, 10263)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_trn = len(data_clas.train_ds.x)\n",
    "num_val = len(data_clas.valid_ds.x)\n",
    "num_trn, num_val, num_trn+num_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([3136, 2439, 2084, 1578]), array([332, 284, 213, 197]))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_LabelCounts = np.unique(data_clas.train_ds.y.items, return_counts=True)[1]\n",
    "val_LabelCounts = np.unique(data_clas.valid_ds.y.items, return_counts=True)[1]\n",
    "trn_LabelCounts, val_LabelCounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.6604958319800801,\n",
       "  0.7359532315686912,\n",
       "  0.7743856230377828,\n",
       "  0.8291653134134459],\n",
       " [0.6764132553606238,\n",
       "  0.723196881091618,\n",
       "  0.7923976608187134,\n",
       "  0.8079922027290448])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_weights = [1 - count/num_trn for count in trn_LabelCounts]\n",
    "val_weights = [1 - count/num_val for count in val_LabelCounts]\n",
    "trn_weights, val_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training (Loss = FlattenedLoss of weighted LabelSmoothingCrossEntropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 344 ms, sys: 24 ms, total: 368 ms\n",
      "Wall time: 364 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_tcu_jurisp_reduzido_sp15_multifit_v2', bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, pretrained=False, drop_mult=0.3, \n",
    "                                  metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_tcu_jurisp_reduzido_sp15_multifit_v2');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Change loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_weights = torch.FloatTensor(trn_weights).cuda()\n",
    "learn_c.loss_func = FlattenedLoss(WeightedLabelSmoothingCrossEntropy, weight=loss_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of WeightedLabelSmoothingCrossEntropy()"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_c.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xV9f3H8dfnZi+SQJgJe4psAoqroBZXHdSF6ydi9YfWVe2wy9ra9le1ttVaV62iddXVOuusiAoIYQ8B2YSRBEgC2ev7+yM3GGkCgeTck+S+n4/HfXDvPefe876H5H7yPd9zvl9zziEiIuEr4HcAERHxlwqBiEiYUyEQEQlzKgQiImFOhUBEJMxF+h3gcKWlpbk+ffr4HUNEpE1ZuHDhLudc54aWtblC0KdPH7KysvyOISLSppjZ5saW6dCQiEiYUyEQEQlzKgQiImFOhUBEJMypEIiIhDkVAhGRMKdCICIS5lQIQqy8qpq/z9vMhrwiv6OIiABt8IKylvLwrPW8s3Inj/9PJp2TYkKyza17Svjuc4tYll1IdESA6yb257qJ/YmNigjJ9kVEGhKWheCNpdu5+53VAHzn6SxeuOZY4qKb/mX8Zc4+7nl3DRVVNURFGJGBAAkxkaSnxpERvPXulED3DrEEAgbAf1bn8L1/LKXGOX5/4Uhmr83j/g+/5I2l27nrvGEcPyDNk88qInIo1tZmKMvMzHTNGWJieXYhFzwyh+HpyVx1fF9ueH4Rk4d25aHLxhIR/NIGqK5xX3tcJ3dvGVMemkNReRV90hKoqq6hsrqGfWVV7NxbRv3dGRsVoG9aIl2SYvh4bR5H9+jAQ5eNoXenBABmr83j56+tYPPuEk4cmMaNJw9kfN+OR/zZREQaY2YLnXOZDS4Lp0KQu7eMcx78jIiA8doNx5OWGMOTn23kl2+s4uoT+vKzs45i8dYCnp23hbeWb+foHsn86eJR9OwYD0BJRRVTH5vHutwiXvzfCQxLT/7a+1dU1bCjsJSte0rZvKeYDXnFbMgrYvPuEo4fkMZPzzrqvw4DlVVW89ScTfz1kw3sKqpgfN+OfO/UQUzo3+nIdpCISANUCKj9wr34sXms3bmPV647jqE9Ouxf9ss3VvLkZ5vo3SmezbtLSIiOYPLR3fhgVQ4Y/O7bIzh9WDdmPLOQD7/I4a//k8kpR3VtyY9FaUU1z8/fwqOz15O7r5zfThnOJeN7teg2RCR8HawQhE0fwWtLtrF0awGPXD72a0UA4GdnDSW/uIINu4r5zZRhnDsqncSYSLbsLuGmFxbz3ecWMbR7B1bt2MudZw9t8SIAEBcdwfQT+nLpMb2Y8cxCfvzqcqqqa7hiQp8W35aISH1h0yJwzrFkawGje6Ue1usqq2u47721PPLxeqYd14c7zzn6sLd9uMqrqvnus4v44Itcfv6toVx9Ql/Ptyki7ZsODbWAXUXldEqIxuy/O5C9UFFVw03PL+adlTu57JheTOjfiSHdOtCnUzyREbr8Q0QOjwpBG1VZXcNPXl3Oq4u3UV1T+/8UHRlg+vF9+dHpg0NWlESk7VMfQRsVFRHg3gtHctd5w1iXW8Sanfv4z5pcHvl4PXFREdx86kC/I4pIO6BC0AbERkUwLD2ZYenJTBmdTmxkBH/8YC0dE6LUmSwizaZC0MYEAsbd5w+nsLSCO15fSXJ8NOeM7NGi23DOsSy7kI/X5tGrYzzD0pPpl5aw/yrpg6mucby/aif/WrydEwelcen4Xk0+hOWcY9GWfJ6Zt4V+aQnccPIAHf4SCQEVgjYoMiLAg5eO4X/+Np9b/7GEbfmlXDGhN4kxzfvv3FNcwb8Wb+PFrK2s3rnva8sSYyLp3Sme0spqisqqKCqvIj46grG9UxnXpyOZfTqyPLuAxz/dyObdJSTFRPLOyp3MXpvH3eePICU+usFtOufYW1rFh6tzmDlnU+04TJEBKqpq2FtWyU/OPErFQMRj6ixuw/aWVXLjc4v5eG0eyXFRTDuuD9OO60Nqwte/dKtrHNn5JWzIKyZ3XxkVVTWUB287CktZn1vMhl1F5OwtB2BERjIXZfbkrOHd2bm3jOXbClmxrZCte0qIj4kkKSaSxJhI9pRUkLUpny17SvZva1TPFK49qR/fHNqVp+Zs4u53VtM5MYbfXziSpNgo1ubsY23OPtbnFZGdX8q2/FL2lVcB0L9zAtOO78u3R6dzzzureWruZq46vg93fGuoioFIM+msoXZu8ZZ8Hpq1nvdX5RAVYSTHRZMQE0FcVAQ1zrFpdwkVVTUNvrZDbCT9uyTSLy2Rfp0TOHlIF47q3qHBdRuzs7CMRVvy6dohljG9Ur72pb0su4Cbnl/Mpt1fFYvoiAB90xLo2TGOjNR40lPiOLpHByb077T/tc45fvXmKp78bBPTjuvDL85WMRBpDhWCMLFm5z5eXZzN3tJKSiqqKamoBqBfWgL9OyfSv0sC3ZLjiIkMEB0ZIDoiQExkwPMv2KLyKv61eBudEqIZ1C2J3h2bdi2Ec47fvPUFj3+6kb5pCZw0MI0TB3bm2P6dmn0YTCTcqBBIm+Wc46WsbP69YgfzNuyhtLKa6MgA9188ijOGd/c7nkib4ct1BGb2BPAtINc5N6yB5ROB14CNwadedc79yqs80jaZGReN68lF43pSXlXNws353PvuGm7+xxLSkmIY10fDdos0l5djFcwETj/EOp8450YFbyoCclAxkREc1z+NJ64cR0ZKHNc8ncW6XE35KdJcnhUC59xsYI9X7y/hKzUhmplXjScyYEx7cj65+8r8jiTSpvk9etkEM1tqZv82s0aH9TSza80sy8yy8vLyQplPWqleneJ5Yto4dhdVcMXj83lm3mbW5RbR1vq8RFoDTzuLzawP8GYjfQQdgBrnXJGZnQnc75w75OA56iyW+matyeXHry5nR2FtqyAtMYazhnfjJ2cdRUxk0+ehFmnvWuWgc865vfXuv21mD5lZmnNul1+ZpO2ZOLgLc24/mc27S5i3YTefrtvFU3M3sy6viEevyNRppiJN4NuhITPrZsET2M1sfDDLbr/ySNtlZvRJS2Dq+F48eOkY7rtwJPM27OHSv85jd1G53/FEWj3PCoGZPQ/MBQabWbaZXW1mM8xsRnCVC4AVZrYUeACY6nSAV1rA+WMzePTysazZuY8LH51Ldn7JoV8kEsZ0QZm0Wws27WH6zAWkxEfx0v8eR7fkWL8jifjmYH0Efp81JOKZcX068ux3jiG/uJLL//Y5e4or/I4k0iqpEEi7NiIjhcevzGTrnhKufGI++8oq/Y4k0uqoEEi7d2y/Tjx8+Ri+2LGXq5/Koqyy2u9IIq2KCoGEhZOHdOUPF49iwaY9XPDIHNbnaWgKkToqBBI2zhnZg8euyCQ7v5RvPfApL8zfoiuRRVAhkDDzzaFdefeWkxjTO4XbX13Odc8sUr+BhD0VAgk7XTvE8vfpx/DjM4bw/hc5XP/sIiqrG57BTSQcqBBIWAoEjP/9Rn/+b8pwPvlyFz//1wodJpKwpYFYJKxdNK4nW/aU8OBH6+jZMZ7vThrgdySRkFMhkLB32+RBbM0v4d5315CRGse5o9L9jiQSUjo0JGHPzLjnghGM79OR215cyg9fXqqZzySsqBCIUDsN5l+vzOTSY3rx2pLtfPOPH3Pt01kszy70O5qI51QIRIKS46L41bnDmHP7ydw4aQCfb9zDlIc+45WF2X5HE/GUCoHIATolxnDr5MF88qNJHNOvI7e9tJS/fLROZxVJu6VCINKIDrFRPDltPOeO6sG9767hjtdWUl2jYiDtj84aEjmI6MgAf7xoFF07xPLY7A3kl1Rw/9TRRATM72giLUaFQOQQAgHjJ2ceRWp8NHe/s5qk2Ch+O2UYwZlWRdo8FQKRJrpuYn/2lVXy0Kz1pMZH8cPTh/gdSaRFqBCIHIYfnDaY/JK6YhDNNSf18zuSSLOpEIgcBjPj1+cNY29pJb95+wvSkqKZMjrD71gizaKzhkQOU0TA+MPFIzm2X0duf2U5K7frojNp21QIRI5ATGQED146htT4aGY8s5DCEs1pIG2XCoHIEUpLjOGhy8ews7CMW/6xmBpdYyBtlAqBSDOM6ZXKHd8aykdr8vjzf9b5HUfkiKgQiDTT5cf25tuj0/nTh2t5f1WO33FEDpsKgUgzmRm/mTKc4enJ3Pj8IpZsLfA7kshhUSEQaQFx0RH87cpxdEmK5eqZC9i0q9jvSCJNpkIg0kI6J8Uw86px1DjHtCfns7uo3O9IIk2iQiDSgvp1TuTxK8exo7CMq5/KoqKqxu9IIoekQiDSwsb2TuWeC0awZGsBby/f4XcckUNSIRDxwNkjetA3LYG/z9vsdxSRQ1IhEPFAIGBcfmxvFm7O1xAU0uqpEIh45IIxGcRGBXhGrQJp5TwrBGb2hJnlmtmKQ6w3zsyqzewCr7KI+CE5PopzR6bzr8XbKSzVWETSennZIpgJnH6wFcwsArgbeNfDHCK+uWJCb0orq3llYbbfUUQa5VkhcM7NBvYcYrUbgVeAXK9yiPhpWHoyo3ul8My8zTinQemkdfKtj8DM0oEpwCN+ZRAJhSuO7c2GXcV8tm6331FEGuRnZ/GfgB8556oPtaKZXWtmWWaWlZeXF4JoIi3nzOHd6ZgQzdNzN/kdRaRBfhaCTOAFM9sEXAA8ZGbnNbSic+4x51ymcy6zc+fOocwo0myxURFMHdeTD77IYW3OPr/jiPwX3wqBc66vc66Pc64P8DJwvXPuX37lEfHSNSf2IyE6kt+/u8bvKCL/xcvTR58H5gKDzSzbzK42sxlmNsOrbYq0VqkJ0Vx7Uj/eW5XDoi35fscR+Rpra2cyZGZmuqysLL9jiBy24vIqvnHvR/TvnMgL1x6LmfkdScKImS10zmU2tExXFouESEJMJDdMGsDnG/cw+8tdfscR2U+FQCSELjmmFxmpcdz77mpNdi+thgqBSAjFREZw6zcHsWLbXt5eoSGqpXVQIRAJsXNHpTO4axK/+/dqSiqq/I4jokIgEmoRAeOX5x5Ndn4pf/rgS7/jiKgQiPjh2H6dmDquJ49/soEV2zRfgfhLhUDEJz8+4yg6JsRw+6vLqKrW3MbiHxUCEZ8kx0dx5zlDWbFtLzPnbPI7joQxFQIRH501vDunDOnCfe+tZeueEr/jSJhSIRDxkZnxq/OGYQb3vadxiMQfKgQiPktPiePicT15a/kOcvaW+R1HwpAKgUgrMO24PlTVOE10L75QIRBpBXp3SuDUo7ry7OdbKKs85FxNIi1KhUCklbjq+D7sKa7gtSXb/I4iYUaFQKSVmNCvE0O6JfHEp5s00b2ElAqBSCthZkw/oS9rcvYxZ70mupfQUSEQaUXOGdmDTgnRPPnZRr+jSBhRIRBpRWKjIrjsmF58uDqXjbuK/Y4jYUKFQKSVuXxCb6IiAjwya73fUSRMqBCItDJdkmKZOq4nryzK1rATEhIqBCKt0HUT+xMw4yG1CiQEVAhEWqHuyXFcNC6DlxduZVtBqd9xpJ1TIRBppa6bOACAh2et8zmJtHdNKgRm1t/MYoL3J5rZTWaW4m00kfCWnhLHhZk9eXFBNjsK1SoQ7zS1RfAKUG1mA4C/AX2B5zxLJSIAXD+xPzXO8bD6CsRDTS0ENc65KmAK8Cfn3PeA7t7FEhGAjNR4LhibwQvzt6pVIJ5paiGoNLNLgCuBN4PPRXkTSUTq++6k2r6C+z/40uck0l41tRBcBUwAfuOc22hmfYFnvIslInV6dozn8mN782LWVtbl7vM7jrRDTSoEzrlVzrmbnHPPm1kqkOSc+53H2UQk6IaTBxAfHck972g6S2l5TT1raJaZdTCzjsBS4Ekz+4O30USkTseEaGZ8ox/vrcph4eY9fseRdqaph4aSnXN7gW8DTzrnxgKnehdLRA40/YS+dE6K4Xf/Xq35CqRFNbUQRJpZd+AivuosFpEQio+O5OZTBrJgUz4ffpHrdxwJsVXb91JQUuHJeze1EPwKeBdY75xbYGb9AJ3CIBJiF4/rSd+0BO55dzXVNWoVhAvnHFMe+syzsaea2ln8knNuhHPuuuDjDc658z1JJCKNiooIcOs3B7E2p4g3l233O46ESF5ROeVVNaSnxHny/k3tLM4ws3+aWa6Z5ZjZK2aWcYjXPBFcf0Ujy881s2VmtsTMsszshCP5ACLh5qzh3RnUNZEHPvxSrYIwsS2/9mLCjFQfCwHwJPA60ANIB94IPncwM4HTD7L8Q2Ckc24UMB14vIlZRMJaIGDcfMog1ucVq1UQJrL3F4J4T96/qYWgs3PuSedcVfA2E+h8sBc452YDjZ7n5pwrcl+d+pAA6E8bkSY6Y1g3BndN4n61CsJCXSFI97lFsMvMLjeziODtcmB3czduZlPMbDXwFrWtgsbWuzZ4+CgrLy+vuZsVafMCAePmUweyIa+YN5aqVdDeZeeXkBofRWJMpCfv39RCMJ3aU0d3AjuAC6gddqJZnHP/dM4NAc4D7jrIeo855zKdc5mdOx+0ISISNk4/uhtDuiXxwIdfUlVd43cc8VB2fqlnrQFo+llDW5xz5zjnOjvnujjnzqP24rIWETyM1N/M0lrqPUXau0DAuOXUgWzYVczrahW0a9sKSslI8aZ/AJo3Q9mtzdmwmQ0wMwveHwNE0wKHm0TCyeSh3RjavQN3v7Oa3H1lfscRDzjnyM4v8eyMIWheIbCDLjR7HpgLDDazbDO72sxmmNmM4CrnAyvMbAnwF+Bip+vmRQ5LIGD8/sKRFJZWcv0zi6io0iGi9mZ3cQVllTWeFoLm9Dwc9EvbOXfJIZbfDdzdjO2LCDC0RwfuuWAkNz2/mF+9uZJfnzfc70jSgrw+dRQOUQjMbB8Nf+Eb4F15EpHDcs7IHqzcXsijH29gWI9kpo7v5XckaSHZ+SUAZHT0qUXgnEvybMsi0qJ+eNoQVm3fyx2vraR/l0TG9enodyRpAXVXFXs1vAQ0r49ARFqRiIDx50tGk54ax+WPf85by3b4HUlaQHZ+KclxUSTFejc7sAqBSDuSEh/NSzMmMDw9me8+t4gHPvxScxe0cV6fMQQqBCLtTlpiDM9ecwzfHp3OH95fy80vLKGsstrvWHKEsvNLVQhE5PDFREZw30Uj+cFpg3l96Xbu/1DTh7RFtdcQlHp6xhCoEIi0W2bGdycNYMrodJ74dCPbCkr9jiSHKb+kktLKak87ikGFQKTdu23yIBxw37tr/I4ih2n/qaM6NCQizZGRGs/04/vy6uJtrNhW6HccOQyhuJgMVAhEwsL1k/qTGh/Fb976QmcRtSF1LQIvRx4FFQKRsNAhNoqbTxnI3A27+WhNrt9xpImy80vpEBtJcpx31xCACoFI2Lj0mN70TUvgt2+vZkehOo7bgm0hOGMIVAhEwkZ0ZICfnnkU6/OKOO53/+GiR+fyzLzN5BdX+B1NGuH1hDR1VAhEwsipQ7vyn9smcsspg9hdVM7P/rWCb9z7EWtz9vkdTQ4QinkI6qgQiISZvmkJ3HzqQD649Ru8ccMJxEZFMH3mAnYVlfsdTeopKKmkuKJah4ZExDtmxvCMZB6/MpNdReVc+3SWhqJoRb46dVQtAhHx2IiMFP5w0SgWbSnghy8v0+mlrcS2gtBcTAYqBCICnDm8+/5xiR78zzq/4wj1WgQeTlpfR4VARAC4fmJ/zh3Vgz99+CUrt+sKZL9l55eSFBNJh7jmzCjcNCoEIgLU9hn88pyjSY2P4sevLqe6RoeI/JSdX0J6ahxm5vm2VAhEZL+U+Gh+cfbRLMsuZOacTX7HCWuhmIegjgqBiHzNt0Z0Z9Lgztz33pr9Y91I6O0oLKN7sgqBiPjAzLjrvGEA/PxfK3QWkQ9KKqooLK2ke0psSLanQiAi/yUjNZ7bJg/mozV5vLFsh99xws72gjIAeqhFICJ+mnZcH0ZmJHPn6yt11XGI1Q0K2D1ZLQIR8VFEwLj3wpEUlVXpEFGI7ahrEXg8RWUdFQIRadSgrkl875uD+PeKnbypQ0Qhs72wFDPo2kEtAhFpBa45sS8je6bw89dWkLuvzO84YWFHQRlpiTFER4bmK1qFQEQOKjIiwH0XjqCkopqf/VOHiEJhe2EpPULUPwAqBCLSBAO6JPH9yYN4b1WOziIKgVBeQwAqBCLSRFef0I/h6cn8+s1VFJVX+R2n3XLOsaOgNGTXEIAKgYg0UUSg9kKzvKJy7v9grd9x2q29ZVUUV1SH7BoCUCEQkcMwqmcKU8f15InPNrFmp6a39ML+awjUIhCR1uoHpw0hKTaSO15Tx7EX6q4haBd9BGb2hJnlmtmKRpZfZmbLgrc5ZjbSqywi0nI6JkTzg9MG8/nGPby+dLvfcdqd7cEWQY920iKYCZx+kOUbgW8450YAdwGPeZhFRFrQ1HG9GJGRzK/f+oLCkkq/47QrOwrKiAgYXZLaQSFwzs0G9hxk+RznXH7w4Twgw6ssItKyIgLGr88bRkFJBdf8PYvyKk1631K2F5bSNSmGiID3E9LUaS19BFcD/25soZlda2ZZZpaVl5cXwlgi0pgRGSn8/sKRzN+4h1tfXEqNZjRrETsKyugeojGG6vheCMxsErWF4EeNreOce8w5l+mcy+zcuXPowonIQZ07Kp0fnzGEt5bt4P/+/YXfcdqFHYWlIRt1tI73syIfhJmNAB4HznDO7fYzi4gcmWtP6sf2glL++slGuifHMf2Evn5HarOcc+woLGPy0d1Cul3fCoGZ9QJeBa5wzunqFJE2ysy44+yj2bm3jLveWkXXDrGcNaK737HapD3FFZRX1YS8ReDl6aPPA3OBwWaWbWZXm9kMM5sRXOUOoBPwkJktMbMsr7KIiLciAsb9U0cztlcq3/vHEuZtUAP/SOwoDP01BOBhi8A5d8khln8H+I5X2xeR0IqNiuDxKzO54JG5XPN0Fi/NmMCQbh38jtWmbC8I/TUE0Ao6i0Wk/UiJj+ap6eOJj45g2hML9n+xSdP41SJQIRCRFpWeEsfMq8ZTXF7F1MfmsSy7wO9Ibcb2wlKiIwJ0SogO6XZVCESkxR3VvQNPXT2eyuoazn94Dn/7dKPGJWqCHQVldEuOJRDCi8lAhUBEPDKmVypv33QiEwd34a43V/Gdp7LIL67wO1ar5sc1BKBCICIeSk2I5rErxnLn2UP55Mtd3PbSUr8jtWrbC8roEeKrisHnC8pEpP0zM6Yd35ei8ip+/95aVm4v5OgeyX7HanWqaxw5e8vUIhCR9uuKCX1IjInk4Vnr/Y7SKu0qKqeqxoV8nCFQIRCREEmOi+KyY3vx9vIdbNpV7HecVmf/NQRqEYhIe3b1CX2JjAjw6Gy1Cg7k1zUEoEIgIiHUJSmWC8dm8MrCbeTsLfM7Tqvi11XFoEIgIiH2vyf1p6qmhsc/2eB3lFYlO7+UuKgIkuOiQr5tFQIRCaleneI5e2QPnv18CwUluq6gzrwNuxnVMwWz0F5MBioEIuKD6yb2p6Simr99utHvKK3C9oJSVu/cx6Qh/ky8pUIgIiE3pFsHzhrenb99upFdReV+x/HdrDW1U/BOGtzFl+2rEIiIL26dPIjyqhr+8tE6v6P47qM1uaSnxDGgS6Iv21chEBFf9O+cyAVjMnh23hay80v8juOb8qpqPlu3i0lDOvvSPwAqBCLio5tPHQgGf/rgS7+j+GbBxnxKKqp9OywEKgQi4qMeKXFccWxvXl2UzZc5+/yO44uP1uQSHRlgQv9OvmVQIRARX10/sT9xURHc995av6P44qM1uRzbrxPx0f6NAapCICK+6pQYwzUn9eOdlTtZtCXf7zghtXl3MRvyipk02J/TRuuoEIiI775zYj+6JMXwy9dXUlMTPjOZ+X3aaB0VAhHxXWJMJLefMYSl2YW8vCjb7zghM2tNLn3TEuiTluBrDhUCEWkVzhuVzpheKdzzzmr2llX6HcdzZZXVzFm/m4k+HxYCFQIRaSUCAePOc45md3EFD4TB6aRz1u+ivKrG98NCoEIgIq3IiIwULs7sycw5m1iX275PJ31xQTap8VEc06+j31FUCESkdfn+aYOJi47gF6+vpLK6xu84nthZWMb7X+RwUWZPYiIj/I6jQiAirUtaYgw/PH0In63bzQUPz2FjO5zW8oUFW6iucVx6TC+/owAqBCLSCl1xbG8eumwMm3aXcNYDn/CPBVtwrn2cVlpVXcML87dy0qDO9O7k79lCdVQIRKRVOnN4d9655URG9UzhR68s57aXlraLYvDBF7ns3FvG5a2kNQAqBCLSinVPjuOZq4/hhkkDeHXRNp6bv8XvSM327Oeb6Z4cy8lD/D9bqI4KgYi0aoGAces3B3HiwDTuenMV63KL/I50xDbuKuaTL3dxyfheREa0nq/f1pNERKQRgYBx34UjiYuK4JZ/LKaiqm2eTfTc55uJDBhTx/X0O8rXqBCISJvQpUMsd58/ghXb9vKH9498pNJZa3J9mSu5pKKKlxZmM/nornTpEBvy7R+Mf+OeiogcpslHd+OS8b14dPZ6hnRL4ozh3Q7rPPwNeUVc98wiSiurGdglkZMGhWZ4h5KKKqbPXMDe0kqmH983JNs8HOZVL7yZPQF8C8h1zg1rYPkQ4ElgDPBT59zvm/K+mZmZLisrq0WzikjbUVJRxXl/+Yy1OUUkREdw4sDOnHxUF7okxeAc1DhHTGQEE/p3IiLw1dSPldU1nP/wHDbvLiE1PooaB+997yRio7y9oKukooqrnlzAgk17+MNFozhvdLqn22uMmS10zmU2tMzLFsFM4EHg6UaW7wFuAs7zMIOItDPx0ZG8fsMJzN2wmw9W5fDBFzm8s3Lnf613woA0HrhkNB0TogH40wdrWZZdyMOXjSE5PopL//o5f/loHbdNHnzEWdbnFbFiWyFnDOtOdOR/H2kvLq/iqpkLyNq0hz9ePIpzR/lTBA7Fs0LgnJttZn0OsjwXyDWzs7zKICLtU2xUBJMGd2HS4C78+rxhrM0poriiioAZAYOlWwu4660vOPvPn/LQZWMoq6zmoVnruTizJ2cM7w7At0en88jH6zl3VA8GdEk67AzVNY7rn1nEmpx93J28musmDeCizPmjFokAAAr0SURBVAxiIiPYXlDKh6tzeWH+Flbv3Mf9U0dz9sgeLb0bWoxnh4YAgoXgzYYODdVb506g6GCHhszsWuBagF69eo3dvHlzywYVkXZnWXYB1z2ziLx95XSIiyQxJpK3bjqRhJjav393FZVzyn0fM6RbEi9ceyxmdoh3/LoXs7byw5eXcd3E/ny+YTeLthTQrUMsHROiWbVjLwC9OsbzkzOHcPqw7i3++Q6XX4eGWoxz7jHgMajtI/A5joi0ASMyUnjjxhO4+YXFfL5hD49fOW5/EYDaMY1uP2MIP351OS8tzOaizKaf0llaUc0f3lvLyJ4p/PC02kNLc9bv5pGP11NeWcPtZwzh1KO60L9z4mEXGD+0iUIgInIkOiZE89RV49lbVklKfPR/Lb84syf/XLyNO19fyYiMZIZ06/C15SUVVby8MJuzhnenU2LM/uef+GwjO/eWcf/UUfu/6I8fkMbxA9K8/UAe0XUEItKuBQLWYBGoW/bgJaNJio3kmqezyC+u2L9sX1klVz4xnzteW8nZf/6UJVsLANhdVM7Ds9Zz6lFdOaZfp5B8Bq95VgjM7HlgLjDYzLLN7Gozm2FmM4LLu5lZNnAr8LPgOh0O9p4iIi2tS4dYHrl8LDmF5dzw/CKqqmsoLKnk8r/NZ/GWAn50+hACAeOiR+by/Pwt/Pk/6yitrOb2M4b4Hb3FeNpZ7AVdRyAiXngpays/eHkZl4zvxbLsAr7MKeLBS0cz+ehu5BdXcPM/ljB7bR4Alx7Ti99OGe5z4sPT5juLRUS8dmFmT1Zu38vMOZuIiQzw2P+MZWJwPuHUhGienDaO+z9Yy9srdnLLKQN9Ttuy1CIQEQmqrK7h/g++5KRBnRnf1/+5hFuSWgQiIk0QFRHg+6cd+ZXGbZXOGhIRCXMqBCIiYU6FQEQkzKkQiIiEORUCEZEwp0IgIhLmVAhERMKcCoGISJhrc1cWm1ke0NDMNMlA4UGeO3B53eOG1kkDdh1BvIYyNHWdxvI19Lih+83NfrBsh1quff/Vc17lP9J9f+BjL352/Nz39e+H474/WL4Dl/d2znVucA3nXLu4AY8d7LkDl9c9bmgdIKulMjR1ncbyHeqzHPg5jjR7U/I3NXu47nsv8x/pvg/Fz46f+z4U+Vvzvm9O/vq39nRo6I1DPHfg8jeasE5LZGjqOo3la+hxQ/ebm70p79HU7Ac+p31/aF7t+wMfe5Hfz33f1O0fTFve9015j0Nuo80dGgoFM8tyjQzO1Nq15eyg/H5qy9mhbef3O3t7ahG0pMf8DtAMbTk7KL+f2nJ2aNv5fc2uFoGISJhTi0BEJMypEIiIhLl2XwjM7AkzyzWzFUfw2rFmttzM1pnZA2Zm9ZbdaGZrzGylmd3Tsqn3b6PFs5vZnWa2zcyWBG9ntnzy/Rk82ffB5d83M2dmaS2X+Gvv78W+v8vMlgX3+3tm1qPlk+/P4EX+e81sdfAz/NPMUlo+uWfZLwz+rtaYmSedss3J3cj7XWlmXwZvV9Z7/qC/G0fkSM9dbSs34CRgDLDiCF47H5gAGPBv4Izg85OAD4CY4OMubSj7ncD32+q+Dy7rCbxL7YWFaW0lO9Ch3jo3AY+0pX0PTAYig/fvBu5uQ9mPAgYDs4DM1pQ7mKnPAc91BDYE/00N3k892Gdszq3dtwicc7OBPfWfM7P+ZvaOmS00s0/MbMiBrzOz7tT+4s51tXv/aeC84OLrgN8558qD28htQ9lDxsP8fwR+CHh2poMX2Z1ze+utmtAG87/nnKsKrjoPyGhD2b9wzq3xIm9zczfiNOB959we51w+8D5wule/2+2+EDTiMeBG59xY4PvAQw2skw5k13ucHXwOYBBwopl9bmYfm9k4T9N+XXOzA9wQbN4/YWap3kVtULPym9k5wDbn3FKvgzag2fvezH5jZluBy4A7PMzakJb42akzndq/RkOlJbOHUlNyNyQd2Frvcd1n8eQzht3k9WaWCBwHvFTv0FpMQ6s28FzdX3CR1DbXjgXGAS+aWb9ghfZMC2V/GLgr+Pgu4D5qf6k919z8ZhYP/JTaQxQh1UL7HufcT4GfmtmPgRuAX7Rw1Aa1VP7ge/0UqAKebcmMjWnJ7KF0sNxmdhVwc/C5AcDbZlYBbHTOTaHxz+LJZwy7QkBtK6jAOTeq/pNmFgEsDD58ndovzPpN3wxge/B+NvBq8It/vpnVUDtoVJ6XwWmB7M65nHqv+yvwppeBD9Dc/P2BvsDS4C9WBrDIzMY753a28uwHeg54ixAVAloof7DT8lvAKV7/4VNPS+/7UGkwN4Bz7kngSQAzmwVMc85tqrdKNjCx3uMMavsSsvHiM3rRadLabkAf6nXgAHOAC4P3DRjZyOsWUPtXf12nzJnB52cAvwreH0RtE87aSPbu9db5HvBCW9r3B6yzCY86iz3a9wPrrXMj8HJb2vfA6cAqoLOXub38ucHDzuIjzU3jncUbqT3ykBq837Epn/GIcnv9H+r3DXge2AFUUltNr6b2r8p3gKXBH+w7GnltJrACWA88yFdXYkcDzwSXLQJObkPZ/w4sB5ZR+1dUdy+ye5X/gHU24d1ZQ17s+1eCzy+jdiCw9La074F11P7RsyR48+SsJ4+yTwm+VzmQA7zbWnLTQCEIPj89uM/XAVcdzu/G4d40xISISJgL17OGREQkSIVARCTMqRCIiIQ5FQIRkTCnQiAiEuZUCKRdMLOiEG/vcTMb2kLvVW21I5KuMLM3DjWqp5mlmNn1LbFtEdAMZdJOmFmRcy6xBd8v0n01wJqn6mc3s6eAtc653xxk/T7Am865YaHIJ+2fWgTSbplZZzN7xcwWBG/HB58fb2ZzzGxx8N/BweenmdlLZvYG8J6ZTTSzWWb2stWOw/9s3djvweczg/eLgoPJLTWzeWbWNfh8/+DjBWb2qya2Wuby1QB7iWb2oZktstrx588NrvM7oH+wFXFvcN0fBLezzMx+2YK7UcKACoG0Z/cDf3TOjQPOBx4PPr8aOMk5N5raEUB/W+81E4ArnXMnBx+PBm4BhgL9gOMb2E4CMM85NxKYDVxTb/v3B7d/yPFggmPnnELtFd8AZcAU59wYaufAuC9YiG4H1jvnRjnnfmBmk4GBwHhgFDDWzE461PZE6oTjoHMSPk4FhtYb+bGDmSUBycBTZjaQ2pEbo+q95n3nXP0x5ec757IBzGwJtWPJfHrAdir4avC+hcA3g/cn8NVY8c8Bv28kZ1y9915I7djzUDuWzG+DX+o11LYUujbw+snB2+Lg40RqC8PsRrYn8jUqBNKeBYAJzrnS+k+a2Z+Bj5xzU4LH22fVW1x8wHuU17tfTcO/M5Xuq862xtY5mFLn3CgzS6a2oHwXeIDaOQs6A2Odc5VmtgmIbeD1Bvyfc+7Rw9yuCKBDQ9K+vUftmP8AmFndcMDJwLbg/Wkebn8etYekAKYeamXnXCG1U1h+38yiqM2ZGywCk4DewVX3AUn1XvouMD04/j1mlm5mXVroM0gYUCGQ9iLezLLr3W6l9ks1M9iBuora4cMB7gH+z8w+AyI8zHQLcKuZzQe6A4WHeoFzbjG1I1VOpXbil0wzy6K2dbA6uM5u4LPg6ab3Oufeo/bQ01wzWw68zNcLhchB6fRREY8EZ1Qrdc45M5sKXOKcO/dQrxMJNfURiHhnLPBg8EyfAkI0JajI4VKLQEQkzKmPQEQkzKkQiIiEORUCEZEwp0IgIhLmVAhERMLc/wOLSBnKDO+IFAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 2e-1\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.749393</td>\n",
       "      <td>0.631815</td>\n",
       "      <td>0.868421</td>\n",
       "      <td>0.869856</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.628098</td>\n",
       "      <td>0.542874</td>\n",
       "      <td>0.916179</td>\n",
       "      <td>0.915374</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.716813</td>\n",
       "      <td>0.927593</td>\n",
       "      <td>0.720273</td>\n",
       "      <td>0.683785</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.643462</td>\n",
       "      <td>0.551566</td>\n",
       "      <td>0.905458</td>\n",
       "      <td>0.907423</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.672351</td>\n",
       "      <td>0.656076</td>\n",
       "      <td>0.863548</td>\n",
       "      <td>0.857651</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.534614</td>\n",
       "      <td>0.461940</td>\n",
       "      <td>0.951267</td>\n",
       "      <td>0.950608</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(2, slice(lr/(2.6**4),lr), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.623553</td>\n",
       "      <td>0.508049</td>\n",
       "      <td>0.924951</td>\n",
       "      <td>0.924046</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.489635</td>\n",
       "      <td>0.447499</td>\n",
       "      <td>0.958090</td>\n",
       "      <td>0.958690</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(2, slice(lr/2/(2.6**4),lr/2), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.467003</td>\n",
       "      <td>0.448250</td>\n",
       "      <td>0.962963</td>\n",
       "      <td>0.963586</td>\n",
       "      <td>00:23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.471006</td>\n",
       "      <td>0.450991</td>\n",
       "      <td>0.963938</td>\n",
       "      <td>0.962333</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.437463</td>\n",
       "      <td>0.431128</td>\n",
       "      <td>0.965887</td>\n",
       "      <td>0.964934</td>\n",
       "      <td>00:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.409270</td>\n",
       "      <td>0.429323</td>\n",
       "      <td>0.969786</td>\n",
       "      <td>0.968492</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(4, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.412864</td>\n",
       "      <td>0.429698</td>\n",
       "      <td>0.964912</td>\n",
       "      <td>0.963988</td>\n",
       "      <td>00:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.402781</td>\n",
       "      <td>0.427383</td>\n",
       "      <td>0.966862</td>\n",
       "      <td>0.965871</td>\n",
       "      <td>00:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.401838</td>\n",
       "      <td>0.428162</td>\n",
       "      <td>0.968811</td>\n",
       "      <td>0.967614</td>\n",
       "      <td>00:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.399231</td>\n",
       "      <td>0.426054</td>\n",
       "      <td>0.968811</td>\n",
       "      <td>0.967874</td>\n",
       "      <td>00:20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2')\n",
    "learn_c.fit_one_cycle(4, slice(lr/100/(2.6**4),lr/100), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.405359</td>\n",
       "      <td>0.430013</td>\n",
       "      <td>0.971735</td>\n",
       "      <td>0.970652</td>\n",
       "      <td>00:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.389910</td>\n",
       "      <td>0.425390</td>\n",
       "      <td>0.970760</td>\n",
       "      <td>0.969534</td>\n",
       "      <td>00:20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2')\n",
    "learn_c.fit_one_cycle(2, slice(lr/1000/(2.6**4),lr/1000), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.26 s, sys: 236 ms, total: 1.5 s\n",
      "Wall time: 2.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_tcu_jurisp_reduzido_sp15_multifit_v2', bs=bs, num_workers=1);\n",
    "\n",
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3\n",
    "\n",
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2', purge=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWUAAAEmCAYAAAC6SYF5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAfK0lEQVR4nO3dd3yV9f3+8dc7CSsERGRJAjJkWxUBBXFrVQTUftWvokVxlK9Wq7hHl6gtHXZY10+sFUWLVm1VlqMqFhFliQOQoYgShgzZQZKT9++P+4QGTEISObk/x1zPxyMPco9z39c5hIs7n3Pf9zF3R0REwpARdwAREfkvlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyhIcM2tgZuPNbKOZPfMttnOBmb2yN7PFxcyONrOFceeQ1DOdpyzVZWbnA9cBXYHNwFzgV+7+1rfc7lDgJ8CR7l70rYMGzswc6OTuS+LOIvHTkbJUi5ldB/wZ+DXQEmgLPACcsRc2fwCwqDYUcmWYWVbcGaQGubu+9FWlL2AfYAtwTgXr1CMq7RXJrz8D9ZLLjgOWA9cDXwIrgYuTy0YCO4DC5D4uBW4Hnii17XaAA1nJ6WHAp0RH60uBC0rNf6vU444EZgIbk38eWWrZFOBOYFpyO68Azcp5biX5byqV/0zgNGARsB64rdT6hwPTgQ3Jde8D6iaX/Sf5XLYmn++5pbZ/M7AKGFsyL/mYjsl9HJacbg2sBY6L+2dDX9/+S0fKUh39gPrAvypY56dAX+BQ4BCiYvpZqeWtiMo9l6h47zezfd39l0RH30+7e467P1JREDNrCPwFGODujYiKd24Z6zUFJibX3Q/4IzDRzPYrtdr5wMVAC6AucEMFu25F9BrkAr8AHgZ+CPQCjgZ+YWYdkusmgGuBZkSv3YnAjwHc/ZjkOockn+/TpbbflOi3huGld+zunxAV9pNmlg08Coxx9ykV5JU0oVKW6tgPWOsVDy9cANzh7l+6+xqiI+ChpZYXJpcXuvskoqPELtXMUwwcZGYN3H2lu88rY52BwGJ3H+vuRe4+DvgYGFxqnUfdfZG7FwD/IPoPpTyFROPnhcBTRIV7j7tvTu5/HnAwgLvPdvd3kvv9DHgIOLYSz+mX7v51Ms8u3P1hYDHwLrA/0X+C8h2gUpbqWAc028NYZ2tgWanpZcl5O7exW6lvA3KqGsTdtxL9yn85sNLMJppZ10rkKcmUW2p6VRXyrHP3RPL7ktJcXWp5QcnjzayzmU0ws1VmtonoN4FmFWwbYI27b9/DOg8DBwH3uvvXe1hX0oRKWapjOrCdaBy1PCuIfvUu0TY5rzq2AtmlpluVXujuL7v794mOGD8mKqs95SnJlF/NTFXxIFGuTu7eGLgNsD08psLToswsh2ic/hHg9uTwjHwHqJSlytx9I9E46v1mdqaZZZtZHTMbYGa/S642DviZmTU3s2bJ9Z+o5i7nAseYWVsz2we4tWSBmbU0s9OTY8tfEw2DJMrYxiSgs5mdb2ZZZnYu0B2YUM1MVdEI2ARsSR7FX7Hb8tVAh288qmL3ALPd/TKisfL/961TShBUylIt7v5HonOUfwasAb4ArgKeT65yFzAL+AD4EJiTnFedfb0KPJ3c1mx2LdIMorM4VhCdkXAsyTfRdtvGOmBQct11RGdODHL3tdXJVEU3EL2JuJnoKP7p3ZbfDjxmZhvM7H/3tDEzOwM4lWjIBqK/h8PM7IK9llhio4tHREQCoiPlSH1gBvA+0bvmI5PznwQWAh8BfwPqJOffSPQr9dzksgTR6UtSipnVN7MZZva+mc0zs5F7fpSUMLNTzWyhmS0xs1vizpMuzKyNmb1hZguSP3fXxJ2pKlJ6pGxmpxKNfWUCf3X336RsZ9+OAQ2JxiPrAG8B1xAV7eTkOn8nOtH/wd0eO5joHNQTaiRpGjEzAxq6+xYz2/m6uvs7MUcLnpllEl2I8n2iC0lmAkPcfX6swdKAme0P7O/uc8ysEdGQ15np8tql7Eg5+UN1PzCA6A2VIWbWPVX7+5acqJAhKuU6yXmTkn860ZF0XhmPHUL0ppbsxiNlva6yZ4cDS9z9U3ffQXQu9N64hP07L3mu+pzk95uBBex66mPQUjl8kW4/VJlEwxFfAq8SnZRfog7RhQ8v7faYbKI3XJ6riYDpyMwyzWzn6+ru7+7pMQJEJfJFqenlpFGxhMLM2gE92fXfc9BSNnxhZmcDpyZP2Sm589cR7n7VbusNp+QyUsvqZfX3TUmeytpnn8b886kxXH39bcyb/zEAo+//A1u3buPam36+y7r/e9YZXDDkbM44e2hZm6pRh3ZrG3eEChUVFbH000/Ia9OWBg0axB1nF3s6YTgOX331FZs2beSAA9oBsG7dOrZt20qbNuH9PYf6q08ikWDxokW0atWKJvvG2ytl+XzZZ6xdu/YbP36pvPtUWT/r3/j7c/fRwGiAjOwWXq/LHs8ISqntwFvzNzLo/GtZMvY1bhs+gJbt8jj3+r+ye7bzL/4Rz7363jfmx+Gt6ffGHWGPfn3XSLKzGzLiuopuKVHzMjLCq+V3pk/nV3fezvhJLwPw+9+OAuDGm2+t6GGxCPEMrsLCQs46czDnX/BDrh5xXdxxytS/b58y56dy+GI50KbUdB7Vv6IrpZrtm8M+OdHRW/16dTjhiC4s/Gw1w37Qj+8f2Y0Lbx3zjR+8xjn1OarXgYyf8kEckdPCmjVr2LBhAwAFBQW88fprdOlS1hXQsrveffqwZMliPlu6lB07dvDM008xcNDpccdKC+7OFcMvo0vXrsEWckVSeaQ8E+hkZu2JLmU9j+gE+uC0ataYh+8YSmZGBhkZxnOvzmHy1I/YPPMePl+5nimPXQ/AC6/PZdToaFj59OMP4bV3Pmbb9h1xRg/aqlUrGX7pMBKJBMXFxZx19jkMGDgo7lhpISsriz/dcx+DB55CIpHgomGX0L1Hj7hjpYXpb0/j70+O5aCDvscRvXsCMPLOX3HqgNNiTlY5qT4l7jSi6/Mzgb+5+68qWj+E4Yt0te7d8IcvQhXi8EU6CXH4Ih3079uHObNn1eiYMslbMk5K5T5ERL5LdEWfiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAsuIOUNqh3doydfq9ccdISy2HPh53hLS15smL4o6Q1sws7ghpqbxXTUfKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEpCsuAOkg0QiwdH9+tC6dS7PPj8+7jhByd0vm9FXHkXLJg0oLoZHX1vEg5MXMOaaY+jUeh8A9smuy8ZtO+h/83jqZGbwl+H96NlhP4rduWnMDN6avzrmZxGmV15+iRuuu4ZEIsGwSy7jxptuiTtSWvi/yy5h8qQJNG/RgtlzP4o7TpWlrJTN7G/AIOBLdz8oVfupCQ/cew9dunZj86ZNcUcJTlHCuW3sLN5fup6c+llMHTWI1z9YwbB7/rNznV8P7c3GbTsAGHZiJwD63vgizRrX55+3nsSxt03APZb4wUokEoy4+komTn6V3Lw8jurbh0GDTqdb9+5xRwve0IuGcfmPr+KySy6MO0q1pHL4Ygxwagq3XyPyly/npcmTuOjiS+OOEqTVGwp4f+l6ALZsL2Jh/kZaN83eZZ0f9G3Hs9OWAtA1rwlTPlwJwNpN29m4dQeHdWhWs6HTwMwZM+jY8UDad+hA3bp1Oefc85gw/oW4Y6WFo44+hqZNm8Ydo9pSVsru/h9gfaq2X1NuuuFa7hr1WzIyNPy+J22bN+Tg9k2ZtWTtznn9u7Xky40FfLJqMwAfLVvPwD5tyMwwDmiew6Ed9iN3v+zyNllrrViRT15em53Tubl55Ofnx5hIakrsTWNmw81slpnNWrt2TdxxdjF54gSaN29Oz8N6xR0leA3rZfHEdcdzy2Mz2VxQuHP+2Ue259m3l+6cfvyNJeSv28Z/Rg3itxf14d1FX5Io1tjF7ryM8RwziyGJ1LTY3+hz99HAaIDDevUO6l/nO9OnMWnieF55eTLbt29n86ZNXDpsKI+MGRt3tKBkZRpPXH8c/3jrU16c8fnO+ZkZxumHt+XoWyfsnJcodm59fObO6X/fMYAlKzVWv7vc3DyWL/9i53R+/nJat24dYyKpKbEfKYds5F2jWPTpF8xftJQxY8dx7HEnqJDLcP/l/VmYv5H7Js7fZf7x39ufRSs2smL9tp3zGtTNJLte1s7lRcXOwvyNNZo3HfTu04clSxbz2dKl7Nixg2eefoqBg06PO5bUgNiPlCW99evSgvOP6chHy9Yz7beDARg5bg6vzM3n7CPb88y0pbus33yf+jx/2/cpdmfF+m386L6pccQOXlZWFn+65z4GDzyFRCLBRcMuoXuPHnHHSgsX/nAIU9+cwtq1a+nYLo+f/2Ikwy5Jnzfqrayxq72yYbNxwHFAM2A18Et3f6SixxzWq7dPnT6zolWkHK2GPh53hLS15smL4o4gtVD/I3oze/asb7xRkLIjZXcfkqpti4h8V2lMWUQkICplEZGAqJRFRAKiUhYRCYhKWUQkICplEZGAqJRFRAKiUhYRCYhKWUQkICplEZGAqJRFRAKiUhYRCYhKWUQkICplEZGAqJRFRAKiUhYRCYhKWUQkICplEZGAqJRFRAKiUhYRCYhKWUQkICplEZGAZJW3wMzGA17ecnc/PSWJRERqsXJLGbi7xlKIiAhQQSm7+5s1GURERCo+UgbAzDoBo4DuQP2S+e7eIYW5RERqpcq80fco8CBQBBwPPA6MTWUoEZHaqjKl3MDdXwPM3Ze5++3ACamNJSJSO+1x+ALYbmYZwGIzuwrIB1qkNpaISO1UmSPlEUA2cDXQCxgKXJTKUCIitdUej5TdfWby2y3AxamNIyJSu1Xm7Is3KOMiEnfXuLKIyF5WmTHlG0p9Xx84i+hMDBER2csqM3wxe7dZ08wsJReWGJCZYanY9Hfe6rEXxh0hbQ24/+24I6S1CVf0iztCWirvHhaVGb5oWmoyg+jNvlZ7I5SIiOyqMsMXs4lK3YiGLZYCl6YylIhIbVWZUu7m7ttLzzCzeinKIyJSq1XmPOWyBtym7+0gIiJS8f2UWwG5QAMz60k0fAHQmOhiEhER2csqGr44BRgG5AF/4L+lvAm4LbWxRERqp4rup/wY8JiZneXuz9VgJhGRWqsyY8q9zKxJyYSZ7Wtmd6Uwk4hIrVWZUh7g7htKJtz9K+C01EUSEam9KlPKmaVPgTOzBoBOiRMRSYHKnKf8BPCamT2anL4YeCx1kUREaq/K3Pvid2b2AXAS0RkYLwEHpDqYiEhtVJnhC4BVQDHRHeJOBBakLJGISC1W0cUjnYHzgCHAOuBpos/pO76GsomI1DoVDV98DEwFBrv7EgAzu7ZGUomI1FIVDV+cRTRs8YaZPWxmJ/Lfq/pERCQFyi1ld/+Xu58LdAWmANcCLc3sQTM7uYbyiYjUKnt8o8/dt7r7k+4+iOg+GHOBW1KeTESkFqrs2RcAuPt6d39IH5oqIpIaVSplERFJLZWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBKSiD06VpFdefokbrruGRCLBsEsu48ab9MErlbF9+3ZOPvFYvv76axJFRZz5P2fxs1+MjDtWMJrn1OXWkzvRtGEd3GHCR6t5bu5KGtXL4hendaZV43qs2vQ1IyctZMvXCQAOyW3MVce2JyvD2FhQyIjn5sX8LMKUSCQ4ul8fWrfO5dnnx8cdp0pSVspm1gZ4HGgFFAOj3f2eVO0vVRKJBCOuvpKJk18lNy+Po/r2YdCg0+nWvXvc0YJXr149Jr38Gjk5ORQWFnLS8Udz8ikDOPyIvnFHC0Ki2Hlw6mcsXrOVBnUyeGjIIcz6fAOndmvBnC82Mm5WPkN653J+7zxGT1tGw7qZjDi+Aze/MJ8vN++gSYM6cT+FYD1w7z106dqNzZs2xR2lylI5fFEEXO/u3YC+wJVmlnZNNnPGDDp2PJD2HTpQt25dzjn3PCaMfyHuWGnBzMjJyQGgsLCQwsJCzPSB6CXWbytk8ZqtABQUFvP5+gKa5dTlyI5NeXn+lwC8PP9L+ndsCsBJXZsz9ZN1fLl5BwAbCgrjCR64/OXLeWnyJC66+NK4o1RLykrZ3Ve6+5zk95uBBUBuqvaXKitW5JOX12bndG5uHvn5+TEmSi+JRIK+fXrSLq8lJ5x4En0OPyLuSEFq2ageB7ZoyIJVW2iaXYf126LCXb+tkH2TR8R5TerTqF4WfzqrBw+ddzAnd20eZ+Rg3XTDtdw16rdkZKTnW2Y1ktrM2gE9gXfLWDbczGaZ2aw1a9fURJwqcfdvzNPRXuVlZmbyzsz3WPTpF8yeNZN58z6KO1Jw6tfJ4I6BXbj/zaVs25Eod73MDKNzixxufWEBNz4/n6FH5JHXpH4NJg3f5IkTaN68OT0P6xV3lGpLeSmbWQ7wHDDC3b8xwOPuo929t7v3bt4svP/5c3PzWL78i53T+fnLad26dYyJ0lOTJk04+phjefXll+KOEpTMDOOOgV3498I1TP1kPRAdHTfNjo6Om2bX4avkMMWaLTuYsWwD24uK2bS9iA/yN9GxWcPYsofonenTmDRxPN07t2fY0CG8OeV1Lh02NO5YVZLSUjazOkSF/KS7/zOV+0qV3n36sGTJYj5bupQdO3bwzNNPMXDQ6XHHSgtr1qxhw4YNABQUFPDG66/RpUvXmFOF5aaTOrJsfQHPvLdy57y3P13PKd1bAHBK9xa8nSzraZ+s5+DcRmQY1MvKoFvLRiz7qiCW3KEaedcoFn36BfMXLWXM2HEce9wJPDJmbNyxqiSVZ18Y8AiwwN3/mKr9pFpWVhZ/uuc+Bg88hUQiwUXDLqF7jx5xx0oLq1atZPilw0gkEhQXF3PW2ecwYOCguGMF46DWjTi5Wws+WbuVh88/BIC/vr2McbPy+eVpnTmtRwu+3Pw1t09cBMDnXxUw47MNPHLBobjDxHmr+WzdtjifgqSAlTVmulc2bHYUMBX4kOiUOIDb3H1SeY/p1au3T3t3VkryfNcVF6fm77E2GPjg9LgjpLUJV/SLO0JaOrpfH+bMnvWNN6hSdqTs7m8BekdMRKQK0vOcERGR7yiVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEpCsuAPI3uFxB0hjL/5f37gjpLWL//5e3BHS0mfrt5U5X0fKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVciW88vJLHNyjCz26Hsjvf/ebuOOkle6d23P4YQfTr09Pju7XJ+44aeW+v/yZww/7Hkf0OpiLLzyf7du3xx0pKFf0b8vD536Pu8/otnPeAU0bcNfALvzu9K6MGtSFjs2yd3lMx/2yeerCnhxxQJOajltpKStlM6tvZjPM7H0zm2dmI1O1r1RKJBKMuPpKXhg/mfc+mM8zT41jwfz5ccdKK5NeeZ3pM99j6vSZcUdJGyvy83nogXt5c9oM3p39AcWJBM8981TcsYIyZcl6fv3qkl3m/bBXLs/OXclNL37MP95byQ975+5cZgYX9M5l7opNNR21SlJ5pPw1cIK7HwIcCpxqZn1TuL+UmDljBh07Hkj7Dh2oW7cu55x7HhPGvxB3LKkFioqKKCgooKioiG0F22i1f+u4IwVlweotbNmR2GWeAw3qZAKQXTeTr7YV7lw2oFtz3l32FZu2F9VkzCpLWSl7ZEtysk7yy1O1v1RZsSKfvLw2O6dzc/PIz8+PMVF6MYwzBp7CUX1787e/jo47TtponZvLT0ZcT4/O7ejUPpfGjffhxJNOjjtW8B6bsZyhvXN54JyDGNo7l7/PXgHAvtl1OLxtE15ZuDbmhHtm7qnrSTPLBGYDBwL3u/vNZawzHBienOwCLExZoOrZF2gMbAXWAk2BhsAXcYZKI3WAfYANQGfgc2BLhY8QgEygI9HrtgboAHwFrI8zVGg6d+5cd8KECZ06d+48D+DRRx9tM2XKlM2PPfbYhqFDh7a9/PLL6/fv33/RpEmTOtx9992rX3/99a3PPvtsuwkTJmwcM2bMVzHHP8Ddm+8+M6WlvHMnZk2AfwE/cfePUr7DvcjM+gG3A/u5e28zuxXA3UfFGiyNmNms5Gt3O7DF3e+OO1PozOwc4FTgkORrdyHQ191/HHO00LQDJgAHJac3Ak0Az8jImFVcXNyZ6KBqKWDJdZoB24gOBp+v0bSVUCNnX7j7BmAK0Q9ZupkJdALqmlld4DzgxXgjpQcza2hmjUq+B04G0uo/5Rh9DvQFMszMgBOBBfFGSgsrgGMBBg8e3AhYnJzfnqjA2wHPAj8mwEKG1J590Tx5hIyZNQBOAj5O1f5Sxd2LgKuIfvVeAPzD3efFmypttATeAroDM4CJ7v5SvJHSg7u/S1Qe3YAPif6talB+V+OA6UTDnsuBS4EfAX8A3r/zzjtz+e/QaNpI2fCFmR0MPEY0NpZBVGZ3pGRnNcDMhru7/lFUg1676tNrV33p+trVyJiyiIhUjq7oExEJiEpZRCQgKmURkYBkxR0gVGbWFTgDyCW6EnEF8KK767QkkQCZ2eFEFxPPNLPuRKfgfuzuk2KOViU6Ui6Dmd0MPEV0svkMonOVDRhnZrfEmU2+28ysq5mdaGY5u81Px3P8a4yZ/RL4C/CgmY0C7gNygFvM7KexhqsinX1RBjNbBPRw98Ld5tcF5rl7p3iSpTczu9jdH407R6jM7GrgSqLz4Q8FrnH3F5LL5rj7YXHmC5mZfUj0mtUDVgF57r4peY3Eu+5+cKwBq0BHymUrBsq6Jdf+yWVSPWl5+9Ya9COgl7ufCRwH/NzMrkkus3IfJQBF7p5w923AJ+6+CcDdC0izf7MaUy7bCOA1M1vMf2881JboxkpXxZYqDZjZB+UtIrrCT8qXWXJnRXf/zMyOA541swNQKe/JDjPLTpZyr5KZZrYPaVbKGr4oh5llAIcTvdFnRJdxznT3RIUPrOXMbDVwCtEdzXZZBLzt7ropcDnM7HXgOnefW2peFvA34AJ3z4wtXODMrJ67f13G/GbA/u7+YQyxqkVHyuVw92LgnbhzpKEJQE7pYilhZlNqPk5auRDY5Q7syXuvXGhmD8UTKT2UVcjJ+WuJbrmbNnSkLCISEL3RJyISEJWyiEhAVMqSFswsYWZzzewjM3vGzLL3/Khyt3WcmU1Ifn96RRcEmVkTM6vyp32Y2e1mdkN1M0rtpVKWdFHg7oe6+0HADuDy0gstUuWfZ3d/0d1/U8EqTYg+pUKkRqiUJR1NBQ40s3ZmtsDMHgDmAG3M7GQzm25mc5JH1DkQXaZsZh+b2VvA/5RsyMyGmdl9ye9bmtm/zOz95NeRwG+Ajsmj9N8n17vRzGaa2QdmNrLUtn5qZgvN7N9En4YhUmUqZUkryfN2BxB9RBJE5fe4u/ck+sTxnwEnJS9JngVcZ2b1gYeBwcDRQKtyNv8X4E13PwQ4DJgH3EJ0hdih7n6jmZ1M9JmNhxNd1tvLzI4xs15En9/Yk6j0++zlpy61hM5TlnTRwMxKzn2eCjxCdCn8MncvOZ+8L9HnAU6LPmuUukSf4dYVWOruiwHM7AnK/uy2E4jOFSZ5kdBGM9t3t3VOTn69l5zOISrpRsC/kleUYWb6cF2pFpWypIsCdz+09Ixk8W4tPQt41d2H7LbeoUS3X90bDBjl7rtczGFmI/biPqQW0/CFfJe8A/Q3swMBzCzbzDoTfYp6ezPrmFxvSDmPfw24IvnYTDNrDGwmOgou8TJwSamx6lwzawH8B/iBmTUws0ZEQyUiVaZSlu8Md18DDCO67/UHRCXd1d23Ew1XTEy+0besnE1cAxyfvA3kbKLbt64jGg75yMx+7+6vAH8HpifXexZo5O5zgKeBucBzREMsIlWmy6xFRAKiI2URkYColEVEAqJSFhEJiEpZRCQgKmURkYColEVEAqJSFhEJyP8HHE4o31mcM8EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds,y,losses = learn_c.get_preds(with_loss=True)\n",
    "predictions = np.argmax(preds, axis = 1)\n",
    "\n",
    "interp = ClassificationInterpretation(learn_c, preds, y, losses)\n",
    "interp.plot_confusion_matrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[327   3   0   2]\n",
      " [  4 279   0   1]\n",
      " [  0   3 206   4]\n",
      " [  0   5   8 184]]\n",
      "accuracy global: 0.9707602339181286\n",
      "accuracy on class 0: 98.49397590361446\n",
      "accuracy on class 1: 98.23943661971832\n",
      "accuracy on class 2: 96.71361502347418\n",
      "accuracy on class 3: 93.4010152284264\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(np.array(y), np.array(predictions))\n",
    "print(cm)\n",
    "\n",
    "## acc\n",
    "print(f'accuracy global: {(cm[0,0]+cm[1,1]+cm[2,2]+cm[3,3])/(cm.sum())}')\n",
    "\n",
    "# acc neg, acc pos\n",
    "print(f'accuracy on class 0: {cm[0,0]/(cm.sum(1)[0])*100}') \n",
    "print(f'accuracy on class 1: {cm[1,1]/(cm.sum(1)[1])*100}')\n",
    "print(f'accuracy on class 2: {cm[2,2]/(cm.sum(1)[2])*100}')\n",
    "print(f'accuracy on class 3: {cm[3,3]/(cm.sum(1)[3])*100}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁a ▁assistência ▁judicial ▁por ▁parte ▁do ▁xxmaj ▁poder ▁xxmaj ▁público ▁aos ▁membros ▁do ▁xxmaj ▁ministério ▁xxmaj ▁público ▁da ▁xxmaj ▁união , ▁em ▁ações ▁propostas ▁por ▁pessoas ▁física s ▁ou ▁jurídica s ▁por ▁eles ▁investiga das , ▁deverá ▁pa u tar - se ▁pelos ▁seguintes ▁critérios : &lt; ▁/ ▁p &gt;&lt; p &gt; a ) ▁nas ▁hipótese s ▁em ▁que ▁as ▁demanda s ▁judiciais ▁mo vidas ▁contra ▁os ▁membros ▁do</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁nos ▁processos ▁li cita tório s ▁sob ▁a ▁modalidade ▁pre gão ▁que ▁se ▁de stin em ▁ao ▁fornecimento ▁de ▁bens ▁e ▁serviços ▁comuns ▁de ▁informática ▁e ▁auto ma ção , ▁verifica do ▁empate ▁entre ▁propostas ▁comerciais , ▁a ▁xxmaj ▁administração ▁xxmaj ▁pública ▁xxmaj ▁federal ▁deverá ▁adotar ▁os ▁seguintes ▁procedimento s : ▁i . ▁analisar , ▁primeiro , ▁se ▁algum ▁dos ▁li cita ntes ▁está ▁oferta ndo ▁bem ▁ou</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁os ▁requisitos ▁a ▁serem ▁pre en chi dos , ▁no ▁momento ▁do ▁ ó bit o ▁do ▁institui dor , ▁para ▁a ▁habilita ção ▁e ▁manutenção ▁da ▁qualidade ▁de ▁filha ▁maior ▁sol t eira , ▁como ▁depende nte ▁de ▁pen são ▁são : ▁a ) ▁ser ▁sol t eira , ▁viúva ▁ou ▁de s qui tada , ▁independente mente ▁da ▁idade ▁( pode ▁ser ▁maior ▁ou ▁menor ▁de</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁é ▁permitida ▁a ▁utilização ▁do ▁chama mento ▁público ▁para ▁per m uta ▁de ▁imóveis ▁da ▁xxmaj ▁união ▁como ▁mecanismo ▁de ▁pro spec ção ▁de ▁mercado , ▁para ▁fim ▁de ▁identificar ▁os ▁imóveis ▁ele g íveis ▁que ▁a ten da m ▁às ▁necessidades ▁da ▁xxmaj ▁união , ▁com ▁atendimento ▁aos ▁princípios ▁da ▁im pessoa lidade , ▁moral idade ▁e ▁publicidade , ▁deve ndo , ▁posteriormente , ▁ser ▁utilizadas ▁várias ▁fontes</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁como ▁alternativa ▁à ▁re ten ção ▁ cau te lar ▁de ▁pagamento s , ▁a ▁xxmaj ▁administração ▁pode ▁pro pi cia r ▁ao ▁contratado ▁a ▁oportunidade ▁de ▁oferecer ▁nova ▁ fia nça ▁bancária ▁ou ▁outra ▁garantia ▁de ▁alta ▁ liquid ez ▁dentre ▁aquela s ▁prevista s ▁no ▁art . ▁ 56 , ▁ § ▁1 o , ▁da ▁xxmaj ▁lei ▁8 . 6 66 ▁/ ▁1993, ▁de ▁a</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.show_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions some random sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(Category 1, tensor(1), tensor([0.0193, 0.7045, 0.0242, 0.2520]))\n"
     ]
    }
   ],
   "source": [
    "# Get the prediction\n",
    "test_text = \"A medida cautelar do TCU que determina a suspensão de licitação por falhas no edital não impede o órgão ou a entidade de rever seu ato convocatório, valendo-se do poder de autotutela (art. 49 da Lei 8.666/1993 c/c o art. 9º da Lei 10.520/2002) , com o objetivo de, antecipando-se a eventual deliberação do Tribunal, promover de modo próprio a anulação da licitação e o refazimento do edital, livre dos vícios apontados.\"\n",
    "pred = learn_c.predict(test_text)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.239\" style=\"background-color: rgba(219, 219, 235, 0.5);\">▁xxbos</span> <span title=\"0.614\" style=\"background-color: rgba(130, 127, 187, 0.5);\">▁a</span> <span title=\"0.832\" style=\"background-color: rgba(91, 53, 149, 0.5);\">▁medida</span> <span title=\"0.505\" style=\"background-color: rgba(156, 152, 199, 0.5);\">▁</span> <span title=\"0.982\" style=\"background-color: rgba(65, 4, 127, 0.5);\">cau</span> <span title=\"1.000\" style=\"background-color: rgba(63, 0, 125, 0.5);\">te</span> <span title=\"0.565\" style=\"background-color: rgba(142, 138, 192, 0.5);\">lar</span> <span title=\"0.126\" style=\"background-color: rgba(238, 236, 244, 0.5);\">▁do</span> <span title=\"0.075\" style=\"background-color: rgba(244, 242, 248, 0.5);\">▁xxup</span> <span title=\"0.190\" style=\"background-color: rgba(228, 227, 239, 0.5);\">▁</span> <span title=\"0.295\" style=\"background-color: rgba(207, 207, 229, 0.5);\">t</span> <span title=\"0.124\" style=\"background-color: rgba(239, 237, 245, 0.5);\">cu</span> <span title=\"0.078\" style=\"background-color: rgba(244, 242, 248, 0.5);\">▁que</span> <span title=\"0.166\" style=\"background-color: rgba(232, 230, 241, 0.5);\">▁determina</span> <span title=\"0.144\" style=\"background-color: rgba(236, 234, 243, 0.5);\">▁a</span> <span title=\"0.227\" style=\"background-color: rgba(221, 221, 236, 0.5);\">▁suspensão</span> <span title=\"0.109\" style=\"background-color: rgba(240, 239, 246, 0.5);\">▁de</span> <span title=\"0.150\" style=\"background-color: rgba(234, 233, 243, 0.5);\">▁li</span> <span title=\"0.084\" style=\"background-color: rgba(243, 241, 247, 0.5);\">cita</span> <span title=\"0.036\" style=\"background-color: rgba(248, 247, 250, 0.5);\">ção</span> <span title=\"0.087\" style=\"background-color: rgba(243, 241, 247, 0.5);\">▁por</span> <span title=\"0.236\" style=\"background-color: rgba(220, 220, 236, 0.5);\">▁falha</span> <span title=\"0.034\" style=\"background-color: rgba(248, 247, 250, 0.5);\">s</span> <span title=\"0.044\" style=\"background-color: rgba(247, 246, 250, 0.5);\">▁no</span> <span title=\"0.090\" style=\"background-color: rgba(243, 241, 247, 0.5);\">▁e</span> <span title=\"0.130\" style=\"background-color: rgba(238, 236, 244, 0.5);\">di</span> <span title=\"0.119\" style=\"background-color: rgba(239, 237, 245, 0.5);\">tal</span> <span title=\"0.067\" style=\"background-color: rgba(245, 243, 248, 0.5);\">▁não</span> <span title=\"0.130\" style=\"background-color: rgba(238, 236, 244, 0.5);\">▁impede</span> <span title=\"0.032\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁o</span> <span title=\"0.079\" style=\"background-color: rgba(243, 242, 247, 0.5);\">▁órgão</span> <span title=\"0.117\" style=\"background-color: rgba(240, 238, 245, 0.5);\">▁ou</span> <span title=\"0.155\" style=\"background-color: rgba(234, 232, 242, 0.5);\">▁a</span> <span title=\"0.290\" style=\"background-color: rgba(208, 208, 230, 0.5);\">▁entidade</span> <span title=\"0.075\" style=\"background-color: rgba(244, 242, 248, 0.5);\">▁de</span> <span title=\"0.162\" style=\"background-color: rgba(232, 231, 242, 0.5);\">▁rever</span> <span title=\"0.195\" style=\"background-color: rgba(227, 226, 239, 0.5);\">▁seu</span> <span title=\"0.451\" style=\"background-color: rgba(169, 167, 207, 0.5);\">▁ato</span> <span title=\"0.658\" style=\"background-color: rgba(122, 113, 179, 0.5);\">▁convoca</span> <span title=\"0.061\" style=\"background-color: rgba(245, 244, 249, 0.5);\">tório</span> <span title=\"0.057\" style=\"background-color: rgba(246, 244, 249, 0.5);\">,</span> <span title=\"0.133\" style=\"background-color: rgba(237, 235, 244, 0.5);\">▁vale</span> <span title=\"0.039\" style=\"background-color: rgba(247, 246, 250, 0.5);\">ndo</span> <span title=\"0.049\" style=\"background-color: rgba(247, 245, 249, 0.5);\">-</span> <span title=\"0.094\" style=\"background-color: rgba(242, 240, 247, 0.5);\">se</span> <span title=\"0.104\" style=\"background-color: rgba(241, 239, 246, 0.5);\">▁do</span> <span title=\"0.232\" style=\"background-color: rgba(221, 220, 236, 0.5);\">▁poder</span> <span title=\"0.505\" style=\"background-color: rgba(156, 152, 199, 0.5);\">▁de</span> <span title=\"0.492\" style=\"background-color: rgba(159, 155, 200, 0.5);\">▁auto</span> <span title=\"0.174\" style=\"background-color: rgba(231, 229, 241, 0.5);\">tu</span> <span title=\"0.153\" style=\"background-color: rgba(234, 232, 242, 0.5);\">te</span> <span title=\"0.080\" style=\"background-color: rgba(243, 242, 247, 0.5);\">la</span> <span title=\"0.024\" style=\"background-color: rgba(249, 248, 251, 0.5);\">▁(</span> <span title=\"0.018\" style=\"background-color: rgba(250, 249, 251, 0.5);\">art</span> <span title=\"0.015\" style=\"background-color: rgba(250, 249, 252, 0.5);\">.</span> <span title=\"0.018\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁</span> <span title=\"0.046\" style=\"background-color: rgba(247, 246, 250, 0.5);\">49</span> <span title=\"0.017\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁da</span> <span title=\"0.017\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁xxmaj</span> <span title=\"0.033\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁lei</span> <span title=\"0.039\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁8</span> <span title=\"0.095\" style=\"background-color: rgba(242, 240, 246, 0.5);\">.</span> <span title=\"0.263\" style=\"background-color: rgba(214, 215, 233, 0.5);\">6</span> <span title=\"0.297\" style=\"background-color: rgba(207, 207, 229, 0.5);\">66</span> <span title=\"0.016\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁/</span> <span title=\"0.053\" style=\"background-color: rgba(246, 245, 249, 0.5);\">▁1993</span> <span title=\"0.060\" style=\"background-color: rgba(245, 244, 249, 0.5);\">▁c</span> <span title=\"0.041\" style=\"background-color: rgba(247, 246, 250, 0.5);\">▁/</span> <span title=\"0.086\" style=\"background-color: rgba(243, 241, 247, 0.5);\">▁c</span> <span title=\"0.015\" style=\"background-color: rgba(250, 249, 252, 0.5);\">▁o</span> <span title=\"0.009\" style=\"background-color: rgba(251, 250, 252, 0.5);\">▁art</span> <span title=\"0.016\" style=\"background-color: rgba(250, 249, 251, 0.5);\">.</span> <span title=\"0.019\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁9</span> <span title=\"0.011\" style=\"background-color: rgba(251, 250, 252, 0.5);\">o</span> <span title=\"0.018\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁da</span> <span title=\"0.030\" style=\"background-color: rgba(249, 247, 251, 0.5);\">▁xxmaj</span> <span title=\"0.025\" style=\"background-color: rgba(249, 248, 251, 0.5);\">▁lei</span> <span title=\"0.037\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁10</span> <span title=\"0.242\" style=\"background-color: rgba(219, 219, 235, 0.5);\">.</span> <span title=\"0.285\" style=\"background-color: rgba(210, 210, 231, 0.5);\">5</span> <span title=\"0.306\" style=\"background-color: rgba(204, 205, 228, 0.5);\">20</span> <span title=\"0.087\" style=\"background-color: rgba(243, 241, 247, 0.5);\">▁/</span> <span title=\"0.130\" style=\"background-color: rgba(238, 236, 244, 0.5);\">▁2002</span> <span title=\"0.025\" style=\"background-color: rgba(249, 248, 251, 0.5);\">)</span> <span title=\"0.022\" style=\"background-color: rgba(249, 248, 251, 0.5);\">▁</span> <span title=\"0.011\" style=\"background-color: rgba(251, 250, 252, 0.5);\">,</span> <span title=\"0.013\" style=\"background-color: rgba(250, 249, 252, 0.5);\">▁com</span> <span title=\"0.042\" style=\"background-color: rgba(247, 246, 250, 0.5);\">▁o</span> <span title=\"0.062\" style=\"background-color: rgba(245, 244, 249, 0.5);\">▁objetivo</span> <span title=\"0.054\" style=\"background-color: rgba(246, 245, 249, 0.5);\">▁de</span> <span title=\"0.043\" style=\"background-color: rgba(247, 246, 250, 0.5);\">,</span> <span title=\"0.106\" style=\"background-color: rgba(240, 239, 246, 0.5);\">▁antecipa</span> <span title=\"0.066\" style=\"background-color: rgba(245, 243, 248, 0.5);\">ndo</span> <span title=\"0.051\" style=\"background-color: rgba(246, 245, 249, 0.5);\">-</span> <span title=\"0.040\" style=\"background-color: rgba(247, 246, 250, 0.5);\">se</span> <span title=\"0.053\" style=\"background-color: rgba(246, 245, 249, 0.5);\">▁a</span> <span title=\"0.081\" style=\"background-color: rgba(243, 242, 247, 0.5);\">▁eventual</span> <span title=\"0.104\" style=\"background-color: rgba(241, 239, 246, 0.5);\">▁de</span> <span title=\"0.248\" style=\"background-color: rgba(218, 218, 235, 0.5);\">libera</span> <span title=\"0.099\" style=\"background-color: rgba(241, 240, 246, 0.5);\">ção</span> <span title=\"0.042\" style=\"background-color: rgba(247, 246, 250, 0.5);\">▁do</span> <span title=\"0.139\" style=\"background-color: rgba(236, 235, 244, 0.5);\">▁xxmaj</span> <span title=\"0.245\" style=\"background-color: rgba(219, 219, 235, 0.5);\">▁tribunal</span> <span title=\"0.068\" style=\"background-color: rgba(245, 243, 248, 0.5);\">,</span> <span title=\"0.131\" style=\"background-color: rgba(238, 236, 244, 0.5);\">▁promover</span> <span title=\"0.038\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁de</span> <span title=\"0.064\" style=\"background-color: rgba(245, 243, 248, 0.5);\">▁modo</span> <span title=\"0.050\" style=\"background-color: rgba(247, 245, 249, 0.5);\">▁próprio</span> <span title=\"0.052\" style=\"background-color: rgba(246, 245, 249, 0.5);\">▁a</span> <span title=\"0.116\" style=\"background-color: rgba(240, 238, 245, 0.5);\">▁an</span> <span title=\"0.162\" style=\"background-color: rgba(232, 231, 242, 0.5);\">ulação</span> <span title=\"0.099\" style=\"background-color: rgba(241, 240, 246, 0.5);\">▁da</span> <span title=\"0.166\" style=\"background-color: rgba(232, 230, 241, 0.5);\">▁li</span> <span title=\"0.033\" style=\"background-color: rgba(248, 247, 250, 0.5);\">cita</span> <span title=\"0.010\" style=\"background-color: rgba(251, 250, 252, 0.5);\">ção</span> <span title=\"0.015\" style=\"background-color: rgba(250, 249, 252, 0.5);\">▁e</span> <span title=\"0.019\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁o</span> <span title=\"0.074\" style=\"background-color: rgba(244, 243, 248, 0.5);\">▁re</span> <span title=\"0.214\" style=\"background-color: rgba(224, 223, 238, 0.5);\">fa</span> <span title=\"0.213\" style=\"background-color: rgba(224, 223, 238, 0.5);\">zi</span> <span title=\"0.121\" style=\"background-color: rgba(239, 237, 245, 0.5);\">mento</span> <span title=\"0.070\" style=\"background-color: rgba(245, 243, 248, 0.5);\">▁do</span> <span title=\"0.078\" style=\"background-color: rgba(244, 242, 248, 0.5);\">▁e</span> <span title=\"0.026\" style=\"background-color: rgba(249, 248, 251, 0.5);\">di</span> <span title=\"0.026\" style=\"background-color: rgba(249, 248, 251, 0.5);\">tal</span> <span title=\"0.039\" style=\"background-color: rgba(247, 246, 250, 0.5);\">,</span> <span title=\"0.080\" style=\"background-color: rgba(243, 242, 247, 0.5);\">▁livre</span> <span title=\"0.076\" style=\"background-color: rgba(244, 242, 248, 0.5);\">▁dos</span> <span title=\"0.117\" style=\"background-color: rgba(240, 238, 245, 0.5);\">▁v</span> <span title=\"0.056\" style=\"background-color: rgba(246, 244, 249, 0.5);\">ício</span> <span title=\"0.068\" style=\"background-color: rgba(245, 243, 248, 0.5);\">s</span> <span title=\"0.140\" style=\"background-color: rgba(236, 235, 244, 0.5);\">▁apontado</span> <span title=\"0.224\" style=\"background-color: rgba(222, 222, 237, 0.5);\">s</span> <span title=\"0.221\" style=\"background-color: rgba(223, 222, 237, 0.5);\">.</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The darker the word-shading in the below example, the more it contributes to the classification. \n",
    "txt_ci = TextClassificationInterpretation.from_learner(learn_c)\n",
    "txt_ci.show_intrinsic_attention(test_text,cmap=plt.cm.Purples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.2395, 0.6145, 0.8318, 0.5050, 0.9821, 1.0000, 0.5648, 0.1259, 0.0754,\n",
       "        0.1899, 0.2945, 0.1241, 0.0776, 0.1660, 0.1436, 0.2270, 0.1089, 0.1496,\n",
       "        0.0836, 0.0363, 0.0872, 0.2356, 0.0342, 0.0443, 0.0896, 0.1303, 0.1193,\n",
       "        0.0666, 0.1301, 0.0322, 0.0787, 0.1167, 0.1554, 0.2903, 0.0753, 0.1617,\n",
       "        0.1949, 0.4512, 0.6584, 0.0610, 0.0571, 0.1333, 0.0393, 0.0493, 0.0936,\n",
       "        0.1043, 0.2322, 0.5054, 0.4923, 0.1743, 0.1532, 0.0799, 0.0238, 0.0178,\n",
       "        0.0152, 0.0184, 0.0459, 0.0170, 0.0167, 0.0335, 0.0389, 0.0949, 0.2626,\n",
       "        0.2967, 0.0161, 0.0526, 0.0600, 0.0407, 0.0855, 0.0151, 0.0091, 0.0165,\n",
       "        0.0194, 0.0110, 0.0180, 0.0304, 0.0254, 0.0369, 0.2418, 0.2850, 0.3064,\n",
       "        0.0873, 0.1301, 0.0248, 0.0222, 0.0110, 0.0131, 0.0415, 0.0622, 0.0540,\n",
       "        0.0430, 0.1060, 0.0663, 0.0510, 0.0401, 0.0531, 0.0810, 0.1038, 0.2476,\n",
       "        0.0991, 0.0421, 0.1390, 0.2452, 0.0683, 0.1305, 0.0378, 0.0639, 0.0499,\n",
       "        0.0519, 0.1165, 0.1615, 0.0988, 0.1658, 0.0335, 0.0099, 0.0145, 0.0193,\n",
       "        0.0740, 0.2143, 0.2131, 0.1213, 0.0701, 0.0779, 0.0260, 0.0257, 0.0394,\n",
       "        0.0801, 0.0763, 0.1167, 0.0558, 0.0679, 0.1402, 0.2245, 0.2207],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt_ci.intrinsic_attention(test_text)[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Text</th>\n",
       "      <th>Prediction</th>\n",
       "      <th>Actual</th>\n",
       "      <th>Loss</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁a ▁adoção , ▁por ▁conselho ▁de ▁fiscalização ▁profissional , ▁da ▁modalidade ▁de ▁li cita ção ▁convite ▁para ▁a ▁contratação ▁de ▁serviços ▁ad voca t ício s ▁que ▁possam ▁ser ▁considerados ▁como ▁objeto ▁comum ▁ inf ring e ▁o ▁disposto ▁no ▁art . ▁4 o ▁do ▁xxmaj ▁decreto ▁5 . 4 50 ▁/ ▁2005, ▁que ▁determina ▁a ▁utilização ▁do ▁pre gão , ▁prefere ncial mente ▁na ▁forma ▁eletrônica .</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4.31</td>\n",
       "      <td>0.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁a ▁concessão ▁de ▁pen são ▁da ▁xxmaj ▁lei ▁3. 37 3 ▁/ ▁1958 ▁a ▁filho ▁maior ▁in vá lido ▁requer ▁ lau do ▁peri cial ▁emitido ▁por ▁junta ▁médica ▁oficial ▁que ▁a tes te ▁a ▁in vali dez ▁e ▁sua ▁pre ex ist ência ▁no ▁momento ▁do ▁ ó bit o ▁do ▁institui dor .</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3.58</td>\n",
       "      <td>0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁falecido ▁o ▁responsável , ▁a ▁obriga ção ▁de ▁re para r ▁o ▁da no ▁re ca i ▁sobre ▁o ▁seu ▁ esp ólio ▁ou , ▁caso ▁concluída ▁a ▁partilha , ▁aos ▁sucesso res ▁até ▁o ▁limite ▁do ▁valor ▁do ▁patrimônio ▁transferido . ▁xxmaj ▁ ante ▁o ▁seu ▁caráter ▁ persona l íssimo , ▁a ▁multa ▁não ▁se ▁trans fer e ▁aos ▁sucesso res .</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3.04</td>\n",
       "      <td>0.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁a ▁xxmaj ▁administração ▁deve ▁a ten tar ▁para ▁os ▁per cent uais ▁aplicado s ▁de ▁xxup ▁b di ▁sobre ▁serviços , ▁materiais ▁e ▁equipamentos , ▁de ▁forma ▁a ▁ corri gir ▁eventual ▁disto r ção ▁com para tivamente ▁aos ▁preços ▁de ▁mercado , ▁avalia ndo ▁quanto ▁a ▁estes ▁dois ▁últimos ▁( mate ria is ▁e ▁equipamentos ) ▁a ▁possibilidade ▁de ▁ela ▁própria ▁rea vali ar ▁as ▁compra s ▁de</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3.03</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁é ▁possível ▁a ▁ respons a bil ização ▁de ▁agentes ▁políticos ▁nas ▁hipótese s ▁de ▁( i ) ▁prática ▁de ▁ato ▁administrativo ▁de ▁gestão ▁ou ▁outro ▁ato , ▁o miss ivo ▁ou ▁com is sivo , ▁que ▁esta bel eça ▁corre lação ▁com ▁as ▁irregular idades ▁a pura das ; ▁( ii ) ▁conduta ▁rei ter ada ▁de ▁da no ▁ao ▁e r ário ▁em ▁decorrência ▁da ▁execução ▁deficiente</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.97</td>\n",
       "      <td>0.90</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# tabulation showing the first k texts in top_losses along with their prediction, actual,loss, and probability of actual class.\n",
    "# max_len is the maximum number of tokens displayed. If max_len=None, it will display all tokens.\n",
    "txt_ci.show_top_losses(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning \"backward Classifier\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')  # \"error\", \"ignore\", \"always\", \"default\", \"module\" or \"on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 100 ms, sys: 20 ms, total: 120 ms\n",
      "Wall time: 501 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = load_data(path, f'{lang}_databunch_lm_tcu_jurisp_reduzido_sp15_multifit_bwd_v2', bs=bs, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.25 s, sys: 420 ms, total: 1.67 s\n",
      "Wall time: 2.46 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = (TextList.from_df(df_trn_val, path, cols=reviews, processor=SPProcessor.load(dest), vocab=data_lm.vocab)\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_from_df(cols=label)\n",
    "    .databunch(bs=bs, num_workers=1, backwards=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 176 ms, sys: 16 ms, total: 192 ms\n",
      "Wall time: 193 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas.save(f'{lang}_textlist_class_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get weights to penalize loss function of the majority class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 420 ms, sys: 4 ms, total: 424 ms\n",
      "Wall time: 398 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_tcu_jurisp_reduzido_sp15_multifit_bwd_v2', bs=bs, num_workers=1, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9237, 1026, 10263)"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_trn = len(data_clas.train_ds.x)\n",
    "num_val = len(data_clas.valid_ds.x)\n",
    "num_trn, num_val, num_trn+num_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([3136, 2439, 2084, 1578]), array([332, 284, 213, 197]))"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_LabelCounts = np.unique(data_clas.train_ds.y.items, return_counts=True)[1]\n",
    "val_LabelCounts = np.unique(data_clas.valid_ds.y.items, return_counts=True)[1]\n",
    "trn_LabelCounts, val_LabelCounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.6604958319800801,\n",
       "  0.7359532315686912,\n",
       "  0.7743856230377828,\n",
       "  0.8291653134134459],\n",
       " [0.6764132553606238,\n",
       "  0.723196881091618,\n",
       "  0.7923976608187134,\n",
       "  0.8079922027290448])"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_weights = [1 - count/num_trn for count in trn_LabelCounts]\n",
    "val_weights = [1 - count/num_val for count in val_LabelCounts]\n",
    "trn_weights, val_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training (Loss = FlattenedLoss of weighted LabelSmoothingCrossEntropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 404 ms, sys: 0 ns, total: 404 ms\n",
      "Wall time: 378 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_tcu_jurisp_reduzido_sp15_multifit_bwd_v2', bs=bs, num_workers=1, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, drop_mult=0.3, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_tcu_jurisp_reduzido_sp15_multifit_bwd_v2');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Change loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_weights = torch.FloatTensor(trn_weights).cuda()\n",
    "learn_c.loss_func = FlattenedLoss(WeightedLabelSmoothingCrossEntropy, weight=loss_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of WeightedLabelSmoothingCrossEntropy()"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_c.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xUVf7/8dcnvZCEhIQWCIEQmnRCVUDEgr3vil1gEde17K5+d9XVLe6qrPqz7arLIqKoWNFdV6UqIlJDD72m0NKAkN7O74+ZYIBJAebOnUk+z8djHpncc2fmPWmfnHvOPVeMMSillFKn8rM7gFJKKe+kBUIppZRLWiCUUkq5pAVCKaWUS1oglFJKuRRgdwB3io2NNYmJiXbHUEopn7FmzZpcY0ycq7YmVSASExNJTU21O4ZSSvkMEUmvq00PMSmllHJJC4RSSimXtEAopZRySQuEUkopl7RAKKWUckkLhFJKKZe0QCillHJJC8Q52pVdyPzNh+yOoZRSbmdZgRCRGSKSLSJpDew3WESqROSmWtuqRGS98/ZfqzKeq01Zx7jxjWVMnrWGbYcK7I6jlFJuZWUPYiYwrr4dRMQfmArMO6WpxBjT33m7xqJ852R95lFunb6CFsEBtAgO4LVFu+yOpJRSbmVZgTDGLAHyG9jtAeAzINuqHFZYm3GEO6avJDosiI/uHcbdIxL5Ou0gOw4ftzuaUkq5jW1jECISD1wPvOmiOUREUkVkhYhc18DzTHbum5qTk2NJ1trS9h/jzrdW0apFEB9OHkaH6DAmXtCZsEB/Xl200/LXV0opT7FzkPpl4HfGmCoXbQnGmBTgVuBlEUmq60mMMdOMMSnGmJS4OJcLErpNSXkVD364joiQAD6cPJz2LUMBiA4P4q4RiXy16SA7tRehlGoi7CwQKcCHIrIPuAl4vaa3YIw54Py4B1gMDLAp40mmzt3GnpwiXri5H22jQk5qmzSyi6MX8a2ORSilmgbbCoQxprMxJtEYkwh8CvzSGPOFiESLSDCAiMQC5wNb7MpZ48dducxcto+7RyRyftfY09pjwoO4c0Qi/9t4gF3Z2otQSvk+K6e5zgaWA91FJEtEJorIFBGZ0sBDewKpIrIB+A54zhhja4E4VlLBI59soEtcOL8b16PO/X4xsguhgf68vFDHIpRSvs+yCwYZY8afwb5317q/DOhjRaaz9ecvN5N9vIzP7htBaJB/nfvFhAcxaWQXXl20kxsHZTOme2sPplRKKffSM6kbsGDLYeas3c/9Y7rSv2PLBve/f0wSya1b8PicTRwvrfBAQqWUsoYWiHocL63gyS/S6NE2ggcu6tqoxwQH+PP3m/pyuKCUZ77eZnHC+hljbH19pZRv0wJRj7/P3c7h46U8d2NfAv0b/6UakBDNpJFdmL0qgx935VqY0LXdOYU8NmcjPZ+ay8SZq8k5XubxDO5SVlnFqr35Pv0elPJVlo1B+LrUffm8tzKdu0ckNurQ0ql+c0k3Fmw5zO8+28i8h0cRHuz+L/VbS/eydGcObaNCaRcVQuuIYBZty2bh1sME+vsxtkdrFm3L5vJXlvD8Tf0Y08M3xkSOFVewcOthFm49zJIdORSVVxEREsATV/Tk54M7IiJ2R1SqWZCmdBgiJSXFpKamnvPzlFVWceWrSykpr2L+r8/+j/vqffn87F/LSYprweDEGHrHR9K7fRR94qPw8zu3P3LvLt/HU//ZTEJMGEVlleQVlQPQMiyQO4Z14s7hicRFBLP90HEe+nAd2w4d587hnXj8ip6EBNY90G6nqmrD+yvTeX7edo6XVtImMpixPdswIqkVs5ans3JvPiOSWvHsDX3o1Crc7rhKNQkissZ5YvLpbVogTvfywh28vHAnb98z+JxnIn2cmsl/1u8nbX8Bx0ocg9Yjk2N54/ZBtDjLwrNgy2HunZXKRT1a8687UvD3E0orqsguKCMuIvi0mValFVW8MG8705fupXd8JG/ePogO0WFn/Z6OFpfzxbr9JLeJcHlOyNlYm3GEJ79IY/OBAs7v2opHL+tBvw5RJ3oL1dWG2aszeO7rbVRUV/P8Tf24ul97t7y2Us2ZFoh6VFZV8/2OHDYfKGDzgWNsPlBA1pESrunXnlfHu+8EbmMMWUdKmLf5EM9+s43ubSKYec9gWkeGNPzgWtZnHuWWacvp3iaC2ZOHERbU+CKzaOthHv5oPQF+wqvjBzAy+cyWJsk6UsxbS/fy0epMissdK6T8LKUDT1zZi6jQwEY/T35ROV9vOkjmkWL2Hykh60gJ6zOP0iYymCev6sWVfdrVeRjp4LESHpy9jjXpR3jh5n7cMLDDSe0bMo+yYMthLu7V5qwODSrV3GiBqEd1taHPn+ZRXFFF51bh9GwfSd/4KG4b1ums/8NvyHfbs7n//bVEhwXxzoTBdG0d0eBjqqoN6zKOcO+sNYQF+zPnvvOJiwg+49fem1vElFlr2Jl9nN9e2p0po5Pwd3G4a1d2IYu3Z5N9vIzDBaUcPFbKmvQjCHBN//bcM6Iz36Qd5M3vd9M6IoRnb+zTqN5WZn4xt7+1kvS8YgL9hfiWocRHhzIwIZp7Ryc16mteXF7JpHdSWb4nj6k39OVngztSWlHFSwt38O8le6h2/kj37RDF7cM6cU2/9l57WE0pu2mBaMDWgwV0jAmzrCC4krb/GHe/vZqKqmqmjE7iugHtaRcVeqLdGENGfjE/7spj6a4clu3O42hxBS3DAvnsvhEkxbU469cuLq/kd59t4ssNB+jRNoLHr+jJqG6O3sSx4gpeWriDWSvSqao2BAX40TYyhDaRwQxIiObuEYknFikEx3/sj3yygZ3ZhVzSqw0PjU2md3yUy9fddqiAO99aRVllNf+6YxBDEmPOeiymtKKKybPWsGRHDvddmMS8tEPsyS1i/JCOPHBRMou2Hubd5enszC4kMiSAGwZ24NahCXRr03AxVqo50QLhpTLzi/ntJxtYtTcfETg/KZYLu8ex/dBxlu3OY//REgDaRoZwQXIsI5NjGZUcR3R40Dm/tjGGrzcd4rm5W8nML2F0tzjO79qKNxbv5lhJBeOHJPDARcm0iQxucNZQWWUV077fw7Qf9nC8tJJLe7XhgYuS6dkuggDn9OA16fnc8/ZqQoP8mTVxqFv+UJdWVHH/+2tZtC2b+JahTL2xLxck/zQmYoxhxZ58PliVwdy0g1RUGVI6RTOwUzQHjjoObR04WkJMeBCjusUxMjmWwYkx2ttQzYoWCC+3L7eIz9ftZ866LDLzS2gZFsjwLq0YkdSK4UmtSIprYdnUzrLKKt5dls5r3+6koLSSoZ1j+OPV59GrfeQZP9exkgpm/riP6UsdhQIgPMifyNBA8orKiW8ZyrsThtAx5uwHyE9VXlnNwq2HGd0trt7ZZnmFZXy2NosPV2WSdaSE9i1DiI8OpX1UKAeOlbB67xHKq6oJDvDjwbHJ/PLCJJ1Oq5oFLRA+whjDgWOltIsMOedpsGfqSFE5mUeK6RMfdc5/GI+VVPDVxoPkHC+joLSCYyUVBPr78dtLuxHb4szHTdzNGHPaeywur2TV3nxmr8pg3ubD/PLCJB69rLsWCdXkaYFQqpGqqw1PfJHG7FUZTLqgM09c2VOLhGrS6isQeia1UrX4+QnPXN+b4AA/pi/dS1llNX++5jyP9+iU8gZaIJQ6hYjwx6t7ERzgx7+W7CHAX3jqql7ak1DNjhYIpVwQEX5/eQ/Kq6p5+8d9tI0M4d7RdV4aXakmSQuEUnUQEZ68shc5x8t49pttxEUEn3bmtlJNmRYIperh5ye8+LN+5BeV83+fbqRVi2BGdzuzJUqU8lV6PQilGhAc4M+/7hhEcpsIpsxaw7PfbGVXdqHdsZSynKXTXEVkBnAVkG2M6V3PfoOBFcDPjTGfOrfdBfzBuctfjTHvNPR6Os1VWSn7eCl/+DyNRduyqao2DExoydiebcgrLCcjv5jM/GJaRwbzz9sGEhnS+MULlbKTbedBiMgooBB4t64CISL+wAKgFJhhjPlURGKAVCAFMMAaYJAx5kh9r6cFQnlC9vFSvli3n49Ts9iVXUhooD8JMWHER4eyZEcO53eN5a27Uk4sM6KUN7PtPAhjzBIRSWxgtweAz4DBtbZdBiwwxuQDiMgCYBww24KYSp2R1hEhTB6VxC9GdqGgpJLI0IATU2DfX5nOE5+n8czX23jq6l42J1Xq3Ng6SC0i8cD1wEWcXCDigcxan2c5t7l6jsnAZICEhARrgirlgogQFXbyoaTbhnZiV3YhM37cS3KbFowfoj+TynfZ3Qd+GfidMabqlO2uzkhyeSzMGDPNGJNijEmJi9PZJcp+T1zRk9Hd4njyizSW7c61O45SZ83uApECfCgi+4CbgNdF5DocPYaOtfbrABzwfDylzlyAvx+v3TqAxNhw7p6x2nERo+qms+aZaj5sLRDGmM7GmERjTCLwKfBLY8wXwDzgUhGJFpFo4FLnNqV8QmRIIB9NHsaF3eP429dbuXX6ihPX91DKV1haIERkNrAc6C4iWSIyUUSmiMiU+h7nHJx+GljtvP2lZsBaKV/RqkUw/7pjEH+/sS+bso4x7qUlfLct2+5YSjWaLvetlAdk5BUzeVYquYVlLH50jEcvb6tUfeqb5mr3GIRSzUJCqzCm3tiX3MJy3ly82+44SjWKFgilPKRfx5Zc2789//5hDwd0PEL5AC0QSnnQo5d1xwAvzNtudxSlGqQFQikP6hAdxsQLOjNn3X42ZR2zO45S9dICoZSH/fLCJFqFB/HXr7bQlCaJqKZHC4RSHhYREsjDl3Rj5d58Fm3Vaa/Ke2mBUMoG4wd3JL5lKLNWpNsdRak6aYFQygYB/n5cPyCeH3bmkH281O44SrmkBUIpm1w/MJ5qA/9dr8uMKe+kBUIpmyTFtaBfhyjmrN1vdxSlXNICoZSNrh8Qz5aDBWw/dNzuKEqdRguEUja6ul97AvyEOeuy7I6i1Gm0QChlo1YtghndLY7/rDtAlV4zQnkZLRBK2ez6gfEcKihlxZ48u6ModRItEErZ7OKebYgIDuCztXqYSXkXLRBK2Swk0J8r+rRjbtohissr7Y6j1AlaIJTyAjcMjKe4vIoPV2XaHUWpE7RAKOUFhnSOYVS3OJ6ft52MvGK74ygFaIFQyiuICM/d0IcAP+HRTzdQrTOalBfQAqGUl2jfMpQnr+rFyr35vLt8n91xlLKuQIjIDBHJFpG0OtqvFZGNIrJeRFJF5IJabVXO7etF5L9WZVTK29yc0oELu8cxde520vOK7I6jmjkrexAzgXH1tC8C+hlj+gMTgOm12kqMMf2dt2sszKiUVxERnr2hDwH+wqOfbNRDTcpWlhUIY8wSIL+e9kLz0+W0wgH9TVAKaBcVypNX9mLVvny+2nTQ7jiqGbN1DEJErheRbcBXOHoRNUKch51WiMh1DTzHZOe+qTk5OZbmVcpTbhrUgS5x4by+eLdellTZxtYCYYz53BjTA7gOeLpWU4IxJgW4FXhZRJLqeY5pxpgUY0xKXFycxYmV8gw/P2HK6CS2Hixg8Q79x0fZwytmMTkPRyWJSKzz8wPOj3uAxcAA+9IpZY/r+sfTLiqEN77bbXcU1UzZViBEpKuIiPP+QCAIyBORaBEJdm6PBc4HttiVUym7BAX4MXlUF1bty2f1vjqH85SyjJXTXGcDy4HuIpIlIhNFZIqITHHuciOQJiLrgX8CP3cOWvcEUkVkA/Ad8JwxRguEapZuGZxATHgQr3+3y+4oqhkKsOqJjTHjG2ifCkx1sX0Z0MeqXEr5ktAgf+4ZkciLC3aw5UABvdpH2h1JNSNeMQahlKrbncMTCQ/y543vdSxCeZYWCKW8XFRYILcP68RXGw+wK1uvXa08RwuEUj5g8qguhAb68+L8HXZHUc2IFgilfECrFsFMGtmFb9IOsSHzqN1xVDOhBUIpHzFpZGeiwwJ5ft52u6OoZkILhFI+IiIkkPvHdGXprlyW7cq1O45qBrRAKOVDbh/WifZRIUydt13XaFKW0wKhlA8JCfTnoYuT2ZB5lPlbDtsdRzVxWiCU8jE3DnSs9Pr3udsoq6yyO45qwrRAKOVjAvz9eOqqXuzOKeLVRTvtjqOaMC0QSvmgC7u35uZBHXjz+z1szNJpr8oaWiCU8lF/uKoXcS2CeeSTDXqoSVlCC4RSPioqNJBnb+zDjsOFvLJQDzUp99MCoZQPG3PiUNNuPcNauZ0WCKV83B+u6kXriBAe/3yTnhuh3EoLhFI+Lio0kEcu687mAwUs0HMjlBtpgVCqCbiuf3s6tQrjlUU7tReh3EYLhFJNQIC/Hw9clMzmAwUs3JptdxzVRGiBUKqJqOlFvLxwh/YilFtYWiBEZIaIZItIWh3t14rIRhFZLyKpInJBrba7RGSn83aXlTmVagoC/P341ZiubD5QwCLtRSg3sLoHMRMYV0/7IqCfMaY/MAGYDiAiMcAfgaHAEOCPIhJtbVSlfN/1A+IdvYhF2otQ587SAmGMWQLk19NeaH76KQ4Hau5fBiwwxuQbY44AC6i/0Cil+KkXkbZfexHq3Nk+BiEi14vINuArHL0IgHggs9ZuWc5trh4/2Xl4KjUnJ8fasEr5gOsHxJMQE8bri3fZHUX5ONsLhDHmc2NMD+A64GnnZnG1ax2Pn2aMSTHGpMTFxVkVUymfEeDvx8QLOrM24yhr0o/YHUf5MNsLRA3n4agkEYnF0WPoWKu5A3DAlmBK+aCbBnUgMiSAt5busTuK8mGNKhAikiQiwc77F4rIgyLS8lxfXES6iog47w8EgoA8YB5wqYhEOwenL3VuU0o1QnhwALcO7cTctENk5hfbHUf5qMb2ID4DqkSkK/AW0Bn4oKEHichsYDnQXUSyRGSiiEwRkSnOXW4E0kRkPfBP4OfGIR/H4abVzttfnNuUUo1094hE/ER4+8d9dkdRPiqgkftVG2MqReR64GVjzGsisq6hBxljxjfQPhWYWkfbDGBGI/MppU7RNiqEq/u156PVGTx0cTJRoYF2R1I+prE9iAoRGQ/cBfzPuU1/2pTychMv6ExReRUfrc6wO4ryQY0tEPcAw4G/GWP2ikhn4D3rYiml3KF3fBTDu7Ti7R/3UVFVbXcc5WMaVSCMMVuMMQ8aY2Y7B40jjDHPWZxNKeUGk0Z25uCxUj5ft9/uKMrHNHYW02IRiXQugbEBeFtE/p+10ZRS7jCme2sGJLTk6S+3kJ5XZHcc5UMae4gpyhhTANwAvG2MGQRcbF0spZS7+PkJr94yABH41QfrKKussjuS8hGNLRABItIO+Bk/DVIrpXxEx5gwnr+5H5v2H+O5b7bZHUf5iMYWiL/gOFFttzFmtYh0AXZaF0sp5W6XndeWu0ck8vaP+5i3+ZDdcZQPaOwg9SfGmL7GmPucn+8xxtxobTSllLs9dkUP+sRH8egnG9iTU2h3HOXlGjtI3UFEPnde/OewiHwmIh2sDqeUcq/gAH/+cesAAv39uOnN5WzIPGp3JOXFGnuI6W3gv0B7HMtuf+ncppTyMZ1ahfPpfSMID/bnlmkrWLxdrxuhXGtsgYgzxrxtjKl03mYCura2Uj6qc2w4n903gs6x4Ux6J5U5a7PsjqS8UGMLRK6I3C4i/s7b7ThWXVVK+ajWESF8dO8whnSO4Tcfb2Bdhl47Qp2ssQViAo4proeAg8BNOJbfUEr5sIiQQP59ZwrRYYH841u9Ap06WWNnMWUYY64xxsQZY1obY67DcdKcUsrHhQcHMOH8zizalk3a/mN2x1Fe5FyuKPcbt6VQStnqzhGJRAQH6HWs1UnOpUC4um60UsoHRYUGcteIRL5JO8TOw8ftjqPOwIerMnhsziZLnvtcCoRxWwqllO0mXNCZkAB/Xl+82+4o6gys3JvPDztzLHnueguEiBwXkQIXt+M4zolQSjURMeFB3D4sgf+s36+rvvqQ3MIyWrUItuS56y0QxpgIY0yki1uEMaaxlytVSvmIX4zsQoC/H29oL8Jn5BWWExseZMlzn8shJqVUE9M6MoRbBnfkkzVZzNcF/XxCXlEZrVr4WIEQkRnOtZvS6mi/TUQ2Om/LRKRfrbZ9IrJJRNaLSKpVGZVSp3v0su70iY/i/g/WsnDLYbvjqHoYY8grLLfnENM5mgmMq6d9LzDaGNMXeBqYdkr7GGNMf2NMikX5lFIuRIQE8s6EIfRsF8l976/h221aJLxVQUklldWGVr52iMkYswTIr6d9mTGm5tz+FYCuDquUl4gKDWTWhKF0bxvBlFlr+U4X9PNKuUVlAMT6YA/iTEwEvqn1uQHmi8gaEZlc3wNFZLKIpIpIak6ONVO9lGqOosICeW/iUJLbtGDKrDWs2lvn/3vKJrnHm3iBEJExOArE72ptPt8YMxC4HLhfREbV9XhjzDRjTIoxJiUuTheYVcqdWoYF8e6EIcS3DGXiO6vZdqjA7kiqlryicgDfG6RuDBHpC0wHrjXGnFgd1hhzwPkxG/gcGGJPQqVUqxbBvDtxCGFB/tz51ioy84vtjqSc8godPYgmVyBEJAGYA9xhjNlRa3u4iETU3AcuBVzOhFJKeUaH6DDenTCU0ooq7pyxilznHyZlr9xCRw8iJszHCoSIzAaWA91FJEtEJorIFBGZ4tzlKaAV8Pop01nbAEtFZAOwCvjKGDPXqpxKqcbp3jaCt+4ezIGjJUx8J5XSiiq7IzV7eUVlRIcFEuBvzZ9yy86GNsaMb6B9EjDJxfY9QL/TH6GUstvgxBheuaU/U95by+NzNvHiz/ohout22sXKcyDACwaplVK+ZVzvdvz64m7MWbef6T/stTtOs5ZbWEasReMPoAVCKXUWHrioK1f0acuz32xlsZ4jYRvtQSilvI6fn/DCzf3o3jaSB2avY3dOod2RmqXcwjLLFuoDLRBKqbMUFhTAv+8cRJC/H1NmraGorNLuSM1KeWU1BaWV2oNQSnmnDtFhvDZ+ALtzCnlsziaM0euIeUq+xSfJgRYIpdQ5GtE1lt9e2p3/bjjAu8vT7Y7TbNSci9IqXHsQSikvdt/oJC7u2Zq/frWFNelHGn6AOmc1y2zoLCallFfz8xNevLk/baNCuP/9tSeWgFDWsXqhPtACoZRyk6iwQN64bRB5RWW8uGBHww9Q5ySvyNp1mEALhFLKjXrHR3FzSkc+Tc3i0LFSu+M0aXmF5QQF+NEi2LIFMbRAKKXc677RSVQZw79/2GN3lCYtt7Cc2PAgS5c60QKhlHKrjjFhXNc/nvdXputYhIXyisosPQcCtEAopSzwyzFJlFVW89ZSXavJKo5lNqwbfwAtEEopCyTFteDKPu14d3k6x4or7I7TJOUVllk6gwm0QCilLHL/mK4UllUyc9k+u6M0OcYYcrUHoZTyVT3bRXJJrzbM+HEv+4+W2B2nSTleVkl5VTWxFp5FDVoglFIWemhsMiXlVYz6+3fcOyuVH3fl6npNbpBXaP06TKAFQillod7xUSz67Wh+MbILq/bmc9v0lVz+yg86u+kc1Xz9dBaTUsqndYwJ4/eX92D5Y2N5/qa+7Mwu5P/pmdbnJLemB2HhtSDAwgIhIjNEJFtE0upov01ENjpvy0SkX622cSKyXUR2icjvrcqolPKckEB/bk7pyB3DOjF7VQZbDxbYHcln1Syz4cuzmGYC4+pp3wuMNsb0BZ4GpgGIiD/wT+ByoBcwXkR6WZhTKeVBD1+cTGRoIE//b4uOR5ylmjGIGF/tQRhjlgD59bQvM8bUrAu8AujgvD8E2GWM2WOMKQc+BK61KqdSyrNahgXxm0u6sWx3HvO3HLY7jk/KLSwjKjSQoABrRwm8ZQxiIvCN8348kFmrLcu5zSURmSwiqSKSmpOTY2FEpZS73DokgeTWLXjm662UVVbZHcfneOIsavCCAiEiY3AUiN/VbHKxW539UGPMNGNMijEmJS4uzoqISik3C/D348mrepGeV8zbP+6zO47PyS0ss/wcCLC5QIhIX2A6cK0xJs+5OQvoWGu3DsABT2dTSllrVLc4xvZozT++3UXOcZ32eibyipp4D0JEEoA5wB3GmNpz3lYDySLSWUSCgFuA/9qRUSllrcev7ElpRRUvLdRpr2cir7DMtwuEiMwGlgPdRSRLRCaKyBQRmeLc5SmgFfC6iKwXkVQAY0wl8CtgHrAV+NgYs9mqnEop+yTFteD2YZ34cFUGOw4ftzuOT6isquZIcYXlU1wBLLsUkTFmfAPtk4BJdbR9DXxtRS6llHd5aGwyc9Zm8bevtvLOhCF2x/F6+UU1y2w08TEIpZSKDg/iwbHJfL8jh8Xbs+2O4/VqzqKOtfgcCNACoZTyAncM70SnVmE88/VWKquq7Y7j1WrOotYehFKqWQgO8Of343qw43AhH6VmNvyAZsxTK7mCFgillJcY17stgxOjeXXRTiq0F1GnminBTf48CKWUqiEi3DsqicMFZSzUJTjqlHmkmIjgACJDLZtjdIIWCKWU1xjTozXxLUN5b2W63VG8VkZ+MQmtwhBxteiEe2mBUEp5DX8/YfyQjvy4K489OYV2x/FKGXnFJMSEeeS1tEAopbzKzwZ3JMBPeH9lht1RvE5VtSHrSAkJrbRAKKWaodYRIVzWuy2frsmitEJXeq3tUEEp5VXV2oNQSjVftw/txLGSCr7coOt01paRVwxAp5hwj7yeFgillNcZ1iWGrq1b8J4eZjpJRn4RAJ30EJNSqrkSEW4bmsCGzKOk7T9mdxyvkZFfTICf0C4qxCOvpwVCKeWVbhjYgdBAf95boVNea6TnFRMfHUqAv2f+dGuBUEp5pajQQK7p157/rD9AQWmF3XG8Qma+56a4ghYIpZQXu31YJ0oqqvhi3X67o3iFdC0QSinl0KdDFH07RPHeinSMqfPS9M3CsZIKjhZXaIFQSqkatw1NYMfhQlLTj9gdxVaZ+c4prh6awQRaIJRSXu7qfu2JCAlo9oPV6c5zIDpqD0IppRzCggK4cWAHvtl0iLzCMrvj2CbD2YNoEoeYRGSGiGSLSFod7T1EZLmIlInII6e07RORTSKyXkRSrcqolPINtw1NoLyqmk/WZNkdxTYZ+UXEhAcREV3FrHUAAA/8SURBVBLosde0sgcxExhXT3s+8CDwQh3tY4wx/Y0xKe4OppTyLcltIhjSOYYPVmZQXd08B6szPDyDCSwsEMaYJTiKQF3t2caY1YBOcFZKNej2YZ3IyC/mh125dkexRXpesUcHqMF7xyAMMF9E1ojI5Pp2FJHJIpIqIqk5OTkeiqeU8rTLzmtDXEQwryzc0ex6ERVV1Rw4WtJ0ehDn6HxjzEDgcuB+ERlV147GmGnGmBRjTEpcXJznEiqlPCo4wJ/fjevB2oyjzGlmJ87tP1JCtfHsADV4aYEwxhxwfswGPgeG2JtIKeUNbhgQz8CEljz3zdZmtfyGHTOYwAsLhIiEi0hEzX3gUsDlTCilVPPi5yf85dre5BWV8/KCnXbH8Zj0EyfJeeY6EDUCrHpiEZkNXAjEikgW8EcgEMAY86aItAVSgUigWkQeBnoBscDnzgtyBwAfGGPmWpVTKeVbesdHceuQBN5Zvo+fD+5I97YRdkeyXGZ+MUEBfrSOCPbo61pWIIwx4xtoPwR0cNFUAPSzJJRSqkl45NLufLXpIE/9J40PJw/D+Q9lk5WeV0RCTBh+fp59n153iEkppRoSHR7Eo5d1Z+XefB7/PK3Jn2Gdke/5GUygBUIp5aNuGZzA3SMS+Tg1kwufX8wbi3dTWlFldyy3M8aQ4exBeJoWCKWUT/L3E/50zXnMe3gkQzrHMHXuNsa++P2JVU+biryicorKqzx+khxogVBK+biurSN46+7BfDBpKEeKy3n2m612RzpjhwtKOVpcftr2YyUVJy6WZEcPwrJBaqWU8qQRXWP5xcguvLJoJ2vSjzCoU7TdkRolu6CUC59fTElFFR1jQukb35IuceGszTjCyj35VFYbWkcE0yc+yuPZtAehlGoyJo/qQlxEMM98vdVnrkA3feleyiqreGhsMn3jW7Jx/1Fe+3YX2QVlTBrZhc/uG87yx8bSOjLE49m0B6GUajLCgwP49cXdePzzTczbfIhxvdvZHaleR4rKeW9FOlf3a8+vL+l2YntpRRUhgf42JnPQHoRSqkn5WUoHurZuwdS526moqrY7Tr1mLttHcXkVv7yw60nbvaE4gBYIpVQTE+Dvx2OX92BvbhEfrMw46+fZcfg4hWWVjd7/TA9pFZZVMnPZPi7p1cZrzwbXAqGUanIu6tGaYV1ieGnhDuaszaK88sx6EjsPH+fyV35gwszVVDViafG5aYcY9fx37M0tavRrvL8inWMlFdw/pmvDO9tEC4RSqskREf56XW9iWwTzm483cMHUb/nHtzvJLzp9KumpjDH8+cst+Iuwam8+b36/u8H9X/t2J5n5JTz84bpGHdYqraji3z/s5YKusfTv2LLR78vTdJBaKdUkdW0dwYJfj2LJzlzeWrqXF+bv4KWFOxmUEM3o7nGM7hbHee0jT1vHaf6Wwyzdlcufru5FavoRXlqwg/Pr+UO+NuMomw8UcHHPNizcepiXFuzg/8b1OGmfwrJKMvKK8fMDfxHmbzlMbmEZ948ZYNn7dwfxlalgjZGSkmJSU1PtjqGU8kI7Dx/nP+sPsHhHNmn7CwA4r30k0+5MIb5lKOD4z/6Sl74nNNCfrx8cSVF5FVe88gMB/sJXD46kRfDp/1M/9OE6vt2azYrHx/KXL7fw8ZpMPpg0jOFJrQCYm3aQP3yRRm7hyb2XQZ2i+XTKcNsXGhSRNcaYFJdtWiCUUs1N9vFSFm3N5pmvthIS5M/0O1Po17El//xuF8/P2877k4ZyftdYAFbtzeeWacu5YWAHXrj55IWmc46XMeK5Rdw2tBN/uuY8isoqueq1pZRWVPHBL4bx4vzt/G/jQc5rH8m9o5MI9BOqjKHawJDEGNpGef7chlPVVyD0EJNSqtlpHRHC+CEJpHSK5p6Zq/n5tOU8cUVP/vndLi7t1eZEcQAY0jmGX43pyqvf7mJwYjQ/H5xwou2j1RlUVBnuGN4JcJyH8cot/bnh9WVc9OJiAvyE317SjSkXJhHo73tDvr6XWCml3CS5TQRf3H8+vdpF8uR/NlNZbfjDlb1O2+/BscmMTI7lsTmb+GbTQQAqq6p5f2UGF3SNJSmuxYl9+3ZoyZ+vPY8LusbyvwdG8sDYZJ8sDqCHmJRSitKKKqbO3Ua3NhGMH5Lgcp/i8krueGsVG7OOMv2uwZSUVzLlvbVMu2MQl57X1sOJ3UfHIJRSyg2OlVRwy7QV7M0tpEN0GCXlVSz5vzH4e/hKb+5UX4HwzX6PUkrZICo0kHcnDKFdVCi7sgu5dWiCTxeHhuggtVJKnYG4iGDemzSUd5fvOzE43VRZ1oMQkRkiki0iaXW09xCR5SJSJiKPnNI2TkS2i8guEfm9VRmVUupsxLcM5bHLexIZEmh3FEtZeYhpJjCunvZ84EHghdobRcQf+CdwOdALGC8ip08rUEopZSnLCoQxZgmOIlBXe7YxZjVQcUrTEGCXMWaPMaYc+BC41qqcSimlXPPGQep4ILPW51nObS6JyGQRSRWR1JycHMvDKaVUc+GNBcLVlIA65+IaY6YZY1KMMSlxcXEWxlJKqebFGwtEFtCx1ucdgAM2ZVFKqWbLGwvEaiBZRDqLSBBwC/BfmzMppVSzY9l5ECIyG7gQiBWRLOCPQCCAMeZNEWkLpAKRQLWIPAz0MsYUiMivgHmAPzDDGLPZqpxKKaVcs6xAGGPGN9B+CMfhI1dtXwNfW5FLKaVU4zSptZhEJAdIP2VzFHCsgW21P3d1v+ZjLJB7lvFc5WhMuzvzw9m/h4by17dPfXlP/byh+5r/zPdp6Georvfjzvz15WuoXX+Hrc3fyRjjeoaPMaZJ34BpDW2r/bmr+7U+prozR2Pa3Zn/XN5DQ/nP5D2caX53fA80f93b6no/7szfmPfgid8BzX9mj/HGQWp3+7IR275s4L6r53BHjsa0+0r++vapL++pnzfm/tnQ/HVvq+v9uDN/Y57D138HfD3/aZrUISariUiqqWNZXF/h6+9B89tL89vL0/mbQw/CnabZHcANfP09aH57aX57eTS/9iCUUkq5pD0IpZRSLmmBUEop5VKzLRANXdCogccOEpFNzgsavSoiUqvtAefFjjaLyN/dm/qkDG7PLyJ/EpH9IrLeebvC/clPymHJ98DZ/oiIGBGJdV/i0zJY8T14WkQ2Or/+80WkvfuTn8hgRf7nRWSb8z18LiIt3Z/8RAYr8t/s/N2tFhFLBoPPJXcdz3eXiOx03u6qtb3e35FGOds5tb5+A0YBA4G0s3jsKmA4jpVnvwEud24fAywEgp2ft/ax/H8CHvHl74GzrSOOpVrSgVhfyg9E1trnQeBNH8t/KRDgvD8VmOpj+XsC3YHFQIo35XZmSjxlWwywx/kx2nk/ur73eCa3ZtuDMC4uaCQiSSIyV0TWiMgPItLj1MeJSDscv8TLjeO78C5wnbP5PuA5Y0yZ8zWyfSy/R1n4Hl4C/o96lol3ByvyG2MKau0ajoXvwaL8840xlc5dV1DHcjpenH+rMWa7VZnPJXcdLgMWGGPyjTFHgAXAOHf9njfbAlGHacADxphBwCPA6y72icexJHmN2hc06gaMFJGVIvK9iAy2NO3pzjU/wK+chwdmiEi0dVHrdE7vQUSuAfYbYzZYHbQO5/w9EJG/iUgmcBvwlIVZXXHHz1CNCTj+c/Ukd+b3pMbkdqWuC6y55T1atlifrxGRFsAI4JNah+qCXe3qYlvNf3kBOLp5w4DBwMci0sVZwS3lpvxvAE87P38aeBHHL7lHnOt7EJEw4Akchzk8zk3fA4wxTwBPiMhjwK9wrIRsOXfldz7XE0Al8L47M9bHnfk9qb7cInIP8JBzW1fgaxEpB/YaY66n7vfilveoBeInfsBRY0z/2htFxB9Y4/z0vzj+iNbuNte+oFEWMMdZEFaJSDWOxbU8cS3Uc85vjDlc63H/Bv5nZWAXzvU9JAGdgQ3OX7QOwFoRGWIcqwdbzR0/Q7V9AHyFhwoEbsrvHCi9ChjriX+OanH3199TXOYGMMa8DbwNICKLgbuNMftq7ZKF47IKNTrgGKvIwh3v0YpBGF+5AYnUGigClgE3O+8L0K+Ox63G0UuoGfy5wrl9CvAX5/1uOLp+4kP529Xa59fAh772PThln31YOEht0fcgudY+DwCf+lj+ccAWIM7qnx0rf36wcJD6bHNT9yD1XhxHLqKd92Ma8x4bldMT30RvvAGzgYNABY5qOxHHf59zgQ3OH/Kn6nhsCpAG7Ab+wU9npAcB7znb1gIX+Vj+WcAmYCOO/7TaWZXfqvdwyj77sHYWkxXfg8+c2zfiWFwt3sfy78Lxj9F6583KWVhW5L/e+VxlwGFgnrfkxkWBcG6f4Py67wLuOZPfkYZuutSGUkopl3QWk1JKKZe0QCillHJJC4RSSimXtEAopZRySQuEUkopl7RAqCZNRAo9/HrTRaSXm56rShyruqaJyJcNrYwqIi1F5JfueG2lQK8op5o4ESk0xrRw4/MFmJ8Wo7NU7ewi8g6wwxjzt3r2TwT+Z4zp7Yl8qunTHoRqdkQkTkQ+E5HVztv5zu1DRGSZiKxzfuzu3H63iHwiIl8C80XkQhFZLCKfiuPaB+/XrLXv3J7ivF/oXHhvg4isEJE2zu1Jzs9Xi8hfGtnLWc5PCxK2EJFFIrJWHOv9X+vc5zkgydnreN6576PO19koIn9245dRNQNaIFRz9ArwkjFmMHAjMN25fRswyhgzAMcqqs/Uesxw4C5jzEXOzwcADwO9gC7A+S5eJxxYYYzpBywBflHr9V9xvn6D6+M41xIai+PsdoBS4HpjzEAc1yB50Vmgfg/sNsb0N8Y8KiKXAsnAEKA/MEhERjX0ekrV0MX6VHN0MdCr1sqZkSISAUQB74hIMo6VLwNrPWaBMab2Gv6rjDFZACKyHsfaOktPeZ1yflrwcA1wifP+cH5am/8D4IU6cobWeu41ONb6B8faOs84/9hX4+hZtHHx+Eudt3XOz1vgKBhL6ng9pU6iBUI1R37AcGNMSe2NIvIa8J0x5nrn8fzFtZqLTnmOslr3q3D9u1Rhfhrkq2uf+pQYY/qLSBSOQnM/8CqO60TEAYOMMRUisg8IcfF4AZ41xvzrDF9XKUAPManmaT6O6ywAICI1yyxHAfud9++28PVX4Di0BXBLQzsbY47huPzoIyISiCNntrM4jAE6OXc9DkTUeug8YILzegOISLyItHbTe1DNgBYI1dSFiUhWrdtvcPyxTXEO3G7BsUw7wN+BZ0XkR8DfwkwPA78RkVVAO+BYQw8wxqzDsdLnLTguwpMiIqk4ehPbnPvkAT86p8U+b4yZj+MQ1nIR2QR8yskFRKl66TRXpTzMeeW7EmOMEZFbgPHGmGsbepxSnqZjEEp53iDgH86ZR0fx4GVdlToT2oNQSinlko5BKKWUckkLhFJKKZe0QCillHJJC4RSSimXtEAopZRy6f8DfG+PwO4rhRQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 2e-1\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.766112</td>\n",
       "      <td>0.790079</td>\n",
       "      <td>0.781676</td>\n",
       "      <td>0.780847</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.633976</td>\n",
       "      <td>0.567282</td>\n",
       "      <td>0.898636</td>\n",
       "      <td>0.898657</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.757286</td>\n",
       "      <td>0.706500</td>\n",
       "      <td>0.842105</td>\n",
       "      <td>0.833556</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.656997</td>\n",
       "      <td>0.558589</td>\n",
       "      <td>0.896686</td>\n",
       "      <td>0.897445</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.672892</td>\n",
       "      <td>0.761602</td>\n",
       "      <td>0.809942</td>\n",
       "      <td>0.817644</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.529195</td>\n",
       "      <td>0.470551</td>\n",
       "      <td>0.949318</td>\n",
       "      <td>0.950640</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(2, slice(lr/(2.6**4),lr), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.635578</td>\n",
       "      <td>0.555220</td>\n",
       "      <td>0.906433</td>\n",
       "      <td>0.906872</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.547100</td>\n",
       "      <td>0.463824</td>\n",
       "      <td>0.946394</td>\n",
       "      <td>0.947524</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(2, slice(lr/2/(2.6**4),lr/2), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.512873</td>\n",
       "      <td>0.447683</td>\n",
       "      <td>0.955166</td>\n",
       "      <td>0.955316</td>\n",
       "      <td>00:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.503354</td>\n",
       "      <td>0.461661</td>\n",
       "      <td>0.954191</td>\n",
       "      <td>0.953338</td>\n",
       "      <td>00:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.464976</td>\n",
       "      <td>0.430627</td>\n",
       "      <td>0.965887</td>\n",
       "      <td>0.965267</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.430950</td>\n",
       "      <td>0.434405</td>\n",
       "      <td>0.965887</td>\n",
       "      <td>0.965663</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(4, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.441612</td>\n",
       "      <td>0.432574</td>\n",
       "      <td>0.965887</td>\n",
       "      <td>0.965280</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.422067</td>\n",
       "      <td>0.429031</td>\n",
       "      <td>0.967836</td>\n",
       "      <td>0.967265</td>\n",
       "      <td>00:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.430873</td>\n",
       "      <td>0.432398</td>\n",
       "      <td>0.970760</td>\n",
       "      <td>0.970493</td>\n",
       "      <td>00:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.427765</td>\n",
       "      <td>0.433643</td>\n",
       "      <td>0.966862</td>\n",
       "      <td>0.966608</td>\n",
       "      <td>00:22</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(4, slice(lr/100/(2.6**4),lr/100), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.424508</td>\n",
       "      <td>0.429726</td>\n",
       "      <td>0.968811</td>\n",
       "      <td>0.968703</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')\n",
    "learn_c.fit_one_cycle(1, slice(lr/1000/(2.6**4),lr/1000), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.428017</td>\n",
       "      <td>0.431108</td>\n",
       "      <td>0.970760</td>\n",
       "      <td>0.970493</td>\n",
       "      <td>00:20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(1, slice(lr/1000/(2.6**4),lr/1000), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2');\n",
    "learn_c.to_fp32().export(f'{lang}_classifier_tcu_jurisp_reduzido_sp15_multifit_bwd_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.83 s, sys: 176 ms, total: 2.01 s\n",
      "Wall time: 907 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_tcu_jurisp_reduzido_sp15_multifit_bwd_v2', bs=bs, num_workers=1, backwards=True)\n",
    "\n",
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3\n",
    "\n",
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2', purge=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWUAAAEmCAYAAAC6SYF5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAeHklEQVR4nO3deXwU9f3H8dcnCadcUojcosipVTAgCEi96ol3q6KCiEipWu96tVaptfbXWltbj4r1ALVq1daWw6taFBEVUFCQUwHlviHcIfn8/pgJDZiEJLKZ75L38/HIg53Z2Zn3DsubyXdnds3dERGRMGQkHUBERP5HpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsgTHzGqZ2SgzW29mL36L9VxsZm/szWxJMbNjzGx20jkk9UznKUtFmdlFwA1AByAXmArc4+7vfcv19gd+AvR09x3fOmjgzMyBtu4+L+kskjwdKUuFmNkNwB+BXwMHAK2Ah4Gz9sLqDwTmVIVCLgszy0o6g1Qid9ePfsr1A9QHNgI/LGWZGkSlvST++SNQI77vWGARcCOwAlgKXBbfNwzYDuTF27gcuAt4psi6WwMOZMXTA4EviY7W5wMXF5n/XpHH9QQmAevjP3sWuW8ccDcwIV7PG0CjEp5bYf6bi+Q/GzgNmAOsAW4vsvxRwERgXbzsg0D1+L534+eyKX6+FxRZ/y3AMuDpwnnxY9rE2zgynm4GrAKOTfq1oZ9v/6MjZamIo4GawD9LWeZnQA+gM3AEUTH9vMj9TYjKvTlR8T5kZvu7+51ER98vuHsdd3+8tCBmth/wJ+BUd69LVLxTi1muITAmXvY7wP3AGDP7TpHFLgIuA7KB6sBNpWy6CdE+aA78AngMuATIAY4BfmFmB8fL5gPXA42I9t0JwJUA7t4nXuaI+Pm+UGT9DYl+axhSdMPu/gVRYT9rZrWBJ4Gn3H1cKXklTaiUpSK+A6zy0ocXLgZ+6e4r3H0l0RFw/yL358X357n7WKKjxPYVzFMAHGZmtdx9qbvPKGaZ04G57v60u+9w9+eAWcAZRZZ50t3nuPsW4O9E/6GUJI9o/DwPeJ6ocB9w99x4+zOAwwHcfYq7fxBvdwHwKPC9MjynO919W5xnF+7+GDAX+BBoSvSfoOwDVMpSEauBRnsY62wGLCwyvTCet3Mdu5X6ZqBOeYO4+yaiX/mHAkvNbIyZdShDnsJMzYtMLytHntXunh/fLizN5UXu31L4eDNrZ2ajzWyZmW0g+k2gUSnrBljp7lv3sMxjwGHAn9192x6WlTShUpaKmAhsJRpHLckSol+9C7WK51XEJqB2kekmRe9099fd/ftER4yziMpqT3kKMy2uYKbyeIQoV1t3rwfcDtgeHlPqaVFmVodonP5x4K54eEb2ASplKTd3X080jvqQmZ1tZrXNrJqZnWpmv40Xew74uZk1NrNG8fLPVHCTU4E+ZtbKzOoDtxXeYWYHmNmZ8djyNqJhkPxi1jEWaGdmF5lZlpldAHQCRlcwU3nUBTYAG+Oj+B/vdv9y4OBvPKp0DwBT3H0w0Vj5X751SgmCSlkqxN3vJzpH+efASuBr4GrglXiRXwGTgU+Bz4CP43kV2dabwAvxuqawa5FmEJ3FsYTojITvEb+Jtts6VgN942VXE5050dfdV1UkUzndRPQmYi7RUfwLu91/FzDCzNaZ2fl7WpmZnQWcQjRkA9Hfw5FmdvFeSyyJ0cUjIiIB0ZFypCbwETCN6F3zYfH8Z4HZwHTgCaBaPH9/otPBPo0fd1hlhk0XZtbSzP5rZjPNbIaZXZt0pnRiZqeY2Wwzm2dmtyadJ12Y2RNmtsLMpiedpSJSWspp9KLaBhxPdD5tZ6JfDXsQlXIH4LtALWBwvPztROOchwMDiMb35Jt2ADe6e0ei/XmVmXVKOFNaMLNM4CHgVKKx737ad2X2FNG/4bSUslJOsxeVE71BBNHRcLV43tj4Tyc6Im4RL9MJeCu+PYvoCrMDKilr2ojPGf44vp0LzGTXU9CkZEcB89z9S3ffTnQu9N64hH2f5+7vEr2/kJZSeaScbi+qTKKj3xXAm0Qn5ReqRnThw2vx9DTg3Pj2UUSnWrVASmRmrYEu7LpfpWTNid48LbQI/YdWJaTsjT4z+wFwSnzKTuEnf3V396t3W24IhZeRWlaO1dw/JXnKqn79evzj+ae45sbbmfH5LACGP/R7Nm3azPU33wFA3bp1+ON999DliO/y2fSZdGh/CFdceQOfflbchWSVo3PHVolte0/y8/OZO2cOTZo0ocH+yf79FmdPJwwnYe3atWzYsJ4DD2wNwOrVq9m8eRMtW4b79xySbdu28cUX8+jU6dCko5Ro4cIFrFq16hsvv1R++lRxr/Vv/A/g7sOB4QAZtbO9Rvs9nhGUUluB9z5fT9+Lrmfe029x+5BTOaB1Cy648a8UZtsOXHn/BKLProFZY45nSbUcarRP7gUw4YM/J7bt0uTl5XHe2Wdw0cWXcM11NyQdp1hm4dXyBxMncs/ddzFq7OsA/O7/7gXgp7fcVtrDJLZwwQLOPbsvEz6cnHSUEvXq3rXY+akcvlgEtCwy3YKKX9GVUo32r0P9OrUAqFmjGsd3b8/sBcsZeM7RfL9nRwbc9hRFf6OoX6cW1bIyAbjsnJ689/E8cjft6YrYqsfd+fGQwbTv0CHYQg5V127dmDdvLgvmz2f79u28+MLznN73zKRjSSVI5ZHyJKCtmR1EdCnrhUQn0AenSaN6PPbL/mRmZJCRYbz85se8On46uZMe4Kulaxg34kYA/vX2VO4d/hodDm7CX+/uT35+AbO+XMbQYc8m/AzCNPH9Cfzt2ac57LDv0r1rFwCG3X0Pp5x6WsLJwpeVlcUfHniQM04/mfz8fC4dOIhOh4b7q3hIBlzSj/HvjGPVqlW0ad2CO34xjIGDLk86Vpml9OIRMzuN6Pr8TOAJd7+ntOVDGL5IV2s+CnP4Ih2EOHwh+75e3bsyZcrkSh1TJv5IxrGp3IaIyL5EV/SJiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAQkK+kARXXp2IoJHz6YdIy0lN1/ZNIR0tbSEf2TjpDWMjMs6Qj7FB0pi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISECykg4Quh8NHsSrY0fTODubKVOnJx0nOM0b1ubRK3tzQIOaFDg89dYcHnltFk9e04e2TesBUH+/6qzftJ3et40mK9N4cEhPjmjdkKxM47nxX3L/v7Rfi9Op3UHUqVOXzMxMsrKyGD9xUtKR0sYbr7/GTTdcS35+PgMHDeanN9+adKQyS1kpm9kTQF9ghbsflqrtpFr/Swcy9MqrGTxoQNJRgrSjwPnZM5OZtmANdWpm8e6v+/L2Z0u57E/v7lzmnkty2LA5D4BzuremRlYGR98yilrVM/novrN4acJ8vlq1KamnELSxb7xNo0aNko6RVvLz87numqsY8+qbNG/Rgt49utG375l07NQp6Whlksrhi6eAU1K4/krR+5g+NGzYMOkYwVq+bgvTFqwBYOPWHcxevJ5mDWvvssw5PVrz0vvzAXCc2jWyyMwwalXPIm9HAblb8io9t+y7Jn30EW3aHMJBBx9M9erV+eEFFzJ61L+SjlVmKStld38XWJOq9Ut4WjXaj8NbN2TyvFU75/XskM2K9Vv4YlkuAK98uJDN23Yw95EfMuPP5/Kn0TNYu2l7UpGDZhhnnX4yvXt05Ym/Dk86TtpYsmQxLVq03DndvHkLFi9enGCi8kl8TNnMhgBDAFq2apVwGqmo/Wpk8fT1x3LryEm7HPn+oOdBvPT+gp3TOW0akV/gtLvyRRrsV4PX7zyZcdOXsmDFxgRSh+0/496jabNmrFixgjNPO4l27TvQ+5g+SccKnrt/Y56ZJZCkYhI/+8Ldh7t7V3fv2rhR46TjSAVkZRrPXH8sf5/wJaMmfbVzfmaGceZRrfjHxAU7553f6yD+M20JO/KdVRu28sGclXQ5+DuVHzoNNG3WDIDs7GzOOOtspkz6KOFE6aF58xYsWvT1zunFixfRLN6X6SDxUpb099CQnsxeso6Hxs7cZf5x323KnCXrWbJm8855X6/aRJ9DmwBQu0YW3Q5pxJwl6ys1bzrYtGkTubm5O2+//Z836XRo2r5fXqm6duvGvHlzWTB/Ptu3b+fFF57n9L5nJh2rzBIfvgjdgEv6Mf6dcaxatYo2rVtwxy+GMXDQ5UnHCkaP9tn069OG6V+t5b17+wLwyxc+4Y2piznv6Na7DF0APPbGbB4e2pMPf3cmBjzzzhfM+Gpd5QcP3Irly+l3/rkA7Nixg/Mv7Mf3T077980rRVZWFn944EHOOP1k8vPzuXTgIDodemjSscrMiht/2SsrNnsOOBZoBCwH7nT3x0t7TE5OV5/w4eSU5NnXZfcfmXSEtLV0RP+kI6S1zIz0Ga8NSa/uXZkyZfI3dl7KjpTdvV+q1i0isq/SmLKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAVEpi4gERKUsIhIQlbKISEBUyiIiAckq6Q4zGwV4Sfe7+5kpSSQiUoWVWMrAfZWWQkREgFJK2d3fqcwgIiJS+pEyAGbWFrgX6ATULJzv7genMJeISJVUljf6ngQeAXYAxwEjgadTGUpEpKoqSynXcve3AHP3he5+F3B8amOJiFRNexy+ALaaWQYw18yuBhYD2amNJSJSNZXlSPk6oDZwDZAD9AcuTWUoEZGqao9Hyu4+Kb65EbgstXFERKq2spx98V+KuYjE3TWuLCKyl5VlTPmmIrdrAucRnYkhIiJ7WVmGL6bsNmuCmenCksAsH9k/6Qhp6/RHJiYdIa2NGnp00hHSUkmfYVGW4YuGRSYziN7sa7I3QomIyK7KMnwxhajUjWjYYj5weSpDiYhUVWUp5Y7uvrXoDDOrkaI8IiJVWlnOU36/mHkahBMRSYHSPk+5CdAcqGVmXYiGLwDqEV1MIiIie1lpwxcnAwOBFsDv+V8pbwBuT20sEZGqqbTPUx4BjDCz89z95UrMJCJSZZVlTDnHzBoUTpjZ/mb2qxRmEhGpsspSyqe6+7rCCXdfC5yWukgiIlVXWUo5s+gpcGZWC9ApcSIiKVCW85SfAd4ysyfj6cuAEamLJCJSdZXlsy9+a2afAicSnYHxGnBgqoOJiFRFZRm+AFgGFBB9QtwJwMyUJRIRqcJKu3ikHXAh0A9YDbxA9D19x1VSNhGRKqe04YtZwHjgDHefB2Bm11dKKhGRKqq04YvziIYt/mtmj5nZCfzvqj4REUmBEkvZ3f/p7hcAHYBxwPXAAWb2iJmdVEn5RESqlD2+0efum9z9WXfvS/Q5GFOBW1OeTESkCirr2RcAuPsad39UX5oqIpIa5SplERFJLZWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgFRKYuIBKS0L04V4EeDB/Hq2NE0zs5mytTpScdJK4u+/prBgy5l+bJlZGRkMGjwFVz1k2uTjhWMxnWqc+tJbWlYuxruMHr6cv4xbSl1a2Rxx6ntaFKvBss2bOOXr85m47Z8AI5oXo+r+hxEVoaxfmse1788I+FnEab8/HyOObobzZo156VXRiUdp1xSVspm1hIYCTQBCoDh7v5AqraXKv0vHcjQK69m8KABSUdJO5lZWdz72/vo0uVIcnNz6dW9K8ef8H06duqUdLQg5Bc4fxm/gLkrN1GrWgZ/ufAIpny9jpM7ZvPJ1+t5bspi+uU0p19OCx57fyH7Vc/k2uMO5tZXPmfFxu00qFUt6acQrIf//ADtO3Qkd8OGpKOUWyqHL3YAN7p7R6AHcJWZpd2/xt7H9KFhw4ZJx0hLTZs2pUuXIwGoW7cu7Tt0ZMmSxQmnCseazXnMXbkJgC15BXy1dguN9qtOr4Mb8vrMFQC8PnMFvdtEr78T2jfmvXmrWbFxOwDrtuQlEzxwixct4rVXx3LpZZcnHaVCUnak7O5LgaXx7Vwzmwk0Bz5P1TYlXAsXLGDatE/odlT3pKME6YC6NTik8X7MXL6R/WtXY83mqHDXbM7beUTcskFNMjON+889lNrVM3l56lLenLUyydhBuvmm6/nVvf9Hbm5u0lEqpFLe6DOz1kAX4MNi7htiZpPNbPLKVXqB7Ys2btxIvwt+wG/v+wP16tVLOk5walbLYNjp7Xn43fls3p5f4nKZGUa77Drc/u+Z3PzK5/Q/qgUtGtSsxKThe3XMaBo3bkyXI3OSjlJhKS9lM6sDvAxc5+7fGOBx9+Hu3tXduzZu1DjVcaSS5eXlcdEFP+DCfhdx9jnnJh0nOJkZxrDT2vOf2SsZ/8UaANZuzqNh7ejouGHtajuHKVZu3M6khevYuqOADVt38OniDbRptF9i2UP0wcQJjB0zik7tDmJg/368M+5tLh/YP+lY5ZLSUjazakSF/Ky7/yOV25LwuDs/HjKY9h06cM11NyQdJ0g/PaENX63ZwkufLN057/0v13Byx2wATu6YzYQvo7Ke8OUavtusLhkGNbIy6NikLgvXbEkkd6iG/epe5nz5NZ/Pmc9TTz/H9449nsefejrpWOWSslI2MwMeB2a6+/2p2k6qDbikH8ceczRzZs+mTesWPPXE40lHShsT35/A3559mnf++1+6d+1C965deO3VsUnHCsZhTetyUsdsOresz/B+RzC83xF0P7ABz01ZTE6r+owc0IWcVvV5bnL05uhXa7cwaeE6/npxZx6+4HDGzljOgjWbE34WsreZu6dmxWa9gfHAZ0SnxAHc7u4l/qvMyenqEz6cnJI8+7pU/T1WBac/MjHpCGlt1NCjk46Qlo45uhsfT5lsu89P5dkX7wHf2KCIiJRMl1mLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAREpSwiEhCVsohIQFTKIiIBUSmLiAQkK+kAsneYWdIR0tY/ruiedIS0Nvj5qUlHSEsL1mwudr6OlEVEAqJSFhEJiEpZRCQgKmURkYColEVEAqJSFhEJiEpZRCQgKmURkYColEVEAqJSFhEJiEpZRCQgKmURkYColEVEAqJSFhEJiEpZRCQgKmURkYColEVEAqJSFhEJiEpZRCQgKmURkYColEVEAqJSFhEJiEpZRCQgKmURkYColEVEAqJSFhEJiEpZRCQgKmURkYColEVEAqJSFhEJiEpZRCQgKmURkYColEVEAqJSFhEJiEpZRCQgWUkHCN2PBg/i1bGjaZydzZSp05OOk1a078rn6h8N5vXXxtCocTYTJ08D4I7bb+b1sWOoVr06Bx10MA89+jj1GzRIOGkYftSzFV2a12PD1h3cPGoWAAfuX4vLe7SkWqZRUABPfPg1X6zeTE7L+pzfuSkF7hQUwMjJi5i9YlPCz6B4KTtSNrOaZvaRmU0zsxlmNixV20ql/pcO5F+jX0s6RlrSviuffv0H8NIrY3aZd9zxJ/L+5GlM+OgT2rRty/33/SahdOF5Z95qfvPWF7vMuyinGS9PW8Zto2fz4rSlXJTTDIDpS3O5ZdQsbhs9m0ffX8gVR7dKInKZpHL4YhtwvLsfAXQGTjGzHincXkr0PqYPDRs2TDpGWtK+K59evfuw/2776/gTTyIrK/qFtlu3HixZvDiJaEGatWITG7fl7zLPgVrVo1qrXS2TtVvyANi2o2DnMjWyMqIFA5Wy4Qt3d2BjPFkt/gl4V4iE7ZmRT3LOD85POkbQRk5axG0nHsIlOc0xgztfnbPzvq4t63Phkc2oXzOL3+52hB0Si7ozRSs3ywSmAIcAD7n7LcUsMwQYEk+2B2anLFDFVQc6AJ8mHSQNad+VT3WgLTAjnm4ErAKaAPsB4bZJAtq1a1d99OjRbdu1azcD4Mknn2w5bty43BEjRqzr379/q6FDh9bs1avXnKKPOeWUU+rccccdzXafn4AD3b3x7jNTWso7N2LWAPgn8BN3T7t3fMysNTDT3WslHCXtaN+VT7y/Rrv7YfH0ZODPwFDgBHffnFy6ILUGRgOHxdPrgQaAZ2RkTC4oKGgH1CvmcfOBbkT/4QWlUk6Jc/d1wDjglMrYnsg+pB5wC3CmCrlMlgDfAzjjjDPqAnPj+YcAFt8+kug3ktWVnq4MUnn2ReP4CBkzqwWcCMxK1fZSxcyeAyYCNcxskZldnnSmdKF9Vz5F9lf7IvurFVAXeNPMpprZXxINGZad+wtYBFwOXAH8Hph29913N+d/Q6PnAdOBqcBDwAUE+h5XyoYvzOxwYASQSVT+f3f3X6ZkY5XAzIa4+/Ckc6Qj7buK076ruHTdd5UypiwiImWjy6xFRAKiUhYRCYhKWUQkIPpAohKYWQfgLKA50bu0S4B/u/vMRIOJSLHM7Ciii4knmVknolNwZ7n72ISjlYuOlIthZrcAzxOd1/gRMCm+/ZyZ3ZpkNtm3mVkHMzvBzOrsNl/n+JfCzO4E/gQ8Ymb3Ag8CdYBbzexniYYrJ519UQwzmwMc6u55u82vDsxw97bJJEtvZnaZuz+ZdI5Qmdk1wFXATKIP8brW3f8V3/exux+ZZL6QmdlnRPusBrAMaOHuG+JrJD5098MTDVgOOlIuXgHQrJj5TeP7pGLS8uNbK9EVQI67nw0cC9xhZtfG91mJjxKAHe6eH1/1+IW7bwBw9y2k2b9ZjSkX7zrgLTObC3wdz2tFdKnm1YmlSgNmVtIHDxlwQGVmSUOZ7r4RwN0XmNmxwEtmdiAq5T3Zbma141LOKZxpZvVJs1LW8EUJzCwDOIrojT4juoxzkrvnl/rAKs7MlgMnA2t3vwt4392L+w1EADN7G7jB3acWmZcFPAFc7O6ZiYULnJnVcPdtxcxvBDR1988SiFUhOlIugbsXAB8knSMNjQbqFC2WQmY2rvLjpJUBwI6iM9x9BzDAzB5NJlJ6KK6Q4/mrCPCT4EqjI2URkYDojT4RkYColEVEAqJSlrRgZvnx5wlPN7MXzaz2t1jXsWY2Or59ZmkXBJlZAzO7sgLbuMvMbqpoRqm6VMqSLra4e+f4a5K2E3090k4WKffr2d3/7e6/KWWRBkC5S1mkolTKko7GA4eYWWszm2lmDwMfAy3N7CQzm2hmH8dH1HUgukzZzGaZ2XvAuYUrMrOBZvZgfPsAM/unmU2Lf3oCvwHaxEfpv4uX+6mZTTKzT81sWJF1/czMZpvZf4i+DUOk3FTKklbi83ZPBQrPO20PjHT3LsAm4OfAifElyZOBG8ysJvAYcAZwDNE3QxfnT8A77n4E0fe4zQBuJbpCrLO7/9TMTiL6tumjiC7rzTGzPmaWA1wIdCEq/W57+alLFaHzlCVd1DKzwnOfxwOPE10Kv9DdC88n7wF0AiaYGURfjjkR6ADMd/e5AGb2DP/77raijic6V5j4IqH1Zrb/bsucFP98Ek/XISrpusA/C7/c1Mz+/a2erVRZKmVJF1vcvXPRGXHxbio6C3jT3fvttlxn9t6XZBpwr7vvcjGHmV23F7chVZiGL2Rf8gHQy8wOATCz2mbWjuhb1A8yszbxcv1KePxbwI/jx2aaWT0gl+gouNDrwKAiY9XNzSwbeBc4x8xqmVldoqESkXJTKcs+w91XAgOJPvf6U6KS7uDuW4mGK8bEb/QtLGEV1wLHxR8DOYXo41tXEw2HTDez37n7G8DfgInxci8Bdd39Y+AFoq+wf5loiEWk3HSZtYhIQHSkLCISEJWyiEhAVMoiIgFRKYuIBESlLCISEJWyiEhAVMoiIgH5fzmYn1bqSI7GAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds,y,losses = learn_c.get_preds(with_loss=True)\n",
    "predictions = np.argmax(preds, axis = 1)\n",
    "\n",
    "interp = ClassificationInterpretation(learn_c, preds, y, losses)\n",
    "interp.plot_confusion_matrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[329   2   0   1]\n",
      " [  1 278   5   0]\n",
      " [  1   2 206   4]\n",
      " [  1   1  12 183]]\n",
      "accuracy global: 0.9707602339181286\n",
      "accuracy on class 0: 99.09638554216868\n",
      "accuracy on class 1: 97.88732394366197\n",
      "accuracy on class 2: 96.71361502347418\n",
      "accuracy on class 3: 92.89340101522842\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(np.array(y), np.array(predictions))\n",
    "print(cm)\n",
    "\n",
    "## acc\n",
    "print(f'accuracy global: {(cm[0,0]+cm[1,1]+cm[2,2]+cm[3,3])/(cm.sum())}')\n",
    "\n",
    "# acc neg, acc pos\n",
    "print(f'accuracy on class 0: {cm[0,0]/(cm.sum(1)[0])*100}') \n",
    "print(f'accuracy on class 1: {cm[1,1]/(cm.sum(1)[1])*100}')\n",
    "print(f'accuracy on class 2: {cm[2,2]/(cm.sum(1)[2])*100}')\n",
    "print(f'accuracy on class 3: {cm[3,3]/(cm.sum(1)[3])*100}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>. ▁ ) ▁1993 ▁/ 66 6 . ▁8 ▁lei ▁xxmaj ▁da o ▁3 . art ▁( ” ▁administração ▁xxmaj ▁a ▁para a jos ta ▁van ▁mais ▁proposta ▁a r ciona le se ▁“ ▁a ▁modo ▁de , s ▁interessado uais vent ▁e ▁os ▁entre ” mia no ▁iso ▁da ▁constitucional ▁princípio ▁do ância v er ▁obs ▁a r anti gar ▁“ ▁necessariamente ▁deverá ▁administração ▁xxmaj ▁a , ▁questão ▁em</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁2002. ▁/ 20 5 . ▁10 ▁lei ▁xxmaj ▁da o ▁9 . ▁art ▁do ▁força ▁por gão ▁pre ▁xxmaj ▁ao mente aria idi s ▁sub ável c pli ▁a ▁1993, ▁/ 66 6 . ▁8 ▁lei ▁xxmaj ▁da , o ▁2 § ▁ , ▁45 . ▁art ▁no ▁disposto ▁o ▁observado , ▁público ▁interesse ▁ao á ▁atender ▁que ▁oferta ▁da ▁sorteio ▁ao r ▁procede ▁1991, ▁/ 48 2 . ▁8</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>. são ▁pen ▁de ▁benefício ▁do ▁econômica ▁dependência ▁a ize ter ac scar ▁de ▁que ▁renda r feri u ▁a ) ▁c ▁ou ; ▁ocupação ▁dessa corrente ▁de ▁aposentadoria ▁receber ▁ou , ▁indireta ▁xxmaj ▁ou ▁direta ▁xxmaj ▁pública ▁xxmaj ▁administração ▁xxmaj ▁na ▁efetivo ▁cargo ▁ocupar ) ▁b ; ▁estável ▁união ▁de ▁situação ▁na ▁encontrar ▁se ▁ou ▁casamento ído ▁contra ▁ter ) ▁a : ▁benefício ▁do ▁percepção ▁à ▁direito ▁do tivas</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>. ▁escolhida ▁opção ▁a ▁para ção ▁motiva ▁adequada ▁a ▁observar ndo ▁deve ▁1998, ▁/ 36 6 . ▁9 ▁lei ▁xxmaj ▁da , o ▁2 § ▁ , ▁30 . ▁art ▁do ▁e ▁1993 ▁/ 66 6 . ▁8 ▁lei ▁xxmaj ▁da , ▁i ▁xxup so inci ▁ , ▁17 . ▁art ▁do ▁termos ▁nos , tório cita ▁li ▁procedimento ▁o ▁realizar ▁ou ▁1993, ▁/ 66 6 . ▁8 ▁lei ▁xxmaj</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>. ▁vigor ▁em mente ida ▁val ▁estiver ▁alternativa ▁medida ▁a ▁enquanto ▁eficácia ▁sua e ▁suspend ▁mas , s ▁pagamento ▁de ção ten ▁re ▁de lar te cau ▁ ▁a ▁revoga ▁não ▁referência ▁em ▁alternativa ▁medida ▁da ▁adoção ▁a , ▁casos s ▁nesse ▁xxmaj . ▁valores ir titu res ▁ ▁a ▁contratada ▁empresa ▁a ne ▁conde ▁que ▁tribunal ▁xxmaj ▁deste dão r có ▁a ▁eventual ▁de ▁julgado ▁em ▁trânsito ▁o ▁após</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.show_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions some random sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(Category 3, tensor(3), tensor([0.0095, 0.1228, 0.0891, 0.7785]))\n"
     ]
    }
   ],
   "source": [
    "# Get the prediction\n",
    "test_text = \"A medida cautelar do TCU que determina a suspensão de licitação por falhas no edital não impede o órgão ou a entidade de rever seu ato convocatório, valendo-se do poder de autotutela (art. 49 da Lei 8.666/1993 c/c o art. 9º da Lei 10.520/2002) , com o objetivo de, antecipando-se a eventual deliberação do Tribunal, promover de modo próprio a anulação da licitação e o refazimento do edital, livre dos vícios apontados.\"\n",
    "pred = learn_c.predict(test_text)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.044\" style=\"background-color: rgba(247, 246, 250, 0.5);\">.</span> <span title=\"0.060\" style=\"background-color: rgba(245, 244, 249, 0.5);\">s</span> <span title=\"0.128\" style=\"background-color: rgba(238, 236, 244, 0.5);\">▁apontado</span> <span title=\"0.116\" style=\"background-color: rgba(240, 238, 245, 0.5);\">s</span> <span title=\"0.239\" style=\"background-color: rgba(219, 219, 235, 0.5);\">ício</span> <span title=\"0.227\" style=\"background-color: rgba(221, 221, 236, 0.5);\">▁v</span> <span title=\"0.027\" style=\"background-color: rgba(249, 247, 251, 0.5);\">▁dos</span> <span title=\"0.045\" style=\"background-color: rgba(247, 246, 250, 0.5);\">▁livre</span> <span title=\"0.032\" style=\"background-color: rgba(248, 247, 250, 0.5);\">,</span> <span title=\"0.067\" style=\"background-color: rgba(245, 243, 248, 0.5);\">tal</span> <span title=\"0.070\" style=\"background-color: rgba(245, 243, 248, 0.5);\">di</span> <span title=\"0.033\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁e</span> <span title=\"0.047\" style=\"background-color: rgba(247, 245, 249, 0.5);\">▁do</span> <span title=\"0.178\" style=\"background-color: rgba(230, 229, 240, 0.5);\">mento</span> <span title=\"0.379\" style=\"background-color: rgba(187, 188, 219, 0.5);\">zi</span> <span title=\"0.160\" style=\"background-color: rgba(233, 232, 242, 0.5);\">fa</span> <span title=\"0.089\" style=\"background-color: rgba(243, 241, 247, 0.5);\">▁re</span> <span title=\"0.014\" style=\"background-color: rgba(250, 249, 252, 0.5);\">▁o</span> <span title=\"0.025\" style=\"background-color: rgba(249, 248, 251, 0.5);\">▁e</span> <span title=\"0.055\" style=\"background-color: rgba(246, 244, 249, 0.5);\">ção</span> <span title=\"0.071\" style=\"background-color: rgba(244, 243, 248, 0.5);\">cita</span> <span title=\"0.074\" style=\"background-color: rgba(244, 243, 248, 0.5);\">▁li</span> <span title=\"0.028\" style=\"background-color: rgba(249, 247, 251, 0.5);\">▁da</span> <span title=\"0.074\" style=\"background-color: rgba(244, 243, 248, 0.5);\">ulação</span> <span title=\"0.048\" style=\"background-color: rgba(247, 245, 249, 0.5);\">▁an</span> <span title=\"0.049\" style=\"background-color: rgba(247, 245, 249, 0.5);\">▁a</span> <span title=\"0.109\" style=\"background-color: rgba(240, 238, 245, 0.5);\">▁próprio</span> <span title=\"0.098\" style=\"background-color: rgba(241, 240, 246, 0.5);\">▁modo</span> <span title=\"0.081\" style=\"background-color: rgba(243, 242, 247, 0.5);\">▁de</span> <span title=\"0.116\" style=\"background-color: rgba(240, 238, 245, 0.5);\">▁promover</span> <span title=\"0.101\" style=\"background-color: rgba(241, 240, 246, 0.5);\">,</span> <span title=\"0.227\" style=\"background-color: rgba(221, 221, 236, 0.5);\">▁tribunal</span> <span title=\"0.100\" style=\"background-color: rgba(241, 240, 246, 0.5);\">▁xxmaj</span> <span title=\"0.035\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁do</span> <span title=\"0.129\" style=\"background-color: rgba(238, 236, 244, 0.5);\">ção</span> <span title=\"0.284\" style=\"background-color: rgba(210, 210, 231, 0.5);\">libera</span> <span title=\"0.015\" style=\"background-color: rgba(250, 249, 252, 0.5);\">▁de</span> <span title=\"0.018\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁eventual</span> <span title=\"0.006\" style=\"background-color: rgba(251, 250, 252, 0.5);\">▁a</span> <span title=\"0.011\" style=\"background-color: rgba(251, 250, 252, 0.5);\">se</span> <span title=\"0.014\" style=\"background-color: rgba(250, 249, 252, 0.5);\">-</span> <span title=\"0.049\" style=\"background-color: rgba(247, 245, 249, 0.5);\">ndo</span> <span title=\"0.071\" style=\"background-color: rgba(244, 243, 248, 0.5);\">▁antecipa</span> <span title=\"0.011\" style=\"background-color: rgba(251, 250, 252, 0.5);\">,</span> <span title=\"0.009\" style=\"background-color: rgba(251, 250, 252, 0.5);\">▁de</span> <span title=\"0.015\" style=\"background-color: rgba(250, 249, 252, 0.5);\">▁objetivo</span> <span title=\"0.008\" style=\"background-color: rgba(251, 250, 252, 0.5);\">▁o</span> <span title=\"0.010\" style=\"background-color: rgba(251, 250, 252, 0.5);\">▁com</span> <span title=\"0.020\" style=\"background-color: rgba(249, 248, 251, 0.5);\">,</span> <span title=\"0.035\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁</span> <span title=\"0.060\" style=\"background-color: rgba(245, 244, 249, 0.5);\">)</span> <span title=\"0.038\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁2002</span> <span title=\"0.042\" style=\"background-color: rgba(247, 246, 250, 0.5);\">▁/</span> <span title=\"0.086\" style=\"background-color: rgba(243, 241, 247, 0.5);\">20</span> <span title=\"0.056\" style=\"background-color: rgba(246, 244, 249, 0.5);\">5</span> <span title=\"0.017\" style=\"background-color: rgba(250, 249, 251, 0.5);\">.</span> <span title=\"0.017\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁10</span> <span title=\"0.011\" style=\"background-color: rgba(251, 250, 252, 0.5);\">▁lei</span> <span title=\"0.010\" style=\"background-color: rgba(251, 250, 252, 0.5);\">▁xxmaj</span> <span title=\"0.028\" style=\"background-color: rgba(249, 247, 251, 0.5);\">▁da</span> <span title=\"0.058\" style=\"background-color: rgba(246, 244, 249, 0.5);\">o</span> <span title=\"0.014\" style=\"background-color: rgba(250, 249, 252, 0.5);\">▁9</span> <span title=\"0.020\" style=\"background-color: rgba(249, 248, 251, 0.5);\">.</span> <span title=\"0.019\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁art</span> <span title=\"0.011\" style=\"background-color: rgba(251, 250, 252, 0.5);\">▁o</span> <span title=\"0.019\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁c</span> <span title=\"0.017\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁/</span> <span title=\"0.027\" style=\"background-color: rgba(249, 248, 251, 0.5);\">▁c</span> <span title=\"0.053\" style=\"background-color: rgba(246, 245, 249, 0.5);\">▁1993</span> <span title=\"0.090\" style=\"background-color: rgba(242, 240, 247, 0.5);\">▁/</span> <span title=\"0.155\" style=\"background-color: rgba(234, 232, 242, 0.5);\">66</span> <span title=\"0.046\" style=\"background-color: rgba(247, 246, 250, 0.5);\">6</span> <span title=\"0.022\" style=\"background-color: rgba(249, 248, 251, 0.5);\">.</span> <span title=\"0.032\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁8</span> <span title=\"0.036\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁lei</span> <span title=\"0.005\" style=\"background-color: rgba(251, 250, 252, 0.5);\">▁xxmaj</span> <span title=\"0.015\" style=\"background-color: rgba(250, 249, 252, 0.5);\">▁da</span> <span title=\"0.030\" style=\"background-color: rgba(249, 247, 251, 0.5);\">49</span> <span title=\"0.019\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁</span> <span title=\"0.029\" style=\"background-color: rgba(249, 247, 251, 0.5);\">.</span> <span title=\"0.027\" style=\"background-color: rgba(249, 248, 251, 0.5);\">art</span> <span title=\"0.055\" style=\"background-color: rgba(246, 245, 249, 0.5);\">▁(</span> <span title=\"0.153\" style=\"background-color: rgba(234, 232, 242, 0.5);\">la</span> <span title=\"0.179\" style=\"background-color: rgba(230, 229, 240, 0.5);\">te</span> <span title=\"0.104\" style=\"background-color: rgba(241, 239, 246, 0.5);\">tu</span> <span title=\"0.143\" style=\"background-color: rgba(236, 234, 243, 0.5);\">▁auto</span> <span title=\"0.037\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁de</span> <span title=\"0.035\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁poder</span> <span title=\"0.034\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁do</span> <span title=\"0.061\" style=\"background-color: rgba(245, 244, 249, 0.5);\">se</span> <span title=\"0.019\" style=\"background-color: rgba(250, 249, 251, 0.5);\">-</span> <span title=\"0.049\" style=\"background-color: rgba(247, 245, 249, 0.5);\">ndo</span> <span title=\"0.058\" style=\"background-color: rgba(246, 244, 249, 0.5);\">▁vale</span> <span title=\"0.050\" style=\"background-color: rgba(247, 245, 249, 0.5);\">,</span> <span title=\"0.170\" style=\"background-color: rgba(231, 230, 241, 0.5);\">tório</span> <span title=\"0.314\" style=\"background-color: rgba(202, 203, 227, 0.5);\">▁convoca</span> <span title=\"0.171\" style=\"background-color: rgba(231, 230, 241, 0.5);\">▁ato</span> <span title=\"0.070\" style=\"background-color: rgba(245, 243, 248, 0.5);\">▁seu</span> <span title=\"0.138\" style=\"background-color: rgba(236, 235, 244, 0.5);\">▁rever</span> <span title=\"0.038\" style=\"background-color: rgba(248, 247, 250, 0.5);\">▁de</span> <span title=\"0.084\" style=\"background-color: rgba(243, 241, 247, 0.5);\">▁entidade</span> <span title=\"0.019\" style=\"background-color: rgba(250, 249, 251, 0.5);\">▁a</span> <span title=\"0.060\" style=\"background-color: rgba(245, 244, 249, 0.5);\">▁ou</span> <span title=\"0.120\" style=\"background-color: rgba(239, 237, 245, 0.5);\">▁órgão</span> <span title=\"0.068\" style=\"background-color: rgba(245, 243, 248, 0.5);\">▁o</span> <span title=\"0.171\" style=\"background-color: rgba(231, 230, 241, 0.5);\">▁impede</span> <span title=\"0.100\" style=\"background-color: rgba(241, 240, 246, 0.5);\">▁não</span> <span title=\"0.209\" style=\"background-color: rgba(225, 224, 238, 0.5);\">tal</span> <span title=\"0.108\" style=\"background-color: rgba(240, 239, 246, 0.5);\">di</span> <span title=\"0.065\" style=\"background-color: rgba(245, 243, 248, 0.5);\">▁e</span> <span title=\"0.059\" style=\"background-color: rgba(245, 244, 249, 0.5);\">▁no</span> <span title=\"0.183\" style=\"background-color: rgba(229, 228, 240, 0.5);\">s</span> <span title=\"0.434\" style=\"background-color: rgba(173, 172, 210, 0.5);\">▁falha</span> <span title=\"0.045\" style=\"background-color: rgba(247, 246, 250, 0.5);\">▁por</span> <span title=\"0.058\" style=\"background-color: rgba(246, 244, 249, 0.5);\">ção</span> <span title=\"0.094\" style=\"background-color: rgba(242, 240, 247, 0.5);\">cita</span> <span title=\"0.081\" style=\"background-color: rgba(243, 242, 247, 0.5);\">▁li</span> <span title=\"0.160\" style=\"background-color: rgba(232, 231, 242, 0.5);\">▁de</span> <span title=\"0.314\" style=\"background-color: rgba(202, 203, 227, 0.5);\">▁suspensão</span> <span title=\"0.096\" style=\"background-color: rgba(242, 240, 246, 0.5);\">▁a</span> <span title=\"0.189\" style=\"background-color: rgba(228, 227, 239, 0.5);\">▁determina</span> <span title=\"0.084\" style=\"background-color: rgba(243, 241, 247, 0.5);\">▁que</span> <span title=\"0.105\" style=\"background-color: rgba(241, 239, 246, 0.5);\">cu</span> <span title=\"0.079\" style=\"background-color: rgba(243, 242, 247, 0.5);\">t</span> <span title=\"0.069\" style=\"background-color: rgba(245, 243, 248, 0.5);\">▁</span> <span title=\"0.102\" style=\"background-color: rgba(241, 239, 246, 0.5);\">▁xxup</span> <span title=\"0.297\" style=\"background-color: rgba(206, 206, 229, 0.5);\">▁do</span> <span title=\"1.000\" style=\"background-color: rgba(63, 0, 125, 0.5);\">lar</span> <span title=\"0.922\" style=\"background-color: rgba(75, 23, 135, 0.5);\">te</span> <span title=\"0.487\" style=\"background-color: rgba(161, 157, 202, 0.5);\">cau</span> <span title=\"0.357\" style=\"background-color: rgba(192, 193, 222, 0.5);\">▁</span> <span title=\"0.251\" style=\"background-color: rgba(217, 217, 234, 0.5);\">▁medida</span> <span title=\"0.095\" style=\"background-color: rgba(242, 240, 246, 0.5);\">▁a</span> <span title=\"0.155\" style=\"background-color: rgba(234, 232, 242, 0.5);\">▁xxbos</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The darker the word-shading in the below example, the more it contributes to the classification. \n",
    "txt_ci = TextClassificationInterpretation.from_learner(learn_c)\n",
    "txt_ci.show_intrinsic_attention(test_text,cmap=plt.cm.Purples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.0440, 0.0597, 0.1282, 0.1160, 0.2387, 0.2270, 0.0274, 0.0446, 0.0321,\n",
       "        0.0675, 0.0699, 0.0328, 0.0472, 0.1784, 0.3785, 0.1595, 0.0891, 0.0143,\n",
       "        0.0246, 0.0554, 0.0706, 0.0741, 0.0281, 0.0738, 0.0480, 0.0486, 0.1095,\n",
       "        0.0978, 0.0815, 0.1162, 0.1015, 0.2267, 0.1001, 0.0349, 0.1285, 0.2844,\n",
       "        0.0148, 0.0181, 0.0062, 0.0109, 0.0138, 0.0493, 0.0709, 0.0110, 0.0093,\n",
       "        0.0148, 0.0079, 0.0101, 0.0200, 0.0354, 0.0605, 0.0379, 0.0420, 0.0856,\n",
       "        0.0557, 0.0167, 0.0172, 0.0105, 0.0098, 0.0284, 0.0583, 0.0142, 0.0201,\n",
       "        0.0193, 0.0110, 0.0186, 0.0167, 0.0271, 0.0528, 0.0899, 0.1545, 0.0456,\n",
       "        0.0219, 0.0315, 0.0358, 0.0052, 0.0151, 0.0303, 0.0187, 0.0293, 0.0269,\n",
       "        0.0546, 0.1531, 0.1788, 0.1044, 0.1432, 0.0371, 0.0347, 0.0338, 0.0615,\n",
       "        0.0187, 0.0492, 0.0584, 0.0502, 0.1698, 0.3144, 0.1714, 0.0701, 0.1377,\n",
       "        0.0382, 0.0842, 0.0190, 0.0597, 0.1201, 0.0680, 0.1715, 0.0999, 0.2089,\n",
       "        0.1084, 0.0650, 0.0587, 0.1827, 0.4336, 0.0451, 0.0576, 0.0937, 0.0808,\n",
       "        0.1604, 0.3141, 0.0955, 0.1888, 0.0840, 0.1050, 0.0788, 0.0690, 0.1024,\n",
       "        0.2974, 1.0000, 0.9221, 0.4872, 0.3569, 0.2507, 0.0954, 0.1546],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt_ci.intrinsic_attention(test_text)[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Text</th>\n",
       "      <th>Prediction</th>\n",
       "      <th>Actual</th>\n",
       "      <th>Loss</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁consideram - se ▁i liquid áveis ▁as ▁conta s , ▁ordena ndo - se ▁o ▁seu ▁tran ca mento , ▁em ▁razão ▁da ▁imp os s ibilidade ▁do ▁exercício ▁de ▁ampla ▁defesa , ▁pelo ▁longo ▁de curso ▁de ▁tempo ▁entre ▁a ▁prática ▁do ▁ato ▁e ▁a ▁cita ção ▁do ▁responsável .</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3.10</td>\n",
       "      <td>0.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁como ▁regra ▁geral , ▁sujeita ▁a ▁po nder ação ▁no ▁caso ▁concreto , ▁o ▁parcela mento ▁do ▁objeto ▁deve ▁ser ▁adotado ▁na ▁contratação ▁de ▁serviços ▁de ▁maior ▁especialização ▁técnica , ▁sendo ▁de s ne ces s ário ▁nos ▁serviços ▁de ▁menor ▁especialização .</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.76</td>\n",
       "      <td>0.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁falecido ▁o ▁responsável , ▁a ▁obriga ção ▁de ▁re para r ▁o ▁da no ▁re ca i ▁sobre ▁o ▁seu ▁ esp ólio ▁ou , ▁caso ▁concluída ▁a ▁partilha , ▁aos ▁sucesso res ▁até ▁o ▁limite ▁do ▁valor ▁do ▁patrimônio ▁transferido . ▁xxmaj ▁ ante ▁o ▁seu ▁caráter ▁ persona l íssimo , ▁a ▁multa ▁não ▁se ▁trans fer e ▁aos ▁sucesso res .</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.64</td>\n",
       "      <td>0.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁ao ▁firma r ▁termo ▁de ▁parceria ▁com ▁xxmaj ▁os cip ▁que ▁em ▁a ve nça ▁anterior ▁deixou ▁de ▁obedece r ▁normas ▁técnicas ▁na ▁execução ▁de ▁projeto ▁semelhante ▁e ▁de ▁mesma ▁natureza , ▁apresentando ▁erros ▁graves ▁na ▁presta ção ▁dos ▁serviços , ▁o ▁gesto r ▁assume ▁o ▁risco ▁de ▁in sucesso ▁e ▁de ▁prejuízo ▁ao ▁e r ário , ▁responde ndo ▁sol ida ria mente ▁pelo ▁da no .</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.61</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁a ▁apresentação ▁de ▁presta ção ▁conta s ▁fora ▁de ▁prazo ▁ ajusta do , ▁sem ▁justifica tiva ▁para ▁a ▁falta , ▁não ▁e li de ▁a ▁respectiva ▁o missão .</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2.61</td>\n",
       "      <td>0.07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# tabulation showing the first k texts in top_losses along with their prediction, actual,loss, and probability of actual class.\n",
    "# max_len is the maximum number of tokens displayed. If max_len=None, it will display all tokens.\n",
    "txt_ci.show_top_losses(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = load_data(path, f'{lang}_textlist_class_tcu_jurisp_reduzido_sp15_multifit_v2', bs=bs, num_workers=1)\n",
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, drop_mult=0.3, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_v2', purge=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9708), tensor(0.9707))"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds,targs = learn_c.get_preds(ordered=True)\n",
    "accuracy(preds,targs),f1(preds,targs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas_bwd = load_data(path, f'{lang}_textlist_class_tcu_jurisp_reduzido_sp15_multifit_bwd_v2', bs=bs, num_workers=1, backwards=True)\n",
    "learn_c_bwd = text_classifier_learner(data_clas_bwd, AWD_LSTM, config=config, drop_mult=0.3, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c_bwd.load(f'{lang}clas_tcu_jurisp_reduzido_sp15_multifit_bwd_v2', purge=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9708), tensor(0.9708))"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds_b,targs_b = learn_c_bwd.get_preds(ordered=True)\n",
    "accuracy(preds_b,targs_b),f1(preds_b,targs_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_avg = (preds+preds_b)/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9795), tensor(0.9795))"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(preds_avg,targs_b),f1(preds_avg,targs_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[330   2   0   0]\n",
      " [  0 282   2   0]\n",
      " [  0   1 207   5]\n",
      " [  0   2   9 186]]\n",
      "accuracy global: 0.97953216374269\n",
      "accuracy on class 0: 99.3975903614458\n",
      "accuracy on class 1: 99.29577464788733\n",
      "accuracy on class 2: 97.1830985915493\n",
      "accuracy on class 3: 94.41624365482234\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "predictions = np.argmax(preds_avg, axis = 1)\n",
    "cm = confusion_matrix(np.array(targs_b), np.array(predictions))\n",
    "print(cm)\n",
    "\n",
    "## acc\n",
    "print(f'accuracy global: {(cm[0,0]+cm[1,1]+cm[2,2]+cm[3,3])/(cm.sum())}')\n",
    "\n",
    "# acc neg, acc pos\n",
    "print(f'accuracy on class 0: {cm[0,0]/(cm.sum(1)[0])*100}') \n",
    "print(f'accuracy on class 1: {cm[1,1]/(cm.sum(1)[1])*100}')\n",
    "print(f'accuracy on class 2: {cm[2,2]/(cm.sum(1)[2])*100}')\n",
    "print(f'accuracy on class 3: {cm[3,3]/(cm.sum(1)[3])*100}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}