training-bert/BERT Fine-Tuning Quora Question pairs with PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BERT Fine-Tuning Quora Question pairs with PyTorch\n",
    "This research is based on the toturial [BERT Fine-Tuning Tutorial with PyTorch](https://mccormickml.com/2019/07/22/BERT-fine-tuning/).\n",
    "# Introduction\n",
    "In this research I'd like to use BERT with the huggingface PyTorch library to fine-tune a model which will perform best in question pairs classification. \n",
    "\n",
    "So firstly let's talk about the model and the dataset:\n",
    "# Bert\n",
    "Bidirectional Encoder Representations from Transformers (BERT) was released, and pretrained, in late 2018 by Google (see original model code [here](https://github.com/google-research/bert)) for NLP (Natural Language Processing) tasks. Bert was created originally by [Jacob Devlin](https://www.linkedin.com/in/jacob-devlin-135ab048) with two corpora in pre-training: BookCorpus and English Wikipedia.\n",
    "\n",
    "BERT consists of 12 Transformer Encoding layers (or 24 for large BERT). If you stack Transformer Decoding layers you'll GPT model to generate senetances.\n",
    "\n",
    "You can more information inthe those videos: [Transformer Neural Networks - EXPLAINED! (Attention is all you need)](https://youtu.be/TQQlZhbC5ps) [BERT Neural Network - EXPLAINED!](https://youtu.be/xI0HHN5XKDo)\n",
    "\n",
    "# Quora Question Pairs Dataset\n",
    "[Quora](https://www.quora.com/) is a question-and-answer website where questions are asked, answered, followed, and edited by Internet users, either factually or in the form of opinions. Quora was co-founded by former Facebook employees Adam D'Angelo and Charlie Cheever in June 2009. website was made available to the public for the first time on June 21, 2010. Today the website is available in many languages.\n",
    "\n",
    "Over 100 million people visit Quora every month, so it's no surprise that many people ask similarly worded questions. Multiple questions with the same intent can cause seekers to spend more time finding the best answer to their question, and make writers feel they need to answer multiple versions of the same question.\n",
    "\n",
    "The goal is to predict which of the provided pairs of questions contain two questions with the same meaning. The ground truth is the set of labels that have been supplied by human experts. The dataset itself can be downloaded from kaggle: [here](https://www.kaggle.com/c/quora-question-pairs/).\n",
    "\n",
    "## Data fields\n",
    "* id - the id of a training set question pair\n",
    "* qid1, qid2 - unique ids of each question (only available in train.csv)\n",
    "* question1, question2 - the full text of each question\n",
    "* is_duplicate - the target variable, set to 1 if question1 and question2 have essentially the same meaning, and 0 otherwise.\n",
    "\n",
    "The set itself contains more than 400K pairs of questions.\n",
    "\n",
    "# Fire up pandas and Read the Dataset\n",
    "I'm going to read the first 5K records due to performance issue.\n",
    "\n",
    "**Pay Attention**: in the full set there are some missing values. So before training a model the missing values should be addressed. Here I won't have any."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid1</th>\n",
       "      <th>qid2</th>\n",
       "      <th>question1</th>\n",
       "      <th>question2</th>\n",
       "      <th>is_duplicate</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>What is the step by step guide to invest in sh...</td>\n",
       "      <td>What is the step by step guide to invest in sh...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>What is the story of Kohinoor (Koh-i-Noor) Dia...</td>\n",
       "      <td>What would happen if the Indian government sto...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>How can I increase the speed of my internet co...</td>\n",
       "      <td>How can Internet speed be increased by hacking...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>Why am I mentally very lonely? How can I solve...</td>\n",
       "      <td>Find the remainder when [math]23^{24}[/math] i...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>Which one dissolve in water quikly sugar, salt...</td>\n",
       "      <td>Which fish would survive in salt water?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    qid1  qid2                                          question1  \\\n",
       "id                                                                  \n",
       "0      1     2  What is the step by step guide to invest in sh...   \n",
       "1      3     4  What is the story of Kohinoor (Koh-i-Noor) Dia...   \n",
       "2      5     6  How can I increase the speed of my internet co...   \n",
       "3      7     8  Why am I mentally very lonely? How can I solve...   \n",
       "4      9    10  Which one dissolve in water quikly sugar, salt...   \n",
       "\n",
       "                                            question2  is_duplicate  \n",
       "id                                                                   \n",
       "0   What is the step by step guide to invest in sh...             0  \n",
       "1   What would happen if the Indian government sto...             0  \n",
       "2   How can Internet speed be increased by hacking...             0  \n",
       "3   Find the remainder when [math]23^{24}[/math] i...             0  \n",
       "4             Which fish would survive in salt water?             0  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas\n",
    "\n",
    "\n",
    "\n",
    "questions_dataset = pandas.read_csv(\"train.csv\", index_col=\"id\", nrows=5000)\n",
    "questions_dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what we got:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 5000 entries, 0 to 4999\n",
      "Data columns (total 5 columns):\n",
      " #   Column        Non-Null Count  Dtype \n",
      "---  ------        --------------  ----- \n",
      " 0   qid1          5000 non-null   int64 \n",
      " 1   qid2          5000 non-null   int64 \n",
      " 2   question1     5000 non-null   object\n",
      " 3   question2     5000 non-null   object\n",
      " 4   is_duplicate  5000 non-null   int64 \n",
      "dtypes: int64(3), object(2)\n",
      "memory usage: 234.4+ KB\n"
     ]
    }
   ],
   "source": [
    "questions_dataset.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenizing\n",
    "To feed our text to BERT, it must be split into tokens, and then these tokens must be mapped to their index in the tokenizer vocabulary. \n",
    "\n",
    "The tokenization must be performed by the tokenizer included with BERT–the below cell will download this for us. We’ll be using the “uncased” version here. This Tokenizer is build upon the WordPiece tokenizer. you can read more about it [here](https://huggingface.co/transformers/model_doc/bert.html#berttokenizerfast).\n",
    "\n",
    "Loading BERT tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertTokenizerFast\n",
    "\n",
    "\n",
    "\n",
    "tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased', do_lower_case=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split a sentence into tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original: What is the step by step guide to invest in share market in india?\n",
      "Tokenized: ['what', 'is', 'the', 'step', 'by', 'step', 'guide', 'to', 'invest', 'in', 'share', 'market', 'in', 'india', '?']\n",
      "Token IDs: [2054, 2003, 1996, 3357, 2011, 3357, 5009, 2000, 15697, 1999, 3745, 3006, 1999, 2634, 1029]\n"
     ]
    }
   ],
   "source": [
    "print(f\"Original: {questions_dataset['question1'][0]}\")\n",
    "print(f\"Tokenized: {tokenizer.tokenize(questions_dataset['question1'][0])}\")\n",
    "print(f\"Token IDs: {tokenizer.convert_tokens_to_ids(tokenizer.tokenize(questions_dataset['question1'][0]))}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can combine ``tokenize`` and ``convert_tokens_to_ids`` with the method ``encode``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original: What is the step by step guide to invest in share market?\n",
      "Tokenized: [101, 2054, 2003, 1996, 3357, 2011, 3357, 5009, 2000, 15697, 1999, 3745, 3006, 1029, 102]\n"
     ]
    }
   ],
   "source": [
    "print(f\"Original: {questions_dataset['question2'][0]}\")\n",
    "print(f\"Tokenized: {tokenizer.encode(questions_dataset['question2'][0])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see we got two extra tokens ``101`` and ``102``. Let's see what they are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[CLS] what is the step by step guide to invest in share market? [SEP]'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode([101, 2054, 2003, 1996, 3357, 2011, 3357, 5009, 2000, 15697, 1999, 3745, 3006, 1029, 102])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Special Tokens\n",
    "``[SEP]`` - At the end of every sentence, we need to append the special ``[SEP]`` token.\n",
    "\n",
    "This token is an artifact of two-sentence tasks, where BERT is given two separate sentences and asked to determine something (remeber BERT was trained intially to perdict question-answering task).\n",
    "\n",
    "``[CLS]`` - For classification tasks, we must prepend the special ``[CLS]`` token to the beginning of every sentence.\n",
    "\n",
    "This token has special significance. BERT consists of 12 Transformer layers. Each transformer takes in a list of token embeddings, and produces the same number of embeddings on the output.\n",
    "\n",
    "So now let's see how to encode the two questions together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[CLS] what is the step by step guide to invest in share market in india? [SEP] what is the step by step guide to invest in share market? [SEP]'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_pair = tokenizer.encode(questions_dataset['question1'][0], questions_dataset['question2'][0])\n",
    "tokenizer.decode(encoded_pair)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence Length\n",
    "\n",
    "The sentences in our dataset obviously have varying lengths, so how does BERT handle this?\n",
    "\n",
    "BERT has two constraints:\n",
    "\n",
    "1. All sentences must be padded or truncated to a single, fixed length.\n",
    "2. The maximum sentence length is 512 tokens.\n",
    "Padding is done with a special ``[PAD]`` token, which is at index 0 in the BERT vocabulary.\n",
    "\n",
    "The maximum length does impact training and evaluation speed.\n",
    "\n",
    "Before we are ready to encode our text, though, we need to decide on a maximum sentence length for padding / truncating to.\n",
    "\n",
    "Let's find the maximum length of question pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\transformers\\lib\\site-packages\\tqdm\\std.py:668: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version\n",
      "  from pandas import Panel\n",
      "100%|███████████████████████████████████████████████████████████████████████████| 5000/5000 [00:00<00:00, 12987.09it/s]\n",
      "100%|███████████████████████████████████████████████████████████████████████████| 5000/5000 [00:00<00:00, 12531.32it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "310"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "\n",
    "\n",
    "tqdm.pandas()\n",
    "questions_dataset[\"question1_length\"] = questions_dataset[\"question1\"].progress_apply(lambda question: \n",
    "                                                                                      len(tokenizer.tokenize(question)))\n",
    "questions_dataset[\"question2_length\"] = questions_dataset[\"question2\"].progress_apply(lambda question: \n",
    "                                                                                      len(tokenizer.tokenize(question)))\n",
    "questions_dataset[\"joint_length\"] = questions_dataset[\"question1_length\"] + questions_dataset[\"question2_length\"]\n",
    "questions_dataset[\"joint_length\"].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will choose max length of 310.\n",
    "\n",
    "# Training & Validation Split\n",
    "\n",
    "I'm going to spit the set into 3200 records for train (64%), 800 records for validation (16%) and 1000 records for test (20%).\n",
    "\n",
    "Let's Create train, validation and test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question1</th>\n",
       "      <th>question2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3444</th>\n",
       "      <td>Is there a way to get a user's friends' email ...</td>\n",
       "      <td>Is there any way to get a user's email address...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2063</th>\n",
       "      <td>How can I get rid of cellulite on my stomach?</td>\n",
       "      <td>I am in good shape but have a trouble spot of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3714</th>\n",
       "      <td>Among SC, ST, NC and OBC which is the lowest c...</td>\n",
       "      <td>Is the 'Tili' caste in West Bengal SC/ST/OBC?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2671</th>\n",
       "      <td>Are you tolerant if you tolerate the intolerant?</td>\n",
       "      <td>Are you tolerant?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2154</th>\n",
       "      <td>Are food allergies more common in USA?</td>\n",
       "      <td>Why are food allergies so common in USA?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              question1  \\\n",
       "id                                                        \n",
       "3444  Is there a way to get a user's friends' email ...   \n",
       "2063      How can I get rid of cellulite on my stomach?   \n",
       "3714  Among SC, ST, NC and OBC which is the lowest c...   \n",
       "2671   Are you tolerant if you tolerate the intolerant?   \n",
       "2154             Are food allergies more common in USA?   \n",
       "\n",
       "                                              question2  \n",
       "id                                                       \n",
       "3444  Is there any way to get a user's email address...  \n",
       "2063  I am in good shape but have a trouble spot of ...  \n",
       "3714      Is the 'Tili' caste in West Bengal SC/ST/OBC?  \n",
       "2671                                  Are you tolerant?  \n",
       "2154           Why are food allergies so common in USA?  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(questions_dataset[[\"question1\", \"question2\"]], \n",
    "                                                    questions_dataset[\"is_duplicate\"], test_size=0.2, random_state=42)\n",
    "X_train, X_validation, y_train, y_validation = train_test_split(X_train, y_train, test_size=0.2, random_state=42)\n",
    "\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id\n",
       "3444    0\n",
       "2063    1\n",
       "3714    0\n",
       "2671    0\n",
       "2154    0\n",
       "Name: is_duplicate, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenize Dataset and Create Dataloader\n",
    "\n",
    "Now we’re ready to perform the real tokenization.\n",
    "\n",
    "The tokenizer.encode_plus function combines multiple steps for us:\n",
    "\n",
    "1. Split the sentence into tokens.\n",
    "2. Add the special ``[CLS]`` and ``[SEP]`` tokens.\n",
    "3. Map the tokens to their IDs.\n",
    "4. Pad or truncate all sentences to the same length.\n",
    "5. Create the attention masks which explicitly differentiate real tokens from ``[PAD]`` tokens.\n",
    "\n",
    "Documentation is [here](https://huggingface.co/transformers/main_classes/tokenizer.html?highlight=encode_plus#transformers.PreTrainedTokenizer.encode_plus)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[  101,  2003,  2045,  1037,  2126,  2000,  2131,  1037,  5310,  1005,\n",
       "          1055,  2814,  1005, 10373,  4769,  2083,  9130, 17928,  2005, 25718,\n",
       "          1029,   102,  2003,  2045,  2151,  2126,  2000,  2131,  1037,  5310,\n",
       "          1005,  1055, 10373,  4769,  2083, 10474,  1051,  4887,  2705, 17928,\n",
       "          1029,   102,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,\n",
       "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_length = 310\n",
    "tokenizer.encode_plus(X_train.iloc[0][\"question1\"], X_train.iloc[0][\"question2\"], max_length=max_length, \n",
    "                      pad_to_max_length=True, return_attention_mask=True, return_tensors='pt', truncation=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what we got:\n",
    "\n",
    "1. input_ids - Token ids padded with 0 at the end.\n",
    "2. token_type_ids - This array indicates bert what is the first sentence and what is the seconds. In case of classification of only one sentance this array is redundant.\n",
    "3. attention_mask - The “Attention Mask” is simply an array of 1s and 0s indicating which tokens are padding and which aren’t. This mask tells the “Self-Attention” mechanism in BERT not to incorporate these ``[PAD]`` tokens into its interpretation of the sentence.\n",
    "\n",
    "All the array are in the size of ``max_length``.\n",
    "\n",
    "Now let's create our datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "from tqdm import tqdm\n",
    "from torch.utils.data import TensorDataset\n",
    "\n",
    "\n",
    "\n",
    "def convert_to_dataset_torch(data: pandas.DataFrame, labels: pandas.Series) -> TensorDataset:\n",
    "    input_ids = []\n",
    "    attention_masks = []\n",
    "    token_type_ids = []\n",
    "    for _, row in tqdm(data.iterrows(), total=data.shape[0]):\n",
    "        encoded_dict = tokenizer.encode_plus(row[\"question1\"], row[\"question2\"], max_length=max_length, pad_to_max_length=True, \n",
    "                      return_attention_mask=True, return_tensors='pt', truncation=True)\n",
    "        # Add the encoded sentences to the list.\n",
    "        input_ids.append(encoded_dict['input_ids'])\n",
    "        token_type_ids.append(encoded_dict[\"token_type_ids\"])\n",
    "        # And its attention mask (simply differentiates padding from non-padding).\n",
    "        attention_masks.append(encoded_dict['attention_mask'])\n",
    "    \n",
    "    # Convert the lists into tensors.\n",
    "    input_ids = torch.cat(input_ids, dim=0)\n",
    "    token_type_ids = torch.cat(token_type_ids, dim=0)\n",
    "    attention_masks = torch.cat(attention_masks, dim=0)\n",
    "    labels = torch.tensor(labels.values)\n",
    "    \n",
    "    return TensorDataset(input_ids, attention_masks, token_type_ids, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████| 3200/3200 [00:02<00:00, 1588.09it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████| 800/800 [00:00<00:00, 1754.39it/s]\n"
     ]
    }
   ],
   "source": [
    "train = convert_to_dataset_torch(X_train, y_train)\n",
    "validation = convert_to_dataset_torch(X_validation, y_validation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ll also create an iterator for our dataset using the torch DataLoader class. This helps save on memory during training because, unlike a for loop, with an iterator the entire dataset does not need to be loaded into memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import multiprocessing\n",
    "\n",
    "from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n",
    "\n",
    "# The DataLoader needs to know our batch size for training, so we specify it \n",
    "# here.\n",
    "batch_size = 32\n",
    "\n",
    "core_number = multiprocessing.cpu_count()\n",
    "\n",
    "# Create the DataLoaders for our training and validation sets.\n",
    "# We'll take training samples in random order. \n",
    "train_dataloader = DataLoader(\n",
    "            train,  # The training samples.\n",
    "            sampler = RandomSampler(train), # Select batches randomly\n",
    "            batch_size = batch_size, # Trains with this batch size.\n",
    "            num_workers = core_number\n",
    "        )\n",
    "\n",
    "# For validation the order doesn't matter, so we'll just read them sequentially.\n",
    "validation_dataloader = DataLoader(\n",
    "            validation, # The validation samples.\n",
    "            sampler = SequentialSampler(validation), # Pull out batches sequentially.\n",
    "            batch_size = batch_size, # Evaluate with this batch size.\n",
    "            num_workers = core_number\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Trainig the Classification Model\n",
    "Now that our input data is properly formatted, it’s time to fine tune the BERT model.\n",
    "## BertForSequenceClassification\n",
    "For this task, we first want to modify the pre-trained BERT model to give outputs for classification, and then we want to continue training the model on our dataset until that the entire model, end-to-end, is well-suited for our task.\n",
    "\n",
    "Thankfully, the huggingface pytorch implementation includes a set of interfaces designed for a variety of NLP tasks. Though these interfaces are all built on top of a trained BERT model, each has different top layers and output types designed to accomodate their specific NLP task.\n",
    "\n",
    "We’ll be using [BertForSequenceClassification](https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification). This is the normal BERT model with an added single linear layer on top for classification that we will use as a sentence classifier. As we feed input data, the entire pre-trained BERT model and the additional untrained classification layer is trained on our specific task.\n",
    "\n",
    "OK, let’s load BERT! There are a few different pre-trained BERT models available. “bert-base-uncased” means the version that has only lowercase letters (“uncased”) and is the smaller version of the two (“base” vs “large”)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import BertForSequenceClassification\n",
    "\n",
    "\n",
    "\n",
    "# Load BertForSequenceClassification, the pretrained BERT model with a single \n",
    "# linear classification layer on top. \n",
    "bert_model = BertForSequenceClassification.from_pretrained(\n",
    "    \"bert-base-uncased\", # Use the 12-layer BERT model, with an uncased vocab.\n",
    "    num_labels=2, # The number of output labels--2 for binary classification.\n",
    "                    # You can increase this for multi-class tasks.   \n",
    "    output_attentions=False, # Whether the model returns attentions weights.\n",
    "    output_hidden_states=False, # Whether the model returns all hidden-states.\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizer & Learning Rate Scheduler\n",
    "Now that we have our model loaded we need to grab the training hyperparameters from within the stored model.\n",
    "\n",
    "For the purposes of fine-tuning, the authors recommend choosing from the following values (from Appendix A.3 of the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf)):\n",
    "\n",
    "* Batch size: 16, 32\n",
    "* Learning rate (Adam): 5e-5, 3e-5, 2e-5\n",
    "* Number of epochs: 2, 3, 4\n",
    "\n",
    "I chose:\n",
    "* Batch size: 32\n",
    "* Learning rate: 2e-5\n",
    "* Number of epochs: 2\n",
    "\n",
    "All due to hardware performance issues.\n",
    "\n",
    "Let's create our optimizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AdamW\n",
    "\n",
    "\n",
    "\n",
    "# Note: AdamW is a class from the huggingface library (as opposed to pytorch)\n",
    "adamw_optimizer = AdamW(bert_model.parameters(),\n",
    "                  lr = 2e-5, # args.learning_rate - default is 5e-5, our notebook had 2e-5\n",
    "                  eps = 1e-8 # args.adam_epsilon  - default is 1e-8.\n",
    "                )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The epsilon parameter eps = 1e-8 is “a very small number to prevent any division by zero in the implementation”.\n",
    "\n",
    "Let's create the learning rate scheduler:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import get_linear_schedule_with_warmup\n",
    "\n",
    "# Number of training epochs. The BERT authors recommend between 2 and 4. \n",
    "epochs = 2\n",
    "\n",
    "# Total number of training steps is [number of batches] x [number of epochs]. \n",
    "# (Note that this is not the same as the number of training samples).\n",
    "total_steps = len(train_dataloader) * epochs\n",
    "\n",
    "# Create the learning rate scheduler.\n",
    "scheduler = get_linear_schedule_with_warmup(adamw_optimizer, \n",
    "                                            num_warmup_steps = 0, \n",
    "                                            num_training_steps = total_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Loop\n",
    "Helper function for formatting elapsed times as ``hh:mm:ss``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import datetime\n",
    "\n",
    "def format_time(elapsed):\n",
    "    '''\n",
    "    Takes a time in seconds and returns a string hh:mm:ss\n",
    "    '''\n",
    "    # Round to the nearest second.\n",
    "    elapsed_rounded = int(round((elapsed)))\n",
    "    \n",
    "    # Format as hh:mm:ss\n",
    "    return str(datetime.timedelta(seconds=elapsed_rounded))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method is to train one batch over our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_batch(dataloader, model, optimizer, epoch):\n",
    "    total_train_loss = 0\n",
    "    \n",
    "    for batch in tqdm(dataloader, desc=f\"Training epoch:{epoch}\", unit=\"batch\"):\n",
    "        # Unpack batch from dataloader.\n",
    "        input_ids, attention_masks, token_type_ids, labels = batch\n",
    "        \n",
    "        # clear any previously calculated gradients before performing a backward pass.\n",
    "        model.zero_grad()\n",
    "        \n",
    "        # Perform a forward pass (evaluate the model on this training batch).\n",
    "        loss, _ = model(input_ids, \n",
    "                             token_type_ids=token_type_ids, \n",
    "                             attention_mask=attention_masks, \n",
    "                             labels=labels)\n",
    "        total_train_loss += loss.item()\n",
    "\n",
    "        # Perform a backward pass to calculate the gradients.\n",
    "        loss.backward()\n",
    "\n",
    "        # Clip the norm of the gradients to 1.0.\n",
    "        # This is to help prevent the \"exploding gradients\" problem.\n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
    "\n",
    "        # Update parameters and take a step using the computed gradient.\n",
    "        optimizer.step()\n",
    "\n",
    "        # Update the learning rate.\n",
    "        scheduler.step()\n",
    "        \n",
    "    return total_train_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method is evalute one batch over our model. We're going to use scikit learn [accuracy_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "def eval_batch(dataloader, model, metric=accuracy_score):\n",
    "    total_eval_accuracy = 0\n",
    "    total_eval_loss = 0\n",
    "    predictions , predicted_labels = [], []\n",
    "    \n",
    "    for batch in tqdm(dataloader, desc=\"Evaluating\", unit=\"batch\"):\n",
    "        # Unpack batch from dataloader.\n",
    "        input_ids, attention_masks, token_type_ids, labels = batch\n",
    "        \n",
    "        # Tell pytorch not to bother with constructing the compute graph during\n",
    "        # the forward pass, since this is only needed for backprop (training).\n",
    "        with torch.no_grad():\n",
    "            # Forward pass, calculate logit predictions.\n",
    "            loss, logits = model(input_ids, \n",
    "                                   token_type_ids=token_type_ids, \n",
    "                                   attention_mask=attention_masks,\n",
    "                                   labels=labels)\n",
    "        total_eval_loss += loss.item()\n",
    "        \n",
    "        # Calculate the accuracy for this batch of validation sentences, and\n",
    "        # accumulate it over all batches.\n",
    "        y_pred = numpy.argmax(logits.detach().numpy(), axis=1).flatten()\n",
    "        total_eval_accuracy += metric(labels, y_pred)\n",
    "        \n",
    "        predictions.extend(logits.detach().numpy().tolist())\n",
    "        predicted_labels.extend(y_pred.tolist())\n",
    "    \n",
    "    return total_eval_accuracy, total_eval_loss, predictions ,predicted_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Main loop method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "# Set the seed value all over the place to make this reproducible.\n",
    "seed_val = 42\n",
    "\n",
    "random.seed(seed_val)\n",
    "numpy.random.seed(seed_val)\n",
    "torch.manual_seed(seed_val)\n",
    "\n",
    "\n",
    "def train(train_dataloader, validation_dataloader, model, optimizer, epochs):\n",
    "    # We'll store a number of quantities such as training and validation loss, \n",
    "    # validation accuracy, and timings.\n",
    "    training_stats = []\n",
    "    \n",
    "    # Measure the total training time for the whole run.\n",
    "    total_t0 = time.time()\n",
    "    \n",
    "    for epoch in range(0, epochs):\n",
    "        \n",
    "        # Measure how long the training epoch takes.\n",
    "        t0 = time.time()\n",
    "        \n",
    "        # Reset the total loss for this epoch.\n",
    "        total_train_loss = 0\n",
    "        \n",
    "        # Put the model into training mode. \n",
    "        model.train()\n",
    "        \n",
    "        total_train_loss = fit_batch(train_dataloader, model, optimizer, epoch)\n",
    "        \n",
    "        # Calculate the average loss over all of the batches.\n",
    "        avg_train_loss = total_train_loss / len(train_dataloader)\n",
    "        \n",
    "        # Measure how long this epoch took.\n",
    "        training_time = format_time(time.time() - t0)\n",
    "        \n",
    "        t0 = time.time()\n",
    "        \n",
    "        # Put the model in evaluation mode--the dropout layers behave differently\n",
    "        # during evaluation.\n",
    "        model.eval()\n",
    "        \n",
    "        total_eval_accuracy, total_eval_loss, _, _ = eval_batch(validation_dataloader, model)\n",
    "        \n",
    "        # Report the final accuracy for this validation run.\n",
    "        avg_val_accuracy = total_eval_accuracy / len(validation_dataloader)\n",
    "        \n",
    "        print(f\"  Accuracy: {avg_val_accuracy}\")\n",
    "    \n",
    "        # Calculate the average loss over all of the batches.\n",
    "        avg_val_loss = total_eval_loss / len(validation_dataloader)\n",
    "    \n",
    "        # Measure how long the validation run took.\n",
    "        validation_time = format_time(time.time() - t0)\n",
    "    \n",
    "        print(f\"  Validation Loss: {avg_val_loss}\")\n",
    "    \n",
    "        # Record all statistics from this epoch.\n",
    "        training_stats.append(\n",
    "            {\n",
    "                'epoch': epoch,\n",
    "                'Training Loss': avg_train_loss,\n",
    "                'Valid. Loss': avg_val_loss,\n",
    "                'Valid. Accur.': avg_val_accuracy,\n",
    "                'Training Time': training_time,\n",
    "                'Validation Time': validation_time\n",
    "            }\n",
    "        )\n",
    "        \n",
    "\n",
    "    print(\"\")\n",
    "    print(\"Training complete!\")\n",
    "\n",
    "    print(f\"Total training took {format_time(time.time()-total_t0)}\")\n",
    "    return training_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’re ready to kick off the training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training epoch:0: 100%|████████████████████████████████████████████████████████| 100/100 [2:56:55<00:00, 106.16s/batch]\n",
      "Evaluating: 100%|███████████████████████████████████████████████████████████████████| 25/25 [14:06<00:00, 33.84s/batch]\n",
      "Training epoch:1:   0%|                                                                     | 0/100 [00:00<?, ?batch/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Accuracy: 0.69875\n",
      "  Validation Loss: 0.516389410495758\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training epoch:1: 100%|████████████████████████████████████████████████████████| 100/100 [2:55:47<00:00, 105.48s/batch]\n",
      "Evaluating: 100%|███████████████████████████████████████████████████████████████████| 25/25 [14:03<00:00, 33.76s/batch]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Accuracy: 0.755\n",
      "  Validation Loss: 0.46126050233840943\n",
      "\n",
      "Training complete!\n",
      "Total training took 6:20:54\n"
     ]
    }
   ],
   "source": [
    "training_stats = train(train_dataloader, validation_dataloader, bert_model, adamw_optimizer, epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s view the summary of the training process:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Valid. Loss</th>\n",
       "      <th>Valid. Accur.</th>\n",
       "      <th>Training Time</th>\n",
       "      <th>Validation Time</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>epoch</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.567917</td>\n",
       "      <td>0.516389</td>\n",
       "      <td>0.69875</td>\n",
       "      <td>2:56:56</td>\n",
       "      <td>0:14:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.424198</td>\n",
       "      <td>0.461261</td>\n",
       "      <td>0.75500</td>\n",
       "      <td>2:55:48</td>\n",
       "      <td>0:14:04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Training Loss  Valid. Loss  Valid. Accur. Training Time Validation Time\n",
       "epoch                                                                         \n",
       "0           0.567917     0.516389        0.69875       2:56:56         0:14:06\n",
       "1           0.424198     0.461261        0.75500       2:55:48         0:14:04"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stats = pandas.DataFrame(training_stats).set_index('epoch')\n",
    "df_stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xUZfbH8c9JCCX0JkiABAVEasAYuomiuyi2VRQQC6IiYu+FtaFY0UWsi2L5KS66oq4Fl1U0dIHQmwUxgdBrAKlJzu+PZzIkcdJgJjflvF+vvJy5c+/cM1Hnm+c+954rqooxxhiTV5jXBRhjjCmdLCCMMcYEZAFhjDEmIAsIY4wxAVlAGGOMCcgCwhhjTEAWEMZTIvKNiFwT7HVLMxGJEREVkUq+5/l+rrzrHsO+HhKRt46nXlNxWUCYYhORfTl+skTkQI7ng4vzXqp6rqq+F+x1i0tE6onIlyKSLiIbReS+Qtb/SUSGBlh+u4gkF2ffwfpcIpIoIml53vspVb3+eN87wL6GiMisYL+vKV2O6a8SU7Gpao3sxyKSAlyvqt/lXU9EKqlqRknWdhzuBaoCJwJVgLaFrP8ecDXwdp7lV/leM6bMsxGECZrsv2BF5H4R2Qy8IyJ1ReQrEdkmIrt8j5vm2CZJRK73PR4iIrNEZIxv3d9F5NxjXLeFiMwQkb0i8p2IvCoiHxRQfgawVVX3q+ouVZ1dyMd9H+glItE59nkq0BH4l4j0E5HFIrJHRNaLyGMF/N5yfq5w32faLiJrgX551r1WRFb7PtdaEbnRt7w68A3QJMdoromIPJbzc4vIhSKyUkR2+/Z7ao7XUkTkHhFZ5htJfSQiVQv5PQT6PD1EZIHvPRaISI8crw3x1b3X9+9ssG95SxGZ7ttmu4h8VNz9muCzgDDB1hioB0QDw3D/jb3je94cOAC8UsD2XYGfgQbAc8AEEZFjWPdDYD5QH3gM95d9QeYDgwIdNgpEVdOAH/K879XAFFXdDvzhe14H9yV/k4hcXIS3vgE4H+gMxAH987y+1fd6LeBa4B8i0kVV/wDOBTaqag3fz8acG4pIa+BfwB1AQ2AK8KWIVM6x2uVAX6AFLuyGFKHmnPuoB3wNjMP97l8EvhaR+r4QGwecq6o1gR7AEt+mTwD/A+oCTYGXi7NfExoWECbYsoBHVfWQqh5Q1R2qOtn3l/leYDSQUMD2qar6pqpm4g7VnAg0Ks66ItIcOB14RFUPq+os4Iv8digiLYHxQCLwgIhc61teRUQOi0jtfDZ9D19AiEgYMNi3DFVNUtXlqpqlqstwX8wFfe5slwNjVXW9qu4Ens75oqp+raq/qTMd96XauwjvCzAA+FpVv1XVI8AYoBruizrbOFXd6Nv3l0BsEd87Wz/gV1V9X1UzVPVfwE/ABb7Xs4D2IlJNVTep6krf8iO4PyKaqOpB378z4zELCBNs21T1YPYTEYkUkX+KSKqI7AFmAHVEJDyf7TdnP1DV/b6HNYq5bhNgZ45lAOsLqPk64FtVnQH8FXjCFxLdgMWqmp7Pdp8CJ4pIN1y4ROL+ekZEuorID75Da+nAcNxIpzBN8tSamvNFETlXRH4UkZ0ishs4r4jvm/3e/vdT1SzfvqJyrLM5x+P95P+7L9I+fFKBKN8oZwDud7FJRL4WkTa+de4DBJjvOwRWpJGcCS0LCBNsedsD3w2cAnRV1VrAGb7l+R02CoZNQD0RicyxrFkB61fCzUGgqr/jDrE8B7wFjMpvI18AfYI7lHQVMElVD/te/hA3ammmqrWBNyjaZ96Up9bm2Q9EpAowGfeXfyNVrYM7TJT9voW1Zt6I+ys9+/3Et68NRairqHLtw6d59j5UdaqqnoMb7f0EvOlbvllVb1DVJsCNwGu+kZ3xkAWECbWauHmH3b7j04+GeoeqmgokA4+JSGUR6c7RQxyBfAoMEJGLfSObPcBS4GQK/9J9D/dX8aXkPnupJm4Uc1BE4oErilj+x8BtItJUROoCD+R4rTLuDKttQIZvUv4vOV7fAtQv4JDYx0A/EekjIhG48D4EzClibXmJiFTN+YMLrNYicoWIVBKRAbgzwr4SkUa+SfLqvv3uAzJ9b3SZHD15YRfu9555jHWZILGAMKE2FnecezvwI/DfEtrvYKA7sAN4EvgI96X0J6o6F/cF/ijuy2kq7ovuUtwZSZ0L2M8MIB3YoKoLciwfAYwSkb3AI7gv56J407f/pcAiXHhl17kXuM33Xrt8NX+R4/WfcHMda31nKTXJ8zl/Bq7ETQBvx4XmBTlGPcXVAxf+OX/ScZPod+N+9/cB5/sm7sN8yzcCO3FzMiN873U6ME9E9vk+0+2+0ZzxkNgNg0xF4Dtt8idVDfkIxpjywkYQplwSkdNF5GQRCRORvsBFwOde12VMWWJXUpvyqjHu8Ex9IA24SVUXe1uSMWWLHWIyxhgTkB1iMsYYE1C5OsTUoEEDjYmJ8boMY4wpMxYuXLhdVRsGeq1cBURMTAzJycXqtGyMMRWaiOS98t3PDjEZY4wJyALCGGNMQBYQxhhjAipXcxDGmPLjyJEjpKWlcfDgwcJXNoWqWrUqTZs2JSIiosjbWEAYY0qltLQ0atasSUxMDPnfM8oUhaqyY8cO0tLSaNGiRZG3q/CHmCZOhJgYCAtz/5w40euKjDEABw8epH79+hYOQSAi1K9fv9ijsQo9gpg4EYYNg/2+28qkprrnAIMHe1eXMcaxcAieY/ldVugRxMiRR8Mh2/79brkxxlR0FTog1q0r3nJjTMWxY8cOYmNjiY2NpXHjxkRFRfmfHz5c8C00kpOTue222wrdR48ePQpdx0sVOiCaNw+8/IQTSrYOY8zxC/Z8Yv369VmyZAlLlixh+PDh3Hnnnf7nlStXJiMjI99t4+LiGDduXKH7mDPnWG/mVzIqdECMHg2RkbmXicCWLTB8OKTnd6t6Y0ypkj2fmJoKqkfnE4N90smQIUO46667OPPMM7n//vuZP38+PXr0oHPnzvTo0YOff/4ZgKSkJM4//3wAHnvsMYYOHUpiYiInnXRSruCoUaOGf/3ExET69+9PmzZtGDx4MNmdtqdMmUKbNm3o1asXt912m/99S0KFnqTOnogeOdIdVmreHB59FFauhH/8A776Ct54A0rw34cxJoA77oAlS/J//ccf4VCeG8ru3w/XXQdvvhl4m9hYGDu2+LX88ssvfPfdd4SHh7Nnzx5mzJhBpUqV+O6773jooYeYPHnyn7b56aef+OGHH9i7dy+nnHIKN91005+uR1i8eDErV66kSZMm9OzZk9mzZxMXF8eNN97IjBkzaNGiBYMGDSp+wcehQgcEuJAIdMbS5Ze7/7guuAAGDYKXXoKGAfsdGmO8ljccClt+PC677DLCw8MBSE9P55prruHXX39FRDhy5EjAbfr160eVKlWoUqUKJ5xwAlu2bKFp06a51omPj/cvi42NJSUlhRo1anDSSSf5r10YNGgQ48ePD/6HykeFD4j8xMfDwoXw9NPuUNS338K4cTBwoDsMZYwpOYX9pR8T4w4r5RUdDUlJwa2levXq/scPP/wwZ555Jp999hkpKSkkJiYG3KZKlSr+x+Hh4QHnLwKt4/UN3Sr0HERhKld2h5wWLYKTToIrroALL4S0NK8rM8bkFGg+MTLSLQ+l9PR0oqKiAHj33XeD/v5t2rRh7dq1pKSkAPDRRx8FfR8FsYAogvbtYc4cePFFmDYN2rWD8eMhK8vryowx4A4Tjx/vRgwi7p/jx4f+gtf77ruPBx98kJ49e5KZmRn0969WrRqvvfYaffv2pVevXjRq1IjatWsHfT/5Cek9qUWkL/ASEA68parP5Hk9EfgP8Ltv0aeqOsr3Wh3gLaA9oMBQVZ1b0P7i4uI01DcM+u03uOEG+OEHSEx0E2AtW4Z0l8ZUSKtXr+bUU0/1ugzP7du3jxo1aqCq3HzzzbRq1Yo777zzmN4r0O9URBaqalyg9UM2ghCRcOBV4FygLTBIRNoGWHWmqsb6fkblWP4S8F9VbQN0AlaHqtbiOPlkN4p480136KlDBxgzBgo4JdoYY47Zm2++SWxsLO3atSM9PZ0bb7yxxPYdykNM8cAaVV2rqoeBScBFRdlQRGoBZwATAFT1sKruDlmlxSQC118Pq1bBOefAvfdCjx6wfLnXlRljypvsC/RWrVrFxIkTicw72RJCoQyIKGB9judpvmV5dReRpSLyjYi08y07CdgGvCMii0XkLRGpHmBbRGSYiCSLSPK2bduC+gEKExUF//kPTJoEKSnQpYub1A7FqXXGGFPSQhkQgU4GzTvhsQiIVtVOwMvA577llYAuwOuq2hn4A3gg0E5UdbyqxqlqXEMPLlQQgQED3GhiwAAYNQpOOw3mzSvxUowxJqhCGRBpQLMcz5sCG3OuoKp7VHWf7/EUIEJEGvi2TVPV7K/ZT3CBUWo1aAAffOCuvk5Ph+7d4a674I8/vK7MGGOOTSgDYgHQSkRaiEhlYCDwRc4VRKSx+JqUi0i8r54dqroZWC8ip/hW7QOsCmGtQdOvn2vVMXy4a9fRoYOb1DbGmLImZAGhqhnALcBU3BlIH6vqShEZLiLDfav1B1aIyFJgHDBQj553eyswUUSWAbHAU6GqNdhq1YLXXnNXcIaHw9lnu1Njd5eaaXZjTGESExOZOnVqrmVjx45lxIgR+a6ffZr9eeedx+4A/8M/9thjjBkzpsD9fv7556xadfTv4UceeYTvvvuuuOUHRUgvlFPVKaraWlVPVtXRvmVvqOobvsevqGo7Ve2kqt1UdU6ObZf45hY6qurFqrorlLWGQkICLFsG990Hb78Nbdu6SW1jTPBNXD6RmLExhD0eRszYGCYuP75WroMGDWLSpEm5lk2aNKlIDfOmTJlCnTp1jmm/eQNi1KhRnH322cf0XsfLrqQOsWrV4Nln3aR1gwZw8cWun9PWrV5XZkz5MXH5RIZ9OYzU9FQUJTU9lWFfDjuukOjfvz9fffUVh3ynJaakpLBx40Y+/PBD4uLiaNeuHY8++mjAbWNiYti+fTsAo0eP5pRTTuHss8/2twMHd33D6aefTqdOnbj00kvZv38/c+bM4YsvvuDee+8lNjaW3377jSFDhvDJJ58AMG3aNDp37kyHDh0YOnSov7aYmBgeffRRunTpQocOHfjpp5+O+XPnZM36SkhcHCQnw3PPwRNPuOZ/L73kWgFY8z9jCnbHf+9gyeb8+33/mPYjhzJzn1++/8h+rvvPdby5MHC/79jGsYztm38XwPr16xMfH89///tfLrroIiZNmsSAAQN48MEHqVevHpmZmfTp04dly5bRsWPHgO+xcOFCJk2axOLFi8nIyKBLly6cdtppAFxyySXccMMNAPz9739nwoQJ3HrrrVx44YWcf/759O/fP9d7HTx4kCFDhjBt2jRat27N1Vdfzeuvv84dd9wBQIMGDVi0aBGvvfYaY8aM4a233sr3sxWVjSBKUOXK8Pe/w+LF0Lo1XHWVu9fE+vWFb2uMyV/ecChseVHlPMyUfXjp448/pkuXLnTu3JmVK1fmOhyU18yZM/nb3/5GZGQktWrV4sILL/S/tmLFCnr37k2HDh2YOHEiK1euLLCWn3/+mRYtWtC6dWsArrnmGmbMmOF//ZJLLgHgtNNO8zf3O142gvBA27Ywaxa88go89JB7/txzcOON7naJxpjcCvpLHyBmbAyp6X/u9x1dO5qkIUnHvN+LL76Yu+66i0WLFnHgwAHq1q3LmDFjWLBgAXXr1mXIkCEcPHiwwPeQfA4RDBkyhM8//5xOnTrx7rvvklRIX/LC+uZltwvPr534sbCvI4+Eh8Ptt8OKFdCtG4wY4Zr//fKL15UZU/aM7jOayIjcLSgiIyIZ3ef4+n3XqFGDxMREhg4dyqBBg9izZw/Vq1endu3abNmyhW+++abA7c844ww+++wzDhw4wN69e/nyyy/9r+3du5cTTzyRI0eOMDHHvVFr1qzJ3r17//Rebdq0ISUlhTVr1gDw/vvvk5CQcFyfrzAWEB5r0QL+9z+YMMGd8dSpkxtNWPM/Y4pucIfBjL9gPNG1oxGE6NrRjL9gPIM7HH+/70GDBrF06VIGDhxIp06d6Ny5M+3atWPo0KH07NmzwG27dOnCgAEDiI2N5dJLL6V3797+15544gm6du3KOeecQ5s2bfzLBw4cyPPPP0/nzp357bff/MurVq3KO++8w2WXXUaHDh0ICwtj+PDhhFJI232XtJJo9x1KGzfCzTfD55+7vk5vv+0Cw5iKyNp9B1+pafdtiq9JE/j0U/j3v91d6+Li4OGHrfmfMcYbFhCljAj07++a/11xBTz5JHTu7O5oZ4wxJckCopSqXx/eew+++cY1/OvVy01q79vndWXGlJzydAjca8fyu7SAKOX69nVnOo0YAePGueZ/337rdVXGhF7VqlXZsWOHhUQQqCo7duygatWqxdrOJqnLkJkz3Z3sfvkFrr0WXngB6tb1uipjQuPIkSOkpaUVep2BKZqqVavStGlTIiIici0vaJLaLpQrQ3r3hqVL3U2JnnvOHX567TX429+8rsyY4IuIiKBFixZel1Gh2SGmMqZqVXjqKZg/Hxo3hksugcsug82bva7MGFPeWECUUV26uJB46in48kvXruO996AcHTE0xnjMAqIMi4iABx+EJUvg1FNhyBA491xI/XNLGmOMKTYLiHKgTRs3gf3yy64JYLt2rhFgVpbXlRljyjILiHIiLAxuucXdD7tXL7j1VjjjDMhxfxJjjCmWkAaEiPQVkZ9FZI2IPBDg9UQRSReRJb6fR/K8Hi4ii0Xkq1DWWZ5ER7uzm959112N3akTPP00HDnidWXGmLImZAEhIuHAq8C5QFtgkIi0DbDqTFWN9f2MyvPa7cDqUNVYXonANde4gLjgAnfPifh4d6MiY4wpqlCOIOKBNaq6VlUPA5OAi4q6sYg0BfoBx3/fvAqqcWPX+G/yZNi0CU4/3U1q23VHxpiiCGVARAE5b6aZ5luWV3cRWSoi34hIuxzLxwL3ATbVepwuuQRWr4arr4ZnnnGHnWbN8roqY0xpF8qACHSfvbxn6S8ColW1E/Ay8DmAiJwPbFXVhYXuRGSYiCSLSPK2bduOt+Zyq25dd3+JqVNd+/Devd2kdoAbVxljDBDagEgDmuV43hTYmHMFVd2jqvt8j6cAESLSAOgJXCgiKbhDU2eJyAeBdqKq41U1TlXjGjZsGIKPUb785S+u+d9tt7k2He3bu9Awxpi8QhkQC4BWItJCRCoDA4Evcq4gIo3Fd0dvEYn31bNDVR9U1aaqGuPb7ntVvTKEtVYoNWrASy+5w0yRka5j7DXXwM6dXldmjClNQhYQqpoB3AJMxZ2J9LGqrhSR4SKSfSPV/sAKEVkKjAMGanlqL1vK9ejhzmwaORI+/NBdjf3JJ9auwxjjWLtvA7h2HdddB4sWue6wr74KJ57odVXGmFCze1KbQsXGwrx57iynKVNc87933rHRhDEVmQWE8atUCe6/H5Ytc3euGzrUTWr//rvXlRljvGABYf6kdWtISnJnOf34ozvTadw4yMz0ujJjTEmygDABhYXBTTe55n8JCXD77e7aidXW+MSYCsMCwhSoeXP4+mt4/33XGTY2Fp580pr/GVMRWECYQonAlVe60cPFF8PDD0NcHCws9Dp3Y0xZZgFhiuyEE+Cjj+Czz2DbNtch9v774cABryszxoSCBYQptosvdq3Ehw6F555zzf9mzPC6KmNMsFlAmGNSpw68+SZ89x1kZLiJ7BEjYM8eryszxgSLBYQ5Ln36wPLlcOed8MYb7pTYKVO8rsoYEwwWEOa4Va8OL74Ic+ZAzZrQrx9cdRVs3+51ZcaY42EBYYKmWzfXy+mRR2DSJNeu46OPrF2HMWWVBYQJqipV4PHH3Smw0dEwcKCb1N64sfBtjTGliwWECYmOHWHuXHj+efjf/9xo4q23bDRhTFliAWFCplIluOceN4kdGws33ABnnw1r13pdmTGmKCwgTMi1bAnffw///CcsWODOdPrHP6z5nzGlnQWEKRFhYTBsmLvA7qyz4K673B3tVqzwujJjTH4sIEyJatoUvvzS3eJ07Vro0sVNah8+7HVlxpi8LCBMiROBQYPcaOKyy+Cxx+C009zhJ2NM6RHSgBCRviLys4isEZEHAryeKCLpIrLE9/OIb3kzEflBRFaLyEoRuT2UdRpvNGwIEyfCF1/Arl3uOop77oH9+72uzBgDIQwIEQkHXgXOBdoCg0SkbYBVZ6pqrO9nlG9ZBnC3qp4KdANuzmdbUw5ccIG7MdENN8ALL7hTZJOSvK7KGBPKEUQ8sEZV16rqYWAScFFRNlTVTaq6yPd4L7AaiApZpcZztWu7Xk7ff++en3km3HgjpKd7W5cxFVkoAyIKWJ/jeRqBv+S7i8hSEflGRNrlfVFEYoDOwLxAOxGRYSKSLCLJ27ZtO/6qjafOPBOWLXOHmt56y11g9+WXXldlTMUUyoCQAMvyXke7CIhW1U7Ay8Dnud5ApAYwGbhDVQM2klbV8aoap6pxDRs2DELZxmuRke4K7LlzoV49uPBCuOIKd5MiY0zJCWVApAHNcjxvCuTqyKOqe1R1n+/xFCBCRBoAiEgELhwmquqnoSpy4vKJxIyNIezxMGLGxjBx+cRQ7coUU3y86+n0+OPwySdw6qnu9Fhr12FMyQhlQCwAWolICxGpDAwEvsi5gog0FhHxPY731bPDt2wCsFpVXwxVgROXT2TYl8NITU9FUVLTUxn25TALiVKkcmXXHXbxYndF9uDBbkSRluZ1ZcaUfyELCFXNAG4BpuImmT9W1ZUiMlxEhvtW6w+sEJGlwDhgoKoq0BO4Cjgrxymw5wW7xpHTRrL/SO5zKvcf2c/IaSODvStznNq1g9mz3X0npk1zcxP//CdkZXldmTHll2g5Gq/HxcVpcnJykdcPezwM/dO0iPPEmU+QEJ1AfFQ8VSpVCVaJJgjWrnWnxH7/PSQmuluftmzpdVXGlE0islBV4wK+VpEDImZsDKnpqX9aHhEWQUZWBopStVJVejTrQWJ0IgkxCXSN6mqBUQqowoQJcPfdrk3HE0/AHXe4DrLGmKKzgMhH9hxEzsNMkRGRjL9gPOe2PJeZqTNJSklieup0lmxe4g+M7k27kxiTSGJMIvFR8VStVDUUH8cUwYYNMGKEuxo7Ls6FRseOXldlTNlhAVGAicsnMnLaSNalr6N57eaM7jOawR0G/2m9XQd2MXPd0cBYvGkxilIlvArdm3UnMdoFRtemXS0wSpgq/PvfcMstrmXHQw+5nyo20DOmUBYQIbD74O5cI4zFmxeTpVlUCa9Ct6bd/COMbk27WWCUkB073GGmDz5wk9gTJrj+TsaY/FlAlIDdB3cza90sklKSSEpJyhUYXZt29Y8wujXtRrWIap7UWFFMmeLadGzY4ALjiSegenWvqzKmdLKA8ED6wfSjgZGaxKJNi8jSLCqHV6ZrVFf/CKN70+4WGCGwZw888AC8/jq0aOHOdOrTx+uqjCl9LCBKgfSD6cxeP9s/wli4aaE/MOKj4v0jjO7NuhMZEel1ueXGjBlw/fXw669w3XUwZgzUqeN1VcaUHhYQpdCeQ3uYvW62f4SxcONCMjWTiLAIujbtSkJ0AokxifRo1sMC4zgdOODadYwZAyec4EYVFxWpr7Ax5Z8FRBmQHRjTU6eTlJJE8sZkf2DER8WTGJNIQnQCPZr1oHplO6B+LBYudKOIpUvh8sth3Dho1MjrqozxlgVEGbT30F5mr5/N9JTpJKUmsWDDAjI1k0phlfyHpBJiXGDUqFzD63LLjCNH4LnnYNQoqFEDXnrJ9XeSQL2HjakALCDKgX2H9+UaYSzYuICMrAwqhVXi9Can+0cYPZv3tMAogtWr3Whi7lw491x3s6Lmzb2uypiSZwFRDu07vI856+f4r8OYv2G+PzDimsT5J70tMPKXmQmvvgoPPghhYfDsszB8uHtsTEVhAVEB/HH4D39gJKUm+QMjXMJdYPhOq+3ZrCc1q9T0utxS5fff3XUT334LvXu7O9m1bu11VcaUDAuICuiPw38wN22u/7Ta+RvmcyTrCOESzmlNTss1wqhVpZbX5XpOFd57D+688+hZT3ffbc3/TPlnAWHYf2Q/c9fP9Y8w5qXN8wdGlxO7+EcYvZr3qtCBsWkT3HwzfPYZdOkCb78NnTp5XZUxoWMBYf5k/5H9/Jj2o3+EMW/DPA5nHiZMwlxgRB8NjNpVa3tdbombPNkFxY4dcP/98Pe/Q1VrqWXKIQsIU6gDRw4cDYzUJH5M+zFXYGRfuNe7ee8KExg7d8Jdd7lDT23auOZ/PXp4XZUxwWUBYYrtwJEDzNswzz/CmJs21x8YnRt3PhoY0b2pU7V8966YOhWGDYP1611L8aeectdQGFMeeBYQItIXeAkIB95S1WfyvJ4I/Af43bfoU1UdVZRtA7GACJ3swMi+cG/u+rkcyjyEIHQ+sbP/wr3ezXtTt1pdr8sNur173T0mXn3VXS8xfjz85S9eV2XM8TvugBCR6sABVc0SkdZAG+AbVT1SwDbhwC/AOUAasAAYpKqrcqyTCNyjqucXd9tALCBKzsGMg8xLm+e/cG/O+jn+wIhtHOu/cO+M6DPKVWDMmuWa//38MwwZAi++CHXLz8czFVAwAmIh0BuoC/wIJAP7VfXPt147uk134DFV/avv+YMAqvp0jnUSCRwQhW4biAWEdw5mHGT+hvn+Ecac9XM4mHEQQejUuJN/0rt3dG/qVavndbnH5eBB16rjueegYUM3qrjkEq+rMubYBCMgFqlqFxG5Faimqs+JyGJV7VzANv2Bvqp6ve/5VUBXVb0lxzqJwGTcKGEjLixWFmXbQCwgSo9DGYeYv2G+/0rv2etn+wOjY6OO/tNqz4g+o8wGxuLFMHQoLFkCl14Kr7wCjRt7XZUxxVNQQBT1MiDx/VU/GLiuiNsGan+WN40WAdGquk9EzgM+B1oVcdvswoYBwwCaWzOdUqNKpSr0ju5N7+jePMzDHMo4xIKNC/yT3uMXjueleS8BuMCIPhoY9SPre1x90YPO/xYAABn+SURBVHTuDPPnuzbijz8O338P//gHXH21Nf8z5UNRRxAJwN3AbFV9VkROAu5Q1dsK2KbYh4lEJAWIw4WEHWIqxw5nHmbBhgX+02rnrJ/D/iP7AehwQodcI4wGkQ08rrZwP/3k5iZmz3aT1//8J8TEeF2VMYUL6llMIhIG1FDVPYWsVwk30dwH2ICbaL5CVVfmWKcxsEVVVUTigU+AaNyZSwVuG4gFRNl1OPMwyRuT/SOM2etn+wOj/Qntc40wGlZv6HG1gWVluZsRPfCAa93x9NPuYjtr/mdKs2DMQXwIDAcygYVAbeBFVX2+kO3OA8bivvDfVtXRIjIcQFXfEJFbgJuADOAAcJeqzslv28LqtIAoPw5nHmbhxoX+EcasdbNyBUb2dRgJ0QmlLjBSU13zv6lToWdP1/yvTRuvqzImsGAExBJVjRWRwcBpwP3AQlXtGNxSj48FRPl1JPMICzct9I8wZq2bxR9H/gCgXcN2RwMjJoETqp/gcbVuBPH++3DHHfDHH/Doo3DvvRAR4XVlxuQWjIBYCcQCHwKvqOp0EVmqqqWqjZkFRMVxJPMIizYtyjXC2Hd4HwBtG7bNNcJoVMO7+4pu2eKuvv7kE4iNde06unTxrBxj/iQYAXEbbtSwFOgHNAc+UNXewSz0eFlAVFzZgZF94d7MdTP9gXFqg1P9YZEQk0DjGiV/Luqnn7r5iG3b3EjikUegWrUSL8OYPwlJqw0RqaSqGcdVWZBZQJhsGVkZLjB8F+7NTJ3J3sN7AWjToI2/NUhCdAIn1jyxRGratQvuuce1EG/d2o0mevUqkV0bk69gjCBqA48CZ/gWTQdGqWp60KoMAgsIk5+MrAwWb1rsv3BvRuoMf2CcUv8U/2m1JREY330HN9wAKSluVPH001DTbvJnPBKMgJgMrADe8y26CuikqqWqwYAFhCmqjKwMlmxekisw9hxyZ263rt/af1ptQkwCTWo2Cfr+9+1z95gYNw6aNXPXTfTtG/TdGFOooJ3FVNgyr1lAmGOVmZXpD4zsQ1Lph9wAuVW9VrlGGFG1ooK237lz4brrYPVqdwX2iy9C/bJxIbkpJ4IREHOBe1V1lu95T2CMqnYPaqXHyQLCBEtmViZLtyz1n1Y7I3WGPzBa1muZa4TRtFbT49rXoUPw5JPwzDNQr57r6dS/v7XrMCUjGAHRCfg/3AVyALuAa1R1WdCqDAILCBMqmVmZLNuyzD/CmJE6g90HdwNwct2Tc40wmtVudkz7WLrUjSYWLoSLL4bXXoMTS2b+3FRgQTuLSURqAajqHhG5Q1XHBqnGoLCAMCUlMyuT5VuX5xph7Dq4C3CBkX0dRmJMYrECIyPDNfx75BGoUsUdcrr2WhtNmNAJ1Wmu61S1VLVPtYAwXsnSLJZvWe4fYUxPme4PjJPqnpQrMJrXLvx/m19+cWc6zZgBZ5/t7mDXokWoP4WpiEIVEOtV9djG0iFiAWFKiyzNYsXWFf4RxvTU6ew8sBOAFnVakBCT4J/HiK4THfg9slww3HcfZGa6e2HfcguEh5fkJzHlnY0gjPFYdmBkX7g3PWU6Ow7sACCmTox//iIxJpGYOjG5tl2/3jX/++Yb6N7dNf9r29aDD2HKpWMOCBHZS+Ab9QjuznJFveFQibCAMGVFlmaxcutKf2uQ6anT2b5/OwDRtaNzTXq7wBA+/BBuvx327oWHH3Yji8qVPf0YphwIyQiiNLKAMGVVlmaxatsqf1gkpST5A6N57eYuMKITaV8zkRcejuGjSULHjq5dR1zA/7WNKRoLCGPKGFX9U2Bs278NgGa1mnFyeCJLPk8kfWkid1/XglGPizX/M8fEAsKYMk5VWb19da5J761/bHUvpjel5o5Ehp+byI1/SeSkuichdl6sKSILCGPKGVXlp+0/kZSSxMfzk5i5fjqZ1bYA0KRGFGe2SPTPY5xc92QLDJMvCwhjyrl9+5RbH/+Z96YnUbVNElXbTGfXkc0ARNWMynVabct6LS0wjJ8FhDEVxLx5MHQorFqlXDDkFxKuSSJ5hzsstXmfC4wmNZvkunCvVb1WFhgVmGcBISJ9gZeAcOAtVX0mn/VOB34EBqjqJ75ldwLX406zXQ5cq6oHC9qfBYQxrvnf00+7C+tq14aXX4bLL1fW7PrVP4eRlJLEpn2bADixxom5rsNoXb+1BUYF4klAiEg48AtwDpAGLAAGqeqqAOt9CxwE3lbVT0QkCpgFtFXVAyLyMTBFVd8taJ8WEMYctXy5a/63YAFceKFr/hfl61SuqqzZucbfGiQpJYmNezcC0LhG41yBcUr9UywwyrGCAiKUF7rFA2tUda2viEnARcCqPOvdCkwGTg9QWzUROQJEAhtDWKsx5U6HDu5+E2PHugvr2raFMWPg+utBRGhVvxWt6rfihtNu8AdG9im1SSlJTFoxCYBG1RvlCow2DdpYYFQQoRxB9Af6qur1vudXAV1V9ZYc60QBHwJnAROAr3IcYrodGA0cAP6nqoPz2c8wYBhA8+bNT0tNTQ3J5zGmLFuzxjX/S0qCM8+EN9+Ek0/Of31V5bddv/lbg/zw+w9s2LsBgBOqn+C/cC8hJoFTG5xqgVGGeTWCCPRfTN40Ggvcr6qZOf8DE5G6uNFGC2A38G8RuVJVP/jTG6qOB8aDO8QUpNqNKVdatoRp01wfp3vvdaOLJ590rTsCNf8TEVrWa0nLei25rst1qCprd631X4PxQ8oPfLzyY8AFRs5JbwuM8iOUI4juwGOq+lff8wcBVPXpHOv8ztEgaQDsx40GInCjj+t8610NdFPVEQXt0+YgjClcWhrcdBN89RXEx7t2He3bF+89VJXfd/9+NDB+/4H1e9YD0DCyYa7Tats2bGuBUYp5NUldCTdJ3QfYgJukvkJVV+az/rv4DjGJSFfgbdy8xAHgXSBZVV8uaJ8WEMYUjSp89BHceiukp8PIkfDgg8fe/E9VSdmdkmvSe136OgAaRDbINcJo27AtYRIWxE9jjoeXp7mehzuMFI47Q2m0iAwHUNU38qz7LrnnIB4HBgAZwGLgelU9VND+LCCMKZ7t291hpg8/dKOICRPcqCIY/IHh+0lNd/OD9avVzzXCaHdCOwsMD9mFcsaYAn31FQwfDps2wZ13wqhREBkZ3H2k7E7xT3onpSSRsjsFcIFxRvQZ/hFG+xPaW2CUIAsIY0yh0tPh/vvhn/+Ek05yE9pnnhm6/aXuTs11Wu3vu38HoF61ei4wfCOMDo06WGCEkAWEMabIkpLctRK//QbDhsFzz7krskNtXfo6N8LwzWOs3bUWgLpV65IQk+Cfx+jYqKMFRhBZQBhjimX/fnjsMXjhBWjcGN54Ay64oGRrWJ++PtcI47ddvwEuMM6IPiNXYISH2Y26j5UFhDHmmCQnu+Z/y5fDwIEwbhw0bOhNLdmBkT2PsWbnGgDqVK2TKzA6NepkgVEMFhDGmGN2+DA8+yw88QTUquVCYtAg8PrShrQ9aUxPme4fZfy681cAalep7Z/0TohOILZxrAVGASwgjDHHbeVK1/xv3jzo1w9efx2aNfO6qqM27NmQa4Txy45fABcYvaN7+ye9LTBys4AwxgRFZqZrHz5ypGvR8fzzrsdTWCmcM964d6N/0nt66nR+3vEzALWq1KJ3897+02pjG8dSKSyUXYdKNwsIY0xQrV3rznCaNg0SElzzv1atvK6qYJv2bvIfjpqeOp2ftv8EQM3KNXONMDqf2LlCBYYFhDEm6FTh7bfh7rvdTYpGjXIX2VUqI9+tm/dtznVabc7A6NW8l3+E0eXELuU6MCwgjDEhs3EjjBgB//kPxMW5dh0dO3pdVfFt3reZGakz/KfVrt6+GoAalWu4wIg+GhgR4REeVxs8FhDGmJBShU8+gVtugZ07XeO/kSOhShWvKzt2W/ZtORoYqUms2ubudVajcg16NuvpH2GcduJpZTowLCCMMSVixw53mOn9990d7CZMgG7dvK4qOLb+sTXXCGPlNteYunpEdXo17+W/DiOuSVyZCgwLCGNMifrmG7jxRnfvidtvdzcnql7d66qCa9sf23KNMFZsXQG4wOjZvGeuwKgcfox91EuABYQxpsTt2eMONb32GrRoAePHw9lne11V6GQHRvaZUsu3LgcgMiLSf0gqITqB06NOL1WBYQFhjPHMzJnuArtff3VtO154AerU8bqq0Nu+f7sLDN+Fe8u2LAOgWqVq9Gze039P7/ioeE8DwwLCGOOpAwfcabDPPw8nnOBGFRdf7HVVJWvH/h3+Q1LTU6ezdMtSwAVGj2Y9/JPepzc5nSqVSm523wLCGFMqLFzoRhNLl8Jll7mrshs18roqb+zYv4OZ62YeDYzNS1GUqpWqusDwnVYbHxUf0sCwgDDGlBpHjriRxOOPQ40aMHYsXHml983/vLbzwE5mph4NjCWbl/gDo3vT7v4RRteorv7AmLh8IiOnjWRd+jqa127O6D6jGdxhcLH2awFhjCl1Vq92o4m5c6FvX3cnu+bNva6q9Nh1YJd/hJGUkpQrMLo17Ua9qvX4+tevOZR5yL9NZEQk4y8YX6yQ8CwgRKQv8BIQDrylqs/ks97pwI/AAFX9xLesDvAW0B5QYKiqzi1ofxYQxpQtmZluPuLBB90I4pln4KabSmfzP6/tOrCLWetm+U+rXbRpUcD1omtHk3JHSpHf15OAEJFw4BfgHCANWAAMUtVVAdb7FjgIvJ0jIN4DZqrqWyJSGYhU1d0F7dMCwpiyKSXFNf/79lvo1cvdD/uUU7yuqnQLezwM5c/f34KQ9WhWkd+noIAIZU7HA2tUda2qHgYmARcFWO9WYDKwNXuBiNQCzgAmAKjq4cLCwRhTdsXEwNSp8M47sGIFdOrkRhMZGV5XVno1rx34eFx+y49FKAMiClif43mab5mfiEQBfwPeyLPtScA24B0RWSwib4lIwOswRWSYiCSLSPK2bduCV70xpkSJwJAhbm6iXz932KlrV1iyxOvKSqfRfUYTGRGZa1lkRCSj+4wO2j5CGRCBzknIOx4aC9yvqpl5llcCugCvq2pn4A/ggUA7UdXxqhqnqnENvbpZrjEmaBo3hsmTXfO/DRtch9iRI+HgQa8rK10GdxjM+AvGE107GkGIrh1d7AnqwoRyDqI78Jiq/tX3/EEAVX06xzq/czRIGgD7gWG4CesfVTXGt15v4AFV7VfQPm0OwpjyZedOd7+Jd9+FNm3c3ETPnl5XVb54NQexAGglIi18k8wDgS9yrqCqLVQ1xhcEnwAjVPVzVd0MrBeR7GmqPkCuyW1jTPlXr56bl5g61V2N3bs33HYb7NvndWUVQ8gCQlUzgFuAqcBq4GNVXSkiw0VkeBHe4lZgoogsA2KBp0JVqzGmdPvLX9zk9S23wCuvQPv28L//eV1V+WcXyhljypTZs90Fdj//7Ca1X3jBjTTMsfHqEJMxxgRdz57uzKaHHjp6Y6LJk72uqnyygDDGlDlVq8Lo0ZCcDE2aQP/+7mfzZq8rK18sIIwxZVZsLMyb5y6q++orN5p49113j2xz/CwgjDFlWkQE3H+/ayHerh1cey389a+ufYc5PhYQxphy4ZRTYPp0ePVV1yG2fXt3v4msorclMnlYQBhjyo2wMBgxwp0Sm33NRO/ern2HKT4LCGNMuRMdDVOmwP/9H/z0k5ureOopd7MiU3QWEMaYckkErroKVq1y978eORLi42FR4NsomAAsIIwx5VqjRvDRR/DZZ+402Ph41yn2wAGvKyv9LCCMMRXCxRe70cSQIe602NhYmDnT66pKNwsIY0yFUbeu6wj77bdw+DCccQbcfDPs3et1ZaWTBYQxpsI5+2x3ptMdd8Drr7vrJ775xuuqSh8LCGNMhVS9OvzjH675X40acN55cPXVsGOH15WVHhYQxpgKrXt3WLwYHn4Y/vUv167j3/+2dh1gAWGMMVSpAqNGwcKF0KwZXH45XHIJbNzodWXesoAwxhifjh3hxx/huefgv/91o4kJEyruaMICwhhjcqhUCe69F5Ytg06d4Prr4ZxzYO1arysreRYQxhgTQKtW8MMP7iyn+fOhQwcYOxYyM72urOSENCBEpK+I/Cwia0TkgQLWO11EMkWkf57l4SKyWES+CmWdxhgTSFgYDB8OK1dCYiLceSf06uUuuKsIQhYQIhIOvAqcC7QFBolI23zWexaYGuBtbgesD6MxxlPNmrkbEk2cCL/+Cp07wxNPuIvtyrNQjiDigTWqulZVDwOTgIsCrHcrMBnYmnOhiDQF+gFvhbBGY4wpEhG44grXOvySS+CRRyAuDhYs8Lqy0AllQEQB63M8T/Mt8xORKOBvwBsBth8L3AcUeLsPERkmIskikrxt27bjq9gYYwrRsKG7XuI//3EX1XXrBvfdB/v3e11Z8IUyICTAsrwni40F7lfVXNM+InI+sFVVFxa2E1Udr6pxqhrXsGHDY6/WGGOK4cIL3VzEddfB88+7M56mT/e6quAKZUCkAc1yPG8K5L3sJA6YJCIpQH/gNRG5GOgJXOhbPgk4S0Q+CGGtxhhTbLVrw/jxMG2au7VpYiLcdBPs2eN1ZcERyoBYALQSkRYiUhkYCHyRcwVVbaGqMaoaA3wCjFDVz1X1QVVt6ls+EPheVa8MYa3GGHPMzjoLli+Hu+92gdGuHXz9tddVHb+QBYSqZgC34M5OWg18rKorRWS4iAwP1X6NMcYLkZEwZgzMnQt16sD558PgwVCWp0ZFy9E15HFxcZqcnOx1GcaYCu7wYXj6aRg92h2GevllGDDAnQlV2ojIQlWNC/SaXUltjDFBVrkyPPqou//1SSfBoEFw0UWwYYPXlRWPBYQxxoRI+/YwZw688AJ8951r/vfmm2Wn+Z8FhDHGhFB4ONx1l5vEPu00GDYM+vSB337zurLCWUAYY0wJOPlkdzrs+PHuvhMdOriRRWlu/mcBYYwxJUQEbrjBXWB39tlwzz3ujnYrVnhdWWAWEMYYU8KiolyrjkmTICUFunSBxx4rfc3/LCCMMcYDIu7U11Wr3C1OH3/cBcX8+V5XdpQFhDHGeKhBA/jgA9dOPD3dHXK6++7S0fzPAsIYY0qBfv3cjYmGDYMXX3ST2D/84G1NFhDGGFNK1KrlbnGalOTuZnfWWS4wdu/2ph4LCGOMKWUSEmDZMnefiQkTXPO/L74ofLtgs4AwxphSqFo1ePZZmDcP6td3rToGDoStWwvfNlgsIIwxphSLi4PkZHcP7M8+c+06Jk4smXYdFhDGGFPKVa4Mf/87LF4MrVrBlVfCBRfAuHEQE+PmK2JiXHAEU6Xgvp0xxphQadsWZs2CV16Be+/NfVOi1FQ3oQ3uPhTBYCMIY4wpQ8LD4fbboWHDP7+2fz+MHBm8fVlAGGNMGbRpU+Dl69YFbx8WEMYYUwY1b1685cfCAsIYY8qg0aPdfbBziox0y4MlpAEhIn1F5GcRWSMiDxSw3ukikiki/X3Pm4nIDyKyWkRWisjtoazTGGPKmsGD3b0loqNd47/oaPc8WBPUEMKzmEQkHHgVOAdIAxaIyBequirAes8CU3MszgDuVtVFIlITWCgi3+bd1hhjKrLBg4MbCHmFcgQRD6xR1bWqehiYBFwUYL1bgcmA//pAVd2kqot8j/cCq4GoENZqjDEmj1AGRBSwPsfzNPJ8yYtIFPA34I383kREYoDOwLx8Xh8mIskikrxt27bjLNkYY0y2UAaEBFiW9+LwscD9qhrwrqwiUgM3urhDVfcEWkdVx6tqnKrGNQx0YrAxxphjEsorqdOAZjmeNwU25lknDpgkIgANgPNEJENVPxeRCFw4TFTVT0NYpzHGmABCGRALgFYi0gLYAAwErsi5gqq2yH4sIu8CX/nCQYAJwGpVfTGENRpjjMlHyAJCVTNE5Bbc2UnhwNuqulJEhvtez3feAegJXAUsF5ElvmUPqeqUgva5cOHC7SKSeowlNwC2H+O2xhjjpeP5/orO7wXRkugZWwaISLKqxnldhzHGFFeovr/sSmpjjDEBWUAYY4wJyALiqPFeF2CMMccoJN9fNgdhjDEmIBtBGGOMCcgCwhhjTEAVPiCK2pLcGGNKGxF5W0S2isiKULx/hQ6IHC3JzwXaAoNEpK23VRljTJG9C/QN1ZtX6ICg6C3JjTGm1FHVGcDOUL1/RQ+IQluSG2NMRVXRA6IoLcmNMaZCqugBUZSW5MYYUyFV9IDwtyQXkcq4luRfeFyTMcaUChU6IFQ1A8huSb4a+FhVV3pblTHGFI2I/AuYC5wiImkicl1Q399abRhjjAmkQo8gjDHG5M8CwhhjTEAWEMYYYwKygDDGGBOQBYQxxpiALCCMKQYRyRSRJTl+gtYBWERiQtWV05hjUcnrAowpYw6oaqzXRRhTEmwEYUwQiEiKiDwrIvN9Py19y6NFZJqILPP9s7lveSMR+UxElvp+evjeKlxE3hSRlSLyPxGp5tmHMhWeBYQxxVMtzyGmATle26Oq8cArwFjfsleA/1PVjsBEYJxv+Thguqp2AroA2VfwtwJeVdV2wG7g0hB/HmPyZVdSG1MMIrJPVWsEWJ4CnKWqa0UkAtisqvVFZDtwoqoe8S3fpKoNRGQb0FRVD+V4jxjgW1Vt5Xt+PxChqk+G/pMZ82c2gjAmeDSfx/mtE8ihHI8zsXlC4yELCGOCZ0COf871PZ6D6xIMMBiY5Xs8DbgJ3K1vRaRWSRVpTFHZXyfGFE81EVmS4/l/VTX7VNcqIjIP94fXIN+y24C3ReReYBtwrW/57cB4X/fNTFxYbAp59cYUg81BGBMEvjmIOFXd7nUtxgSLHWIyxhgTkI0gjDHGBGQjCGOMMQFZQBhjjAnIAsIYY0xAFhDGGGMCsoAwxhgT0P8DgslVDlYXqA0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "\n",
    "\n",
    "pyplot.plot(df_stats['Training Loss'], 'b-o', label=\"Training\")\n",
    "pyplot.plot(df_stats['Valid. Loss'], 'g-o', label=\"Validation\")\n",
    "pyplot.title(\"Training & Validation Loss\")\n",
    "pyplot.xlabel(\"Epoch\")\n",
    "pyplot.ylabel(\"Loss\")\n",
    "pyplot.legend()\n",
    "pyplot.xticks(df_stats.index.values.tolist())\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performance On Test Set\n",
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 1138.19it/s]\n"
     ]
    }
   ],
   "source": [
    "test = convert_to_dataset_torch(X_test, y_test)\n",
    "test_dataloader = DataLoader(test,  sampler=SequentialSampler(test), batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate on Test Set\n",
    "With the test set prepared, we can apply our fine-tuned model to generate predictions on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluating: 100%|███████████████████████████████████████████████████████████████████| 32/32 [17:21<00:00, 32.55s/batch]\n"
     ]
    }
   ],
   "source": [
    "bert_model.eval()\n",
    "\n",
    "_, _,_ ,predicted_labels = eval_batch(test_dataloader, bert_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot confusion matix over the test results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD4CAYAAADM6gxlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd5wURfrH8c93F5Ccc1BQQMUEgogKiJgzmDGgguHOcIYznjlnzzv1d4oRDChGMCBJEQwgCEhGQBSWHJScdvf5/dG9y7DM7M6G2WVnn/fr1a+dqanurm60pqa66imZGc4550qPlJIugHPOufzxits550oZr7idc66U8YrbOedKGa+4nXOulPGK2znnSpliqbglZUiaErE1zyXvhiI435uSFoTnmiTpiAIc41VJbcLX/8rx2Q+FLWN4nKz7Ml3SZ5Jq5pG/raRTCnnO9pKmSZon6b+SFCVPHUnfSNog6YUcnz0iaVHOfydJfwuPO0XSd1n3LvzsifAap0s6vzDld86BimMct6QNZla1qPPmcow3gc/N7ENJJwBPm9nBhTheocuU13El9Qd+NbNHcsl/GdDBzK4rxDl/Am4AxgFfAv81s6E58lQB2gEHAgdGnk9SJ+APYG7kPZFU3czWha/PAK4xs5MknQrcCJwM7AF8C3TPyuucy78S6SqRVFXSqLA1PE3SmVHyNJI0JqJF2iVMP0HSj+G+H0jKq0IdA7QM9705ouV3Y5hWRdIXkn6JbBFKGi2pg6THgUphOd4JP9sQ/n0/sgUctvTPlpQq6SlJEyRNlXR1HLflR6BJeJyOkn6QNDn8u6+kCsCDwPlhWc4Py/56eJ7J0e5jznsKVDezHy34xh4A9MiZz8w2mtl3wJYon40zs6VR0iMr4ipAVougDfCtmaWb2UbgF+CkOO6Hcy4WM0v4BmQAU8LtE6AcQQUCUBeYx47W/4bw7z+Bu8LXqUC1MO8YoEqYfjtwb5TzvQmcE74+FxgPtAemEVQqVYEZBK3Ks4FXIvatEf4dTdC6zS5TRJ6sMvYE+oevKwCLgErAVcDdYfoewESgRZRyboi4vg+Ak8L31YFy4evjgI/C15cBL0Ts/yhwcfi6JvBreH2NgS+jnK8DMDLifReCXyax/t12Ol+0sudIuxaYH96HVmHaCcD3QOXw3+834J/F8d+db74l61Zu16o8ITabWdusN5LKA49K6gpkErQ0GwDLIvaZALwe5v3UzKZIOpqgBfd92DVbgaClGs1Tku4GVgJ9gWOBTyxo9SHpY4KK6yvgaUlPEFRiY/NxXUOB/0rag6AVOcbMNofdMwdLOifMVwNoBSzIsX8lSVOA5sDPwIiI/P0ltSJouZaPcf4TgDMk3RK+rwjsaWazgGh94bv0Z7OjZVxoZvYi8KKkC4G7gUvNbLikw4AfCP4tfgTSi+qczpVFJTWq5CKgHtA+rNCXE1Q62cxsDNAVWAy8Jak3QcUzwszahlsbM+sb4xy3hnmON7PpRK+0MLNf2dEaf0zSvfFehJltIWiZnwicD7wXfiTg+ohytjCz4VEOkfWFthfBl9C1YfpDwDdmdiBwOjnuTQQBZ0ecJ6vSjiUNaBrxvimwJM8Lzb/3iOiCMbNHsv4twjLPTcA5nSszSqrirgGsMLPtko4hqLh2ImmvMM8rwGvAoQQP1I6SlNVnXVlS6zjPOQboEe5ThaCbY6ykxsAmM3sbeDo8T07bw5Z/NO8BlxO03oeFacOAv2ftI6l1eM6ozGwt8A/glnCfGgRfWBB0V2RZT9BllGUYcH3WyBBJ7WKdIzzPUmC9pE7hPr2BwbntE6/w10GWUwkr57C/v074+mDgYCDal5hzLl7F0R/Drn3EdQl+Mk8EXgVmAc0j8wKXAtOBycBYwj5ioDtBN8rUcDsjyvneJOzjzpF+c3jM6cCNYdqJ4XGmhMfN6tceHfH6ibCM7+S8HoJujNXAGxFpKQT9z9PCc31D2Heex335DLgEOIKgv/p7gtb37+HntcMyTiFo4VcCXo44z+dhvqh93OFnHcK884EX2PFs4QzgwYh8vwNrgA0ELfU2YfqT4fvM8O/9Yfp/CJ4bTAmv94AwvSIwM9zGAW2L478533xL5q1YhgM655wrOj5z0jnnSplknjm5OBztgaS6kn4v7HGjnKdHjhmCD0o6rgiOGznz8xdJx8axz7/yypPH/nuE49LnSRof699I0ldhmWZIeklSal775/j3HxLlmM8Xxb+7c2VFcbW4N9uOkQ9tzez3YjhnBtAnwefoQTA8EQAzu9fMRhbRsW+1YMTJjcBLceQvVMVNMGTyTzNrCfyboF8/mvPM7BCCWZX1CMbJ57V/5L//GZEHk9SBYAy6cy5OyTxz8jngJkm7jFWXdGvErMYHItLvkTRb0ghJA7PGR0u6Msz/i6SPwpEpRxI80HsqLOM+YUv5HEknSxoUcdxukj7LZ/mzZM+oDPf/VNLPYYv3qjAt2uzOiyX9FKa9nNUyzsWZQP/w9YfAsVmjVSLZjhmS5QiGMGY9JIlr/0hhmZ4CbsujbM65CMVVcWdVKlMkfUIwlbqnmR0KHAM8E+V/8guBYWGr8xBgiqS6BBM7jgv3nUgwUiSahcB3BKM0simYHNMK6Ai0BdpL6hq2/M4mmE15FsHoiywfm9lhYUtzFtDXzH4AhrBjvPj8iPwjgE4RQwDPB97PZ/mznAR8GvG+j5m1D8v3D0l1zOwOdrRqL5K0f3jOo8L7l0Ewdj4reFYHdtWEYMYjZpYOrAXqRCuQpGHACoLhiR/GsX9FSRMljZMUOcX+OmCIRZlC75yLLZlnTkIwJG8I8EVE2gnhNjl8X5WgIq8GDDazzWEZP4vY50BJDxP8pK/KjvHaUZlZuqSvgNMlfUgwrvk2IL8zP58E6gOdItL/Ialn+LpZWPbVOfY9lmBS0YTwPJUIKlrM7IoY54t7VqWZnSipIvAOwfDMEXnsv6eZLZG0N/C1pGnAZoJulm4xyuOci6G4Ku6cImdOblfw4HCXmZNhxX4qwczJp4A/CWZO9ornJGY2T8GU8vMikgU8ZmYvR+aVdFMuh3oT6GFmvyiI0NctjtO/TzATcg0wwczWh78q4i3/rcDHBBNz+hP8MuhGELvkCDPbJGk00WdViiCGyp1xnCdLGsEXQVrYvVQjLHtUZrYlfNB4JkHFHXN/M1sS/v0tLHM7goq7JTAv/HKpLGle2EfunMtFWZg5+QhwS8T7YUCfrL5lSU0k1SfoVjldUsXws1Mj9qkGLA1b/xdFpOecyRhpdFjmKwkqcfJbfjPLJJjYkiLpRIL79mdYae/Hzi3xyNmdo4BzwutCUu3wfuZmCMGkJ4BzgK8txyB/Bc8mGoWvyxHEQ5md2/6SailidA9wFDDTzL4ws4Zm1tzMmhPMXvVK27k4lFTF/Q7QQdJEgopwdpQ83Qj6tScT9D3/x8xWEkwBHyhpKkFFuF9uJzKzGcCkiPfDgXeBH8Of7B8C1cxsAkHl8wtBS3ciQT8twD0EEQZH5Cjre8CtCkKq7pPjvBnA5wRxqD8P0wpSfgMeJuhq+QooF+77ULh/ln7AVEnvmNlMgr704WHeEUBWhRurj/s1oI6keQT97ndkfRD+aoEg8uCQ8Ji/EHS/vJTH/vsDEyX9QjCj8vGwfM65AvKZkxEkVTWzDZIqE8Q2ucrMJuW1n3POFaeS6uPeXfVTMKGmIkEfsVfazrndjre4nXOulPFYJc45V8p4xe2cc6WMV9zOOVfKeMXtnHOljFfczjlXyhRXPO6Gkt6TNF/STElfxjHjMdaxuoSR8aZIqpTPff+mYNHhQpHUXJJJeigira6k7ZJeyGPfbgoiC8b6/AxJd8T63DnnEl5xh/E5PgFGm9k+ZtaGIHZ0gwIe8iLg6TAS3ub87GhmL5nZgAKeN6ffgNMi3p9LsOZiXroBUStuSeXMbIiZPV744jnnklVxtLiPAbabWfZiAGY2xczGKvCUgnjb0ySdD9mt0tGSPlQQH/udMO8VBAGj7g3Tukn6POu4kl4Ig0Ah6fGwdT9V0tNh2v3aEWO7rYIwo1MlfSKpVpg+WtITCmJZ/6owDngUm4FZEdPHzwciY3CfrmAlmMmSRkpqoGBVmL8RxAmfEv56eFPSs5K+AZ6QdFlWq13S4KxfCJKuVhhr2zlXthXHzMkDgZ9jfHYWQUzsQwhWfp8gaUz4WTvgAGAJwWrnR5nZq5I6E6xm/qGCaHm7kFQb6AnsFwY6irbCygDgejP7VtKDwH0Eq80AlDOzjpJOCdNjLUf2HnCBpGUEMa+XEKywDkHQqk7h+a8AbjOzf0p6iWB196wvk75Aa4IY3RlZXzyhqwhCwC4A/kkYVErS3yD4BRGjXM65JFbSU947AwPDgEzLJX0LHAasA34yszTIDnLUnKAyjMc6gsUaXpX0BWGQpyySagA1zezbMKk/8EFElo/Dvz+H543lK4JgT8vZEQEwS1OCxRMaEcTdXpDLcT4I78FOzGy5pHsJgjP1NLOsMKleYTtXhhVHV8kMgqD+0eS2tNXWiNcZRP+SSWfna6gI2SuwdAQ+IlgX8qt4C5vj3LHOS3iebQSV+z/Dc0V6HnjBzA4CriZ63OwsG3P57CCChRIa55LHOVeGFEfF/TWwh6QrsxIkHaZgNZsxwPmSUiXVA7oCP+Xj2H8AbRSsMF6DYOUXFMTTrmFmXxJ0f7SN3MnM1gJ/RvRfXwJ8S8E8A9xuZjlXoakBLA5fXxqRnlsM751I6kgQFrYdcIukFgUso3MuiSS8qyTs4+0JPBcOc9sC/E5QoY4BjiCI7WwE/cDLFCwSEM+xFylYlHcqMJcdy5FVAwYrWF5LQLTVbS4FXgpDuP4GXF7A65tB9NEk9wMfSFpMEDc7q9L9DPhQwQLJ18c6roLFB14BLg+X/fonwVJu3Qla8N5l4lwZ5dEBnXOulPGZk845V8p4xe2cc6VMwvu4Nw97wftiEqxD77dKughJb86faSVdhDIhfdvi3EaaxWX7qt9i1jnl6+5d6OPvDrzF7ZxLLhnbY29xCke6Tc6amS2ptqQRkuaGf2tF5L1T0jxJcySdmIAr2oVX3M655JKZGXuL3w3ArIj3dwCjzKwVMCp8j4I1ai8gmOV9EvB/klKL5Dpy4RW3cy6pWEZ6zC0ekpoCpwKvRiSfSTDDmvBvj4j098xsq5ktAOYRTP5LqJKe8u6cc0UrH10iMTwH3MbOE+UamNlSADNbKql+mN6EYJ5GlrQwLaG8xe2cSy6WGXOTdJWkiRHbVZG7SjoNWGFmsQLj5RTtYWfCB2R4i9s5l1Ry6xIxs35Av1x2Pwo4I4wMWhGoLultgiB4jcLWdiNgRZg/DWgWsX9TgiihCeUtbudccslIj73lwczuNLOmZtac4KHj12Z2MTCEHTGHLgUGh6+HEIR23iOMJdSK/MVbKhBvcTvnkkvmLhGSi8LjwKAwfv5CghWvMLMZYbykmQTRSq+NFqK5qHnF7ZxLLnGOHsmLmY0GRoevVxNGH42S7xHgkSI5aZy84nbOJZciqrh3Z15xO+eSSjH0VJQ4r7idc8nFW9zOOVfKFH4Czm7PK27nXHKxfMUkKZW84nbOJRfvKnHOuVIm3Stu55wrVXxUiXPOlTbeVeKcc6WMV9zOOVfK5G+lm1IpKSrurdvT6fOfj9ienkF6pnFc23245pRO/O/L8Xz84wxqVa0EwPWnHUGXA5qzPSODBwZ+zexFK8nIzOS0w/aj7wkddjnu2o1buO3Nr1iyZh2Na1fnqctPonrligC8Nnwin46bSUqKuP3srhy5/14AzFy4gnvfGcnW7el0brMXt53dFSkp1ifdRUpKCoOGv8nyZSu59uJ/AnBh33O5sM+5ZKRnMGbk9zzz0Au77Nf5mE7c8fDNpKam8NE7Q3j1+QEA1KhZnaf7PUyTZo1ZvGgJ/7zyLtatXQ/AFf+4lLMvPJ2MjEweu+sZvh89vvgudDdx4gndePbZB0lNSeH1Nwby5FMv7vR59erVGND/eZo1a0K5cqk8++xL9B8wCIAb/nElffr0wsyYPn02fa+4ma1btwJw7TWXc801l5Oens7QoaO4485iDbtR9LzFXTpUKJfKK9f3pPIeFdiekcHlz31E5/2bA3Bxt7ZceuyhO+UfMXke29Mz+PDOC9m8bTtnPfoOJ7VvTZM61XfK9/rInzm8dVP6HN+B10dM5PURP3PjmUcxf+kahk36lY/uvIiV6zZw9QufMvieS0hNSeGRQd9wzwXHcHDzhlz30hC+n/UHnds0L6Y7UbwuufJ8fpv7O1WqVQGg41Ht6X5SV3oecxHbt22ndt1au+yTkpLCXY/fypXnXc/yJSt4f9ibfDNsLPN/XcAV1/dm/NiJvPr8AK64vjdXXN+bZx9+kX1at+CUHsdzRtde1G9Yl1c/eIFTjziXzDLQssqSkpLCf//zCCed0ou0tKWM+/FLPvt8OLNmzc3Oc83fL2PWrF/p0fMy6tatzczpY3h34CfUq1eH667tw0GHHMOWLVsY+O5LnH/emQx4axDdjj6SM04/kXaHHse2bduoV69OCV5lESkDFXdSxOOWROU9KgCQnpFJekYmuTVyJbF563bSMzLZuj2d8qmpVK1YYZd8o6f9xukd9wfg9I77882037LTTzy0NRXKp9KkTg2a1avJ9D+Ws3LtRjZu2cYhLRohidM67s83U38r+gveDTRoVJ+uxx/FR+8Mzk47/9KzePX5AWzfFsxcW7Pqz132O+jQNixakEbaH0vYvj2dLz8dwTEndQXgmJO68un7XwDw6ftf0P3ko7PTv/x0BNu3bWfxwqUsWpDGQYe2SfQl7lY6HtaO+fN/Z8GChWzfvp1BgwZzxuk7LyhuZlStWhWAqlWrsGbNX6SHQ+PKlStHpUoVSU1NpXKlSixdugyAq6/uzZNPvci2bdsAWLlydTFeVYIUzWLBu7WYFbekaZKmRtmmSZpanIWMR0ZmJuc9MZDu/3qNTvs246DmDQF4b+xUzn38Xe57ZyTrNm0B4Li2+1Bpj/Icf/drnHTfm/Tu3o4aVSrucszV6zdRr0bQmqxXowpr1m8GYMXaDTSsVTU7X4OaVVnx10ZWrN1Ag5qR6VVYsXZjwq65JN3x0E088+ALZGbuWKWp+T570v7wtgwc+hpvfvI/Dmy7/y77NWhYn6VLlme/X75kBQ0a1gOgTr3arFoRVByrVqzObrE3aFiPZYt37LNs6QoaNKxPWdK4SUMWpe1YWCVt8VIaN264U54X/+8N9t+vFYv+mMSUSaO4+Z/3YWYsWbKMZ//9Egvm/0TawsmsXbeOESPHANCq1d507tyRH777jK9HfkiH9ocU63UlRCEWUigtcmtxnwacHmXLSo8pcl231778vqjKmqvUlBQG3d6LYQ9ezvQ/ljNvyWrO63wQn9/bm/dv60XdGlV45pPvAJj+x3JSJIY/3Icv77uUt76ZTNqqtXGfy6KsKCfFSC/oBe3Gjj7+KNasWsPMqbN3Sk8tl0r1mtXodXJfnnnweZ555dFdd45yQyyPJfqiPSOwaDc7icVzD044oRu//DKDZnsdSvvDTuA/zz1MtWpVqVmzBmecfiItW3ei2V6HUqVKZS688CwAypVLpWbNGhzZ+XRuv+NhBr77UrFcT0KV5YrbzP7IbcvtoGbWz8w6mFmHvqccVfSlzkX1ynvQoVUTvp/1B3WqVyY1JYWUFHHWEQcwfWHQahs68VeO2n8vyqemUrtaZdq2aMSMhSt2OVadapVZGbaYV67dSO1qwUPOBjWrsuzPDdn5lv+1gXo1qtCgZlWW/xWZvjG7xZ5M2nU8hG4ndmX4hE94+uWHOfyoDjz+4v0sX7KCkV+MBmDa5JlkZmZSq07NnfZdvnQFjRo3yH7foHF9VixbBcDqlWuoWz/oY61bv052V8uypSto2GTHPg0b1WfF8pWJvMTdzuK0pTRr2jj7fdMmjVi6dPlOeS7rfT6ffPolAPPn/87vvy9iv31bcuyxXVjw+0JWrVpDeno6n3w6lCM6dcg+7qefDgVgwsQpZGZmUrdu7WK6qgQpy10lWSR1kjRB0gZJ2yRlSFpXHIWL15r1m1m3KXhCvmVbOuPnLKJFg1rZlS7A11Pn07JRUCk0qlWNn+amYWZs3rqdab8vo0WDXR+kHX1gCz77aRYAn/00i24H7R2kH9SCYZN+Zdv2DBavXsvClX9x4F4NqFejCpUrVmDqgmWYGZ9H7JNMnnvk/zi23emccFhPbrn6bsZ/P5E7rr2fUUO/5fDOQYWw197NKF++PH+u/munfadPnsWeezejyZ6NKF++HKf0OJ5vhgU/278ZNpYe558KQI/zT+Wbr7LSx3BKj+MpX6E8TfZsxJ57N2PapJnFeMUlb8LEKbRs2YLmzYP7et55Z/LZ58N3yrNw0WK6d+8MQP36dWndem9+W/AHixYu5vDDD6VSpaA7sPsxnZk9O3ioOXjIMI45JmhctWq1NxUqVGDVqjXFeGUJkJERe0sS8YwqeYFg0cwPgA5Ab6BlIguVX6vWbeSet0eQaUamGSe0bUXXA1tw14DhzFm8Cgka167O3ecfA8D5XQ/i3ndGcfZj74IZZ3RqQ+smdQF44N1RnNP5QA7YswF9jm/PbW98xSfjZtKoVjWeuvxkAFo2qsPx7Vpx1qNvk5qawp3nHk1qSvAdeNd53YLhgNvSOarNXnRus1fJ3JQS8MnAz3joubv59Nt32b5tO3f94wEA6jWoy4PP3sXfL7qJjIwMHrnzafq9919SUlP4ZOBnzJ+zAIBXn+/Ps688ylkXnsHSxcu4+Yp/ATB/zgK+GjKSIWPfIyM9g4fveKpMjSgByMjI4IYb7+bLL94lNSWFN/u/z8yZv3LVlZcA0O+Vt3jk0ed4/dV/M3nSSCRx512Psnr1n6xe/Scff/wFE34aRnp6OlOmzOCVV98B4I033+PVV55hyuRRbNu2nT59byzJyywaSdQlEovy6iuUNNHMOkiaamYHh2k/mNmR8Zxg87AXylZnZAno0Putki5C0pvzZ1pJF6FMSN+2uNCPhTYPuDNmnVOp92NJ8dgpnhb3JkkVgCmSngSWAsnXceucSw5J1CUSSzzjuC8BUoHrgI1AM+DsRBbKOecKrAyMKsmzxR0xgmQz8EBii+Occ4VjmcnfO5tnxS1pAew60NbMkm+4hHOu9EuilnUs8fRxR0ZfqgicC5TygZ7OuaSV7n3cmNnqiG2xmT0HdC+GsjnnXP6VgQk48XSVRIbWSyFogVdLWImcc64wysCokni6Sp6JeJ0OLADOS0xxnHOukLziBqCvme0Um1RSiwSVxznnCqcMjCqJZxz3h3GmOedcibP0jJhbXiRVlPSTpF8kzZD0QJheW9IISXPDv7Ui9rlT0jxJcySdGPvoRSdmi1vSfsABQA1JZ0V8VJ1gdIlzzu1+CtdVshXobmYbJJUHvpM0FDgLGGVmj0u6A7gDuF1SG4JYTgcAjYGRklqbWUL7a3LrKtmXIPZ2TXaOv70euDKRhXLOuQIrxOgRC4I3ZcVmLh9uBpwJdAvT+wOjgdvD9PfMbCuwQNI8oCPwY4ELEYeYFbeZDQYGSzrCzBJaCOecKzK5tLglXQVcFZHUz8z65ciTCvxMEAX1RTMbL6mBmS0FMLOlkrKWYGoCjIvYPS1MS6h4Hk7+TdIsM/sLIOzbecbM+iS2aM45l3+59WWHlXS/mBmCPBlAW0k1gU8kHZhL9mjRBhP+dDSeh5MHZ1XaAGb2J9AucUVyzrlCyLTYWz6E9d5o4CRguaRGAOHfrCWz0ggC72VpCiwhweKpuFNyPEGtTXwtdeecK36FWAFHUr2wpY2kSsBxwGxgCHBpmO1SYHD4eghwgaQ9wmHSrYCfiviKdhHvBJwfJGUNATwXiLIKrHPOlTxLL9TU9kZA/7CfOwUYZGafS/oRGCSpL7CQoB7EzGZIGgTMJJigeG2iR5RAfGFdB0iaSBCfRMBZZla2FvxzzpUehZiAY2ZTidIVbGargWNj7PMI8EiBT1oAcXV5hBX1TEn7AL0kDTKz3DrsnXOuZHh0wKAjXtKNkn4CZhCshtMr4SVzzrkCsIzMmFuyyG3m5JUEFXRTYBBwBTDYzPK1Cs7IS78vVAFd3qbMGFjSRUh6lRp3KekiuHiVgVgluXWVvEgw++dCM5sIICn574hzrlQr5MPJUiG3irsxwZPTZyU1IGh1ly+WUjnnXEGlJ3/7MmYft5mtMrP/mVlXgqepa4EVkmZJ8uGAzrndkmVazC1ZxDMBBzNLM7Onzaw90IMggpZzzu1+0i32liTyPQPSzOYA+XpA6ZxzxcWSqIKOxaeuO+eSSjJ1icTiFbdzLqlYekmXIPHi6uPOIun+BJXDOeeKhKXH3pJFvipu4IyElMI554pKZi5bkshvV0m0oOHOObfbyEyilnUs+a242yekFM45V0QsI/nbl/mquM0siX5sOOeSUVmopXxUiXMuqWSme4vbOedKlcwy0FWS31ElAEi6vKgL4pxzRcEyFXNLFgWquPEp78653VRmhmJuySK3hRSmxvoIaJCY4jjnXOEkUwUdS2593A2AE4E/c6QL+CFhJXLOuUJIpi6RWHKruD8HqprZlJwfSBqdsBI551whlOkWt5n1zeWzCxNTHOecK5yMzII+uis9fDigcy6plIWukqT4ajrk31dzwvSXOHr0k9lp1Q/Yi85fPEjXkY/RZdgj1Gy3DwB1ux5El2GPcPQ3T9Bl2CPUOeqAqMcsX7MKnd7/F8f88Cyd3v8X5WtUyf6s5fVn0v3Hf3PMd89Qr9vB2ek1Dm7B0d88Qfcf/80BD1+aoKstWRkZGZxz2bVcc+t9AAz7eixnXnQ1B3U+hemzft0l/9JlKzjsuJ688e6HUY+3dt16rrjhX5xyfl+uuOFfrF23PvuzVwa8z8nn9eG0C67g+/E/Z6fPmD2Xnpf8nZPP68Oj//4fZskffxngxBO6MWP6GGbP/I7bbr02ap6jux7BxAnD+WXK13w9csc9v/66vkyZPIpfpnzNP66/Ijv93ntu5o8FE5k4YTgTJwzn5JO6J/w6Eq0sjCpJiop70fvfMr7X4zultbnnQn595iPGHHcncwDcVbcAACAASURBVJ78gP3vCXp3tq1Zz0+9n+bbY25nyg3/o90L10Q9Zsvrz2TV2Ol8c+TNrBo7nZbXB4ERq7ZuQuMeRzD66FsZd+HjHPR4H0gJ/oM46Ik+TL3lVb4+4iaq7t2Q+t0PSeBVl4y3PxjM3s33zH7fcu+9eO7Re2jf9sCo+Z/4bz+6dOoQ83ivvjWITh3a8uX7r9GpQ1tee3sQAPMX/MHQUd8y+O2XeOnZh3no6RfIyMgA4KGnX+C+2//Bl++/xsK0JXw3bmIRXuHuKSUlhf/+5xFOO/1iDjrkGM4/vwf7799qpzw1alTn+ecfpedZl3FI2+6c3+tqAA44YF/69r2QI448lUPbH8+ppxxHy5Ytsvf7z39focNhJ9DhsBMY+tXXxXpdiZCRmRJzSxZJcSVrxs1m218bdkozM8pVqwRAuWqV2bIsGByzbvrvbF0evF4/O43UPcqTUmHXHqOGJ7Zn0aAxACwaNIaGJ3UI0zuw5NMfydyWzuaFK9m4YBm12rVkj/o1KV+1En/+PDfcZ2z2Psli2YqVjPnhJ84+/cTstH2a70mLvZpGzT9qzA80bdyQfVrsFfOY34z9kTNPPg6AM08+jq/H/AjA12PHcfKxR1OhQgWaNm7Ink0bM23Wr6xctYaNGzfR9sD9kcQZJx3L12N/LMKr3D11PKwd8+f/zoIFC9m+fTuDBg3mjIh/B4BeF/Tk00+HsmjREgBWrlwNwH77tWL8+Els3ryFjIwMxowdR48zTyr2ayguZrG3ZBGz4pY0TdLUKNu0XMZ47zZm3DuANvdcxHE/v0Cb+y5i9qPv7ZKn0WkdWTv9dzK37RoHco96Ndi64i8Atq74iwp1qwNQsVEtNi9ZnZ1vy9I1VGxUi4qNarN56ZqI9NVUbFS7qC+rRD3xn5e5+Zq+SHl/32/avIXX3/6Aa/pclGu+1X/+Rb26wX2qV7c2a/5aC8CKlatp2KBedr4G9euyYuUqlq9cRYP6dXek16vL8pWrSXaNmzRkUdqS7Pdpi5fSuHHDnfK0arU3NWvWYNSIDxg/bigXX3wOADNmzKZLl07Url2LSpUqcvJJ3WnatHH2ftf8/XIm/TyCV/o9Q82aNYrnghKorLe4TwNOj7Jlpcck6SpJEyVN/GrTvKIqa77sdenxzLjvLUa2v44Z973FIc9etdPnVfdtyv53X8jUW1/N34EVpZ/MiBqpPJn6Xkd/P57atWpywH6t8s4MvPjaW1xyfk8qV65UoPMZu947oejpydN1GZOiXGTO/77KlUul/aEHc/qZvTnl1Au5684badVqb2bPnsdTT73IV0MH8uXn7/DL1JlkpAfdTi+9PIDW+x1J+w4nsGzZCp568t5iuZ5EyshUzC0vkppJ+kbSLEkzJN0QpteWNELS3PBvrYh97pQ0T9IcSSfGPnrRyW044B8FPaiZ9QP6AXzWsFeJ1F7NzuvKjLv7A7B0yDgOeebK7M8qNqrNYa/fzOTr/49Nf6yIuv/WlWvZo35Ntq74iz3q12TbqnUAbFmyhkqN6+x0rC3L/gzSI1rYFRvVYeuynHOXSq/JU2cy+rtxjP1xAlu3bWfjxk3c/sCTPHHfbVHzT5sxhxHffMez//ca6zdsRBJ7VKjAhefsvIhSnVo1WblqDfXq1mblqjXUDlt8DerVZdnyldn5lq9YRb16dWhYrx7LV6zakb5yFfXr1iHZLU5bSrOIVnLTJo1YunT5znkWL2X16jVs2rSZTZs2M/a7cRx8cBvmzv2NN958jzfeDH51PvzQHaSlLQVgRcS9fPW1dxj8af9iuJrEMivUN3k68E8zmySpGvCzpBHAZcAoM3tc0h3AHcDtktoAFwAHAI2BkZJam1lGoS4iD3n+dpDUSdIESRskbZOUIWldIgtVFLYs+5M6R+4PQN3OB7Dxt2UAlKtemY5v38bsR9/jzwm7joLIsmz4zzQ7rysQfAksG/ZzdnrjHkeQUqEclfasR5W9G/Ln5HlsXfEX6Ru3UPPQluE+XbL3SQY3/f1yRn36NsM/6s9TD9xBx/aHxKy0AQb872mGf9Sf4R/15+LzenBl7/N3qbQBunXuxOChIwEYPHQkx3Q5AoBjOndi6Khv2bZtG2lLlrEwbQkH7d+aenVrU7lyJX6ZPgszY8hXozimc6fEXPRuZMLEKbRs2YLmzZtRvnx5zjvvTD77fPhOeYZ8NozORx1OamoqlSpVpGPHdsyeHTxzqVcv+HJr1qwxPXqczHvvfwpAw4b1s/fvcebJzJgxp5iuKHEyTDG3vJjZUjObFL5eD8wCmgBnAlnfav2BHuHrM4H3zGyrmS0A5gEdi/iSdhHPOO4XCL5RPgA6AL2BloksVH4d+r/rqXPk/lSoXY3jJr3AnKc+ZOotr3DAQ71RuVQyt27P7hJp0edEqrRoQKubetLqpp4AjLvgMbatWsfBz1zJHwNGsfaX35j3/BDa97uBZhd2Y/Pi1fx85XMAbJiTxtIh4+g25mksPYPpd74BmcGPiqm3v07b//yN1IoVWPH1FFaM2mXSadIZ+e33PPbv/7Hmr7Vcc+t97Ndqb/r9+5Fc97n3sec4r8cpHLh/a6645Dz+ec+jfPz5MBo1qMezD98FBKNVTuzehTMuuppyqancdfM1pKamAnDPLddx9yPPsmXrVrp0OowuRxyW8OssaRkZGdxw4918+cW7pKak8Gb/95k581euuvISAPq98hazZ89j2PBvmDxpJJmZmbz++sDsiviD91+hdp1abN+ezj/+cRd/hc8SHn/sbg45pA1mxh9/pPH3a24vsWssKrn1ZUu6CojsN+0X9hBEy9scaAeMBxqY2VIIKndJWd94TYBxEbulhWkJpbz6YSVNNLMOkqaa2cFh2g9mdmQ8JyiprpKy5KTpuVeUrvAqNe5S0kUoE9K3LS70E4uxDc+JWed0WfZhXMeXVBX4FnjEzD6W9JeZ1Yz4/E8zqyXpReBHM3s7TH8N+NLMPircVeQunhb3JkkVgCmSngSWAlXy2Mc550pEPF0iuZFUHvgIeMfMPg6Tl0tqFLa2GwFZD8fSgGYRuzcFlpBg8YyPuQRIBa4DNhIU8uxEFso55woqg5SYW14UDN95DZhlZs9GfDQEyJoOfSkwOCL9Akl7SGoBtAJ+KrKLiSHPFnfE6JLN+AIKzrndXCHXCj6KoLE6TVLWQ6p/AY8DgyT1BRYC5wKY2QxJg4CZBCNSrk30iBKIo+KWtAB2HTxrZnsnpETOOVcIGdEmVcTJzL4j6qwMAI6Nsc8jQLE+aIqnjzty3nZFgm+a5JoS6JxLGoWpuEuLPDt9zGx1xLbYzJ4DSn8IMedcUspU7C1ZxNNVcmjE2xSCFni1hJXIOecKoSy0uOPpKnkm4nU6sAA4LzHFcc65wkkvA8Fr4qm4+5rZb5EJ4bAX55zb7ZSFGX/xjOOOtnRJ9OVMnHOuhKUr9pYsYra4Je1HEPGqhqSzIj6qTjC6xDnndjtlvY97X4LY2zXZOf72euDKqHs451wJS6bRI7HkFo97MDBY0hFmlvxrQznnkkLCpy3uBuLp4/6bpMioWLUkvZ7AMjnnXIGV6T7uCAeb2V9Zb8zsT0ntElgm55wrsELGKikV4qm4UyTVMrM/IVh7Lc79nHOu2GUkUcs6lngn4PwgKWsI4LnAo4krknPOFVxZ6OOOJ6zrAEkTCeKTCDjLzGYmvGTOOVcAZXpUSaSwop4paR+gl6RBZnZgYovmnHP5l17SBSgG8azy3kjSjZJ+AmYQrIbTK+Elc865AshQ7C1ZxKy4JV0p6WuCBTPrAlcAS83sATObVlwFdM65/MjMZUsWuXWVvAj8CFxoZhMBJOU7fkvPNWMKWDQXN1+B3LlsGWUgzFRuFXdjghEkz0pqAAwCyhdLqZzbDaVvW1zSRXBxKAujSmJ2lZjZKjP7n5l1JVhrbS2wQtIsST4c0Dm3WyoLXSXxTHnHzNLM7Gkzaw/0ALYmtljOOVcw6bKYW7LI9wxIM5sDPJCAsjjnXKGVha4Sn7runEsqmWX84aRzzpU6ZaHFHVcfdxZJ9yeoHM45VyQysJhbsshXxQ2ckZBSOOdcESkLo0ry21WSRJNGnXPJKJla1rHkt+Jun5BSOOdcESkLFXe+ukrMLJl+bTjnklAmFnOLh6TXJa2QND0irbakEZLmhn9rRXx2p6R5kuZIOjEBl7SL/PZxO+fcbq0IHk6+CZyUI+0OYJSZtQJGhe+R1Aa4ADgg3Of/JKUWxXXkxitu51xSKWzFbWZjgDU5ks8E+oev+xPMIM9Kf8/MtprZAmAe0LHwV5G7XPu4Je0XFqwJYMASYIiZzUp0wZxzriBy68+VdBVwVURSPzPrF8dhG5jZUgAzWyqpfpjeBBgXkS8tTEuomBW3pNsJFkx4D/gpTG4KDJT0npk9nujCOedcfmVY7JZ1WEnHU1HHK9pIu4Q/Hc2txd0XOMDMtkcmSnqWYCUcr7idc7udjMSM2F4uqVHY2m4ErAjT04BmEfmaEvRMJFRufdyZBDG5c2pEco1ld84lkQRNwBkCXBq+vhQYHJF+gaQ9JLUAWrGjhyJhcmtx3wiMkjQXWBSm7Qm0BK5LdMGcc64gCtviljQQ6AbUlZQG3EfQwzBIUl9gIcEiM5jZDEmDgJkE6xRfa2YJD5ciy6U/SFIKwRPSJgR9OWnAhPwUrFyFJsk/Gt6VCb4CTrEo9OzsnnueHrPO+WThZ0kx+zvXUSXhhJtxueVxzrndSVkI65r047hPPKEbM6aPYfbM77jt1muj5jm66xFMnDCcX6Z8zdcjP8xOr1GjOu+/14/p075l2tTRdDo8mPH/wP23MunnEUycMJyhX7xLo0YNiuVadld53eOaNWvw4QevMunnEfz4/ecccMC+ee5bq1ZNvvpyILNmfMdXXw6kZs0axXItrvQrC9EBc+0qKQol2VWSkpLCrBljOemUXqSlLWXcj19y8SXXMGvW3Ow8NWpUZ+yYwZx62kUsWrSEevXqsHLlagBef+05vvtuPK+/MZDy5ctTuXIl1q5dR7VqVVm/fgMA113bh/33b821191RItdY0uK5x088djcbNm7koYf/zb777sPz/3mUE046P9d9H3/sLtas+Ysnn3qR2269llq1anDnv0p2qVPvKikWhe7KOLnZyTHrnKGLhiZFV0lSt7g7HtaO+fN/Z8GChWzfvp1BgwZzxuk7hxLodUFPPv10KIsWBSN4sirtatWq0qXz4bz+xkAAtm/fztq16wCyK22AKlUqk+gvv91ZPPd4//1b8/XX3wEwZ8589tqrKfXr181139NPP5EBb30AwIC3PuCMM3LOQHYuurIQ1jWpK+7GTRqyKG3HkMq0xUtp3LjhTnlatdqbmjVrMGrEB4wfN5SLLz4HgL333otVq1bz2qv/ZsJPw3j5paeoXLlS9n4PPXg7C+ZPoFevntz/wFPFc0G7oXju8dRpM+nZ4xQADuvQlr32akrTJo1y3bdB/bosWxYMlV22bAX169VJ9KW4JJFBZswtWcRVcUs6KoyI9auk3yQtkPRbogtXWNKuv4pyto7LlUul/aEHc/qZvTnl1Au5684badVqb8qlptKu3UG8/PIADut4Ihs3buL223aMgrzn3idosc9hDBz4Cddec3nCr2V3Fc89fuLJF6hZqwYTJwzn2mv7MHnKdNIzMuLa17n8yrDMmFuyiDce92vATcDPxLGkW2Q8AKXWICWlSoELWBiL05bSrOmOOURNmzRi6dLlO+dZvJTVq9ewadNmNm3azNjvxnHwwW347rvxpKUt5acJkwH4+OMvuO3WXYevD3zvE4YMHsADDz6T2IvZTcVzj9ev38AVV96c/X7er+NYsGAhlStVirnv8hWraNiwPsuWraBhw/qsCLuwnMuLJdFDyFji7SpZa2ZDzWyFma3O2mJlNrN+ZtbBzDqUVKUNMGHiFFq2bEHz5s0oX7485513Jp99PnynPEM+G0bnow4nNTWVSpUq0rFjO2bPnsvy5StJS1tC69b7ANC9e2dmzfoVgJYtW2Tvf/ppJzBnzvziu6jdTDz3uEaN6pQvXx6Avn0uZOx341m/fkOu+37+2XB6X3IuAL0vOZfPPhtWvBfmSq0Ms5hbsoi3xf2NpKeAj4GtWYlmNikhpSoiGRkZ3HDj3Xz5xbukpqTwZv/3mTnzV6668hIA+r3yFrNnz2PY8G+YPGkkmZmZvP76QGbMmAPADTfdw4D+z1OhQnkWLFhI3yuCVuOjj9xJ69b7kJmZycKFi7nm2rI5ogTiu8f779eKN17/DxmZGcya9StXXnVLrvsCPPHUi7z37ktcflkvFi1azPm9ri6xa3SlS3oS9WXHEtdwQEnfREk2M+ue174+c9IlCx8OWCwKPVyvU+NuMeuccUtGJ8VwwLha3GZ2TKIL4pxzRSGZRo/EElfFLakGQaCVrmHSt8CDZrY2UQVzzrmCSKbRI7HE+3DydWA9cF64rQPeSFShnHOuoMws5pYs4n04uY+ZnR3x/gFJUxJRIOecK4yy0FUSb4t7s6TOWW8kHQVsTkyRnHOu4HwCzg5/B/qHfd0iWAH5skQVyjnnCiozibpEYol3VMkU4BBJ1cP36xJaKuecK6BkalnHkmvFLeliM3tb0s050gEws2cTWDbnnMu3Ml9xA1nz1asluiDOOVcUykKskryWLns5/PtA8RTHOecKJyPxa/WWuHjDuj4pqbqk8pJGSVol6eJEF8455/KrLIwqiXc44AnhA8nTCFZ6bw3cmrBSOedcAfkEnB3Kh39PAQaa2ZpoQfCdc66kJVPLOpZ4K+7PJM0mmHRzjaR6wJbEFcs55wqmLFTcca/yLqkWsM7MMiRVBqqb2bK89vOwri5ZeFjXYlHon/J1q7eOWeesWvdrUnQV5DWOu7uZfS3prIi0yCwfJ6pgzjlXEBmZyd/izqur5Gjga+D0KJ8ZXnE753Yz3lVSBLyrxCUL7yopFoXuyqhauUXMOmfDpgVJ0VUS7zjuRyXVjHhfS9LDiSuWc84VjI/j3uFkM/sr642Z/UkwNNA553YrmZYZc0sW8Q4HTJW0h5ltBZBUCdgjccVyzrmCSaaJNrHEW3G/DYyS9AbBQ8k+QP94dkzftrjU9SlJusrM+pV0OZKZ3+PEK6v3eNvWtFJX5+RXfsZxnwQcR/DwYLiZDUtkwUqSpIlm1qGky5HM/B4nnt/j5BVvixtgFpBuZiMlVZZUzczWJ6pgzjnnoot3VMmVwIfAy2FSE+DTRBXKOedcbPGOKrkWOApYB2Bmc4H6iSrUbqDM9QuWAL/Hief3OEnF1cctabyZHS5pspm1k1QOmGRmBye+iM455yLF2+L+VtK/gEqSjgc+AD5LXLGcc87FEm+LW8AVwAkEo0qGAa9aWRgw6Zxzu5k8W9ySUoBpZvaKmZ1rZueErxNaaUt6XdIKSdMLuP9oSXMk/SLpe0n7FqIsl0l6IXz9N0m9c8nbXNKFBTjHm5LOKWgZC0LSSeE9mifpjgLs7/e4GEjKkDRF0nRJn0WGnyii4/8uqW74ekNRHtslRp4Vt5llAr9I2rMYyhPpTeCkQh7jIjM7hGCy0FM5P5SUmt8DmtlLZjYglyzNgXxXKsUtvPYXgZOBNkAvSW0KcCi/x4m32czamtmBwBqCwQKuDIu3j7sRMCNcKHhI1pbIgpnZGIL/SIvCGKAlBC0KSQ9KGg8cIeliST+FLZqXsyoaSZdL+lXStwQjagjT75d0S/i6paSRYYtzkqR9gMeBLuHxbpKUKukpSRMkTZV0dbivJL0gaaakLyj+UTodgXlm9puZbQPeA84sxPH8HhePHwmG4yJpH0lfSfpZ0lhJ+4XpDSR9Et6zXyQdGaZ/GuadIemqErwGV0jxTsB5IKGlSLzTgWnh6yrAdDO7V9L+wO3AUWa2XdL/ARdJGkFwze2BtcA3wOQox30HeNzMPpFUkeCL8A7gFjM7DYJpx8BaMztM0h7A95KGA+2AfYGDgAbATOD1RFx8DE2ARRHv04DDC3E8v8cJFn7hHQu8Fib1A/5mZnMlHQ78H9Ad+C/wrZn1DPepGubvE64XWwmYIOkjM1tdzJfhikBeK+BUBP5G0JKaBrxmZunFUbAi8o6kzcDvwPVhWgbwUfj6WIKKY4KClX0qASsIKrDRZrYSQNL7BCvbZ5NUDWhiZp8AmNmWMD1nGU4ADo7oW60BtAK6Eiy8nAEskfR1EVxvfkSL51CQ5xZ+jxOvkqQpBF1EPwMjJFUFjgQ+iLgfWYHfugO9AcJrXxum/0NSz/B1M4J75BV3KZRXi7s/sB0Yy46+0BsSXah4hC2Jn8O3Q8zs3ijZLjKziTnStoT/MUNQefU3sztzHLsHeVdi8QayEXB9ztgukk6J4xyJlEbwP2+WpsCSyAx+j3cbm82sraQawOcEfdxvAn+ZWdt4DiCpG0GsoSPMbJOk0UDFxBTXJVpefdxtzOxiM3sZOAfoUgxliouZZYQPbNrGqFDiMQo4R1J9AEm1Je0FjAe6SaojqTxwbpTzrwPSwgoISXsoWER5PVAtIusw4O/hcZDUWlIVgj7hC8L+2UbAMQW8hoKaALSS1EJSBeACYKfnFn6Pdy9mthb4B3ALsBlYIOlcyO7PPyTMOgr4e5ieKqk6wa+QP8NKez+gU7FfgCsyeVXc27NeFHcXiaSBBA9i9pWUJqlvUZ/DzGYCdwPDJU0FRgCNzGwpcH94/pHApBiHuITg5+dU4AegITAVSA8fCt0EvErQtzpJwdDGlwl+6XwCzCXogvof8G1RX19uwn/P6wgqvVnAIDObkYDzlNl7nAhmNhn4heCL9iKgr6RfgBnseLh8A3CMpGkEv5gOAL4CyoX38SFgXHGX3RWdXCfgSMoANma9Jeif3BS+NjOrnvASOuec20nCFwt2zjlXtOIdx+2cc2434RW3c86VMl5xO+dcKeMVt3POlTLFVnFL6inJsuIplFbhOOQRkuaGf2tFybOvgjgaWds6STeGn70fkf57OCMOSceHcSSmhX+7RxyvfZg+T9J/FWXqoHOu7Ci2USWSBhEEqxplZvcn8DypEbP2EnH8J4E1Zva4glCotczs9tzKAywGDjezP3J89gxBjI0HJbUDlpvZEkkHAsPMLCuY0E8EY3PHAV8C/zWzoQm5QOfcbq9YWtxhXIWjgL4EEwey0lMlPR22JqdKuj5MP0zSD+EEi58kVVNEvOYwz+fhNN5o0ejuVRApbrqkflktVEWJNCfpLUlnRhz3HUln5HI5ZxKEAiD82yOPyz8WmB+l0hZwHjAQgokVZpY15XwGUDGcKdgIqG5mP4Yx0AfEcU7nXBIrrq6SHsBXZvYrsEbSoWH6VUALoF24fuU74fTr94EbwjjPxxFM781NVjS6w83sO+AFMzssjF9cCTgtzPcO8GJ43COBpQSz7i4HUBAL4kjgS0lfSmoc5VwNwll/hH/zChV6AWHlnEMXghb23CifnQ1MNrOtBFH80iI+SwvTnHNlVHFV3L0I4j0T/u0Vvj4OeClrOr2ZrSEIw7nUzCaEaevimG4fGY0Ogum+48Mpv92BAxQl0pyZbTKzb4GWYSyNXsBHZpZuZqdEtIALJPwSOoNgjc6cehGlQpd0APAEcHVWUpR9fdaUc2VYvPG4C0xSHYLK80BJBqQCJuk2wqnzOXeJkgaQzs5fNJGRzbKj0SkIRft/QAczWyTp/jBvbg/03iKI+3AB0CePS1ouqZGZLQ27MVbkkvdkYJKZLY9MlFQOOIsg3GlkelOC+Bq9zWx+mJxGELkvyy5R/JxzZUtxtLjPAQaY2V5m1tzMmgELgM7AcOBvYUWGpNrAbKCxpMPCtGrh578DbSWlSGpGsIJLNFkV+qqwb/0cyDXSHAQhMm8M8+UVaGkIcGn4+lJgcC55o7aqCX5pzDaz7C4QBesIfgHcaWbfZ6WH3THrJXUK+8V753FO51ySK46KuxdBKzLSRwRrBr4KLASmhhHOLgyX0TofeD5MG0FQGX9PUOFPA54mRjQ5M/sLeCXM9ylB+NIs0SLNEbaIZwFvZGXMpY/7ceB4SXOB48P3SGos6cuI/SuHn38c5RjR+r2vI1iw4p6I4YJZ/ed/D+/VPGA+4CNKnCvDPMgU2ZXsNODQMOaxc87ttsr8zElJxxF0zzzvlbZzrjTwFrdzzpUyZb7F7ZxzpU1xzZzMyBG7o7mCtQa/CWc9vpDLvqdJmhzOdpwp6epYeRNJhY9R0lbSuDB9oqSOOfbdM7wXt0SkPSJpkaQNib9C51xpUSxdJZI2mFnVHGlVgHbAgcCBZnZdlP3KA38AHc0sTdIeQHMzm1OIsojgujPzuV+hYpRIGg7828yGKlh9/DYz6xaR/yMgExhvZk+HaZ0Irn9uzvvnnCu7SqyrxMw2htPTt+SSrRrBJKHV4T5bsyptSQ0kfRK2xH+RdGSYfnMYo2R6RGu3uaRZkv6PYBhhM0m3KohnMlXSA3EUubAxSgzIWqOzBhGTaMKx5b8RxCjJZmbjsqbXO+dcluKquCtFdB/kHNMdUzgFfgjwh6SBki6SlFXm/wLfhnFHDgVmSGpPEHfkcKATcKWCqHsQTKUfYGbtwtetCCbxtAXaS+oKuY7fLmyMkhuBpyQtIhiHfmd4virA7UA8Xx7OOVdsFfdmM2sbbj3zs6OZXUHQev0JuAV4PfyoO/C/ME9GOJSvM/BJ2JrfQDD5pUuY/w8zGxe+PiHcJhO0wPcjqMhJYIySvwM3hTNHbwJeC9MfIOhC8X5s51xcEh6rpCiY2TRgmqS3CGZPXhYja27xSDbmyPeYmb2cj2IUNkbJpQQxtSGo0F8NXx8OnBP2odcEMiVtMbOYD2ydc2Xbbj0cUFJVhTG3Q20JHtYBjCJoxWbF9a4OjAF6SKocdkH0BMZGOfQwoE8YywRJTSKml8dS2BglS4Cjw9fdgbkAZtYljOHSHHgO/pmFoAAAAKxJREFUeNQrbedcbkq04pb0O/AscJmkNEltcmYBbpM0R8ESXw+wo7V9A0H41mnAz8ABZjaJIGDUT8B44FUzm5zzvGY2HHgX+DHc/0OCB6GJjFFyJfBMGH/lUYJY5LnS/7dvh0gAAjEQBLOWV5/mnTwEg0OcZaluHxMxIlVJziTXzBzPftZuBvg/n5MAZT59KgHgTbgBygg3QBnhBigj3ABlhBugjHADlBFugDI3a9NfJx4GUeUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ds_utils.metrics import plot_confusion_matrix\n",
    "\n",
    "\n",
    "\n",
    "plot_confusion_matrix(y_test, predicted_labels, [1, 0])\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving the Fine-Tuned Model\n",
    "First let's save our model with the method ``save_pretrained``, then save our tokenizer with ``save_pretrained``.\n",
    "\n",
    "**Pay Attension**: tokenizer's ``save_pretrained`` need to get the path to save as a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('C:\\\\my repo\\\\Quora-Qusetions-Pairs-App\\\\training-bert\\\\model_save\\\\vocab.txt',\n",
       " 'C:\\\\my repo\\\\Quora-Qusetions-Pairs-App\\\\training-bert\\\\model_save\\\\special_tokens_map.json',\n",
       " 'C:\\\\my repo\\\\Quora-Qusetions-Pairs-App\\\\training-bert\\\\model_save\\\\added_tokens.json')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "\n",
    "\n",
    "output_dir = Path(\"__file__\").parents[0].absolute().joinpath(\"model_save\")\n",
    "output_dir.mkdir(exist_ok=True)\n",
    "\n",
    "# Save a trained model, configuration and tokenizer using `save_pretrained()`.\n",
    "# They can then be reloaded using `from_pretrained()`\n",
    "model_to_save = bert_model.module if hasattr(bert_model, 'module') else bert_model  # Take care of distributed/parallel training\n",
    "model_to_save.save_pretrained(output_dir)\n",
    "tokenizer.save_pretrained(str(output_dir.absolute()))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}