Multiple Choice/CosmosQA/CosmosQA - Multiple Choice Using BERT.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CosmosQA: Multiple Choice\n",
    "\n",
    "Dataset Source: https://www.kaggle.com/datasets/zjjc123/cosmosqa"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Git LFS initialized.\n"
     ]
    }
   ],
   "source": [
    "import os, sys, random, ast\n",
    "os.environ['TOKENIZERS_PARALLELISM']='false'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import datasets\n",
    "from datasets import Dataset, DatasetDict\n",
    "\n",
    "import torch\n",
    "\n",
    "from dataclasses import dataclass\n",
    "\n",
    "import transformers\n",
    "from transformers import AutoTokenizer, AutoModelForMultipleChoice\n",
    "from transformers import Trainer, TrainingArguments, set_seed\n",
    "from transformers.tokenization_utils_base import PreTrainedTokenizerBase, PaddingStrategy\n",
    "\n",
    "from typing import Optional, Union\n",
    "\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "\n",
    "!git lfs install"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Library Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        Python: 3.9.7 \n",
      "         NumPy: 1.23.3\n",
      "        Pandas: 1.4.4\n",
      "         Torch: 1.12.1\n",
      "      Datasets: 2.8.0\n",
      "  Transformers: 4.26.1\n"
     ]
    }
   ],
   "source": [
    "print(\"Python:\".rjust(15), sys.version[0:6])\n",
    "print(\"NumPy:\".rjust(15), np.__version__)\n",
    "print(\"Pandas:\".rjust(15), pd.__version__)\n",
    "print(\"Torch:\".rjust(15), torch.__version__)\n",
    "print(\"Datasets:\".rjust(15), datasets.__version__)\n",
    "print(\"Transformers:\".rjust(15), transformers.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load & Process Training Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>answer0</th>\n",
       "      <th>answer1</th>\n",
       "      <th>answer2</th>\n",
       "      <th>answer3</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Good Old War and person L : I saw both of these bands Wednesday night , and they both blew me away . seriously . Good Old War is acoustic and makes me smile . I really can not help but be happy when I listen to them ; I think it 's the fact that they seemed so happy themselves when they played .</td>\n",
       "      <td>In the future , will this person go to see other bands play ?</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>This person likes music and likes to see the show , they will see other bands play .</td>\n",
       "      <td>This person only likes Good Old War and Person L , no other bands .</td>\n",
       "      <td>Other Bands is not on tour and this person can not see them .</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I mean it : not one person said ANYTHING to me . They would have if I did something wrong , right ? But , this morning , I got a call from my guy at the temp agency and he said that I was no longer needed in that position , that I did n't need to go out there .</td>\n",
       "      <td>Why might have the temp agency tell me I am not needed at that position ?</td>\n",
       "      <td>The company hiring the temp workers might have had a change of mind .</td>\n",
       "      <td>The temp agency hiring the temp workers might have had a change of mind .</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>I might have had a change of mind working for the company .</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Leaving my shift Thursday day shift I arrived the same time as my partner just after six that evening and before long the radio erupted in dispatch tones . A car fleeing the police has crashed and landed on its roof with four separate people entrapped inside . Our medic unit is dispatched along with multiple other ambulances and Rescue Companies .</td>\n",
       "      <td>What may have caused the radio to erupt with dispatch tones ?</td>\n",
       "      <td>My partner needed a medic unit .</td>\n",
       "      <td>Someone was running from the ambulances after they got into a wreck .</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>Someone was running from the cops and got into a wreck .</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>So , last day in Seattle , and my flight was at 1:30 . I got to chit chat with my old manager ( more like a mentor ) , and left Seattle feeling really good and inspired . .</td>\n",
       "      <td>Why did I chit chat with my old manager ?</td>\n",
       "      <td>Because my flight was at 1:30 .</td>\n",
       "      <td>Because I left Seattle feeling really good and inspired .</td>\n",
       "      <td>Because it 's my last day in Seattle .</td>\n",
       "      <td>Because I enjoy talking to him .</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ms. Mumma informed Mr. Dail that evidence had been found in his case . \" I fell out of my chair and burst into tears . Evidence meant I was going home , and I knew that , \" said Mr. Dail . He immediately told Ms. Mumma , \" Test anything and everything you can find .</td>\n",
       "      <td>Why did I burst into tears ?</td>\n",
       "      <td>Because Ms. Mumma was informative .</td>\n",
       "      <td>Because I had an intense emotional response to the new development .</td>\n",
       "      <td>Because Mr. Dail was informed .</td>\n",
       "      <td>Because I fell out of my chair .</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25257</th>\n",
       "      <td>My dad in old age ... he would have been wise in some matters . Important matters of the day to day . Wise in his right - brained blueness . Wise as each white hair in his beard .</td>\n",
       "      <td>What may be a fact about their dad ?</td>\n",
       "      <td>He 's a different person now .</td>\n",
       "      <td>He died .</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>He does n't talk to them anymore .</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25258</th>\n",
       "      <td>As the house was in the higher part of the town we had the view on the lake from our window :) Again , we got ready and went to the same place where we had the \" Vin d'honneur \" to again have some drinks and snacks with other guests . Gave us some strength before we hit the road . I must say I was n't too chatty ( am I ever ?</td>\n",
       "      <td>Why might we have gone back to the same place we had the \" Vin d'honneur \" .</td>\n",
       "      <td>It 's a place I do n't have to be chatty because it 's loud .</td>\n",
       "      <td>I liked the restaurant by the river and wanted to go back .</td>\n",
       "      <td>We enjoyed the restaurant the first time around so we returned .</td>\n",
       "      <td>I like the food and it helps me sleep when we hit the road .</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25259</th>\n",
       "      <td>Then there 's is my baby pitbull . Not to play favorite , but he has managed to find a deeper spot in my heart . He is chocolate and carmel swirled I 'd like to say . He 's never been loud , or obnoxious , or too rowdy , or too lazy .</td>\n",
       "      <td>What is the narrator describing ?</td>\n",
       "      <td>They 're talking their cat .</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>They 're talking their pet .</td>\n",
       "      <td>They 're talking their lazy cat .</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25260</th>\n",
       "      <td>Then there 's is my baby pitbull . Not to play favorite , but he has managed to find a deeper spot in my heart . He is chocolate and carmel swirled I 'd like to say . He 's never been loud , or obnoxious , or too rowdy , or too lazy .</td>\n",
       "      <td>What is the narrator describing ?</td>\n",
       "      <td>They 're talking their cat .</td>\n",
       "      <td>They 're talking about their dog .</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>They 're talking their lazy cat .</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25261</th>\n",
       "      <td>The good news : Michael came to visit me yesterday . We went cruising north of here . Then we went to the gym . Then we went walking around once it stopped raining . The bad news : I did n't get pictures .</td>\n",
       "      <td>What happened after Michael came to visit ?</td>\n",
       "      <td>We hung out the entire day .</td>\n",
       "      <td>We made sure to get lots of pictures .</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>We watched movies at home .</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>25262 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                             context  \\\n",
       "0                                                           Good Old War and person L : I saw both of these bands Wednesday night , and they both blew me away . seriously . Good Old War is acoustic and makes me smile . I really can not help but be happy when I listen to them ; I think it 's the fact that they seemed so happy themselves when they played .   \n",
       "1                                                                                              I mean it : not one person said ANYTHING to me . They would have if I did something wrong , right ? But , this morning , I got a call from my guy at the temp agency and he said that I was no longer needed in that position , that I did n't need to go out there .   \n",
       "2      Leaving my shift Thursday day shift I arrived the same time as my partner just after six that evening and before long the radio erupted in dispatch tones . A car fleeing the police has crashed and landed on its roof with four separate people entrapped inside . Our medic unit is dispatched along with multiple other ambulances and Rescue Companies .   \n",
       "3                                                                                                                                                                                       So , last day in Seattle , and my flight was at 1:30 . I got to chit chat with my old manager ( more like a mentor ) , and left Seattle feeling really good and inspired . .   \n",
       "4                                                                                         Ms. Mumma informed Mr. Dail that evidence had been found in his case . \" I fell out of my chair and burst into tears . Evidence meant I was going home , and I knew that , \" said Mr. Dail . He immediately told Ms. Mumma , \" Test anything and everything you can find .   \n",
       "...                                                                                                                                                                                                                                                                                                                                                              ...   \n",
       "25257                                                                                                                                                                            My dad in old age ... he would have been wise in some matters . Important matters of the day to day . Wise in his right - brained blueness . Wise as each white hair in his beard .   \n",
       "25258                        As the house was in the higher part of the town we had the view on the lake from our window :) Again , we got ready and went to the same place where we had the \" Vin d'honneur \" to again have some drinks and snacks with other guests . Gave us some strength before we hit the road . I must say I was n't too chatty ( am I ever ?   \n",
       "25259                                                                                                                     Then there 's is my baby pitbull . Not to play favorite , but he has managed to find a deeper spot in my heart . He is chocolate and carmel swirled I 'd like to say . He 's never been loud , or obnoxious , or too rowdy , or too lazy .   \n",
       "25260                                                                                                                     Then there 's is my baby pitbull . Not to play favorite , but he has managed to find a deeper spot in my heart . He is chocolate and carmel swirled I 'd like to say . He 's never been loud , or obnoxious , or too rowdy , or too lazy .   \n",
       "25261                                                                                                                                                  The good news : Michael came to visit me yesterday . We went cruising north of here . Then we went to the gym . Then we went walking around once it stopped raining . The bad news : I did n't get pictures .   \n",
       "\n",
       "                                                                           question  \\\n",
       "0                     In the future , will this person go to see other bands play ?   \n",
       "1         Why might have the temp agency tell me I am not needed at that position ?   \n",
       "2                     What may have caused the radio to erupt with dispatch tones ?   \n",
       "3                                         Why did I chit chat with my old manager ?   \n",
       "4                                                      Why did I burst into tears ?   \n",
       "...                                                                             ...   \n",
       "25257                                          What may be a fact about their dad ?   \n",
       "25258  Why might we have gone back to the same place we had the \" Vin d'honneur \" .   \n",
       "25259                                             What is the narrator describing ?   \n",
       "25260                                             What is the narrator describing ?   \n",
       "25261                                   What happened after Michael came to visit ?   \n",
       "\n",
       "                                                                     answer0  \\\n",
       "0                                                None of the above choices .   \n",
       "1      The company hiring the temp workers might have had a change of mind .   \n",
       "2                                           My partner needed a medic unit .   \n",
       "3                                            Because my flight was at 1:30 .   \n",
       "4                                        Because Ms. Mumma was informative .   \n",
       "...                                                                      ...   \n",
       "25257                                         He 's a different person now .   \n",
       "25258          It 's a place I do n't have to be chatty because it 's loud .   \n",
       "25259                                           They 're talking their cat .   \n",
       "25260                                           They 're talking their cat .   \n",
       "25261                                           We hung out the entire day .   \n",
       "\n",
       "                                                                                    answer1  \\\n",
       "0      This person likes music and likes to see the show , they will see other bands play .   \n",
       "1                 The temp agency hiring the temp workers might have had a change of mind .   \n",
       "2                     Someone was running from the ambulances after they got into a wreck .   \n",
       "3                                 Because I left Seattle feeling really good and inspired .   \n",
       "4                      Because I had an intense emotional response to the new development .   \n",
       "...                                                                                     ...   \n",
       "25257                                                                             He died .   \n",
       "25258                           I liked the restaurant by the river and wanted to go back .   \n",
       "25259                                                           None of the above choices .   \n",
       "25260                                                    They 're talking about their dog .   \n",
       "25261                                                We made sure to get lots of pictures .   \n",
       "\n",
       "                                                                   answer2  \\\n",
       "0      This person only likes Good Old War and Person L , no other bands .   \n",
       "1                                              None of the above choices .   \n",
       "2                                              None of the above choices .   \n",
       "3                                   Because it 's my last day in Seattle .   \n",
       "4                                          Because Mr. Dail was informed .   \n",
       "...                                                                    ...   \n",
       "25257                                          None of the above choices .   \n",
       "25258     We enjoyed the restaurant the first time around so we returned .   \n",
       "25259                                         They 're talking their pet .   \n",
       "25260                                          None of the above choices .   \n",
       "25261                                          None of the above choices .   \n",
       "\n",
       "                                                             answer3  label  \n",
       "0      Other Bands is not on tour and this person can not see them .      1  \n",
       "1        I might have had a change of mind working for the company .      0  \n",
       "2           Someone was running from the cops and got into a wreck .      3  \n",
       "3                                   Because I enjoy talking to him .      3  \n",
       "4                                   Because I fell out of my chair .      1  \n",
       "...                                                              ...    ...  \n",
       "25257                             He does n't talk to them anymore .      1  \n",
       "25258   I like the food and it helps me sleep when we hit the road .      2  \n",
       "25259                              They 're talking their lazy cat .      2  \n",
       "25260                              They 're talking their lazy cat .      1  \n",
       "25261                                    We watched movies at home .      0  \n",
       "\n",
       "[25262 rows x 7 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parent_dir = \"/Users/leedunn/Desktop/Projects_to_Train/MC\"\n",
    "\n",
    "training_file_location = os.path.join(parent_dir, \"train.csv\")\n",
    "\n",
    "train_ds = pd.read_csv(training_file_location)\n",
    "\n",
    "train_ds = train_ds.drop(columns=[\"id\"])\n",
    "\n",
    "train_ds.reset_index(drop=True, inplace=True)\n",
    "\n",
    "train_ds"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load & Process Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>answer0</th>\n",
       "      <th>answer1</th>\n",
       "      <th>answer2</th>\n",
       "      <th>answer3</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Do i need to go for a legal divorce ? I wanted to marry a woman but she is not in the same religion , so i am not concern of the marriage inside church . I will do the marriage registered with the girl who i am going to get married . But legally will there be any complication , like if the other woman comes back one day , will the girl who i am going to get married now will be in trouble or Is there any complication ?</td>\n",
       "      <td>Why is this person asking about divorce ?</td>\n",
       "      <td>If he gets married in the church he wo nt have to get a divorce .</td>\n",
       "      <td>He wants to get married to a different person .</td>\n",
       "      <td>He wants to know if he does nt like this girl can he divorce her ?</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Do i need to go for a legal divorce ? I wanted to marry a woman but she is not in the same religion , so i am not concern of the marriage inside church . I will do the marriage registered with the girl who i am going to get married . But legally will there be any complication , like if the other woman comes back one day , will the girl who i am going to get married now will be in trouble or Is there any complication ?</td>\n",
       "      <td>Why is he worried about getting married ?</td>\n",
       "      <td>He was married before and she might come back one day .</td>\n",
       "      <td>He wants the girl he is going to marry get in trouble .</td>\n",
       "      <td>He knows that he will be committing polygamy .</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I was walking home from the store , when I saw an old man laying on the sidewalk , bleeding . The right side of his face was all covered in blood . He was conscious but seemed dazed and probably intoxicated . Nearby there was a young man dialing his cell phone .</td>\n",
       "      <td>What may happen after the young man makes his call ?</td>\n",
       "      <td>An ambulance would likely come to the scene</td>\n",
       "      <td>The taxi would pick up the young man</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>The bus would arrive at the stop soon</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>I was walking home from the store , when I saw an old man laying on the sidewalk , bleeding . The right side of his face was all covered in blood . He was conscious but seemed dazed and probably intoxicated . Nearby there was a young man dialing his cell phone .</td>\n",
       "      <td>What may happen after the young man makes his call ?</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>The bus would arrive at the stop soon</td>\n",
       "      <td>The taxi would pick up the young man</td>\n",
       "      <td>Medical personnel would come to help the old man</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I was walking home from the store , when I saw an old man laying on the sidewalk , bleeding . The right side of his face was all covered in blood . He was conscious but seemed dazed and probably intoxicated . Nearby there was a young man dialing his cell phone .</td>\n",
       "      <td>What may have happened to the old man ?</td>\n",
       "      <td>He was waiting on a ride</td>\n",
       "      <td>He likely fell on the sidewalk and hit his head while intoxicated</td>\n",
       "      <td>He was waiting on the taxi</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2980</th>\n",
       "      <td>Not going in but the antibiotic they put into the IV just kill my arm . And after the procedure was over , I was throwing up all over again . I laid in bed the rest of the afternoon with DH and felt pretty decent by tonight .</td>\n",
       "      <td>What 's a possible reason the writer was throwing up ?</td>\n",
       "      <td>They felt pretty decent by the night .</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>They were not going in .</td>\n",
       "      <td>They have an illness .</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2981</th>\n",
       "      <td>This morning was my appointment with the psychiatrist . And may I give Mike another little pat ? He had to get up at 4:25 a.m. yesterday to go into work early to accumulate hours for this day off , and then he stayed an hour and a half late , too , not getting home until 7:00 , and he 's going to have to do that twice more . That 's a high price he 's paying . Anyway , it was worth it .</td>\n",
       "      <td>What 's a possible reason the writer has an appointment with a psychiatrist ?</td>\n",
       "      <td>Because the writer may give Mike another little pat .</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>Because it 's in the morning .</td>\n",
       "      <td>Because Mike had to get up at 4:25 a.m. yesterday .</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2982</th>\n",
       "      <td>Those who do n't pass this training will have to give up their jetpack . Fortunately they 'll get their money back . Each jetpack is equipped with a ballistic parachute , and though it 's more dangerous than most other current flying options , Glenn Martin hopes to sell between 10 and 20 devices by this time next year .</td>\n",
       "      <td>What may happen if you do n't pass your training ?</td>\n",
       "      <td>I would n't be able to keep my equipment .</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>You 'd have to choose other flying options .</td>\n",
       "      <td>You would be given a ballistic parachute .</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2983</th>\n",
       "      <td>She had her first field practice with her new team last night . It looks like they are going to do really well . The coach knows his stuff . Yesterday was the single most craziest busiest day of the entire summer , and I was in mellow contentment .</td>\n",
       "      <td>What is the narrator 's general outlook at the moment ?</td>\n",
       "      <td>Optimistic</td>\n",
       "      <td>Pensive</td>\n",
       "      <td>Observant</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2984</th>\n",
       "      <td>She had her first field practice with her new team last night . It looks like they are going to do really well . The coach knows his stuff . Yesterday was the single most craziest busiest day of the entire summer , and I was in mellow contentment .</td>\n",
       "      <td>What is the narrator 's general outlook at the moment ?</td>\n",
       "      <td>Happy</td>\n",
       "      <td>Pensive</td>\n",
       "      <td>Observant</td>\n",
       "      <td>None of the above choices .</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2985 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                    context  \\\n",
       "0     Do i need to go for a legal divorce ? I wanted to marry a woman but she is not in the same religion , so i am not concern of the marriage inside church . I will do the marriage registered with the girl who i am going to get married . But legally will there be any complication , like if the other woman comes back one day , will the girl who i am going to get married now will be in trouble or Is there any complication ?   \n",
       "1     Do i need to go for a legal divorce ? I wanted to marry a woman but she is not in the same religion , so i am not concern of the marriage inside church . I will do the marriage registered with the girl who i am going to get married . But legally will there be any complication , like if the other woman comes back one day , will the girl who i am going to get married now will be in trouble or Is there any complication ?   \n",
       "2                                                                                                                                                                    I was walking home from the store , when I saw an old man laying on the sidewalk , bleeding . The right side of his face was all covered in blood . He was conscious but seemed dazed and probably intoxicated . Nearby there was a young man dialing his cell phone .   \n",
       "3                                                                                                                                                                    I was walking home from the store , when I saw an old man laying on the sidewalk , bleeding . The right side of his face was all covered in blood . He was conscious but seemed dazed and probably intoxicated . Nearby there was a young man dialing his cell phone .   \n",
       "4                                                                                                                                                                    I was walking home from the store , when I saw an old man laying on the sidewalk , bleeding . The right side of his face was all covered in blood . He was conscious but seemed dazed and probably intoxicated . Nearby there was a young man dialing his cell phone .   \n",
       "...                                                                                                                                                                                                                                                                                                                                                                                                                                     ...   \n",
       "2980                                                                                                                                                                                                      Not going in but the antibiotic they put into the IV just kill my arm . And after the procedure was over , I was throwing up all over again . I laid in bed the rest of the afternoon with DH and felt pretty decent by tonight .   \n",
       "2981                                  This morning was my appointment with the psychiatrist . And may I give Mike another little pat ? He had to get up at 4:25 a.m. yesterday to go into work early to accumulate hours for this day off , and then he stayed an hour and a half late , too , not getting home until 7:00 , and he 's going to have to do that twice more . That 's a high price he 's paying . Anyway , it was worth it .   \n",
       "2982                                                                                                      Those who do n't pass this training will have to give up their jetpack . Fortunately they 'll get their money back . Each jetpack is equipped with a ballistic parachute , and though it 's more dangerous than most other current flying options , Glenn Martin hopes to sell between 10 and 20 devices by this time next year .   \n",
       "2983                                                                                                                                                                               She had her first field practice with her new team last night . It looks like they are going to do really well . The coach knows his stuff . Yesterday was the single most craziest busiest day of the entire summer , and I was in mellow contentment .   \n",
       "2984                                                                                                                                                                               She had her first field practice with her new team last night . It looks like they are going to do really well . The coach knows his stuff . Yesterday was the single most craziest busiest day of the entire summer , and I was in mellow contentment .   \n",
       "\n",
       "                                                                           question  \\\n",
       "0                                         Why is this person asking about divorce ?   \n",
       "1                                         Why is he worried about getting married ?   \n",
       "2                              What may happen after the young man makes his call ?   \n",
       "3                              What may happen after the young man makes his call ?   \n",
       "4                                           What may have happened to the old man ?   \n",
       "...                                                                             ...   \n",
       "2980                         What 's a possible reason the writer was throwing up ?   \n",
       "2981  What 's a possible reason the writer has an appointment with a psychiatrist ?   \n",
       "2982                             What may happen if you do n't pass your training ?   \n",
       "2983                        What is the narrator 's general outlook at the moment ?   \n",
       "2984                        What is the narrator 's general outlook at the moment ?   \n",
       "\n",
       "                                                                answer0  \\\n",
       "0     If he gets married in the church he wo nt have to get a divorce .   \n",
       "1               He was married before and she might come back one day .   \n",
       "2                           An ambulance would likely come to the scene   \n",
       "3                                           None of the above choices .   \n",
       "4                                              He was waiting on a ride   \n",
       "...                                                                 ...   \n",
       "2980                             They felt pretty decent by the night .   \n",
       "2981              Because the writer may give Mike another little pat .   \n",
       "2982                         I would n't be able to keep my equipment .   \n",
       "2983                                                         Optimistic   \n",
       "2984                                                              Happy   \n",
       "\n",
       "                                                                answer1  \\\n",
       "0                       He wants to get married to a different person .   \n",
       "1               He wants the girl he is going to marry get in trouble .   \n",
       "2                                  The taxi would pick up the young man   \n",
       "3                                 The bus would arrive at the stop soon   \n",
       "4     He likely fell on the sidewalk and hit his head while intoxicated   \n",
       "...                                                                 ...   \n",
       "2980                                        None of the above choices .   \n",
       "2981                                        None of the above choices .   \n",
       "2982                                        None of the above choices .   \n",
       "2983                                                            Pensive   \n",
       "2984                                                            Pensive   \n",
       "\n",
       "                                                                 answer2  \\\n",
       "0     He wants to know if he does nt like this girl can he divorce her ?   \n",
       "1                         He knows that he will be committing polygamy .   \n",
       "2                                            None of the above choices .   \n",
       "3                                   The taxi would pick up the young man   \n",
       "4                                             He was waiting on the taxi   \n",
       "...                                                                  ...   \n",
       "2980                                            They were not going in .   \n",
       "2981                                      Because it 's in the morning .   \n",
       "2982                        You 'd have to choose other flying options .   \n",
       "2983                                                           Observant   \n",
       "2984                                                           Observant   \n",
       "\n",
       "                                                  answer3  label  \n",
       "0                             None of the above choices .      1  \n",
       "1                             None of the above choices .      0  \n",
       "2                   The bus would arrive at the stop soon      0  \n",
       "3        Medical personnel would come to help the old man      3  \n",
       "4                             None of the above choices .      1  \n",
       "...                                                   ...    ...  \n",
       "2980                               They have an illness .      3  \n",
       "2981  Because Mike had to get up at 4:25 a.m. yesterday .      1  \n",
       "2982           You would be given a ballistic parachute .      0  \n",
       "2983                          None of the above choices .      0  \n",
       "2984                          None of the above choices .      0  \n",
       "\n",
       "[2985 rows x 7 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_file_location = os.path.join(parent_dir, \"valid.csv\")\n",
    "\n",
    "eval_ds = pd.read_csv(eval_file_location)\n",
    "\n",
    "eval_ds = eval_ds.drop(columns=[\"id\"])\n",
    "\n",
    "eval_ds.reset_index(drop=True, inplace=True)\n",
    "\n",
    "eval_ds"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visual Question Length (in Words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABccAAAK9CAYAAADhdNQKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABhpklEQVR4nO3debhd490//vfJdBJJzomQQZpRoghCBZEihBAEVTyGloZSbcWslKetIR0ENbam8jzSR6NKa04NaSShxBQNGkPRKC2JMYMgiWT9/vDL/joyyCFxwnq9rmtfOfte91rrs9a+z7p45869q4qiKAIAAAAAACXSqKELAAAAAACAz5pwHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHADgc6x79+45+OCDG7oMVgGnn356qqqq8vrrr6/0c91xxx3ZZJNN0rx581RVVWXGjBkr/ZxLc/DBB6d79+713q8hf3c+y88KAIClE44DAKwiRo4cmaqqqjzyyCNL3L7ddttlww03/NTn+fOf/5zTTz/9Ux/ni+iFF15IVVVVfvnLXzZ0KUv1i1/8IjfddFODnf+NN97IvvvumxYtWuTiiy/O1VdfnZYtW37sfpdcckmqqqrSr1+/ep/z5Zdfzumnn57Jkyd/goo/e1tssUWqqqpy6aWXrvRzrQr3xjMFAPi8Eo4DAHyOPfPMM7niiivqtc+f//znnHHGGSupIla2hg7HH3744cyePTs//elPc+ihh+bAAw9M06ZNP3a/UaNGpXv37nnooYfy3HPP1eucL7/8cs4444wlBsBXXHFFnnnmmXodb2V69tln8/DDD6d79+4ZNWrUSj/fsu7NZ8UzBQD4vBKOAwB8jlVXVy9XMLkqmTNnTkOXwKfw6quvJknatGmz3PtMnTo1999/f84777y0a9duuUPj999/P/PmzVtmn6ZNm6a6unq5a1nZfve736V9+/Y599xzc//99+eFF15o6JLqeOeddxq6BACAVYZwHADgc+yj6ybPnz8/Z5xxRtZZZ500b948a6yxRrbeeuuMGTMmyQfrM1988cVJkqqqqsprkTlz5uSEE05Ily5dUl1dnXXXXTe//OUvUxRFnfO+++67Ofroo7PmmmumdevW2WOPPfKf//wnVVVVdZZXWLS28pNPPplvfOMbWX311bP11lsnSR5//PEcfPDBWXvttdO8efN07Ngx3/72t/PGG2/UOdeiY/zjH//IgQcemNra2rRr1y4/+clPUhRFXnrppXzta19LTU1NOnbsmHPPPXex+/Tiiy/m6aef/lT3+sPmzp2b0047Lb169Up1dXW6dOmSk046KXPnzq3Tr6qqKkceeWRuuummbLjhhqmurs4GG2yQO+64Y7Fjjh8/PptttlmaN2+enj175vLLL69c+4ePN2fOnPz2t7+tfHYfXTd7xowZOfjgg9OmTZvU1tbmkEMOWe5A9Prrr0/fvn3TokWLrLnmmjnwwAPzn//8p7J9u+22y9ChQ5Mkm2+++RLPvySjRo3K6quvniFDhmSfffZZYjj+4SVtLrjggvTs2TPV1dW55JJLsvnmmydJDjnkkMp1jxw5MsmS1xxfuHBhLrzwwmy00UZp3rx52rVrl5133nmpSxYtMmPGjBx77LGV8d+rV6+cddZZWbhw4cde4yLXXHNN9tlnn+y2226pra3NNddcs9S+r7/+evbdd9/U1NRkjTXWyDHHHJP33nuvTp8xY8Zk6623Tps2bdKqVausu+66+e///u8kH4yZZd2bRUsxTZo0KQMGDMhqq61W2ffmm2/OkCFD0qlTp1RXV6dnz5756U9/mgULFixW54MPPphdd901q6++elq2bJk+ffrkwgsvTPLxz5Rrr702ffv2TevWrVNTU5ONNtqosi8AQENr0tAFAABQ18yZM5f4RX3z58//2H1PP/30nHnmmTnssMOyxRZbZNasWXnkkUfy6KOPZscdd8x3v/vdvPzyyxkzZkyuvvrqOvsWRZE99tgj48aNy6GHHppNNtkkd955Z0488cT85z//yfnnn1/pe/DBB+e6667LQQcdlC233DITJkzIkCFDllrXf/3Xf2WdddbJL37xi0rQPmbMmPzzn//MIYccko4dO2bKlCn5zW9+kylTpuSBBx6oE7AlyX777Zf1118/I0aMyOjRo/Ozn/0sbdu2zeWXX57tt98+Z511VkaNGpUf/OAH2XzzzTNgwIDKvt/61rcyYcKExUL+T2LhwoXZY4898te//jWHH3541l9//TzxxBM5//zz849//GOxJU/++te/5oYbbsgRRxyR1q1b56KLLsree++dF198MWussUaS5G9/+1t23nnnrLXWWjnjjDOyYMGCDB8+PO3atatzrKuvvrry2R5++OFJkp49e9bps++++6ZHjx4588wz8+ijj+bKK69M+/btc9ZZZy3zukaOHJlDDjkkm2++ec4888xMnz49F154Ye6777787W9/S5s2bfKjH/0o6667bn7zm99k+PDh6dGjx2LnX5JRo0Zlr732SrNmzXLAAQfk0ksvzcMPP1wJdj/sqquuynvvvZfDDz881dXV+frXv57Zs2fn1FNPzeGHH55tttkmSfLVr351qec79NBDM3LkyOyyyy457LDD8v777+fee+/NAw88kM0222yJ+7zzzjvZdttt85///Cff/e5307Vr19x///055ZRT8sorr+SCCy742Ot88MEH89xzz+Wqq65Ks2bNstdee2XUqFGVQPqj9t1333Tv3j1nnnlmHnjggVx00UV566238n//939JkilTpmS33XZLnz59Mnz48FRXV+e5557LfffdlyRZf/31M3z48GXemzfeeCO77LJL9t9//xx44IHp0KFDkg8+71atWuX4449Pq1atcvfdd+fUU0/NrFmzcs4551T2HzNmTHbbbbestdZaOeaYY9KxY8c89dRTue2223LMMccs85kyZsyYHHDAAdlhhx0q4++pp57Kfffdl2OOOeZj7ycAwEpXAACwSrjqqquKJMt8bbDBBnX26datWzF06NDK+4033rgYMmTIMs8zbNiwYkn/GXjTTTcVSYqf/exnddr32WefoqqqqnjuueeKoiiKSZMmFUmKY489tk6/gw8+uEhSnHbaaZW20047rUhSHHDAAYud75133lms7fe//32RpLjnnnsWO8bhhx9eaXv//feLzp07F1VVVcWIESMq7W+99VbRokWLOvekKIpi2223XeI1f9TUqVOLJMU555yz1D5XX3110ahRo+Lee++t037ZZZcVSYr77ruv0pakaNasWeXeFUVRPPbYY0WS4le/+lWlbffddy9WW2214j//+U+l7dlnny2aNGmyWN0tW7Zc7PqK4v/dp29/+9t12r/+9a8Xa6yxxjKve968eUX79u2LDTfcsHj33Xcr7bfddluRpDj11FMrbYvG6cMPP7zMYy7yyCOPFEmKMWPGFEVRFAsXLiw6d+5cHHPMMXX6Lbr3NTU1xauvvlpn28MPP1wkKa666qrFjj906NCiW7dulfd33313kaQ4+uijF+u7cOHCys8f/d356U9/WrRs2bL4xz/+UWefk08+uWjcuHHx4osvfuy1HnnkkUWXLl0q57nrrruKJMXf/va3Ov0WfVZ77LFHnfYjjjiiSFI89thjRVEUxfnnn18kKV577bWlnnNZ92bRuL/ssssW27ak37/vfve7xWqrrVa89957RVF88HvWo0ePolu3bsVbb71Vp++H7+XSninHHHNMUVNTU7z//vtLrR8AoCFZVgUAYBVz8cUXZ8yYMYu9+vTp87H7tmnTJlOmTMmzzz5b7/P++c9/TuPGjXP00UfXaT/hhBNSFEVuv/32JKksCXLEEUfU6XfUUUct9djf+973Fmtr0aJF5ef33nsvr7/+erbccsskyaOPPrpY/8MOO6zyc+PGjbPZZpulKIoceuihlfY2bdpk3XXXzT//+c86+44fP36FzBpPPlh6ZP311896662X119/vfLafvvtkyTjxo2r03/QoEF1Zlf36dMnNTU1lRoXLFiQv/zlL9lzzz3TqVOnSr9evXpll112qXd9H73X22yzTd54443MmjVrqfs88sgjefXVV3PEEUekefPmlfYhQ4ZkvfXWy+jRo+tdxyKjRo1Khw4dMnDgwCQfLL2x33775dprr13iEh577733YjPm6+NPf/pTqqqqctpppy227aP/GuHDrr/++myzzTZZffXV63yugwYNyoIFC3LPPfcs87zvv/9+/vCHP2S//farnGf77bdP+/btl7rG+rBhw+q8X/Q79Oc//znJ/1vX/eabb67X0i4fVl1dnUMOOWSx9g///s2ePTuvv/56ttlmm7zzzjuVJYj+9re/ZerUqTn22GMXW2N+WfdykTZt2mTOnDmVZZ0AAFY1wnEAgFXMFltskUGDBi32Wn311T923+HDh2fGjBn58pe/nI022ignnnhiHn/88eU677/+9a906tQprVu3rtO+/vrrV7Yv+rNRo0bp0aNHnX69evVa6rE/2jdJ3nzzzRxzzDHp0KFDWrRokXbt2lX6zZw5c7H+Xbt2rfO+trY2zZs3z5prrrlY+1tvvbXUWj6tZ599NlOmTEm7du3qvL785S8n+X9fWLm0upNk9dVXr9T46quv5t13313i/VvWPV2aj55v0bhZ1j1Z9Nmuu+66i21bb731Ktvra8GCBbn22mszcODATJ06Nc8991yee+659OvXL9OnT8/YsWMX22dJY6U+nn/++XTq1Clt27at137PPvts7rjjjsU+10GDBiVZ/HP9qLvuuiuvvfZatthii8p1Tp06NQMHDszvf//7JYbb66yzTp33PXv2TKNGjSpf4rnffvtlq622ymGHHZYOHTpk//33z3XXXVevoPxLX/pSmjVrtlj7lClT8vWvfz21tbWpqalJu3btcuCBByb5f79/zz//fJJkww03XO7zfdgRRxyRL3/5y9lll13SuXPnfPvb317ievsAAA3FmuMAAF8gAwYMyPPPP5+bb745d911V6688sqcf/75ueyyy+rMvP6sfXiW6iL77rtv7r///px44onZZJNN0qpVqyxcuDA777zzEsO/xo0bL1dbkhU2S3xJFi5cmI022ijnnXfeErd36dKlzvvPusaGuCdLc/fdd+eVV17Jtddem2uvvXax7aNGjcpOO+1Up21JY+WzsHDhwuy444456aSTlrh90V9+LM2i2eH77rvvErdPmDChMnt+aT46G7tFixa55557Mm7cuIwePTp33HFH/vCHP2T77bfPXXfdtdTP+qPH+KgZM2Zk2223TU1NTYYPH56ePXumefPmefTRR/PDH/7wE89S/6j27dtn8uTJufPOO3P77bfn9ttvz1VXXZVvfetb+e1vf7tCzgEA8GkIxwEAvmDatm2bQw45JIccckjefvvtDBgwIKeffnolHF/acgjdunXLX/7yl8yePbvO7PFFSyx069at8ufChQszderUOjNfn3vuueWu8a233srYsWNzxhln5NRTT620f5LlYD5rPXv2zGOPPZYddthhuZaW+Djt27dP8+bNl3j/ltS2Is75UYs+22eeeaayPMwizzzzTGV7fY0aNSrt27fPxRdfvNi2G264ITfeeGMuu+yyjw3E63PNPXv2zJ133pk333yzXrPHe/bsmbfffrsyU7w+5syZk5tvvjn77bdf9tlnn8W2H3300Rk1atRi4fizzz5bZ6b8c889l4ULF6Z79+6VtkaNGmWHHXbIDjvskPPOOy+/+MUv8qMf/Sjjxo3LoEGDPtF4GD9+fN54443ccMMNdb64durUqXX6LVoO6O9///sy78uyamjWrFl233337L777lm4cGGOOOKIXH755fnJT37yif5lBADAimRZFQCAL5A33nijzvtWrVqlV69emTt3bqWtZcuWST6YPfphu+66axYsWJBf//rXddrPP//8VFVVVda/Hjx4cJLkkksuqdPvV7/61XLXuWjG60dnM19wwQXLfYz6ePHFFysh/6e177775j//+U+uuOKKxba9++67mTNnTr2O17hx4wwaNCg33XRTXn755Ur7c889V1nn/cNatmy52Gf3aW222WZp3759Lrvssjpj5fbbb89TTz2VIUOG1PuY7777bm644Ybstttu2WeffRZ7HXnkkZk9e3ZuueWWjz3W0sbskuy9994piiJnnHHGYtuWNXt+3333zcSJE3PnnXcutm3GjBl5//33l7rvjTfemDlz5mTYsGFLvNbddtstf/rTn+rc2ySL/aXBot+hRb9rb7755mLn2mSTTZKkcqz63JtFlvT7N2/evMV+pzfddNP06NEjF1xwwWLH//C+S6vho8+jRo0aVb474aP3AgCgIZg5DgDwBdK7d+9st9126du3b9q2bZtHHnkkf/zjH3PkkUdW+vTt2zfJB7NZBw8enMaNG2f//ffP7rvvnoEDB+ZHP/pRXnjhhWy88ca56667cvPNN+fYY4+tzCLt27dv9t5771xwwQV54403suWWW2bChAn5xz/+kWT5ZvnW1NRkwIABOfvsszN//vx86Utfyl133bXYzNUV5Vvf+lYmTJiw3EuLjB07Nu+9995i7XvuuWcOOuigXHfddfne976XcePGZauttsqCBQvy9NNP57rrrsudd96ZzTbbrF71nX766bnrrruy1VZb5fvf/37lLyk23HDDTJ48uU7fvn375i9/+UvOO++8dOrUKT169Ei/fv3qdb6Patq0ac4666wccsgh2XbbbXPAAQdk+vTpufDCC9O9e/ccd9xx9T7mLbfcktmzZ2ePPfZY4vYtt9wy7dq1y6hRo7Lffvst81g9e/ZMmzZtctlll6V169Zp2bJl+vXrt8T1yQcOHJiDDjooF110UZ599tnKMj333ntvBg4cWOd34cNOPPHE3HLLLdltt91y8MEHp2/fvpkzZ06eeOKJ/PGPf8wLL7yw2Pr2i4waNSprrLFGvvrVry5x+x577JErrrgio0ePzl577VVpnzp1avbYY4/svPPOmThxYn73u9/lG9/4RjbeeOMkH3yHwD333JMhQ4akW7duefXVV3PJJZekc+fO2Xrrret9bxb56le/mtVXXz1Dhw7N0Ucfnaqqqlx99dWL/X40atQol156aXbfffdssskmOeSQQ7LWWmvl6aefzpQpUyp/kbC0Z8phhx2WN998M9tvv306d+6cf/3rX/nVr36VTTbZpPJdBgAADaoAAGCVcNVVVxVJiocffniJ27fddttigw02qNPWrVu3YujQoZX3P/vZz4otttiiaNOmTdGiRYtivfXWK37+858X8+bNq/R5//33i6OOOqpo165dUVVVVXz4Pwlnz55dHHfccUWnTp2Kpk2bFuuss05xzjnnFAsXLqxz3jlz5hTDhg0r2rZtW7Rq1arYc889i2eeeaZIUowYMaLS77TTTiuSFK+99tpi1/Pvf/+7+PrXv160adOmqK2tLf7rv/6rePnll4skxWmnnfaxxxg6dGjRsmXL5bpP2267bbE8/+k7derUIslSX1dffXVRFEUxb9684qyzzio22GCDorq6ulh99dWLvn37FmeccUYxc+bMyvGSFMOGDVvsPB/93IqiKMaOHVt85StfKZo1a1b07NmzuPLKK4sTTjihaN68eZ1+Tz/9dDFgwICiRYsWRZLKcZZ2nxaNq6lTp37s9f/hD38ovvKVrxTV1dVF27Zti29+85vFv//97yUeb2njdJHdd9+9aN68eTFnzpyl9jn44IOLpk2bFq+//nrl3p9zzjlL7HvzzTcXvXv3Lpo0aVIkKa666qqiKD4YB926davT9/333y/OOeecYr311iuaNWtWtGvXrthll12KSZMmVfos6TOYPXt2ccoppxS9evUqmjVrVqy55prFV7/61eKXv/xlnd+hD5s+fXrRpEmT4qCDDlrqdb7zzjvFaqutVnz9618viuL/fVZPPvlksc8++xStW7cuVl999eLII48s3n333cp+Y8eOLb72ta8VnTp1Kpo1a1Z06tSpOOCAA4p//OMfy3VvlvS7sMh9991XbLnllkWLFi2KTp06FSeddFJx5513FkmKcePG1en717/+tdhxxx2L1q1bFy1btiz69OlT/OpXv6pzv5f0TPnjH/9Y7LTTTkX79u2LZs2aFV27di2++93vFq+88spS7xUAwGepqiga4Jt5AAD4wpk8eXK+8pWv5He/+12++c1vNnQ5Xwh77rlnpkyZ8rlYix0AAD5vrDkOAEC9vfvuu4u1XXDBBWnUqFGdL/hj+X30nj777LP585//nO22265hCgIAgC84a44DAFBvZ599diZNmpSBAwemSZMmuf3223P77bfn8MMPT5cuXRq6vM+ltddeOwcffHDWXnvt/Otf/8qll16aZs2a5aSTTmro0gAA4AvJsioAANTbmDFjcsYZZ+TJJ5/M22+/na5du+aggw7Kj370ozRpYv7FJ3HIIYdk3LhxmTZtWqqrq9O/f//84he/yKabbtrQpQEAwBeScBwAAAAAgNKx5jgAAAAAAKUjHAcAAAAAoHQsCLkcFi5cmJdffjmtW7dOVVVVQ5cDAAAAAMASFEWR2bNnp1OnTmnUaNlzw4Xjy+Hll19Oly5dGroMAAAAAACWw0svvZTOnTsvs49wfDm0bt06yQc3tKampoGrAQAAAABgSWbNmpUuXbpUMt1lEY4vh0VLqdTU1AjHAQAAAABWccuzPLYv5AQAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlE6Thi4AAD6Pup88uqFLqOOFEUMaugQAAAD4XDFzHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACidVSYcHzFiRKqqqnLsscdW2t57770MGzYsa6yxRlq1apW9994706dPr7Pfiy++mCFDhmS11VZL+/btc+KJJ+b999+v02f8+PHZdNNNU11dnV69emXkyJGfwRUBAAAAALCqWiXC8YcffjiXX355+vTpU6f9uOOOy6233prrr78+EyZMyMsvv5y99tqrsn3BggUZMmRI5s2bl/vvvz+//e1vM3LkyJx66qmVPlOnTs2QIUMycODATJ48Occee2wOO+yw3HnnnZ/Z9QEAAAAAsGpp8HD87bffzje/+c1cccUVWX311SvtM2fOzP/8z//kvPPOy/bbb5++ffvmqquuyv33358HHnggSXLXXXflySefzO9+97tssskm2WWXXfLTn/40F198cebNm5ckueyyy9KjR4+ce+65WX/99XPkkUdmn332yfnnn98g1wsAAAAAQMNr8HB82LBhGTJkSAYNGlSnfdKkSZk/f36d9vXWWy9du3bNxIkTkyQTJ07MRhttlA4dOlT6DB48OLNmzcqUKVMqfT567MGDB1eOsSRz587NrFmz6rwAAAAAAPjiaNKQJ7/22mvz6KOP5uGHH15s27Rp09KsWbO0adOmTnuHDh0ybdq0Sp8PB+OLti/atqw+s2bNyrvvvpsWLVosdu4zzzwzZ5xxxie+LgAAAAAAVm0NNnP8pZdeyjHHHJNRo0alefPmDVXGEp1yyimZOXNm5fXSSy81dEkAAAAAAKxADRaOT5o0Ka+++mo23XTTNGnSJE2aNMmECRNy0UUXpUmTJunQoUPmzZuXGTNm1Nlv+vTp6dixY5KkY8eOmT59+mLbF21bVp+ampolzhpPkurq6tTU1NR5AQAAAADwxdFg4fgOO+yQJ554IpMnT668Nttss3zzm9+s/Ny0adOMHTu2ss8zzzyTF198Mf3790+S9O/fP0888UReffXVSp8xY8akpqYmvXv3rvT58DEW9Vl0DAAAAAAAyqfB1hxv3bp1NtxwwzptLVu2zBprrFFpP/TQQ3P88cenbdu2qampyVFHHZX+/ftnyy23TJLstNNO6d27dw466KCcffbZmTZtWn784x9n2LBhqa6uTpJ873vfy69//eucdNJJ+fa3v52777471113XUaPHv3ZXjAAAAAAAKuMBv1Czo9z/vnnp1GjRtl7770zd+7cDB48OJdccklle+PGjXPbbbfl+9//fvr375+WLVtm6NChGT58eKVPjx49Mnr06Bx33HG58MIL07lz51x55ZUZPHhwQ1wSAAAAAACrgKqiKIqGLmJVN2vWrNTW1mbmzJnWHwcgSdL95FXrXyC9MGJIQ5cAAAAADa4+WW6DrTkOAAAAAAANRTgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACidJg1dAADw6XU/eXRDl1DHCyOGNHQJAAAAsExmjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0mjR0AQCwPLqfPLqhSwAAAAC+QMwcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEqnSUMXAAB88XQ/eXRDl7CYF0YMaegSAAAAWIWYOQ4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdBo0HL/00kvTp0+f1NTUpKamJv3798/tt99e2f7ee+9l2LBhWWONNdKqVavsvffemT59ep1jvPjiixkyZEhWW221tG/fPieeeGLef//9On3Gjx+fTTfdNNXV1enVq1dGjhz5WVweAAAAAACrqAYNxzt37pwRI0Zk0qRJeeSRR7L99tvna1/7WqZMmZIkOe6443Lrrbfm+uuvz4QJE/Lyyy9nr732quy/YMGCDBkyJPPmzcv999+f3/72txk5cmROPfXUSp+pU6dmyJAhGThwYCZPnpxjjz02hx12WO68887P/HoBAAAAAFg1VBVFUTR0ER/Wtm3bnHPOOdlnn33Srl27XHPNNdlnn32SJE8//XTWX3/9TJw4MVtuuWVuv/327Lbbbnn55ZfToUOHJMlll12WH/7wh3nttdfSrFmz/PCHP8zo0aPz97//vXKO/fffPzNmzMgdd9yxXDXNmjUrtbW1mTlzZmpqalb8RQPwsbqfPLqhS+Bz7oURQxq6BAAAAFay+mS5q8ya4wsWLMi1116bOXPmpH///pk0aVLmz5+fQYMGVfqst9566dq1ayZOnJgkmThxYjbaaKNKMJ4kgwcPzqxZsyqzzydOnFjnGIv6LDrGksydOzezZs2q8wIAAAAA4IujwcPxJ554Iq1atUp1dXW+973v5cYbb0zv3r0zbdq0NGvWLG3atKnTv0OHDpk2bVqSZNq0aXWC8UXbF21bVp9Zs2bl3XffXWJNZ555ZmprayuvLl26rIhLBQAAAABgFdHg4fi6666byZMn58EHH8z3v//9DB06NE8++WSD1nTKKadk5syZlddLL73UoPUAAAAAALBiNWnoApo1a5ZevXolSfr27ZuHH344F154Yfbbb7/MmzcvM2bMqDN7fPr06enYsWOSpGPHjnnooYfqHG/69OmVbYv+XNT24T41NTVp0aLFEmuqrq5OdXX1Crk+AAAAAABWPQ0+c/yjFi5cmLlz56Zv375p2rRpxo4dW9n2zDPP5MUXX0z//v2TJP37988TTzyRV199tdJnzJgxqampSe/evSt9PnyMRX0WHQMAAAAAgPJp0Jnjp5xySnbZZZd07do1s2fPzjXXXJPx48fnzjvvTG1tbQ499NAcf/zxadu2bWpqanLUUUelf//+2XLLLZMkO+20U3r37p2DDjooZ599dqZNm5Yf//jHGTZsWGXm9/e+9738+te/zkknnZRvf/vbufvuu3Pddddl9OjRDXnpAAAAAAA0oAYNx1999dV861vfyiuvvJLa2tr06dMnd955Z3bcccckyfnnn59GjRpl7733zty5czN48OBccskllf0bN26c2267Ld///vfTv3//tGzZMkOHDs3w4cMrfXr06JHRo0fnuOOOy4UXXpjOnTvnyiuvzODBgz/z6wUAAAAAYNVQVRRF0dBFrOpmzZqV2trazJw5MzU1NQ1dDkApdT/Zv/jh03lhxJCGLgEAAICVrD5Z7iq35jgAAAAAAKxswnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA69Q7HX3rppfz73/+uvH/ooYdy7LHH5je/+c0KLQwAAAAAAFaWeofj3/jGNzJu3LgkybRp07LjjjvmoYceyo9+9KMMHz58hRcIAAAAAAArWr3D8b///e/ZYostkiTXXXddNtxww9x///0ZNWpURo4cuaLrAwAAAACAFa7e4fj8+fNTXV2dJPnLX/6SPfbYI0my3nrr5ZVXXlmx1QEAAAAAwEpQ73B8gw02yGWXXZZ77703Y8aMyc4775wkefnll7PGGmus8AIBAAAAAGBFq3c4ftZZZ+Xyyy/PdtttlwMOOCAbb7xxkuSWW26pLLcCAAAAAACrsib13WG77bbL66+/nlmzZmX11VevtB9++OFZbbXVVmhxAAAAAACwMtR75niSFEWRSZMm5fLLL8/s2bOTJM2aNROOAwAAAADwuVDvmeP/+te/svPOO+fFF1/M3Llzs+OOO6Z169Y566yzMnfu3Fx22WUro04AAAAAAFhh6j1z/Jhjjslmm22Wt956Ky1atKi0f/3rX8/YsWNXaHEAAAAAALAy1Hvm+L333pv7778/zZo1q9PevXv3/Oc//1lhhQEAAAAAwMpS75njCxcuzIIFCxZr//e//53WrVuvkKIAAAAAAGBlqnc4vtNOO+WCCy6ovK+qqsrbb7+d0047LbvuuuuKrA0AAAAAAFaKei+rcu6552bw4MHp3bt33nvvvXzjG9/Is88+mzXXXDO///3vV0aNAAAAAACwQtU7HO/cuXMee+yxXHvttXn88cfz9ttv59BDD803v/nNOl/QCQAAAAAAq6p6h+NJ0qRJkxx44IEruhYAAAAAAPhMLFc4fssttyz3AffYY49PXAwAAAAAAHwWlisc33PPPZfrYFVVVVmwYMGnqQcAAAAAAFa65QrHFy5cuLLrAAAAAACAz0yjhi4AAAAAAAA+a58oHB87dmx222239OzZMz179sxuu+2Wv/zlLyu6NgAAAAAAWCnqHY5fcskl2XnnndO6descc8wxOeaYY1JTU5Ndd901F1988cqoEQAAAAAAVqjlWnP8w37xi1/k/PPPz5FHHllpO/roo7PVVlvlF7/4RYYNG7ZCCwQAAAAAgBWt3jPHZ8yYkZ133nmx9p122ikzZ85cIUUBAAAAAMDKVO9wfI899siNN964WPvNN9+c3XbbbYUUBQAAAAAAK1O9l1Xp3bt3fv7zn2f8+PHp379/kuSBBx7IfffdlxNOOCEXXXRRpe/RRx+94ioFAAAAAIAVpKooiqI+O/To0WP5DlxVlX/+85+fqKhVzaxZs1JbW5uZM2empqamocsBKKXuJ49u6BL4nHthxJCGLgEAAICVrD5Zbr1njk+dOvUTFwYAAAAAAKuCeq85DgAAAAAAn3f1njleFEX++Mc/Zty4cXn11VezcOHCOttvuOGGFVYcAAAAAACsDPUOx4899thcfvnlGThwYDp06JCqqqqVURcAAAAAAKw09Q7Hr7766txwww3ZddddV0Y9AAAAAACw0tV7zfHa2tqsvfbaK6MWAAAAAAD4TNQ7HD/99NNzxhln5N13310Z9QAAAAAAwEpX72VV9t133/z+979P+/bt07179zRt2rTO9kcffXSFFQcAAAAAACtDvcPxoUOHZtKkSTnwwAN9IScAAAAAAJ9L9Q7HR48enTvvvDNbb731yqgHAAAAAABWunqvOd6lS5fU1NSsjFoAAAAAAOAzUe9w/Nxzz81JJ52UF154YSWUAwAAAAAAK1+9l1U58MAD884776Rnz55ZbbXVFvtCzjfffHOFFQcAAAAAACtDvcPxCy64YCWUAQAAAAAAn516h+NDhw5dGXUAAAAAAMBnpt7h+Ie99957mTdvXp02X9YJAAAAAMCqrt5fyDlnzpwceeSRad++fVq2bJnVV1+9zgsAAAAAAFZ19Q7HTzrppNx999259NJLU11dnSuvvDJnnHFGOnXqlP/7v/9bGTUCAAAAAMAKVe9lVW699db83//9X7bbbrsccsgh2WabbdKrV69069Yto0aNyje/+c2VUScAAAAAAKww9Z45/uabb2bttddO8sH64m+++WaSZOutt84999yzYqsDAAAAAICVoN7h+Nprr52pU6cmSdZbb71cd911ST6YUd6mTZsVWhwAAAAAAKwM9V5W5ZBDDsljjz2WbbfdNieffHJ23333/PrXv878+fNz3nnnrYwaAQA+te4nj27oEup4YcSQhi4BAACg1Oodjh933HGVnwcNGpSnnnoqjz76aHr16pU+ffqs0OIAAAAAAGBlqHc4/lHdu3dP9+7dV0ApAAAAAADw2VjuNccnTpyY2267rU7b//3f/6VHjx5p3759Dj/88MydO3eFFwgAAAAAACvacofjw4cPz5QpUyrvn3jiiRx66KEZNGhQTj755Nx6660588wzV0qRAAAAAACwIi13OD558uTssMMOlffXXntt+vXrlyuuuCLHH398Lrroolx33XUrpUgAAAAAAFiRljscf+utt9KhQ4fK+wkTJmSXXXapvN98883z0ksvrdjqAAAAAABgJVjucLxDhw6ZOnVqkmTevHl59NFHs+WWW1a2z549O02bNl3xFQIAAAAAwAq23OH4rrvumpNPPjn33ntvTjnllKy22mrZZpttKtsff/zx9OzZc6UUCQAAAAAAK1KT5e3405/+NHvttVe23XbbtGrVKr/97W/TrFmzyvb//d//zU477bRSigQAAAAAgBVpucPxNddcM/fcc09mzpyZVq1apXHjxnW2X3/99WnVqtUKLxAAAAAAAFa05Q7HF6mtrV1ie9u2bT91MQAAAAAA8FlY7jXHAQAAAADgi0I4DgAAAABA6QjHAQAAAAAoneUKxzfddNO89dZbSZLhw4fnnXfeWalFAQAAAADAyrRc4fhTTz2VOXPmJEnOOOOMvP322yu1KAAAAAAAWJmaLE+nTTbZJIcccki23nrrFEWRX/7yl2nVqtUS+5566qkrtEAAAAAAAFjRliscHzlyZE477bTcdtttqaqqyu23354mTRbftaqqSjgOAAAAAMAqb7nC8XXXXTfXXnttkqRRo0YZO3Zs2rdvv1ILAwAAAACAlWW5wvEPW7hw4cqoAwAAAAAAPjP1DseT5Pnnn88FF1yQp556KknSu3fvHHPMMenZs+cKLQ4AAAAAAFaGRvXd4c4770zv3r3z0EMPpU+fPunTp08efPDBbLDBBhkzZszKqBEAAAAAAFaoes8cP/nkk3PcccdlxIgRi7X/8Ic/zI477rjCigMAAAAAgJWh3jPHn3rqqRx66KGLtX/729/Ok08+uUKKAgAAAACAlane4Xi7du0yefLkxdonT56c9u3br4iaAAAAAABgpar3sirf+c53cvjhh+ef//xnvvrVryZJ7rvvvpx11lk5/vjjV3iBAAAAAACwotU7HP/JT36S1q1b59xzz80pp5ySJOnUqVNOP/30HH300Su8QAAAAAAAWNHqHY5XVVXluOOOy3HHHZfZs2cnSVq3br3CCwMAAAAAgJWl3uH4hwnFAQAAAAD4PKr3F3ICAAAAAMDnnXAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAErnE4XjRx55ZN58880VXQsAAAAAAHwmljsc//e//135+Zprrsnbb7+dJNloo43y0ksvrfjKAAAAAABgJWmyvB3XW2+9rLHGGtlqq63y3nvv5aWXXkrXrl3zwgsvZP78+SuzRgAAAAAAWKGWe+b4jBkzcv3116dv375ZuHBhdt1113z5y1/O3Llzc+edd2b69Okrs04AAAAAAFhhljscnz9/frbYYouccMIJadGiRf72t7/lqquuSuPGjfO///u/6dGjR9Zdd92VWSsAAAAAAKwQy72sSps2bbLJJptkq622yrx58/Luu+9mq622SpMmTfKHP/whX/rSl/Lwww+vzFoBAAAAAGCFWO6Z4//5z3/y4x//ONXV1Xn//ffTt2/fbLPNNpk3b14effTRVFVVZeutt16ZtQIAAAAAwAqx3OH4mmuumd133z1nnnlmVltttTz88MM56qijUlVVlR/84Aepra3NtttuuzJrBQAAAACAFWK5w/GPqq2tzb777pumTZvm7rvvztSpU3PEEUesyNoAAAAAAGClWO41xz/s8ccfz5e+9KUkSbdu3dK0adN07Ngx++233wotDgAAAAAAVoZPFI536dKl8vPf//73T3zyM888MzfccEOefvrptGjRIl/96ldz1llnZd111630ee+993LCCSfk2muvzdy5czN48OBccskl6dChQ6XPiy++mO9///sZN25cWrVqlaFDh+bMM89Mkyb/7/LGjx+f448/PlOmTEmXLl3y4x//OAcffPAnrh3gi6z7yaMbugQAAACAleoTL6uyIkyYMCHDhg3LAw88kDFjxmT+/PnZaaedMmfOnEqf4447Lrfeemuuv/76TJgwIS+//HL22muvyvYFCxZkyJAhmTdvXu6///789re/zciRI3PqqadW+kydOjVDhgzJwIEDM3ny5Bx77LE57LDDcuedd36m1wsAAAAAwKqhqiiKoqGLWOS1115L+/btM2HChAwYMCAzZ85Mu3btcs0112SfffZJkjz99NNZf/31M3HixGy55Za5/fbbs9tuu+Xll1+uzCa/7LLL8sMf/jCvvfZamjVrlh/+8IcZPXp0nVnu+++/f2bMmJE77rjjY+uaNWtWamtrM3PmzNTU1KyciwdYhZg5DivfCyOGNHQJAAAAXzj1yXIbdOb4R82cOTNJ0rZt2yTJpEmTMn/+/AwaNKjSZ7311kvXrl0zceLEJMnEiROz0UYb1VlmZfDgwZk1a1amTJlS6fPhYyzqs+gYHzV37tzMmjWrzgsAAAAAgC+OVSYcX7hwYY499thstdVW2XDDDZMk06ZNS7NmzdKmTZs6fTt06JBp06ZV+nw4GF+0fdG2ZfWZNWtW3n333cVqOfPMM1NbW1t5fXiNdQAAAAAAPv9WmXB82LBh+fvf/55rr722oUvJKaeckpkzZ1ZeL730UkOXBAAAAADACtSkoQtIkiOPPDK33XZb7rnnnnTu3LnS3rFjx8ybNy8zZsyoM3t8+vTp6dixY6XPQw89VOd406dPr2xb9Oeitg/3qampSYsWLRarp7q6OtXV1Svk2gAAAAAAWPU06Mzxoihy5JFH5sYbb8zdd9+dHj161Nnet2/fNG3aNGPHjq20PfPMM3nxxRfTv3//JEn//v3zxBNP5NVXX630GTNmTGpqatK7d+9Knw8fY1GfRccAAAAAAKBcGnTm+LBhw3LNNdfk5ptvTuvWrStrhNfW1qZFixapra3NoYcemuOPPz5t27ZNTU1NjjrqqPTv3z9bbrllkmSnnXZK7969c9BBB+Xss8/OtGnT8uMf/zjDhg2rzP7+3ve+l1//+tc56aST8u1vfzt33313rrvuuowePbrBrh0AAAAAgIbToDPHL7300sycOTPbbbdd1lprrcrrD3/4Q6XP+eefn9122y177713BgwYkI4dO+aGG26obG/cuHFuu+22NG7cOP3798+BBx6Yb33rWxk+fHilT48ePTJ69OiMGTMmG2+8cc4999xceeWVGTx48Gd6vQAAAAAArBqqiqIoGrqIVd2sWbNSW1ubmTNnpqampqHLAVjpup/sX9bAyvbCiCENXQIAAMAXTn2y3AadOQ4AAAAAAA1BOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6TRq6AACAMup+8uiGLqGOF0YMaegSAAAAPlNmjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAACl06QhT37PPffknHPOyaRJk/LKK6/kxhtvzJ577lnZXhRFTjvttFxxxRWZMWNGttpqq1x66aVZZ511Kn3efPPNHHXUUbn11lvTqFGj7L333rnwwgvTqlWrSp/HH388w4YNy8MPP5x27drlqKOOykknnfRZXioAwCqt+8mjG7qEOl4YMaShSwAAAL7gGnTm+Jw5c7Lxxhvn4osvXuL2s88+OxdddFEuu+yyPPjgg2nZsmUGDx6c9957r9Lnm9/8ZqZMmZIxY8bktttuyz333JPDDz+8sn3WrFnZaaed0q1bt0yaNCnnnHNOTj/99PzmN79Z6dcHAAAAAMCqqaooiqKhi0iSqqqqOjPHi6JIp06dcsIJJ+QHP/hBkmTmzJnp0KFDRo4cmf333z9PPfVUevfunYcffjibbbZZkuSOO+7Irrvumn//+9/p1KlTLr300vzoRz/KtGnT0qxZsyTJySefnJtuuilPP/30EmuZO3du5s6dW3k/a9asdOnSJTNnzkxNTc1KvAsAq4ZVbQYpUD5mjgMAAJ/ErFmzUltbu1xZ7iq75vjUqVMzbdq0DBo0qNJWW1ubfv36ZeLEiUmSiRMnpk2bNpVgPEkGDRqURo0a5cEHH6z0GTBgQCUYT5LBgwfnmWeeyVtvvbXEc5955pmpra2tvLp06bIyLhEAAAAAgAayyobj06ZNS5J06NChTnuHDh0q26ZNm5b27dvX2d6kSZO0bdu2Tp8lHePD5/ioU045JTNnzqy8XnrppU9/QQAAAAAArDIa9As5V1XV1dWprq5u6DIAAAAAAFhJVtmZ4x07dkySTJ8+vU779OnTK9s6duyYV199tc72999/P2+++WadPks6xofPAQAAAABAuayy4XiPHj3SsWPHjB07ttI2a9asPPjgg+nfv3+SpH///pkxY0YmTZpU6XP33Xdn4cKF6devX6XPPffck/nz51f6jBkzJuuuu25WX331z+hqAAAAAABYlTRoOP72229n8uTJmTx5cpIPvoRz8uTJefHFF1NVVZVjjz02P/vZz3LLLbfkiSeeyLe+9a106tQpe+65Z5Jk/fXXz84775zvfOc7eeihh3LfffflyCOPzP77759OnTolSb7xjW+kWbNmOfTQQzNlypT84Q9/yIUXXpjjjz++ga4aAAAAAICG1qBrjj/yyCMZOHBg5f2iwHro0KEZOXJkTjrppMyZMyeHH354ZsyYka233jp33HFHmjdvXtln1KhROfLII7PDDjukUaNG2XvvvXPRRRdVttfW1uauu+7KsGHD0rdv36y55po59dRTc/jhh392FwoAAAAAwCqlqiiKoqGLWNXNmjUrtbW1mTlzZmpqahq6HICVrvvJoxu6BKDkXhgxpKFLAAAAPofqk+WusmuOAwAAAADAyiIcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6TRq6AACS7iePbugSAAAAAErFzHEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEqnSUMXAAAAH9X95NENXcJiXhgxpKFLAAAAViAzxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOk0aegCAADg86D7yaMbuoQ6XhgxpKFLAACAzzUzxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSadLQBQAAAPXX/eTRDV1CHS+MGNLQJQAAQL2YOQ4AAAAAQOkIxwEAAAAAKB3LqgCltKr9U3QAAAAAPltmjgMAAAAAUDrCcQAAAAAASkc4DgAAAABA6QjHAQAAAAAoHeE4AAAAAAClIxwHAAAAAKB0hOMAAAAAAJSOcBwAAAAAgNIRjgMAAAAAUDrCcQAAAAAASqdJQxcAAAB8/nU/eXRDl1DHCyOGNHQJAACs4swcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6wnEAAAAAAEpHOA4AAAAAQOkIxwEAAAAAKB3hOAAAAAAApSMcBwAAAACgdITjAAAAAACUjnAcAAAAAIDSEY4DAAAAAFA6TRq6AAAAgBWt+8mjG7qEOl4YMaShSwAA4CPMHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKp0lDFwAAAPBF1/3k0Q1dwmJeGDGkoUsAAGhQwnHgM7Eq/g8hAAAAAOVlWRUAAAAAAErHzHEAAIASWtX+ZZ9lXgCAz5pwHAAAgAYnrAcAPmuWVQEAAAAAoHRKFY5ffPHF6d69e5o3b55+/frloYceauiSAAAAAABoAKUJx//whz/k+OOPz2mnnZZHH300G2+8cQYPHpxXX321oUsDAAAAAOAzVlUURdHQRXwW+vXrl8033zy//vWvkyQLFy5Mly5dctRRR+Xkk09e5r6zZs1KbW1tZs6cmZqams+iXPjCWdXWkAQAgM8Ta6ADwPKpT5Zbii/knDdvXiZNmpRTTjml0taoUaMMGjQoEydOXKz/3LlzM3fu3Mr7mTNnJvngxgKfzMK57zR0CQAA8LnV9bjrG7oEPuf+fsbghi4BVrgNT7uzoUuow+/ZqmFRhrs8c8JLEY6//vrrWbBgQTp06FCnvUOHDnn66acX63/mmWfmjDPOWKy9S5cuK61GAAAAgJWl9oKGrgC++PyerVpmz56d2traZfYpRTheX6ecckqOP/74yvuFCxfmzTffzBprrJGqqqoGrKxhzJo1K126dMlLL71kWRm+EIxpvmiMab5ojGm+aIxpvoiMa75ojGm+aMo8pouiyOzZs9OpU6eP7VuKcHzNNddM48aNM3369Drt06dPT8eOHRfrX11dnerq6jptbdq0WZklfi7U1NSU7peJLzZjmi8aY5ovGmOaLxpjmi8i45ovGmOaL5qyjumPmzG+SKOVXMcqoVmzZunbt2/Gjh1baVu4cGHGjh2b/v37N2BlAAAAAAA0hFLMHE+S448/PkOHDs1mm22WLbbYIhdccEHmzJmTQw45pKFLAwAAAADgM1aacHy//fbLa6+9llNPPTXTpk3LJptskjvuuGOxL+lkcdXV1TnttNMWW2oGPq+Mab5ojGm+aIxpvmiMab6IjGu+aIxpvmiM6eVTVRRF0dBFAAAAAADAZ6kUa44DAAAAAMCHCccBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4yzTxRdfnO7du6d58+bp169fHnrooYYuCT6x008/PVVVVXVe6623XkOXBcvtnnvuye67755OnTqlqqoqN910U53tRVHk1FNPzVprrZUWLVpk0KBBefbZZxumWFgOHzemDz744MWe2zvvvHPDFAvL4cwzz8zmm2+e1q1bp3379tlzzz3zzDPP1Onz3nvvZdiwYVljjTXSqlWr7L333pk+fXoDVQzLtjxjervttlvsWf29732vgSqGZbv00kvTp0+f1NTUpKamJv3798/tt99e2e4ZzefNx41pz+iPJxxnqf7whz/k+OOPz2mnnZZHH300G2+8cQYPHpxXX321oUuDT2yDDTbIK6+8Unn99a9/beiSYLnNmTMnG2+8cS6++OIlbj/77LNz0UUX5bLLLsuDDz6Yli1bZvDgwXnvvfc+40ph+XzcmE6SnXfeuc5z+/e///1nWCHUz4QJEzJs2LA88MADGTNmTObPn5+ddtopc+bMqfQ57rjjcuutt+b666/PhAkT8vLLL2evvfZqwKph6ZZnTCfJd77znTrP6rPPPruBKoZl69y5c0aMGJFJkyblkUceyfbbb5+vfe1rmTJlShLPaD5/Pm5MJ57RH6eqKIqioYtg1dSvX79svvnm+fWvf50kWbhwYbp06ZKjjjoqJ598cgNXB/V3+umn56abbsrkyZMbuhT41KqqqnLjjTdmzz33TPLBrPFOnTrlhBNOyA9+8IMkycyZM9OhQ4eMHDky+++/fwNWCx/vo2M6+WDm+IwZMxabUQ6fF6+99lrat2+fCRMmZMCAAZk5c2batWuXa665Jvvss0+S5Omnn87666+fiRMnZsstt2zgimHZPjqmkw9mJW6yySa54IILGrY4+ITatm2bc845J/vss49nNF8Ii8b0oYce6hm9HMwcZ4nmzZuXSZMmZdCgQZW2Ro0aZdCgQZk4cWIDVgafzrPPPptOnTpl7bXXzje/+c28+OKLDV0SrBBTp07NtGnT6jy3a2tr069fP89tPtfGjx+f9u3bZ9111833v//9vPHGGw1dEiy3mTNnJvngf1KTZNKkSZk/f36dZ/V6662Xrl27elbzufDRMb3IqFGjsuaaa2bDDTfMKaecknfeeachyoN6WbBgQa699trMmTMn/fv394zmc++jY3oRz+hla9LQBbBqev3117NgwYJ06NChTnuHDh3y9NNPN1BV8On069cvI0eOzLrrrptXXnklZ5xxRrbZZpv8/e9/T+vWrRu6PPhUpk2bliRLfG4v2gafNzvvvHP22muv9OjRI88//3z++7//O7vssksmTpyYxo0bN3R5sEwLFy7Msccem6222iobbrhhkg+e1c2aNUubNm3q9PWs5vNgSWM6Sb7xjW+kW7du6dSpUx5//PH88Ic/zDPPPJMbbrihAauFpXviiSfSv3//vPfee2nVqlVuvPHG9O7dO5MnT/aM5nNpaWM68YxeHsJxoDR22WWXys99+vRJv3790q1bt1x33XU59NBDG7AyAJbkw8sBbbTRRunTp0969uyZ8ePHZ4cddmjAyuDjDRs2LH//+999vwlfGEsb04cffnjl54022ihrrbVWdthhhzz//PPp2bPnZ10mfKx11103kydPzsyZM/PHP/4xQ4cOzYQJExq6LPjEljame/fu7Rm9HCyrwhKtueaaady48WLfyjx9+vR07NixgaqCFatNmzb58pe/nOeee66hS4FPbdGz2XObL7K11147a665puc2q7wjjzwyt912W8aNG5fOnTtX2jt27Jh58+ZlxowZdfp7VrOqW9qYXpJ+/foliWc1q6xmzZqlV69e6du3b84888xsvPHGufDCCz2j+dxa2pheEs/oxQnHWaJmzZqlb9++GTt2bKVt4cKFGTt2bJ11i+Dz7O23387zzz+ftdZaq6FLgU+tR48e6dixY53n9qxZs/Lggw96bvOF8e9//ztvvPGG5zarrKIocuSRR+bGG2/M3XffnR49etTZ3rdv3zRt2rTOs/qZZ57Jiy++6FnNKunjxvSSTJ48OUk8q/ncWLhwYebOnesZzRfGojG9JJ7Ri7OsCkt1/PHHZ+jQodlss82yxRZb5IILLsicOXNyyCGHNHRp8In84Ac/yO67755u3brl5ZdfzmmnnZbGjRvngAMOaOjSYLm8/fbbdf6Gf+rUqZk8eXLatm2brl275thjj83PfvazrLPOOunRo0d+8pOfpFOnTtlzzz0brmhYhmWN6bZt2+aMM87I3nvvnY4dO+b555/PSSedlF69emXw4MENWDUs3bBhw3LNNdfk5ptvTuvWrStr1NbW1qZFixapra3NoYcemuOPPz5t27ZNTU1NjjrqqPTv3z9bbrllA1cPi/u4Mf3888/nmmuuya677po11lgjjz/+eI477rgMGDAgffr0aeDqYXGnnHJKdtlll3Tt2jWzZ8/ONddck/Hjx+fOO+/0jOZzaVlj2jN6ORWwDL/61a+Krl27Fs2aNSu22GKL4oEHHmjokuAT22+//Yq11lqraNasWfGlL32p2G+//YrnnnuuocuC5TZu3LgiyWKvoUOHFkVRFAsXLix+8pOfFB06dCiqq6uLHXbYoXjmmWcatmhYhmWN6XfeeafYaaedinbt2hVNmzYtunXrVnznO98ppk2b1tBlw1ItaTwnKa666qpKn3fffbc44ogjitVXX71YbbXViq9//evFK6+80nBFwzJ83Jh+8cUXiwEDBhRt27Ytqquri169ehUnnnhiMXPmzIYtHJbi29/+dtGtW7eiWbNmRbt27YoddtihuOuuuyrbPaP5vFnWmPaMXj5VRVEUn2UYDwAAAAAADc2a4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAfG49/fTT2XLLLdO8efNssskmDV3OEm233XY59thjP9G+I0eOTJs2bVZoPSvCwQcfnD333HO5+r7wwgupqqrK5MmTV2pNSdK9e/dccMEFK/08AAB8MQjHAQBY6V577bU0a9Ysc+bMyfz589OyZcu8+OKLn/q4p512Wlq2bJlnnnkmY8eOXWz7ZZddltatW+f999+vtL399ttp2rRptttuuzp9x48fn6qqqjz//POfuq4VZb/99ss//vGPFXa89dZbL9XV1Zk2bdpy9V9asH3hhRdm5MiRK6yu+qjvNdTHZxmur6p/8QEAUCbCcQAAVrqJEydm4403TsuWLfPoo4+mbdu26dq166c+7vPPP5+tt9463bp1yxprrLHY9oEDB+btt9/OI488Umm7995707Fjxzz44IN57733Ku3jxo1L165d07Nnz3rXURRFnQB+RWnRokXat2+/Qo7117/+Ne+++2722Wef/Pa3v/3Y/vPmzVvqttra2gYJdut7DSvDggULsnDhwgY5NwAAK5ZwHACAle7+++/PVlttleSDgHPRz8uycOHCDB8+PJ07d051dXU22WST3HHHHZXtVVVVmTRpUoYPH56qqqqcfvrpix1j3XXXzVprrZXx48dX2saPH5+vfe1r6dGjRx544IE67QMHDkySzJ07N0cffXTat2+f5s2bZ+utt87DDz9cp29VVVVuv/329O3bN9XV1fnrX/+aOXPm5Fvf+lZatWqVtdZaK+eee+5iNV1yySVZZ5110rx583To0CH77LPPUu/BR2cXn3766dlkk01y9dVXp3v37qmtrc3++++f2bNnf+z9/J//+Z984xvfyEEHHZT//d//XWx79+7d89Of/jTf+ta3UlNTk8MPPzw9evRIknzlK19JVVVVZbb9R5dVWbhwYc4+++z06tUr1dXV6dq1a37+858vtZa///3v2WWXXdKqVat06NAhBx10UF5//fVPfQ1JMnv27BxwwAFp2bJlvvSlL+Xiiy+ubCuKIqeffnq6du2a6urqdOrUKUcffXSSD5a/+de//pXjjjsuVVVVqaqqSvL/PoNbbrklvXv3TnV1dV588cU8/PDD2XHHHbPmmmumtrY22267bR599NE6tcyYMSPf/e5306FDhzRv3jwbbrhhbrvttowfPz6HHHJIZs6cWTnXovFbn/EBAMCnIxwHAGClePHFF9OmTZu0adMm5513Xi6//PK0adMm//3f/52bbropbdq0yRFHHLHU/S+88MKce+65+eUvf5nHH388gwcPzh577JFnn302SfLKK69kgw02yAknnJBXXnklP/jBD5Z4nIEDB2bcuHGV9+PGjct2222XbbfdttL+7rvv5sEHH6yE4yeddFL+9Kc/5be//W0effTR9OrVK4MHD86bb75Z59gnn3xyRowYkaeeeip9+vTJiSeemAkTJuTmm2/OXXfdlfHjx9cJTB955JEcffTRGT58eJ555pnccccdGTBgQL3u6/PPP5+bbropt912W2677bZMmDAhI0aMWOY+s2fPzvXXX58DDzwwO+64Y2bOnJl77713sX6//OUvs/HGG+dvf/tbfvKTn+Shhx5KkvzlL3/JK6+8khtuuGGJxz/llFMyYsSI/OQnP8mTTz6Za665Jh06dFhi3xkzZmT77bfPV77ylTzyyCO54447Mn369Oy7774r5BrOOeecyjWcfPLJOeaYYzJmzJgkyZ/+9Kecf/75ufzyy/Pss8/mpptuykYbbZQkueGGG9K5c+cMHz48r7zySl555ZXKMd95552cddZZufLKKzNlypS0b98+s2fPztChQ/PXv/41DzzwQNZZZ53suuuulb+oWLhwYXbZZZfcd999+d3vfpcnn3wyI0aMSOPGjfPVr341F1xwQWpqairn+sEPfrBCxgcAAPVQAADASjB//vxi6tSpxWOPPVY0bdq0eOyxx4rnnnuuaNWqVTFhwoRi6tSpxWuvvbbU/Tt16lT8/Oc/r9O2+eabF0cccUTl/cYbb1ycdtppy6zjiiuuKFq2bFnMnz+/mDVrVtGkSZPi1VdfLa655ppiwIABRVEUxdixY4skxb/+9a/i7bffLpo2bVqMGjWqcox58+YVnTp1Ks4+++yiKIpi3LhxRZLipptuqvSZPXt20axZs+K6666rtL3xxhtFixYtimOOOaYoiqL405/+VNTU1BSzZs1a9s37/1111VVFbW1t5f1pp51WrLbaanX2P/HEE4t+/fot8zi/+c1vik022aTy/phjjimGDh1ap0+3bt2KPffcs07b1KlTiyTF3/72tzrtQ4cOLb72ta8VRVEUs2bNKqqrq4srrrhiief+6DF++tOfFjvttFOdPi+99FKRpHjmmWc+9TXsvPPOddr222+/YpdddimKoijOPffc4stf/nIxb968JZ6jW7duxfnnn1+n7aqrriqSFJMnT15qbUVRFAsWLChat25d3HrrrUVRFMWdd95ZNGrUaKnX9NHPtijqPz4AAPh0zBwHAGClaNKkSbp3756nn346m2++efr06ZNp06alQ4cOGTBgQLp3754111xzifvOmjUrL7/88mLLr2y11VZ56qmn6lXHdtttlzlz5uThhx/Ovffemy9/+ctp165dtt1228q64+PHj8/aa6+drl275vnnn8/8+fPrnLtp06bZYostFjv3ZpttVvn5+eefz7x589KvX79KW9u2bbPuuutW3u+4447p1q1b1l577Rx00EEZNWpU3nnnnXpdT/fu3dO6devK+7XWWiuvvvrqMvf53//93xx44IGV9wceeGCuv/76xZZj+fD1LK+nnnoqc+fOzQ477LBc/R977LGMGzcurVq1qrzWW2+9JFnml6Eu7zX0799/sfeLPrf/+q//yrvvvpu111473/nOd3LjjTcu11rxzZo1S58+feq0TZ8+Pd/5zneyzjrrpLa2NjU1NXn77bcrXzQ7efLkdO7cOV/+8pc/9viLrIjxAQDA8hOOAwCwUmywwQZp1apVDjrooDz00ENp1apVdthhh7zwwgtp1apVNthgg8+kjl69eqVz584ZN25cxo0bl2233TZJ0qlTp3Tp0iX3339/xo0bl+23377ex27ZsmW9+rdu3TqPPvpofv/732ettdbKqaeemo033jgzZsxY7mM0bdq0zvuqqqplfkHkk08+mQceeCAnnXRSmjRpkiZNmmTLLbfMO++8k2uvvfZTXU/ywZeG1sfbb7+d3XffPZMnT67zevbZZ5e6hEh9rmFZunTpkmeeeSaXXHJJWrRokSOOOCIDBgzI/PnzP/YaF61BvsjQoUMzefLkXHjhhbn//vszefLkrLHGGpUvMq3vfUlWzPgAAGD5CccBAFgp/vznP2fy5Mnp2LFjfve732Xy5MnZcMMNc8EFF2Ty5Mn585//vNR9a2pq0qlTp9x333112u+777707t273rUMHDgw48ePz/jx4ytfKpkkAwYMyO23356HHnqost54z54906xZszrnnj9/fh5++OFlnrtnz55p2rRpHnzwwUrbW2+9lX/84x91+jVp0iSDBg3K2WefnccffzwvvPBC7r777npf0/L6n//5nwwYMCCPPfZYnTD6+OOPz//8z/8sc99mzZolSRYsWLDUPuuss05atGiRsWPHLlc9m266aaZMmZLu3bunV69edV5LC+frcw0f/pLVRe/XX3/9yvsWLVpk9913z0UXXZTx48dn4sSJeeKJJyrXu6xr/bD77rsvRx99dHbddddssMEGqa6urvOlon369Mm///3vxT7/RZZ2rs96fAAAlFmThi4AAIAvpm7dumXatGmZPn16vva1r6WqqipTpkzJ3nvvnbXWWutj9z/xxBNz2mmnpWfPntlkk01y1VVXZfLkyRk1alS9axk4cGCGDRuW+fPnV2aOJ8m2226bI488MvPmzauE4y1btsz3v//9nHjiiWnbtm26du2as88+O++8804OPfTQpZ6jVatWOfTQQ3PiiSdmjTXWSPv27fOjH/0ojRr9v/kot912W/75z39mwIABWX311fPnP/85CxcurLP0yoo0f/78XH311Rk+fHg23HDDOtsOO+ywnHfeeZkyZcpSZ/G3b98+LVq0yB133JHOnTunefPmqa2trdOnefPm+eEPf5iTTjopzZo1y1ZbbZXXXnstU6ZMWeL9GjZsWK644ooccMABOemkk9K2bds899xzufbaa3PllVemcePGn+oa7rvvvpx99tnZc889M2bMmFx//fUZPXp0kmTkyJFZsGBB+vXrl9VWWy2/+93v0qJFi3Tr1i3JB0vW3HPPPdl///1TXV291GV/kg/+UuDqq6/OZpttllmzZuXEE0+sM1t82223zYABA7L33nvnvPPOS69evfL000+nqqoqO++8c7p375633347Y8eOzcYbb5zVVlstd99992c6PgAAys7McQAAVprx48dn8803T/PmzfPQQw+lc+fOyxWMJ8nRRx+d448/PieccEI22mij3HHHHbnllluyzjrr1LuOgQMH5t13302vXr3SoUOHSvu2226b2bNnZ911161T14gRI7L33nvnoIMOyqabbprnnnsud955Z1ZfffVlnuecc87JNttsk9133z2DBg3K1ltvnb59+1a2t2nTJjfccEO23377rL/++rnsssvy+9//fqUtMXPLLbfkjTfeyNe//vXFtq2//vpZf/31lzl7vEmTJrnoooty+eWXp1OnTvna1762xH4/+clPcsIJJ+TUU0/N+uuvn/3222+p66Av+hcBCxYsyE477ZSNNtooxx57bNq0aVPnLxI+6TWccMIJeeSRR/KVr3wlP/vZz3Leeedl8ODBST64/1dccUW22mqr9OnTJ3/5y19y6623Zo011kiSDB8+PC+88EJ69uyZdu3aLfW+JB/MZn/rrbey6aab5qCDDsrRRx+d9u3b1+nzpz/9KZtvvnkOOOCA9O7dOyeddFJltvhXv/rVfO9738t+++2Xdu3a5eyzz/7MxwcAQNlVFUVRNHQRAAAAAADwWTJzHAAAAACA0hGOAwAAAABQOsJxAAAAAABKRzgOAAAAAEDpCMcBAAAAACgd4TgAAAAAAKUjHAcAAAAAoHSE4wAAAAAAlI5wHAAAAACA0hGOAwAAAABQOsJxAAAAAABK5/8DsXJP1LVdT18AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The maximum length of the questions (in words) is 33 words.\n"
     ]
    }
   ],
   "source": [
    "train_ds['question_length'] = train_ds['question'].str.split().str.len()\n",
    "\n",
    "n_bin=35\n",
    "train_ds.hist('question_length', grid=False, figsize=(18,8), range=(0,n_bin), bins=n_bin)\n",
    "plt.title('Histogram: Length of Article Abstracts')\n",
    "plt.xlabel('# of Words in Article Abstracts')\n",
    "plt.ylabel('# of Samples')\n",
    "plt.show()\n",
    "\n",
    "print(f\"The maximum length of the questions (in words) is {max(train_ds['question_length'])} words.\")\n",
    "train_ds = train_ds.drop(columns=['question_length'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert from DataFrame to Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = Dataset.from_pandas(train_ds)\n",
    "eval_dataset = Dataset.from_pandas(eval_ds)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assign Training & Validation Datasets to DatasetDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Dataset Shape: (25262, 7)\n",
      "Evaluation Dataset Shape: (2985, 7)\n",
      "{'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answer0': Value(dtype='string', id=None), 'answer1': Value(dtype='string', id=None), 'answer2': Value(dtype='string', id=None), 'answer3': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}\n"
     ]
    }
   ],
   "source": [
    "ds = DatasetDict({\n",
    "    'train' : train_dataset,\n",
    "    'eval' : eval_dataset,\n",
    "})\n",
    "\n",
    "print(\"Training Dataset Shape:\", ds['train'].shape)\n",
    "print(\"Evaluation Dataset Shape:\", ds['eval'].shape)\n",
    "\n",
    "print(ds['train'].features)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Function to Display an Example Sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_one_example(sample):\n",
    "    print(f\"Context: {sample['context']} {sample['question']}\")\n",
    "    print(f\" 0 - {sample['answer0']}\")\n",
    "    print(f\" 1 - {sample['answer1']}\")\n",
    "    print(f\" 2 - {sample['answer2']}\")\n",
    "    print(f\" 3 - {sample['answer3']}\")\n",
    "    print(f\"\\nGround Truth: option {sample['label']}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display an Example Sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context: Leaving my shift Thursday day shift I arrived the same time as my partner just after six that evening and before long the radio erupted in dispatch tones . A car fleeing the police has crashed and landed on its roof with four separate people entrapped inside . Our medic unit is dispatched along with multiple other ambulances and Rescue Companies . What may have caused the radio to erupt with dispatch tones ?\n",
      " 0 - My partner needed a medic unit .\n",
      " 1 - Someone was running from the ambulances after they got into a wreck .\n",
      " 2 - None of the above choices .\n",
      " 3 - Someone was running from the cops and got into a wreck .\n",
      "\n",
      "Ground Truth: option 3\n"
     ]
    }
   ],
   "source": [
    "show_one_example(ds['train'][2])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic Values/Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_CKPT = \"bert-base-uncased\"\n",
    "MODEL_NAME = MODEL_CKPT.split(\"/\")[-1] + \"-Cosmos_QA\"\n",
    "\n",
    "STRATEGY = \"epoch\"\n",
    "LEARNING_RATE = 5e-5\n",
    "\n",
    "BATCH_SIZE = 16\n",
    "NUM_OF_EPOCHS = 3\n",
    "\n",
    "WEIGHT_DECAY = 0.01\n",
    "REPORTS_TO = \"tensorboard\"\n",
    "\n",
    "set_seed = 42"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instantiate Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT, use_fast=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Preprocessing Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "ending_names = [\"answer0\", \"answer1\", \"answer2\", \"answer3\"]\n",
    "\n",
    "def preprocess_function(samples):\n",
    "    first_sentences = [[context] * len(ending_names) for context in samples['context']]\n",
    "    \n",
    "    question_headers = samples['question']\n",
    "    second_sentences = [[f\"{header} {samples[end][i]}\" for end in ending_names] for i, header in enumerate(question_headers)]\n",
    "    \n",
    "    first_sentences = sum(first_sentences, [])\n",
    "    second_sentences = sum(second_sentences, [])\n",
    "    \n",
    "    tokenized_samples = tokenizer(first_sentences, second_sentences, truncation=True)\n",
    "    return {k: [v[i:i + len(ending_names)] for i in range(0, len(v), len(ending_names))] for k, v in tokenized_samples.items()}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Preprocessing Function to Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0622fdf88f134b13afd13ac1a2e269c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/26 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b1102da7dd94532af6f1815f9659235",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "encoded_ds = ds.map(preprocess_function, batched=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'context': Value(dtype='string', id=None),\n",
       " 'question': Value(dtype='string', id=None),\n",
       " 'answer0': Value(dtype='string', id=None),\n",
       " 'answer1': Value(dtype='string', id=None),\n",
       " 'answer2': Value(dtype='string', id=None),\n",
       " 'answer3': Value(dtype='string', id=None),\n",
       " 'label': Value(dtype='int64', id=None),\n",
       " 'input_ids': Sequence(feature=Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), length=-1, id=None),\n",
       " 'token_type_ids': Sequence(feature=Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None), length=-1, id=None),\n",
       " 'attention_mask': Sequence(feature=Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None), length=-1, id=None)}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_ds['train'].features"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instantiate Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMultipleChoice: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForMultipleChoice from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForMultipleChoice from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForMultipleChoice were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = AutoModelForMultipleChoice.from_pretrained(MODEL_CKPT)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define TrainingArguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "args = TrainingArguments(\n",
    "    output_dir=MODEL_NAME,\n",
    "    evaluation_strategy=STRATEGY,\n",
    "    learning_rate=LEARNING_RATE,\n",
    "    per_device_train_batch_size=BATCH_SIZE,\n",
    "    per_device_eval_batch_size=BATCH_SIZE,\n",
    "    num_train_epochs=NUM_OF_EPOCHS,\n",
    "    weight_decay=WEIGHT_DECAY,\n",
    "    report_to=REPORTS_TO,\n",
    "    logging_first_step=True,\n",
    "    hub_private_repo=True,\n",
    "    push_to_hub=True\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Data Collator Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class DataCollatorForMultipleChoice:\n",
    "    \"\"\"\n",
    "    Data collator that will dynamically pad inputs \n",
    "    for the multiple choices received.\n",
    "    \"\"\"\n",
    "\n",
    "    tokenizer: PreTrainedTokenizerBase\n",
    "    padding: Union[bool, str, PaddingStrategy] = True\n",
    "    max_length: Optional[int] = None\n",
    "    pad_to_multiple_of: Optional[int] = None\n",
    "    \n",
    "    def __call__(self, features):\n",
    "        label_name = \"label\" if \"label\" in features[0].keys() else \"labels\"\n",
    "        labels = [feature.pop(label_name) for feature in features]\n",
    "        batch_size = len(features)\n",
    "        num_choices = len(features[0][\"input_ids\"])\n",
    "        flattened_features = [[{k: v[i] for k, v in feature.items()} \n",
    "                               for i in range(num_choices)] for feature in features]\n",
    "        flattened_features = sum(flattened_features, [])\n",
    "        \n",
    "        batch = self.tokenizer.pad(\n",
    "            flattened_features,\n",
    "            padding=self.padding,\n",
    "            max_length=self.max_length,\n",
    "            pad_to_multiple_of=self.pad_to_multiple_of,\n",
    "            return_tensors=\"pt\",\n",
    "        )\n",
    "        \n",
    "        # Unflatten\n",
    "        batch = {k: v.view(batch_size, num_choices, -1) for k, v in batch.items()}\n",
    "        # Add back labels\n",
    "        batch[\"labels\"] = torch.tensor(labels, dtype=torch.int64)\n",
    "        return batch"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Function to Compute Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(predictions):\n",
    "    preds, labels = predictions\n",
    "    preds = np.argmax(preds, axis=1)\n",
    "    return {\"accuracy\": (preds == labels).astype(np.float32).mean().item()}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instantiate Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning https://huggingface.co/DunnBC22/bert-base-uncased-Cosmos_QA into local empty directory.\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=args,\n",
    "    train_dataset=encoded_ds[\"train\"],\n",
    "    eval_dataset=encoded_ds[\"eval\"],\n",
    "    tokenizer=tokenizer,\n",
    "    data_collator=DataCollatorForMultipleChoice(tokenizer),\n",
    "    compute_metrics=compute_metrics,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The following columns in the training set don't have a corresponding argument in `BertForMultipleChoice.forward` and have been ignored: answer1, answer3, context, answer0, answer2, question. If answer1, answer3, context, answer0, answer2, question are not expected by `BertForMultipleChoice.forward`,  you can safely ignore this message.\n",
      "/Users/leedunn/Documents/nlpnn/nlp/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 25262\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 16\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 16\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 4737\n",
      "  Number of trainable parameters = 109483009\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "07c2fac1984043ce8862b254f0cdb4d6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/4737 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.3339, 'learning_rate': 4.998944479628457e-05, 'epoch': 0.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to bert-base-uncased-Cosmos_QA/checkpoint-500\n",
      "Configuration saved in bert-base-uncased-Cosmos_QA/checkpoint-500/config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.2061, 'learning_rate': 4.472239814228415e-05, 'epoch': 0.32}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in bert-base-uncased-Cosmos_QA/checkpoint-500/pytorch_model.bin\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/checkpoint-500/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/checkpoint-500/special_tokens_map.json\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/special_tokens_map.json\n",
      "Saving model checkpoint to bert-base-uncased-Cosmos_QA/checkpoint-1000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.0631, 'learning_rate': 3.944479628456829e-05, 'epoch': 0.63}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Configuration saved in bert-base-uncased-Cosmos_QA/checkpoint-1000/config.json\n",
      "Model weights saved in bert-base-uncased-Cosmos_QA/checkpoint-1000/pytorch_model.bin\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/checkpoint-1000/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/checkpoint-1000/special_tokens_map.json\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/special_tokens_map.json\n",
      "Saving model checkpoint to bert-base-uncased-Cosmos_QA/checkpoint-1500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.9734, 'learning_rate': 3.416719442685244e-05, 'epoch': 0.95}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Configuration saved in bert-base-uncased-Cosmos_QA/checkpoint-1500/config.json\n",
      "Model weights saved in bert-base-uncased-Cosmos_QA/checkpoint-1500/pytorch_model.bin\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/checkpoint-1500/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/checkpoint-1500/special_tokens_map.json\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/special_tokens_map.json\n",
      "The following columns in the evaluation set don't have a corresponding argument in `BertForMultipleChoice.forward` and have been ignored: answer1, answer3, context, answer0, answer2, question. If answer1, answer3, context, answer0, answer2, question are not expected by `BertForMultipleChoice.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 2985\n",
      "  Batch size = 16\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f72739ba45f84a86bde6cb6cbf3a0dae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/187 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.9463162422180176, 'eval_accuracy': 0.6237856149673462, 'eval_runtime': 5178.4001, 'eval_samples_per_second': 0.576, 'eval_steps_per_second': 0.036, 'epoch': 1.0}\n",
      "{'loss': 0.6396, 'learning_rate': 2.8889592569136586e-05, 'epoch': 1.27}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to bert-base-uncased-Cosmos_QA/checkpoint-2000\n",
      "Configuration saved in bert-base-uncased-Cosmos_QA/checkpoint-2000/config.json\n",
      "Model weights saved in bert-base-uncased-Cosmos_QA/checkpoint-2000/pytorch_model.bin\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/checkpoint-2000/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/checkpoint-2000/special_tokens_map.json\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/special_tokens_map.json\n",
      "Saving model checkpoint to bert-base-uncased-Cosmos_QA/checkpoint-2500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5421, 'learning_rate': 2.3611990711420732e-05, 'epoch': 1.58}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Configuration saved in bert-base-uncased-Cosmos_QA/checkpoint-2500/config.json\n",
      "Model weights saved in bert-base-uncased-Cosmos_QA/checkpoint-2500/pytorch_model.bin\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/checkpoint-2500/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/checkpoint-2500/special_tokens_map.json\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/special_tokens_map.json\n",
      "Saving model checkpoint to bert-base-uncased-Cosmos_QA/checkpoint-3000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5363, 'learning_rate': 1.833438885370488e-05, 'epoch': 1.9}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Configuration saved in bert-base-uncased-Cosmos_QA/checkpoint-3000/config.json\n",
      "Model weights saved in bert-base-uncased-Cosmos_QA/checkpoint-3000/pytorch_model.bin\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/checkpoint-3000/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/checkpoint-3000/special_tokens_map.json\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/special_tokens_map.json\n",
      "The following columns in the evaluation set don't have a corresponding argument in `BertForMultipleChoice.forward` and have been ignored: answer1, answer3, context, answer0, answer2, question. If answer1, answer3, context, answer0, answer2, question are not expected by `BertForMultipleChoice.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 2985\n",
      "  Batch size = 16\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ceb916e623624725b58f5647c9d608da",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/187 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.098717212677002, 'eval_accuracy': 0.6157453656196594, 'eval_runtime': 5215.1053, 'eval_samples_per_second': 0.572, 'eval_steps_per_second': 0.036, 'epoch': 2.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to bert-base-uncased-Cosmos_QA/checkpoint-3500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.3025, 'learning_rate': 1.3056786995989024e-05, 'epoch': 2.22}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Configuration saved in bert-base-uncased-Cosmos_QA/checkpoint-3500/config.json\n",
      "Model weights saved in bert-base-uncased-Cosmos_QA/checkpoint-3500/pytorch_model.bin\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/checkpoint-3500/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/checkpoint-3500/special_tokens_map.json\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/special_tokens_map.json\n",
      "Adding files tracked by Git LFS: ['.DS_Store']. This may take a bit of time if the files are large.\n",
      "Saving model checkpoint to bert-base-uncased-Cosmos_QA/checkpoint-4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.1767, 'learning_rate': 7.779185138273169e-06, 'epoch': 2.53}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Configuration saved in bert-base-uncased-Cosmos_QA/checkpoint-4000/config.json\n",
      "Model weights saved in bert-base-uncased-Cosmos_QA/checkpoint-4000/pytorch_model.bin\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/checkpoint-4000/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/checkpoint-4000/special_tokens_map.json\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/special_tokens_map.json\n",
      "Saving model checkpoint to bert-base-uncased-Cosmos_QA/checkpoint-4500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.1852, 'learning_rate': 2.501583280557315e-06, 'epoch': 2.85}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Configuration saved in bert-base-uncased-Cosmos_QA/checkpoint-4500/config.json\n",
      "Model weights saved in bert-base-uncased-Cosmos_QA/checkpoint-4500/pytorch_model.bin\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/checkpoint-4500/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/checkpoint-4500/special_tokens_map.json\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/special_tokens_map.json\n",
      "The following columns in the evaluation set don't have a corresponding argument in `BertForMultipleChoice.forward` and have been ignored: answer1, answer3, context, answer0, answer2, question. If answer1, answer3, context, answer0, answer2, question are not expected by `BertForMultipleChoice.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 2985\n",
      "  Batch size = 16\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3cc0c5dab04412ca2cf1102b004ee21",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/187 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.7425875663757324, 'eval_accuracy': 0.6000000238418579, 'eval_runtime': 5131.8614, 'eval_samples_per_second': 0.582, 'eval_steps_per_second': 0.036, 'epoch': 3.0}\n",
      "{'train_runtime': 676721.6107, 'train_samples_per_second': 0.112, 'train_steps_per_second': 0.007, 'train_loss': 0.6021223496806906, 'epoch': 3.0}\n"
     ]
    }
   ],
   "source": [
    "train_results = trainer.train()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The following columns in the evaluation set don't have a corresponding argument in `BertForMultipleChoice.forward` and have been ignored: answer1, answer3, context, answer0, answer2, question. If answer1, answer3, context, answer0, answer2, question are not expected by `BertForMultipleChoice.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 2985\n",
      "  Batch size = 16\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2780f207e5484f1a91606cbdcfdbe3ef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/187 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.7425875663757324, 'eval_accuracy': 0.6000000238418579, 'eval_runtime': 5128.7565, 'eval_samples_per_second': 0.582, 'eval_steps_per_second': 0.036, 'epoch': 3.0}\n"
     ]
    }
   ],
   "source": [
    "metrics = trainer.evaluate()\n",
    "\n",
    "print(metrics)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Push to HuggingFace Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to bert-base-uncased-Cosmos_QA\n",
      "Configuration saved in bert-base-uncased-Cosmos_QA/config.json\n",
      "Model weights saved in bert-base-uncased-Cosmos_QA/pytorch_model.bin\n",
      "tokenizer config file saved in bert-base-uncased-Cosmos_QA/tokenizer_config.json\n",
      "Special tokens file saved in bert-base-uncased-Cosmos_QA/special_tokens_map.json\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c81718fa683b4e139c0f811ff8fe28ac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file pytorch_model.bin:   0%|          | 32.0k/418M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b3eb9c6ae5543aa96de024e6133c885",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file runs/.DS_Store: 100%|##########| 6.00k/6.00k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "282657745aa645a998ed7926c082e048",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file runs/Mar06_20-39-06_Lees-Air/events.out.tfevents.1678156754.Lees-Air.71037.0: 100%|##########| 6.5…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a47b34b3f3144d78c491d629625b5e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file runs/Mar06_20-39-06_Lees-Air/events.out.tfevents.1678838605.Lees-Air.71037.2: 100%|##########| 363…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0e81ffa72c7043ebbfb58dcd4598928f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file .DS_Store: 100%|##########| 10.0k/10.0k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "remote: Scanning LFS files of refs/heads/main for validity...        \n",
      "remote: LFS file scan complete.        \n",
      "To https://huggingface.co/DunnBC22/bert-base-uncased-Cosmos_QA\n",
      "   e7a4348..ceb9c55  main -> main\n",
      "\n",
      "Dropping the following result as it does not have all the necessary fields:\n",
      "{'metrics': [{'name': 'Accuracy', 'type': 'accuracy', 'value': 0.6000000238418579}]}\n",
      "To https://huggingface.co/DunnBC22/bert-base-uncased-Cosmos_QA\n",
      "   ceb9c55..1df8283  main -> main\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'https://huggingface.co/DunnBC22/bert-base-uncased-Cosmos_QA/commit/ceb9c55fa94544556c5d6712ba3b9cd87f42299e'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.push_to_hub()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes & Other Takeaways From This Project\n",
    "****\n",
    "- It looks like 67.5 percent is the State of the Art value for BERT (the 'Large' BERT checkpoint, not the base checkpoint) on Multiple Choice with the SWAG dataset. Having done some smaller initial test runs and trying this project with distilBERT originally, I am sure that with more training (4 or 5 epochs instead of 3), I could achieve metrics more in line with the State of the Art value. (Same response as the Social IQa project, but for good reason.)\n",
    "****"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Citation\n",
    "- Model Checkpoint\n",
    "@article{DBLP:journals/corr/abs-1810-04805,\n",
    "author = {Jacob Devlin and Ming{-}Wei Chang and Kenton Lee and Kristina Toutanova}, title = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language Understanding}, journal = {CoRR}, volume = {abs/1810.04805}, year = {2018}, url = {http://arxiv.org/abs/1810.04805%7D, archivePrefix = {arXiv}, eprint = {1810.04805}, timestamp = {Tue, 30 Oct 2018 20:39:56 +0100}, biburl = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib%7D, bibsource = {dblp computer science bibliography, https://dblp.org%7D\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nlp_ml",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "0b7a754c5aaaa44a21e2f2148f50b931a5abdffe7ef71537f5954179554f39df"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}