Sentiment Analysis/Sentiment Analysis of Commodity News - Gold (Transformer Comparison)/News About Gold - Sentiment Analysis - SqueezeBERT with W&B.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e67daac8",
   "metadata": {},
   "source": [
    "## News About Gold\n",
    "\n",
    "Dataset Source: https://www.kaggle.com/datasets/ankurzing/sentiment-analysis-in-commodity-market-gold"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d2c358a6",
   "metadata": {},
   "source": [
    "#### Import Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "617d6a4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Git LFS initialized.\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "os.environ['TOKENIZERS_PARALLELISM']='false'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import wandb\n",
    "\n",
    "import torch\n",
    "\n",
    "import transformers\n",
    "from transformers import AutoTokenizer, Trainer, TrainingArguments\n",
    "from transformers import AutoModelForSequenceClassification, set_seed\n",
    "from transformers import DataCollatorWithPadding\n",
    "\n",
    "import datasets\n",
    "from datasets import Dataset, DatasetDict\n",
    "\n",
    "import evaluate\n",
    "\n",
    "!git lfs install"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "48ed1f2c",
   "metadata": {},
   "source": [
    "#### Name Project for Weights & Biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6d236c30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: WANDB_PROJECT=News_About_Gold_SqueezeBERT\n",
      "env: WANDB_LOG_MODEL='checkpoint'\n",
      "env: WANDB_NOTEBOOK_NAME=\"News About Gold - Sentiment Analysis - SqueezeBERT with W&B.ipynb\"\n",
      "env: WANDB_WATCH=all\n"
     ]
    }
   ],
   "source": [
    "%env WANDB_PROJECT=News_About_Gold_SqueezeBERT\n",
    "%env WANDB_LOG_MODEL='checkpoint'\n",
    "%env WANDB_NOTEBOOK_NAME=\"News About Gold - Sentiment Analysis - SqueezeBERT with W&B.ipynb\"\n",
    "%env WANDB_WATCH=all"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b4382b16",
   "metadata": {},
   "source": [
    "#### Log In To Weights & Biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "10e8c823",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find \"News About Gold - Sentiment Analysis - SqueezeBERT with W&B.ipynb\".\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mdunnbc22\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wandb.login()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d9f8e4b7",
   "metadata": {},
   "source": [
    "#### Display Library Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a1e3f193",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      Library   Version\n",
      "****************************\n",
      "            Python : 3.9.12\n",
      "             NumPy : 1.24.3\n",
      "  Weights & Biases : 0.15.3\n",
      "            Pandas : 2.0.1\n",
      "             Torch : 2.0.0\n",
      "          Datasets : 2.11.0\n",
      "      Transformers : 4.28.1\n",
      "          Evaluate : 0.4.0\n"
     ]
    }
   ],
   "source": [
    "n = 20\n",
    "\n",
    "print(\"Library  \".rjust(15), \"Version\")\n",
    "print(\"*\" * (n + 8))\n",
    "\n",
    "print(\"Python :\".rjust(n), sys.version[0:6])\n",
    "print(\"NumPy :\".rjust(n), np.__version__)\n",
    "print(\"Weights & Biases :\".rjust(n), wandb.__version__)\n",
    "print(\"Pandas :\".rjust(n), pd.__version__)\n",
    "print(\"Torch :\".rjust(n), torch.__version__)\n",
    "print(\"Datasets :\".rjust(n), datasets.__version__)\n",
    "print(\"Transformers :\".rjust(n), transformers.__version__)\n",
    "print(\"Evaluate :\".rjust(n), evaluate.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "242bbe6e",
   "metadata": {},
   "source": [
    "#### Ingest Data, Drop Unnecessary Columns, & Rename Remaining Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ece9f605",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>april gold down 20 cents to settle at $1,116.1...</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gold suffers third straight daily decline</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Gold futures edge up after two-session decline</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dent research : is gold's day in the sun comin...</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Gold snaps three-day rally as Trump, lawmakers...</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10565</th>\n",
       "      <td>gold seen falling from 3-week high this week</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10566</th>\n",
       "      <td>dominic frisby : now looks like a good time to...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10567</th>\n",
       "      <td>Gold heading for worst week since November on ...</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10568</th>\n",
       "      <td>august gold up $7.60 at $878.80 an ounce on nymex</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10569</th>\n",
       "      <td>december gold down $1 at $749 an ounce on nymex</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10570 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text     label\n",
       "0      april gold down 20 cents to settle at $1,116.1...  negative\n",
       "1              gold suffers third straight daily decline  negative\n",
       "2         Gold futures edge up after two-session decline  positive\n",
       "3      dent research : is gold's day in the sun comin...      none\n",
       "4      Gold snaps three-day rally as Trump, lawmakers...  negative\n",
       "...                                                  ...       ...\n",
       "10565       gold seen falling from 3-week high this week  negative\n",
       "10566  dominic frisby : now looks like a good time to...  positive\n",
       "10567  Gold heading for worst week since November on ...  negative\n",
       "10568  august gold up $7.60 at $878.80 an ounce on nymex  positive\n",
       "10569    december gold down $1 at $749 an ounce on nymex  negative\n",
       "\n",
       "[10570 rows x 2 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_names = [\"Dates\",\n",
    "             \"URL\", \n",
    "             \"text\",\n",
    "             \"Price Direction Up\",\n",
    "             \"Price Direction Constant\",\n",
    "             \"Price Direction Down\",\n",
    "             \"Asset Comparision\",\n",
    "             \"Past Information\",\n",
    "             \"Future Information\",\n",
    "             \"label\",\n",
    "             ]\n",
    "\n",
    "data_types = {\"Dates\": str,\n",
    "             \"URL\": str, \n",
    "             \"text\": str,\n",
    "             \"Price Direction Up\": np.uint8,\n",
    "             \"Price Direction Constant\": np.uint8,\n",
    "             \"Price Direction Down\": np.uint8,\n",
    "             \"Asset Comparision\": np.uint8,\n",
    "             \"Past Information\": np.uint8,\n",
    "             \"Future Information\": np.uint8,\n",
    "             \"label\": str\n",
    "             }\n",
    "\n",
    "data = pd.read_csv(\"gold-dataset.csv\",\n",
    "                   names=col_names,\n",
    "                   dtype=data_types,\n",
    "                   engine=\"c\",\n",
    "                   header=0)\n",
    "\n",
    "data['text'] = data['text'].apply(lambda x: x.strip())\n",
    "\n",
    "data = data.drop(columns=\n",
    "                 [\n",
    "                     'URL',\n",
    "                     \"Price Direction Up\",\n",
    "                     \"Price Direction Constant\",\n",
    "                     \"Price Direction Down\",\n",
    "                     \"Asset Comparision\",\n",
    "                     \"Past Information\",\n",
    "                     \"Future Information\",\n",
    "                     'Dates'\n",
    "                    ]\n",
    "                 )\n",
    "\n",
    "data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "50224af9",
   "metadata": {},
   "source": [
    "#### More Information About Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "640de3ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 10570 entries, 0 to 10569\n",
      "Data columns (total 2 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   text    10570 non-null  object\n",
      " 1   label   10570 non-null  object\n",
      "dtypes: object(2)\n",
      "memory usage: 165.3+ KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "983738d4",
   "metadata": {},
   "source": [
    "#### Title Case All Values in Label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "06988715",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>april gold down 20 cents to settle at $1,116.1...</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gold suffers third straight daily decline</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Gold futures edge up after two-session decline</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dent research : is gold's day in the sun comin...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Gold snaps three-day rally as Trump, lawmakers...</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text     label\n",
       "0  april gold down 20 cents to settle at $1,116.1...  Negative\n",
       "1          gold suffers third straight daily decline  Negative\n",
       "2     Gold futures edge up after two-session decline  Positive\n",
       "3  dent research : is gold's day in the sun comin...      None\n",
       "4  Gold snaps three-day rally as Trump, lawmakers...  Negative"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['label'] = data['label'].apply(lambda x: x.title())\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "09269047",
   "metadata": {},
   "source": [
    "#### Define Function to Create Bar Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5c540131",
   "metadata": {},
   "outputs": [],
   "source": [
    "def barplot_visual(feature_to_visualize: pd.Series,\n",
    "                   plot_title: str,\n",
    "                   x_label: str,\n",
    "                   y_label: str = \"# of Samples\",\n",
    "                   rotation: np.int8 = 0,\n",
    "                   annotation: np.bool_ = True\n",
    "                   ):\n",
    "    \"\"\"\n",
    "    This function returns a histogram for a feature.\n",
    "    \n",
    "    Inputs: \n",
    "    - feature_to_visualize: Feature to plot in the form \n",
    "    of pandas Series.\n",
    "    - plot_title: The title of the plot/visual.\n",
    "    - x_label: The string label value for the x label.\n",
    "    - y_label: The string label value for the y label.\n",
    "    - rotation: The integer value of the rotation of \n",
    "        x-tick values\n",
    "    - \n",
    "    \n",
    "    Output:\n",
    "    - A Bar Plot.\n",
    "    \"\"\"\n",
    "    \n",
    "    assert type(feature_to_visualize) == pd.Series, \\\n",
    "        \"Make sure the dataset input type is a Pandas Series.\"\n",
    "    assert type(plot_title) != 'str', \\\n",
    "        \"The title value must be a string value.\"\n",
    "    assert type(x_label) !=  'str', \\\n",
    "        \"The x label value must be a string value.\"\n",
    "    assert type(y_label) != 'str', \\\n",
    "        \"The y label value must be a string value.\"\n",
    "    assert type(rotation) != np.int8, \\\n",
    "        \"The rotation value must be an integer value.\"\n",
    "    \n",
    "    ax = feature_to_visualize.value_counts().plot.bar( \n",
    "                                                      figsize=(12,8),\n",
    "                                                      rot=rotation, \n",
    "                                                      legend=False,\n",
    "                                                      color=[\"lightgreen\", \n",
    "                                                             \"blue\", \n",
    "                                                             \"brown\", \n",
    "                                                             \"lightblue\",\n",
    "                                                             \"tan\", \n",
    "                                                             \"yellow\", \n",
    "                                                             \"orange\",\n",
    "                                                             \"green\", \n",
    "                                                             \"red\",\n",
    "                                                             \"purple\"]\n",
    "                                                      )\n",
    "    \n",
    "    if annotation:\n",
    "        ax.bar_label(ax.containers[0])\n",
    "    \n",
    "    plt.suptitle('')\n",
    "    plt.title(plot_title)\n",
    "    plt.ylabel(y_label)\n",
    "    plt.xlabel(x_label)\n",
    "    \n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0f372f4c",
   "metadata": {},
   "source": [
    "#### Visualization: Distribution of Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ff001fa1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/YAAAK9CAYAAACO8PdPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABcs0lEQVR4nO3de3zP9f//8ft7ZrPzDNvMYZb1YZMRPmUohzDHiCKUQyvlMx9MJTrIIdFBDh3QyaYPCUWisDCEHMbklByjGBU2c9jGXr8/+nr/erdh02Z7crteLu/Lxev5fL6er8dr9fJ23+tksyzLEgAAAAAAMJJTURcAAAAAAACuH8EeAAAAAACDEewBAAAAADAYwR4AAAAAAIMR7AEAAAAAMBjBHgAAAAAAgxHsAQAAAAAwGMEeAAAAAACDEewBAAAAADAYwR4AgOswYsQI2Wy2G7KtJk2aqEmTJvblxMRE2Ww2zZs374Zsv3fv3qpSpcoN2db1Sk9P1+OPP67AwEDZbDYNGjSoqEsCAOCGIdgDAG55cXFxstls9k+pUqUUFBSkqKgoTZ48WWfOnCmQ7Rw9elQjRoxQcnJygcxXkIpzbXnx6quvKi4uTv369dMnn3yiRx999IpjMzMzNWnSJN15553y9vaWr6+vatSoob59++rHH3+8gVUDAFAwnIu6AAAAiotRo0YpJCREWVlZSklJUWJiogYNGqS33npLCxcuVEREhH3siy++qKFDh+Zr/qNHj2rkyJGqUqWKateunef1li1blq/tXI+r1fbBBx8oOzu70Gv4J1asWKH69evr5ZdfvubYzp0765tvvlG3bt30xBNPKCsrSz/++KMWLVqkBg0aqHr16jegYgAACg7BHgCA/9O6dWvVq1fPvjxs2DCtWLFC7dq10/3336/du3fLzc1NkuTs7Cxn58L9Gj137pzc3d3l4uJSqNu5lpIlSxbp9vPixIkTCg8Pv+a4TZs2adGiRRozZoyef/55h7533nlHp0+fLqQKc7pw4YJcXFzk5MQFlACAf4ZvEgAArqJZs2Z66aWX9PPPP+t///ufvT23e+wTEhLUqFEj+fr6ytPTU9WqVbOHx8TERP373/+WJPXp08d+2X9cXJykP++jv+OOO5SUlKR7771X7u7u9nX/fo/9ZZcuXdLzzz+vwMBAeXh46P7779eRI0ccxlSpUkW9e/fOse5f57xWbbndY3/27Fk9/fTTqlSpklxdXVWtWjW9+eabsizLYZzNZlP//v21YMEC3XHHHXJ1dVWNGjW0ZMmS3H/gf3PixAlFR0crICBApUqVUq1atRQfH2/vv/y8gYMHD2rx4sX22g8dOpTrfPv375ckNWzYMEdfiRIlVKZMGYe2X3/9VdHR0QoKCpKrq6tCQkLUr18/ZWZm2sccOHBADz30kPz8/OTu7q769etr8eLFDvNcrnP27Nl68cUXVaFCBbm7uystLU2StGHDBrVq1Uo+Pj5yd3dX48aNtXbtWoc5zpw5o0GDBqlKlSpydXWVv7+/WrRooS1btuTpZwkAuHlxxh4AgGt49NFH9fzzz2vZsmV64oknch2zc+dOtWvXThERERo1apRcXV21b98+ezgLCwvTqFGjNHz4cPXt21f33HOPJKlBgwb2Of744w+1bt1aDz/8sB555BEFBARcta4xY8bIZrPpueee04kTJzRx4kQ1b95cycnJ9isL8iIvtf2VZVm6//77tXLlSkVHR6t27dpaunSpnn32Wf3666+aMGGCw/jvvvtOX3zxhf7zn//Iy8tLkydPVufOnXX48OEcQfqvzp8/ryZNmmjfvn3q37+/QkJCNHfuXPXu3VunT5/WwIEDFRYWpk8++USxsbGqWLGinn76aUlSuXLlcp0zODhYkjRz5kw1bNjwqlddHD16VHfddZdOnz6tvn37qnr16vr11181b948nTt3Ti4uLjp+/LgaNGigc+fOacCAASpTpozi4+N1//33a968eXrggQcc5hw9erRcXFz0zDPPKCMjQy4uLlqxYoVat26tunXr6uWXX5aTk5OmT5+uZs2aac2aNbrrrrskSU899ZTmzZun/v37Kzw8XH/88Ye+++477d69W3Xq1LnifgAAbgEWAAC3uOnTp1uSrE2bNl1xjI+Pj3XnnXfal19++WXrr1+jEyZMsCRZv/322xXn2LRpkyXJmj59eo6+xo0bW5KsqVOn5trXuHFj+/LKlSstSVaFChWstLQ0e/ucOXMsSdakSZPsbcHBwVavXr2uOefVauvVq5cVHBxsX16wYIElyXrllVccxj344IOWzWaz9u3bZ2+TZLm4uDi0bdu2zZJkvf322zm29VcTJ060JFn/+9//7G2ZmZlWZGSk5enp6bDvwcHBVtu2ba86n2VZVnZ2tv1nHRAQYHXr1s169913rZ9//jnH2J49e1pOTk65/n+RnZ1tWZZlDRo0yJJkrVmzxt535swZKyQkxKpSpYp16dIly7L+/3+z2267zTp37pzDPLfffrsVFRVln9OyLOvcuXNWSEiI1aJFC3ubj4+PFRMTc819BADcergUHwCAPPD09Lzq0/F9fX0lSV9++eV1P2jO1dVVffr0yfP4nj17ysvLy7784IMPqnz58vr666+va/t59fXXX6tEiRIaMGCAQ/vTTz8ty7L0zTffOLQ3b95cVatWtS9HRETI29tbBw4cuOZ2AgMD1a1bN3tbyZIlNWDAAKWnp2vVqlX5rt1ms2np0qV65ZVXVLp0aX366aeKiYlRcHCwunbtar/HPjs7WwsWLFD79u0dnrvw13ku13jXXXepUaNG9j5PT0/17dtXhw4d0q5duxzW69Wrl8PVFMnJydq7d6+6d++uP/74Q7///rt+//13nT17Vvfdd59Wr15t///J19dXGzZs0NGjR/O93wCAmxvBHgCAPEhPT3cI0X/XtWtXNWzYUI8//rgCAgL08MMPa86cOfkK+RUqVMjXg/Juv/12h2WbzabQ0NAr3l9eUH7++WcFBQXl+HmEhYXZ+/+qcuXKOeYoXbq0Tp06dc3t3H777TkeLnel7eSVq6urXnjhBe3evVtHjx7Vp59+qvr162vOnDnq37+/JOm3335TWlqa7rjjjmvWWK1atRztV6oxJCTEYXnv3r2S/gz85cqVc/h8+OGHysjIUGpqqiTp9ddf144dO1SpUiXdddddGjFixDV/OQIAuDUQ7AEAuIZffvlFqampCg0NveIYNzc3rV69Wt9++60effRR/fDDD+ratatatGihS5cu5Wk7+bkvPq/+/oC/y/JaU0EoUaJEru3W3x60VxTKly+vhx9+WKtXr9btt9+uOXPm6OLFi4W2vb//N778i5833nhDCQkJuX48PT0lSV26dNGBAwf09ttvKygoSG+88YZq1KiR4woJAMCth2APAMA1fPLJJ5KkqKioq45zcnLSfffdp7feeku7du3SmDFjtGLFCq1cuVLSlUP29bp8tvcyy7K0b98+hyfYly5dOtdXuP39THJ+agsODtbRo0dz3Jrw448/2vsLQnBwsPbu3ZvjqoeC3o705yX+ERERysrK0u+//65y5crJ29tbO3bsuGaNe/bsydGe1xov36Lg7e2t5s2b5/r56+sGy5cvr//85z9asGCBDh48qDJlymjMmDH53V0AwE2GYA8AwFWsWLFCo0ePVkhIiHr06HHFcSdPnszRVrt2bUlSRkaGJMnDw0OSCuxd6TNmzHAI1/PmzdOxY8fUunVre1vVqlX1/fffO7yebdGiRTlei5ef2tq0aaNLly7pnXfecWifMGGCbDabw/b/iTZt2iglJUWfffaZve3ixYt6++235enpqcaNG+d7zr179+rw4cM52k+fPq3169erdOnSKleunJycnNSxY0d99dVX2rx5c47xl682aNOmjTZu3Kj169fb+86ePav3339fVapUUXh4+FXrqVu3rqpWrao333xT6enpOfp/++03SX9eYXH5kvzL/P39FRQUZP//CwBw6+J1dwAA/J9vvvlGP/74oy5evKjjx49rxYoVSkhIUHBwsBYuXKhSpUpdcd1Ro0Zp9erVatu2rYKDg3XixAm99957qlixov3BalWrVpWvr6+mTp0qLy8veXh46O67785x33Ve+fn5qVGjRurTp4+OHz+uiRMnKjQ01OGVfI8//rjmzZunVq1aqUuXLtq/f7/+97//OTzMLr+1tW/fXk2bNtULL7ygQ4cOqVatWlq2bJm+/PJLDRo0KMfc16tv376aNm2aevfuraSkJFWpUkXz5s3T2rVrNXHixKs+8+BKtm3bpu7du6t169a655575Ofnp19//VXx8fE6evSoJk6caL914NVXX9WyZcvUuHFj9e3bV2FhYTp27Jjmzp2r7777Tr6+vho6dKg+/fRTtW7dWgMGDJCfn5/i4+N18OBBff755zmeD/B3Tk5O+vDDD9W6dWvVqFFDffr0UYUKFfTrr79q5cqV8vb21ldffaUzZ86oYsWKevDBB1WrVi15enrq22+/1aZNmzR+/Pjr+vkCAG4iRftQfgAAit7l191d/ri4uFiBgYFWixYtrEmTJjm8Vu2yv7/ubvny5VaHDh2soKAgy8XFxQoKCrK6detm/fTTTw7rffnll1Z4eLjl7Ozs8Hq5xo0bWzVq1Mi1viu97u7TTz+1hg0bZvn7+1tubm5W27Ztc31t2/jx460KFSpYrq6uVsOGDa3NmzfnmPNqtf39dXeW9ecr3WJjY62goCCrZMmS1u2332698cYbDq9ss6w/X3eX2yvarvQavr87fvy41adPH6ts2bKWi4uLVbNmzVxfyZfX190dP37cGjdunNW4cWOrfPnylrOzs1W6dGmrWbNm1rx583KM//nnn62ePXta5cqVs1xdXa3bbrvNiomJsTIyMuxj9u/fbz344IOWr6+vVapUKeuuu+6yFi1a5DDP5f9mc+fOzbWurVu3Wp06dbLKlCljubq6WsHBwVaXLl2s5cuXW5ZlWRkZGdazzz5r1apVy/Ly8rI8PDysWrVqWe+999419xkAcPOzWVYxeHINAAAAAAC4LtxjDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGMy5qAswQXZ2to4ePSovLy/ZbLaiLgcAAAAAcJOzLEtnzpxRUFCQnJyufk6eYJ8HR48eVaVKlYq6DAAAAADALebIkSOqWLHiVccQ7PPAy8tL0p8/UG9v7yKuBgAAAABws0tLS1OlSpXsefRqCPZ5cPnye29vb4I9AAAAAOCGycvt4Dw8DwAAAAAAgxHsAQAAAAAwGMEeAAAAAACDEewBAAAAADAYwR4AAAAAAIMR7AEAAAAAMBjBHgAAAAAAgxHsAQAAAAAwGMEeAAAAAACDEewBAAAAADAYwR4AAAAAAIMR7AEAAAAAMBjBHgAAAAAAgxHsAQAAAAAwGMEeN4Vx48bJZrNp0KBBOfosy1Lr1q1ls9m0YMECh74BAwaobt26cnV1Ve3atXOsm5iYqA4dOqh8+fLy8PBQ7dq1NXPmzMLZCQAAAAC4DgR7GG/Tpk2aNm2aIiIicu2fOHGibDbbFdd/7LHH1LVr11z71q1bp4iICH3++ef64Ycf1KdPH/Xs2VOLFi0qkNoBAAAA4J9yLuoCgH8iPT1dPXr00AcffKBXXnklR39ycrLGjx+vzZs3q3z58jn6J0+eLEn67bff9MMPP+Tof/755x2WBw4cqGXLlumLL75Qu3btCmgvAAAAAOD6ccYeRouJiVHbtm3VvHnzHH3nzp1T9+7d9e677yowMLDAtpmamio/P78Cmw8AAAAA/gnO2MNYs2fP1pYtW7Rp06Zc+2NjY9WgQQN16NChwLY5Z84c+6X/AAAAAFAcEOxhpCNHjmjgwIFKSEhQqVKlcvQvXLhQK1as0NatWwtsmytXrlSfPn30wQcfqEaNGgU2LwAAAAD8E1yKDyMlJSXpxIkTqlOnjpydneXs7KxVq1Zp8uTJcnZ2VkJCgvbv3y9fX197vyR17txZTZo0yff2Vq1apfbt22vChAnq2bNnAe8NAAAAAFw/ztjDSPfdd5+2b9/u0NanTx9Vr15dzz33nMqWLasnn3zSob9mzZqaMGGC2rdvn69tJSYmql27dnrttdfUt2/ff1w7AAAAABQkgj2M5OXlpTvuuMOhzcPDQ2XKlLG35/bAvMqVKyskJMS+vG/fPqWnpyslJUXnz59XcnKyJCk8PFwuLi5auXKl2rVrp4EDB6pz585KSUmRJLm4uPAAPQAAAADFAsEet7THH39cq1atsi/feeedkqSDBw+qSpUqio+P17lz5zR27FiNHTvWPq5x48ZKTEy80eUCAAAAQA42y7Ksoi6iuEtLS5OPj49SU1Pl7e1d1OUAAAAAAG5y+cmhPDwPAAAAAACDcSk+cph0alJRl4ACNLD0wKIuAQAAAEAh4ow9AAAAAAAGI9gDAAAAAGAwgj0AAAAAAAYj2AMAAAAAYDCCPQAAAAAABiPYAwAAAABgMII9AAAAAAAGI9gDAAAAAGAwgj0AAAAAAAYj2AMAAAAAYDCCPQAAAAAABiPYAwAAAABgMII9AAAAAAAGI9gDAAAAAGAwgj0AAAAAAAYj2AMAAAAAYDCCPQAAAAAABiPYAwAAAABgMII9AAAAAAAGI9gDAAAAAGAwgj0AAAAAAAYj2AMAAAAAYDCCPQAAAAAABiPYAwAAAABgMII9AAAAAAAGI9gDAAAAAGAwgj0AAAAAAAYj2AMAAAAAYDCCPQAAAAAABiPYAwAAAABgMII9AAAAAAAGI9gDAAAAAGAwgj0AAAAAAAYj2AMAAAAAYDCCPQAAAAAABiPYAwAAAABgMII9AAAAAAAGI9gDAAAAAGAwgj0AAAAAAAYj2AMAAAAAYDCCPQAAAAAABiPYAwAAAABgMII9AAAAAAAGI9gDAAAAAGAwgj0AAAAAAAYj2AMAAAAAYDCCPQAAAAAABiPYAwAAAABgMII9AAAAAAAGI9gDAAAAAGCwYhPsx40bJ5vNpkGDBtnbLly4oJiYGJUpU0aenp7q3Lmzjh8/7rDe4cOH1bZtW7m7u8vf31/PPvusLl686DAmMTFRderUkaurq0JDQxUXF3cD9ggAAAAAgMJXLIL9pk2bNG3aNEVERDi0x8bG6quvvtLcuXO1atUqHT16VJ06dbL3X7p0SW3btlVmZqbWrVun+Ph4xcXFafjw4fYxBw8eVNu2bdW0aVMlJydr0KBBevzxx7V06dIbtn8AAAAAABSWIg/26enp6tGjhz744AOVLl3a3p6amqqPPvpIb731lpo1a6a6detq+vTpWrdunb7//ntJ0rJly7Rr1y7973//U+3atdW6dWuNHj1a7777rjIzMyVJU6dOVUhIiMaPH6+wsDD1799fDz74oCZMmFAk+wsAAAAAQEEq8mAfExOjtm3bqnnz5g7tSUlJysrKcmivXr26KleurPXr10uS1q9fr5o1ayogIMA+JioqSmlpadq5c6d9zN/njoqKss+Rm4yMDKWlpTl8AAAAAAAojpyLcuOzZ8/Wli1btGnTphx9KSkpcnFxka+vr0N7QECAUlJS7GP+Guov91/uu9qYtLQ0nT9/Xm5ubjm2PXbsWI0cOfK69wsAAAAAgBulyM7YHzlyRAMHDtTMmTNVqlSpoiojV8OGDVNqaqr9c+TIkaIuCQAAAACAXBVZsE9KStKJEydUp04dOTs7y9nZWatWrdLkyZPl7OysgIAAZWZm6vTp0w7rHT9+XIGBgZKkwMDAHE/Jv7x8rTHe3t65nq2XJFdXV3l7ezt8AAAAAAAojoos2N93333avn27kpOT7Z969eqpR48e9j+XLFlSy5cvt6+zZ88eHT58WJGRkZKkyMhIbd++XSdOnLCPSUhIkLe3t8LDw+1j/jrH5TGX5wAAAAAAwGRFdo+9l5eX7rjjDoc2Dw8PlSlTxt4eHR2twYMHy8/PT97e3vrvf/+ryMhI1a9fX5LUsmVLhYeH69FHH9Xrr7+ulJQUvfjii4qJiZGrq6sk6amnntI777yjIUOG6LHHHtOKFSs0Z84cLV68+MbuMAAAAAAAhaBIH553LRMmTJCTk5M6d+6sjIwMRUVF6b333rP3lyhRQosWLVK/fv0UGRkpDw8P9erVS6NGjbKPCQkJ0eLFixUbG6tJkyapYsWK+vDDDxUVFVUUuwQAAAAAQIGyWZZlFXURxV1aWpp8fHyUmpp6S9xvP+nUpKIuAQVoYOmBRV0CAAAAgHzKTw4t8vfYAwAAAACA60ewBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAFLgpU6YoIiJC3t7e8vb2VmRkpL755ht7f0pKih599FEFBgbKw8NDderU0eeff+4wx5gxY9SgQQO5u7vL19f3qtv7448/VLFiRdlsNp0+fboQ9ggAAKD4ItgDAApcxYoVNW7cOCUlJWnz5s1q1qyZOnTooJ07d0qSevbsqT179mjhwoXavn27OnXqpC5dumjr1q32OTIzM/XQQw+pX79+19xedHS0IiIiCm1/AAAAijOCPQCgwLVv315t2rTR7bffrn/9618aM2aMPD099f3330uS1q1bp//+97+66667dNttt+nFF1+Ur6+vkpKS7HOMHDlSsbGxqlmz5lW3NWXKFJ0+fVrPPPNMoe4TAABAcUWwBwAUqkuXLmn27Nk6e/asIiMjJUkNGjTQZ599ppMnTyo7O1uzZ8/WhQsX1KRJk3zNvWvXLo0aNUozZsyQkxNfaQAA4NbkXNQFAABuTtu3b1dkZKQuXLggT09PzZ8/X+Hh4ZKkOXPmqGvXripTpoycnZ3l7u6u+fPnKzQ0NM/zZ2RkqFu3bnrjjTdUuXJlHThwoLB2BQAAoFgj2AMACkW1atWUnJys1NRUzZs3T7169dKqVasUHh6ul156SadPn9a3336rsmXLasGCBerSpYvWrFlzzUvvLxs2bJjCwsL0yCOPFPKeAAAAFG8EewBAoXBxcbGfga9bt642bdqkSZMmaciQIXrnnXe0Y8cO1ahRQ5JUq1YtrVmzRu+++66mTp2ap/lXrFih7du3a968eZIky7IkSWXLltULL7ygkSNHFsJeAQAAFD8EewDADZGdna2MjAydO3dOknLcE1+iRAllZ2fneb7PP/9c58+fty9v2rRJjz32mNasWaOqVasWTNEAAAAGINgDAArcsGHD1Lp1a1WuXFlnzpzRrFmzlJiYqKVLl6p69eoKDQ3Vk08+qTfffFNlypTRggULlJCQoEWLFtnnOHz4sE6ePKnDhw/r0qVLSk5OliSFhobK09MzR3j//fffJUlhYWHXfO89AADAzYRgDwAocCdOnFDPnj117Ngx+fj4KCIiQkuXLlWLFi0kSV9//bWGDh2q9u3bKz09XaGhoYqPj1ebNm3scwwfPlzx8fH25TvvvFOStHLlynw/PR8AAOBmZrMu35SIK0pLS5OPj49SU1Pl7e1d1OUUukmnJhV1CShAA0sPLOoSAAAAAORTfnIoL/0FAAAAAMBgBHsAAAAAAAzGPfYAYBCbragrQEHiZjgAAFAQOGMPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBijTYT5kyRREREfL29pa3t7ciIyP1zTff2PsvXLigmJgYlSlTRp6enurcubOOHz/uMMfhw4fVtm1bubu7y9/fX88++6wuXrzoMCYxMVF16tSRq6urQkNDFRcXdyN2DwAAAACAQlekwb5ixYoaN26ckpKStHnzZjVr1kwdOnTQzp07JUmxsbH66quvNHfuXK1atUpHjx5Vp06d7OtfunRJbdu2VWZmptatW6f4+HjFxcVp+PDh9jEHDx5U27Zt1bRpUyUnJ2vQoEF6/PHHtXTp0hu+vwAAAAAAFDSbZVlWURfxV35+fnrjjTf04IMPqly5cpo1a5YefPBBSdKPP/6osLAwrV+/XvXr19c333yjdu3a6ejRowoICJAkTZ06Vc8995x+++03ubi46LnnntPixYu1Y8cO+zYefvhhnT59WkuWLMlTTWlpafLx8VFqaqq8vb0LfqeLmUmnJhV1CShAA0sPLOoSUIBstqKuAAWpeH0DAwCA4iQ/ObTY3GN/6dIlzZ49W2fPnlVkZKSSkpKUlZWl5s2b28dUr15dlStX1vr16yVJ69evV82aNe2hXpKioqKUlpZmP+u/fv16hzkuj7k8R24yMjKUlpbm8AEAAAAAoDgq8mC/fft2eXp6ytXVVU899ZTmz5+v8PBwpaSkyMXFRb6+vg7jAwIClJKSIklKSUlxCPWX+y/3XW1MWlqazp8/n2tNY8eOlY+Pj/1TqVKlgthVAAAAAAAKXJEH+2rVqik5OVkbNmxQv3791KtXL+3atatIaxo2bJhSU1PtnyNHjhRpPQAAAAAAXIlzURfg4uKi0NBQSVLdunW1adMmTZo0SV27dlVmZqZOnz7tcNb++PHjCgwMlCQFBgZq48aNDvNdfmr+X8f8/Un6x48fl7e3t9zc3HKtydXVVa6urgWyfwAAAAAAFKYiP2P/d9nZ2crIyFDdunVVsmRJLV++3N63Z88eHT58WJGRkZKkyMhIbd++XSdOnLCPSUhIkLe3t8LDw+1j/jrH5TGX5wAAAAAAwGRFesZ+2LBhat26tSpXrqwzZ85o1qxZSkxM1NKlS+Xj46Po6GgNHjxYfn5+8vb21n//+19FRkaqfv36kqSWLVsqPDxcjz76qF5//XWlpKToxRdfVExMjP2M+1NPPaV33nlHQ4YM0WOPPaYVK1Zozpw5Wrx4cVHuOgAAAAAABaJIg/2JEyfUs2dPHTt2TD4+PoqIiNDSpUvVokULSdKECRPk5OSkzp07KyMjQ1FRUXrvvffs65coUUKLFi1Sv379FBkZKQ8PD/Xq1UujRo2yjwkJCdHixYsVGxurSZMmqWLFivrwww8VFRV1w/cXAAAAAICCVuzeY18c8R57mIz32N9ceI/9zYVvYAAAcCVGvsceAAAAAADkH8EeAAAAAACDEewBAAAAADAYwR4AAAAAAIMR7AEAAAAAMBjBHgAAAAAAgxHsAQAAAAAwGMEeAAAAAACDEewBAAAAADAYwR4AAAAAAIMR7AEAAAAAMBjBHgAAAAAAgxHsAQAAAAAwGMEeAAAAAACDEewBAAAAADAYwR4AAAAAAIMR7AEAAAAAMBjBHgAAAAAAgxHsAQAAAAAwGMEeAAAAAACDEewBAAAAADAYwR4AAAAAAIMR7AEAAAAAMBjBHgAAAAAAgxHsAQAAAAAwGMEeAAAAAACDEewBAAAAADAYwR4AAAAAAIMR7AEAAAAAMBjBHgAAAAAAgxHsAQAAAAAwGMEeAAAAAACDEewBAAAAADAYwR4AAAAAAIMR7AEAAAAAMBjBHgAAAAAAgxHsAQAAAAAwGMEeAAAAAACDEewBAAAAADBYvoP9kSNH9Msvv9iXN27cqEGDBun9998v0MIAAAAAAMC15TvYd+/eXStXrpQkpaSkqEWLFtq4caNeeOEFjRo1qsALBAAAAAAAV5bvYL9jxw7dddddkqQ5c+bojjvu0Lp16zRz5kzFxcUVdH0AAAAAAOAq8h3ss7Ky5OrqKkn69ttvdf/990uSqlevrmPHjhVsdQAAAAAA4KryHexr1KihqVOnas2aNUpISFCrVq0kSUePHlWZMmUKvEAAAAAAAHBl+Q72r732mqZNm6YmTZqoW7duqlWrliRp4cKF9kv0AQAAAADAjeGc3xWaNGmi33//XWlpaSpdurS9vW/fvnJ3dy/Q4gAAAAAAwNVd13vsLctSUlKSpk2bpjNnzkiSXFxcCPYAAAAAANxg+T5j//PPP6tVq1Y6fPiwMjIy1KJFC3l5eem1115TRkaGpk6dWhh1AgAAAACAXOT7jP3AgQNVr149nTp1Sm5ubvb2Bx54QMuXLy/Q4gAAAAAAwNXl+4z9mjVrtG7dOrm4uDi0V6lSRb/++muBFQYAAAAAAK4t32fss7OzdenSpRztv/zyi7y8vAqkKAAAAAAAkDf5DvYtW7bUxIkT7cs2m03p6el6+eWX1aZNm4KsDQAAAAAAXEO+L8UfP368oqKiFB4ergsXLqh79+7au3evypYtq08//bQwagQAAAAAAFeQ72BfsWJFbdu2TbNnz9YPP/yg9PR0RUdHq0ePHg4P0wMAAAAAAIUv38FekpydnfXII48UdC0AAAAAACCf8hTsFy5cmOcJ77///usuBgAAAAAA5E+egn3Hjh3zNJnNZsv1ifkAAAAAAKBw5CnYZ2dnF3YdAAAAAADgOuT7dXcAAAAAAKD4uK5gv3z5crVr105Vq1ZV1apV1a5dO3377bcFXRsAAAAAALiGfAf79957T61atZKXl5cGDhyogQMHytvbW23atNG7775bGDUCAAAAAIAryPfr7l599VVNmDBB/fv3t7cNGDBADRs21KuvvqqYmJgCLRAAAAAAAFxZvs/Ynz59Wq1atcrR3rJlS6WmphZIUQAAAAAAIG/yHezvv/9+zZ8/P0f7l19+qXbt2hVIUQAAAAAAIG/yfSl+eHi4xowZo8TEREVGRkqSvv/+e61du1ZPP/20Jk+ebB87YMCAgqsUAAAAAADkYLMsy8rPCiEhIXmb2GbTgQMHrquo4iYtLU0+Pj5KTU2Vt7d3UZdT6CadmlTUJaAADSw9sKhLQAGy2Yq6AhSk/H0DAwCAW0l+cmi+z9gfPHjwugsDAAAAAAAF67reYw8AAAAAAIqHfJ+xtyxL8+bN08qVK3XixAllZ2c79H/xxRcFVhwAAAAAALi6fAf7QYMGadq0aWratKkCAgJk44ZPAAAAAACKTL6D/SeffKIvvvhCbdq0KYx6AAAAAABAPuT7HnsfHx/ddttthVELAAAAAADIp3wH+xEjRmjkyJE6f/58YdQDAAAAAADyId+X4nfp0kWffvqp/P39VaVKFZUsWdKhf8uWLQVWHAAAAAAAuLp8B/tevXopKSlJjzzyCA/PAwAAAACgiOU72C9evFhLly5Vo0aNCqMeAAAAAACQD/m+x75SpUry9vYujFoAAAAAAEA+5TvYjx8/XkOGDNGhQ4cKoRwAAAAAAJAf+b4U/5FHHtG5c+dUtWpVubu753h43smTJwusOAAAAAAAcHX5DvYTJ04shDIAAAAAAMD1uK6n4gMAAAAAgOIh38H+ry5cuKDMzEyHNh6sBwAAAADAjZPvh+edPXtW/fv3l7+/vzw8PFS6dGmHDwAAAAAAuHHyHeyHDBmiFStWaMqUKXJ1ddWHH36okSNHKigoSDNmzCiMGgEAAAAAwBXk+1L8r776SjNmzFCTJk3Up08f3XPPPQoNDVVwcLBmzpypHj16FEadAAAAAAAgF/k+Y3/y5Enddtttkv68n/7y6+0aNWqk1atXF2x1AAAAAADgqvId7G+77TYdPHhQklS9enXNmTNH0p9n8n19fQu0OAAAAAAAcHX5DvZ9+vTRtm3bJElDhw7Vu+++q1KlSik2NlbPPvtsgRcIAAAAAACuLN/32MfGxtr/3Lx5c+3evVtbtmxRaGioIiIiCrQ4AAAAAABwdf/oPfaSVKVKFVWpUqUASgEAAAAAAPmV50vx169fr0WLFjm0zZgxQyEhIfL391ffvn2VkZFR4AUCAAAAAIAry3OwHzVqlHbu3Glf3r59u6Kjo9W8eXMNHTpUX331lcaOHVsoRQIAAAAAgNzlOdgnJyfrvvvusy/Pnj1bd999tz744AMNHjxYkydPtj8hHwAAAAAA3Bh5DvanTp1SQECAfXnVqlVq3bq1ffnf//63jhw5UrDVAQAAAACAq8pzsA8ICLC/vz4zM1NbtmxR/fr17f1nzpxRyZIlC75CAAAAAABwRXkO9m3atNHQoUO1Zs0aDRs2TO7u7rrnnnvs/T/88IOqVq1aKEUCAAAAAIDc5fl1d6NHj1anTp3UuHFjeXp6Kj4+Xi4uLvb+jz/+WC1btiyUIgEAAAAAQO7yHOzLli2r1atXKzU1VZ6enipRooRD/9y5c+Xp6VngBQIAAAAAgCvLc7C/zMfHJ9d2Pz+/f1wMAAAAAADInzzfYw8AAAAAAIofgj0AAAAAAAYj2AMAAAAAYLA8Bfs6dero1KlTkqRRo0bp3LlzhVoUAAAAAADImzwF+927d+vs2bOSpJEjRyo9Pb1QiwIAAAAAAHmTp6fi165dW3369FGjRo1kWZbefPPNK77abvjw4QVaIAAAAAAAuLI8Bfu4uDi9/PLLWrRokWw2m7755hs5O+dc1WazEewBAAAAALiB8hTsq1WrptmzZ0uSnJyctHz5cvn7+xdqYQAAAAAA4NryFOz/Kjs7uzDqAAAAAAAA1yHfwV6S9u/fr4kTJ2r37t2SpPDwcA0cOFBVq1Yt0OIAAAAAAMDV5fs99kuXLlV4eLg2btyoiIgIRUREaMOGDapRo4YSEhIKo0YAAAAAAHAF+T5jP3ToUMXGxmrcuHE52p977jm1aNGiwIoDAAAAAABXl+8z9rt371Z0dHSO9scee0y7du0qkKIAAAAAAEDe5DvYlytXTsnJyTnak5OTeVI+AAAAAAA3WL4vxX/iiSfUt29fHThwQA0aNJAkrV27Vq+99poGDx5c4AUCAAAAAIAry3ewf+mll+Tl5aXx48dr2LBhkqSgoCCNGDFCAwYMKPACAQAAAADAleU72NtsNsXGxio2NlZnzpyRJHl5eRV4YQAAAAAA4Nqu6z32lxHoAQAAAAAoWvl+eB4AAAAAACg+CPYAAAC3kNWrV6t9+/YKCgqSzWbTggULHPqPHz+u3r17KygoSO7u7mrVqpX27t2bY57169erWbNm8vDwkLe3t+69916dP3/e3v/TTz+pQ4cOKlu2rLy9vdWoUSOtXLmysHcPAG5JBHsAAIBbyNmzZ1WrVi29++67Ofosy1LHjh114MABffnll9q6dauCg4PVvHlznT171j5u/fr1atWqlVq2bKmNGzdq06ZN6t+/v5yc/v8/Ldu1a6eLFy9qxYoVSkpKUq1atdSuXTulpKTckP0EgFuJzbIsq6iLKO7S0tLk4+Oj1NRUeXt7F3U5hW7SqUlFXQIK0MDSA4u6BBQgm62oK0BB4hsYRc1ms2n+/Pnq2LGjpD/PslerVk07duxQjRo1JEnZ2dkKDAzUq6++qscff1ySVL9+fbVo0UKjR4/Odd7ff/9d5cqV0+rVq3XPPfdIks6cOSNvb28lJCSoefPmhb9zAGC4/OTQ6zpj379/f508efK6igMAAEDxlJGRIUkqVaqUvc3JyUmurq767rvvJEknTpzQhg0b5O/vrwYNGiggIECNGze290tSmTJlVK1aNc2YMUNnz57VxYsXNW3aNPn7+6tu3bo3dqcA4BaQ52D/yy+/2P88a9YspaenS5Jq1qypI0eOXNfGx44dq3//+9/y8vKSv7+/OnbsqD179jiMuXDhgmJiYlSmTBl5enqqc+fOOn78uMOYw4cPq23btnJ3d5e/v7+effZZXbx40WFMYmKi6tSpI1dXV4WGhiouLu66agYAALhZVa9eXZUrV9awYcN06tQpZWZm6rXXXtMvv/yiY8eOSZIOHDggSRoxYoSeeOIJLVmyRHXq1NF9991nvxffZrPp22+/1datW+Xl5aVSpUrprbfe0pIlS1S6dOki2z8AuFnlOdhXr15dwcHB6t69uy5cuGAP84cOHVJWVtZ1bXzVqlWKiYnR999/r4SEBGVlZally5YO93DFxsbqq6++0ty5c7Vq1SodPXpUnTp1svdfunRJbdu2VWZmptatW6f4+HjFxcVp+PDh9jEHDx5U27Zt1bRpUyUnJ2vQoEF6/PHHtXTp0uuqGwAA4GZUsmRJffHFF/rpp5/k5+cnd3d3rVy5Uq1bt7bfP5+dnS1JevLJJ9WnTx/deeedmjBhgqpVq6aPP/5Y0p/36sfExMjf319r1qzRxo0b1bFjR7Vv397+CwIAQMHJ83vsT58+rS1btmjNmjX64osv1KZNGwUEBCgjI0NLly5Vp06dFBAQkK+NL1myxGE5Li5O/v7+SkpK0r333qvU1FR99NFHmjVrlpo1ayZJmj59usLCwvT999+rfv36WrZsmXbt2qVvv/1WAQEBql27tkaPHq3nnntOI0aMkIuLi6ZOnaqQkBCNHz9ekhQWFqbvvvtOEyZMUFRUVL5qBgAAuJnVrVtXycnJSk1NVWZmpsqVK6e7775b9erVkySVL19ekhQeHu6wXlhYmA4fPixJWrFihRYtWqRTp07Z7wt97733lJCQoPj4eA0dOvQG7hEA3PzyfMY+KytLd911l55++mm5ublp69atmj59ukqUKKGPP/5YISEhqlat2j8qJjU1VZLk5+cnSUpKSlJWVpbDA1YuXyK2fv16SX8+lbVmzZoOv1SIiopSWlqadu7caR/z94e0REVF2ef4u4yMDKWlpTl8AAAAbiU+Pj4qV66c9u7dq82bN6tDhw6SpCpVqigoKCjH7ZM//fSTgoODJUnnzp2TJIen5F9evnzGHwBQcPJ8xt7X11e1a9dWw4YNlZmZqfPnz6thw4ZydnbWZ599pgoVKmjTpk3XXUh2drYGDRqkhg0b6o477pAkpaSkyMXFRb6+vg5jAwIC7K9KSUlJyXGlwOXla41JS0vT+fPn5ebm5tA3duxYjRw58rr3BQAAoLhKT0/Xvn377MsHDx5UcnKy/Pz8VLlyZc2dO1flypVT5cqVtX37dg0cOFAdO3ZUy5YtJf15//yzzz6rl19+WbVq1VLt2rUVHx+vH3/8UfPmzZMkRUZGqnTp0urVq5eGDx8uNzc3ffDBB/bbIwEABSvPwf7XX3/V+vXrtW7dOl28eFF169bVv//9b2VmZmrLli2qWLGiGjVqdN2FxMTEaMeOHQ5PVC0qw4YN0+DBg+3LaWlpqlSpUhFWBAAAUDA2b96spk2b2pcv/5unV69eiouL07FjxzR48GAdP35c5cuXV8+ePfXSSy85zDFo0CBduHBBsbGxOnnypGrVqqWEhARVrVpVklS2bFktWbJEL7zwgpo1a6asrCzVqFFDX375pWrVqnXjdhYAbhF5DvZly5ZV+/bt1b59e02dOlWrV6/W7t271bNnTz3zzDN69NFHddddd2nVqlX5LqJ///5atGiRVq9erYoVK9rbAwMDlZmZqdOnTzuctT9+/LgCAwPtYzZu3Ogw3+Wn5v91zN+fpH/8+HF5e3vnOFsvSa6urnJ1dc33fgAAABR3TZo0kWVZV+wfMGCABgwYcM15hg4detV75evVq8eDigHgBrmu99hLf9531aVLF5UsWVIrVqzQwYMH9Z///Cdfc1iWpf79+2v+/PlasWKFQkJCHPrr1q2rkiVLavny5fa2PXv26PDhw4qMjJT056Ve27dv14kTJ+xjEhIS5O3tbX+oS2RkpMMcl8dcngMAAAAAAFPl+Yz9X/3www+qUKGCJCk4OFglS5ZUYGCgunbtmq95YmJiNGvWLH355Zfy8vKy3xPv4+MjNzc3+fj4KDo6WoMHD5afn5+8vb313//+V5GRkapfv74kqWXLlgoPD9ejjz6q119/XSkpKXrxxRcVExNjP+v+1FNP6Z133tGQIUP02GOPacWKFZozZ44WL158PbsPAACQq1k1ahR1CShA3f/vQcwAUNxdV7D/6/3mO3bsuO6NT5kyRdKfl4T91fTp09W7d29J0oQJE+Tk5KTOnTsrIyNDUVFReu+99+xjS5QooUWLFqlfv36KjIyUh4eHevXqpVGjRtnHhISEaPHixYqNjdWkSZNUsWJFffjhh7zqDgAAAABgPJt1tZusIOnPh+f5+PgoNTXV/i7Wm9mkU5OKugQUoIGlBxZ1CShANltRV4CCxDfwzYcz9jcXztgDKEr5yaHXfY89AAAAAAAoegR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMRrAHAAAAAMBgBHsAAAAAAAxGsAcAAAAAwGAEewAAAAAADEawBwAAAADAYAR7AAAAAAAMVqTBfvXq1Wrfvr2CgoJks9m0YMECh37LsjR8+HCVL19ebm5uat68ufbu3esw5uTJk+rRo4e8vb3l6+ur6OhopaenO4z54YcfdM8996hUqVKqVKmSXn/99cLeNQAAAAAAbogiDfZnz55VrVq19O677+ba//rrr2vy5MmaOnWqNmzYIA8PD0VFRenChQv2MT169NDOnTuVkJCgRYsWafXq1erbt6+9Py0tTS1btlRwcLCSkpL0xhtvaMSIEXr//fcLff8AAAAAAChszkW58datW6t169a59lmWpYkTJ+rFF19Uhw4dJEkzZsxQQECAFixYoIcffli7d+/WkiVLtGnTJtWrV0+S9Pbbb6tNmzZ68803FRQUpJkzZyozM1Mff/yxXFxcVKNGDSUnJ+utt95y+AUAAAAAAAAmKrb32B88eFApKSlq3ry5vc3Hx0d333231q9fL0lav369fH197aFekpo3by4nJydt2LDBPubee++Vi4uLfUxUVJT27NmjU6dO5brtjIwMpaWlOXwAAAAAACiOim2wT0lJkSQFBAQ4tAcEBNj7UlJS5O/v79Dv7OwsPz8/hzG5zfHXbfzd2LFj5ePjY/9UqlTpn+8QAAAAAACFoNgG+6I0bNgwpaam2j9Hjhwp6pIAAAAAAMhVsQ32gYGBkqTjx487tB8/ftzeFxgYqBMnTjj0X7x4USdPnnQYk9scf93G37m6usrb29vhAwAAAABAcVRsg31ISIgCAwO1fPlye1taWpo2bNigyMhISVJkZKROnz6tpKQk+5gVK1YoOztbd999t33M6tWrlZWVZR+TkJCgatWqqXTp0jdobwAAAAAAKBxFGuzT09OVnJys5ORkSX8+MC85OVmHDx+WzWbToEGD9Morr2jhwoXavn27evbsqaCgIHXs2FGSFBYWplatWumJJ57Qxo0btXbtWvXv318PP/ywgoKCJEndu3eXi4uLoqOjtXPnTn322WeaNGmSBg8eXER7DQAAAABAwSnS191t3rxZTZs2tS9fDtu9evVSXFychgwZorNnz6pv3746ffq0GjVqpCVLlqhUqVL2dWbOnKn+/fvrvvvuk5OTkzp37qzJkyfb+318fLRs2TLFxMSobt26Klu2rIYPH86r7gAAAAAANwWbZVlWURdR3KWlpcnHx0epqam3xP32k05NKuoSUIAGlh5Y1CWgANlsRV0BChLfwDefWTVqFHUJKEDdd+4s6hIA3MLyk0OL7T32AAAAAADg2gj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAUExNmTJFERER8vb2lre3tyIjI/XNN99Ikg4dOiSbzZbrZ+7cuQ7zxMXFKSIiQqVKlZK/v79iYmKKYndQSJyLugAAAAAAQO4qVqyocePG6fbbb5dlWYqPj1eHDh20detWVa9eXceOHXMY//777+uNN95Q69at7W1vvfWWxo8frzfeeEN33323zp49q0OHDt3gPUFhItgDAAAAQDHVvn17h+UxY8ZoypQp+v7771WjRg0FBgY69M+fP19dunSRp6enJOnUqVN68cUX9dVXX+m+++6zj4uIiCj84nHDcCk+AAAAABjg0qVLmj17ts6ePavIyMgc/UlJSUpOTlZ0dLS9LSEhQdnZ2fr1118VFhamihUrqkuXLjpy5MiNLB2FjGAPAAAAAMXY9u3b5enpKVdXVz311FOaP3++wsPDc4z76KOPFBYWpgYNGtjbDhw4oOzsbL366quaOHGi5s2bp5MnT6pFixbKzMy8kbuBQkSwBwAAAIBirFq1akpOTtaGDRvUr18/9erVS7t27XIYc/78ec2aNcvhbL0kZWdnKysrS5MnT1ZUVJTq16+vTz/9VHv37tXKlStv5G6gEHGPPQAAAAAUYy4uLgoNDZUk1a1bV5s2bdKkSZM0bdo0+5h58+bp3Llz6tmzp8O65cuXlySHM/zlypVT2bJldfjw4RtQPW4EztgDAAAAgEGys7OVkZHh0PbRRx/p/vvvV7ly5RzaGzZsKEnas2ePve3kyZP6/fffFRwcXPjF4obgjD0AAAAAFFPDhg1T69atVblyZZ05c0azZs1SYmKili5dah+zb98+rV69Wl9//XWO9f/1r3+pQ4cOGjhwoN5//315e3tr2LBhql69upo2bXojdwWFiGAPAAAAAMXUiRMn1LNnTx07dkw+Pj6KiIjQ0qVL1aJFC/uYjz/+WBUrVlTLli1znWPGjBmKjY1V27Zt5eTkpMaNG2vJkiUqWbLkjdoNFDKbZVlWURdR3KWlpcnHx0epqany9vYu6nIK3aRTk4q6BBSggaUHFnUJKEA2W1FXgILEN/DNZ1aNGkVdAgpQ9507i7oEALew/ORQ7rEHAAAAAMBgBHsAAAAAAAzGPfYAAAAAbnpf7DlW1CWgAHWqVr6oSyhWOGMPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMEI9gAAAAAAGIxgDwAAAACAwQj2AAAAAAAYjGAPAAAAAIDBCPYAAAAAABiMYA8AAAAAgMFuqWD/7rvvqkqVKipVqpTuvvtubdy4sahLAgAAAADgH7llgv1nn32mwYMH6+WXX9aWLVtUq1YtRUVF6cSJE0VdGgAAAAAA1+2WCfZvvfWWnnjiCfXp00fh4eGaOnWq3N3d9fHHHxd1aQAAAAAAXDfnoi7gRsjMzFRSUpKGDRtmb3NyclLz5s21fv36HOMzMjKUkZFhX05NTZUkpaWlFX6xxcCFtAtFXQIKUFqJW+P/W8BEt8jXyi3l3KVLRV0CCtCt8m+/W8W59DNFXQIKUFqaR1GXUOgu/x1kWdY1x94Swf7333/XpUuXFBAQ4NAeEBCgH3/8Mcf4sWPHauTIkTnaK1WqVGg1AoVlqIYWdQkArsDHp6grAHA1T3CQAigGzpw5I59r/H10SwT7/Bo2bJgGDx5sX87OztbJkydVpkwZ2Wy2IqwMBSUtLU2VKlXSkSNH5O3tXdTlAPgLjk+geOMYBYovjs+bi2VZOnPmjIKCgq459pYI9mXLllWJEiV0/Phxh/bjx48rMDAwx3hXV1e5uro6tPn6+hZmiSgi3t7e/KUHFFMcn0DxxjEKFF8cnzePa52pv+yWeHiei4uL6tatq+XLl9vbsrOztXz5ckVGRhZhZQAAAAAA/DO3xBl7SRo8eLB69eqlevXq6a677tLEiRN19uxZ9enTp6hLAwAAAADgut0ywb5r16767bffNHz4cKWkpKh27dpasmRJjgfq4dbg6uqql19+OcctFwCKHscnULxxjALFF8fnrctm5eXZ+QAAAAAAoFi6Je6xBwAAAADgZkWwBwAAAADAYAR7AAAAAAAMRrDHLSMxMVE2m02nT5++6rgqVapo4sSJN6QmAP8cxywAAIUvr/+WRtEg2KPY6d27t2w2m2w2m1xcXBQaGqpRo0bp4sWL/2jeBg0a6NixY/Lx8ZEkxcXFydfXN8e4TZs2qW/fvv9oW8DN4vLxOG7cOIf2BQsWyGaz3dBaOGaBglGcjmvgVnSjjsFDhw7JZrMpOTm5wOZE8UWwR7HUqlUrHTt2THv37tXTTz+tESNG6I033vhHc7q4uCgwMPCaf2GWK1dO7u7u/2hbwM2kVKlSeu2113Tq1KmiLiVXHLNA/hX34xq42RWnYzAzM7OoS0ABINijWHJ1dVVgYKCCg4PVr18/NW/eXAsXLtSpU6fUs2dPlS5dWu7u7mrdurX27t1rX+/nn39W+/btVbp0aXl4eKhGjRr6+uuvJTlePpSYmKg+ffooNTXVfnXAiBEjJDle1tu9e3d17drVobasrCyVLVtWM2bMkCRlZ2dr7NixCgkJkZubm2rVqqV58+YV/g8JuEGaN2+uwMBAjR079opjvvvuO91zzz1yc3NTpUqVNGDAAJ09e9bef+zYMbVt21Zubm4KCQnRrFmzclxC/9Zbb6lmzZry8PBQpUqV9J///Efp6emSxDELFLC8HNeff/65atSoIVdXV1WpUkXjx4936K9SpYpeffVVPfbYY/Ly8lLlypX1/vvvO4w5cuSIunTpIl9fX/n5+alDhw46dOhQYewSYJSC+G612WxasGCBwzq+vr6Ki4uTJIWEhEiS7rzzTtlsNjVp0kTSn1cMdOzYUWPGjFFQUJCqVasmSfrkk09Ur149eXl5KTAwUN27d9eJEycKbqdRqAj2MIKbm5syMzPVu3dvbd68WQsXLtT69etlWZbatGmjrKwsSVJMTIwyMjK0evVqbd++Xa+99po8PT1zzNegQQNNnDhR3t7eOnbsmI4dO6Znnnkmx7gePXroq6++socLSVq6dKnOnTunBx54QJI0duxYzZgxQ1OnTtXOnTsVGxurRx55RKtWrSqknwZwY5UoUUKvvvqq3n77bf3yyy85+vfv369WrVqpc+fO+uGHH/TZZ5/pu+++U//+/e1jevbsqaNHjyoxMVGff/653n///Rz/WHByctLkyZO1c+dOxcfHa8WKFRoyZIgkjlmgoF3ruE5KSlKXLl308MMPa/v27RoxYoReeukle2C4bPz48apXr562bt2q//znP+rXr5/27Nkj6c9fqkVFRcnLy0tr1qzR2rVr5enpqVatWnGGELe8gvhuvZaNGzdKkr799lsdO3ZMX3zxhb1v+fLl2rNnjxISErRo0SJJfx6zo0eP1rZt27RgwQIdOnRIvXv3/mc7ihvHAoqZXr16WR06dLAsy7Kys7OthIQEy9XV1erYsaMlyVq7dq197O+//265ublZc+bMsSzLsmrWrGmNGDEi13lXrlxpSbJOnTplWZZlTZ8+3fLx8ckxLjg42JowYYJlWZaVlZVllS1b1poxY4a9v1u3blbXrl0ty7KsCxcuWO7u7ta6desc5oiOjra6det2PbsPFCt/PR7r169vPfbYY5ZlWdb8+fOty18h0dHRVt++fR3WW7NmjeXk5GSdP3/e2r17tyXJ2rRpk71/7969liT7sZabuXPnWmXKlLEvc8wCBSMvx3X37t2tFi1aOKz37LPPWuHh4fbl4OBg65FHHrEvZ2dnW/7+/taUKVMsy7KsTz75xKpWrZqVnZ1tH5ORkWG5ublZS5cuLZR9A0xQEN+tlmVZkqz58+c7jPHx8bGmT59uWZZlHTx40JJkbd26Ncf2AwICrIyMjKvWuWnTJkuSdebMGcuycv5bGsULZ+xRLC1atEienp4qVaqUWrdura5du6p3795ydnbW3XffbR9XpkwZVatWTbt375YkDRgwQK+88ooaNmyol19+WT/88MM/qsPZ2VldunTRzJkzJUlnz57Vl19+qR49ekiS9u3bp3PnzqlFixby9PS0f2bMmKH9+/f/o20Dxc1rr72m+Ph4+/F22bZt2xQXF+dwDERFRSk7O1sHDx7Unj175OzsrDp16tjXCQ0NVenSpR3m+fbbb3XfffepQoUK8vLy0qOPPqo//vhD586dy3ONHLNA/lzpuN69e7caNmzo0NawYUPt3btXly5dsrdFRETY/2yz2RQYGGi/Gmfbtm3at2+fvLy87Mean5+fLly4wPEG/J/r/W79p2rWrCkXFxeHtqSkJLVv316VK1eWl5eXGjduLEk6fPjwP94eCp9zURcA5KZp06aaMmWKXFxcFBQUJGdnZy1cuPCa6z3++OOKiorS4sWLtWzZMo0dO1bjx4/Xf//73+uupUePHmrcuLFOnDihhIQEubm5qVWrVpJkv9x38eLFqlChgsN6rq6u171NoDi69957FRUVpWHDhjlcmpeenq4nn3xSAwYMyLFO5cqV9dNPP11z7kOHDqldu3bq16+fxowZIz8/P3333XeKjo5WZmZmvh6OxzEL5N2Vjuu8KlmypMOyzWZTdna2pD+Pt7p169p/0fZX5cqVu656gZvN9X63Sn8eb5ZlOfRdvj31Wjw8PByWz549q6ioKEVFRWnmzJkqV66cDh8+rKioKG6dMQTBHsWSh4eHQkNDHdrCwsJ08eJFbdiwQQ0aNJAk/fHHH9qzZ4/Cw8Pt4ypVqqSnnnpKTz31lIYNG6YPPvgg12Dv4uLicNbhSho0aKBKlSrps88+0zfffKOHHnrI/g+Z8PBwubq66vDhw/bfagI3s3Hjxql27dr2B+1IUp06dbRr164cx+xl1apV08WLF7V161bVrVtX0p9nzv/6JOCkpCRlZ2dr/PjxcnL682KyOXPmOMzDMQsUjtyO67CwMK1du9Zh3Nq1a/Wvf/1LJUqUyNO8derU0WeffSZ/f395e3sXaM3AzeR6vlulP39BduzYMfvy3r17Ha5yu3xGPi/fnT/++KP++OMPjRs3TpUqVZIkbd68Od/7gqLDpfgwxu23364OHTroiSee0Hfffadt27bpkUceUYUKFdShQwdJ0qBBg7R06VIdPHhQW7Zs0cqVKxUWFpbrfFWqVFF6erqWL1+u33///aqX+3bv3l1Tp05VQkKC/ZJeSfLy8tIzzzyj2NhYxcfHa//+/dqyZYvefvttxcfHF+wPACgGatasqR49emjy5Mn2tueee07r1q1T//79lZycrL179+rLL7+0P+CnevXqat68ufr27auNGzdq69at6tu3r9zc3OyvnwwNDVVWVpbefvttHThwQJ988ommTp3qsG2OWaBw5HZcP/3001q+fLlGjx6tn376SfHx8XrnnXdyfWjllfTo0UNly5ZVhw4dtGbNGh08eFCJiYkaMGBArg8LA25V1/PdKknNmjXTO++8o61bt2rz5s166qmnHK6i8ff3l5ubm5YsWaLjx48rNTX1ijVUrlxZLi4u9u/hhQsXavTo0YWzwygUBHsYZfr06apbt67atWunyMhIWZalr7/+2v6X2KVLlxQTE6OwsDC1atVK//rXv/Tee+/lOleDBg301FNPqWvXripXrpxef/31K263R48e2rVrlypUqJDjnsPRo0frpZde0tixY+3bXbx4sf0VI8DNZtSoUfZLbaU/77FdtWqVfvrpJ91zzz268847NXz4cAUFBdnHzJgxQwEBAbr33nv1wAMP6IknnpCXl5dKlSolSapVq5beeustvfbaa7rjjjs0c+bMHK8A4pgFCs/fj+s6depozpw5mj17tu644w4NHz5co0aNytfl+u7u7lq9erUqV66sTp06KSwsTNHR0bpw4QJn8IG/uZ7v1vHjx6tSpUq655571L17dz3zzDMOt645Oztr8uTJmjZtmoKCguwnwnJTrlw5xcXFae7cuQoPD9e4ceP05ptvFs7OolDYrL/fmAEAQCH75ZdfVKlSJfsD8wAAAHD9CPYAgEK3YsUKpaenq2bNmjp27JiGDBmiX3/9VT/99FOOh28BAAAgf3h4HgCg0GVlZen555/XgQMH5OXlpQYNGmjmzJmEegAAgALAGXsAAAAAAAzGw/MAAAAAADAYwR4AAAAAAIMR7AEAAAAAMBjBHgAAAAAAgxHsAQAAAAAwGMEeAADkYLPZtGDBgqIuAwAA5AHBHgAAg/Tu3Vs2m002m00lS5ZUSEiIhgwZogsXLhTodo4dO6bWrVsX6JxXc/DgQXXv3l1BQUEqVaqUKlasqA4dOujHH3+8YTUAAGAq56IuAAAA5E+rVq00ffp0ZWVlKSkpSb169ZLNZtNrr71WYNsIDAwssLmuJSsrSy1atFC1atX0xRdfqHz58vrll1/0zTff6PTp04W63ZIlSxba/AAA3CicsQcAwDCurq4KDAxUpUqV1LFjRzVv3lwJCQn2/uzsbI0dO1YhISFyc3NTrVq1NG/ePHtfxYoVNWXKFIc5t27dKicnJ/3888+Scl6Kf+TIEXXp0kW+vr7y8/NThw4ddOjQIUnSjh075OTkpN9++02SdPLkSTk5Oenhhx+2r//KK6+oUaNGue7Pzp07tX//fr333nuqX7++goOD1bBhQ73yyiuqX7++fdwvv/yibt26yc/PTx4eHqpXr542bNhg758yZYqqVq0qFxcXVatWTZ988onDdmw2m6ZMmaL7779fHh4eGjNmjCTpyy+/VJ06dVSqVCnddtttGjlypC5evJin/xYAABQHBHsAAAy2Y8cOrVu3Ti4uLva2sWPHasaMGZo6dap27typ2NhYPfLII1q1apWcnJzUrVs3zZo1y2GemTNnqmHDhgoODs6xjaysLEVFRcnLy0tr1qzR2rVr5enpqVatWikzM1M1atRQmTJltGrVKknSmjVrHJYladWqVWrSpEmu+1CuXDk5OTlp3rx5unTpUq5j0tPT1bhxY/36669auHChtm3bpiFDhig7O1uSNH/+fA0cOFBPP/20duzYoSeffFJ9+vTRypUrHeYZMWKEHnjgAW3fvl2PPfaY1qxZo549e2rgwIHatWuXpk2bpri4OHvoBwDACBYAADBGr169rBIlSlgeHh6Wq6urJclycnKy5s2bZ1mWZV24cMFyd3e31q1b57BedHS01a1bN8uyLGvr1q2WzWazfv75Z8uyLOvSpUtWhQoVrClTptjHS7Lmz59vWZZlffLJJ1a1atWs7Oxse39GRobl5uZmLV261LIsy+rUqZMVExNjWZZlDRo0yHr22Wet0qVLW7t377YyMzMtd3d3a9myZVfcr3feecdyd3e3vLy8rKZNm1qjRo2y9u/fb++fNm2a5eXlZf3xxx+5rt+gQQPriSeecGh76KGHrDZt2jjs06BBgxzG3Hfffdarr77q0PbJJ59Y5cuXv2KtAAAUN5yxBwDAME2bNlVycrI2bNigXr16qU+fPurcubMkad++fTp37pxatGghT09P+2fGjBnav3+/JKl27doKCwuzn7VftWqVTpw4oYceeijX7W3btk379u2Tl5eXfT4/Pz9duHDBPmfjxo2VmJhon69Zs2a69957lZiYqE2bNikrK0sNGza84j7FxMQoJSVFM2fOVGRkpObOnasaNWrYbzFITk7WnXfeKT8/v1zX3717d475GzZsqN27dzu01atXL8e+jRo1yuFn9cQTT+jYsWM6d+7cFesFAKA44eF5AAAYxsPDQ6GhoZKkjz/+WLVq1dJHH32k6OhopaenS5IWL16sChUqOKzn6upq/3OPHj00a9YsDR06VLNmzVKrVq1UpkyZXLeXnp6uunXraubMmTn6ypUrJ0lq0qSJBg0apL1792rXrl1q1KiRfvzxRyUmJurUqVOqV6+e3N3dr7pfXl5eat++vdq3b69XXnlFUVFReuWVV9SiRQu5ubnl/Qd0FR4eHjn2beTIkerUqVOOsaVKlSqQbQIAUNg4Yw8AgMGcnJz0/PPP68UXX9T58+cVHh4uV1dXHT58WKGhoQ6fSpUq2dfr3r27duzYoaSkJM2bN089evS44jbq1KmjvXv3yt/fP8ecPj4+kqSaNWuqdOnSeuWVV1S7dm15enqqSZMmWrVqlRITE694f/2V2Gw2Va9eXWfPnpUkRUREKDk5WSdPnsx1fFhYmNauXevQtnbtWoWHh191O3Xq1NGePXty7FdoaKicnPhnEgDADHxjAQBguIceekglSpTQu+++Ky8vLz3zzDOKjY1VfHy89u/fry1btujtt99WfHy8fZ0qVaqoQYMGio6O1qVLl3T//fdfcf4ePXqobNmy6tChg9asWaODBw8qMTFRAwYM0C+//CLpzyB+7733aubMmfYQHxERoYyMDC1fvlyNGze+4vzJycnq0KGD5s2bp127dmnfvn366KOP9PHHH6tDhw6SpG7duikwMFAdO3bU2rVrdeDAAX3++edav369JOnZZ59VXFycpkyZor179+qtt97SF198oWeeeeaqP7vhw4drxowZGjlypHbu3Kndu3dr9uzZevHFF/P0swcAoDgg2AMAYDhnZ2f1799fr7/+us6ePavRo0frpZde0tixYxUWFqZWrVpp8eLFCgkJcVivR48e2rZtmx544IGrXuru7u6u1atXq3LlyurUqZPCwsIUHR2tCxcuyNvb2z6ucePGunTpkj3YOzk56d5775XNZrvq/fUVK1ZUlSpVNHLkSN19992qU6eOJk2apJEjR+qFF16QJLm4uGjZsmXy9/dXmzZtVLNmTY0bN04lSpSQJHXs2FGTJk3Sm2++qRo1amjatGmaPn36Na8UiIqK0qJFi7Rs2TL9+9//Vv369TVhwoRc3w4AAEBxZbMsyyrqIgAAAAAAwPXhjD0AAAAAAAYj2AMAAAAAYDCCPQAAAAAABiPYAwAAAABgMII9AAAAAAAGI9gDAAAAAGAwgj0AAAAAAAYj2AMAAAAAYDCCPQAAAAAABiPYAwAAAABgMII9AAAAAAAG+3/k9ba61QsDggAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1200x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "label_distribution = data['label']\n",
    "\n",
    "barplot_visual(label_distribution,\n",
    "               \"Distribution of Scores\",\n",
    "               \"Review Score\",\n",
    "               \"# of Samples\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c6f699e6",
   "metadata": {},
   "source": [
    "#### Visualization: Distribution of Input Text Length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a43c1696",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkQAAAHHCAYAAABeLEexAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/kElEQVR4nO3de3wOZ/7/8feNHAhJHBOpCKLOh7ZUGq1GsEItVbYOtW2ooirtosra3Rbdtgirh92W2kXotlq21G5tqZI4RFCHOFfDOrQlKJIQJCTX74/+cn97i5Bbc+dOMq/n4zGPdmau+5rPZHLnfpu55h6bMcYIAADAwsq5uwAAAAB3IxABAADLIxABAADLIxABAADLIxABAADLIxABAADLIxABAADLIxABAADLIxABAADLIxABcJuOHTuqRYsWxbKtGTNmqEGDBipfvrzuueeeYtkmgNKDQASUcnFxcbLZbNq+fbu7S7mpkydPavLkyUpOTnZbDV9++aXGjx+vBx98UAsWLNAbb7xRYNvBgwercuXKxVjdrR04cECTJ0/WsWPHbtnu2LFjstlshZpu11dR1waUBhXcXQCAsu3kyZOaMmWK6tWr57YzM+vWrVO5cuU0b948eXp6uqWGO3XgwAFNmTJFHTt2VL169QpsV7NmTX3wwQcOy/7yl7/o+++/15tvvpmvbXHWBpQGBCIAZd6ZM2dUsWLFUheGnOHj46Pf/va3Dss+/vhjXbhwId9yAPlxyQywiB9++EFPP/20AgIC5OXlpebNm2v+/PkObRISEmSz2bRkyRK9/vrrqlOnjry9vdW5c2cdPnw4X5/vvvuuGjRooIoVK6pdu3bauHGjOnbsqI4dO9r7u//++yVJQ4YMsV+yiYuLc+jnwIEDioyMVKVKlXTXXXcpNja2UPt0/fp1/fnPf1ZoaKi8vLxUr149/eEPf1BWVpa9jc1m04IFC5SZmVng9m+nXr16+vWvf61NmzapXbt28vb2VoMGDbRo0SKHdnmXLzds2KARI0aoevXq8vX11VNPPaULFy44tLXZbJo8efJNtzV48GB7f48//rgkKTIy0l5/QkKCU/X/XFZWliZNmqSGDRvKy8tLwcHBGj9+vMPPLDo6Wt7e3jp48KDDa6OiolS1alWdPHnSJbUB7sQZIsACTp8+rQceeEA2m00xMTGqWbOmvvjiCw0dOlQZGRkaPXq0Q/tp06apXLlyGjdunNLT0xUbG6tBgwZp69at9jazZ89WTEyMOnTooDFjxujYsWPq3bu3qlatqjp16kiSmjZtqldffVWvvPKKhg8frg4dOkiS2rdvb+/nwoUL6tatm/r06aN+/frpX//6lyZMmKCWLVuqe/fut9yvZ555RgsXLtRvfvMbvfjii9q6daumTp2qgwcPavny5ZKkDz74QHPnztW2bdv0j3/8I9/2C+vw4cP6zW9+o6FDhyo6Olrz58/X4MGD1aZNGzVv3tyhbUxMjPz9/TV58mQdOnRIs2fP1vHjx+2Bs7AefvhhvfDCC3rnnXf0hz/8QU2bNpUk+3+dlZubq169emnTpk0aPny4mjZtqr179+rNN9/Ut99+q88++0yS9Pbbb2vdunWKjo5WUlKSypcvr/fff19ffvmlPvjgAwUFBRV5bYDbGQCl2oIFC4wk8/XXXxfYZujQoaZ27drmxx9/dFg+YMAA4+fnZy5fvmyMMSY+Pt5IMk2bNjVZWVn2dm+//baRZPbu3WuMMSYrK8tUr17d3H///ebatWv2dnFxcUaSiYiIsC/7+uuvjSSzYMGCfHVFREQYSWbRokX2ZVlZWSYwMND07dv3lvudnJxsJJlnnnnGYfm4ceOMJLNu3Tr7sujoaOPj43PL/m7VNiQkxEgyGzZssC87c+aM8fLyMi+++KJ9Wd6xaNOmjcnOzrYvj42NNZLMihUr7MskmUmTJuXbfkhIiImOjrbPL1261Egy8fHxhar/53r06GFCQkLs8x988IEpV66c2bhxo0O7OXPmGEkmMTHRvmz16tVGknnttdfM//73P1O5cmXTu3dvh9f9ktqAkoZLZkAZZ4zRp59+qp49e8oYox9//NE+RUVFKT09XTt37nR4zZAhQxzG2+Sd2fnf//4nSdq+fbvOnTunYcOGqUKF/zvRPGjQIFWtWtWp+ipXruwwxsXT01Pt2rWzb6sg//3vfyVJY8eOdVj+4osvSpJWrlzpVB2306xZM/vPQfppYHLjxo1vWufw4cPl4eFhnx85cqQqVKhgr9ldli5dqqZNm6pJkyYOvwedOnWSJMXHx9vbdu3aVSNGjNCrr76qPn36yNvbW++//767SgdcjktmQBl39uxZpaWlae7cuZo7d+5N25w5c8Zhvm7dug7zeSEnbxzM8ePHJUkNGzZ0aFehQgWn7zaqU6dOvstIVatW1Z49e275uuPHj6tcuXL5aggMDJS/v7+9xqJy488kr84bxwZJ0t133+0wX7lyZdWuXdvtt6enpKTo4MGDBd5lduPvwcyZM7VixQolJyfro48+Uq1atYqjTMAtCERAGZebmytJ+u1vf6vo6OibtmnVqpXDfPny5W/azhhTtMUVwbacGZPzSxTXzyQnJ6dI+/u53NxctWzZUrNmzbrp+uDgYIf5Xbt22UPS3r17NXDgQJfVBrgbgQgo42rWrKkqVaooJydHXbp0KZI+Q0JCJP000DgyMtK+/Pr16zp27JhDwHJVYAkJCVFubq5SUlIcBvKePn1aaWlp9hrdISUlxeHncunSJZ06dUqPPPKIfVnVqlWVlpbm8Lrs7GydOnXKYVlR/vxCQ0O1e/dude7c+bb9ZmZmasiQIWrWrJnat2+v2NhYPfbYY/a7Bou6NsDdGEMElHHly5dX37599emnn2rfvn351p89e9bpPtu2bavq1avr73//u65fv25f/uGHH+a7hOTj4yNJ+T78f6m8cPHWW285LM87+9GjR48i3Z4z5s6dq2vXrtnnZ8+erevXrzvcNRcaGqoNGzbke92NZ4iK8ufXr18//fDDD/r73/+eb92VK1eUmZlpn58wYYJOnDihhQsXatasWapXr56io6Mdbs931bEF3IEzREAZMX/+fK1atSrf8t/97neaNm2a4uPjFRYWpmHDhqlZs2Y6f/68du7cqa+++krnz593aluenp6aPHmynn/+eXXq1En9+vXTsWPHFBcXp9DQUIczB6GhofL399ecOXNUpUoV+fj4KCwsTPXr1/9F+9u6dWtFR0dr7ty5SktLU0REhLZt26aFCxeqd+/eDmdoilt2drY6d+6sfv366dChQ3rvvff00EMPqVevXvY2zzzzjJ599ln17dtXv/rVr7R7926tXr1aNWrUcOjrnnvuUfny5TV9+nSlp6fLy8tLnTp1uqPxPE8++aSWLFmiZ599VvHx8XrwwQeVk5Ojb775RkuWLNHq1avVtm1brVu3Tu+9954mTZqk++67T5K0YMECdezYUS+//LL9e6KKsjbA7dx6jxuAXyzvVu+Cpu+++84YY8zp06fNqFGjTHBwsPHw8DCBgYGmc+fOZu7cufa+8m67X7p0qcM2jh49etNb59955x0TEhJivLy8TLt27UxiYqJp06aN6datm0O7FStWmGbNmpkKFSo49BMREWGaN2+eb5+io6MdbhcvyLVr18yUKVNM/fr1jYeHhwkODjYTJ040V69ezdffL73tvkePHvnaRkREOHzFQN6xWL9+vRk+fLipWrWqqVy5shk0aJA5d+6cw2tzcnLMhAkTTI0aNUylSpVMVFSUOXz4cL7b7o0x5u9//7tp0KCBKV++vFO3ud94270xxmRnZ5vp06eb5s2bGy8vL1O1alXTpk0bM2XKFJOenm4yMjJMSEiIue+++xy+UsEYY8aMGWPKlStnkpKSfnFtQEljM8YFoyQBWFJubq5q1qypPn363PSyTFkXFxenIUOG6Ouvv1bbtm3dXQ4AJzCGCMAduXr1ar47rBYtWqTz58/bH90BAKUFY4gA3JEtW7ZozJgxevzxx1W9enXt3LlT8+bNU4sWLezPuAKA0oJABOCO1KtXT8HBwXrnnXd0/vx5VatWTU899ZSmTZtWpp8qD6BsYgwRAACwPMYQAQAAyyMQAQAAy2MMUSHk5ubq5MmTqlKlCl9VDwBAKWGM0cWLFxUUFKRy5W59DohAVAgnT57M99BDAABQOnz33XeqU6fOLdsQiAqhSpUqkn76gfr6+rq5GgAAUBgZGRkKDg62f47fCoGoEPIuk/n6+hKIAAAoZQoz3IVB1QAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIquLsAAK5hs7mmX2Nc0y8AuBNniAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOW5NRBNnTpV999/v6pUqaJatWqpd+/eOnTokEObq1evatSoUapevboqV66svn376vTp0w5tTpw4oR49eqhSpUqqVauWXnrpJV2/ft2hTUJCgu677z55eXmpYcOGiouLc/XuAQCAUsKtgWj9+vUaNWqUtmzZojVr1ujatWvq2rWrMjMz7W3GjBmj//znP1q6dKnWr1+vkydPqk+fPvb1OTk56tGjh7Kzs7V582YtXLhQcXFxeuWVV+xtjh49qh49eigyMlLJyckaPXq0nnnmGa1evbpY9xcAAJRMNmOMcXcRec6ePatatWpp/fr1evjhh5Wenq6aNWvqo48+0m9+8xtJ0jfffKOmTZsqKSlJDzzwgL744gv9+te/1smTJxUQECBJmjNnjiZMmKCzZ8/K09NTEyZM0MqVK7Vv3z77tgYMGKC0tDStWrXqtnVlZGTIz89P6enp8vX1dc3OA0XMZnNNvyXnLwYA3Jozn98lagxRenq6JKlatWqSpB07dujatWvq0qWLvU2TJk1Ut25dJSUlSZKSkpLUsmVLexiSpKioKGVkZGj//v32Nj/vI69NXh8AAMDaKri7gDy5ubkaPXq0HnzwQbVo0UKSlJqaKk9PT/n7+zu0DQgIUGpqqr3Nz8NQ3vq8dbdqk5GRoStXrqhixYoO67KyspSVlWWfz8jI+OU7CAAASqwSc4Zo1KhR2rdvnz7++GN3l6KpU6fKz8/PPgUHB7u7JAAA4EIlIhDFxMTo888/V3x8vOrUqWNfHhgYqOzsbKWlpTm0P336tAIDA+1tbrzrLG/+dm18fX3znR2SpIkTJyo9Pd0+fffdd794HwEAQMnl1kBkjFFMTIyWL1+udevWqX79+g7r27RpIw8PD61du9a+7NChQzpx4oTCw8MlSeHh4dq7d6/OnDljb7NmzRr5+vqqWbNm9jY/7yOvTV4fN/Ly8pKvr6/DBAAAyi633mX23HPP6aOPPtKKFSvUuHFj+3I/Pz/7mZuRI0fqv//9r+Li4uTr66vnn39ekrR582ZJP912f8899ygoKEixsbFKTU3Vk08+qWeeeUZvvPGGpJ9uu2/RooVGjRqlp59+WuvWrdMLL7yglStXKioq6rZ1cpcZSiPuMgNgdc58frs1ENkK+Iu9YMECDR48WNJPX8z44osvavHixcrKylJUVJTee+89++UwSTp+/LhGjhyphIQE+fj4KDo6WtOmTVOFCv83ZjwhIUFjxozRgQMHVKdOHb388sv2bdwOgQilEYEIgNWVmkBUWhCIUBoRiABYXan9HiIAAAB3KDHfQwSgdHDFmSfOOgFwN84QAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAyyMQAQAAy6vg7gIAwGZzTb/GuKZfAGUPZ4gAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDluTUQbdiwQT179lRQUJBsNps+++wzh/WDBw+WzWZzmLp16+bQ5vz58xo0aJB8fX3l7++voUOH6tKlSw5t9uzZow4dOsjb21vBwcGKjY119a4BAIBSxK2BKDMzU61bt9a7775bYJtu3brp1KlT9mnx4sUO6wcNGqT9+/drzZo1+vzzz7VhwwYNHz7cvj4jI0Ndu3ZVSEiIduzYoRkzZmjy5MmaO3euy/YLAACULhXcufHu3bure/fut2zj5eWlwMDAm647ePCgVq1apa+//lpt27aVJP31r3/VI488opkzZyooKEgffvihsrOzNX/+fHl6eqp58+ZKTk7WrFmzHIITAACwrhI/highIUG1atVS48aNNXLkSJ07d86+LikpSf7+/vYwJEldunRRuXLltHXrVnubhx9+WJ6envY2UVFROnTokC5cuHDTbWZlZSkjI8NhAgAAZVeJDkTdunXTokWLtHbtWk2fPl3r169X9+7dlZOTI0lKTU1VrVq1HF5ToUIFVatWTampqfY2AQEBDm3y5vPa3Gjq1Kny8/OzT8HBwUW9awAAoARx6yWz2xkwYID9/1u2bKlWrVopNDRUCQkJ6ty5s8u2O3HiRI0dO9Y+n5GRQSgCAKAMK9FniG7UoEED1ahRQ4cPH5YkBQYG6syZMw5trl+/rvPnz9vHHQUGBur06dMObfLmCxqb5OXlJV9fX4cJAACUXaUqEH3//fc6d+6cateuLUkKDw9XWlqaduzYYW+zbt065ebmKiwszN5mw4YNunbtmr3NmjVr1LhxY1WtWrV4dwAAAJRIbg1Ely5dUnJyspKTkyVJR48eVXJysk6cOKFLly7ppZde0pYtW3Ts2DGtXbtWjz76qBo2bKioqChJUtOmTdWtWzcNGzZM27ZtU2JiomJiYjRgwAAFBQVJkp544gl5enpq6NCh2r9/vz755BO9/fbbDpfEAACAxRk3io+PN5LyTdHR0eby5cuma9eupmbNmsbDw8OEhISYYcOGmdTUVIc+zp07ZwYOHGgqV65sfH19zZAhQ8zFixcd2uzevds89NBDxsvLy9x1111m2rRpTtWZnp5uJJn09PRfvM9AcZGYAFibM5/fNmOMcWMeKxUyMjLk5+en9PR0xhOh1LDZ3F2B+/HXDbA2Zz6/S9UYIgAAAFcgEAEAAMsjEAEAAMsjEAEAAMsjEAEAAMsjEAEAAMsjEAEAAMsjEAEAAMsjEAEAAMsjEAEAAMsjEAEAAMtzOhCdOHFCN3v8mTFGJ06cKJKiAAAAipPTgah+/fo6e/ZsvuXnz59X/fr1i6QoAACA4uR0IDLGyHaTx2hfunRJ3t7eRVIUAABAcapQ2IZjx46VJNlsNr388suqVKmSfV1OTo62bt2qe+65p8gLBAAAcLVCB6Jdu3ZJ+ukM0d69e+Xp6Wlf5+npqdatW2vcuHFFXyEAAICLFToQxcfHS5KGDBmit99+W76+vi4rCgAAoDg5PYYoNja2wDC0d+/eX1wQAABAcXM6ELVs2VIrV67Mt3zmzJlq165dkRQFAABQnJwORGPHjlXfvn01cuRIXblyRT/88IM6d+6s2NhYffTRR66oEQAAwKWcDkTjx49XUlKSNm7cqFatWqlVq1by8vLSnj179Nhjj7miRgAAAJe6o0d3NGzYUC1atNCxY8eUkZGh/v37KzAwsKhrAwAAKBZOB6LExES1atVKKSkp2rNnj2bPnq3nn39e/fv314ULF1xRIwAAgEs5HYg6deqk/v37a8uWLWratKmeeeYZ7dq1SydOnFDLli1dUSMAAIBLFfp7iPJ8+eWXioiIcFgWGhqqxMREvf7660VWGAAAQHGxmZs9ur4QDh8+rCNHjujhhx9WxYoVC3zGWVmQkZEhPz8/paen84WUKDXK6NvRKXf21w1AWeHM57fTl8zOnTunzp07q1GjRnrkkUd06tQpSdLQoUN5dAdwh2y2op8AAIXndCAaM2aMPDw8dOLECYcHvPbv319ffPFFkRYHAABQHO5oDNHq1atVp04dh+V33323jh8/XmSFAQAAFBenzxBlZmY6nBnKc/78eXl5eRVJUQAAAMXJ6UDUoUMHLVq0yD5vs9mUm5ur2NhYRUZGFmlxAAAAxaHQl8waNGigr7/+WrGxsercubO2b9+u7OxsjR8/Xvv379f58+eVmJjoyloBAABcotBniI4dO6acnBy1aNFC3377rR566CE9+uijyszMVJ8+fbRr1y6Fhoa6slYAAACXcHpQtST5+fnpj3/8Y1HXAgAA4BZOBaLVq1fLz8/vlm169er1iwoCAAAobk4Foujo6Fuut9lsysnJ+UUFAQAAFDen7jJLTU1Vbm5ugRNhCAAAlEaFDkRl9TllAAAAhQ5Ed/gMWAAAgBKv0IEoOjpaFStWdGUtAAAAblHoQdULFixwZR0AAABu4/SjOwAAAMoaAhEAALA8AhEAALC8QgeiunXrKiYmRl9++aWuX7/uypoAAACKVaED0QcffCAvLy+NGjVKNWrUUP/+/fXhhx8qLS3NheUBAAC4ns3cwRcM7d+/X//+97+1YsUKJScnq3379urVq5d69eqlBg0auKJOt8rIyJCfn5/S09Pl6+vr7nJQBvG9p67B16cB1ubM5/cdjSFq3ry5Jk6cqC1btujYsWMaOHCg1q5dqxYtWqhFixZauXLlHRUOAADgDnd0hqggly9f1urVq1WlShV16dKlqLp1O84QwdU4Q+QanCECrM2Zz2+nnnZ/O5UqVdJjjz1WlF0CAAC4HLfdAwAAyyMQAQAAyyMQAQAAy/vFgSgjI0OfffaZDh48WBT1AAAAFDunA1G/fv30t7/9TZJ05coVtW3bVv369VOrVq306aefFnmBAAAAruZ0INqwYYM6dOggSVq+fLmMMUpLS9M777yj1157rcgLBAAAcDWnA1F6erqqVasmSVq1apX69u2rSpUqqUePHkpJSSnyAgEAAFzN6UAUHByspKQkZWZmatWqVeratask6cKFC/L29i7yAgEAAFzN6S9mHD16tAYNGqTKlSsrJCREHTt2lPTTpbSWLVsWdX0AAAAu53Qgeu6559SuXTt99913+tWvfqVy5X46ydSgQQPGEAEAgFKpSJ9lVlbxLDO4Gs8ycw3+ugHWVuTPMhs7dmyhNz5r1qxCtwUAACgJChWIdu3a5TC/c+dOXb9+XY0bN5YkffvttypfvrzatGlT9BUCAAC4WKECUXx8vP3/Z82apSpVqmjhwoWqWrWqpJ/uMBsyZIj9+4kAAABKE6fHEN1111368ssv1bx5c4fl+/btU9euXXXy5MkiLbAkYAwRXI0xRK7BGCLA2pz5/Hb6e4gyMjJ09uzZfMvPnj2rixcvOtsdAACA2zkdiB577DENGTJEy5Yt0/fff6/vv/9en376qYYOHao+ffq4okYAAACXcvp7iObMmaNx48bpiSee0LVr137qpEIFDR06VDNmzCjyAgEAAFzNqTFEOTk5SkxMVMuWLeXp6akjR45IkkJDQ+Xj4+OyIt2NMURwNcYQuQZjiABrK/LvIcpTvnx5de3aVQcPHlT9+vXVqlWrX1QoAABASeD0GKIWLVrof//7X5FsfMOGDerZs6eCgoJks9n02WefOaw3xuiVV15R7dq1VbFiRXXp0kUpKSkObc6fP69BgwbJ19dX/v7+Gjp0qC5duuTQZs+ePerQoYO8vb0VHBys2NjYIqkfAACUDU4Hotdee03jxo3T559/rlOnTikjI8NhckZmZqZat26td99996brY2Nj9c4772jOnDnaunWrfHx8FBUVpatXr9rbDBo0SPv379eaNWv0+eefa8OGDRo+fLh9fUZGhrp27aqQkBDt2LFDM2bM0OTJkzV37lxndx0AAJRVxkk2m80+lStXzj7lzd8pSWb58uX2+dzcXBMYGGhmzJhhX5aWlma8vLzM4sWLjTHGHDhwwEgyX3/9tb3NF198YWw2m/nhhx+MMca89957pmrVqiYrK8veZsKECaZx48aFri09Pd1IMunp6Xe6e8At/TTahamoJwDW5sznt9N3mf38W6td6ejRo0pNTVWXLl3sy/z8/BQWFqakpCQNGDBASUlJ8vf3V9u2be1tunTponLlymnr1q167LHHlJSUpIcfflienp72NlFRUZo+fbouXLhg/7ZtAABgXU4HooiICFfUkU9qaqokKSAgwGF5QECAfV1qaqpq1arlsL5ChQqqVq2aQ5v69evn6yNv3c0CUVZWlrKysuzzzl4KBAAApYvTgUiS0tLSNG/ePB08eFCS1Lx5cz399NPy8/Mr0uLcZerUqZoyZYq7ywDwC7ni6wy4lR8om5weVL19+3aFhobqzTff1Pnz53X+/HnNmjVLoaGh2rlzZ5EVFhgYKEk6ffq0w/LTp0/b1wUGBurMmTMO669fv67z5887tLlZHz/fxo0mTpyo9PR0+/Tdd9/98h0CAAAlltOBaMyYMerVq5eOHTumZcuWadmyZTp69Kh+/etfa/To0UVWWP369RUYGKi1a9fal2VkZGjr1q0KDw+XJIWHhystLU07duywt1m3bp1yc3MVFhZmb7Nhwwb7t2pL0po1a9S4ceMCxw95eXnJ19fXYQIAAGXXHZ0hmjBhgipU+L+rbRUqVND48eO1fft2p/q6dOmSkpOTlZycLOmngdTJyck6ceKEbDabRo8erddee03//ve/tXfvXj311FMKCgpS7969JUlNmzZVt27dNGzYMG3btk2JiYmKiYnRgAEDFBQUJEl64okn5OnpqaFDh2r//v365JNP9Pbbb2vs2LHO7joAACirnL2FrVatWmb16tX5lq9atcrUqlXLqb7i4+ONpHxTdHS0MeanW+9ffvllExAQYLy8vEznzp3NoUOHHPo4d+6cGThwoKlcubLx9fU1Q4YMMRcvXnRos3v3bvPQQw8ZLy8vc9ddd5lp06Y5VSe33cPV3H17OhO38gNlkTOf3049y0ySXnjhBS1fvlwzZ85U+/btJUmJiYl66aWX1LdvX7311ltFGthKAp5lBlfjWWalB4OqgdLDZc8yk6SZM2fKZrPpqaee0vXr1yVJHh4eGjlypKZNm3ZnFQMAALhRoc8QHT161OH7fC5fvuzwtPtKlSq5psISgDNEcDXOEJUenCECSg+XnCEKDQ1VSEiIIiMj1alTJ0VGRqply5a/uFgAAAB3K3QgWrdunRISEpSQkKDFixcrOztbDRo0sIejyMjIfN8qDQAAUBo4Pahakq5evarNmzfbA9K2bdt07do1NWnSRPv373dFnW7FJTO4GpfMSg8umQGlhzOf33cUiPJkZ2crMTFRX3zxhd5//31dunRJOTk5d9pdiUUggqsRiEoPAhFQerjsLrPs7Gxt2bJF8fHxSkhI0NatWxUcHKyHH35Yf/vb34rtwa8AAABFqdCBqFOnTtq6davq16+viIgIjRgxQh999JFq167tyvoAAABcrtCBaOPGjapdu7Y6deqkjh07KiIiQtWrV3dlbQAAAMWi0M8yS0tL09y5c1WpUiVNnz5dQUFBatmypWJiYvSvf/1LZ8+edWWdAAAALnPHg6ovXryoTZs22ccT7d69W3fffbf27dtX1DW6HYOq4WoMqi49GFQNlB7OfH47/bT7PD4+PqpWrZqqVaumqlWrqkKFCjp48OCddgcAAOA2hR5DlJubq+3btyshIUHx8fFKTExUZmam7rrrLkVGRurdd99VZGSkK2sFAABwiUIHIn9/f2VmZiowMFCRkZF688031bFjR4WGhrqyPgAAAJcrdCCaMWOGIiMj1ahRI1fWAwAAUOwKHYhGjBjhyjoAAADc5o4HVQMAAJQVBCIAAGB5BCIAAGB5BCIAAGB5BCIAAGB5BCIAAGB5hb7tHgDPHAOAsoozRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIIRAAAwPIquLsAAChNbDbX9GuMa/oFUDicIQIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZXogPR5MmTZbPZHKYmTZrY11+9elWjRo1S9erVVblyZfXt21enT5926OPEiRPq0aOHKlWqpFq1aumll17S9evXi3tXAABACVbiH93RvHlzffXVV/b5ChX+r+QxY8Zo5cqVWrp0qfz8/BQTE6M+ffooMTFRkpSTk6MePXooMDBQmzdv1qlTp/TUU0/Jw8NDb7zxRrHvCwAAKJlKfCCqUKGCAgMD8y1PT0/XvHnz9NFHH6lTp06SpAULFqhp06basmWLHnjgAX355Zc6cOCAvvrqKwUEBOiee+7Rn//8Z02YMEGTJ0+Wp6dnce8OAAAogUr0JTNJSklJUVBQkBo0aKBBgwbpxIkTkqQdO3bo2rVr6tKli71tkyZNVLduXSUlJUmSkpKS1LJlSwUEBNjbREVFKSMjQ/v37y9wm1lZWcrIyHCYAABA2VWiA1FYWJji4uK0atUqzZ49W0ePHlWHDh108eJFpaamytPTU/7+/g6vCQgIUGpqqiQpNTXVIQzlrc9bV5CpU6fKz8/PPgUHBxftjgEAgBKlRF8y6969u/3/W7VqpbCwMIWEhGjJkiWqWLGiy7Y7ceJEjR071j6fkZFBKAIAoAwr0WeIbuTv769GjRrp8OHDCgwMVHZ2ttLS0hzanD592j7mKDAwMN9dZ3nzNxuXlMfLy0u+vr4OEwAAKLtKVSC6dOmSjhw5otq1a6tNmzby8PDQ2rVr7esPHTqkEydOKDw8XJIUHh6uvXv36syZM/Y2a9aska+vr5o1a1bs9QMAgJKpRF8yGzdunHr27KmQkBCdPHlSkyZNUvny5TVw4ED5+flp6NChGjt2rKpVqyZfX189//zzCg8P1wMPPCBJ6tq1q5o1a6Ynn3xSsbGxSk1N1Z/+9CeNGjVKXl5ebt47AABQUpToQPT9999r4MCBOnfunGrWrKmHHnpIW7ZsUc2aNSVJb775psqVK6e+ffsqKytLUVFReu+99+yvL1++vD7//HONHDlS4eHh8vHxUXR0tF599VV37RIAACiBbMYY4+4iSrqMjAz5+fkpPT2d8UQWZ7O5uwKUVfwlBoqeM5/fpWoMEQAAgCsQiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOVVcHcBAADJZiv6Po0p+j6BsoozRAAAwPI4Q4QyyxX/4gYAlE2cIQIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZHIAIAAJZXwd0FAABcw2ZzTb/GuKZfwJ04QwQAACyPQAQAACyPQAQAACyPQAQAACyPQAQAACyPQAQAACyPQAQAACyPQAQAACyPQAQAACyPQAQAACyPR3fA7Vz1eAEAAAqLM0QAAMDyCEQAAMDyuGQGAHCKKy5zG1P0fQLO4AwRAACwPAIRAACwPAIRAACwPAIRAACwPAIRAACwPO4yAwC4HXeuwd0sdYbo3XffVb169eTt7a2wsDBt27bN3SUBAIASwDKB6JNPPtHYsWM1adIk7dy5U61bt1ZUVJTOnDnj7tJKFZut6CcAANzNMoFo1qxZGjZsmIYMGaJmzZppzpw5qlSpkubPn+/u0gAALuCKf8Dxj7iyyxKBKDs7Wzt27FCXLl3sy8qVK6cuXbooKSnJjZUBAEobQlbZZIlB1T/++KNycnIUEBDgsDwgIEDffPNNvvZZWVnKysqyz6enp0uSMjIyXFtoEfLzc3cFAIDCKk2h6P9/JJYKeZ/bphAj7C0RiJw1depUTZkyJd/y4OBgN1QDAEDJURr/wX3x4kX53aZwSwSiGjVqqHz58jp9+rTD8tOnTyswMDBf+4kTJ2rs2LH2+bS0NIWEhOjEiRO3/YGWVhkZGQoODtZ3330nX19fd5dT5Ni/0q+s72NZ3z+p7O8j+1fyGGN08eJFBQUF3batJQKRp6en2rRpo7Vr16p3796SpNzcXK1du1YxMTH52nt5ecnLyyvfcj8/v1LzS3CnfH19y/Q+sn+lX1nfx7K+f1LZ30f2r2Qp7IkMSwQiSRo7dqyio6PVtm1btWvXTm+99ZYyMzM1ZMgQd5cGAADczDKBqH///jp79qxeeeUVpaam6p577tGqVavyDbQGAADWY5lAJEkxMTE3vUR2O15eXpo0adJNL6OVFWV9H9m/0q+s72NZ3z+p7O8j+1e62Uxh7kUDAAAowyzxxYwAAAC3QiACAACWRyACAACWRyACAACWRyD6/959913Vq1dP3t7eCgsL07Zt227ZfunSpWrSpIm8vb3VsmVL/fe//y2mSp03depU3X///apSpYpq1aql3r1769ChQ7d8TVxcnGw2m8Pk7e1dTBU7Z/LkyflqbdKkyS1fU5qOnyTVq1cv3z7abDaNGjXqpu1L+vHbsGGDevbsqaCgINlsNn322WcO640xeuWVV1S7dm1VrFhRXbp0UUpKym37dfZ97Cq32r9r165pwoQJatmypXx8fBQUFKSnnnpKJ0+evGWfd/J77kq3O4aDBw/OV2+3bt1u229pOIaSbvp+tNlsmjFjRoF9lqRjWJjPhatXr2rUqFGqXr26KleurL59++Z74sON7vS9WxIQiCR98sknGjt2rCZNmqSdO3eqdevWioqK0pkzZ27afvPmzRo4cKCGDh2qXbt2qXfv3urdu7f27dtXzJUXzvr16zVq1Cht2bJFa9as0bVr19S1a1dlZmbe8nW+vr46deqUfTp+/HgxVey85s2bO9S6adOmAtuWtuMnSV9//bXD/q1Zs0aS9Pjjjxf4mpJ8/DIzM9W6dWu9++67N10fGxurd955R3PmzNHWrVvl4+OjqKgoXb16tcA+nX0fu9Kt9u/y5cvauXOnXn75Ze3cuVPLli3ToUOH1KtXr9v268zvuavd7hhKUrdu3RzqXbx48S37LC3HUJLDfp06dUrz58+XzWZT3759b9lvSTmGhflcGDNmjP7zn/9o6dKlWr9+vU6ePKk+ffrcst87ee+WGAamXbt2ZtSoUfb5nJwcExQUZKZOnXrT9v369TM9evRwWBYWFmZGjBjh0jqLypkzZ4wks379+gLbLFiwwPj5+RVfUb/ApEmTTOvWrQvdvrQfP2OM+d3vfmdCQ0NNbm7uTdeXpuMnySxfvtw+n5ubawIDA82MGTPsy9LS0oyXl5dZvHhxgf04+z4uLjfu381s27bNSDLHjx8vsI2zv+fF6Wb7GB0dbR599FGn+inNx/DRRx81nTp1umWbknwMb/xcSEtLMx4eHmbp0qX2NgcPHjSSTFJS0k37uNP3bklh+TNE2dnZ2rFjh7p06WJfVq5cOXXp0kVJSUk3fU1SUpJDe0mKiooqsH1Jk56eLkmqVq3aLdtdunRJISEhCg4O1qOPPqr9+/cXR3l3JCUlRUFBQWrQoIEGDRqkEydOFNi2tB+/7Oxs/fOf/9TTTz8tm81WYLvSdPx+7ujRo0pNTXU4Rn5+fgoLCyvwGN3J+7gkSU9Pl81mk7+//y3bOfN7XhIkJCSoVq1aaty4sUaOHKlz584V2LY0H8PTp09r5cqVGjp06G3bltRjeOPnwo4dO3Tt2jWH49GkSRPVrVu3wONxJ+/dksTygejHH39UTk5Ovkd4BAQEKDU19aavSU1Ndap9SZKbm6vRo0frwQcfVIsWLQps17hxY82fP18rVqzQP//5T+Xm5qp9+/b6/vvvi7HawgkLC1NcXJxWrVql2bNn6+jRo+rQoYMuXrx40/al+fhJ0meffaa0tDQNHjy4wDal6fjdKO84OHOM7uR9XFJcvXpVEyZM0MCBA2/5wExnf8/drVu3blq0aJHWrl2r6dOna/369erevbtycnJu2r40H8OFCxeqSpUqt72cVFKP4c0+F1JTU+Xp6ZkvpN/uszGvTWFfU5JY6tEdkEaNGqV9+/bd9rp1eHi4wsPD7fPt27dX06ZN9f777+vPf/6zq8t0Svfu3e3/36pVK4WFhSkkJERLliwp1L/YSpt58+ape/fuCgoKKrBNaTp+Vnbt2jX169dPxhjNnj37lm1L2+/5gAED7P/fsmVLtWrVSqGhoUpISFDnzp3dWFnRmz9/vgYNGnTbGxdK6jEs7OdCWWf5M0Q1atRQ+fLl842cP336tAIDA2/6msDAQKfalxQxMTH6/PPPFR8frzp16jj1Wg8PD9177706fPiwi6orOv7+/mrUqFGBtZbW4ydJx48f11dffaVnnnnGqdeVpuOXdxycOUZ38j52t7wwdPz4ca1Zs+aWZ4du5na/5yVNgwYNVKNGjQLrLY3HUJI2btyoQ4cOOf2elErGMSzocyEwMFDZ2dlKS0tzaH+7z8a8NoV9TUli+UDk6empNm3aaO3atfZlubm5Wrt2rcO/sH8uPDzcob0krVmzpsD27maMUUxMjJYvX65169apfv36TveRk5OjvXv3qnbt2i6osGhdunRJR44cKbDW0nb8fm7BggWqVauWevTo4dTrStPxq1+/vgIDAx2OUUZGhrZu3VrgMbqT97E75YWhlJQUffXVV6pevbrTfdzu97yk+f7773Xu3LkC6y1txzDPvHnz1KZNG7Vu3drp17rzGN7uc6FNmzby8PBwOB6HDh3SiRMnCjwed/LeLVHcPKi7RPj444+Nl5eXiYuLMwcOHDDDhw83/v7+JjU11RhjzJNPPml+//vf29snJiaaChUqmJkzZ5qDBw+aSZMmGQ8PD7N371537cItjRw50vj5+ZmEhARz6tQp+3T58mV7mxv3ccqUKWb16tXmyJEjZseOHWbAgAHG29vb7N+/3x27cEsvvviiSUhIMEePHjWJiYmmS5cupkaNGubMmTPGmNJ//PLk5OSYunXrmgkTJuRbV9qO38WLF82uXbvMrl27jCQza9Yss2vXLvtdVtOmTTP+/v5mxYoVZs+ePebRRx819evXN1euXLH30alTJ/PXv/7VPn+793FJ2b/s7GzTq1cvU6dOHZOcnOzwnszKyipw/273e17cbrWPFy9eNOPGjTNJSUnm6NGj5quvvjL33Xefufvuu83Vq1ftfZTWY5gnPT3dVKpUycyePfumfZTkY1iYz4Vnn33W1K1b16xbt85s377dhIeHm/DwcId+GjdubJYtW2afL8x7t6QiEP1/f/3rX03dunWNp6enadeundmyZYt9XUREhImOjnZov2TJEtOoUSPj6elpmjdvblauXFnMFReepJtOCxYssLe5cR9Hjx5t/3kEBASYRx55xOzcubP4iy+E/v37m9q1axtPT09z1113mf79+5vDhw/b15f245dn9erVRpI5dOhQvnWl7fjFx8ff9Hcybx9yc3PNyy+/bAICAoyXl5fp3Llzvv0OCQkxkyZNclh2q/dxcbrV/h09erTA92R8fLy9jxv373a/58XtVvt4+fJl07VrV1OzZk3j4eFhQkJCzLBhw/IFm9J6DPO8//77pmLFiiYtLe2mfZTkY1iYz4UrV66Y5557zlStWtVUqlTJPPbYY+bUqVP5+vn5awrz3i2pbMYY45pzTwAAAKWD5ccQAQAAEIgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgAAIDlEYgA4AZr165V06ZN7U9mnzx5su655x73FvULObMPc+bMUc+ePV1bEFDCEIgACxk8eLB69+5d7NuNi4uTv7//Ldt07NhRNputwKljx46/qIZ69erprbfeKlTb8ePH609/+pPKly8vSRo3bly+598545tvvpHNZtOWLVsclj/wwAPy9vbW1atX7cuuXr0qb29vzZs3746390s9/fTT2rlzpzZu3Oi2GoDiRiACUCIsW7ZMp06d0qlTp7Rt2zZJ0ldffWVftmzZsmKpY9OmTTpy5Ij69u1rX1a5cuU7egBrniZNmigwMFAJCQn2ZRcvXtTOnTtVs2ZNh6CUlJSkrKwsderU6Y62de3atTuuM4+np6eeeOIJvfPOO7+4L6C0IBABFtaxY0e98MILGj9+vKpVq6bAwEBNnjzZoY3NZtPs2bPVvXt3VaxYUQ0aNNC//vUv+/qEhATZbDalpaXZlyUnJ8tms+nYsWNKSEjQkCFDlJ6ebj/bc+M2JNm3HxgYqJo1a0qSqlevbl924MABdejQQRUrVlRwcLBeeOEFZWZmSpIWLVqkypUrKyUlxd7fc889pyZNmujy5cvq2LGjjh8/rjFjxthrKMjHH3+sX/3qV/L29rYvu/FyU96ZtpkzZ6p27dqqXr26Ro0adcswEhkZ6RCINm3apEaNGqlnz54OyxMSEhQSEmJ/+vjs2bMVGhoqT09PNW7cWB988IFDv3nHp1evXvLx8dHrr78uSZo2bZoCAgJUpUoVDR061OEsVN522rVrJx8fH/n7++vBBx/U8ePH7et79uypf//737py5UqB+wSUJQQiwOIWLlwoHx8fbd26VbGxsXr11Ve1Zs0ahzYvv/yy+vbtq927d2vQoEEaMGCADh48WKj+27dvr7feeku+vr72sz3jxo1zqsYjR46oW7du6tu3r/bs2aNPPvlEmzZtUkxMjCTpqaee0iOPPKJBgwbp+vXrWrlypf7xj3/oww8/VKVKlbRs2TLVqVNHr776qr2GgmzcuFFt27a9bU3x8fE6cuSI4uPjtXDhQsXFxSkuLq7A9pGRkdq0aZOuX79uf33Hjh0VERGh+Ph4h34jIyMlScuXL9fvfvc7vfjii9q3b59GjBihIUOGOLSXfgpsjz32mPbu3aunn35aS5Ys0eTJk/XGG29o+/btql27tt577z17++vXr6t3796KiIjQnj17lJSUpOHDhzsExbZt2+r69evaunXrbX8WQJng7qfLAig+0dHR5tFHH7XPR0REmIceesihzf33328mTJhgn5dknn32WYc2YWFhZuTIkcaY/3sq+IULF+zrd+3aZSSZo0ePGmOMWbBggfHz8yt0nXlPhN+1a5cxxpihQ4ea4cOHO7TZuHGjKVeunLly5Yoxxpjz58+bOnXqmJEjR5qAgADz+uuvO7QPCQkxb7755m237efnZxYtWuSwbNKkSaZ169b2+ejoaBMSEmKuX79uX/b444+b/v37F9hvSkqKkWQ2b95sjPnp57xkyRJz8uRJ4+XlZa5cuWIuX75svLy8zMKFC40xxrRv394MGzbMoZ/HH3/cPPLII/Z5SWb06NEObcLDw81zzz3nsCwsLMy+D+fOnTOSTEJCwi1/FlWrVjVxcXG3bAOUFZwhAiyuVatWDvO1a9fWmTNnHJaFh4fnmy/sGaKisHv3bsXFxaly5cr2KSoqSrm5uTp69KgkqWrVqpo3b579EtPvf//7O9rWlStXHC6XFaR58+b2QdfSzX9uP9ewYUPVqVNHCQkJysjI0K5duxQREaHatWurbt26SkpKso8fyjtDdPDgQT344IMO/Tz44IP5fvY3ntE6ePCgwsLCHJb9/BhWq1ZNgwcPVlRUlHr27Km33377pmfNKlasqMuXL9/mJwGUDQQiwOI8PDwc5m02m3Jzcwv9+nLlfvozYoyxLyuKgb0/d+nSJY0YMULJycn2affu3UpJSVFoaKi93YYNG1S+fHmdOnXKPr7IWTVq1NCFCxdu2+5Ofm4dO3ZUfHy8Nm7cqLvvvlu1atWSJPtls/j4eDVs2FDBwcFO1ezj4+NUe0lasGCBkpKS1L59e33yySdq1KhRvrvgzp8/bx/PBZR1BCIAt3XjB+WWLVvUtGlTSbJ/YP78DENycrJDe09PT/t3+tyJ++67TwcOHFDDhg3zTZ6enpKkzZs3a/r06frPf/6jypUr28cXOVvDvffeqwMHDtxxrbcSGRmpzZs3a82aNQ5fI/Dwww8rISFBCQkJ9rNDktS0aVMlJiY69JGYmKhmzZrdcjtNmzbNN/bnxmMo/bSvEydO1ObNm9WiRQt99NFH9nVHjhzR1atXde+99zqzi0CpRSACcFtLly7V/Pnz9e2332rSpEnatm2bPXDkndGYPHmyUlJStHLlSv3lL39xeH29evV06dIlrV27Vj/++KPTl2EmTJigzZs3KyYmRsnJyUpJSdGKFSvsNVy8eFFPPvmkXnjhBXXv3l0ffvihPvnkE4e74erVq6cNGzbohx9+0I8//ljgtqKiorRp0yan6iusyMhIZWZmav78+YqIiLAvj4iI0NatW7Vt2zaHQPTSSy8pLi5Os2fPVkpKimbNmqVly5bddlD67373O82fP18LFiywH7P9+/fb1x89elQTJ05UUlKSjh8/ri+//FIpKSn2kCv9NLi8QYMGDmfggLKMQATgtqZMmaKPP/5YrVq10qJFi7R48WL7WQoPDw8tXrxY33zzjVq1aqXp06frtddec3h9+/bt9eyzz6p///6qWbOmYmNjndp+q1attH79en377bfq0KGD7r33Xr3yyisKCgqS9FMA8PHx0RtvvCFJatmypd544w2NGDFCP/zwgyTp1Vdf1bFjxxQaGnrLy0CDBg3S/v37dejQIadqLIz69esrJCREFy9edAhEdevWVVBQkLKzsx3OHPXu3Vtvv/22Zs6cqebNm+v999/XggULbvsllf3799fLL7+s8ePHq02bNjp+/LhGjhxpX1+pUiV988036tu3rxo1aqThw4dr1KhRGjFihL3N4sWLNWzYsCLbd6Cks5mfX/gHgBvYbDYtX77cLd9w7S4vvfSSMjIy9P7777u7FLfYv3+/OnXqpG+//VZ+fn7uLgcoFpwhAoAb/PGPf1RISIhTg8vLklOnTmnRokWEIVgKZ4gA3JIVzxABsJ4K7i4AQMnGv5kAWAGXzAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOURiAAAgOX9PzUz64FyQe7iAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The longest input text is approximately 21 words long.\n"
     ]
    }
   ],
   "source": [
    "data['text_length'] = data['text'].str.split().apply(len)\n",
    "\n",
    "data.hist(\"text_length\", color=\"Blue\", grid=False, bins=21)\n",
    "plt.title(\"Length of Input Text\")\n",
    "plt.suptitle('')\n",
    "plt.xlabel('Input Text (in Words)')\n",
    "plt.ylabel('Words / Text')\n",
    "plt.show()\n",
    "\n",
    "max_len = max(data['text_length'])\n",
    "\n",
    "print(f\"The longest input text is approximately {max_len} words long.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "93a8f30d",
   "metadata": {},
   "source": [
    "#### Remove Unnecessary Feature(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9f6a0768",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 10570 entries, 0 to 10569\n",
      "Data columns (total 2 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   text    10570 non-null  object\n",
      " 1   label   10570 non-null  object\n",
      "dtypes: object(2)\n",
      "memory usage: 165.3+ KB\n"
     ]
    }
   ],
   "source": [
    "data = data.drop(columns=['text_length'])\n",
    "data.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "642da7c1",
   "metadata": {},
   "source": [
    "#### Create Conversions Between Labels With String & Integer Data Types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "becf8e84",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['None', 'Negative', 'Neutral', 'Positive']\n",
      "4\n",
      "{0: 'None', 1: 'Negative', 2: 'Neutral', 3: 'Positive'}\n",
      "{'None': 0, 'Negative': 1, 'Neutral': 2, 'Positive': 3}\n"
     ]
    }
   ],
   "source": [
    "UNIQUE_LABEL_VALUES = ['None', \n",
    "                       'Negative', \n",
    "                       'Neutral', \n",
    "                       'Positive']\n",
    "\n",
    "NUM_LABELS = len(UNIQUE_LABEL_VALUES)\n",
    "\n",
    "id2label = {idx:label for idx, label in enumerate(UNIQUE_LABEL_VALUES)}\n",
    "label2id = {label:idx for idx, label in enumerate(UNIQUE_LABEL_VALUES)}\n",
    "\n",
    "print(UNIQUE_LABEL_VALUES)\n",
    "print(NUM_LABELS)\n",
    "print(id2label)\n",
    "print(label2id)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d593410c",
   "metadata": {},
   "source": [
    "#### Convert Label Values From String to Integer Data Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "87f57cbf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>april gold down 20 cents to settle at $1,116.1...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gold suffers third straight daily decline</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Gold futures edge up after two-session decline</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dent research : is gold's day in the sun comin...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Gold snaps three-day rally as Trump, lawmakers...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  label\n",
       "0  april gold down 20 cents to settle at $1,116.1...      1\n",
       "1          gold suffers third straight daily decline      1\n",
       "2     Gold futures edge up after two-session decline      3\n",
       "3  dent research : is gold's day in the sun comin...      0\n",
       "4  Gold snaps three-day rally as Trump, lawmakers...      1"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['label'] = data['label'].replace(label2id)\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ffbe2526",
   "metadata": {},
   "source": [
    "#### Convert DataFrame to Datasets & Split into Training/Evaluation Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2ddcc257",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (8456, 2)\n",
      "Eval data shape: (2114, 2)\n"
     ]
    }
   ],
   "source": [
    "dataset = Dataset.from_pandas(data)\n",
    "\n",
    "train_eval = dataset.train_test_split(test_size=0.20)\n",
    "\n",
    "ds = DatasetDict({\n",
    "    'train' : train_eval['train'],\n",
    "    'eval' : train_eval['test']\n",
    "})\n",
    "\n",
    "print('Training data shape:', ds['train'].shape)\n",
    "print('Eval data shape:', ds['eval'].shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c7c2c71b",
   "metadata": {},
   "source": [
    "#### Determine Longest Tokenized Input Length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "28a5084e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The longest tokenized input text length is approximately 41 tokens.\n"
     ]
    }
   ],
   "source": [
    "MODEL_CKPT = \"squeezebert/squeezebert-uncased\"\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT, \n",
    "                                             truncation=True, \n",
    "                                             padding=True,\n",
    "                                             batched=True, \n",
    "                                             batch_size=None)\n",
    "\n",
    "tokenized_input_text = [len(tokenizer.encode(a)) for a in ds['train']['text']]\n",
    "\n",
    "max_tokenized_input_text = max(tokenized_input_text)\n",
    "\n",
    "print(f\"The longest tokenized input text length is approximately {max_tokenized_input_text} tokens.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a8e6121e",
   "metadata": {},
   "source": [
    "#### Basic Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "cf6efed0",
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_OF_EPOCHS = 5\n",
    "MAX_INPUT_LENGTH = max_tokenized_input_text\n",
    "\n",
    "DEVICE = \"cpu\"\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "STRATEGY = \"epoch\"\n",
    "MODEL_CKPT = \"squeezebert/squeezebert-uncased\"\n",
    "\n",
    "MODEL_NAME = f\"{MODEL_CKPT.split('/')[-1]}-News_About_Gold\"\n",
    "REPORTS_TO = [\"tensorboard\", \"wandb\"]\n",
    "\n",
    "LR = 2e-5\n",
    "WANDB_NAME = \"News_About_Gold-SqueezeBERT\"\n",
    "\n",
    "set_seed(42)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "54187dcd",
   "metadata": {},
   "source": [
    "#### Tokenize Whole Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c41a9840",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "297ef60580924051a2d50c7b24cb1e42",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/8456 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1cb52d9c7b5d4ab9948cdc21791abc70",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/2114 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None), 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), 'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None), 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None)}\n",
      "Dataset({\n",
      "    features: ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
      "    num_rows: 8456\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT)\n",
    "\n",
    "def tokenize(batch):\n",
    "    return tokenizer(batch['text'], \n",
    "                     truncation=True, \n",
    "                     padding=True, \n",
    "                     max_length=MAX_INPUT_LENGTH, \n",
    "                     return_tensors=\"pt\")\n",
    "\n",
    "encoded_ds = ds.map(tokenize, \n",
    "                    batched=True, \n",
    "                    batch_size=None)\n",
    "\n",
    "print(encoded_ds['train'].features)\n",
    "print(encoded_ds['train'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "879e7fd8",
   "metadata": {},
   "source": [
    "#### Initialize Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "bda2bb12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d3a6d683c7ff4d38bd0e82a5d8e5a2c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading pytorch_model.bin:   0%|          | 0.00/103M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at squeezebert/squeezebert-uncased were not used when initializing SqueezeBertForSequenceClassification: ['cls.seq_relationship.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing SqueezeBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing SqueezeBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of SqueezeBertForSequenceClassification were not initialized from the model checkpoint at squeezebert/squeezebert-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = (\n",
    "    AutoModelForSequenceClassification.from_pretrained(MODEL_CKPT, \n",
    "                                                       num_labels=NUM_LABELS)\n",
    "    ).to(DEVICE)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c05c8359",
   "metadata": {},
   "source": [
    "#### Define Data Collator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "9d6b1099",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collator = DataCollatorWithPadding(tokenizer, \n",
    "                                        padding=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5b4af8d7",
   "metadata": {},
   "source": [
    "#### Metrics Evaluation Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f032424d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(p):\n",
    "    \"\"\"\n",
    "    This function takes the predictions output from \n",
    "    trainer.evaluate & returns metrics (accuracy, F1, \n",
    "    recall, & precision) for a multiclass \n",
    "    classification project.\n",
    "    \"\"\"\n",
    "    accuracy_metric = evaluate.load(\"accuracy\")\n",
    "    accuracy = accuracy_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                             axis=1), \n",
    "                                       references=p.label_ids)['accuracy']\n",
    "    \n",
    "    ### ------------------- F1 scores -------------------\n",
    "    \n",
    "    f1_score_metric = evaluate.load(\"f1\")\n",
    "    weighted_f1_score = f1_score_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                      axis=1), \n",
    "                                                references=p.label_ids, \n",
    "                                                average='weighted')[\"f1\"]\n",
    "    \n",
    "    micro_f1_score = f1_score_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                   axis=1), \n",
    "                                             references=p.label_ids, \n",
    "                                             average='micro')['f1']\n",
    "    \n",
    "    macro_f1_score = f1_score_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                   axis=1), \n",
    "                                             references=p.label_ids, \n",
    "                                             average='macro')[\"f1\"]\n",
    "    \n",
    "    ### ------------------- recall -------------------\n",
    "    \n",
    "    recall_metric = evaluate.load(\"recall\")\n",
    "    weighted_recall = recall_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                  axis=1), \n",
    "                                            references=p.label_ids, \n",
    "                                            average='weighted')[\"recall\"]\n",
    "    \n",
    "    micro_recall = recall_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                               axis=1), \n",
    "                                         references=p.label_ids, \n",
    "                                         average='micro')[\"recall\"]\n",
    "    \n",
    "    macro_recall = recall_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                               axis=1), \n",
    "                                         references=p.label_ids, \n",
    "                                         average='macro')[\"recall\"]\n",
    "    \n",
    "    ### ------------------- precision -------------------\n",
    "    \n",
    "    precision_metric = evaluate.load(\"precision\")\n",
    "    weighted_precision = precision_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                        axis=1), \n",
    "                                                  references=p.label_ids, \n",
    "                                                  average='weighted')[\"precision\"]\n",
    "    \n",
    "    micro_precision = precision_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                     axis=1), \n",
    "                                               references=p.label_ids, \n",
    "                                               average='micro')[\"precision\"]\n",
    "    \n",
    "    macro_precision = precision_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                     axis=1), \n",
    "                                               references=p.label_ids, \n",
    "                                               average='macro')[\"precision\"]\n",
    "    \n",
    "    return {\"accuracy\" : accuracy, \n",
    "            \"Weighted F1\" : weighted_f1_score,\n",
    "            \"Micro F1\" : micro_f1_score,\n",
    "            \"Macro F1\" : macro_f1_score,\n",
    "            \"Weighted Recall\" : weighted_recall,\n",
    "            \"Micro Recall\" : micro_recall,\n",
    "            \"Macro Recall\" : macro_recall,\n",
    "            \"Weighted Precision\" : weighted_precision,\n",
    "            \"Micro Precision\" : micro_precision,\n",
    "            \"Macro Precision\" : macro_precision\n",
    "            }"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "08334475",
   "metadata": {},
   "source": [
    "#### Trainer Arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "0358228f",
   "metadata": {},
   "outputs": [],
   "source": [
    "args = TrainingArguments(output_dir=MODEL_NAME, \n",
    "                         num_train_epochs=NUM_OF_EPOCHS, \n",
    "                         per_device_train_batch_size=BATCH_SIZE, \n",
    "                         per_device_eval_batch_size=BATCH_SIZE, \n",
    "                         weight_decay=0.01,\n",
    "                         learning_rate=LR,\n",
    "                         evaluation_strategy=STRATEGY,\n",
    "                         logging_strategy=STRATEGY,\n",
    "                         logging_first_step=True,\n",
    "                         save_strategy=STRATEGY, \n",
    "                         report_to=REPORTS_TO,\n",
    "                         run_name=WANDB_NAME,\n",
    "                         group_by_length=True,\n",
    "                         hub_private_repo=True,\n",
    "                         push_to_hub=True\n",
    "                         )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bb7b50e7",
   "metadata": {},
   "source": [
    "#### Subclass Trainer to Handle Class Imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ea55bf10",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomTrainer(Trainer):\n",
    "    def compute_loss(self, model, inputs, return_outputs=False):\n",
    "        labels = inputs.get(\"labels\")\n",
    "        outputs = model(**inputs)\n",
    "        logits = outputs.get(\"logits\")\n",
    "        loss_fct = torch.nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0, 3.0, 4.0]))\n",
    "        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))\n",
    "        return (loss, outputs) if return_outputs else loss"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "191fe09e",
   "metadata": {},
   "source": [
    "#### Build Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3551a802",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning https://huggingface.co/DunnBC22/squeezebert-uncased-News_About_Gold into local empty directory.\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(model=model, \n",
    "                  args=args, \n",
    "                  train_dataset=encoded_ds['train'], \n",
    "                  eval_dataset=encoded_ds['eval'],\n",
    "                  tokenizer=tokenizer,\n",
    "                  data_collator=data_collator,\n",
    "                  compute_metrics=compute_metrics)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cc60f187",
   "metadata": {},
   "source": [
    "#### Train the Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a0e50f1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/briandunn/Documents/deep_learning/dl/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>/Users/briandunn/Desktop/current projects/Sentiment Analysis of Commodity News (Gold)/wandb/run-20230606_152308-vgs25u68</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/dunnbc22/News_About_Gold_SqueezeBERT/runs/vgs25u68' target=\"_blank\">News_About_Gold-SqueezeBERT</a></strong> to <a href='https://wandb.ai/dunnbc22/News_About_Gold_SqueezeBERT' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/dunnbc22/News_About_Gold_SqueezeBERT' target=\"_blank\">https://wandb.ai/dunnbc22/News_About_Gold_SqueezeBERT</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/dunnbc22/News_About_Gold_SqueezeBERT/runs/vgs25u68' target=\"_blank\">https://wandb.ai/dunnbc22/News_About_Gold_SqueezeBERT/runs/vgs25u68</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b14e76fa2c144509a46a5892c8de1400",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/665 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a SqueezeBertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.4231, 'learning_rate': 1.9969924812030075e-05, 'epoch': 0.01}\n",
      "{'loss': 0.8756, 'learning_rate': 1.6000000000000003e-05, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "06d9da324afb45269a445fb2960a1f1c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/34 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/briandunn/Documents/deep_learning/dl/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/Users/briandunn/Documents/deep_learning/dl/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.452942818403244, 'eval_accuracy': 0.869914853358562, 'eval_Weighted F1': 0.8557169411489111, 'eval_Micro F1': 0.869914853358562, 'eval_Macro F1': 0.6559577437042121, 'eval_Weighted Recall': 0.869914853358562, 'eval_Micro Recall': 0.869914853358562, 'eval_Macro Recall': 0.6726779588705305, 'eval_Weighted Precision': 0.8436710960059746, 'eval_Micro Precision': 0.869914853358562, 'eval_Macro Precision': 0.6414383471396911, 'eval_runtime': 45.2965, 'eval_samples_per_second': 46.67, 'eval_steps_per_second': 0.751, 'epoch': 1.0}\n",
      "{'loss': 0.4097, 'learning_rate': 1.2e-05, 'epoch': 2.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e546a27015d8487ab0737bf2ef10b0d9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/34 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.31955790519714355, 'eval_accuracy': 0.902554399243141, 'eval_Weighted F1': 0.8981990872046596, 'eval_Micro F1': 0.902554399243141, 'eval_Macro F1': 0.7826498896982309, 'eval_Weighted Recall': 0.902554399243141, 'eval_Micro Recall': 0.902554399243141, 'eval_Macro Recall': 0.763545504951241, 'eval_Weighted Precision': 0.9058995948483624, 'eval_Micro Precision': 0.902554399243141, 'eval_Macro Precision': 0.8743465956410477, 'eval_runtime': 46.2695, 'eval_samples_per_second': 45.689, 'eval_steps_per_second': 0.735, 'epoch': 2.0}\n",
      "{'loss': 0.3147, 'learning_rate': 8.000000000000001e-06, 'epoch': 3.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "403fc11eb70b49f18a5c894d48f39896",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/34 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.2824004292488098, 'eval_accuracy': 0.9115421002838221, 'eval_Weighted F1': 0.9110995447372147, 'eval_Micro F1': 0.9115421002838221, 'eval_Macro F1': 0.8469888698710597, 'eval_Weighted Recall': 0.9115421002838221, 'eval_Micro Recall': 0.9115421002838221, 'eval_Macro Recall': 0.8318917244382616, 'eval_Weighted Precision': 0.9138453932389957, 'eval_Micro Precision': 0.9115421002838221, 'eval_Macro Precision': 0.8751072643902021, 'eval_runtime': 44.6739, 'eval_samples_per_second': 47.321, 'eval_steps_per_second': 0.761, 'epoch': 3.0}\n",
      "{'loss': 0.2685, 'learning_rate': 4.000000000000001e-06, 'epoch': 4.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aaae9dc80bd34d74a10ffdf989b4afde",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/34 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.26486265659332275, 'eval_accuracy': 0.9186376537369915, 'eval_Weighted F1': 0.918722301523608, 'eval_Micro F1': 0.9186376537369915, 'eval_Macro F1': 0.8681032986279441, 'eval_Weighted Recall': 0.9186376537369915, 'eval_Micro Recall': 0.9186376537369915, 'eval_Macro Recall': 0.8601906345542645, 'eval_Weighted Precision': 0.9202617437649651, 'eval_Micro Precision': 0.9186376537369915, 'eval_Macro Precision': 0.8797074213841205, 'eval_runtime': 44.116, 'eval_samples_per_second': 47.919, 'eval_steps_per_second': 0.771, 'epoch': 4.0}\n",
      "{'loss': 0.2479, 'learning_rate': 0.0, 'epoch': 5.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72ce744992c341baa69e033205447042",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/34 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.26428648829460144, 'eval_accuracy': 0.9167455061494797, 'eval_Weighted F1': 0.9165697427628363, 'eval_Micro F1': 0.9167455061494797, 'eval_Macro F1': 0.8749369503373983, 'eval_Weighted Recall': 0.9167455061494797, 'eval_Micro Recall': 0.9167455061494797, 'eval_Macro Recall': 0.8684302125750863, 'eval_Weighted Precision': 0.9168439952197603, 'eval_Micro Precision': 0.9167455061494797, 'eval_Macro Precision': 0.8821869512033577, 'eval_runtime': 44.094, 'eval_samples_per_second': 47.943, 'eval_steps_per_second': 0.771, 'epoch': 5.0}\n",
      "{'train_runtime': 3111.9997, 'train_samples_per_second': 13.586, 'train_steps_per_second': 0.214, 'train_loss': 0.42408741363009117, 'epoch': 5.0}\n"
     ]
    }
   ],
   "source": [
    "training_results = trainer.train()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "277a72fe",
   "metadata": {},
   "source": [
    "#### Save Model & Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "565ab306",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Several commits (2) will be pushed upstream.\n",
      "The progress bars may be unreliable.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0edf9743b0e245c8a385ad5b4784be25",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file pytorch_model.bin:   0%|          | 1.00/195M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f2ad5ec77014368a9cf349ba2aafc46",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file runs/Jun06_15-23-02_Brians-Mac-mini/events.out.tfevents.1686082988.Brians-Mac-mini.65964.0:   0%| …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/squeezebert-uncased-News_About_Gold\n",
      "   19166bd..81f0873  main -> main\n",
      "\n",
      "To https://huggingface.co/DunnBC22/squeezebert-uncased-News_About_Gold\n",
      "   81f0873..6000862  main -> main\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** train metrics *****\n",
      "  epoch                    =        5.0\n",
      "  train_loss               =     0.4241\n",
      "  train_runtime            = 0:51:51.99\n",
      "  train_samples_per_second =     13.586\n",
      "  train_steps_per_second   =      0.214\n"
     ]
    }
   ],
   "source": [
    "trainer.save_model()\n",
    "trainer.log_metrics(\"train\", training_results.metrics)\n",
    "trainer.save_metrics(\"train\", training_results.metrics)\n",
    "trainer.save_state()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5ab72d49",
   "metadata": {},
   "source": [
    "#### Push Model to Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "83c80523",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/squeezebert-uncased-News_About_Gold\n",
      "   6000862..83db6c2  main -> main\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'https://huggingface.co/DunnBC22/squeezebert-uncased-News_About_Gold/commit/83db6c2557b6fb4c76394fd7a444e3d4e2ddc964'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.push_to_hub(commit_message=\"All Dunn!!!\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d9d70523",
   "metadata": {},
   "source": [
    "#### Save Metrics to CSV File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "bc187ed8",
   "metadata": {},
   "outputs": [],
   "source": [
    "api = wandb.Api()\n",
    "\n",
    "# Specify what run for which to record metrics\n",
    "run = api.run(\"dunnbc22/News_About_Gold_SqueezeBERT/vgs25u68\")\n",
    "\n",
    "# save metrics for run to csv file\n",
    "metrics_dataframe = run.history()\n",
    "metrics_dataframe.to_csv(\"metrics.csv\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bae0b700",
   "metadata": {},
   "source": [
    "#### Finish Logging to Weights & Biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "d2682262",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/Macro F1</td><td>▁▅▇██</td></tr><tr><td>eval/Macro Precision</td><td>▁████</td></tr><tr><td>eval/Macro Recall</td><td>▁▄▇██</td></tr><tr><td>eval/Micro F1</td><td>▁▆▇██</td></tr><tr><td>eval/Micro Precision</td><td>▁▆▇██</td></tr><tr><td>eval/Micro Recall</td><td>▁▆▇██</td></tr><tr><td>eval/Weighted F1</td><td>▁▆▇██</td></tr><tr><td>eval/Weighted Precision</td><td>▁▇▇██</td></tr><tr><td>eval/Weighted Recall</td><td>▁▆▇██</td></tr><tr><td>eval/accuracy</td><td>▁▆▇██</td></tr><tr><td>eval/loss</td><td>█▃▂▁▁</td></tr><tr><td>eval/runtime</td><td>▅█▃▁▁</td></tr><tr><td>eval/samples_per_second</td><td>▄▁▆██</td></tr><tr><td>eval/steps_per_second</td><td>▄▁▆██</td></tr><tr><td>train/epoch</td><td>▁▂▂▄▄▅▅▇▇███</td></tr><tr><td>train/global_step</td><td>▁▂▂▄▄▅▅▇▇███</td></tr><tr><td>train/learning_rate</td><td>█▇▅▄▂▁</td></tr><tr><td>train/loss</td><td>█▅▂▁▁▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/Macro F1</td><td>0.87494</td></tr><tr><td>eval/Macro Precision</td><td>0.88219</td></tr><tr><td>eval/Macro Recall</td><td>0.86843</td></tr><tr><td>eval/Micro F1</td><td>0.91675</td></tr><tr><td>eval/Micro Precision</td><td>0.91675</td></tr><tr><td>eval/Micro Recall</td><td>0.91675</td></tr><tr><td>eval/Weighted F1</td><td>0.91657</td></tr><tr><td>eval/Weighted Precision</td><td>0.91684</td></tr><tr><td>eval/Weighted Recall</td><td>0.91675</td></tr><tr><td>eval/accuracy</td><td>0.91675</td></tr><tr><td>eval/loss</td><td>0.26429</td></tr><tr><td>eval/runtime</td><td>44.094</td></tr><tr><td>eval/samples_per_second</td><td>47.943</td></tr><tr><td>eval/steps_per_second</td><td>0.771</td></tr><tr><td>train/epoch</td><td>5.0</td></tr><tr><td>train/global_step</td><td>665</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.2479</td></tr><tr><td>train/total_flos</td><td>283497743040960.0</td></tr><tr><td>train/train_loss</td><td>0.42409</td></tr><tr><td>train/train_runtime</td><td>3111.9997</td></tr><tr><td>train/train_samples_per_second</td><td>13.586</td></tr><tr><td>train/train_steps_per_second</td><td>0.214</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">News_About_Gold-SqueezeBERT</strong> at: <a href='https://wandb.ai/dunnbc22/News_About_Gold_SqueezeBERT/runs/vgs25u68' target=\"_blank\">https://wandb.ai/dunnbc22/News_About_Gold_SqueezeBERT/runs/vgs25u68</a><br/>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>./wandb/run-20230606_152308-vgs25u68/logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wandb.finish()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6cad95d7",
   "metadata": {},
   "source": [
    "### Notes & Other Takeaways From This Project\n",
    "\n",
    "****\n",
    "- This project is part of a transformer comparison with the News About Gold Dataset.\n",
    "- The results are pretty good and it took less time to train than most of the other projects.\n",
    "****"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e28d083d",
   "metadata": {},
   "source": [
    "### Citation\n",
    "\n",
    "- Model Checkpoint\n",
    "\n",
    "    > @article{2020_SqueezeBERT, author = {Forrest N. Iandola and Albert E. Shaw and Ravi Krishna and Kurt W. Keutzer}, title = {{SqueezeBERT}: What can computer vision teach NLP about efficient neural networks?}, journal = {arXiv:2006.11316}, year = {2020}}\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 64-bit ('anaconda3')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "a4d9b73f1482e108b8bee9fd1a7eb6f7460da6f2684402434766fba1dc39798b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}