Multiclass Classification/Apple iPhone SE Reviews/Apple IPhone Reviews - MC CLF - Bert-Base.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e67daac8",
   "metadata": {},
   "source": [
    "## Apple IPhone SE Reviews\n",
    "\n",
    "Dataset Source: https://www.kaggle.com/datasets/kmldas/apple-iphone-se-reviews-ratings"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d2c358a6",
   "metadata": {},
   "source": [
    "#### Import Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "617d6a4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Git LFS initialized.\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "os.environ['TOKENIZERS_PARALLELISM']='false'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import wandb\n",
    "\n",
    "import torch\n",
    "\n",
    "import transformers\n",
    "from transformers import AutoTokenizer, Trainer, TrainingArguments\n",
    "from transformers import AutoModelForSequenceClassification, set_seed\n",
    "from transformers import DataCollatorWithPadding\n",
    "\n",
    "import datasets\n",
    "from datasets import Dataset, DatasetDict\n",
    "\n",
    "import evaluate\n",
    "\n",
    "!git lfs install"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a4a5cb6d",
   "metadata": {},
   "source": [
    "#### Name Project for Weights & Biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "705901e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: WANDB_PROJECT=initial_Apple_iPhone_Reviews\n",
      "env: WANDB_LOG_MODEL='checkpoint'\n",
      "env: WANDB_NOTEBOOK_NAME=\"Apple IPhone Reviews - MC CLF - Bert-Base.ipynb\"\n",
      "env: WANDB_WATCH=all\n"
     ]
    }
   ],
   "source": [
    "%env WANDB_PROJECT=initial_Apple_iPhone_Reviews\n",
    "%env WANDB_LOG_MODEL='checkpoint'\n",
    "%env WANDB_NOTEBOOK_NAME=\"Apple IPhone Reviews - MC CLF - Bert-Base.ipynb\"\n",
    "%env WANDB_WATCH=all"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "be581ab6",
   "metadata": {},
   "source": [
    "#### Log In To Weights & Biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "997b034c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find \"Apple IPhone Reviews - MC CLF - Bert-Base.ipynb\".\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mdunnbc22\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wandb.login()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d9f8e4b7",
   "metadata": {},
   "source": [
    "#### Display Library Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "a1e3f193",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "           Library   Version\n",
      "****************************\n",
      "            Python : 3.9.12\n",
      "             NumPy : 1.24.3\n",
      "  Weights & Biases : 0.15.3\n",
      "            Pandas : 2.0.1\n",
      "             Torch : 2.0.0\n",
      "          Datasets : 2.11.0\n",
      "      Transformers : 4.28.1\n",
      "          Evaluate : 0.4.0\n"
     ]
    }
   ],
   "source": [
    "n = 20\n",
    "\n",
    "print(\"Library  \".rjust(n), \"Version\")\n",
    "print(\"*\" * (n + 8))\n",
    "\n",
    "print(\"Python :\".rjust(n), sys.version[0:6])\n",
    "print(\"NumPy :\".rjust(n), np.__version__)\n",
    "print(\"Weights & Biases :\".rjust(n), wandb.__version__)\n",
    "print(\"Pandas :\".rjust(n), pd.__version__)\n",
    "print(\"Torch :\".rjust(n), torch.__version__)\n",
    "print(\"Datasets :\".rjust(n), datasets.__version__)\n",
    "print(\"Transformers :\".rjust(n), transformers.__version__)\n",
    "print(\"Evaluate :\".rjust(n), evaluate.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "242bbe6e",
   "metadata": {},
   "source": [
    "#### Ingest Data, Drop Unnecessary Columns, & Rename Remaining Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ece9f605",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5 Stars</td>\n",
       "      <td>Super! : Great camera for pics and videos Batt...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5 Stars</td>\n",
       "      <td>Must buy! : Great device. Let me tell the Pros...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5 Stars</td>\n",
       "      <td>Great product : Who all loves older size i.e.,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5 Stars</td>\n",
       "      <td>Simply awesome : This iPhone SE is the best ph...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5 Stars</td>\n",
       "      <td>Classy product : This is my second iphone afte...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9708</th>\n",
       "      <td>5 Stars</td>\n",
       "      <td>Terrific purchase : Absolutely brilliantREAD M...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9709</th>\n",
       "      <td>5 Stars</td>\n",
       "      <td>Classy product : Superb phone. This is my 4th ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9710</th>\n",
       "      <td>5 Stars</td>\n",
       "      <td>Awesome : very niceREAD MOREAwesome : very nic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9711</th>\n",
       "      <td>5 Stars</td>\n",
       "      <td>Super! : Loving it as of now. Good Product .RE...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9712</th>\n",
       "      <td>5 Stars</td>\n",
       "      <td>Terrific purchase : NiceElegant ElectricREAD M...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9713 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        label                                               text\n",
       "0     5 Stars  Super! : Great camera for pics and videos Batt...\n",
       "1     5 Stars  Must buy! : Great device. Let me tell the Pros...\n",
       "2     5 Stars  Great product : Who all loves older size i.e.,...\n",
       "3     5 Stars  Simply awesome : This iPhone SE is the best ph...\n",
       "4     5 Stars  Classy product : This is my second iphone afte...\n",
       "...       ...                                                ...\n",
       "9708  5 Stars  Terrific purchase : Absolutely brilliantREAD M...\n",
       "9709  5 Stars  Classy product : Superb phone. This is my 4th ...\n",
       "9710  5 Stars  Awesome : very niceREAD MOREAwesome : very nic...\n",
       "9711  5 Stars  Super! : Loving it as of now. Good Product .RE...\n",
       "9712  5 Stars  Terrific purchase : NiceElegant ElectricREAD M...\n",
       "\n",
       "[9713 rows x 2 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv('APPLE_iPhone_SE.csv', \n",
    "                   names=['label', 'comment', 'reviews'],\n",
    "                   dtype={'label': np.float32, \n",
    "                           'comment': str, \n",
    "                           'reviews': str},\n",
    "                   engine=\"c\",\n",
    "                   header=0)\n",
    "\n",
    "data['text'] = data['comment'] + \" : \" + data['reviews']\n",
    "\n",
    "data.drop(columns=['comment', 'reviews'], inplace=True)\n",
    "\n",
    "data['text'] = data['text'].apply(lambda x: x.split(\"(\\\\N{\")[0]) + \\\n",
    "               data['text'].apply(lambda x: x.split(\"})\")[-1])\n",
    "\n",
    "data['label'] = data['label'].astype(np.int8).astype('str') + \" Stars\"\n",
    "\n",
    "data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "50224af9",
   "metadata": {},
   "source": [
    "#### More Information About Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "640de3ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 9713 entries, 0 to 9712\n",
      "Data columns (total 2 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   label   9713 non-null   object\n",
      " 1   text    9713 non-null   object\n",
      "dtypes: object(2)\n",
      "memory usage: 151.9+ KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "09269047",
   "metadata": {},
   "source": [
    "#### Define Function to Create Bar Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5c540131",
   "metadata": {},
   "outputs": [],
   "source": [
    "def barplot_visual(feature_to_visualize: pd.Series,\n",
    "                   plot_title: str,\n",
    "                   x_label: str,\n",
    "                   y_label: str = \"# of Samples\",\n",
    "                   rotation: np.int8 = 0,\n",
    "                   annotation: np.bool_ = True\n",
    "                   ):\n",
    "    \"\"\"\n",
    "    This function returns a histogram for a feature.\n",
    "    \n",
    "    Inputs: \n",
    "    - feature_to_visualize: Feature to plot in the form \n",
    "    of pandas Series.\n",
    "    - plot_title: The title of the plot/visual.\n",
    "    - x_label: The string label value for the x label.\n",
    "    - y_label: The string label value for the y label.\n",
    "    - rotation: The integer value of the rotation of \n",
    "        x-tick values\n",
    "    - \n",
    "    \n",
    "    Output:\n",
    "    - A Bar Plot.\n",
    "    \"\"\"\n",
    "    \n",
    "    assert type(feature_to_visualize) == pd.Series, \\\n",
    "        \"Make sure the dataset input type is a Pandas Series.\"\n",
    "    assert type(plot_title) != 'str', \\\n",
    "        \"The title value must be a string value.\"\n",
    "    assert type(x_label) !=  'str', \\\n",
    "        \"The x label value must be a string value.\"\n",
    "    assert type(y_label) != 'str', \\\n",
    "        \"The y label value must be a string value.\"\n",
    "    assert type(rotation) != np.int8, \\\n",
    "        \"The rotation value must be an integer value.\"\n",
    "    \n",
    "    ax = feature_to_visualize.value_counts().plot.bar( \n",
    "                                                      figsize=(12,8),\n",
    "                                                      rot=rotation, \n",
    "                                                      legend=False,\n",
    "                                                      color=[\"lightgreen\", \n",
    "                                                             \"blue\", \n",
    "                                                             \"brown\", \n",
    "                                                             \"lightblue\",\n",
    "                                                             \"tan\", \n",
    "                                                             \"yellow\", \n",
    "                                                             \"orange\",\n",
    "                                                             \"green\", \n",
    "                                                             \"red\",\n",
    "                                                             \"purple\"]\n",
    "                                                      )\n",
    "    \n",
    "    if annotation:\n",
    "        ax.bar_label(ax.containers[0])\n",
    "    \n",
    "    plt.suptitle('')\n",
    "    plt.title(plot_title)\n",
    "    plt.ylabel(y_label)\n",
    "    plt.xlabel(x_label)\n",
    "    \n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0f372f4c",
   "metadata": {},
   "source": [
    "#### Visualization: Distribution of Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ff001fa1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/YAAAK9CAYAAACO8PdPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABmWUlEQVR4nO3dfXyP9f////trm53ZOTuxzMxJmLNCschJToYROSlShPjUe8KUs5JQ0VtvoRLKeZFQCDlZzmNOWq2c57QRm4pt5mRjO35/9PX69WrDps3LYbfr5XJcLl7P43E8X49jHRe67zizGIZhCAAAAAAAmJKDvRsAAAAAAAC3j2APAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAMBtGDVqlCwWyx35rsaNG6tx48bWz5s2bZLFYtGSJUvuyPc/99xzKlu27B35rtuVnp6u559/XkFBQbJYLBo4cKC9WwIA4I4h2AMAirw5c+bIYrFYF1dXVwUHBysyMlLvv/++Lly4UCDfc/r0aY0aNUoJCQkFMl9Bupt7y4uxY8dqzpw5evHFF/Xpp5/q2WefvWFtZmamJk+erAcffFBeXl7y8fFR1apV1bdvXx08ePAOdg0AQMFwsncDAADcLcaMGaOwsDBdvXpVSUlJ2rRpkwYOHKj33ntPX3/9tWrUqGGtHTFihIYNG5av+U+fPq3Ro0erbNmyeuCBB/K83bp16/L1PbfjZr198sknys7OLvQe/o0NGzaoXr16euONN25Z27FjR61evVpdu3ZVnz59dPXqVR08eFArV67UI488osqVK9+BjgEAKDgEewAA/p9WrVqpTp061s/Dhw/Xhg0b1KZNGz3++OM6cOCA3NzcJElOTk5ycircf0YvXbokd3d3OTs7F+r33EqxYsXs+v15cfbsWYWHh9+ybvfu3Vq5cqXefvttvfrqqzbrPvzwQ6WkpBRShzlduXJFzs7OcnDgAkoAwL/DvyQAANzEY489ptdff12//vqrPvvsM+t4bvfYx8bGqkGDBvLx8ZGHh4cqVapkDY+bNm3SQw89JEnq2bOn9bL/OXPmSPrrPvpq1aopPj5eDRs2lLu7u3Xbf95jf11WVpZeffVVBQUFqXjx4nr88cd18uRJm5qyZcvqueeey7Ht3+e8VW+53WN/8eJFvfzyywoJCZGLi4sqVaqk//3vfzIMw6bOYrGoX79+WrZsmapVqyYXFxdVrVpVa9asyf0H/g9nz55V7969FRgYKFdXV9WsWVNz5861rr/+vIHjx49r1apV1t5PnDiR63xHjx6VJNWvXz/HOkdHR5UoUcJm7LffflPv3r0VHBwsFxcXhYWF6cUXX1RmZqa15tixY+rcubP8/Pzk7u6uevXqadWqVTbzXO9z4cKFGjFihO677z65u7srLS1NkrRz5061bNlS3t7ecnd3V6NGjbRt2zabOS5cuKCBAweqbNmycnFxUUBAgJo3b64ffvghTz9LAMC9izP2AADcwrPPPqtXX31V69atU58+fXKt2bdvn9q0aaMaNWpozJgxcnFx0ZEjR6zhrEqVKhozZoxGjhypvn376tFHH5UkPfLII9Y5/vzzT7Vq1UpdunTRM888o8DAwJv29fbbb8tisWjo0KE6e/asJk2apGbNmikhIcF6ZUFe5KW3vzMMQ48//rg2btyo3r1764EHHtDatWs1ePBg/fbbb5o4caJN/XfffaevvvpK//nPf+Tp6an3339fHTt2VGJiYo4g/XeXL19W48aNdeTIEfXr109hYWFavHixnnvuOaWkpGjAgAGqUqWKPv30U8XExKh06dJ6+eWXJUn+/v65zhkaGipJmj9/vurXr3/Tqy5Onz6thx9+WCkpKerbt68qV66s3377TUuWLNGlS5fk7Oys5ORkPfLII7p06ZL69++vEiVKaO7cuXr88ce1ZMkSPfHEEzZzvvnmm3J2dtYrr7yijIwMOTs7a8OGDWrVqpVq166tN954Qw4ODpo9e7Yee+wxbd26VQ8//LAk6YUXXtCSJUvUr18/hYeH688//9R3332nAwcOqFatWjfcDwBAEWAAAFDEzZ4925Bk7N69+4Y13t7exoMPPmj9/MYbbxh//2d04sSJhiTj999/v+Ecu3fvNiQZs2fPzrGuUaNGhiRj2rRpua5r1KiR9fPGjRsNScZ9991npKWlWccXLVpkSDImT55sHQsNDTV69Ohxyzlv1luPHj2M0NBQ6+dly5YZkoy33nrLpq5Tp06GxWIxjhw5Yh2TZDg7O9uM/fTTT4Yk44MPPsjxXX83adIkQ5Lx2WefWccyMzONiIgIw8PDw2bfQ0NDjaioqJvOZxiGkZ2dbf1ZBwYGGl27djWmTJli/Prrrzlqu3fvbjg4OOR6XGRnZxuGYRgDBw40JBlbt261rrtw4YIRFhZmlC1b1sjKyjIM4///b1auXDnj0qVLNvNUrFjRiIyMtM5pGIZx6dIlIywszGjevLl1zNvb24iOjr7lPgIAih4uxQcAIA88PDxu+nR8Hx8fSdLy5ctv+0FzLi4u6tmzZ57ru3fvLk9PT+vnTp06qVSpUvrmm29u6/vz6ptvvpGjo6P69+9vM/7yyy/LMAytXr3aZrxZs2YqX7689XONGjXk5eWlY8eO3fJ7goKC1LVrV+tYsWLF1L9/f6Wnp2vz5s357t1isWjt2rV666235Ovrq88//1zR0dEKDQ3VU089Zb3HPjs7W8uWLVPbtm1tnrvw93mu9/jwww+rQYMG1nUeHh7q27evTpw4of3799ts16NHD5urKRISEnT48GE9/fTT+vPPP/XHH3/ojz/+0MWLF9W0aVNt2bLFejz5+Pho586dOn36dL73GwBwbyPYAwCQB+np6TYh+p+eeuop1a9fX88//7wCAwPVpUsXLVq0KF8h/7777svXg/IqVqxo89lisahChQo3vL+8oPz6668KDg7O8fOoUqWKdf3flSlTJsccvr6+On/+/C2/p2LFijkeLnej78krFxcXvfbaazpw4IBOnz6tzz//XPXq1dOiRYvUr18/SdLvv/+utLQ0VatW7ZY9VqpUKcf4jXoMCwuz+Xz48GFJfwV+f39/m2XGjBnKyMhQamqqJGn8+PHau3evQkJC9PDDD2vUqFG3/OUIAKBoINgDAHALp06dUmpqqipUqHDDGjc3N23ZskXffvutnn32Wf3888966qmn1Lx5c2VlZeXpe/JzX3xe/fMBf9fltaeC4OjomOu48Y8H7dlDqVKl1KVLF23ZskUVK1bUokWLdO3atUL7vn/+N77+i593331XsbGxuS4eHh6SpCeffFLHjh3TBx98oODgYL377ruqWrVqjiskAABFD8EeAIBb+PTTTyVJkZGRN61zcHBQ06ZN9d5772n//v16++23tWHDBm3cuFHSjUP27bp+tvc6wzB05MgRmyfY+/r65voKt3+eSc5Pb6GhoTp9+nSOWxMOHjxoXV8QQkNDdfjw4RxXPRT090h/XeJfo0YNXb16VX/88Yf8/f3l5eWlvXv33rLHQ4cO5RjPa4/Xb1Hw8vJSs2bNcl3+/rrBUqVK6T//+Y+WLVum48ePq0SJEnr77bfzu7sAgHsMwR4AgJvYsGGD3nzzTYWFhalbt243rDt37lyOsQceeECSlJGRIUkqXry4JBXYu9LnzZtnE66XLFmiM2fOqFWrVtax8uXLa8eOHTavZ1u5cmWO1+Llp7fWrVsrKytLH374oc34xIkTZbFYbL7/32jdurWSkpL0xRdfWMeuXbumDz74QB4eHmrUqFG+5zx8+LASExNzjKekpCguLk6+vr7y9/eXg4OD2rdvrxUrVuj777/PUX/9aoPWrVtr165diouLs667ePGiPv74Y5UtW1bh4eE37ad27doqX768/ve//yk9PT3H+t9//13SX1dYXL8k/7qAgAAFBwdbjy8AQNHF6+4AAPh/Vq9erYMHD+ratWtKTk7Whg0bFBsbq9DQUH399ddydXW94bZjxozRli1bFBUVpdDQUJ09e1YfffSRSpcubX2wWvny5eXj46Np06bJ09NTxYsXV926dXPcd51Xfn5+atCggXr27Knk5GRNmjRJFSpUsHkl3/PPP68lS5aoZcuWevLJJ3X06FF99tlnNg+zy29vbdu2VZMmTfTaa6/pxIkTqlmzptatW6fly5dr4MCBOea+XX379tX06dP13HPPKT4+XmXLltWSJUu0bds2TZo06abPPLiRn376SU8//bRatWqlRx99VH5+fvrtt980d+5cnT59WpMmTbLeOjB27FitW7dOjRo1Ut++fVWlShWdOXNGixcv1nfffScfHx8NGzZMn3/+uVq1aqX+/fvLz89Pc+fO1fHjx/Xll1/meD7APzk4OGjGjBlq1aqVqlatqp49e+q+++7Tb7/9po0bN8rLy0srVqzQhQsXVLp0aXXq1Ek1a9aUh4eHvv32W+3evVsTJky4rZ8vAOAeYt+H8gMAYH/XX3d3fXF2djaCgoKM5s2bG5MnT7Z5rdp1/3zd3fr164127doZwcHBhrOzsxEcHGx07drV+OWXX2y2W758uREeHm44OTnZvF6uUaNGRtWqVXPt70avu/v888+N4cOHGwEBAYabm5sRFRWV62vbJkyYYNx3332Gi4uLUb9+feP777/PMefNevvn6+4M469XusXExBjBwcFGsWLFjIoVKxrvvvuuzSvbDOOv193l9oq2G72G75+Sk5ONnj17GiVLljScnZ2N6tWr5/pKvry+7i45Odl45513jEaNGhmlSpUynJycDF9fX+Oxxx4zlixZkqP+119/Nbp37274+/sbLi4uRrly5Yzo6GgjIyPDWnP06FGjU6dOho+Pj+Hq6mo8/PDDxsqVK23muf7fbPHixbn29eOPPxodOnQwSpQoYbi4uBihoaHGk08+aaxfv94wDMPIyMgwBg8ebNSsWdPw9PQ0ihcvbtSsWdP46KOPbrnPAIB7n8Uw7oIn1wAAAAAAgNvCPfYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMSd7N2AG2dnZOn36tDw9PWWxWOzdDgAAAADgHmcYhi5cuKDg4GA5ONz8nDzBPg9Onz6tkJAQe7cBAAAAAChiTp48qdKlS9+0hmCfB56enpL++oF6eXnZuRsAAAAAwL0uLS1NISEh1jx6MwT7PLh++b2XlxfBHgAAAABwx+TldnAengcAAAAAgIkR7AEAAAAAMDGCPQAAAAAAJkawBwAAAADAxOwa7MuWLSuLxZJjiY6OliRduXJF0dHRKlGihDw8PNSxY0clJyfbzJGYmKioqCi5u7srICBAgwcP1rVr12xqNm3apFq1asnFxUUVKlTQnDlz7tQuAgAAAABQqOwa7Hfv3q0zZ85Yl9jYWElS586dJUkxMTFasWKFFi9erM2bN+v06dPq0KGDdfusrCxFRUUpMzNT27dv19y5czVnzhyNHDnSWnP8+HFFRUWpSZMmSkhI0MCBA/X8889r7dq1d3ZnAQAAAAAoBBbDMAx7N3HdwIEDtXLlSh0+fFhpaWny9/fXggUL1KlTJ0nSwYMHVaVKFcXFxalevXpavXq12rRpo9OnTyswMFCSNG3aNA0dOlS///67nJ2dNXToUK1atUp79+61fk+XLl2UkpKiNWvW5KmvtLQ0eXt7KzU1ldfdAQAAAAAKXX5y6F1zj31mZqY+++wz9erVSxaLRfHx8bp69aqaNWtmralcubLKlCmjuLg4SVJcXJyqV69uDfWSFBkZqbS0NO3bt89a8/c5rtdcnyM3GRkZSktLs1kAAAAAALgb3TXBftmyZUpJSdFzzz0nSUpKSpKzs7N8fHxs6gIDA5WUlGSt+Xuov77++rqb1aSlpeny5cu59jJu3Dh5e3tbl5CQkH+7ewAAAAAAFIq7JtjPnDlTrVq1UnBwsL1b0fDhw5WammpdTp48ae+WAAAAAADIlZO9G5CkX3/9Vd9++62++uor61hQUJAyMzOVkpJic9Y+OTlZQUFB1ppdu3bZzHX9qfl/r/nnk/STk5Pl5eUlNze3XPtxcXGRi4vLv94vAAAAAAAK211xxn727NkKCAhQVFSUdax27doqVqyY1q9fbx07dOiQEhMTFRERIUmKiIjQnj17dPbsWWtNbGysvLy8FB4ebq35+xzXa67PAQAAAACAmdk92GdnZ2v27Nnq0aOHnJz+/wsIvL291bt3bw0aNEgbN25UfHy8evbsqYiICNWrV0+S1KJFC4WHh+vZZ5/VTz/9pLVr12rEiBGKjo62nnF/4YUXdOzYMQ0ZMkQHDx7URx99pEWLFikmJsYu+wsAAAAAQEGy+6X43377rRITE9WrV68c6yZOnCgHBwd17NhRGRkZioyM1EcffWRd7+joqJUrV+rFF19URESEihcvrh49emjMmDHWmrCwMK1atUoxMTGaPHmySpcurRkzZigyMvKO7B8AAAAAAIXprnqP/d2K99jfvt9++01Dhw7V6tWrdenSJVWoUEGzZ89WnTp1JEkWiyXX7caPH6/BgwdLkn755RcNHjxY27ZtU2ZmpmrUqKE333xTTZo0sdbv3r1bw4YNU3x8vCwWix5++GGNHz9eNWvWLPydBAAAAIACZsr32OPec/78edWvX1/FihXT6tWrtX//fk2YMEG+vr7WmjNnztgss2bNksViUceOHa01bdq00bVr17RhwwbFx8erZs2aatOmjfWVhunp6WrZsqXKlCmjnTt36rvvvpOnp6ciIyN19erVO77fAAAAAHAnccY+Dzhjf3uGDRumbdu2aevWrXnepn379rpw4YL1gYd//PGH/P39tWXLFj366KOSpAsXLsjLy0uxsbFq1qyZvv/+ez300ENKTExUSEiIJGnPnj2qUaOGDh8+rAoVKhT8zgEAAABAIeKMPe4KX3/9terUqaPOnTsrICBADz74oD755JMb1icnJ2vVqlXq3bu3daxEiRKqVKmS5s2bp4sXL+ratWuaPn26AgICVLt2bUlSpUqVVKJECc2cOVOZmZm6fPmyZs6cqSpVqqhs2bKFvZsAAAAAYFcEexSaY8eOaerUqapYsaLWrl2rF198Uf3799fcuXNzrZ87d648PT3VoUMH65jFYtG3336rH3/8UZ6ennJ1ddV7772nNWvWWC/p9/T01KZNm/TZZ5/Jzc1NHh4eWrNmjVavXm3zpgUAAAAAuBcR7FFosrOzVatWLY0dO1YPPvig+vbtqz59+mjatGm51s+aNUvdunWTq6urdcwwDEVHRysgIEBbt27Vrl271L59e7Vt21ZnzpyRJF2+fFm9e/dW/fr1tWPHDm3btk3VqlVTVFSULl++fEf2FQAAAADshdOZKDSlSpVSeHi4zViVKlX05Zdf5qjdunWrDh06pC+++MJmfMOGDVq5cqXOnz9vva/ko48+UmxsrObOnathw4ZpwYIFOnHihOLi4uTg8NfvqhYsWCBfX18tX75cXbp0KaQ9BAAAAAD744w9Ck39+vV16NAhm7FffvlFoaGhOWpnzpyp2rVr53g93aVLlyTJGtivc3BwUHZ2trXGwcHB5tV51z9frwEAAACAexXBHoUmJiZGO3bs0NixY3XkyBEtWLBAH3/8saKjo23q0tLStHjxYj3//PM55oiIiJCvr6969Oihn376yfpO++PHjysqKkqS1Lx5c50/f17R0dE6cOCA9u3bp549e8rJycnmXfcAAAAAcC8i2KPQPPTQQ1q6dKk+//xzVatWTW+++aYmTZqkbt262dQtXLhQhmGoa9euOeYoWbKk1qxZo/T0dD322GOqU6eOvvvuOy1fvtx6dr9y5cpasWKFfv75Z0VEROjRRx/V6dOntWbNGpUqVeqO7CsAAAAA2Avvsc8D3mMPAAAAALiTeI89AAAAAABFBE/FLwImn59s7xaKnAG+A+zdAgAAAIAigjP2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAAAAAEyMYA8AAAAAgIkR7AEAAAAAMDGCPQAAAAAAJkawBwAAAADAxAj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAAAAAEyMYA8AAAAAgIkR7AEAAAAAMDGCPQAAAAAAJkawBwAAAADAxAj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAAAAAEyMYA8AAAAAgIkR7AEAAAAAMDGCPQAAAAAAJkawBwAAAADAxAj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAAAAAEyMYA8AAAAAgIkR7AEAAAAAMDGCPQAAAAAAJkawBwAAAADAxAj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAAAAAEyMYA8AAAAAgIkR7AEAAAAAMDGCPQAAAAAAJkawBwAAAADAxAj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAAAAAEyMYA8AAAAAgIkR7AEAAAAAMDGCPQAAAAAAJkawBwAAAADAxAj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAAAAAEyMYA8AAAAAgIkR7AEAAAAAMDGCPQAAAAAAJkawBwAAAADAxAj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAAAAAEyMYA8AAAAAgIkR7AEAAAAAMDGCPQAAAAAAJmb3YP/bb7/pmWeeUYkSJeTm5qbq1avr+++/t643DEMjR45UqVKl5ObmpmbNmunw4cM2c5w7d07dunWTl5eXfHx81Lt3b6Wnp9vU/Pzzz3r00Ufl6uqqkJAQjR8//o7sHwAAAAAAhcmuwf78+fOqX7++ihUrptWrV2v//v2aMGGCfH19rTXjx4/X+++/r2nTpmnnzp0qXry4IiMjdeXKFWtNt27dtG/fPsXGxmrlypXasmWL+vbta12flpamFi1aKDQ0VPHx8Xr33Xc1atQoffzxx3d0fwEAAAAAKGgWwzAMe335sGHDtG3bNm3dujXX9YZhKDg4WC+//LJeeeUVSVJqaqoCAwM1Z84cdenSRQcOHFB4eLh2796tOnXqSJLWrFmj1q1b69SpUwoODtbUqVP12muvKSkpSc7OztbvXrZsmQ4ePHjLPtPS0uTt7a3U1FR5eXkV0N7fOZPPT7Z3C0XOAN8B9m4BAAAAgInlJ4fa9Yz9119/rTp16qhz584KCAjQgw8+qE8++cS6/vjx40pKSlKzZs2sY97e3qpbt67i4uIkSXFxcfLx8bGGeklq1qyZHBwctHPnTmtNw4YNraFekiIjI3Xo0CGdP38+R18ZGRlKS0uzWQAAAAAAuBvZNdgfO3ZMU6dOVcWKFbV27Vq9+OKL6t+/v+bOnStJSkpKkiQFBgbabBcYGGhdl5SUpICAAJv1Tk5O8vPzs6nJbY6/f8ffjRs3Tt7e3tYlJCSkAPYWAAAAAICCZ9dgn52drVq1amns2LF68MEH1bdvX/Xp00fTpk2zZ1saPny4UlNTrcvJkyft2g8AAAAAADdi12BfqlQphYeH24xVqVJFiYmJkqSgoCBJUnJysk1NcnKydV1QUJDOnj1rs/7atWs6d+6cTU1uc/z9O/7OxcVFXl5eNgsAAAAAAHcjuwb7+vXr69ChQzZjv/zyi0JDQyVJYWFhCgoK0vr1663r09LStHPnTkVEREiSIiIilJKSovj4eGvNhg0blJ2drbp161prtmzZoqtXr1prYmNjValSJZsn8AMAAAAAYDZ2DfYxMTHasWOHxo4dqyNHjmjBggX6+OOPFR0dLUmyWCwaOHCg3nrrLX399dfas2ePunfvruDgYLVv317SX2f4W7ZsqT59+mjXrl3atm2b+vXrpy5duig4OFiS9PTTT8vZ2Vm9e/fWvn379MUXX2jy5MkaNGiQvXYdAAAAAIAC4WTPL3/ooYe0dOlSDR8+XGPGjFFYWJgmTZqkbt26WWuGDBmiixcvqm/fvkpJSVGDBg20Zs0aubq6Wmvmz5+vfv36qWnTpnJwcFDHjh31/vvvW9d7e3tr3bp1io6OVu3atVWyZEmNHDnS5l33AAAAAACYkV3fY28WvMce+cV77AEAAAD8G6Z5jz0AAAAAAPh3CPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGJ2DfajRo2SxWKxWSpXrmxdf+XKFUVHR6tEiRLy8PBQx44dlZycbDNHYmKioqKi5O7uroCAAA0ePFjXrl2zqdm0aZNq1aolFxcXVahQQXPmzLkTuwcAAAAAQKGz+xn7qlWr6syZM9blu+++s66LiYnRihUrtHjxYm3evFmnT59Whw4drOuzsrIUFRWlzMxMbd++XXPnztWcOXM0cuRIa83x48cVFRWlJk2aKCEhQQMHDtTzzz+vtWvX3tH9BAAAAACgMDjZvQEnJwUFBeUYT01N1cyZM7VgwQI99thjkqTZs2erSpUq2rFjh+rVq6d169Zp//79+vbbbxUYGKgHHnhAb775poYOHapRo0bJ2dlZ06ZNU1hYmCZMmCBJqlKlir777jtNnDhRkZGRd3RfAQAAAAAoaHY/Y3/48GEFBwerXLly6tatmxITEyVJ8fHxunr1qpo1a2atrVy5ssqUKaO4uDhJUlxcnKpXr67AwEBrTWRkpNLS0rRv3z5rzd/nuF5zfY7cZGRkKC0tzWYBAAAAAOBuZNdgX7duXc2ZM0dr1qzR1KlTdfz4cT366KO6cOGCkpKS5OzsLB8fH5ttAgMDlZSUJElKSkqyCfXX119fd7OatLQ0Xb58Ode+xo0bJ29vb+sSEhJSELsLAAAAAECBs+ul+K1atbL+uUaNGqpbt65CQ0O1aNEiubm52a2v4cOHa9CgQdbPaWlphHsAAAAAwF3J7pfi/52Pj4/uv/9+HTlyREFBQcrMzFRKSopNTXJysvWe/KCgoBxPyb/++VY1Xl5eN/zlgYuLi7y8vGwWAAAAAADuRndVsE9PT9fRo0dVqlQp1a5dW8WKFdP69eut6w8dOqTExERFRERIkiIiIrRnzx6dPXvWWhMbGysvLy+Fh4dba/4+x/Wa63MAAAAAAGBmdg32r7zyijZv3qwTJ05o+/bteuKJJ+To6KiuXbvK29tbvXv31qBBg7Rx40bFx8erZ8+eioiIUL169SRJLVq0UHh4uJ599ln99NNPWrt2rUaMGKHo6Gi5uLhIkl544QUdO3ZMQ4YM0cGDB/XRRx9p0aJFiomJseeuAwAAAABQIOx6j/2pU6fUtWtX/fnnn/L391eDBg20Y8cO+fv7S5ImTpwoBwcHdezYURkZGYqMjNRHH31k3d7R0VErV67Uiy++qIiICBUvXlw9evTQmDFjrDVhYWFatWqVYmJiNHnyZJUuXVozZszgVXcAAAAAgHuCxTAMw95N3O3S0tLk7e2t1NRUU95vP/n8ZHu3UOQM8B1g7xYAAAAAmFh+cuhddY89AAAAAADIH4I9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAieU72J88eVKnTp2yft61a5cGDhyojz/+uEAbAwAAAAAAt5bvYP/0009r48aNkqSkpCQ1b95cu3bt0muvvaYxY8YUeIMAAAAAAODG8h3s9+7dq4cffliStGjRIlWrVk3bt2/X/PnzNWfOnILuDwAAAAAA3ES+g/3Vq1fl4uIiSfr222/1+OOPS5IqV66sM2fO3HYj77zzjiwWiwYOHGgdu3LliqKjo1WiRAl5eHioY8eOSk5OttkuMTFRUVFRcnd3V0BAgAYPHqxr167Z1GzatEm1atWSi4uLKlSowC8gAAAAAAD3jHwH+6pVq2ratGnaunWrYmNj1bJlS0nS6dOnVaJEidtqYvfu3Zo+fbpq1KhhMx4TE6MVK1Zo8eLF2rx5s06fPq0OHTpY12dlZSkqKkqZmZnavn275s6dqzlz5mjkyJHWmuPHjysqKkpNmjRRQkKCBg4cqOeff15r1669rV4BAAAAALib5DvY//e//9X06dPVuHFjde3aVTVr1pQkff3119ZL9PMjPT1d3bp10yeffCJfX1/reGpqqmbOnKn33ntPjz32mGrXrq3Zs2dr+/bt2rFjhyRp3bp12r9/vz777DM98MADatWqld58801NmTJFmZmZkqRp06YpLCxMEyZMUJUqVdSvXz916tRJEydOzHevAAAAAADcbfId7Bs3bqw//vhDf/zxh2bNmmUd79u3r6ZNm5bvBqKjoxUVFaVmzZrZjMfHx+vq1as245UrV1aZMmUUFxcnSYqLi1P16tUVGBhorYmMjFRaWpr27dtnrfnn3JGRkdY5cpORkaG0tDSbBQAAAACAu9FtvcfeMAzFx8dr+vTpunDhgiTJ2dlZ7u7u+Zpn4cKF+uGHHzRu3Lgc65KSkuTs7CwfHx+b8cDAQCUlJVlr/h7qr6+/vu5mNWlpabp8+XKufY0bN07e3t7WJSQkJF/7BQAAAADAneKU3w1+/fVXtWzZUomJicrIyFDz5s3l6emp//73v8rIyMjzWfuTJ09qwIABio2Nlaura74bL0zDhw/XoEGDrJ/T0tII9wAAAACAu1K+z9gPGDBAderU0fnz5+Xm5mYdf+KJJ7R+/fo8zxMfH6+zZ8+qVq1acnJykpOTkzZv3qz3339fTk5OCgwMVGZmplJSUmy2S05OVlBQkCQpKCgox1Pyr3++VY2Xl5dN/3/n4uIiLy8vmwUAAAAAgLtRvoP91q1bNWLECDk7O9uMly1bVr/99lue52natKn27NmjhIQE61KnTh1169bN+udixYrZ/LLg0KFDSkxMVEREhCQpIiJCe/bs0dmzZ601sbGx8vLyUnh4uLXmn79wiI2Ntc4BAAAAAICZ5ftS/OzsbGVlZeUYP3XqlDw9PfM8j6enp6pVq2YzVrx4cZUoUcI63rt3bw0aNEh+fn7y8vLSSy+9pIiICNWrV0+S1KJFC4WHh+vZZ5/V+PHjlZSUpBEjRig6OlouLi6SpBdeeEEffvihhgwZol69emnDhg1atGiRVq1ald9dBwAAAADgrpPvM/YtWrTQpEmTrJ8tFovS09P1xhtvqHXr1gXZmyZOnKg2bdqoY8eOatiwoYKCgvTVV19Z1zs6OmrlypVydHRURESEnnnmGXXv3l1jxoyx1oSFhWnVqlWKjY1VzZo1NWHCBM2YMUORkZEF2isAAAAAAPZgMQzDyM8Gp06dUmRkpAzD0OHDh1WnTh0dPnxYJUuW1JYtWxQQEFBYvdpNWlqavL29lZqaasr77Sefn2zvFoqcAb4D7N0CAAAAABPLTw7N96X4pUuX1k8//aSFCxfq559/Vnp6unr37q1u3brd8GF0AAAAAACgcOQ72EuSk5OTnnnmmYLuBQAAAAAA5FOegv3XX3+d5wkff/zx224GAAAAAADkT56Cffv27fM0mcViyfWJ+QAAAAAAoHDkKdhnZ2cXdh8AAAAAAOA25Pt1dwAAAAAA4O5xW8F+/fr1atOmjcqXL6/y5curTZs2+vbbbwu6NwAAAAAAcAv5DvYfffSRWrZsKU9PTw0YMEADBgyQl5eXWrdurSlTphRGjwAAAAAA4Aby/bq7sWPHauLEierXr591rH///qpfv77Gjh2r6OjoAm0QAAAAAADcWL7P2KekpKhly5Y5xlu0aKHU1NQCaQoAAAAAAORNvoP9448/rqVLl+YYX758udq0aVMgTQEAAAAAgLzJ96X44eHhevvtt7Vp0yZFRERIknbs2KFt27bp5Zdf1vvvv2+t7d+/f8F1CgAAAAAAcrAYhmHkZ4OwsLC8TWyx6NixY7fV1N0mLS1N3t7eSk1NlZeXl73bybfJ5yfbu4UiZ4DvAHu3AAAAAMDE8pND833G/vjx47fdGAAAAAAAKFi39R57AAAAAABwd8j3GXvDMLRkyRJt3LhRZ8+eVXZ2ts36r776qsCaAwAAAAAAN5fvYD9w4EBNnz5dTZo0UWBgoCwWS2H0BQAAAAAA8iDfwf7TTz/VV199pdatWxdGPwAAAAAAIB/yfY+9t7e3ypUrVxi9AAAAAACAfMp3sB81apRGjx6ty5cvF0Y/AAAAAAAgH/J9Kf6TTz6pzz//XAEBASpbtqyKFStms/6HH34osOYAAAAAAMDN5TvY9+jRQ/Hx8XrmmWd4eB4AAAAAAHaW72C/atUqrV27Vg0aNCiMfgAAAAAAQD7k+x77kJAQeXl5FUYvAAAAAAAgn/Id7CdMmKAhQ4boxIkThdAOAAAAAADIj3xfiv/MM8/o0qVLKl++vNzd3XM8PO/cuXMF1hwAAAAAALi5fAf7SZMmFUIbAAAAAADgdtzWU/EBAAAAAMDdId/B/u+uXLmizMxMmzEerAcAAAAAwJ2T74fnXbx4Uf369VNAQICKFy8uX19fmwUAAAAAANw5+Q72Q4YM0YYNGzR16lS5uLhoxowZGj16tIKDgzVv3rzC6BEAAAAAANxAvi/FX7FihebNm6fGjRurZ8+eevTRR1WhQgWFhoZq/vz56tatW2H0CQAAAAAAcpHvM/bnzp1TuXLlJP11P/3119s1aNBAW7ZsKdjuAAAAAADATeU72JcrV07Hjx+XJFWuXFmLFi2S9NeZfB8fnwJtDgAAAAAA3Fy+g33Pnj31008/SZKGDRumKVOmyNXVVTExMRo8eHCBNwgAAAAAAG4s3/fYx8TEWP/crFkzHThwQD/88IMqVKigGjVqFGhzAAAAAADg5v7Ve+wlqWzZsipbtmwBtAIAAAAAAPIrz5fix8XFaeXKlTZj8+bNU1hYmAICAtS3b19lZGQUeIMAAAAAAODG8hzsx4wZo3379lk/79mzR71791azZs00bNgwrVixQuPGjSuUJgEAAAAAQO7yHOwTEhLUtGlT6+eFCxeqbt26+uSTTzRo0CC9//771ifkAwAAAACAOyPPwf78+fMKDAy0ft68ebNatWpl/fzQQw/p5MmTBdsdAAAAAAC4qTwH+8DAQOv76zMzM/XDDz+oXr161vUXLlxQsWLFCr5DAAAAAABwQ3kO9q1bt9awYcO0detWDR8+XO7u7nr00Uet63/++WeVL1++UJoEAAAAAAC5y/Pr7t5880116NBBjRo1koeHh+bOnStnZ2fr+lmzZqlFixaF0iQAAAAAAMhdnoN9yZIltWXLFqWmpsrDw0OOjo426xcvXiwPD48CbxAAAAAAANxYnoP9dd7e3rmO+/n5/etmAAAAAABA/uT5HnsAAAAAAHD3IdgDAAAAAGBiBHsAAAAAAEwsT8G+Vq1aOn/+vCRpzJgxunTpUqE2BQAAAAAA8iZPwf7AgQO6ePGiJGn06NFKT08v1KYAAAAAAEDe5Omp+A888IB69uypBg0ayDAM/e9//7vhq+1GjhxZoA0CAAAAAIAby1OwnzNnjt544w2tXLlSFotFq1evlpNTzk0tFgvBHgAAAACAOyhPwb5SpUpauHChJMnBwUHr169XQEBAoTYGAAAAAABuLU/B/u+ys7MLow8AAAAAAHAb8h3sJeno0aOaNGmSDhw4IEkKDw/XgAEDVL58+QJtDgAAAAAA3Fy+32O/du1ahYeHa9euXapRo4Zq1KihnTt3qmrVqoqNjS2MHgEAAAAAwA3k+4z9sGHDFBMTo3feeSfH+NChQ9W8efMCaw4AAAAAANxcvs/YHzhwQL17984x3qtXL+3fv79AmgIAAAAAAHmT72Dv7++vhISEHOMJCQk8KR8AAAAAgDss35fi9+nTR3379tWxY8f0yCOPSJK2bdum//73vxo0aFCBNwgAAAAAAG4s38H+9ddfl6enpyZMmKDhw4dLkoKDgzVq1Cj179+/wBsEAAAAAAA3lu9gb7FYFBMTo5iYGF24cEGS5OnpWeCNAQAAAACAW7ut99hfR6AHAAAAAMC+8v3wPAAAAAAAcPcg2AMAAAAAYGIEewAAAAAATIxgDwAAAACAid1WsO/Xr5/OnTtX0L0AAAAAAIB8ynOwP3XqlPXPCxYsUHp6uiSpevXqOnnyZMF3BgAAAAAAbinPr7urXLmySpQoofr16+vKlSs6efKkypQpoxMnTujq1auF2SMAAAAAALiBPJ+xT0lJ0eLFi1W7dm1lZ2erdevWuv/++5WRkaG1a9cqOTm5MPsEAAAAAAC5yHOwv3r1qh5++GG9/PLLcnNz048//qjZs2fL0dFRs2bNUlhYmCpVqlSYvQIAAAAAgH/I86X4Pj4+euCBB1S/fn1lZmbq8uXLql+/vpycnPTFF1/ovvvu0+7duwuzVwAAAAAA8A95PmP/22+/acSIEXJxcdG1a9dUu3ZtPfroo8rMzNQPP/wgi8WiBg0aFGavAAAAAADgH/Ic7EuWLKm2bdtq3Lhxcnd31+7du/XSSy/JYrHolVdekbe3txo1alSYvQIAAAAAgH+4rffYS5K3t7eefPJJFStWTBs2bNDx48f1n//8J19zTJ06VTVq1JCXl5e8vLwUERGh1atXW9dfuXJF0dHRKlGihDw8PNSxY8ccD+lLTExUVFSU3N3dFRAQoMGDB+vatWs2NZs2bVKtWrXk4uKiChUqaM6cObe72wAAAAAA3FVuK9j//PPPKl26tCQpNDRUxYoVU1BQkJ566ql8zVO6dGm98847io+P1/fff6/HHntM7dq10759+yRJMTExWrFihRYvXqzNmzfr9OnT6tChg3X7rKwsRUVFKTMzU9u3b9fcuXM1Z84cjRw50lpz/PhxRUVFqUmTJkpISNDAgQP1/PPPa+3atbez6wAAAAAA3FUshmEY9m7i7/z8/PTuu++qU6dO8vf314IFC9SpUydJ0sGDB1WlShXFxcWpXr16Wr16tdq0aaPTp08rMDBQkjRt2jQNHTpUv//+u5ydnTV06FCtWrVKe/futX5Hly5dlJKSojVr1uTaQ0ZGhjIyMqyf09LSFBISotTUVHl5eRXi3heOyecn27uFImeA7wB7twAAAADAxNLS0uTt7Z2nHHrbl+IXtKysLC1cuFAXL15URESE4uPjdfXqVTVr1sxaU7lyZZUpU0ZxcXGSpLi4OFWvXt0a6iUpMjJSaWlp1rP+cXFxNnNcr7k+R27GjRsnb29v6xISElKQuwoAAAAAQIGxe7Dfs2ePPDw85OLiohdeeEFLly5VeHi4kpKS5OzsLB8fH5v6wMBAJSUlSZKSkpJsQv319dfX3awmLS1Nly9fzrWn4cOHKzU11bqcPHmyIHYVAAAAAIACl+f32BeWSpUqKSEhQampqVqyZIl69OihzZs327UnFxcXubi42LUHAAAAAADywu7B3tnZWRUqVJAk1a5dW7t379bkyZP11FNPKTMzUykpKTZn7ZOTkxUUFCRJCgoK0q5du2zmu/7U/L/X/PNJ+snJyfLy8pKbm1th7RYAAAAAAHeE3S/F/6fs7GxlZGSodu3aKlasmNavX29dd+jQISUmJioiIkKSFBERoT179ujs2bPWmtjYWHl5eSk8PNxa8/c5rtdcnwMAAAAAADOz6xn74cOHq1WrVipTpowuXLigBQsWaNOmTVq7dq28vb3Vu3dvDRo0SH5+fvLy8tJLL72kiIgI1atXT5LUokULhYeH69lnn9X48eOVlJSkESNGKDo62nop/QsvvKAPP/xQQ4YMUa9evbRhwwYtWrRIq1atsueuAwAAAABQIOwa7M+ePavu3bvrzJkz8vb2Vo0aNbR27Vo1b95ckjRx4kQ5ODioY8eOysjIUGRkpD766CPr9o6Ojlq5cqVefPFFRUREqHjx4urRo4fGjBljrQkLC9OqVasUExOjyZMnq3Tp0poxY4YiIyPv+P4CAAAAAFDQ7rr32N+N8vP+wLsR77G/83iPPQAAAIB/w5TvsQcAAAAAAPlHsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmJhdg/24ceP00EMPydPTUwEBAWrfvr0OHTpkU3PlyhVFR0erRIkS8vDwUMeOHZWcnGxTk5iYqKioKLm7uysgIECDBw/WtWvXbGo2bdqkWrVqycXFRRUqVNCcOXMKe/cAAAAAACh0dg32mzdvVnR0tHbs2KHY2FhdvXpVLVq00MWLF601MTExWrFihRYvXqzNmzfr9OnT6tChg3V9VlaWoqKilJmZqe3bt2vu3LmaM2eORo4caa05fvy4oqKi1KRJEyUkJGjgwIF6/vnntXbt2ju6vwAAAAAAFDSLYRiGvZu47vfff1dAQIA2b96shg0bKjU1Vf7+/lqwYIE6deokSTp48KCqVKmiuLg41atXT6tXr1abNm10+vRpBQYGSpKmTZumoUOH6vfff5ezs7OGDh2qVatWae/evdbv6tKli1JSUrRmzZpb9pWWliZvb2+lpqbKy8urcHa+EE0+P9neLRQ5A3wH2LsFAAAAACaWnxx6V91jn5qaKkny8/OTJMXHx+vq1atq1qyZtaZy5coqU6aM4uLiJElxcXGqXr26NdRLUmRkpNLS0rRv3z5rzd/nuF5zfY5/ysjIUFpams0CAAAAAMDd6K4J9tnZ2Ro4cKDq16+vatWqSZKSkpLk7OwsHx8fm9rAwEAlJSVZa/4e6q+vv77uZjVpaWm6fPlyjl7GjRsnb29v6xISElIg+wgAAAAAQEG7a4J9dHS09u7dq4ULF9q7FQ0fPlypqanW5eTJk/ZuCQAAAACAXDnZuwFJ6tevn1auXKktW7aodOnS1vGgoCBlZmYqJSXF5qx9cnKygoKCrDW7du2yme/6U/P/XvPPJ+knJyfLy8tLbm5uOfpxcXGRi4tLgewbAAAAAACFya5n7A3DUL9+/bR06VJt2LBBYWFhNutr166tYsWKaf369daxQ4cOKTExUREREZKkiIgI7dmzR2fPnrXWxMbGysvLS+Hh4daav89xveb6HAAAAAAAmJVdz9hHR0drwYIFWr58uTw9Pa33xHt7e8vNzU3e3t7q3bu3Bg0aJD8/P3l5eemll15SRESE6tWrJ0lq0aKFwsPD9eyzz2r8+PFKSkrSiBEjFB0dbT3r/sILL+jDDz/UkCFD1KtXL23YsEGLFi3SqlWr7LbvAAAAAAAUBLuesZ86dapSU1PVuHFjlSpVyrp88cUX1pqJEyeqTZs26tixoxo2bKigoCB99dVX1vWOjo5auXKlHB0dFRERoWeeeUbdu3fXmDFjrDVhYWFatWqVYmNjVbNmTU2YMEEzZsxQZGTkHd1fAAAAAAAK2l31Hvu7Fe+xR37xHnsAAAAA/4Zp32MPAAAAAADyh2APAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gDwL2zZskVt27ZVcHCwLBaLli1bZrPeYrHkurz77rvWmrfffluPPPKI3N3d5ePjk+M75syZc8N5zp49W8h7CAAAgLsdwR4A/oWLFy+qZs2amjJlSq7rz5w5Y7PMmjVLFotFHTt2tNZkZmaqc+fOevHFF3Od46mnnsoxT2RkpBo1aqSAgIBC2S8AAACYh5O9GwAAM2vVqpVatWp1w/VBQUE2n5cvX64mTZqoXLly1rHRo0dL+uvMfG7c3Nzk5uZm/fz7779rw4YNmjlz5r/oHAAAAPcKgj0A3CHJyclatWqV5s6d+6/mmTdvntzd3dWpU6cC6gwAAABmxqX4AHCHzJ07V56enurQocO/mmfmzJl6+umnbc7iAwAAoOjijD0A3CGzZs1St27d5OrqettzxMXF6cCBA/r0008LsDMAAACYGcEeAO6ArVu36tChQ/riiy/+1TwzZszQAw88oNq1axdQZwAAADA7LsUHgDtg5syZql27tmrWrHnbc6Snp2vRokXq3bt3AXYGAAAAs+OMPQD8C+np6Tpy5Ij18/Hjx5WQkCA/Pz+VKVNGkpSWlqbFixdrwoQJuc6RmJioc+fOKTExUVlZWUpISJAkVahQQR4eHta6L774QteuXdMzzzxTeDsEAAAA0yHYA8C/8P3336tJkybWz4MGDZIk9ejRw/r6uoULF8owDHXt2jXXOUaOHGnzpPwHH3xQkrRx40Y1btzYOj5z5kx16NBBPj4+BbsTAAAAMDWLYRiGvZu426Wlpcnb21upqany8vKydzv5Nvn8ZHu3UOQM8B1g7xYAAAAAmFh+cij32AMAAAAAYGJcig/gnmCx2LuDoofrvQAAAO4OnLEHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDE7Brst2zZorZt2yo4OFgWi0XLli2zWW8YhkaOHKlSpUrJzc1NzZo10+HDh21qzp07p27dusnLy0s+Pj7q3bu30tPTbWp+/vlnPfroo3J1dVVISIjGjx9f2LsGAAAAAMAdYddgf/HiRdWsWVNTpkzJdf348eP1/vvva9q0adq5c6eKFy+uyMhIXblyxVrTrVs37du3T7GxsVq5cqW2bNmivn37WtenpaWpRYsWCg0NVXx8vN59912NGjVKH3/8caHvHwAAAAAAhc3Jnl/eqlUrtWrVKtd1hmFo0qRJGjFihNq1aydJmjdvngIDA7Vs2TJ16dJFBw4c0Jo1a7R7927VqVNHkvTBBx+odevW+t///qfg4GDNnz9fmZmZmjVrlpydnVW1alUlJCTovffes/kFAAAAAAAAZnTX3mN//PhxJSUlqVmzZtYxb29v1a1bV3FxcZKkuLg4+fj4WEO9JDVr1kwODg7auXOntaZhw4Zydna21kRGRurQoUM6f/58rt+dkZGhtLQ0mwUAAAAAgLvRXRvsk5KSJEmBgYE244GBgdZ1SUlJCggIsFnv5OQkPz8/m5rc5vj7d/zTuHHj5O3tbV1CQkL+/Q4BAAAAAFAI7tpgb0/Dhw9XamqqdTl58qS9WwIAAAAAIFd3bbAPCgqSJCUnJ9uMJycnW9cFBQXp7NmzNuuvXbumc+fO2dTkNsffv+OfXFxc5OXlZbMAAAAAAHA3umuDfVhYmIKCgrR+/XrrWFpamnbu3KmIiAhJUkREhFJSUhQfH2+t2bBhg7Kzs1W3bl1rzZYtW3T16lVrTWxsrCpVqiRfX987tDcAAAAAABQOuwb79PR0JSQkKCEhQdJfD8xLSEhQYmKiLBaLBg4cqLfeektff/219uzZo+7duys4OFjt27eXJFWpUkUtW7ZUnz59tGvXLm3btk39+vVTly5dFBwcLEl6+umn5ezsrN69e2vfvn364osvNHnyZA0aNMhOew0AAAAAQMGx6+vuvv/+ezVp0sT6+XrY7tGjh+bMmaMhQ4bo4sWL6tu3r1JSUtSgQQOtWbNGrq6u1m3mz5+vfv36qWnTpnJwcFDHjh31/vvvW9d7e3tr3bp1io6OVu3atVWyZEmNHDmSV90BAAAAAO4JFsMwDHs3cbdLS0uTt7e3UlNTTXm//eTzk+3dQpEzwHeAvVsociwWe3dQ9PCvBwAAQOHJTw69a++xBwAAAAAAt0awBwAAAADAxAj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAAAAAEyMYA8AAAAAgIkR7AEAAAAAMDGCPQAAAAAAJkawBwAAAADAxAj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAAAAAEyMYA8AAAAAgIkR7AEAAAAAMDGCPQAAAAAAJkawBwAAAADAxAj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHsAAJAvo0aNksVisVkqV65sXf9///d/Kl++vNzc3OTv76927drp4MGDNnP8c3uLxaKFCxfe6V0BAOCe4GTvBgAAgPlUrVpV3377rfWzk9P//78UtWvXVrdu3VSmTBmdO3dOo0aNUosWLXT8+HE5Ojpa62bPnq2WLVtaP/v4+NyR3gEAuNcQ7AEAQL45OTkpKCgo13V9+/a1/rls2bJ66623VLNmTZ04cULly5e3rvPx8bnhHAAAIO+4FB8AAOTb4cOHFRwcrHLlyqlbt25KTEzMte7ixYuaPXu2wsLCFBISYrMuOjpaJUuW1MMPP6xZs2bJMIw70ToAAPccgj0AAMiXunXras6cOVqzZo2mTp2q48eP69FHH9WFCxesNR999JE8PDzk4eGh1atXKzY2Vs7Oztb1Y8aM0aJFixQbG6uOHTvqP//5jz744AN77A4AAKZHsAcAAPnSqlUrde7cWTVq1FBkZKS++eYbpaSkaNGiRdaabt266ccff9TmzZt1//3368knn9SVK1es619//XXVr19fDz74oIYOHaohQ4bo3XfftcfuAHnyzjvvyGKxaODAgZKkEydO5PoQSIvFosWLF+fY/s8//1Tp0qVlsViUkpJyZ5sHcM8j2AMAgH/Fx8dH999/v44cOWId8/b2VsWKFdWwYUMtWbJEBw8e1NKlS284R926dXXq1CllZGTciZaBfNm9e7emT5+uGjVqWMdCQkJ05swZm2X06NHy8PBQq1atcszRu3dvm+0BoCAR7AEAwL+Snp6uo0ePqlSpUrmuNwxDhmHcNLQnJCTI19dXLi4uhdUmcFvS09PVrVs3ffLJJ/L19bWOOzo6KigoyGZZunSpnnzySXl4eNjMMXXqVKWkpOiVV1650+0DKCII9gAAIF9eeeUVbd68WSdOnND27dv1xBNPyNHRUV27dtWxY8c0btw4xcfHKzExUdu3b1fnzp3l5uam1q1bS5JWrFihGTNmaO/evTpy5IimTp2qsWPH6qWXXrLzngE5RUdHKyoqSs2aNbtpXXx8vBISEtS7d2+b8f3792vMmDGaN2+eHBz4X28AhYPX3QEAgHw5deqUunbtqj///FP+/v5q0KCBduzYIX9/f129elVbt27VpEmTdP78eQUGBqphw4bavn27AgICJEnFihXTlClTFBMTI8MwVKFCBb333nvq06ePnfcMsLVw4UL98MMP2r179y1rZ86cqSpVquiRRx6xjmVkZKhr16569913VaZMGR07dqww2wVQhBHsAQBAvixcuPCG64KDg/XNN9/cdPuWLVuqZcuWBd0WUKBOnjypAQMGKDY2Vq6urjetvXz5shYsWKDXX3/dZnz48OGqUqWKnnnmmcJsFQC4FB8AAAD4p/j4eJ09e1a1atWSk5OTnJyctHnzZr3//vtycnJSVlaWtXbJkiW6dOmSunfvbjPHhg0btHjxYuv2TZs2lSSVLFlSb7zxxh3dHwD3Ns7YAwBgEguqVrV3C0XO0/v22bsF2EnTpk21Z88em7GePXuqcuXKGjp0qBwdHa3jM2fO1OOPPy5/f3+b+i+//FKXL1+2ft69e7d69eqlrVu3qnz58oW7AwCKFII9AAAA8A+enp6qVq2azVjx4sVVokQJm/EjR45oy5Ytud6C8s/w/scff0iSqlSpIh8fn4JvGkCRxaX4AAAAwG2aNWuWSpcurRYtWti7FQBFGGfsAQAAgDzYtGlTjrGxY8dq7Nixedq+cePGMgyjgLsCAM7YAwAAAABgagR7AAAAAABMjEvxAQAAcNf46tAZe7dQ5HSoVMreLQD4lzhjDwAAAACAiRHsAQAAAAAwMYI9AAAAAAAmRrAHAAAAAMDECPYAAAAAAJgYwR4AAAAAABMj2AMAAAAAYGIEewAAAAAATIxgDwAAAACAiRHsAQAAAAAwMYI9AAAAABRRW7ZsUdu2bRUcHCyLxaJly5bZrE9OTtZzzz2n4OBgubu7q2XLljp8+LBNzdGjR/XEE0/I399fXl5eevLJJ5WcnHwH9wIEewAAAAAooi5evKiaNWtqypQpOdYZhqH27dvr2LFjWr58uX788UeFhoaqWbNmunjxonX7Fi1ayGKxaMOGDdq2bZsyMzPVtm1bZWdn3+ndKbKc7N0AAAAAAMA+WrVqpVatWuW67vDhw9qxY4f27t2rqlWrSpKmTp2qoKAgff7553r++ee1bds2nThxQj/++KO8vLwkSXPnzpWvr682bNigZs2a3bF9Kco4Yw8AAAAAyCEjI0OS5Orqah1zcHCQi4uLvvvuO2uNxWKRi4uLtcbV1VUODg7WGhQ+gj0AAAAAIIfKlSurTJkyGj58uM6fP6/MzEz997//1alTp3TmzBlJUr169VS8eHENHTpUly5d0sWLF/XKK68oKyvLWoPCR7AHAAAAAORQrFgxffXVV/rll1/k5+cnd3d3bdy4Ua1atZKDw19R0t/fX4sXL9aKFSvk4eEhb29vpaSkqFatWtYaFD7usQcAAAAA5Kp27dpKSEhQamqqMjMz5e/vr7p166pOnTrWmhYtWujo0aP6448/5OTkJB8fHwUFBalcuXJ27Lxo4VcoAAAAAICb8vb2lr+/vw4fPqzvv/9e7dq1y1FTsmRJ+fj4aMOGDTp79qwef/xxO3RaNHHGHgAAAACKqPT0dB05csT6+fjx40pISJCfn5/KlCmjxYsXy9/fX2XKlNGePXs0YMAAtW/fXi1atLBuM3v2bFWpUkX+/v6Ki4vTgAEDFBMTo0qVKtljl4okgj0AAAAAFFHff/+9mjRpYv08aNAgSVKPHj00Z84cnTlzRoMGDVJycrJKlSql7t276/XXX7eZ49ChQxo+fLjOnTunsmXL6rXXXlNMTMwd3Y+izmIYhmHvJu52aWlp8vb2VmpqqvXdjGYy+fxke7dQ5AzwHWDvFooci8XeHRQ9/Otx5y34f+8Qxp3z9L599m6hyPnqEE/RvtM6VCpl7xYA5CI/OZR77AEAAAAAMDGCPQAAAAAAJsY99gAAAABwB+1ZNcHeLRQ51aNetncLhYoz9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMrEgF+ylTpqhs2bJydXVV3bp1tWvXLnu3BAAAAADAv1Jkgv0XX3yhQYMG6Y033tAPP/ygmjVrKjIyUmfPnrV3awAAAAAA3LYiE+zfe+899enTRz179lR4eLimTZsmd3d3zZo1y96tAQAAAABw25zs3cCdkJmZqfj4eA0fPtw65uDgoGbNmikuLi5HfUZGhjIyMqyfU1NTJUlpaWmF32whuJJ2xd4tFDlpjuY8VoD8MOlfiaZ2KSvL3i0UOWb9t9/MLqVfsHcLRU5aWnF7t1DkpF/i/8/vNDP+fX69Z8MwbllbJIL9H3/8oaysLAUGBtqMBwYG6uDBgznqx40bp9GjR+cYDwkJKbQecW8ZpmH2bgEodN7e9u4AKHx9ONAB4B4xwt4N3LYLFy7I+xb/HhWJYJ9fw4cP16BBg6yfs7Ozde7cOZUoUUIWi8WOnRUdaWlpCgkJ0cmTJ+Xl5WXvdoBCwXGOooDjHEUBxzmKAo7zO88wDF24cEHBwcG3rC0Swb5kyZJydHRUcnKyzXhycrKCgoJy1Lu4uMjFxcVmzMfHpzBbxA14eXnxFwfueRznKAo4zlEUcJyjKOA4v7Nudab+uiLx8DxnZ2fVrl1b69evt45lZ2dr/fr1ioiIsGNnAAAAAAD8O0XijL0kDRo0SD169FCdOnX08MMPa9KkSbp48aJ69uxp79YAAAAAALhtRSbYP/XUU/r99981cuRIJSUl6YEHHtCaNWtyPFAPdwcXFxe98cYbOW6JAO4lHOcoCjjOURRwnKMo4Di/u1mMvDw7HwAAAAAA3JWKxD32AAAAAADcqwj2AAAAAACYGMEeAAAAAAATI9gDAAAAAGBiBHv8K6NGjZLFYrFZKleufNNtLl26pOHDh6t8+fJydXWVv7+/GjVqpOXLl1trypYtq0mTJhVy90D+vfPOO7JYLBo4cOBN6zjOYTZTp05VjRo15OXlJS8vL0VERGj16tU33YbjHGazZcsWtW3bVsHBwbJYLFq2bNktt8nKytI777yjypUry83NTX5+fqpbt65mzJhhrWncuPEt/10A7pRx48bpoYcekqenpwICAtS+fXsdOnToptvw97n5FZnX3aHwVK1aVd9++631s5PTzQ+rF154QTt37tQHH3yg8PBw/fnnn9q+fbv+/PPPAu8tMzNTzs7OBT4viqbdu3dr+vTpqlGjxi1rOc5hNqVLl9Y777yjihUryjAMzZ07V+3atdOPP/6oqlWr5roNxznM5uLFi6pZs6Z69eqlDh065Gmb0aNHa/r06frwww9Vp04dpaWl6fvvv9f58+cLvD+OcxSEzZs3Kzo6Wg899JCuXbumV199VS1atND+/ftVvHjxXLfh7/N7gAH8C2+88YZRs2bNfG3j7e1tzJkz54brGzVqZEiyWQzDMP744w+jS5cuRnBwsOHm5mZUq1bNWLBgQY5to6OjjQEDBhglSpQwGjdubGRnZxtvvPGGERISYjg7OxulSpUyXnrppXzvK4q2CxcuGBUrVjRiY2ONRo0aGQMGDLhpPcc57gW+vr7GjBkzbrie4xxmJslYunTpLetq1qxpjBo16obre/TokeM4P378uHHt2jWjV69eRtmyZQ1XV1fj/vvvNyZNmpRj23bt2hlvvfWWUapUKaNs2bKGYRjGlClTjAoVKhguLi5GQECA0bFjx3+1ryjazp49a0gyNm/efMMa/j43Py7Fx792+PBhBQcHq1y5curWrZsSExNvWh8UFKRvvvlGFy5cyHX9V199pdKlS2vMmDE6c+aMzpw5I0m6cuWKateurVWrVmnv3r3q27evnn32We3atctm+7lz58rZ2Vnbtm3TtGnT9OWXX2rixImaPn26Dh8+rGXLlql69eoFs/MoMqKjoxUVFaVmzZrlqZ7jHGaWlZWlhQsX6uLFi4qIiLhhHcc5ioKgoCBt2LBBv//+e67rJ0+erIiICPXp08d6nIeEhCg7O1ulS5fW4sWLtX//fo0cOVKvvvqqFi1aZLP9+vXrdejQIcXGxmrlypX6/vvv1b9/f40ZM0aHDh3SmjVr1LBhwzuxq7hHpaamSpL8/PxuWMPf5/cAe/9mAeb2zTffGIsWLTJ++uknY82aNUZERIRRpkwZIy0t7YbbbN682ShdurRRrFgxo06dOsbAgQON7777zqYmNDTUmDhx4i2/Pyoqynj55Zetnxs1amQ8+OCDNjUTJkww7r//fiMzMzN/Owf8P59//rlRrVo14/Lly4ZhGHk6Y89xDjP6+eefjeLFixuOjo6Gt7e3sWrVqpvWc5zDzJTHM/b79u0zqlSpYjg4OBjVq1c3/u///s/45ptvbGry8u+CYRhGdHS0zdn3Hj16GIGBgUZGRoZ17MsvvzS8vLxu+v9SQF5lZWUZUVFRRv369W9ax9/n5scZe/wrrVq1UufOnVWjRg1FRkbqm2++UUpKSo7fRv9dw4YNdezYMa1fv16dOnXSvn379Oijj+rNN9+86XdlZWXpzTffVPXq1eXn5ycPDw+tXbs2xxUCtWvXtvncuXNnXb58WeXKlVOfPn20dOlSXbt27fZ3GkXKyZMnNWDAAM2fP1+urq553o7jHGZUqVIlJSQkaOfOnXrxxRfVo0cP7d+//4b1HOcoCsLDw7V3717t2LFDvXr10tmzZ9W2bVs9//zzt9x2ypQpql27tvz9/eXh4aGPP/44x3FevXp1m/uNmzdvrtDQUJUrV07PPvus5s+fr0uXLhX4fqFoiI6O1t69e7Vw4cKb1vH3+T3A3r9ZwL2nTp06xrBhw/K1zZtvvmkUK1bM+hvr3H4jOG7cOKNEiRLGp59+aiQkJBiHDx82oqKijHbt2llrbvQb80uXLhlff/218dJLLxlBQUFGREQEvyFEnixdutSQZDg6OloXSYbFYjEcHR2Na9eu5XkujnOYTdOmTY2+ffvmaxuOc5iF8njGPjeffvqpIck4duyYYRi5H6+ff/654erqakyZMsX44YcfjMOHDxt9+/a1eTbR9Xvs/+nq1atGbGysMXjwYKNcuXJGhQoVjPPnz99Wryi6oqOjjdKlS1uP0/zi73Nz4Yw9ClR6erqOHj2qUqVK5Wu78PBwXbt2TVeuXJEkOTs7Kysry6Zm27ZtateunZ555hnVrFlT5cqV0y+//JKn+d3c3NS2bVu9//772rRpk+Li4rRnz5589YiiqWnTptqzZ48SEhKsS506ddStWzclJCTI0dExz3NxnMNssrOzlZGRka9tOM5RFISHh0v66yn70o2P80ceeUT/+c9/9OCDD6pChQo6evRonuZ3cnJSs2bNNH78eP388886ceKENmzYULA7gXuWYRjq16+fli5dqg0bNigsLOy25uHvc3PhdXf4V1555RW1bdtWoaGhOn36tN544w05Ojqqa9euN9ymcePG6tq1q+rUqaMSJUpo//79evXVV9WkSRN5eXlJ+us9mVu2bFGXLl3k4uKikiVLqmLFilqyZIm2b98uX19fvffee0pOTrb+43ojc+bMUVZWlurWrSt3d3d99tlncnNzU2hoaIH+LHBv8vT0VLVq1WzGihcvrhIlSuQY/zuOc5jN8OHD1apVK5UpU0YXLlzQggULtGnTJq1du/aG23Ccw2zS09N15MgR6+fjx48rISFBfn5+KlOmTK7bdOrUSfXr19cjjzyioKAgHT9+XMOHD9f999+vypUrS/rrON+5c6dOnDghDw8P+fn5qWLFipo3b57Wrl2rsLAwffrpp9q9e/ctQ9bKlSt17NgxNWzYUL6+vvrmm2+UnZ2tSpUqFdwPAve06OhoLViwQMuXL5enp6eSkpIkSd7e3nJzc8t1G/4+vwfY+5IBmNtTTz1llCpVynB2djbuu+8+46mnnjKOHDly023Gjh1rREREGH5+foarq6tRrlw5o3///sYff/xhrYmLizNq1KhhuLi4WF+n8eeffxrt2rUzPDw8jICAAGPEiBFG9+7db3mpz9KlS426desaXl5eRvHixY169eoZ3377bYH9DFD05OUhSRznMJtevXoZoaGhhrOzs+Hv7280bdrUWLdu3U234TiH2WzcuDHHK7skGT169LjhNh9//LHRpEkTw9/f33B2djbKlCljPPfcc8aJEyesNYcOHTLq1atnuLm5WV93d+XKFeO5554zvL29DR8fH+PFF180hg0bdstL8bdu3Wo0atTI8PX1Ndzc3IwaNWoYX3zxRQH/JHAvy+0Yl2TMnj37htvw97n5WQzDMO78rxMAAAAAAEBB4B57AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AACQg8Vi0bJly+zdBgAAyAOCPQAAJvLcc8/JYrHIYrGoWLFiCgsL05AhQ3TlypUC/Z4zZ86oVatWBTrnzRw/flxPP/20goOD5erqqtKlS6tdu3Y6ePDgHesBAACzcrJ3AwAAIH9atmyp2bNn6+rVq4qPj1ePHj1ksVj03//+t8C+IygoqMDmupWrV6+qefPmqlSpkr766iuVKlVKp06d0urVq5WSklKo31usWLFCmx8AgDuFM/YAAJiMi4uLgoKCFBISovbt26tZs2aKjY21rs/Ozta4ceMUFhYmNzc31axZU0uWLLGuK126tKZOnWoz548//igHBwf9+uuvknJein/y5Ek9+eST8vHxkZ+fn9q1a6cTJ05Ikvbu3SsHBwf9/vvvkqRz587JwcFBXbp0sW7/1ltvqUGDBrnuz759+3T06FF99NFHqlevnkJDQ1W/fn299dZbqlevnrXu1KlT6tq1q/z8/FS8eHHVqVNHO3futK6fOnWqypcvL2dnZ1WqVEmffvqpzfdYLBZNnTpVjz/+uIoXL663335bkrR8+XLVqlVLrq6uKleunEaPHq1r167l6b8FAAB3A4I9AAAmtnfvXm3fvl3Ozs7WsXHjxmnevHmaNm2a9u3bp5iYGD3zzDPavHmzHBwc1LVrVy1YsMBmnvnz56t+/foKDQ3N8R1Xr15VZGSkPD09tXXrVm3btk0eHh5q2bKlMjMzVbVqVZUoUUKbN2+WJG3dutXmsyRt3rxZjRs3znUf/P395eDgoCVLligrKyvXmvT0dDVq1Ei//fabvv76a/30008aMmSIsrOzJUlLly7VgAED9PLLL2vv3r36v//7P/Xs2VMbN260mWfUqFF64okntGfPHvXq1Utbt25V9+7dNWDAAO3fv1/Tp0/XnDlzrKEfAABTMAAAgGn06NHDcHR0NIoXL264uLgYkgwHBwdjyZIlhmEYxpUrVwx3d3dj+/btNtv17t3b6Nq1q2EYhvHjjz8aFovF+PXXXw3DMIysrCzjvvvuM6ZOnWqtl2QsXbrUMAzD+PTTT41KlSoZ2dnZ1vUZGRmGm5ubsXbtWsMwDKNDhw5GdHS0YRiGMXDgQGPw4MGGr6+vceDAASMzM9Nwd3c31q1bd8P9+vDDDw13d3fD09PTaNKkiTFmzBjj6NGj1vXTp083PD09jT///DPX7R955BGjT58+NmOdO3c2WrdubbNPAwcOtKlp2rSpMXbsWJuxTz/91ChVqtQNewUA4G7DGXsAAEymSZMmSkhI0M6dO9WjRw/17NlTHTt2lCQdOXJEly5dUvPmzeXh4WFd5s2bp6NHj0qSHnjgAVWpUsV61n7z5s06e/asOnfunOv3/fTTTzpy5Ig8PT2t8/n5+enKlSvWORs1aqRNmzZZ53vsscfUsGFDbdq0Sbt379bVq1dVv379G+5TdHS0kpKSNH/+fEVERGjx4sWqWrWq9RaDhIQEPfjgg/Lz88t1+wMHDuSYv379+jpw4IDNWJ06dXLs25gxY2x+Vn369NGZM2d06dKlG/YLAMDdhIfnAQBgMsWLF1eFChUkSbNmzVLNmjU1c+ZM9e7dW+np6ZKkVatW6b777rPZzsXFxfrnbt26acGCBRo2bJgWLFigli1bqkSJErl+X3p6umrXrq358+fnWOfv7y9Jaty4sQYOHKjDhw9r//79atCggQ4ePKhNmzbp/PnzqlOnjtzd3W+6X56enmrbtq3atm2rt956S5GRkXrrrbfUvHlzubm55f0HdBPFixfPsW+jR49Whw4dctS6uroWyHcCAFDYOGMPAICJOTg46NVXX9WIESN0+fJlhYeHy8XFRYmJiapQoYLNEhISYt3u6aef1t69exUfH68lS5aoW7duN/yOWrVq6fDhwwoICMgxp7e3tySpevXq8vX11VtvvaUHHnhAHh4eaty4sTZv3qxNmzbd8P76G7FYLKpcubIuXrwoSapRo4YSEhJ07ty5XOurVKmibdu22Yxt27ZN4eHhN/2eWrVq6dChQzn2q0KFCnJw4H+TAADmwL9YAACYXOfOneXo6KgpU6bI09NTr7zyimJiYjR37lwdPXpUP/zwgz744APNnTvXuk3ZsmX1yCOPqHfv3srKytLjjz9+w/m7deumkiVLql27dtq6dauOHz+uTZs2qX///jp16pSkv4J4w4YNNX/+fGuIr1GjhjIyMrR+/Xo1atTohvMnJCSoXbt2WrJkifbv368jR45o5syZmjVrltq1aydJ6tq1q4KCgtS+fXtt27ZNx44d05dffqm4uDhJ0uDBgzVnzhxNnTpVhw8f1nvvvaevvvpKr7zyyk1/diNHjtS8efM0evRo7du3TwcOHNDChQs1YsSIPP3sAQC4GxDsAQAwOScnJ/Xr10/jx4/XxYsX9eabb+r111/XuHHjVKVKFbVs2VKrVq1SWFiYzXbdunXTTz/9pCeeeOKml7q7u7try5YtKlOmjDp06KAqVaqod+/eunLliry8vKx1jRo1UlZWljXYOzg4qGHDhrJYLDe9v7506dIqW7asRo8erbp166pWrVqaPHmyRo8erddee02S5OzsrHXr1ikgIECtW7dW9erV9c4778jR0VGS1L59e02ePFn/+9//VLVqVU2fPl2zZ8++5ZUCkZGRWrlypdatW6eHHnpI9erV08SJE3N9OwAAAHcri2EYhr2bAAAAAAAAt4cz9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYgR7AAAAAABMjGAPAAAAAICJEewBAAAAADAxgj0AAAAAACZGsAcAAAAAwMQI9gAAAAAAmBjBHgAAAAAAEyPYAwAAAABgYv8faNwGWfRqvPcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1200x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "label_distribution = data['label']\n",
    "\n",
    "barplot_visual(label_distribution,\n",
    "               \"Distribution of Scores\",\n",
    "               \"Review Score\",\n",
    "               \"# of Samples\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c6f699e6",
   "metadata": {},
   "source": [
    "#### Visualization: Distribution of Review Length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a43c1696",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAHHCAYAAABZbpmkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA8qUlEQVR4nO3deVxWZf7/8fcNCiIIuLIkIpLlnqVJaOZGYtliUmnDFJmpGVRu2Tgzbk2lUmPlTGk2JdpkllNNk02Wo+CKuJe5UrmVohYCigoC1+8Pv55ft7gAAjccX8/H437kfZ3rPudz7gPc765znfs4jDFGAAAANuXm6gIAAAAqEmEHAADYGmEHAADYGmEHAADYGmEHAADYGmEHAADYGmEHAADYGmEHAADYGmEHAADYGmEHQIXo3r272rRpUynbevnll9WsWTO5u7urffv2lbJNANUHYQeowpKSkuRwOLRhwwZXl3JBBw8e1KRJk7RlyxaX1fD1119r7Nix6tKli+bMmaOXXnrpon0fffRR+fj4VGJ1l7Z9+3ZNmjRJe/fuvWS/vXv3yuFwlOhxuXWVd21AdVDD1QUAqL4OHjyoyZMnq2nTpi4bUVm2bJnc3Nz0zjvvyMPDwyU1lNX27ds1efJkde/eXU2bNr1ov4YNG+q9995zavvrX/+qn376Sa+++mqxvpVZG1AdEHYAVGtHjhyRl5dXtQs6peHt7a3f//73Tm0LFizQsWPHirUDKI7TWIAN/Pzzz3rssccUEBAgT09PtW7dWu+++65Tn5SUFDkcDn300Ud68cUX1bhxY9WqVUu9evXS999/X2ydb7zxhpo1ayYvLy916tRJK1euVPfu3dW9e3drfTfffLMkadCgQdZplKSkJKf1bN++XT169FDt2rV1zTXXKDExsUT7VFBQoL/85S8KDw+Xp6enmjZtqj/+8Y/Ky8uz+jgcDs2ZM0e5ubkX3f7lNG3aVHfddZdWrVqlTp06qVatWmrWrJnmzZvn1O/cKcUVK1Zo2LBhql+/vnx9ffXII4/o2LFjTn0dDocmTZp0wW09+uij1voeeOABSVKPHj2s+lNSUkpV/2/l5eVp4sSJuvbaa+Xp6amQkBCNHTvW6T2Li4tTrVq1tGPHDqfXRkdHq27dujp48GCF1Aa4EiM7QDV3+PBh3XLLLXI4HEpISFDDhg315ZdfavDgwcrJydGIESOc+k+dOlVubm4aM2aMsrOzlZiYqNjYWKWlpVl9Zs6cqYSEBHXt2lUjR47U3r171a9fP9WtW1eNGzeWJLVs2VLPP/+8JkyYoKFDh6pr166SpM6dO1vrOXbsmPr06aP+/fvrwQcf1L/+9S8999xzatu2re64445L7tfjjz+uuXPn6v7779fo0aOVlpamKVOmaMeOHfr0008lSe+9955mz56tdevW6R//+Eex7ZfU999/r/vvv1+DBw9WXFyc3n33XT366KPq0KGDWrdu7dQ3ISFB/v7+mjRpknbt2qWZM2dq3759Vpgsqdtuu01PP/20ZsyYoT/+8Y9q2bKlJFn/La2ioiLdc889WrVqlYYOHaqWLVtq69atevXVV7V79279+9//liS9/vrrWrZsmeLi4pSamip3d3e99dZb+vrrr/Xee+8pODi43GsDXM4AqLLmzJljJJn169dftM/gwYNNUFCQ+eWXX5zaBw4caPz8/MzJkyeNMcYkJycbSaZly5YmLy/P6vf6668bSWbr1q3GGGPy8vJM/fr1zc0332zOnDlj9UtKSjKSTLdu3ay29evXG0lmzpw5xerq1q2bkWTmzZtnteXl5ZnAwEATExNzyf3esmWLkWQef/xxp/YxY8YYSWbZsmVWW1xcnPH29r7k+i7VNzQ01EgyK1assNqOHDliPD09zejRo622c8eiQ4cOJj8/32pPTEw0ksxnn31mtUkyEydOLLb90NBQExcXZz1fuHChkWSSk5NLVP9v9e3b14SGhlrP33vvPePm5mZWrlzp1G/WrFlGklm9erXV9tVXXxlJ5oUXXjA//vij8fHxMf369XN63ZXUBlQ1nMYCqjFjjD7++GPdfffdMsbol19+sR7R0dHKzs7Wpk2bnF4zaNAgp/kt50ZkfvzxR0nShg0b9Ouvv2rIkCGqUeP/D/7Gxsaqbt26parPx8fHaU6Jh4eHOnXqZG3rYv773/9KkkaNGuXUPnr0aEnSF198Uao6LqdVq1bW+yCdneR7/fXXX7DOoUOHqmbNmtbz4cOHq0aNGlbNrrJw4UK1bNlSLVq0cPo56NmzpyQpOTnZ6tu7d28NGzZMzz//vPr3769atWrprbfeclXpQIXjNBZQjR09elRZWVmaPXu2Zs+efcE+R44ccXrepEkTp+fnAsy5eSf79u2TJF177bVO/WrUqFHqq3IaN25c7NRO3bp19e23317ydfv27ZObm1uxGgIDA+Xv72/VWF7Of0/O1Xn+XBxJat68udNzHx8fBQUFufwS7fT0dO3YseOiV2Od/3Pwyiuv6LPPPtOWLVs0f/58NWrUqDLKBFyCsANUY0VFRZKk3//+94qLi7tgn3bt2jk9d3d3v2A/Y0z5FlcO2yrNHJgrUVnvSWFhYbmu77eKiorUtm1bTZ8+/YLLQ0JCnJ5v3rzZCkBbt27VQw89VGG1Aa5G2AGqsYYNG6pOnToqLCxUVFRUuawzNDRU0tlJuz169LDaCwoKtHfvXqfwVFFhJDQ0VEVFRUpPT3eaFHv48GFlZWVZNbpCenq60/ty4sQJHTp0SHfeeafVVrduXWVlZTm9Lj8/X4cOHXJqK8/3Lzw8XN9884169ep12fXm5uZq0KBBatWqlTp37qzExETdd9991tV15V0b4GrM2QGqMXd3d8XExOjjjz/Wd999V2z50aNHS73Ojh07qn79+nr77bdVUFBgtb///vvFTut4e3tLUrEP9it1Lji89tprTu3nRi369u1brtsrjdmzZ+vMmTPW85kzZ6qgoMDp6rLw8HCtWLGi2OvOH9kpz/fvwQcf1M8//6y333672LJTp04pNzfXev7cc89p//79mjt3rqZPn66mTZsqLi7O6RL1ijq2gCswsgNUA++++64WL15crP2ZZ57R1KlTlZycrIiICA0ZMkStWrVSZmamNm3apP/973/KzMws1bY8PDw0adIkPfXUU+rZs6cefPBB7d27V0lJSQoPD3f6P/7w8HD5+/tr1qxZqlOnjry9vRUREaGwsLAr2t8bbrhBcXFxmj17trKystStWzetW7dOc+fOVb9+/ZxGVipbfn6+evXqpQcffFC7du3Sm2++qVtvvVX33HOP1efxxx/XE088oZiYGN1+++365ptv9NVXX6lBgwZO62rfvr3c3d01bdo0ZWdny9PTUz179izT/JmHH35YH330kZ544gklJyerS5cuKiws1M6dO/XRRx/pq6++UseOHbVs2TK9+eabmjhxom666SZJ0pw5c9S9e3eNHz/e+h6k8qwNcDmXXgsG4JLOXe58sceBAweMMcYcPnzYxMfHm5CQEFOzZk0TGBhoevXqZWbPnm2t69yl5wsXLnTaxp49ey54+fiMGTNMaGio8fT0NJ06dTKrV682HTp0MH369HHq99lnn5lWrVqZGjVqOK2nW7dupnXr1sX2KS4uzumS6Ys5c+aMmTx5sgkLCzM1a9Y0ISEhZty4ceb06dPF1nell5737du3WN9u3bo5XWZ/7lgsX77cDB061NStW9f4+PiY2NhY8+uvvzq9trCw0Dz33HOmQYMGpnbt2iY6Otp8//33xS49N8aYt99+2zRr1sy4u7uX6lLv8y89N8aY/Px8M23aNNO6dWvj6elp6tatazp06GAmT55ssrOzTU5OjgkNDTU33XST09cKGGPMyJEjjZubm0lNTb3i2oCqxmFMBcxKBGA7RUVFatiwofr373/BUyV2l5SUpEGDBmn9+vXq2LGjq8sBUArM2QFQzOnTp4tdiTRv3jxlZmZat4sAgOqCOTsAilm7dq1GjhypBx54QPXr19emTZv0zjvvqE2bNtY9kwCguiDsACimadOmCgkJ0YwZM5SZmal69erpkUce0dSpU219d3EA9sScHQAAYGvM2QEAALZG2AEAALbGnB2dvaT24MGDqlOnDl+RDgBANWGM0fHjxxUcHCw3t4uP3xB2JB08eLDYTfIAAED1cODAATVu3Piiywk7kurUqSPp7Jvl6+vr4moAAEBJ5OTkKCQkxPocvxjCjv7/3X19fX0JOwAAVDOXm4LCBGUAAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrNVxdwNXO4XB+boxr6gAAwK4Y2QEAALZG2AEAALZG2AEAALbGnJ1Kdv4cHQAAULEY2QEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALZG2AEAALbm0rBTWFio8ePHKywsTF5eXgoPD9df/vIXGWOsPsYYTZgwQUFBQfLy8lJUVJTS09Od1pOZmanY2Fj5+vrK399fgwcP1okTJyp7dwAAQBXk0rAzbdo0zZw5U3//+9+1Y8cOTZs2TYmJifrb3/5m9UlMTNSMGTM0a9YspaWlydvbW9HR0Tp9+rTVJzY2Vtu2bdOSJUu0aNEirVixQkOHDnXFLgEAgCrGYX47jFLJ7rrrLgUEBOidd96x2mJiYuTl5aV//vOfMsYoODhYo0eP1pgxYyRJ2dnZCggIUFJSkgYOHKgdO3aoVatWWr9+vTp27ChJWrx4se6880799NNPCg4OvmwdOTk58vPzU3Z2tnx9fStmZ/+Pw3Hp5a47GgAAVC8l/fx26chO586dtXTpUu3evVuS9M0332jVqlW64447JEl79uxRRkaGoqKirNf4+fkpIiJCqampkqTU1FT5+/tbQUeSoqKi5ObmprS0tAtuNy8vTzk5OU4PAABgTzVcufE//OEPysnJUYsWLeTu7q7CwkK9+OKLio2NlSRlZGRIkgICApxeFxAQYC3LyMhQo0aNnJbXqFFD9erVs/qcb8qUKZo8eXJ57w4AAKiCXDqy89FHH+n999/X/PnztWnTJs2dO1evvPKK5s6dW6HbHTdunLKzs63HgQMHKnR7AADAdVw6svPss8/qD3/4gwYOHChJatu2rfbt26cpU6YoLi5OgYGBkqTDhw8rKCjIet3hw4fVvn17SVJgYKCOHDnitN6CggJlZmZarz+fp6enPD09K2CPAABAVePSkZ2TJ0/Kzc25BHd3dxUVFUmSwsLCFBgYqKVLl1rLc3JylJaWpsjISElSZGSksrKytHHjRqvPsmXLVFRUpIiIiErYCwAAUJW5dGTn7rvv1osvvqgmTZqodevW2rx5s6ZPn67HHntMkuRwODRixAi98MILat68ucLCwjR+/HgFBwerX79+kqSWLVuqT58+GjJkiGbNmqUzZ84oISFBAwcOLNGVWAAAwN5cGnb+9re/afz48XryySd15MgRBQcHa9iwYZowYYLVZ+zYscrNzdXQoUOVlZWlW2+9VYsXL1atWrWsPu+//74SEhLUq1cvubm5KSYmRjNmzHDFLgEAgCrGpd+zU1XwPTsAAFQ/1eJ7dgAAACoaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANgaYQcAANhaDVcXAGcOh/NzY1xTBwAAdsHIDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsLUari4Al+ZwOD83xjV1AABQXTGyAwAAbI2wAwAAbI2wAwAAbI2wAwAAbM3lYefnn3/W73//e9WvX19eXl5q27atNmzYYC03xmjChAkKCgqSl5eXoqKilJ6e7rSOzMxMxcbGytfXV/7+/ho8eLBOnDhR2bsCAACqIJeGnWPHjqlLly6qWbOmvvzyS23fvl1//etfVbduXatPYmKiZsyYoVmzZiktLU3e3t6Kjo7W6dOnrT6xsbHatm2blixZokWLFmnFihUaOnSoK3YJAABUMQ5jXHcx8x/+8AetXr1aK1euvOByY4yCg4M1evRojRkzRpKUnZ2tgIAAJSUlaeDAgdqxY4datWql9evXq2PHjpKkxYsX684779RPP/2k4ODgy9aRk5MjPz8/ZWdny9fXt/x28ALOv5S8tLj0HACAs0r6+V2mkZ39+/frQhnJGKP9+/eXeD3/+c9/1LFjRz3wwANq1KiRbrzxRr399tvW8j179igjI0NRUVFWm5+fnyIiIpSamipJSk1Nlb+/vxV0JCkqKkpubm5KS0u74Hbz8vKUk5Pj9AAAAPZUprATFhamo0ePFmvPzMxUWFhYidfz448/aubMmWrevLm++uorDR8+XE8//bTmzp0rScrIyJAkBQQEOL0uICDAWpaRkaFGjRo5La9Ro4bq1atn9TnflClT5OfnZz1CQkJKXDMAAKheyhR2jDFyXOB8zIkTJ1SrVq0Sr6eoqEg33XSTXnrpJd14440aOnSohgwZolmzZpWlrBIbN26csrOzrceBAwcqdHsAAMB1SnW7iFGjRkmSHA6Hxo8fr9q1a1vLCgsLlZaWpvbt25d4fUFBQWrVqpVTW8uWLfXxxx9LkgIDAyVJhw8fVlBQkNXn8OHD1nYCAwN15MgRp3UUFBQoMzPTev35PD095enpWeI6AQBA9VWqsLN582ZJZ0d2tm7dKg8PD2uZh4eHbrjhBmsicUl06dJFu3btcmrbvXu3QkNDJZ09XRYYGKilS5da4SYnJ0dpaWkaPny4JCkyMlJZWVnauHGjOnToIElatmyZioqKFBERUZrdAwAANlSqsJOcnCxJGjRokF5//fUrvnJp5MiR6ty5s1566SU9+OCDWrdunWbPnq3Zs2dLOjuCNGLECL3wwgtq3ry5wsLCNH78eAUHB6tfv36Szo4E9enTxzr9debMGSUkJGjgwIEluhILAADYW5kuPT969KgaNmx4wWVbt25V27ZtS7yuRYsWady4cUpPT1dYWJhGjRqlIUOGWMuNMZo4caJmz56trKws3XrrrXrzzTd13XXXWX0yMzOVkJCgzz//XG5uboqJidGMGTPk4+NTohq49BwAgOqnpJ/fZQo7gYGBeuedd9S3b1+n9ldeeUXjx4/XqVOnSl+xCxF2AACofir0e3ZGjRqlmJgYDR8+XKdOndLPP/+sXr16KTExUfPnzy9z0QAAAOWtTGFn7NixSk1N1cqVK9WuXTu1a9dOnp6e+vbbb3XfffeVd40AAABlVuZ7Y1177bVq06aN9u7dq5ycHA0YMOCil3oDAAC4SpnCzurVq9WuXTulp6fr22+/1cyZM/XUU09pwIABOnbsWHnXCAAAUGZlCjs9e/bUgAEDtHbtWrVs2VKPP/64Nm/erP3795fqSiwAAICKVqrv2Tnn66+/Vrdu3ZzawsPDtXr1ar344ovlUhgAAEB5KNOl5+d8//33+uGHH3TbbbfJy8vrovfMquq49BwAgOqnQi89//XXX9WrVy9dd911uvPOO3Xo0CFJ0uDBg0t1uwgAAICKVqawM3LkSNWsWVP79+93uhnogAED9OWXX5ZbcQAAAFeqzHN2vvrqKzVu3NipvXnz5tq3b1+5FAYAAFAeyjSyk5ub6zSic05mZqY8PT2vuCgAAIDyUqaw07VrV82bN8967nA4VFRUpMTERPXo0aPcigMAALhSpTqN1axZM61fv16JiYnq1auXNmzYoPz8fI0dO1bbtm1TZmamVq9eXVG1AgAAlFqpRnb27t2rwsJCtWnTRrt379att96qe++9V7m5uerfv782b96s8PDwiqoVAACg1Mo0QVmS/Pz89Kc//ak8awEAACh3pQ47X331lfz8/C7Z55577ilzQQAAAOWp1GEnLi7ukssdDocKCwvLXBAAAEB5KvXVWBkZGSoqKrrog6ADAACqklKFnep43ysAAHB1K1XYuYJ7hgIAALhEqcJOXFycvLy8KqoWAACAcleqCcpz5sypqDoAAAAqRJluFwEAAFBdEHYAAICtEXYAAICtlSrsNGnSRAkJCfr6669VUFBQUTUBAACUm1KFnffee0+enp6Kj49XgwYNNGDAAL3//vvKysqqoPIAAACujMOU8ctztm3bpv/85z/67LPPtGXLFnXu3Fn33HOP7rnnHjVr1qy866xQOTk58vPzU3Z2tnx9fSt0W1f6vYx81REAAGeV9PO7zHN2WrdurXHjxmnt2rXau3evHnroIS1dulRt2rRRmzZt9MUXX5R11QAAAOWmzCM7F3Py5El99dVXqlOnjqKiospz1RWGkR0AAKqfkn5+l/qu55dTu3Zt3XfffeW9WgAAgDLh0nMAAGBrhB0AAGBrhB0AAGBr5RJ2cnJy9O9//1s7duwoj9XhEhwO5wcAALi0MoWdBx98UH//+98lSadOnVLHjh314IMPql27dvr444/LtUAAAIArUaaws2LFCnXt2lWS9Omnn8oYo6ysLM2YMUMvvPBCuRYIAABwJcoUdrKzs1WvXj1J0uLFixUTE6PatWurb9++Sk9PL9cCAQAArkSZwk5ISIhSU1OVm5urxYsXq3fv3pKkY8eOqVatWuVaIAAAwJUo05cKjhgxQrGxsfLx8VFoaKi6d+8u6ezprbZt25ZnfQAAAFekTGHnySefVKdOnXTgwAHdfvvtcnM7O0DUrFkz5uwAAIAqpdzvjVUdVad7Y52PowcAuFqV+72xRo0aVeKNT58+vcR9AQAAKlKJw87mzZudnm/atEkFBQW6/vrrJUm7d++Wu7u7OnToUL4VAgAAXIESh53k5GTr39OnT1edOnU0d+5c1a1bV9LZK7EGDRpkff8OAABAVVCmOTvXXHONvv76a7Vu3dqp/bvvvlPv3r118ODBciuwMjBnBwCA6qekn99l+p6dnJwcHT16tFj70aNHdfz48bKsEgAAoEKUKezcd999GjRokD755BP99NNP+umnn/Txxx9r8ODB6t+/f3nXCAAAUGZl+p6dWbNmacyYMfrd736nM2fOnF1RjRoaPHiwXn755XItEJd2/mkxTmsBAOCs1HN2CgsLtXr1arVt21YeHh764YcfJEnh4eHy9vaukCIrWnWes3M+wg4A4GpR7t+zc467u7t69+6tHTt2KCwsTO3atbuiQgEAACpSmebstGnTRj/++GN51wIAAFDuyhR2XnjhBY0ZM0aLFi3SoUOHlJOT4/QAAACoKsr0PTvnbvwpSY7fTEIxxsjhcKiwsLB8qqskzNkBAKD6qbA5O5LztykDAABUZWUKO926dSvvOgAAACpEmcKOJGVlZemdd97Rjh07JEmtW7fWY489Jj8/v3IrDgAA4EqVaYLyhg0bFB4erldffVWZmZnKzMzU9OnTFR4erk2bNpV3jQAAAGVWpgnKXbt21bXXXqu3335bNWqcHRwqKCjQ448/rh9//FErVqwo90IrEhOUAQCofkr6+V2msOPl5aXNmzerRYsWTu3bt29Xx44ddfLkydJX7EKEHQAAqp8Kveu5r6+v9u/fX6z9wIEDqlOnTllWCQAAUCHKFHYGDBigwYMH68MPP9SBAwd04MABLViwQI8//rgeeuih8q4RAACgzMp0NdYrr7wih8OhRx55RAUFBZKkmjVravjw4Zo6dWq5FggAAHAlSjVnZ8+ePQoLC7Oenzx50umu57Vr1y7/CisBc3YAAKh+KmTOTnh4uMLCwvTYY4/pn//8p44dO6a2bduqbdu2Vxx0pk6dKofDoREjRlhtp0+fVnx8vOrXry8fHx/FxMTo8OHDTq/bv3+/+vbtq9q1a6tRo0Z69tlnrdEmAACAUoWdZcuWKS4uTj/++KOGDBmiJk2aqHnz5ho2bJgWLFhQLIiU1Pr16/XWW2+pXbt2Tu0jR47U559/roULF2r58uU6ePCg+vfvby0vLCxU3759lZ+frzVr1mju3LlKSkrShAkTylQHAACwnzJdei6dHXVZs2aNUlJSlJKSonXr1unMmTNq0aKFtm3bVuL1nDhxQjfddJPefPNNvfDCC2rfvr1ee+01ZWdnq2HDhpo/f77uv/9+SdLOnTvVsmVLpaam6pZbbtGXX36pu+66SwcPHlRAQIAkadasWXruued09OhReXh4lKgGTmMBAFD9VOil55JUq1Yt9ezZU3/+8581efJkPf300/Lx8dHOnTtLtZ74+Hj17dtXUVFRTu0bN27UmTNnnNpbtGihJk2aKDU1VZKUmpqqtm3bWkFHkqKjo5WTk3PJwJWXl6ecnBynBwAAsKdSX42Vn5+vtWvXKjk5WSkpKUpLS1NISIhuu+02/f3vfy/VTUIXLFigTZs2af369cWWZWRkyMPDQ/7+/k7tAQEBysjIsPr8NuicW35u2cVMmTJFkydPLnGdAACg+ipV2OnZs6fS0tIUFhambt26adiwYZo/f76CgoJKveEDBw7omWee0ZIlS1SrVq1Sv/5KjBs3TqNGjbKe5+TkKCQkpFJrAAAAlaNUp7FWrlyp+vXrq2fPnurVq5duv/32MgUd6expqiNHjuimm25SjRo1VKNGDS1fvlwzZsxQjRo1FBAQoPz8fGVlZTm97vDhwwoMDJQkBQYGFpsUfe75uT4X4unpKV9fX6cHAACwp1KFnaysLM2ePVu1a9fWtGnTFBwcrLZt2yohIUH/+te/dPTo0RKvq1evXtq6dau2bNliPTp27KjY2Fjr3zVr1tTSpUut1+zatUv79+9XZGSkJCkyMlJbt27VkSNHrD5LliyRr6+vWrVqVZpdsw2Hw/kBAMDVrsxXY0nS8ePHtWrVKmv+zjfffKPmzZvru+++K9P6unfvbl2NJUnDhw/Xf//7XyUlJcnX11dPPfWUJGnNmjWSzl563r59ewUHBysxMVEZGRl6+OGH9fjjj+ull14q8XbtdDXW+bg6CwBgVyX9/C7T7SLO8fb2Vr169VSvXj3VrVtXNWrU0I4dO65klU5effVVubm5KSYmRnl5eYqOjtabb75pLXd3d9eiRYs0fPhwRUZGytvbW3FxcXr++efLrQYAAFC9lWpkp6ioSBs2bFBKSoqSk5O1evVq5ebm6pprrlGPHj2sR2hoaEXWXO4Y2QEAoPqpkJEdf39/5ebmKjAwUD169NCrr76q7t27Kzw8/IoLBgAAqAilCjsvv/yyevTooeuuu66i6gEAAChXpQo7w4YNq6g6AAAAKkSZbxcBAABQHRB2AACArRF2AACArRF2AACArRF2AACArV3RNyjj8rg/FQAArsXIDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDVuBGpz59+I1BjX1AEAgKswsgMAAGyNsAMAAGyNsAMAAGyNsAMAAGyNsAMAAGyNsAMAAGyNsAMAAGyNsAMAAGyNsAMAAGyNsAMAAGyNsAMAAGyNsAMAAGyNsAMAAGyNu55fZbgLOgDgasPIDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDXCDgAAsDW+QRlO+IZlAIDdMLIDAABsjZGdq9z5IzkAANgNIzsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWXBp2pkyZoptvvll16tRRo0aN1K9fP+3atcupz+nTpxUfH6/69evLx8dHMTExOnz4sFOf/fv3q2/fvqpdu7YaNWqkZ599VgUFBZW5KwAAoIpyadhZvny54uPjtXbtWi1ZskRnzpxR7969lZuba/UZOXKkPv/8cy1cuFDLly/XwYMH1b9/f2t5YWGh+vbtq/z8fK1Zs0Zz585VUlKSJkyY4IpdAgAAVYzDGGNcXcQ5R48eVaNGjbR8+XLddtttys7OVsOGDTV//nzdf//9kqSdO3eqZcuWSk1N1S233KIvv/xSd911lw4ePKiAgABJ0qxZs/Tcc8/p6NGj8vDwuOx2c3Jy5Ofnp+zsbPn6+pbrPjkc5bq6Sld1fjoAAHBW0s/vKjVnJzs7W5JUr149SdLGjRt15swZRUVFWX1atGihJk2aKDU1VZKUmpqqtm3bWkFHkqKjo5WTk6Nt27ZVYvVXB4fD+QEAQFVXw9UFnFNUVKQRI0aoS5cuatOmjSQpIyNDHh4e8vf3d+obEBCgjIwMq89vg8655eeWXUheXp7y8vKs5zk5OeW1GwAAoIqpMiM78fHx+u6777RgwYIK39aUKVPk5+dnPUJCQip8mwAAwDWqRNhJSEjQokWLlJycrMaNG1vtgYGBys/PV1ZWllP/w4cPKzAw0Opz/tVZ556f63O+cePGKTs723ocOHCgHPcGAABUJS4NO8YYJSQk6NNPP9WyZcsUFhbmtLxDhw6qWbOmli5darXt2rVL+/fvV2RkpCQpMjJSW7du1ZEjR6w+S5Yska+vr1q1anXB7Xp6esrX19fpAQAA7Mmlc3bi4+M1f/58ffbZZ6pTp441x8bPz09eXl7y8/PT4MGDNWrUKNWrV0++vr566qmnFBkZqVtuuUWS1Lt3b7Vq1UoPP/ywEhMTlZGRoT//+c+Kj4+Xp6enK3fPFpiEDACo7lx66bnjIp+kc+bM0aOPPirp7JcKjh49Wh988IHy8vIUHR2tN9980+kU1b59+zR8+HClpKTI29tbcXFxmjp1qmrUKFmW49LzsuPSdACAq5T087tKfc+OqxB2yo6fHgCAq1TL79kBAAAob4QdAABga4QdAABga4QdAABga4QdAABga1Xm3lions6/2oyrswAAVQ0jOwAAwNYIOwAAwNYIOwAAwNaYs4MKw3weAEBVwMgOAACwNcIOAACwNcIOAACwNebsoFzZ/S7vAIDqh5EdAABga4zsoNJwdRYAwBUY2QEAALZG2AEAALZG2AEAALZG2AEAALbGBGVUGUxgBgBUBEZ2AACArTGyA5fhCwgBAJWBkR0AAGBrhB0AAGBrnMZClcWEZQBAeWBkBwAA2BojO6i2GPkBAJQEIzsAAMDWCDsAAMDWCDsAAMDWCDsAAMDWmKAM22DCMgDgQhjZAQAAtkbYAQAAtsZpLNgWp7UAABIjOwAAwOYIOwAAwNY4jYVq4/zTUgAAlARhB1cN5vAAwNWJ01gAAMDWCDsAAMDWCDsAAMDWCDsAAMDWmKAM/B8mMAOAPRF2cNW60kvZCUcAUD1wGgsAANgaIztACfGlhgBQPRF2gHJyuTDEaS4AcA1OYwEAAFtjZAeoJIz8AIBrEHaAKqIywxBXkgG4mnAaCwAA2BojOwAui5EgANUZYQeoprgUHgBKhtNYAADA1hjZAaoJRnIAoGwIOwBKPSeHOTyojvi5vXoRdgBcMb5DCEBVRtgBUEx5nzKr6DDE/7GjLC73c8PPlX0QdgBUOxUdxkrzocYHYuUp7XEv72PB6d7qi7ADoMop7w+JK/mQquwPKD4gXae0YYqLBqoPwg4Al7vch0Zln1ZD+SG8XRzvTeUh7ADAJVxpMHJ1sKroD9TyPAV4Pj78UV5s86WCb7zxhpo2bapatWopIiJC69atc3VJAK5CDofzo7o5v/7y3p/q/N5UtOr+s1OV2SLsfPjhhxo1apQmTpyoTZs26YYbblB0dLSOHDni6tIA4IpcLnyU9gPyavpArWr7eqX1VHQQtTNbhJ3p06dryJAhGjRokFq1aqVZs2apdu3aevfdd11dGgBUKleHHz6Ay4+rj6WdVPuwk5+fr40bNyoqKspqc3NzU1RUlFJTU11YGQDYDx+o1dPVftyq/QTlX375RYWFhQoICHBqDwgI0M6dOy/4mry8POXl5VnPs7OzJUk5OTkVVyiqHX4cqi+OnT1wHK/MpULN5QLP/30sWvz8Lr38fKXtX1bnPrfNZWazV/uwUxZTpkzR5MmTi7WHhIS4oBpUVef/sqL64NjZA8fRdS733pf22FT0sTx+/Lj8LrGRah92GjRoIHd3dx0+fNip/fDhwwoMDLzga8aNG6dRo0ZZz4uKipSZman69evLcYG4m5OTo5CQEB04cEC+vr7luwMoM45L1cWxqZo4LlUXx6ZsjDE6fvy4goODL9mv2ocdDw8PdejQQUuXLlW/fv0knQ0vS5cuVUJCwgVf4+npKU9PT6c2f3//y27L19eXH8IqiONSdXFsqiaOS9XFsSm9S43onFPtw44kjRo1SnFxcerYsaM6deqk1157Tbm5uRo0aJCrSwMAAC5mi7AzYMAAHT16VBMmTFBGRobat2+vxYsXF5u0DAAArj62CDuSlJCQcNHTVlfK09NTEydOLHbqC67Fcam6ODZVE8el6uLYVCyHudz1WgAAANVYtf9SQQAAgEsh7AAAAFsj7AAAAFsj7AAAAFsj7FzGG2+8oaZNm6pWrVqKiIjQunXrXF3SVWfSpElyOBxOjxYtWljLT58+rfj4eNWvX18+Pj6KiYkp9o3auHIrVqzQ3XffreDgYDkcDv373/92Wm6M0YQJExQUFCQvLy9FRUUpPT3dqU9mZqZiY2Pl6+srf39/DR48WCdOnKjEvbCnyx2bRx99tNjvUJ8+fZz6cGzK35QpU3TzzTerTp06atSokfr166ddu3Y59SnJ36/9+/erb9++ql27tho1aqRnn31WBQUFlbkr1R5h5xI+/PBDjRo1ShMnTtSmTZt0ww03KDo6WkeOHHF1aVed1q1b69ChQ9Zj1apV1rKRI0fq888/18KFC7V8+XIdPHhQ/fv3d2G19pSbm6sbbrhBb7zxxgWXJyYmasaMGZo1a5bS0tLk7e2t6OhonT592uoTGxurbdu2acmSJVq0aJFWrFihoUOHVtYu2Nbljo0k9enTx+l36IMPPnBazrEpf8uXL1d8fLzWrl2rJUuW6MyZM+rdu7dyc3OtPpf7+1VYWKi+ffsqPz9fa9as0dy5c5WUlKQJEya4YpeqL4OL6tSpk4mPj7eeFxYWmuDgYDNlyhQXVnX1mThxornhhhsuuCwrK8vUrFnTLFy40GrbsWOHkWRSU1MrqcKrjyTz6aefWs+LiopMYGCgefnll622rKws4+npaT744ANjjDHbt283ksz69eutPl9++aVxOBzm559/rrTa7e78Y2OMMXFxcebee++96Gs4NpXjyJEjRpJZvny5MaZkf7/++9//Gjc3N5ORkWH1mTlzpvH19TV5eXmVuwPVGCM7F5Gfn6+NGzcqKirKanNzc1NUVJRSU1NdWNnVKT09XcHBwWrWrJliY2O1f/9+SdLGjRt15swZp+PUokULNWnShONUifbs2aOMjAyn4+Dn56eIiAjrOKSmpsrf318dO3a0+kRFRcnNzU1paWmVXvPVJiUlRY0aNdL111+v4cOH69dff7WWcWwqR3Z2tiSpXr16kkr29ys1NVVt27Z1uiNAdHS0cnJytG3btkqsvnoj7FzEL7/8osLCwmK3nAgICFBGRoaLqro6RUREKCkpSYsXL9bMmTO1Z88ede3aVcePH1dGRoY8PDyK3ciV41S5zr3Xl/p9ycjIUKNGjZyW16hRQ/Xq1eNYVbA+ffpo3rx5Wrp0qaZNm6bly5frjjvuUGFhoSSOTWUoKirSiBEj1KVLF7Vp00aSSvT3KyMj44K/V+eWoWRsc7sI2Ncdd9xh/btdu3aKiIhQaGioPvroI3l5ebmwMqB6GDhwoPXvtm3bql27dgoPD1dKSop69erlwsquHvHx8fruu++c5hui8jCycxENGjSQu7t7sVnxhw8fVmBgoIuqgiT5+/vruuuu0/fff6/AwEDl5+crKyvLqQ/HqXKde68v9fsSGBhYbHJ/QUGBMjMzOVaVrFmzZmrQoIG+//57SRybipaQkKBFixYpOTlZjRs3ttpL8vcrMDDwgr9X55ahZAg7F+Hh4aEOHTpo6dKlVltRUZGWLl2qyMhIF1aGEydO6IcfflBQUJA6dOigmjVrOh2nXbt2af/+/RynShQWFqbAwECn45CTk6O0tDTrOERGRiorK0sbN260+ixbtkxFRUWKiIio9JqvZj/99JN+/fVXBQUFSeLYVBRjjBISEvTpp59q2bJlCgsLc1pekr9fkZGR2rp1q1MYXbJkiXx9fdWqVavK2RE7cPUM6apswYIFxtPT0yQlJZnt27eboUOHGn9/f6dZ8ah4o0ePNikpKWbPnj1m9erVJioqyjRo0MAcOXLEGGPME088YZo0aWKWLVtmNmzYYCIjI01kZKSLq7af48ePm82bN5vNmzcbSWb69Olm8+bNZt++fcYYY6ZOnWr8/f3NZ599Zr799ltz7733mrCwMHPq1ClrHX369DE33nijSUtLM6tWrTLNmzc3Dz30kKt2yTYudWyOHz9uxowZY1JTU82ePXvM//73P3PTTTeZ5s2bm9OnT1vr4NiUv+HDhxs/Pz+TkpJiDh06ZD1Onjxp9bnc36+CggLTpk0b07t3b7NlyxazePFi07BhQzNu3DhX7FK1Rdi5jL/97W+mSZMmxsPDw3Tq1MmsXbvW1SVddQYMGGCCgoKMh4eHueaaa8yAAQPM999/by0/deqUefLJJ03dunVN7dq1zX333WcOHTrkwortKTk52Ugq9oiLizPGnL38fPz48SYgIMB4enqaXr16mV27djmt49dffzUPPfSQ8fHxMb6+vmbQoEHm+PHjLtgbe7nUsTl58qTp3bu3adiwoalZs6YJDQ01Q4YMKfY/bRyb8nehYyLJzJkzx+pTkr9fe/fuNXfccYfx8vIyDRo0MKNHjzZnzpyp5L2p3hzGGFPZo0kAAACVhTk7AADA1gg7AADA1gg7AADA1gg7AADA1gg7AADA1gg7AADA1gg7AADA1gg7AK4qS5cuVcuWLa07fk+aNEnt27d3bVFXqDT7MGvWLN19990VWxBQxRB2AJt49NFH1a9fv0rfblJSkvz9/S/Zp3v37nI4HBd9dO/e/YpqaNq0qV577bUS9R07dqz+/Oc/y93dXZI0ZswYp3sTldbOnTvlcDi0du1ap/ZbbrlFtWrV0unTp62206dPq1atWnrnnXfKvL0r9dhjj2nTpk1auXKly2oAKhthB0CF++STT3To0CEdOnRI69atkyT973//s9o++eSTSqlj1apV+uGHHxQTE2O1+fj4qH79+mVeZ4sWLRQYGKiUlBSr7fjx49q0aZMaNmzoFIJSU1OVl5ennj17lmlbZ86cKXOd53h4eOh3v/udZsyYccXrAqoLwg5gU927d9fTTz+tsWPHql69egoMDNSkSZOc+jgcDs2cOVN33HGHvLy81KxZM/3rX/+ylqekpMjhcCgrK8tq27JlixwOh/bu3auUlBQNGjRI2dnZ1ijN+duQZG0/MDBQDRs2lCTVr1/fatu+fbu6du0qLy8vhYSE6Omnn1Zubq4kad68efLx8VF6erq1vieffFItWrTQyZMn1b17d+3bt08jR460ariYBQsW6Pbbb1etWrWstvNPAZ0bIXvllVcUFBSk+vXrKz4+/pJBo0ePHk5hZ9WqVbruuut09913O7WnpKQoNDTUuvv1zJkzFR4eLg8PD11//fV67733nNZ77vjcc8898vb21osvvihJmjp1qgICAlSnTh0NHjzYafTo3HY6deokb29v+fv7q0uXLtq3b5+1/O6779Z//vMfnTp16qL7BNgJYQewsblz58rb21tpaWlKTEzU888/ryVLljj1GT9+vGJiYvTNN98oNjZWAwcO1I4dO0q0/s6dO+u1116Tr6+vNUozZsyYUtX4ww8/qE+fPoqJidG3336rDz/8UKtWrVJCQoIk6ZFHHtGdd96p2NhYFRQU6IsvvtA//vEPvf/++6pdu7Y++eQTNW7cWM8//7xVw8WsXLlSHTt2vGxNycnJ+uGHH5ScnKy5c+cqKSlJSUlJF+3fo0cPrVq1SgUFBdbru3fvrm7duik5OdlpvT169JAkffrpp3rmmWc0evRofffddxo2bJgGDRrk1F86G8buu+8+bd26VY899pg++ugjTZo0SS+99JI2bNigoKAgvfnmm1b/goIC9evXT926ddO3336r1NRUDR061CkEduzYUQUFBUpLS7vsewHYgqvvRAqgfMTFxZl7773Xet6tWzdz6623OvW5+eabzXPPPWc9l2SeeOIJpz4RERFm+PDhxpj/fzftY8eOWcs3b95sJJk9e/YYY4yZM2eO8fPzK3Gde/bsMZLM5s2bjTHGDB482AwdOtSpz8qVK42bm5s5deqUMcaYzMxM07hxYzN8+HATEBBgXnzxRaf+oaGh5tVXX73stv38/My8efOc2iZOnGhuuOEG63lcXJwJDQ01BQUFVtsDDzxgBgwYcNH1pqenG0lmzZo1xpiz7/NHH31kDh48aDw9Pc2pU6fMyZMnjaenp5k7d64xxpjOnTubIUOGOK3ngQceMHfeeaf1XJIZMWKEU5/IyEjz5JNPOrVFRERY+/Drr78aSSYlJeWS70XdunVNUlLSJfsAdsHIDmBj7dq1c3oeFBSkI0eOOLVFRkYWe17SkZ3y8M033ygpKUk+Pj7WIzo6WkVFRdqzZ48kqW7dunrnnXes0z5/+MMfyrStU6dOOZ3CupjWrVtbE5ilC79vv3XttdeqcePGSklJUU5OjjZv3qxu3bopKChITZo0UWpqqjVf59zIzo4dO9SlSxen9XTp0qXYe3/+SNSOHTsUERHh1PbbY1ivXj09+uijio6O1t13363XX3/9gqNdXl5eOnny5GXeCcAeCDuAjdWsWdPpucPhUFFRUYlf7+Z29k+EMcZqK49Jsr914sQJDRs2TFu2bLEe33zzjdLT0xUeHm71W7Fihdzd3XXo0CFrPk9pNWjQQMeOHbtsv7K8b927d1dycrJWrlyp5s2bq1GjRpJkncpKTk7Wtddeq5CQkFLV7O3tXar+kjRnzhylpqaqc+fO+vDDD3XdddcVu1osMzPTmj8F2B1hB7jKnf8huHbtWrVs2VKSrA/D344MbNmyxam/h4eH9Z01ZXHTTTdp+/btuvbaa4s9PDw8JElr1qzRtGnT9Pnnn8vHx8eaz1PaGm688UZt3769zLVeSo8ePbRmzRotWbLE6VL62267TSkpKUpJSbFGdSSpZcuWWr16tdM6Vq9erVatWl1yOy1btiw21+b8Yyid3ddx48ZpzZo1atOmjebPn28t++GHH3T69GndeOONpdlFoNoi7ABXuYULF+rdd9/V7t27NXHiRK1bt84KE+dGIiZNmqT09HR98cUX+utf/+r0+qZNm+rEiRNaunSpfvnll1KfGnnuuee0Zs0aJSQkaMuWLUpPT9dnn31m1XD8+HE9/PDDevrpp3XHHXfo/fff14cffuh01VjTpk21YsUK/fzzz/rll18uuq3o6GitWrWqVPWVVI8ePZSbm6t3331X3bp1s9q7deumtLQ0rVu3zinsPPvss0pKStLMmTOVnp6u6dOn65NPPrnsBO9nnnlG7777rubMmWMds23btlnL9+zZo3Hjxik1NVX79u3T119/rfT0dCvASmcnajdr1sxp5AywM8IOcJWbPHmyFixYoHbt2mnevHn64IMPrNGFmjVr6oMPPtDOnTvVrl07TZs2TS+88ILT6zt37qwnnnhCAwYMUMOGDZWYmFiq7bdr107Lly/X7t271bVrV914442aMGGCgoODJZ39cPf29tZLL70kSWrbtq1eeuklDRs2TD///LMk6fnnn9fevXsVHh5+yVMzsbGx2rZtm3bt2lWqGksiLCxMoaGhOn78uFPYadKkiYKDg5Wfn+804tOvXz+9/vrreuWVV9S6dWu99dZbmjNnzmW/YHHAgAEaP368xo4dqw4dOmjfvn0aPny4tbx27drauXOnYmJidN1112no0KGKj4/XsGHDrD4ffPCBhgwZUm77DlR1DvPbk/EArioOh0OffvqpS7552VWeffZZ5eTk6K233nJ1KS6xbds29ezZU7t375afn5+rywEqBSM7AK4qf/rTnxQaGlqqidp2cujQIc2bN4+gg6sKIzvAVexqHNkBcPWp4eoCALgO/68D4GrAaSwAAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBrhB0AAGBr/w86YPtHGCXurwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The longest input text is approximately 227 words long.\n"
     ]
    }
   ],
   "source": [
    "data['text_length'] = data['text'].str.split().apply(len)\n",
    "\n",
    "data.hist(\"text_length\", color=\"Blue\", grid=False, bins=114)\n",
    "plt.title(\"Length of Input Text\")\n",
    "plt.suptitle('')\n",
    "plt.xlabel('Input Text (in Words)')\n",
    "plt.ylabel('Words / Text')\n",
    "plt.show()\n",
    "\n",
    "max_len = max(data['text_length'])\n",
    "\n",
    "print(f\"The longest input text is approximately {max_len} words long.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "93a8f30d",
   "metadata": {},
   "source": [
    "#### Remove Unnecessary Feature(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9f6a0768",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 9713 entries, 0 to 9712\n",
      "Data columns (total 2 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   label   9713 non-null   object\n",
      " 1   text    9713 non-null   object\n",
      "dtypes: object(2)\n",
      "memory usage: 151.9+ KB\n"
     ]
    }
   ],
   "source": [
    "data = data.drop(columns=['text_length'])\n",
    "data.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a8e6121e",
   "metadata": {},
   "source": [
    "#### Basic Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cf6efed0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'1 Stars': 0, '2 Stars': 1, '3 Stars': 2, '4 Stars': 3, '5 Stars': 4}\n",
      "{0: '1 Stars', 1: '2 Stars', 2: '3 Stars', 3: '4 Stars', 4: '5 Stars'}\n"
     ]
    }
   ],
   "source": [
    "NUM_OF_EPOCHS = 6\n",
    "MAX_INPUT_LENGTH = int(220 * 1.25)\n",
    "\n",
    "DEVICE = \"cpu\"\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "STRATEGY = \"epoch\"\n",
    "MODEL_CKPT = \"bert-base-uncased\"\n",
    "\n",
    "MODEL_NAME = MODEL_CKPT.split(\"/\")[-1] + \"-apple_iphone_se_reviews\"\n",
    "REPORTS_TO = [\"tensorboard\", \"wandb\"]\n",
    "\n",
    "WANDB_NAME = \"apple_iphone_se_reviews\"\n",
    "\n",
    "LR = 2e-5\n",
    "set_seed(42)\n",
    "\n",
    "NUM_LABELS = 5\n",
    "label2id = {\n",
    "    \"1 Stars\": 0,\n",
    "    \"2 Stars\": 1,\n",
    "    \"3 Stars\": 2,\n",
    "    \"4 Stars\": 3,\n",
    "    \"5 Stars\": 4\n",
    "    }\n",
    "\n",
    "id2label = {label:idx for idx, label in label2id.items()}\n",
    "\n",
    "print(label2id)\n",
    "print(id2label)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "36b4a1f2",
   "metadata": {},
   "source": [
    "#### Convert Labels From String to Integer Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ccdb1125",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 9713 entries, 0 to 9712\n",
      "Data columns (total 2 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   label   9713 non-null   int64 \n",
      " 1   text    9713 non-null   object\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 151.9+ KB\n"
     ]
    }
   ],
   "source": [
    "data['label'] = data['label'].replace(label2id)\n",
    "\n",
    "data.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ffbe2526",
   "metadata": {},
   "source": [
    "#### Convert DataFrame to Datasets & Split into Training/Evaluation Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2ddcc257",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (7770, 2)\n",
      "Eval data shape: (1943, 2)\n"
     ]
    }
   ],
   "source": [
    "dataset = Dataset.from_pandas(data)\n",
    "\n",
    "train_eval = dataset.train_test_split(test_size=0.20)\n",
    "\n",
    "ds = DatasetDict({\n",
    "    'train' : train_eval['train'],\n",
    "    'eval' : train_eval['test']\n",
    "})\n",
    "\n",
    "print('Training data shape:', ds['train'].shape)\n",
    "print('Eval data shape:', ds['eval'].shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "54187dcd",
   "metadata": {},
   "source": [
    "#### Tokenize Whole Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c41a9840",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c883b88b4c4248629c033aebdd5ec2e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/7770 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fd1ba134c9b542dd9ce28fe9fa81f969",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/1943 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'label': Value(dtype='int64', id=None), 'text': Value(dtype='string', id=None), 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), 'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None), 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None)}\n",
      "Dataset({\n",
      "    features: ['label', 'text', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
      "    num_rows: 7770\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT)\n",
    "\n",
    "def tokenize(batch):\n",
    "    return tokenizer(batch['text'], \n",
    "                     truncation=True, \n",
    "                     padding=True, \n",
    "                     max_length=MAX_INPUT_LENGTH, \n",
    "                     return_tensors=\"pt\")\n",
    "\n",
    "encoded_ds = ds.map(tokenize, \n",
    "                    batched=True, \n",
    "                    batch_size=None)\n",
    "\n",
    "print(encoded_ds['train'].features)\n",
    "print(encoded_ds['train'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "879e7fd8",
   "metadata": {},
   "source": [
    "#### Initialize Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "bda2bb12",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = (\n",
    "    AutoModelForSequenceClassification.from_pretrained(\n",
    "        MODEL_CKPT, \n",
    "        num_labels=NUM_LABELS,\n",
    "        id2label=id2label,\n",
    "        label2id=label2id)\n",
    "    ).to(DEVICE)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "345df9e5",
   "metadata": {},
   "source": [
    "#### Data Collator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "65a74699",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collator = DataCollatorWithPadding(tokenizer, \n",
    "                                        padding=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5b4af8d7",
   "metadata": {},
   "source": [
    "#### Metrics Evaluation Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f032424d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(p):\n",
    "    \"\"\"\n",
    "    This function takes the predictions output from \n",
    "    trainer.evaluate & returns metrics (accuracy, F1, \n",
    "    recall, & precision) for a multiclass \n",
    "    classification project.\n",
    "    \"\"\"\n",
    "    accuracy_metric = evaluate.load(\"accuracy\")\n",
    "    accuracy = accuracy_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                             axis=1), \n",
    "                                       references=p.label_ids)['accuracy']\n",
    "    \n",
    "    ### ------------------- F1 scores -------------------\n",
    "    \n",
    "    f1_score_metric = evaluate.load(\"f1\")\n",
    "    weighted_f1_score = f1_score_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                      axis=1), \n",
    "                                                references=p.label_ids, \n",
    "                                                average='weighted')[\"f1\"]\n",
    "    \n",
    "    micro_f1_score = f1_score_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                   axis=1), \n",
    "                                             references=p.label_ids, \n",
    "                                             average='micro')['f1']\n",
    "    \n",
    "    macro_f1_score = f1_score_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                   axis=1), \n",
    "                                             references=p.label_ids, \n",
    "                                             average='macro')[\"f1\"]\n",
    "    \n",
    "    ### ------------------- recall -------------------\n",
    "    \n",
    "    recall_metric = evaluate.load(\"recall\")\n",
    "    weighted_recall = recall_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                  axis=1), \n",
    "                                            references=p.label_ids, \n",
    "                                            average='weighted')[\"recall\"]\n",
    "    \n",
    "    micro_recall = recall_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                               axis=1), \n",
    "                                         references=p.label_ids, \n",
    "                                         average='micro')[\"recall\"]\n",
    "    \n",
    "    macro_recall = recall_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                               axis=1), \n",
    "                                         references=p.label_ids, \n",
    "                                         average='macro')[\"recall\"]\n",
    "    \n",
    "    ### ------------------- precision -------------------\n",
    "    \n",
    "    precision_metric = evaluate.load(\"precision\")\n",
    "    weighted_precision = precision_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                        axis=1), \n",
    "                                                  references=p.label_ids, \n",
    "                                                  average='weighted')[\"precision\"]\n",
    "    \n",
    "    micro_precision = precision_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                     axis=1), \n",
    "                                               references=p.label_ids, \n",
    "                                               average='micro')[\"precision\"]\n",
    "    \n",
    "    macro_precision = precision_metric.compute(predictions=np.argmax(p.predictions, \n",
    "                                                                     axis=1), \n",
    "                                               references=p.label_ids, \n",
    "                                               average='macro')[\"precision\"]\n",
    "    \n",
    "    return {\"accuracy\" : accuracy, \n",
    "            \"Weighted F1\" : weighted_f1_score,\n",
    "            \"Micro F1\" : micro_f1_score,\n",
    "            \"Macro F1\" : macro_f1_score,\n",
    "            \"Weighted Recall\" : weighted_recall,\n",
    "            \"Micro Recall\" : micro_recall,\n",
    "            \"Macro Recall\" : macro_recall,\n",
    "            \"Weighted Precision\" : weighted_precision,\n",
    "            \"Micro Precision\" : micro_precision,\n",
    "            \"Macro Precision\" : macro_precision\n",
    "            }"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "08334475",
   "metadata": {},
   "source": [
    "#### Trainer Arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0358228f",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_args = TrainingArguments(output_dir=MODEL_NAME, \n",
    "                                  num_train_epochs=NUM_OF_EPOCHS, \n",
    "                                  per_device_train_batch_size=BATCH_SIZE, \n",
    "                                  per_device_eval_batch_size=BATCH_SIZE, \n",
    "                                  weight_decay=0.01,\n",
    "                                  learning_rate=LR,\n",
    "                                  run_name=WANDB_NAME,\n",
    "                                  evaluation_strategy=STRATEGY,\n",
    "                                  logging_strategy='steps',\n",
    "                                  logging_steps=25,\n",
    "                                  logging_first_step=True,\n",
    "                                  save_strategy=STRATEGY, \n",
    "                                  report_to=REPORTS_TO,\n",
    "                                  group_by_length=True,\n",
    "                                  hub_private_repo=True,\n",
    "                                  push_to_hub=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "54c91820",
   "metadata": {},
   "source": [
    "#### Subclass Trainer to Handle Class Imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "83a1f992",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomTrainer(Trainer):\n",
    "    def compute_loss(self, model, inputs, return_outputs=False):\n",
    "        labels = inputs.get(\"labels\")\n",
    "        outputs = model(**inputs)\n",
    "        logits = outputs.get(\"logits\")\n",
    "        loss_fct = torch.nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0]))\n",
    "        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))\n",
    "        return (loss, outputs) if return_outputs else loss"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "191fe09e",
   "metadata": {},
   "source": [
    "#### Build Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3551a802",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning https://huggingface.co/DunnBC22/bert-base-uncased-apple_iphone_se_reviews into local empty directory.\n"
     ]
    }
   ],
   "source": [
    "trainer = CustomTrainer(model=model, \n",
    "                  args=training_args, \n",
    "                  train_dataset=encoded_ds['train'], \n",
    "                  eval_dataset=encoded_ds['eval'],\n",
    "                  tokenizer=tokenizer,\n",
    "                  data_collator=data_collator,\n",
    "                  compute_metrics=compute_metrics)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cc60f187",
   "metadata": {},
   "source": [
    "#### Train the Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a0e50f1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/briandunn/Documents/deep_learning/dl/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>/Users/briandunn/Desktop/current projects/Apple IPhone Reviews/wandb/run-20230603_224052-24kqq415</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/dunnbc22/initial_Apple_iPhone_Reviews/runs/24kqq415' target=\"_blank\">apple_iphone_se_reviews</a></strong> to <a href='https://wandb.ai/dunnbc22/initial_Apple_iPhone_Reviews' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/dunnbc22/initial_Apple_iPhone_Reviews' target=\"_blank\">https://wandb.ai/dunnbc22/initial_Apple_iPhone_Reviews</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/dunnbc22/initial_Apple_iPhone_Reviews/runs/24kqq415' target=\"_blank\">https://wandb.ai/dunnbc22/initial_Apple_iPhone_Reviews/runs/24kqq415</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5727bb7a1e54a69940ee29ba1230181",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/732 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 1.4888, 'learning_rate': 1.997267759562842e-05, 'epoch': 0.01}\n",
      "{'loss': 0.8403, 'learning_rate': 1.9316939890710386e-05, 'epoch': 0.2}\n",
      "{'loss': 0.5108, 'learning_rate': 1.8633879781420766e-05, 'epoch': 0.41}\n",
      "{'loss': 0.2703, 'learning_rate': 1.795081967213115e-05, 'epoch': 0.61}\n",
      "{'loss': 0.251, 'learning_rate': 1.726775956284153e-05, 'epoch': 0.82}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f9de40b322ef42979539376091468bf5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/briandunn/Documents/deep_learning/dl/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/Users/briandunn/Documents/deep_learning/dl/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.14221155643463135, 'eval_accuracy': 0.9315491507977355, 'eval_Weighted F1': 0.9238489786917867, 'eval_Micro F1': 0.9315491507977355, 'eval_Macro F1': 0.6725766195055591, 'eval_Weighted Recall': 0.9315491507977355, 'eval_Micro Recall': 0.9315491507977355, 'eval_Macro Recall': 0.7048002061244156, 'eval_Weighted Precision': 0.922153602062566, 'eval_Micro Precision': 0.9315491507977355, 'eval_Macro Precision': 0.6650345556098448, 'eval_runtime': 604.284, 'eval_samples_per_second': 3.215, 'eval_steps_per_second': 0.051, 'epoch': 1.0}\n",
      "{'loss': 0.1868, 'learning_rate': 1.6584699453551914e-05, 'epoch': 1.02}\n",
      "{'loss': 0.1232, 'learning_rate': 1.5901639344262295e-05, 'epoch': 1.23}\n",
      "{'loss': 0.0969, 'learning_rate': 1.5218579234972679e-05, 'epoch': 1.43}\n",
      "{'loss': 0.0803, 'learning_rate': 1.4535519125683062e-05, 'epoch': 1.64}\n",
      "{'loss': 0.0767, 'learning_rate': 1.3852459016393445e-05, 'epoch': 1.84}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8f5b0ac125542ce869c3478e2a6fc79",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.0870295986533165, 'eval_accuracy': 0.9660319094184251, 'eval_Weighted F1': 0.9652816763756681, 'eval_Micro F1': 0.9660319094184251, 'eval_Macro F1': 0.9526253676167743, 'eval_Weighted Recall': 0.9660319094184251, 'eval_Micro Recall': 0.9660319094184251, 'eval_Macro Recall': 0.94617094919798, 'eval_Weighted Precision': 0.9664565777979365, 'eval_Micro Precision': 0.9660319094184251, 'eval_Macro Precision': 0.9612749531619397, 'eval_runtime': 604.691, 'eval_samples_per_second': 3.213, 'eval_steps_per_second': 0.051, 'epoch': 2.0}\n",
      "{'loss': 0.1022, 'learning_rate': 1.3169398907103825e-05, 'epoch': 2.05}\n",
      "{'loss': 0.0577, 'learning_rate': 1.2486338797814207e-05, 'epoch': 2.25}\n",
      "{'loss': 0.0595, 'learning_rate': 1.1803278688524591e-05, 'epoch': 2.46}\n",
      "{'loss': 0.058, 'learning_rate': 1.1120218579234973e-05, 'epoch': 2.66}\n",
      "{'loss': 0.0487, 'learning_rate': 1.0437158469945357e-05, 'epoch': 2.87}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1330ed75b170421cac614f7af4c647d2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.08450928330421448, 'eval_accuracy': 0.9727225939269172, 'eval_Weighted F1': 0.9727144755734425, 'eval_Micro F1': 0.9727225939269172, 'eval_Macro F1': 0.952785825410173, 'eval_Weighted Recall': 0.9727225939269172, 'eval_Micro Recall': 0.9727225939269172, 'eval_Macro Recall': 0.9572503329927302, 'eval_Weighted Precision': 0.9729587634022647, 'eval_Micro Precision': 0.9727225939269172, 'eval_Macro Precision': 0.9499769769665448, 'eval_runtime': 604.6727, 'eval_samples_per_second': 3.213, 'eval_steps_per_second': 0.051, 'epoch': 3.0}\n",
      "{'loss': 0.0563, 'learning_rate': 9.754098360655738e-06, 'epoch': 3.07}\n",
      "{'loss': 0.0381, 'learning_rate': 9.071038251366122e-06, 'epoch': 3.28}\n",
      "{'loss': 0.0519, 'learning_rate': 8.387978142076504e-06, 'epoch': 3.48}\n",
      "{'loss': 0.037, 'learning_rate': 7.704918032786886e-06, 'epoch': 3.69}\n",
      "{'loss': 0.0335, 'learning_rate': 7.021857923497268e-06, 'epoch': 3.89}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a38fb223adc842a783c7a3927885e14b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.08056594431400299, 'eval_accuracy': 0.9716932578486875, 'eval_Weighted F1': 0.9714921580208616, 'eval_Micro F1': 0.9716932578486875, 'eval_Macro F1': 0.9553596215764735, 'eval_Weighted Recall': 0.9716932578486875, 'eval_Micro Recall': 0.9716932578486875, 'eval_Macro Recall': 0.955951511610887, 'eval_Weighted Precision': 0.9716594857536344, 'eval_Micro Precision': 0.9716932578486875, 'eval_Macro Precision': 0.9557910049179299, 'eval_runtime': 604.3625, 'eval_samples_per_second': 3.215, 'eval_steps_per_second': 0.051, 'epoch': 4.0}\n",
      "{'loss': 0.0344, 'learning_rate': 6.338797814207651e-06, 'epoch': 4.1}\n",
      "{'loss': 0.0309, 'learning_rate': 5.655737704918033e-06, 'epoch': 4.3}\n",
      "{'loss': 0.0209, 'learning_rate': 4.9726775956284154e-06, 'epoch': 4.51}\n",
      "{'loss': 0.0386, 'learning_rate': 4.289617486338798e-06, 'epoch': 4.71}\n",
      "{'loss': 0.0326, 'learning_rate': 3.6065573770491806e-06, 'epoch': 4.92}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fea6e8c7010d4801ac01d64f785603e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.09323258697986603, 'eval_accuracy': 0.9716932578486875, 'eval_Weighted F1': 0.9717104454292538, 'eval_Micro F1': 0.9716932578486875, 'eval_Macro F1': 0.9565895989386686, 'eval_Weighted Recall': 0.9716932578486875, 'eval_Micro Recall': 0.9716932578486875, 'eval_Macro Recall': 0.9598220794839089, 'eval_Weighted Precision': 0.9717870490347844, 'eval_Micro Precision': 0.9716932578486875, 'eval_Macro Precision': 0.9537952599499476, 'eval_runtime': 657.4739, 'eval_samples_per_second': 2.955, 'eval_steps_per_second': 0.047, 'epoch': 5.0}\n",
      "{'loss': 0.0282, 'learning_rate': 2.923497267759563e-06, 'epoch': 5.12}\n",
      "{'loss': 0.0328, 'learning_rate': 2.2404371584699454e-06, 'epoch': 5.33}\n",
      "{'loss': 0.0123, 'learning_rate': 1.557377049180328e-06, 'epoch': 5.53}\n",
      "{'loss': 0.0213, 'learning_rate': 8.743169398907105e-07, 'epoch': 5.74}\n",
      "{'loss': 0.0293, 'learning_rate': 1.912568306010929e-07, 'epoch': 5.94}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d1e76c846fc64241afa6a50bcb6fc352",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.09516146034002304, 'eval_accuracy': 0.9711785898095728, 'eval_Weighted F1': 0.9711026555584843, 'eval_Micro F1': 0.9711785898095728, 'eval_Macro F1': 0.9561228225250444, 'eval_Weighted Recall': 0.9711785898095728, 'eval_Micro Recall': 0.9711785898095728, 'eval_Macro Recall': 0.9579555213168207, 'eval_Weighted Precision': 0.9711287510733905, 'eval_Micro Precision': 0.9711785898095728, 'eval_Macro Precision': 0.9548077724606774, 'eval_runtime': 618.7267, 'eval_samples_per_second': 3.14, 'eval_steps_per_second': 0.05, 'epoch': 6.0}\n",
      "{'train_runtime': 55012.5121, 'train_samples_per_second': 0.847, 'train_steps_per_second': 0.013, 'train_loss': 0.11243759266665725, 'epoch': 6.0}\n"
     ]
    }
   ],
   "source": [
    "training_results = trainer.train()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "277a72fe",
   "metadata": {},
   "source": [
    "#### Save Model & Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "565ab306",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Several commits (2) will be pushed upstream.\n",
      "The progress bars may be unreliable.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "74c443f9b2db489da2a557a6d8afa736",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file pytorch_model.bin:   0%|          | 1.00/418M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a4757ee85e0469cb3b4379b31bc2fb5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file runs/Jun03_22-40-44_Brians-Mac-mini/events.out.tfevents.1685850052.Brians-Mac-mini.4549.0:   0%|  …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/bert-base-uncased-apple_iphone_se_reviews\n",
      "   dd59368..0566f8f  main -> main\n",
      "\n",
      "To https://huggingface.co/DunnBC22/bert-base-uncased-apple_iphone_se_reviews\n",
      "   0566f8f..4323499  main -> main\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** train metrics *****\n",
      "  epoch                    =         6.0\n",
      "  train_loss               =      0.1124\n",
      "  train_runtime            = 15:16:52.51\n",
      "  train_samples_per_second =       0.847\n",
      "  train_steps_per_second   =       0.013\n"
     ]
    }
   ],
   "source": [
    "trainer.save_model()\n",
    "trainer.log_metrics(\"train\", training_results.metrics)\n",
    "trainer.save_metrics(\"train\", training_results.metrics)\n",
    "trainer.save_state()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5ab72d49",
   "metadata": {},
   "source": [
    "#### Push Model to Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "83c80523",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/bert-base-uncased-apple_iphone_se_reviews\n",
      "   4323499..03effbd  main -> main\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'https://huggingface.co/DunnBC22/bert-base-uncased-apple_iphone_se_reviews/commit/03effbdb4b62ab1b4839045087900637e1c8d06f'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.push_to_hub(commit_message=\"All Dunn!!!\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dc662bec",
   "metadata": {},
   "source": [
    "#### Finish Logging to Weights & Biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a7bb50fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/Macro F1</td><td>▁█████</td></tr><tr><td>eval/Macro Precision</td><td>▁█████</td></tr><tr><td>eval/Macro Recall</td><td>▁█████</td></tr><tr><td>eval/Micro F1</td><td>▁▇████</td></tr><tr><td>eval/Micro Precision</td><td>▁▇████</td></tr><tr><td>eval/Micro Recall</td><td>▁▇████</td></tr><tr><td>eval/Weighted F1</td><td>▁▇████</td></tr><tr><td>eval/Weighted Precision</td><td>▁▇████</td></tr><tr><td>eval/Weighted Recall</td><td>▁▇████</td></tr><tr><td>eval/accuracy</td><td>▁▇████</td></tr><tr><td>eval/loss</td><td>█▂▁▁▂▃</td></tr><tr><td>eval/runtime</td><td>▁▁▁▁█▃</td></tr><tr><td>eval/samples_per_second</td><td>████▁▆</td></tr><tr><td>eval/steps_per_second</td><td>████▁▆</td></tr><tr><td>train/epoch</td><td>▁▁▁▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇████</td></tr><tr><td>train/global_step</td><td>▁▁▁▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇████</td></tr><tr><td>train/learning_rate</td><td>███▇▇▇▇▆▆▆▆▅▅▅▅▄▄▄▄▃▃▃▃▂▂▂▂▁▁▁</td></tr><tr><td>train/loss</td><td>█▅▃▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/Macro F1</td><td>0.95612</td></tr><tr><td>eval/Macro Precision</td><td>0.95481</td></tr><tr><td>eval/Macro Recall</td><td>0.95796</td></tr><tr><td>eval/Micro F1</td><td>0.97118</td></tr><tr><td>eval/Micro Precision</td><td>0.97118</td></tr><tr><td>eval/Micro Recall</td><td>0.97118</td></tr><tr><td>eval/Weighted F1</td><td>0.9711</td></tr><tr><td>eval/Weighted Precision</td><td>0.97113</td></tr><tr><td>eval/Weighted Recall</td><td>0.97118</td></tr><tr><td>eval/accuracy</td><td>0.97118</td></tr><tr><td>eval/loss</td><td>0.09516</td></tr><tr><td>eval/runtime</td><td>618.7267</td></tr><tr><td>eval/samples_per_second</td><td>3.14</td></tr><tr><td>eval/steps_per_second</td><td>0.05</td></tr><tr><td>train/epoch</td><td>6.0</td></tr><tr><td>train/global_step</td><td>732</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.0293</td></tr><tr><td>train/total_flos</td><td>6588488565351000.0</td></tr><tr><td>train/train_loss</td><td>0.11244</td></tr><tr><td>train/train_runtime</td><td>55012.5121</td></tr><tr><td>train/train_samples_per_second</td><td>0.847</td></tr><tr><td>train/train_steps_per_second</td><td>0.013</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">apple_iphone_se_reviews</strong> at: <a href='https://wandb.ai/dunnbc22/initial_Apple_iPhone_Reviews/runs/24kqq415' target=\"_blank\">https://wandb.ai/dunnbc22/initial_Apple_iPhone_Reviews/runs/24kqq415</a><br/>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>./wandb/run-20230603_224052-24kqq415/logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wandb.finish()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6cad95d7",
   "metadata": {},
   "source": [
    "### Notes & Other Takeaways From This Project\n",
    "\n",
    "****\n",
    "- I completed a version of this project with Microsoft's MPNet-base transformer as well, but it did not turn out as well.\n",
    "\n",
    "- Of the projects that I am utilizing Weights & Biases with the training, this is the first one to complete training. I am glad I have the opportunity to learn a new and relevant tool. I look forward to exploring it further in the near future!\n",
    "****"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d7bd7e95",
   "metadata": {},
   "source": [
    "### Citation\n",
    "\n",
    "- Model Checkpoint (Bert-Base)\n",
    "\n",
    "    > @article{DBLP:journals/corr/abs-1810-04805, author = {Jacob Devlin and Ming{-}Wei Chang and Kenton Lee and Kristina Toutanova}, title = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language Understanding}, journal = {CoRR}, volume = {abs/1810.04805}, year = {2018}, url = {http://arxiv.org/abs/1810.04805}, archivePrefix = {arXiv}, eprint = {1810.04805}, timestamp = {Tue, 30 Oct 2018 20:39:56 +0100}, biburl = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib}, bibsource = {dblp computer science bibliography, https://dblp.org}}\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 64-bit ('anaconda3')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "a4d9b73f1482e108b8bee9fd1a7eb6f7460da6f2684402434766fba1dc39798b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}