Binary Classification/Malignant Comments/Malignant Comments - I-BERT.ipynb

{"cells":[{"attachments":{},"cell_type":"markdown","metadata":{"id":"YI17H3Melofy"},"source":["## Malignant Comments\n","\n","Dataset Source: https://www.kaggle.com/datasets/surekharamireddy/malignant-comment-classification"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"DswtRoySlof0"},"source":["#### Import Necessary Libraries"]},{"cell_type":"code","execution_count":1,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":15618,"status":"ok","timestamp":1681825432664,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"Rae-2jwElof0","outputId":"f56ba99e-3cc8-4553-eb79-84d881718550"},"outputs":[{"name":"stdout","output_type":"stream","text":["Git LFS initialized.\n"]}],"source":["import os, sys\n","os.environ['TOKENIZERS_PARALLELISM']='false'\n","\n","import numpy as np\n","import pandas as pd\n","import matplotlib.pyplot as plt\n","\n","import datasets\n","from datasets import Dataset, DatasetDict\n","\n","import torch\n","\n","import transformers\n","from transformers import AutoModelForSequenceClassification, AutoTokenizer\n","from transformers import TrainingArguments, Trainer, set_seed, DataCollatorWithPadding\n","\n","import evaluate\n","\n","!git lfs install"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"C1fukwgzlof1"},"source":["#### Display Library Versions"]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":9,"status":"ok","timestamp":1681825452264,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"-xu22V8llof1","outputId":"691d198f-e3d7-4189-c7f8-7abd73a7e125"},"outputs":[{"name":"stdout","output_type":"stream","text":["        Python: 3.9.7 \n","         NumPy: 1.23.3\n","        Pandas: 1.4.4\n","      Datasets: 2.8.0\n","  Transformers: 4.28.1\n","         Torch: 2.0.0\n","      Evaluate: 0.2.2\n"]}],"source":["print(\"Python:\".rjust(15), sys.version[0:6])\n","print(\"NumPy:\".rjust(15), np.__version__)\n","print(\"Pandas:\".rjust(15), pd.__version__)\n","print(\"Datasets:\".rjust(15), datasets.__version__)\n","print(\"Transformers:\".rjust(15), transformers.__version__)\n","print(\"Torch:\".rjust(15), torch.__version__)\n","print(\"Evaluate:\".rjust(15), evaluate.__version__)"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"Vob1xXeDlof2"},"source":["#### Load & Start Preprocessing Dataset"]},{"cell_type":"code","execution_count":3,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":424},"executionInfo":{"elapsed":2995,"status":"ok","timestamp":1681825533014,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"znplYD_rlof2","outputId":"58de724a-0330-4164-a083-6ba7132d6543"},"outputs":[{"name":"stdout","output_type":"stream","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 159571 entries, 0 to 159570\n","Data columns (total 2 columns):\n"," #   Column  Non-Null Count   Dtype \n","---  ------  --------------   ----- \n"," 0   text    159571 non-null  object\n"," 1   label   159571 non-null  int64 \n","dtypes: int64(1), object(1)\n","memory usage: 2.4+ MB\n"]}],"source":["parent_dir = \"/Users/leedunn/Desktop/Projects_to_Train/Malignant Text/data\"\n","\n","data_file_location = os.path.join(parent_dir, \"train.csv\")\n","\n","data = pd.read_csv(data_file_location)\n","\n","data = data.rename(columns={\"comment_text\": \"text\"})\n","\n","data = data.drop(columns=['id', \"malignant\", \"highly_malignant\", \"rude\", \"loathe\"])\n","\n","data['text'] = data['text'].apply(lambda x: x.strip())\n","data['text'] = data['text'].replace(r'\\n', \" \", regex=True)\n","\n","data['label'] = data['abuse'] + data['threat']\n","\n","data['label'] = data['label'].apply(lambda x: 1 if x > 0 else 0)\n","\n","data = data.drop(columns=['threat', 'abuse'])\n","\n","data.info()"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"zksx8xsklof2"},"source":["#### Define Dictionaries to Convert Between String & Integer Class Values"]},{"cell_type":"code","execution_count":4,"metadata":{"id":"hd_T8NJllof2"},"outputs":[{"name":"stdout","output_type":"stream","text":["['clean', 'malignant']\n","2\n","{'clean': 0, 'malignant': 1}\n","{0: 'clean', 1: 'malignant'}\n"]}],"source":["labels = ['clean', 'malignant']\n","\n","id2label = {idx:label for idx, label in enumerate(labels)}\n","label2id = {label:idx for idx, label in enumerate(labels)}\n","\n","NUM_OF_LABELS = len(labels)\n","\n","print(labels)\n","print(NUM_OF_LABELS)\n","print(label2id)\n","print(id2label)"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"xUshgDR8lof3"},"source":["#### Visual: Histogram of Text Lengths"]},{"cell_type":"code","execution_count":5,"metadata":{"id":"8gKf23Wqlof3","outputId":"51f36894-6440-4bfd-ae17-a30aac9cb806"},"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAABccAAAK9CAYAAADhdNQKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABo7klEQVR4nOzde5xVdb0//tdwGxCYQVRuiYBQKCpaqMhXRS0SFVNPdtQ0RfOShZlgKpR5qyNkmVrHMk8lVmppJ60kL4iAN7yE4S0lJRBLwVJhBJXr/v3Rj30cuY7OMAPr+Xw89qPZn/VZa73XZrEe04uP711RKpVKAQAAAACAAmnW2AUAAAAAAMDGJhwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDjCcQAAAAAACkc4DgAAAABA4QjHAQAAAAAoHOE4AAAAAACFIxwHAGCD9ezZMyeeeGJjl0ETcNFFF6WioiL/+te/Gvxcd955Z3bbbbe0bt06FRUVWbBgQYOfEwCAzZ9wHACgoMaPH5+Kior86U9/WuP2/fffPzvvvPMHPs8f//jHXHTRRR/4OJujOXPmpKKiIt/97ncbu5S1uvTSS3Pbbbc12vlfe+21HHXUUWnTpk2uvvrq/OIXv0jbtm3Xuc+sWbPyhS98Idtvv31at26dqqqq7L333rnqqqvy9ttvb6TKmzZ/LwEAkhaNXQAAAJuOmTNnplmzuq2v+OMf/5irr75aELeJuvTSS/OZz3wmRxxxRKOc/7HHHsubb76Zb37zmxkyZMh650+YMCH/+Z//mcrKypxwwgnZeeeds3Tp0jzwwAM555xz8swzz+Taa6/dCJU3bf5eAgAIxwEAqIPKysrGLqHOFi9evN6VxjRdr776apKkQ4cO6507e/bsHHPMMenRo0fuvffedO3atbxtxIgReeGFFzJhwoSGKhUAgE2MtioAAGyw9/YcX7ZsWS6++OJ8+MMfTuvWrbPVVltln332ycSJE5MkJ554Yq6++uokSUVFRfm1yuLFi3P22Wene/fuqaysTN++ffPd7343pVKp1nnffvvtnHnmmdl6663Tvn37HHbYYfnHP/6RioqKWitfV/XB/stf/pJjjz02W265ZfbZZ58kyZNPPpkTTzyx3GqjS5cu+fznP5/XXnut1rlWHeOvf/1rPve5z6W6ujrbbLNNvvGNb6RUKuWll17K4YcfnqqqqnTp0iWXX375ap/T3Llz89xzz32gz/rdlixZkgsvvDB9+vRJZWVlunfvnnPPPTdLliypNa+ioiJnnHFGbrvttuy8886prKzMTjvtlDvvvHO1Y06ZMiW77757Wrdund69e+fHP/5x+drffbzFixfn+uuvL//Zvbfn/IIFC3LiiSemQ4cOqa6uzkknnZS33nprg67rlltuyYABA9KmTZtsvfXW+dznPpd//OMf5e37779/hg8fniTZY4891nj+d7vsssuyaNGi/PSnP60VjK/Sp0+ffOUrXym/X758eb75zW+md+/eqaysTM+ePfO1r31ttc+1Z8+eOfTQQ8ufWZs2bbLLLrtkypQpSZLf/va32WWXXdK6desMGDAgf/7zn2vtf+KJJ6Zdu3aZO3duDj300LRr1y4f+tCHyn83nnrqqXz84x9P27Zt06NHj9x4442r1b5gwYKcddZZ5b8rffr0ybe//e2sXLmyPOfdbXquvfba8nXtscceeeyxx2rVs66/lwAARWHlOABAwS1cuHCNX6q4bNmy9e570UUXZezYsTnllFOy5557pqamJn/605/y+OOP55Of/GS+8IUv5OWXX87EiRPzi1/8ota+pVIphx12WCZPnpyTTz45u+22W+66666cc845+cc//pErrriiPPfEE0/MzTffnOOPPz577bVXpk6dmmHDhq21rv/8z//Mhz/84Vx66aXloH3ixIn529/+lpNOOildunQpt9d45pln8vDDD68WDh599NHZcccdM27cuEyYMCHf+ta30rFjx/z4xz/Oxz/+8Xz729/ODTfckK9+9avZY489Mnjw4PK+J5xwQqZOnbpayP9+rFy5MocddlgeeOCBnHbaadlxxx3z1FNP5Yorrshf//rX1fqBP/DAA/ntb3+bL33pS2nfvn2+//3v58gjj8zcuXOz1VZbJUn+/Oc/56CDDkrXrl1z8cUXZ8WKFbnkkkuyzTbb1DrWL37xi/Kf7WmnnZYk6d27d605Rx11VHr16pWxY8fm8ccfz09+8pN06tQp3/72t9d5XePHj89JJ52UPfbYI2PHjs38+fNz1VVX5cEHH8yf//zndOjQIV//+tfTt2/fXHvttbnkkkvSq1ev1c7/bn/4wx+y/fbb5//9v/+3QZ/tKaeckuuvvz6f+cxncvbZZ+eRRx7J2LFj8+yzz+bWW2+tNfeFF17Isccemy984Qv53Oc+l+9+97v51Kc+lWuuuSZf+9rX8qUvfSlJMnbs2Bx11FGrtSBasWJFDj744AwePDiXXXZZbrjhhpxxxhlp27Ztvv71r+e4447Lpz/96VxzzTU54YQTMmjQoPTq1StJ8tZbb2W//fbLP/7xj3zhC1/Idtttl4ceeihjxozJK6+8kiuvvLJWrTfeeGPefPPNfOELX0hFRUUuu+yyfPrTn87f/va3tGzZcp1/LwEACqUEAEAhXXfddaUk63zttNNOtfbp0aNHafjw4eX3u+66a2nYsGHrPM+IESNKa/q187bbbislKX3rW9+qNf6Zz3ymVFFRUXrhhRdKpVKpNH369FKS0llnnVVr3oknnlhKUrrwwgvLYxdeeGEpSemzn/3saud76623Vhu76aabSklK991332rHOO2008pjy5cvL2277balioqK0rhx48rjb7zxRqlNmza1PpNSqVTab7/91njN7zV79uxSktJ3vvOdtc75xS9+UWrWrFnp/vvvrzV+zTXXlJKUHnzwwfJYklKrVq3Kn12pVCo98cQTpSSlH/zgB+WxT33qU6Utttii9I9//KM89vzzz5datGixWt1t27Zd7fpKpf/7nD7/+c/XGv+P//iP0lZbbbXO6166dGmpU6dOpZ133rn09ttvl8dvv/32UpLSBRdcUB5bdZ8+9thj6zzmwoULS0lKhx9++DrnrTJjxoxSktIpp5xSa/yrX/1qKUnp3nvvLY/16NGjlKT00EMPlcfuuuuuUpJSmzZtSi+++GJ5/Mc//nEpSWny5MnlseHDh5eSlC699NLy2Kp7p6KiovSrX/2qPP7cc8+tdl9/85vfLLVt27b017/+tVato0ePLjVv3rw0d+7cUqn0f/fTVlttVXr99dfL8373u9+VkpT+8Ic/lMfW9vcSAKBItFUBACi4q6++OhMnTlzt1b9///Xu26FDhzzzzDN5/vnn63zeP/7xj2nevHnOPPPMWuNnn312SqVS7rjjjiQptwRZtTJ3lS9/+ctrPfbpp5++2libNm3KP7/zzjv517/+lb322itJ8vjjj682/5RTTin/3Lx58+y+++4plUo5+eSTy+MdOnRI375987e//a3WvlOmTKmXVePJv1uP7Ljjjtlhhx3yr3/9q/z6+Mc/niSZPHlyrflDhgyptbq6f//+qaqqKte4YsWK3HPPPTniiCPSrVu38rw+ffrk4IMPrnN97/2s991337z22mupqalZ6z5/+tOf8uqrr+ZLX/pSWrduXR4fNmxYdthhh/fVF3zV+dq3b79B8//4xz8mSUaNGlVr/Oyzz06S1Wro169fBg0aVH4/cODAJMnHP/7xbLfddquNv/eeSGrfU6vunbZt2+aoo44qj/ft2zcdOnSotf8tt9ySfffdN1tuuWWte2DIkCFZsWJF7rvvvlrnOfroo7PllluW3++7775rrQkAoMi0VQEAKLg999wzu++++2rjq4K4dbnkkkty+OGH5yMf+Uh23nnnHHTQQTn++OM3KFh/8cUX061bt9XCzB133LG8fdX/NmvWrNxiYpU+ffqs9djvnZskr7/+ei6++OL86le/Kn/J4yoLFy5cbf67A88kqa6uTuvWrbP11luvNv7evuX16fnnn8+zzz67WsuTVd57Le+tO/n3n+Ubb7xRnv/222+v8fNb12e6Nu8936pQ9o033khVVdUa91n1Z9u3b9/Vtu2www554IEH6lzHqnO9+eabGzR/1X313mvu0qVLOnToUK5xlTXdD0nSvXv3NY6v+rxXad269Wp/htXV1dl2221Xa+lTXV1da//nn38+Tz755Pu+B979ZwIAwP8RjgMA8L4NHjw4s2bNyu9+97vcfffd+clPfpIrrrgi11xzTa1Vshvbu1eJr3LUUUfloYceyjnnnJPddtst7dq1y8qVK3PQQQfV+lLDVZo3b75BY0nqbZX4mqxcuTK77LJLvve9761x+3vD2Y1dY2N8JmtSVVWVbt265emnn67Tfhv6RZRru84Nvf4Psv/KlSvzyU9+Mueee+4a537kIx95XzUBABSdcBwAgA+kY8eOOemkk3LSSSdl0aJFGTx4cC666KJyOL628LFHjx6555578uabb9ZaPf7cc8+Vt6/635UrV2b27Nn58Ic/XJ73wgsvbHCNb7zxRiZNmpSLL744F1xwQXn8/bSD2dh69+6dJ554Ip/4xCc2OMhdl06dOqV169Zr/PzWNFYf53yvVX+2M2fOLLeHWWXmzJnl7XV16KGH5tprr820adNqtUBZWw0rV67M888/X/6vFZJk/vz5WbBgwfuuoSH07t07ixYtypAhQ+rtmA3x5woAsKnRcxwAgPftve1E2rVrlz59+mTJkiXlsbZt2yZJFixYUGvuIYcckhUrVuS///u/a41fccUVqaioKPe/Hjp0aJLkhz/8Ya15P/jBDza4zlUrad+7cvbKK6/c4GPUxdy5c8sh/wd11FFH5R//+Ef+53/+Z7Vtb7/9dhYvXlyn4zVv3jxDhgzJbbfdlpdffrk8/sILL5T7vL9b27ZtV/uz+6B23333dOrUKddcc02te+WOO+7Is88+m2HDhr2v45577rlp27ZtTjnllMyfP3+17bNmzcpVV12V5N/3X7L6PbBqhf77raEhHHXUUZk2bVruuuuu1bYtWLAgy5cvr/Mx1/b3EgCgSKwcBwDgfevXr1/233//DBgwIB07dsyf/vSn/OY3v8kZZ5xRnjNgwIAkyZlnnpmhQ4emefPmOeaYY/KpT30qBxxwQL7+9a9nzpw52XXXXXP33Xfnd7/7Xc4666zyl0oOGDAgRx55ZK688sq89tpr2WuvvTJ16tT89a9/TbJhK2CrqqoyePDgXHbZZVm2bFk+9KEP5e67787s2bMb4FNJTjjhhEydOnWD21hMmjQp77zzzmrjRxxxRI4//vjcfPPNOf300zN58uTsvffeWbFiRZ577rncfPPNueuuu9bYM35dLrrootx9993Ze++988UvfrH8jxQ777xzZsyYUWvugAEDcs899+R73/teunXrll69epW/dPL9atmyZb797W/npJNOyn777ZfPfvazmT9/fq666qr07NkzI0eOfF/H7d27d2688cYcffTR2XHHHXPCCSdk5513ztKlS/PQQw/llltuyYknnpgk2XXXXTN8+PBce+21WbBgQfbbb788+uijuf7663PEEUfkgAMO+EDXWJ/OOeec/P73v8+hhx6aE088MQMGDMjixYvz1FNP5Te/+U3mzJmzWi/89Vnb30sAgCIRjgMA8L6deeaZ+f3vf5+77747S5YsSY8ePfKtb30r55xzTnnOpz/96Xz5y1/Or371q/zyl79MqVTKMccck2bNmuX3v/99Lrjggvz617/Oddddl549e+Y73/lOzj777Frn+fnPf54uXbrkpptuyq233pohQ4bk17/+dfr27ZvWrVtvUK033nhjvvzlL+fqq69OqVTKgQcemDvuuCPdunWr18/k/bjzzjtz5513rjbes2fP7LzzzrnttttyxRVX5Oc//3luvfXWbLHFFtl+++3zla98ZbV+0xtiwIABueOOO/LVr3413/jGN9K9e/dccsklefbZZ1db8f69730vp512Ws4///y8/fbbGT58+AcOx5PkxBNPzBZbbJFx48blvPPOS9u2bfMf//Ef+fa3v50OHTq87+MedthhefLJJ/Od73wnv/vd7/KjH/0olZWV6d+/fy6//PKceuqp5bk/+clPsv3222f8+PG59dZb06VLl4wZMyYXXnjhB76++rTFFltk6tSpufTSS3PLLbfk5z//eaqqqvKRj3wkF198cflLQOtibX8vAQCKpKLkW1kAANgEzZgxIx/96Efzy1/+Mscdd1xjl7NZOOKII/LMM89sEr3YAQDgg9JzHACAJu/tt99ebezKK69Ms2bNMnjw4EaoaNP33s/0+eefzx//+Mfsv//+jVMQAABsZNqqAADQ5F122WWZPn16DjjggLRo0SJ33HFH7rjjjpx22mnp3r17Y5e3Sdp+++1z4oknZvvtt8+LL76YH/3oR2nVqlXOPffcxi4NAAA2Cm1VAABo8iZOnJiLL744f/nLX7Jo0aJst912Of744/P1r389LVpY7/F+nHTSSZk8eXLmzZuXysrKDBo0KJdeemk+9rGPNXZpAACwUQjHAQAAAAAoHD3HAQAAAAAoHOE4AAAAAACFo0HjBli5cmVefvnltG/fPhUVFY1dDgAAAAAAa1AqlfLmm2+mW7duadZs3WvDheMb4OWXX0737t0buwwAAAAAADbASy+9lG233Xadc4TjG6B9+/ZJ/v2BVlVVNXI1AAAAAACsSU1NTbp3717OdNdFOL4BVrVSqaqqEo4DAAAAADRxG9Ie2xdyAgAAAABQOMJxAAAAAAAKRzgOAAAAAEDhCMcBAAAAACgc4TgAAAAAAIUjHAcAAAAAoHCE4wAAAAAAFI5wHAAAAACAwhGOAwAAAABQOMJxAAAAAAAKRzgOAAAAAEDhCMcBAAAAACgc4TgAAAAAAIUjHAcAAAAAoHCE4wAAAAAAFI5wHAAAAACAwhGOAwAAAABQOMJxAAAAAAAKRzgOAAAAAEDhCMcBAAAAACgc4TgAAAAAAIUjHAcAAAAAoHCE4wAAAAAAFI5wHAAAAACAwhGOAwAAAABQOMJxAAAAAAAKp0VjFwDvV8/RE9a6bc64YRuxEgAAAABgU2PlOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKJxGDcd/9KMfpX///qmqqkpVVVUGDRqUO+64o7x9//33T0VFRa3X6aefXusYc+fOzbBhw7LFFlukU6dOOeecc7J8+fJac6ZMmZKPfexjqaysTJ8+fTJ+/PiNcXkAAAAAADRRLRrz5Ntuu23GjRuXD3/4wymVSrn++utz+OGH589//nN22mmnJMmpp56aSy65pLzPFltsUf55xYoVGTZsWLp06ZKHHnoor7zySk444YS0bNkyl156aZJk9uzZGTZsWE4//fTccMMNmTRpUk455ZR07do1Q4cO3bgXDAAAAABAk1BRKpVKjV3Eu3Xs2DHf+c53cvLJJ2f//ffPbrvtliuvvHKNc++4444ceuihefnll9O5c+ckyTXXXJPzzjsv//znP9OqVaucd955mTBhQp5++unyfsccc0wWLFiQO++8c4NqqqmpSXV1dRYuXJiqqqoPfI3Uj56jJ6x125xxwzZiJQAAAABAU1CXLLfJ9BxfsWJFfvWrX2Xx4sUZNGhQefyGG27I1ltvnZ133jljxozJW2+9Vd42bdq07LLLLuVgPEmGDh2ampqaPPPMM+U5Q4YMqXWuoUOHZtq0aWutZcmSJampqan1AgAAAABg89GobVWS5KmnnsqgQYPyzjvvpF27drn11lvTr1+/JMmxxx6bHj16pFu3bnnyySdz3nnnZebMmfntb3+bJJk3b16tYDxJ+f28efPWOaempiZvv/122rRps1pNY8eOzcUXX1zv1woAAAAAQNPQ6OF43759M2PGjCxcuDC/+c1vMnz48EydOjX9+vXLaaedVp63yy67pGvXrvnEJz6RWbNmpXfv3g1W05gxYzJq1Kjy+5qamnTv3r3BzgcAAAAAwMbV6G1VWrVqlT59+mTAgAEZO3Zsdt1111x11VVrnDtw4MAkyQsvvJAk6dKlS+bPn19rzqr3Xbp0WeecqqqqNa4aT5LKyspUVVXVegEAAAAAsPlo9HD8vVauXJklS5ascduMGTOSJF27dk2SDBo0KE899VReffXV8pyJEyemqqqq3Jpl0KBBmTRpUq3jTJw4sVZfcwAAAAAAiqVR26qMGTMmBx98cLbbbru8+eabufHGGzNlypTcddddmTVrVm688cYccsgh2WqrrfLkk09m5MiRGTx4cPr3758kOfDAA9OvX78cf/zxueyyyzJv3rycf/75GTFiRCorK5Mkp59+ev77v/875557bj7/+c/n3nvvzc0335wJEyY05qUDAAAAANCIGjUcf/XVV3PCCSfklVdeSXV1dfr375+77rorn/zkJ/PSSy/lnnvuyZVXXpnFixene/fuOfLII3P++eeX92/evHluv/32fPGLX8ygQYPStm3bDB8+PJdcckl5Tq9evTJhwoSMHDkyV111Vbbddtv85Cc/ydChQxvjktlIeo5e9z9+zBk3bCNVAgAAAAA0RRWlUqnU2EU0dTU1Namurs7ChQv1H29C1heAr4twHAAAAAA2P3XJcptcz3EAAAAAAGhownEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAAqnRWMXAI2h5+gJa902Z9ywjVgJAAAAANAYrBwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDjCcQAAAAAACkc4DgAAAABA4QjHAQAAAAAoHOE4AAAAAACFIxwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDjCcQAAAAAACkc4DgAAAABA4QjHAQAAAAAoHOE4AAAAAACFIxwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDjCcQAAAAAACkc4DgAAAABA4QjHAQAAAAAoHOE4AAAAAACFIxwHAAAAAKBwhOMAAAAAABROo4bjP/rRj9K/f/9UVVWlqqoqgwYNyh133FHe/s4772TEiBHZaqut0q5duxx55JGZP39+rWPMnTs3w4YNyxZbbJFOnTrlnHPOyfLly2vNmTJlSj72sY+lsrIyffr0yfjx4zfG5QEAAAAA0EQ1aji+7bbbZty4cZk+fXr+9Kc/5eMf/3gOP/zwPPPMM0mSkSNH5g9/+ENuueWWTJ06NS+//HI+/elPl/dfsWJFhg0blqVLl+ahhx7K9ddfn/Hjx+eCCy4oz5k9e3aGDRuWAw44IDNmzMhZZ52VU045JXfddddGv14AAAAAAJqGilKpVGrsIt6tY8eO+c53vpPPfOYz2WabbXLjjTfmM5/5TJLkueeey4477php06Zlr732yh133JFDDz00L7/8cjp37pwkueaaa3Leeefln//8Z1q1apXzzjsvEyZMyNNPP10+xzHHHJMFCxbkzjvvXGMNS5YsyZIlS8rva2pq0r179yxcuDBVVVUNePXURc/RExrkuHPGDWuQ4wIAAAAADaumpibV1dUblOW22Eg1rdeKFStyyy23ZPHixRk0aFCmT5+eZcuWZciQIeU5O+ywQ7bbbrtyOD5t2rTssssu5WA8SYYOHZovfvGLeeaZZ/LRj34006ZNq3WMVXPOOuustdYyduzYXHzxxfV+jWwaPkjoLlgHAAAAgE1Do38h51NPPZV27dqlsrIyp59+em699db069cv8+bNS6tWrdKhQ4da8zt37px58+YlSebNm1crGF+1fdW2dc2pqanJ22+/vcaaxowZk4ULF5ZfL730Un1cKgAAAAAATUSjrxzv27dvZsyYkYULF+Y3v/lNhg8fnqlTpzZqTZWVlamsrGzUGgAAAAAAaDiNHo63atUqffr0SZIMGDAgjz32WK666qocffTRWbp0aRYsWFBr9fj8+fPTpUuXJEmXLl3y6KOP1jre/Pnzy9tW/e+qsXfPqaqqSps2bRrqsgAAAAAAaMIava3Ke61cuTJLlizJgAED0rJly0yaNKm8bebMmZk7d24GDRqUJBk0aFCeeuqpvPrqq+U5EydOTFVVVfr161ee8+5jrJqz6hgAAAAAABRPo64cHzNmTA4++OBst912efPNN3PjjTdmypQpueuuu1JdXZ2TTz45o0aNSseOHVNVVZUvf/nLGTRoUPbaa68kyYEHHph+/frl+OOPz2WXXZZ58+bl/PPPz4gRI8ptUU4//fT893//d84999x8/vOfz7333pubb745Eya8/y9dBAAAAABg09ao4firr76aE044Ia+88kqqq6vTv3//3HXXXfnkJz+ZJLniiivSrFmzHHnkkVmyZEmGDh2aH/7wh+X9mzdvnttvvz1f/OIXM2jQoLRt2zbDhw/PJZdcUp7Tq1evTJgwISNHjsxVV12VbbfdNj/5yU8ydOjQjX69AAAAAAA0DRWlUqnU2EU0dTU1Namurs7ChQtTVVXV2OXw/+s5uumt/p8zblhjlwAAAAAAhVWXLLfRv5AT1qYpht8AAAAAwOahyX0hJwAAAAAANDThOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKJwWjV0AFEXP0RPWuX3OuGEbqRIAAAAAwMpxAAAAAAAKRzgOAAAAAEDhCMcBAAAAACgcPcehHq2vrzgAAAAA0DRYOQ4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4LRq7AGD9eo6esM7tc8YN20iVAAAAAMDmwcpxAAAAAAAKx8px2Aysa2W5VeUAAAAAsDrhODQR62udAgAAAADUH21VAAAAAAAoHOE4AAAAAACFIxwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDjCcQAAAAAACkc4DgAAAABA4QjHAQAAAAAoHOE4AAAAAACFIxwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDjCcQAAAAAACkc4DgAAAABA4QjHAQAAAAAoHOE4AAAAAACFIxwHAAAAAKBwWjR2AUDD6jl6wlq3zRk3bCNWAgAAAABNh3AcCmxdwXkiPAcAAABg86WtCgAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4jRqOjx07NnvssUfat2+fTp065YgjjsjMmTNrzdl///1TUVFR63X66afXmjN37twMGzYsW2yxRTp16pRzzjkny5cvrzVnypQp+djHPpbKysr06dMn48ePb+jLg01ez9ET1voCAAAAgE1Zo4bjU6dOzYgRI/Lwww9n4sSJWbZsWQ488MAsXry41rxTTz01r7zySvl12WWXlbetWLEiw4YNy9KlS/PQQw/l+uuvz/jx43PBBReU58yePTvDhg3LAQcckBkzZuSss87KKaeckrvuumujXSsAAAAAAE1Hi8Y8+Z133lnr/fjx49OpU6dMnz49gwcPLo9vscUW6dKlyxqPcffdd+cvf/lL7rnnnnTu3Dm77bZbvvnNb+a8887LRRddlFatWuWaa65Jr169cvnllydJdtxxxzzwwAO54oorMnTo0Ia7QAAAAAAAmqQm1XN84cKFSZKOHTvWGr/hhhuy9dZbZ+edd86YMWPy1ltvlbdNmzYtu+yySzp37lweGzp0aGpqavLMM8+U5wwZMqTWMYcOHZpp06atsY4lS5akpqam1gsAAAAAgM1Ho64cf7eVK1fmrLPOyt57752dd965PH7sscemR48e6datW5588smcd955mTlzZn77298mSebNm1crGE9Sfj9v3rx1zqmpqcnbb7+dNm3a1No2duzYXHzxxfV+jQAAAAAANA1NJhwfMWJEnn766TzwwAO1xk877bTyz7vssku6du2aT3ziE5k1a1Z69+7dILWMGTMmo0aNKr+vqalJ9+7dG+RcAAAAAABsfE2ircoZZ5yR22+/PZMnT8622267zrkDBw5MkrzwwgtJki5dumT+/Pm15qx6v6pP+drmVFVVrbZqPEkqKytTVVVV6wUAAAAAwOajUcPxUqmUM844I7feemvuvffe9OrVa737zJgxI0nStWvXJMmgQYPy1FNP5dVXXy3PmThxYqqqqtKvX7/ynEmTJtU6zsSJEzNo0KB6uhIAAAAAADYljRqOjxgxIr/85S9z4403pn379pk3b17mzZuXt99+O0kya9asfPOb38z06dMzZ86c/P73v88JJ5yQwYMHp3///kmSAw88MP369cvxxx+fJ554InfddVfOP//8jBgxIpWVlUmS008/PX/7299y7rnn5rnnnssPf/jD3HzzzRk5cmSjXTsAAAAAAI2nUcPxH/3oR1m4cGH233//dO3atfz69a9/nSRp1apV7rnnnhx44IHZYYcdcvbZZ+fII4/MH/7wh/Ixmjdvnttvvz3NmzfPoEGD8rnPfS4nnHBCLrnkkvKcXr16ZcKECZk4cWJ23XXXXH755fnJT36SoUOHbvRrBgAAAACg8VWUSqVSYxfR1NXU1KS6ujoLFy7Uf3wj6jl6QmOXwDrMGTessUsAAAAAgFrqkuU2iS/kBAAAAACAjalFYxcAFM/6/qsAq9IBAAAAaGhWjgMAAAAAUDjCcQAAAAAACkc4DgAAAABA4QjHAQAAAAAoHOE4AAAAAACF06KxCwA2Tz1HT2jsEgAAAABgrYTjwPsi/AYAAABgU6atCgAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCadHYBQC8V8/RE9a6bc64YRuxEgAAAAA2V1aOAwAAAABQOMJxAAAAAAAKRzgOAAAAAEDhCMcBAAAAACgc4TgAAAAAAIXTorELAKiLnqMnrHXbnHHDNmIlAAAAAGzKrBwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDjCcQAAAAAACkc4DgAAAABA4QjHAQAAAAAoHOE4AAAAAACFIxwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMKpczj+0ksv5e9//3v5/aOPPpqzzjor1157bb0WBgAAAAAADaXO4fixxx6byZMnJ0nmzZuXT37yk3n00Ufz9a9/PZdcckm9FwgAAAAAAPWtzuH4008/nT333DNJcvPNN2fnnXfOQw89lBtuuCHjx4+v7/oAAAAAAKDe1TkcX7ZsWSorK5Mk99xzTw477LAkyQ477JBXXnmlfqsDAAAAAIAGUOdwfKeddso111yT+++/PxMnTsxBBx2UJHn55Zez1VZb1XuBAAAAAABQ3+ocjn/729/Oj3/84+y///757Gc/m1133TVJ8vvf/77cbgUAAAAAAJqyFnXdYf/998+//vWv1NTUZMsttyyPn3baadliiy3qtTiA+tRz9IS1bpszbthGrAQAAACAxlbnleNJUiqVMn369Pz4xz/Om2++mSRp1aqVcBwAAAAAgE1CnVeOv/jiiznooIMyd+7cLFmyJJ/85CfTvn37fPvb386SJUtyzTXXNESdAAAAAABQb+ocjn/lK1/J7rvvnieeeKLWF3D+x3/8R0499dR6LQ6gLtbVNqWxrK8m7VwAAAAAGkedw/H7778/Dz30UFq1alVrvGfPnvnHP/5Rb4UBAAAAAEBDqXM4vnLlyqxYsWK18b///e9p3759vRQFsLFZ4Q0AAABQLHX+Qs4DDzwwV155Zfl9RUVFFi1alAsvvDCHHHJIfdYGAAAAAAANos4rxy+//PIMHTo0/fr1yzvvvJNjjz02zz//fLbeeuvcdNNNDVEjAAAAAADUqzqH49tuu22eeOKJ/OpXv8qTTz6ZRYsW5eSTT85xxx2XNm3aNESNAAAAAABQr+ocjidJixYt8rnPfa6+awEAAAAAgI1ig8Lx3//+9xt8wMMOO+x9FwMAAAAAABvDBoXjRxxxxAYdrKKiIitWrPgg9QAAAAAAQIPboHB85cqVDV0HAAAAAABsNM0auwAAAAAAANjY3lc4PmnSpBx66KHp3bt3evfunUMPPTT33HNPfdcGAAAAAAANos7h+A9/+MMcdNBBad++fb7yla/kK1/5SqqqqnLIIYfk6quvbogaAQAAAACgXm1Qz/F3u/TSS3PFFVfkjDPOKI+deeaZ2XvvvXPppZdmxIgR9VogAAAAAADUtzqvHF+wYEEOOuig1cYPPPDALFy4sF6KAgAAAACAhlTnleOHHXZYbr311pxzzjm1xn/3u9/l0EMPrbfCAIqg5+gJa902Z9ywjVgJAAAAQLHUORzv169f/uu//itTpkzJoEGDkiQPP/xwHnzwwZx99tn5/ve/X5575pln1l+lAAAAAABQT+ocjv/0pz/Nlltumb/85S/5y1/+Uh7v0KFDfvrTn5bfV1RUCMcBAAAAAGiS6hyOz549uyHqAGjStD8BAAAA2LzU+Qs5AQAAAABgU1fnleOlUim/+c1vMnny5Lz66qtZuXJlre2//e1v6604AAAAAABoCHVeOX7WWWfl+OOPz+zZs9OuXbtUV1fXetXF2LFjs8cee6R9+/bp1KlTjjjiiMycObPWnHfeeScjRozIVlttlXbt2uXII4/M/Pnza82ZO3duhg0bli222CKdOnXKOeeck+XLl9eaM2XKlHzsYx9LZWVl+vTpk/Hjx9f10gEAAAAA2EzUeeX4L37xi/z2t7/NIYcc8oFPPnXq1IwYMSJ77LFHli9fnq997Ws58MAD85e//CVt27ZNkowcOTITJkzILbfckurq6pxxxhn59Kc/nQcffDBJsmLFigwbNixdunTJQw89lFdeeSUnnHBCWrZsmUsvvTTJv/ukDxs2LKeffnpuuOGGTJo0Kaecckq6du2aoUOHfuDrAIptXf3IAQAAAGiaKkqlUqkuO/Tq1St33HFHdthhh3ov5p///Gc6deqUqVOnZvDgwVm4cGG22Wab3HjjjfnMZz6TJHnuueey4447Ztq0adlrr71yxx135NBDD83LL7+czp07J0muueaanHfeefnnP/+ZVq1a5bzzzsuECRPy9NNPl891zDHHZMGCBbnzzjvXW1dNTU2qq6uzcOHCVFVV1ft1s2YCR4rOF30CAAAA1E1dstw6t1W56KKLcvHFF+ftt99+3wWuzcKFC5MkHTt2TJJMnz49y5Yty5AhQ8pzdthhh2y33XaZNm1akmTatGnZZZddysF4kgwdOjQ1NTV55plnynPefYxVc1Yd472WLFmSmpqaWi8AAAAAADYfdW6rctRRR+Wmm25Kp06d0rNnz7Rs2bLW9scff/x9FbJy5cqcddZZ2XvvvbPzzjsnSebNm5dWrVqlQ4cOteZ27tw58+bNK895dzC+avuqbeuaU1NTk7fffjtt2rSptW3s2LG5+OKL39d1ANSX9f3XE1aWAwAAALx/dQ7Hhw8fnunTp+dzn/tcOnfunIqKinopZMSIEXn66afzwAMP1MvxPogxY8Zk1KhR5fc1NTXp3r17I1YEAAAAAEB9qnM4PmHChNx1113ZZ5996q2IM844I7fffnvuu+++bLvttuXxLl26ZOnSpVmwYEGt1ePz589Ply5dynMeffTRWsebP39+eduq/1019u45VVVVq60aT5LKyspUVlbWy7UBAAAAAND01LnnePfu3evtSylLpVLOOOOM3Hrrrbn33nvTq1evWtsHDBiQli1bZtKkSeWxmTNnZu7cuRk0aFCSZNCgQXnqqafy6quvludMnDgxVVVV6devX3nOu4+xas6qYwAAAAAAUCx1Dscvv/zynHvuuZkzZ84HPvmIESPyy1/+MjfeeGPat2+fefPmZd68eeUv+6yurs7JJ5+cUaNGZfLkyZk+fXpOOumkDBo0KHvttVeS5MADD0y/fv1y/PHH54knnshdd92V888/PyNGjCiv/j799NPzt7/9Leeee26ee+65/PCHP8zNN9+ckSNHfuBrAAAAAABg01NRKpVKddlhyy23zFtvvZXly5dniy22WO0LOV9//fUNP/la+pVfd911OfHEE5Mk77zzTs4+++zcdNNNWbJkSYYOHZof/vCH5ZYpSfLiiy/mi1/8YqZMmZK2bdtm+PDhGTduXFq0+L+uMVOmTMnIkSPzl7/8Jdtuu22+8Y1vlM+xPjU1Namurs7ChQvrbdU867e+LyOEovOFnAAAAAC11SXLrXM4fv31169z+/Dhw+tyuE2CcLxxCMdh3YTjAAAAALXVJcut8xdybo7hNwAAAAAAxVLncPzd3nnnnSxdurTWmJXVAAAAAAA0dXX+Qs7FixfnjDPOSKdOndK2bdtsueWWtV4AAAAAANDU1TkcP/fcc3PvvffmRz/6USorK/OTn/wkF198cbp165af//znDVEjAAAAAADUqzq3VfnDH/6Qn//859l///1z0kknZd99902fPn3So0eP3HDDDTnuuOMaok4AAAAAAKg3dV45/vrrr2f77bdP8u/+4q+//nqSZJ999sl9991Xv9UBAAAAAEADqHM4vv3222f27NlJkh122CE333xzkn+vKO/QoUO9FgcAAAAAAA2hzuH4SSedlCeeeCJJMnr06Fx99dVp3bp1Ro4cmXPOOafeCwQAAAAAgPpW557jI0eOLP88ZMiQPPvss3n88cfTp0+f9O/fv16LAwAAAACAhlDncPy9evbsmZ49e9ZDKQAAAAAAsHFscFuVadOm5fbbb6819vOf/zy9evVKp06dctppp2XJkiX1XiAAAAAAANS3DQ7HL7nkkjzzzDPl90899VROPvnkDBkyJKNHj84f/vCHjB07tkGKBAAAAACA+rTBbVVmzJiRb37zm+X3v/rVrzJw4MD8z//8T5Kke/fuufDCC3PRRRfVe5EArK7n6Alr3TZn3LCNWAkAAADApmeDV46/8cYb6dy5c/n91KlTc/DBB5ff77HHHnnppZfqtzoAAAAAAGgAGxyOd+7cObNnz06SLF26NI8//nj22muv8vY333wzLVu2rP8KAQAAAACgnm1wOH7IIYdk9OjRuf/++zNmzJhsscUW2Xfffcvbn3zyyfTu3btBigQAAAAAgPq0wT3Hv/nNb+bTn/509ttvv7Rr1y7XX399WrVqVd7+s5/9LAceeGCDFAkAAAAAAPVpg8PxrbfeOvfdd18WLlyYdu3apXnz5rW233LLLWnXrl29FwgAAAAAAPVtg8PxVaqrq9c43rFjxw9cDAAAAAAAbAwb3HMcAAAAAAA2F8JxAAAAAAAKRzgOAAAAAEDhbFA4/rGPfSxvvPFGkuSSSy7JW2+91aBFAQAAAABAQ9qgcPzZZ5/N4sWLkyQXX3xxFi1a1KBFAQAAAABAQ2qxIZN22223nHTSSdlnn31SKpXy3e9+N+3atVvj3AsuuKBeCwQAAAAAgPq2QeH4+PHjc+GFF+b2229PRUVF7rjjjrRosfquFRUVwnEAAAAAAJq8ilKpVKrLDs2aNcu8efPSqVOnhqqpyampqUl1dXUWLlyYqqqqxi6nMHqOntDYJcBma864YY1dAgAAAEC9q0uWu0Erx99t5cqV77swAAAAAABoCuocjifJrFmzcuWVV+bZZ59NkvTr1y9f+cpX0rt373otDgAAAAAAGkKzuu5w1113pV+/fnn00UfTv3//9O/fP4888kh22mmnTJw4sSFqBAAAAACAelXnleOjR4/OyJEjM27cuNXGzzvvvHzyk5+st+IAAAAAAKAh1Hnl+LPPPpuTTz55tfHPf/7z+ctf/lIvRQEAAAAAQEOqczi+zTbbZMaMGauNz5gxI506daqPmgAAAAAAoEHVua3KqaeemtNOOy1/+9vf8v/+3/9Lkjz44IP59re/nVGjRtV7gQDUv56jJ6x125xxwzZiJQAAAACNo87h+De+8Y20b98+l19+ecaMGZMk6datWy666KKceeaZ9V4gAAAAAADUtzqH4xUVFRk5cmRGjhyZN998M0nSvn37ei8MAAAAAAAaSp3D8XcTigMAAAAAsCmq8xdyAgAAAADApk44DgAAAABA4QjHAQAAAAAoHOE4AAAAAACF877C8TPOOCOvv/56fdcCAAAAAAAbxQaH43//+9/LP994441ZtGhRkmSXXXbJSy+9VP+VAQAAAABAA2mxoRN32GGHbLXVVtl7773zzjvv5KWXXsp2222XOXPmZNmyZQ1ZIwAAAAAA1KsNXjm+YMGC3HLLLRkwYEBWrlyZQw45JB/5yEeyZMmS3HXXXZk/f35D1gkAAAAAAPVmg8PxZcuWZc8998zZZ5+dNm3a5M9//nOuu+66NG/ePD/72c/Sq1ev9O3btyFrBQAAAACAerHBbVU6dOiQ3XbbLXvvvXeWLl2at99+O3vvvXdatGiRX//61/nQhz6Uxx57rCFrBQAAAACAerHBK8f/8Y9/5Pzzz09lZWWWL1+eAQMGZN99983SpUvz+OOPp6KiIvvss09D1goAAAAAAPWiolQqleq605Zbbpn77rsvzz77bE444YR06dIl8+fPz5577pmpU6c2RJ2NqqamJtXV1Vm4cGGqqqoau5zC6Dl6QmOXAIU0Z9ywtW5b39/Lde0LAAAA0NDqkuVu8Mrx96qurs5RRx2Vli1b5t57783s2bPzpS996f0eDgAAAAAANpoN7jn+bk8++WQ+9KEPJUl69OiRli1bpkuXLjn66KPrtTgAAAAAAGgI7ysc7969e/nnp59+ut6KAQAAAACAjeF9t1UBAAAAAIBNlXAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOG0aOwCAGhaeo6e0NglAAAAADQ4K8cBAAAAACgc4TgAAAAAAIWjrQoA9WZdLVnmjBu2ESsBAAAAWDcrxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAAqnRWMXAEAx9Bw9YZ3b54wbtpEqAQAAAGjkleP33XdfPvWpT6Vbt26pqKjIbbfdVmv7iSeemIqKilqvgw46qNac119/Pccdd1yqqqrSoUOHnHzyyVm0aFGtOU8++WT23XfftG7dOt27d89ll13W0JcGAAAAAEAT1qjh+OLFi7Prrrvm6quvXuucgw46KK+88kr5ddNNN9Xaftxxx+WZZ57JxIkTc/vtt+e+++7LaaedVt5eU1OTAw88MD169Mj06dPzne98JxdddFGuvfbaBrsuAAAAAACatkZtq3LwwQfn4IMPXuecysrKdOnSZY3bnn322dx555157LHHsvvuuydJfvCDH+SQQw7Jd7/73XTr1i033HBDli5dmp/97Gdp1apVdtppp8yYMSPf+973aoXoAAAAAAAUR5P/Qs4pU6akU6dO6du3b774xS/mtddeK2+bNm1aOnToUA7Gk2TIkCFp1qxZHnnkkfKcwYMHp1WrVuU5Q4cOzcyZM/PGG2+s8ZxLlixJTU1NrRcAAAAAAJuPJh2OH3TQQfn5z3+eSZMm5dvf/namTp2agw8+OCtWrEiSzJs3L506daq1T4sWLdKxY8fMmzevPKdz58615qx6v2rOe40dOzbV1dXlV/fu3ev70gAAAAAAaESN2lZlfY455pjyz7vsskv69++f3r17Z8qUKfnEJz7RYOcdM2ZMRo0aVX5fU1MjIAcAAAAA2Iw06ZXj77X99ttn6623zgsvvJAk6dKlS1599dVac5YvX57XX3+93Ke8S5cumT9/fq05q96vrZd5ZWVlqqqqar0AAAAAANh8bFLh+N///ve89tpr6dq1a5Jk0KBBWbBgQaZPn16ec++992blypUZOHBgec59992XZcuWledMnDgxffv2zZZbbrlxLwAAAAAAgCahUcPxRYsWZcaMGZkxY0aSZPbs2ZkxY0bmzp2bRYsW5ZxzzsnDDz+cOXPmZNKkSTn88MPTp0+fDB06NEmy44475qCDDsqpp56aRx99NA8++GDOOOOMHHPMMenWrVuS5Nhjj02rVq1y8skn55lnnsmvf/3rXHXVVbXapgAAAAAAUCyNGo7/6U9/ykc/+tF89KMfTZKMGjUqH/3oR3PBBRekefPmefLJJ3PYYYflIx/5SE4++eQMGDAg999/fyorK8vHuOGGG7LDDjvkE5/4RA455JDss88+ufbaa8vbq6urc/fdd2f27NkZMGBAzj777FxwwQU57bTTNvr1AgAAAADQNFSUSqVSYxfR1NXU1KS6ujoLFy7Uf3wj6jl6QmOXAGxEc8YNa+wSAAAAgE1cXbLcTarnOAAAAAAA1AfhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCadHYBQBAkvQcPWGt2+aMG7YRKwEAAACKQDgOQJPXUMH5uo77QY8NAAAANG3CcQA2aQJuAAAA4P3QcxwAAAAAgMIRjgMAAAAAUDjCcQAAAAAACkc4DgAAAABA4QjHAQAAAAAoHOE4AAAAAACFIxwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDjCcQAAAAAACkc4DgAAAABA4bRo7AIAoCH1HD2hsUsAAAAAmiDhOACsxbqC9Tnjhm3ESgAAAID6pq0KAAAAAACFIxwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDgtGrsAANgc9Rw9Ya3b5owbthErAQAAANbEynEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBw9xwFgI9OPHAAAABqfcBwA3od1BdwAAABA06etCgAAAAAAhWPlOAA0Ietbka7tCgAAANQPK8cBAAAAACgc4TgAAAAAAIUjHAcAAAAAoHCE4wAAAAAAFI4v5ASATci6vrDTl3UCAADAhrNyHAAAAACAwhGOAwAAAABQONqqAADatQAAAFA4Vo4DAAAAAFA4wnEAAAAAAApHWxUAKIh1tU4BAACAorFyHAAAAACAwhGOAwAAAABQOMJxAAAAAAAKRzgOAAAAAEDhCMcBAAAAACgc4TgAAAAAAIUjHAcAAAAAoHCE4wAAAAAAFI5wHAAAAACAwmnR2AUAAE1bz9ET1rl9zrhhG6kSAAAAqD/CcQDYTKwvxAYAAAD+j7YqAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhdOisQsAADZfPUdPWOf2OeOGbaRKAAAAoDYrxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcBo1HL/vvvvyqU99Kt26dUtFRUVuu+22WttLpVIuuOCCdO3aNW3atMmQIUPy/PPP15rz+uuv57jjjktVVVU6dOiQk08+OYsWLao158knn8y+++6b1q1bp3v37rnssssa+tIAoDB6jp6w1hcAAAA0VY0aji9evDi77rprrr766jVuv+yyy/L9738/11xzTR555JG0bds2Q4cOzTvvvFOec9xxx+WZZ57JxIkTc/vtt+e+++7LaaedVt5eU1OTAw88MD169Mj06dPzne98JxdddFGuvfbaBr8+AAAAAACapopSqVRq7CKSpKKiIrfeemuOOOKIJP9eNd6tW7ecffbZ+epXv5okWbhwYTp37pzx48fnmGOOybPPPpt+/frlsccey+67754kufPOO3PIIYfk73//e7p165Yf/ehH+frXv5558+alVatWSZLRo0fntttuy3PPPbdBtdXU1KS6ujoLFy5MVVVV/V88a2TFIcDmb864YY1dAgAAAJuRumS5Tbbn+OzZszNv3rwMGTKkPFZdXZ2BAwdm2rRpSZJp06alQ4cO5WA8SYYMGZJmzZrlkUceKc8ZPHhwORhPkqFDh2bmzJl544031njuJUuWpKamptYLAAAAAIDNR5MNx+fNm5ck6dy5c63xzp07l7fNmzcvnTp1qrW9RYsW6dixY605azrGu8/xXmPHjk11dXX51b179w9+QQAAAAAANBlNNhxvTGPGjMnChQvLr5deeqmxSwIAAAAAoB412XC8S5cuSZL58+fXGp8/f355W5cuXfLqq6/W2r58+fK8/vrrteas6RjvPsd7VVZWpqqqqtYLAAAAAIDNR5MNx3v16pUuXbpk0qRJ5bGampo88sgjGTRoUJJk0KBBWbBgQaZPn16ec++992blypUZOHBgec59992XZcuWledMnDgxffv2zZZbbrmRrgYAAAAAgKakUcPxRYsWZcaMGZkxY0aSf38J54wZMzJ37txUVFTkrLPOyre+9a38/ve/z1NPPZUTTjgh3bp1yxFHHJEk2XHHHXPQQQfl1FNPzaOPPpoHH3wwZ5xxRo455ph069YtSXLsscemVatWOfnkk/PMM8/k17/+da666qqMGjWqka4aAAAAAIDG1qIxT/6nP/0pBxxwQPn9qsB6+PDhGT9+fM4999wsXrw4p512WhYsWJB99tknd955Z1q3bl3e54YbbsgZZ5yRT3ziE2nWrFmOPPLIfP/73y9vr66uzt13350RI0ZkwIAB2XrrrXPBBRfktNNO23gXCgDUWc/RE9a5fc64YRupEgAAADZHFaVSqdTYRTR1NTU1qa6uzsKFC/Uf34jWF4oAsOlbV8AtHAcAAKCu6pLlNurKcQCA92td4bngHAAAgPVpsl/ICQAAAAAADcXKcQCg0TRUCy2rygEAAFgfK8cBAAAAACgc4TgAAAAAAIUjHAcAAAAAoHD0HAcACmV9fc71JAcAACgGK8cBAAAAACgc4TgAAAAAAIUjHAcAAAAAoHCE4wAAAAAAFI5wHAAAAACAwmnR2AUAAGwqeo6esM7tc8YN20iVAAAA8EFZOQ4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhaPnOADAu6yvrzgAAACbByvHAQAAAAAoHOE4AAAAAACFIxwHAAAAAKBw9BwHANgI1tXLfM64YRuxEgAAABIrxwEAAAAAKCDhOAAAAAAAhaOtCgBAI1tXy5VE2xUAAICGYOU4AAAAAACFIxwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDjCcQAAAAAACqdFYxcAALC56Dl6QmOXAAAAwAaychwAAAAAgMKxchwAYBO2vtXqc8YN20iVAAAAbFqsHAcAAAAAoHCsHAcAaOL0MgcAAKh/Vo4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOHoOQ4AsBlbV7/yOeOGbcRKAAAAmhbhOI3KF4wBAAAAAI1BWxUAAAAAAApHOA4AAAAAQOFoqwIAUFAfpL3ZuvqVr++4ep0DAABNgZXjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcFo0dgEAAGx6eo6e0NglAAAAfCDCcQAANgvrCuznjBu2ESsBAAA2BcJxAAA2KiE2AADQFOg5DgAAAABA4QjHAQAAAAAoHOE4AAAAAACFo+c4AACbjHX1K/8g++l1DgAAxSMcBwCg8HxJKAAAFI9wHAAAGolQHgAAGo+e4wAAAAAAFI5wHAAAAACAwtFWBQCAJuP9fuEmAABAXVk5DgAAAABA4Vg5DgAA6/BBVrP7Uk0AAGi6rBwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMLRcxwAABrIB+lXDgAANKwmvXL8oosuSkVFRa3XDjvsUN7+zjvvZMSIEdlqq63Srl27HHnkkZk/f36tY8ydOzfDhg3LFltskU6dOuWcc87J8uXLN/alAAAAAADQhDT5leM77bRT7rnnnvL7Fi3+r+SRI0dmwoQJueWWW1JdXZ0zzjgjn/70p/Pggw8mSVasWJFhw4alS5cueeihh/LKK6/khBNOSMuWLXPppZdu9GsBAID6sL4V6XPGDdtIlQAAwKaryYfjLVq0SJcuXVYbX7hwYX7605/mxhtvzMc//vEkyXXXXZcdd9wxDz/8cPbaa6/cfffd+ctf/pJ77rknnTt3zm677ZZvfvObOe+883LRRRelVatWG/tyAAAAAABoApp8OP7888+nW7duad26dQYNGpSxY8dmu+22y/Tp07Ns2bIMGTKkPHeHHXbIdtttl2nTpmWvvfbKtGnTsssuu6Rz587lOUOHDs0Xv/jFPPPMM/noRz+6xnMuWbIkS5YsKb+vqalpuAsEAIA10K8cAAAaVpPuOT5w4MCMHz8+d955Z370ox9l9uzZ2XffffPmm29m3rx5adWqVTp06FBrn86dO2fevHlJknnz5tUKxldtX7VtbcaOHZvq6uryq3v37vV7YQAAAAAANKomvXL84IMPLv/cv3//DBw4MD169MjNN9+cNm3aNNh5x4wZk1GjRpXf19TUCMgBAAAAADYjTXrl+Ht16NAhH/nIR/LCCy+kS5cuWbp0aRYsWFBrzvz588s9yrt06ZL58+evtn3VtrWprKxMVVVVrRcAAAAAAJuPJr1y/L0WLVqUWbNm5fjjj8+AAQPSsmXLTJo0KUceeWSSZObMmZk7d24GDRqUJBk0aFD+67/+K6+++mo6deqUJJk4cWKqqqrSr1+/RrsOAABoSOvqVz5n3LAG2xcAADYlTToc/+pXv5pPfepT6dGjR15++eVceOGFad68eT772c+muro6J598ckaNGpWOHTumqqoqX/7ylzNo0KDstddeSZIDDzww/fr1y/HHH5/LLrss8+bNy/nnn58RI0aksrKyka8OAAA2Pl/0CQAA/9akw/G///3v+exnP5vXXnst22yzTfbZZ588/PDD2WabbZIkV1xxRZo1a5YjjzwyS5YsydChQ/PDH/6wvH/z5s1z++2354tf/GIGDRqUtm3bZvjw4bnkkksa65IAAAAAAGgCKkqlUqmxi2jqampqUl1dnYULF+o/Xs+sXAIA2HRoqwIAQFNXlyy3Sa8cBwAAmg79yAEA2Jw0a+wCAAAAAABgYxOOAwAAAABQOMJxAAAAAAAKR89xAABgk7W+L3jXCx0AgLWxchwAAAAAgMKxchwAAGjS1rc6HAAA3g/hOAAA8IF9kPYmDRl+v99ja8cCALD501YFAAAAAIDCsXIcAABocFqj/J91fRZWrAMAbDxWjgMAAAAAUDhWjgMAANSR1d8AAJs+4TgAAMB7aAMDALD501YFAAAAAIDCEY4DAAAAAFA42qoAAABsAtbX6kWvcwCAuhGOAwAANBEfpNf5B9lXsA4AFJG2KgAAAAAAFI5wHAAAAACAwtFWBQAAoB59kPYmjWVdNa+v5coH2RcAoDEJxwEAAOAD8oWpALDp0VYFAAAAAIDCsXIcAACABtFQLVc2xVXa2s8AQNMjHAcAAGCtNsUe6gAAG0JbFQAAAAAACsfKcQAAADa6TbE1CgCweRGOAwAAsFnR33vD+JwAKDrhOAAAAE2OXudNm2AdgM2BcBwAAADSeIG8FjMA0DiE4wAAABTGprgi3SptAGgYwnEAAAAAgE2Qf0D9YITjAAAAQC2b4gp7AKgr4TgAAABQb6xiBGBTIRwHAACATZQgGgDeP+E4AAAAbIa0RgGAdROOAwAAABvFBwnsrYQHoL4JxwEAAIBNnhYzANRVs8YuAAAAAAAANjYrxwEAAIAm74O0ZFnfvlaWAxSTcBwAAAAotIZqyfJBjqtNDEDDE44DAAAAsF7CfmBzIxwHAAAAeB8+SKuXhjqvkBpgwwnHAQAAAGi0sL9I9L+HpkU4DgAAALAWDRUYb2rHbUhWswONpVljFwAAAAAAABubleMAAAAAfCBNccX6ptjCxCp62LiE4wAAAAA0SZtiwA1sOoTjAAAAABROQ63Sboqr6IE1E44DAAAAQBP3QUL3dYX9DXVc2BQIx2lw/sUUAAAAaAgNlTnIMhre5tZffXO7nqIQjgMAAAAA9e79/iPDB+k1r089dSEcBwAAAADqbHNbYd9YfegF9o1HOA4AAAAAm7EihdgNte/m9hnyb80auwAAAAAAANjYrBwHAAAAAGgkVqU3HivHAQAAAAAoHOE4AAAAAACFIxwHAAAAAKBwhOMAAAAAABSOcBwAAAAAgMIRjgMAAAAAUDgtGrsANn09R09o7BIAAAAAAOrEynEAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCKVQ4fvXVV6dnz55p3bp1Bg4cmEcffbSxSwIAAAAAoBEUJhz/9a9/nVGjRuXCCy/M448/nl133TVDhw7Nq6++2tilAQAAAACwkRUmHP/e976XU089NSeddFL69euXa665JltssUV+9rOfNXZpAAAAAABsZC0au4CNYenSpZk+fXrGjBlTHmvWrFmGDBmSadOmrTZ/yZIlWbJkSfn9woULkyQ1NTUNX+wmaOWStxq7BAAAAADgXYqaZa667lKptN65hQjH//Wvf2XFihXp3LlzrfHOnTvnueeeW23+2LFjc/HFF6823r179warEQAAAACgvlRf2dgVNK4333wz1dXV65xTiHC8rsaMGZNRo0aV369cuTKvv/56ttpqq1RUVDRiZY2jpqYm3bt3z0svvZSqqqrGLgfqnXucInCfs7lzj7O5c49TBO5zNnfucYqgKdznpVIpb775Zrp167beuYUIx7feeus0b9488+fPrzU+f/78dOnSZbX5lZWVqaysrDXWoUOHhixxk1BVVeXhzWbNPU4RuM/Z3LnH2dy5xykC9zmbO/c4RdDY9/n6VoyvUogv5GzVqlUGDBiQSZMmlcdWrlyZSZMmZdCgQY1YGQAAAAAAjaEQK8eTZNSoURk+fHh233337LnnnrnyyiuzePHinHTSSY1dGgAAAAAAG1lhwvGjjz46//znP3PBBRdk3rx52W233XLnnXeu9iWdrK6ysjIXXnjhaq1mYHPhHqcI3Ods7tzjbO7c4xSB+5zNnXucItjU7vOKUqlUauwiAAAAAABgYypEz3EAAAAAAHg34TgAAAAAAIUjHAcAAAAAoHCE4wAAAAAAFI5wnHW6+uqr07Nnz7Ru3ToDBw7Mo48+2tglwfsyduzY7LHHHmnfvn06deqUI444IjNnzqw1Z//9909FRUWt1+mnn95IFUPdXXTRRavdwzvssEN5+zvvvJMRI0Zkq622Srt27XLkkUdm/vz5jVgx1E3Pnj1Xu8crKioyYsSIJJ7jbJruu+++fOpTn0q3bt1SUVGR2267rdb2UqmUCy64IF27dk2bNm0yZMiQPP/887XmvP766znuuONSVVWVDh065OSTT86iRYs24lXA2q3rHl+2bFnOO++87LLLLmnbtm26deuWE044IS+//HKtY6zp+T9u3LiNfCWwdut7lp944omr3cMHHXRQrTme5TRl67vH1/Q7ekVFRb7zne+U5zTVZ7lwnLX69a9/nVGjRuXCCy/M448/nl133TVDhw7Nq6++2tilQZ1NnTo1I0aMyMMPP5yJEydm2bJlOfDAA7N48eJa80499dS88sor5ddll13WSBXD+7PTTjvVuocfeOCB8raRI0fmD3/4Q2655ZZMnTo1L7/8cj796U83YrVQN4899lit+3vixIlJkv/8z/8sz/EcZ1OzePHi7Lrrrrn66qvXuP2yyy7L97///VxzzTV55JFH0rZt2wwdOjTvvPNOec5xxx2XZ555JhMnTsztt9+e++67L6eddtrGugRYp3Xd42+99VYef/zxfOMb38jjjz+e3/72t5k5c2YOO+yw1eZecskltZ7vX/7ylzdG+bBB1vcsT5KDDjqo1j1800031druWU5Ttr57/N339iuvvJKf/exnqaioyJFHHllrXlN8lrdo7AJour73ve/l1FNPzUknnZQkueaaazJhwoT87Gc/y+jRoxu5OqibO++8s9b78ePHp1OnTpk+fXoGDx5cHt9iiy3SpUuXjV0e1JsWLVqs8R5euHBhfvrTn+bGG2/Mxz/+8STJddddlx133DEPP/xw9tprr41dKtTZNttsU+v9uHHj0rt37+y3337lMc9xNjUHH3xwDj744DVuK5VKufLKK3P++efn8MMPT5L8/Oc/T+fOnXPbbbflmGOOybPPPps777wzjz32WHbfffckyQ9+8IMccsgh+e53v5tu3bpttGuBNVnXPV5dXV3+h85V/vu//zt77rln5s6dm+2226483r59e893mqx13eerVFZWrvUe9iynqVvfPf7ee/t3v/tdDjjggGy//fa1xpvis9zKcdZo6dKlmT59eoYMGVIea9asWYYMGZJp06Y1YmVQPxYuXJgk6dixY63xG264IVtvvXV23nnnjBkzJm+99VZjlAfv2/PPP59u3bpl++23z3HHHZe5c+cmSaZPn55ly5bVeq7vsMMO2W677TzX2SQtXbo0v/zlL/P5z38+FRUV5XHPcTYns2fPzrx582o9u6urqzNw4MDys3vatGnp0KFDOUxJkiFDhqRZs2Z55JFHNnrN8EEtXLgwFRUV6dChQ63xcePGZauttspHP/rRfOc738ny5csbp0B4n6ZMmZJOnTqlb9+++eIXv5jXXnutvM2znM3J/PnzM2HChJx88smrbWuKz3Irx1mjf/3rX1mxYkU6d+5ca7xz58557rnnGqkqqB8rV67MWWedlb333js777xzefzYY49Njx490q1btzz55JM577zzMnPmzPz2t79txGphww0cODDjx49P375988orr+Tiiy/Ovvvum6effjrz5s1Lq1atVvs/mp07d868efMap2D4AG677bYsWLAgJ554YnnMc5zNzarn85p+J1+1bd68eenUqVOt7S1atEjHjh0939nkvPPOOznvvPPy2c9+NlVVVeXxM888Mx/72MfSsWPHPPTQQxkzZkxeeeWVfO9732vEamHDHXTQQfn0pz+dXr16ZdasWfna176Wgw8+ONOmTUvz5s09y9msXH/99Wnfvv1qLTyb6rNcOA4UzogRI/L000/X6sWcpFY/t1122SVdu3bNJz7xicyaNSu9e/fe2GVCnb37P3Pr379/Bg4cmB49euTmm29OmzZtGrEyqH8//elPc/DBB9f6z4w9xwE2XcuWLctRRx2VUqmUH/3oR7W2jRo1qvxz//7906pVq3zhC1/I2LFjU1lZubFLhTo75phjyj/vsssu6d+/f3r37p0pU6bkE5/4RCNWBvXvZz/7WY477ri0bt261nhTfZZrq8Iabb311mnevHnmz59fa3z+/PlNrjcQ1MUZZ5yR22+/PZMnT8622267zrkDBw5MkrzwwgsbozSodx06dMhHPvKRvPDCC+nSpUuWLl2aBQsW1Jrjuc6m6MUXX8w999yTU045ZZ3zPMfZ1K16Pq/rd/IuXbrk1VdfrbV9+fLlef311z3f2WSsCsZffPHFTJw4sdaq8TUZOHBgli9fnjlz5mycAqGebb/99tl6663Lv6N4lrO5uP/++zNz5sz1/p6eNJ1nuXCcNWrVqlUGDBiQSZMmlcdWrlyZSZMmZdCgQY1YGbw/pVIpZ5xxRm699dbce++96dWr13r3mTFjRpKka9euDVwdNIxFixZl1qxZ6dq1awYMGJCWLVvWeq7PnDkzc+fO9Vxnk3PdddelU6dOGTZs2DrneY6zqevVq1e6dOlS69ldU1OTRx55pPzsHjRoUBYsWJDp06eX59x7771ZuXJl+R+IoClbFYw///zzueeee7LVVlutd58ZM2akWbNmq7WhgE3F3//+97z22mvl31E8y9lc/PSnP82AAQOy6667rnduU3mWa6vCWo0aNSrDhw/P7rvvnj333DNXXnllFi9enJNOOqmxS4M6GzFiRG688cb87ne/S/v27ct926qrq9OmTZvMmjUrN954Yw455JBstdVWefLJJzNy5MgMHjw4/fv3b+TqYcN89atfzac+9an06NEjL7/8ci688MI0b948n/3sZ1NdXZ2TTz45o0aNSseOHVNVVZUvf/nLGTRoUPbaa6/GLh022MqVK3Pddddl+PDhadHi/36V9RxnU7Vo0aJa/3XD7NmzM2PGjHTs2DHbbbddzjrrrHzrW9/Khz/84fTq1Svf+MY30q1btxxxxBFJkh133DEHHXRQTj311FxzzTVZtmxZzjjjjBxzzDG12g5BY1nXPd61a9d85jOfyeOPP57bb789K1asKP+e3rFjx7Rq1SrTpk3LI488kgMOOCDt27fPtGnTMnLkyHzuc5/Llltu2ViXBbWs6z7v2LFjLr744hx55JHp0qVLZs2alXPPPTd9+vTJ0KFDk3iW0/St7/eV5N//gH/LLbfk8ssvX23/Jv0sL8E6/OAHPyhtt912pVatWpX23HPP0sMPP9zYJcH7kmSNr+uuu65UKpVKc+fOLQ0ePLjUsWPHUmVlZalPnz6lc845p7Rw4cLGLRzq4Oijjy517dq11KpVq9KHPvSh0tFHH1164YUXytvffvvt0pe+9KXSlltuWdpiiy1K//Ef/1F65ZVXGrFiqLu77rqrlKQ0c+bMWuOe42yqJk+evMbfUYYPH14qlUqllStXlr7xjW+UOnfuXKqsrCx94hOfWO3+f+2110qf/exnS+3atStVVVWVTjrppNKbb77ZCFcDq1vXPT579uy1/p4+efLkUqlUKk2fPr00cODAUnV1dal169alHXfcsXTppZeW3nnnnca9MHiXdd3nb731VunAAw8sbbPNNqWWLVuWevToUTr11FNL8+bNq3UMz3KasvX9vlIqlUo//vGPS23atCktWLBgtf2b8rO8olQqlRo8gQcAAAAAgCZEz3EAAAAAAApHOA4AAAAAQOEIxwEAAAAAKBzhOAAAAAAAhSMcBwAAAACgcITjAAAAAAAUjnAcAAAAAIDCEY4DAAAAAFA4wnEAAFiL5557LnvttVdat26d3XbbrbHLWaP9998/Z5111vvad/z48enQoUO91gMAAJsK4TgAAJu8f/7zn2nVqlUWL16cZcuWpW3btpk7d+4HPu6FF16Ytm3bZubMmZk0adJq26+55pq0b98+y5cvL48tWrQoLVu2zP77719r7pQpU1JRUZFZs2Z94Lrqy9FHH52//vWvH/g4pVIp1157bQYOHJh27dqlQ4cO2X333XPllVfmrbfeqodKm6aePXvmyiuvbOwyAAB4n4TjAABs8qZNm5Zdd901bdu2zeOPP56OHTtmu+22+8DHnTVrVvbZZ5/06NEjW2211WrbDzjggCxatCh/+tOfymP3339/unTpkkceeSTvvPNOeXzy5MnZbrvt0rt37zrXUSqVagXw9aVNmzbp1KnTBz7O8ccfn7POOiuHH354Jk+enBkzZuQb3/hGfve73+Xuu++uh0oBAKD+CccBANjkPfTQQ9l7772TJA888ED553VZuXJlLrnkkmy77baprKzMbrvtljvvvLO8vaKiItOnT88ll1ySioqKXHTRRasdo2/fvunatWumTJlSHpsyZUoOP/zw9OrVKw8//HCt8QMOOCBJsmTJkpx55pnp1KlTWrdunX322SePPfZYrbkVFRW54447MmDAgFRWVuaBBx7I4sWLc8IJJ6Rdu3bp2rVrLr/88tVq+uEPf5gPf/jDad26dTp37pzPfOYza/0M3ttW5aKLLspuu+2WX/ziF+nZs2eqq6tzzDHH5M0331zrMW6++ebccMMNuemmm/K1r30te+yxR3r27JnDDz889957b/ma1/d5z5kzJxUVFbn55puz7777pk2bNtljjz3y17/+NY899lh23333tGvXLgcffHD++c9/lvc78cQTc8QRR+TSSy9N586d06FDh1xyySVZvnx5zjnnnHTs2DHbbrttrrvuulp1v/TSSznqqKPSoUOHdOzYMYcffnjmzJmz2nG/+93vpmvXrtlqq60yYsSILFu2LMm/29m8+OKLGTlyZCoqKlJRUbHWzwgAgKZJOA4AwCZp7ty56dChQzp06JDvfe97+fGPf5wOHTrka1/7Wm677bZ06NAhX/rSl9a6/1VXXZXLL7883/3ud/Pkk09m6NChOeyww/L8888nSV555ZXstNNOOfvss/PKK6/kq1/96hqPc8ABB2Ty5Mnl95MnT87++++f/fbbrzz+9ttv55FHHikHxeeee27+93//N9dff30ef/zx9OnTJ0OHDs3rr79e69ijR4/OuHHj8uyzz6Z///4555xzMnXq1PKK7ClTpuTxxx8vz//Tn/6UM888M5dccklmzpyZO++8M4MHD67T5zpr1qzcdtttuf3223P77bdn6tSpGTdu3Frn33DDDenbt28OP/zw1bZVVFSkuro6yfo/71UuvPDCnH/++Xn88cfTokWLHHvssTn33HNz1VVX5f77788LL7yQCy64oNY+9957b15++eXcd999+d73vpcLL7wwhx56aLbccss88sgjOf300/OFL3whf//735Mky5Yty9ChQ9O+ffvcf//9efDBB9OuXbscdNBBWbp0afm4kydPzqxZszJ58uRcf/31GT9+fMaPH58k+e1vf5ttt902l1xySV555ZW88sordfqcAQBoAkoAALAJWrZsWWn27NmlJ554otSyZcvSE088UXrhhRdK7dq1K02dOrU0e/bs0j//+c+17t+tW7fSf/3Xf9Ua22OPPUpf+tKXyu933XXX0oUXXrjOOv7nf/6n1LZt29KyZctKNTU1pRYtWpReffXV0o033lgaPHhwqVQqlSZNmlRKUnrxxRdLixYtKrVs2bJ0ww03lI+xdOnSUrdu3UqXXXZZqVQqlSZPnlxKUrrtttvKc958881Sq1atSjfffHN57LXXXiu1adOm9JWvfKVUKpVK//u//1uqqqoq1dTUrPvD+/9dd911perq6vL7Cy+8sLTFFlvU2v+cc84pDRw4cK3H2HHHHUuHHXbYes+1vs979uzZpSSln/zkJ+XtN910UylJadKkSeWxsWPHlvr27Vt+P3z48FKPHj1KK1asKI/17du3tO+++5bfL1++vNS2bdvSTTfdVCqVSqVf/OIXpb59+5ZWrlxZnrNkyZJSmzZtSnfddVet4y5fvrw85z//8z9LRx99dPl9jx49SldcccV6rx0AgKbJynEAADZJLVq0SM+ePfPcc89ljz32SP/+/TNv3rx07tw5gwcPTs+ePbP11luvcd+ampq8/PLLq7Vf2XvvvfPss8/WqY79998/ixcvzmOPPZb7778/H/nIR7LNNttkv/32K/cdnzJlSrbffvtst912mTVrVpYtW1br3C1btsyee+652rl333338s+zZs3K0qVLM3DgwPJYx44d07dv3/L7T37yk+nRo0e23377HH/88bnhhhvq/IWYPXv2TPv27cvvu3btmldffXWt80ul0nqPWZfPu3///uWfO3funCTZZZddao29t56ddtopzZo1qzXn3fs0b948W221VXm/J554Ii+88ELat2+fdu3apV27dunYsWPeeeedWl+YutNOO6V58+bl9+v7LAAA2LS0aOwCAADg/dhpp53y4osvZtmyZVm5cmXatWuX5cuXZ/ny5WnXrl169OiRZ555psHr6NOnT7bddttMnjw5b7zxRvbbb78kSbdu3dK9e/c89NBDmTx5cj7+8Y/X+dht27at0/z27dvn8ccfz5QpU3L33XfnggsuyEUXXZTHHnusVm/xdWnZsmWt9xUVFVm5cuVa53/kIx/Jc889V6c6N/T8q/p4v3fsvfWsqeZ1XceiRYsyYMCA3HDDDaudf5tttlnncdf1WQAAsGmxchwAgE3SH//4x8yYMSNdunTJL3/5y8yYMSM777xzrrzyysyYMSN//OMf17pvVVVVunXrlgcffLDW+IMPPph+/frVuZYDDjggU6ZMyZQpU7L//vuXxwcPHpw77rgjjz76aLnfeO/evdOqVata5162bFkee+yxdZ67d+/eadmyZR555JHy2BtvvJG//vWvtea1aNEiQ4YMyWWXXZYnn3wyc+bMyb333lvna9pQxx57bP7617/md7/73WrbSqVSFi5cWO+f9wf1sY99LM8//3w6deqUPn361Hqt6pG+IVq1apUVK1Y0YKUAADQk4TjA/9feHYMkE8dhHH+EBikThZaoMKgjqTXnkIKmEAqMiEKQiBYd5KgxhKJoa6vBoUFpaQiHWkTU0eGGCprCoaFokYvKBnuHF4SGN4wK3rjvZz3u/r/7T8fD8fwBAL9SIBCQx+PR3d2dIpGIBgYGdHl5qbm5OQ0PDysQCHx4v2ma2t3d1fHxsa6vr7WxsSHLspRMJj89SzgcVqVSkWVZrT/HJWliYkIHBwd6fX1theNdXV1aW1uTaZo6OzvT1dWVVlZW9PT0pHg8/s81PB6P4vG4TNNUoVDQxcWFYrHYuzqRfD6v/f19WZalWq2mo6MjNZvNd9Ur3y0ajWp+fl4LCwva3t5WtVpVrVZTPp/X1NRU61DS79zvr1pcXFRPT48ikYjK5bJubm5ULBaVSCRah3a2Y3BwUKVSSbe3t3p4ePjBiQEAAPATqFUBAADAr1UsFhUKheR2u1Uul9Xf36/e3t627k0kEqrX60qlUrq/v9fo6KhOT09lGMan5wiHw3p+flYwGGz1ZEt/w3HbtjUyMvJurp2dHTWbTS0tLcm2bY2Pj+v8/Fx+v//Ddfb29vT4+KiZmRl1d3crlUqpXq+3rvt8Pp2cnGhzc1MvLy8yDEO5XE5jY2Offqd2uVwuZbNZHR4eKpPJaGtrSx0dHTIMQ8vLy5qenpb0vfv9VZ2dnSqVSlpfX9fs7Kxs21ZfX58mJyfl9Xrbfk46ndbq6qqGhobUaDTa6l8HAADA/8P1xhccAAAAAAAAAMBhqFUBAAAAAAAAADgO4TgAAAAAAAAAwHEIxwEAAAAAAAAAjkM4DgAAAAAAAABwHMJxAAAAAAAAAIDjEI4DAAAAAAAAAByHcBwAAAAAAAAA4DiE4wAAAAAAAAAAxyEcBwAAAAAAAAA4DuE4AAAAAAAAAMBxCMcBAAAAAAAAAI7zB7O16MLa6blvAAAAAElFTkSuQmCC","text/plain":["<Figure size 1800x800 with 1 Axes>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["The maximum length of Speeches is 1411 words.\n"]}],"source":["data['text_length'] = data['text'].str.split().str.len()\n","\n","n_bin=170\n","data.hist('text_length', grid=False, figsize=(18,8), range=(0,n_bin + 1), bins=(n_bin + 1))\n","plt.title('Histogram: Length of Comment')\n","plt.xlabel('# of Words in Comment')\n","plt.ylabel('# of Samples')\n","plt.show()\n","\n","print(f\"The maximum length of Speeches is {max(data['text_length'])} words.\")"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"Z5LLbXellof3"},"source":["#### Visual: Tweet Length by Class"]},{"cell_type":"code","execution_count":6,"metadata":{"id":"kZduaMjslof3","outputId":"3181f2e3-86f5-4979-e7e7-e9afb51b1433"},"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAA2sAAAKaCAYAAACgDMGYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABDfElEQVR4nO3deZyVdd3/8fcAMrINiIqAIiAooAIqJD9yA9QUTbPsdsPANSs1k1Za3FowScMF9bZMMzW7NZfUsocLSCZZYmR1I4GCUoKaBCOLCMz5/UHM7QQog7NcOM/n43Eew/le1znX5wz20FfXOdcpK5VKpQAAAFAozRp7AAAAANYn1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAYAsxb968lJWV5Xvf+95mPf7mm29OWVlZnn766TqeDID6INYACu7555/PWWedlV122SVbb711Kioqst9+++XKK6/MihUrGnu8QvjlL3+Ziy66aJP3HzZsWPbcc8/6G+g9qu3rKYopU6akrKxsk24AvLsWjT0AABv34IMP5r/+679SXl6e0aNHZ88998xbb72VJ554Il/84hfz17/+NTfccENjj9nofvnLX2bSpElbZOBsyJb6evr165ef/OQnNdbGjRuXtm3b5mtf+1ojTQWw5RJrAAU1d+7cnHDCCenevXsee+yxdOnSpXrb2WefnTlz5uTBBx9sxAmhph122CEnn3xyjbVLL70022233XrrALw7b4MEKKjLLrssS5cuzY033lgj1Nbp3bt3zjvvvOr7q1evzje/+c306tUr5eXl6dGjR7761a9m5cqVNR7Xo0ePfPjDH86UKVMyePDgtGrVKv3798+UKVOSJHfffXf69++frbfeOoMGDcof//jHGo8/5ZRT0rZt27z00kv58Ic/nLZt22bHHXfMpEmTkiR//vOfM2LEiLRp0ybdu3fP7bffvt7sixcvzuc+97l069Yt5eXl6d27d7773e+mqqqqep+3fz7rhhtuqH5dH/jAB/KHP/yhxjzrjl3Xb7P71a9+lQMOOCBt2rRJu3btcuSRR+avf/3rBn8f//jHP3LMMcekbdu22X777fOFL3wha9asqbHv66+/nk984hOpqKhIhw4dMmbMmPzpT39KWVlZbr755lq9nnf6nbyb5cuX56yzzsq2226bioqKjB49Ov/617+qt48ZMybbbbddVq1atd5jP/ShD6VPnz6bfKy3K5VK6dGjRz7ykY+st+3NN99M+/btc9ZZZyX5v7dU/uxnP8tXv/rVdO7cOW3atMnRRx+d+fPnr/f4p556Kocffnjat2+f1q1b56CDDspvf/vbzZoToDBKABTSjjvuWNpll102ef8xY8aUkpQ+/vGPlyZNmlQaPXp0KUnpmGOOqbFf9+7dS3369Cl16dKldNFFF5W+//3vl3bcccdS27ZtS7feemtp5513Ll166aWlSy+9tNS+fftS7969S2vWrKlxnK233rq0++67lz71qU+VJk2aVPrgBz9YSlK66aabSl27di198YtfLF199dWlPfbYo9S8efPSCy+8UP34ZcuWlQYMGFDadtttS1/96ldL119/fWn06NGlsrKy0nnnnVe939y5c0tJSnvvvXepd+/epe9+97ulyy67rLTddtuVdtppp9Jbb71VKpVKpSeffLJ06KGHlpKUfvKTn1Tf3slBBx1U2mOPPd5xn1tuuaVUVlZWOvzww0tXX3116bvf/W6pR48epQ4dOpTmzp273u9jjz32KJ122mml6667rnTssceWkpSuvfba6v3WrFlTGjp0aKl58+alc845p3TNNdeUDj300NLAgQOrf3fv9no29XeyMTfddFMpSal///6lAw44oHTVVVeVzj777FKzZs1KBx54YKmqqqpUKpVKDz/8cClJ6f7776/x+AULFpSaN29euuSSS97xOG+3xx57lA466KDq+1/72tdKW221Ven111+vsd///M//lJKUpk6dWiqVSqXJkydXzzpgwIDSFVdcUfrKV75S2nrrrUu77bZbafny5dWPffTRR0stW7YsDR06tHT55ZeXvv/975cGDBhQatmyZempp57a5FkBikasARTQkiVLSklKH/nIRzZp/xkzZpSSlM4444wa61/4whdKSUqPPfZY9Vr37t1LSUpPPvlk9dqvf/3rUpJSq1atSi+++GL1+n//93+XkpQmT55cvbYuCr/zne9Ur/3rX/8qtWrVqlRWVla64447qtefe+65UpLShRdeWL32zW9+s9SmTZvS3/72txqzfuUrXyk1b9689NJLL5VKpf8Lk2233ba0aNGi6v3uu+++9ULi7LPPLtXm/398t1h74403Sh06dCideeaZNdYXLlxYat++fY31db+P/wyYvffeuzRo0KDq+z//+c9LSUoTJ06sXluzZk1pxIgRNWLtnV5PbX4nG7Iu1gYNGlQj7C677LJSktJ9991XPddOO+1UOv7442s8/oorriiVlZXViO9385+xNmvWrFKS0nXXXVdjv6OPPrrUo0eP6mBcF2s77rhjqbKysnq/dVF35ZVXlkqlUqmqqqq06667lg477LDqx5ZKpdLy5ctLPXv2LB166KGbPCtA0XgbJEABVVZWJknatWu3Sfv/8pe/TJKMHTu2xvrnP//5JFnvs2277757hg4dWn1/yJAhSZIRI0Zk5513Xm/9hRdeWO+YZ5xxRvWfO3TokD59+qRNmzY57rjjqtf79OmTDh061Hj8nXfemQMOOCDbbLNN/vnPf1bfDjnkkKxZsyZTp06tcZzjjz8+22yzTfX9Aw44YKMz1ZWHH344ixcvzoknnlhjxubNm2fIkCGZPHnyeo/51Kc+VeP+AQccUGPGhx56KFtttVXOPPPM6rVmzZrl7LPPrvV87/V38slPfjJbbbVV9f1Pf/rTadGiRfU/R82aNcuoUaPyi1/8Im+88Ub1frfddls++MEPpmfPnrWeeZ3ddtstQ4YMyW233Va9tmjRovzqV7/KqFGj1nvL5+jRo2v87+DjH/94unTpUj3rjBkzMnv27Jx00kl5/fXXq/+uli1bloMPPjhTp06t8fZagC2JC4wAFFBFRUWS1PgP5Xfy4osvplmzZundu3eN9c6dO6dDhw558cUXa6y/PciSpH379kmSbt26bXD97Z9nSpKtt94622+//Xr77rTTTuv9x3b79u1rPH727Nl59tln13v8Oq+++uo7zrouUv5zpro0e/bsJGvjdUPW/f2ss6HfxzbbbFNjxhdffDFdunRJ69ata+z3n39nm+K9/k523XXXGvfbtm2bLl26ZN68edVro0ePzne/+93cc889GT16dGbNmpXp06fn+uuvr/W8/2n06NE555xz8uKLL6Z79+658847s2rVqnziE59411nLysrSu3fv6lnX/V2NGTNmo8dbsmRJjbgF2FKINYACqqioSNeuXfOXv/ylVo/b1AtrNG/evFbrpVKpzh5fVVWVQw89NF/60pc2uO9uu+22WTPVpXVnYn7yk5+kc+fO621v0aLmvz43NmN9aYjfye67755Bgwbl1ltvzejRo3PrrbemZcuWNc6cbq4TTjgh559/fm677bZ89atfza233prBgwdv1oVL1v1dTZgwIXvttdcG92nbtu17GReg0Yg1gIL68Ic/nBtuuCHTpk2r8ZbFDenevXuqqqoye/bs9OvXr3r9lVdeyeLFi9O9e/f6HneT9erVK0uXLs0hhxxSZ89Z11+y3KtXryRJp06d6mzO7t27Z/LkyVm+fHmNs2tz5sxZb9/6/tLo2bNnZ/jw4dX3ly5dmgULFuSII46osd/o0aMzduzYLFiwILfffnuOPPLIOjlD1bFjxxx55JG57bbbMmrUqPz2t7/NxIkTNzrr25VKpcyZMycDBgxI8n9/VxUVFXX6zxRAEfjMGkBBfelLX0qbNm1yxhln5JVXXllv+/PPP58rr7wySar/I/s//4P3iiuuSJIceeSR9TtsLRx33HGZNm1afv3rX6+3bfHixVm9enWtn7NNmzbVj68Lhx12WCoqKvKd73xng5evf+211zbrOVetWpUf/OAH1WtVVVXVl+l/u7p+Pf/phhtuqPG6rrvuuqxevTojR46ssd+JJ56YsrKynHfeeXnhhRfq9LvSPvGJT+R///d/88UvfjHNmzfPCSecsMH9brnllhpvB77rrruyYMGC6lkHDRqUXr165Xvf+16WLl263uM35+8KoCicWQMoqF69euX222/P8ccfn379+mX06NHZc88989Zbb+XJJ5/MnXfemVNOOSVJMnDgwIwZMyY33HBDFi9enIMOOii///3v8+Mf/zjHHHNMjbMoje2LX/xifvGLX+TDH/5wTjnllAwaNCjLli3Ln//859x1112ZN29etttuu1o956BBg5Ikn/3sZ3PYYYe943/8r/Paa6/lW9/61nrrPXv2zKhRo3LdddflE5/4RPbZZ5+ccMIJ2X777fPSSy/lwQcfzH777ZdrrrmmVjMec8wx2XffffP5z38+c+bMSd++ffOLX/wiixYtSlLzbNrmvJ7aeOutt3LwwQfnuOOOy6xZs3Lttddm//33z9FHH11jv+233z6HH3547rzzznTo0KFOo//II4/MtttumzvvvDMjR45Mp06dNrhfx44ds//+++fUU0/NK6+8kokTJ6Z3797VF2pp1qxZfvjDH2bkyJHZY489cuqpp2bHHXfMP/7xj0yePDkVFRW5//7762xugIYk1gAK7Oijj86zzz6bCRMm5L777st1112X8vLyDBgwIJdffnmNKwv+8Ic/zC677JKbb74599xzTzp37pxx48blwgsvbMRXsL7WrVvn8ccfz3e+853ceeedueWWW1JRUZHddtstF198cfVFTWrjYx/7WM4999zccccdufXWW1Mqld41bl599dV84xvfWG/94IMPzqhRo3LSSSela9euufTSSzNhwoSsXLkyO+64Yw444ICceuqptZ6xefPmefDBB3Peeeflxz/+cZo1a5aPfvSjufDCC7Pffvtl6623fk+vpzauueaa3HbbbbnggguyatWqnHjiibnqqqs2+PbL0aNH54EHHshxxx2X8vLyOpuhZcuWOf7443Pttddu8MIi63z1q1/Ns88+m/Hjx+eNN97IwQcfnGuvvbbGW0mHDRuWadOm5Zvf/GauueaaLF26NJ07d86QIUOqv2QbYEtUVqrPT2gDAO/o3nvvzUc/+tE88cQT2W+//Rp7nPXcd999OeaYYzJ16tTqrwioK+eff35uvPHGLFy4cL2rZE6ZMiXDhw/PnXfemY9//ON1elyALYXPrAFAA1mxYkWN+2vWrMnVV1+dioqK7LPPPo001Tv7wQ9+kF122SX7779/nT7vm2++mVtvvTXHHnvseqEGwFreBgkADeTcc8/NihUrMnTo0KxcuTJ33313nnzyyXznO99Jq1atGnu8Gu644448++yzefDBB3PllVfW2RUqX3311TzyyCO566678vrrr+e8886rk+cFeD8SawDQQEaMGJHLL788DzzwQN5888307t07V199dc4555zGHm09J554Ytq2bZvTTz89n/nMZ+rsef/3f/83o0aNSqdOnXLVVVdt9LvRAPCZNQAAgEJq1M+sTZ06NUcddVS6du2asrKy3HvvvTW2l5WVbfA2YcKE6n169Oix3vZLL720gV8JAABA3WrUWFu2bFkGDhy4wS8ETZIFCxbUuP3oRz9KWVlZjj322Br7XXLJJTX2O/fccxtifAAAgHrTqJ9ZGzlyZEaOHLnR7Z07d65x/7777svw4cOzyy671Fhv167devvWRlVVVV5++eW0a9euzj5ADQAAsCGlUilvvPFGunbtmmbNNn7+bIu5wMgrr7ySBx98MD/+8Y/X23bppZfmm9/8ZnbeeeecdNJJOf/889OixcZf2sqVK7Ny5crq+//4xz+y++6718vcAAAAGzJ//vzstNNOG92+xcTaj3/847Rr1y4f+9jHaqx/9rOfzT777JOOHTvmySefzLhx47JgwYJcccUVG32u8ePH5+KLL15vff78+amoqKjz2QEAANaprKxMt27d0q5du3fcrzBXgywrK8s999yTY445ZoPb+/btm0MPPTRXX331Oz7Pj370o5x11llZunRpysvLN7jPf55ZW/fLWrJkiVgDAADqVWVlZdq3b/+u/bFFnFn7zW9+k1mzZuVnP/vZu+47ZMiQrF69OvPmzUufPn02uE95eflGQw4AAKAIGvVqkJvqxhtvzKBBgzJw4MB33XfGjBlp1qxZOnXq1ACTAQAA1I9GPbO2dOnSzJkzp/r+3LlzM2PGjHTs2DE777xzkrWnCO+8885cfvnl6z1+2rRpeeqppzJ8+PC0a9cu06ZNy/nnn5+TTz4522yzTYO9DgAAgLrWqLH29NNPZ/jw4dX3x44dmyQZM2ZMbr755iTJHXfckVKplBNPPHG9x5eXl+eOO+7IRRddlJUrV6Znz545//zzq58HAABgS1WYC4w0pk39gB8AAMB7tan9sUV8Zg0AAKCpEWsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAAqoRWMPANS/5cuT557bvMeuWJHMm5f06JG0arV5z9G3b9K69eY9FgCgqRJr0AQ891wyaFDjHX/69GSffRrv+AAAWyKxBk1A375rg2lzzJyZnHxycuutSb9+m398AABqR6xBE9C69Xs/s9Wvn7NjAAANyQVGAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABdSosTZ16tQcddRR6dq1a8rKynLvvffW2H7KKaekrKysxu3www+vsc+iRYsyatSoVFRUpEOHDjn99NOzdOnSBnwVAAAAda9RY23ZsmUZOHBgJk2atNF9Dj/88CxYsKD69tOf/rTG9lGjRuWvf/1rHn744TzwwAOZOnVqPvnJT9b36AAAAPWqRWMefOTIkRk5cuQ77lNeXp7OnTtvcNvMmTPz0EMP5Q9/+EMGDx6cJLn66qtzxBFH5Hvf+166du1a5zMDAAA0hMJ/Zm3KlCnp1KlT+vTpk09/+tN5/fXXq7dNmzYtHTp0qA61JDnkkEPSrFmzPPXUUxt9zpUrV6aysrLGDQAAoEgKHWuHH354brnlljz66KP57ne/m8cffzwjR47MmjVrkiQLFy5Mp06dajymRYsW6dixYxYuXLjR5x0/fnzat29ffevWrVu9vg4AAIDaatS3Qb6bE044ofrP/fv3z4ABA9KrV69MmTIlBx988GY/77hx4zJ27Njq+5WVlYINAAAolEKfWftPu+yyS7bbbrvMmTMnSdK5c+e8+uqrNfZZvXp1Fi1atNHPuSVrPwdXUVFR4wYAAFAkW1Ss/f3vf8/rr7+eLl26JEmGDh2axYsXZ/r06dX7PPbYY6mqqsqQIUMaa0wAAID3rFHfBrl06dLqs2RJMnfu3MyYMSMdO3ZMx44dc/HFF+fYY49N586d8/zzz+dLX/pSevfuncMOOyxJ0q9fvxx++OE588wzc/3112fVqlU555xzcsIJJ7gSJAAAsEVr1DNrTz/9dPbee+/svffeSZKxY8dm7733zgUXXJDmzZvn2WefzdFHH53ddtstp59+egYNGpTf/OY3KS8vr36O2267LX379s3BBx+cI444Ivvvv39uuOGGxnpJAAAAdaJRz6wNGzYspVJpo9t//etfv+tzdOzYMbfffntdjgUAANDotqjPrAEAADQVYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABRQo8ba1KlTc9RRR6Vr164pKyvLvffeW71t1apV+fKXv5z+/funTZs26dq1a0aPHp2XX365xnP06NEjZWVlNW6XXnppA78SAACAutWosbZs2bIMHDgwkyZNWm/b8uXL88wzz+Qb3/hGnnnmmdx9992ZNWtWjj766PX2veSSS7JgwYLq27nnntsQ4wMAANSbFo158JEjR2bkyJEb3Na+ffs8/PDDNdauueaa7LvvvnnppZey8847V6+3a9cunTt3rtdZAQAAGtIW9Zm1JUuWpKysLB06dKixfumll2bbbbfN3nvvnQkTJmT16tWNMyAAAEAdadQza7Xx5ptv5stf/nJOPPHEVFRUVK9/9rOfzT777JOOHTvmySefzLhx47JgwYJcccUVG32ulStXZuXKldX3Kysr63V2AACA2toiYm3VqlU57rjjUiqVct1119XYNnbs2Oo/DxgwIC1btsxZZ52V8ePHp7y8fIPPN378+Fx88cX1OjMAAMB7Ufi3Qa4LtRdffDEPP/xwjbNqGzJkyJCsXr068+bN2+g+48aNy5IlS6pv8+fPr+OpAQAA3ptCn1lbF2qzZ8/O5MmTs+22277rY2bMmJFmzZqlU6dOG92nvLx8o2fdAAAAiqBRY23p0qWZM2dO9f25c+dmxowZ6dixY7p06ZKPf/zjeeaZZ/LAAw9kzZo1WbhwYZKkY8eOadmyZaZNm5annnoqw4cPT7t27TJt2rScf/75Ofnkk7PNNts01ssCAAB4zxo11p5++ukMHz68+v66z5+NGTMmF110UX7xi18kSfbaa68aj5s8eXKGDRuW8vLy3HHHHbnooouycuXK9OzZM+eff36Nz7EBAABsiRo11oYNG5ZSqbTR7e+0LUn22Wef/O53v6vrsQAAABpd4S8wAgAA0BSJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACqnWsXXLJJVm+fPl66ytWrMgll1xSJ0MBAAA0dbWOtYsvvjhLly5db3358uW5+OKL62QoAACApq7WsVYqlVJWVrbe+p/+9Kd07NixToYCAABo6lps6o7bbLNNysrKUlZWlt12261GsK1ZsyZLly7Npz71qXoZEgAAoKnZ5FibOHFiSqVSTjvttFx88cVp37599baWLVumR48eGTp0aL0MCQAA0NRscqyNGTMmSdKzZ8988IMfzFZbbVVvQwEAADR1mxxr6xx00EGpqqrK3/72t7z66qupqqqqsf3AAw+ss+EAAACaqlrH2u9+97ucdNJJefHFF1MqlWpsKysry5o1a+psOAAAgKaq1rH2qU99KoMHD86DDz6YLl26bPDKkAAAALw3tY612bNn56677krv3r3rYx4AAACyGd+zNmTIkMyZM6c+ZgEAAODfan1m7dxzz83nP//5LFy4MP3791/vqpADBgyos+EAAACaqlrH2rHHHpskOe2006rXysrKUiqVXGAEAACgjtT6bZBz585d7/bCCy9U/6yNqVOn5qijjkrXrl1TVlaWe++9t8b2UqmUCy64IF26dEmrVq1yyCGHZPbs2TX2WbRoUUaNGpWKiop06NAhp59+epYuXVrblwUAAFAotY617t27v+OtNpYtW5aBAwdm0qRJG9x+2WWX5aqrrsr111+fp556Km3atMlhhx2WN998s3qfUaNG5a9//WsefvjhPPDAA5k6dWo++clP1vZlAQAAFEqtYy1JfvKTn2S//fZL165d8+KLLyZJJk6cmPvuu69WzzNy5Mh861vfykc/+tH1tpVKpUycODFf//rX85GPfCQDBgzILbfckpdffrn6DNzMmTPz0EMP5Yc//GGGDBmS/fffP1dffXXuuOOOvPzyy5vz0gAAAAqh1rF23XXXZezYsTniiCOyePHi6s+odejQIRMnTqyzwebOnZuFCxfmkEMOqV5r3759hgwZkmnTpiVJpk2blg4dOmTw4MHV+xxyyCFp1qxZnnrqqTqbBQAAoKHVOtauvvrq/OAHP8jXvva1NG/evHp98ODB+fOf/1xngy1cuDBJssMOO9RY32GHHaq3LVy4MJ06daqxvUWLFunYsWP1PhuycuXKVFZW1rgBAAAUyWZdYGTvvfdeb728vDzLli2rk6Hq2/jx49O+ffvqW7du3Rp7JAAAgBpqHWs9e/bMjBkz1lt/6KGH0q9fv7qYKUnSuXPnJMkrr7xSY/2VV16p3ta5c+e8+uqrNbavXr06ixYtqt5nQ8aNG5clS5ZU3+bPn19ncwMAANSFWn/P2tixY3P22WfnzTffTKlUyu9///v89Kc/zfjx4/PDH/6wzgbr2bNnOnfunEcffTR77bVXkqSysjJPPfVUPv3pTydJhg4dmsWLF2f69OkZNGhQkuSxxx5LVVVVhgwZstHnLi8vT3l5eZ3NCgAAUNdqHWtnnHFGWrVqla9//etZvnx5TjrppHTt2jVXXnllTjjhhFo919KlSzNnzpzq+3Pnzs2MGTPSsWPH7Lzzzvnc5z6Xb33rW9l1113Ts2fPfOMb30jXrl1zzDHHJEn69euXww8/PGeeeWauv/76rFq1Kuecc05OOOGEdO3atbYvDQAAoDDKSqVSaXMfvHz58ixdunS9i3xsqilTpmT48OHrrY8ZMyY333xzSqVSLrzwwtxwww1ZvHhx9t9//1x77bXZbbfdqvddtGhRzjnnnNx///1p1qxZjj322Fx11VVp27btJs9RWVmZ9u3bZ8mSJamoqNis1wLvV888kwwalEyfnuyzT2NPAwCw5dvU/nhPsfZ+IdZg48QaAEDd2tT+qPXbIF9//fVccMEFmTx5cl599dVUVVXV2L5o0aLaTwsAAEANtY61T3ziE5kzZ05OP/307LDDDikrK6uPuQAAAJq0Wsfab37zmzzxxBMZOHBgfcwDAABANuN71vr27ZsVK1bUxywAAAD8W61j7dprr83Xvva1PP7443n99ddTWVlZ4wYAAMB7V+u3QXbo0CGVlZUZMWJEjfVSqZSysrKsWbOmzoYDAABoqmoda6NGjcpWW22V22+/3QVGAAAA6kmtY+0vf/lL/vjHP6ZPnz71MQ8AAADZjM+sDR48OPPnz6+PWQAAAPi3Wp9ZO/fcc3Peeefli1/8Yvr375+tttqqxvYBAwbU2XAAAABNVa1j7fjjj0+SnHbaadVrZWVlLjACAABQh2oda3Pnzq2POQAAAHibWsda9+7d62MOAAAA3qbWsZYkL7/8cp544om8+uqrqaqqqrHts5/9bJ0MBgAA0JTVOtZuvvnmnHXWWWnZsmW23XbbGt+zVlZWJtYAAADqQK1j7Rvf+EYuuOCCjBs3Ls2a1frK/wAAAGyCWtfW8uXLc8IJJwg1AACAelTr4jr99NNz55131scsAAAA/Fut3wY5fvz4fPjDH85DDz20wS/FvuKKK+psOAAAgKZqs2Lt17/+dfr06ZMk611gBACA94fly5Pnntu8x65Ykcybl/TokbRqtXnP0bdv0rr15j0W3g9qHWuXX355fvSjH+WUU06ph3EAACiK555LBg1qvONPn57ss0/jHR8aW61jrby8PPvtt199zAIAQIH07bs2mDbHzJnJyScnt96a9Ou3+ceHpqzWsXbeeefl6quvzlVXXVUf8wAAUBCtW7/3M1v9+jk7Bpur1rH2+9//Po899lgeeOCB7LHHHutdYOTuu++us+EAAACaqlrHWocOHfKxj32sPmYBAADg32odazfddFN9zAEAAMDb1DrW1nnttdcya9asJEmfPn2y/fbb19lQAAAATV2z2j5g2bJlOe2009KlS5cceOCBOfDAA9O1a9ecfvrpWb58eX3MCAAA0OTUOtbGjh2bxx9/PPfff38WL16cxYsX57777svjjz+ez3/+8/UxIwAAQJNT67dB/vznP89dd92VYcOGVa8dccQRadWqVY477rhcd911dTkfAABAk1TrM2vLly/PDjvssN56p06dvA0SAACgjtQ61oYOHZoLL7wwb775ZvXaihUrcvHFF2fo0KF1OhwAAEBTVeu3QV555ZU57LDDstNOO2XgwIFJkj/96U/Zeuut8+tf/7rOBwQAAGiKah1re+65Z2bPnp3bbrstzz33XJLkxBNPzKhRo9KqVas6HxAAAKAp2qzvWWvdunXOPPPMup4FAACAf9vkz6xNnz49w4cPT2Vl5XrblixZkuHDh+dPf/pTnQ4HAADQVG1yrF1++eUZMWJEKioq1tvWvn37HHrooZkwYUKdDgcAANBUbXKsPfXUU/nIRz6y0e1HHXVUnnzyyToZCgAAoKnb5Fj7xz/+kXbt2m10e9u2bbNgwYI6GQoAAKCp2+RY23777TNr1qyNbn/uueey3Xbb1clQAAAATd0mx9ohhxySb3/72xvcViqV8u1vfzuHHHJInQ0GAADQlG3ypfu//vWvZ9CgQRkyZEg+//nPp0+fPknWnlG7/PLL87e//S0333xzfc0JAADQpGxyrPXq1SuPPPJITjnllJxwwgkpKytLsvas2u67756HH344vXv3rrdBAQAAmpJafSn24MGD85e//CUzZszI7NmzUyqVsttuu2Wvvfaqp/EAAACaplrF2jp77bWXQAMAAKhHm3yBEQAAABqOWAMAACggsQYAAFBAtYq11atX55JLLsnf//73+poHAACA1DLWWrRokQkTJmT16tX1NQ8AAADZjLdBjhgxIo8//nh9zAIAAMC/1frS/SNHjsxXvvKV/PnPf86gQYPSpk2bGtuPPvroOhsOAACgqap1rH3mM59JklxxxRXrbSsrK8uaNWve+1QAAABNXK1jraqqqj7mAAAA4G3e06X733zzzbqaAwAAgLepdaytWbMm3/zmN7Pjjjumbdu2eeGFF5Ik3/jGN3LjjTfW+YAAAABNUa1j7dvf/nZuvvnmXHbZZWnZsmX1+p577pkf/vCHdTocAABAU1XrWLvllltyww03ZNSoUWnevHn1+sCBA/Pcc8/V6XAAAABNVa1j7R//+Ed69+693npVVVVWrVpVJ0MBAAA0dbWOtd133z2/+c1v1lu/6667svfee9fJUAAAAE1drS/df8EFF2TMmDH5xz/+kaqqqtx9992ZNWtWbrnlljzwwAP1MSMAAECTU+szax/5yEdy//3355FHHkmbNm1ywQUXZObMmbn//vtz6KGH1seMAAAATU6tz6wlyQEHHJCHH364rmcBAADg3zYr1pLk6aefzsyZM5Os/RzboEGD6mwoAACApq7Wsfb3v/89J554Yn7729+mQ4cOSZLFixfngx/8YO64447stNNOdT0jAABAk1Prz6ydccYZWbVqVWbOnJlFixZl0aJFmTlzZqqqqnLGGWfUx4wAAABNTq3PrD3++ON58skn06dPn+q1Pn365Oqrr84BBxxQp8MBAAA0VbU+s9atW7cNfvn1mjVr0rVr1zoZCgAAoKmr9Zm1CRMm5Nxzz82kSZMyePDgJGsvNnLeeefle9/7Xp0PCPyf2bOTN95o2GP++zpC1T8bUrt2ya67NvxxAQCKoKxUKpVq84Btttkmy5cvz+rVq9OixdrWW/fnNm3a1Nh30aJFdTdpPaqsrEz79u2zZMmSVFRUNPY4sEGzZye77dbYUzS8v/1NsAFsiZ55Jhk0KJk+Pdlnn8aeBoplU/uj1mfWJk6c+F7mAjbTujNqt96a9OvXcMddsSKZNy/p0SNp1arhjjtzZnLyyQ1/JhEAoChqHWtjxoypjzk2qkePHnnxxRfXW//MZz6TSZMmZdiwYXn88cdrbDvrrLNy/fXXN9SI0KD69Wv4/4dyv/0a9ngAALyHL8VuKH/4wx+yZs2a6vt/+ctfcuihh+a//uu/qtfOPPPMXHLJJdX3W7du3aAzAgAA1LXCx9r2229f4/6ll16aXr165aCDDqpea926dTp37tzQowEAANSbWl+6vzG99dZbufXWW3PaaaelrKysev22227Ldtttlz333DPjxo3L8uXLG3FKAACA967wZ9be7t57783ixYtzyimnVK+ddNJJ6d69e7p27Zpnn302X/7ylzNr1qzcfffdG32elStXZuXKldX3Kysr63NsAACAWnvPsVZZWZnHHnssffr0Sb96vkTdjTfemJEjR9b48u1PfvKT1X/u379/unTpkoMPPjjPP/98evXqtcHnGT9+fC6++OJ6nRUAAOC9qPXbII877rhcc801SZIVK1Zk8ODBOe644zJgwID8/Oc/r/MB13nxxRfzyCOP5IwzznjH/YYMGZIkmTNnzkb3GTduXJYsWVJ9mz9/fp3OCgAA8F7VOtamTp2aAw44IElyzz33pFQqZfHixbnqqqvyrW99q84HXOemm25Kp06dcuSRR77jfjNmzEiSdOnSZaP7lJeXp6KiosYNAACgSGoda0uWLEnHjh2TJA899FCOPfbYtG7dOkceeWRmz55d5wMmSVVVVW666aaMGTMmLVr83zs3n3/++Xzzm9/M9OnTM2/evPziF7/I6NGjc+CBB2bAgAH1MgsAAEBDqPVn1rp165Zp06alY8eOeeihh3LHHXckSf71r39l6623rvMBk+SRRx7JSy+9lNNOO63GesuWLfPII49k4sSJWbZsWbp165Zjjz02X//61+tlDgAAgIZS61j73Oc+l1GjRqVt27bp3r17hg0blmTt2yP79+9f1/MlST70oQ+lVCqtt96tW7c8/vjj9XJMAACAxlTrWPvMZz6TfffdN/Pnz8+hhx6aZs3WvpNyl112qdfPrAEAADQlm3Xp/sGDB2fw4ME11t7twh8AAABsuk2KtbFjx27yE15xxRWbPQwAAABrbVKs/fGPf6xx/5lnnsnq1avTp0+fJMnf/va3NG/ePIMGDar7CQEAAJqgTYq1yZMnV//5iiuuSLt27fLjH/8422yzTZK1V4I89dRTq79/DQAAgPem1t+zdvnll2f8+PHVoZYk22yzTb71rW/l8ssvr9PhAAAAmqpax1plZWVee+219dZfe+21vPHGG3UyFAAAQFNX61j76Ec/mlNPPTV33313/v73v+fvf/97fv7zn+f000/Pxz72sfqYEQAAoMmp9aX7r7/++nzhC1/ISSedlFWrVq19khYtcvrpp2fChAl1PiAAAEBTVKtYW7NmTZ5++ul8+9vfzoQJE/L8888nSXr16pU2bdrUy4AAAABNUa1irXnz5vnQhz6UmTNnpmfPnhkwYEB9zQUAANCk1foza3vuuWdeeOGF+pgFAACAf6t1rH3rW9/KF77whTzwwANZsGBBKisra9wAAAB472p9gZEjjjgiSXL00UenrKyser1UKqWsrCxr1qypu+kAAACaqFrH2uTJk+tjDgAAAN6m1rF20EEH1cccAAAAvE2tYy1JFi9enBtvvDEzZ85Mkuyxxx457bTT0r59+zodDgAAoKmq9QVGnn766fTq1Svf//73s2jRoixatChXXHFFevXqlWeeeaY+ZgQAAGhyan1m7fzzz8/RRx+dH/zgB2nRYu3DV69enTPOOCOf+9znMnXq1DofEgAAoKmpdaw9/fTTNUItSVq0aJEvfelLGTx4cJ0OBwAA0FTV+m2QFRUVeemll9Zbnz9/ftq1a1cnQwEAADR1tY61448/Pqeffnp+9rOfZf78+Zk/f37uuOOOnHHGGTnxxBPrY0YAAIAmp9Zvg/ze976XsrKyjB49OqtXr06SbLXVVvn0pz+dSy+9tM4HBAAAaIo2Odbmzp2bnj17pmXLlrnyyiszfvz4PP/880mSXr16pXXr1vU2JAAAQFOzybHWq1evdO/ePcOHD8+IESMyfPjw9O/fvz5nAwAAaLI2OdYee+yxTJkyJVOmTMlPf/rTvPXWW9lll12qw2348OHZYYcd6nNWAACAJmOTY23YsGEZNmxYkuTNN9/Mk08+WR1vP/7xj7Nq1ar07ds3f/3rX+trVgAAgCaj1hcYSZKtt946I0aMyP7775/hw4fnV7/6Vf77v/87zz33XF3PBwAA0CTVKtbeeuut/O53v8vkyZMzZcqUPPXUU+nWrVsOPPDAXHPNNTnooIPqa04AAIAmZZNjbcSIEXnqqafSs2fPHHTQQTnrrLNy++23p0uXLvU5HwAAQJO0ybH2m9/8Jl26dMmIESMybNiwHHTQQdl2223rczYAAIAmq9mm7rh48eLccMMNad26db773e+ma9eu6d+/f84555zcddddee211+pzTgAAgCZlk8+stWnTJocffngOP/zwJMkbb7yRJ554IpMnT85ll12WUaNGZdddd81f/vKXehsWAACgqdjkM2v/qU2bNunYsWM6duyYbbbZJi1atMjMmTPrcjYAAIAma5PPrFVVVeXpp5/OlClTMnny5Pz2t7/NsmXLsuOOO2b48OGZNGlShg8fXp+zAgAANBmbHGsdOnTIsmXL0rlz5wwfPjzf//73M2zYsPTq1as+5wMAAGiSNjnWJkyYkOHDh2e33Xarz3kAAABILWLtrLPOqs85AAAAeJvNvsAIAAAA9UesAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABRQoWPtoosuSllZWY1b3759q7e/+eabOfvss7Ptttumbdu2OfbYY/PKK6804sQAAAB1o9CxliR77LFHFixYUH174oknqredf/75uf/++3PnnXfm8ccfz8svv5yPfexjjTgtAABA3WjR2AO8mxYtWqRz587rrS9ZsiQ33nhjbr/99owYMSJJctNNN6Vfv3753e9+l//3//5fQ48KAABQZwp/Zm327Nnp2rVrdtlll4waNSovvfRSkmT69OlZtWpVDjnkkOp9+/btm5133jnTpk17x+dcuXJlKisra9wAAACKpNCxNmTIkNx888156KGHct1112Xu3Lk54IAD8sYbb2ThwoVp2bJlOnToUOMxO+ywQxYuXPiOzzt+/Pi0b9+++tatW7d6fBUAAAC1V+i3QY4cObL6zwMGDMiQIUPSvXv3/M///E9atWq12c87bty4jB07tvp+ZWWlYAMAAAql0GfW/lOHDh2y2267Zc6cOencuXPeeuutLF68uMY+r7zyygY/4/Z25eXlqaioqHEDAAAoki0q1pYuXZrnn38+Xbp0yaBBg7LVVlvl0Ucfrd4+a9asvPTSSxk6dGgjTgkAAPDeFfptkF/4whdy1FFHpXv37nn55Zdz4YUXpnnz5jnxxBPTvn37nH766Rk7dmw6duyYioqKnHvuuRk6dKgrQQIAAFu8Qsfa3//+95x44ol5/fXXs/3222f//ffP7373u2y//fZJku9///tp1qxZjj322KxcuTKHHXZYrr322kaeGgAA4L0rdKzdcccd77h96623zqRJkzJp0qQGmggAAKBhbFGfWQMAAGgqxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKKAWjT0AAAD1a/bs5I03GvaYM2fW/NmQ2rVLdt214Y8LdU2sAQC8j82eney2W+Md/+STG+e4f/ubYGPLJ9YAAN7H1p1Ru/XWpF+/hjvuihXJvHlJjx5Jq1YNd9yZM9cGYkOfSYT6INYAAJqAfv2SffZp2GPut1/DHg/eb1xgBAAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQC0aewBg06xYsfbnzJmNO0dDWfc6171uAICmRqzBFmLevLU/Tz65UcdocPPmJfvt19hTAAA0PLEGW4gePdb+vPXWpF+/Rh2lQcycuTZM171uAICmRqzBFqJVq7U/+/VL9tmncWdpSOteNwBAU+MCIwAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACKnSsjR8/Ph/4wAfSrl27dOrUKcccc0xmzZpVY59hw4alrKysxu1Tn/pUI00MAABQNwoda48//njOPvvs/O53v8vDDz+cVatW5UMf+lCWLVtWY78zzzwzCxYsqL5ddtlljTQxAABA3WjR2AO8k4ceeqjG/ZtvvjmdOnXK9OnTc+CBB1avt27dOp07d27o8QAAAOpNoc+s/aclS5YkSTp27Fhj/bbbbst2222XPffcM+PGjcvy5cvf8XlWrlyZysrKGjcAAIAiKfSZtberqqrK5z73uey3337Zc889q9dPOumkdO/ePV27ds2zzz6bL3/5y5k1a1buvvvujT7X+PHjc/HFFzfE2AAAAJtli4m1s88+O3/5y1/yxBNP1Fj/5Cc/Wf3n/v37p0uXLjn44IPz/PPPp1evXht8rnHjxmXs2LHV9ysrK9OtW7f6GRwAAGAzbBGxds455+SBBx7I1KlTs9NOO73jvkOGDEmSzJkzZ6OxVl5envLy8jqfEwAAoK4UOtZKpVLOPffc3HPPPZkyZUp69uz5ro+ZMWNGkqRLly71PB0AAED9KXSsnX322bn99ttz3333pV27dlm4cGGSpH379mnVqlWef/753H777TniiCOy7bbb5tlnn83555+fAw88MAMGDGjk6QEAADZfoWPtuuuuS7L2i6/f7qabbsopp5ySli1b5pFHHsnEiROzbNmydOvWLccee2y+/vWvN8K0AAAAdafQsVYqld5xe7du3fL444830DQAAAANZ4v6njUAAICmQqwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACqhFYw8AAED9WbFi7c+ZMxt3joay7nWue92wJRNrAADvY/Pmrf158smNOkaDmzcv2W+/xp4C3huxBgDwPtajx9qft96a9OvXqKM0iJkz14bputcNWzKxBgDwPtaq1dqf/fol++zTuLM0pHWvG7ZkLjACAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABeRqkLCFWL587c9nnmnY465Ysfa7anr0aNgrazWVL28FANgYsQZbiOeeW/vzzDMbd46G1q5dY08AANA4xBpsIY45Zu3Pvn2T1q0b7rjrvly0Mb5MtV27ZNddG/aYAABFIdZgC7HddskZZzTe8Zval6kCADQ2FxgBAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAhJrAAAABSTWAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAWjT2AAAA1J/ly9f+fOaZhj3uihXJvHlJjx5Jq1YNd9yZMxvuWFDfxBoAwPvYc8+t/XnmmY07R0Nr166xJ4D3TqwBALyPHXPM2p99+yatWzfccWfOTE4+Obn11qRfv4Y7brI21HbdtWGPCfVBrAEAvI9tt11yxhmNd/x+/ZJ99mm848OWzAVGAAAACkisAQAAFJBYAwAAKCCxBgAAUEBiDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAYk1AACAAmrR2AMAAFBMy5cnzz23eY+dObPmz83Rt2/SuvXmPx62dGINAIANeu65ZNCg9/YcJ5+8+Y+dPj3ZZ5/3dnzYkok1AAA2qG/ftcG0OVasSObNS3r0SFq12vzjQ1P2vom1SZMmZcKECVm4cGEGDhyYq6++Ovvuu29jjwWF4G0sAGyO1q3f25mt/faru1mgKXpfxNrPfvazjB07Ntdff32GDBmSiRMn5rDDDsusWbPSqVOnxh4PGp23sQAAbHnKSqVSqbGHeK+GDBmSD3zgA7nmmmuSJFVVVenWrVvOPffcfOUrX3nXx1dWVqZ9+/ZZsmRJKioq6ntcaHDv5cxaXb2NxZk1AIC1NrU/tvgza2+99VamT5+ecePGVa81a9YshxxySKZNm7bBx6xcuTIrV66svl9ZWVnvc0Jj8jYWAIAtzxb/PWv//Oc/s2bNmuywww411nfYYYcsXLhwg48ZP3582rdvX33r1q1bQ4wKAACwybb4WNsc48aNy5IlS6pv8+fPb+yRAAAAatji3wa53XbbpXnz5nnllVdqrL/yyivp3LnzBh9TXl6e8vLyhhgPAABgs2zxZ9ZatmyZQYMG5dFHH61eq6qqyqOPPpqhQ4c24mQAAACbb4s/s5YkY8eOzZgxYzJ48ODsu+++mThxYpYtW5ZTTz21sUcDAADYLO+LWDv++OPz2muv5YILLsjChQuz11575aGHHlrvoiMAAABbivfF96y9V75nDQAAaCib2h9b/GfWAAAA3o/EGgAAQAGJNQAAgAISawAAAAUk1gAAAApIrAEAABSQWAMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQC0ae4AiKJVKSZLKyspGngQAAHi/W9cd6zpkY8RakjfeeCNJ0q1bt0aeBAAAaCreeOONtG/ffqPby0rvlnNNQFVVVV5++eW0a9cuZWVljT0OFEplZWW6deuW+fPnp6KiorHHAWAL4d8fsHGlUilvvPFGunbtmmbNNv7JNGfWkjRr1iw77bRTY48BhVZRUeFftgDUmn9/wIa90xm1dVxgBAAAoIDEGgAAQAGJNeAdlZeX58ILL0x5eXljjwLAFsS/P+C9c4ERAACAAnJmDQAAoIDEGgAAQAGJNQAAgAISawAAAAUk1oCNmjRpUnr06JGtt946Q4YMye9///vGHgmAgps6dWqOOuqodO3aNWVlZbn33nsbeyTYYok1YIN+9rOfZezYsbnwwgvzzDPPZODAgTnssMPy6quvNvZoABTYsmXLMnDgwEyaNKmxR4Etnkv3Axs0ZMiQfOADH8g111yTJKmqqkq3bt1y7rnn5itf+UojTwfAlqCsrCz33HNPjjnmmMYeBbZIzqwB63nrrbcyffr0HHLIIdVrzZo1yyGHHJJp06Y14mQAAE2HWAPW889//jNr1qzJDjvsUGN9hx12yMKFCxtpKgCApkWsAQAAFJBYA9az3XbbpXnz5nnllVdqrL/yyivp3LlzI00FANC0iDVgPS1btsygQYPy6KOPVq9VVVXl0UcfzdChQxtxMgCApqNFYw8AFNPYsWMzZsyYDB48OPvuu28mTpyYZcuW5dRTT23s0QAosKVLl2bOnDnV9+fOnZsZM2akY8eO2XnnnRtxMtjyuHQ/sFHXXHNNJkyYkIULF2avvfbKVVddlSFDhjT2WAAU2JQpUzJ8+PD11seMGZObb7654QeCLZhYAwAAKCCfWQMAACggsQYAAFBAYg0AAKCAxBoAAEABiTUAAIACEmsAAAAFJNYAAAAKSKwBAAAUkFgDAAAoILEGAABQQGINAACggMQaAABAAf1/+2ID0HxCVOcAAAAASUVORK5CYII=","text/plain":["<Figure size 1000x800 with 1 Axes>"]},"metadata":{},"output_type":"display_data"}],"source":["data.boxplot(\"text_length\", by=\"label\", grid=False, showfliers=False, color=\"Blue\", figsize=(10,8))\n","\n","plt.suptitle('')\n","plt.title('Comment Length by Type')\n","plt.xlabel('')\n","plt.ylabel('Words per Comment')\n","plt.show()"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"kqgMPq7elof3"},"source":["#### Final Data Preprocessing"]},{"cell_type":"code","execution_count":7,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":236},"executionInfo":{"elapsed":9,"status":"error","timestamp":1681825560450,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"rG5s5ceDlof3","outputId":"1bd0b103-31b8-425d-f0aa-1647f7e32ceb"},"outputs":[{"name":"stdout","output_type":"stream","text":["<class 'pandas.core.frame.DataFrame'>\n","Int64Index: 146214 entries, 0 to 159570\n","Data columns (total 2 columns):\n"," #   Column  Non-Null Count   Dtype \n","---  ------  --------------   ----- \n"," 0   text    146214 non-null  object\n"," 1   label   146214 non-null  int64 \n","dtypes: int64(1), object(1)\n","memory usage: 3.3+ MB\n"]}],"source":["data = data[data['text_length'] < 171]\n","\n","data = data.drop(columns=[\"text_length\"])\n","\n","data.info()"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"XWYCTizQlof3"},"source":["#### Number of Samples in Each Class"]},{"cell_type":"code","execution_count":8,"metadata":{"executionInfo":{"elapsed":6,"status":"aborted","timestamp":1681825560451,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"ih7P8VEmlof4"},"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAlUAAAHHCAYAAACWQK1nAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABJQklEQVR4nO3de1wWdf7//+eFyEH0AvEAUqhoJp7SQkXyUG4kJh0s3TxVVqS7LZhKmboWamWWZql5yraybTXNSrewMMIKt1hUlDykZoWn3Avyg3ApJaDM7w9/zNcrMKEGAXncb7e53bzm/ZqZ11xoPJuZ633ZDMMwBAAAgD/ErbobAAAAuBwQqgAAACxAqAIAALAAoQoAAMAChCoAAAALEKoAAAAsQKgCAACwAKEKAADAAoQqAAAACxCqgDouOztbQ4cOVZMmTWSz2TR//vzqbqlK3H///WrdunV1t3FBb731lkJDQ1W/fn35+flVdztl2Gw2xcXFXZJj1fSfFXAhhCqghvr6669ls9m0f/9+SdJLL71UJb9oJk6cqI0bN2rq1Kl66623NHDgwAvWnjp1StOnT1fnzp3l4+OjJk2aqFu3bho/fryOHTtmeW91xb59+3T//ferbdu2evXVV7V8+fIL1s6YMUM2m+2Ci8PhuISdV47T6dTMmTPVtWtXNWzYUN7e3urcubMmT57M3x9cFtyruwEA5UtPT5e/v7+uvvpqSVJaWpp69epl+XE2bdqkO+64Q4899thv1hUXF6tfv37at2+fRo8erXHjxunUqVPas2ePVq1apTvvvFNBQUGW91cXfP755yopKdGCBQt01VVXVWibpUuXqmHDhmXW18SrXJL0ww8/KDIyUocPH9af//xnjR07Vh4eHtq5c6dee+01rVu3Tt9++211twn8IYQqoIbasmWLevbsKZvNJulcqIqPj7f8ODk5ORX6Rbx+/Xrt2LFDK1eu1MiRI13GTp8+raKiIst7qytycnIkVS4QDR06VE2bNq2ijqx15swZ3XXXXcrOztbnn3+uPn36uIzPmjVLzz//fDV1B1iH239ADXLixAkdP35cx48fV3p6ujp37qzjx49rz549Onr0qNq1a6fjx4/r1KlTF93XDz/8oD//+c/y9/dXgwYN1KtXL23YsMEcX7FihWw2mwzD0OLFi83bRxfy/fffS5J69+5dZszLy0t2u918vXPnTt1///1q06aNvLy8FBgYqAcffFD/93//57Jd6a2sb7/9Vvfcc498fX3VrFkzPfnkkzIMQ0eOHNEdd9whu92uwMBAzZs3z2X7zz//XDabTWvWrNHf//53BQYGysfHR7fffruOHDly0feopKRE8+fPV6dOneTl5aWAgAD95S9/0YkTJ1zqtm3bpqioKDVt2lTe3t4KCQnRgw8+eNH9S9KSJUvUqVMneXp6KigoSLGxscrLyzPHW7durenTp0uSmjVrJpvNphkzZlRo37+lqKhICQkJCgsLk6+vr3x8fNS3b1999tlnZWpLr5J16dJFXl5eatasmQYOHKht27aVqV2/fr06d+4sT09PderUSUlJSRft5b333tPXX3+tadOmlQlUkmS32zVr1qzf3McLL7yg66+/Xk2aNJG3t7fCwsL07rvvlqlLTk5Wnz595Ofnp4YNG6p9+/b6+9//7lLz8ssvq1OnTmrQoIEaN26s7t27a9WqVRc9D+CiDAA1RqtWrQxJF11Gjx79m/txOBxGQECA0ahRI2PatGnGiy++aHTt2tVwc3Mz3n//fcMwDOP777833nrrLUOScfPNNxtvvfWW8dZbb11wn6tWrTIkGU899ZRRUlLym8d/4YUXjL59+xpPPfWUsXz5cmP8+PGGt7e30bNnT5dtp0+fbkgyunXrZowYMcJYsmSJER0dbUgyXnzxRaN9+/bGww8/bCxZssTo3bu3Icn44osvzO0/++wzQ5LRpUsX45prrjFefPFFY8qUKYaXl5dx9dVXGz///LNZO3r0aKNVq1YufT700EOGu7u7MWbMGGPZsmXG5MmTDR8fH6NHjx5GUVGRYRiGkZ2dbTRu3Ni4+uqrjblz5xqvvvqqMW3aNKNDhw6/+R6cf36RkZHGyy+/bMTFxRn16tVz2f+6deuMO++805BkLF261HjrrbeMr7/++qL73L9/v/HTTz+5LCdOnDDrfvrpJ6NFixZGfHy8sXTpUmPOnDlG+/btjfr16xs7duxw2ef9999vSDJuueUWY/78+cYLL7xg3HHHHcbLL79s1kgyunbtarRo0cJ4+umnjfnz5xtt2rQxGjRoYBw/fvw334eRI0cakozDhw9f9D0zjPJ/VldeeaXxt7/9zVi0aJHx4osvGj179jQkGYmJiWbN7t27DQ8PD6N79+7GggULjGXLlhmPPfaY0a9fP7Nm+fLlhiRj6NChxiuvvGIsWLDAiImJMR555JEK9Qb8FkIVUIP85z//MZKTk40nn3zScHd3Nz7++GMjOTnZuOWWW4zu3bsbycnJRnJysrFnz57f3M+ECRMMScbmzZvNdSdPnjRCQkKM1q1bG2fPnjXXSzJiY2Mv2tvPP/9stG/f3pBktGrVyrj//vuN1157zcjOzi639tfefvttQ5KRmppqrisNCGPHjjXXnTlzxrjyyisNm81mPPfcc+b6EydOGN7e3i6BsjRUXXHFFYbT6TTXv/POO4YkY8GCBea6X/+i3rx5syHJWLlypUufSUlJLuvXrVtnSDK2bt160ffofDk5OYaHh4cxYMAAl/d70aJFhiTj9ddfL/M+/PTTTxfdb2lteUv79u3NujNnzhiFhYUu2544ccIICAgwHnzwQXPdpk2bDEnlhorzA7Akw8PDw/juu+/MdV9//bUhySV8lefaa681fH19L3pupcoLVb/+O1VUVGR07tzZ+NOf/mSue+mlly76Pt5xxx1Gp06dKtwLUBnc/gNqkN69eysyMlKnTp1Sjx49NHDgQPPh3ltvvVWRkZGKjIxUx44df3M/H330kXr27Olyq6Vhw4YaO3asDh48qG+++abSvXl7eys9PV2TJk2SdO72YUxMjFq0aKFx48apsLDQpbbU6dOndfz4cfMh++3bt5fZ90MPPWT+uV69eurevbsMw1BMTIy53s/PT+3bt9cPP/xQZvv77rtPjRo1Ml8PHTpULVq00EcffXTB81m7dq18fX118803m7dcjx8/rrCwMDVs2NC8TVb6nFNiYqKKi4t/8z0636effqqioiJNmDBBbm7/7z+1Y8aMkd1ud7kV+3u89957Sk5OdlneeOMNc7xevXry8PCQdO72Xm5urs6cOaPu3bu7/Azee+892Ww28xbk+X59OzgyMlJt27Y1X19zzTWy2+3l/kzO53Q6XX4+v8f5f6dOnDih/Px89e3b1+VcSn9W//73v1VSUlLufvz8/HT06FFt3br1D/UDlIdQBdQQ+fn55i/2lJQUhYeH6/jx4/r222+1Z88ede3aVcePH1d+fv5F93Xo0CG1b9++zPoOHTqY47+Hr6+v5syZo4MHD+rgwYN67bXX1L59ey1atEhPP/20WZebm6vx48crICBA3t7eatasmUJCQszz/LWWLVuWOY6Xl1eZB7F9fX3LPO8kSe3atXN5bbPZdNVVV+ngwYMXPJcDBw4oPz9fzZs3V7NmzVyWU6dOmQ+P33DDDRoyZIhmzpyppk2b6o477tAbb7zhEiLLU/oe//rn4OHhoTZt2vzun0Gpfv36mSG7dImIiHCpefPNN3XNNdfIy8tLTZo0UbNmzbRhwwaXn8H333+voKAg+fv7X/SYv/45SVLjxo3L/Zmcz2636+TJkxU8s/IlJiaqV69e8vLykr+/v5o1a6alS5e6nMuwYcPUu3dvPfTQQwoICNDw4cP1zjvvuASsyZMnq2HDhurZs6fatWun2NhYffnll3+oN6AUoQqoIe644w7zl/rOnTs1f/58NWvWzPylfOedd6pZs2a64447qrnTc1q1aqUHH3xQX375pfz8/LRy5Upz7O6779arr76qv/71r3r//ff1ySefmA80l3cFoV69ehVaJ0mGYVjSf0lJiZo3b17mak/p8tRTT0k6F9DeffddpaWlKS4uTj/++KMefPBBhYWFVegDA9XlX//6lzn31WuvvaakpCQlJyfrT3/60wWv4lzM7/2ZhIaGKj8/v0IfHijP5s2bdfvtt8vLy0tLlizRRx99pOTkZI0cOdLl2N7e3kpNTdWnn36qe++9Vzt37tSwYcN088036+zZs5LO/Y/F/v37tXr1avXp00fvvfee+vTpU+6VOqCymFIBqCHmzZunEydOKC0tTTNnzlRiYqLc3d318ssv68cff9Rzzz0n6dyVgYtp1aqVOWno+fbt22eOW6Vx48Zq27atdu/eLencrZmUlBTNnDlTCQkJZt2BAwcsO+av/XrfhmHou+++0zXXXHPBbdq2batPP/1UvXv3drm1dCG9evVSr169NGvWLK1atUqjRo3S6tWrXW5dnq/0Pd6/f7/atGljri8qKlJWVpYiIyMrcmq/27vvvqs2bdro/fffd7mN9+vw0LZtW23cuFG5ubkVulr1e9x22216++239a9//UtTp06t9PbvvfeevLy8tHHjRnl6eprrz7/dWcrNzU033XSTbrrpJr344ot69tlnNW3aNH322Wfme+7j46Nhw4Zp2LBhKioq0l133aVZs2Zp6tSp8vLy+v0nijqPK1VADREWFqbIyEidOXNGnTt3Np+nys7OdrnFExYWdtF9DRo0SFu2bFFaWpq5rqCgQMuXL1fr1q0v+kxWeb7++msdP368zPpDhw7pm2++Ma+olV7N+PXVi6r8+pt//vOfLreX3n33Xf3vf//TLbfccsFt7r77bp09e9bltmWpM2fOmNMenDhxosy5dOvWTZJ+8xZgZGSkPDw8tHDhQpftX3vtNeXn5ys6Oroip/a7lfdzSE9Pd/k7IUlDhgyRYRiaOXNmmX1YdVVw6NCh6tKli2bNmlXm+JJ08uRJTZs27YLb16tXTzabzbzaJEkHDx7U+vXrXepyc3PLbPvrn9Wvp/Xw8PBQx44dZRhGpZ6ZA8rDlSqghvnyyy91/fXXSzr3kPeOHTvKzLNzMVOmTNHbb7+tW265RY888oj8/f315ptvKisrS++9957Lg9MVlZycrOnTp+v2229Xr1691LBhQ/3www96/fXXVVhYaM6tZLfb1a9fP82ZM0fFxcW64oor9MknnygrK6vSx6wof39/9enTRw888ICys7M1f/58XXXVVRozZswFt7nhhhv0l7/8RbNnz1ZmZqYGDBig+vXr68CBA1q7dq0WLFigoUOH6s0339SSJUt05513qm3btjp58qReffVV2e12DRo06IL7b9asmaZOnaqZM2dq4MCBuv3227V//34tWbJEPXr00D333POHzvndd98td0b1m2++WQEBAbr11lv1/vvv684771R0dLSysrK0bNkydezY0eW2Zf/+/XXvvfdq4cKFOnDggAYOHKiSkhJt3rxZ/fv3t+T7/urXr6/3339fkZGR6tevn+6++2717t1b9evXN2fkb9y48QXnqoqOjtaLL76ogQMHauTIkcrJydHixYt11VVXaefOnWbdU089pdTUVEVHR6tVq1bKycnRkiVLdOWVV5of2hgwYIACAwPVu3dvBQQEaO/evVq0aJGio6P/8MP0AFMqADXImTNnjIYNG5rzRf3nP/8xJBk5OTmV3tf3339vDB061PDz8zO8vLyMnj17uszpU0oVnFLhhx9+MBISEoxevXoZzZs3N9zd3Y1mzZoZ0dHRxqZNm1xqjx49atx5552Gn5+f4evra/z5z382jh07Zkgypk+fbtZdaCqB0aNHGz4+PmV6uOGGG1w+Dl86pcLbb79tTJ061WjevLnh7e1tREdHG4cOHSqzz19/TN8wzs1bFBYWZnh7exuNGjUyunTpYjz++OPGsWPHDMMwjO3btxsjRowwWrZsaXh6ehrNmzc3br31VmPbtm0Xfc8M49wUCqGhoUb9+vWNgIAA4+GHH3aZT+q33ofy/NaUCpKMzz77zDCMc9MhPPvss0arVq0MT09P49prrzUSExPLfR/OnDljzJ071wgNDTU8PDyMZs2aGbfccouRkZFh1lzo70mrVq0uOm9aqRMnThgJCQlGly5djAYNGhheXl5G586djalTpxr/+9//zLryenzttdeMdu3aGZ6enkZoaKjxxhtvmO9FqZSUFOOOO+4wgoKCDA8PDyMoKMgYMWKE8e2335o1r7zyitGvXz+jSZMmhqenp9G2bVtj0qRJRn5+foXOAfgtNsOw6PouAFxin3/+ufr376+1a9dq6NCh1d0OgDqOZ6oAAAAsQKgCAACwAKEKAADAAjxTBQAAYAGuVAEAAFiAUAUAAGABJv+8hEpKSnTs2DE1atSozLe/AwCAmskwDJ08eVJBQUG/OXkyoeoSOnbsmIKDg6u7DQAA8DscOXJEV1555QXHCVWXUOlXIBw5ckR2u72auwEAABXhdDoVHBx80a8yIlRdQqW3/Ox2O6EKAIBa5mKP7vCgOgAAgAUIVQAAABYgVAEAAFiAUAUAAGABQhUAAIAFCFUAAAAWIFQBAABYoFpDVWpqqm677TYFBQXJZrNp/fr1F6z961//KpvNpvnz57usz83N1ahRo2S32+Xn56eYmBidOnXKpWbnzp3q27evvLy8FBwcrDlz5pTZ/9q1axUaGiovLy916dJFH330kcu4YRhKSEhQixYt5O3trcjISB04cOB3nzsAALi8VGuoKigoUNeuXbV48eLfrFu3bp3++9//KigoqMzYqFGjtGfPHiUnJysxMVGpqakaO3asOe50OjVgwAC1atVKGRkZmjt3rmbMmKHly5ebNV999ZVGjBihmJgY7dixQ4MHD9bgwYO1e/dus2bOnDlauHChli1bpvT0dPn4+CgqKkqnT5+24J0AAAC1nlFDSDLWrVtXZv3Ro0eNK664wti9e7fRqlUr46WXXjLHvvnmG0OSsXXrVnPdxx9/bNhsNuPHH380DMMwlixZYjRu3NgoLCw0ayZPnmy0b9/efH333Xcb0dHRLscNDw83/vKXvxiGYRglJSVGYGCgMXfuXHM8Ly/P8PT0NN5+++0Kn2N+fr4hycjPz6/wNgAAoHpV9Pd3jX6mqqSkRPfee68mTZqkTp06lRlPS0uTn5+funfvbq6LjIyUm5ub0tPTzZp+/frJw8PDrImKitL+/ft14sQJsyYyMtJl31FRUUpLS5MkZWVlyeFwuNT4+voqPDzcrClPYWGhnE6nywIAAC5PNTpUPf/883J3d9cjjzxS7rjD4VDz5s1d1rm7u8vf318Oh8OsCQgIcKkpfX2xmvPHz9+uvJryzJ49W76+vuYSHBz8m+cLAABqrxobqjIyMrRgwQKtWLHiol9gWFNNnTpV+fn55nLkyJHqbgkAAFSRGhuqNm/erJycHLVs2VLu7u5yd3fXoUOH9Oijj6p169aSpMDAQOXk5Lhsd+bMGeXm5iowMNCsyc7OdqkpfX2xmvPHz9+uvJryeHp6ym63uywAAODyVGND1b333qudO3cqMzPTXIKCgjRp0iRt3LhRkhQREaG8vDxlZGSY223atEklJSUKDw83a1JTU1VcXGzWJCcnq3379mrcuLFZk5KS4nL85ORkRURESJJCQkIUGBjoUuN0OpWenm7WAACAus29Og9+6tQpfffdd+brrKwsZWZmyt/fXy1btlSTJk1c6uvXr6/AwEC1b99ektShQwcNHDhQY8aM0bJly1RcXKy4uDgNHz7cnH5h5MiRmjlzpmJiYjR58mTt3r1bCxYs0EsvvWTud/z48brhhhs0b948RUdHa/Xq1dq2bZs57YLNZtOECRP0zDPPqF27dgoJCdGTTz6poKAgDR48uIrfpctD6ykbqrsFXEIHn4uu7hYA4JKr1lC1bds29e/f33wdHx8vSRo9erRWrFhRoX2sXLlScXFxuummm+Tm5qYhQ4Zo4cKF5rivr68++eQTxcbGKiwsTE2bNlVCQoLLXFbXX3+9Vq1apSeeeEJ///vf1a5dO61fv16dO3c2ax5//HEVFBRo7NixysvLU58+fZSUlCQvL68/+C4AAIDLgc0wDKO6m6grnE6nfH19lZ+fX+eer+JKVd3ClSoAl5OK/v6usc9UAQAA1CaEKgAAAAsQqgAAACxAqAIAALAAoQoAAMAChCoAAAALEKoAAAAsQKgCAACwAKEKAADAAoQqAAAACxCqAAAALECoAgAAsAChCgAAwAKEKgAAAAsQqgAAACxAqAIAALAAoQoAAMAChCoAAAALEKoAAAAsQKgCAACwAKEKAADAAoQqAAAACxCqAAAALECoAgAAsAChCgAAwAKEKgAAAAsQqgAAACxAqAIAALAAoQoAAMAChCoAAAALEKoAAAAsQKgCAACwAKEKAADAAoQqAAAACxCqAAAALECoAgAAsAChCgAAwAKEKgAAAAsQqgAAACxQraEqNTVVt912m4KCgmSz2bR+/XpzrLi4WJMnT1aXLl3k4+OjoKAg3XfffTp27JjLPnJzczVq1CjZ7Xb5+fkpJiZGp06dcqnZuXOn+vbtKy8vLwUHB2vOnDllelm7dq1CQ0Pl5eWlLl266KOPPnIZNwxDCQkJatGihby9vRUZGakDBw5Y92YAAIBarVpDVUFBgbp27arFixeXGfv555+1fft2Pfnkk9q+fbvef/997d+/X7fffrtL3ahRo7Rnzx4lJycrMTFRqampGjt2rDnudDo1YMAAtWrVShkZGZo7d65mzJih5cuXmzVfffWVRowYoZiYGO3YsUODBw/W4MGDtXv3brNmzpw5WrhwoZYtW6b09HT5+PgoKipKp0+froJ3BgAA1DY2wzCM6m5Ckmw2m9atW6fBgwdfsGbr1q3q2bOnDh06pJYtW2rv3r3q2LGjtm7dqu7du0uSkpKSNGjQIB09elRBQUFaunSppk2bJofDIQ8PD0nSlClTtH79eu3bt0+SNGzYMBUUFCgxMdE8Vq9evdStWzctW7ZMhmEoKChIjz76qB577DFJUn5+vgICArRixQoNHz68QufodDrl6+ur/Px82e323/M21Vqtp2yo7hZwCR18Lrq6WwAAy1T093eteqYqPz9fNptNfn5+kqS0tDT5+fmZgUqSIiMj5ebmpvT0dLOmX79+ZqCSpKioKO3fv18nTpwwayIjI12OFRUVpbS0NElSVlaWHA6HS42vr6/Cw8PNGgAAULe5V3cDFXX69GlNnjxZI0aMMFOiw+FQ8+bNXerc3d3l7+8vh8Nh1oSEhLjUBAQEmGONGzeWw+Ew151fc/4+zt+uvJryFBYWqrCw0HztdDorfL4AAKB2qRVXqoqLi3X33XfLMAwtXbq0utupsNmzZ8vX19dcgoODq7slAABQRWp8qCoNVIcOHVJycrLLvczAwEDl5OS41J85c0a5ubkKDAw0a7Kzs11qSl9frOb88fO3K6+mPFOnTlV+fr65HDlypMLnDQAAapcaHapKA9WBAwf06aefqkmTJi7jERERysvLU0ZGhrlu06ZNKikpUXh4uFmTmpqq4uJisyY5OVnt27dX48aNzZqUlBSXfScnJysiIkKSFBISosDAQJcap9Op9PR0s6Y8np6estvtLgsAALg8VWuoOnXqlDIzM5WZmSnp3APhmZmZOnz4sIqLizV06FBt27ZNK1eu1NmzZ+VwOORwOFRUVCRJ6tChgwYOHKgxY8Zoy5Yt+vLLLxUXF6fhw4crKChIkjRy5Eh5eHgoJiZGe/bs0Zo1a7RgwQLFx8ebfYwfP15JSUmaN2+e9u3bpxkzZmjbtm2Ki4uTdO6TiRMmTNAzzzyjDz74QLt27dJ9992noKCg3/y0IgAAqDuqdUqFzz//XP379y+zfvTo0ZoxY0aZB8xLffbZZ7rxxhslnZv8My4uTh9++KHc3Nw0ZMgQLVy4UA0bNjTrd+7cqdjYWG3dulVNmzbVuHHjNHnyZJd9rl27Vk888YQOHjyodu3aac6cORo0aJA5bhiGpk+fruXLlysvL099+vTRkiVLdPXVV1f4fJlSAXUFUyoAuJxU9Pd3jZmnqi4gVKGuIFQBuJxclvNUAQAA1FSEKgAAAAsQqgAAACxAqAIAALAAoQoAAMAChCoAAAALEKoAAAAsQKgCAACwAKEKAADAAoQqAAAACxCqAAAALECoAgAAsAChCgAAwAKEKgAAAAsQqgAAACxAqAIAALAAoQoAAMAChCoAAAALEKoAAAAsQKgCAACwAKEKAADAAoQqAAAACxCqAAAALECoAgAAsAChCgAAwAKEKgAAAAsQqgAAACxAqAIAALAAoQoAAMAChCoAAAALEKoAAAAsQKgCAACwAKEKAADAAoQqAAAACxCqAAAALECoAgAAsAChCgAAwAKEKgAAAAsQqgAAACxAqAIAALBAtYaq1NRU3XbbbQoKCpLNZtP69etdxg3DUEJCglq0aCFvb29FRkbqwIEDLjW5ubkaNWqU7Ha7/Pz8FBMTo1OnTrnU7Ny5U3379pWXl5eCg4M1Z86cMr2sXbtWoaGh8vLyUpcuXfTRRx9VuhcAAFB3VWuoKigoUNeuXbV48eJyx+fMmaOFCxdq2bJlSk9Pl4+Pj6KionT69GmzZtSoUdqzZ4+Sk5OVmJio1NRUjR071hx3Op0aMGCAWrVqpYyMDM2dO1czZszQ8uXLzZqvvvpKI0aMUExMjHbs2KHBgwdr8ODB2r17d6V6AQAAdZfNMAyjupuQJJvNpnXr1mnw4MGSzl0ZCgoK0qOPPqrHHntMkpSfn6+AgACtWLFCw4cP1969e9WxY0dt3bpV3bt3lyQlJSVp0KBBOnr0qIKCgrR06VJNmzZNDodDHh4ekqQpU6Zo/fr12rdvnyRp2LBhKigoUGJiotlPr1691K1bNy1btqxCvVSE0+mUr6+v8vPzZbfbLXnfaovWUzZUdwu4hA4+F13dLQCAZSr6+7vGPlOVlZUlh8OhyMhIc52vr6/Cw8OVlpYmSUpLS5Ofn58ZqCQpMjJSbm5uSk9PN2v69etnBipJioqK0v79+3XixAmz5vzjlNaUHqcivQAAgLrNvbobuBCHwyFJCggIcFkfEBBgjjkcDjVv3txl3N3dXf7+/i41ISEhZfZROta4cWM5HI6LHudivZSnsLBQhYWF5mun0/kbZwwAAGqzGnul6nIwe/Zs+fr6mktwcHB1twQAAKpIjQ1VgYGBkqTs7GyX9dnZ2eZYYGCgcnJyXMbPnDmj3Nxcl5ry9nH+MS5Uc/74xXopz9SpU5Wfn28uR44cuchZAwCA2qrGhqqQkBAFBgYqJSXFXOd0OpWenq6IiAhJUkREhPLy8pSRkWHWbNq0SSUlJQoPDzdrUlNTVVxcbNYkJyerffv2aty4sVlz/nFKa0qPU5FeyuPp6Sm73e6yAACAy1O1hqpTp04pMzNTmZmZks49EJ6ZmanDhw/LZrNpwoQJeuaZZ/TBBx9o165duu+++xQUFGR+QrBDhw4aOHCgxowZoy1btujLL79UXFychg8frqCgIEnSyJEj5eHhoZiYGO3Zs0dr1qzRggULFB8fb/Yxfvx4JSUlad68edq3b59mzJihbdu2KS4uTpIq1AsAAKjbqvVB9W3btql///7m69KgM3r0aK1YsUKPP/64CgoKNHbsWOXl5alPnz5KSkqSl5eXuc3KlSsVFxenm266SW5ubhoyZIgWLlxojvv6+uqTTz5RbGyswsLC1LRpUyUkJLjMZXX99ddr1apVeuKJJ/T3v/9d7dq10/r169W5c2ezpiK9AACAuqvGzFNVFzBPFeoK5qkCcDmp9fNUAQAA1CaEKgAAAAsQqgAAACxAqAIAALAAoQoAAMAChCoAAAALEKoAAAAsQKgCAACwAKEKAADAAoQqAAAACxCqAAAALECoAgAAsAChCgAAwAKEKgAAAAsQqgAAACxAqAIAALAAoQoAAMAChCoAAAALVDpUHTlyREePHjVfb9myRRMmTNDy5cstbQwAAKA2qXSoGjlypD777DNJksPh0M0336wtW7Zo2rRpeuqppyxvEAAAoDaodKjavXu3evbsKUl655131LlzZ3311VdauXKlVqxYYXV/AAAAtUKlQ1VxcbE8PT0lSZ9++qluv/12SVJoaKj+97//WdsdAABALVHpUNWpUyctW7ZMmzdvVnJysgYOHChJOnbsmJo0aWJ5gwAAALVBpUPV888/r1deeUU33nijRowYoa5du0qSPvjgA/O2IAAAQF3jXtkNbrzxRh0/flxOp1ONGzc2148dO1YNGjSwtDkAAIDa4nfNU2UYhjIyMvTKK6/o5MmTkiQPDw9CFQAAqLMqfaXq0KFDGjhwoA4fPqzCwkLdfPPNatSokZ5//nkVFhZq2bJlVdEnAABAjVbpK1Xjx49X9+7ddeLECXl7e5vr77zzTqWkpFjaHAAAQG1R6StVmzdv1ldffSUPDw+X9a1bt9aPP/5oWWMAAAC1SaWvVJWUlOjs2bNl1h89elSNGjWypCkAAIDaptKhasCAAZo/f7752maz6dSpU5o+fboGDRpkZW8AAAC1RqVv/82bN09RUVHq2LGjTp8+rZEjR+rAgQNq2rSp3n777aroEQAAoMardKi68sor9fXXX2v16tXauXOnTp06pZiYGI0aNcrlwXUAAIC6pNKhSpLc3d11zz33WN0LAABArVWhUPXBBx9UeIelX7AMAABQl1QoVA0ePLhCO7PZbOV+MhAAAOByV6FQVVJSUtV9AAAA1Gq/67v/AAAA4Op3haqUlBTdeuutatu2rdq2batbb71Vn376qdW9AQAA1BqVDlVLlizRwIED1ahRI40fP17jx4+X3W7XoEGDtHjx4qroEQAAoMardKh69tln9dJLL+ntt9/WI488okceeUSrVq3SSy+9pGeffdbS5s6ePasnn3xSISEh8vb2Vtu2bfX000/LMAyzxjAMJSQkqEWLFvL29lZkZKQOHDjgsp/c3FyNGjVKdrtdfn5+iomJ0alTp1xqdu7cqb59+8rLy0vBwcGaM2dOmX7Wrl2r0NBQeXl5qUuXLvroo48sPV8AAFB7VTpU5eXlaeDAgWXWDxgwQPn5+ZY0Ver555/X0qVLtWjRIu3du1fPP/+85syZo5dfftmsmTNnjhYuXKhly5YpPT1dPj4+ioqK0unTp82aUaNGac+ePUpOTlZiYqJSU1M1duxYc9zpdGrAgAFq1aqVMjIyNHfuXM2YMUPLly83a7766iuNGDFCMTEx2rFjhwYPHqzBgwdr9+7dlp4zAAConWzG+Zd9KmDkyJG69tprNWnSJJf1L7zwgrZt26bVq1db1tytt96qgIAAvfbaa+a6IUOGyNvbW//6179kGIaCgoL06KOP6rHHHpMk5efnKyAgQCtWrNDw4cO1d+9edezYUVu3blX37t0lSUlJSRo0aJCOHj2qoKAgLV26VNOmTZPD4ZCHh4ckacqUKVq/fr327dsnSRo2bJgKCgqUmJho9tKrVy9169ZNy5Ytq9D5OJ1O+fr6Kj8/X3a73ZL3qLZoPWVDdbeAS+jgc9HV3QIAWKaiv78rfaWqY8eOmjVrlqKjo/XMM8/omWee0a233qpZs2apc+fOWrhwobn8Uddff71SUlL07bffSpK+/vpr/ec//9Ett9wiScrKypLD4VBkZKS5ja+vr8LDw5WWliZJSktLk5+fnxmoJCkyMlJubm5KT083a/r162cGKkmKiorS/v37deLECbPm/OOU1pQepzyFhYVyOp0uCwAAuDxV+mtqXnvtNTVu3FjffPONvvnmG3O9n5+fyxUlm82mRx555A81N2XKFDmdToWGhqpevXo6e/asZs2apVGjRkmSHA6HJCkgIMBlu4CAAHPM4XCoefPmLuPu7u7y9/d3qQkJCSmzj9Kxxo0by+Fw/OZxyjN79mzNnDmzsqcNAABqoUqHqqysrKroo1zvvPOOVq5cqVWrVqlTp07KzMzUhAkTFBQUpNGjR1+yPn6vqVOnKj4+3nztdDoVHBxcjR0BAICq8ru+UPlSmTRpkqZMmaLhw4dLkrp06aJDhw5p9uzZGj16tAIDAyVJ2dnZatGihblddna2unXrJkkKDAxUTk6Oy37PnDmj3Nxcc/vAwEBlZ2e71JS+vlhN6Xh5PD095enpWdnTBgAAtVClQ5VhGHr33Xf12WefKScnp8xX2Lz//vuWNffzzz/Lzc31sa969eqZxwwJCVFgYKBSUlLMEOV0OpWenq6HH35YkhQREaG8vDxlZGQoLCxMkrRp0yaVlJQoPDzcrJk2bZqKi4tVv359SVJycrLat2+vxo0bmzUpKSmaMGGC2UtycrIiIiIsO18AAFB7VfpB9QkTJujee+9VVlaWGjZsKF9fX5fFSrfddptmzZqlDRs26ODBg1q3bp1efPFF3XnnnZLOPbc1YcIEPfPMM/rggw+0a9cu3XfffQoKCjK/BLpDhw4aOHCgxowZoy1btujLL79UXFychg8frqCgIEnnPtHo4eGhmJgY7dmzR2vWrNGCBQtcbt2NHz9eSUlJmjdvnvbt26cZM2Zo27ZtiouLs/ScAQBA7VTpKRX8/f31r3/9S4MGDaqqnkwnT57Uk08+qXXr1iknJ0dBQUEaMWKEEhISzE/qGYah6dOna/ny5crLy1OfPn20ZMkSXX311eZ+cnNzFRcXpw8//FBubm4aMmSIFi5cqIYNG5o1O3fuVGxsrLZu3aqmTZtq3Lhxmjx5sks/a9eu1RNPPKGDBw+qXbt2mjNnTqXeB6ZUQF3BlAoALicV/f1d6VAVEhKijz/+WKGhoX+4ybqGUIW6glAF4HJSZfNUzZgxQzNnztQvv/zyhxoEAAC4nFT6QfW7775bb7/9tpo3b67WrVubD3aX2r59u2XNAQAA1BaVDlWjR49WRkaG7rnnHgUEBMhms1VFXwAAALVKpUPVhg0btHHjRvXp06cq+gEAAKiVKv1MVXBwcJ17yBoAAOBiKh2q5s2bp8cff1wHDx6sgnYAAABqp0rf/rvnnnv0888/q23btmrQoEGZB9Vzc3Mtaw4AAKC2qHSomj9/fhW0AQAAULv9rk//AQAAwFWlQ9X5Tp8+raKiIpd1PMQOAADqoko/qF5QUKC4uDg1b95cPj4+aty4scsCAABQF1U6VD3++OPatGmTli5dKk9PT/3jH//QzJkzFRQUpH/+859V0SMAAECNV+nbfx9++KH++c9/6sYbb9QDDzygvn376qqrrlKrVq20cuVKjRo1qir6BAAAqNEqfaUqNzdXbdq0kXTu+anSKRT69Omj1NRUa7sDAACoJSodqtq0aaOsrCxJUmhoqN555x1J565g+fn5WdocAABAbVHpUPXAAw/o66+/liRNmTJFixcvlpeXlyZOnKhJkyZZ3iAAAEBtUOlnqiZOnGj+OTIyUnv37tX27dt11VVX6ZprrrG0OQAAgNriD81TJUmtW7dW69atLWgFAACg9qrw7b+0tDQlJia6rPvnP/+pkJAQNW/eXGPHjlVhYaHlDQIAANQGFQ5VTz31lPbs2WO+3rVrl2JiYhQZGakpU6boww8/1OzZs6ukSQAAgJquwqEqMzNTN910k/l69erVCg8P16uvvqr4+HgtXLjQ/CQgAABAXVPhUHXixAkFBASYr7/44gvdcsst5usePXroyJEj1nYHAABQS1Q4VAUEBJjzUxUVFWn79u3q1auXOX7y5EnVr1/f+g4BAABqgQqHqkGDBmnKlCnavHmzpk6dqgYNGqhv377m+M6dO9W2bdsqaRIAAKCmq/CUCk8//bTuuusu3XDDDWrYsKHefPNNeXh4mOOvv/66BgwYUCVNAgAA1HQVDlVNmzZVamqq8vPz1bBhQ9WrV89lfO3atWrYsKHlDQIAANQGlZ7809fXt9z1/v7+f7gZAACA2qrS3/0HAACAsghVAAAAFiBUAQAAWKBCoeq6667TiRMnJJ37upqff/65SpsCAACobSoUqvbu3auCggJJ0syZM3Xq1KkqbQoAAKC2qdCn/7p166YHHnhAffr0kWEYeuGFFy44fUJCQoKlDQIAANQGFQpVK1as0PTp05WYmCibzaaPP/5Y7u5lN7XZbIQqAABQJ1UoVLVv316rV6+WJLm5uSklJUXNmzev0sYAAABqk0pP/llSUlIVfQAAANRqlQ5VkvT9999r/vz52rt3rySpY8eOGj9+PF+oDAAA6qxKz1O1ceNGdezYUVu2bNE111yja665Runp6erUqZOSk5OrokcAAIAar9JXqqZMmaKJEyfqueeeK7N+8uTJuvnmmy1rDgAAoLao9JWqvXv3KiYmpsz6Bx98UN98840lTQEAANQ2lQ5VzZo1U2ZmZpn1mZmZVfKJwB9//FH33HOPmjRpIm9vb3Xp0kXbtm0zxw3DUEJCglq0aCFvb29FRkbqwIEDLvvIzc3VqFGjZLfb5efnp5iYmDITmO7cuVN9+/aVl5eXgoODNWfOnDK9rF27VqGhofLy8lKXLl300UcfWX6+AACgdqp0qBozZozGjh2r559/Xps3b9bmzZv13HPP6S9/+YvGjBljaXMnTpxQ7969Vb9+fX388cf65ptvNG/ePDVu3NismTNnjhYuXKhly5YpPT1dPj4+ioqK0unTp82aUaNGac+ePUpOTlZiYqJSU1M1duxYc9zpdGrAgAFq1aqVMjIyNHfuXM2YMUPLly83a7766iuNGDFCMTEx2rFjhwYPHqzBgwdr9+7dlp4zAAConWyGYRiV2cAwDM2fP1/z5s3TsWPHJElBQUGaNGmSHnnkEdlsNsuamzJlir788ktt3rz5gr0EBQXp0Ucf1WOPPSZJys/PV0BAgFasWKHhw4dr79696tixo7Zu3aru3btLkpKSkjRo0CAdPXpUQUFBWrp0qaZNmyaHwyEPDw/z2OvXr9e+ffskScOGDVNBQYESExPN4/fq1UvdunXTsmXLKnQ+TqdTvr6+ys/Pl91u/93vS23UesqG6m4Bl9DB56KruwUAsExFf39X+kqVzWbTxIkTdfToUeXn5ys/P19Hjx7V+PHjLQ1UkvTBBx+oe/fu+vOf/6zmzZvr2muv1auvvmqOZ2VlyeFwKDIy0lzn6+ur8PBwpaWlSZLS0tLk5+dnBipJioyMlJubm9LT082afv36mYFKkqKiorR//37zi6TT0tJcjlNaU3qc8hQWFsrpdLosAADg8lTpUHW+Ro0aqVGjRlb1UsYPP/ygpUuXql27dtq4caMefvhhPfLII3rzzTclSQ6HQ5IUEBDgsl1AQIA55nA4yjzr5e7uLn9/f5ea8vZx/jEuVFM6Xp7Zs2fL19fXXIKDgyt1/gAAoPb4Q6GqqpWUlOi6667Ts88+q2uvvVZjx47VmDFjKny7rbpNnTrVvJqXn5+vI0eOVHdLAACgitToUNWiRQt17NjRZV2HDh10+PBhSVJgYKAkKTs726UmOzvbHAsMDFROTo7L+JkzZ5Sbm+tSU94+zj/GhWpKx8vj6ekpu93usgAAgMtTjQ5VvXv31v79+13Wffvtt2rVqpUkKSQkRIGBgUpJSTHHnU6n0tPTFRERIUmKiIhQXl6eMjIyzJpNmzappKRE4eHhZk1qaqqKi4vNmuTkZLVv3978pGFERITLcUprSo8DAADqthodqiZOnKj//ve/evbZZ/Xdd99p1apVWr58uWJjYyWde2h+woQJeuaZZ/TBBx9o165duu+++xQUFKTBgwdLOndla+DAgRozZoy2bNmiL7/8UnFxcRo+fLiCgoIkSSNHjpSHh4diYmK0Z88erVmzRgsWLFB8fLzZy/jx45WUlKR58+Zp3759mjFjhrZt26a4uLhL/r4AAICa53eFqri4OOXm5lrdSxk9evTQunXr9Pbbb6tz5856+umnNX/+fI0aNcqsefzxxzVu3DiNHTtWPXr00KlTp5SUlCQvLy+zZuXKlQoNDdVNN92kQYMGqU+fPi5zUPn6+uqTTz5RVlaWwsLC9OijjyohIcFlLqvrr7/eDHVdu3bVu+++q/Xr16tz585V/j4AAICar8LzVB09elRXXnmlJMnf31+ZmZlq2bKlObM4n2y7OOapQl3BPFUALicV/f1d4S9UDg0NVZMmTdS7d2+dPn1aR44cUcuWLXXw4EGXZ5EAAADqogrf/svLy9PatWsVFhamkpISDRo0SFdffbUKCwu1cePGMp+MAwAAqEsqHKqKi4vVs2dPPfroo/L29taOHTv0xhtvqF69enr99dcVEhKi9u3bV2WvAAAANVaFb//5+fmpW7du6t27t4qKivTLL7+od+/ecnd315o1a3TFFVdo69atVdkrAABAjVXhK1U//vijnnjiCXl6eurMmTMKCwtT3759VVRUpO3bt8tms6lPnz5V2SsAAECNVeFQ1bRpU912222aPXu2GjRooK1bt2rcuHGy2Wx67LHH5OvrqxtuuKEqewUAAKixfvfkn76+vrr77rtVv359bdq0SVlZWfrb3/5mZW8AAAC1RoWfqTrfzp07dcUVV0iSWrVqpfr16yswMFDDhg2ztDkAAIDa4neFqvMn+ty9e7dlzQAAANRWNfq7/wAAAGoLQhUAAIAFCFUAAAAWIFQBAABYgFAFAABgAUIVAACABQhVAAAAFiBUAQAAWIBQBQAAYAFCFQAAgAUIVQAAABYgVAEAAFiAUAUAAGABQhUAAIAFCFUAAAAWIFQBAABYgFAFAABgAUIVAACABQhVAAAAFiBUAQAAWIBQBQAAYAFCFQAAgAUIVQAAABYgVAEAAFiAUAUAAGABQhUAAIAFCFUAAAAWIFQBAABYgFAFAABgAUIVAACABQhVAAAAFqhVoeq5556TzWbThAkTzHWnT59WbGysmjRpooYNG2rIkCHKzs522e7w4cOKjo5WgwYN1Lx5c02aNElnzpxxqfn888913XXXydPTU1dddZVWrFhR5viLFy9W69at5eXlpfDwcG3ZsqUqThMAANRCtSZUbd26Va+88oquueYal/UTJ07Uhx9+qLVr1+qLL77QsWPHdNddd5njZ8+eVXR0tIqKivTVV1/pzTff1IoVK5SQkGDWZGVlKTo6Wv3791dmZqYmTJighx56SBs3bjRr1qxZo/j4eE2fPl3bt29X165dFRUVpZycnKo/eQAAUOPZDMMwqruJizl16pSuu+46LVmyRM8884y6deum+fPnKz8/X82aNdOqVas0dOhQSdK+ffvUoUMHpaWlqVevXvr4449166236tixYwoICJAkLVu2TJMnT9ZPP/0kDw8PTZ48WRs2bNDu3bvNYw4fPlx5eXlKSkqSJIWHh6tHjx5atGiRJKmkpETBwcEaN26cpkyZUqHzcDqd8vX1VX5+vux2u5VvUY3XesqG6m4Bl9DB56KruwUAsExFf3/XiitVsbGxio6OVmRkpMv6jIwMFRcXu6wPDQ1Vy5YtlZaWJklKS0tTly5dzEAlSVFRUXI6ndqzZ49Z8+t9R0VFmfsoKipSRkaGS42bm5siIyPNmvIUFhbK6XS6LAAA4PLkXt0NXMzq1au1fft2bd26tcyYw+GQh4eH/Pz8XNYHBATI4XCYNecHqtLx0rHfqnE6nfrll1904sQJnT17ttyaffv2XbD32bNna+bMmRU7UQAAUKvV6CtVR44c0fjx47Vy5Up5eXlVdzuVNnXqVOXn55vLkSNHqrslAABQRWp0qMrIyFBOTo6uu+46ubu7y93dXV988YUWLlwod3d3BQQEqKioSHl5eS7bZWdnKzAwUJIUGBhY5tOApa8vVmO32+Xt7a2mTZuqXr165daU7qM8np6estvtLgsAALg81ehQddNNN2nXrl3KzMw0l+7du2vUqFHmn+vXr6+UlBRzm/379+vw4cOKiIiQJEVERGjXrl0un9JLTk6W3W5Xx44dzZrz91FaU7oPDw8PhYWFudSUlJQoJSXFrAEAAHVbjX6mqlGjRurcubPLOh8fHzVp0sRcHxMTo/j4ePn7+8tut2vcuHGKiIhQr169JEkDBgxQx44dde+992rOnDlyOBx64oknFBsbK09PT0nSX//6Vy1atEiPP/64HnzwQW3atEnvvPOONmz4f59Yi4+P1+jRo9W9e3f17NlT8+fPV0FBgR544IFL9G4AAICarEaHqop46aWX5ObmpiFDhqiwsFBRUVFasmSJOV6vXj0lJibq4YcfVkREhHx8fDR69Gg99dRTZk1ISIg2bNigiRMnasGCBbryyiv1j3/8Q1FRUWbNsGHD9NNPPykhIUEOh0PdunVTUlJSmYfXAQBA3VQr5qm6XDBPFeoK5qkCcDm5rOapAgAAqOkIVQAAABYgVAEAAFiAUAUAAGABQhUAAIAFCFUAAAAWIFQBAABYgFAFAABgAUIVAACABQhVAAAAFiBUAQAAWIBQBQAAYAFCFQAAgAUIVQAAABYgVAEAAFiAUAUAAGABQhUAAIAFCFUAAAAWIFQBAABYgFAFAABgAUIVAACABQhVAAAAFiBUAQAAWIBQBQAAYAFCFQAAgAUIVQAAABYgVAEAAFiAUAUAAGABQhUAAIAFCFUAAAAWIFQBAABYgFAFAABgAUIVAACABQhVAAAAFiBUAQAAWIBQBQAAYAFCFQAAgAUIVQAAABYgVAEAAFiAUAUAAGCBGh2qZs+erR49eqhRo0Zq3ry5Bg8erP3797vUnD59WrGxsWrSpIkaNmyoIUOGKDs726Xm8OHDio6OVoMGDdS8eXNNmjRJZ86ccan5/PPPdd1118nT01NXXXWVVqxYUaafxYsXq3Xr1vLy8lJ4eLi2bNli+TkDAIDaqUaHqi+++EKxsbH673//q+TkZBUXF2vAgAEqKCgwayZOnKgPP/xQa9eu1RdffKFjx47prrvuMsfPnj2r6OhoFRUV6auvvtKbb76pFStWKCEhwazJyspSdHS0+vfvr8zMTE2YMEEPPfSQNm7caNasWbNG8fHxmj59urZv366uXbsqKipKOTk5l+bNAAAANZrNMAyjupuoqJ9++knNmzfXF198oX79+ik/P1/NmjXTqlWrNHToUEnSvn371KFDB6WlpalXr176+OOPdeutt+rYsWMKCAiQJC1btkyTJ0/WTz/9JA8PD02ePFkbNmzQ7t27zWMNHz5ceXl5SkpKkiSFh4erR48eWrRokSSppKREwcHBGjdunKZMmVKh/p1Op3x9fZWfny+73W7lW1PjtZ6yobpbwCV08Lno6m4BACxT0d/fNfpK1a/l5+dLkvz9/SVJGRkZKi4uVmRkpFkTGhqqli1bKi0tTZKUlpamLl26mIFKkqKiouR0OrVnzx6z5vx9lNaU7qOoqEgZGRkuNW5uboqMjDRrylNYWCin0+myAACAy1OtCVUlJSWaMGGCevfurc6dO0uSHA6HPDw85Ofn51IbEBAgh8Nh1pwfqErHS8d+q8bpdOqXX37R8ePHdfbs2XJrSvdRntmzZ8vX19dcgoODK3/iAACgVqg1oSo2Nla7d+/W6tWrq7uVCps6dary8/PN5ciRI9XdEgAAqCLu1d1ARcTFxSkxMVGpqam68sorzfWBgYEqKipSXl6ey9Wq7OxsBQYGmjW//pRe6acDz6/59ScGs7OzZbfb5e3trXr16qlevXrl1pTuozyenp7y9PSs/AkDAIBap0ZfqTIMQ3FxcVq3bp02bdqkkJAQl/GwsDDVr19fKSkp5rr9+/fr8OHDioiIkCRFRERo165dLp/SS05Olt1uV8eOHc2a8/dRWlO6Dw8PD4WFhbnUlJSUKCUlxawBAAB1W42+UhUbG6tVq1bp3//+txo1amQ+v+Tr6ytvb2/5+voqJiZG8fHx8vf3l91u17hx4xQREaFevXpJkgYMGKCOHTvq3nvv1Zw5c+RwOPTEE08oNjbWvIr017/+VYsWLdLjjz+uBx98UJs2bdI777yjDRv+3yfW4uPjNXr0aHXv3l09e/bU/PnzVVBQoAceeODSvzEAAKDGqdGhaunSpZKkG2+80WX9G2+8ofvvv1+S9NJLL8nNzU1DhgxRYWGhoqKitGTJErO2Xr16SkxM1MMPP6yIiAj5+Pho9OjReuqpp8yakJAQbdiwQRMnTtSCBQt05ZVX6h//+IeioqLMmmHDhumnn35SQkKCHA6HunXrpqSkpDIPrwMAgLqpVs1TVdsxTxXqCuapAnA5uSznqQIAAKipCFUAAAAWIFQBAABYgFAFAABgAUIVAACABQhVAAAAFiBUAQAAWIBQBQAAYAFCFQAAgAUIVQAAABYgVAEAAFiAUAUAAGABQhUAAIAFCFUAAAAWIFQBAABYgFAFAABgAUIVAACABQhVAAAAFiBUAQAAWIBQBQAAYAFCFQAAgAUIVQAAABYgVAEAAFiAUAUAAGAB9+puAABQu7WesqG6W8AldPC56OpuocbiShUAAIAFCFUAAAAWIFQBAABYgFAFAABgAUIVAACABQhVAAAAFiBUAQAAWIBQBQAAYAFCFQAAgAUIVQAAABYgVAEAAFiAUAUAAGABQhUAAIAFCFUAAAAWIFRV0uLFi9W6dWt5eXkpPDxcW7Zsqe6WAABADUCoqoQ1a9YoPj5e06dP1/bt29W1a1dFRUUpJyenulsDAADVjFBVCS+++KLGjBmjBx54QB07dtSyZcvUoEEDvf7669XdGgAAqGaEqgoqKipSRkaGIiMjzXVubm6KjIxUWlpaNXYGAABqAvfqbqC2OH78uM6ePauAgACX9QEBAdq3b1+52xQWFqqwsNB8nZ+fL0lyOp1V12gNVVL4c3W3gEuoLv4dr8v491231MV/36XnbBjGb9YRqqrQ7NmzNXPmzDLrg4ODq6Eb4NLxnV/dHQCoKnX53/fJkyfl6+t7wXFCVQU1bdpU9erVU3Z2tsv67OxsBQYGlrvN1KlTFR8fb74uKSlRbm6umjRpIpvNVqX9ovo5nU4FBwfryJEjstvt1d0OAAvx77tuMQxDJ0+eVFBQ0G/WEaoqyMPDQ2FhYUpJSdHgwYMlnQtJKSkpiouLK3cbT09PeXp6uqzz8/Or4k5R09jtdv6jC1ym+Pddd/zWFapShKpKiI+P1+jRo9W9e3f17NlT8+fPV0FBgR544IHqbg0AAFQzQlUlDBs2TD/99JMSEhLkcDjUrVs3JSUllXl4HQAA1D2EqkqKi4u74O0+4Hyenp6aPn16mVvAAGo//n2jPDbjYp8PBAAAwEUx+ScAAIAFCFUAAAAWIFQBAABYgFAFAABgAUIVUAUWL16s1q1by8vLS+Hh4dqyZUt1twTAAqmpqbrtttsUFBQkm82m9evXV3dLqEEIVYDF1qxZo/j4eE2fPl3bt29X165dFRUVpZycnOpuDcAfVFBQoK5du2rx4sXV3QpqIKZUACwWHh6uHj16aNGiRZLOfZ1RcHCwxo0bpylTplRzdwCsYrPZtG7dOvOrywCuVAEWKioqUkZGhiIjI811bm5uioyMVFpaWjV2BgCoaoQqwELHjx/X2bNny3x1UUBAgBwORzV1BQC4FAhVAAAAFiBUARZq2rSp6tWrp+zsbJf12dnZCgwMrKauAACXAqEKsJCHh4fCwsKUkpJirispKVFKSooiIiKqsTMAQFVzr+4GgMtNfHy8Ro8ere7du6tnz56aP3++CgoK9MADD1R3awD+oFOnTum7774zX2dlZSkzM1P+/v5q2bJlNXaGmoApFYAqsGjRIs2dO1cOh0PdunXTwoULFR4eXt1tAfiDPv/8c/Xv37/M+tGjR2vFihWXviHUKIQqAAAAC/BMFQAAgAUIVQAAABYgVAEAAFiAUAUAAGABQhUAAIAFCFUAAAAWIFQBAABYgFAFABVks9m0fv366m4DQA1FqAKA/5/D4dC4cePUpk0beXp6Kjg4WLfddpvLdzkCwIXw3X8AIOngwYPq3bu3/Pz8NHfuXHXp0kXFxcXauHGjYmNjtW/fvupuEUANx5UqAJD0t7/9TTabTVu2bNGQIUN09dVXq1OnToqPj9d///vfcreZPHmyrr76ajVo0EBt2rTRk08+qeLiYnP866+/Vv/+/dWoUSPZ7XaFhYVp27ZtkqRDhw7ptttuU+PGjeXj46NOnTrpo48+uiTnCqBqcKUKQJ2Xm5urpKQkzZo1Sz4+PmXG/fz8yt2uUaNGWrFihYKCgrRr1y6NGTNGjRo10uOPPy5JGjVqlK699lotXbpU9erVU2ZmpurXry9Jio2NVVFRkVJTU+Xj46NvvvlGDRs2rLJzBFD1CFUA6rzvvvtOhmEoNDS0Uts98cQT5p9bt26txx57TKtXrzZD1eHDhzVp0iRzv+3atTPrDx8+rCFDhqhLly6SpDZt2vzR0wBQzbj9B6DOMwzjd223Zs0a9e7dW4GBgWrYsKGeeOIJHT582ByPj4/XQw89pMjISD333HP6/vvvzbFHHnlEzzzzjHr37q3p06dr586df/g8AFQvQhWAOq9du3ay2WyVehg9LS1No0aN0qBBg5SYmKgdO3Zo2rRpKioqMmtmzJihPXv2KDo6Wps2bVLHjh21bt06SdJDDz2kH374Qffee6927dql7t276+WXX7b83ABcOjbj9/4vGgBcRm655Rbt2rVL+/fvL/NcVV5envz8/GSz2bRu3ToNHjxY8+bN05IlS1yuPj300EN69913lZeXV+4xRowYoYKCAn3wwQdlxqZOnaoNGzZwxQqoxbhSBQCSFi9erLNnz6pnz5567733dODAAe3du1cLFy5UREREmfp27drp8OHDWr16tb7//nstXLjQvAolSb/88ovi4uL0+eef69ChQ/ryyy+1detWdejQQZI0YcIEbdy4UVlZWdq+fbs+++wzcwxA7cSD6gCgcw+Kb9++XbNmzdKjjz6q//3vf2rWrJnCwsK0dOnSMvW33367Jk6cqLi4OBUWFio6OlpPPvmkZsyYIUmqV6+e/u///k/33XefsrOz1bRpU911112aOXOmJOns2bOKjY3V0aNHZbfbNXDgQL300kuX8pQBWIzbfwAAABbg9h8AAIAFCFUAAAAWIFQBAABYgFAFAABgAUIVAACABQhVAAAAFiBUAQAAWIBQBQAAYAFCFQAAgAUIVQAAABYgVAEAAFiAUAUAAGCB/w+g8VpXGTdFbwAAAABJRU5ErkJggg==","text/plain":["<Figure size 640x480 with 1 Axes>"]},"metadata":{},"output_type":"display_data"}],"source":["class_counts = data['label'].value_counts()\n","\n","class_counts.plot.bar()\n","plt.suptitle('')\n","plt.title('# of Samples of Each Class')\n","plt.xlabel('Class')\n","plt.ylabel('# of Samples')\n","plt.xticks(rotation = 0)\n","plt.legend('', frameon=False)\n","plt.show()"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"r8AvzHcLlof4"},"source":["#### Convert DataFrame to Dataset"]},{"cell_type":"code","execution_count":9,"metadata":{"executionInfo":{"elapsed":7,"status":"aborted","timestamp":1681825560452,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"TKpuexuklof4"},"outputs":[{"data":{"text/plain":["Dataset({\n","    features: ['text', 'label', '__index_level_0__'],\n","    num_rows: 146214\n","})"]},"execution_count":9,"metadata":{},"output_type":"execute_result"}],"source":["dataset = Dataset.from_pandas(data)\n","\n","dataset"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"m7BbSvYhlof4"},"source":["#### Split Dataset into Training Evaluation Datasets"]},{"cell_type":"code","execution_count":10,"metadata":{"executionInfo":{"elapsed":1337,"status":"aborted","timestamp":1681825560452,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"WLG2lp4olof4"},"outputs":[{"name":"stdout","output_type":"stream","text":["Training Dataset Shape: (116971, 3)\n","Evaluation Dataset Shape: (29243, 3)\n"]}],"source":["train_eval = dataset.train_test_split(train_size=0.80)\n","\n","ds = DatasetDict({\n","    'train' : train_eval['train'],\n","    'eval' : train_eval['test'],\n","})\n","\n","print(\"Training Dataset Shape:\", ds['train'].shape)\n","print(\"Evaluation Dataset Shape:\", ds['eval'].shape)"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"9nJka3nflof4"},"source":["#### Basic Values/Constants"]},{"cell_type":"code","execution_count":11,"metadata":{"executionInfo":{"elapsed":1334,"status":"aborted","timestamp":1681825560453,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"4nBu-l_Nlof4"},"outputs":[{"name":"stdout","output_type":"stream","text":["This project will utilize the CPU to train and evaluate.\n","This project will have a maximum length for the input text of 220 tokens.\n","The model checkpoint used for this project is: kssteven/ibert-roberta-base\n"]}],"source":["set_seed = 42\n","NUM_OF_EPOCHS = 1\n","\n","BATCH_SIZE = 64\n","LEARNING_RATE = 2e-5\n","\n","MAX_LENGTH = 220\n","MODEL_CKPT = \"kssteven/ibert-roberta-base\"\n","MODEL_NAME = MODEL_CKPT.split(\"/\")[-1] + \"-Abusive_Or_Threatening_Speech\"\n","\n","DEVICE = torch.device(\"cpu\")\n","\n","STRATEGY = \"epoch\"\n","REPORTS_TO = \"tensorboard\"\n","\n","print(f\"This project will utilize the {str(DEVICE).upper()} to train and evaluate.\")\n","print(f\"This project will have a maximum length for the input text of {MAX_LENGTH} tokens.\")\n","print(f\"The model checkpoint used for this project is: {MODEL_CKPT}\")"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"7liPWrBolof4"},"source":["#### Tokenize Entire Dataset"]},{"cell_type":"code","execution_count":12,"metadata":{"executionInfo":{"elapsed":4,"status":"aborted","timestamp":1681825561343,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"VSAjrx9Tlof4"},"outputs":[{"data":{"application/vnd.jupyter.widget-view+json":{"model_id":"46322330a73d41e8a8256c9333251252","version_major":2,"version_minor":0},"text/plain":["  0%|          | 0/1 [00:00<?, ?ba/s]"]},"metadata":{},"output_type":"display_data"},{"data":{"application/vnd.jupyter.widget-view+json":{"model_id":"d89bd9903e654d93bb7d2dffbdc25142","version_major":2,"version_minor":0},"text/plain":["  0%|          | 0/1 [00:00<?, ?ba/s]"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Training data shape: (116971, 5)\n","Validation data shape: (29243, 5)\n","{'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None), '__index_level_0__': Value(dtype='int64', id=None), 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None)}\n","Dataset({\n","    features: ['text', 'label', '__index_level_0__', 'input_ids', 'attention_mask'],\n","    num_rows: 116971\n","})\n"]}],"source":["# Define Tokenizer\n","tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT)\n","\n","# Tokenize Method\n","def tokenize(batch):\n","    return tokenizer(batch['text'], \n","                     padding=True, \n","                     truncation=True, \n","                     max_length=MAX_LENGTH, \n","                     return_tensors=\"pt\")\n","\n","# Tokenize entire dataset\n","encoded_data = ds.map(tokenize, batched=True, batch_size=None)\n","\n","print('Training data shape:', encoded_data['train'].shape)\n","print('Validation data shape:', encoded_data['eval'].shape)\n","\n","print(encoded_data['train'].features)\n","print(encoded_data['train'])"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"jZ0t2_Ohlof4"},"source":["#### Instantiate Model"]},{"cell_type":"code","execution_count":13,"metadata":{"executionInfo":{"elapsed":4,"status":"aborted","timestamp":1681825561344,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"vauFY6fOlof4"},"outputs":[{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at kssteven/ibert-roberta-base were not used when initializing IBertForSequenceClassification: ['lm_head.layer_norm.weight', 'ibert.pooler.dense.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.dense.weight', 'lm_head.bias', 'lm_head.decoder.weight', 'ibert.pooler.dense.weight']\n","- This IS expected if you are initializing IBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing IBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of IBertForSequenceClassification were not initialized from the model checkpoint at kssteven/ibert-roberta-base and are newly initialized: ['ibert.encoder.layer.9.pre_output_act.x_max', 'ibert.encoder.layer.11.attention.self.value_activation.x_min', 'ibert.encoder.layer.0.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.11.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.6.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.4.attention.output.output_activation.x_max', 'ibert.encoder.layer.10.output.dense.weight_integer', 'ibert.encoder.layer.3.intermediate.output_activation.x_max', 'ibert.encoder.layer.7.output.output_activation.x_max', 'ibert.encoder.layer.8.attention.self.output_activation.x_max', 'ibert.encoder.layer.7.pre_output_act.x_max', 'ibert.encoder.layer.7.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.0.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.4.output.ln_input_act.x_max', 'ibert.encoder.layer.4.output.output_activation.x_min', 'ibert.encoder.layer.7.attention.self.output_activation.x_min', 'ibert.encoder.layer.3.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.11.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.0.attention.self.value_activation.x_min', 'ibert.encoder.layer.9.attention.self.softmax.act.x_max', 'ibert.encoder.layer.3.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.11.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.4.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.5.attention.self.key.bias_integer', 'ibert.encoder.layer.11.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.4.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.9.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.11.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.4.attention.self.key_activation.x_min', 'ibert.encoder.layer.0.attention.output.output_activation.x_max', 'ibert.encoder.layer.11.attention.self.value_activation.x_max', 'ibert.encoder.layer.0.attention.self.key_activation.x_min', 'ibert.embeddings.output_activation.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.2.attention.output.output_activation.x_max', 'ibert.encoder.layer.10.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.key.weight_integer', 'ibert.encoder.layer.10.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.9.attention.self.value.bias_integer', 'ibert.encoder.layer.3.output.ln_input_act.x_min', 'ibert.encoder.layer.10.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.10.intermediate.output_activation.x_max', 'ibert.encoder.layer.2.pre_output_act.x_max', 'ibert.encoder.layer.0.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.6.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.0.output.dense.bias_integer', 'ibert.encoder.layer.3.attention.self.value.bias_integer', 'ibert.encoder.layer.2.attention.self.key.bias_integer', 'ibert.encoder.layer.3.attention.output.LayerNorm.shift', 'ibert.encoder.layer.1.attention.output.output_activation.x_max', 'ibert.encoder.layer.5.attention.output.dense.bias_integer', 'ibert.encoder.layer.0.attention.self.key.bias_integer', 'ibert.encoder.layer.6.attention.output.LayerNorm.shift', 'ibert.encoder.layer.1.pre_intermediate_act.x_min', 'ibert.encoder.layer.7.attention.output.output_activation.x_min', 'ibert.encoder.layer.6.attention.self.key_activation.x_max', 'ibert.encoder.layer.1.pre_output_act.x_max', 'ibert.encoder.layer.1.attention.self.key.bias_integer', 'ibert.encoder.layer.6.attention.self.output_activation.x_min', 'ibert.encoder.layer.7.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.7.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.9.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.6.output.ln_input_act.x_min', 'ibert.encoder.layer.9.output.ln_input_act.x_max', 'ibert.encoder.layer.3.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.1.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.value.weight_integer', 'ibert.encoder.layer.8.output.ln_input_act.x_min', 'ibert.encoder.layer.6.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.2.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.output_activation.x_max', 'ibert.encoder.layer.2.intermediate.output_activation.x_min', 'ibert.encoder.layer.5.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.9.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.5.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.1.attention.self.query.weight_integer', 'ibert.encoder.layer.5.pre_intermediate_act.x_min', 'ibert.encoder.layer.0.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.6.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.5.output.dense.fc_scaling_factor', 'ibert.encoder.layer.5.attention.self.key_activation.x_max', 'ibert.encoder.layer.0.attention.self.key_activation.x_max', 'ibert.encoder.layer.1.attention.self.output_activation.x_max', 'ibert.encoder.layer.3.attention.self.value_activation.x_min', 'ibert.encoder.layer.0.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.2.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.1.pre_intermediate_act.x_max', 'ibert.encoder.layer.3.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.2.attention.output.LayerNorm.shift', 'ibert.encoder.layer.6.attention.self.query_activation.x_max', 'ibert.encoder.layer.5.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.6.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.2.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.10.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.0.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.6.output.dense.bias_integer', 'ibert.embeddings.embeddings_act1.x_max', 'ibert.encoder.layer.8.attention.self.query_activation.x_max', 'ibert.encoder.layer.0.pre_output_act.x_max', 'ibert.encoder.layer.7.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.6.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.8.attention.output.ln_input_act.x_min', 'ibert.embeddings.embeddings_act1.x_min', 'ibert.encoder.layer.10.pre_intermediate_act.x_max', 'classifier.dense.weight', 'ibert.encoder.layer.4.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.7.output.ln_input_act.x_min', 'ibert.encoder.layer.1.intermediate.output_activation.x_min', 'ibert.encoder.layer.10.attention.self.key_activation.x_max', 'ibert.encoder.layer.11.attention.self.query_activation.x_min', 'ibert.encoder.layer.4.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.output.dense.bias_integer', 'ibert.encoder.layer.2.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.4.pre_intermediate_act.x_min', 'ibert.encoder.layer.3.output.dense.fc_scaling_factor', 'ibert.encoder.layer.6.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.9.output.dense.fc_scaling_factor', 'ibert.encoder.layer.2.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.1.attention.output.output_activation.x_min', 'ibert.encoder.layer.7.attention.self.softmax.act.x_min', 'ibert.embeddings.output_activation.x_max', 'ibert.encoder.layer.3.output.output_activation.x_max', 'ibert.encoder.layer.11.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.1.attention.self.query.bias_integer', 'ibert.encoder.layer.3.attention.self.softmax.act.x_max', 'ibert.encoder.layer.4.attention.self.key.bias_integer', 'ibert.encoder.layer.9.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.10.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.3.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.11.attention.self.value.weight_integer', 'ibert.encoder.layer.2.attention.self.key_activation.x_max', 'ibert.encoder.layer.11.attention.self.key.bias_integer', 'ibert.encoder.layer.0.attention.self.value.bias_integer', 'ibert.encoder.layer.9.attention.self.softmax.act.x_min', 'ibert.encoder.layer.10.attention.self.value.bias_integer', 'ibert.encoder.layer.8.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.11.output.dense.bias_integer', 'ibert.encoder.layer.3.intermediate.output_activation.x_min', 'ibert.encoder.layer.1.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.7.output.dense.bias_integer', 'ibert.encoder.layer.9.pre_intermediate_act.x_max', 'ibert.encoder.layer.4.attention.self.query.weight_integer', 'ibert.encoder.layer.8.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.output.LayerNorm.shift', 'ibert.encoder.layer.2.output.dense.weight_integer', 'ibert.encoder.layer.4.output.dense.bias_integer', 'ibert.encoder.layer.8.attention.self.value_activation.x_max', 'ibert.encoder.layer.11.output.LayerNorm.shift', 'ibert.encoder.layer.11.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.4.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.0.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.query_activation.x_max', 'ibert.encoder.layer.2.pre_intermediate_act.x_max', 'ibert.encoder.layer.4.attention.output.output_activation.x_min', 'ibert.encoder.layer.4.intermediate.dense.bias_integer', 'ibert.encoder.layer.5.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.7.attention.self.query.bias_integer', 'ibert.encoder.layer.2.output.LayerNorm.shift', 'ibert.encoder.layer.5.attention.self.query_activation.x_min', 'ibert.encoder.layer.4.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.11.attention.output.dense.weight_integer', 'ibert.encoder.layer.10.intermediate.dense.bias_integer', 'ibert.encoder.layer.2.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.7.attention.self.key.bias_integer', 'ibert.encoder.layer.5.attention.self.value.bias_integer', 'ibert.encoder.layer.11.attention.self.key.weight_integer', 'ibert.encoder.layer.10.attention.self.query.bias_integer', 'ibert.encoder.layer.4.attention.output.dense.fc_scaling_factor', 'ibert.embeddings.word_embeddings.weight_integer', 'ibert.encoder.layer.2.output.ln_input_act.x_max', 'ibert.encoder.layer.2.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.4.attention.self.softmax.act.x_min', 'ibert.encoder.layer.6.attention.self.key.bias_integer', 'ibert.encoder.layer.6.output.dense.weight_integer', 'ibert.encoder.layer.11.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.1.output.ln_input_act.x_min', 'ibert.encoder.layer.3.pre_output_act.x_max', 'ibert.encoder.layer.9.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.0.pre_intermediate_act.x_min', 'ibert.embeddings.embeddings_act2.x_min', 'ibert.encoder.layer.9.output.output_activation.act_scaling_factor', 'ibert.embeddings.embeddings_act1.act_scaling_factor', 'ibert.encoder.layer.7.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.11.attention.output.output_activation.x_max', 'ibert.encoder.layer.1.attention.self.key.weight_integer', 'ibert.encoder.layer.7.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.3.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.9.pre_output_act.x_min', 'ibert.encoder.layer.5.output.output_activation.x_max', 'ibert.encoder.layer.0.attention.output.dense.weight_integer', 'ibert.encoder.layer.5.intermediate.output_activation.x_min', 'ibert.encoder.layer.0.attention.self.value.weight_integer', 'ibert.encoder.layer.5.pre_intermediate_act.x_max', 'ibert.encoder.layer.7.pre_intermediate_act.x_min', 'ibert.encoder.layer.8.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.10.output.dense.fc_scaling_factor', 'ibert.encoder.layer.1.attention.output.dense.weight_integer', 'ibert.encoder.layer.4.attention.output.dense.weight_integer', 'ibert.encoder.layer.3.pre_intermediate_act.x_max', 'ibert.encoder.layer.9.output.output_activation.x_max', 'ibert.encoder.layer.8.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.11.intermediate.output_activation.x_max', 'ibert.encoder.layer.7.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.9.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.output.output_activation.x_min', 'ibert.embeddings.position_embeddings.weight_integer', 'ibert.encoder.layer.10.pre_output_act.x_min', 'ibert.encoder.layer.11.intermediate.dense.bias_integer', 'ibert.encoder.layer.6.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.7.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.2.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.7.output.dense.fc_scaling_factor', 'ibert.encoder.layer.5.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.4.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.value.weight_integer', 'ibert.encoder.layer.1.intermediate.dense.weight_integer', 'ibert.encoder.layer.2.attention.self.key.weight_integer', 'ibert.encoder.layer.9.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.key.weight_integer', 'ibert.encoder.layer.11.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.11.pre_output_act.x_min', 'ibert.encoder.layer.7.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.5.attention.output.LayerNorm.shift', 'ibert.encoder.layer.8.attention.output.LayerNorm.shift', 'ibert.encoder.layer.9.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.8.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.5.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.1.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.11.output.output_activation.x_min', 'ibert.embeddings.LayerNorm.activation.x_min', 'ibert.encoder.layer.2.intermediate.dense.bias_integer', 'ibert.encoder.layer.3.attention.self.softmax.act.x_min', 'ibert.encoder.layer.6.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.4.attention.self.value.weight_integer', 'ibert.encoder.layer.7.attention.self.value.bias_integer', 'ibert.encoder.layer.7.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.4.pre_output_act.x_min', 'ibert.encoder.layer.7.attention.output.output_activation.x_max', 'ibert.encoder.layer.9.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.query_activation.x_min', 'ibert.encoder.layer.7.attention.self.key_activation.x_max', 'ibert.encoder.layer.10.attention.output.LayerNorm.activation.x_min', 'ibert.embeddings.position_embeddings.weight_scaling_factor', 'ibert.encoder.layer.1.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.10.output.output_activation.x_max', 'ibert.encoder.layer.0.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.4.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.2.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.4.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.10.intermediate.output_activation.x_min', 'ibert.encoder.layer.5.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.3.output.dense.weight_integer', 'ibert.encoder.layer.7.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.7.intermediate.output_activation.x_min', 'ibert.encoder.layer.9.intermediate.dense.bias_integer', 'ibert.encoder.layer.1.pre_output_act.x_min', 'ibert.encoder.layer.3.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.4.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.0.pre_output_act.x_min', 'ibert.encoder.layer.5.attention.self.query_activation.x_max', 'ibert.encoder.layer.8.pre_intermediate_act.x_max', 'ibert.encoder.layer.9.attention.self.output_activation.x_max', 'ibert.encoder.layer.6.attention.output.output_activation.x_min', 'ibert.encoder.layer.8.attention.self.output_activation.x_min', 'ibert.encoder.layer.3.attention.output.dense.weight_integer', 'ibert.encoder.layer.10.attention.output.output_activation.x_max', 'ibert.encoder.layer.2.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.output.dense.weight_integer', 'ibert.encoder.layer.5.output.LayerNorm.shift', 'ibert.encoder.layer.5.pre_output_act.x_max', 'ibert.encoder.layer.2.attention.self.output_activation.x_min', 'ibert.encoder.layer.11.attention.self.softmax.act.x_min', 'ibert.encoder.layer.7.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.1.intermediate.output_activation.x_max', 'ibert.encoder.layer.11.pre_intermediate_act.x_min', 'ibert.encoder.layer.11.output.ln_input_act.x_min', 'ibert.encoder.layer.8.output.ln_input_act.x_max', 'ibert.encoder.layer.2.attention.self.output_activation.x_max', 'ibert.encoder.layer.4.output.dense.weight_integer', 'ibert.encoder.layer.3.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.6.output.LayerNorm.shift', 'ibert.encoder.layer.11.attention.self.output_activation.x_max', 'ibert.encoder.layer.3.pre_output_act.x_min', 'ibert.encoder.layer.10.attention.self.query_activation.x_max', 'ibert.encoder.layer.4.attention.self.output_activation.x_max', 'ibert.encoder.layer.0.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.4.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.key.bias_integer', 'ibert.encoder.layer.10.attention.self.value_activation.x_max', 'ibert.encoder.layer.0.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.2.output.output_activation.x_min', 'ibert.encoder.layer.4.attention.self.query.bias_integer', 'ibert.encoder.layer.11.attention.self.output_activation.x_min', 'ibert.encoder.layer.5.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.9.attention.output.LayerNorm.shift', 'ibert.encoder.layer.6.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.1.attention.self.value_activation.x_max', 'ibert.encoder.layer.4.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.8.attention.self.query.weight_integer', 'ibert.encoder.layer.2.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.11.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.9.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.8.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.5.intermediate.dense.weight_integer', 'ibert.encoder.layer.5.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.0.intermediate.dense.bias_integer', 'ibert.encoder.layer.6.attention.output.dense.bias_integer', 'ibert.encoder.layer.8.output.output_activation.x_min', 'ibert.encoder.layer.7.attention.output.LayerNorm.shift', 'ibert.encoder.layer.11.attention.self.softmax.act.x_max', 'ibert.encoder.layer.6.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.0.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.1.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.6.output.output_activation.x_max', 'ibert.encoder.layer.6.attention.self.value.bias_integer', 'ibert.encoder.layer.4.output.LayerNorm.shift', 'ibert.encoder.layer.8.pre_output_act.x_max', 'ibert.encoder.layer.9.attention.self.value.weight_integer', 'ibert.encoder.layer.6.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.value.weight_integer', 'classifier.dense.bias', 'ibert.encoder.layer.10.attention.output.output_activation.x_min', 'ibert.embeddings.output_activation.x_min', 'ibert.encoder.layer.3.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.7.output.dense.weight_integer', 'ibert.encoder.layer.0.attention.output.dense.bias_integer', 'ibert.encoder.layer.1.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.10.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.10.attention.self.key_activation.x_min', 'ibert.encoder.layer.7.intermediate.dense.bias_integer', 'ibert.encoder.layer.10.attention.self.softmax.act.x_min', 'ibert.encoder.layer.11.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.11.attention.self.query_activation.x_max', 'ibert.encoder.layer.10.attention.self.softmax.act.x_max', 'ibert.encoder.layer.6.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.value_activation.x_max', 'ibert.encoder.layer.11.output.dense.weight_integer', 'ibert.encoder.layer.0.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.query.bias_integer', 'ibert.encoder.layer.1.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.0.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.key.weight_integer', 'ibert.encoder.layer.2.pre_intermediate_act.x_min', 'ibert.encoder.layer.2.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.0.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.3.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.LayerNorm.shift', 'ibert.encoder.layer.0.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.5.intermediate.dense.bias_integer', 'ibert.encoder.layer.11.attention.self.query.bias_integer', 'ibert.encoder.layer.7.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.4.attention.self.value_activation.x_min', 'ibert.encoder.layer.11.attention.self.value.bias_integer', 'ibert.encoder.layer.0.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.0.output.ln_input_act.x_max', 'ibert.encoder.layer.9.attention.self.key.bias_integer', 'ibert.encoder.layer.1.output.ln_input_act.x_max', 'ibert.encoder.layer.1.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.self.key_activation.x_max', 'ibert.encoder.layer.5.output.dense.bias_integer', 'ibert.encoder.layer.2.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.key_activation.x_max', 'ibert.encoder.layer.11.pre_intermediate_act.x_max', 'ibert.encoder.layer.8.output.output_activation.x_max', 'ibert.encoder.layer.11.attention.self.key_activation.x_min', 'ibert.encoder.layer.6.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.7.attention.self.value_activation.x_min', 'ibert.encoder.layer.10.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.9.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.3.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.10.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.6.attention.self.value_activation.x_min', 'ibert.encoder.layer.1.attention.self.value.weight_integer', 'ibert.encoder.layer.4.attention.output.dense.bias_integer', 'ibert.encoder.layer.8.attention.self.value_activation.x_min', 'ibert.encoder.layer.9.attention.output.dense.bias_integer', 'ibert.encoder.layer.7.intermediate.output_activation.x_max', 'ibert.encoder.layer.9.attention.self.query.weight_integer', 'ibert.encoder.layer.1.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.10.attention.self.output_activation.x_max', 'ibert.encoder.layer.6.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.5.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.2.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.1.output.output_activation.x_min', 'ibert.encoder.layer.5.attention.self.value_activation.x_min', 'ibert.encoder.layer.0.attention.self.output_activation.x_min', 'ibert.embeddings.embeddings_act2.act_scaling_factor', 'ibert.encoder.layer.1.output.dense.weight_integer', 'ibert.encoder.layer.3.attention.self.key_activation.x_min', 'ibert.encoder.layer.8.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.9.output.output_activation.x_min', 'ibert.encoder.layer.8.attention.output.dense.weight_integer', 'ibert.encoder.layer.9.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.query_activation.x_min', 'ibert.encoder.layer.1.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.4.attention.output.LayerNorm.shift', 'ibert.encoder.layer.8.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.6.attention.output.dense.weight_integer', 'ibert.encoder.layer.3.output.LayerNorm.activation.act_scaling_factor', 'classifier.out_proj.weight', 'ibert.encoder.layer.1.attention.output.dense.bias_integer', 'ibert.encoder.layer.10.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.query_activation.x_min', 'ibert.encoder.layer.2.attention.output.output_activation.x_min', 'ibert.encoder.layer.5.output.output_activation.x_min', 'ibert.encoder.layer.10.attention.self.value.weight_integer', 'ibert.encoder.layer.1.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.softmax.act.x_max', 'ibert.encoder.layer.4.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.8.attention.output.dense.bias_integer', 'ibert.encoder.layer.9.intermediate.output_activation.x_max', 'ibert.encoder.layer.7.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.9.attention.self.output_activation.x_min', 'ibert.encoder.layer.11.pre_output_act.x_max', 'ibert.encoder.layer.5.attention.self.output_activation.x_min', 'ibert.encoder.layer.0.attention.self.output_activation.x_max', 'ibert.encoder.layer.6.intermediate.dense.weight_integer', 'ibert.encoder.layer.3.attention.self.output_activation.x_min', 'ibert.encoder.layer.4.attention.self.softmax.act.x_max', 'ibert.encoder.layer.2.pre_output_act.x_min', 'ibert.encoder.layer.10.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.0.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.self.softmax.act.x_max', 'ibert.encoder.layer.5.output.ln_input_act.x_min', 'ibert.encoder.layer.9.attention.self.key.weight_integer', 'ibert.encoder.layer.5.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.11.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.5.attention.self.key_activation.x_min', 'ibert.encoder.layer.4.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.10.attention.output.dense.bias_integer', 'ibert.encoder.layer.4.intermediate.output_activation.x_min', 'ibert.encoder.layer.3.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.9.output.dense.weight_integer', 'ibert.encoder.layer.10.output.LayerNorm.shift', 'ibert.encoder.layer.2.output.dense.bias_integer', 'ibert.encoder.layer.4.output.dense.fc_scaling_factor', 'ibert.encoder.layer.3.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.softmax.act.x_max', 'ibert.encoder.layer.5.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.10.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.value_activation.x_max', 'ibert.encoder.layer.1.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.1.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.7.output.ln_input_act.x_max', 'ibert.encoder.layer.8.output.dense.fc_scaling_factor', 'ibert.encoder.layer.2.intermediate.dense.weight_integer', 'ibert.encoder.layer.11.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.5.pre_output_act.x_min', 'ibert.encoder.layer.1.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.0.pre_intermediate_act.x_max', 'ibert.encoder.layer.4.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.9.attention.self.key_activation.x_max', 'ibert.encoder.layer.7.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.6.intermediate.dense.bias_integer', 'ibert.encoder.layer.5.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.9.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.5.attention.output.output_activation.x_max', 'ibert.encoder.layer.4.attention.self.value_activation.x_max', 'ibert.encoder.layer.3.attention.self.query.weight_integer', 'ibert.encoder.layer.4.pre_output_act.x_max', 'ibert.encoder.layer.3.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.5.intermediate.output_activation.x_max', 'ibert.encoder.layer.6.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.2.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.0.output.dense.fc_scaling_factor', 'ibert.encoder.layer.4.attention.self.output_activation.x_min', 'ibert.encoder.layer.7.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.7.output.LayerNorm.activation.x_max', 'ibert.embeddings.word_embeddings.weight_scaling_factor', 'ibert.encoder.layer.9.attention.self.query.bias_integer', 'ibert.embeddings.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.7.attention.self.query_activation.x_max', 'ibert.encoder.layer.1.attention.self.value_activation.x_min', 'ibert.encoder.layer.11.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.0.attention.output.LayerNorm.shift', 'ibert.encoder.layer.1.attention.self.key_activation.x_min', 'ibert.encoder.layer.7.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.key.weight_integer', 'ibert.encoder.layer.0.attention.output.output_activation.x_min', 'ibert.encoder.layer.3.attention.self.output_activation.x_max', 'ibert.encoder.layer.3.intermediate.dense.weight_integer', 'ibert.encoder.layer.10.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.11.output.output_activation.x_max', 'ibert.encoder.layer.0.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.9.attention.self.key_activation.x_min', 'ibert.encoder.layer.3.attention.self.query.bias_integer', 'ibert.encoder.layer.6.attention.self.key_activation.x_min', 'classifier.out_proj.bias', 'ibert.encoder.layer.4.output.ln_input_act.x_min', 'ibert.encoder.layer.1.attention.self.value.bias_integer', 'ibert.encoder.layer.3.output.ln_input_act.x_max', 'ibert.encoder.layer.0.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.7.pre_intermediate_act.x_max', 'ibert.encoder.layer.0.output.dense.weight_integer', 'ibert.encoder.layer.10.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.6.output.output_activation.x_min', 'ibert.encoder.layer.4.attention.self.value.bias_integer', 'ibert.encoder.layer.7.attention.self.query.weight_integer', 'ibert.encoder.layer.1.attention.self.key_activation.x_max', 'ibert.encoder.layer.9.output.ln_input_act.x_min', 'ibert.encoder.layer.7.output.output_activation.x_min', 'ibert.encoder.layer.3.output.output_activation.x_min', 'ibert.encoder.layer.2.attention.self.softmax.act.x_max', 'ibert.encoder.layer.7.output.LayerNorm.shift', 'ibert.encoder.layer.10.output.ln_input_act.x_max', 'ibert.encoder.layer.6.output.dense.fc_scaling_factor', 'ibert.encoder.layer.4.attention.self.query_activation.x_min', 'ibert.encoder.layer.4.attention.self.query_activation.x_max', 'ibert.encoder.layer.8.attention.self.softmax.act.x_min', 'ibert.encoder.layer.1.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.2.output.ln_input_act.x_min', 'ibert.encoder.layer.1.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.7.intermediate.dense.weight_integer', 'ibert.encoder.layer.7.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.7.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.10.attention.self.query_activation.x_min', 'ibert.encoder.layer.8.pre_output_act.x_min', 'ibert.encoder.layer.3.attention.self.key.bias_integer', 'ibert.encoder.layer.0.attention.self.softmax.act.x_min', 'ibert.encoder.layer.3.pre_intermediate_act.x_min', 'ibert.encoder.layer.4.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.6.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.7.attention.self.key_activation.x_min', 'ibert.encoder.layer.9.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.8.attention.output.output_activation.x_min', 'ibert.encoder.layer.10.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.11.intermediate.dense.weight_integer', 'ibert.encoder.layer.2.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.11.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.6.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.7.attention.self.query_activation.x_min', 'ibert.encoder.layer.9.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.4.attention.self.key.weight_integer', 'ibert.encoder.layer.6.pre_intermediate_act.x_max', 'ibert.encoder.layer.9.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.10.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.5.output.dense.weight_integer', 'ibert.encoder.layer.4.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.10.output.output_activation.x_min', 'ibert.encoder.layer.9.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.4.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.8.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.3.output.LayerNorm.shift', 'ibert.encoder.layer.6.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.7.attention.output.dense.bias_integer', 'ibert.encoder.layer.1.attention.self.softmax.act.x_min', 'ibert.encoder.layer.0.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.output.output_activation.x_max', 'ibert.encoder.layer.6.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.3.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.7.attention.self.key.weight_integer', 'ibert.encoder.layer.11.attention.self.query.weight_integer', 'ibert.encoder.layer.0.intermediate.dense.weight_integer', 'ibert.encoder.layer.3.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.3.attention.self.query_activation.x_min', 'ibert.encoder.layer.5.attention.output.dense.weight_integer', 'ibert.encoder.layer.8.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.1.output.LayerNorm.shift', 'ibert.encoder.layer.7.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.0.attention.self.query_activation.x_min', 'ibert.encoder.layer.3.attention.self.value.weight_integer', 'ibert.encoder.layer.9.attention.self.value_activation.x_max', 'ibert.encoder.layer.2.output.output_activation.x_max', 'ibert.encoder.layer.2.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.1.attention.self.query_activation.x_min', 'ibert.encoder.layer.6.pre_intermediate_act.x_min', 'ibert.encoder.layer.10.output.dense.bias_integer', 'ibert.encoder.layer.6.pre_output_act.x_max', 'ibert.encoder.layer.0.attention.self.query.bias_integer', 'ibert.embeddings.token_type_embeddings.weight_scaling_factor', 'ibert.encoder.layer.0.output.ln_input_act.x_min', 'ibert.encoder.layer.0.output.output_activation.x_min', 'ibert.encoder.layer.1.attention.self.query_activation.x_max', 'ibert.encoder.layer.2.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.3.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.query.weight_integer', 'ibert.encoder.layer.7.attention.self.softmax.act.x_max', 'ibert.encoder.layer.10.pre_output_act.x_max', 'ibert.encoder.layer.0.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.1.output.dense.fc_scaling_factor', 'ibert.encoder.layer.0.attention.self.key.weight_integer', 'ibert.encoder.layer.4.attention.self.key_activation.x_max', 'ibert.encoder.layer.8.attention.self.query.bias_integer', 'ibert.encoder.layer.9.output.dense.bias_integer', 'ibert.encoder.layer.8.output.dense.weight_integer', 'ibert.encoder.layer.6.attention.self.softmax.act.x_min', 'ibert.encoder.layer.10.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.8.intermediate.output_activation.x_min', 'ibert.encoder.layer.2.intermediate.output_activation.x_max', 'ibert.encoder.layer.0.attention.self.softmax.act.x_max', 'ibert.encoder.layer.2.attention.self.softmax.act.x_min', 'ibert.encoder.layer.6.pre_output_act.x_min', 'ibert.encoder.layer.3.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.7.attention.output.dense.weight_integer', 'ibert.encoder.layer.11.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.0.attention.self.query_activation.x_max', 'ibert.encoder.layer.4.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.output.output_activation.x_min', 'ibert.encoder.layer.5.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.5.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.6.attention.self.query.bias_integer', 'ibert.encoder.layer.0.attention.self.value_activation.x_max', 'ibert.encoder.layer.2.output.dense.fc_scaling_factor', 'ibert.encoder.layer.7.attention.self.value_activation.x_max', 'ibert.encoder.layer.7.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.output_activation.x_max', 'ibert.encoder.layer.10.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.7.attention.self.value.weight_integer', 'ibert.encoder.layer.2.attention.self.query.bias_integer', 'ibert.encoder.layer.6.attention.self.softmax.act.x_max', 'ibert.encoder.layer.8.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.10.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.10.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.6.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.1.output.dense.bias_integer', 'ibert.encoder.layer.9.attention.output.output_activation.x_min', 'ibert.encoder.layer.8.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.4.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.9.attention.output.dense.weight_integer', 'ibert.encoder.layer.10.pre_intermediate_act.x_min', 'ibert.encoder.layer.3.attention.output.output_activation.x_max', 'ibert.encoder.layer.0.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.output_activation.x_min', 'ibert.encoder.layer.4.output.output_activation.x_max', 'ibert.encoder.layer.2.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.key_activation.x_min', 'ibert.encoder.layer.11.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.10.attention.output.LayerNorm.shift', 'ibert.encoder.layer.7.attention.self.output_activation.x_max', 'ibert.encoder.layer.7.pre_output_act.x_min', 'ibert.encoder.layer.9.attention.self.value_activation.x_min', 'ibert.encoder.layer.10.output.ln_input_act.x_min', 'ibert.encoder.layer.3.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.9.intermediate.dense.weight_integer', 'ibert.encoder.layer.8.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.self.query_activation.x_max', 'ibert.encoder.layer.2.attention.output.dense.bias_integer', 'ibert.encoder.layer.4.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.8.attention.self.key.bias_integer', 'ibert.embeddings.token_type_embeddings.weight_integer', 'ibert.encoder.layer.0.attention.self.query.weight_integer', 'ibert.encoder.layer.7.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.4.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.1.attention.self.softmax.act.act_scaling_factor', 'ibert.embeddings.LayerNorm.shift', 'ibert.encoder.layer.8.attention.output.output_activation.x_max', 'ibert.encoder.layer.8.attention.self.value.bias_integer', 'ibert.encoder.layer.5.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.4.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.7.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.1.output.output_activation.x_max', 'ibert.encoder.layer.3.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.9.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.4.intermediate.dense.weight_integer', 'ibert.encoder.layer.5.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.9.attention.self.query_activation.x_min', 'ibert.encoder.layer.5.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.5.attention.self.query.weight_integer', 'ibert.encoder.layer.5.attention.self.softmax.act.x_min', 'ibert.encoder.layer.11.output.dense.fc_scaling_factor', 'ibert.encoder.layer.8.intermediate.output_activation.x_max', 'ibert.encoder.layer.0.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.1.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.9.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.5.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.self.key.weight_integer', 'ibert.encoder.layer.11.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.0.intermediate.output_activation.x_min', 'ibert.encoder.layer.10.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.6.intermediate.output_activation.x_min', 'ibert.encoder.layer.5.attention.self.value.weight_integer', 'ibert.encoder.layer.8.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.9.output.LayerNorm.shift', 'ibert.encoder.layer.4.pre_intermediate_act.x_max', 'ibert.encoder.layer.9.intermediate.output_activation.x_min', 'ibert.encoder.layer.5.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.8.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.1.attention.self.output_activation.x_min', 'ibert.encoder.layer.2.attention.self.value_activation.x_min', 'ibert.encoder.layer.5.attention.output.output_activation.x_min', 'ibert.encoder.layer.6.output.ln_input_act.x_max', 'ibert.encoder.layer.3.intermediate.dense.bias_integer', 'ibert.encoder.layer.2.attention.self.query.weight_integer', 'ibert.encoder.layer.10.attention.self.query.weight_integer', 'ibert.encoder.layer.11.output.ln_input_act.x_max', 'ibert.encoder.layer.3.attention.self.value_activation.x_max', 'ibert.encoder.layer.2.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.value.bias_integer', 'ibert.encoder.layer.7.output.output_activation.act_scaling_factor', 'ibert.embeddings.LayerNorm.activation.x_max', 'ibert.encoder.layer.6.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.9.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.value_activation.x_max', 'ibert.encoder.layer.9.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.output.dense.bias_integer', 'ibert.encoder.layer.3.output.dense.bias_integer', 'ibert.encoder.layer.8.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.5.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.5.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.self.query_activation.x_max', 'ibert.encoder.layer.8.output.dense.bias_integer', 'ibert.encoder.layer.8.pre_intermediate_act.x_min', 'ibert.encoder.layer.0.output.LayerNorm.shift', 'ibert.encoder.layer.0.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.8.attention.self.key_activation.x_min', 'ibert.encoder.layer.11.attention.output.LayerNorm.shift', 'ibert.embeddings.embeddings_act2.x_max', 'ibert.encoder.layer.8.intermediate.dense.bias_integer', 'ibert.encoder.layer.5.output.ln_input_act.x_max', 'ibert.encoder.layer.1.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.3.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.4.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.6.attention.output.output_activation.x_max', 'ibert.encoder.layer.1.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.10.intermediate.dense.weight_integer', 'ibert.encoder.layer.0.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.8.intermediate.dense.weight_integer', 'ibert.encoder.layer.0.intermediate.output_activation.x_max', 'ibert.encoder.layer.9.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.10.attention.self.value_activation.x_min', 'ibert.encoder.layer.2.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.11.intermediate.output_activation.x_min', 'ibert.encoder.layer.1.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.3.attention.self.key_activation.x_max', 'ibert.encoder.layer.0.output.output_activation.x_max', 'ibert.encoder.layer.6.intermediate.output_activation.x_max', 'ibert.encoder.layer.9.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.1.intermediate.dense.bias_integer', 'ibert.encoder.layer.2.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.4.intermediate.output_activation.x_max', 'ibert.encoder.layer.1.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.10.attention.output.dense.weight_integer', 'ibert.encoder.layer.9.pre_intermediate_act.x_min']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]}],"source":["model = AutoModelForSequenceClassification.from_pretrained(\n","    MODEL_CKPT,\n","    num_labels=NUM_OF_LABELS,\n","    id2label=id2label,\n","    label2id=label2id\n","    ).to(DEVICE)"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"AkIZ7Q8Llof5"},"source":["#### Define Function to Compute Metrics"]},{"cell_type":"code","execution_count":14,"metadata":{"executionInfo":{"elapsed":5,"status":"aborted","timestamp":1681825561345,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"ZS-cTQ9plof5"},"outputs":[],"source":["def compute_metrics(p):\n","    accuracy_metric = evaluate.load(\"accuracy\")\n","    accuracy = accuracy_metric.compute(predictions=np.argmax(p.predictions, axis=1), \n","                                       references=p.label_ids)['accuracy']\n","    \n","    ### ------------------- F1 scores -------------------\n","    \n","    f1_score_metric = evaluate.load(\"f1\")\n","    f1_score = f1_score_metric.compute(predictions=np.argmax(p.predictions, axis=1), \n","                                       references=p.label_ids)[\"f1\"]\n","    \n","    ### ------------------- recall -------------------\n","    \n","    recall_metric = evaluate.load(\"recall\")\n","    recall = recall_metric.compute(predictions=np.argmax(p.predictions, axis=1), \n","                                   references=p.label_ids)[\"recall\"]\n","    \n","    ### ------------------- precision -------------------\n","    \n","    precision_metric = evaluate.load(\"precision\")\n","    precision = precision_metric.compute(predictions=np.argmax(p.predictions, axis=1), \n","                                         references=p.label_ids)[\"precision\"]\n","    \n","    return {\"accuracy\" : accuracy, \n","            \"F1\" : f1_score,\n","            \"Recall\" : recall,\n","            \"Precision\" : precision,\n","            }"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"1EYt1FZslof5"},"source":["#### Define TrainingArguments"]},{"cell_type":"code","execution_count":15,"metadata":{"executionInfo":{"elapsed":5,"status":"aborted","timestamp":1681825561345,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"WP3ndiTRlof5"},"outputs":[],"source":["training_args = TrainingArguments(\n","    output_dir=MODEL_NAME,\n","    num_train_epochs=NUM_OF_EPOCHS,\n","    learning_rate=LEARNING_RATE,\n","    per_device_train_batch_size=BATCH_SIZE, \n","    per_device_eval_batch_size=BATCH_SIZE, \n","    weight_decay=0.01,\n","    report_to=REPORTS_TO,\n","    log_level=\"error\",\n","    logging_strategy='steps', \n","    logging_steps=50,\n","    evaluation_strategy=STRATEGY,\n","    save_strategy=STRATEGY,\n","    logging_first_step=True,\n","    load_best_model_at_end=True,\n","    metric_for_best_model=\"eval_F1\",\n","    greater_is_better=True,\n","    disable_tqdm=False,\n","    hub_private_repo=True,\n","    push_to_hub=True\n",")"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"3sPPrpwIlof5"},"source":["#### Define Data Collator"]},{"cell_type":"code","execution_count":16,"metadata":{"executionInfo":{"elapsed":5,"status":"aborted","timestamp":1681825561345,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"kJ05TD8rlof5"},"outputs":[],"source":["data_collator = DataCollatorWithPadding(tokenizer, padding=True)"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"uzEpgs7Slof5"},"source":["#### Subclass Trainer to Handle Class Imbalance"]},{"cell_type":"code","execution_count":17,"metadata":{"id":"P93xfT6Vlof5"},"outputs":[],"source":["class CustomTrainer(Trainer):\n","    def compute_loss(self, model, inputs, return_outputs=False):\n","        labels = inputs.get(\"labels\")\n","        outputs = model(**inputs)\n","        logits = outputs.get(\"logits\")\n","        loss_fct = torch.nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0]))\n","        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))\n","        return (loss, outputs) if return_outputs else loss"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"0JlxDWivlof5"},"source":["#### Instantiate Trainer"]},{"cell_type":"code","execution_count":18,"metadata":{"id":"TUOElPI1lof5","outputId":"2d90b472-5ff9-4bc8-c550-5f778b329504"},"outputs":[{"name":"stderr","output_type":"stream","text":["Cloning https://huggingface.co/DunnBC22/ibert-roberta-base-Abusive_Or_Threatening_Speech into local empty directory.\n"]}],"source":["trainer = CustomTrainer(model=model, \n","                  args=training_args, \n","                  compute_metrics=compute_metrics,\n","                  train_dataset=encoded_data['train'], \n","                  eval_dataset=encoded_data['eval'], \n","                  data_collator=data_collator,\n","                  tokenizer=tokenizer)"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"umV50LBJlof5"},"source":["#### Train Model"]},{"cell_type":"code","execution_count":19,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":166},"executionInfo":{"elapsed":6,"status":"error","timestamp":1681825562575,"user":{"displayName":"hill billy","userId":"11912902376012624422"},"user_tz":300},"id":"8Nd7rywHlof5","outputId":"148377c5-7c4e-458f-e492-46da70a5c165"},"outputs":[{"name":"stderr","output_type":"stream","text":["/Users/leedunn/Documents/nlpnn/nlp/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n","  warnings.warn(\n"]},{"data":{"application/vnd.jupyter.widget-view+json":{"model_id":"29708929e86e4a958dbe8f151a145294","version_major":2,"version_minor":0},"text/plain":["  0%|          | 0/1828 [00:00<?, ?it/s]"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["{'loss': 0.7234, 'learning_rate': 1.9989059080962803e-05, 'epoch': 0.0}\n","{'loss': 0.3222, 'learning_rate': 1.9452954048140046e-05, 'epoch': 0.03}\n","{'loss': 0.132, 'learning_rate': 1.890590809628009e-05, 'epoch': 0.05}\n","{'loss': 0.1327, 'learning_rate': 1.8358862144420135e-05, 'epoch': 0.08}\n","{'loss': 0.1067, 'learning_rate': 1.7811816192560176e-05, 'epoch': 0.11}\n","{'loss': 0.1085, 'learning_rate': 1.726477024070022e-05, 'epoch': 0.14}\n","{'loss': 0.0917, 'learning_rate': 1.6717724288840264e-05, 'epoch': 0.16}\n","{'loss': 0.1098, 'learning_rate': 1.617067833698031e-05, 'epoch': 0.19}\n","{'loss': 0.1011, 'learning_rate': 1.562363238512035e-05, 'epoch': 0.22}\n","{'loss': 0.0979, 'learning_rate': 1.5076586433260396e-05, 'epoch': 0.25}\n","{'loss': 0.1076, 'learning_rate': 1.452954048140044e-05, 'epoch': 0.27}\n","{'loss': 0.1102, 'learning_rate': 1.3982494529540483e-05, 'epoch': 0.3}\n","{'loss': 0.0987, 'learning_rate': 1.3435448577680525e-05, 'epoch': 0.33}\n","{'loss': 0.0958, 'learning_rate': 1.2888402625820571e-05, 'epoch': 0.36}\n","{'loss': 0.0906, 'learning_rate': 1.2341356673960614e-05, 'epoch': 0.38}\n","{'loss': 0.1108, 'learning_rate': 1.1794310722100657e-05, 'epoch': 0.41}\n","{'loss': 0.1282, 'learning_rate': 1.12472647702407e-05, 'epoch': 0.44}\n","{'loss': 0.0975, 'learning_rate': 1.0700218818380745e-05, 'epoch': 0.46}\n","{'loss': 0.0944, 'learning_rate': 1.0153172866520788e-05, 'epoch': 0.49}\n","{'loss': 0.088, 'learning_rate': 9.606126914660832e-06, 'epoch': 0.52}\n","{'loss': 0.0919, 'learning_rate': 9.059080962800875e-06, 'epoch': 0.55}\n","{'loss': 0.0934, 'learning_rate': 8.51203501094092e-06, 'epoch': 0.57}\n","{'loss': 0.0924, 'learning_rate': 7.964989059080964e-06, 'epoch': 0.6}\n","{'loss': 0.0975, 'learning_rate': 7.417943107221007e-06, 'epoch': 0.63}\n","{'loss': 0.0857, 'learning_rate': 6.87089715536105e-06, 'epoch': 0.66}\n","{'loss': 0.0917, 'learning_rate': 6.323851203501094e-06, 'epoch': 0.68}\n","{'loss': 0.0924, 'learning_rate': 5.776805251641139e-06, 'epoch': 0.71}\n","{'loss': 0.0873, 'learning_rate': 5.229759299781182e-06, 'epoch': 0.74}\n","{'loss': 0.0732, 'learning_rate': 4.682713347921226e-06, 'epoch': 0.77}\n","{'loss': 0.1094, 'learning_rate': 4.135667396061269e-06, 'epoch': 0.79}\n","{'loss': 0.0918, 'learning_rate': 3.5886214442013136e-06, 'epoch': 0.82}\n","{'loss': 0.0919, 'learning_rate': 3.041575492341357e-06, 'epoch': 0.85}\n","{'loss': 0.0875, 'learning_rate': 2.4945295404814006e-06, 'epoch': 0.88}\n","{'loss': 0.0868, 'learning_rate': 1.9474835886214445e-06, 'epoch': 0.9}\n","{'loss': 0.0969, 'learning_rate': 1.400437636761488e-06, 'epoch': 0.93}\n","{'loss': 0.0973, 'learning_rate': 8.533916849015319e-07, 'epoch': 0.96}\n","{'loss': 0.0771, 'learning_rate': 3.063457330415755e-07, 'epoch': 0.98}\n"]},{"data":{"application/vnd.jupyter.widget-view+json":{"model_id":"e1fd02394b394b55816d47cb1f5db2f6","version_major":2,"version_minor":0},"text/plain":["  0%|          | 0/457 [00:00<?, ?it/s]"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["{'eval_loss': 0.08016789704561234, 'eval_accuracy': 0.9741476592688849, 'eval_F1': 0.7772539776075428, 'eval_Recall': 0.8609660574412533, 'eval_Precision': 0.7083780880773362, 'eval_runtime': 13441.5223, 'eval_samples_per_second': 2.176, 'eval_steps_per_second': 0.034, 'epoch': 1.0}\n","{'train_runtime': 331532.8168, 'train_samples_per_second': 0.353, 'train_steps_per_second': 0.006, 'train_loss': 0.10460054101646599, 'epoch': 1.0}\n"]}],"source":["train_results = trainer.train()"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"-KmSywCHlof5"},"source":["#### Save Model & Metrics on Training Dataset"]},{"cell_type":"code","execution_count":20,"metadata":{"colab":{"referenced_widgets":["2bd1bb076e20406eb5a5cf5bbc492cae","4e622a4c359a46569dc449769c71d646"]},"id":"gJ3iXuR-lof5","outputId":"68a43ad8-999a-4724-9d97-d5490d22f341"},"outputs":[{"name":"stderr","output_type":"stream","text":["Several commits (2) will be pushed upstream.\n","The progress bars may be unreliable.\n"]},{"data":{"application/vnd.jupyter.widget-view+json":{"model_id":"a275a59c2fd44cdb8c4b0dccd883bc10","version_major":2,"version_minor":0},"text/plain":["Upload file pytorch_model.bin:   0%|          | 1.00/949M [00:00<?, ?B/s]"]},"metadata":{},"output_type":"display_data"},{"data":{"application/vnd.jupyter.widget-view+json":{"model_id":"77187b3ae4a84f3bb8cf49874c52c547","version_major":2,"version_minor":0},"text/plain":["Upload file runs/May05_15-41-09_Lees-Air/events.out.tfevents.1683319277.Lees-Air.57344.0:   0%|          | 1.0…"]},"metadata":{},"output_type":"display_data"},{"name":"stderr","output_type":"stream","text":["To https://huggingface.co/DunnBC22/ibert-roberta-base-Abusive_Or_Threatening_Speech\n","   9d0356a..2c198db  main -> main\n","\n","To https://huggingface.co/DunnBC22/ibert-roberta-base-Abusive_Or_Threatening_Speech\n","   2c198db..c1cbd37  main -> main\n","\n"]},{"name":"stdout","output_type":"stream","text":["***** train metrics *****\n","  epoch                    =                 1.0\n","  train_loss               =              0.1046\n","  train_runtime            = 3 days, 20:05:32.81\n","  train_samples_per_second =               0.353\n","  train_steps_per_second   =               0.006\n"]}],"source":["trainer.save_model()\n","trainer.log_metrics(\"train\", train_results.metrics)\n","trainer.save_metrics(\"train\", train_results.metrics)\n","trainer.save_state()"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"fCkk__fqlof6"},"source":["#### Push Model to HuggingFace Hub"]},{"cell_type":"code","execution_count":21,"metadata":{"id":"BpR4tiIFlof6"},"outputs":[{"name":"stderr","output_type":"stream","text":["To https://huggingface.co/DunnBC22/ibert-roberta-base-Abusive_Or_Threatening_Speech\n","   c1cbd37..987e7dd  main -> main\n","\n"]},{"data":{"text/plain":["'https://huggingface.co/DunnBC22/ibert-roberta-base-Abusive_Or_Threatening_Speech/commit/987e7ddf3b7455cce3513272f2ad1fc7fab3db5a'"]},"execution_count":21,"metadata":{},"output_type":"execute_result"}],"source":["trainer.push_to_hub(commit_message=\"All Dunn!!!\")"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"GpNxw--Alof5"},"source":["#### Evaluate Model"]},{"cell_type":"code","execution_count":22,"metadata":{"colab":{"referenced_widgets":["27c83af28dab49b89cb08e72925344c5"]},"id":"QcHxdny1lof6","outputId":"47bb1af0-1553-466b-86c8-673d8a1e99dd"},"outputs":[{"data":{"application/vnd.jupyter.widget-view+json":{"model_id":"07f51adc79514373a336e3f9f430fe8b","version_major":2,"version_minor":0},"text/plain":["  0%|          | 0/457 [00:00<?, ?it/s]"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["***** eval metrics *****\n","  epoch                   =        1.0\n","  eval_F1                 =     0.7773\n","  eval_Precision          =     0.7084\n","  eval_Recall             =      0.861\n","  eval_accuracy           =     0.9741\n","  eval_loss               =     0.0802\n","  eval_runtime            = 3:40:13.33\n","  eval_samples_per_second =      2.213\n","  eval_steps_per_second   =      0.035\n"]}],"source":["metrics = trainer.evaluate()\n","trainer.log_metrics(\"eval\", metrics)\n","trainer.save_metrics(\"eval\", metrics)"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"RnF_C77Wlof6"},"source":["### Notes & Other Takeaways From This Project\n","****\n","- These text inputs are slightly different than the inputs from other projects that I have trained. The inputs are not spam and the intent is not to measure sentiment, but rather this project seeks to find any reviews that need to be flagged, reviewed and potentially removed. \n","****"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"SAgsejO9lof6"},"source":["### Citations\n","- Model Checkpoint\n","    > @article{kim2021bert, title={I-BERT: Integer-only BERT Quantization}, author={Kim, Sehoon and Gholami, Amir and Yao, Zhewei and Mahoney, Michael W and Keutzer, Kurt}, journal={arXiv preprint arXiv:2101.01321}, year={2021}}\n","\n"]}],"metadata":{"accelerator":"GPU","colab":{"provenance":[]},"gpuClass":"standard","kernelspec":{"display_name":"nlp_ml","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.7"},"orig_nbformat":4,"vscode":{"interpreter":{"hash":"0b7a754c5aaaa44a21e2f2148f50b931a5abdffe7ef71537f5954179554f39df"}}},"nbformat":4,"nbformat_minor":0}