Causal Language Modeling/2000 Clean Medical Articles/2,000 Clean Medical Articles - CLM.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2,000 Cleaned Medical Articles: Causal Language Modeling\n",
    "\n",
    "Dataset Source: https://www.kaggle.com/datasets/trikialaaa/2k-clean-medical-articles-medicalnewstoday"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Necessary & Missing Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "NOTE: Redirects are currently not supported in Windows or MacOs.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Git LFS initialized.\n"
     ]
    }
   ],
   "source": [
    "import os, sys, glob, math\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import datasets\n",
    "from datasets import ClassLabel, Dataset, DatasetDict\n",
    "\n",
    "import transformers\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM, DataCollatorForLanguageModeling\n",
    "from transformers import TrainingArguments, Trainer, set_seed\n",
    "\n",
    "!git lfs install"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Library Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        Python: 3.9.12\n",
      "        Pandas: 1.5.0\n",
      "      Datasets: 2.9.0\n",
      "  Transformers: 4.26.1\n"
     ]
    }
   ],
   "source": [
    "print(\"Python:\".rjust(15), sys.version[0:6])\n",
    "print(\"Pandas:\".rjust(15), pd.__version__)\n",
    "print(\"Datasets:\".rjust(15), datasets.__version__)\n",
    "print(\"Transformers:\".rjust(15), transformers.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1979 entries, 0 to 0\n",
      "Data columns (total 1 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   text    1979 non-null   object\n",
      "dtypes: object(1)\n",
      "memory usage: 30.9+ KB\n"
     ]
    }
   ],
   "source": [
    "parent_dir = \"/Users/briandunn/Desktop/nlp_ds/Causal Language Modeling/2k Clean Medical Articles/articles\"\n",
    "\n",
    "data_values = [pd.read_csv(filename, sep=\"\\t\", engine=\"c\", header=None, names=[\"text\"]) for filename in glob.glob(os.path.join(parent_dir, \"*txt\"))]\n",
    "\n",
    "data = pd.concat(data_values)\n",
    "\n",
    "data = data.drop_duplicates()\n",
    "data = data.dropna()\n",
    "\n",
    "data = data.apply(lambda x: x.str.strip())\n",
    "\n",
    "data.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visual: Histogram of Input Lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The maximum length of this dataset is approximately 17093 words.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABZkAAAK9CAYAAABGqWLxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAA9hAAAPYQGoP6dpAABIPklEQVR4nO3debxVZb348e9hOoznAMoYMyIoiiYqclUGRZG4DmWO5GylomYaKtergBOo3fRWRo5gqTiUaDmjApYoCohzCIRCTtxMOAx6QM76/dGL/WPLeB44cA6836/Xfsle+1lrPfvwuDt9Wq1dkGVZFgAAAAAAkKDatp4AAAAAAABVl8gMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAsJ1r165dnH766dt6Gtu9m266KTp06BDVq1ePvffee1tPZ4dTUFAQ559//raeBgDADklkBgCoQsaOHRsFBQUxbdq0db7ep0+f2GOPPTb7PE8++WQMHz58s4+zo3j22Wfj0ksvjQMPPDDGjBkT119//VpjJk2aFAUFBZv02FKmTJkSw4cPj0WLFm3S+NNPPz3q16+/xc6/pZX3/QAAsHXU2NYTAACgYs2aNSuqVSvftQVPPvlk3HrrrULzJnrhhReiWrVqcdddd0WtWrXWOWa33XaL3//+93nbhg4dGvXr148rrriiQuY1ZcqUGDFiRJx++unRsGHDCjnH1rS9vR8AgO2FyAwAsJ0rLCzc1lMot2XLlkW9evW29TQ22cKFC6NOnTrrDcwREc2aNYsf/OAHedtGjRoVO++881rbAQCgKnG7DACA7dw378m8cuXKGDFiRHTq1Clq164dO+20Uxx00EExYcKEiPj3LRNuvfXWiIh13sJh2bJlcckll0Tr1q2jsLAwOnfuHD//+c8jy7K883755Zdx4YUXxs477xwNGjSIo446Kj766KMoKCjIu0J6+PDhUVBQEO+++26cfPLJ0ahRozjooIMiIuLNN9+M008/PTp06BC1a9eO5s2bx5lnnhmff/553rlWH+P999+PH/zgB1FcXBxNmjSJK6+8MrIsiwULFsTRRx8dRUVF0bx58/if//mfTfrZff3113HNNddEx44do7CwMNq1axf/9V//FaWlpbkxBQUFMWbMmFi2bFnuZzV27NhNOv66LFq0KC666KLcz3eXXXaJG264IcrKyiIiIsuy6Nu3bzRp0iQWLlyY22/FihWx5557RseOHWPZsmUxfPjwGDJkSEREtG/fPje3Dz74IHluq02dOjWOOOKIKC4ujrp160bv3r3jpZdeyhuz+u9kzpw5uSuPi4uL44wzzojly5fnjd2UtbKp7+fRRx+NPfbYIwoLC6Nr167x9NNPb/b7BQBgw1zJDABQBS1evDj++c9/rrV95cqVG913+PDhMXLkyDj77LNj//33j5KSkpg2bVrMmDEjDjvssPjxj38cH3/8cUyYMGGt2ztkWRZHHXVUTJw4Mc4666zYe++945lnnokhQ4bERx99FDfffHNu7Omnnx4PPfRQnHLKKXHAAQfE5MmTY+DAgeud13HHHRedOnWK66+/PhesJ0yYEH//+9/jjDPOiObNm8c777wTt99+e7zzzjvxyiuvrHX/4hNOOCF22223GDVqVDzxxBNx7bXXRuPGjeO2226LQw45JG644Ya477774mc/+1nst99+0atXrw3+rM4+++y455574vvf/35ccsklMXXq1Bg5cmS89957MX78+IiI+P3vfx+33357vPrqq3HnnXdGRMR//Md/bPTvYV2WL18evXv3jo8++ih+/OMfR5s2bWLKlCkxdOjQ+OSTT+KWW26JgoKCuPvuu6Nbt25xzjnnxCOPPBIREcOGDYt33nknJk2aFPXq1Yvvfe978f7778e4cePi5ptvjp133jkiIpo0aZI0t9VeeOGFGDBgQHTv3j2GDRsW1apVizFjxsQhhxwSf/nLX2L//ffPG3/88cdH+/btY+TIkTFjxoy48847o2nTpnHDDTfkxmzKWtmU9/PXv/41HnnkkTjvvPOiQYMG8ctf/jKOPfbYmD9/fuy0006b9b4BANiADACAKmPMmDFZRGzw0bVr17x92rZtm5122mm553vttVc2cODADZ5n8ODB2bp+VXz00UeziMiuvfbavO3f//73s4KCgmzOnDlZlmXZ9OnTs4jILrroorxxp59+ehYR2bBhw3Lbhg0blkVEdtJJJ611vuXLl6+1bdy4cVlEZC+++OJax/jRj36U2/b1119nrVq1ygoKCrJRo0bltn/xxRdZnTp18n4m6zJz5swsIrKzzz47b/vPfvazLCKyF154IbfttNNOy+rVq7fB461L165ds969e+eeX3PNNVm9evWy999/P2/c5ZdfnlWvXj2bP39+btttt92WRUR27733Zq+88kpWvXr1tX7eN910UxYR2bx58zZpPht7H2VlZVmnTp2y/v37Z2VlZbnty5cvz9q3b58ddthhuW2r/07OPPPMvGN897vfzXbaaafc8/KslQ29n4jIatWqlVuDWZZlb7zxRhYR2a9+9auNvncAANK5XQYAQBV06623xoQJE9Z6dOvWbaP7NmzYMN55552YPXt2uc/75JNPRvXq1ePCCy/M237JJZdElmXx1FNPRUTkblFw3nnn5Y274IIL1nvsc845Z61tderUyf35q6++in/+859xwAEHRETEjBkz1hp/9tln5/5cvXr12HfffSPLsjjrrLNy2xs2bBidO3eOv//97+udS8S/32tExMUXX5y3/ZJLLomIiCeeeGKD+6d4+OGH4+CDD45GjRrFP//5z9yjX79+sWrVqnjxxRdzY3/0ox9F//7944ILLohTTjklOnbsGNdff/0Wn9OaZs6cGbNnz46TTz45Pv/889z8li1bFoceemi8+OKLudt6rPbNv9eDDz44Pv/88ygpKYmItLWyPv369YuOHTvmnnfr1i2Kioo2+ncNAMDmcbsMAIAqaP/994999913re2r4+SGXH311XH00UfHrrvuGnvssUccccQRccopp2xSoP7www+jZcuW0aBBg7ztu+22W+711f+sVq1atG/fPm/cLrvsst5jf3NsRMS//vWvGDFiRDzwwAN59x+O+PctQ76pTZs2ec+Li4ujdu3auVsrrLn9m/d1/qbV7+Gbc27evHk0bNgw9163pNmzZ8ebb7653ltafPNncNddd0XHjh1j9uzZMWXKlLwoXxFW/w8Tp5122nrHLF68OBo1apR7/s2/k9WvffHFF1FUVJS0Vtbnm+dafb4vvvii3McCAGDTicwAADuYXr16xdy5c+Oxxx6LZ599Nu688864+eab47e//W3elcBb27oC6fHHHx9TpkyJIUOGxN577x3169ePsrKyOOKII9a6Yjbi31cvb8q2iFjriwrX55v3fa5IZWVlcdhhh8Wll166ztd33XXXvOeTJk3KfQnhW2+9FT179qzw+UVE3HTTTbH33nuvc0z9+vXznm/uz788tua5AAD4/0RmAIAdUOPGjeOMM86IM844I5YuXRq9evWK4cOH5yLz+sJq27Zt47nnnoslS5bkXc38t7/9Lff66n+WlZXFvHnzolOnTrlxc+bM2eQ5fvHFF/H888/HiBEj4qqrrsptT7nNR4rV72H27Nm5K7UjIj777LNYtGhR7r1uSR07doylS5dGv379Njr2k08+iQsuuCAOP/zwqFWrVvzsZz+L/v37581rSwfy1beiKCoq2qQ5boryrJWtGfwBANh07skMALCD+eZtIurXrx+77LJL7orYiIh69epFRMSiRYvyxn7nO9+JVatWxa9//eu87TfffHMUFBTEgAEDIiKif//+ERHxm9/8Jm/cr371q02e5+qrUr95Feott9yyycfYHN/5znfWeb5f/OIXERExcODALX7O448/Pl5++eV45pln1npt0aJF8fXXX+ee//CHP4yysrK466674vbbb48aNWrEWWedlffzWt/fY6ru3btHx44d4+c//3ksXbp0rdf/7//+r9zHLM9a2dLvBwCALcOVzAAAO5jdd989+vTpE927d4/GjRvHtGnT4g9/+EOcf/75uTHdu3ePiIgLL7ww+vfvH9WrV48TTzwxjjzyyOjbt29cccUV8cEHH8Ree+0Vzz77bDz22GNx0UUX5a507d69exx77LFxyy23xOeffx4HHHBATJ48Od5///2I2LQrUouKiqJXr15x4403xsqVK+Nb3/pWPPvsszFv3rwK+Kmsba+99orTTjstbr/99li0aFH07t07Xn311bjnnnvimGOOib59+27xcw4ZMiT+9Kc/xX/+53/G6aefHt27d49ly5bFW2+9FX/4wx/igw8+iJ133jnGjBkTTzzxRIwdOzZatWoVEf+Osj/4wQ9i9OjRuS/RW/33eMUVV8SJJ54YNWvWjCOPPDIXa9dl5cqVce211661vXHjxnHeeefFnXfeGQMGDIiuXbvGGWecEd/61rfio48+iokTJ0ZRUVH8+c9/Ltd7Ls9aSXk/AABUPJEZAGAHc+GFF8af/vSnePbZZ6O0tDTatm0b1157bQwZMiQ35nvf+15ccMEF8cADD8S9994bWZbFiSeeGNWqVYs//elPcdVVV8WDDz4YY8aMiXbt2sVNN90Ul1xySd55fve730Xz5s1j3LhxMX78+OjXr188+OCD0blz56hdu/YmzfX++++PCy64IG699dbIsiwOP/zweOqpp6Jly5Zb9GeyPnfeeWd06NAhxo4dG+PHj4/mzZvH0KFDY9iwYRVyvrp168bkyZPj+uuvj4cffjh+97vfRVFRUey6664xYsSIKC4ujn/84x/x05/+NI488si8L+AbNGhQ/PGPf4xLL700BgwYEO3bt4/99tsvrrnmmvjtb38bTz/9dO62FBuKsitWrIgrr7xyre0dO3aM8847L/r06RMvv/xyXHPNNfHrX/86li5dGs2bN48ePXrEj3/846T3valrJeX9AABQ8Qoy34IBAMBWMnPmzPj2t78d9957bwwaNGhbT4dKzFoBAKg63JMZAIAK8eWXX6617ZZbbolq1apFr169tsGMqKysFQCAqs3tMgAAqBA33nhjTJ8+Pfr27Rs1atSIp556Kp566qn40Y9+FK1bt97W06MSsVYAAKo2t8sAAKBCTJgwIUaMGBHvvvtuLF26NNq0aROnnHJKXHHFFVGjhmsd+P+sFQCAqk1kBgAAAAAgmXsyAwAAAACQTGQGAAAAACDZVr/BWVlZWXz88cfRoEGDKCgo2NqnBwAAAABgE2RZFkuWLImWLVtGtWrrv155q0fmjz/+2DdEAwAAAABUEQsWLIhWrVqt9/WtHpkbNGgQEf+eWFFR0dY+PQAAAAAAm6CkpCRat26da7rrs9Uj8+pbZBQVFYnMAAAAAACV3MZue+yL/wAAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAycoVmVetWhVXXnlltG/fPurUqRMdO3aMa665JrIsq6j5AQAAAABQidUoz+AbbrghRo8eHffcc0907do1pk2bFmeccUYUFxfHhRdeWFFzBAAAAACgkipXZJ4yZUocffTRMXDgwIiIaNeuXYwbNy5effXVCpkcAAAAAACVW7lul/Ef//Ef8fzzz8f7778fERFvvPFG/PWvf40BAwasd5/S0tIoKSnJewAAAAAAsH0o15XMl19+eZSUlESXLl2ievXqsWrVqrjuuuti0KBB691n5MiRMWLEiM2eKAAAAAAAlU+5rmR+6KGH4r777ov7778/ZsyYEffcc0/8/Oc/j3vuuWe9+wwdOjQWL16ceyxYsGCzJw0AAAAAQOVQkGVZtqmDW7duHZdffnkMHjw4t+3aa6+Ne++9N/72t79t0jFKSkqiuLg4Fi9eHEVFReWfMQAAAAAAFW5TW265rmRevnx5VKuWv0v16tWjrKwsbZYAAAAAAFRp5bon85FHHhnXXXddtGnTJrp27Rqvv/56/OIXv4gzzzyzouYHAAAAAEAlVq7bZSxZsiSuvPLKGD9+fCxcuDBatmwZJ510Ulx11VVRq1atTTqG22UAAAAAAFR+m9pyyxWZtwSRGQAAAACg8quQezIDAAAAAMCaRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgWbkic7t27aKgoGCtx+DBgytqfgAAAAAAVGI1yjP4tddei1WrVuWev/3223HYYYfFcccdt8UnBgAAAABA5VeuyNykSZO856NGjYqOHTtG7969t+ikAAAAAACoGsoVmde0YsWKuPfee+Piiy+OgoKC9Y4rLS2N0tLS3POSkpLUUwIAAAAAUMkkf/Hfo48+GosWLYrTTz99g+NGjhwZxcXFuUfr1q1TTwkAAAAAQCVTkGVZlrJj//79o1atWvHnP/95g+PWdSVz69atY/HixVFUVJRyagAAAAAAKlhJSUkUFxdvtOUm3S7jww8/jOeeey4eeeSRjY4tLCyMwsLClNMAAAAAAFDJJd0uY8yYMdG0adMYOHDglp4PAAAAAABVSLkjc1lZWYwZMyZOO+20qFEj+XsDAQAAAADYDpQ7Mj/33HMxf/78OPPMMytiPgAAAAAAVCHlvhT58MMPj8TvCgQAAAAAYDuTdE9mAAAAAACIEJkBAAAAANgMIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQrNyR+aOPPoof/OAHsdNOO0WdOnVizz33jGnTplXE3AAAAAAAqORqlGfwF198EQceeGD07ds3nnrqqWjSpEnMnj07GjVqVFHzAwAAAACgEitXZL7hhhuidevWMWbMmNy29u3bb/FJAQAAAABQNZTrdhl/+tOfYt99943jjjsumjZtGt/+9rfjjjvu2OA+paWlUVJSkvcAAAAAAGD7UK7I/Pe//z1Gjx4dnTp1imeeeSbOPffcuPDCC+Oee+5Z7z4jR46M4uLi3KN169abPWkAAAAAACqHgizLsk0dXKtWrdh3331jypQpuW0XXnhhvPbaa/Hyyy+vc5/S0tIoLS3NPS8pKYnWrVvH4sWLo6ioaDOmDgAAAABARSkpKYni4uKNttxyXcncokWL2H333fO27bbbbjF//vz17lNYWBhFRUV5DwAAAAAAtg/liswHHnhgzJo1K2/b+++/H23btt2ikwIAAAAAoGooV2T+6U9/Gq+88kpcf/31MWfOnLj//vvj9ttvj8GDB1fU/AAAAAAAqMTKFZn322+/GD9+fIwbNy722GOPuOaaa+KWW26JQYMGVdT8AAAAAACoxMr1xX9bwqbeLBoAAAAAgG2nQr74DwAAAAAA1iQyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMnKFZmHDx8eBQUFeY8uXbpU1NwAAAAAAKjkapR3h65du8Zzzz33/w9Qo9yHAAAAAABgO1HuQlyjRo1o3rx5RcwFAAAAAIAqptz3ZJ49e3a0bNkyOnToEIMGDYr58+dvcHxpaWmUlJTkPQAAAAAA2D6UKzL36NEjxo4dG08//XSMHj065s2bFwcffHAsWbJkvfuMHDkyiouLc4/WrVtv9qSputpd/kSVPDYAAAAAsG7liswDBgyI4447Lrp16xb9+/ePJ598MhYtWhQPPfTQevcZOnRoLF68OPdYsGDBZk8aAAAAAIDKYbO+ta9hw4ax6667xpw5c9Y7prCwMAoLCzfnNAAAAAAAVFLlvifzmpYuXRpz586NFi1abKn5AAAAAABQhZQrMv/sZz+LyZMnxwcffBBTpkyJ7373u1G9evU46aSTKmp+AAAAAABUYuW6XcY//vGPOOmkk+Lzzz+PJk2axEEHHRSvvPJKNGnSpKLmBwAAAABAJVauyPzAAw9U1DwAAAAAAKiCNuuezAAAAAAA7NhEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACDZZkXmUaNGRUFBQVx00UVbaDoAAAAAAFQlyZH5tddei9tuuy26deu2JecDAAAAAEAVkhSZly5dGoMGDYo77rgjGjVqtKXnBAAAAABAFZEUmQcPHhwDBw6Mfv36bXRsaWlplJSU5D0AAAAAANg+lDsyP/DAAzFjxowYOXLkJo0fOXJkFBcX5x6tW7cu9ySpvNpd/sQWGbOlbOhcW3MeAAAAALCjKFdkXrBgQfzkJz+J++67L2rXrr1J+wwdOjQWL16ceyxYsCBpogAAAAAAVD41yjN4+vTpsXDhwthnn31y21atWhUvvvhi/PrXv47S0tKoXr163j6FhYVRWFi4ZWYLAAAAAEClUq7IfOihh8Zbb72Vt+2MM86ILl26xGWXXbZWYAYAAAAAYPtWrsjcoEGD2GOPPfK21atXL3baaae1tgMAAAAAsP0r9xf/AQAAAADAauW6knldJk2atAWmAQAAAABAVeRKZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZOWKzKNHj45u3bpFUVFRFBUVRc+ePeOpp56qqLkBAAAAAFDJlSsyt2rVKkaNGhXTp0+PadOmxSGHHBJHH310vPPOOxU1PwAAAAAAKrEa5Rl85JFH5j2/7rrrYvTo0fHKK69E165dt+jEAAAAAACo/MoVmde0atWqePjhh2PZsmXRs2fP9Y4rLS2N0tLS3POSkpLUUwIAAAAAUMmU+4v/3nrrrahfv34UFhbGOeecE+PHj4/dd999veNHjhwZxcXFuUfr1q03a8JUjHaXP7FJz7+5fWP7rmv8mq9t7Lwb23dTx6/vGOU9LwAAAACQr9yRuXPnzjFz5syYOnVqnHvuuXHaaafFu+++u97xQ4cOjcWLF+ceCxYs2KwJAwAAAABQeZT7dhm1atWKXXbZJSIiunfvHq+99lr87//+b9x2223rHF9YWBiFhYWbN0sAAAAAACqlcl/J/E1lZWV591wGAAAAAGDHUa4rmYcOHRoDBgyINm3axJIlS+L++++PSZMmxTPPPFNR8wMAAAAAoBIrV2ReuHBhnHrqqfHJJ59EcXFxdOvWLZ555pk47LDDKmp+AAAAAABUYuWKzHfddVdFzQMAAAAAgCpos+/JDAAAAADAjktkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKVKzKPHDky9ttvv2jQoEE0bdo0jjnmmJg1a1ZFzQ0AAAAAgEquXJF58uTJMXjw4HjllVdiwoQJsXLlyjj88MNj2bJlFTU/AAAAAAAqsRrlGfz000/nPR87dmw0bdo0pk+fHr169dqiEwMAAAAAoPIrV2T+psWLF0dEROPGjdc7prS0NEpLS3PPS0pKNueUAAAAAABUIslf/FdWVhYXXXRRHHjggbHHHnusd9zIkSOjuLg492jdunXqKdlC2l3+xGaNXd/+39y+oecbG7t626acf1PmsynveX3n21Sbs+/WOB4AAAAAVITkyDx48OB4++2344EHHtjguKFDh8bixYtzjwULFqSeEgAAAACASibpdhnnn39+PP744/Hiiy9Gq1atNji2sLAwCgsLkyYHAAAAAEDlVq7InGVZXHDBBTF+/PiYNGlStG/fvqLmBQAAAABAFVCuyDx48OC4//7747HHHosGDRrEp59+GhERxcXFUadOnQqZIAAAAAAAlVe57sk8evToWLx4cfTp0ydatGiRezz44IMVNT8AAAAAACqxct8uAwAAAAAAVivXlcwAAAAAALAmkRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIVu7I/OKLL8aRRx4ZLVu2jIKCgnj00UcrYFoAAAAAAFQF5Y7My5Yti7322ituvfXWipgPAAAAAABVSI3y7jBgwIAYMGBARcwFAAAAAIAqptyRubxKS0ujtLQ097ykpKSiTwkAAAAAwFZS4V/8N3LkyCguLs49WrduXdGn3OG0u/yJpNdXb1/zn2uO/ebrmzOXDR1jY/NYc9ymHn9D72lDz7+5bVP3X9fPbVOk7rc5NuU86/t5bm3b6rwAAAAAbLoKj8xDhw6NxYsX5x4LFiyo6FMCAAAAALCVVPjtMgoLC6OwsLCiTwMAAAAAwDZQ4VcyAwAAAACw/Sr3lcxLly6NOXPm5J7PmzcvZs6cGY0bN442bdps0ckBAAAAAFC5lTsyT5s2Lfr27Zt7fvHFF0dExGmnnRZjx47dYhMDAAAAAKDyK3dk7tOnT2RZVhFzAQAAAACginFPZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDAAAAABAMpEZAAAAAIBkIjMAAAAAAMlEZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyURmAAAAAACSicwAAAAAACQTmQEAAAAASCYyAwAAAACQTGQGAAAAACCZyAwAAAAAQDKRGQAAAACAZEmR+dZbb4127dpF7dq1o0ePHvHqq69u6XkBAAAAAFAFlDsyP/jgg3HxxRfHsGHDYsaMGbHXXntF//79Y+HChRUxPwAAAAAAKrFyR+Zf/OIX8cMf/jDOOOOM2H333eO3v/1t1K1bN+6+++6KmB8AAAAAAJVYjfIMXrFiRUyfPj2GDh2a21atWrXo169fvPzyy+vcp7S0NEpLS3PPFy9eHBERJSUlKfNlHcpKl2/w5/nN11c/X/Ofa1rXtm9uX328NZ+vuc/6jrGx19Y3fs1zbcr4jY0t75h1vd91zWnNn+umWHPs+v6etrRNOe7WmsvGbKvzAgAAAPD/G1iWZRscV5BtbMQaPv744/jWt74VU6ZMiZ49e+a2X3rppTF58uSYOnXqWvsMHz48RowYsamnAAAAAACgElmwYEG0atVqva+X60rmFEOHDo2LL74497ysrCz+9a9/xU477RQFBQUVffpKpaSkJFq3bh0LFiyIoqKibT0d2OKscXYE1jnbO2ucHYF1zvbOGmdHYJ2zvassazzLsliyZEm0bNlyg+PKFZl33nnnqF69enz22Wd52z/77LNo3rz5OvcpLCyMwsLCvG0NGzYsz2m3O0VFRT4A2a5Z4+wIrHO2d9Y4OwLrnO2dNc6OwDpne1cZ1nhxcfFGx5Tri/9q1aoV3bt3j+effz63raysLJ5//vm822cAAAAAALBjKPftMi6++OI47bTTYt999439998/brnllli2bFmcccYZFTE/AAAAAAAqsXJH5hNOOCH+7//+L6666qr49NNPY++9946nn346mjVrVhHz264UFhbGsGHD1rp9CGwvrHF2BNY52ztrnB2Bdc72zhpnR2Cds72ramu8IMuybFtPAgAAAACAqqlc92QGAAAAAIA1icwAAAAAACQTmQEAAAAASCYyAwAAAACQTGTeim699dZo165d1K5dO3r06BGvvvrqtp4SrGXkyJGx3377RYMGDaJp06ZxzDHHxKxZs/LG9OnTJwoKCvIe55xzTt6Y+fPnx8CBA6Nu3brRtGnTGDJkSHz99dd5YyZNmhT77LNPFBYWxi677BJjx46t6LcHERExfPjwtdZwly5dcq9/9dVXMXjw4Nhpp52ifv36ceyxx8Znn32WdwxrnMqsXbt2a63xgoKCGDx4cET4HKdqevHFF+PII4+Mli1bRkFBQTz66KN5r2dZFldddVW0aNEi6tSpE/369YvZs2fnjfnXv/4VgwYNiqKiomjYsGGcddZZsXTp0rwxb775Zhx88MFRu3btaN26ddx4441rzeXhhx+OLl26RO3atWPPPfeMJ598cou/X3Y8G1rjK1eujMsuuyz23HPPqFevXrRs2TJOPfXU+Pjjj/OOsa7P/1GjRuWNscbZljb2WX766aevtYaPOOKIvDE+y6nMNrbG1/U7ekFBQdx00025MVX1s1xk3koefPDBuPjii2PYsGExY8aM2GuvvaJ///6xcOHCbT01yDN58uQYPHhwvPLKKzFhwoRYuXJlHH744bFs2bK8cT/84Q/jk08+yT3W/EBbtWpVDBw4MFasWBFTpkyJe+65J8aOHRtXXXVVbsy8efNi4MCB0bdv35g5c2ZcdNFFcfbZZ8czzzyz1d4rO7auXbvmreG//vWvudd++tOfxp///Od4+OGHY/LkyfHxxx/H9773vdzr1jiV3WuvvZa3vidMmBAREccdd1xujM9xqpply5bFXnvtFbfeeus6X7/xxhvjl7/8Zfz2t7+NqVOnRr169aJ///7x1Vdf5cYMGjQo3nnnnZgwYUI8/vjj8eKLL8aPfvSj3OslJSVx+OGHR9u2bWP69Olx0003xfDhw+P222/PjZkyZUqcdNJJcdZZZ8Xrr78exxxzTBxzzDHx9ttvV9ybZ4ewoTW+fPnymDFjRlx55ZUxY8aMeOSRR2LWrFlx1FFHrTX26quvzvt8v+CCC3KvWeNsaxv7LI+IOOKII/LW8Lhx4/Je91lOZbaxNb7m2v7kk0/i7rvvjoKCgjj22GPzxlXJz/KMrWL//ffPBg8enHu+atWqrGXLltnIkSO34axg4xYuXJhFRDZ58uTctt69e2c/+clP1rvPk08+mVWrVi379NNPc9tGjx6dFRUVZaWlpVmWZdmll16ade3aNW+/E044Ievfv/+WfQOwDsOGDcv22muvdb62aNGirGbNmtnDDz+c2/bee+9lEZG9/PLLWZZZ41Q9P/nJT7KOHTtmZWVlWZb5HKfqi4hs/PjxuedlZWVZ8+bNs5tuuim3bdGiRVlhYWE2bty4LMuy7N13380iInvttddyY5566qmsoKAg++ijj7Isy7Lf/OY3WaNGjXLrPMuy7LLLLss6d+6ce3788cdnAwcOzJtPjx49sh//+Mdb9D2yY/vmGl+XV199NYuI7MMPP8xta9u2bXbzzTevdx9rnMpkXev8tNNOy44++uj17uOznKpkUz7Ljz766OyQQw7J21ZVP8tdybwVrFixIqZPnx79+vXLbatWrVr069cvXn755W04M9i4xYsXR0RE48aN87bfd999sfPOO8cee+wRQ4cOjeXLl+dee/nll2PPPfeMZs2a5bb1798/SkpK4p133smNWfPfidVj/DvB1jJ79uxo2bJldOjQIQYNGhTz58+PiIjp06fHypUr89Znly5dok2bNrn1aY1TlaxYsSLuvffeOPPMM6OgoCC33ec425N58+bFp59+mrcmi4uLo0ePHnmf3Q0bNox99903N6Zfv35RrVq1mDp1am5Mr169olatWrkx/fv3j1mzZsUXX3yRG2PtUxksXrw4CgoKomHDhnnbR40aFTvttFN8+9vfjptuuinvVkfWOFXBpEmTomnTptG5c+c499xz4/PPP8+95rOc7clnn30WTzzxRJx11llrvVYVP8trVNiRyfnnP/8Zq1atyvsvahERzZo1i7/97W/baFawcWVlZXHRRRfFgQceGHvssUdu+8knnxxt27aNli1bxptvvhmXXXZZzJo1Kx555JGIiPj000/Xud5Xv7ahMSUlJfHll19GnTp1KvKtsYPr0aNHjB07Njp37hyffPJJjBgxIg4++OB4++2349NPP41atWqt9V/YmjVrttH1u/q1DY2xxtnaHn300Vi0aFGcfvrpuW0+x9nerF6X61qTa67Zpk2b5r1eo0aNaNy4cd6Y9u3br3WM1a81atRovWt/9TFga/jqq6/isssui5NOOimKiopy2y+88MLYZ599onHjxjFlypQYOnRofPLJJ/GLX/wiIqxxKr8jjjgivve970X79u1j7ty58V//9V8xYMCAePnll6N69eo+y9mu3HPPPdGgQYO8WzNGVN3PcpEZWK/BgwfH22+/nXev2ojIu9/VnnvuGS1atIhDDz005s6dGx07dtza04RyGzBgQO7P3bp1ix49ekTbtm3joYceEsbY7tx1110xYMCAaNmyZW6bz3GAqmvlypVx/PHHR5ZlMXr06LzXLr744tyfu3XrFrVq1Yof//jHMXLkyCgsLNzaU4VyO/HEE3N/3nPPPaNbt27RsWPHmDRpUhx66KHbcGaw5d19990xaNCgqF27dt72qvpZ7nYZW8HOO+8c1atXj88++yxv+2effRbNmzffRrOCDTv//PPj8ccfj4kTJ0arVq02OLZHjx4RETFnzpyIiGjevPk61/vq1zY0pqioSORjq2vYsGHsuuuuMWfOnGjevHmsWLEiFi1alDdmzc9sa5yq4sMPP4znnnsuzj777A2O8zlOVbd6XW7o9+3mzZuv9aXbX3/9dfzrX//aIp/vfq9na1gdmD/88MOYMGFC3lXM69KjR4/4+uuv44MPPogIa5yqp0OHDrHzzjvn/Y7is5ztwV/+8peYNWvWRn9Pj6g6n+Ui81ZQq1at6N69ezz//PO5bWVlZfH8889Hz549t+HMYG1ZlsX5558f48ePjxdeeGGt/wvGusycOTMiIlq0aBERET179oy33nor7z/8V/8SvPvuu+fGrPnvxOox/p1gW1i6dGnMnTs3WrRoEd27d4+aNWvmrc9Zs2bF/Pnzc+vTGqeqGDNmTDRt2jQGDhy4wXE+x6nq2rdvH82bN89bkyUlJTF16tS8z+5FixbF9OnTc2NeeOGFKCsry/0PLT179owXX3wxVq5cmRszYcKE6Ny5czRq1Cg3xtpnW1gdmGfPnh3PPfdc7LTTThvdZ+bMmVGtWrXc7QWscaqaf/zjH/H555/n/Y7is5ztwV133RXdu3ePvfbaa6Njq8xneYV9pSB5HnjggaywsDAbO3Zs9u6772Y/+tGPsoYNG+Z9aztUBueee25WXFycTZo0Kfvkk09yj+XLl2dZlmVz5szJrr766mzatGnZvHnzssceeyzr0KFD1qtXr9wxvv7662yPPfbIDj/88GzmzJnZ008/nTVp0iQbOnRobszf//73rG7dutmQIUOy9957L7v11luz6tWrZ08//fRWf8/seC655JJs0qRJ2bx587KXXnop69evX7bzzjtnCxcuzLIsy84555ysTZs22QsvvJBNmzYt69mzZ9azZ8/c/tY4VcGqVauyNm3aZJdddlnedp/jVFVLlizJXn/99ez111/PIiL7xS9+kb3++uvZhx9+mGVZlo0aNSpr2LBh9thjj2VvvvlmdvTRR2ft27fPvvzyy9wxjjjiiOzb3/52NnXq1Oyvf/1r1qlTp+ykk07Kvb5o0aKsWbNm2SmnnJK9/fbb2QMPPJDVrVs3u+2223JjXnrppaxGjRrZz3/+8+y9997Lhg0bltWsWTN76623tt4Pg+3Shtb4ihUrsqOOOipr1apVNnPmzLzf00tLS7Msy7IpU6ZkN998czZz5sxs7ty52b333ps1adIkO/XUU3PnsMbZ1ja0zpcsWZL97Gc/y15++eVs3rx52XPPPZfts88+WadOnbKvvvoqdwyf5VRmG/t9JcuybPHixVndunWz0aNHr7V/Vf4sF5m3ol/96ldZmzZtslq1amX7779/9sorr2zrKcFaImKdjzFjxmRZlmXz58/PevXqlTVu3DgrLCzMdtlll2zIkCHZ4sWL847zwQcfZAMGDMjq1KmT7bzzztkll1ySrVy5Mm/MxIkTs7333jurVatW1qFDh9w5oKKdcMIJWYsWLbJatWpl3/rWt7ITTjghmzNnTu71L7/8MjvvvPOyRo0aZXXr1s2++93vZp988kneMaxxKrtnnnkmi4hs1qxZedt9jlNVTZw4cZ2/o5x22mlZlmVZWVlZduWVV2bNmjXLCgsLs0MPPXSt9f/5559nJ510Ula/fv2sqKgoO+OMM7IlS5bkjXnjjTeygw46KCssLMy+9a1vZaNGjVprLg899FC26667ZrVq1cq6du2aPfHEExX2vtlxbGiNz5s3b72/p0+cODHLsiybPn161qNHj6y4uDirXbt2tttuu2XXX399XpzLMmucbWtD63z58uXZ4YcfnjVp0iSrWbNm1rZt2+yHP/zhWhfn+SynMtvY7ytZlmW33XZbVqdOnWzRokVr7V+VP8sLsizLKu46aQAAAAAAtmfuyQwAAAAAQDKRGQAAAACAZCIzAAAAAADJRGYAAAAAAJKJzAAAAAAAJBOZAQAAAABIJjIDAAAAAJBMZAYAAAAAIJnIDADAdqVPnz5x0UUXbetpVHkffPBBFBQUxMyZM7f1VAAAqOREZgAAtpjf/va30aBBg/j6669z25YuXRo1a9aMPn365I2dNGlSFBQUxNy5c7fyLCuPefPmxcknnxwtW7aM2rVrR6tWreLoo4+Ov/3tb9t6agAAsMlEZgAAtpi+ffvG0qVLY9q0abltf/nLX6J58+YxderU+Oqrr3LbJ06cGG3atImOHTuW+zxZluWF7Mpu5cqV69x22GGHxeLFi+ORRx6JWbNmxYMPPhh77rlnLFq0aOtPEgAAEonMAABsMZ07d44WLVrEpEmTctsmTZoURx99dLRv3z5eeeWVvO19+/aNiIjS0tK48MILo2nTplG7du046KCD4rXXXssbW1BQEE899VR07949CgsL469//WssW7YsTj311Khfv360aNEi/ud//metOf3mN7+JTp06Re3ataNZs2bx/e9/f73zHzt2bDRs2DAeffTR3D79+/ePBQsW5I177LHHYp999onatWtHhw4dYsSIEXnRu6CgIEaPHh1HHXVU1KtXL6677rq1zvXOO+/E3Llz4ze/+U0ccMAB0bZt2zjwwAPj2muvjQMOOCA37rLLLotdd9016tatGx06dIgrr7wyL1oPHz489t5777j77rujTZs2Ub9+/TjvvPNi1apVceONN0bz5s2jadOma81h9RwHDBgQderUiQ4dOsQf/vCH9f5sIiLefvvtGDBgQNSvXz+aNWsWp5xySvzzn//c4D4AAGz/RGYAALaovn37xsSJE3PPJ06cGH369InevXvntn/55ZcxderUXGS+9NJL449//GPcc889MWPGjNhll12if//+8a9//Svv2JdffnmMGjUq3nvvvejWrVsMGTIkJk+eHI899lg8++yzMWnSpJgxY0Zu/LRp0+LCCy+Mq6++OmbNmhVPP/109OrVa4PzX758eVx33XXxu9/9Ll566aVYtGhRnHjiibnX//KXv8Spp54aP/nJT+Ldd9+N2267LcaOHbtWxB0+fHh897vfjbfeeivOPPPMtc7TpEmTqFatWvzhD3+IVatWrXc+DRo0iLFjx8a7774b//u//xt33HFH3HzzzXlj5s6dG0899VQ8/fTTMW7cuLjrrrti4MCB8Y9//CMmT54cN9xwQ/z3f/93TJ06NW+/K6+8Mo499th44403YtCgQXHiiSfGe++9t855LFq0KA455JD49re/HdOmTYunn346Pvvsszj++OM3+PMEAGAHkAEAwBZ0xx13ZPXq1ctWrlyZlZSUZDVq1MgWLlyY3X///VmvXr2yLMuy559/PouI7MMPP8yWLl2a1axZM7vvvvtyx1ixYkXWsmXL7MYbb8yyLMsmTpyYRUT26KOP5sYsWbIkq1WrVvbQQw/ltn3++edZnTp1sp/85CdZlmXZH//4x6yoqCgrKSnZpLmPGTMmi4jslVdeyW177733sojIpk6dmmVZlh166KHZ9ddfn7ff73//+6xFixa55xGRXXTRRRs9369//eusbt26WYMGDbK+fftmV199dTZ37twN7nPTTTdl3bt3zz0fNmxYVrdu3bz32L9//6xdu3bZqlWrcts6d+6cjRw5Mm+O55xzTt6xe/TokZ177rlZlmXZvHnzsojIXn/99SzLsuyaa67JDj/88LzxCxYsyCIimzVr1kbfKwAA2y9XMgMAsEX16dMnli1bFq+99lr85S9/iV133TWaNGkSvXv3zt2XedKkSdGhQ4do06ZNzJ07N1auXBkHHnhg7hg1a9aM/ffff62ravfdd9/cn+fOnRsrVqyIHj165LY1btw4OnfunHt+2GGHRdu2baNDhw5xyimnxH333RfLly/f4Pxr1KgR++23X+55ly5domHDhrm5vPHGG3H11VdH/fr1c48f/vCH8cknn+Qde825rs/gwYPj008/jfvuuy969uwZDz/8cHTt2jUmTJiQG/Pggw/GgQceGM2bN4/69evHf//3f8f8+fPzjtOuXbto0KBB7nmzZs1i9913j2rVquVtW7hwYd5+PXv2XOv5+q5kfuONN2LixIl577tLly4RETv0lzcCAOB2GQAAbGG77LJLtGrVKiZOnBgTJ06M3r17R0REy5Yto3Xr1jFlypSYOHFiHHLIIeU+dr169co1vkGDBjFjxowYN25ctGjRIq666qrYa6+9NuuL9ZYuXRojRoyImTNn5h5vvfVWzJ49O2rXrl3uuTZo0CCOPPLIuO666+KNN96Igw8+OK699tqIiHj55Zdj0KBB8Z3vfCcef/zxeP311+OKK66IFStW5B2jZs2aec8LCgrWua2srCzlLUfEv9/3kUcemfe+Z86cGbNnz97oLUgAANi+icwAAGxxffv2jUmTJsWkSZOiT58+ue29evWKp556Kl599dXc/Zg7duwYtWrVipdeeik3buXKlfHaa6/F7rvvvt5zdOzYMWrWrJl3n+Evvvgi3n///bxxNWrUiH79+sWNN94Yb775ZnzwwQfxwgsvrPe4X3/9dUybNi33fNasWbFo0aLYbbfdIiJin332iVmzZsUuu+yy1mPNK4dTFBQURJcuXWLZsmURETFlypRo27ZtXHHFFbHvvvtGp06d4sMPP9ysc6xpzS9iXP189fv8pn322SfeeeedaNeu3Vrvu7zxHwCA7UuNbT0BAAC2P3379o3BgwfHypUrc1cyR0T07t07zj///FixYkUuMterVy/OPffcGDJkSDRu3DjatGkTN954YyxfvjzOOuus9Z6jfv36cdZZZ8WQIUNip512iqZNm8YVV1yRF3off/zx+Pvf/x69evWKRo0axZNPPhllZWV5t9T4ppo1a8YFF1wQv/zlL6NGjRpx/vnnxwEHHBD7779/RERcddVV8Z//+Z/Rpk2b+P73vx/VqlWLN954I95+++3cFcibYubMmTFs2LA45ZRTYvfdd49atWrF5MmT4+67747LLrssIiI6deoU8+fPjwceeCD222+/eOKJJ2L8+PGbfI6Nefjhh2PfffeNgw46KO6777549dVX46677lrn2MGDB8cdd9wRJ510Ulx66aXRuHHjmDNnTjzwwANx5513RvXq1bfYvAAAqFpEZgAAtri+ffvGl19+GV26dIlmzZrltvfu3TuWLFkSnTt3jhYtWuS2jxo1KsrKyuKUU06JJUuWxL777hvPPPNMNGrUaIPnuemmm3K3cWjQoEFccsklsXjx4tzrDRs2jEceeSSGDx8eX331VXTq1CnGjRsXXbt2Xe8x69atG5dddlmcfPLJ8dFHH8XBBx+cF1779+8fjz/+eFx99dVxww03RM2aNaNLly5x9tlnl+tn1KpVq2jXrl2MGDEiPvjggygoKMg9/+lPfxoREUcddVT89Kc/jfPPPz9KS0tj4MCBceWVV8bw4cPLda71GTFiRDzwwANx3nnnRYsWLWLcuHHrvXq8ZcuW8dJLL8Vll10Whx9+eJSWlkbbtm3jiCOO2OwruAEAqNoKsizLtvUkAACgMhg7dmxcdNFFm3XP5qqioKAgxo8fH8ccc8y2ngoAAFWcSw4AAAAAAEgmMgMAAAAAkMztMgAAAAAASOZKZgAAAAAAkonMAAAAAAAkE5kBAAAAAEgmMgMAAAAAkExkBgAAAAAgmcgMAAAAAEAykRkAAAAAgGQiMwAAAAAAyf4fPZT7RYrRmEQAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data['input_len'] = data['text'].str.split().str.len()\n",
    "\n",
    "max_len = max(data['input_len'])\n",
    "print(f\"The maximum length of this dataset is approximately {max_len} words.\")\n",
    "\n",
    "n = max_len\n",
    "data.hist('input_len', grid=False, figsize=(18,8), range=(0, n), bins=n)\n",
    "plt.suptitle('')\n",
    "plt.title('Histogram of Text Length')\n",
    "plt.ylabel('')\n",
    "plt.xlabel('Words per Sample')\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Remove Overly Length Input Texts & Remove Unnecessary Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1934 entries, 0 to 0\n",
      "Data columns (total 1 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   text    1934 non-null   object\n",
      "dtypes: object(1)\n",
      "memory usage: 30.2+ KB\n"
     ]
    }
   ],
   "source": [
    "data = data[data[\"input_len\"] < 2500]\n",
    "\n",
    "data = data.drop(columns=[\"input_len\"])\n",
    "\n",
    "data.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert DataFrame to Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = Dataset.from_pandas(data)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split Dataset into Training & Evaluation Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (1547, 2)\n",
      "Eval data shape: (387, 2)\n"
     ]
    }
   ],
   "source": [
    "train_eval_dataset = dataset.train_test_split(test_size=0.20)\n",
    "\n",
    "ds = DatasetDict({\n",
    "    'train' : train_eval_dataset[\"train\"],\n",
    "    'eval' : train_eval_dataset[\"test\"]\n",
    "})\n",
    "\n",
    "print('Training data shape:', ds['train'].shape)\n",
    "print('Eval data shape:', ds['eval'].shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic Values/Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "BLOCK_SIZE = 128\n",
    "MODEL_CKPT = \"distilgpt2\"\n",
    "\n",
    "MODEL_NAME = MODEL_CKPT.split(\"/\")[-1] + \"-2k_clean_medical_articles_causal_language_model\"\n",
    "set_seed = 42\n",
    "\n",
    "BATCH_SIZE = 1000 \n",
    "NUM_OF_EPOCHS = 3\n",
    "\n",
    "WEIGHT_DECAY = 0.01\n",
    "STRATEGY = \"epoch\"\n",
    "\n",
    "REPORTS_TO = \"tensorboard\"\n",
    "LEARNING_RATE = 2e-5"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tokenize Entire Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT, use_fast=True)\n",
    "\n",
    "def tokenizer_function(samples):\n",
    "    return tokenizer(samples[\"text\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Tokenizer Function to Entire Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1bf206b417d24367929f5e934f51e14b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1211 > 1024). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d55b2cb23fab42099c46d83fc9d822d2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['input_ids', 'attention_mask'],\n",
       "    num_rows: 1547\n",
       "})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_ds = ds.map(tokenizer_function, \n",
    "                      batched=True,\n",
    "                      remove_columns=ds[\"train\"].column_names,)\n",
    "\n",
    "tokenized_ds[\"train\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocess Tokenized Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def group_texts(samples):\n",
    "    concatenated_examples = {k: sum(samples[k], []) for k in samples.keys()}\n",
    "    total_length = len(concatenated_examples[list(samples.keys())[0]])\n",
    "    total_length = (total_length // BLOCK_SIZE) * BLOCK_SIZE\n",
    "    result = {\n",
    "        k: [t[i : i + BLOCK_SIZE] for i in range(0, total_length, BLOCK_SIZE)]\n",
    "        for k, t in concatenated_examples.items()\n",
    "    }\n",
    "    result[\"labels\"] = result[\"input_ids\"].copy()\n",
    "    return result"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Preprocessing Function to Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   "
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b6923ae47e1349d2a34031b4d572675e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " "
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "14bdf5c8e15d48648769d86da683a5d0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    "
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4328ff1915d04064ab146e6e54bb70e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df8ee6f33724437fb267a9d35dca7155",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "clm_ds = tokenized_ds.map(\n",
    "    group_texts,\n",
    "    batched=True,\n",
    "    num_proc=2,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instantiate Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = AutoModelForCausalLM.from_pretrained(MODEL_CKPT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Data Collator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.pad_token = tokenizer.eos_token\n",
    "data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define TrainingArguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "args = TrainingArguments(\n",
    "    output_dir=MODEL_NAME,\n",
    "    evaluation_strategy=STRATEGY,\n",
    "    learning_rate=LEARNING_RATE,\n",
    "    weight_decay=WEIGHT_DECAY,\n",
    "    num_train_epochs=NUM_OF_EPOCHS,\n",
    "    report_to=REPORTS_TO,\n",
    "    hub_private_repo=True,\n",
    "    push_to_hub=True\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instantiate Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning https://huggingface.co/DunnBC22/distilgpt2-2k_clean_medical_articles_causal_language_model into local empty directory.\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=args,\n",
    "    train_dataset=clm_ds[\"train\"],\n",
    "    eval_dataset=clm_ds[\"eval\"],\n",
    "    data_collator=data_collator,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/briandunn/Documents/nlpnn/nlp_ml/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 15921\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 8\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 8\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 5973\n",
      "  Number of trainable parameters = 81912576\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f372fbae00734805acd684fd89139de9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-500\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-500/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-500/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 3.2551, 'learning_rate': 1.832579943077181e-05, 'epoch': 0.25}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-500/pytorch_model.bin\n",
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-1000\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-1000/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-1000/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 3.1543, 'learning_rate': 1.6651598861543614e-05, 'epoch': 0.5}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-1000/pytorch_model.bin\n",
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-1500\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-1500/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-1500/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 3.1211, 'learning_rate': 1.4977398292315422e-05, 'epoch': 0.75}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-1500/pytorch_model.bin\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 4029\n",
      "  Batch size = 8\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09f03611a8a04f6ea2e5f15000f14fa6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 2.9739990234375, 'eval_runtime': 2324.7462, 'eval_samples_per_second': 1.733, 'eval_steps_per_second': 0.217, 'epoch': 1.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-2000\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-2000/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-2000/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 3.0909, 'learning_rate': 1.3303197723087226e-05, 'epoch': 1.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-2000/pytorch_model.bin\n",
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-2500\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-2500/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-2500/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 3.0184, 'learning_rate': 1.1628997153859034e-05, 'epoch': 1.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-2500/pytorch_model.bin\n",
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-3000\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-3000/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-3000/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 3.0144, 'learning_rate': 9.95479658463084e-06, 'epoch': 1.51}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-3000/pytorch_model.bin\n",
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-3500\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-3500/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-3500/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 2.998, 'learning_rate': 8.280596015402646e-06, 'epoch': 1.76}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-3500/pytorch_model.bin\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 4029\n",
      "  Batch size = 8\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39b9376415d24970bba4a8a22312ab64",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 2.9366679191589355, 'eval_runtime': 2374.4091, 'eval_samples_per_second': 1.697, 'eval_steps_per_second': 0.212, 'epoch': 2.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-4000\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-4000/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-4000/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 2.981, 'learning_rate': 6.606395446174452e-06, 'epoch': 2.01}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-4000/pytorch_model.bin\n",
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-4500\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-4500/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-4500/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 2.9605, 'learning_rate': 4.932194876946258e-06, 'epoch': 2.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-4500/pytorch_model.bin\n",
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-5000\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-5000/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-5000/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 2.9476, 'learning_rate': 3.2579943077180647e-06, 'epoch': 2.51}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-5000/pytorch_model.bin\n",
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-5500\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-5500/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-5500/generation_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 2.9484, 'learning_rate': 1.5837937384898713e-06, 'epoch': 2.76}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/checkpoint-5500/pytorch_model.bin\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 4029\n",
      "  Batch size = 8\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f08f474447542fca3bd3ff0cb974dff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 2.9268078804016113, 'eval_runtime': 2449.0702, 'eval_samples_per_second': 1.645, 'eval_steps_per_second': 0.206, 'epoch': 3.0}\n",
      "{'train_runtime': 90899.7407, 'train_samples_per_second': 0.525, 'train_steps_per_second': 0.066, 'train_loss': 3.036437467137274, 'epoch': 3.0}\n"
     ]
    }
   ],
   "source": [
    "train_results = trainer.train()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 4029\n",
      "  Batch size = 8\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6819127d66c54bfca5f2132b2467987e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Perplexity: 18.67\n"
     ]
    }
   ],
   "source": [
    "evaluation_results = trainer.evaluate()\n",
    "print(f\"Perplexity: {math.exp(evaluation_results['eval_loss']):.2f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Push Model to HuggingFace Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to distilgpt2-2k_clean_medical_articles_causal_language_model\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/config.json\n",
      "Configuration saved in distilgpt2-2k_clean_medical_articles_causal_language_model/generation_config.json\n",
      "Model weights saved in distilgpt2-2k_clean_medical_articles_causal_language_model/pytorch_model.bin\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4fbfa8e19bb0409db1dbac957ac6b326",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "621f04d05ffd46bcacd404ebbbd98742",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8aad86e10f2f4f20991392d71646ba8a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "remote: Scanning LFS files of refs/heads/main for validity...        \n",
      "remote: LFS file scan complete.        \n",
      "To https://huggingface.co/DunnBC22/distilgpt2-2k_clean_medical_articles_causal_language_model\n",
      "   e3c7ec6..b0ccf87  main -> main\n",
      "\n",
      "Dropping the following result as it does not have all the necessary fields:\n",
      "{'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}\n",
      "To https://huggingface.co/DunnBC22/distilgpt2-2k_clean_medical_articles_causal_language_model\n",
      "   b0ccf87..10df069  main -> main\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'https://huggingface.co/DunnBC22/distilgpt2-2k_clean_medical_articles_causal_language_model/commit/b0ccf876668dc3bd09a53f5ce5af39ecec767a5f'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.push_to_hub()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes & Other Takeaways From This Project\n",
    "****\n",
    "- DistilGPT2 reached a perplexity of 21.1 after fine tuning, so the perlexity score here of 18.67 is pretty good.\n",
    "****"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Citations\n",
    "\n",
    "- Model Checkpoint\n",
    "    - @inproceedings{sanh2019distilbert,\n",
    "  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},\n",
    "  author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},\n",
    "  booktitle={NeurIPS EMC^2 Workshop},\n",
    "  year={2019}\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nlp_ml",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "0b7a754c5aaaa44a21e2f2148f50b931a5abdffe7ef71537f5954179554f39df"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}