Building a Handwritten Digits Classifier with Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a Handwritten Digits Classifier with Neural Networks\n",
    "by Nicholas Archambault\n",
    "\n",
    "Deep neural networks have been used to reach state-of-the-art performance on image classification tasks in the last decade. For some image classification tasks, deep neural networks actually perform as well as or slightly better than the human benchmark.\n",
    "\n",
    "This project will explore the effectiveness of deep, feedforward neural networks at classifying images of digits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing the Data\n",
    "\n",
    "We'll first import the appropriate packages as well as a pre-loaded Python dataset of handwritten digits from the University of California - Irvine and transform the resulting 2D array of data values into a Pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'target', 'target_names', 'images', 'DESCR'])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "digits_data = load_digits()\n",
    "digits_data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert key aspects to series and dataframe objects\n",
    "labels = pd.Series(digits_data[\"target\"])\n",
    "data = pd.DataFrame(digits_data[\"data\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After this transformation, each image is represented as a row of pixel values. We'll need to reshape the image to its original 28x28 pixel format and plot it on a coordinate grid in order to visualize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1a1f8d9748>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPgAAAD8CAYAAABaQGkdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAACtlJREFUeJzt3V9onfUdx/HPZ1HZ/FOsazekqYsBKchgtoaCFITVZdQpuospLShMBr1SlA2s7m53eiPuYghSdYKd0lQFEacTVJywOZO226ypo60dzapryir+GaxUv7vIKXRdtjzp+T1/ztf3C4L5c8jve4jvPs85OXl+jggByOlLbQ8AoD4EDiRG4EBiBA4kRuBAYgQOJEbgQGIEDiRG4EBiZ9XxTZctWxYjIyN1fOtWHTt2rNH1ZmZmGltryZIlja01PDzc2FpDQ0ONrdWkgwcP6ujRo17odrUEPjIyosnJyTq+dasmJiYaXW/Lli2NrTU+Pt7YWvfdd19jay1durSxtZo0NjZW6XacogOJETiQGIEDiRE4kBiBA4kROJAYgQOJETiQWKXAbW+w/a7tfbbvqXsoAGUsGLjtIUm/kHStpMslbbJ9ed2DAehflSP4Wkn7IuJARByX9JSkG+sdC0AJVQJfIenQKR/P9D4HoOOqBD7fX6z818XUbW+2PWl7cnZ2tv/JAPStSuAzklae8vGwpMOn3ygiHo6IsYgYW758ean5APShSuBvSbrM9qW2z5G0UdJz9Y4FoIQF/x48Ik7Yvl3SS5KGJD0aEXtqnwxA3ypd8CEiXpD0Qs2zACiMV7IBiRE4kBiBA4kROJAYgQOJETiQGIEDiRE4kFgtO5tk1eROI5L03nvvNbZWk9syXXTRRY2ttX379sbWkqSbbrqp0fUWwhEcSIzAgcQIHEiMwIHECBxIjMCBxAgcSIzAgcQIHEisys4mj9o+YvvtJgYCUE6VI/gvJW2oeQ4ANVgw8Ih4XdI/GpgFQGE8BgcSKxY4WxcB3VMscLYuArqHU3QgsSq/JntS0u8krbI9Y/tH9Y8FoIQqe5NtamIQAOVxig4kRuBAYgQOJEbgQGIEDiRG4EBiBA4kRuBAYgO/ddHU1FRjazW5lZAk7d+/v7G1RkdHG1trfHy8sbWa/P9DYusiAA0icCAxAgcSI3AgMQIHEiNwIDECBxIjcCAxAgcSI3AgsSoXXVxp+1Xb07b32L6zicEA9K/Ka9FPSPpJROy0fYGkKdsvR8Q7Nc8GoE9V9iZ7PyJ29t7/WNK0pBV1Dwagf4t6DG57RNJqSW/O8zW2LgI6pnLgts+X9LSkuyLio9O/ztZFQPdUCtz22ZqLe1tEPFPvSABKqfIsuiU9Imk6Ih6ofyQApVQ5gq+TdKuk9bZ3996+V/NcAAqosjfZG5LcwCwACuOVbEBiBA4kRuBAYgQOJEbgQGIEDiRG4EBiBA4kNvB7kx07dqyxtdasWdPYWlKz+4U16corr2x7hC8MjuBAYgQOJEbgQGIEDiRG4EBiBA4kRuBAYgQOJEbgQGJVLrr4Zdt/sP3H3tZFP2tiMAD9q/JS1X9JWh8Rn/Qun/yG7V9HxO9rng1An6pcdDEkfdL78OzeW9Q5FIAyqm58MGR7t6Qjkl6OCLYuAgZApcAj4rOIuELSsKS1tr85z23YugjomEU9ix4RH0p6TdKGWqYBUFSVZ9GX276w9/5XJH1H0t66BwPQvyrPol8s6XHbQ5r7B2F7RDxf71gASqjyLPqfNLcnOIABwyvZgMQIHEiMwIHECBxIjMCBxAgcSIzAgcQIHEiMrYsWYXx8vLG1MmvyZ7Z06dLG1uoijuBAYgQOJEbgQGIEDiRG4EBiBA4kRuBAYgQOJEbgQGKVA+9dG32Xba7HBgyIxRzB75Q0XdcgAMqrurPJsKTrJG2tdxwAJVU9gj8o6W5Jn9c4C4DCqmx8cL2kIxExtcDt2JsM6JgqR/B1km6wfVDSU5LW237i9BuxNxnQPQsGHhH3RsRwRIxI2ijplYi4pfbJAPSN34MDiS3qii4R8ZrmdhcFMAA4ggOJETiQGIEDiRE4kBiBA4kROJAYgQOJETiQ2MBvXdTk1jRTU//3720GWpPbCU1OTja21s0339zYWl3EERxIjMCBxAgcSIzAgcQIHEiMwIHECBxIjMCBxAgcSKzSK9l6V1T9WNJnkk5ExFidQwEoYzEvVf12RBytbRIAxXGKDiRWNfCQ9BvbU7Y31zkQgHKqnqKvi4jDtr8m6WXbeyPi9VNv0At/syRdcsklhccEcCYqHcEj4nDvv0ckPStp7Ty3YesioGOqbD54nu0LTr4v6buS3q57MAD9q3KK/nVJz9o+eftfRcSLtU4FoIgFA4+IA5K+1cAsAArj12RAYgQOJEbgQGIEDiRG4EBiBA4kRuBAYgQOJDbwWxeNjo42tlaTW+5I0sTERMq1mrRly5a2R2gVR3AgMQIHEiNwIDECBxIjcCAxAgcSI3AgMQIHEiNwILFKgdu+0PYO23ttT9u+qu7BAPSv6ktVfy7pxYj4ge1zJJ1b40wAClkwcNtLJF0t6YeSFBHHJR2vdywAJVQ5RR+VNCvpMdu7bG/tXR8dQMdVCfwsSWskPRQRqyV9Kume029ke7PtSduTs7OzhccEcCaqBD4jaSYi3ux9vENzwf8Hti4CumfBwCPiA0mHbK/qfeoaSe/UOhWAIqo+i36HpG29Z9APSLqtvpEAlFIp8IjYLWms5lkAFMYr2YDECBxIjMCBxAgcSIzAgcQIHEiMwIHECBxIjMCBxNibbBHuv//+xtaSmt1Xa2ysuRcqTk1NNbbWFx1HcCAxAgcSI3AgMQIHEiNwIDECBxIjcCAxAgcSI3AgsQUDt73K9u5T3j6yfVcTwwHoz4IvVY2IdyVdIUm2hyT9TdKzNc8FoIDFnqJfI2l/RPy1jmEAlLXYwDdKenK+L7B1EdA9lQPvbXpwg6SJ+b7O1kVA9yzmCH6tpJ0R8fe6hgFQ1mIC36T/cXoOoJsqBW77XEnjkp6pdxwAJVXdm+yfkr5a8ywACuOVbEBiBA4kRuBAYgQOJEbgQGIEDiRG4EBiBA4k5ogo/03tWUmL/ZPSZZKOFh+mG7LeN+5Xe74REQv+VVctgZ8J25MR0dwGWQ3Ket+4X93HKTqQGIEDiXUp8IfbHqBGWe8b96vjOvMYHEB5XTqCAyisE4Hb3mD7Xdv7bN/T9jwl2F5p+1Xb07b32L6z7ZlKsj1ke5ft59uepSTbF9reYXtv72d3Vdsz9aP1U/Tetdb/orkrxsxIekvSpoh4p9XB+mT7YkkXR8RO2xdImpL0/UG/XyfZ/rGkMUlLIuL6tucpxfbjkn4bEVt7Fxo9NyI+bHuuM9WFI/haSfsi4kBEHJf0lKQbW56pbxHxfkTs7L3/saRpSSvanaoM28OSrpO0te1ZSrK9RNLVkh6RpIg4PshxS90IfIWkQ6d8PKMkIZxke0TSaklvtjtJMQ9KulvS520PUtiopFlJj/Uefmy1fV7bQ/WjC4F7ns+leWrf9vmSnpZ0V0R81PY8/bJ9vaQjETHV9iw1OEvSGkkPRcRqSZ9KGujnhLoQ+Iyklad8PCzpcEuzFGX7bM3FvS0islyRdp2kG2wf1NzDqfW2n2h3pGJmJM1ExMkzrR2aC35gdSHwtyRdZvvS3pMaGyU91/JMfbNtzT2Wm46IB9qep5SIuDcihiNiRHM/q1ci4paWxyoiIj6QdMj2qt6nrpE00E+KVrpscp0i4oTt2yW9JGlI0qMRsaflsUpYJ+lWSX+2vbv3uZ9GxAstzoSF3SFpW+9gc0DSbS3P05fWf00GoD5dOEUHUBMCBxIjcCAxAgcSI3AgMQIHEiNwIDECBxL7NyyRs2/TGgiSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "first_image = data.iloc[0]\n",
    "np_image = first_image.values\n",
    "np_image = np_image.reshape(8,8)\n",
    "\n",
    "plt.imshow(np_image, cmap=\"gray_r\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot multiple images on the same grid in order to get a sense of their varied appearances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1a1f9c1390>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAADiCAYAAABupy2YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAEKpJREFUeJzt3U9sVtXWx/HfentDYgxaDMWoIJVEb8IEog0Tk1fvQKMj6kAjI2DSyWtiGXGHMNOBsQ7uhNxImRgjgwID458JOqWNbdQb7w1gCQ2JtsSSmzgwmPUOANNA7V7tc3bPWfT7mdA2i7M3P58uHw5ndZu7CwCQx/+0vQEAwOrQuAEgGRo3ACRD4waAZGjcAJAMjRsAkqFxA0AyNG4ASIbGDQDJ/KXGRbdu3eqDg4M9X+eXX34p1szNzRVrHnroodB627dvL9b09fWFrrWS2dlZLSws2Fp+b1PZ/v7778Wab7/9tlizadOm0Hq7d+8O1TVhampqwd0HVvv7msr2+vXrxZqrV68Wa6J76e/vD9U1Ya3ZSuub7+zsbM/r3LFt27ZizY4dO3peZzV9IdS4zewVSR9I6pP0T3d/Z6X6wcFBTU5ORi69otOnTxdrjh49Wqx56aWXQuu9886KfyxJ0pYtW0LXWsnQ0NAfH7eV7eLiYrEm8k0W/UZsYs9RZnZlycfhfJvKdnx8vFgzOjparHnvvfdC6+3fvz9U14S1Ziutb76HDx/ueZ07Dhw4UKwZGxvreZ2lfaGkeKvEzPok/UPSq5J2SzpgZuv39uk+RrZ1kW89ZNuuyD3ufZIuuvtld/9N0seS1u9/8fc3sq2LfOsh2xZFGvcTkpbekJu7/TX0jmzrIt96yLZFkca93M3ye34WrJmNmNmkmU3Oz8/3vrONgWzrKuZLtmvGa7dFkcY9J2npP5lul3Tt7iJ3P+HuQ+4+NDCwpn903ojItq5ivmS7Zrx2WxRp3BckPW1mT5nZJklvSjpXd1sbBtnWRb71kG2Lio8DuvtNM3tL0ue69djPh+7+ffWdbQBkWxf51kO27Qo9x+3un0r6tPJe7hF5RvvHH38s1kQGeSTpkUceKdZ88sknxZrXX389tJ7UXrYRN27cKNbMzMyErnX27NliTY3nkZvON/L8e+QZ7YiDBw+G6s6fP1+s2bt3b4+7uVdbr91jx44Va95+++1izfT0dGi9Jod5msLIOwAkQ+MGgGRo3ACQDI0bAJKhcQNAMjRuAEiGxg0AydC4ASCZKifgRExNTRVrIsM1ly5dKtbs2rUrtKfIgQuRfa9mAKctkR9G36T1/GH/NUUGcCKDSydPnizWnDlzJrSn4eHhYk0Xh0jWKjI4EzkVKDooVWN4qVe84waAZGjcAJAMjRsAkqFxA0AyNG4ASIbGDQDJ0LgBIBkaNwAk09oATuRUmmeffbZYEx2uiXjuuecau1bXNTWAs2fPnkauk8Xg4GCx5uGHHy7WRE5xiQz7SLGBn8i1IkMrWURe3x988EHoWtGTiEoOHTrUyHUk3nEDQDo0bgBIhsYNAMnQuAEgGRo3ACRD4waAZGjcAJAMjRsAkun0AE7kRJomRfa0ZcuWddhJb8bGxoo1MzMz67CTjen8+fPFmsjpNtFTa06dOlWs+eqrr4o1WU4pipxcE8lk586dofUiw0vr3Rd4xw0AydC4ASAZGjcAJEPjBoBkaNwAkAyNGwCSoXEDQDI0bgBIhsYNAMm0NjkZmTSamppqZK3IRKQkTU5OFmveeOONXrdTXeSIpMh039mzZ3vfzAa0d+/eRmqmp6dD60WmBL/55ptiTZbJyddee61YE5lMjXwPSLHj6tZbqHGb2ayk/0r6XdJNdx+quamNhGzrIt96yLY9q3nH/Td3X6i2k42NbOsi33rItgXc4waAZKKN2yV9YWZTZjayXIGZjZjZpJlNzs/PN7fD+x/Z1rVivmTbE167LYk27ufd/VlJr0r6PzP737sL3P2Euw+5+9DAwECjm7zPkW1dK+ZLtj3htduSUON292u3f/1Z0oSkfTU3tZGQbV3kWw/ZtqfYuM3sQTPbfOdjSS9L+q72xjYCsq2LfOsh23ZFnip5VNKEmd2p/8jdP6u6q42DbOsi33rItkXFxu3ulyXtaXrhXbt2FWsiAzGnT59upCbq6NGjjV2rVrb9/f3FmsjxZpEBnC4fgVYr3yZEhj/Gx8er72Ot2sw2cuRYZGimi4M1UTwOCADJ0LgBIBkaNwAkQ+MGgGRo3ACQDI0bAJKhcQNAMjRuAEimtRNwIgM47777brEmMhAzNBT7+e5NnbiTQWRIJzLocOXKldB6kdNcIqfCZBAZboqc0LK4uBha74UXXijWjI6Ohq6VQWR4aXh4uP5GWsQ7bgBIhsYNAMnQuAEgGRo3ACRD4waAZGjcAJAMjRsAkqFxA0Ay5u7NX9RsXtLSyYytkhYaX6i+Wvve6e5rOvKabEPWlO8y2Uo58+1cthKv3YBwtlUa9z2LmE26e2x8sUMy7DvDHpeTZd9Z9rlUlj1n2efdurBvbpUAQDI0bgBIZr0a94l1WqdpGfadYY/LybLvLPtcKsues+zzbq3ve13ucQMAmsOtEgBIpnrjNrNXzOzfZnbRzP5ee70mmNmsmX1rZtNmNtn2fv5MxmylHPmSbV0Z8+1StlVvlZhZn6T/SHpJ0pykC5IOuPu/qi3aADOblTTk7p19xjRrtlL38yXburLm26Vsa7/j3ifportfdvffJH0saX/lNTcKsq2HbOsi3x7VbtxPSLq65PO521/rOpf0hZlNmdlI25v5E1mzlbqfL9nWlTXfzmRb+8xJW+ZrGR5jed7dr5nZNklfmtkP7v5125u6S9Zspe7nS7Z1Zc23M9nWfsc9J2nHks+3S7pWec2eufu127/+LGlCt/5q1zUps5VS5Eu2daXMt0vZ1m7cFyQ9bWZPmdkmSW9KOld5zZ6Y2YNmtvnOx5JelvRdu7taVrpspTT5km1d6fLtWrZVb5W4+00ze0vS55L6JH3o7t/XXLMBj0qaMDPpVj4fuftn7W7pXkmzlRLkS7Z1Jc23U9kyOQkAyTA5CQDJ0LgBIBkaNwAkQ+MGgGRo3ACQDI0bAJKhcQNAMjRuAEiGxg0AydC4ASAZGjcAJEPjBoBkaNwAkAyNGwCSoXEDQDI0bgBIhsYNAMnQuAEgGRo3ACRD4waAZGjcAJAMjRsAkqFxA0AyNG4ASIbGDQDJ0LgBIBkaNwAkQ+MGgGRo3ACQDI0bAJL5S42Lbt261QcHB3u+ztWrV4s1169fL9b09fWF1vvrX/9arNm0aVPoWiuZnZ3VwsKCreX3NpXt4uJisebSpUvFmsceeyy03uOPPx6qa8LU1NSCuw+s9vc1le1PP/1UrJmbmyvWPPDAA6H1duzYUazZvHlz6Fola81Wai7fyPf87OxssaZrr93V9IVQ4zazVyR9IKlP0j/d/Z2V6gcHBzU5ORm59IpGR0eLNePj48Wa/v7+0Hrnzp0r1jTxwhsaGvrj47ayPXv2bLFmeHi4WDMyMhJa79ixY6G6JpjZlSUfh/NtKtuxsbFizZEjR4o1zzzzTGPrvfjii6Frlaw1W6m5fCPf84cPHy7WdO21u7QvlBRvlZhZn6R/SHpV0m5JB8xs95p3hz+QbV3kWw/Ztityj3ufpIvuftndf5P0saT9dbe1YZBtXeRbD9m2KNK4n5C09Gbz3O2voXdkWxf51kO2LYo07uVulvs9RWYjZjZpZpPz8/O972xjINu6ivmS7Zrx2m1RpHHPSVr6z9bbJV27u8jdT7j7kLsPDQys6R+dNyKyrauYL9muGa/dFkUa9wVJT5vZU2a2SdKbksqPXyCCbOsi33rItkXFxwHd/aaZvSXpc9167OdDd/+++s42ALKti3zrIdt2hZ7jdvdPJX1aeS9rcv78+WJN5LnPaF3Tz3S2le3ExESx5syZM8Wa999/P7ReJNtDhw6FrrUaXX7tlszMzITqIv+dmnqOe6m2so0M1+zZs6dYc/z48dB6kXmGvXv3hq7VFEbeASAZGjcAJEPjBoBkaNwAkAyNGwCSoXEDQDI0bgBIhsYNAMlUOQGnKZEfEB8RfTg+MsyTQeTP0dTgUmTQQYoNMdQYwGlD5M8ROUgBy4vkG3l9Hzx4MLReZMCJARwAwIpo3ACQDI0bAJKhcQNAMjRuAEiGxg0AydC4ASAZGjcAJNPpAZyIyMP4kQfoJWl6errH3eTR1Ikog4ODja0XOV2o6ROIaoic0IK1i7zmIgM40QG/Lg7m8Y4bAJKhcQNAMjRuAEiGxg0AydC4ASAZGjcAJEPjBoBkaNwAkEynB3CaGmSIDptETrGIDOlEh1JquXHjRrFmvffY39+/ruu1KTrw1ZSNlG2Tov2li/nyjhsAkqFxA0AyNG4ASIbGDQDJ0LgBIBkaNwAkQ+MGgGRo3ACQDI0bAJIJTU6a2ayk/0r6XdJNdx/qdeGmJhDHx8d73cqq1otMW61mKrFGtnv27CnWnDx5stdlVmVxcbFYE5lcXa0a+Zas99FlTR1Dt1ptZNuk6JFkkeMR19tqRt7/5u4L1XaysZFtXeRbD9m2gFslAJBMtHG7pC/MbMrMRmpuaAMi27rItx6ybUn0Vsnz7n7NzLZJ+tLMfnD3r5cW3P4PNyJJTz75ZMPbvK+RbV0r5ku2PeG125LQO253v3b7158lTUjat0zNCXcfcvehgYGBZnd5HyPbukr5ku3a8dptT7Fxm9mDZrb5zseSXpb0Xe2NbQRkWxf51kO27YrcKnlU0oSZ3an/yN0/q7qrjYNs6yLfesi2RcXG7e6XJZUfDMaqkW1d5FsP2bartaPLxsbGijWRIZ3Iw/HDw8ORLbV+5FhTIn+OyPBBZLgpOmzS5LW6rqmjrnbu3BmqqzG41GWRYa7R0dFizczMTGi9SB86duxYsabUh3799dfQfiSe4waAdGjcAJAMjRsAkqFxA0AyNG4ASIbGDQDJ0LgBIBkaNwAk09oATmQgIzKkE6k5cuRIZEs6ePBgsaat00aadubMmWJNZIghOmwSGfhpanClbZGBr8hQR3QgKfI9EBkQySKS3alTpxpbL/K9cuPGjWLN8ePHm9iOJN5xA0A6NG4ASIbGDQDJ0LgBIBkaNwAkQ+MGgGRo3ACQDI0bAJIxd2/+ombzkq4s+dJWSQuNL1RfrX3vdPc1HXlNtiFryneZbKWc+XYuW4nXbkA42yqN+55FzCbdfaj6Qg3LsO8Me1xOln1n2edSWfacZZ9368K+uVUCAMnQuAEgmfVq3CfWaZ2mZdh3hj0uJ8u+s+xzqSx7zrLPu7W+73W5xw0AaA63SgAgmeqN28xeMbN/m9lFM/t77fWaYGazZvatmU2b2WTb+/kzGbOVcuRLtnVlzLdL2Va9VWJmfZL+I+klSXOSLkg64O7/qrZoA8xsVtKQu3f2GdOs2Urdz5ds68qab5eyrf2Oe5+ki+5+2d1/k/SxpP2V19woyLYesq2LfHtUu3E/Ienqks/nbn+t61zSF2Y2ZWYjbW/mT2TNVup+vmRbV9Z8O5Nt7TMnbZmvZXiM5Xl3v2Zm2yR9aWY/uPvXbW/qLlmzlbqfL9nWlTXfzmRb+x33nKQdSz7fLula5TV75u7Xbv/6s6QJ3fqrXdekzFZKkS/Z1pUy3y5lW7txX5D0tJk9ZWabJL0p6VzlNXtiZg+a2eY7H0t6WdJ37e5qWemyldLkS7Z1pcu3a9lWvVXi7jfN7C1Jn0vqk/Shu39fc80GPCppwsykW/l85O6ftbuleyXNVkqQL9nWlTTfTmXL5CQAJMPkJAAkQ+MGgGRo3ACQDI0bAJKhcQNAMjRuAEiGxg0AydC4ASCZ/wfWh1qXhRUWTwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 8 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(2,4)\n",
    "\n",
    "ax[0,0].imshow(data.iloc[0].values.reshape(8,8), cmap=\"gray_r\")\n",
    "ax[0,1].imshow(data.iloc[99].values.reshape(8,8), cmap=\"gray_r\")\n",
    "ax[0,2].imshow(data.iloc[199].values.reshape(8,8), cmap=\"gray_r\")\n",
    "ax[0,3].imshow(data.iloc[299].values.reshape(8,8), cmap=\"gray_r\")\n",
    "\n",
    "ax[1,0].imshow(data.iloc[999].values.reshape(8,8), cmap=\"gray_r\")\n",
    "ax[1,1].imshow(data.iloc[1099].values.reshape(8,8), cmap=\"gray_r\")\n",
    "ax[1,2].imshow(data.iloc[1199].values.reshape(8,8), cmap=\"gray_r\")\n",
    "ax[1,3].imshow(data.iloc[1299].values.reshape(8,8), cmap=\"gray_r\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K-Nearest Neighbors Model\n",
    "\n",
    "Our baseline model uses a k-nearest neighbors algorithm; we establish a data pipeline to create, test, and establish the model's accuracy with 4-fold cross validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9677233358079684"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "## FUNCTION PIPELINE\n",
    "\n",
    "# Training function; renders 50/50 train/test split of data\n",
    "def train(neighbors, train_features, train_labels):\n",
    "    knn = KNeighborsRegressor(n_neighbors = neighbors)\n",
    "    knn.fit(train_features, train_labels)\n",
    "    return knn\n",
    "\n",
    "# Testing function\n",
    "def test(model, test_features, test_labels):\n",
    "    predictions = model.predict(test_features)\n",
    "    train_test_df = pd.DataFrame()\n",
    "    train_test_df[\"correct_labels\"] = test_labels\n",
    "    train_test_df[\"predictions\"] = predictions\n",
    "    accuracy = sum(train_test_df[\"correct_labels\"] == train_test_df[\"predictions\"])/len(train_test_df)\n",
    "    \n",
    "    return accuracy\n",
    "\n",
    "# 4-fold cross validation of testing results\n",
    "def cross_validate(k):\n",
    "    fold_accuracies = []\n",
    "    kf = KFold(n_splits = 4, random_state = 2)\n",
    "    for train_index, test_index in kf.split(data):\n",
    "        train_features, test_features = data.loc[train_index], data.loc[test_index]\n",
    "        train_labels, test_labels = labels.loc[train_index], labels.loc[test_index]\n",
    "        \n",
    "        model = train(k, train_features, train_labels)\n",
    "        accuracy = test(model, test_features, test_labels)\n",
    "        fold_accuracies.append(accuracy)\n",
    "        \n",
    "    return fold_accuracies\n",
    "\n",
    "# Run pipeline for model with k = 1\n",
    "knn_one = cross_validate(1)\n",
    "\n",
    "# Must take mean of cross-validated accuracies\n",
    "np.mean(knn_one)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a k-value of 1 and 4-fold cross validation, our model yields a respectable accuracy value of 96.8%.\n",
    "\n",
    "We can then establish a range of values for k and run the model and plot the results for each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgEAAAEWCAYAAAD/3UTfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Wd4XNW59vH/I7n3JjfZsuSGKzbggnujGAKmBjCEFhwCoThAQuAczjkJJyQhb8IhhAAxNr2YDqZXyw13XMAGjIply3Lv3SrP+2G2kkHIlixpNJLm/l2XLs3sNs+WZM89a6+9lrk7IiIiEnviol2AiIiIRIdCgIiISIxSCBAREYlRCgEiIiIxSiFAREQkRikEiIiIxCiFABGJCDO7wsw+inYdJTGz0WaWXUmv9ZSZ/b4yXkukNBQCRAJmttbMjphZqyLLl5uZm1lylOpKMbMCM3skGq9fVu7+vLufEe06KoqZDTGz/WbWuJh1y8zs5mjUJVIeCgEi35cJTCx8YmZ9gfrRKweAq4CdwGVmVrcyX9jMalXm61Vl7j4fyAYuCl9uZn2AXsCL0ahLpDwUAkS+71lCb7qFrgaeCd/AzOqa2V/MbJ2ZbTazx8ysfrCuuZm9Y2ZbzWxn8LhD2L6pZva/ZjbPzPaa2UdFWx6KcRVwD5ALnFuklt5m9rGZ7Qhq+Y9gebyZ/YeZpQevs9TMOppZctCqUSvsGKlmNil4fE1Q2/+Z2Q7gt2bWxcw+M7PtZrbNzJ43s2Zh+3c0s9eDc95uZg+HHWtu2HY9wmr91swuCVt3tpmtDmrdYGa/KvpDCH7uu4I33cJlCWZ20Mxam1mr4Oe9K3iNOWZ23P/HmdmtQS0diln9NN//+yB4/q67bw/2f8XMNpnZbjObbWa9j/I63/v5BMvczLqGnW+xf2ciFUUhQOT7FgBNzKynmcUDlwLPFdnmfqA70B/oCiQC/x2siwOeBDoBScBB4OEi+18OXAu0BuoAP3jDK2RmI4AOwHTgZcLegIJm6U+AD4D2QS2fBqtvJ9SicTbQBPgpcKAU5w8wGMgI6rsPMOCPwWv0BDoCvw1qiAfeAbKAZEI/i+nFnEdD4GPgheC4E4FHwt4gpwE/d/fGQB/gs6LHcPfDwOuEtdQAlwCz3H0LcAehT+oJQBvgP4DjGhfdzP4LuAYY5e7F9RN4FhhhZknB9nGEfp/hQfF9oFtwnl8Azx9PDWGO9XcmUiEUAkR+qLA14HTgG2BD4QozM+BnwG3uvsPd9wJ/AC4DcPft7v6aux8I1t0HjCpy/CfdfY27HyT0xt7/GLVcDbzv7jsJvYGeZWatg3XnAJvc/a/ufsjd97r7wmDdJOAed//WQ1YUflIthRx3/7u757n7QXdPc/eP3f2wu28FHgg7p0GEwsGv3X1/UMfcYo55DrDW3Z8MjvsF8BpwcbA+F+hlZk3cfWewvjgv8P0QcHmwrPAY7YBO7p7r7nO89JOjmJk9AJwJjAnO8wfcfT0wC/hJsGgcUA94N2ybJ4LfxWFCYamfmTUtZR3/KoZj/J2JVBSFAJEfepbQm8s1FLkUQOhTZgNgadDsvIvQJ/EEADNrYGb/NLMsM9sDzAaaBZ+YC20Ke3wAaFRcEUHT748JPkkG16TXBbVB6BN5+lHO4VjrSrK+SB2tzWx60Ey/h1DLSOEljI5AlrvnlXDMTsDgwp9Z8HO7AmgbrL+IUKtFlpnNMrMhRznOZ0B9MxtsZp0IBag3gnX/D0gDPjKzDDO7q/SnTDPgeuCP7r67hG3DLwlcCbzg7rnwr8swfwouw+wB1gbblXTJp6hj/p2JVBSFAJEi3D2LUAfBswk1P4fbRqiJv7e7Nwu+mrp74Rv5HcAJwGB3bwKMDJZbGUq5gFBT/iPBNeZNhJqEC9+A1gNdjrLv0dbtD743CFvWtsg2RT89/zFYdmJwTj/h3+ezHkiykjsQrifUbN8s7KuRu98I4O6L3f08Qk3obxJqIfkBdy8I1k0kFIbeCT4lE3z6vsPdOxPqO3G7mY0roa5COwm1VjxpZsNK2PZ1INHMxgAX8v2geDlwHnAa0JTQJRIo/ve/n7Dfg5mF/x5K+jsTqRAKASLFuw4Y6+77wxcGb0KPA/9X2CxvZolmdmawSWNC/3nvMrMWwP+Uo4argSeAvoQ+8fYHhgH9LXTXwjtAWzP7ZdCJrLGZDQ72nQr8r5l1s5ATzaxl0My9AfhJ8Kn1pxw9SBRqDOwLzikR+HXYukXARuBPZtbQzOod5U30HaC7mV1pZrWDr4FB34s6FhpToGnwiXoPkH+Mel4g1FfjCv59KQAzO8fMugZN6YXHONZxvsfdU4NjvhH2cyxuu/3Aq4T6fmS5+5Kw1Y2Bw8B2Qm/wfzjGS64AeptZfzOrR9DPIniNkv7ORCqEQoBIMdw9vch/7uF+Q6jZeUHQ5PsJoU//AA8SuqVwG6FOhh+U5fWDN9txwIPuvinsa2lwzKuDT8CnE/rUuwn4DhgTHOIBQp+YPyL0hjiNf9/q+DNCb+Tbgd7A5yWU8zvgZGA3oWvf/2odcff84PW7ErpUkU3oDfp7glrPIHRNOyeo936g8JbHK4G1wc/zBv59zf0Hgn4P+wn1RXg/bFU3Qr+LfcB84JHgjR0ze9+COyeOxd0/JtRpc4aZnXKMTZ8mdImj6OWiZwh1ktwArCb0N3C011oD3BvU/B1QtC/Fsf7ORCqElb7fjIiIiNQkagkQERGJUQoBIiIiMUohQEREJEZFNASY2XgLDQ+aVtw9u2bWycw+NbOVFhq6tEOwfIyFJm0p/DpkZucH654ys8ywdccaaEVERESOImIdA4PBUdYQ6r2cDSwGJrr76rBtXiF0n+/TZjYWuNbdryxynBaEesh2cPcDZvZUsM+rpa2lVatWnpycXN5TEhERqRaWLl26zd1LHFwqkjOEDQLS3D0DwMymExpEY3XYNr2A24LHMwkNElLUxYSGTS3tuOc/kJyczJIlR7vbS0REpGYxs6zSbBfJywGJfH/40exgWbgV/HtazguAxmbWssg2l/HDKTrvCy4h/J8dZWpVM7vezJaY2ZKtW4sdBlxERCSmRTIEFDdMZtFrD78CRpnZMkITkmwA/jUGuZm1IzRa2odh+9wN9AAGAi0IDajxwxdyn+LuA9x9QEKChtsWEREpKpKXA7IJTS5SqAOhkcL+xd1zCI29jZk1Ai4qMnnHJcAbhZNzBPtsDB4eNrMnOcY0rCIiInJ0kWwJWAx0M7MUM6tDqFl/RvgGZtbKQvNxQ+gT/hNFjjGRIpcCgtaBwqk2zwe+ikDtIiIiNV7EQkAwtejNhJryvwZedvdVZnavmU0INhsNfGtma4A2hOZeB8DMkgm1JMwqcujnzexL4EtC03P+PlLnICIiUpPFxNwBAwYMcN0dICIiscLMlrr7gJK204iBIiIiMUoh4Dh9snozzy3I4kheQbRLERERKReFgOP03pcbuefNrxj3QCqvLc0mv6DmX04REZGaSSHgOP31kn48ec1AmtSrzR2vrODMB2fz3pcbKVAYEBGRakYh4DiZGWN6tObtm4fzyBUnA/CL57/g3IfnMvObLcRCR0sREakZFALKKC7OOLtvOz785UgeuKQfew/lce1Ti7n4sfnMT98e7fJERERKpFsEK0hufgEvL1nPQ59+x+Y9hxnetRW/OvME+ndsFtHXFRERKaq0twgqBFSwQ7n5PLcgi0dS09mx/win9WzDHWd0p2e7JpXy+iIiIgoBYaIxWNC+w3k8OTeTKXMy2Hc4j3NObM9tp3Wjc0KjSq1DRERij0JAmGiOGLj7QC5T5qTz5Ly1HM4r4KKTE7l1XDc6NG8QlXpERKTmUwgIUxWGDd627zCPzEznuYVZuDuXD0riprFdad24XlTrEhGRmkchIExVCAGFcnYd5O+fpfHKkvXUijeuHprMDSO70LxhnWiXJiIiNYRCQJiqFAIKZW3fz4OffMebyzfQqE4trhuRwnXDU2hcr3a0SxMRkWpOISBMVQwBhdZs3ssDH63hg1WbaN6gNjeM6sJVQ5KpXyc+2qWJiEg1pRAQpiqHgEJfZu/mLx99y6w1W0loXJdbxnblsoFJ1Kml8ZxEROT4KASEqQ4hoNCizB385aNvWZS5g8Rm9Zl8WjcuPCmRWvEKAyIiUjqlDQF6Z6liBqW04KXrT+WZnw6iZaM63PnqSs54cDZvr8jRJEUiIlKhFAKqIDNjZPcE3rppGP+88hRqx8Vxy4vL+NHf5/LJ6s2apEhERCqEQkAVZmac2bst700ewd8u68/BI3lMemYJFzzyOfPStkW7PBERqeYiGgLMbLyZfWtmaWZ2VzHrO5nZp2a20sxSzaxD2Lp8M1sefM0IW55iZgvN7Dsze8nMavwN9vFxxnn9E/n49lH86cK+bNlziCumLmTilAUszdoZ7fJERKSailjHQDOLB9YApwPZwGJgoruvDtvmFeAdd3/azMYC17r7lcG6fe7+g4H2zexl4HV3n25mjwEr3P3RY9VSnToGlsah3HxeXLSOf8xMY9u+I4zt0Zo7zuhO7/ZNo12aiIhUAVWhY+AgIM3dM9z9CDAdOK/INr2AT4PHM4tZ/z1mZsBY4NVg0dPA+RVWcTVRr3Y81w5LYfadY7hz/AkszdrJjx6ay03Pf0Haln3RLk9ERKqJSIaARGB92PPsYFm4FcBFweMLgMZm1jJ4Xs/MlpjZAjMrfKNvCexy97xjHBMAM7s+2H/J1q1by3suVVKDOrX4xeiuzL5zDLeO7Urqt1s44/9mccfLK1i/40C0yxMRkSoukiHAillW9NrDr4BRZrYMGAVsAArf4JOCpozLgQfNrEspjxla6D7F3Qe4+4CEhIQynUB10bR+bW4/4wRm3zmG64an8M7KHMb+NZV73vySzXsORbs8ERGpoiIZArKBjmHPOwA54Ru4e467X+juJwH/GSzbXbgu+J4BpAInAduAZmZW62jHjGUtG9XlP3/Ui9l3juHSgR15afF6Rv55Jve9u5od+49EuzwREaliIhkCFgPdgt78dYDLgBnhG5hZKzMrrOFu4IlgeXMzq1u4DTAMWO2hXowzgYuDfa4G3orgOVRLbZrU4/fn9+WzO0ZzzontmTY3kxH3f8YDH33LnkO50S5PRESqiIiFgOC6/c3Ah8DXwMvuvsrM7jWzCcFmo4FvzWwN0Aa4L1jeE1hiZisIven/Keyugt8At5tZGqE+AtMidQ7VXccWDfjrJf346LaRjD6hNQ99lsaI+2fySGoaB47klXwAERGp0TR3QAz5asNuHvh4DZ99s4VWjepy05guXD44ibq1NGOhiEhNUhVuEZQqpk9iU564ZiCv3TiUbq0b8bu3V3Pu3+eyOmdPtEsTEZEoUAiIQad0as6L15/Kk9cMZOeBXM7/xzymzE7XBEUiIjFGISCGjenRmg9/OZIxPRL4w3vfcMXUheTsOhjtskREpJIoBMS4Fg3r8NhPTuH+i/qyInsX4x+czTsrddeliEgsUAgQzIxLBybx3q0j6JzQiJtfWMbtLy1nr24nFBGp0RQC5F+SWzXk1RuGMHlcN95akcNZf5vD4rU7ol2WiIhEiEKAfE+t+DhuO707L/98CHFmXPrP+fzlw2/JzS+IdmkiIlLBFAKkWKd0as57k0dw8SkdeHhmGhc9+jkZWzVDoYhITaIQIEfVqG4t/nxxPx77ycms23GAHz00lxcWriMWBpgSEYkFCgFSovF92vHhL0cyILk5//HGl/zsmaVs33c42mWJiEg5KQRIqbRpUo+nrx3Ef5/Ti9nfbeXMB+cw85st0S5LRETKQSFASi0uzvjp8BRm3DyMVo3qcO1Ti/nvt77i4JH8aJcmIiJloBAgx61H2ya8edMwJg1P4Zn5WZz78Fy+2rA72mWJiMhxUgiQMqlXO557zunF85MGs+9QHhc8Mo9HU9PJ1/wDIiLVhkKAlMuwrq344JcjOL1XG+7/4Bsuf3wBGzT/gIhItaAQIOXWrEEd/nH5yfzlx/34asNuxj84m7eWb4h2WSIiUgKFAKkQZsbFp3Tg/ckj6d6mMZOnL2fy9GXsPqj5B0REqiqFAKlQSS0b8NL1p3L76d15Z+VGzv7bHBZkbI92WSIiUgyFAKlwteLjuHVcN167cSi1442Jjy/g/g++4Uie5h8QEalKIhoCzGy8mX1rZmlmdlcx6zuZ2admttLMUs2sQ7C8v5nNN7NVwbpLw/Z5yswyzWx58NU/kucgZde/YzPevXUElw3syKOp6Vz46DzStmj+ARGRqiJiIcDM4oF/AGcBvYCJZtaryGZ/AZ5x9xOBe4E/BssPAFe5e29gPPCgmTUL2+/X7t4/+FoeqXOQ8mtYtxZ/vPBE/nnlKWzYeZBz/j6HZ+ev1fwDIiJVQCRbAgYBae6e4e5HgOnAeUW26QV8GjyeWbje3de4+3fB4xxgC5AQwVolws7s3ZYPfzmSwSkt+a+3VvHTpxazda/mHxARiaZIhoBEYH3Y8+xgWbgVwEXB4wuAxmbWMnwDMxsE1AHSwxbfF1wm+D8zq1vci5vZ9Wa2xMyWbN26tTznIRWkdZN6PHXtQH43oTefp29n/IOz+WT15miXJSISsyIZAqyYZUXbgH8FjDKzZcAoYAOQ968DmLUDngWudffCXmV3Az2AgUAL4DfFvbi7T3H3Ae4+ICFBjQhVhZlx9dBk3r5lOK2b1GPSM0v4jze+5MCRvJJ3FhGRChXJEJANdAx73gHICd/A3XPc/UJ3Pwn4z2DZbgAzawK8C9zj7gvC9tnoIYeBJwlddpBqpnubxrx501B+PrIzLy5axzkPzWVl9q5olyUiElMiGQIWA93MLMXM6gCXATPCNzCzVmZWWMPdwBPB8jrAG4Q6Db5SZJ92wXcDzge+iuA5SATVrRXP3Wf35PlJgzmYm8+Fj3zOP2amaf4BEZFKErEQ4O55wM3Ah8DXwMvuvsrM7jWzCcFmo4FvzWwN0Aa4L1h+CTASuKaYWwGfN7MvgS+BVsDvI3UOUjmGdmnFB5NHMr5PW/7fh99y2ZT5rN9xINpliYjUeBYLt2oNGDDAlyxZEu0ypATuzpvLN/Dfb64C4N7ze3N+/0RCjT4iIlJaZrbU3QeUtJ1GDJQqw8y44KQOvDd5BD3aNea2l1Zwy4vL2H1A8w+IiESCQoBUOR1bNGD69UP49Zkn8MFXmxj/t9l8nr4t2mWJiNQ4CgFSJcXHGTeN6crrvxhK/drxXDF1IX9872sO5+VHuzQRkRpDIUCqtBM7NOOdW4dz+aAk/jk7g/P/8TlrNu+NdlkiIjWCQoBUeQ3q1OK+C/oy9aoBbNlziHP/Ppen5mVq/gERkXJSCJBq47RebfjglyMZ2qUlv317NVc/uZgtew5FuywRkWpLIUCqlYTGdXnimoH873m9WZixnTMfnM2HqzZFuywRkWpJIUCqHTPjyiHJvHvrcBKb1+fnzy7lrtdWsn2fZiUUETkeCgFSbXVt3ZjXbxzGjaO78NKS9Qz902fc/fpK0rao46CISGloxECpEdK27GPa3Exe/yKbw3kFjD4hgZ+N6MzQLi014qCIxJzSjhioECA1yvZ9h3l+4Tqemb+WbfuO0KNtYyaN6My5/dpRt1Z8tMsTEakUCgFhFAJiz6HcfGasyGHanEy+3byXhMZ1uXpIJy4f3IkWDetEuzwRkYhSCAijEBC73J25aduYNjeT1G+3Uq92HBed3IGfDk+hS0KjaJcnIhIRpQ0BtSqjGJFoMTNGdEtgRLcE1mzeyxNzM3llaTbPL1zHuB6tuW54CkPUb0BEYpRaAiTmbNt3mOcWZPHs/Cy27z9Cr3ZNmDQihXNObE+dWrphRkSqP10OCKMQIMU5lJvPW8s3MHVOJt9t2UfrxnW5emgyVwxOolkD9RsQkepLISCMQoAci7sz+7ttTJ2TwZzvtlG/djwXn9KBa4cl01n9BkSkGlIICKMQIKX17aa9TJubwZvLcsgtKGBcjzZMGpHC4JQW6jcgItWGQkAYhQA5Xlv3HubZBVk8tyCLHfuP0CexCZOGd+bsvu3Ub0BEqrzShoCI/m9mZuPN7FszSzOzu4pZ38nMPjWzlWaWamYdwtZdbWbfBV9Xhy0/xcy+DI75kOnjmURAQuO63H56dz6/ayx/vLAvB4/k88uXljPyzzN5NDWd3Qdyo12iiEi5RawlwMzigTXA6UA2sBiY6O6rw7Z5BXjH3Z82s7HAte5+pZm1AJYAAwAHlgKnuPtOM1sETAYWAO8BD7n7+8eqRS0BUl4FBc6s77YybU4mc9NC/QYuGdCBa4elkNyqYbTLExH5nqowTsAgIM3dM4KCpgPnAavDtukF3BY8ngm8GTw+E/jY3XcE+34MjDezVKCJu88Plj8DnA8cMwSIlFdcnDHmhNaMOaE1X2/cw7S5mbywaB3PLMji9J5tmDSiMwOTm6vfgIhUK5G8HJAIrA97nh0sC7cCuCh4fAHQ2MxaHmPfxODxsY4pElE92zXhLz/ux7zfjOXmMV1ZvHYHl/xzPuf9Yx5vLd9Abn5BtEsUESmVSIaA4j4SFb328CtglJktA0YBG4C8Y+xbmmOGXtzsejNbYmZLtm7dWvqqRUqpdZN63HHGCXx+1zjuu6AP+w7nMXl6qN/AP2els/ug+g2ISNUWyRCQDXQMe94ByAnfwN1z3P1Cdz8J+M9g2e5j7JsdPD7qMcOOPcXdB7j7gISEhPKei8hR1a8TzxWDO/HJbaN44poBpLRqyB/f/4Yhf/yU385YxbrtB6JdoohIsUoMAWZ2s5k1L8OxFwPdzCzFzOoAlwEzihy7lZkV1nA38ETw+EPgDDNrHrz2GcCH7r4R2GtmpwZ3BVwFvFWG2kQqXFycMbZHG1742am8e+twxvdpy/MLsxj9l5nc8OxSlqzdQSzckisi1UdpWgLaAovN7OXglr9S9Xxy9zzgZkJv6F8DL7v7KjO718wmBJuNBr41szVAG+C+YN8dwP8SChKLgXsLOwkCNwJTgTQgHXUKlCqod/umPHBJf+b+Ziw3ju7C/IztXPzYfM5/5HPeXpFDnvoNiEgVUKpbBIM3/jOAawndtvcyMM3d0yNbXsXQLYISbQeO5PHaFxt4Ym4mmdv2k9isPtcMTebSQR1pUq92tMsTkRqmQgcL8lBS2BR85QHNgVfN7M/lqlIkRjSoU4srT+3Ep7ePYupVA+jYoj73vfc1Q/7wKfe+vZr1O9RvQEQqX4ktAWZ2K3A1sI1QM/yb7p4bXMv/zt27RL7M8lFLgFRFX23YzbS5mby9IocCd8b3act1wztzSqeydMEREfm3Cps7wMzuJdT0n1XMup7u/nXZy6wcCgFSlW3afYin56/l+QVZ7DmUx0lJzbhxVBdO69mGuDgNPiQix68iQ8CpwCp33xs8bwz0cveFFVJpJVAIkOpg/+E8Xl2azdS5GazfcZCurRtxw6gunNe/PbXjNWmRiJReRYaAZcDJQb8AgssAS9z95AqptBIoBEh1kpdfwLtfbuTR1HS+2bSX9k3r8bORnblsYBL168RHuzwRqQYqsmOgeVhScPcCIjvngEhMqxUfx3n9E3l/8gievGYgic3r87u3VzPs/s946NPv2HXgSLRLFJEaojQhIMPMbjWz2sHXZCAj0oWJxDozY0yP1rxyw1BeuWEIJ3VsxgMfr2Honz7j9++sZtPuQ9EuUUSqudJcDmgNPASMJTRO/6fAL919S+TLqxi6HCA1xTeb9vBYajpvr9xInMGFJ3Xg56M60zmhUbRLE5EqpML6BNQECgFS06zfcYApszN4ecl6juQXcFafttw4qit9OzSNdmkiUgVUZMfAesB1QG+gXuFyd/9peYusLAoBUlNt3XuYpz7P5Jn5Wew9lMfwrq34xeguDOnSklKO8C0iNVBFdgx8ltD8AWcCswjN3Le3fOWJSEVIaFyXX5/Zg8/vGstdZ/Xg2817uXzqQs7/xzw++GojBQU1v6VPRMquVLcIuvtJZrbS3U80s9qEZvQbWzkllp9aAiRWHMrN5/UvNvDP2elkbT9A54SG3DCqC+f3T6ROLY01IBIrKrIlIDf4vsvM+gBNgeRy1CYiEVKvdjyXD07i09tH8feJJ1G3Vjx3vrqSUf9vJtPmZrL/cF60SxSRKqQ0LQGTgNeAvsBTQCPgv9z9nxGvroKoJUBilbsza81WHk1NZ2HmDpo1qM01Q5O5ekgyzRvWiXZ5IhIhFdIxMBgd8GJ3f7kii6tsCgEisDRrJ4+mpvPJ15upXzueiYOSmDQihfbN6ke7NBGpYBV5d8Bsdx9ZYZVFgUKAyL+t2byXx1LTeWtFDnEG5/dP5OejutC1tcYaEKkpKjIE/BdwEHgJ2F+43N13lLfIyqIQIPJD2TsP8PjsDKYvDo01cGavttw4ugv9OjaLdmkiUk4VGQIyi1ns7t65rMVVNoUAkaPbtu8wT3++lqc/X8ueQ3kM7dKSG0d3YXjXVhprQKSa0oiBYRQCREq291AuLy5ax9Q5mWzZe5i+iU25cXQXzuzdlvg4hQGR6qQiWwKuKm65uz9TxtoqnUKASOkdzgvGGpiVztrtB+jcqiE/H9WZ809KpG4tTWUsUh1UZAj4e9jTesA44At3v7gURYwH/gbEA1Pd/U9F1icBTwPNgm3ucvf3zOwK4Ndhm54InOzuy80sFWhHqJ8CwBklTWakECBy/PILnA++2sQjqWmsytlDmyZ1mTS8MxMHJ9GormYTF6nKInY5wMyaAs+6+4QStosH1gCnA9nAYmCiu68O22YKsMzdHzWzXsB77p5c5Dh9gbcK+yAEIeBX7l7qd3WFAJGyc3fmfLeNR1PTmZ+xnab1a3P1kE5cPTSZlo3qRrs8ESlGaUNAWeL8AaBbKbYbBKS5e0ZQ0HTgPGB12DYONAkeNwVyijnORODFMtQpIhXAzBjZPYGR3RP4Yt1OHktN56HP0pgyJ4PLBibxs5GdSdRYAyLVUokhwMzeJvRmDaFhhnsBpRk8KBFYH/Y8GxhcZJvfAh+Z2S1AQ+C0Yo5zKaHwEO5JM8snNJLh772Y5gwzux64HiApKakU5YpISU5Oas6UqwaQtmUvj6Zm8NyCLJ5bkMXqn87VAAAbvUlEQVSE/u25cVQXurVpHO0SReQ4lKZPwKiwp3lAlrtnl3hgsx8DZ7r7pOD5lcAgd78lbJvbgxr+amZDgGlAH3cvCNYPJtSXoG/YPonuvsHMGhMKAc+V1ElRlwNEImPDroNMnZPB9EXrOZibz+m92nDj6C6cnNQ82qWJxLSKvBywDtjo7oeCA9c3s2R3X1vCftlAx7DnHfhhc/91wHgAd59vZvWAVkBhR7/LKHIpwN03BN/3mtkLhC47VJs7FURqksRm9fmfc3tzy9huPBWMNfDx6s2c2rkFN47uyshuGmtApCorzSyCrwAFYc/zg2UlWQx0M7MUM6tD6A19RpFt1hG62wAz60no7oOtwfM44MfA9MKNzayWmbUKHtcGzgG+KkUtIhJBLRrW4fbTuzPvrrHc86OeZG7bz9VPLOLsh+by2tJsjuQVlHwQEal0pQkBtdz9SOGT4HGJ04+5ex5wM/Ah8DXwsruvMrN7zazwzoI7gJ+Z2QpCn/ivCbu+PxLILuxYGKgLfGhmK4HlwAbg8VKcg4hUgkZ1azFpRGdm3zmG+y/qS25+AXe8soLh93/GP2amsXP/kZIPIiKVpjR9Aj4G/u7uM4Ln5wG3uvu4SqivQqhPgEh0FE5lPG1uJnO+20a92nFcfEoHfjoshc4JmrBIJFIqcrCgLsDzQPtgUTZwlbunlbvKSqIQIBJ932zaw7Q5mby1PIfcggLG9WjDpBEpDE5poX4DIhWswgcLMrNGwfZ7y1tcZVMIEKk6tuw9xHPzs3h2QRY7D+TSJ7EJk4Z35uy+7ahTqzRXKEWkJBXZEvAH4M/uvit43hy4w93vqZBKK4FCgEjVcyg3NEfBtLkZpG/dT9sm9bh6aDKXD0qiaYPa0S5PpFqryBCwzN1PKrLsC3c/uZw1VhqFAJGqq6Ag1G9g6twM5qVtp37teC4Z0IFrh6WQ3KphtMsTqZYqcpyAeDOr6+6HgwPXJ9RLX0Sk3OLijDE9WjOmR2tW5+xh2txMXli0jmcWZHF6zzZMGtGZgcnN1W9AJAJK0xJwJzABeDJYdC0ww93/HOHaKoxaAkSqly17DvHM/CyeW5jFrgO5nNihKdcNT+Hsvu2oHa9+AyIlqdCOgcGUwKcBBuwE2rn7TeWuspIoBIhUTweP5PPaF9k8MTeTjG37ade0HtcMTeayQUk0ra9+AyJHU9EhoD9wOXAJkAm85u4Pl7vKSqIQIFK9FRQ4M7/dwtQ5mczP2E6DOvFcMqAjPx2WQlLLBtEuT6TKKXefADPrTmio34nAduAlQqFhTIVVKSJSCnFxxriebRjXsw1fbdjNE3MzeW5BFs/MX8sZvdoyaUQKp3RSvwGR43XUlgAzKwDmANcVDgxkZhnu3rkS66sQagkQqXk27T7EM/PX8vzCdew+mEu/js2YNDyFs/q0pZb6DUiMK/flADO7gFBLwFDgA0IT+Ux195SKLLQyKASI1FwHjuTx2tJsps3NZO32AyQ2q881Q5O5dFBHmtRTvwGJTRU5TkBD4HxClwXGAk8Db7j7RxVRaGVQCBCp+QoKnE+/2cLUORkszNxBwzrxXDowiWuHJdOxhfoNSGyp8GGDg4O2IDS976XuPrYc9VUqhQCR2PJl9m6mzc3gnZUbKXBnfJ+2XDe8M6d0ah7t0kQqRURCQHWlECASmzbuPsjTn2fxwsIs9hzK46SkZkwa3pkze7dRvwGp0RQCwigEiMS2/YfzeHVpNk/MyyQr6Ddw7bBkLh3YkcbqNyA1kEJAGIUAEQHIL3A++Xoz0+ZksmjtDhrVrcVlAztyzbBkOjRXvwGpORQCwigEiEhRK9bvYtrcTN79ciMAZ/Vpy6QRnenfsVmUKxMpP4WAMAoBInI0ObsO8vTna3lh0Tr2HspjQKfmTBqRwum92hIfp8GHpHpSCAijECAiJdl3OI9XlqzniXmZrN9xkI4t6vPTYSn8eEBHGtUtzYSrIlVHaUNARLvHmtl4M/vWzNLM7K5i1ieZ2UwzW2ZmK83s7GB5spkdNLPlwddjYfucYmZfBsd8yDROqIhUgEZ1a3HtsBRSfzWGx35yMm0a1+N3b69myB8/5Q/vfc2m3YeiXaJIhYtYS4CZxQNrgNOBbGAxMNHdV4dtMwVY5u6Pmlkv4D13TzazZOAdd+9TzHEXAZOBBcB7wEPu/v6xalFLgIiUxbJ1O5k2N5P3v9pEvBmXDerIDaO60L5Z/WiXJnJM5Z5AqAIMAtLcPSMoaDpwHrA6bBsHmgSPmwI5xzqgmbUDmrj7/OD5M4RGMzxmCBARKYuTkprz8OXNWb/jAI+kpvHCwnVMX7SeHw/owC/GdCVRYUCquUheDkgE1oc9zw6Whfst8BMzyyb0qf6WsHUpwWWCWWY2IuyY2SUcEwAzu97MlpjZkq1bt5bjNEQk1nVs0YA/Xngiqb8ezY8HdODlJesZ/f9mcvfrK1m/40C0yxMps0iGgOKu1Re99jAReMrdOwBnA8+aWRywEUhy95OA24EXzKxJKY8ZWug+xd0HuPuAhISEMp+EiEihDs0bcN8FfZn16zFcNjCJ15ZuYMxfUvnNqytZt11hQKqfSIaAbKBj2PMO/LC5/zrgZYCgib8e0MrdD7v79mD5UiAd6B4cs0MJxxQRiaj2zerzv+f3Ydado/nJqZ14Y/kGxvw1lV+9soK12/ZHuzyRUotkCFgMdDOzFDOrQ2ha4hlFtlkHjAMws56EQsBWM0sIOhZiZp2BbkCGu28E9prZqcFdAVcBb0XwHEREjqpd0/r8dkJv5tw5hquGdOLtFTmM/Wsqt7+0nIyt+6JdnkiJIjpOQHDL34NAPPCEu99nZvcCS9x9RnBHwONAI0LN+ne6+0dmdhFwL5AH5AP/4+5vB8ccADwF1CfUIfAWL+EkdHeAiFSGLXsPMWVWBs8tzOJIXgET+rXn5rHd6Nq6UbRLkxijwYLCKASISGXauvcwj8/J4Nn5WRzKy+ecE9tz69iudGvTONqlSYxQCAijECAi0bB932Een5PJM/PXcjA3n7P7tOOWcV3p0bZJifuKlIdCQBiFABGJph37jzBtbgZPf57FvsN5nNWnLbeO60bPdgoDEhkKAWEUAkSkKth14AjT5mby1Ly17D2cxxm92nDruG70SWwa7dKkhlEICKMQICJVye4DuTwxL5Mn5mWy91Aep/VszeRx3enbQWFAKoZCQBiFABGpivYcyuWpeWuZNjeT3QdzGdujNbeO60b/js2iXZpUcwoBYRQCRKQq23sol2fmZ/H4nAx2HchlVPcEJp/WjZOTmke7NKmmFALCKASISHWw73Aez8xfy+OzM9h5IJcR3VoxeVw3BiS3iHZpUs0oBIRRCBCR6mT/4TyeW5DFlNkZbN9/hGFdWzJ5XHcGpSgMSOkoBIRRCBCR6ujAkTxeWLiOx2ZlsG3fYU7t3ILJ47ozpEvLaJcmVZxCQBiFABGpzg4eyeeFRet4bFY6W/ceZlBKCyaP68bQLi0JTaMi8n0KAWEUAkSkJjiUm8/0Ret4dFY6m/ccZkCn5kw+rRvDu7ZSGJDvUQgIoxAgIjXJodx8Xl6ynkdT09m4+xAnJzXj1nHdGNU9QWFAAIWA71EIEJGa6HBePq8syebR1HQ27DpIv47NmDyuK2NOaK0wEOMUAsIoBIhITXYkr4DXvsjmHzPTyN55kBM7NOXWsd0Y11NhIFYpBIRRCBCRWJCbX8DrX2Tz8Mw01u84SO/2Tbh1XDfO6NVGYSDGKASEUQgQkViSm1/Am8s28PDMNLK2H6BnuybcOrYrZ/ZuS1ycwkAsUAgIoxAgIrEoL7+AGStyePizNDK27adH28bcMrYb4/u0JV5hoEZTCAijECAisSy/wHl7RQ5//+w70rfup3NCQ24Y2YXzT0qkTq24aJcnEaAQEEYhQEQkFAbe/2ojj6amsypnD22b1GPSiBQmDkqiYd1a0S5PKlBpQ0BEI6CZjTezb80szczuKmZ9kpnNNLNlZrbSzM4Olp9uZkvN7Mvg+9iwfVKDYy4PvlpH8hxERGqK+DjjnBPb884tw3n6p4NIbtWA37/7NUP/9BkPfLyGHfuPRLtEqWQRawkws3hgDXA6kA0sBia6++qwbaYAy9z9UTPrBbzn7slmdhKw2d1zzKwP8KG7Jwb7pAK/cvdSf7RXS4CISPG+WLeTx1LT+Wj1ZurXjueyQR2ZNKIzic3qR7s0KYfStgREsv1nEJDm7hlBQdOB84DVYds40CR43BTIAXD3ZWHbrALqmVlddz8cwXpFRGLOyUnNmXLVAL7bvJfHZmXw7Pwsnp2fxfknJXLDqM50bd042iVKBEXyckAisD7seXawLNxvgZ+YWTbwHnBLMce5iFBrQXgAeDK4FPBfdpSbX83sejNbYmZLtm7dWuaTEBGJBd3aNOavl/Qj9dej+cmpnXhnZQ6nPTCb659ZwvL1u6JdnkRIJENAcW/ORa89TASecvcOwNnAs2b2r5rMrDdwP/DzsH2ucPe+wIjg68riXtzdp7j7AHcfkJCQUI7TEBGJHR2aN+C3E3rz+V3juHVsVxZm7uD8f8xj4pQFzF6zlVjoTB5LIhkCsoGOYc87EDT3h7kOeBnA3ecD9YBWAGbWAXgDuMrd0wt3cPcNwfe9wAuELjuIiEgFatGwDrefcQLz7hrLPT/qSca2fVz1xCLOfXgu767cSH6BwkBNEMkQsBjoZmYpZlYHuAyYUWSbdcA4ADPrSSgEbDWzZsC7wN3uPq9wYzOrZWaFIaE2cA7wVQTPQUQkpjWqW4tJIzoz+84x3H9RX/YfzuemF77gtAdmMX3ROg7n5Ue7RCmHiI4TENzy9yAQDzzh7veZ2b3AEnefEdwR8DjQiNClgjvd/SMzuwe4G/gu7HBnAPuB2UDt4JifALe7+zH/CnV3gIhIxcgvcD5ctYlHU9P5csNuWjeuy6QRKVw+uBONNNZAlaHBgsIoBIiIVCx3Z17adh5JTePz9O00qVeLq4cmc83QZFo2qhvt8mKeQkAYhQARkchZvn4Xj6Wm8+HqTdStFcdlA5OYNCKFDs0bRLu0mKUQEEYhQEQk8tK27OWfszJ4Y9kGHDivX3tuGN2F7m001kBlUwgIoxAgIlJ5cnYdZOqcTF5ctI6Dufmc1rMNN47uwimdmke7tJihEBBGIUBEpPLt3H+Epz5fy9Pz17LrQC6DU1pw4+gujOqewFHGeZMKohAQRiFARCR69h/OY/ri9Uydk8HG3Yfo1a4JN47uwtl92xEfpzAQCQoBYRQCRESi70heAW8u38Bjs9LJ2LqfTi0b8PORXbjw5ETq1Y6Pdnk1ikJAGIUAEZGqI7/A+Xj1Jh5JTWdl9m4SGtfluuEpXDE4icb1ake7vBpBISCMQoCISNXj7sxP384jqenMTdtG43q1uGpIJ64dlkIrjTVQLgoBYRQCRESqtpXZu3g0NZ0PVm2iTnwclw7syM9GdKZjC401UBYKAWEUAkREqof0rfuYMiuD15dlU+Bw7ontuHF0V05oq7EGjodCQBiFABGR6mXT7kNMnZPBC4vWceBIPuN6tOYXY7pwSqcW0S6tWlAICKMQICJSPe06cISnP8/iqc8z2Xkgl0HJobEGRp+gsQaORSEgjEKAiEj1duBIHi8tXs/jszPI2X2IHm0bc/3IzpzZuy0NNXvhDygEhFEIEBGpGY7kFTBjRQ6PzUonbcs+6teOZ1zP1kzo155RJyRQt5bGGwCFgO9RCBARqVkKCpwlWTt5e0UO7325ke37j9C4Xi3G927LhP7tGdK5JbXi46JdZtQoBIRRCBARqbny8guYl76dGctz+GjVJvYezqNVozqc3bcdE/q15+Sk5sTF2PDECgFhFAJERGLDodx8Ur/dytsrcvjk680czisgsVl9zjmxHef2a0/v9k1iokOhQkAYhQARkdiz73Aen6zezIwVOcxes5W8AqdzQkPOPbE9E/q3p0tCo2iXGDEKAWEUAkREYtuuA0d4/6tNzFiew4LM7bhDr3ZNmNC/Pef2a09is/rRLrFCVYkQYGbjgb8B8cBUd/9TkfVJwNNAs2Cbu9z9vWDd3cB1QD5wq7t/WJpjFkchQERECm3Zc4h3Vm5kxooclq/fBcApnZozoV97zu7bjoTG1X/egqiHADOLB9YApwPZwGJgoruvDttmCrDM3R81s17Ae+6eHDx+ERgEtAc+AboHux3zmMVRCBARkeKs33GAGStyeHtFDt9s2kucwdAurZjQrz1n9mlL0/rVc1bD0oaASI6wMAhIc/eMoKDpwHlA+Bu2A02Cx02BnODxecB0dz8MZJpZWnA8SnFMERGRUunYogE3jenKTWO6smbzXt5ekcOMFTnc+dpK7nnzK0adkMC5/dpzWs/WNKhT8wYliuQZJQLrw55nA4OLbPNb4CMzuwVoCJwWtu+CIvsmBo9LOiYAZnY9cD1AUlLS8VcvIiIxpXubxtxxxgncfnp3VmbvZsaKHN5ZmcPHqzdTv3Y8p/dqw7n92jOye6saMyhRJENAcfdgFL32MBF4yt3/amZDgGfNrM8x9i1u5Idir2e4+xRgCoQuB5S6ahERiWlmRr+OzejXsRn/eXZPFq3dwYwVObz/ZagfQZN6tRjfpy0T+iUypEtL4qvxGASRDAHZQMew5x34d3N/oeuA8QDuPt/M6gGtSti3pGOKiIhUiLg449TOLTm1c0t+N6E3c9O28fbyHN77chMvL8mmVaO6/KhvaJTCk5OaV7sxCCIZAhYD3cwsBdgAXAZcXmSbdcA44Ckz6wnUA7YCM4AXzOwBQh0DuwGLCLUQlHRMERGRClc7Po4xJ7RmzAmtOZSbz8xvtjBjRQ4vLl7P0/OzQoMS9QuNUtirXfUYlChiIcDd88zsZuBDQrfzPeHuq8zsXmCJu88A7gAeN7PbCDXrX+Oh2xVWmdnLhDr85QE3uXs+QHHHjNQ5iIiIFKde7XjO6tuOs/q2Y++hXD4OBiWaNieTf87KoEtCQ87t154J/drTuQoPSqTBgkRERCrIjv1HeP+rjcxYnsOitTtwhz6JTTj3xNCgRO0raVCiqI8TUJUoBIiISGXbtPsQ76wMjUGwIns3AAOTQ4MSndW3Ha0aRW5QIoWAMAoBIiISTVnb9/9rDII1m/cRH2cM7dKSCf3ac0bvih+USCEgjEKAiIhUFd9s2vOvQLB+x0HqxMdx+eAkfjuhd4W9RlUYMVBERESK6NG2CT3aNuFXZ5zA8vW7eHvFRto1rReVWhQCREREosDMOCmpOSclNY9aDcWNwCciIiIxQCFAREQkRikEiIiIxCiFABERkRilECAiIhKjFAJERERilEKAiIhIjFIIEBERiVExMWywmW0FsirwkK2AbRV4vGjSuVQ9NeU8QOdSFdWU8wCdy7F0cveEkjaKiRBQ0cxsSWnGZK4OdC5VT005D9C5VEU15TxA51IRdDlAREQkRikEiIiIxCiFgLKZEu0CKpDOpeqpKecBOpeqqKacB+hcyk19AkRERGKUWgJERERilEKAiIhIjFIIOA5m9oSZbTGzr6JdS3mYWUczm2lmX5vZKjObHO2aysrM6pnZIjNbEZzL76JdU3mZWbyZLTOzd6JdS3mY2Voz+9LMlpvZkmjXU1Zm1szMXjWzb4J/M0OiXVNZmNkJwe+i8GuPmf0y2nWVlZndFvyb/8rMXjSzetGuqSzMbHJwDqui8ftQn4DjYGYjgX3AM+7eJ9r1lJWZtQPaufsXZtYYWAqc7+6ro1zacTMzAxq6+z4zqw3MBSa7+4Iol1ZmZnY7MABo4u7nRLuesjKztcAAd6/Wg7mY2dPAHHefamZ1gAbuvivadZWHmcUDG4DB7l6RA6lVCjNLJPRvvZe7HzSzl4H33P2p6FZ2fMysDzAdGAQcAT4AbnT37yqrBrUEHAd3nw3siHYd5eXuG939i+DxXuBrIDG6VZWNh+wLntYOvqptsjWzDsCPgKnRrkXAzJoAI4FpAO5+pLoHgMA4IL06BoAwtYD6ZlYLaADkRLmesugJLHD3A+6eB8wCLqjMAhQCYpyZJQMnAQujW0nZBc3ny4EtwMfuXm3PBXgQuBMoiHYhFcCBj8xsqZldH+1iyqgzsBV4MrhEM9XMGka7qApwGfBitIsoK3ffAPwFWAdsBHa7+0fRrapMvgJGmllLM2sAnA10rMwCFAJimJk1Al4Dfunue6JdT1m5e7679wc6AIOCJrZqx8zOAba4+9Jo11JBhrn7ycBZwE3B5bTqphZwMvCou58E7Afuim5J5RNc0pgAvBLtWsrKzJoD5wEpQHugoZn9JLpVHT93/xq4H/iY0KWAFUBeZdagEBCjguvnrwHPu/vr0a6nIgTNtKnA+CiXUlbDgAnBtfTpwFgzey66JZWdu+cE37cAbxC67lndZAPZYa1LrxIKBdXZWcAX7r452oWUw2lAprtvdfdc4HVgaJRrKhN3n+buJ7v7SEKXmyutPwAoBMSkoDPdNOBrd38g2vWUh5klmFmz4HF9Qv85fBPdqsrG3e929w7unkyoufYzd692n24AzKxh0OmUoPn8DEJNn9WKu28C1pvZCcGicUC160BbxESq8aWAwDrgVDNrEPx/No5Q36Zqx8xaB9+TgAup5N9Nrcp8serOzF4ERgOtzCwb+B93nxbdqspkGHAl8GVwLR3gP9z9vSjWVFbtgKeD3s5xwMvuXq1vrash2gBvhP5/phbwgrt/EN2SyuwW4PmgGT0DuDbK9ZRZcN35dODn0a6lPNx9oZm9CnxBqPl8GdV3COHXzKwlkAvc5O47K/PFdYugiIhIjNLlABERkRilECAiIhKjFAJERERilEKAiIhIjFIIEBERiVEKASLyPWaWXNJMmWaWGXbvfOGyB83szvIcV0Qql0KAiJTFdEIDGgFgZnHAxcBLUatIRI6bQoCIHJWZdQ4mzhlYZNWLhIUAQjPtrXX3rOAT/xwz+yL4+sFwrmZ2jZk9HPb8HTMbHTw+w8zmB/u+EsxxISIRoBAgIsUKmvtfA65198Xh69x9JVBgZv2CReGz0m0BTg8mD7oUeOg4XrMVcA9wWrD/EuD2cp2IiByVhg0WkeIkAG8BF7n7qqNs8yJwmZmtIjSj238Hy2sDD5tZfyAf6H4cr3sq0AuYFww5XAeYf/zli0hpKASISHF2A+sJzTNxrBDwETALWBnMFghwG7AZ6EeotfFQMfvm8f2WyHrBdwM+dveJ5apeREpFlwNEpDhHgPOBq8zs8uI2cPd0YDvwJ74/81lTYKO7FxCaqCq+mN3XAv3NLM7MOvLvaYYXAMPMrCuEJrwxs+NpSRCR46AQICLFcvf9wDnAbWZ23lE2exHoAbwRtuwR4GozW0DoUsD+YvabB2QCXwJ/ITQbHO6+FbgGeNHMVhIKBT3KfTIiUizNIigiIhKj1BIgIiISoxQCREREYpRCgIiISIxSCBAREYlRCgEiIiIxSiFAREQkRikEiIiIxKj/D1XzHM8BoUA+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Update parameters\n",
    "k_values = list(range(1,10))\n",
    "k_accuracies = []\n",
    "\n",
    "# Enfold into model pipeline\n",
    "for k in k_values:\n",
    "    accuracy = cross_validate(k)\n",
    "    mean_accuracy = np.mean(accuracy)\n",
    "    k_accuracies.append(mean_accuracy)\n",
    "    \n",
    "plt.figure(figsize = (8,4))\n",
    "plt.plot(k_values, k_accuracies)\n",
    "plt.title(\"Mean Accuracies vs. k Value\")\n",
    "plt.xlabel(\"k Value\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the plot that a k-nearest neighbors model with 4-fold cross validation generally yields diminishing accuracy as k is increased. The most accurate model features a k-value of 1.\n",
    "\n",
    "We will use neural networks with increasing numbers of hidden layers to generate more accurate models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Single Layer Neural Network\n",
    "\n",
    "The power of neural networks lies in their ability to construct and learn from intermediate stage models within hidden layers. We'll test a single-layer network to gauge its comparison in accuracy to the k-nearest neighbors model.\n",
    "\n",
    "The data pipeline remains largely the same aside from the replacement of the k-nearest neighbors estimator object with a neural network object from scikit Learn's `neural_network.MLPClassifier` class. We iterate over an array of values for neuron quantities within the single hidden layer (8, 16, 32, 64, 128, 256) and continue to use 4-fold cross validation to assess the accuracy of each iteration, plotting the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import packages\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAEWCAYAAABhUT6OAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmYXGWZ9/HvL510ks6edEgg6SRsAkH2CKiDbCKLIgqvCrIqiguMM76irzoMIurgAuMGIkEjICrDMOpERIFhiDuQIKAGRAIC6U4CnaQ7ay/p7vv945zunK70Uh26Ul3Vv8911dVnr7uePlV3PU895zyKCMzMzKx8jSh2AGZmZlZYTvZmZmZlzsnezMyszDnZm5mZlTknezMzszLnZG9mZlbmnOzNckg6V9J9xY5jKJIUkvYp0nPvJ+kxSZskfWSA+x4j6elBiuMWSZ8fjGOZ7SpO9mVA0vOSWiVV5yx/PP1wnlekuPaU1CHpW8V4/p0VET+IiDcVO47+SDou/f/ekLP8t5IuKlJYhfQJYElETIiIb+SulHSgpPskNUhqlPSopNMAIuI3EbHfLo94xxiL9mWpN5KuSuN6R2bZyGJ+dtjgc7IvH38HzumckXQQMLZ44QBwAdAAnC1p9K58Ykkjd+XzFdEW4IJS+1Deyf/PXGB5H+t/BtwPzAB2Az4CbNyJ5ylbfZT7euBqSRW7IIaCP4ftyMm+fHyfJLl2uhC4LbuBpNGSrpX0oqSXJH1b0th03RRJd0uqT2tGd0uandl3iaTPSfpd2ox6X25LQg8uAK4AtgGn58RyoKT7Ja1PY/l0urxC0qclPZs+z6OSaiTNS2saIzPHWCLpfen0RWlsX5W0HrhK0t6S/lfSOklrJf1A0uTM/jWSfpy+5nWSrs8c67eZ7fbPxPq0pHdm1p0m6ck01jpJl+cWQlrujZJenVk2XVKTpN0kVafl3Zg+x28k5fvebARuAT7T08q01nZ7Zr5bOaZl+HlJv5e0WdLPJE1Ly2qjpKU9fJE4TdJzaZl+JRurpPdKeio9h+6VNDezLiRdKukZ4Jle4n2rpOVpWSyRdEC6/H+B44Hr0zhflbNfNbAncHNEtKaP30XEb9P1x0mqzWz/vKTLJf1J0gZJ/yFpTGb9JyStlrRK0vvUR41c0luUtKI1puV4cE/b9aWvc1XSxyX9V87235T0tXR6kqTvpvHWpf/PinTdDu+LXkL4JdAKnNdLfH19dnR7v6TLuspLyc8eN0q6R9IW4Pg05tvS994Lkq7oPI86j5c+X4Okv0s6NXPsi9Lzb1O67tyBlvewFBF+lPgDeB54I/A0cABQAawkqQkFMC/d7mvAYmAqMIGkJnRNum4acBZQla77T+CnmedYAjwLvIqkxWAJ8MU+YjoGaAGmAN8EFmfWTQBWAx8DxqTzR6XrPg78GdgPEHBIGtu89LWMzInpfen0RUAb8I/AyDTGfYCTgNHAdODXwNfS7SuAJ4CvAuPSOP4hc6zfptPj0rJ8T3rcw4G1wIHp+tXAMen0FODwXspjEfCFzPylwC/T6WuAbwOj0scxgPL4vx8H1AIzSWqw+6XLfwtclE5fBdye2adbOaZluALYG5gEPAn8jeR8GknyhfF7mf0DeJDkHJqTbtv5P3hbeqwD0n2vAH6fs+/96b5je3g9ryJpqTgpLYdPpMerzP1/97CvSL5A3J3GMaOnssp5zzwC7JHG8xTwwXTdKcAa4ECS98P309j3SdffAnw+nT4ceBk4iuScujA99uhe4uw6Ts7yvs7V3dNymZzOj0yf84h0/qfATSTn6m7p6/pAb++LHp77KuB24K3Ac2nZjyT/z46LSN8vPb3OtLw2AK8nqWCOITmv/js91jyS8+jizPG2Ae9Py/RDwKr0fzyO7uf67qTvRT/6+bwodgB+DMI/cXuyv4IkcZxC8qHa9YZN3yhbgL0z+70W+HsvxzwUaMjMLwGuyMx/mDRZ9bL/d0i/LKTPsw3YLZ0/B3isl/2eBs7oYfk8+k/2L/ZTTm/rfN40pvrs8TLbdX14Ae8CfpOz/ibgM+n0i8AHgIn9PPcbgecy878DLkinr04/+HZIAv0c8zjSBAZ8GfiPdHqgyf5fMuuvA36RmT8deDwzH8ApOefBA+n0L0g/sNP5EcBWYG5m3xP6eD3/CtyZs38dcFzu/7uX/WcD15N8Ke0gSZj75pZV5j1zXmb+y8C30+lFpIksnd+H3pP9jcDnejiHj+0lxh6TfV/naqZs359OvwV4Mp2eQfKlemxm23OABwfwvug6R4CHSZJr3p8d5Jfsb8usq0hjnp9Z9gGS/hidx1uRWVeVHm8mSbJvJKmY7PDFxY/eH27GLy/fB95N8ma5LWfddJI3zaNpc2MjSdPddABJVZJuSpvUNpJ8UE5W99/X1mSmtwLjewoibd57B/ADgIj4A0lSfHe6SQ3JB3JP+lrXn5U5cewm6Y60aXMjSe2l86eHGuCFiGjr55hzgaM6yywtt3NJPngg+dA5DXhB0q8kvbaX4/wvMFbSUWnT9qHAT9J1XyGpwd6XNk9+Mv+X3OVLwMmSDtmJfV/KTDf1MJ/7f86W8wsktWNIyurrmXJaT5IoZvWyb6490uMBEBEd6fazet0jIyJqI+KyiNg7jWULO74Psno7n/fIibOvmOcCH8s5P2rYXiZ56edcBbiV7U3s55G81zuffxSwOvP8N5HU8POJP9cVwL+Q1L479fnZkadsDNVAJZn/dTqd/T93/W8iYms6OT4itpB8Af8gyWv+uaT9BxDHsOVkX0Yi4gWSjnqnAT/OWb2W5IP7wIiYnD4mRUTnB9zHSJrOj4qIicAb0uXaiVDeDkwEviVpjaQ1JG/kzj4FK0majXvS27ot6d+qzLKZOdtEzvw16bKD09d0Httfz0pgjvrvKLYS+FWmzCZHxPiI+BBARCyNiDNIPlx/CtzZ00HSxHUnSa3r3cDdEbEpXbcpIj4WEXuR1KT/r6QT+4kr9/jrSJpaP5ezagt9l9nOqMlMzyFpYoWkrD6QU1ZjI+L32VD7OO4qkuQFgCSlz1U30AAjYiVwA/Dq/rbtwWqSVoJONb1tSPKav5Dzmqsi4kcDfM6+zlVIzq2DlfT7eAvpF+n0+VuA6szzT4yIAzP79lXm3UTE/SRfPD+cWdzfZ0e3c0xST+dYNoa1JC19czPL5pDn/zki7o2Ik0ia8P8K3JzPfsOdk335uZikqXRLdmGabG4GvippNwBJsySdnG4ygeQN3ShpKr10+MrThSRNoQeR1GAPJfm97lAlVwncDcyU9M9px58Jko5K9/0O8DlJ+ypxsKRpEVFP8mFwnpJOfO+l9y8MnSYAm9PXNIukP0CnR0g+1L8oaZykMZJe38Mx7gZeJel8SaPSx2skHSCpUsk1+ZMiYhvJb4ntfcTzQ5JaybnpNNDVwWufNLl1HqOv4/Tm34HXkfxm3ulx4A2S5kiaBHxqJ46b6+NKOnTWAP8E/Ee6/NvApyQdCF0dx97R20F6cCfwZkknShpF8gW0Bfh937t1dTD9bFqOI5R02Hsv8NAAnj8bx3vS/3EVcGUf294MfDBtsVF6Lr1Z0oQ+9qlMz7fORwV9n6tERDNwF8l580hEvJguXw3cB1wnaWL62veWdOxOvO5O/0LSX6Lzufv77HgCOFDSoUo6OV7V18Ejop2kjL+QvvfnAv+XpDWjT5JmKOnEOY7k3NjMzr1Xhh0n+zITEc9GxLJeVv8/km/tD6VNhf9DUpuHpFY4luRb90MkzXQDln5QnUjSuWhN5vFoeswL0xrtSSS12DUkHauOTw/x7yQfBPeRJL7vsv0SwveTfAiuI+k81V8S+CxJB6oNwM/JtHakHzink/we+yJJR7d35R4gjfVNwNkkNc81JE3mnZcSng88n5bnB+mlN3N6rIdJakF7kPwG22lfkv/FZuAPwLciYgmApF8ovVKhPxGxkeS356mZZfeTJOM/AY+SfHl5pf47PdbjJOX63fS5fkJSNnek5fEX4NTeDtJD/E+TlN83Sc7D04HTI6I1j91bSX5f/h+S8+YvJMngonyfPxPHL4BvkHREXEHyPyE9Xu62y0jOy+tJLjNdkcdzLif5Yt35eA99nKsZt5J8gf5+zvILSJrFn0xjuIuk1rtTIuJ3JF+Gs3r97IiIv5H0O/kfkvfyb+nfP5K8F55Lt/8hSQWhPyNIvgSuIvmZ6Fi6t0JYLxSRdwuPmdmwo+Tyv7+Q9LDvr49HIeOYQ9JsPTP9YmeWN9fszcxySHp7+jPNFJLWip8VOdGPIGnqvsOJ3naGk72Z2Y4+QHJp5rMkvwl/qFiBpL9PbyT56euV9KWxYczN+GZmZmXONXszM7MyVzaDhVRXV8e8efOKHYaZmdku8+ijj66NiH5vcFQ2yX7evHksW9bbFWdmZmblR9IL/W9V4GZ8SacoGSVsRU+3AJU0V9IDSkaeWqLuo6y1KxlJ6nFJiwsZp5mZWTkrWM0+vSvUDSQ9SGuBpZIWR8STmc2uJRkg4VZJJ5DcMvL8dF1TRBxaqPjMzMyGi0LW7I8kGbnoufQOWHcAZ+RsMx94IJ1+sIf1ZmZm9goVMtnPovtIR7XsOHrVEySjhkEyeMoESdPS+TGSlkl6SNLbenoCSZek2yyrr68fzNjNzMzKRiGTfU+jpeVe1H85cKykx0jucVwHdN6lak5ELCAZIexrknYY9CQiFkbEgohYMH36QEZbNDMzGz4K2Ru/lu5DQ85m+1CYAETEKuBMAEnjgbMiYkNmHRHxnKQlwGHs/DjnZmZmw1Yha/ZLgX0l7SmpkmTUsG696iVVp/d8hmTozUXp8imSRnduQzI8arZjn5mZmeWpYDX7iGiTdBlwL1ABLIqI5ZKuBpZFxGLgOOAaSQH8Grg03f0A4CZJHSRfSL6Y04vfzMxsSIoINre0Ub+phbWbW1m7uSV5bGrhbYfNYq/p43d5TAW9qU5E3APck7Psysz0XSRjL+fu93uScZvNzMyKLiLY2NzG2s0taRJPknc2mddvbk2XtdDS1rHDMSQ4cNak8kv2ZmZmQ1VEsKFpG2s3t/ByZ+LuTOSbM4k8XdfavmMCrxghpo6rpHr8aKrHV7J39TiqJyTT1eNHM33C6HTdaKaOq6RiRE991wvPyd7MzMpGR0fQmCbwzhr4Ds3pm1tYu6mVdVta2Na+48ivI0eIaeMru5L0vrtNoHpCJdPHb0/c09OEPqWqkhFFSuAD4WRvZmZDWntH0LC1tStJ129uZu2m1rTpvHuNfN2WVto7dkzgoyq0PVGPH80BMyemNfAkaU/P1MInjR1VEgl8IJzszcxsl2tr72D91tbtSbuH5vPOGvn6LS30kL+pHDkirW1XsvukMRw0axLVE7bXyJMaeCXTx49h4tiRSOWVwAfCyd7MzAbFtvYO1m9pzSTu1pwk3tKV3NdvbSV6SOCjR47oqmHPnlLFYXMmd0ve1eMru2rkE8cM7wQ+EE72ZmbWq9a2DtZtaenWbJ5N5tkObQ1bt/V4jLGjKrp+4547rYoj5k1Jm9PTWnimOX38aCfwQnCyNzMbZlra2nvseV6/Kf0NPJPMNzT1nMDHVVZQPSH5/Xvv6eM5aq+pOzSfd06PG+1UU2z+D5iZlYHmbe3da9yZpF2/uXvNfFNzW4/HmDB6ZNdlY/vNnMDre2g+7+yRPrayYhe/QnslnOzNzIaora1tae/zHX/zzu3Qtrml5wQ+cczIrmbyA3afyBsy14Bvb0JP5seMcgIvV072Zma7SESwpbW9W/N59q5ruTXzra3tPR5nctWortr2q2dN6rrue/r40d16o08bX8nokU7g5mRvZvaKRASbWtp2uHXq2k1pIs+phTdv6/k2qlOqKrtq2IfWpD3QO2/kkmk+nzquksqRhRzDzMqRk72ZWY6IYGNTW07z+cDvgz5t3PZa9rxpVd16nnf2Tp+eJvCRFU7gVjhO9mY2LEQEjVu39XjXtdwbuazr5T7oIwTTMh3Wcu+Dnr0XejHvg26Wy8nezEpWR9dtVDM17l7ug752cwttPdyGLZ/7oHfOl8p90M1yOdmb2ZDS3hGs39Jzsh7M+6B3/g5ejvdBN8vlZG9mBdfWeRvVfprP+7wPesWIrt+5+7oPeudAJr4Lm9l2TvZmtlOy90Gv76EDW773Qe/8ndv3QTcrHCd7M9tBa1sHdY1NvLh+KyvTx6oNzXnfB73zN27fB91saHCyNxuGIoL6zS1pIk+Sejaxr97Y3K0mXlkxgt0nj6F6/Gj2mj7O90E3KzF+V5qVqa2tbT0m8hfXb2Vlw9Ydbu4yY+JoaqZUcfRe06iZWkXN1CrmpI/dJox2JzazEuZkb1ai2juC1RuSZF6bTeoNSVJfu7m12/bjKiuomVrFntXjeMOrpncl8pqpY5k9pcr3RTcrY072ZkNURLChaVtaK9+ezGsbkr91DU3drhuvGCH2mDyGOVOreOMBM3aonU+pcg91s+GqoMle0inA14EK4DsR8cWc9XOBRcB0YD1wXkTUZtZPBJ4CfhIRlxUyVrNiaGlrp64h0xGuoYkX122voecORTp1XCU1U8Zy0KxJvPmg3bsl85mTxjDKt1w1sx4ULNlLqgBuAE4CaoGlkhZHxJOZza4FbouIWyWdAFwDnJ9Z/zngV4WK0azQIoL6TS1dyfvFdU3dmtrX5HaEGzmCmiljmTO1igXzpqTN7FXUTEma2yeMGVW8F2NmJauQNfsjgRUR8RyApDuAM4Bssp8PfDSdfhD4aecKSUcAM4BfAgsKGKfZK7K5pa1b57fahu5N7rkd4WZOHEPN1LG8du9pSTKfUsWcaUntfPp4d4Qzs8FXyGQ/C1iZma8FjsrZ5gngLJKm/rcDEyRNAxqA60hq+Sf29gSSLgEuAZgzZ86gBW6W1dbeweoNzd16sr+4vrNj3FbWbeneEW786JHUTK1i7+njOH6/6V2/nddMqWL2lLHuCGdmu1whk31P1ZPce2hdDlwv6SLg10Ad0AZ8GLgnIlb21aEoIhYCCwEWLFjQw/25zPrXORra9kS+tdv156sad+wIN2ty0tT+pgNndP1uXjMl+TvZHeHMbIgpZLKvBWoy87OBVdkNImIVcCaApPHAWRGxQdJrgWMkfRgYD1RK2hwRnyxgvFbGmre173BHuKSpvYna9VvZ1NK9I9y0cZXMnlrFITWTOf2Q3bsSec3UKnafNMZjj5tZSSlksl8K7CtpT5Ia+9nAu7MbSKoG1kdEB/Apkp75RMS5mW0uAhY40VtfOjqSO8K92C2Rb7/+fM3G5m7bjx45oqtGfuS8Kdtr5+ljvO8CZ2ZlpGCfaBHRJuky4F6SS+8WRcRySVcDyyJiMXAccI2kIGnGv7RQ8Vjp29S8jZXrm7p6smcT+8qGJlrbtneEkzo7wlXx+n2qk8vTpo3tqqFXuyOcmQ0jip6GoipBCxYsiGXLlhU7DHsFOjvCvZibyNNkvj6nI9yEtCNcksirqJkytmt+1pSxjB7pjnBmVt4kPRoR/V6x5rZKK6pt7R3c8rvnuf3hF6htaKI90xFu5AgxK73m/OQ9JnW7veucqVUes9zMLE9O9lY0v3mmnqsWL+fZ+i28bu9pnH7wHtRM3V47nznRHeHMzAaDk73tcivXb+ULP3+KXy5fw9xpVXz3wgWceMCMYodlZla2nOxtl2ne1s63f/UsNy55lhESHz95Py7+hz19kxkzswJzsreCiwjuXf4Sn//5k9Q2NPHmg3fnX047gD0mjy12aGZmw4KTvRXUipc389mfLec3z6xlvxkT+OH7j+J1e1cXOywzs2HFyd4KYlPzNr75vytY9Nu/M7aygs+cPp/zj57rDndmZkXgZG+DKiL46eN1/Ns9f6V+UwvvXDCbT5yyP9XjRxc7NDOzYcvJ3gbNX+o2cNXi5Sx7oYFDZk/i5gsWcGjN5GKHZWY27DnZ2yvWsKWVa+97mh898iJTqir50lkH8Y4janw7WjOzIcLJ3nZae0fwo0de5Nr7nmZTcxsXvHYeHz3pVUwaO6rYoZmZWYaTve2UZc+v5zOLl7N81UaO3msqV731QPafObHYYZmZWQ+c7G1AXt7YzDW/+Cs/eayO3SeN4ZvnHMZbDt7d96g3MxvCnOwtL61tHdzy+7/z9f95hm3twaXH782lx+9DVaVPITOzoc6f1NavX/+tnqt+tpzn6rdwwv67ceVb5jOvelyxwzIzszw52VuvVq7fyud//iT3Ln+JedOqWHTRAk7Y3wPWmJmVGid720HztnZuXPIs3/7V9gFr3nfMnowe6QFrzMxKkZO9dekcsOZzdz9JXWMTbzl4dz7tAWvMzEqek70BOw5Y86P3H81r955W7LDMzGwQONkPc5uat/GNB57he797nrGVFVx1+nzO84A1ZmZlxcl+mIoIfvJYHdf84q+s3dzCO4+o4eOn7OcBa8zMylBBk72kU4CvAxXAdyLiiznr5wKLgOnAeuC8iKhNl/843W8U8M2I+HYhYx1O/lK3gc8sXs6jLzRwSM1kvnPBAg7xgDVmZmWrYMleUgVwA3ASUAsslbQ4Ip7MbHYtcFtE3CrpBOAa4HxgNfC6iGiRNB74S7rvqkLFOxw0bGnlK+mANVOrKvny/zmY/3P4bA9YY2ZW5gpZsz8SWBERzwFIugM4A8gm+/nAR9PpB4GfAkREa2ab0YB/QH4F2juCHz7yItelA9Zc9Lp5/PMbPWCNmdlwUchkPwtYmZmvBY7K2eYJ4CySpv63AxMkTYuIdZJqgJ8D+wAfd61+5yx9fj2f+e/lPLk6GbDms299NfvNnFDssMzMbBcqZLLvqW04cuYvB66XdBHwa6AOaAOIiJXAwZL2AH4q6a6IeKnbE0iXAJcAzJkzZ3CjL3G5A9Zc/+7DePNBHrDGzGw4KmSyrwVqMvOzgW6187S2fiZA+tv8WRGxIXcbScuBY4C7ctYtBBYCLFiwIPeLxLDU2tbB9373d77xQDJgzWXH78OHj9/bA9aYmQ1jhcwAS4F9Je1JUmM/G3h3dgNJ1cD6iOgAPkXSMx9Js4F1EdEkaQrweuDfCxhrWfjV3+r5bDpgzYn778a/esAaMzOjgMk+ItokXQbcS3IJ3aKIWC7pamBZRCwGjgOukRQkzfiXprsfAFyXLhdwbUT8uVCxlrqV67fyubuf5L4nkwFrvnfRazh+/92KHZaZmQ0RiiiP1u8FCxbEsmXLih3GLtXU2s6Nv3qWm371LBUjxGUn7MPF/+ABa8zMhgtJj0bEgv628w+5JSgZsGYNn7v7Keoamzj9kD349Gn7s/skD1hjZmY7crIvMa1tHfzzfzzGPX9ew/4zJ3DHJUdz9F4esMbMzHrnZF9CWtra+fDtf+SBv77MJ07Zj0uO2csD1piZWb+c7EtE87Z2Pnj7oyx5up4vvP3VnHvU3GKHZGZmJcLJvgQ0b2vn/bct47cr1vLFMw/i7CN9AyEzM8ufk/0Q19TazvtuW8rvn13Hl846mHcuqOl/JzMzswwn+yFsa2sb771lKY/8fT3XveMQzjx8drFDMjOzEuRkP0Rtbmnjvd9byrIX1vPVdx3KGYfOKnZIZmZWopzsh6BNzdt4z/eW8tjKRr5+9mGcfsgexQ7JzMxKmJP9ELOxeRsXLnqEP9du4JvnHMZpB+1e7JDMzKzEOdkPIRuatnHBokdYXreB6999OKe8emaxQzIzszLgZD9ENG5t5fzvPsJf12zkxvOO4KT5M4odkpmZlQkn+yGgYUsr5333YZ55aTM3nX8EJ+zvRG9mZoPHyb7I1m1u4dzvPMxza7ew8IIjOG4/D01rZmaDy8m+iNZubuHcmx/m+XVb+O6FCzhm3+nFDsnMzMqQk32RvLypmXNvfpiVDVv53kWv4XX7VBc7JDMzK1NO9kXw0sZmzrn5IVY3NnPLe470ELVmZlZQ/Y6PKukySVN2RTDDwZoNzZy98CHWbGjm1vc60ZuZWeHlMxj6TGCppDslnSJJhQ6qXK1qbOJdC/9A/aYWbnvvkRy559Rih2RmZsNAv8k+Iq4A9gW+C1wEPCPp3yTtXeDYykptw1betfAPrN/cym0XH8mCeU70Zma2a+RTsyciAliTPtqAKcBdkr5cwNjKxsr1W3nXTQ+xYes2bn/fURw+x7+KmJnZrtNvBz1JHwEuBNYC3wE+HhHbJI0AngE+UdgQS9sL67ZwzsKH2NLazg/edzQHzZ5U7JDMzGyYyadmXw2cGREnR8R/RsQ2gIjoAN7S147pb/xPS1oh6ZM9rJ8r6QFJf5K0RNLsdPmhkv4gaXm67l078dqK7u9rt/Cumx5i67Z2fvC+o5zozcysKPJJ9vcA6ztnJE2QdBRARDzV206SKoAbgFOB+cA5kubnbHYtcFtEHAxcDVyTLt8KXBARBwKnAF+TNDm/lzQ0PFu/mbMX/oHW9g5++L6jefUsJ3ozMyuOfJL9jcDmzPyWdFl/jgRWRMRzEdEK3AGckbPNfOCBdPrBzvUR8beIeCadXgW8DJTM7eVWvLyJsxc+RFt78KP3H838PSYWOyQzMxvG8kn2SjvoAV3N9/ncjGcWsDIzX5suy3oCOCudfjswQVK3C88lHQlUAs/uEJh0iaRlkpbV19fnEVLh/e2lJNFHwB2XHM1+MycUOyQzMxvm8kn2z0n6iKRR6eOfgOfy2K+n6/EjZ/5y4FhJjwHHAnUkvf2TA0i7A98H3pN+yeh+sIiFEbEgIhZMn178iv9Tqzdy9sKHGCFxxyVHs+8MJ3ozMyu+fJL9B4HXkSTiWuAo4JI89qsFajLzs4FV2Q0iYlVEnBkRhwH/ki7bACBpIvBz4IqIeCiP5yu6D93+KKMqkkS/z27jix2OmZkZkEdzfES8DJy9E8deCuwraU+SLwpnA+/ObiCpGlif1to/BSxKl1cCPyHpvPefO/Hcu1xTazvPr9vKx056FXtNd6I3M7OhI5/r7McAFwMHAmM6l0fEe/vaLyLaJF0G3AtUAIsiYrmkq4FlEbEYOA64RlIAvwYuTXd/J/AGYJqki9JlF0XE4wN4bbtUXWMTALOmjC1yJGZmZt3l09Hu+8BfgZNJLo87F+j1krusiLiH5NK97LIrM9N3AXf1sN/7+NN0AAARn0lEQVTtwO35PMdQsaoz2U92sjczs6Eln9/s94mIfwW2RMStwJuBgwobVulxzd7MzIaqfJL9tvRvo6RXA5OAeQWLqETVNTRRMULMnDim/43NzMx2oXya8Rem49lfASwGxgP/WtCoSlBdYxMzJ45hZEVeYwuZmZntMn0m+3Swm40R0UDSgW6vXRJVCapraPLv9WZmNiT1WQ1NL4m7bBfFUtLqGpv8e72ZmQ1J+bQ53y/pckk1kqZ2PgoeWQlpa+9gzcZm1+zNzGxIyuc3+87r6S/NLAvcpN9lzcZm2jvCNXszMxuS8rmD3p67IpBSVtfga+zNzGzoyucOehf0tDwibhv8cEqTr7E3M7OhLJ9m/NdkpscAJwJ/BJzsU501+z0mOdmbmdnQk08z/j9m5yVNIrmFrqXqGpuYNq6SsZUVxQ7FzMxsBztzB5itwL6DHUgp82V3ZmY2lOXzm/3PSHrfQ/LlYD5wZyGDKjV1DU3sN3NCscMwMzPrUT6/2V+bmW4DXoiI2gLFU3IigrrGJk7Yf7dih2JmZtajfJL9i8DqiGgGkDRW0ryIeL6gkZWItZtbaWnrcDO+mZkNWfn8Zv+fQEdmvj1dZmQuu/M19mZmNkTlk+xHRkRr50w6XVm4kEpL1w11XLM3M7MhKp9kXy/prZ0zks4A1hYupNJS17gVgNmTq4ociZmZWc/y+c3+g8APJF2fztcCPd5Vbzha1djM+NEjmTg2n6I0MzPb9fK5qc6zwNGSxgOKiE2FD6t01Kbj2EsqdihmZmY96rcZX9K/SZocEZsjYpOkKZI+vyuCKwW+oY6ZmQ11+fxmf2pENHbOREQDcFrhQiotdQ1b3RPfzMyGtHySfYWk0Z0zksYCo/vYvoukUyQ9LWmFpE/2sH6upAck/UnSEkmzM+t+KalR0t35PFcxbGrexsbmNtfszcxsSMsn2d8OPCDpYkkXA/cDt/a3k6QK4AbgVJJb7J4jaX7OZtcCt0XEwcDVwDWZdV8Bzs8jvqLxNfZmZlYK+k32EfFl4PPAASRJ+5fA3DyOfSSwIiKeS6/NvwM4I2eb+cAD6fSD2fUR8QAwpDsD+hp7MzMrBfmOereG5C56Z5GMZ/9UHvvMAlZm5mvTZVlPpMcEeDswQdK0PGNC0iWSlklaVl9fn+9ug6azZj/bNXszMxvCek32kl4l6UpJTwHXkyRuRcTxEXF9b/tlD9HDssiZvxw4VtJjwLFAHclgO3mJiIURsSAiFkyfPj3f3QZNXUMTlRUjqB6fVxcGMzOzoujrOvu/Ar8BTo+IFQCSPjqAY9cCNZn52cCq7AYRsQo4Mz32eOCsiNgwgOcoqtrGJvaYPIYRI3yNvZmZDV19NeOfRdJ8/6CkmyWdSM+19d4sBfaVtKekSuBsYHF2A0nVkjpj+BSwaADHL7q6Bl9jb2ZmQ1+vyT4ifhIR7wL2B5YAHwVmSLpR0pv6O3BEtAGXAfeS/MZ/Z0Qsl3R15l77xwFPS/obMAP4Quf+kn5DMrreiZJqJZ28My+wkOoam9wT38zMhrx8bpe7BfgByf3xpwLvAD4J3JfHvvcA9+QsuzIzfRdwVy/7HtPf8YupeVs79ZtamOUBcMzMbIjLtzc+ABGxPiJuiogTChVQqVi9oRnwZXdmZjb0DSjZ23ad19jvMXlMkSMxMzPrm5P9TvI49mZmViqc7HdSXUMTEsyc5Jq9mZkNbU72O6m2sYkZE8ZQOdJFaGZmQ5sz1U5a5XHszcysRDjZ7yRfY29mZqXCyX4ntHcEqxubXbM3M7OS4GS/E17e1ExbR7hmb2ZmJcHJfid4HHszMyslTvY7wePYm5lZKXGy3wm1rtmbmVkJcbLfCXWNTUypGkVVZb/jCJmZmRWdk/1O8Dj2ZmZWSpzsd4KvsTczs1LiZD9AEZHU7D0AjpmZlQgn+wFq2LqNpm3tbsY3M7OS4WQ/QF3X2LsZ38zMSoST/QB1jWPvmr2ZmZUIJ/sBqnXN3szMSoyT/QDVNTZRVVnB5KpRxQ7FzMwsL072A5T0xB+LpGKHYmZmlpeCJntJp0h6WtIKSZ/sYf1cSQ9I+pOkJZJmZ9ZdKOmZ9HFhIeMciLpG31DHzMxKS8GSvaQK4AbgVGA+cI6k+TmbXQvcFhEHA1cD16T7TgU+AxwFHAl8RtKUQsU6EL6hjpmZlZpC1uyPBFZExHMR0QrcAZyRs8184IF0+sHM+pOB+yNifUQ0APcDpxQw1rxsaWmjces29nCyNzOzElLIZD8LWJmZr02XZT0BnJVOvx2YIGlanvsi6RJJyyQtq6+vH7TAe7Oqc2hbN+ObmVkJKWSy76kHW+TMXw4cK+kx4FigDmjLc18iYmFELIiIBdOnT3+l8farttGX3ZmZWekp5BittUBNZn42sCq7QUSsAs4EkDQeOCsiNkiqBY7L2XdJAWPNS53HsTczsxJUyJr9UmBfSXtKqgTOBhZnN5BULakzhk8Bi9Lpe4E3SZqSdsx7U7qsqOoamxg5Quw2YUyxQzEzM8tbwZJ9RLQBl5Ek6aeAOyNiuaSrJb013ew44GlJfwNmAF9I910PfI7kC8NS4Op0WVHVNTSx++QxVIzwNfZmZlY6CtmMT0TcA9yTs+zKzPRdwF297LuI7TX9IcGX3ZmZWSnyHfQGwOPYm5lZKXKyz1NrWwcvbWp25zwzMys5TvZ5WrOhmQiY7WZ8MzMrMU72eapNx7F3zd7MzEqNk32e6jyOvZmZlSgn+zzVpXfP232yr7E3M7PS4mSfp7qGJnabMJrRIyuKHYqZmdmAONnnyePYm5lZqXKyz5NvqGNmZqXKyT4PHR3B6kZfY29mZqXJyT4P9ZtbaG3v8DX2ZmZWkpzs81DroW3NzKyEOdnnYVVj5zX2vi++mZmVHif7PHReY++avZmZlSIn+zzUNTQxaewoxo8u6IjAZmZmBeFknwdfdmdmZqXMyT4PdQ1N7OFkb2ZmJcrJvh8RQV1jE7P9e72ZmZUoJ/t+bGxqY3NLm5vxzcysZDnZ98Pj2JuZWalzsu+Hx7E3M7NS52TfD19jb2Zmpa6gyV7SKZKelrRC0id7WD9H0oOSHpP0J0mnpcsrJX1P0p8lPSHpuELG2Ze6hibGjBrBtHGVxQrBzMzsFSlYspdUAdwAnArMB86RND9nsyuAOyPiMOBs4Fvp8vcDRMRBwEnAdZKK0gpR15hcdiepGE9vZmb2ihUygR4JrIiI5yKiFbgDOCNnmwAmptOTgFXp9HzgAYCIeBloBBYUMNZe+YY6ZmZW6gqZ7GcBKzPztemyrKuA8yTVAvcA/5gufwI4Q9JISXsCRwA1uU8g6RJJyyQtq6+vH+z4gaQZ39fYm5lZKStksu+p3Tty5s8BbomI2cBpwPfT5vpFJF8OlgFfA34PtO1wsIiFEbEgIhZMnz59UIMHaGptZ92WVtfszcyspBVyZJdautfGZ7O9mb7TxcApABHxB0ljgOq06f6jnRtJ+j3wTAFj7ZF74puZWTkoZM1+KbCvpD0lVZJ0wFucs82LwIkAkg4AxgD1kqokjUuXnwS0RcSTBYy1R3Uex97MzMpAwWr2EdEm6TLgXqACWBQRyyVdDSyLiMXAx4CbJX2UpIn/oogISbsB90rqAOqA8wsVZ1+6bqjjmr2ZmZWwgg7QHhH3kHS8yy67MjP9JPD6HvZ7HtivkLHlY1VjExUjxIwJo4sdipmZ2U7zHfT6UNfYxMyJYxhZ4WIyM7PS5SzWh7qGJjfhm5lZyXOy70NdYxOzfdmdmZmVOCf7XrS1d7BmY7Nr9mZmVvKc7HuxZmMz7R3hG+qYmVnJc7LvhS+7MzOzcuFk34vtN9Rxsjczs9LmZN+Lzpr9Hk72ZmZW4pzse1HX2ET1+ErGjKoodihmZmaviJN9LzyOvZmZlQsn+174hjpmZlYunOx7EBGu2ZuZWdlwsu9BS1sHp756JkfMnVLsUMzMzF6xgo56V6rGjKrga2cfVuwwzMzMBoVr9mZmZmXOyd7MzKzMOdmbmZmVOSd7MzOzMudkb2ZmVuac7M3MzMqck72ZmVmZc7I3MzMrc4qIYscwKCTVAy/0sUk1sHYXhVOuXIaDw+U4OFyOg8PlODiKVY5zI2J6fxuVTbLvj6RlEbGg2HGUMpfh4HA5Dg6X4+BwOQ6OoV6ObsY3MzMrc072ZmZmZW44JfuFxQ6gDLgMB4fLcXC4HAeHy3FwDOlyHDa/2ZuZmQ1Xw6lmb2ZmNiw52ZuZmZW5sk/2kk6R9LSkFZI+Wex4Somk5yX9WdLjkpaly6ZKul/SM+nfKcWOc6iRtEjSy5L+klnWY7kp8Y30/PyTpMOLF/nQ0ks5XiWpLj0nH5d0Wmbdp9JyfFrSycWJemiRVCPpQUlPSVou6Z/S5T4fB6CPciyZ87Gsk72kCuAG4FRgPnCOpPnFjarkHB8Rh2auH/0k8EBE7As8kM5bd7cAp+Qs663cTgX2TR+XADfuohhLwS3sWI4AX03PyUMj4h6A9H19NnBgus+30vf/cNcGfCwiDgCOBi5Ny8rn48D0Vo5QIudjWSd74EhgRUQ8FxGtwB3AGUWOqdSdAdyaTt8KvK2IsQxJEfFrYH3O4t7K7Qzgtkg8BEyWtPuuiXRo66Uce3MGcEdEtETE34EVJO//YS0iVkfEH9PpTcBTwCx8Pg5IH+XYmyF3PpZ7sp8FrMzM19L3P8i6C+A+SY9KuiRdNiMiVkPyBgB2K1p0paW3cvM5OnCXpU3MizI/I7kc+yFpHnAY8DA+H3daTjlCiZyP5Z7s1cMyX2uYv9dHxOEkTXuXSnpDsQMqQz5HB+ZGYG/gUGA1cF263OXYB0njgf8C/jkiNva1aQ/LXI6pHsqxZM7Hck/2tUBNZn42sKpIsZSciFiV/n0Z+AlJM9RLnc166d+XixdhSemt3HyODkBEvBQR7RHRAdzM9qZRl2MvJI0iSVA/iIgfp4t9Pg5QT+VYSudjuSf7pcC+kvaUVEnSYWJxkWMqCZLGSZrQOQ28CfgLSfldmG52IfDfxYmw5PRWbouBC9Je0EcDGzqbV21HOb8fv53knISkHM+WNFrSniQdzB7Z1fENNZIEfBd4KiL+PbPK5+MA9FaOpXQ+jizmkxdaRLRJugy4F6gAFkXE8iKHVSpmAD9JznFGAj+MiF9KWgrcKeli4EXgHUWMcUiS9CPgOKBaUi3wGeCL9Fxu9wCnkXTg2Qq8Z5cHPET1Uo7HSTqUpEn0eeADABGxXNKdwJMkPacvjYj2YsQ9xLweOB/4s6TH02WfxufjQPVWjueUyvno2+WamZmVuXJvxjczMxv2nOzNzMzKnJO9mZlZmXOyNzMzK3NO9mZmZmXOyd5sGJIUkq7LzF8u6aoihmRmBeRkbzY8tQBnSqoezIOmN2Px54rZEOM3pdnw1AYsBD6au0LSdEn/JWlp+nh9uvwqSZdntvuLpHnp4ylJ3wL+CNRIOkfSn9NtvpTZZ7OkL0h6QtJDkmaky9+RbvuEpF8X+sWbDTdO9mbD1w3AuZIm5Sz/OskY3a8BzgK+k8ex9iMZGvUwYBvwJeAEkgFCXiOpcwjVccBDEXEI8Gvg/enyK4GT0+VvfQWvycx6UNa3yzWz3kXERkm3AR8BmjKr3gjMT2+VDDCxc5yEPryQjn8O8BpgSUTUA0j6AfAG4KdAK3B3ut2jwEnp9O+AW9JbjP4YMxtUTvZmw9vXSJrev5dZNgJ4bURkvwAgqY3urYFjMtNbspv28XzbYvs9uttJP4Mi4oOSjgLeDDwu6dCIWDegV2JmvXIzvtkwFhHrgTuBizOL7wMu65xJB/qAZKCPw9NlhwN79nLYh4FjJVVLqgDOAX7VVxyS9o6IhyPiSmAt3YcHNbNXyMnezK4Dsr3yPwIskPQnSU8CH0yX/xcwNR3160PA33o6WDok6qeAB4EngD9GRH9DIX+ls0MfyW/5T+z0qzGzHXjUOzMzszLnmr2ZmVmZc7I3MzMrc072ZmZmZc7J3szMrMw52ZuZmZU5J3szM7My52RvZmZW5v4/7yTqm4kecqwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "## FUNCTION PIPELINE\n",
    "\n",
    "# Neural network training function\n",
    "def train_nn(neural_arch, train_features, train_labels):\n",
    "    nn = MLPClassifier(hidden_layer_sizes = neural_arch, max_iter = 10000)\n",
    "    nn.fit(train_features, train_labels)\n",
    "    return nn\n",
    "\n",
    "# Testing function\n",
    "def test_nn(model, test_features, test_labels):\n",
    "    predictions = model.predict(test_features)\n",
    "    train_test_df = pd.DataFrame()\n",
    "    train_test_df[\"correct_labels\"] = test_labels\n",
    "    train_test_df[\"predictions\"] = predictions\n",
    "    accuracy = sum(train_test_df[\"correct_labels\"] == train_test_df[\"predictions\"])/len(train_test_df)\n",
    "    \n",
    "    return accuracy\n",
    "\n",
    "# Cross-validation function\n",
    "def cross_validate_nn(neural_arch):\n",
    "    fold_accuracies = []\n",
    "    kf = KFold(n_splits = 4, random_state = 2)\n",
    "    for train_index, test_index in kf.split(data):\n",
    "        train_features, test_features = data.loc[train_index], data.loc[test_index]\n",
    "        train_labels, test_labels = labels.loc[train_index], labels.loc[test_index]\n",
    "        \n",
    "        model = train_nn(neural_arch, train_features, train_labels)\n",
    "        accuracy = test_nn(model, test_features, test_labels)\n",
    "        fold_accuracies.append(accuracy)\n",
    "        \n",
    "    return fold_accuracies\n",
    "\n",
    "# Define list of neural parameters\n",
    "neural_layers = [\n",
    "    (8,),\n",
    "    (16,),\n",
    "    (32,),\n",
    "    (64,),\n",
    "    (128,),\n",
    "    (256,)\n",
    "]\n",
    "\n",
    "neural_accuracies = []\n",
    "\n",
    "# Iterate through neural parameters, resulting in list of comparative accuracies\n",
    "for n in neural_layers:\n",
    "    accuracy = cross_validate_nn(n)\n",
    "    mean_accuracy = np.mean(accuracy)\n",
    "    neural_accuracies.append(mean_accuracy)\n",
    "\n",
    "x = [i[0] for i in neural_layers]\n",
    "\n",
    "plt.figure(figsize = (8,4))\n",
    "plt.plot(x, neural_accuracies)\n",
    "plt.title(\"Mean Accuracies vs. Number of Single Layer Neurons\")\n",
    "plt.xlabel(\"Neurons\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.8842514229151199,\n",
       " 0.9193046275674339,\n",
       " 0.9276540460282108,\n",
       " 0.9376676565206633,\n",
       " 0.9421244741400645,\n",
       " 0.9482417718386538]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neural_accuracies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plot, we see that a greater number of neurons generally leads to a more accurate model: the iteration with 256 neurons achieved close to 95% accuracy, up from the ~88% accuracy achieved by the model with only 8 neurons. Simple accuracy computes the number of correct classifications the model made, but doesn't tell us anything about false or true positives or false or true negatives, meaning these better-fitting models could actually be overfitting.\n",
    "\n",
    "Given that the k-nearest neighbors model achieved over 96% accuracy, there seem to be no advantages to using a single layer neural network instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Double Layer Network\n",
    "\n",
    "We can alter the model so it takes two identical hidden layers of 64, 128 and 256 neurons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgEAAAEWCAYAAAD/3UTfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xe8HGXZ//HPl3RIIwUUUoHQLCR4BJTei0rLT0lAlEcRFQI2UBBBxAdBBUUFC4+ioPQeEAFFAjaQE9JIQkIIhJwUCC0JIZCc5Pr9MfeSyea0lD179uz3/Xrt68zcU/a+Z3fOXHPPtTOKCMzMzKz6bFbuCpiZmVl5OAgwMzOrUg4CzMzMqpSDADMzsyrlIMDMzKxKOQgwMzOrUg4CzDYBSSdJeqjc9WiLJIWkHcr03jtJmiBpqaSzylEHs7bMQUCVkvSCpBWS+hWVT0z/tIeUqV5DJa2W9MtyvP+GiogbIuKwctejOZIOSJ/v1UXl/5R0SpmqVUrfBMZFRI+I+Hl+gqSpkt5Mr1WS3s6Nf7tUFZK0UNI+pVr/hpD0pfS9OLOo/BVJe5WrXlZ6DgKq2/PA6MKIpA8A3cpXHQA+A7wOjJLUpTXfWFLH1ny/MloGfKZcgd6G2sDPZzAwtaEJEfG+iOgeEd2BfwBjCuMR8YONqWtb1sR2fA04X9LmZayDtTIHAdXtj2QH3YLPAtfnZ5DURdLlkl6U9JKkX0vqlqZtKek+SYskvZ6GB+SWHSfp+5L+lbpjHyrueWjAZ4DvACuBTxTV5X2S/irptVSXb6fyDpK+Lem59D7jJQ2UNCSd3XTMrWOcpFPT8Cmpbj+V9BpwkaTtJf1d0qvpLOgGSb1zyw+UdGdq86uSrsqt65+5+XbO1XWGpE/lph0laVqq6zxJZxdvhLTd35D0/lxZf0nLJW0lqV/a3m+k9/iHpJbuz28AfwC+29BESRdJ+lNufK3tmLbh/0r6dzprvldS37Stlkh6soEA4yhJs9M2/XG+rpI+J2l6+g49KGlwblpIOkPSs8CzjdT36HRW/0aq2y6p/O/AgcBVqZ47tnD7FNa7UNL70vCpqS7bpfExkm5Ow90kXS1pgaS61L5O6/NeaT39Jf0lfbdek3SPpPemaSdL+lfR/OcX1eFKSXNTvX+hFERLOkLSLEkXSHoJ+FUjVZgATAHObGhi2s8uyH2O7+4bhfdoYPvtk4Yvk3SjpFskLSUL8hvdbrk6fzttj3mSTsqt+xhJz6R9aK58qWeDOQiobo8DPSXtIqkDcALwp6J5fgjsCAwHdgC2BS5M0zYDfk92tjUIWA5cVbT8icD/AFsBnYF1DngFkvYFBgA3A7eSC1Ak9QD+BjwAbJPq8nCa/HWyHo2jgJ7A54C3WtB+gD2B2al+lwACLk3vsQswELgo1aEDcB8wBxhCti1ubqAdWwB/BW5M6x0N/LJwQAF+B3wxInoA7wf+XryOiHgHuJNcTw3wKeDRiHgZ+AZQB/QHtga+DazPPcAvAUZK2mk9lskbBZxMtg22B/5D9l3oA0xn3QDjOKAG2B04huwzQtKxqe7Hp7b8A7ipaNljyT6nXYsrkQ7sNwFfTcvfD9wrqXNEHMTaZ/gz17ONjwEHpOH9yL4n++fGH03D3wM+CHwA+FBa5pvr+V6Q7U+/JtuXhqayn6a/dwIfKAQhyUlkgXxhvgGpDjuR7bPn5uYdAnQi+z43dcD8DvBNST0bmHYOcBiwT3qvlbn6tcRI4DqgF3AHzW+3wWT74zbAGODXkrqnadcCn0n70HCyz9k2RET4VYUv4AXgELKd/lLgCLIDV0eyg8kQsh1wGbB9brmPAM83ss7hwOu58XHAd3LjpwMPNFGn3wJ3595nJbBVGh8NTGhkuRnAMQ2UD0lt6VhUp1PT8CnAi81sp2ML75vqtCi/vtx8pwD/TMMnAP8omv4b4Ltp+EXgi0DPZt77EGB2bvxfZP/4AC4G7gF2WM/P/QCgLg3/CLglDf8TOCUNXwT8qbHtmLbh+bnpVwB/yY1/ApiYGw/giKLvwcNp+C/A53PTNiML4Abnlj2oifZcANxatPw84IDiz7uZ7bLOfMAZhXWTBQBfAP6QxhcCu6bhefk6kgU5zzTxXguBfVpQp72ABbnx3wMXpOEa4GWy/bUjsALYNjfvgcD0NHwE2X7cqYn3+hLwtzQ8FvheGn4F2CsNPw/snVtmaPqslN5jVmPtBC4DHiqa3uh2S+tbDGyWm74EGJ6GXyI7ueixPt9/v9Z9uSfA/kh2tn4KRZcCyM6sNgfGp67WN8jOxPsDSNpc0m8kzZG0hOzMqXc6Yy5YmBt+C+hOA5RdYvgkcANARPyH7GB5YpplIPBcI21oalpz5hbVYytJN6fuxyVkPSOFSxgDgTkRUd/MOgcDexa2WdpuJwHvSdNHkvVazJH0qKSPNLKevwPdJO2ZusiHA3elaT8GZgEPpe7ZcxtZR1N+CBwuabcNWPal3PDyBsaLP+f8dp5DdnYH2bb6WW47vUZ2UNm2kWWLbZPWB0BErE7zb9voEi33KLC/pEFkB9E7gf0k7UwWbEyXJLLPdU5uuTmF91d2aamQbDiyqTeT1EPStcouvS0BHmLNdw+ys+hCl/ingZvSd3EbsrP8qbnteDdZL1TBwohY2cJ2XwB8RVLfXN1E9v2/P/ceE9J26Nvwatbx7ufY3HZLFqXPsyD//+NYsv3oxbSNa1pYByviIKDKRcQcsgj/KLJ/cnmvkP1Df19E9E6vXpElUkHWJb0TsGdE9CTrIoXsn/j6Oo6sK/+X6VriQrJ/CIVLAnPJup0b0ti0ZelvPtHpPUXzFHehX5rKPpja9GnWtGcuMEjNJzXNJeu27517dY+ILwNExJMRcQzZP+m7yS59rCP9A7yVrBfkROC+iFiapi2NiG9ExHZkZ95fl3RwM/UqXv+rwJXA94smLaPpbbYhBuaGBwHz0/Bcsksj+W3VLSL+na9qE+udTxZIAGsdrOZtgjpPBTqQnSU/mrbXm2TfycciIQt0B+eWG1R4/4g4KNYkG97RzPudS9bN/uH03TuMtfelR4GuyrL1R7HmUsACoJ6sxy6/n+YPzi2+VBQRk8iC/W/lyoI1Z+75z6prRLxC0XcmXdvvU7zqovU1ut1aUMf/RMTHyS6FPcS6l5CshRwEGMDnyXbuZfnCdBD6P+CnkrYCkLStpMPTLD3IgoQ3JPWhkUSzFvos2XW+D5Cd8Q4H9gaGK/vVwn3AeyR9VVnSXA9Je6Zlfwt8X9IwZT4oqW9ELCL7p/LplNT0ORoPJAp6kP2jf0PStmTXQQv+S/YP9zJJW0jqKmnvBtZxH7CjsmSuTun14ZR70VnZPQV6pTOzJcCqJupzI9nlhZPSMACSPi5ph3TQK6yjqfU05ifAR8nyHwomkp3xDpLUCzhvA9Zb7BxliaQDga8At6TyXwPnaU0CXi9Jn1yP9d4KfEzSwenA8w3gHeDfTS/WvHSgeozsenTh+v+jReOQHYC+qyw5civgfNbNrSnWOX1/Cq8OZN+9t8i+e/3ILtUV1+ePwDXAaxFRm8pXku07P1OWMCplCayHbnjr+S5Z8JMPBn9N9t0fCO/2mhWSd6cDfXKfw/do/viyIduNtO+NSnkLK4GlbNh333AQYEBEPFf4h9KAb5F1Oz+euij/Rnb2D9lZZDeyHoPHyc4e1ls62B4MXBkRC3Ov8Wmdn01nwIeSnfUuJMsUPzCt4idkB4OHyA6Iv2PNTx2/QHYgfxV4H80fHL5Hlry2GPgzud6RiFiV3n8HsksVdWQH6LWkuh5GdrY2P9X3h0DhJ48nAy+k7fklst6GBkXEE2RnWduQXT8vGEb2WbxJlpT3y4gYB6Asw7xFv3OPiCVkuQF9cmV/JTtITwbGkwU1G+uetK6JZNv1d+m97iLbNjen7fE0cGRLVxoRM8i23y/IvoefAD4RESs2QZ0hO9j3IAsGGhqHLFF2GlnPwUSy3I0fNbPeh8kC6MLrPOBysu7/V8lyNO5vYLnryALlPxaVf5Xsu1ZL9t19gOx7ukHSdr2dtX8y/COy79zflWX4/5tsXyH1BnyF7HJeHdl3/pVm3mZDtlvB58guHywm65n5bAuXsyLKgkszM2vrlP1K5iVg54h4sdz1scrnngAzs8pxJtkdEB0A2CbhuzaZmVWAlCz7FnB0ueti7YcvB5iZmVUpXw4wMzOrUlVxOaBfv34xZMiQclfDzMysVYwfP/6ViOjf3HxVEQQMGTKE2trGfgFnZmbWvkia0/xcvhxgZmZWtRwEmJmZVSkHAWZmZlXKQYCZmVmVchBgZmZWpRwEmJmZVSkHAWZmZlWqKu4TYGZmVi4RwdsrV7Pk7ZUsWb6SxctXpuH6d8sG9tmcY4Zv2+p1cxBgZmbWjHfqV2UH79yBe8nb9Y0e1AvTlqRpK1c1/Zyew9+3tYMAMzOzUli5avXaB+e3VzZwUM/G1xzU18z/Tv3qJtffucNm9OzWiZ7dOtKrWyd6d+vEoD6b07Nrx6y8ayd6pek9u3ZKZWumde5YnqvzDgLMzKzNW7U6WNrgQXrNgXzx8pXrHOgL8y9fuarJ9XfcTOkgvebgvE2vbtlBOx2oiw/c+YN6104dWmlLbFoOAszMrORWrw6WvlO/zoF7TXd6w9MKB/U336lvcv2bidzBOjswb9+/+1rj+TP1NQf1rKxbpw5IaqWt0XY4CDAzs2ZFBMtWrFq327zRg/raZUvfqSeavixOj64d1zrDHtRn87UO1L0aOiPvls2/RefqPIhvLAcBZmZVIJ+h3uTBuzBe1M2+9O16Vq1u+ijevUvHtbrLt+ndlZ279Wi2K71nt05079KRDpv5IN7aHASYmVWIt1euajALvbGDevH18eYy1Lt16vDugblXt07079GF7ftv0cTZ+JoDeY+uHenYwbeeqTQOAszMWklxhnrDPy1byeLl+evj65Gh3nGzdJDOzrh7b96ZQX23oGfXjrmkt7XPwAvz9yhjhrqVj4MAM7MWql+1mjffqV+vn5blf4q2QRnqvbutm9zWyEG9UjPUrXwcBJhZ1WgsQ72xn5YVn42vT4Z64Zr39j3WzlDvtfm6Z+OF+bt22szJbdaqShoESDoC+BnQAfhtRFxWNH0wcC3QH3gN+HRE1KVpq4ApadYXI+LoVH4w8GOy5x68CZwSEbNK2Q4zaxuKM9QXv9XwT8sau0beXIa6BD26dFzrDLuQod6rga70nl075g7qzlC3ylOyIEBSB+Bq4FCgDnhS0tiImJab7XLg+oi4TtJBwKXAyWna8ogY3sCqfwUcExHTJZ0OfAc4pVTtMLNNp5Ch3ujNXt4qOnAXdbNvWIZ6twYz1Bs6qPfo0pHNnKFuVaSUPQF7ALMiYjaApJuBY4B8ELAr8LU0/AhwdwvWG0DPNNwLmL9JamtmLdJQhnpLbvayvhnqhYN0cYb6urdeXZO13r2LM9TN1kcpg4Btgbm58Tpgz6J5JgEjyS4ZHAf0kNQ3Il4FukqqBeqByyKiECCcCtwvaTmwBNiroTeXdBpwGsCgQYM2TYvM2oEV9auz26828dOydQ/qa+Zf0UyGepeOm611xl3IUO/VwIF7zUHdGepm5VDKIKChPrXiU4CzgasknQI8BswjO+gDDIqI+ZK2A/4uaUpEPEfWc3BURDwh6RzgJ2SBwdpvFHENcA1ATU1NM/epMqsc9atWs/Tt+nXPuBt8IMq6P0Vbrwz1dHBuKEM9/1M0Z6ibVaZSBgF1wMDc+ACKuu4jYj5wPICk7sDIiFicm0ZEzJY0DhghaQmwW0Q8kVZxC/BACdtgtsk1lKHe2E/LljRwUG8uQ73DZlrn4LxDylDPktjWPXDnf27mDHWz6lHKIOBJYJikoWRn+KOAE/MzSOoHvBYRq4HzyH4pgKQtgbci4p00z97Aj4DXgV6SdoyImWRJh9NL2AazFlv2Tj13T5zHK0tXNPmUszc3IEN9cN/Nm+5Kz52hO0PdzFqqZEFARNRLGgM8SPYTwWsjYqqki4HaiBgLHABcKinILgeckRbfBfiNpNVkPwW8rPCrAklfAO5I014HPleqNpi1VETw9Vsn8uDUl4CiDPVu62aoN9aV3mvzTnTv7Ax1M2sdiuYe69QO1NTURG1tbbmrYe3YH/71PBfdO43zjtyZz+8z1BnqZlZWksZHRE1z8/mOgWYbaXLdG1xy/3QO2WUrTttvO3fFm1nF8OmK2UZYvHwlZ9z4FFv16Mrln9zNAYCZVRT3BJhtoIjgW7dPZsEbb3PLFz9C7807l7tKZmbrxT0BZhvoun+/wANTF/KtI3bmQ4O3LHd1zMzWm4MAsw0wue4NfnD/Mxyyy1acuu/QclfHzGyDOAgwW0+FPIB+3Ts7D8DMKppzAszWQ0Rw7h3OAzCz9sE9AWbr4fr/zOEvTy/km0fs5DwAM6t4DgLMWmhK3WIu+fN0Dt55K07dZ7tyV8fMbKM5CDBrgSVvr50H4Nv6mll74JwAs2YU8gDmvbGcW7+4F1tu4TwAM2sf3BNg1ow/Pj6H+6cs5JuH78SHBvcpd3XMzDYZBwFmTXh63mL+977pHLTzVnxhX+cBmFn74iDArBGFPIC+3TtzhfMAzKwdck6AWQMigvPumELd68u55TTnAZhZ++SeALMG/OnxOfx5ygLOOXwnaoY4D8DM2icHAWZFnp63mO/fN50Dd+rPac4DMLN2zEGAWc7SfB7Ap4Y7D8DM2jXnBJglEcG5d67JA+jjPAAza+fcE2CW/OmJF/nz5AWcfZjzAMysOjgIMCPlAdw7jQN26s8X93MegJlVBwcBVvWWvr2SMTc+RZ8tOvMT5wGYWRUpaRAg6QhJMyTNknRuA9MHS3pY0mRJ4yQNyE1bJWlieo3NlUvSJZJmSpou6axStsHat4jgvDunMPf15fzixBHOAzCzqlKyxEBJHYCrgUOBOuBJSWMjYlputsuB6yPiOkkHAZcCJ6dpyyNieAOrPgUYCOwcEaslbVWqNlj7d8MTL3Lf5AV884id+LDzAMysypSyJ2APYFZEzI6IFcDNwDFF8+wKPJyGH2lgekO+DFwcEasBIuLlTVRfqzJT5y/m4vumsf+O/fnSftuXuzpmZq2ulEHAtsDc3HhdKsubBIxMw8cBPST1TeNdJdVKelzSsblltgdOSNP+ImlYQ28u6bQ0T+2iRYs2vjXWrmR5ABPos3lnfvIpPxfAzKpTKYOAhv6rRtH42cD+kiYA+wPzgPo0bVBE1AAnAldKKpyqdQHeTtP+D7i2oTePiGsioiYiavr377+RTbH2pJAH8OJrb/Hz0SPo271LuatkZlYWpQwC6siu3RcMAObnZ4iI+RFxfESMAM5PZYsL09Lf2cA4YERuvXek4buAD5ao/tZO3fjfLA/g64fuyB5DnQdgZtWrlEHAk8AwSUMldQZGAWPzM0jqJ6lQh/NIZ/WStpTUpTAPsDdQSCi8GzgoDe8PzCxhG6ydmTp/Md+7dxr77difL+/vPAAzq24lCwIioh4YAzwITAdujYipki6WdHSa7QBghqSZwNbAJal8F6BW0iSyhMHLcr8quAwYKWkK2a8JTi1VG6x9efOdesbcOIEtN+/ET50HYGaGIoov07c/NTU1UVtbW+5qWBlFBF+5eSL3TZ7PTV/Yiz2369v8QmZmFUrS+JQ71yTfMdCqwk3/ncvYSfP5xmE7OQAwM0scBFi7N23+Ei66dyr7DuvnPAAzsxwHAdauZXkAT2V5ACf4uQBmZnklu22wWblFBN++cwovvLqMm76wF/18PwAzs7W4J8DarZufzPIAvn7ojs4DMDNrgIMAa5emL1jCRWOzPIDTD9ih3NUxM2uTHARYu/PmO/WcccNT9OrmPAAzs6Y4J8DalYjg/LuyPIAbnQdgZtYk9wRYu3LLk3O5Z+J8vnbIjuzlPAAzsyY5CLB2Y/qCJXy3kAdwoPMAzMya4yDA2oVl79Rzxo1r8gA6OA/AzKxZzgmwihcRfOfup3nhlWXccKrzAMzMWso9AVbxbq2dy10T5vHVQ3bkI9s7D8DMrKUcBFhFe2bhEi68Zyr77NCPM5wHYGa2XhwEWMVa9k49p9/wFD2dB2BmtkEcBFhFyucB/GzUcPr3cB6Amdn6chBgFem22jrumjCPrxy8Ix/dvl+5q2NmVpEcBFjFeWbhEi6452n23qEvYw5yHoCZ2YZyEGAVZVl6LkDPbp248oQRzgMwM9sIDgKsYkQEF9z9NM87D8DMbJNwEGAV47bxddw5YR5nHTzMeQBmZptASYMASUdImiFplqRzG5g+WNLDkiZLGidpQG7aKkkT02tsA8v+QtKbpay/tR0zFi7lwnue5qPb9+XMg4aVuzpmZu1CyW4bLKkDcDVwKFAHPClpbERMy812OXB9RFwn6SDgUuDkNG15RAxvZN01QO9S1d3alrdWZM8F6N6lE1eO8v0AzMw2lVL2BOwBzIqI2RGxArgZOKZonl2Bh9PwIw1MX0cKLn4MfHMT1tXasAvunspzi97k56OGs1WPruWujplZu1HKIGBbYG5uvC6V5U0CRqbh44Aekgo3f+8qqVbS45KOzS0zBhgbEQtKUWlrW26rncsdT9XxlYOH8dEdnAdgZrYplfIpgg312UbR+NnAVZJOAR4D5gH1adqgiJgvaTvg75KmAMuBTwIHNPvm0mnAaQCDBg3akPpbmc18aSkXOA/AzKxkShkE1AEDc+MDgPn5GSJiPnA8gKTuwMiIWJybRkTMljQOGEEWBOwAzJIEsLmkWRGxzh1jIuIa4BqAmpqa4uDD2ri3VmTPBXAegJlZ6ZTycsCTwDBJQyV1BkYBa2X5S+onqVCH84BrU/mWkroU5gH2BqZFxJ8j4j0RMSQihgBvNRQAWOW78J4sD+BnzgMwMyuZkgUBEVFPdv3+QWA6cGtETJV0saSj02wHADMkzQS2Bi5J5bsAtZImkSUMXlb0qwJrx26rncvt4+s466Bh7O08ADOzklFE0z3lksYAN0TE661TpU2vpqYmamtry10Na4GZLy3l6Kv+yYiBW/KnU/f0ZQAzsw0gaXxE1DQ3X0t6At5D9hv/W9PNf/xf2UrirRXZcwG6d+nIz5wHYGZWcs0GARHxHWAY8DvgFOBZST+QtH2J62ZV5sJ7pjJr0ZtcecIIturpPAAzs1JrUU5AZNcMFqZXPbAlcLukH5WwblZFbh9fx+3j6zjzoGHsM8x5AGZmraHZnwhKOgv4LPAK8FvgnIhYmbL6n8V37rON9OxLS7ng7qfZa7s+fOVg3w/AzKy1tOQ+Af2A4yNiTr4wIlZL+nhpqmXVonA/gC26dODno0Y4D8DMrBW15HLA/cBrhRFJPSTtCRAR00tVMasO3015AD89YbjzAMzMWllLgoBfAflH9i5LZWYb5Y7xddw2vo4zD9yBfYf1L3d1zMyqTkuCAEXuZgIRsZrS3m7YqsCsl5fynbufZs+hffjKITuWuzpmZlWpJUHAbElnSeqUXl8BZpe6YtZ+LV+xitNveIrNO3fg56OdB2BmVi4tCQK+BHyU7Al/dcCepKfzmW2I7459mmdfzvIAtnYegJlZ2TTbrR8RL5M9/Mdso935VB231tZx5kE7sN+OzgMwMyunltwnoCvweeB9wLunbRHxuRLWy9qhWS8v5fy7Uh6A7wdgZlZ2Lbkc8Eey5wccDjwKDACWlrJS1v4sX7GKM26Y8G4eQMcOpXyKtZmZtURL/hPvEBEXAMsi4jrgY8AHSlsta28uGjuVmS8vdR6AmVkb0pIgYGX6+4ak9wO9gCElq5G1O3dNqOOW2rmccYDzAMzM2pKW/N7/GklbAt8BxgLdgQtKWitrN2a9/Cbn3/U0ewztw1cPcR6AmVlb0mQQkB4StCQiXgceA7ZrlVpZu5DlATxF107ZcwGcB2Bm1rY0+V853R1wTCvVxdqZ7907lRkvZXkA7+nlPAAzs7amJadmf5V0tqSBkvoUXiWvmVW0uyfM4+Yn53LGgduzv/MAzMzapJbkBBTuB3BGrizwpQFrxHOL3uTbd01hjyF9+JqfC2Bm1ma15I6BQ1ujItY+vL0ylwfg+wGYmbVpLblj4GcaKo+I6zd9dazSfe/eqTyzcCl/+J8POw/AzKyNa8lp2odzr32Bi4CjW7JySUdImiFplqRzG5g+WNLDkiZLGidpQG7aKkkT02tsrvyGtM6nJV0rqVNL6mKld8/Eedz037mcfsD2HLDTVuWujpmZNaMllwPOzI9L6kV2K+EmSeoAXA0cSvb0wScljY2IabnZLgeuj4jrJB0EXAqcnKYtj4jhDaz6BuDTafhG4FTgV83Vx0rruUVv8u07p/DhIVvy9UOdB2BmVgk25ILtW0BL7vqyBzArImZHxArgZuCYonl2BR5Ow480MH0dEXF/JMB/yZ5lYGVUyAPo3HEz5wGYmVWQZv9bS7pX0tj0ug+YAdzTgnVvC8zNjdelsrxJwMg0fBzQQ1LfNN5VUq2kxyUd20C9OpH1GjzQSL1PS8vXLlq0qAXVtQ31vXun8czCpfzkhOG8t1e3clfHzMxaqCU/Ebw8N1wPzImIuhYspwbKomj8bOAqSaeQ3ZFwXnoPgEERMV/SdsDfJU2JiOdyy/4SeCwi/tHQm0fENcA1ADU1NcXva5tIlgfwIl8+YHsOdB6AmVlFaUkQ8CKwICLeBpDUTdKQiHihmeXqgIG58QHA/PwMETEfOD6ttzswMiIW56YREbMljQNGAM+leb8L9Ae+2IL6W4nMTnkANYO35BvOAzAzqzgtuXh7G7A6N74qlTXnSWCYpKGSOgOjyB5A9C5J/dLzCQDOA65N5VtK6lKYB9gbmJbGTwUOB0an2xpbGby9chVn3DiBzh034xcnOg/AzKwSteQ/d8eU2AdAGu7c3EIRUU/23IEHgenArRExVdLFkgo/MTwAmCFpJrA1cEkq3wWolTSJLGHwstyvCn6d5v1P+vnghS1og21iF983jekLljgPwMysgrXkcsAiSUdHxFgASccAr7Rk5RFxP3B/UdmFueHbgdsbWO7fwAcaWWdL6mwlNHbSfG584kW+tL/zAMzMKllLDqhfAm6QdFUarwMavIugtX/Pv7KM8+6YnOUBHOb5fOoVAAAT3klEQVQ8ADOzStaSmwU9B+yVEvcUEUtLXy1ri95euYrTc/cD6OQ8ADOzitaS+wT8QFLviHgzIpampL3/bY3KWdvy/UIewKeGs01v5wGYmVW6lpzKHRkRbxRGIuJ14KjSVcnaonsnzeeGJ17ki/tvx4E7Ow/AzKw9aEkQ0KHwcz3I7hMAdGlifmtnnn9lGefdOYUPDd6Ssw/bqdzVMTOzTaQliYF/Ah6W9Ps0/j/AdaWrkrUlhecCdOwgfuE8ADOzdqUliYE/kjQZOITsVsAPAINLXTFrG/73z9OYtmAJ155S4zwAM7N2pqWndQvJ7ho4EjiY7OY/1s7dO2k+f3r8Rb6433YctPPW5a6OmZltYo32BEjakexWv6OBV4FbyH4ieGAr1c3K6IWUB7D7oN6cfbjzAMzM2qOmLgc8A/wD+EREzAKQ9LVWqZWVVfZcgJQHcOLuzgMwM2unmvrvPpLsMsAjkv5P0sE0/Hhga2cu+fN0ps5fwhWf3I1tnQdgZtZuNRoERMRdEXECsDMwDvgasLWkX0k6rJXqZ63svsnz+ePjczhtv+04eBfnAZiZtWfN9vNGxLKIuCEiPg4MACYC55a8ZtbqXnhlGefeMYURg3pzjvMAzMzavfW62BsRr0XEbyLioFJVyMqjkAfQYTNxlfMAzMyqgh/LawD84P4sD+C3n6lxHoCZWZXw6Z7x58kLuP4/c/jCvkM5ZFfnAZiZVQsHAVVuzqvL+NYdkxkxqDffPGLnclfHzMxakYOAKvZO/Zo8AD8XwMys+jgnoIr94M/TeXpelgcwYMvNy10dMzNrZT71q1L3T1nAdf+Zw6n7OA/AzKxaOQioQnNeXca3bp/M8IHOAzAzq2YOAqrMO/WrGHPjBCS46sQRdO7or4CZWbUq6RFA0hGSZkiaJWmduwxKGizpYUmTJY2TNCA3bZWkiek1Nlc+VNITkp6VdIukzqVsQ3tz6f3PMGXeYq741HDnAZiZVbmSBQGSOgBXA0cCuwKjJe1aNNvlwPUR8UHgYuDS3LTlETE8vY7Olf8Q+GlEDANeBz5fqja0N3+ZsoA//PsFPr/PUA51HoCZWdUrZU/AHsCsiJgdESuAm4FjiubZFXg4DT/SwPS1SBJwEHB7KroOOHaT1bgde/HVt/jm7ZPZbWBvvuU8ADMzo7RBwLbA3Nx4XSrLm0T2yGKA44Aekvqm8a6SaiU9LqlwoO8LvBER9U2sEwBJp6XlaxctWrSxbalo79SvYsxNT2V5AKOdB2BmZplSHg3UQFkUjZ8N7C9pArA/MA8oHOAHRUQNcCJwpaTtW7jOrDDimoioiYia/v37b1AD2otL73+GyXWLufyTuzGwj/MAzMwsU8qbBdUBA3PjA4D5+RkiYj5wPICk7sDIiFicm0ZEzJY0DhgB3AH0ltQx9Qass05b2wNPZ3kAn9t7KIe97z3lro6ZmbUhpewJeBIYlrL5OwOjgLH5GST1k1Sow3nAtal8S0ldCvMAewPTIiLIcgf+X1rms8A9JWxDRXvx1bc45/bJ7DagF+ce6TwAMzNbW8mCgHSmPgZ4EJgO3BoRUyVdLKmQ7X8AMEPSTGBr4JJUvgtQK2kS2UH/soiYlqZ9C/i6pFlkOQK/K1UbKtm7eQDAVSfu7jwAMzNbh7KT6/atpqYmamtry12NVvW9e6fy+3+9wG9O/hCH+zKAmVlVkTQ+5dU1yaeH7dADTy/k9/96gf/Ze4gDADMza5SDgHZm7mtvcc7tk9htQC/OO3KXclfHzMzaMAcB7ciK+tWMufEpwHkAZmbWvFL+RNBa2WV/eYZJdYv59ac/5PsBmJlZs3yq2E48OHUh1/7ref5n7yEc8X7nAZiZWfMcBLQDc197i3Num8QHnQdgZmbrwUFAhSvkAQRw1WjnAZiZWcs5J6DC/fCBQh7A7gzq6zwAMzNrOZ82VrCHpi7kd/98nlM+OoQj3v/eclfHzMwqjIOACjX3tbc4+7ZJfGDbXpx3lJ8LYGZm689BQAVaUb+aMTdNIAKuPnF3unTsUO4qmZlZBXJOQAX60QPPMGnuG/zqJOcBmJnZhnNPQIV5aOpCfpvyAI78gPMAzMxswzkIqCDOAzAzs03JQUCFWFG/mjNTHsBVJ45wHoCZmW005wRUiB8/+AwT577BL0/ancF9tyh3dczMrB1wT0AF+Ou0l/i/fzzPZz8ymKOcB2BmZpuIg4A2ru71LA/g/dv25Nsf83MBzMxs03EQ0IZlzwWYwOrV4fsBmJnZJuecgDaskAdw9YnOAzAzs03PPQFt1N9SHsBnPjKYj33QeQBmZrbpOQhog+a9sZxvFPIAjnIegJmZlUZJgwBJR0iaIWmWpHMbmD5Y0sOSJksaJ2lA0fSekuZJuipXNlrSlLTMA5L6lbINrW3lqtWMufEpVq0Orhq9O107OQ/AzMxKo2RBgKQOwNXAkcCuwGhJuxbNdjlwfUR8ELgYuLRo+veBR3Pr7Aj8DDgwLTMZGFOaFpTHjx+cwYQX3+CykR9gSD/nAZiZWemUsidgD2BWRMyOiBXAzcAxRfPsCjychh/JT5f0IWBr4KHc/EqvLSQJ6AnML031W9/D01/imsdmc/Jeg/n4B7cpd3XMzKydK2UQsC0wNzdel8ryJgEj0/BxQA9JfSVtBlwBnJOfOSJWAl8GppAd/HcFftfQm0s6TVKtpNpFixZtbFtKrpAH8L5tenK+7wdgZmatoJRBgBooi6Lxs4H9JU0A9gfmAfXA6cD9EZEPIpDUiSwIGAFsQ3Y54LyG3jwiromImoio6d+//0Y1pNRWrlrNmTc+Rf2q7H4AzgMwM7PWUMr7BNQBA3PjAyjquo+I+cDxAJK6AyMjYrGkjwD7Sjod6A50lvQmcEda7rm0zK3AOgmHlebyB2fw1Itv8IvRI5wHYGZmraaUQcCTwDBJQ8nO8EcBJ+ZnSJn9r0XEarIz+msBIuKk3DynADURca6kbYBdJfWPiEXAocD0Erah5P7+zEv85rHZfHqvQXxiN+cBmJlZ6ynZ5YCIqCfL3H+Q7EB9a0RMlXSxpKPTbAcAMyTNJEsCvKSZdc4Hvgc8JmkyMBz4QYmaUHLz31jO12+dxK7v7cl3Plb8wwkzM7PSUkTxZfr2p6amJmpra8tdjbWsXLWaUdc8zjMLlnDfWfsy1JcBzMxsE5E0PiJqmpvPzw4ok8sfmsH4Oa/z89EjHACYmVlZ+LbBZfDIMy/zm0dnc9KegzjaeQBmZlYmDgJaWZYHMJFd3tuTCz7uPAAzMysfBwGtaOWq1Zx50wRW1K/mlyf5fgBmZlZezgloRVc8NJPxc17nZ6OGOw/AzMzKzj0BreSRZ17m148+x4l7DuKY4cV3TzYzM2t9DgJawYLFa/IALnQegJmZtREOAkqsftVqzrwxywO4+sQRzgMwM7M2wzkBJXbFX2dSm/IAtuvfvdzVMTMze5d7AkrokRkv86txzzF6D+cBmJlZ2+MgoEQWLF7ON26dxM7v6cF3P+E8ADMza3scBJRA/arVnHXTBN5ZuYqrfT8AMzNro5wTUAI/+etMnnwhywPY3nkAZmbWRrknYBMbN+NlfjnuOUbvMdB5AGZm1qY5CNiEFi5+m6+/mwfwvnJXx8zMrEkOAjaRQh7A284DMDOzCuGcgE3kp3+byX9feI0rT3AegJmZVQb3BGwCj85cxNWPPMeoDw/k2BHOAzAzs8rgIGAjLVz8Nl+7ZSI7v6cHFx3tPAAzM6scDgI2Qv2q1Zx1c5YHcNWJzgMwM7PK4pyAjXDl357lv8+/xk9P2I0dtnIegJmZVRb3BGygx2Yu4upxszihZiDHjRhQ7uqYmZmtt5IGAZKOkDRD0ixJ5zYwfbCkhyVNljRO0oCi6T0lzZN0Va6ss6RrJM2U9IykkaVsQ0NeWpLlAey4lfMAzMyscpUsCJDUAbgaOBLYFRgtqfhJOpcD10fEB4GLgUuLpn8feLSo7Hzg5YjYMa23eHpJFe4HsDzdD6BbZ+cBmJlZZSplT8AewKyImB0RK4CbgWOK5tkVeDgNP5KfLulDwNbAQ0XLfI4ULETE6oh4pQR1b9TPHn6WJ55/jf899v3OAzAzs4pWyiBgW2BubrwuleVNAgrd+ccBPST1lbQZcAVwTn5mSb3T4PclPSXpNklbN/Tmkk6TVCupdtGiRRvblndt27sbJ+81mON3dx6AmZlVtlIGAWqgLIrGzwb2lzQB2B+YB9QDpwP3R8Tcovk7AgOAf0XE7sB/yC4prPtGEddERE1E1PTv338jmrG2UXsM4vvHvn+Trc/MzKxcSvkTwTpgYG58ADA/P0NEzAeOB5DUHRgZEYslfQTYV9LpQHegs6Q3gfOAt4C70ipuAz5fwjaYmZm1W6UMAp4EhkkaSnaGPwo4MT+DpH7AaxGxmuwAfy1ARJyUm+cUoCYizk3j9wIHAH8HDgamlbANZmZm7VbJLgdERD0wBngQmA7cGhFTJV0s6eg02wHADEkzyZIAL2nBqr8FXCRpMnAy8I1NXnkzM7MqoIjiy/TtT01NTdTW1pa7GmZmZq1C0viIqGluPt8x0MzMrEo5CDAzM6tSDgLMzMyqlIMAMzOzKlUViYGSFgFzyl2PFuoHtOqtkFuB21QZ3KbK0N7a1N7aA22jTYMjotk75VVFEFBJJNW2JKOzkrhNlcFtqgztrU3trT1QWW3y5QAzM7Mq5SDAzMysSjkIaHuuKXcFSsBtqgxuU2Vob21qb+2BCmqTcwLMzMyqlHsCzMzMqpSDADMzsyrlIKCMJO0kaWLutUTSVyVdJGlervyocte1KZKulfSypKdzZX0k/VXSs+nvlqlckn4uaZakyZJ2L1/NG9dIm34s6ZlU77sk9U7lQyQtz31evy5fzRvXSJsa/a5JOi99TjMkHV6eWjeukfbckmvLC5ImpvJK+YwGSnpE0nRJUyV9JZVX7P7URJsqdn9qok2Vtz9FhF9t4AV0ABYCg4GLgLPLXaf1qPt+wO7A07myHwHnpuFzgR+m4aOAvwAC9gKeKHf916NNhwEd0/APc20akp+vrb4aaVOD3zVgV2AS0AUYCjwHdCh3G5prT9H0K4ALK+wzei+wexruAcxMn0XF7k9NtKli96cm2lRx+5N7AtqOg4HnIqJS7mz4roh4DHitqPgY4Lo0fB1wbK78+sg8DvSW9N7WqWnLNdSmiHgoIurT6OPAgFav2EZo5HNqzDHAzRHxTkQ8D8wC9ihZ5TZAU+2RJOBTwE2tWqmNFBELIuKpNLwUmA5sSwXvT421qZL3pyY+p8a02f3JQUDbMYq1/2GNSd1k1xa6/irM1hGxALIdBtgqlW8LzM3NV0fTO09b9TmyM7CCoZImSHpU0r7lqtQGaui7Vumf077ASxHxbK6soj4jSUOAEcATtJP9qahNeRW7PzXQporanxwEtAGSOgNHA7elol8B2wPDgQVk3ZrthRooq6jfqUo6H6gHbkhFC4BBETEC+Dpwo6Se5arfemrsu1bpn9No1g6qK+ozktQduAP4akQsaWrWBsra5OfUWJsqeX9qoE0Vtz85CGgbjgSeioiXACLipYhYFRGrgf+jjXQbraeXCt2S6e/LqbwOGJibbwAwv5XrtsEkfRb4OHBSpIt9qYvv1TQ8nux6347lq2XLNfFdq9jPSVJH4HjglkJZJX1GkjqRHVhuiIg7U3FF70+NtKmi96eG2lSJ+5ODgLZhrbOWomt6xwFPr7NE2zcW+Gwa/ixwT678MymreS9gcaGbs62TdATwLeDoiHgrV95fUoc0vB0wDJhdnlqunya+a2OBUZK6SBpK1qb/tnb9NtAhwDMRUVcoqJTPKOUy/A6YHhE/yU2q2P2psTZV8v7URJsqb38qd2Zitb+AzYFXgV65sj8CU4DJZF+e95a7ns204Sayrq+VZBHv54G+wMPAs+lvnzSvgKvJovspQE25678ebZpFdl1vYnr9Os07EphKlv37FPCJctd/PdrU6HcNOD99TjOAI8td/5a0J5X/AfhS0byV8hntQ9ZNPDn3PTuqkvenJtpUsftTE22quP3Jtw02MzOrUr4cYGZmVqUcBJiZmVUpBwFmZmZVykGAmZlZlXIQYGZmVqUcBJjZuySFpCty42dLuqiMVTKzEnIQYGZ57wDHS+q3KVeabmbj/zdmbYx3SjPLqweuAb5WPCHdye0OSU+m196p/CJJZ+fmezo9E35Iet76L8lu+jJQ0mhJU9I8P8wt86akSyRNkvS4pK1T+SfTvJMkPVbqxptVGwcBZlbsauAkSb2Kyn8G/DQiPkx2V7fftmBdO5E96nYE2Z39fggcRPaAlQ9LKjwSdwvg8YjYDXgM+EIqvxA4PJUfvRFtMrMGdCx3BcysbYmIJZKuB84ClucmHQLsmt02HYCekno0s7o5kT3nHuDDwLiIWAQg6QZgP+BuYAVwX5pvPHBoGv4X8AdJtwLvPnjGzDYNBwFm1pArybrwf58r2wz4SETkAwMk1bN2r2LX3PCy/KxNvN/KWHMP81Wk/00R8SVJewIfAyZKGh7pCXNmtvF8OcDM1hERrwG3kj1kqOAhYExhRNLwNPgCsHsq2x0Y2shqnwD2l9QvPSVuNPBoU/WQtH1EPBERFwKvsPbjWM1sIzkIMLPGXAHkfyVwFlAjabKkacCXUvkdQB9JE4EvAzMbWllkj7g9D3iE9IS4iLinoXlzflxIJCTLFZi0wa0xs3X4KYJmZmZVyj0BZmZmVcpBgJmZWZVyEGBmZlalHASYmZlVKQcBZmZmVcpBgJmZWZVyEGBmZlal/j9lk9merTYi5wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Alter neural parameters\n",
    "neural_layers_two = [\n",
    "    (64,64),\n",
    "    (128,128),\n",
    "    (256,256)\n",
    "]\n",
    "\n",
    "neural_accuracies_two = []\n",
    "\n",
    "# Run function pipeline for new parameters\n",
    "for n in neural_layers_two:\n",
    "    accuracy = cross_validate_nn(n)\n",
    "    mean_accuracy = np.mean(accuracy)\n",
    "    neural_accuracies_two.append(mean_accuracy)\n",
    "\n",
    "x = [i[0] for i in neural_layers_two]\n",
    "\n",
    "plt.figure(figsize = (8,4))\n",
    "plt.plot(x, neural_accuracies_two)\n",
    "plt.title(\"Mean Accuracies vs. Number of Two-Layer Neurons\")\n",
    "plt.xlabel(\"Neurons\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9460146003464489, 0.957706013363029, 0.9588121752041574]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neural_accuracies_two"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using two layers marginally improved accuracy. We should be wary of overfitting, but using 4-fold cross validation also provides a bit more assurance that the model is generalizing to achieve the extra 1% in simple accuracy over the single hidden layer networks we tried earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Triple Layer Network\n",
    "\n",
    "To help prevent overfitting, we can increase the validation to 6-fold as we incorporate a third hidden layer. Neuron values are 10, 64, and 128."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAEWCAYAAABhUT6OAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XeYVPXZ//H3h75IWwFRWQTsYtcFLEk0xlgSo1Fi7CgaERN/qSbRVGNiTDHteewKKjZEjdGYPNGEaIwNWETsKGKh6iK9s7v3749zVs4OC7uUYXZnP6/rmmtPnbnn7Jm553zPOfdXEYGZmZkVr1aFDsDMzMzyy8nezMysyDnZm5mZFTknezMzsyLnZG9mZlbknOzNzMyKnJO9WT0knSXp8ULH0RRJCkm7Fui195A0WdISSV/fyHXPk/R0vmIza8qc7IuEpHclrZbUI2f6i+mXc78CxdVfUo2k6wvx+psqIu6OiGMKHUdDJB2Z/n+vy5n+tKTzChRWPn0PeDIiOkfE/2RnSHpV0tL0US1pZWb8BwWKNxvf7ZJ+Ueg4siT1S/efv+VMv0vSFQUKy/LAyb64vAOcUTsiaV+gpHDhADAUWACcLqn91nxhSW225usV0DJgaKF+0G2qTfz/9AVerW9GROwdEZ0iohPwX+CS2vGI+OVGxiZJRfX92MD2PkTS4QWOwfKoqHZm406S5FrrXGB0dgFJ7SVdI+l9SR9IulFSSTqvVNKjkiolLUiHyzLrPinp55KeSZtRH89tSajHUOBHwBrgCzmx7C3pn5Lmp7H8IJ3eWtIPJL2dvs4kSX0yRyFtMs/xpKSvpMPnpbH9QdJ84ApJu0j6t6SPJM2TdLekbpn1+0j6c/qeP5J0bea5ns4st2cm1qmSvpyZ9zlJr6WxzpJ0ae5GSLf7Qkn7ZKb1lLRC0naSeqTbe2H6Gv/diGSzELgd+Gl9MyVdIemuzHid7Zhuw19IejY9Cv6rpO7ptlosaWI9PyQ+J2l6uk1/m41V0vmSXk/3occk9c3MC0lfk/QW8NZ64j1RyVH6wjS2vdLp/wY+DVybxrl7I7dP7vNfk8b2jqTjM9OflHSVpGeA5cDOkrpKGilpTvq//YWk1o15rxsZ058kzUi39yRJn0ynby9puaTumWUPTvfXtg3F0JjtnfoNsN5WB0knKGklXJjuJ/vlvMaumfGPWzCUtDzNlPR9SXOB29LpF0qalu7rj0jaMef5Rkh6K31P10lSOm9XSf+RtCjd9+5r7DZu6Zzsi8vzQBdJe6VfSKcBd+Us82tgd+AAYFegN/CTdF4rkg9jX2AnYAVwbc76ZwLDgO2AdsA6ia1W+oVVBowBxpL5ISKpM/Av4B/Ajmks49LZ3yZpofgc0AU4n+TLtzEGA9PT+K4CBFydvsZeQB/gijSG1sCjwHtAP5JtMaae97EN8E/gnvR5zwCul7R3ushI4KKI6AzsA/w79zkiYhXwZzItL8CXgf9ExIfAd4CZQE+gF/ADYGNqWV8FDJG0x0ask3U6cA7JNtgFeI5kX9gWeJ11f0icDJQDBwEnkfyPkPTFNPZT0vfyX+DenHW/SPJ/GpAbRJrA7wW+ma7/d+CvktpFxFHUPWJ/cxPe52BgKtCDJMGNrE0kqXOA4UBnkv3iDqCKZP88EDgGqP1x2Zj32lgTST6T25LsZ/dL6hARc4EnSfaVWmcDYyJizeZu74zrgN0lHZ07Q9JBwCjgIqA7cBPwiBrfUrd9+r76AsMlHUXymfwysAPJds793J0ADAT2T5c7Np3+c+BxoJTku+V/GxmDRYQfRfAA3gWOJjmKvho4jiRBtSFJGv1IEt8yYJfMeocC76znOQ8AFmTGnwR+lBn/KvCPDcR0K/CXzOusAbZLx88AJq9nvanASfVM75e+lzY5MX0lHT4PeL+B7fTF2tdNY6rMPl9mufOAp9Ph04D/5sy/CfhpOvw+yRdhlwZe+2hgemb8GWBoOnwl8DCw60b+348EZqbDvwHuS4efBs5Lh68A7lrfdky34Q8z838H/F9m/AvAi5nxAI7L2Q/GpcP/B1yQmdeK5Ida38y6R23g/fwYGJuz/izgyNz/dwPbZZ3l0v/ptMx4xzSe7TPrXJmZ3wtYBZRkpp0BPNGY91pPTLcDv2jk/3UBsH9m/3smHW4NzAUGbaHt/fG+kP4fn0+n3wVckQ7fAPw8Z72pwBGZ19i1vvdJsn+uBjpk5o8EfpMZ70Ty3dAv83yfyMwfC1yWDo8GbgbKNuZz4kf4yL4I3Uly9H0eOU34JL/8OwKT0ua4hSRH1j0BJHWUdJOk9yQtBp4CumWbLUm+aGotJ/mgrkPJqYFTgbsBIuI5kqR4ZrpIH+Dt9byHDc1ryIycOLaTNCZtgl1M8iVWe+qhD/BeRFQ18Jx9gcG12yzdbmeRHLEADCFphXgvbWI8dD3P82+gRNLgtKn1AOChdN5vgWnA40qaxy9r/Fv+2K+BYyXtvwnrfpAZXlHPeO7/Obud3yNpOYFkW/0ps53mk/zI7L2edXPtmD4fABFRky7fe71rbJyP99+IqG0tyr63bGx9gbbAnMz7uYmkdad2fr3vVclpqNqLA29sKChJ30mb4helz9WVtfvpw8AASTsDnwUWRcSEhmJYz3vakFuAXpK+kDO9L/CdnP2/D2v/5w2pjIiVmfHc//FS4KOcmNf3PfM9kvc3QcmpnvMbGUOL52RfZCLiPZIL9T5H0mycNY/ki3vviOiWPrpGckETJE3JewCDI6IL8Kl0uth4J5M0wV8vaW56vq43a5vyZ5A0F9dnffOWpX87ZqZtn7NMbtP31em0/dL3dDZr388MYCc1fNHQDJLm9m6ZR6eIuBggIiZGxEkkSeAvJEci60gT11iSo8MzgUcjYkk6b0lEfCcidiY5kv62pM80EFfu838E/JGkqTNrGRveZpuiT2Z4J2B2OjyD5JRGdluVRMSz2VA38LyzSZILkFwol77WrC0Qc2NkY5tBcmTfI/NeukTE3pn59b7XiPhlrL04cMSGXjA93fV9kubq0ojoBiwi3U/TRDmW5AfmOSQ/6LMxbs72XrtQxBrgZyT7T/YzPwO4Kuc1OkZE7emC5WzcZzL3f7wNyemBBv/HETE3Ii6MiB1JWtOuV4FuA21unOyL0wUkTXfLshPTZHML8AdJ2wFI6i2p9nxYZ5IfAwslbct6LvhqpHNJzvPtS3IEewBwOHCAkrsEHgW2l/RNJRevdZY0OF33VuDnknZTYj9J3SOikuQL4WwlF/Gdz/p/MNTqDCxN31Nv4LuZeROAOcCvJG0jqYPqvyL5UZLzmedIaps+BqbXRrRTck9+1/TLcjFQvYF47iFplj0rHQY+vgBq1zS51T7Hhp5nfX4PHEZyfUKtF4FPSdpJUlfg8k143lzfVXJBZx/gG0DthVI3ApfXXs+g5AK3UzfieccCn5f0GSUXoH2HJOE+u+HVtryImENyfvh3krpIaqXkgs8j0kU25b22Tvez2kc7kn20ivSUkqSfkPxQzhpN0lp3InWvw9nc7Z3rTqA9yWnAWrcAI9IWKaWflc8rue4Gkv3rzPQzeRxwBBt2DzBM0gFKzvv/EhgfEe82FJykU7X2ouEFJD8kNuVz0uI42RehiHg7IirWM/v7JM3Fz6fN2v8iOZqH5KiwhKQF4HmSJv6NlibVzwB/TH+J1z4mpc95bnpE+1mSo9i5JFcKfzp9it+TfOk/TpL4RrL2FsILSRL2R8DeNJwEfkZyEdki4G9kWjsiojp9/V1JTjHMJEnEdaSxHkNyEdvsNN5fk3wpQnK09W66PUeQtB7UKyLGkxxp70hyvrXWbiT/i6UkF8ddHxFPAkj6PzXyPvGIWExy7n7bzLR/kiTjl4BJJD9eNtfD6XO9SLJdR6av9RDJthmTbo9XgOPX9yT1xD+VZPv9L8l++AXgCxGxegvEvCmGklyI+hpJcnmA5KKyTX2vl5H8oK59/Bt4jGRfeJOkeXslOU3vEfEMUAO8kE2Km7u9c6WfiZ9Sd/+pIPncXUuyDaaR/PCo9Q2S/1Pt6a2/NPAa40iuzXiQ5Mf2LiSfrcYYCIyXtBR4BPhGRLzTyHVbNEVszAW/ZmZWCEpuPbwnIm4tdCzW/DjZm5k1cZIGktxd06f2Og+zjeFmfDOzJkzSHSSneL7pRG+bykf2ZmZmRc5H9mZmZkWuaDol6NGjR/Tr16/QYZiZmW01kyZNmhcRPRtarmiSfb9+/aioWN/dZmZmZsVH0nsNL+VmfDMzs6KX12Qv6Tgl3YFOq6/Wt6S+ksZJeklJ95LZ7lR3UtKF6utKug/tl89YzczMilXekr2SzlOuI6nmNAA4Q1JuF4vXAKMjYj+SXr+uzswbDfw2IvYCBgEf5itWMzOzYpbPI/tBJN1JTk9LXY4h6fc6awBr+zB/onZ++qOgTVrmk4hYmumhyszMzDZCPpN9b+rWd57Jut1UTiHpHhSSXtI6S+oO7E7SccmfJU2W9FvV7WYVAEnDJVVIqqisrMzDWzAzM2v+8pns6+sWNbeCz6XAEZImk/SUNIuk96c2wCfT+QOBnanb8ULyZBE3R0R5RJT37NngnQdmZmYtUj6T/Uzq9nldxto+rwGIiNkRcUpEHAj8MJ22KF13cnoKoIqkF6WD8hirmZlZ0crnffYTgd0k9Sc5Yj8dODO7gKQewPy0n/XLSfo/r123VFLPtA/zowDfRG9mZs3G6qoa5ixawawFK5i5YAUzFyzn5IPK6N9jm60eS96SfURUSbqEpK/m1sCoiHhV0pVARUQ8AhwJXC0pgKeAr6XrVku6FBgnSST9Zt+Sr1jNzMw21qqqauYsXPlxIp+1cG1Sn7lgBXMXryTb/Uwrwd69uxYk2RdNRzjl5eXhCnpmZralrKqqZvbClR8n79q/tUfqHyypm8xbtxLbd+lAWWkJZaUd078l9C4toU9pR7bv2oG2rbfs2XNJkyKivKHliqZcrpmZ2cZYuaaaWQvrNrNn/364ZFWd5Vu3Ejt260DvbiV8Yrce6yT17bt0oM0WTuZbipO9mZkVpZVrqnOS+Iq0qT0Zr8xJ5m1aiR27JYn7yD160rvb2kRetm1HenVu32STeUOc7M3MrFlasbqaWQuXM2OdI/PkaH3e0rrJvG3rtcn8qD22S5N4ycdJvVeXDrRuVd9d482fk72ZmTVJy1ZV1TkSz21u/2jZ6jrLt2vdit6lJfTuVsLRe22X08zekZ6d2xdtMm+Ik72ZmRXE0lVVaQLPuQAuvap9fm4yb9OKsm7JBW/H7Nh1bRN7bTLv1J5WLTSZN8TJ3szM8mLJyjWZI/K1TewzFybDC5evqbN8+zbJkXlZaUf26d314yTeu1sJfUpL6OFkvsmc7M3MbJMsXrmGmfNzj8jXJvVFK+om8w5tW33crL5/Wbc6V7KXlXakR6d2JKVVbEtzsjczs3otWrGm7hF5nXPny1m8sqrO8iVtW3+cvA/aqfTje8xrk3r3bZzMC8XJ3sysBYqINJnnNLHXVoNbsIIlq+om847tWn98FF7er7TOBXC9u5WwrZN5k+Vkb2ZWhCKChctzk3m2pOsKluYk823atabPtknyHtx/2zpXspeVltCtY1sn82bKyd7MrBmKCOYvW13vufLaI/Nlq6vrrNO5fZuPm9UP2bl7nfPlZaUldC1xMi9WTvZmZk1QRPBRmsxzz5XXJvUVa+om8y4d2lBW2pG+3bfh8F17rHMBXNeStgV6N1ZoTvZmZgUQEcxburr+C+DSI/WVa2rqrNO1pC1lpSXs3HMbPrV7T3p3W5vIe6dH5mb1cbI3M8uDmppg3tJVzNhAM/uqqrrJvFvHJJnv2rMTR+7es04i711aQpcOTua2aZzszcw2QU1NULl01fpvTVu4gtU5yXzbbdpRVlrCHr0685k9t6tzAVzv0hI6tfdXsuWH9ywzs3pU1wQfLllZT/enyfDshStZXV03mXdPk/leO3ThswN6pRfDra0Ct42TuRWI9zwza5Gqa4IPFq+s06yeLeU6e+EK1lRHnXV6dGpP79IS9u7dlWP32f7jI/M+pSXs2K2Eju38lWpNk/dMMytKVdU1fLBkFTPn55wrT+8zn71wBVU1dZN5z87tKSstYd/eXTl+nx3qXMneu1sJJe1aF+jdmG0eJ3sza5aqqmuYs2hlpkhM3Z7T5i5auU4y3y5N5gf06cYJ++1Qp5Rr724ldGjrZG7FycnezJqkNdU1zF20khmZc+XZ+8znLl5JdSaZS9Crcwd6l5ZwcN91S7nu6GRuLZiTvZkVxOqqGuYsWlHvBXCzFq5gzqIVZA/MJdi+SwfKSksY1H/bOveYl5WWsEO3DrRv42RuVh8nezPLi1VV1cxZuHKdc+XZI/PIJPNWHyfzjmld9rW3pJWVlrBD1xLatWlVuDdk1ow52ZvZJllVVc3shSvXOVdee6T+wZJ1k/kOXZPiMIfu0r1OKdc+pR3ZvmsH2rZ2MjfLByd7M6vXyjXVzFpYXzN78vfDJavqLN+6ldiha9LM/ondeqzTzO5kblY4TvZmLdh7Hy3jnXnLMvXY1ybzypxk3qaV2KFbB8q6deSI3XvWOTLvXVrC9l060MbJ3KxJcrI3a4Eigl/94w1u+s/0j6e1bS12TI/GP71HNpknf3t16UDrVu7+1Kw5ymuyl3Qc8CegNXBrRPwqZ35fYBTQE5gPnB0RM9N51cDL6aLvR8SJ+YzVrKWoqq7h8j+/zP2TZnL6wD4MObiMstIStuvsZG5WrPKW7CW1Bq4DPgvMBCZKeiQiXsssdg0wOiLukHQUcDVwTjpvRUQckK/4zFqilWuqueSeyfzr9Q/4xmd245tH74bkBG9W7PJ5gm0QMC0ipkfEamAMcFLOMgOAcenwE/XMN7MtZPHKNQwdNYFxb3zAz07cm299dncnerMWIp/JvjcwIzM+M52WNQUYkg6fDHSW1D0d7yCpQtLzkr5Y3wtIGp4uU1FZWbklYzcrKh8uWclpNz3P5PcX8KfTD+Tcw/oVOiQz24rymezrO2SInPFLgSMkTQaOAGYBVem8nSKiHDgT+KOkXdZ5soibI6I8Isp79uy5BUM3Kx7vf7ScU298jnfnLWPkuQM5cf8dCx2SmW1l+bxAbybQJzNeBszOLhARs4FTACR1AoZExKLMPCJiuqQngQOBt/MYr1nReX3OYoaOmsCa6hruuXAwB+5UWuiQzKwA8nlkPxHYTVJ/Se2A04FHsgtI6iGpNobLSa7MR1KppPa1ywCHA9kL+8ysARPemc+Xb3qONq3E/Rcd6kRv1oLlLdlHRBVwCfAY8DowNiJelXSlpNrb6I4Epkp6E+gFXJVO3wuokDSF5MK9X+VcxW9mG/Cv1z7gnJHj6dm5PQ9cfBi79epc6JDMrIAUkXsavXkqLy+PioqKQodhVnD3V8zgsj+/zN47duG28wbSvVP7QodkZnkiaVJ6fdsGuYKeWRG5+am3+eXf3+ATu/bgxnMOplN7f8TNzMnerChky99+ft8d+P1p+7tvdzP7mJO9WTNXVV3DDx56mbEVMzlr8E5cedI+LntrZnU42Zs1YyvXVPP1eyfz+Gsf8PXP7Ma3XP7WzOrhZG/WTC1euYYL76hg/DvzueILAzjv8P6FDsnMmigne7NmqHLJKs4dNYE3P1jCn04/gJMOyK1EbWa2lpO9WTPz/kfLOWfUeD5cvIpbzy3nyD22K3RIZtbEOdmbNSPZ8rd3XziYg1wVz8wawcnerJmY+O58zr99Itu0a8M9Fx3qqnhm1mhO9mbNwLjXP+Crd79A724ljL5gEGWlHQsdkpk1I072Zk3cg5Nm8r0HX3L5WzPbZE72Zk3YLU9N56q/v87hu3bnpnPKXf7WzDaJvznMmqCI4Nf/mMqN/3mbz+27PX847QCXvzWzTeZkb9bEVFXX8MOHXuG+ihmcOXgnfu7yt2a2mZzszZqQOuVvj9qVb312d5e/NbPN5mRv1kQsWbmGC0dX8Pz0+fz0CwMY5vK3ZraFONmbNQGVS1Zx3m0TmDrX5W/NbMtzsjcrsBnzl3POyPHMXbySW84t59Muf2tmW5iTvVkBvTF3MUNHTmBVVQ13f+UQDu7r8rdmtuU52ZsVyMR353PB7RPp2K4N9484lN1d/tbM8sTJ3qwA/v3GB1x8l8vfmtnW4WRvtpX9+YWZfPeBlxiwQxduH+byt2aWf072ZlvRrf+dzi/+9jqH7dKdm4e6/K2ZbR3+pjHbCiKC3z42leuffJvj99meP57u8rdmtvU42ZvlWVV1DT/6yyuMmTiDMwbtxC++6PK3ZrZ1Odmb5dHKNdV8Y8xkHnv1A/7fUbvybZe/NbMCaJXPJ5d0nKSpkqZJuqye+X0ljZP0kqQnJZXlzO8iaZaka/MZp1k+LFm5hmG3TeSxVz/gp18YwHeO2cOJ3swKIm/JXlJr4DrgeGAAcIakATmLXQOMjoj9gCuBq3Pm/xz4T75iNMuXeUtXccYtzzPx3fn88bQDXOfezAoqn0f2g4BpETE9IlYDY4CTcpYZAIxLh5/Izpd0MNALeDyPMZptcTPmL+dLNzzLtA+Xcsu55XzxQNe5N7PCymey7w3MyIzPTKdlTQGGpMMnA50ldZfUCvgd8N0NvYCk4ZIqJFVUVlZuobDNNt0bcxcz5IZnmb9sNXd/ZbDr3JtZk5DPZF/fycnIGb8UOELSZOAIYBZQBXwV+HtEzGADIuLmiCiPiPKePXtuiZjNNlnFu/P58o3PIcH9Iw7j4L7bFjokMzMgv1fjzwT6ZMbLgNnZBSJiNnAKgKROwJCIWCTpUOCTkr4KdALaSVoaEetc5GfWFPz7jQ/46t0vsEPXEkafP4g+27r8rZk1HflM9hOB3ST1JzliPx04M7uApB7A/IioAS4HRgFExFmZZc4Dyp3oral6aPJMLr3/JfbaoTO3DxtED5e/NbMmJm/N+BFRBVwCPAa8DoyNiFclXSnpxHSxI4Gpkt4kuRjvqnzFY5YPI59+h2/dN4XB/bfl3gsPcaI3syZJEbmn0Zun8vLyqKioKHQY1kJEBNc8PpXrnnib4/ZOyt92aOvyt2a2dUmaFBHlDS3nCnpmG6m6JvjRX17m3gkzOGNQH37xxX1d/tbMmjQne7ONsHJNNd8c8yL/eHUuX/v0Llzqqnhm1gw42Zs10pKVaxg+ehLPTf+IH58wgAs+4ap4ZtY8ONmbNcK8pas477YJvDFnCX84bX9OPrCs4ZXMzJoIJ3uzBsyYv5yhoyYwZ9EKbhlazqf3dFU8M2tenOzNNmDq3CUMHTWeFaurueuCwZT3c1U8M2t+nOzN1mPSe/M5//YK2rdpxdgRh7Ln9l0KHZKZ2SZpsKiOpEsklW6NYMyaiife+JCzbh1Pace2PHjxYU70ZtasNaaC3vbAREljJR0n32dkRe4vk2dx4egKdunZiQcuPsx17s2s2Wsw2UfEj4DdgJHAecBbkn4paZc8x2a21Y16+h2+ed+LDOy3LWOGu/ytmRWHRtXGj6Sm7tz0UQWUAg9I+k0eYzPbaiKCax6bypWPvsZxe2/PbcMG0rlD20KHZWa2RTR4gZ6krwPnAvOAW4HvRsQaSa2At4Dv5TdEs/yqrgl+/PAr3DP+fU4f2IerTnb5WzMrLo25Gr8HcEpEvJedGBE1kk7IT1hmW8eqqmq+dd+L/P3luXz1yF347rEuf2tmxacxyf7vwPzaEUmdgQERMT4iXs9bZGZ5tnRVFcNHV/Ds2x/xo8/vxVc+uXOhQzIzy4vGnLO/AViaGV+WTjNrtuYtXcUZNz/P+Hfm87tT93eiN7Oi1pgje0Wm0/u0+d7FeKzZmrlgOUNHTmDWwhXcfM7BfGavXoUOycwsrxpzZD9d0tcltU0f3wCm5zsws3x484MlDLnhWeYtXcVdXxnsRG9mLUJjkv0I4DBgFjATGAwMz2dQZvkw6b0FnHrjc0TA2BGHMtB17s2shWiwOT4iPgRO3wqxmOXNk1M/5OK7XqBXl/bcecFgV8UzsxalMffZdwAuAPYGOtROj4jz8xiX2Rbz8Iuz+M7YKeyxfWduHzaInp1dFc/MWpbGNOPfSVIf/1jgP0AZsCSfQZltKbc98w7fGPMiB/ct5d7hhzjRm1mL1Jhkv2tE/BhYFhF3AJ8H9s1vWGabJyL43eNT+dlfX+OYAb244/xBdHH5WzNroRpzC92a9O9CSfuQ1Mfvl7eIzDZTtvztaeV9uOrkfWjTulHdQJiZFaXGJPub0/7sfwQ8AnQCfpzXqMw2Ubb87cVH7sL3XP7WzGzDyT7t7GZxRCwAngJcZsyarKWrqrjozgqemebyt2ZmWRts24yIGuCSrRSL2Sb7aOkqzrzleZ6f7vK3Zma5GnMi85+SLpXUR9K2tY/GPLmk4yRNlTRN0mX1zO8raZyklyQ9KaksM32SpBclvSppxEa+L2tBZi5Yzqk3PsfUuUu46eyDGXJwWaFDMjNrUhpzzr72fvqvZaYFDTTpS2oNXAd8lqTy3kRJj0TEa5nFrgFGR8Qdko4CrgbOAeYAh0XEKkmdgFfSdWc36l1Zi/HmB0sYOnICy1ZXcddXBrsqnplZPRpTQa//Jj73IGBaREwHkDQGOAnIJvsBwLfS4SeAv6SvuTqzTHsa1wJhLcwL7y9g2G0TademFWMvOpS9duhS6JDMzJqkxlTQG1rf9IgY3cCqvYEZmfHauvpZU4AhwJ+Ak4HOkrpHxEeS+gB/A3YFvuujesuqLX+7XZf23OXyt2ZmG9SYI+aBmccngSuAExuxXn33O0XO+KXAEZImA0eQdLZTBRARMyJiP5Jkf66kdbonkzRcUoWkisrKykaEZMXg4Rdn8ZU7KujfYxseGHGYE72ZWQMa04z//7LjkrqSlNBtyEygT2a8DKhzdJ4erZ+SPm8nYEhELMpdRtKrJD80HsiZdzNwM0B5eXnuDwkrQrc/8w4/e/Q1BvXbllvOLXdVPDOzRtiUc+HLgd0asdxEYDdJ/SW1I+k575HsApJ6pPfyA1wOjEqnl0kqSYdLgcOBqZsQqxWJiOD3j0/lir++xtF7ufytmdnGaMw5+7+ytvm9FclFdWMbWi8iqiRdAjwGtAZGRcSrkq4EKiLiEeCYS+GXAAAVMklEQVRI4GpJQVK0p/aK/72A36XTBVwTES9v1DuzolFdE/zk4Ve4e/z7fLm8jF+evK/L35qZbQRFbLj1W9IRmdEq4L2ImJnXqDZBeXl5VFRUFDoM28JWVVXz7fum8LeX53DRETtz2XF7uvytmVlK0qSIKG9oucbcZ/8+MCciVqZPXCKpX0S8u5kxmm3Q0lVVjLhzEk9Pm8cPPrcnwz+1S6FDMjNrlhrTFno/UJMZr06nmeXN/GWrOeuW53lu+kdcc+r+TvRmZpuhMUf2bbJFbiJidXrBnVlezFq4gnNGjmfWghXcdPbBHD1gnbsuzcxsIzTmyL5S0sf31Us6CZiXv5CsJXvrgyV86YZnqVyyijsvGOxEb2a2BTTmyH4EcLeka9PxmUC9VfXMNscL7y/g/Nsn0rZ1K+4bfigDdnT5WzOzLaExRXXeBg5Ji94oIpbkPyxraf7zZiUj7pxEz85J+dudursqnpnZltJgM76kX0rqFhFLI2KJpFJJv9gawVnL8MiU2Xzljon067END1x8qBO9mdkW1phz9sdHxMLakYhYAHwufyFZSzL6uXf5xpjJHLhTKfdddAjbde5Q6JDMzIpOY87Zt5bUPiJWQXKfPUm3s2abLCL447/e4k/j3uLovXpx7ZkH0qFt60KHZWZWlBqT7O8Cxkm6LR0fBtyRv5Cs2FXXBFc88ip3Pv8epx5cxtWnuPytmVk+NeYCvd9Iegk4mqRO/T+AvvkOzIrTqqpqvj12Cn97aQ4XfWpnLjve5W/NzPKtMUf2AHNJquh9GXgHeDBvEVnRWraqiovS8reXH78nFx3hqnhmZlvDepO9pN1JuqU9A/gIuI/k1rtPb6XYrIjMX7aaYbdN4JXZi/nNl/bjy+V9Ch2SmVmLsaEj+zeA/wJfiIhpAJK+tVWisqIya+EKho4cz8wFK7jx7IP5rKvimZltVRu6KmoISfP9E5JukfQZknP2Zo027cOk/O2Hi1cx+vxBTvRmZgWw3mQfEQ9FxGnAnsCTwLeAXpJukHTMVorPmrHJ7y/gSzc+x5rq4L6LDmXwzt0LHZKZWYvU4P1OEbEsIu6OiBOAMuBF4LK8R2bN2lNvVnLWrePp0qEtD17sOvdmZoW0UTc3R8T8iLgpIo7KV0DW/P11ymwuuGMiO23bkQdGHErf7tsUOiQzsxatsbfemTXKnc+9y08eeZWBfbfllnPL6VrSttAhmZm1eE72tkXULX+7HdeeeZDL35qZNRFO9rbZamqCK/76KqOfe48vHVzGr1z+1sysSXGyt82yuqqGb499kUdfmsPwT+3M5S5/a2bW5DjZ2yZbtqqKEXdN4r9vufytmVlT5mRvm2T+stUMu30iL89c6PK3ZmZNnJO9bbTZC1dwzsjxzEjL3x6z9/aFDsnMzDbAyd42yrQPlzJ05HiWrKxi9PmDOMRV8czMmjwne2u0F2csZNhtE2jdqhVjLjqEvXfsWuiQzMysEfJ6f5Sk4yRNlTRN0joldiX1lTRO0kuSnpRUlk4/QNJzkl5N552WzzitYf99q5Izb3meTh3a8ODFhzrRm5k1I3lL9pJaA9cBxwMDgDMkDchZ7BpgdETsB1wJXJ1OXw4MjYi9geOAP0rqlq9YbcMefWk259+elL99cMRhLn9rZtbM5PPIfhAwLSKmR8RqYAxwUs4yA4Bx6fATtfMj4s2IeCsdng18CPTMY6y2Hnc+9y7/797JHNCnG/dddCjbdelQ6JDMzGwj5TPZ9wZmZMZnptOypgBD0uGTgc6S6lzxJWkQ0A54O/cFJA2XVCGporKycosFbrXlb9/kxw+/ylF7bMfo8we7zr2ZWTOVz2RfXxm1yBm/FDhC0mTgCGAWUPXxE0g7AHcCwyKiZp0ni7g5IsojorxnTx/4byk1NcEVj7zKH//1FkMOKuPGcw6mpJ3r3JuZNVf5vBp/JpCttFIGzM4ukDbRnwIgqRMwJCIWpeNdgL8BP4qI5/MYp2WsrqrhO/dP4a9TZnPhJ/tz+fF70aqVy9+amTVn+Uz2E4HdJPUnOWI/HTgzu4CkHsD89Kj9cmBUOr0d8BDJxXv35zFGy1i+uooRd73AU29WctnxezLC5W/NzIpC3prxI6IKuAR4DHgdGBsRr0q6UtKJ6WJHAlMlvQn0Aq5Kp38Z+BRwnqQX08cB+YrVYMGy1Zx5y3iefquS3wzZz4nezKyIKCL3NHrzVF5eHhUVFYUOo1mavXAFQ0dN4P35y/nfMw7kWJe/NTNrFiRNiojyhpZzBb0Wrrb87eKVVdwxbBCH7uLyt2ZmxcbJvgWbMmMh5902gdatxJjhh7BPb1fFMzMrRk72LdTTb81j+J0VdO/UjjvPH0y/Hq6KZ2ZWrJzsW6C/vTSHb943mV16dmL0+YNcFc/MrMg52bcwdz7/Hj95+BUO3qmUkecOpGtHV8UzMyt2TvYtRETwP+Om8Yd/vclRe27HdWce5Kp4ZmYthJN9C1BTE/zsr69yx3PvccpBvfn1kP1o2zqvvRubmVkT4mRf5FZX1XDp/VN4ZMpsvvKJ/vzgcy5/a2bW0jjZF7Fs+dvvH7cnI47YGcmJ3syspXGyL1ILl69m2O0TmTJjIb8esi+nDdyp0CGZmVmBONkXoTmLVjB05ATem7+c6886mOP2cflbM7OWzMm+yLxduZShIyewaMUal781MzPAyb6ovDRzIefdNhGBy9+amdnHnOyLxNNvzeOiOyso3aYdd14wmP4uf2tmZikn+yLw95fn8M0xL9K/xzaMvmAQvVz+1szMMpzsm7m7x7/Hj/7i8rdmZrZ+TvbNVERw7b+n8bt/uvytmZltmJN9M1RTE1z56Gvc/uy7nHJgb379JZe/NTOz9XOyb2ZWV9Xw3Qem8PCLs7ngE/35ocvfmplZA5zsm5Hlq6u4+K4X+M+blXz32D346pG7uPytmZk1yMm+mVi4fDXn3z6RF2cs5Fen7Mvpg1z+1szMGsfJvhmYu2glQ0eN5915y7n+rIM4bp8dCh2SmZk1I072Tdz0yqWck5a/vf38gRy2S49Ch2RmZs2Mk30T5vK3Zma2JTjZN1HPTJvH8NEVdOvYjjsvGMTOPTsVOiQzM2umnOyboNryt/16dGT0+YPZvqvL35qZ2abLayUWScdJmippmqTL6pnfV9I4SS9JelJSWWbePyQtlPRoPmNsau4Z/z5fu+cF9i3rytiLDnWiNzOzzZa3ZC+pNXAdcDwwADhD0oCcxa4BRkfEfsCVwNWZeb8FzslXfE1NUv72LX7w0MscuXtP7rpgMN06tit0WGZmVgTyeWQ/CJgWEdMjYjUwBjgpZ5kBwLh0+Ins/IgYByzJY3xNRm3522sef5OTD+zNzUPLXefezMy2mHwm+97AjMz4zHRa1hRgSDp8MtBZUvfGvoCk4ZIqJFVUVlZuVrCFsqa6hm+PfZHbnnmX8w/vz+9O3d917s3MbIvKZ1apr45r5IxfChwhaTJwBDALqGrsC0TEzRFRHhHlPXv23PRIC2T56iouHF3BX16czXeP3YMfn+A692ZmtuXl82r8mUCfzHgZMDu7QETMBk4BkNQJGBIRi/IYU5ORLX/7y5P35czBLn9rZmb5kc9kPxHYTVJ/kiP204EzswtI6gHMj4ga4HJgVB7jaTKy5W+vO/Mgjt/X5W/NzCx/8taMHxFVwCXAY8DrwNiIeFXSlZJOTBc7Epgq6U2gF3BV7fqS/gvcD3xG0kxJx+Yr1q1peuVShtzwLLMWrOD2YQOd6M3MLO8UkXsavXkqLy+PioqKQoexQS/PXMR5t00A4PZhg9i3zOVvzcxs00maFBHlDS3nCnpbybNvz2P46El0LWnr8rdmZrZVOdlvBf94ZQ5fv9flb83MrDCc7PPs3gnv88OHXuaAPt0Ydd5AV8UzM7Otzsk+TyKC6598m98+NpUj9+jJ9WcdRMd23txmZrb1OfvkQU1N8Iu/vc6oZ97hiwfsyG9dFc/MzArIyX4LW1Ndw/ceeImHJs9i2OH9+PHnB7gqnpmZFZST/Ra0YnU1X7vnBf79xodceszufO3TuyI50ZuZWWE52W8hi5av4fw7JvLC+wu46uR9OGtw30KHZGZmBjjZbxFzF63k3FETeGfeMq478yA+56p4ZmbWhDjZb6Z35i3j7FvHs3D5am4bNpDDd+1R6JDMzMzqcLLfDK/MWsS5oyYQwL3DD2G/sm6FDsnMzGwdTvabKFv+dvQFg9jF5W/NzKyJcrLfBP94ZS5fv3cyfbt35M4LXP7WzMyaNif7jTRmwvv84KGX2b9PN25z+VszM2sGnOwbKVv+9ojde3LD2S5/a2ZmzYOzVSPU1ARX/f11Rj79DifuvyPXnLo/7dq4/K2ZmTUPTvYNWFNdw/cfeIk/T57FeYf14ycnuPytmZk1L072G5Atf/udz+7OJUe5/K2ZmTU/TvbrsWj5Gi64YyKT3l/AL764D2cf4vK3ZmbWPDnZ1+Ojpas469bxvF25lGvPOIjP7+fyt2Zm1nw52dejc4e29O+xDT/6/AA+sZvL35qZWfPmZF+Pdm1accPZBxc6DDMzsy3C94+ZmZkVOSd7MzOzIudkb2ZmVuSc7M3MzIpcXpO9pOMkTZU0TdJl9czvK2mcpJckPSmpLDPvXElvpY9z8xmnmZlZMctbspfUGrgOOB4YAJwhaUDOYtcAoyNiP+BK4Op03W2BnwKDgUHATyWV5itWMzOzYpbPI/tBwLSImB4Rq4ExwEk5ywwAxqXDT2TmHwv8MyLmR8QC4J/AcXmM1czMrGjlM9n3BmZkxmem07KmAEPS4ZOBzpK6N3JdJA2XVCGporKycosFbmZmVkzyWVSnvh5jImf8UuBaSecBTwGzgKpGrktE3AzcDCCpUtJ7mxNwE9YDmFfoIJoYb5N1eZvU5e2xLm+TdTX3bdKojlvymexnAn0y42XA7OwCETEbOAVAUidgSEQskjQTODJn3Sc39GIR0XPzQ26aJFVERHmh42hKvE3W5W1Sl7fHurxN1tVStkk+m/EnArtJ6i+pHXA68Eh2AUk9JNXGcDkwKh1+DDhGUml6Yd4x6TQzMzPbSHlL9hFRBVxCkqRfB8ZGxKuSrpR0YrrYkcBUSW8CvYCr0nXnAz8n+cEwEbgynWZmZmYbKa8d4UTE34G/50z7SWb4AeCB9aw7irVH+i3dzYUOoAnyNlmXt0ld3h7r8jZZV4vYJopY57o3MzMzKyIul2tmZlbknOzNzMyKnJN9EyOpj6QnJL0u6VVJ30inbyvpn2lfAf9saeWDJbWWNFnSo+l4f0nj0+1xX3rHR4shqZukByS9ke4rh3of0bfSz8wrku6V1KGl7SeSRkn6UNIrmWn17hdK/E/ad8lLkg4qXOT5s55t8tv0s/OSpIckdcvMuzzdJlMlHVuYqLc8J/umpwr4TkTsBRwCfC3tU+AyYFxE7EZSYnidjoWK3DdI7uqo9WvgD+n2WABcUJCoCudPwD8iYk9gf5Jt02L3EUm9ga8D5RGxD9Ca5Hbflraf3M66pcXXt18cD+yWPoYDN2ylGLe221l3m/wT2Cftl+VNklu/Sb9rTwf2Tte5Pu3npdlzsm9iImJORLyQDi8h+RLvTdJvwB3pYncAXyxMhFtf2hvi54Fb03EBR7H2To6Wtj26AJ8CRgJExOqIWEgL3kdSbYASSW2AjsAcWth+EhFPAbm3Ka9vvziJpCOyiIjngW6Sdtg6kW499W2TiHg8vT0c4HmSwm2QbJMxEbEqIt4BppH089LsOdk3YZL6AQcC44FeETEHkh8EwHaFi2yr+yPwPaAmHe8OLMx8WOvtO6GI7QxUArelpzZulbQNLXgfiYhZJL1ovk+S5BcBk2jZ+0mt9e0XjeqDpAU4H/i/dLhot4mTfROVlg9+EPhmRCwudDyFIukE4MOImJSdXM+iLeke0jbAQcANEXEgsIwW1GRfn/Q89ElAf2BHYBuSZupcLWk/aUhL/xwh6Yckp07vrp1Uz2JFsU2c7JsgSW1JEv3dEfHndPIHtU1s6d8PCxXfVnY4cKKkd0m6ST6K5Ei/W9pcC/X0u1DkZgIzI2J8Ov4ASfJvqfsIwNHAOxFRGRFrgD8Dh9Gy95Na69svGuy/pJhJOhc4ATgr1hacKdpt4mTfxKTno0cCr0fE7zOzHgHOTYfPBR7e2rEVQkRcHhFlEdGP5MKZf0fEWcATwJfSxVrM9gCIiLnADEl7pJM+A7xGC91HUu8Dh0jqmH6GardJi91PMta3XzwCDE2vyj8EWFTb3F/sJB0HfB84MSKWZ2Y9Apwuqb2k/iQXL04oRIxbmivoNTGSPgH8F3iZteeof0By3n4ssBPJF9upLa2/AElHApdGxAmSdiY50t8WmAycHRGrChnf1iTpAJILFtsB04FhJD/eW+w+IulnwGkkzbKTga+QnG9tMfuJpHtJ+hzpAXwA/BT4C/XsF+mPomtJrjpfDgyLiIpCxJ1P69kmlwPtgY/SxZ6PiBHp8j8kOY9fRXIa9f9yn7M5crI3MzMrcm7GNzMzK3JO9mZmZkXOyd7MzKzIOdmbmZkVOSd7MzOzIudkb9YCSQpJv8uMXyrpigKGZGZ55GRv1jKtAk6R1GNLPmlaoMXfK2ZNjD+UZi1TFXAz8K3cGZJ6SnpQ0sT0cXg6/QpJl2aWe0VSv/TxuqTrgReAPpLOkPRyusyvM+sslXSVpCmSnpfUK51+arrsFElP5fvNm7U0TvZmLdd1wFmSuuZM/xNJH/ADgSGkXQs3YA+S7lIPBNaQ9CN/FHAAMFBSbbeq25BUK9sfeAq4MJ3+E+DYdPqJm/GezKwebRpexMyKUUQsljQa+DqwIjPraGBAUk0VgC6SOjfwdO+lfaIDDASejIhKAEl3A58iKdu6Gng0XW4S8Nl0+BngdkljSTqxMbMtyMnerGX7I0nT+22Zaa2AQyMi+wMASVXUbQ3skBlell10A6+3JtPDWDXpd1BEjJA0GPg88KKkAyLio/U9iZltHDfjm7VgaUc5Y4ELMpMfBy6pHUk73QF4l6QrXSQdRNJ3fH3GA0dI6iGpNXAG8J8NxSFpl4gYHxE/AeZRt5tRM9tMTvZm9juSHsFqfR0ol/SSpNeAEen0B4FtJb0IXAy8Wd+Tpd2kXk7SvewU4IWIaKhr2d/WXtBHci5/yia/GzNbh3u9MzMzK3I+sjczMytyTvZmZmZFzsnezMysyDnZm5mZFTknezMzsyLnZG9mZlbknOzNzMyK3P8H50MEJpMRpFsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Modify function to use 6-fold cross-validation\n",
    "def cross_validate_nn(neural_arch):\n",
    "    fold_accuracies = []\n",
    "    kf = KFold(n_splits = 6, random_state = 2)\n",
    "    for train_index, test_index in kf.split(data):\n",
    "        train_features, test_features = data.loc[train_index], data.loc[test_index]\n",
    "        train_labels, test_labels = labels.loc[train_index], labels.loc[test_index]\n",
    "        \n",
    "        model = train_nn(neural_arch, train_features, train_labels)\n",
    "        accuracy = test_nn(model, test_features, test_labels)\n",
    "        fold_accuracies.append(accuracy)\n",
    "        \n",
    "    return fold_accuracies\n",
    "\n",
    "# Three-layer network list\n",
    "neural_layers_three = [\n",
    "    (10,10,10),\n",
    "    (64,64,64),\n",
    "    (128,128,128)\n",
    "]\n",
    "\n",
    "neural_accuracies_three = []\n",
    "\n",
    "# Run function pipeline\n",
    "for n in neural_layers_three:\n",
    "    accuracy = cross_validate_nn(n)\n",
    "    mean_accuracy = np.mean(accuracy)\n",
    "    neural_accuracies_three.append(mean_accuracy)\n",
    "\n",
    "x = [i[0] for i in neural_layers_three]\n",
    "\n",
    "plt.figure(figsize = (8,4))\n",
    "plt.plot(x, neural_accuracies_three)\n",
    "plt.title(\"Mean Accuracies vs. Number of Three-Layer Neurons\")\n",
    "plt.xlabel(\"Neurons\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9026142697881828, 0.9510330732069862, 0.9588127090301004]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neural_accuracies_three"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "Using three hidden layers improved accuracy to nearly 96%, even with 6-fold cross validation and only 128 neurons. This seems to be in line with the research literature out there about deep neural networks for computer vision. Having more layers and more neurons tends to improve the network's performance."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}