pytorch-lightning_ipynb/cnn/cnn-lenet5-quickdraw.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8382b5bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch            : 1.10.1\n",
      "pytorch_lightning: 1.6.0.dev0\n",
      "torchmetrics     : 0.6.2\n",
      "matplotlib       : 3.3.4\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -p torch,pytorch_lightning,torchmetrics,matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "50d23702",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext pycodestyle_magic\n",
    "%flake8_on --ignore W291,W293,E703"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9672fdf6",
   "metadata": {},
   "source": [
    "<a href=\"https://pytorch.org\"><img src=\"https://raw.githubusercontent.com/pytorch/pytorch/master/docs/source/_static/img/pytorch-logo-dark.svg\" width=\"90\"/></a> &nbsp; &nbsp;&nbsp;&nbsp;<a href=\"https://www.pytorchlightning.ai\"><img src=\"https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/docs/source/_static/images/logo.svg\" width=\"150\"/></a>\n",
    "\n",
    "# Model Zoo -- LeNet-5 Trained on QuickDraw"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ecbe454",
   "metadata": {},
   "source": [
    "This notebook implements the classic LeNet-5 convolutional network [1] and applies it to MNIST digit classification. The basic architecture is shown in the figure below:\n",
    "\n",
    "![](../../pytorch_ipynb/images/lenet/lenet-5_1.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8855551c",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "LeNet-5 is commonly regarded as the pioneer of convolutional neural networks, consisting of a very simple architecture (by modern standards). In total, LeNet-5 consists of only 7 layers. 3 out of these 7 layers are convolutional layers (C1, C3, C5), which are connected by two average pooling layers (S2 & S4). The penultimate layer is a fully connexted layer (F6), which is followed by the final output layer. The additional details are summarized below:\n",
    "\n",
    "- All convolutional layers use 5x5 kernels with stride 1.\n",
    "- The two average pooling (subsampling) layers are 2x2 pixels wide with stride 1.\n",
    "- Throughrout the network, tanh sigmoid activation functions are used. (**In this notebook, we replace these with ReLU activations**)\n",
    "- The output layer uses 10 custom Euclidean Radial Basis Function neurons for the output layer. (**In this notebook, we replace these with softmax activations**)\n",
    "- The expected input size is 32x32; so, here, we rescale the Quickdraw images from 28x28 to 32x32 to match this input dimension. Alternatively, we would have to change the \n",
    "achieve error rate below 1% on the MNIST data set, which was very close to the state of the art at the time (produced by a boosted ensemble of three LeNet-4 networks).\n",
    "\n",
    "\n",
    "### References\n",
    "\n",
    "- [1] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. [Gradient-based learning applied to document recognition](https://ieeexplore.ieee.org/document/726791). Proceedings of the IEEE, november 1998."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "244b1b58",
   "metadata": {},
   "source": [
    "## General settings and hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b0d68bc",
   "metadata": {},
   "source": [
    "- Here, we specify some general hyperparameter values and general settings\n",
    "- Note that for small datatsets, it is not necessary and better not to use multiple workers as it can sometimes cause issues with too many open files in PyTorch. So, if you have problems with the data loader later, try setting `NUM_WORKERS = 0` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "96da32cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "NUM_EPOCHS = 10\n",
    "LEARNING_RATE = 0.001\n",
    "NUM_WORKERS = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20a830a6",
   "metadata": {},
   "source": [
    "## Implementing a Neural Network using PyTorch Lightning's `LightningModule`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1155c0e2",
   "metadata": {},
   "source": [
    "- In this section, we set up the main model architecture using the `LightningModule` from PyTorch Lightning.\n",
    "- We start with defining our neural network  model in pure PyTorch, and then we use it in the `LightningModule` to get all the extra benefits that PyTorch Lightning provides."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "01d25119",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "class PyTorchLeNet5(torch.nn.Module):\n",
    "\n",
    "    def __init__(self, num_classes, grayscale=False):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.grayscale = grayscale\n",
    "        self.num_classes = num_classes\n",
    "\n",
    "        if self.grayscale:\n",
    "            in_channels = 1\n",
    "        else:\n",
    "            in_channels = 3\n",
    "\n",
    "        self.features = torch.nn.Sequential(\n",
    "            torch.nn.Conv2d(in_channels, 6, kernel_size=5),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.MaxPool2d(kernel_size=2),\n",
    "            torch.nn.Conv2d(6, 16, kernel_size=5),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.MaxPool2d(kernel_size=2)\n",
    "        )\n",
    "\n",
    "        self.classifier = torch.nn.Sequential(\n",
    "            torch.nn.Linear(16*5*5, 120),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.Linear(120, 84),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.Linear(84, num_classes),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.features(x)\n",
    "        x = torch.flatten(x, start_dim=1)\n",
    "        logits = self.classifier(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ee923af2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pytorch_lightning as pl\n",
    "import torchmetrics\n",
    "\n",
    "\n",
    "# LightningModule that receives a PyTorch model as input\n",
    "class LightningModel(pl.LightningModule):\n",
    "    def __init__(self, model, learning_rate):\n",
    "        super().__init__()\n",
    "\n",
    "        self.learning_rate = learning_rate\n",
    "        # The inherited PyTorch module\n",
    "        self.model = model\n",
    "\n",
    "        # Save settings and hyperparameters to the log directory\n",
    "        # but skip the model parameters\n",
    "        self.save_hyperparameters(ignore=['model'])\n",
    "\n",
    "        # Set up attributes for computing the accuracy\n",
    "        self.train_acc = torchmetrics.Accuracy()\n",
    "        self.valid_acc = torchmetrics.Accuracy()\n",
    "        self.test_acc = torchmetrics.Accuracy()\n",
    "        \n",
    "    # Defining the forward method is only necessary \n",
    "    # if you want to use a Trainer's .predict() method (optional)\n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "        \n",
    "    # A common forward step to compute the loss and labels\n",
    "    # this is used for training, validation, and testing below\n",
    "    def _shared_step(self, batch):\n",
    "        features, true_labels = batch\n",
    "        logits = self(features)\n",
    "        loss = torch.nn.functional.cross_entropy(logits, true_labels)\n",
    "        predicted_labels = torch.argmax(logits, dim=1)\n",
    "\n",
    "        return loss, true_labels, predicted_labels\n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"train_loss\", loss)\n",
    "        \n",
    "        # To account for Dropout behavior during evaluation\n",
    "        self.model.eval()\n",
    "        with torch.no_grad():\n",
    "            _, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.train_acc.update(predicted_labels, true_labels)\n",
    "        self.log(\"train_acc\", self.train_acc, on_epoch=True, on_step=False)\n",
    "        self.model.train()\n",
    "        return loss  # this is passed to the optimzer for training\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"valid_loss\", loss)\n",
    "        self.valid_acc(predicted_labels, true_labels)\n",
    "        self.log(\"valid_acc\", self.valid_acc,\n",
    "                 on_epoch=True, on_step=False, prog_bar=True)\n",
    "\n",
    "    def test_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.test_acc(predicted_labels, true_labels)\n",
    "        self.log(\"test_acc\", self.test_acc, on_epoch=True, on_step=False)\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n",
    "        return optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edc328e9",
   "metadata": {},
   "source": [
    "## Setting up the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6bd485d",
   "metadata": {},
   "source": [
    "- In this section, we are going to set up our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40a1f6e6",
   "metadata": {},
   "source": [
    "- Here, we are going to use Google's Quickdraw dataset (https://quickdraw.withgoogle.com). \n",
    "- In particular we will be working with an arbitrary subset of 10 categories in PNG format:\n",
    "\n",
    "    label_dict = {\n",
    "             \"lollipop\": 0,\n",
    "             \"binoculars\": 1,\n",
    "             \"mouse\": 2,\n",
    "             \"basket\": 3,\n",
    "             \"penguin\": 4,\n",
    "             \"washing machine\": 5,\n",
    "             \"canoe\": 6,\n",
    "             \"eyeglasses\": 7,\n",
    "             \"beach\": 8,\n",
    "             \"screwdriver\": 9,\n",
    "    }\n",
    "    \n",
    "For more details on obtaining and preparing the dataset, please see the\n",
    "\n",
    "- [custom-data-loader-quickdraw.ipynb](../../pytorch_ipynb/mechanics/custom-data-loader-quickdraw.ipynb)\n",
    "\n",
    "notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cfe6405",
   "metadata": {},
   "source": [
    "### Inspecting the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0ce1ffa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fddd2d42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 28)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAOk0lEQVR4nO3df6jVdZ7H8dc704zGfrje1Pyxd3aIWCnWkZNsGdISK/2AMqgYKTGINCiYCaWVWWr8S6x2lIViwCkb26YmYYysbHaiopqQ6Ciu2spuGq5eveqVpBoJXfW9f9xvO1e938/3es73nO+x9/MBl3Pu930+9/v24Ot+zz2f7/d8zN0F4PvvvKobANAehB0IgrADQRB2IAjCDgRxfjt3NmbMGO/u7m7nLoFQdu3apUOHDtlgtabCbmY3S/pXScMkPefuy1KP7+7uVr1eb2aXABJqtVpureGX8WY2TNKzkm6RNEXSHDOb0ujPA9BazfzNPl3SDnf/wt2PSfqdpDvKaQtA2ZoJ+wRJewZ835NtO4WZzTezupnV+/r6mtgdgGY0E/bB3gQ449xbd1/p7jV3r3V1dTWxOwDNaCbsPZImDfh+oqR9zbUDoFWaCfunkq40sx+a2QhJP5G0rpy2AJSt4ak3dz9uZo9I+nf1T72tcvfPSusMQKmammd39/WS1pfUC4AW4nRZIAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgmhqFVcAjVmwYEFube7cucmxN9xwQ0P7bCrsZrZL0jeSTkg67u61Zn4egNYp48j+D+5+qISfA6CF+JsdCKLZsLukP5rZRjObP9gDzGy+mdXNrN7X19fk7gA0qtmwz3D3aZJukfSwmc08/QHuvtLda+5e6+rqanJ3ABrVVNjdfV92e1DSa5Kml9EUgPI1HHYzu8jMRn13X9IsSdvKagxAuZp5N36spNfM7Luf87K7/6GUroBz3JEjR5L1lStX5tamTJmSHNv2eXZ3/0LS3zU6HkB7MfUGBEHYgSAIOxAEYQeCIOxAEFzimnH3ZP26667Lrd11113JsYsWLWqoJ5y79u/f3/DYcePGldjJX3BkB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgmGfPvPHGG8n6J598klt77LHHym4H57gDBw40PHbs2LEldvIXHNmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjm2TPLli1L1q+66qrc2uzZs0vuBue6Zq5nZ54dQFMIOxAEYQeCIOxAEIQdCIKwA0EQdiCIMPPsH330UbK+YcOGZP25557LrZ13Hr8zcapmrmev7HPjzWyVmR00s20Dto02s3fM7PPs9rKWdAegNEM5JP1G0s2nbVss6V13v1LSu9n3ADpYYdjd/UNJX562+Q5Jq7P7qyXNLrctAGVr9I/Nse7eK0nZ7eV5DzSz+WZWN7N6X19fg7sD0KyWv7Pk7ivdvebuta6urlbvDkCORsN+wMzGS1J2e7C8lgC0QqNhXydpXnZ/nqTXy2kHQKsUzrOb2SuSbpQ0xsx6JP1C0jJJa8zsAUm7Jd3dyibL8OSTTybrRXOb9957b5nt4HuuaJ59xIgRubVLL7205G76FYbd3efklG4quRcALcSpX0AQhB0IgrADQRB2IAjCDgTxvbnEdfv27cn6+vXrk/WnnnoqWR85cuRZ94S4iqbeLr889wxzmVnZ7UjiyA6EQdiBIAg7EARhB4Ig7EAQhB0IgrADQXxv5tmXLl2arI8aNSpZf/DBB8tsB+e4nTt3Juv79u1L1puZZ28VjuxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EMQ5Nc++Z8+e3Nqrr76aHLto0aJk/ZJLLmmoJ0k6duxYsr53795kvaenJ1n/8svTl9o71axZs3JrF154YXLsuezw4cPJeur/xIsvvpgcW7SE9/Dhw5P1xx9/PFmfNm1ast4KHNmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IIiOmmcvugZ4zpy8BWWlEydOJMdu2rQpWb/22muT9dRc+P79+5NjW+3uu/NXzF6zZk0bOzmTu+fW3nrrreTYF154IVkvGn/06NHcWq1WS4697777kvWXXnopWb/zzjuT9auvvjpZb4XCI7uZrTKzg2a2bcC2JWa218w2Z1+3trZNAM0aysv430i6eZDtK9x9avaVXm4FQOUKw+7uH0pKn68JoOM18wbdI2a2JXuZf1neg8xsvpnVzaze19fXxO4ANKPRsP9K0o8kTZXUK+mXeQ9095XuXnP3WldXV4O7A9CshsLu7gfc/YS7n5T0a0nTy20LQNkaCruZjR/w7Z2StuU9FkBnKJxnN7NXJN0oaYyZ9Uj6haQbzWyqJJe0S9KCMppZt25dsv7xxx/n1i644ILk2KJrwidOnJisX3/99Q2PnTBhQrI+efLkZP3pp59O1rdu3Zqst1JqHl2Sbrrpptza+++/nxxb9Lw++uijyfrcuXNza1OmTEmOffPNN5P1onn2kydPJutVKAy7uw92JsvzLegFQAtxuiwQBGEHgiDsQBCEHQiCsANBdNQlrkWXHaYsXrw4WV+yZEnDP7tqK1asSNZ37NiRW7vnnnuSY4cNG5asX3zxxcl66uO9pfT02rPPPpsc+9BDDyXr553Hseps8GwBQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBAdNc9+6NChhsfedtttJXbSWYouIz1+/HhubefOncmxR44cSdaLlqP+9ttvk/WUGTNmJOvMo5eLZxMIgrADQRB2IAjCDgRB2IEgCDsQBGEHguioefaia6dTvv766xI76SxFK+lMmjQpt7Zx48ay2znFe++9l6ynPkq6mTn6oUj9/MOHDyfHFv27hg8fnqwXfQx2FTiyA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQHTXPnpovLrJly5ZkPTXf22pfffVVsl702etF/7aiJaFbaeTIkQ2PLTo34uWXX07Wn3nmmWR9w4YNZ93TUM2ePTtZHz16dMv23ajCI7uZTTKz981su5l9ZmY/zbaPNrN3zOzz7Pay1rcLoFFDeRl/XNJCd/9bSX8v6WEzmyJpsaR33f1KSe9m3wPoUIVhd/ded9+U3f9G0nZJEyTdIWl19rDVkma3qEcAJTirN+jMrFvSjyV9Immsu/dK/b8QJF2eM2a+mdXNrN7X19dkuwAaNeSwm9kPJP1e0s/cfchXnbj7SnevuXut6IIOAK0zpLCb2XD1B/237r4223zAzMZn9fGSDramRQBlKJx6MzOT9Lyk7e6+fEBpnaR5kpZlt68328wVV1yRrM+cOTO3tnDhwuTYtWvXJuv9/8x8vb29ubW9e/cmx7b6Us6lS5e29OenTJ48OVlPPa+33357cuzRo0eT9WuuuSZZX7ZsWW5t3LhxybFFU2dFH4PdiYYyzz5D0lxJW81sc7bt5+oP+Roze0DSbkl3t6RDAKUoDLu7/0lS3q/n6s5UAXBWOF0WCIKwA0EQdiAIwg4EQdiBIDrqEtcib7/9dm5t+fLluTVJ+uCDD5L1889PPxWpedWi8wPGjx+frBddojpt2rRkvbu7O1lvpaKPTH7iiSdya7t3706Ovf/++5P11HkXOBNHdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0Iwty9bTur1Wper9fbtj8gmlqtpnq9PuhVqhzZgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IIjCsJvZJDN738y2m9lnZvbTbPsSM9trZpuzr1tb3y6ARg1lkYjjkha6+yYzGyVpo5m9k9VWuPu/tK49AGUZyvrsvZJ6s/vfmNl2SeklTAB0nLP6m93MuiX9WNIn2aZHzGyLma0ys8tyxsw3s7qZ1fv6+prrFkDDhhx2M/uBpN9L+pm7fy3pV5J+JGmq+o/8vxxsnLuvdPeau9e6urqa7xhAQ4YUdjMbrv6g/9bd10qSux9w9xPuflLSryVNb12bAJo1lHfjTdLzkra7+/IB2wcuTXqnpG3ltwegLEN5N36GpLmStprZ5mzbzyXNMbOpklzSLkkLWtAfgJIM5d34P0ka7HOo15ffDoBW4Qw6IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEObu7duZWZ+k/xmwaYykQ21r4Ox0am+d2pdEb40qs7e/dvdBP/+trWE/Y+dmdXevVdZAQqf21ql9SfTWqHb1xst4IAjCDgRRddhXVrz/lE7trVP7kuitUW3prdK/2QG0T9VHdgBtQtiBICoJu5ndbGb/ZWY7zGxxFT3kMbNdZrY1W4a6XnEvq8zsoJltG7BttJm9Y2afZ7eDrrFXUW8dsYx3YpnxSp+7qpc/b/vf7GY2TNJ/S/pHST2SPpU0x93/s62N5DCzXZJq7l75CRhmNlPSnyW96O5XZ9uekvSluy/LflFe5u7/1CG9LZH056qX8c5WKxo/cJlxSbMl3a8Kn7tEX/eoDc9bFUf26ZJ2uPsX7n5M0u8k3VFBHx3P3T+U9OVpm++QtDq7v1r9/1naLqe3juDuve6+Kbv/jaTvlhmv9LlL9NUWVYR9gqQ9A77vUWet9+6S/mhmG81sftXNDGKsu/dK/f95JF1ecT+nK1zGu51OW2a8Y567RpY/b1YVYR9sKalOmv+b4e7TJN0i6eHs5SqGZkjLeLfLIMuMd4RGlz9vVhVh75E0acD3EyXtq6CPQbn7vuz2oKTX1HlLUR/4bgXd7PZgxf38v05axnuwZcbVAc9dlcufVxH2TyVdaWY/NLMRkn4iaV0FfZzBzC7K3jiRmV0kaZY6bynqdZLmZffnSXq9wl5O0SnLeOctM66Kn7vKlz9397Z/SbpV/e/I75T0z1X0kNPX30j6j+zrs6p7k/SK+l/W/a/6XxE9IOmvJL0r6fPsdnQH9fZvkrZK2qL+YI2vqLcb1P+n4RZJm7OvW6t+7hJ9teV543RZIAjOoAOCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIP4PUkBl65Uqj0EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "import pandas as pd\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "df = pd.read_csv('quickdraw_png_set1_train.csv', index_col=0)\n",
    "df.head()\n",
    "\n",
    "main_dir = 'quickdraw-png_set1/'\n",
    "\n",
    "img = Image.open(os.path.join(main_dir, df.index[2]))\n",
    "img = np.asarray(img, dtype=np.uint8)\n",
    "print(img.shape)\n",
    "plt.imshow(np.array(img), cmap='binary')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0acc2af0",
   "metadata": {},
   "source": [
    "### Creating a custom Dataset and DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "433bb7ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import transforms\n",
    "\n",
    "\n",
    "class QuickdrawDataset(Dataset):\n",
    "    \"\"\"Custom Dataset for loading Quickdraw images\"\"\"\n",
    "\n",
    "    def __init__(self, txt_path, img_dir, transform=None):\n",
    "    \n",
    "        df = pd.read_csv(txt_path, sep=\",\", index_col=0)\n",
    "        self.img_dir = img_dir\n",
    "        self.txt_path = txt_path\n",
    "        self.img_names = df.index.values\n",
    "        self.y = df['Label'].values\n",
    "        self.transform = transform\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        img = Image.open(os.path.join(self.img_dir,\n",
    "                                      self.img_names[index]))\n",
    "        \n",
    "        if self.transform is not None:\n",
    "            img = self.transform(img)\n",
    "        \n",
    "        label = self.y[index]\n",
    "        return img, label\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.y.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "993c35e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_transform = transforms.Compose([transforms.ToTensor()])\n",
    "\n",
    "train_dataset = QuickdrawDataset(txt_path='quickdraw_png_set1_train.csv',\n",
    "                                 img_dir='quickdraw-png_set1/',\n",
    "                                 transform=custom_transform)\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset,\n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          shuffle=True,\n",
    "                          num_workers=NUM_WORKERS) \n",
    "\n",
    "valid_dataset = QuickdrawDataset(txt_path='quickdraw_png_set1_valid.csv',\n",
    "                                 img_dir='quickdraw-png_set1/',\n",
    "                                 transform=custom_transform)\n",
    "\n",
    "valid_loader = DataLoader(dataset=valid_dataset,\n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          shuffle=False,\n",
    "                          num_workers=NUM_WORKERS) \n",
    "\n",
    "test_dataset = QuickdrawDataset(txt_path='quickdraw_png_set1_test.csv',\n",
    "                                img_dir='quickdraw-png_set1/',\n",
    "                                transform=custom_transform)\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset,\n",
    "                         batch_size=BATCH_SIZE,\n",
    "                         shuffle=False,\n",
    "                         num_workers=NUM_WORKERS) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7a921ec4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training label distribution:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(0, 90149),\n",
       " (1, 87122),\n",
       " (2, 125242),\n",
       " (3, 82893),\n",
       " (4, 178028),\n",
       " (5, 84541),\n",
       " (6, 86707),\n",
       " (7, 158087),\n",
       " (8, 87143),\n",
       " (9, 81109)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "\n",
    "train_counter = Counter()\n",
    "for images, labels in train_loader:\n",
    "    train_counter.update(labels.tolist())\n",
    "\n",
    "print('\\nTraining label distribution:')\n",
    "sorted(train_counter.items(), key=lambda pair: pair[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d9bc4966",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Validation label distribution:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(0, 12944),\n",
       " (1, 12260),\n",
       " (2, 17938),\n",
       " (3, 11773),\n",
       " (4, 25347),\n",
       " (5, 12089),\n",
       " (6, 12220),\n",
       " (7, 22679),\n",
       " (8, 12657),\n",
       " (9, 11668)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_counter = Counter()\n",
    "for images, labels in valid_loader:\n",
    "    valid_counter.update(labels.tolist())\n",
    "\n",
    "print('\\nValidation label distribution:')\n",
    "sorted(valid_counter.items(), key=lambda pair: pair[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6500b4c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Test label distribution:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(0, 25756),\n",
       " (1, 24808),\n",
       " (2, 35646),\n",
       " (3, 23792),\n",
       " (4, 50416),\n",
       " (5, 24221),\n",
       " (6, 24840),\n",
       " (7, 44996),\n",
       " (8, 25138),\n",
       " (9, 23536)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_counter = Counter()\n",
    "for images, labels in test_loader:\n",
    "    test_counter.update(labels.tolist())\n",
    "\n",
    "print('\\nTest label distribution:')\n",
    "sorted(test_counter.items(), key=lambda pair: pair[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b2b8c34",
   "metadata": {},
   "source": [
    "### Performance baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f525eb46",
   "metadata": {},
   "source": [
    "- Especially for imbalanced datasets, it's quite useful to compute a performance baseline.\n",
    "- In classification contexts, a useful baseline is to compute the accuracy for a scenario where the model always predicts the majority class -- you want your model to be better than that!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a7f564f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4, 50416)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "majority_class = test_counter.most_common(1)[0]\n",
    "majority_class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "88a73add",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy when always predicting the majority class:\n",
      "0.17 (16.63%)\n"
     ]
    }
   ],
   "source": [
    "baseline_acc = majority_class[1] / sum(test_counter.values())\n",
    "print('Accuracy when always predicting the majority class:')\n",
    "print(f'{baseline_acc:.2f} ({baseline_acc*100:.2f}%)')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45b342bb",
   "metadata": {},
   "source": [
    "### Setting up a `DataModule`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3568e334",
   "metadata": {},
   "source": [
    "- There are three main ways we can prepare the dataset for Lightning. We can\n",
    "  1. make the dataset part of the model;\n",
    "  2. set up the data loaders as usual and feed them to the fit method of a Lightning Trainer -- the Trainer is introduced in the next subsection;\n",
    "  3. create a LightningDataModule.\n",
    "- Here, we are going to use approach 3, which is the most organized approach. The `LightningDataModule` consists of several self-explanatory methods as we can see below:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d9dc13c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from torch.utils.data.dataset import random_split\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "class DataModule(pl.LightningDataModule):\n",
    "    def __init__(self, data_path='./'):\n",
    "        super().__init__()\n",
    "        self.data_path = data_path\n",
    "        \n",
    "    def prepare_data(self):\n",
    "        # We assume the dataset is already downloaded, otherwise\n",
    "        # put the code for downloading it here\n",
    "        self.resize_transform = transforms.Compose(\n",
    "            [transforms.Resize((32, 32)),\n",
    "             transforms.ToTensor()])\n",
    "        return\n",
    "\n",
    "    def setup(self, stage=None):\n",
    "        # Note transforms.ToTensor() scales input images\n",
    "        # to 0-1 range\n",
    "        self.train = QuickdrawDataset( \n",
    "            txt_path=os.path.join(\n",
    "                self.data_path, 'quickdraw_png_set1_train.csv'),\n",
    "            img_dir=os.path.join(self.data_path, 'quickdraw-png_set1/'),\n",
    "            transform=self.resize_transform)\n",
    "\n",
    "        self.valid = QuickdrawDataset( \n",
    "            txt_path=os.path.join(\n",
    "                self.data_path, 'quickdraw_png_set1_valid.csv'),\n",
    "            img_dir=os.path.join(self.data_path, 'quickdraw-png_set1/'),\n",
    "            transform=self.resize_transform)        \n",
    "\n",
    "        self.test = QuickdrawDataset( \n",
    "            txt_path=os.path.join(\n",
    "                self.data_path, 'quickdraw_png_set1_test.csv'),\n",
    "            img_dir=os.path.join(self.data_path, 'quickdraw-png_set1/'),\n",
    "            transform=self.resize_transform)\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        train_loader = DataLoader(dataset=self.train, \n",
    "                                  batch_size=BATCH_SIZE, \n",
    "                                  drop_last=True,\n",
    "                                  shuffle=True,\n",
    "                                  num_workers=NUM_WORKERS)\n",
    "        return train_loader\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        valid_loader = DataLoader(dataset=self.valid, \n",
    "                                  batch_size=BATCH_SIZE, \n",
    "                                  drop_last=False,\n",
    "                                  shuffle=False,\n",
    "                                  num_workers=NUM_WORKERS)\n",
    "        return valid_loader\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        test_loader = DataLoader(dataset=self.test, \n",
    "                                 batch_size=BATCH_SIZE, \n",
    "                                 drop_last=False,\n",
    "                                 shuffle=False,\n",
    "                                 num_workers=NUM_WORKERS)\n",
    "        return test_loader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebb73cb1",
   "metadata": {},
   "source": [
    "- Note that the `prepare_data` method is usually used for steps that only need to be executed once, for example, downloading the dataset; the `setup` method defines the the dataset loading -- if you run your code in a distributed setting, this will be called on each node / GPU. \n",
    "- Next, lets initialize the `DataModule`; we use a random seed for reproducibility (so that the data set is shuffled the same way when we re-execute this code):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4bcb535f",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(1) \n",
    "data_module = DataModule(data_path='./')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2474ebe2",
   "metadata": {},
   "source": [
    "## Training the model using the PyTorch Lightning Trainer class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcf5df82",
   "metadata": {},
   "source": [
    "- Next, we initialize our model.\n",
    "- Also, we define a call back so that we can obtain the model with the best validation set performance after training.\n",
    "- PyTorch Lightning offers [many advanced logging services](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) like Weights & Biases. Here, we will keep things simple and use the `CSVLogger`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5efbb4d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_lightning.callbacks import ModelCheckpoint\n",
    "from pytorch_lightning.loggers import CSVLogger\n",
    "\n",
    "\n",
    "pytorch_model = PyTorchLeNet5(\n",
    "    num_classes=10, grayscale=True)\n",
    "\n",
    "lightning_model = LightningModel(\n",
    "    model=pytorch_model, learning_rate=LEARNING_RATE)\n",
    "\n",
    "callbacks = [ModelCheckpoint(\n",
    "    save_top_k=1, mode='max', monitor=\"valid_acc\")]  # save top 1 model \n",
    "logger = CSVLogger(save_dir=\"logs/\", name=\"my-model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e56fece3",
   "metadata": {},
   "source": [
    "- Now it's time to train our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "114513ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jovyan/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:90: LightningDeprecationWarning: Setting `Trainer(progress_bar_refresh_rate=50)` is deprecated in v1.5 and will be removed in v1.7. Please pass `pytorch_lightning.callbacks.progress.TQDMProgressBar` with `refresh_rate` directly to the Trainer's `callbacks` argument instead. Or, to disable the progress bar pass `enable_progress_bar = False` to the Trainer.\n",
      "  rank_zero_deprecation(\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name      | Type          | Params\n",
      "--------------------------------------------\n",
      "0 | model     | PyTorchLeNet5 | 61.7 K\n",
      "1 | train_acc | Accuracy      | 0     \n",
      "2 | valid_acc | Accuracy      | 0     \n",
      "3 | test_acc  | Accuracy      | 0     \n",
      "--------------------------------------------\n",
      "61.7 K    Trainable params\n",
      "0         Non-trainable params\n",
      "61.7 K    Total params\n",
      "0.247     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation sanity check: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4e5225d94d144c9b87b02e0ccab245d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training took 21.72 min in total.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "\n",
    "trainer = pl.Trainer(\n",
    "    max_epochs=NUM_EPOCHS,\n",
    "    callbacks=callbacks,\n",
    "    progress_bar_refresh_rate=50,  # recommended for notebooks\n",
    "    accelerator=\"auto\",  # Uses GPUs or TPUs if available\n",
    "    devices=\"auto\",  # Uses all available GPUs/TPUs if applicable\n",
    "    logger=logger,\n",
    "    log_every_n_steps=1000)\n",
    "\n",
    "start_time = time.time()\n",
    "trainer.fit(model=lightning_model, datamodule=data_module)\n",
    "\n",
    "runtime = (time.time() - start_time)/60\n",
    "print(f\"Training took {runtime:.2f} min in total.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a19c0dbe",
   "metadata": {},
   "source": [
    "## Evaluating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceaf63b5",
   "metadata": {},
   "source": [
    "- After training, let's plot our training ACC and validation ACC using pandas, which, in turn, uses matplotlib for plotting (you may want to consider a [more advanced logger](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) that does that for you):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7d870b0f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Epoch', ylabel='ACC'>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABBkUlEQVR4nO3dd3wVVfr48c+TTholjZAACaGEHiD0lqAooIJYEAuoKyKurG11LfvdXUV3f+6urq4rgshiX7FiA0SFBESQKkKABEIPLYWWAOnn98dc4CYESMK9uSnP+/W6r8ydmTP3mUO4T+bMmXPEGINSSilVWW6uDkAppVTdoolDKaVUlWjiUEopVSWaOJRSSlWJJg6llFJV4uHqAGpCcHCwiYqKqlbZkydP4ufn59iA6jCtj3O0LsrS+iirPtTHunXrso0xIeXXN4jEERUVxdq1a6tVNjk5mYSEBMcGVIdpfZyjdVGW1kdZ9aE+RGRPReu1qUoppVSVaOJQSilVJZo4lFJKVUmDuMehlKpfioqKyMjIID8/39WhXFDjxo3ZunWrq8OoFB8fHyIjI/H09KzU/po4lFJ1TkZGBgEBAURFRSEirg6nQrm5uQQEBLg6jEsyxpCTk0NGRgbR0dGVKqNNVUqpOic/P5+goKBamzTqEhEhKCioSldvmjiUUnWSJg3HqWpdauK4iOS0TL7ZUejqMJRSqlbRxHERK3bkMC+9iLyCYleHopRStYYmjotI7BBKiYHl27NdHYpSqhY5duwYr7/+epXLjRo1imPHjlW53F133cWnn35a5XLOoonjIuKjmtLIA5JSM10dilKqFrlQ4igpKblouQULFtCkSRMnRVVztDvuRXi6u9E5yJ2ktEyMMXozTqla6NmvN7PlwAmHHrNTi0D+cl3nC25/8skn2bFjB3FxcXh6euLv7094eDgbNmxgy5YtXH/99ezZs4fCwkIeeughJk+eDJwbNy8vL4+RI0cyaNAgVqxYQUREBF9++SWNGjW6ZGyLFy/mscceo7i4mN69ezNjxgy8vb158skn+eqrr/Dw8OCqq67ixRdf5JNPPuHZZ5/F3d2dxo0bs2zZMofUjyaOS+ge4s7alAI2HzhBl4jGrg5HKVULvPDCC6SkpLBhwwaSk5O55pprSElJOfscxJw5c/D09MTDw4PevXtz4403EhQUVOYY27dv58MPP+TNN99k3LhxfPbZZ9xxxx0X/dz8/HzuuusuFi9eTPv27Zk4cSIzZsxg4sSJzJs3j9TUVETkbHPYtGnTWLRoEREREdVqIrsQTRyX0DXEHbB6WGniUKr2udiVQU3p06dPmYfnXn31VT777DPc3NzYt28f27dvPy9xREdHExcXB0CvXr3YvXv3JT8nLS2N6Oho2rdvD8Cdd97J9OnTmTp1Kj4+PkyaNIlrrrmGa6+9FoCBAwdy1113MW7cOG644QbHnCx6j+OSmni70S2yMUv0PodS6gLs591ITk7mhx9+4IcffuDXX3+lR48eFT5c5+3tfXbZ3d2d4uJL9940xlS43sPDg9WrV3PjjTfyxRdfMGLECABmzpzJ888/z759+4iLiyMnJ6eqp1YhTRyVkNAhlF/2HePISX2mQykFAQEB5ObmVrjt+PHjNG3aFF9fX1JTU/n5558d9rmxsbHs3r2b9PR0AN577z2GDh1KXl4ex48fZ9SoUbzyyits2LABgB07dtC3b1+mTZtGcHAw+/btc0gc2lRVCcNiQ3l18XaWbcvi+h4Rrg5HKeViQUFBDBw4kC5dutCoUSPCwsLObhsxYgQzZ86kf//+dOzYkX79+jnsc318fHjrrbe4+eabz94cnzJlCkeOHGHMmDHk5+djjOHll18G4PHHH2f79u0YY7jiiivo3r27Q+LQxFEJ3SIaE+TnxZLUTE0cSikA/ve//1W43tvbm4ULF1Y4yOGZ+xjBwcGkpKScXf/YY49d9LPefvvts8tXXHEFv/zyS5nt4eHhrF69+rxyn3/++UWPW13aVFUJbm7C0A4hLN2WRUlpxW2MSinVUGjiqKRhsaEcP13EL3uPujoUpVQ99cADDxAXF1fm9dZbb7k6rPNoU1UlDW4XgrubkJSWSXxUM1eHo5Sqh6ZPn+7qECpFrzgqqXEjT3q1bsqS1CxXh6KUUi7l1MQhIiNEJE1E0kXkyQq2jxGRjSKyQUTWisgg2/oOtnVnXidE5GHbtmdEZL/dtlHOPAd7w2JD2XrwBIeO197pKpVSytmcljhExB2YDowEOgG3ikincrstBrobY+KA3wCzAYwxacaYONv6XsApYJ5duZfPbDfGLHDWOZSX2CEUgKQ0fRhQKdVwOfOKow+QbozZaYwpBOYCY+x3MMbkmXOPQvoBFXVZugLYYYzZ48RYK6V9mD8RTRrpU+RKqQbNmYkjArB/TDHDtq4MERkrIqnAfKyrjvLGAx+WWzfV1sQ1R0SaOirgSxEREjqE8FN6NgXFFx8+WSml7Pn7+wNw4MABbrrppgr3SUhIYO3atRc8RlRUFNnZrp8fyJm9qioag/y8KwpjzDxgnogMAZ4Drjx7ABEvYDTwlF2RGbb9jO3nS1SQcERkMjAZICwsjOTk5GqdRF5eXpmyIUXFnCos4c0vkukS7F6tY9Zl5eujIdO6KKsm66Nx48YXHPKjtigpKTkvxjMPBb711lsVxl9SUsLJkycveG7GGPLy8sqMc+Uo+fn5lf73c2biyABa2r2PBA5caGdjzDIRiRGRYGPMmZQ6ElhvjDlst9/ZZRF5E/jmAsebBcwCiI+PNwkJCdU6ieTkZOzL9iks5vWN33PEO5yEhPK3bOq/8vXRkGldlFWT9bF169ZzT2UvfBIObXLsBzTvCiNfuOguTzzxBK1bt+a3v/0tAM888wwiwrJlyzh69CgFBQX87W9/Y8yYcy30AQEB7N69m2uvvZaUlBROnz7N3XffzZYtW+jYsSOFhYX4+fmd98T5GSKCv78/AQEB/Otf/2LOnDkATJo0iYcffpiTJ08ybtw4MjIyKCkp4U9/+hO33HJLhXN1lOfj40OPHj0qVT3OTBxrgHYiEg3sx2pyus1+BxFpi3X/wohIT8ALsB++8VbKNVOJSLgx5qDt7VgghRrk6+VB/zZBJKVl8ufrGl7iUEpZxo8fz8MPP3w2cXz88cd8++23PPLIIwQGBrJ7926uvPJKRo8efcFJ4GbMmIGvry8bN25k48aN9OzZs1KfvW7dOt566y1WrVqFMYa+ffsydOhQdu7cSYsWLZg/fz5gDbh45MiRCufquBxOSxzGmGIRmQosAtyBOcaYzSIyxbZ9JnAjMFFEioDTwC1nbpaLiC8wHLiv3KH/ISJxWE1VuyvY7nTDYkP5y1eb2ZV9kuhgv0sXUEo5zyWuDJylR48eZGZmcuDAAbKysmjatCnh4eE88sgjZ2fa279/P4cPH6Z58+YVHmPZsmU8+OCDAHTr1o1u3bpV6rOXL1/O2LFjzw7nfsMNN/Djjz8yYsQIHnvsMZ544gmuvfZaBg8eTHFxcYVzdVwOpz7HYYxZYIxpb4yJMcb81bZupi1pYIz5uzGms61bbX9jzHK7sqeMMUHGmOPljjnBGNPVGNPNGDPa7uqjxpztlqu9q5Rq0G666SY+/fRTPvroI8aPH88HH3xAVlYW69at46effiIsLKzCuTjsVWdK6gvNy9G+fXvWrVtH165deeqpp5g2bdoF5+q4HPrkeDW0CvIlJsRPn+dQqoEbP348c+fO5dNPP+Wmm27i+PHjhIaG4unpybJly9iz5+JPEQwZMoQPPvgAgJSUFDZu3Fipzx0yZAhffPEFp06d4uTJk8ybN4/Bgwdz4MABfH19ueOOO3jsscdYv379BefquBw6VlU1JXYI5d2VezhZUIyft1ajUg1R586dyc3NJSIigvDwcG6//Xauu+464uPj6dy5M7GxsRctf//993P33XfTrVs34uLi6NOnT6U+t2fPntx1111n9580aRI9evRg0aJFPP7447i5ueHp6cmMGTPIzc2tcK6Oy6HfeNU0LDaU2ct38VN6Nld1rrj9UilV/23adK5HV3BwMCtXrgQ4bz6OvLw8wHoW48xcHI0aNWLu3LmV/iz7eckfffRRHn300TLbr776aq6++urzylU0V8fl0KaqaoqPaoa/twdJaTrooVKqYdErjmry8nBjUNtgktMyMcZU6waXUkpVpG/fvhQUFJRZ995779G1a1cXRVSWJo7LMCw2lG83H2LrwVw6tQh0dThKNSj1+Q+2VatW1ejnXaiX1oVoU9VlSOgQAuhouUrVNB8fH3Jycqr8hafOZ4whJycHHx+fSpfRK47LEBroQ5eIQJJSM3kgsa2rw1GqwYiMjCQjI4OsrNp7jzE/P79KX8au5OPjQ2RkZKX318RxmRI7hDI9KZ1jpwpp4uvl6nCUahA8PT2Jjo52dRgXlZycXOmxn+oabaq6TImxoZQaWLqt9v7lo5RSjqSJ4zJ1j2xCMz8vkrVbrlKqgdDEcZnc3YSh7UNITsukpFRv1Cml6j9NHA6QGBvK0VNFbNh3zNWhKKWU02nicIAh7YJxE0jWbrlKqQZAE4cDNPH1olfrpizRYdaVUg2AJg4HSYwNZfOBExw+cfGx95VSqq7TxOEgZyZ30uYqpVR9p4nDQWKbBxDe2Eebq5RS9Z4mDgcRERI6hLJ8ezaFxaWuDkcppZzGqYlDREaISJqIpIvIkxVsHyMiG0Vkg4isFZFBdtt2i8imM9vs1jcTke9FZLvtZ1NnnkNVDIsN5WRhCWt2H3F1KEop5TROSxwi4g5MB0YCnYBbRaRTud0WA92NMXHAb4DZ5bYnGmPijDHxduueBBYbY9rZyp+XkFxlQEwQXu5uJGlzlVKqHnPmFUcfIN0Ys9MYUwjMBcbY72CMyTPnxkX2Ayrz6PUY4B3b8jvA9Y4J9/L5eXvQt00zlugNcqVUPebM0XEjgH127zOAvuV3EpGxwP8DQoFr7DYZ4DsRMcAbxphZtvVhxpiDAMaYgyISWtGHi8hkYDJAWFgYycnJ1TqJvLy8KpVt5V7Ej1mFfLxgCaG+9e8WUlXroz7TuihL66Os+lwfzkwcFU3Ndd4VhTFmHjBPRIYAzwFX2jYNNMYcsCWG70Uk1RizrLIfbks0swDi4+NNQkJCVeMHrKGRq1I2KvskH6Qmc6pxNAkDa/ewz9VR1fqoz7QuytL6KKs+14cz/yTOAFravY8EDlxoZ1tSiBGRYNv7A7afmcA8rKYvgMMiEg5g+1mr2oWigv1oE+zHEh0tVylVTzkzcawB2olItIh4AeOBr+x3EJG2Yps0WER6Al5Ajoj4iUiAbb0fcBWQYiv2FXCnbflO4EsnnkO1JHQI5eedOZwqLHZ1KEop5XBOSxzGmGJgKrAI2Ap8bIzZLCJTRGSKbbcbgRQR2YDVA+sW283yMGC5iPwKrAbmG2O+tZV5ARguItuB4bb3tcqw2FAKi0tZkZ7j6lCUUsrhnDp1rDFmAbCg3LqZdst/B/5eQbmdQPcLHDMHuMKxkTpW7+im+Hm5k5SWyZWdwlwdjlJKOVT96/ZTC3h7uDOwbTBJqZmc622slFL1gyYOJxkWG8qB4/mkHc51dShKKeVQmjicJME2Wm5SqvauUkrVL5o4nKR5Yx86hQfq8CNKqXpHE4cTJcaGsG7vUY6fKnJ1KEop5TCaOJxoWGwoJaWGZdu1uUopVX9o4nCiuJZNaeLrSZIOeqiUqkc0cTiRu5swtH0IS9OyKC3VbrlKqfpBE4eTDYsNJedkIb9mHHN1KEop5RCaOJxsSLsQ3ASSdNBDpVQ9oYnDyZr6edGjVVPtlquUqjc0cdSAYbGhbNp/nMzcfFeHopRSl00TRw1I6BACQLI2Vyml6gFNHDWgU3ggYYHeJGu3XKVUPaCJowaICIkdQvlxWzZFJaWuDkcppS6LJo4akhgbSm5BMWt2H3F1KEopdVk0cdSQQW2D8XQXvc+hlKrzNHHUED9vD/pGB7FEu+Uqpeo4pyYOERkhImkiki4iT1awfYyIbBSRDSKyVkQG2da3FJEkEdkqIptF5CG7Ms+IyH5bmQ0iMsqZ5+BIibGhpGfmse/IKVeHopRS1ea0xCEi7sB0YCTQCbhVRDqV220x0N0YEwf8BphtW18M/N4Y0xHoBzxQruzLxpg426vMnOa1WaKtW64OeqiUqsucecXRB0g3xuw0xhQCc4Ex9jsYY/LMuUm5/QBjW3/QGLPetpwLbAUinBhrjWgT4k9UkK82Vyml6jQPJx47Athn9z4D6Ft+JxEZC/w/IBS4poLtUUAPYJXd6qkiMhFYi3VlcrSCcpOByQBhYWEkJydX6yTy8vKqXbYi7fwLSd5+ikWLk/B2F4cdt6Y4uj7qMq2LsrQ+yqrX9WGMccoLuBmYbfd+AvCfi+w/BPih3Dp/YB1wg926MMAd62rpr8CcS8XSq1cvU11JSUnVLluRpWmZpvUT35jFWw859Lg1xdH1UZdpXZSl9VFWfagPYK2p4DvVmU1VGUBLu/eRwIEL7WyMWQbEiEgwgIh4Ap8BHxhjPrfb77AxpsQYUwq8idUkVmf0bdOMRp7uJKVqt1ylVN3kzMSxBmgnItEi4gWMB76y30FE2oqI2JZ7Al5Ajm3df4Gtxph/lSsTbvd2LJDixHNwOG8Pdwa2DWZJauaZKyillKpTnJY4jDHFwFRgEdbN7Y+NMZtFZIqITLHtdiOQIiIbsHpg3WK7PBqI1bQ1rIJut/8QkU0ishFIBB5x1jlQUoR3vuNvZA+LDWX/sdOkZ+Y5/NhKKeVszrw5jrG6yi4ot26m3fLfgb9XUG45UOGdY2PMBAeHeWHzH6VnyjfQpzc0i3bYYc+MlrskNZN2YQEOO65SStUEfXL8YvpOwa20EN4dAycueHumylo0aURs8wDtlquUqpM0cVxMWGc2dnsGTh2xkkee425oD4sNZe2eo5zIL3LYMZVSqiZo4riE3MB2cNtHcGwfvDcWTp/3yEi1JMaGUlJq+HFbtkOOp5RSNUUTR2VEDYTx70N2GnxwMxTkXvYhe7RsQuNGnjr8iFKqztHEUVltr4Sb3oL96+HDW6Ho9GUdzsPdjSHtQ0hOy6S0VLvlKqXqDk0cVdHxWhg7E3Yvh48nQnHhZR1uWGwI2XmFpBw47qAAlVLK+TRxVFW3cXDty7D9O/h8EpQUV/tQQ9qFIIL2rlJK1SmaOKoj/m64+m+w5Uv46ndQWr15xIP8vYlr2YQkTRxKqTpEE0d19X8AEp6GX/8HCx+Hag4fMqxDKL9mHCcrt8DBASqllHNo4rgcQ/8AAx6ENbPhh79UK3kkxoYCsHSbDnqolKobNHFcDhEYPg3i74Gf/g3LXqzyITq3CCQ0wFu75Sql6gynjlXVIIjAqBeh6BQkPQ9eftD/t1UoLiR0CGFhyiGKSkrxdNdcrpSq3fRbyhHc3GD0a9BxNCx6Cta9XaXiw2JDyc0vZt0exzyVrpRSzlSpxCEifiLiZltuLyKjbRMtqTPcPeDG/0Lb4fD1w7Dxk0oXHdg2GE930eYqpVSdUNkrjmWAj4hEAIuBu4G3nRVUneXhBbe8B1GDYN59kDq/UsUCfDzpHdVMu+UqpeqEyiYOMcacAm7Amjd8LNDJeWHVYZ6N4NYPoUUP+OQuSF9cqWLDYkPZdjiPjKOnnBufUkpdpkonDhHpD9wOnPkzWm+sX4h3ANzxKQR3gLm3w54VlyyS0MHqlpuUpt1ylVK1W2UTx8PAU8A82/SvbYAkp0VVHzRqChPmQeNI+GCcNTjiRcSE+NGqmS/J2lyllKrlKpU4jDFLjTGjjTF/t90kzzbGPHipciIyQkTSRCRdRJ6sYPsYEdlom1N8rYgMulRZEWkmIt+LyHbbz6aVPNea5x8CE78E36bw/g1weMsFdxUREjuE8NOObPKLSmowSKWUqprK9qr6n4gEiogfsAVIE5HHL1HGHZgOjMS6H3KriJS/L7IY6G6MiQN+A8yuRNkngcXGmHa28uclpFqlcQRM/Ao8fKxZBHN2XHDXxNhQ8otKWbkzpwYDVEqpqqlsU1UnY8wJ4HpgAdAKmHCJMn2AdGPMTmNMITAXGGO/gzEmz5iz43T4AaYSZccA79iW37HFVLs1i7auPEwJvDMaju2tcLd+bYLw8XTT5iqlVK1W2cThaXtu43rgS2NMEee+5C8kAthn9z7Dtq4MERkrIqlYN91/U4myYcaYgwC2n6GVPAfXCukAE76AwlzryiP30Hm7+Hi6MzAmmCVpmZhqDpqolFLOVtmeUW8Au4FfgWUi0ho4cYkyUsG6874NjTHzgHkiMgR4DriysmUv+uEik4HJAGFhYSQnJ1el+Fl5eXnVLluRwI5/pPuvfyZ/xnA2xP2VIq/AMtsj3IpYfKSQD+cn0cK/9j3Y7+j6qMu0LsrS+iirXteHMaZaL8DjEtv7A4vs3j8FPHWJMruA4IuVBdKAcNtyOJB2qVh79eplqispKanaZS9o51Jjngs1ZuZgY04fK7Mp4+gp0/qJb8yspTsc/7kO4JT6qKO0LsrS+iirPtQHsNZU8J1a2ZvjjUXkX7aeT2tF5CWsexIXswZoJyLRIuIFjAe+KnfctiIituWegBeQc4myXwF32pbvBL6szDnUKtFDYNy7cHiz1VW38OTZTRFNGtEhLEBnBVRK1VqVbQuZA+QC42yvE8BbFytgjCkGpgKLgK3Ax8Z6BmSKiEyx7XYjkCIiG7B6Ud1iS3QVlrWVeQEYLiLbgeG293VP+6vhxtmQsRrm3gZF+Wc3JcaGsmb3EXLzi1wYoFJKVayy9zhijDE32r1/1vZlf1HGmAVYvbDs1820W/478PfKlrWtzwGuqFzYtVznsVB0Gr64Hz6927oKcfcksUMIM5fuYPn2bEZ2DXd1lEopVUZlrzhOl3s4byBw2jkhNTBxt1nzeaQtsAZGLC2hV+umBPh46Gi5SqlaqbJXHFOAd0Wkse39Uc7dZ1CXq8+91n2OH/4Cnr54XPcqQ9qHkJSWRWmpwc2tok5mSinlGpVKHMaYX4HuIhJoe39CRB4GNjoxtoZl0MNQmAfL/glefgxrP5X5Gw+y5eAJukQ0vmRxpZSqKVV6UMAYc8JYT5ADPOqEeBq2xD9Cv9/CqpmMzP4vImjvKqVUrXM5T5hp+4mjicDVf4OeE/H9+WWebfadJg6lVK1zOXNq6JgYziAC174ChaeYmPI2O44ZcvLiCfL3dnVkSikFXCJxiEguFScIARo5JSIFbu4wdibHc0/w7J53WPtDO4Ku/52ro1JKKeASTVXGmABjTGAFrwBjjM4A6EzungTc/i4/S3d6bvgzrJwOp464OiqllLqsexzKydy8GvFV7D9YS0dY9DS82B4+uBk2fAj5x10dnlKqgdKrhlpuUKfWjPvlaebf5E/nI4th8zzYPgXcvaHtldDlBmg/Arz9XR2qUqqB0MRRyw1qF4yHmxvv7G7Kn6/7C/7Dp0HGWtj8OWz+AtLmg0cjaH+VNYRJu6vBy9fVYSul6jFNHLVcoI8nV3duzsdrM5j3y376RDcjsUMoCT3/SMxVzyP7VkHK57DlS+vl6QcdRkDnG6wrEk8fV5+CUqqe0cRRB7wyPo4J/VuTlJpJUlomz8/fyvPzt9KqmS+JHUJIiP0D/a/8Gz4HfraSyNavIOUz8A6EDqOs5qw2ieDh5epTUUrVA5o46gBPdzf6tQmiX5sgnhrVkYyjp0hKyyI5NZOP1u7jnZV78PF0Y0BMMIkdHibhN9NoeWyN1Zy19RvYOBd8GkPsddBlLEQPBXdPV5+WUqqO0sRRB0U29WVCv9ZM6Nea/KISft6ZQ3JaFktSM88+ad4u1J/E2Kkk3vBHepduxGPrF9aVyIb3oVEz6DTaas6KGmQ9N6KUUpWkiaOO8/F0J6FDKAkdQvnLdZ3YmX2SpNRMktOyeOunXcxaZvD39mBwu3u5IvExrvTaRJOd38DGT2Dd2+AXAp3GWEmkVX9w0x7aSqmL08RRj4gIMSH+xIT4M2lwG/IKivkpPZvktEySUrNYmHII8KJLxF0M7/kg1/ltJurQItx++QDWzIaAcOh0vXVPJLK3NfyJUkqVo4mjHvP39uDqzs25unNzjDFsPZhLUlomyWmZ/HtZBi+bxjT1vZ3hbSczLnAL3Y4twWvtHFg1Axq3tK5EutwALXpqElFKnaWJo4EQETq1CKRTi0AeSGzLsVOFLNueTXJqJj9sy+Ljky1wkzsYEPkbJjbbQv/TS/Ff9Qay8jVoGmU9I9L5BjClrj4VpZSLOTVxiMgI4N+AOzDbGPNCue23A0/Y3uYB9xtjfhWRDsBHdru2Af5sjHlFRJ4B7gWybNuets1Prqqgia8Xo7u3YHT3FpSUGjZmHLN6aqVlMvnXtkBb2gZM5L7wrSQULyf4p1eR5S8zFIFVgeDTxOqp1ajJBZabVrxeuwQrVec5LXGIiDswHRgOZABrROQrY8wWu912AUONMUdFZCQwC+hrjEkD4uyOsx+YZ1fuZWPMi86KvaFxdxN6tGpKj1ZNeXR4ezJz81malkVyWhbTtjXi8YJuhLpP4J6QNOK9M+jVuinkH7PGyzp9DLK3nVsuvsRU9J6+F0k6ja33F1r28tMms1rq25SDrNtfRIKrA1E1wplXHH2AdGPMTgARmQuMAc4mDmPMCrv9fwYiKzjOFcAOY8weJ8aq7IQG+HBzfEtujm9JUUkp6/YcJSktk7c3hPBqTj7f3JxAdLBfxYWLC6wEkn/cSi7nLR8ru/5EBhzebC0XXGLgRjePc1cz/s0hsIX1ahx5bjkwAnyDtXdYDfpg1R7+OC8Fd4G7j52mRROdcaG+E2OcMx+TiNwEjDDGTLK9n4B1NTH1Avs/BsSe2d9u/RxgvTHmNdv7Z4C7gBPAWuD3xpijFRxvMjAZICwsrNfcuXOrdR55eXn4++sAggBZp0p5ZsUpGnu78X/9GuHr6eC//k0JHsWn8Cg+iUdxHp5FebZl6731OolnUS5ehUfxLsjGu+AIbqa4zGFKxYMC72YUeAdT4B1ke5VdLvRqAnJ5z6/o7wYs2VvEu1sK6djMjbSjJVzRypPbO+qkY1A/fj8SExPXGWPiy6935hVHRd8qFWYpEUkE7gEGlVvvBYwGnrJbPQN4znas54CXgN+c90HGzMJq+iI+Pt4kJCRU+QQAkpOTqW7Z+ij79GJeWlfAp/v9eXNiPO5uLm46Ki2FU9lwYj+cOADH9+N2Yj+NThyg0YkD1hVNzmooKShbTtyt7sf2VyrllwOaX/QJ+4b+u/Huyt28u2UzV3YMZfrtPfnN6z+w/IDh7xMH0MxP72XV598PZyaODKCl3ftI4ED5nUSkGzAbGGmMySm3eSTW1cbhMyvsl0XkTeAbRwatLq5jkDvPjO7M/32Rwj8XpfHkyFjXBuTmBv6h1qtFj4r3McaaBOtMcinzcz8cToHt30HRqXIFBfzDKmgSsxKLd35WhR/XELz10y6e/XoLwzuFMf22nnh5uDGqjSc/HTjN2yt28+jw9q4OUTmRMxPHGqCdiERj3dweD9xmv4OItAI+ByYYY7ZVcIxbgQ/LlQk3xhy0vR0LpDg6cHVxd/RrTeqhE8xcuoPY5gFc3yPC1SFdnAj4BVmv8G4V72OMde/lxIEKkssByEmHnUuhMPdskf4Au16xdVUeC0ExNXAyrjf7x508P38rV3cO4z+3WkkDIMLfjas6hfHOit3cN6QNft7a27++ctq/rDGmWESmAouwuuPOMcZsFpEptu0zgT8DQcDrYvWWKT7TniYivlg9su4rd+h/iEgcVlPV7gq2qxrwl+s6k56Zxx8+20hUsB9xLZu4OqTLI2LddG/UFMI6X3i//BNnE0r6ym9oW7gZljxnvcK7Wwmk0/XQLLrGQq9Js5bt4G8LUhnZpTmv3toDT/eynRDuT4jhuy2H+XD1XiYNbuOiKJWzOfVPAtvzFQvKrZtptzwJmFS+nG3bKaykUn79BAeHqarB092N12/vxejXljP53bV8/btBhAU2gLk/fAKtV2gsGRnutE14GY7ts+ZC2TwPfnjGerXoabsSuR6atHJx0I4xc+kOXliYyjVdw3llfNx5SQOgR6um9G8TxJs/7mRC/9Z4e+gAmvWR9llU1dbMz4vZd8ZzsqCYye+uJb+oxNUhuUaTljBgKty7GB7aCMOnWU/Yf/8neKUrzL4SVk6H4xmujrTapiel88LCVK7r3oJ/l08aRadhzwqCs1bCzqU83j0fz9wM5q9OtTovqHpHGyHVZYltHsjLt8Qx+b11PPX5Jv41rjvSkB/Sa9oaBj5kvY7stKb33fw5LHraerXsZ2vOGgOB4a6OtlL+s3g7L32/jTFxLXjp5u545B2Efatg32rIWA0Hf4XSYroAbIaewHJv4Dsw37sh3oHnHvY87+HOC61rak1E5q5fUbWR/quoy3ZV5+Y8dlV7XvxuG7HNA7hvaMO4SXxJzdrA4EetV3Y6bJkHKfPg2yfg2yeh9YBzScQ/1NXRVujV77bwQ9IPvNw6kzFuGbj9e43VxRmsue4jesKA30HLvqzddpD4Lu0g/xgb0/fw5c9bGN81kHYBxWUfAj1x4NzDoCWFFw/AK+DiCcZ+BIIzPxtHgpevM6qj9is8aSXy/evhwHrr543/hcheDv0YTRzKIR5IbEvqoVxe+DaV9mEBJMbWzi9ClwluC0Met15ZaeeuRBY8Bgv/AK0HnksifsGui/NkNuxbjdm3iv2blnLv8c086F0Ih4H8ltCyD7T8nfWzedcyz7nkHUyG6MEAdO5geDhtKauy3Pn6lkEXvgotOn0uiVQ4ysDxstuP7Dy3ruhkxccUN2gWY8Vn//IPq19D1hQXWl3JD6yH/b9YP7NSzw1EGhgJET2cMoqCJg7lECLCP2/qzq7skzz44S/Me2AAbUMDXB1W7RTSARKesF6ZW6154jd/DvMfhQWPW1++ncdCx9Hg28x5cZSWWJ+fsdpqdtq3yvpixnr6Pqskil0hYxiYeA1uLftA48p3u3Z3E+4b2oYnPtvE8vRsBrcLqXhHz0bWqzrNdsWF5xLNmWRy+igc2QGHNsH+dVa9nuEbfH4yCWpXN5rDSkusMeHsryQOp5y7YvMNsjpkdLzO+hnR06lXsXWgxlRd0cjLnTcnxjP6teVMemctXz4wiMa+Orf5RYV2hGF/hMSnrTG7Nn9u9c76+iH45lFok2DNiRJ7jdXufznyj0PG2nNJImPtuedS/EKgZV9Mz7v434Ewpq33ZmzvGP42titu1Rwd4PoeEbz8/XZeT9px4cRxOTy8wD/Eel1I/nGrXg9tgkMb4VAKrHrj3EgC7t7Wv4F9MgnrbDWBuYoxcGxP2SRx8FcozLO2e/lbD7v2nWIliBY9rZ57NXg1pYlDOVSLJo14Y0Ivxs/6makfruetu3rjUUG3TVWOCDTvYr2G/cn6kkuxJZEvH4CvH4aYYdaVSOyoS3+xGQM5O2xXE7Yb2ZlbAWM15YR2hm7joGVfaNkbmkZjgBe+TeWN9Tu5rW8rnh/TpdpJA8Dbw51Jg6N5fv5Wftl7lB6tLjPxVYdPY+teUusB59aVFEH2dusv9jPJJG0h/PLeuX2atC6XTLo478s59/C5BHHm5+kj1jZ3L+vzu996LkkEtwM313Zz1sShHK5X62b89fqu/OGzjfx1wVb+ct1FHqhT5xOxHiYM7w5XPgMHfrFdiXwB2xdZXyZtr7SSSPsR1nMlhaesL519q8/1djplG8HHu7GVHDqPte5NRPQC77LNiMYY/rZgK2/+uIs7+rVi2ujLSxpn3NqnFf9Zks6M5B3MmnjeWHmu4e4JYZ2sV7dx1jpjIPdQ2WRyaBOkzufsEHvejW2JpMu5ZBLaETyqMKjj6WPWv+fZRPGLNToBWAk9pKP1h8GZ5qbQzrVyDhtNHMopxvVuSeqhXOb8tIvY5gHc0rt+PARX40SsL5CInjD8Oat5afM865W2wGpqCWoL2WlQahslOLg9tB9pu5Hd13p/kRukxhie+2Yrc37axZ39W/PM6M4O61Lt5+3BnQOieHXxdtIzc2vvfS8R6z5LYDi0G35ufeFJ60rNPpmsf+/cjXk3DwjuUDaZNO8GfkG4lRTA3lVlryZy0s8du2k0tOpnJfIWPa3hcLwuMF1BLaOJQznN06Ni2Z6Zy/99kUJMiD/xUU680dsQiFhXDi17w1XPW1cVm+dZN007jLCSRGTvKt1QN8bw7NdbeHvFbu4eGMWfr+3k8Odw7hoQxZvLdjIjeScvjevu0GM7nZcfRMZbrzNKS+HorrLJZNePsNFu0lK/EAafzIEfbT2cAsKt5NB9vPWzRQ/ndnxwMk0cymk83N147daeXP/6T0x5fx1fTh1EhE7y4xhubtZfq636VfsQxhie+Woz76zcwz2Dovm/azo65eHNZn5ejO/TkvdW7uHRq9rX/d8BNzdrQMugGKv574yTOXB4k5VMMrey90gBrQeMtRJFHXnYs7L0rqVyqsa+nrw5MZ6ColLufWctpwqLL11IOV1pqeFPX6bwzso9TB7SxmlJ44x7bQMevrlsp9M+w+X8gqxecAOmwvXT2dXmDqs3XD1LGqCJQ9WAtqH+vHpbD7YeOsHjn2zEWbNOqsopLTX88YsU3v95L1OGxvDUyFinDxPTokkjru8Rwdw1e8nJK7h0AVWraeJQNSKxQyhPjYxl/qaDvLYk/dIFlFOUlhqenreJD1fv5YHEGJ4Y0aHGxhabMjSGguJS3lmxu0Y+TzmPJg5VY+4d3IYbekTw0vfb+DblkKvDaXBKSw1PfLaRuWv28bthbXnsqppLGmBdeV7dqTlvr9hNXoE2WdZlmjhUjRER/nZDV7q3bMKjH28g9dAJV4fUYJSUGh7/dCOfrMvgoSva8ejw9i4Zxfj+hBhO5Bfz4aq9Nf7ZynE0caga5ePpzqwJvQjw8WDSO2u1vbsGlJQaHv/kVz5bn8EjV7bnERclDYDuLZswsG0Qs5fvpKC4gc7fUg9o4lA1LizQh1kT4snMLeD+D9ZTWFy3JvspKC7hg1V7+NfafF5YmMrSbVm1trdYcUkpj368gc9/2c9jV7XnoSvbuTok7h/alsMnCvh8/X5Xh6KqyamJQ0RGiEiaiKSLyJMVbL9dRDbaXitEpLvdtt0isklENojIWrv1zUTkexHZbvvpggFw1OXq3rIJ/7ixG6t3HeHZrze7OpxKOV1Ywpzluxj6j2T+OC+FAydLmf3jTu6cs5puz3zHjTNW8OKiNH5Kz64VsyEWl5TyyMe/8uWGA/xhRAemDnN90gAY2DaIbpGNeWPpDkpKtYddXeS0BwBFxB2YDgwHMoA1IvKVMWaL3W67gKHGmKMiMhKYBfS1255ojMkud+gngcXGmBdsyehJ4AlnnYdynut7RJB6KJeZS3cQGx7IhH6tXR1ShXLzi3j/573M/nEnOScL6RvdjBdv7k5Rxib6DBjMuj1HWbkzh5U7cpixdAevJaXj5e5GXKsm9G8TRP+YIHq0alKj828XlZTy8NwNzN90kKdGxtaqybVEhPuHxnD/B+tZmHKQa7u1cHVIqoqc+eR4HyDdGLMTQETmAmOAs4nDGLPCbv+fgchKHHcMkGBbfgdIRhNHnfX41R3YfjiXZ7/aTEyIHwNiXDiJUTnHThXy1k+7eeunXZzIL2Zo+xCmDmtLb9vQKcn7BT9vD4a0D2FIe2to79z8ItbuPpdI/rNkO/9evB1vDzd6tW5KP1si6R7ZBC8P51zwF5WU8uCHv7Aw5RB/HNWRe4e0ccrnXI6rOzenTYgfM5J3cE3X8IY93XAdJM56GEtEbgJGGGMm2d5PAPoaY6ZeYP/HgFi7/XcBR7GGpnzDGDPLtv6YMaaJXbmjxpjzmqtEZDIwGSAsLKzX3Llzq3UeeXl5+Pv7V6tsfeSM+jhdbHhu5WlOFBr+3L8Rob6uvfV2osDw7e4iluwtIr8Eeoa6c12MJ9GNy14xVKYuThYZth0tYWtOCVuPlLIv17qf4+UO7Zq40bGZO7FB7kQHuuHugNFoi0sNM34tYN3hEm6N9eLqqJqbD6Wqvxs/ZhTx35RCft/Lm64h9W/0o/rw3ZGYmLjOGHPesMbO/Neq6H9BhVlKRBKBe4BBdqsHGmMOiEgo8L2IpBpjllX2w22JZhZAfHy8SUhIqHTg9pKTk6lu2frIWfXRMe4kY6b/xH/TPPjstwPw9675L5KDx08za9lOPly9l4LiUq7t1oIHEmOIbR5Y4f6VrYtr7JaPnixk1a4j/Gy7Ivl0ey5sL8LPy53e0c3ONm11btG4yomksLiUB/63nnWHT/GX6zpx98DoKpW/XFX93RhQXMqCfybx01Fffndzf+cF5iL1+bvDmf87M4CWdu8jgQPldxKRbsBsYKQxJufMemPMAdvPTBGZh9X0tQw4LCLhxpiDIhIOZDrxHFQNiQr24/XbezJxzmoe+WgDb9zRyyHzQVTG3pxTzFi6g8/WZVBiDGN7RHB/QgwxIY7/a7GpnxcjujRnRJfmAGTnFbBq5xFW7sxm5Y4cktOyAAjw8aBvdLOzTVsdmwdetD4Kikv47fvrWZyaybQxnZnYP8rhsTual4cbkwa34blvtrB+71F6umKiJ1Utzkwca4B2IhIN7AfGA7fZ7yAirYDPgQnGmG126/0AN2NMrm35KmCabfNXwJ3AC7afXzrxHFQNGtg2mD9d05Fnvt7Cyz9s4/dXdXDq56Vn5vF6cjpfbjiAuwg3x0cyZWgMLZv5OvVz7QX7e3NNt3Cu6WYNhJd5Ip+VO3POXpH8sNX6u6iJryd9z16RBNM+zP/sfYH8ohLuf38dSWlZPH99F+6opZ0MKjK+d0v+s2Q7M5J38GZtmehJXZLTEocxplhEpgKLAHdgjjFms4hMsW2fCfwZCAJet/0nKLa1p4UB82zrPID/GWO+tR36BeBjEbkH2Avc7KxzUDXvzgFRpB7K5T9L0mkfFsB13R3f42brwRO8lpTOgk0H8fZw464BUdw7uA3NG/s4/LOqKjTQhzFxEYyJiwCs5rMzSWTlzhwWbT4MQJCfF/3aBNEvJojFWw+TnJbF38Z25ba+dWvCLD9vD+4aEMUrP2xn2+Fc2ofV0omeVBlObUg2xiwAFpRbN9NueRIwqYJyO4EKZ3yxNWdd4dhIVW0hIkwb04UdWXk8/umvRAX50TXyEvNrV9KGfcd4bUk6P2w9jL+3B/cPjeGeQdEE+Vdh6s8aFt64EWN7RDK2h9XhMOPoqbNJ5OcdOczfdBAReOGGrozvU7eSxhl39o9i1rKdzEzewb9uiXN1OKoS6l9XBlXneXm4MeOOXoz+z3Imv7eWL6cOJDSg+lcDq3bm8FpSOj9uz6aJryePDm/Pnf2jaOxbcz2OHCWyqS83x/tyc3xLjDHsPXKKohJD29C623unqZ8Xt/ZpxdsrdvPoVe2JbFpzTYWqenTIEVUrBft78+ad8Rw7VcSU99ZVeVwjYwzLtmUxbuZKbpn1M1sPnuCpkbEsf2IYD17Rrk4mjfJEhNZBfnU6aZwxaXA0blLPJ3qqRzRxqFqrc4vGvDSuO+v3HuOP81IqNQGUMYbvtxzm+uk/MXHOavYdPcUz13Vi+RPDuG9ojEu6+apLs5rkIpi7Zh/ZOvDlZSsuKWX59myenreJQ8fzHX58/V+karVRXcN56Ip2/HvxdjqGB3LPoIqfTSgpNSxMsSaJSj2US6tmvvy/G7pyQ8+IGh3qQ1XffUNj+GRdBm//tJvHrnZuj7r6qLC4lBU7slm46RDfbTnE0VNF+Hq5M7xTmMM7fmjiULXeQ1e0I+1QLn+dv4V2of5nh/cAa3iNrzYcYHpyOjuzThIT4sfLt3Tnum4t8HDXC+q6JCbEnxGdm/Puyt3cN7QNAT51vznR2QqKS1i+PZsFmw7x/ZZDnMgvxt/bgys7hjKyazhD24fg4+n4P5w0cahaz81NeGlcd26ccZKp/1vPFw8MJKJpIz5dl8GM5B1kHD1Nx/BAXr+9JyM6N6+xBweV492fEMPClEP8b9XeWjUwY22SX1TC0m1ZLNx0kMVbM8ktKCbQx4PhnZozqmtzBrYNdkqysKeJQ9UJft4evDkxnjG2exfFJYZDJ/KJa9mEZ0d3ZlhsqA6UVw90i2zCoLbBzF6+izsHRDn9C7CuOFVYTHJaFgs2HWRJaianCkto4uvJqK7hjOzanAExwU4bNLMimjhUndGymS8zbMOS9GjVhJfGdWdATJAmjHrmtwkx3DZ7FZ+v31/nHmh0pLyCYpakZrJw00GS0jLJLyolyM+L63tEMKpLOH3bNMPTRc2xmjhUndK3TRAbn7lKb3jXY9aw8415Y9kOxsVHNqh7VSfyi1i89TALNh1i6bYsCotLCQnwZlx8S0Z2CadPdDOHjKJ8uTRxqDpHk0b9JiLcn9CWKe+vY2HKIacMO1ObHDtVyPdbDrMw5RA/bs+iqMTQPNCH2/u2YlTXcHq1alrr7ttp4lBK1TpXdQojJsSP15N3cG23+jfR05GThXy3+RALUg6xIj2b4lJDRJNG3DUgipFdw4mLbFLrkoU9TRxKqVrHzU2YMjSGxz/dyNJtWSR0CHV1SJctK7eARZsPsTDlID/vPEJJqaF1kC+TBrdhVNfmdI1oXGcSpCYOpVStNCYugpe/38bryTvqbOI4fCKfb1MOsWDTQVbvPoIx0CbYj/uHxjCya3M6hQfWmWRhTxOHUqpWOjPR07RvtrBuzxF6tW7m6pAqpbiklA/X7OPdn0+z/dvFALQP8+fBYe0Y1TW8zFwqdZUmDqVUrTW+z7mJnmbfWfsTx56ckzzy0QbW7z1GywA3fj+8PSO7NqdtaP2aZ0QTh1Kq1vL18uCuAdG8/MM20g7l0qF57fwCNsbw0Zp9TPtmCx5uwqu39iDw6DYSEtq5OjSnaDgdpJVSddKdA1rj6+XOzKU7XB1KhbLzCrj33XU8+fkm4lo24duHhzC6nnch1sShlKrVmvh6cVufVnz16wH2HTnl6nDKWJJ6mBGvLGPZ9iz+75qOvH9PX1o0aeTqsJzOqYlDREaISJqIpIvIkxVsv11ENtpeK0Sku219SxFJEpGtIrJZRB6yK/OMiOwXkQ221yhnnoNSyvUmDW5jTfT0Y+2Y6OlUYTFPz9vEb95eS7C/N19PHWTFWIufvXAkp93jEBF3YDowHMgA1ojIV8aYLXa77QKGGmOOishIYBbQFygGfm+MWS8iAcA6EfneruzLxpgXnRW7Uqp2ad7Yhxt7RvLRmn38blg7QgJcN0/8L3uP8shHG9hz5BT3DWnDo1e1b3CjGTjziqMPkG6M2WmMKQTmAmPsdzDGrDDGHLW9/RmItK0/aIxZb1vOBbYCEU6MVSlVy00e0obCklLeXrHLJZ9fXFLKy99v46aZKykqMXx4bz+eGtWxwSUNcG7iiAD22b3P4OJf/vcAC8uvFJEooAewym71VFvz1hwRaeqAWJVStVybEH9GdQnn3ZV7yM0vqtHP3pV9khtnruTfi7czpnsLFj48mH5tgmo0htpEKjOPc7UOLHIzcLUxZpLt/QSgjzHmdxXsmwi8DgwyxuTYrfcHlgJ/NcZ8blsXBmQDBngOCDfG/KaCY04GJgOEhYX1mjt3brXOIy8vD39//2qVrY+0Ps7RuiirJupj9/ESnlmZz83tPbmmjZdTPwusbrbJ+4r5MK0QTze4s7M3fZpXroW/Pvx+JCYmrjPGxJdf78znODKAlnbvI4ED5XcSkW7AbGBkuaThCXwGfHAmaQAYYw7b7fMm8E1FH26MmYV1z4T4+HiTkJBQrZNITk6mumXrI62Pc7Quyqqp+licvYqkA7k8N2GwUyd6ysot4MnPNrI4NZPB7YL5503dqzR3d33+/XBmU9UaoJ2IRIuIFzAe+Mp+BxFpBXwOTDDGbLNbL8B/ga3GmH+VKxNu93YskOKk+JVStdD9CTFk5xXw6boMp33G91usbrbL07P5y3WdeOfuPlVKGvWd0644jDHFIjIVWAS4A3OMMZtFZIpt+0zgz0AQ8Lpt7JZi22XRQGACsElENtgO+bQxZgHwDxGJw2qq2g3c56xzUErVPv3bBBHXsgmzlu1kfO+WDp3o6WRBMdO+3sJHa/fRKTyQuePjaBdWO59WdyWnDjli+6JfUG7dTLvlScCkCsotByrsEG2MmeDgMJVSdYiI8NuEGCa/t475mw4yJs4xHS7X7TnKox9vYO+RU/w2IYaHr2xfo/N41yVaK0qpOufKjmG0C/VnRvIOLreDT1FJKS99l8bNM1dQUmr4+L7+/GFErCaNi9CaUUrVOWcmeko9lEtyWla1j7MjK48bZ6zgP0vSuaFnJAsfGkzvqNo/Cq+raeJQStVJo+NaENGkEa8np1e5rDGG91bu5ppXf2TvkVPMuL0nL97cnQAfTydEWv9o4lBK1Ume7m7cOziaNbuPsmb3kUqXy8zN5+631/CnLzfTNzqI7x4ewsiu4ZcuqM7SxKGUqrNu6d2KZn5ezEiu3JDr36Yc4uqXl7FyRw7TxnTm7bt7Exqo3WyrShOHUqrOauTlzt0DoliSmsnWgycuuF9eQTGPf/IrU95fR2RTX+Y/OJiJ/aPq/BSurqKJQylVp03sH4XfRSZ6Wrv7CCP/vYzP1mcwNbEtn90/gLahdXsoEFfTxKGUqtMa+3pye7/WfP3rAfbmnJvoqbC4lH8uSmXcGysRhE+m9OexqztoN1sH0BpUStV59wyKxsPNjVk/Wlcd6Zm53DDjJ6Yn7eCmXpEseGgwvVprN1tHceqT40opVRPCAn24sVcEH6/NILxxI15dvB0/bw/emNCLqzs3d3V49Y5ecSil6oX7hsRQXFLKPxelMSAmiG8fHqxJw0n0ikMpVS9EBfvx7JgueLu7cXN8pPaYciJNHEqpemNCv9auDqFB0KYqpZRSVaKJQymlVJVo4lBKKVUlmjiUUkpViSYOpZRSVaKJQymlVJVo4lBKKVUlmjiUUkpViVzuRO91gYhkAXuqWTwYyHZgOHWd1sc5WhdlaX2UVR/qo7UxJqT8ygaROC6HiKw1xsS7Oo7aQuvjHK2LsrQ+yqrP9aFNVUoppapEE4dSSqkq0cRxabNcHUAto/VxjtZFWVofZdXb+tB7HEoppapErziUUkpViSYOpZRSVaKJ4yJEZISIpIlIuog86ep4XEVEWopIkohsFZHNIvKQq2OqDUTEXUR+EZFvXB2Lq4lIExH5VERSbb8n/V0dk6uIyCO2/ycpIvKhiPi4OiZH08RxASLiDkwHRgKdgFtFpJNro3KZYuD3xpiOQD/ggQZcF/YeAra6Ooha4t/At8aYWKA7DbReRCQCeBCIN8Z0AdyB8a6NyvE0cVxYHyDdGLPTGFMIzAXGuDgmlzDGHDTGrLct52J9KUS4NirXEpFI4BpgtqtjcTURCQSGAP8FMMYUGmOOuTQo1/IAGomIB+ALHHBxPA6niePCIoB9du8zaOBflgAiEgX0AFa5OBRXewX4A1Dq4jhqgzZAFvCWrelutoj4uTooVzDG7AdeBPYCB4HjxpjvXBuV42niuDCpYF2D7rssIv7AZ8DDxpgTro7HVUTkWiDTGLPO1bHUEh5AT2CGMaYHcBJokPcERaQpVstENNAC8BORO1wbleNp4riwDKCl3ftI6uElZ2WJiCdW0vjAGPO5q+NxsYHAaBHZjdWEOUxE3ndtSC6VAWQYY85chX6KlUgaoiuBXcaYLGNMEfA5MMDFMTmcJo4LWwO0E5FoEfHCusH1lYtjcgkREaz2663GmH+5Oh5XM8Y8ZYyJNMZEYf1eLDHG1Lu/KivLGHMI2CciHWyrrgC2uDAkV9oL9BMRX9v/myuohx0FPFwdQG1ljCkWkanAIqyeEXOMMZtdHJarDAQmAJtEZINt3dPGmAWuC0nVMr8DPrD9kbUTuNvF8biEMWaViHwKrMfqjfgL9XDoER1yRCmlVJVoU5VSSqkq0cShlFKqSjRxKKWUqhJNHEoppapEE4dSSqkq0cShlAOISImIbLB7OezJaRGJEpEURx1Pqculz3Eo5RinjTFxrg5CqZqgVxxKOZGI7BaRv4vIaturrW19axFZLCIbbT9b2daHicg8EfnV9jozXIW7iLxpm+fhOxFp5LKTUg2eJg6lHKNRuaaqW+y2nTDG9AFewxpVF9vyu8aYbsAHwKu29a8CS40x3bHGezozWkE7YLoxpjNwDLjRqWej1EXok+NKOYCI5Blj/CtYvxsYZozZaRso8pAxJkhEsoFwY0yRbf1BY0ywiGQBkcaYArtjRAHfG2Pa2d4/AXgaY56vgVNT6jx6xaGU85kLLF9on4oU2C2XoPcnlQtp4lDK+W6x+7nStryCc1OK3g4sty0vBu6Hs3OaB9ZUkEpVlv7VopRjNLIbORis+bfPdMn1FpFVWH+o3Wpb9yAwR0Qex5o978xosg8Bs0TkHqwri/uxZpJTqtbQexxKOZHtHke8MSbb1bEo5SjaVKWUUqpK9IpDKaVUlegVh1JKqSrRxKGUUqpKNHEopZSqEk0cSimlqkQTh1JKqSr5/9N9jL7+ByiqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEJCAYAAACOr7BbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAzJUlEQVR4nO3deXzcdb3v8dcn+763aZq0TWhLF7qTpigUgiibKAgoRVApSw/KJvdcBT3eq171yjmHc454D1CLVATLUlFcEERBQ1lqE0oX0n1J26Rpk7bT7Pvkc//4TdppOkmTNJNZ8nk+HvPIzG+Z+XyT9vee3/f7W0RVMcYYY3qLCHQBxhhjgpMFhDHGGJ8sIIwxxvhkAWGMMcYnCwhjjDE+WUAYY4zxya8BISJXisgOEdktIg/7mJ8uIq+IyGYRKRWRWZ7pcZ7Xm0Rki4h83591GmOMOZ346zwIEYkEdgKfAqqAMuBmVd3qtcy/A02q+n0RmQ48rqqXiYgAiaraJCLRwLvAA6r6D78Ua4wx5jRRfnzvImC3qu4FEJEXgWuBrV7LzAR+DKCq20UkX0SyVbUGaPIsE+15nDHJsrKyND8/f0jFNjc3k5iYOKR1Q5W1OfyNtvaCtXmw1q9ff1RVx/ia58+AyAUqvV5XAYt6LbMJuB54V0SKgElAHlDj2QNZD0zB2bNYd6YPzM/P54MPPhhSsSUlJRQXFw9p3VBlbQ5/o629YG0eLBHZ39c8fwaE+JjWey/gEeAxEdkIfARsALoAVNUNzBORNOAVEZmlquWnfYjIMmAZQHZ2NiUlJUMqtqmpacjrhiprc/gbbe0Fa/Nw8mdAVAETvF7nAdXeC6hqA7AUwDPuUOF5eC9TJyIlwJXAaQGhqiuAFQCFhYU61BS1bx2jw2hr82hrL1ibh5M/j2IqA6aKSIGIxABLgD94LyAiaZ55AHcCa1S1QUTGePYcEJF44JPAdj/Waowxphe/7UGoapeI3Au8AUQCK1V1i4jc7Zm/HJgBPCsibpzB6zs8q+cAv/SMQ0QAq1X11aHU0dnZSVVVFW1tbf0ul5qayrZt24byESFroG2Oi4sjLy+P6OjoEajKGBMs/NnFhKq+BrzWa9pyr+drgak+1tsMzB+OGqqqqkhOTiY/Px+nF8u3xsZGkpOTh+MjQ8ZA2qyqHDt2jKqqKgoKCkaoMmNMMAj7M6nb2trIzMzsNxxM30SEzMzMM+6BGWPCT9gHBGDhcJbs92fM6OTXLiZjjDHDq9PdTW1jO4frWzlc386h+lZ27u3AHwduWUAYY0yQaOt0c7i+jUP1bRxucALgcH2r53Ubh+vbONLUTu8rJKXGCv/mh3osIPysrq6O559/nq997WuDWu/qq6/m+eefJy0tzT+FGWNGjKrS2N5FTc/G/0QItJ0SAHUtnaetmxwXRU5qHONS45kxLoXs1DjPa8/PlDg2rHvPL3VbQPhZXV0dTzzxxGkB4Xa7iYyM7HO91157rc95xpjg0d2tHGvuoKahjZqGUwPAed3K4fo2mjvcp62blRRDdkoceenxFOank5MaT3bKyQAYlxJHYuyZN9P+GiccVQHx/T9uYWt1g895Z9pg92Xm+BS++5nz+pz/8MMPs2fPHubNm0d0dDRJSUnk5OSwceNGtm7dynXXXUdlZSVtbW088MADLFu2DDh5XammpiauuuoqLrroIt5//31yc3P5/e9/T3x8vM/Pe+qpp1ixYgUdHR1MmTKF5557joSEBGpqarj77rvZu3cvAE8++SSzZ8/m2Wef5dFHH0VEmDNnDs8999ygfwfGhCNVpaG1i5pGZ0N/uL6N2sb2E0FQ09BObYMzrav71D6fyAhhbHIs41LjmDYumYvPHXNiL2CcJwDGpsQSGzX4bc5IGlUBEQiPPPII5eXlbNy4kZKSEj796U9TXl5+4pyClStXkpGRQWtrKwsXLuSGG24gMzPzlPfYtWsXL7zwAk899RRf+MIX+M1vfsOtt97q8/Ouv/567rrrLgC+853v8PTTT3Pfffdx//33c8kll/DKK6/gdrtpampi27Zt/OhHP+K9994jKysLl8vl31+GMUGipaOLmgbvjX3bide1De0c9kxr7+o+bd3U+GiyU2LJTolj8pisE8+zU2IZlxpPTmocWUmxREaE/tF/oyog+vumP1InyhUVFZ1ywtlPf/pTXnnlFQAqKyvZtWvXaQFRUFDAvHnzADj//PPZt29fn+9fXl7Od77zHerq6mhqauKKK64A4G9/+xvPPvssAJGRkaSmpvL2229z4403kpWVBUBGRsZwNdOYgOh0d3O0tZv1+11eAeB80+/Z6Nc2tNPY3nXauvHRkYxLjWNscizzJ6aRneI8dzb+cSeCIC46uL/1D6dRFRDBwPua7SUlJbz55pusXbuWhIQEiouLfZ6QFhsbe+J5ZGQkra2tfb7/bbfdxu9+9zvmzp3LM8880+8VHlXVznEwIcXdrdQ0tFHpaqHqeCuVxz0/Pa8P1bfSrcDba0+sExMZwVjPxn3auGQWTx1z8ht/ShxjPc+TYqPs/0MvFhB+lpycTGNjo8959fX1pKenk5CQwPbt2/nHP87+hnmNjY3k5OTQ2dnJqlWryM3NBeCyyy7jySef5Otf/zput5vm5maKi4u59dZbefDBB8nMzMTlctlehAmo7m7laFP7aRv+ntfVda10uk/294tAdnIcEzLiKSrIYEJ6PI21B7hk4dwT3/zTE6Jtwz9EFhB+lpmZyYUXXsisWbOIj48nOzv7xLwrr7yS5cuXM2fOHKZNm8YFF1xw1p/3gx/8gEWLFjFp0iRmz559Ipwee+wxli1bxtNPP01kZCRPPvkks2bN4l/+5V+45JJLiIyMZP78+TzzzDNnXYMxfVFVXM0dPr/9Vx5v4eDx1tP6/bOSYslLj2dOXhqfnp1DXnoCEzLiyUtPYHxa3GkDvSUlhyieNnYkmxW2/HZP6kAoLCzU3neU27ZtGzNmzDjjunaxvv4N9PcY7EbbvQIC0d62Tjd7jzRzwNVC1fGW0/YCWnod7pmeEH3KRn9CevyJ17lpCcTHDK7Pf7T9jeGs7yi3XlULfc2zPQhjzJDUt3Sy+0gju2ub2F3bxJ4jzeyubaLyeMspZ/omx0aRl5HApMxELpoyhrz0eCZkJJCXHk9eejzJcXYZ+WBlARGi7rnnHt5779SzJx944AGWLl0aoIpMOFJVDje0eYVAk+d5M0eb2k8sFxMVwTlZiczJS+X6BblMHpNEQVYiE9ITSIm3wd9QZQERoh5//PFAl2DCSJe7m/2ullNCYI9nr6DJ65DQlLgopoxN4hPTxzB5TBJTxjqPvPSEsDju35zKAsKYUaS1w82eI957As5j37HmU44OGpcSx+SxidywIJcpY5OY7AmCMUmxtjcwilhAGBOGOrq62XG4kbcrO3nn1a0nguBg3clzaCIEJmUmMnlMEpfNyD6xNzB5TKKNCxjAAsKYkNfl7mZXbRMfVdWz+WAdm6vq2X6okQ63c7hoXPR+zslK4vxJ6dy0cIInBJLIz0oI+msBmcCygDAmhHR3K3uPNvPRwTo2Vdbz0cF6tlTX09bphEFybBSzclNZemE+c/LSaK7axo1XXkqEjQ+YIbCACDJJSUk0NTVRXV3N/fffz8svv3zaMsXFxTz66KMUFvo8dNmECVXlgKuFzVVOEGyuqqP8YMOJQeP46Ehm5abwxaJJzMlLZU5eKvmZiaeEQYlrh4WDGTILiCA1fvx4n+FgwpOqcqi+jc1VThA4gVBPfatzA5mYyAhmjE/hc/NzPWGQxuQxiURFjorbypsAGV0B8frDcPgjn7Pi3V0QOYRfx7jZcNUjfc5+6KGHmDRp0okbBn3ve99DRFizZg3Hjx+ns7OTH/7wh1x77bWnrLdv3z6uueYaysvLaW1tZenSpWzdupUZM2b0e7E+gK9+9auUlZXR2trKjTfeyPe//30AysrKeOCBB2hubiY2Npbf/e53JCQk8NBDD/HGG28gItx1113cd999g/89mEGpbWxzxgy89g6ONnUAEBUhTBuXzNWzxzE7N405eamcm51MTJSFgRlZoysgAmDJkiV8/etfPxEQq1ev5s9//jMPPvggKSkpHD16lAsuuIDPfvazfR4++OSTT5KQkMDmzZvZvHkzCxYs6Pczf/SjH5GRkYHb7eayyy5j8+bNTJ8+nZtuuomXXnqJhQsX0tDQgNvtZsWKFVRUVLBhwwaioqLsnhB+UN/ayabKOjZX1Z0IhEP1zlV7IwSmjE3iknPHMndCKrNzU5mRkzKqLil9Vtoa4OguOLoTju6AIzuZd3gftFwE4+dD7gLInAIR9vscitEVEP1802/107WY5s+fT21tLdXV1Rw5coT09HRycnJ48MEHWbNmDRERERw8eJCamhrGjRvn8z3WrFnD/fffD8CcOXOYM2dOv5+5evVqVqxYQVdXF4cOHWLr1q2ICDk5OSxcuBCAlJQUGhsbefPNN7n77ruJinL+KdjVXM9ep7ubTZV1rNl1lHd2HWFTZR09NxwryEqkqCCD2blON9F541MGdEvJUU0VmmrgyA5PEOw8+bzx0MnlIqIhczKiCht+BaU/c6bHJEHOXCcweh4Z5ziXgjX9sn+ZI+DGG2/k5Zdf5vDhwyxZsoRVq1Zx5MgR1q9fT3R0NPn5+T7vA+FtoCcnVVRU8Oijj1JWVkZ6ejq33XYbbW1tfd77we4JMTz2H2t2AmHnEdbuOUZjexcRAnPy0rj30iksOieTWbmppMbb+QV9cnfB8X2n7A04z3dBe/3J5WKSYcy5cM6lzs+scyFrGqTnQ2QUG0pKKL54sbNe9YdQvcF5lP0cujz/z+JSIWfeqaGRNtFCoxcLiBGwZMkS7rrrLo4ePcrbb7/N6tWrGTt2LNHR0fz9739n//79/a5/8cUXs2rVKi699FLKy8vZvHlzn8s2NDSQmJhIamoqNTU1vP766xQXFzN9+nSqq6spKytj4cKFNDY20tXVxeWXX87y5cspLi4+0cVkexFnVt/aydo9R0/sJVS6nHGh3LR4rpk7nounZvHxyVmkRnVA+W/gsAvc02HsDEidABGjeDyho9nTLbTLEwQ7nOeuPeDuOLlccg5kTYU5X4Ax05znWdMgedyZN+QRkTB2uvOY90VnmrsTjmx3wuKgJzjWPg7dzoEAJGSeGhjj50PKeP/8DkKEBcQIOO+882hsbCQ3N5ecnBxuueUWPvOZz1BYWMi8efOYPn16v+t/9atfZenSpcyZM4d58+ZRVFTU57Jz585l/vz5nHfeeZxzzjlceOGFAMTExPDSSy9x33330draSnx8PK+88gp33nknO3fuZM6cOURHR3PXXXdx7733Dmv7w0GXu5tNVXWs2ekEwkZPt1FiTCQfm5zFXYvPYfHUMeRnJjh7ZK69sOa7TleH97dfgOhEZ4M3dgaMmX7yZ2pe+HyDVYWWY56NvycAerqF6itPLieRkFHgbPjPvcITBNMga4rzLX84RUY7B5WMmw0LvuxM62qHmi2evYwPoXojvPOfoJ5LkieNOz00ksYMb11BzK/3gxCRK4HHgEjg56r6SK/56cBKYDLQBtyuquUiMgF4FhgHdAMrVPWxM32e3Q9icOx+EP3r6TZ6d9cR3t99arfR4qlZLJ46hvkT04juOdRUFfb8DUpXwM43nG+xMz4Li+52ukKO7IDabc632J6fTTUnPzAm+WRweIdHcs6Qg8Nv90Zwdzn9//VVnkel87PhoPOzrvLUYIxOOLkHkHWup2tomjMWEBUzrKWddZs7WqCm/NQ9jaM7Ac+2MnUCjJ93MjBy5kFCYPe6Q+5+ECISCTwOfAqoAspE5A+qutVrsW8DG1X1cyIy3bP8ZUAX8M+q+qGIJAPrReSvvdY1Zlg1tHXy/u5jvLPrCO/sOsoBVwvQ022Uw+KpY/j45EzSEnpt0NobYeMLTjAc2wWJY+Dib0Dh7ZCSc3K5iRc4D28trpOB0RMaO16HDc+dXCY21ekq8d7bGDsDkrL9s8ehCm11Xht/rwCo9wRAYzXoqXd+Iz7d2QtKmwgTPwaZk0+GQkpu6HSrxSTAhCLn0aO9EQ5t9trT2ADb/nhyfnqBExbzboGpnxz5mv3En11MRcBuVd0LICIvAtcC3hv5mcCPAVR1u4jki0i2qh4CDnmmN4rINiC317qj3qJFi2hvbz9l2nPPPcfs2bMDVFFocbqN6k8EwsbKOtzd6uk2yuSOiwpYPDWLgqxE3wP5R3c7obDxeehohNzz4XMr4LzrICp2YEUkZMCkjzsPb81HT9/b2PZH+PCXJ5eJS/Pa25jhCZEZZ+4C6eo4+U3fOwC8p3U0nbpOZIyzkU/Ng4KLnZ/ej5RciE0aWJtDUWwy5F/oPHq0HodDm07uaVR9cPrfMcT5MyByAa/ORqqARb2W2QRcD7wrIkXAJCAPOLHfLSL5wHxg3VALCdcjddatG/KvZFDC6ba0tS3d/Oof+3ln1xHe33OMxrYuxNNt9LXiyVw0JYv5E9P7Pimtuxt2v+kcQrn7TefQylnXQ9E/Qd75w1doYhYULHYePVSh+YhXcGyF2u3OIHibV3dOQuaJwJhwrBPe+OupYdBUw4nukhOfN8bZ0GdNdY4OOrHxn+D8TBwTOnsAIyU+Hc4pdh49urv7Wjok+W0MQkQ+D1yhqnd6Xn8JKFLV+7yWScEZo5gPfARMB+5U1U2e+UnA28CPVPW3fXzOMmAZQHZ29vkvvvjiKfOTkpLIzs4mNTW135Bwu91ERo6uk2kG0mZVpb6+npqaGpqamvpdNlh1uJU1VV38dX8nNS3Ov/fMOOG8rEhmZUUyMyOSpJj+v0BEdjWTc+gtxle/RkLrIdpj0qkefxXV46+gMyZtBFrRD1ViOlwkNleS2HyAhJYDJDYfILG5kih3C+6IGNpjs2iLG0N77BjPz5Ov22Mz6Y4c4B5PCGhqaiIpKYz3Znw4mzZfeumlfY5B+DMgPgZ8T1Wv8Lz+FoCq/riP5QWoAOaoaoOIRAOvAm+o6n8O5DN9DVJ3dnZSVVV1xvMM2traiIuLG8jHhI2BtjkuLo68vDyio0PrGP6m9i5W/WM/T71TwdGmds6flM70hGZuv/pjnNNXt1FvtdudbqRNL0JnM0xYBIv+yRl8jgzy34cq77z1Gosvuzp8jo4aAL8NzAexkBukBsqAqSJSABwElgBf7FVYGtCiqh3AncAaTzgI8DSwbaDh0Jfo6GgKCgrOuFxJSQnz588/m48KOeHa5rqWDp55fx+/eG8f9a2dLJ6axb2XzmfROZmUlJQwecwZvml1u2Hnn2Hdz6DibYiMhdk3QtEy5+iVUCGCOypxVIWDGV5+CwhV7RKRe4E3cA5zXamqW0Tkbs/85cAM4FkRceMMQN/hWf1C4EvARyKy0TPt26r6mr/qNaHvSGM7T79bwXNr99Hc4eZTM7O599IpzJ2QNrA3aHE5Rw+V/RzqDjgDr5f9b1jwFWdMwJhRxq8nynk26K/1mrbc6/laYKqP9d4F7GuPGZDqulZWrNnLC6UH6HR3c82c8Xzt0slMH5cysDeo2eLsLWxeDV2tMOkiuPyHMO3TQ7vCrzFhwv71m5C172gzy9/ew28+rEIVrl+Qy1eLp1CQlXjmld1dsONPsG4F7H8XouKdSzoULYNxs/xfvDEhwALChJydNY08/vfd/HFTNVGREdxcNJFlF59DXnrCGdeN7miAd/4DylZCQ5VzUtenfgDzbw342bDGBBsLCBOcut3OdXLc7c5F1rra2XHwGC+u3UXZnsMkRys/mJvBp8/LJC36AFTvgsrOk+t0dZyyLu52aDjEx7b8HrQTCi6Bq/8Nzr3S7hVgTB8sIIz/qDpnBB/bfeqjqbbXxrvjtDA4cbE0L9OA7wL0HLK/3fM4E4lwjkSKSeRQzifJve57zlnHxph+WUCYs9feCMf2eAJgj1cY7Dn1gm0R0c7F2ZLHOWehRsU4G+7ImJPPPT81MpqKui7W7G1g97EOYmLi+Pi08Vw4fTzxcQme5U5dh6hY59wE72mRMacMNO8qKSHXwsGYAbGAMAPT1e7czOWUvQFPGHhfkRRxLs+QOdkZ9M2c4nlMdqaf4aig7m7lzW01PP733WyqqicnNY5lV5/DkoUTiY+xriBjRpIFhDmpu9sZuD1tT2C3c16A99U7E8c4G/6pn4KMySeDIKMAouMH/dHubuXVzdU88fc97KhpZGJGAj++fjbXL8glNsqCwZhAsIAYjVSh+kPGHXoT3izxCoQ9zjhAj5gk55t/7vkw56aTewIZkyE+bVhK6ejq5ncbDvJEyW72HWth6tgkfnLTPK6Zk0NUpF0czphAsoAYTeoOONcU2vg8HK9gOsCuaOdbf+YUmHKZV5fQFP/dbwBo63TzUlklP3t7D9X1bczKTWH5rQu4fOY4IiLsHEljgoEFRLjraHbuI7BxFVSscablL4ZLvsk/DgkXXP75ET1buLm9i195XUCvcFI6//f62Vxy7piwvCS7MaHMAiIcqcKBfzihsOV3zs1s0iZB8bdh7hJInwRAW13JiIZDdV0rX1lZyq7aJhZPzeKeS+ezqCDDgsGYIGUBEU7qKj1dSKvgeAVEJ8J5n4N5X3RuARnAG77sqmnkyytLaWrr4rk7ilg8dfTc+N2YUGUBEeo6Wnp1IamnC+khmPGZoLgN5Pr9Lm5/5gNioiJ46Z8+xszxA7yInjEmoCwgQpEqVK6DDb/q1YX0rVO6kILBm1truPeFD8lJjefZ24uYkHHm6yUZY4KDBUQoqauEzZ6jkFx7PV1I13m6kD4edPcMXv1BJd/67UecNz6FX9y2kMyk8LmtpTGjgQVEsOtoge2vOl1Ie9/mRBfSxd9wbnsZBF1IvakqT5Ts4d/f2MHiqVksv/V8EmPtn5oxocb+1wajni6kjaug/BVPF9JEKH7Y04WUH+gK+9TdrfyfV7fyzPv7uG7eeP7txrnERAXXno0xZmAsIIJJfRVsegE2vgCuPUHfhdRbe5ebf169iVc3H+LOiwr49tUz7KQ3Y0KYBUSgdbTA9j95upBKONmF9D+DtgvJl8a2Tu7+1Xre232Mb189nWUXTw50ScaYs2QBEUgbX4DXvwntDU4X0iUPwbybg7oLyZcjje3c9otSth9u5D8+P5cbzs8LdEnGmGFgAREou9+E398DEy9wDk+ddGHQdyH5sv9YM19eWUptQzs//0ohl04bG+iSjDHDxAIiEA6Xw+rbYOxM+OJLEJsc6IqGpPxgPbf9ohR3t/L8XYuYPzE90CUZY4aRBcRIazgEz3/BCYVbVodsOLy3+yjLnv2AtIQYnr2jiMljQmOsxBgzcBYQI6m9yQmHtnq4/c+QMj7QFQ3JHzdV8z9Wb+ScrCR+eXsR41LjAl2SMcYPLCBGirsLXr4darY43UrjZge6oiF55r0Kvv/qVhZOyuCprxSSGh8d6JKMMX5iATESVOHPD8GuN+Ca/3Ju0xliVJV/f2MHT5Ts4fKZ2fz05vnERdutQI0JZxYQI2Ht41D2c/j4/VB4e6CrGbQudzff+u1H/Hp9FTcXTeSH180i0k6AMybsWUD427Y/wl++AzOvhU9+P9DVDFprh5t7n/+Qt7bX8sBlU/n6J6faDX6MGSUsIPyp6gP4zV2QVwif+1nInedwvLmDO35ZxobKOn543SxuvSB4LiNujPE/v26xRORKEdkhIrtF5GEf89NF5BUR2SwipSIyy2veShGpFZFyf9boN8f3wfM3QXI23PwiRMcHuqJBqa5r5fM/W0v5wQae+OICCwdjRiG/BYSIRAKPA1cBM4GbRWRmr8W+DWxU1TnAl4HHvOY9A1zpr/r8qvU4rPo8dHfBLS9DYlagKxqUnTWNXP/E+9TUt/HsHUVcNTsn0CUZYwLAn3sQRcBuVd2rqh3Ai8C1vZaZCbwFoKrbgXwRyfa8XgO4/Fiff3R1wEtfAlcFLFkFWVMDXdGgfLDPxY1Pvo9blZf+6WNccE5moEsyxgSIP8cgcoFKr9dVwKJey2wCrgfeFZEiYBKQB9QM9ENEZBmwDCA7O5uSkpIhFdvU1DTkdU9QZfr2nzCu5h22zniQ2n1dsO8s39OPerd5Q20XT2xsJyNO+J/zo6nd+SG1OwNXnz8My985hIy29oK1eTj5MyB8HeqivV4/AjwmIhuBj4ANQNdgPkRVVwArAAoLC7W4uHjQhQKUlJQw1HVPvskjUFMCl/4LMy/5Jr3704KNd5tfKjvA/9vwEbNzU1kZxrcHHZa/cwgZbe0Fa/Nw8mdAVAETvF7nAdXeC6hqA7AUQJxjJys8j9Cz8QUo+THMu8W5HWiIUFUe//tuHv3LTi4+dwxP3rLAbg9qjAH8GxBlwFQRKQAOAkuAL3ovICJpQItnjOJOYI0nNEJLxTvwh/ug4GK45icQIucJdKvyvT9s4Zdr9/O5+bn86w1z7PagxpgT/BYQqtolIvcCbwCRwEpV3SIid3vmLwdmAM+KiBvYCtzRs76IvAAUA1kiUgV8V1Wf9le9Q3ZkB7x0C2ROhi88B1Exga5oQNq73Dy5qZ2yw/u5a3EB37rKbg9qjDmVX/sSVPU14LVe05Z7PV8L+DzMR1Vv9mdtw6KpFlbdCJGx8MXVEJ8W6IoGxN2t3PnLDyg77Lbbgxpj+mSdzUPV0QIvLIGmI7D0T5AeOieSbamu551dR7lpWoyFgzGmTxYQQ9HdDa8sg4Mfwk2/gtzzA13RoJRWOKeXXJBjV2M1xvTNAmIo/vq/nIvwXfFjmHFNoKsZtHUVLvIzE0iPszEHY0zf7JCVwSp9Ctb+NxQtgwu+GuhqBq27Wynb56KoICPQpRhjgpwFxGDsfANe/yacexVc+UjIHM7qbVdtE3UtnRQV2CU0jDH9s4AYqEOb4NdLnVuF3vBziAjN/vvSimMALLI9CGPMGVhADER9Faz6AsSnO4ezxiYFuqIhW1fhIic1jrz00Lr8uDFm5Nkg9Zm0NTjh0NEMd7wByeMCXdGQqSqlFS4+NjnT7gpnjDkjC4j+uDvh17fB0R1wy68h+7xAV3RW9h9robax3QaojTEDYgHRF1X40z/Dnrfgs/8PJn8i0BWdtdJ9zvkPNv5gjBkIG4Poy3s/gQ9/CRf9D1jw5UBXMyxKK1xkJMYweUzojqEYY0aOBYQv5b+FN78Hs26AT/yvQFczbEorXCzMT7fxB2PMgFhA9HZgHbxyN0y4AK59AiLC41d0qL6VA64WO//BGDNg4bH1Gy7H9jgX4EvNhSXPQ3RcoCsaNj3XX7LxB2PMQFlAeER1NsCqzzsvbnkZEsPrm3ZphYuk2Chm5KQEuhRjTIiwo5gAOtuYVf5jaKqCr/zBuflPmCmtcFGYn06k3RTIGDNAtgfR3Q2/v4e0+q3wuSdh4gWBrmjYHWtqZ1dtk53/YIwZFAuItjqo3cbegi85Ry2FobJ9xwEbfzDGDI4FREIG3PkmByaGZziA070UGxXB7Ny0QJdijAkhFhAAMQkheenugSrdd4wFE9OJibI/tzFm4PrcYojIv4nI3T6mPygi/+rfssxwaWjrZGt1g40/GGMGrb+vlNcAK3xMfwz4tH/KMcNt/f7jdKuNPxhjBq+/gFBV7fYxsRsI3/6YMFNa4SIqQpg/MT3QpRhjQkx/AdEiIlN7T/RMa/VfSWY4lVa4mJOXSnxMaN4BzxgTOP0FxP8GXheR20RktuexFPiTZ54Jcm2dbjZX1dn1l4wxQ9LnmdSq+rqIXAd8A7jPM7kcuEFVPxqB2sxZ2nCgjk632viDMWZI+gwIEYkDalT1K72mjxWROFVt83t15qyUVrgQgQWTbPzBGDN4/XUx/RRY7GP6p4D/8k85ZjiV7jvGjHEppMZHB7oUY0wI6i8gLlLV3/aeqKqrgIsH8uYicqWI7BCR3SLysI/56SLyiohsFpFSEZk10HVN/zq6ulm//7id/2CMGbL+AqK/Q1nPeEquiEQCjwNXATOBm0VkZq/Fvg1sVNU5wJdxzrEY6LqmH+XV9bR1dtv4gzFmyPrb0NeKSFHviZ5pRwbw3kXAblXdq6odwIvAtb2WmQm8BaCq24F8Ecke4LqmHz03CFpoAWGMGaL+7gfxDWC1iDwDrPdMK8T5pr9kAO+dC1R6va4CFvVaZhNwPfCuJ3gmAXkDXBcAEVkGLAPIzs6mpKRkAKWdrqmpacjrBqPX17eRkyiUf7C2z2XCrc0DMdraPNraC9bm4dTfYa6lIrII+BpwG6DAFuArOCGx7gzv7auLSnu9fgR4TEQ2Ah8BG4CuAa7bU+cKPJcEKSws1OLi4jOU5VtJSQlDXTfYuLuV+0r+wjVzJlBcPLvP5cKpzQM12to82toL1ubh1O8d5VS1BviuiMwHbsYJh4uB3wzgvauACV6v84DqXu/fACwFEBEBKjyPhDOta/q2/XADjW1dNv5gjDkr/Z0HcS5OV9LNwDHgJUBU9dIBvncZMFVECoCDnvf6Yq/PSANaPOMMdwJrVLVBRM64rulbz/iDHcFkjDkb/e1BbAfeAT6jqrvBudT3QN9YVbtE5F7gDSASWKmqW3ouIa6qy4EZwLMi4ga2Anf0t+6gWzdKlVa4yEuPZ3xafKBLMcaEsP4C4gacb+5/F5E/4xxJNKiruKrqa8BrvaYt93q+FjjtgoB9rWvOTFUprXBxybQxgS7FGBPi+jzMVVVfUdWbgOlACfAgkC0iT4rI5SNUnxmkPUeaOdbcYeMPxpizdsYT3lS1WVVXqeo1OIPFGwE7szlIle3rGX+wK7gaY87OoG5SrKouVf2Zqn7CXwWZs1Na4WJMciz5mQmBLsUYE+LsLvZhprTCRVF+Bs5Rw8YYM3QWEGGk6ngLB+ta7fBWY8ywsIAII3b+gzFmOFlAhJHSChcpcVFMy04OdCnGmDBgARFGSitcFBVkEBFh4w/GmLNnAREmahvb2Hu02bqXjDHDxgIiTJRVHAfs/AdjzPCxgAgTpRXHSIiJ5LzxKYEuxRgTJiwgwsS6ChfnT0onOtL+pMaY4WFbkzBQ19LBjppGivJt/MEYM3wsIMLAB/uOo2rnPxhjhpcFRBgo3eciJjKCuRPSAl2KMSaMWECEgXUVLuZNSCMuOjLQpRhjwogFRIhrbu9iy8F6614yxgw7C4gQt+FAHV3dagFhjBl2FhAhrrTiGBECCyalB7oUY0yYsYAIcesqXMzKTSUptr/bixtjzOBZQISw9i43Gyrr7PwHY4xfWECEsM1V9XR0ddv4gzHGLywgQljPDYIW2h6EMcYPLCBC2LoKF9Oyk0lPjAl0KcaYMGQBEaK63N2s3+ey7iVjjN9YQISorYcaaO5wW0AYY/zGAiJE9Yw/WEAYY/zFAiJEratwkZ+ZQHZKXKBLMcaEKb8GhIhcKSI7RGS3iDzsY36qiPxRRDaJyBYRWeo17wERKfdM/7o/6ww13d1KmY0/GGP8zG8BISKRwOPAVcBM4GYRmdlrsXuArao6FygG/kNEYkRkFnAXUATMBa4Rkan+qjXU7Kptoq6l0+4/bYzxK3/uQRQBu1V1r6p2AC8C1/ZaRoFkEREgCXABXcAM4B+q2qKqXcDbwOf8WGtIKa04BsAi24MwxviRPy/gkwtUer2uAhb1Wua/gT8A1UAycJOqdotIOfAjEckEWoGrgQ98fYiILAOWAWRnZ1NSUjKkYpuamoa87kh7dWMbGXHC7k3r2CMy5PcJpTYPl9HW5tHWXrA2Dyd/BoSvLZf2en0FsBH4BDAZ+KuIvKOq20TkX4G/Ak3AJpw9i9PfUHUFsAKgsLBQi4uLh1RsSUkJQ113JKkqD73/FounZ3LppfPP6r1Cpc3DabS1ebS1F6zNw8mfXUxVwASv13k4ewrelgK/VcduoAKYDqCqT6vqAlW9GKfraZcfaw0ZB1wt1DS02+U1jDF+58+AKAOmikiBiMQAS3C6k7wdAC4DEJFsYBqw1/N6rOfnROB64AU/1hoy1nnOf7DxB2OMv/mti0lVu0TkXuANIBJYqapbRORuz/zlwA+AZ0TkI5wuqYdU9ajnLX7jGYPoBO5R1eP+qjWUlFa4yEiMYcrYpECXYowJc369y4yqvga81mvacq/n1cDlfay72J+1harSChcL89ORsxicNsaYgbAzqUPIofpWDrha7PwHY8yIsIAIIaU2/mCMGUEWECGktMJFUmwUM3JSAl2KMWYUsIAIIaUVLgrz04mMsPEHY4z/WUCEiGNN7eyqbbIL9BljRowFRIgo2+cc5WvjD8aYkWIBESJKK1zERkUwOzct0KUYY0YJC4gQUbrvGAsmphMTZX8yY8zIsK1NCGhs62RrdYONPxhjRpQFRAhYv/843WrjD8aYkWUBEQJKK1xERQjzJ6YHuhRjzChiARECSitczM5LJT4mMtClGGNGEQuIINfW6WZTVZ2NPxhjRpwFRJDbcKCOTrfa+IMxZsRZQAS50goXInD+JAsIY8zIsoAIcqX7jjFjXAqp8dGBLsUYM8pYQASxjq5u1u8/buMPxpiAsIAIYuXV9bR1dtv4gzEmICwggljPDYIWWkAYYwLAAiKIlVa4mDwmkayk2ECXYowZhSwggpS7Wynb57L7TxtjAsYCIkhtP9xAY1uXjT8YYwLGAiJI9Yw/2BFMxphAsYAIUmX7XOSlxzM+LT7QpRhjRikLiCCkqpRWuGzvwRgTUBYQQWjv0WaONnVQlG8BYYwJHAuIIGTjD8aYYGABEYRKK1xkJcVSkJUY6FKMMaOYXwNCRK4UkR0isltEHvYxP1VE/igim0Rki4gs9Zr3oGdauYi8ICJx/qw1mJRWuFhUkIGIBLoUY8wo5reAEJFI4HHgKmAmcLOIzOy12D3AVlWdCxQD/yEiMSKSC9wPFKrqLCASWOKvWoNJ1fEWDta1WveSMSbg/LkHUQTsVtW9qtoBvAhc22sZBZLF+aqcBLiALs+8KCBeRKKABKDaj7UGDRt/MMYEC38GRC5Q6fW6yjPN238DM3A2/h8BD6hqt6oeBB4FDgCHgHpV/Ysfaw0apRUuUuKimJadHOhSjDGjXJQf39tXB7r2en0FsBH4BDAZ+KuIvIPTpXQtUADUAb8WkVtV9VenfYjIMmAZQHZ2NiUlJUMqtqmpacjrDqeSLS2ckxzBmjVv+/2zgqXNI2m0tXm0tReszcPJnwFRBUzwep3H6d1ES4FHVFWB3SJSAUwHJgEVqnoEQER+C3wcOC0gVHUFsAKgsLBQi4uLh1RsSUkJQ113uNQ2tnH4z29xe/FUii+e7PfPC4Y2j7TR1ubR1l6wNg8nf3YxlQFTRaRARGJwBpn/0GuZA8BlACKSDUwD9nqmXyAiCZ7xicuAbX6sNSiUVRwHsCu4GmOCgt/2IFS1S0TuBd7A6TJaqapbRORuz/zlwA+AZ0TkI5wuqYdU9ShwVEReBj7EGbTegGcvIZyVVhwjISaS88anBLoUY4zxaxcTqvoa8Fqvacu9nlcDl/ex7neB7/qzvmCzrsLF+ZPSiY608xeNMYFnW6IgUd/SyY6aRrv+kjEmaFhABIkP9rtQtfMfjDHBwwIiSJRWuIiJjGDuhLRAl2KMMYAFRNBYV+Fi7oRU4qIjA12KMcYAFhBBobm9i/KD9da9ZIwJKhYQQWDDgTq6utXOfzDGBBULiCBQWnGMCIHzJ6UHuhRjjDnBAiIIrKtwMSs3laRYv56WYowxg2IBEWDtXW42VNbZ+Q/GmKBjARFgm6vq6ejqtgFqY0zQsYAIsJ4bBC20PQhjTJCxgAiwdRUupmUnk54YE+hSjDHmFBYQAdTl7mb9Ppd1LxljgpIFRABtPdRAc4fbAsIYE5QsIAKoZ/zBAsIYE4wsIAJoTHIsn507nuyUuECXYowxp7EzswLo2nm5XDsvN9BlGGOMT7YHYYwxxicLCGOMMT5ZQBhjjPHJAsIYY4xPFhDGGGN8soAwxhjjkwWEMcYYnywgjDHG+CSqGugaho2IHAH2D3H1LODoMJYTCqzN4W+0tReszYM1SVXH+JoRVgFxNkTkA1UtDHQdI8naHP5GW3vB2jycrIvJGGOMTxYQxhhjfLKAOGlFoAsIAGtz+Btt7QVr87CxMQhjjDE+2R6EMcYYn0Z9QIjIlSKyQ0R2i8jDga7H30Rkgoj8XUS2icgWEXkg0DWNFBGJFJENIvJqoGsZCSKSJiIvi8h2z9/7Y4Guyd9E5EHPv+tyEXlBRMLublwislJEakWk3Gtahoj8VUR2eX6mD8dnjeqAEJFI4HHgKmAmcLOIzAxsVX7XBfyzqs4ALgDuGQVt7vEAsC3QRYygx4A/q+p0YC5h3nYRyQXuBwpVdRYQCSwJbFV+8QxwZa9pDwNvqepU4C3P67M2qgMCKAJ2q+peVe0AXgSuDXBNfqWqh1T1Q8/zRpyNRtjf1k5E8oBPAz8PdC0jQURSgIuBpwFUtUNV6wJa1MiIAuJFJApIAKoDXM+wU9U1gKvX5GuBX3qe/xK4bjg+a7QHRC5Q6fW6ilGwsewhIvnAfGBdgEsZCT8Bvgl0B7iOkXIOcAT4hadb7ecikhjoovxJVQ8CjwIHgENAvar+JbBVjZhsVT0EzpdAYOxwvOloDwjxMW1UHNYlIknAb4Cvq2pDoOvxJxG5BqhV1fWBrmUERQELgCdVdT7QzDB1OwQrT7/7tUABMB5IFJFbA1tVaBvtAVEFTPB6nUcY7pL2JiLROOGwSlV/G+h6RsCFwGdFZB9ON+InRORXgS3J76qAKlXt2Tt8GScwwtkngQpVPaKqncBvgY8HuKaRUiMiOQCen7XD8aajPSDKgKkiUiAiMTgDWn8IcE1+JSKC0y+9TVX/M9D1jARV/Zaq5qlqPs7f+G+qGtbfLFX1MFApItM8ky4DtgawpJFwALhARBI8/84vI8wH5r38AfiK5/lXgN8Px5tGDcebhCpV7RKRe4E3cI54WKmqWwJclr9dCHwJ+EhENnqmfVtVXwtcScZP7gNWeb787AWWBrgev1LVdSLyMvAhztF6GwjDs6pF5AWgGMgSkSrgu8AjwGoRuQMnKD8/LJ9lZ1IbY4zxZbR3MRljjOmDBYQxxhifLCCMMcb4ZAFhjDHGJwsIY4wxPllAGDMIIuIWkY1ej2E7O1lE8r2v0GlMoI3q8yCMGYJWVZ0X6CKMGQm2B2HMMBCRfSLyryJS6nlM8UyfJCJvichmz8+JnunZIvKKiGzyPHouCREpIk957mnwFxGJD1ijzKhnAWHM4MT36mK6yWteg6oWAf+Nc/VYPM+fVdU5wCrgp57pPwXeVtW5ONdI6jmDfyrwuKqeB9QBN/i1Ncb0w86kNmYQRKRJVZN8TN8HfEJV93ouhnhYVTNF5CiQo6qdnumHVDVLRI4Aeara7vUe+cBfPTd9QUQeAqJV9Ycj0DRjTmN7EMYMH+3jeV/L+NLu9dyNjROaALKAMGb43OT1c63n+fucvO3lLcC7nudvAV+FE/fKThmpIo0ZKPt2YszgxHtdBRecez73HOoaKyLrcL543eyZdj+wUkS+gXOHt54rqj4ArPBcfdONExaH/F28MYNhYxDGDAPPGEShqh4NdC3GDBfrYjLGGOOT7UEYY4zxyfYgjDHG+GQBYYwxxicLCGOMMT5ZQBhjjPHJAsIYY4xPFhDGGGN8+v9roykpeFK/sAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "metrics = pd.read_csv(f\"{trainer.logger.log_dir}/metrics.csv\")\n",
    "\n",
    "aggreg_metrics = []\n",
    "agg_col = \"epoch\"\n",
    "for i, dfg in metrics.groupby(agg_col):\n",
    "    agg = dict(dfg.mean())\n",
    "    agg[agg_col] = i\n",
    "    aggreg_metrics.append(agg)\n",
    "\n",
    "df_metrics = pd.DataFrame(aggreg_metrics)\n",
    "df_metrics[[\"train_loss\", \"valid_loss\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='Loss')\n",
    "df_metrics[[\"train_acc\", \"valid_acc\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='ACC')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18304f53",
   "metadata": {},
   "source": [
    "- The `trainer` automatically saves the model with the best validation accuracy automatically for us, we which we can load from the checkpoint via the `ckpt_path='best'` argument; below we use the `trainer` instance to evaluate the best model on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8bff53a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at logs/my-model/version_36/checkpoints/epoch=8-step=74600.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from checkpoint at logs/my-model/version_36/checkpoints/epoch=8-step=74600.ckpt\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "80900a8ff1054ce0abdc8a2cbcc7b646",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">        Test metric        </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9217909574508667     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m       Test metric       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9217909574508667    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'test_acc': 0.9217909574508667}]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.test(model=lightning_model, datamodule=data_module, ckpt_path='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebe513ab",
   "metadata": {},
   "source": [
    "## Predicting labels of new data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0674611",
   "metadata": {},
   "source": [
    "- You can use the `trainer.predict` method on a new `DataLoader` or `DataModule` to apply the model to new data.\n",
    "- Alternatively, you can also manually load the best model from a checkpoint as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "99fb98a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logs/my-model/version_36/checkpoints/epoch=8-step=74600.ckpt\n"
     ]
    }
   ],
   "source": [
    "path = trainer.checkpoint_callback.best_model_path\n",
    "print(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ab60d544",
   "metadata": {},
   "outputs": [],
   "source": [
    "lightning_model = LightningModel.load_from_checkpoint(\n",
    "    path, model=pytorch_model)\n",
    "lightning_model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb52e61c",
   "metadata": {},
   "source": [
    "- Note that our PyTorch model, which is passed to the Lightning model, requires input arguments. However, this is automatically being taken care of since we used `self.save_hyperparameters()` in our PyTorch model's `__init__` method.\n",
    "- Now, below is an example applying the model manually. Here, pretend that the `test_dataloader` is a new data loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b544b139",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([7, 1, 3, 4, 9])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataloader = data_module.test_dataloader()\n",
    "\n",
    "all_true_labels = []\n",
    "all_predicted_labels = []\n",
    "for batch in test_dataloader:\n",
    "    features, labels = batch\n",
    "    \n",
    "    with torch.no_grad():  # since we don't need to backprop\n",
    "        logits = lightning_model(features)\n",
    "\n",
    "    predicted_labels = torch.argmax(logits, dim=1)\n",
    "    all_predicted_labels.append(predicted_labels)\n",
    "    all_true_labels.append(labels)\n",
    "    \n",
    "all_predicted_labels = torch.cat(all_predicted_labels)\n",
    "all_true_labels = torch.cat(all_true_labels)\n",
    "all_predicted_labels[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9afe2e3",
   "metadata": {},
   "source": [
    "Just as an internal check, if the model was loaded correctly, the test accuracy below should be identical to the test accuracy we saw earlier in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "89cd4554",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.9218 (92.18%)\n"
     ]
    }
   ],
   "source": [
    "test_acc = torch.mean((all_predicted_labels == all_true_labels).float())\n",
    "print(f'Test accuracy: {test_acc:.4f} ({test_acc*100:.2f}%)')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74a2323e",
   "metadata": {},
   "source": [
    "## Single-image usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "331fa78f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "694f1e4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62f39922",
   "metadata": {},
   "source": [
    "- Assume we have a single image as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "0882c122",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAPzklEQVR4nO3de4wVZZrH8d8jN7mMXKTBltFtmeAta9YZW1lhI24mi0pIZGIk+IdhExP8QxJGNK4OCRhjlGy84B/emIUMa2acjGGI93UMGWNGkomtQcAlCEvaGaRDN4IOKqhNP/tHl5MWu95qquqcKnm/n6Rzuuvpt+vx2D/q9Hmr6jV3F4BT32lVNwCgOQg7EAnCDkSCsAORIOxAJIY3c2eTJ0/2tra2Zu4SiEpnZ6cOHjxog9UKhd3MrpX0mKRhkv7L3VeHvr+trU0dHR1FdgkgoL29PbWW+2W8mQ2T9Lik6yRdLOkmM7s4788D0FhF/ma/QtIed9/r7l9J+q2k68tpC0DZioR9mqS/Dvh6X7LtW8xsiZl1mFlHT09Pgd0BKKJI2Ad7E+A75966+1p3b3f39paWlgK7A1BEkbDvk3TOgK9/KGl/sXYANEqRsL8taYaZnWdmIyUtkvRCOW0BKFvuqTd37zWzpZJeU//U23p3f7+0zgCUqtA8u7u/IumVknoB0ECcLgtEgrADkSDsQCQIOxAJwg5EgrADkSDsQCQIOxAJwg5EgrADkSDsQCQIOxAJwg5EgrADkSDsQCQIOxAJwg5EgrADkSDsQCQIOxAJwg5EoqlLNhfV3d2dWpswYUJw7PHjx4P1Y8eO5WlJkmQ26Aq5f3fGGWcE66edFue/uV1dXcF6a2trkzqJQ5y/ZUCECDsQCcIORIKwA5Eg7EAkCDsQCcIORKJW8+xff/11sH722Wen1rLmunt7e3P1VIYpU6YE61dddVWwvnz58mD9yiuvPOmemmXLli2ptdmzZzd032PGjEmtzZ8/Pzj2wQcfDNanT5+eq6cqFQq7mXVKOiLpuKRed28voykA5SvjyP6v7n6whJ8DoIH4mx2IRNGwu6Q/mNk7ZrZksG8wsyVm1mFmHT09PQV3ByCvomGf7e4/kXSdpNvM7DvvNLn7Wndvd/f2lpaWgrsDkFehsLv7/uSxW9ImSVeU0RSA8uUOu5mNNbMffPO5pLmSdpTVGIByFXk3fqqkTcn89nBJv3H3/ynSTF9fX7AeuiZ92bJlwbFz584N1seNGxesjxgxIrWWdX7Aa6+9Fqw/8cQTwXrW87Jx48ZgvUovvvhi7rFPP/10sD5q1Khg/eOPP06tPfTQQ8Gxq1atCtafeeaZYL2Ocofd3fdK+qcSewHQQEy9AZEg7EAkCDsQCcIORIKwA5Go1SWuRW6pnHWZ57x583L/7KKyLmE9/fTTg/U1a9YE6+6eWsu69LfRPvnkk9TaWWedFRy7ZMmgZ2A3xYoVK4L1devWBesjR44ss51ScGQHIkHYgUgQdiAShB2IBGEHIkHYgUgQdiASp8w8e9aSzHU2Z86cYH3lypXB+sGD6ff7rPruQG+88UZqbebMmc1r5AQLFiwI1u+4445gfdu2bcF6e3v9brTMkR2IBGEHIkHYgUgQdiAShB2IBGEHIkHYgUjUap69yLXXWbdzrrNp06YVGn/48OHUWqPn2Y8ePRqs79q1K7WWdZ1/I5133nnBemh5cEl67rnngnXm2QFUhrADkSDsQCQIOxAJwg5EgrADkSDsQCRqNc+edT17aB6+t7e37HaaZvjwYv8bqryWf8+ePcF66J72WUsuN1LWOR2LFi0K1rOWbF69enWh/TdC5pHdzNabWbeZ7RiwbZKZvW5mu5PHiY1tE0BRQ3kZ/ytJ156w7W5Jm919hqTNydcAaiwz7O7+pqRDJ2y+XtKG5PMNkhaU2xaAsuV9g26qu3dJUvI4Je0bzWyJmXWYWUdPT0/O3QEoquHvxrv7Wndvd/f2qm9+CMQsb9gPmFmrJCWP3eW1BKAR8ob9BUmLk88XS3q+nHYANErmBK+ZPSvpakmTzWyfpFWSVkv6nZndIukvkm5sZJPfCM3Df5/vG1/kfvlStdfyb9++PffYOq5h/o2FCxcG64888kiwnnX+wYwZM066p6Iyw+7uN6WUflpyLwAaiNNlgUgQdiAShB2IBGEHIkHYgUjU6hLX0OWQWfWil4lWqWjva9asSa2NHz8+ODbrFOaurq5gfevWrcF6yOjRo3OPbbTLLrssWD/33HOD9fvvvz9Y37BhQ7DeCBzZgUgQdiAShB2IBGEHIkHYgUgQdiAShB2IRK0mp7Mu1ezr60utZc0n11nRefZNmzal1rKWJs5aLrqtrS1Yz7qUc+LE9BsPL126NDi2Sln/T7Lm0RcvXhysh2413draGhybF0d2IBKEHYgEYQciQdiBSBB2IBKEHYgEYQci8b2aZw/5Ps+zDxs2rND4l19+ObU2a9asQj87y5w5c4L1Cy64ILU2derUsttpmqxbTd9+++3Bemie/bHHHsvVUxaO7EAkCDsQCcIORIKwA5Eg7EAkCDsQCcIORKJW8+xffvll7rFjxowpsZPmKjrP3tvbW1InJ2/Xrl3B+ty5c5vUSXONGjUqWF+5cmWwvnz58tTaqlWrgmMnTZoUrKfJPLKb2Xoz6zazHQO23WtmH5nZ1uRjXq69A2iaobyM/5WkawfZ/qi7X5p8vFJuWwDKlhl2d39T0qEm9AKggYq8QbfUzLYlL/NTbzRmZkvMrMPMOrLWFQPQOHnD/qSkH0m6VFKXpIfTvtHd17p7u7u3t7S05NwdgKJyhd3dD7j7cXfvk/RLSVeU2xaAsuUKu5kNvNftzyTtSPteAPWQOc9uZs9KulrSZDPbJ2mVpKvN7FJJLqlT0q1lNHPs2LHcY7/P8+ynnVbs3KZGzrNn3WOgu7s7WM+6b/2pKuu+8cuWLUutbdmyJTh2/vz5uXrKDLu73zTI5nW59gagMpwuC0SCsAORIOxAJAg7EAnCDkSiVpe4Hj16NPfY0aNHB+t79+4N1g8cOJB731mXqF5yySXBetGptyK34M7y6aefBuvuHqyfqlNvWdPEDzzwQO6f3agzTTmyA5Eg7EAkCDsQCcIORIKwA5Eg7EAkCDsQiVrNs3/xxRe5x15zzTXB+ocffpj7Zxc1fHj4ab7hhhsK/fyvvvqq0PiQorcSa21tzf6mCmSd07F+/fpgPetW0UeOHAnWH3/88dTa5ZdfHhybF0d2IBKEHYgEYQciQdiBSBB2IBKEHYgEYQciccrMs+/fvz9Yv/HGG4P1iy66KFgPXWOcdS39e++9F6xnzelm+eCDDwqND/noo48KjT/zzDNL6uS7sn5fnnrqqdTafffdFxz72WefBetLly4N1lesWBGsV7E6Ekd2IBKEHYgEYQciQdiBSBB2IBKEHYgEYQciUat59gsvvDBYnzVrVmqtr68vOHb79u3B+quvvhqsZ827FjFu3LhgfebMmcH6woULy2znW3bv3h2sjxw5MlgP/bcdOnQoOHbNmjXB+sMPPxysh+6nH1oyWZLuueeeYH3SpEnBeh1lHtnN7Bwz+6OZ7TSz981sWbJ9kpm9bma7k8eJjW8XQF5DeRnfK+kOd79I0j9Lus3MLpZ0t6TN7j5D0ubkawA1lRl2d+9y93eTz49I2ilpmqTrJW1Ivm2DpAUN6hFACU7qDToza5P0Y0l/ljTV3buk/n8QJE1JGbPEzDrMrKPo/cwA5DfksJvZOEkbJf3c3f821HHuvtbd2929vYqT/wH0G1LYzWyE+oP+a3f/fbL5gJm1JvVWSd2NaRFAGTKn3szMJK2TtNPdHxlQekHSYkmrk8fnizYzfvz4YP2tt94quovcQtM4n3/+eXBs1rTd1KlTg/URI0YE643U2dkZrGdNG956662ptXXr1gXHjhkzJli/6667gvXQ9NqECROCY09FQ5lnny3pZknbzWxrsu0X6g/578zsFkl/kRS+YBxApTLD7u5/kmQp5Z+W2w6ARuF0WSAShB2IBGEHIkHYgUgQdiAStbrEtc5Cc91Zc7bf5zndw4cPB+tZl6m+9NJLqbUnn3wyOPbmm28O1rNu4Y1v48gORIKwA5Eg7EAkCDsQCcIORIKwA5Eg7EAkmGdH0KOPPhqs33nnncH69OnTU2vDh/Pr10wc2YFIEHYgEoQdiARhByJB2IFIEHYgEoQdiAQTnQgaO3ZssH7++ec3qRMUxZEdiARhByJB2IFIEHYgEoQdiARhByJB2IFIZIbdzM4xsz+a2U4ze9/MliXb7zWzj8xsa/Ixr/HtAshrKCfV9Eq6w93fNbMfSHrHzF5Pao+6+0ONaw9AWYayPnuXpK7k8yNmtlPStEY3BqBcJ/U3u5m1SfqxpD8nm5aa2TYzW29mE1PGLDGzDjPr6OnpKdYtgNyGHHYzGydpo6Sfu/vfJD0p6UeSLlX/kf/hwca5+1p3b3f39paWluIdA8hlSGE3sxHqD/qv3f33kuTuB9z9uLv3SfqlpCsa1yaAoobybrxJWidpp7s/MmB764Bv+5mkHeW3B6AsQ3k3frakmyVtN7OtybZfSLrJzC6V5JI6Jd3agP4AlGQo78b/SZINUnql/HYANApn0AGRIOxAJAg7EAnCDkSCsAORIOxAJAg7EAnCDkSCsAORIOxAJAg7EAnCDkSCsAORIOxAJMzdm7czsx5JHw7YNFnSwaY1cHLq2ltd+5LoLa8ye/sHdx/0/m9NDft3dm7W4e7tlTUQUNfe6tqXRG95Nas3XsYDkSDsQCSqDvvaivcfUtfe6tqXRG95NaW3Sv9mB9A8VR/ZATQJYQciUUnYzexaM9tlZnvM7O4qekhjZp1mtj1Zhrqj4l7Wm1m3me0YsG2Smb1uZruTx0HX2Kuot1os4x1YZrzS567q5c+b/je7mQ2T9IGkf5O0T9Lbkm5y9/9taiMpzKxTUru7V34ChpldJekzSf/t7v+YbPtPSYfcfXXyD+VEd/+PmvR2r6TPql7GO1mtqHXgMuOSFkj6d1X43AX6WqgmPG9VHNmvkLTH3fe6+1eSfivp+gr6qD13f1PSoRM2Xy9pQ/L5BvX/sjRdSm+14O5d7v5u8vkRSd8sM17pcxfoqymqCPs0SX8d8PU+1Wu9d5f0BzN7x8yWVN3MIKa6e5fU/8sjaUrF/ZwocxnvZjphmfHaPHd5lj8vqoqwD7aUVJ3m/2a7+08kXSfptuTlKoZmSMt4N8sgy4zXQt7lz4uqIuz7JJ0z4OsfStpfQR+Dcvf9yWO3pE2q31LUB75ZQTd57K64n7+r0zLegy0zrho8d1Uuf15F2N+WNMPMzjOzkZIWSXqhgj6+w8zGJm+cyMzGSpqr+i1F/YKkxcnniyU9X2Ev31KXZbzTlhlXxc9d5cufu3vTPyTNU/878v8naUUVPaT0NV3Se8nH+1X3JulZ9b+s+1r9r4hukXSmpM2SdiePk2rU2zOStkvapv5gtVbU27+o/0/DbZK2Jh/zqn7uAn015XnjdFkgEpxBB0SCsAORIOxAJAg7EAnCDkSCsAORIOxAJP4fUwOrijzVq+YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from PIL import Image\n",
    "\n",
    "\n",
    "image = Image.open('./quickdraw-png_set1/binoculars/binoculars_093136.png')\n",
    "plt.imshow(image, cmap='Greys')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2625ac92",
   "metadata": {},
   "source": [
    "- Note that we used a resize-transformation in the `DataModule` that rescaled the 28x28 images to size 32x32. We also have to apply the same transformation to any new image that we feed to the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "64dafca9",
   "metadata": {},
   "outputs": [],
   "source": [
    "resize_transform = transforms.Compose(\n",
    "            [transforms.Resize((32, 32)),\n",
    "             transforms.ToTensor()])\n",
    "\n",
    "image_chw = resize_transform(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "003c8d20",
   "metadata": {},
   "source": [
    "- Note that `ToTensor` returns the image in the CHW format. CHW refers to the dimensions and stands for channel, height, and width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "02845a79",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 32, 32])\n"
     ]
    }
   ],
   "source": [
    "print(image_chw.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ecfcca3",
   "metadata": {},
   "source": [
    "- However, the PyTorch / PyTorch Lightning model expectes images in NCHW format, where N stands for the number of images (e.g., in a batch).\n",
    "- We can add the additional channel dimension via `unsqueeze` as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "b442de8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 1, 32, 32])\n"
     ]
    }
   ],
   "source": [
    "image_nchw = image_chw.unsqueeze(0)\n",
    "print(image_nchw.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25690363",
   "metadata": {},
   "source": [
    "- Now that we have the image in the right format, we can feed it to our classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "db887578",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():  # since we don't need to backprop\n",
    "    logits = lightning_model(image_nchw)\n",
    "    probas = torch.softmax(logits, axis=1)\n",
    "    predicted_label = torch.argmax(probas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "c8fdd29f",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_dict = {\n",
    "         0: \"lollipop\",\n",
    "         1: \"binoculars\",\n",
    "         2: \"mouse\",\n",
    "         3: \"basket\",\n",
    "         4: \"penguin\",\n",
    "         5: \"washing machine\",\n",
    "         6: \"canoe\",\n",
    "         7: \"eyeglasses\",\n",
    "         8: \"beach\",\n",
    "         9: \"screwdriver\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "be912947",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted label: binoculars\n",
      "Class-membership probability 99.95%\n"
     ]
    }
   ],
   "source": [
    "print(f'Predicted label: {label_dict[predicted_label.item()]}')\n",
    "print(f'Class-membership probability {probas[0][predicted_label]*100:.2f}%')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}