pytorch_ipynb/cnn/cnn-alexnet-cifar10.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UEBilEjLj5wY"
   },
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 536,
     "status": "ok",
     "timestamp": 1524974472601,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "GOzuY8Yvj5wb",
    "outputId": "c19362ce-f87a-4cc2-84cc-8d7b4b9e6007"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Author: Sebastian Raschka\n",
      "\n",
      "Python implementation: CPython\n",
      "Python version       : 3.8.12\n",
      "IPython version      : 8.0.1\n",
      "\n",
      "torch: 1.10.1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rH4XmErYj5wm"
   },
   "source": [
    "# AlexNet CIFAR-10 Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "References\n",
    "    \n",
    "- [1] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. \"[Imagenet classification with deep convolutional neural networks.](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf)\" In Advances in Neural Information Processing Systems, pp. 1097-1105. 2012.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MkoGLH_Tj5wn"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "ORj09gnrj5wp"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import random\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data.dataset import Subset\n",
    "\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I6hghKPxj5w0"
   },
   "source": [
    "## Model Settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setting a random seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I recommend using a function like the following one prior to using dataset loaders and initializing a model if you want to ensure the data is shuffled in the same manner if you rerun this notebook and the model gets the same initial random weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_all_seeds(seed):\n",
    "    os.environ[\"PL_GLOBAL_SEED\"] = str(seed)\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed_all(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setting cuDNN and PyTorch algorithmic behavior to deterministic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the `set_all_seeds` function above, I recommend setting the behavior of PyTorch and cuDNN to deterministic (this is particulary relevant when using GPUs). We can also define a function for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_deterministic():\n",
    "    if torch.cuda.is_available():\n",
    "        torch.backends.cudnn.benchmark = False\n",
    "        torch.backends.cudnn.deterministic = True\n",
    "    torch.set_deterministic(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23936,
     "status": "ok",
     "timestamp": 1524974497505,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "NnT0sZIwj5wu",
    "outputId": "55aed925-d17e-4c6a-8c71-0d9b3bde5637"
   },
   "outputs": [],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Hyperparameters\n",
    "RANDOM_SEED = 1\n",
    "LEARNING_RATE = 0.0001\n",
    "BATCH_SIZE = 256\n",
    "NUM_EPOCHS = 40\n",
    "\n",
    "# Architecture\n",
    "NUM_CLASSES = 10\n",
    "\n",
    "# Other\n",
    "DEVICE = \"cuda:0\"\n",
    "\n",
    "set_all_seeds(RANDOM_SEED)\n",
    "\n",
    "# Deterministic behavior not yet supported by AdaptiveAvgPool2d\n",
    "#set_deterministic()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import utility functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.insert(0, \"..\") # to include ../helper_evaluate.py etc.\n",
    "\n",
    "from helper_evaluate import compute_accuracy\n",
    "from helper_data import get_dataloaders_cifar10\n",
    "from helper_train import train_classifier_simple_v1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to data/cifar-10-python.tar.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e73f98cd265c48ca8922f5968c31d202",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/170498071 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data/cifar-10-python.tar.gz to data\n"
     ]
    }
   ],
   "source": [
    "### Set random seed ###\n",
    "set_all_seeds(RANDOM_SEED)\n",
    "\n",
    "##########################\n",
    "### Dataset\n",
    "##########################\n",
    "\n",
    "train_transforms = transforms.Compose([transforms.Resize((70, 70)),\n",
    "                                       transforms.RandomCrop((64, 64)),\n",
    "                                       transforms.ToTensor()])\n",
    "\n",
    "test_transforms = transforms.Compose([transforms.Resize((70, 70)),\n",
    "                                      transforms.CenterCrop((64, 64)),\n",
    "                                      transforms.ToTensor()])\n",
    "\n",
    "\n",
    "train_loader, valid_loader, test_loader = get_dataloaders_cifar10(\n",
    "    batch_size=BATCH_SIZE, \n",
    "    num_workers=2, \n",
    "    train_transforms=train_transforms,\n",
    "    test_transforms=test_transforms,\n",
    "    validation_fraction=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Set:\n",
      "\n",
      "Image batch dimensions: torch.Size([256, 3, 64, 64])\n",
      "Image label dimensions: torch.Size([256])\n",
      "tensor([0, 2, 3, 5, 4, 8, 9, 6, 9, 7])\n",
      "\n",
      "Validation Set:\n",
      "Image batch dimensions: torch.Size([256, 3, 64, 64])\n",
      "Image label dimensions: torch.Size([256])\n",
      "tensor([6, 9, 3, 5, 7, 3, 4, 1, 8, 0])\n",
      "\n",
      "Testing Set:\n",
      "Image batch dimensions: torch.Size([256, 3, 64, 64])\n",
      "Image label dimensions: torch.Size([256])\n",
      "tensor([2, 6, 3, 1, 1, 1, 1, 2, 4, 8])\n"
     ]
    }
   ],
   "source": [
    "# Checking the dataset\n",
    "print('Training Set:\\n')\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.size())\n",
    "    print('Image label dimensions:', labels.size())\n",
    "    print(labels[:10])\n",
    "    break\n",
    "    \n",
    "# Checking the dataset\n",
    "print('\\nValidation Set:')\n",
    "for images, labels in valid_loader:  \n",
    "    print('Image batch dimensions:', images.size())\n",
    "    print('Image label dimensions:', labels.size())\n",
    "    print(labels[:10])\n",
    "    break\n",
    "\n",
    "# Checking the dataset\n",
    "print('\\nTesting Set:')\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.size())\n",
    "    print('Image label dimensions:', labels.size())\n",
    "    print(labels[:10])\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### MODEL\n",
    "##########################\n",
    "\n",
    "class AlexNet(nn.Module):\n",
    "\n",
    "    def __init__(self, num_classes):\n",
    "        super(AlexNet, self).__init__()\n",
    "        self.features = nn.Sequential(\n",
    "            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "            \n",
    "            nn.Conv2d(64, 192, kernel_size=5, padding=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "            \n",
    "            nn.Conv2d(192, 384, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            \n",
    "            nn.Conv2d(384, 256, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            \n",
    "            nn.Conv2d(256, 256, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "        )\n",
    "        self.avgpool = nn.AdaptiveAvgPool2d((6, 6))\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(256 * 6 * 6, 4096),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(4096, 4096),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Linear(4096, num_classes)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.features(x)\n",
    "        x = self.avgpool(x)\n",
    "        x = x.view(x.size(0), 256 * 6 * 6)\n",
    "        logits = self.classifier(x)\n",
    "        probas = F.softmax(logits, dim=1)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "_lza9t_uj5w1"
   },
   "outputs": [],
   "source": [
    "torch.manual_seed(RANDOM_SEED)\n",
    "\n",
    "model = AlexNet(NUM_CLASSES)\n",
    "model.to(DEVICE)\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RAodboScj5w6"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 1547
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2384585,
     "status": "ok",
     "timestamp": 1524976888520,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "Dzh3ROmRj5w7",
    "outputId": "5f8fd8c9-b076-403a-b0b7-fd2d498b48d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/040 | Batch 0000/0175 | Loss: 2.3033\n",
      "Epoch: 001/040 | Batch 0050/0175 | Loss: 2.0240\n",
      "Epoch: 001/040 | Batch 0100/0175 | Loss: 1.9445\n",
      "Epoch: 001/040 | Batch 0150/0175 | Loss: 1.8135\n",
      "***Epoch: 001/040 | Train. Acc.: 33.674% | Loss: 1.703\n",
      "***Epoch: 001/040 | Valid. Acc.: 34.880% | Loss: 1.670\n",
      "Time elapsed: 1.05 min\n",
      "Epoch: 002/040 | Batch 0000/0175 | Loss: 1.7606\n",
      "Epoch: 002/040 | Batch 0050/0175 | Loss: 1.5473\n",
      "Epoch: 002/040 | Batch 0100/0175 | Loss: 1.5496\n",
      "Epoch: 002/040 | Batch 0150/0175 | Loss: 1.5093\n",
      "***Epoch: 002/040 | Train. Acc.: 42.819% | Loss: 1.505\n",
      "***Epoch: 002/040 | Valid. Acc.: 43.840% | Loss: 1.491\n",
      "Time elapsed: 2.09 min\n",
      "Epoch: 003/040 | Batch 0000/0175 | Loss: 1.5411\n",
      "Epoch: 003/040 | Batch 0050/0175 | Loss: 1.5485\n",
      "Epoch: 003/040 | Batch 0100/0175 | Loss: 1.3723\n",
      "Epoch: 003/040 | Batch 0150/0175 | Loss: 1.3084\n",
      "***Epoch: 003/040 | Train. Acc.: 49.712% | Loss: 1.336\n",
      "***Epoch: 003/040 | Valid. Acc.: 50.300% | Loss: 1.327\n",
      "Time elapsed: 3.12 min\n",
      "Epoch: 004/040 | Batch 0000/0175 | Loss: 1.4301\n",
      "Epoch: 004/040 | Batch 0050/0175 | Loss: 1.4117\n",
      "Epoch: 004/040 | Batch 0100/0175 | Loss: 1.2894\n",
      "Epoch: 004/040 | Batch 0150/0175 | Loss: 1.1508\n",
      "***Epoch: 004/040 | Train. Acc.: 54.138% | Loss: 1.231\n",
      "***Epoch: 004/040 | Valid. Acc.: 54.300% | Loss: 1.226\n",
      "Time elapsed: 4.16 min\n",
      "Epoch: 005/040 | Batch 0000/0175 | Loss: 1.1781\n",
      "Epoch: 005/040 | Batch 0050/0175 | Loss: 1.2942\n",
      "Epoch: 005/040 | Batch 0100/0175 | Loss: 1.3343\n",
      "Epoch: 005/040 | Batch 0150/0175 | Loss: 1.1216\n",
      "***Epoch: 005/040 | Train. Acc.: 58.536% | Loss: 1.139\n",
      "***Epoch: 005/040 | Valid. Acc.: 58.220% | Loss: 1.152\n",
      "Time elapsed: 5.23 min\n",
      "Epoch: 006/040 | Batch 0000/0175 | Loss: 1.1030\n",
      "Epoch: 006/040 | Batch 0050/0175 | Loss: 1.1732\n",
      "Epoch: 006/040 | Batch 0100/0175 | Loss: 1.1508\n",
      "Epoch: 006/040 | Batch 0150/0175 | Loss: 1.0059\n",
      "***Epoch: 006/040 | Train. Acc.: 58.882% | Loss: 1.132\n",
      "***Epoch: 006/040 | Valid. Acc.: 58.600% | Loss: 1.158\n",
      "Time elapsed: 6.28 min\n",
      "Epoch: 007/040 | Batch 0000/0175 | Loss: 1.0091\n",
      "Epoch: 007/040 | Batch 0050/0175 | Loss: 1.2888\n",
      "Epoch: 007/040 | Batch 0100/0175 | Loss: 1.0148\n",
      "Epoch: 007/040 | Batch 0150/0175 | Loss: 1.0491\n",
      "***Epoch: 007/040 | Train. Acc.: 65.203% | Loss: 0.966\n",
      "***Epoch: 007/040 | Valid. Acc.: 63.880% | Loss: 1.007\n",
      "Time elapsed: 7.31 min\n",
      "Epoch: 008/040 | Batch 0000/0175 | Loss: 0.8920\n",
      "Epoch: 008/040 | Batch 0050/0175 | Loss: 0.9769\n",
      "Epoch: 008/040 | Batch 0100/0175 | Loss: 1.0159\n",
      "Epoch: 008/040 | Batch 0150/0175 | Loss: 1.0733\n",
      "***Epoch: 008/040 | Train. Acc.: 67.181% | Loss: 0.920\n",
      "***Epoch: 008/040 | Valid. Acc.: 65.020% | Loss: 0.974\n",
      "Time elapsed: 8.35 min\n",
      "Epoch: 009/040 | Batch 0000/0175 | Loss: 0.9276\n",
      "Epoch: 009/040 | Batch 0050/0175 | Loss: 0.8630\n",
      "Epoch: 009/040 | Batch 0100/0175 | Loss: 1.1130\n",
      "Epoch: 009/040 | Batch 0150/0175 | Loss: 0.9105\n",
      "***Epoch: 009/040 | Train. Acc.: 66.795% | Loss: 0.920\n",
      "***Epoch: 009/040 | Valid. Acc.: 64.980% | Loss: 0.984\n",
      "Time elapsed: 9.38 min\n",
      "Epoch: 010/040 | Batch 0000/0175 | Loss: 0.8506\n",
      "Epoch: 010/040 | Batch 0050/0175 | Loss: 0.7531\n",
      "Epoch: 010/040 | Batch 0100/0175 | Loss: 0.9312\n",
      "Epoch: 010/040 | Batch 0150/0175 | Loss: 0.9103\n",
      "***Epoch: 010/040 | Train. Acc.: 70.491% | Loss: 0.832\n",
      "***Epoch: 010/040 | Valid. Acc.: 67.560% | Loss: 0.934\n",
      "Time elapsed: 10.42 min\n",
      "Epoch: 011/040 | Batch 0000/0175 | Loss: 0.8196\n",
      "Epoch: 011/040 | Batch 0050/0175 | Loss: 0.7955\n",
      "Epoch: 011/040 | Batch 0100/0175 | Loss: 0.9367\n",
      "Epoch: 011/040 | Batch 0150/0175 | Loss: 0.7501\n",
      "***Epoch: 011/040 | Train. Acc.: 70.826% | Loss: 0.819\n",
      "***Epoch: 011/040 | Valid. Acc.: 66.220% | Loss: 0.950\n",
      "Time elapsed: 11.50 min\n",
      "Epoch: 012/040 | Batch 0000/0175 | Loss: 0.7863\n",
      "Epoch: 012/040 | Batch 0050/0175 | Loss: 0.8496\n",
      "Epoch: 012/040 | Batch 0100/0175 | Loss: 0.7997\n",
      "Epoch: 012/040 | Batch 0150/0175 | Loss: 0.9733\n",
      "***Epoch: 012/040 | Train. Acc.: 73.600% | Loss: 0.757\n",
      "***Epoch: 012/040 | Valid. Acc.: 68.640% | Loss: 0.901\n",
      "Time elapsed: 12.72 min\n",
      "Epoch: 013/040 | Batch 0000/0175 | Loss: 0.8286\n",
      "Epoch: 013/040 | Batch 0050/0175 | Loss: 0.8397\n",
      "Epoch: 013/040 | Batch 0100/0175 | Loss: 0.7478\n",
      "Epoch: 013/040 | Batch 0150/0175 | Loss: 0.8451\n",
      "***Epoch: 013/040 | Train. Acc.: 76.750% | Loss: 0.672\n",
      "***Epoch: 013/040 | Valid. Acc.: 70.260% | Loss: 0.848\n",
      "Time elapsed: 13.96 min\n",
      "Epoch: 014/040 | Batch 0000/0175 | Loss: 0.6818\n",
      "Epoch: 014/040 | Batch 0050/0175 | Loss: 0.7883\n",
      "Epoch: 014/040 | Batch 0100/0175 | Loss: 0.7845\n",
      "Epoch: 014/040 | Batch 0150/0175 | Loss: 0.6714\n",
      "***Epoch: 014/040 | Train. Acc.: 76.462% | Loss: 0.669\n",
      "***Epoch: 014/040 | Valid. Acc.: 69.560% | Loss: 0.876\n",
      "Time elapsed: 15.17 min\n",
      "Epoch: 015/040 | Batch 0000/0175 | Loss: 0.7720\n",
      "Epoch: 015/040 | Batch 0050/0175 | Loss: 0.7569\n",
      "Epoch: 015/040 | Batch 0100/0175 | Loss: 0.6428\n",
      "Epoch: 015/040 | Batch 0150/0175 | Loss: 0.7415\n",
      "***Epoch: 015/040 | Train. Acc.: 78.196% | Loss: 0.622\n",
      "***Epoch: 015/040 | Valid. Acc.: 70.460% | Loss: 0.852\n",
      "Time elapsed: 16.39 min\n",
      "Epoch: 016/040 | Batch 0000/0175 | Loss: 0.6150\n",
      "Epoch: 016/040 | Batch 0050/0175 | Loss: 0.7300\n",
      "Epoch: 016/040 | Batch 0100/0175 | Loss: 0.4870\n",
      "Epoch: 016/040 | Batch 0150/0175 | Loss: 0.6177\n",
      "***Epoch: 016/040 | Train. Acc.: 80.033% | Loss: 0.571\n",
      "***Epoch: 016/040 | Valid. Acc.: 71.500% | Loss: 0.832\n",
      "Time elapsed: 17.62 min\n",
      "Epoch: 017/040 | Batch 0000/0175 | Loss: 0.6556\n",
      "Epoch: 017/040 | Batch 0050/0175 | Loss: 0.6564\n",
      "Epoch: 017/040 | Batch 0100/0175 | Loss: 0.5505\n",
      "Epoch: 017/040 | Batch 0150/0175 | Loss: 0.6272\n",
      "***Epoch: 017/040 | Train. Acc.: 81.415% | Loss: 0.532\n",
      "***Epoch: 017/040 | Valid. Acc.: 71.980% | Loss: 0.836\n",
      "Time elapsed: 18.80 min\n",
      "Epoch: 018/040 | Batch 0000/0175 | Loss: 0.5772\n",
      "Epoch: 018/040 | Batch 0050/0175 | Loss: 0.4951\n",
      "Epoch: 018/040 | Batch 0100/0175 | Loss: 0.4850\n",
      "Epoch: 018/040 | Batch 0150/0175 | Loss: 0.6942\n",
      "***Epoch: 018/040 | Train. Acc.: 82.944% | Loss: 0.486\n",
      "***Epoch: 018/040 | Valid. Acc.: 71.520% | Loss: 0.839\n",
      "Time elapsed: 19.84 min\n",
      "Epoch: 019/040 | Batch 0000/0175 | Loss: 0.4757\n",
      "Epoch: 019/040 | Batch 0050/0175 | Loss: 0.4909\n",
      "Epoch: 019/040 | Batch 0100/0175 | Loss: 0.5568\n",
      "Epoch: 019/040 | Batch 0150/0175 | Loss: 0.5895\n",
      "***Epoch: 019/040 | Train. Acc.: 81.592% | Loss: 0.515\n",
      "***Epoch: 019/040 | Valid. Acc.: 70.840% | Loss: 0.911\n",
      "Time elapsed: 20.87 min\n",
      "Epoch: 020/040 | Batch 0000/0175 | Loss: 0.5108\n",
      "Epoch: 020/040 | Batch 0050/0175 | Loss: 0.5133\n",
      "Epoch: 020/040 | Batch 0100/0175 | Loss: 0.4775\n",
      "Epoch: 020/040 | Batch 0150/0175 | Loss: 0.5364\n",
      "***Epoch: 020/040 | Train. Acc.: 85.272% | Loss: 0.431\n",
      "***Epoch: 020/040 | Valid. Acc.: 72.240% | Loss: 0.850\n",
      "Time elapsed: 21.89 min\n",
      "Epoch: 021/040 | Batch 0000/0175 | Loss: 0.4184\n",
      "Epoch: 021/040 | Batch 0050/0175 | Loss: 0.5490\n",
      "Epoch: 021/040 | Batch 0100/0175 | Loss: 0.4124\n",
      "Epoch: 021/040 | Batch 0150/0175 | Loss: 0.3877\n",
      "***Epoch: 021/040 | Train. Acc.: 86.616% | Loss: 0.384\n",
      "***Epoch: 021/040 | Valid. Acc.: 72.900% | Loss: 0.850\n",
      "Time elapsed: 22.93 min\n",
      "Epoch: 022/040 | Batch 0000/0175 | Loss: 0.3587\n",
      "Epoch: 022/040 | Batch 0050/0175 | Loss: 0.4164\n",
      "Epoch: 022/040 | Batch 0100/0175 | Loss: 0.4908\n",
      "Epoch: 022/040 | Batch 0150/0175 | Loss: 0.5300\n",
      "***Epoch: 022/040 | Train. Acc.: 87.763% | Loss: 0.353\n",
      "***Epoch: 022/040 | Valid. Acc.: 73.160% | Loss: 0.867\n",
      "Time elapsed: 23.97 min\n",
      "Epoch: 023/040 | Batch 0000/0175 | Loss: 0.3409\n",
      "Epoch: 023/040 | Batch 0050/0175 | Loss: 0.3932\n",
      "Epoch: 023/040 | Batch 0100/0175 | Loss: 0.4906\n",
      "Epoch: 023/040 | Batch 0150/0175 | Loss: 0.3842\n",
      "***Epoch: 023/040 | Train. Acc.: 88.516% | Loss: 0.325\n",
      "***Epoch: 023/040 | Valid. Acc.: 72.940% | Loss: 0.867\n",
      "Time elapsed: 25.01 min\n",
      "Epoch: 024/040 | Batch 0000/0175 | Loss: 0.3903\n",
      "Epoch: 024/040 | Batch 0050/0175 | Loss: 0.4127\n",
      "Epoch: 024/040 | Batch 0100/0175 | Loss: 0.3478\n",
      "Epoch: 024/040 | Batch 0150/0175 | Loss: 0.4306\n",
      "***Epoch: 024/040 | Train. Acc.: 90.315% | Loss: 0.284\n",
      "***Epoch: 024/040 | Valid. Acc.: 73.220% | Loss: 0.911\n",
      "Time elapsed: 26.04 min\n",
      "Epoch: 025/040 | Batch 0000/0175 | Loss: 0.2716\n",
      "Epoch: 025/040 | Batch 0050/0175 | Loss: 0.3371\n",
      "Epoch: 025/040 | Batch 0100/0175 | Loss: 0.4309\n",
      "Epoch: 025/040 | Batch 0150/0175 | Loss: 0.4343\n",
      "***Epoch: 025/040 | Train. Acc.: 88.908% | Loss: 0.311\n",
      "***Epoch: 025/040 | Valid. Acc.: 73.000% | Loss: 0.909\n",
      "Time elapsed: 27.07 min\n",
      "Epoch: 026/040 | Batch 0000/0175 | Loss: 0.2467\n",
      "Epoch: 026/040 | Batch 0050/0175 | Loss: 0.2832\n",
      "Epoch: 026/040 | Batch 0100/0175 | Loss: 0.3431\n",
      "Epoch: 026/040 | Batch 0150/0175 | Loss: 0.3218\n",
      "***Epoch: 026/040 | Train. Acc.: 90.547% | Loss: 0.272\n",
      "***Epoch: 026/040 | Valid. Acc.: 72.900% | Loss: 0.925\n",
      "Time elapsed: 28.10 min\n",
      "Epoch: 027/040 | Batch 0000/0175 | Loss: 0.3064\n",
      "Epoch: 027/040 | Batch 0050/0175 | Loss: 0.2874\n",
      "Epoch: 027/040 | Batch 0100/0175 | Loss: 0.3545\n",
      "Epoch: 027/040 | Batch 0150/0175 | Loss: 0.3866\n",
      "***Epoch: 027/040 | Train. Acc.: 92.277% | Loss: 0.230\n",
      "***Epoch: 027/040 | Valid. Acc.: 73.760% | Loss: 0.935\n",
      "Time elapsed: 29.13 min\n",
      "Epoch: 028/040 | Batch 0000/0175 | Loss: 0.1964\n",
      "Epoch: 028/040 | Batch 0050/0175 | Loss: 0.2317\n",
      "Epoch: 028/040 | Batch 0100/0175 | Loss: 0.2595\n",
      "Epoch: 028/040 | Batch 0150/0175 | Loss: 0.3056\n",
      "***Epoch: 028/040 | Train. Acc.: 92.049% | Loss: 0.225\n",
      "***Epoch: 028/040 | Valid. Acc.: 73.340% | Loss: 0.994\n",
      "Time elapsed: 30.16 min\n",
      "Epoch: 029/040 | Batch 0000/0175 | Loss: 0.2118\n",
      "Epoch: 029/040 | Batch 0050/0175 | Loss: 0.2198\n",
      "Epoch: 029/040 | Batch 0100/0175 | Loss: 0.2389\n",
      "Epoch: 029/040 | Batch 0150/0175 | Loss: 0.3052\n",
      "***Epoch: 029/040 | Train. Acc.: 93.170% | Loss: 0.198\n",
      "***Epoch: 029/040 | Valid. Acc.: 73.520% | Loss: 1.004\n",
      "Time elapsed: 31.20 min\n",
      "Epoch: 030/040 | Batch 0000/0175 | Loss: 0.1664\n",
      "Epoch: 030/040 | Batch 0050/0175 | Loss: 0.1880\n",
      "Epoch: 030/040 | Batch 0100/0175 | Loss: 0.1938\n",
      "Epoch: 030/040 | Batch 0150/0175 | Loss: 0.2032\n",
      "***Epoch: 030/040 | Train. Acc.: 93.333% | Loss: 0.188\n",
      "***Epoch: 030/040 | Valid. Acc.: 72.820% | Loss: 1.061\n",
      "Time elapsed: 32.23 min\n",
      "Epoch: 031/040 | Batch 0000/0175 | Loss: 0.2679\n",
      "Epoch: 031/040 | Batch 0050/0175 | Loss: 0.2778\n",
      "Epoch: 031/040 | Batch 0100/0175 | Loss: 0.2026\n",
      "Epoch: 031/040 | Batch 0150/0175 | Loss: 0.2144\n",
      "***Epoch: 031/040 | Train. Acc.: 94.058% | Loss: 0.170\n",
      "***Epoch: 031/040 | Valid. Acc.: 73.500% | Loss: 1.044\n",
      "Time elapsed: 33.27 min\n",
      "Epoch: 032/040 | Batch 0000/0175 | Loss: 0.1634\n",
      "Epoch: 032/040 | Batch 0050/0175 | Loss: 0.2475\n",
      "Epoch: 032/040 | Batch 0100/0175 | Loss: 0.1528\n",
      "Epoch: 032/040 | Batch 0150/0175 | Loss: 0.2810\n",
      "***Epoch: 032/040 | Train. Acc.: 94.471% | Loss: 0.161\n",
      "***Epoch: 032/040 | Valid. Acc.: 73.000% | Loss: 1.065\n",
      "Time elapsed: 34.32 min\n",
      "Epoch: 033/040 | Batch 0000/0175 | Loss: 0.2095\n",
      "Epoch: 033/040 | Batch 0050/0175 | Loss: 0.1590\n",
      "Epoch: 033/040 | Batch 0100/0175 | Loss: 0.1752\n",
      "Epoch: 033/040 | Batch 0150/0175 | Loss: 0.2319\n",
      "***Epoch: 033/040 | Train. Acc.: 95.118% | Loss: 0.141\n",
      "***Epoch: 033/040 | Valid. Acc.: 73.440% | Loss: 1.075\n",
      "Time elapsed: 35.35 min\n",
      "Epoch: 034/040 | Batch 0000/0175 | Loss: 0.1156\n",
      "Epoch: 034/040 | Batch 0050/0175 | Loss: 0.1456\n",
      "Epoch: 034/040 | Batch 0100/0175 | Loss: 0.1519\n",
      "Epoch: 034/040 | Batch 0150/0175 | Loss: 0.1831\n",
      "***Epoch: 034/040 | Train. Acc.: 95.286% | Loss: 0.142\n",
      "***Epoch: 034/040 | Valid. Acc.: 73.820% | Loss: 1.064\n",
      "Time elapsed: 36.40 min\n",
      "Epoch: 035/040 | Batch 0000/0175 | Loss: 0.1532\n",
      "Epoch: 035/040 | Batch 0050/0175 | Loss: 0.1267\n",
      "Epoch: 035/040 | Batch 0100/0175 | Loss: 0.1633\n",
      "Epoch: 035/040 | Batch 0150/0175 | Loss: 0.1263\n",
      "***Epoch: 035/040 | Train. Acc.: 94.638% | Loss: 0.154\n",
      "***Epoch: 035/040 | Valid. Acc.: 73.440% | Loss: 1.093\n",
      "Time elapsed: 37.43 min\n",
      "Epoch: 036/040 | Batch 0000/0175 | Loss: 0.1787\n",
      "Epoch: 036/040 | Batch 0050/0175 | Loss: 0.1622\n",
      "Epoch: 036/040 | Batch 0100/0175 | Loss: 0.1840\n",
      "Epoch: 036/040 | Batch 0150/0175 | Loss: 0.1143\n",
      "***Epoch: 036/040 | Train. Acc.: 95.480% | Loss: 0.132\n",
      "***Epoch: 036/040 | Valid. Acc.: 73.020% | Loss: 1.159\n",
      "Time elapsed: 38.46 min\n",
      "Epoch: 037/040 | Batch 0000/0175 | Loss: 0.1282\n",
      "Epoch: 037/040 | Batch 0050/0175 | Loss: 0.1299\n",
      "Epoch: 037/040 | Batch 0100/0175 | Loss: 0.1869\n",
      "Epoch: 037/040 | Batch 0150/0175 | Loss: 0.1387\n",
      "***Epoch: 037/040 | Train. Acc.: 95.129% | Loss: 0.138\n",
      "***Epoch: 037/040 | Valid. Acc.: 72.640% | Loss: 1.174\n",
      "Time elapsed: 39.50 min\n",
      "Epoch: 038/040 | Batch 0000/0175 | Loss: 0.1137\n",
      "Epoch: 038/040 | Batch 0050/0175 | Loss: 0.1053\n",
      "Epoch: 038/040 | Batch 0100/0175 | Loss: 0.1298\n",
      "Epoch: 038/040 | Batch 0150/0175 | Loss: 0.1280\n",
      "***Epoch: 038/040 | Train. Acc.: 95.429% | Loss: 0.134\n",
      "***Epoch: 038/040 | Valid. Acc.: 73.040% | Loss: 1.230\n",
      "Time elapsed: 40.53 min\n",
      "Epoch: 039/040 | Batch 0000/0175 | Loss: 0.1410\n",
      "Epoch: 039/040 | Batch 0050/0175 | Loss: 0.1084\n",
      "Epoch: 039/040 | Batch 0100/0175 | Loss: 0.1578\n",
      "Epoch: 039/040 | Batch 0150/0175 | Loss: 0.1516\n",
      "***Epoch: 039/040 | Train. Acc.: 97.002% | Loss: 0.090\n",
      "***Epoch: 039/040 | Valid. Acc.: 73.420% | Loss: 1.159\n",
      "Time elapsed: 41.57 min\n",
      "Epoch: 040/040 | Batch 0000/0175 | Loss: 0.1143\n",
      "Epoch: 040/040 | Batch 0050/0175 | Loss: 0.1153\n",
      "Epoch: 040/040 | Batch 0100/0175 | Loss: 0.1493\n",
      "Epoch: 040/040 | Batch 0150/0175 | Loss: 0.2771\n",
      "***Epoch: 040/040 | Train. Acc.: 96.071% | Loss: 0.111\n",
      "***Epoch: 040/040 | Valid. Acc.: 73.520% | Loss: 1.218\n",
      "Time elapsed: 42.61 min\n",
      "Total Training Time: 42.61 min\n"
     ]
    }
   ],
   "source": [
    "log_dict = train_classifier_simple_v1(num_epochs=NUM_EPOCHS, model=model, \n",
    "                                      optimizer=optimizer, device=DEVICE, \n",
    "                                      train_loader=train_loader, valid_loader=valid_loader, \n",
    "                                      logging_interval=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABLpUlEQVR4nO3dd3hT1RvA8e/JaMvee5Ulu6yCgAgoGwERlaUgOBBR/DkRBQXEgeBWBHHjQBAXiuwhe7TsvSxQZssoo7TNOL8/kqYrbVOadJD38zx5mtx77r1vWsibe6bSWiOEEMJ/GXI7ACGEELlLEoEQQvg5SQRCCOHnJBEIIYSfk0QghBB+zpTbAWRV6dKldXBwcG6HIYQQ+Up4eHi01rqMu335LhEEBwcTFhaW22EIIUS+opQ6lt4+qRoSQgg/J4lACCH8nCQCIYTwc/mujUAIkT0Wi4XIyEji4uJyOxThA0FBQVSuXBmz2ezxMZIIhPAzkZGRFClShODgYJRSuR2O8CKtNefPnycyMpLq1at7fJxUDQnhZ+Li4ihVqpQkgZuQUopSpUpl+W5PEoEQfkiSwM3rRv62fpMItNbcO309+05fzu1QhBAiT/GbRPDB0oOEH7tI94/W5HYoQvg9pRSDBw92vbZarZQpU4aePXsCMH/+fCZPnpzhOU6dOsV9990HwLfffstTTz2VpRjeeuutTMsMHTqUefPmZVqucOHCWbp2XuM3ieCpO2vndghCCKdChQqxe/durl+/DsDSpUupVKmSa3/v3r0ZM2ZMhueoWLGiRx/S6fEkEfgLv0kEASYDT3SoidmokFXZhMh93bt3Z8GCBQDMnj2bgQMHuvYl/4Y/dOhQnn76adq0aUONGjVcH/4RERE0bNjQdcyJEyfo1q0bderUYeLEia7tffr0oXnz5jRo0ICZM2cCMGbMGK5fv06TJk144IEHAJg1axYhISE0btw4xd3K6tWr01w7PVprXnzxRRo2bEijRo2YM2cOAKdPn6Zdu3Y0adKEhg0bsmbNGmw2G0OHDnWV/eCDD274d5ldftV9tHCgCYtNE2+1E2Q25nY4QuS6iX/tYe8p77ab1a9YlPG9GmRabsCAAbz++uv07NmTnTt38vDDD7Nmjfuq29OnT7N27Vr2799P7969XVVCyW3evJndu3dTsGBBWrRowV133UVoaChff/01JUuW5Pr167Ro0YJ7772XyZMn8+mnn7J9+3YA9uzZw5tvvsm6desoXbo0Fy5cyNK1E/32229s376dHTt2EB0dTYsWLWjXrh0//fQTXbt2ZezYsdhsNmJjY9m+fTsnT55k9+7dAFy6dCnT35mv+M0dAUDRIEfeuxJnzeVIhBAhISFEREQwe/ZsevTokWHZPn36YDAYqF+/PmfPnnVbpnPnzpQqVYoCBQrQt29f1q5dC8DHH39M48aNadWqFSdOnODQoUNpjl2xYgX33XcfpUuXBqBkyZJZunaitWvXMnDgQIxGI+XKlaN9+/Zs2bKFFi1a8M033zBhwgR27dpFkSJFqFGjBkePHmXUqFEsWrSIokWLZnhuX/KrO4ICAY63ez3BlsuRCJE3ePLN3Zd69+7NCy+8wKpVqzh//ny65QIDA13P06vaTd1tUinFqlWrWLZsGRs2bKBgwYJ06NDBbR97rXW63S49uXZm+9u1a8fq1atZsGABgwcP5sUXX2TIkCHs2LGDxYsXM23aNObOncvXX3+d4fl9xa/uCEwGxx/6l/ATuRyJEALg4Ycf5rXXXqNRo0bZPtfSpUu5cOEC169f548//uC2224jJiaGEiVKULBgQfbv38/GjRtd5c1mMxaLBYCOHTsyd+5cVzJKXjWUFe3atWPOnDnYbDaioqJYvXo1LVu25NixY5QtW5bHHnuMRx55hK1btxIdHY3dbufee+9l0qRJbN26Ndu/gxvlV3cERmci+GTFYZ7vUieXoxFCVK5cmf/9739eOVfbtm0ZPHgwhw8fZtCgQYSGhtKoUSNmzJhBSEgIderUoVWrVq7yw4cPJyQkhGbNmvHjjz8yduxY2rdvj9FopGnTpnz77bdZjuGee+5hw4YNNG7cGKUUU6ZMoXz58nz33XdMnToVs9lM4cKFmTVrFidPnmTYsGHY7XYA3n77ba/8Hm6Eym89aEJDQ/WNLkzz985TPPXTNgAiJt/lzbCEyDf27dtHvXr1cjsM4UPu/sZKqXCtdai78n5VNbQrMia3QxBCiDzHrxKBwSDzqwghRGp+lQhMkgiEECINv0oERkkEQgiRhl8lArkjEEKItPwqERgNSW/3cpyFK3GWXIxGCCHyBj9LBEnPQyYsodGEJbkXjBB+zGg0uiZg69Wrl0/m2ZkxYwazZs3y+nlvRn6WCPzq7QqRZxUoUIDt27eze/duSpYsybRp07x+jREjRjBkyBCvn9dbrNa8M+eZX30yGqWJQIg8p3Xr1pw8eRKADh06kDhgNDo6muDgYMAxLXXfvn3p1q0btWvXZvTo0a7jCxcuzNixY10TyyVODDdhwgTeffdd13lfeuklWrZsyS233OKa5TQ2NpZ+/foREhJC//79ufXWW3E3YPX111+nRYsWNGzYkOHDh6O1Zt++fbRs2dJVJiIigpCQEADCw8Np3749zZs3p2vXrpw+fdoVxyuvvEL79u356KOP+Ouvv7j11ltp2rQpnTp1csUeFRVF586dadasGY8//jjVqlUjOjoagB9++IGWLVvSpEkTHn/8cWy27M+d5ldTTFy32HM7BCHyloVj4Mwu756zfCPonvHqYolsNhvLly/nkUceybTs9u3b2bZtG4GBgdSpU4dRo0ZRpUoVrl27RqtWrXjzzTcZPXo0X3zxBePGjUtzvNVqZfPmzfzzzz9MnDiRZcuW8dlnn1GiRAl27tzJ7t27adKkidtrP/XUU7z22msADB48mL///ptevXqRkJDA0aNHqVGjBnPmzKFfv35YLBZGjRrFn3/+SZkyZZgzZw5jx451TSh36dIl/v33XwAuXrzIxo0bUUrx5ZdfMmXKFN577z0mTpzInXfeycsvv8yiRYtc6yjs27ePOXPmsG7dOsxmMyNHjuTHH3/M9p2PXyWCzvXL8s6i/bkdhhB+L3FRmIiICJo3b07nzp0zPaZjx44UK1YMgPr163Ps2DGqVKlCQECAa4nL5s2bs3TpUrfH9+3b11UmIiICcEwbnTjXUcOGDV3f6FNbuXIlU6ZMITY2lgsXLtCgQQN69epFv379mDt3LmPGjGHOnDnMmTOHAwcOsHv3btd7stlsVKhQwXWu/v37u55HRkbSv39/Tp8+TUJCAtWrV3fF9fvvvwPQrVs3SpQoAcDy5csJDw+nRYsWrt9j2bJlM/3dZcavEkGAURajESIFD7+5e1tiG0FMTAw9e/Zk2rRpPP3005hMJtckbKmni04+HbTRaHTVsZvNZtcU0sm3p5Z4fPIynsy1FhcXx8iRIwkLC6NKlSpMmDDBFVv//v25//776du3L0opateuza5du2jQoAEbNmxwe75ChQq5no8aNYrnnnuO3r17s2rVKiZMmJBhXFprHnroIa9PUOdXbQT2fDbBnhA3u2LFivHxxx/z7rvvYrFYCA4OJjw8HCBb6xF7qm3btsydOxeAvXv3smtX2mqyxA/90qVLc/Xq1RRx1axZE6PRyKRJk1zf9OvUqUNUVJQrEVgsFvbs2eP2+jExMa61mr/77ju3cS1ZsoSLFy8CjruiefPmce7cOcAxXfaxY8du/Bfg5FeJoHyxoNwOQQiRStOmTWncuDE///wzL7zwAtOnT6dNmzauxlFfGjlyJFFRUYSEhPDOO+8QEhLiqn5KVLx4cR577DEaNWpEnz59XNUyifr3788PP/xAv379AAgICGDevHm89NJLNG7cmCZNmrB+/Xq3158wYQL3338/t99+u2t1NIDx48ezZMkSmjVrxsKFC6lQoQJFihShfv36vPHGG3Tp0oWQkBA6d+7saojODr+ahhogeMyCFK9lOmrhb2Qa6iQ2mw2LxUJQUBBHjhyhY8eOHDx4kICAgFyNKz4+HqPRiMlkYsOGDTzxxBOu9ZU9kdVpqH3WRqCUqgLMAsoDdmCm1vqjVGUU8BHQA4gFhmqtc2+ZHiGEX4mNjeWOO+7AYrGgtWb69Om5ngQAjh8/Tr9+/bDb7QQEBPDFF1/49Hq+bCy2As9rrbcqpYoA4UqppVrrvcnKdAdqOx+3AtOdP4UQwueKFCnidtxAbqtduzbbtm3Lsev5rI1Aa3068du91voKsA+olKrY3cAs7bARKK6UqoAQwqfyW5Ww8NyN/G1zpLFYKRUMNAU2pdpVCUi+knwkaZMFSqnhSqkwpVRYVFRUtmJ5qHW1bB0vRH4XFBTE+fPnJRnchLTWnD9/nqCgrHWM8fk4AqVUYeBX4Bmt9eXUu90ckuZfp9Z6JjATHI3F2YmnSdXifLch+92thMivKleuTGRkJNn9UiXypqCgICpXrpylY3yaCJRSZhxJ4Eet9W9uikQCVZK9rgyc8mVMbWuVSfH6UmwCxQvmfuOQEDnFbDa7RrAKAT6sGnL2CPoK2Ke1fj+dYvOBIcqhFRCjtc5+p9gMpF6bJizioi8vJ4QQeZ4v2whuAwYDdyqltjsfPZRSI5RSI5xl/gGOAoeBL4CRPowHwDUUPdGjs/JejwEhhMhJPqsa0lqvxX0bQPIyGnjSVzG4U7KQVAMJIURyfjXFhBBCiLT8avZRpLucEEKk4T93BPsXwDvBcOl4bkcihBB5iv8kgqDiEHcJzh+mW4PyKXbJwBohhD/zn0RQqpbj5/nDfPZAsxS7Xv7Ny0v1CSFEPuI/iaBwWQgsCtGHMBgUT3es7dr185YT7Dt9mYjoa7kYoBBC5A7/SQRKOe4Kzh8CoHfjiil2d/9oDR3eXZULgQkhRO7yn0QAjkQQfRiA4gXNuRyMEELkDf6VCMrWhcuREHeZ0oUDMy8vhBB+wM8SQX3Hz3P7cjcOIYTIQ/w0EezNuJwQQvgR/0oExas6eg6d2ZnbkQghRJ7hX4lAKagcCsdTL5SWZG7YiXT3CSHEzci/EgFA1daOqqHrl9zuHj1P7haEEP7FDxNBK0BD5JZ0i1yLt+ZcPEIIkcv8LxFUag4GExxbn26Rod9szsGAhBAid/lfIggoBBWbwdFV6RbZIstXCiH8iP8lAoDaXeDUVp5tXSy3IxFCiFznn4ngli4A1LuafjvB2kPRbD1+kePnY3MqKiGEyBX+tUJZovIhULg8jeM3A7e4LfLgV0ldTCMm35VDgQkhRM7zzzsCpaB2J8qdXYcRW25HI4QQuco/EwE42gniY2iuDuZ2JEIIkav8NxHU6AAGE3cYt2dadPuJSwz+ahMWm93nYQkhRE7z30QQVAyqtqaDYXumRV/4ZQdrDkXLCmZCiJuS/yYCgNpdqGc4QUWiMyx2+NxVAJRSHDp7xfVaCCFuBv6dCOp0B6C70bORxEpB5w9W0+n9f30ZlRBC5Cj/TgSlaxNdrCEDjSsAndvRCCFErvDvRAAcrNKfWoZTtDXszrRsx/fkTkAIcfPx+0RQrcODnNXFGW78O7dDEUKIXOH3iaBS6ZKU6/gU7Yy7qKeO5XY4QgiR4/w+EQDQ4lFidEGeM/2S25EIIUSOk0QAUKAEi4veT2fjVtoZduR2NEIIkaMkETh1eXQS++xVeN88nRJczu1whBAix2SaCJRSTymlSuREMLlJBRTkWcuTlOAKj5sWZFr+aJQMKhNC3Bw8uSMoD2xRSs1VSnVTSilPTqyU+lopdU4p5bZfplKqg1IqRim13fl4LSuBe5tBwX5dlX/stzLIuIwiZLwOwZ3JupIu33eWF3+RKiUhRP6UaSLQWo8DagNfAUOBQ0qpt5RSNTM59FugWyZl1mitmzgfr3sQr899bu1JUXWdgcblHh/zyHdh/BIeicVmJ+pKvA+jE0II7/OojUBrrYEzzocVKAHMU0pNyeCY1cAFbwSZExLHFe/WNVhvq88w02LMWDM85kxMHJdiE1yv73h3FS3eXMbm//LN2xZCCI/aCJ5WSoUDU4B1QCOt9RNAc+DebF6/tVJqh1JqoVKqQTbPlS1mg+NX0e6WMrQZ8joV1AV6GjZkeEyrt5fT5PWlrteRF68DsPdUjO8CFUIIL/NkqcrSQF+tdYrRVlpru1KqZzauvRWoprW+qpTqAfyBowoqDaXUcGA4QNWqVbNxyfQVCDDy96i2VC9dCAKM7LdXYbjpb35PaAt41CwihBD5kidtBK8BpZx3BqOUUs2S7dt3oxfWWl/WWl91Pv8HMCulSqdTdqbWOlRrHVqmTJkbvWSmGlYqRqFAEyjFF9a7qGc4QQ/DpswPTMXD9nQhhMgTPKkaehX4DiiF4+7gG6XUuOxeWClVPrEHklKqpTOW89k9r7f8Yb+N3fZgxptnUYjruR2OEEL4jCeNxYOAFlrr8Vrr8UAr4IHMDlJKzQY2AHWUUpFKqUeUUiOUUiOcRe4DdiuldgAfAwOcjdJ5QuECQbxqGUY5dYnRpp+RaaqFEDcrT9oIIoAgIM75OhA4ktlBWuuBmez/FPjUg+vnih3ju9B+qpmVAffx0MV5KOA167DcDksIIbzOkzuCeGCPUupbpdQ3wG7gqlLqY6XUx74NL3f9++Id3PH0l8y1tmeIaSm1VKRHxyVvIth9MoZr8Wm7oc7fcYotEdLNVAiR+zxJBL8DrwArgVXAWGAhEO583NyUYrJ1IFd0ASaZvkVh9/jQOIuNnp+sZcQPaX9NT8/exv0zMu6eKoQQOSHTqiGt9XdKqQDgFuemA1pri2/DylsuUJRJ1geZYv6Cx+wLmGnrlWF5k3NMQmKLR/IBZqsPRrH1+EWfxSqEEFmVaSJQSnXA0WsoAkeH+ipKqYecI4f9wvePtGTwV5puhi08Y/qNP2xtOUf68/BNW3mYM5fjuLdZJSApIQAM+Xqzr8MVQogs8aRq6D2gi9a6vda6HdAV+MC3YeUtt9cuAyjGWx/ChJVnTL9mWP7kpet8vPwQ7aeuAsCWdzpDCSFEGp4kArPW+kDiC631QcDsu5Dypk8HNeWELsdPto70M66ioTrq8bE2uyY2IeN5i4QQIrd4kgjClVJfOaeN7qCU+gJ/aCROpWdIRSIm38Un1ns4Q0m+D5icpTWOZ28+4Xb7W//sI/qqzFgqhMg9KrMxXEqpQOBJIHHSndXAZ1rrXPn0Cg0N1WFhYblxaQBqvLyAipxjTsAkArAyIGEcR3Qlj45VKmV7QaIejcrz2QPNvRypEEIkUUqFa61D3e3L8I5AKWUAwrXW72ut+2qt79Faf5BbSSAv2PRKJyJ1WYYljMaIjQ/Mn3ncpTS9nGuxSRuCECL3ZJgItNZ2YIdSyjdTfuZDZYoEAnBQV+F1yxBCDP/R07AxW+eUKeqEELnJkzaCCjhGFi9XSs1PfPg6sLxsyn0hAPxlb81+exVeMf9EEH57kySEyOc8mWtoos+jyGeKF3B0mrJhZLxlKHMCJzHQuIJvbN1v6HwG55wU01Yepl3tMjSoWBSb1piNHi0gJ4QQ2eLJJ00PrfW/yR9AD18Hlpc1qFTM9XyTrscme11GmP664buCxLmJpi4+QK9P13LbOyuoPXYhv4YnzW2099RlPlp2KFtxCyGEO54kgs5utt3YV9+bRKXiBVK8fs9yP+XUJUaY/rrhc46et8P1/HSMY6LX539J2tZn2jo+WHYQu10aloUQ3pVu1ZBS6glgJFBDKbUz2a4iwHpfB5afbNb1+M3WlmdMv3HUXpH59jZZOl4pmBvmfmbTO99bRZMqxUmweT7ZnRBCZEVGbQQ/4Zhl9G1gTLLtV7TWMn9yKi9bHqWSiubjgE8xJNj5w97W42NVBv2GjkZd42jUNW+EKIQQbqVbNaS1jtFaRzgXmIkELDiW6Sos3UnTiieAYQmj2WSvy4cBn/Gh+VMCSfD6dfp9voGJf+3x+nmFEP7LkzWLnwLOAkuBBc7H3z6OK8/7dliLNNtiCeLhhBf509aGPsb1TDHP9Ohce07FeHzdsGMX+WZdhMflhRAiM550H30GqKO1zjMLy+cFt9cuw0Otq/Ho7TWoVLwANV75B4BrFOB/lqc4Yq/Ic+Z5xOhCvGF9kIQM5umLOB+bU2ELIUQanvQaOgF4/pXVTxgNiol3N6RKyYIYDIoejcqn2P+prQ+fW+9iiGkpiwJeohzebVaZvHC/TFYnhPAKTxLBUWCVUuplpdRziQ9fB5bffNi/aYrXdgy8bX2AcZZh1DCcYWHgGLoYtnjtejP+PcK433d77XxCCP/lSSI4jqN9IABH19HEh0gmwGRgfK/6fDkk5eR+P9g60y1+Mmd0KWYGfMAg43KvXTPeavPauYQQ/suTNYvTTDGhlPKkbcHvDLututvt+3VV7kmYyDTzR7xl/goDdn6wuRunlzUrD0RlWmbxnjPEJli5p2nlbF9PCHFzSveOQCm1Ntnz71PtloV3syieAJ6wPMtSWzPeMH/DBNO3XjnvnC3H3W7/YvVRVh+M4vHvw3l2zo40+y02O5+tOkycRe4qhPB3GVUNFUr2vGGqfTJz8g2wYOJZy0j+tYUw1LSEYcaF2T7na3/uQWvN+D93s/3EJW6bvIL1R6J58599DPk6/Xw9e/Nxpiw6wOf/er7kphDi5pRRItDpPHf3WiTTukYpHmnrvproKgV5xPICe+3VGG/+nj8DxlGR6Bu+VrzVTsT5WL7bcIw+09Zx8tJ1Bn2xKdPjYhMcdwLXZC1lIfxeRomguFLqHqXUvc7nfZ2Pe4FiGRzn92YPb8WrPeuz6oUObvdbMXFPwkSmW3vR2HCU9UFP01Ltu+Hr3fHuKo/KPTtnO/N3nAKSbukyW6pUCHHzyygR/Av0Bno6n/dyPnriWLdYZEKlU4HWsFJR4gngHetAesdPIlYHMjdwEk8Y5+PLm63ft53k6dnbiLPYXLFJHhBCpNv7R2s9LCcDuRlVLlHQ7fbkH747dU16JLzFzwFv8JL5Z3oaNzA84TlOUsZncU2Yv4daZQs7YvHZVYQQ+YUsgeVDRkPaW4Jpg5qRekmBCF2BVvGf8oetDQ0Mx1gX9D96G9Z5NZbtJy65nu89fdn1/Ku1/7H31GU3Rwgh/IUkghxUqXgB7gqpkE69vOIZy1M8nvAMAB8HTGOocZHXrt1nWsrEopLVW01dvN9r1xFC5D+SCHLI230b8fPwVpmWW2xvSZ24b1lha8IE8yxGGX/zSTyZ9f89dl7WQBDCX3gyDfX9SqkizufjlFK/KaWa+T60m8vAllWpUtLRZlAwwJhh2XgCeNTyAr/abud58zwGG5d4NZbjF2LZfyapOij1/cnKA+doP3UVfzl7GGXViQuxMiGeEPmIJ3cEr2qtryil2gJdge+A6b4N6+Y27YFmvNDllgzL2DHwouVxVtoa84rpJ542/kYhrnvl+pdiLWmWxrTY7Px7MIrLcRb2OdsQRs3elqLMEz+E80vYCVcX1PTcPmUloW8s80qsQgjf82TOoMQ5CO4Cpmut/1RKTfBdSDeX+5tX5tC5qym2VShWgKfurM2RqGv8vu1kusc6ksEIvgl4h+fM8+ho3Er/hFeJI9CrMV6Ns/LqH7v5ecsJwNG91Z2Fu8+wcPcZAOpXKOrqeSSEyN88uSM4qZT6HOgH/KOUCvTkOKXU10qpc0opt3MlK4ePlVKHlVI7b9bqpqn3N+aPJ29zu++9+xtz6M3uGR4fTTH6JEximrU3jQ1HmR8wjubqgFdjDDt2kWX7zrle7z6ZeS8imaNIiJuHJ4mgH7AY6Ka1vgSUBF704LhvgW4Z7O8O1HY+huOH1U0Gg8JszPxPYMPIVOsARlseI1id4dfAifQzrsSMN6eHcD+iINbDKSi2Hb/In9vTv7sRQuRdniSCCsACrfUhpVQH4H48mH1Ua70aMlyW625glnbYiGMaiwoexHPTeqh1Ncb3qp/u/rm2O7gn4XUAppi/YGXgc9RSkemWz4roqwlut9d/bTEAUVdSNv7ujEy5aN09n63nfz9v90osQoic5Uki+BWwKaVqAV8B1YGfvHDtSjiWwUwU6dyWhlJquFIqTCkVFhWV+Rz8+U29Co46+Yl3N2Rom+AMy+7R1akZ9z0vWx6hLBdZFjiaF00/Y8S3VTUrD5xL8frvne4bjF+at9OncQghvM+TRGDXWluBvsCHWutncdwlZJe7ruxu6ye01jO11qFa69AyZXw39UJu+ePJNuyZ2BVwDPQqUdCx0H34uE5sePnONOVtGJlt60i/hPFE6aI8aZrPmsD/UVnlXJJcf+S82+1zwk643S6EyLs8SQQWpdRAYAjwt3Ob2QvXjgSqJHtdGbixjuv5XKDJSKHApA5cBueoX42jh1F6tutatIifwQTLECqqC/wTMIZvze/QUHl3jYErcRZGu/mmf17GCghxU/AkEQwDWgNvaq3/U0pVB37wwrXnA0OcvYdaATFa69NeOG++90CragAUDvRsRdBvbd3oFD+FFfamdDDu4O/AcYw1/UAg7uv9s6rf5xvdbo+z2j0+x6Ldp9l9MibzgkKIHKc8mY9eKRUAJI6AOqC1tnhwzGygA1AaOAuMx3knobWeoRyT3XyKo2dRLDBMax2W2XlDQ0N1WFimxfI1rTV2nTRpXfCYBR4fW5LLvGuewZ3G7eyyBzM84XlOU8oncfZqXJFPBjZNN8aIyXe5nifuT75NCJFzlFLhWutQd/sy/crp7Cn0HRCBo16/ilLqIWevoHRprQdmsl8DT2Z2fX+klMLopgWlSKCJK/EZd+e8QFEetoymoy2cD8yfsSFoFJ9ZezPFOsDrcW466r6dQAiRv3hSNfQe0EVr3V5r3Q7HNBMf+DYskdwc52R19iysIrPc3px+CeM5qUsx0jSfN0xf0dWwmQAyvZkTQvgZTxKBWWvtGsqqtT6IdxqLhYfKFwsCSLGOwdT7QqhcIv2GZID9uiod499lnq0dD5qW83nAh2wOHMlE0zc0UwdReF7H7865K/GcuxLHvHD3YxksNvfn7zNtHS3f9M1cRCcuxPrkvELczDxJBOFKqa+UUh2cjy+AcF8HJpIoZ09bW7I7gvtDq6RX3KVDnTLEEcgLlhF0jJ/KS5bHOKIr8pBpKb8FTuD3gPHZblBu+eZy/khnvqRDZ6+63b79xCXOXfF+j6MFO09z+5SVrEo15kEIkTFPEsEIYA/wNPA/YK9zm8ghZYsGUiTQxMvd6zpeF3FMOlcoIOMmnv7JksURXYk5tju4N2EijyY8z1Z7LZoYjjDW9CPZXbBy7eFot9tPXMzZb+c7Iy8BsP/MlRy9rhD5XYafJEopAxCutW4IvJ8zIYnUgsxGdk3syumY60z8a69r4flShQMc/bFSua1WKSb2bpDuN/Jl9uYsS2jOeNN3DDMt5nbDTlbZm3CVAnxgvQ+7l9YrWnXgHAlWO1fi3Ddwrz0UzXWLjT+2naRVzVLUr1CU5tVKZHhOrTWX46wUK5C2djKxDSWzRXeEECllmAi01nal1A6lVFWt9fGcCkq4l1gzZFDuP+o61y/H0r1nqVisALXKFkk3ESSaaB1CLIE8aZpPdYNjTqF7jGsZkjCGo7pituPdcSKG2ZvTH2n84FebXM8X7HIMIcmse+kPm47z6h+7WfF8e2qUSTkNduLvJ51fjxAiHZ5OOrdHKbVcKTU/8eHrwERaqb/xpv7Ae7FrHQC6NyoPwOFzGScCUEy1DqBD/Hs0ifucKZb+VFbRrAh8gSeM2f8T7z2ddjrrhEwGoW2JuMAd765Kd9bT5fsct0ARbpbSTKzgSi9RCiHc82To6kSfRyE8kvSN1/FB16leOdYdTurLX7ts4RTfqKM8nAIiQjumjvrMdjcr7E1ZFDiGl8w/c0qX5E97Wy9F73DLuIUZ7r9/xgYAPv/3KA+2qsbv2yJ57PYarveckax0rxVCJEn3jkApVUspdZvW+t/kDxxfvLwz97HIkgCT489Vqbij2+jQNsHsGN8l3fINKxYDoP0tnk/Ut19XpWbc92yx38IU80yeNP5BQeKyEfWN+Wj5IR6bFcZb/+xn32nPGn9TJ0ohhGcyqhr6EHD3PzDWuU/ksHJFg/h0UFNmDG4OOD7wkjeapv4ALBBgBKBGmUJZuo4NI48lPM8yezNeNM9lQ+BTPGeaSylydq6g7ScuAXD2ivtE9M26/4i5nnaAnKQBIbImo6qhYK11mikntdZhSqlg34UkMtIzxPNG3B6NKnDy0nWGtK7GN+sisnSdSxThScszzLLuY6hpEU8Z/2SIcSlr7I2wYeCAvSrTbb3IiY/dYd9sSdOIvOm/C3z+71HCj13k00GOVU4T580ySCYQIksySgRBGezLeEiryBOMBsWI9jWzdY5Nuh6bLPW4RZ3gFdNPdDRs4yoF6GNcT0fjVsZZHqYwsYTpOvgyKbR4cxmfDGzqqv6JtzganZPfEdilakiIG5JRItiilHpMa/1F8o1KqUeQkcV5ymcPNON0TMb1+ANbVqFmmcLY7JoKxQvwwi87Mu3Bk9xBXYWhlpecrzSfmj+hp3EjiwLHuMpMtAzmJ1tH4gm4kbeRoagr8QyYmTQdtrteRdrZb0jygBBZk1EieAb4XSn1AEkf/KFAAHCPj+MSWdCjUeYLxr3dNyTF696NK3Ik6iod3/s3xfaShQK4cC2zaScUT1lGMcfWgbrqOIVVHF0MWxhv/p57jGt5OGE00RTL6tvIkrlhjv4Kyb/9S2OxEDcm3cZirfVZrXUbHN1HI5yPiVrr1lrrMzkTnvClAGPKP//dTSryWs/6Hh6tWGMP4QtbTz6w3sddCW8z0TKYBiqCuQETKcNF7wfsxuqDUdw3fT2QNI7g1KXrHInKbAxFSna7xm6X7qfCP2U6oExrvVJr/YnzsSInghK5w6DUDffFt2PgG1t3BiS8SiUVzWcBH1GcK5Qk7aAybws7dpGDZ6+wdK9jsNn0VUfo+N6/fLvuP4LHLCA2wcr1BJtrZtI4i41n52zn3OWk6rQ2k1cQ6qMZUYXI6zxbC1H4BUXKqa5vxBZdl3HWh5lqnsn2oMcBiNLF6Bj/LpfJWjfWrPh1ayRRqWY0/WLNfwBcuJbAi7/sZMPR83SpX442NUvx+7aTaK35cIBjhbUzl3N+rIQQeYUkAuGilKKMc2bT1NrfUoYWwSV4d8nBTM/zi60DUboYXQ1hlFBX6Wbcws6gxzhmL8s468NsstcjwctLWnz+79F09x2/EMsG52pqS/aedVUbyVQUQjhIIvBjgaaUNYMDW1YhNLgksx5uSYLVzqOzktaGfqRtdXY5F59/vH2NDD94AVbZm7LK7vi23dx6gHuNa+hrXMP3AZOJ12a+tnXjU2sfrvmwJ/LJS9cBGPTFphTbpZupEClJIvBjZYsG8dGAJtxWqzSlCyfdCbRzTkmx6oUOdHh3FQBligRico7UMmbwAepuXeVwXYdwax3esg5yJYQnTH8xzLiIlfYmzLV1YJu9Fpco4uV36N5l59iDX7dGcneTiineuxD+SBKBn7u7SaV09wWXLsTMwc0pFGiiXoWiBJcqxImLsTzRoSafrTri9pjJ94bw5E9b3e67SkG+s3XlO1tXWqj9DDItp4dhM92NW7iuA5hoHcJSW3NiKITVh/80zyfrHvvr1kgaVUrq6mqx2TEbvbMegxD5hSQCkaEuDcq7nhcIMPJGn0YZlq+UyTrKibboumyx1OUtHqCDcTvDjIuZbP6SyeYv2WGvQb+E13wyMM2d2ASb6/nxC7HUTLXOgRA3O/nqI7yqSZXiWSofRXF+sXWgd8Iknkj4Hz9bO9DYcJSp5s9zpOspwPtLkxrAO773Lz9vPs4pZ/sCgNVmR2vHOIPryZKGEDcLSQTihrx3f+M021oGlwTg9tqls3w+KyYW2m9ljHU4n1l709u4ga1BI/glYAIN1H/Zjjcrxvy2K2mQmtbUGruQCfP38MGyg9R7bRGX49LOeCpEfqZ0PlvMIzQ0VIeFhWVeUPjczshL9P50HQBFgkzsmtAVgOPnY2k3dWW2zt3VsIU7DNsYYFoFwDF7WTbb63KZQnxi7ePzhmWlYNLdDRn3x+40+1a/eAdVSxX06fWF8DalVLjWOtTdPmkjEDesQcViDGxZlcdur55i/WBvfEgutrdgsb0Fb1kH8ZF5GncYd1DNcA6AEMMRHkx4xadtCFrD56vdN4j/vesUIzvUSvfY5+Zup2rJgjzT6RbXtqvxVgJNBmmIFnmS/KsUN8xoULzdt1GaReS96TKFGWZ5ieC4n6ge9wPPJjxBC8NBDgQNZVPgSAYal2PC/frG2XXiwnW326/GOa63+2QM01YeTrP/t60n+XDZoRTbGo5fzBM/uO9NJURuk0QgfKJ04aRv6+Puqud6Xr5oRstcZExj4Hf77Tyd8BR77dW4qIvwtvkrVgY8TxvDbpKmnfMtDRyJukrPT9YydfEBj49btu9sitdv/7OPb9flbPuHEO5IIhA+4hh0NrpbHR5pW52IyXcRMfkuWtcsBThmOr1R8+1t6JHwNt0SJvN4wjMUUbH8FPAWvwWMp7Vhj1eiz4iCNNN334jPVx9lwl97sx+QENkkiUD41H3NKqeYyiHxWS2vVCcpFttb0jH+Xd6xDKC2OsnsgDcZalzEo8YFVFO+mS099WC6l3/b5ZPrAPy5/STBYxYQZ5Fuq8J3JBEIn0j87E9dWVPaOald7XJFiJh8F9W80LB8nmJMt/WmbfxHHLFXYIJ5FuPMP7IkYDQzze8x3fwBywOeZ6ppBkXJ2joFnpi9+TgHzlxJsz14zALqvrqQmNgb72767hJH1dNZmR1V+JD0GhI+kfjNP3Xv5Oc630L10oXo2qAcAP88fTsjf9zKvwejsn3NGAozIOFVOhi3E2EvTy/jBvoY11FUxXLcXob7TatpYTjAgIRxnKFUtq+X3DU3S2cCxFnsNH59SbbPr3y4HrQQckcgfCKxLaCA2Zhie5DZyMCWVV3VRYUCTXwztIVrojt3ptwbwqQ+DQFYM/qODK+bOFJ5i67La9ZhhMR/SeO4mbRL+Ij+8a9SQZ1nTeAzvGX6ErMXexv1/Wx9ts8RFnGBDlNXcvBs0t1FPhvmI/IpuSMQPjHlvhBG3VmbYgUzX3fAYFBMf6AZP2w8xtsL91OxWBCnYhxVIf/rWJt+LaoAMLhVNaw2e5ZjicHRHrFJ16NvwkQeNC5jkGkFjQxH2WGvyRUKclRX4Bdbe8jGN+8v1xzl0dtrZFpu7pYTjP51Z4ptu0/GcN+MDQBMXXyAL4Y4xv0krcN8w2EJkSlJBMInAk1GapX1vEG4UKCJx9vX5OG21VHAyB+3cvZKPM92viVFueyuIbBHV+dl62NssDdgpOlPHjQtd+1rrI6wxt6IxfYW3EhCeGPBPo/e88crUo4xePm3nczefML1eunesyzec4auDcq71lTIqtMx1/l2fQQvda2LwSBZRGTMp4lAKdUN+AgwAl9qrSen2t8B+BNI7Ez9m9b6dV/GJPK2xJG3M4e4HQnvNfPtbZif0AYAIzZmmt/nQdNyHsSRGL62duNn2x0c1FWydN6h32zJtEzkxZQf7smTQKLHvw/nv7d7ZOnayT03Z4drac7m1Ure8HmEf/BZIlBKGYFpQGcgEtiilJqvtU7dcXqN1rqnr+IQN5fU3203v9KRlm8td1vWUzaMPGZ5nlrWk7Q07Gek6U8eNi3iYdMirulAfrW14z3r/a4qpuz4bFXakcjpyU77gMVZhZbdNaiFf/BlY3FL4LDW+qjWOgH4Gbjbh9cTfsBgUDSoWNT1umw2RionZ8fAQV2FH2ydaRP/Ke3iP2C85SF22GsyxLSUOQGTqEh0tq8zZZHnI5HtyTLBkairWG32FNNjA8QmWLF78dP+u/URBI9ZwBWZYdWv+DIRVAKS3/NGOrel1loptUMptVAp1cDdiZRSw5VSYUqpsKio7HczFPlbeqOSVzzfngdbVXW9fvrO9CeGy8xxXY7vbF0ZZBnHAwkvU1FFMzfwdWqqkzd8zqyqNXah6/nQb7Yw6e+9tJm8gnunr+dqvJXN/12g/muLqfHKP+meI6t3Fd+tjwDg3JX4GwlZ5FO+TATuWqhS/7PcClTTWjcGPgH+cHcirfVMrXWo1jq0TJn0uxkK/5C6T/2b9zTkq4dCqVGmMBN6JX2XeLhtdUZ3q0ON0oUA6NuskmtZyiGtq3l8vXX2RgxMGEcgFmYFTKYcF7zwLrJu5QHHl6DwYxf5YOlB+n2+Id2yiW3qV+IshB/LnXhF/uHLxuJIIHlLW2XgVPICWuvLyZ7/o5T6TClVWmud/XtwcdNK7JL6eHtHV80Hbk36UDclm+a5eMEARnaolWLK6N6frgXg3maVmbXhmMfX3KOr80jCC/we8Bqbgp4iUpfmii7IIV2JZbbm7NVVOawrkZ3up1kxe/Nxt9s3Hj3PgJkbXSO2h38fjs2u2fd6NwoEGN0e446MX/AvvkwEW4DaSqnqwElgADAoeQGlVHngrNZaK6Va4rhDOe/DmMRN4L5mlQG4p6m7msYbs+iZ2+n24ZoMy+zUNRmYMI7Whr3UMZygh3EzNfQpehsd38xX2Rozw9aLolyjhLrKZV2QnfYanMQ7d7HHL8S6nsemWjIz/NhFmlQpzlM/bQPg2HlHWZuz/cCW6pP98LmrvLv4AB8PbEqAKW3FQGyClR0nLtE4i0uPivzJZ4lAa21VSj0FLMbRffRrrfUepdQI5/4ZwH3AE0opK3AdGKDz25JpIscZDIp+oVnr1pkZTxes36zrsdlWD2yABUxYaaoOc5txN48b/6aDcUeK8vHazDjrsGwPVsvMvdMzHtmceijBK7/tYnPEBbYev0irGmmn2xj541YiL15nz8SuFAqU4UY3O5/+hbXW/wD/pNo2I9nzT4FPfRmD8D/fDG1BiUJZW73MbDQwZ3gr+s/cmKXjrJjYouuyxVqXb61daW44iAk7Foxc1gV53jSPqeaZPGBczsMJL3KBIuRU9VFyqdtVDM6bgDQ9jpzFEsc6JFjtFAr0dXQit8lcQ+Kmc0fdsjTxoEpj0ysdU7y+1c0346y4RBGW25uz2N6CFfZmhOm6PGh5mWnW3jQxHGFr0Aj2BQ5jTcD/GGGcT04tpANQ77VFruk5gscsYONRRwOyN8cZhEVc8GpXVpFzJBEIv1WuaBB3hVRg+gPN0uyb0Ku+x+f5MoNR0DaMTLUOoG/8BFbZGvOPvSVnKMEY889MN39ISS5TkWivToCXnqPR19LG56yJvRxnIXjMAo5GpSyz9/TlNMecvHSd4DELWJ5sxbWVB85x34wNfOvsfuqp37dFpphkT+QOSQTCr7xzbwid6pWjXgXHoLRpg5rRvVGFNOXuqFvW43NWKlEg0zJb9S0MtbzE85aR3J8wnrctA+lqCGNr0AjWBz3NxsAnedX0PeV92Feiywer+XN7ynEQT8/exvBZYUSlM27ggS83pdm288QlAB75Loxenzh6YSVWJR2JusoHSw/ScPxij2J6ds4Ounyw2tO3IHxEEoHwK/UqFOXLh0Ld9pQBqFLS8aGuUKwfc6ePolB8butFr4Q3+MbalVW2xhzRFXnEtJCNQaN4xzST2ioSkw/uEpbsSblucsx1C0v2nsVwg5P57ToZw6LdZ1L0N/1o+SGuxmcee+gby27omsL7pDuAEMkk77NWsXgBjrzVA621a5RvrbKFOXwu5SpnBTPpn1+yUAAXriWk2b5HV2ePtbrrdYg6wkumn+lvWkV/0yoALusCfGi9j69t3W/wHaW0YNdpt9uzkgae+HFritcjfgjnhS6OWWIzyydHoq7y3JztzHrkVqKvyujlvEISgRDJ3Fq9FJEXIykc5PivYTQoQPHbyDYEGA3Ur1CUvacv09NZJfLpoKZUK1Uow3POG9GaOz1Y7H6nrskDlrFUsJynkzGc0iqGpuowr5m/p5nhINOsfdinEwfPaQKwkkDm6z14osO7q9LdF37sIs2rlQAc6ya48+6Sgx5dZ9Lfe9kRGcPUxfuzHKPwHUkEQiTzVt+GPNGhBiVTdT9tVrWE63nDSsV4plNtPlx2iG4NygMQMfkubhm7kIRkC+esH3MnV+Ot1PBwjEKi05Tie1sXAAzYeVV/zzDTYnoaNxGrA4nHTAnluCs5ai/Pj7ZOfGXrjq+6pf4SdoJAk4EfNh7LdExBZktqhkdcBOCHje5HRovcIYlAiGQcC+oUybTcM51u4ZlOKRfNaVylGFucH3QRk+9KsW9M97pMXpj1b8F2DEy0PsQ8W3vaGXZSUUVTXl3AgolDuhJdDOG8av6B50y/cEyXZ629IdcJZJu9FqvtIdi90Az485YT/Lwl7ZoJmdl49Dy1yxamVOGkgQjZXWlNa827Sw7Qq3FF6pYvmvkBwiOSCITwks8Hh9Js0lIqu+lFFFK5WLbOvUcHs8cWnGb7x/TlHuNaOhm2UlZdZLhpgWvfAXtlhia8xGmyNz4iK77fmDR/04CZG6lZphDLn+/g2pbd1dKuJdiYtvIIszYcY9eErtk6l0giiUAILylZKID9k7p59K03wGTg4wFNGfFDeLauacPIPFt75tnaA46qpFaGvdRXx3jBNJcNQaP4yXonM213EaHTdpP1tSOpxiVk1Dvp/SUH+HvXaVYkSxzpkYFr3iWJQAgvCjJnPsPnqz3r8/BtwSilKBpk4nKc97qJ2jGw3t6Q9TRktT2EKeaZDDKtYJBpBetsDVhtD+EKBQmz38JBXZmcmO5Ca83W45doXq1EisV2kpv0916+Wvuf233uXEs16Z4nBn+1ibOX41jybPssH3uzk3EEQuSABhWSqoYeaVsd5fxmnLpR2psO6ir0SZhE+/j3mWHtRQV1npfNs3nL/BVLAl9iX+Aw/gh4lUbqqM9iAEd10b3T17Ni/1kuxbpf+Sx5Epi1IYLrbj7oLcka4gFWH3S/SNX2E5cIHrOAI1Epu/muORTNwbNX3R6TU85diWPj0bw3wbLcEQiRA4oVNLPi+fZpplOoVKIAEedj05Rf+9IdtH1nJQDLnmtHp/dvfPTtMV2eydaBTGYAlYimjIrhVsM+6hmO0dqwl78Cx3HUXp5i6hqHdSUCsXBeF2WpvTl/21pxlYI3fG3ANe7i4W/DPCr/2p97eO3PPawZfQdVSjquvedUDHd9vJb7mld2lRvy9eY0jfIAv22NBGDNwSjXrLLJ2y5yU+9P1nHmcpzbuDOy+mAUDSoWTdHw7k2SCITIITXKFE7TlfTbYS2pPXYhHw1owv9+3u7aXrlE0odvhWJpG58Pv9k9xVKWbq9XulCq+YUUJynDSV2G7bZaYIPSxNDfuJJQwwGs2kRldY5rFKCu4Tgdjdt4xvQr4fbaXNaFOKNLsl9XZbu9JmcpgafVSllZACi526esZNhtwXy3PoI3+jQCYF54ZIoydrvGYFBYbHbirXYKB5pcazAYnQ3TuyJjePWP3R5dc93haB74chMLnm5Lg4rZa+B358zluCwfY7HZGfL1ZuqWL8KiZ9p5PSaQRCBErjIbDa5vh3c3qcT1BFuaEbfJ++6vfKEDJoNKsRJbeoa0rsaEv/ZmWCaaYkyz9XGsr5CMATtdDGH0N66knjpOccNVSqqkapUoXYw/bLfxubUX0Xj/AzPRN+siAHjl911u989YfYQAo4Fl+86y8egFwsd14sdNjjEK/x6MYnDrYA54MKldvNXGqUtxLN5zBoAt/13IViKw2zULdp1m1OxtfNi/CX2ysYhSYrtK6gkBvUkSgRB5SIEAo6s6JLk+TSpSpWRBqpdOGsV8R50yrDtyHgXEW1PWn3eqVzbTEc8ZsWNgkb0li+wtXduKcpWa6jTNDIdoadjPMOMiBhpX8Lm1J9/ZunKZjK6X2Ejs3cbpKYsOpHjdPNn8Rcv2nSMm1uLRVBaj5+3kz+2n6NOkIpB0N5HczshLFAkyp/gbAHR8bxWPtK3BoFururZ9v/EY4+fvAeCZOduzlQgSaR9OWy6JQIh84MMBTdNs+2aY40P6aryVhuMXU7tsYQ456+O/fKgFAJ8MbMqo2du8EsNlCrNN12abrTZf2XpQQ53iFdOPPG+ex1OmP9hkr8dlCrLHXp2V9iYc1JXpZVhPb+MGmhsOUlxdY6+9GmH2W1hjb8RKexOsPv4Iavz6kjTbgscsoG+zSrzfr4lr28JdjjuBxB5cH684zODWwSmO6/3pOgD+e7uHq7EfHF1kX/l9F4NurYrVZsdkNLDywLkM47qeYKNAgJHoq/F8uy6C5zrfku4Yi8xGa3uDJAIh8qi/R7X1aKW1woEmIibfhdaa37aepGnV4q59ocEl3B6zeWxHWr65PFvxHdUVedTyAo2s/9HXuIaOhq1UNUTR07iJl/jZVe6EvQyLbC2wo6hjiOQB4zKGmJYSqUvzRMIz7NI1shXHjfht60me71KHSsUL8Mrvu1xTg+yMvASQYlruOIuN1/9OqmL7au1/HL8Qyys96qXoLrz6YBRDvt7My93rsupAyh5NwWMWsDnZQkj1XlvEmtF38OaCfSzac4Zba5Tk9tpJa1trrQk7dpHQaiV8eieQSBKBEHlUw0pZq6NWSnFvsl41kDSAq0yRwBQfbmWLBGU/QMdV2aVrsMtag4k8REWiMSg73Q2bKaUuc0RXTLNec0HiuN2wk9fM3zMvYAJvWB/kB1sndA73Zr9t8goiJt/FT5uS5j2Kvpo0S+xDX2/mu4dbMnP10RRl3liwD4DgUoV4uG3S7LGJdwFvpzOVSMu3Uibefacvu7rExltSVu39s+sMT/60lcl9G3mlWikzkgiEuIkl1mDoZAO5dk5wTGi39Nl2PPT1Zk7FpO3JUqpQADXLFmbzfxeydL1TlAYNX9h6plsmliAW21uyOb4u75unM8n8LY8a/2GtvRFndAlKqitE62Kstzdgjw722gyr7py8dD3dff8ejCJ4zIJ09x84cyVFFdDSvWfTLevO8O+TRpWfuBjL8FlhfDigCQUDTMz49wgA/yXr9ZXOWDyvkEQgxE3M6MwEdg0j2tckzmKjaJDjg7V2uSLMe6IN98/YkOYDMfzVzgApPgjH9qjHm//s81psFynKw5YX6WXbQD/jKnoZN1BUxXJNB1JIOe5e4rWZeMxEacfdkRkrMRRil70GK+1NiNDlMWLHgJ3GhiPssNdkrw72OIbbJq+44fjnhJ1gTljSZHyJq7TdiInO3l2Ldp+hb7PK7HIz3bfVrrkab6VwJjPA3ghJBELcxAo4F83pUKcMY7rXTbO/YvECrBtzp+sD//tHWqYY1TvurnquqpCgACMRk+/K8FtyVmkMzLffxnz7bYDGjA0LRiqraJqrA4QaDqJRlFGXsGLEipFyXKSPcR2DTO4/xDfZ6zLTehfL7c29FmdOST16+viF2BTrRu88cYk2tUp7/bpK+/J+wwdCQ0N1WJhnIxSFEI7qjzKFA9NdnhOSvvm7G/GauO+XEa1pEVyS4DELKF04IEV9emq/jWxD38/WexRfv9DKzA2LzLxgMgWJ4xYVSXV1mjgCMGNln65GB8N2HjItobKKJtxem1O6FGZsrLfXZ629EUd1xSxdJzdklmyzOio5kVIqXGsd6m6f3BEIcZOrVDztyOTUFjzdFpMh48baFsElAdg9sStGpZgbdoI9p2LSfIh3rFuWplWKM6lPw0xH9E7u24jiBQOynAhiCWK7rsV2XSvF9kO2ysyydeF50y/cYdhObcNJQNPNuAWArfZaxOkAYihEIeK4QgHO62IUUbFUUBcwYiPMXocfrJ04SRk3V3ZHU5B4ChBPAFYKq+uU4Ar7dDWu3MD0HKnnSMoJckcghMhQRncLyfcnSl4uvW+2m1/pyLUEG9VLF2LR7jPZno47M5XVOe4xrKWTcSsWTBTjGtcIogixlFYxxOhCRFEcO4rm6hAAe3Q1jGiKOsc/rLE3YpO9Hod0JcpzkZGmP+liDKM41whUaSfTs2vFWntDltqbc1hX4oi9IucoTnYH1ckdgRAix73fr3GGvYc2vdIRs9FAs0lL0+z7ckgol65beOGXHSm2ly2a1H21dc1SVCpegGsJ1nRnJ02ueEGzR+WSi9Rl+cTWl09sfTMtW4ko+plW0UQdwY7imC5LQxVBF3PKZJWgjSy2t+CkLs0lXZh4zFwjiELEcVaXoJHhP3ob19POmDQ9xmF7RV61DmODvUGW4vc1uSMQQnjFzNVHiE2wpVnCE5LuDJ7pVJtWNUrRqkbaVdOmrTzM1MWOKSOaVS3O1uOXALiveeUUk801r1aCPk0q8uqfe9Kco0rJAgxuVY23/sn6sqAZ01RW0XQyhFNCXeGqLsAKe1OO6Mz6+GvKcZGahlM0U4d4wfwLAHvs1Tihy3KNII7Zy3FClyGOABbab800ErkjEELkWcPb1cy0jLskkeiJ9jWZuvgAVUoWoFKJgmw9foku9cvxfJdbUiQCu9YMbh3MthOX+G3ryRTnWDP6TvafuZwmEbSqUZKNR7M2JiIlRaQuw7e2blk+7iwlOWsvyXoa8qWtB4OMK+hu3ESwOkNxdZV7jWsAOKVLsjA+80TgC5IIhBA+90qPujSvVjLDMgaDYuPLHSkUaMRm1zSvWpyH2jhWcvv3xQ4oFO2mrqRniKPnz9ge9VIkgrmPtwagbvmiHH6zO4/OCnNN9fDz8NYMmLkhm8kg++II5Gtbd762dXdtK0UMpdRlLunCGRzpW1I1JITIN64n2AgyG1BKobXm1T93c1/zKtQoU8g1UC5RvNVG6BvLmHJvCN0bVSDOYuNSrIVWb7ufY+mNPg35L/oaX639jyJBJq4kW0L0w/5N6FS/HA3HL/bp+8tMaLUSzHuizQ0dm1HVkCQCIYRf2X7iEn2mOWYS7R9ahVEda6GUcnWzXb7vLKHVSqaYufTQm90xGw18vyEiRdtEkUATV+LdrzltNirCX+3MyYvXWXsoOsWo7IIBRmKTDdyb2LuBa9rqjLx5T0MeuLVa1t6wU0aJQNYsFkL4lSZVivN+v8YAFAw0UrlEwRRjLTrWK0exgmb+eqqta1viVB0FA1LWpi94+na6NSgPkGadgin3hVA0yEy9CkV59PbqzHgwaaTzH0/e5nr+dMfaPNQmmDrliriNd0T7mvz3dg9+fPRWBrWs6rZMdkkbgRDC7/RuXJHjF2J59Pb0p8BuVLkYK55vz7rD0a61Au5uUpFjF2K5u0lFKhUvQJDZyLQHmvHR8kMMaxPMrpMxvLvkADsjY6heOqnOXylFt4blXa9vKVeEI2/14PiFWKo5FyJ6q29D7p2+IU0ciVOD3OaDqSVc8UnVkBBCeI/WmhMXrlO1VNpRxS/N20mpwgGM7pZ23idIOQDv66GhlC4cSEjl4l6JK9faCJRS3YCPACPwpdZ6cqr9yrm/BxALDNVab83onJIIhBA3K7tdM23lYbo3Kk+tsu6rim5UrowjUEoZgWlAZyAS2KKUmq+1Tr6adnegtvNxKzDd+VMIIfyOwaAY1bF2zl/Xh+duCRzWWh/VWicAPwN3pypzNzBLO2wEiiulKvgwJiGEEKn4MhFUAk4kex3p3JbVMiilhiulwpRSYVFRUal3CyGEyAZfJgJ3U+ylbpDwpAxa65la61CtdWiZMp5ODSuEEMITvkwEkUCVZK8rA6duoIwQQggf8mUi2ALUVkpVV0oFAAOA+anKzAeGKIdWQIzW+rQPYxJCCJGKz3oNaa2tSqmngMU4uo9+rbXeo5Qa4dw/A/gHR9fRwzi6jw7zVTxCCCHc8+nIYq31Pzg+7JNvm5HsuQae9GUMQgghMiZzDQkhhJ/Ld1NMKKWigGM3eHhpINqL4fhafoo3P8UK+Sve/BQr5K9481OskL14q2mt3Xa7zHeJIDuUUmHpDbHOi/JTvPkpVshf8eanWCF/xZufYgXfxStVQ0II4eckEQghhJ/zt0QwM7cDyKL8FG9+ihXyV7z5KVbIX/Hmp1jBR/H6VRuBEEKItPztjkAIIUQqkgiEEMLP+U0iUEp1U0odUEodVkqNyaUYvlZKnVNK7U62raRSaqlS6pDzZ4lk+152xntAKdU12fbmSqldzn0fO1d683asVZRSK5VS+5RSe5RS/8vj8QYppTYrpXY4452Yl+N1XseolNqmlPo7H8Qa4bzOdqVUWF6OVylVXCk1Tym13/nvt3UejrWO83ea+LislHomx+PVWt/0DxxzHR0BagABwA6gfi7E0Q5oBuxOtm0KMMb5fAzwjvN5fWecgUB1Z/xG577NQGsc03gvBLr7INYKQDPn8yLAQWdMeTVeBRR2PjcDm4BWeTVe53WeA34C/s7L/xac14kASqfalifjBb4DHnU+DwCK59VYU8VtBM4A1XI6Xp+9qbz0cP5yFid7/TLwci7FEkzKRHAAqOB8XgE44C5GHJP3tXaW2Z9s+0Dg8xyI+08cy47m+XiBgsBWHMue5sl4cUy5vhy4k6REkCdjdZ47grSJIM/FCxQF/sPZESYvx+om9i7AutyI11+qhjxaCS2XlNPOqbedP8s6t6cXcyXn89TbfUYpFQw0xfEtO8/G66xq2Q6cA5ZqrfNyvB8CowF7sm15NVZwLBi1RCkVrpQanofjrQFEAd84q92+VEoVyqOxpjYAmO18nqPx+ksi8GgltDwmvZhz9L0opQoDvwLPaK0vZ1TUzbYcjVdrbdNaN8HxbbulUqphBsVzLV6lVE/gnNY63NND3GzL6X8Lt2mtmwHdgSeVUu0yKJub8ZpwVL9O11o3Ba7hqFpJT1743aIca7b0Bn7JrKibbdmO118SQV5eCe2sUqoCgPPnOef29GKOdD5Pvd3rlFJmHEngR631b3k93kRa60vAKqBbHo33NqC3UioC+Bm4Uyn1Qx6NFQCt9Snnz3PA70DLPBpvJBDpvBsEmIcjMeTFWJPrDmzVWp91vs7ReP0lEXiyWlpumQ885Hz+EI66+MTtA5RSgUqp6kBtYLPzNvGKUqqVs1fAkGTHeI3z3F8B+7TW7+eDeMsopYo7nxcAOgH782K8WuuXtdaVtdbBOP4trtBaP5gXYwVQShVSShVJfI6jLnt3XoxXa30GOKGUquPc1BHYmxdjTWUgSdVCiXHlXLy+bPzISw8cK6EdxNHKPjaXYpgNnAYsODL4I0ApHI2Gh5w/SyYrP9YZ7wGS9QAAQnH8RzwCfEqqhjEvxdoWx63lTmC789EjD8cbAmxzxrsbeM25PU/Gm+xaHUhqLM6TseKod9/hfOxJ/P+Th+NtAoQ5/y38AZTIq7E6r1MQOA8US7YtR+OVKSaEEMLP+UvVkBBCiHRIIhBCCD8niUAIIfycJAIhhPBzkgiEEMLPSSIQfkspddX5M1gpNcjL534l1ev13jy/EN4kiUAIx0SAWUoESiljJkVSJAKtdZssxiREjpFEIARMBm53zgf/rHPyuqlKqS1KqZ1KqccBlFIdlGONhp+AXc5tfzgnYtuTOBmbUmoyUMB5vh+d2xLvPpTz3Ludc8f3T3buVSppHv0fszSfvBDZYMrtAITIA8YAL2itewI4P9BjtNYtlFKBwDql1BJn2ZZAQ631f87XD2utLzintdiilPpVaz1GKfWUdkyAl1pfHCNfGwOlncesdu5rCjTAMUfMOhxzEq319psVIjW5IxAirS7AEOeU1ptwDPev7dy3OVkSAHhaKbUD2IhjMrDaZKwtMFs7Zko9C/wLtEh27kittR3HlB7BXngvQmRK7giESEsBo7TWi1NsVKoDjmmNk7/uBLTWWscqpVYBQR6cOz3xyZ7bkP+fIofIHYEQcAXHcpyJFgNPOKfhRil1i3PWzdSKARedSaAujqUxE1kSj09lNdDf2Q5RBsfypZu98i6EuEHyjUMIxyyVVmcVz7fARziqZbY6G2yjgD5ujlsEjFBK7cQxE+TGZPtmAjuVUlu11g8k2/47jqUFd+CY3XW01vqMM5EIkStk9lEhhPBzUjUkhBB+ThKBEEL4OUkEQgjh5yQRCCGEn5NEIIQQfk4SgRBC+DlJBEII4ef+D4B7vBn6hA9BAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss_list = log_dict['train_loss_per_batch']\n",
    "\n",
    "plt.plot(loss_list, label='Minibatch loss')\n",
    "plt.plot(np.convolve(loss_list, \n",
    "                     np.ones(200,)/200, mode='valid'), \n",
    "         label='Running average')\n",
    "\n",
    "plt.ylabel('Cross Entropy')\n",
    "plt.xlabel('Iteration')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA2zklEQVR4nO3deXwU9fnA8c+T++LITSBAAoQrnCGgiCCKVlEEBC9aW6i39VevX63an1Zsa6vVqrVVKx6IF0hFBRUvAoiKiNz3kYQQQgI5IPed/f7+mCUETEJIspkcz/v12tfszs7sPDuQefZ7jhhjUEoppQDc7A5AKaVU66FJQSmlVDVNCkoppappUlBKKVVNk4JSSqlqmhSUUkpVc1lSEJHXRSRTRHbUWBckIl+JyH7nMrDGew+JSKKI7BWRS10Vl1JKqbq5sqTwBnDZaeseBBKMMTFAgvM1IjIYuB6Ide7zooi4uzA2pZRStXBZUjDGrAGOnbZ6GrDA+XwBML3G+kXGmDJjzAEgERjjqtiUUkrVzqOFjxdujMkAMMZkiEiYc30PYF2N7dKc6+oVEhJioqKimj1IpZRqzzZu3JhtjAmt7b2WTgp1kVrW1Tr/hojcCtwK0KtXLzZs2ODKuJRSqt0RkYN1vdfSvY+OikgEgHOZ6VyfBvSssV0kkF7bBxhj5hlj4o0x8aGhtSY6pZRSjdTSSWEZMNv5fDawtMb660XEW0SigRhgfQvHppRSHZ7Lqo9EZCEwEQgRkTTgUeAJYLGI3ASkAtcAGGN2ishiYBdQCdxpjKlyVWxKKaVq57KkYIyZVcdbk+rY/nHg8aYet6KigrS0NEpLS5v6UcrJx8eHyMhIPD097Q5FKeViraWhudmkpaXRqVMnoqKiEKmt/VqdDWMMOTk5pKWlER0dbXc4SikXa3fTXJSWlhIcHKwJoZmICMHBwVryUqqDaHdJAdCE0Mz0fCrVcbTLpKCUUm3ZuuQcFv94iNzi8hY/drtrU7BbTk4OkyZZbelHjhzB3d2dE+Mp1q9fj5eXV537btiwgTfffJPnn3++3mOcd955rF27tvmCVkq1CjmFZfzl0918uPkwAA9/5MbFg8OYGRfJBf1D8XB3/e94TQrNLDg4mC1btgAwd+5cAgIC+N3vflf9fmVlJR4etZ/2+Ph44uPjz3gMTQhKtS/GGP67MY2/Lt9NUVkld13Uj4sGhbN0y2GWbkln+fYjhAR4M31Ed2aOimRQRGeXxaJJoQXMmTOHoKAgNm/eTFxcHNdddx333HMPJSUl+Pr6Mn/+fAYMGMDq1at5+umn+eSTT5g7dy6pqakkJyeTmprKPffcw1133QVAQEAAhYWFrF69mrlz5xISEsKOHTsYNWoUb7/9NiLC8uXLue+++wgJCSEuLo7k5GQ++eQTm8+EUup0SVmF/N+H21mXfIzRUYH89aqhxIR3AmBEz648NHkQq/dmsmRTGgu+T+HVbw8Q270zs8dGce3onmf49LPXrpPCYx/vZFd6frN+5uDunXn0ytiz3m/fvn2sWLECd3d38vPzWbNmDR4eHqxYsYI//OEPLFmy5Cf77Nmzh1WrVlFQUMCAAQO44447fjJWYPPmzezcuZPu3bszbtw4vvvuO+Lj47nttttYs2YN0dHRzJpV15ARpZRdyiqr+M/qZF5YlYiPpxt/mzGU6+J74uZ2ascOLw83fhbbjZ/FduNYUTnLthxmyabD7EzP49TZgZpHu04Krck111yDu7t1i4i8vDxmz57N/v37EREqKipq3eeKK67A29sbb29vwsLCOHr0KJGRkadsM2bMmOp1I0aMICUlhYCAAPr06VM9rmDWrFnMmzfPhd9OqbbDGENReRV5JRX4e7nT1a/udr7aHCsq5/MdR1i5JxM/L3eigv3oFexPVLAfvYP9CQnw+kmPvdKKKtJzSzicW8Lh4yWk55awfMcREjMLuXJ4dx6ZMoiwTj5nPHaQvxdzxkUzZ1w0FVWOs4q7odp1UmjML3pX8ff3r37+yCOPcOGFF/Lhhx+SkpLCxIkTa93H29u7+rm7uzuVlZUN2saYWieYVapDcDgMuzLyWZuUzc70fHKLK8grqSC/xFrmlVRQ6bD+RkRgULfOnNc3mHH9QhgdHUSA908vi8eLyvly1xE+2ZbB2qQcqhyGXkF+AHyyLR1HjT85fy93egf7062LDzmFZRzOLSG78NReRG4C/cICmP/r0Vw4IIzG8HRRo3O7TgqtVV5eHj16WLeLeOONN5r98wcOHEhycjIpKSlERUXx3nvvNfsxlGotjDEkZRWxNimbtYk5rDuQQ26xVfru0dWX4AAvuvh60iPQly6+nqc8sgrK+D4phzfXHeTVbw/g7iYMj+zCeX1DGNs3mMO5JXy6LYPvErOpdBh6B/tx+wV9uGJodwZFdEJEKK90cDi3hJScIg5mF5GSU8zBnCIy8koJCfBiUERnenT1pUegL927+tKjqy/duvi47KLeVJoUbPD73/+e2bNn88wzz3DRRRc1++f7+vry4osvctlllxESEsKYMXoTO9U+OByGw7kl7M8sYP/RQnZl5PN9Ug6ZBWWAlQQuGRTOuH7WRT2885mrZO6aFENpRRUbDx63EktSDi99ncS/VyUC0DPIl5vH92HKsAhiu3f+SdWQl4cb0SH+RIf4w4Dm/84tTdpyVUN8fLw5/SY7u3fvZtCgQTZF1HoUFhYSEBCAMYY777yTmJgY7r333kZ/np5X1dKKyyv54cAx9mQUVCeBxMxCSipOTqAc3tmbMdHBVvVP3xB6Bvk2ywj8gtIKNhw8TrC/F0N7dGl3o/pFZKMxptb+71pSaKdeeeUVFixYQHl5OSNHjuS2226zOySlziglu4hVezNZtTeLdck5lFdajandOvsQEx7ArDG9iAkPoH94AP1CO9HFzzUz93by8Wx0XX9bp0mhnbr33nubVDJQqiUUl1ey8eBxVu7JZPXeLA5kFwHQJ9SfX57bm4kDQhkW2ZUuvjpte0vRpKCUconySgefbk9nc2ruT3r/5JVUkl9aUV0S8PJwY2yfYOacF8XEAaH0DvY/w6crV9GkoJRqVrnF5bzzQyoL1qaQWVBGJx8PAv2sHkCdfT3o1sXH+dyTzj6eDOzWifP6huDr5W536ApNCkqpZnIwp4jXvz3A4g1plFRUMT4mhKeuGc6EmJB211DbnmlSUErV6Zv9Wbz27QH8vNwJDfAmJMCbkE7e1vNO3oQEeJGRV8pr3xzgi11H8HATpo3owU3nR7t00jblOpoUmtnEiRN56KGHuPTSS6vXPffcc+zbt48XX3yx1u2ffvpp4uPjufzyy3n33Xfp2rXrKdvUNtvq6T766CP69+/P4MGDAfjjH//IhAkTuPjii5vni6kOpaS8iic+282C7w8S0cUHXy93vi3IJr/0p6PqAbr4evKbiX2ZPTaKsAaMDVCtlyaFZjZr1iwWLVp0SlJYtGgRTz311Bn3Xb58eaOP+9FHHzFlypTqpPCnP/2p0Z+lOrath3K5d/EWkrOKmHNeFA9OHoiPp1XfX1ZZRXZhOdkFZWQXlpFVUIaHuxuXD+2Gn5deTtqD1jnOug27+uqr+eSTTygrs0ZYpqSkkJ6ezrvvvkt8fDyxsbE8+uijte4bFRVFdnY2AI8//jgDBgzg4osvZu/evdXbvPLKK4wePZrhw4czc+ZMiouLWbt2LcuWLeP+++9nxIgRJCUlMWfOHN5//30AEhISGDlyJEOHDuXGG2+sji0qKopHH32UuLg4hg4dyp49e1x5alQrV1Hl4Nmv9jHjpbWUlFfx9k3nMHdqbHVCAPD2cKdHV1+G9+zKpEHhXD+mF1ePitSE0I7Y8i8pIncDtwACvGKMeU5EgoD3gCggBbjWGHO8SQf67EE4sr1pwZ6u21CY/ESdbwcHBzNmzBg+//xzpk2bxqJFi7juuut46KGHCAoKoqqqikmTJrFt2zaGDRtW62ds3LiRRYsWsXnzZiorK4mLi2PUqFEAzJgxg1tuuQWAhx9+mNdee43f/va3TJ06lSlTpnD11Vef8lmlpaXMmTOHhIQE+vfvz69+9Steeukl7rnnHgBCQkLYtGkTL774Ik8//TSvvvpqM5wk1dYkZRVy33tb2JqWx1UjezB3aqyODeigWrykICJDsBLCGGA4MEVEYoAHgQRjTAyQ4HzdJp2oQgKr6mjWrFksXryYuLg4Ro4cyc6dO9m1a1ed+3/zzTdcddVV+Pn50blzZ6ZOnVr93o4dOxg/fjxDhw7lnXfeYefOnfXGsnfvXqKjo+nfvz8As2fPZs2aNdXvz5gxA4BRo0aRkpLS2K+s2iiHw7BgbQpXPP8NB48V8+Iv4nj2uhGaEDowO0oKg4B1xphiABH5GrgKmAZMdG6zAFgNPNCkI9Xzi96Vpk+fzn333cemTZsoKSkhMDCQp59+mh9//JHAwEDmzJlDaWlpvZ9RVxe+OXPm8NFHHzF8+HDeeOMNVq9eXe/nnGluqxNTb9c1Nbdq3YwxvPfjIfJKKvjV2Kiz6uufkl3EA0u28cOBY0wcEMrfZw7TRmJlS5vCDmCCiASLiB9wOdbtg8KNMRkAzmWbnXgkICCAiRMncuONNzJr1izy8/Px9/enS5cuHD16lM8++6ze/SdMmMCHH35ISUkJBQUFfPzxx9XvFRQUEBERQUVFBe+88071+k6dOlFQUPCTzxo4cCApKSkkJlozPr711ltccMEFzfRNlZ1yCsu4acEGHvxgO3/7bA8X/WM1H25Ow+Go/4dAlcPw6jfJXPbPNezKyOfJmUOZP2e0JgQF2FBSMMbsFpEnga+AQmAr0OCfqCJyK3ArQK9evVwSY3OYNWsWM2bMYNGiRQwcOJCRI0cSGxtLnz59GDduXL37nriP84gRI+jduzfjx4+vfu/Pf/4z55xzDr1792bo0KHVieD666/nlltu4fnnn69uYAbw8fFh/vz5XHPNNVRWVjJ69Ghuv/1213xp1WK+2Z/FfYu3kldcwaNXDmZQRGce/3Q39763lfnfpfDwFYMZEx30k/32Hy3g/ve3seVQLpMGhvH4VUPp1kWTgTrJ9qmzReSvQBpwNzDRGJMhIhHAamNMvbOT69TZLUfPa+tQXung6S/3Mm9NMv3CAvjXrJHVg8QcDsNHWw7z98/3ciS/lMlDuvHg5IH0DvanosrBy18n8XxCIv7e7sydGsvU4d11pHEH1eqmzhaRMGNMpoj0AmYAY4FoYDbwhHO51I7YlGqtkrIKuXvRZnYczucX5/Ti4SsGn9KG4OYmzIiLZPKQCF79JpmXvk5ixe6j/OKc3qw/cIxdGflcMSyCx6bGEhLgXc+RVEdmV+fiJSISDFQAdxpjjovIE8BiEbkJSAWusSk2pVoVYwyLNxxi7rJdeHu68fIvR3FpbLc6t/f1cue3k2K4bnRP/vHlPhZ8n0Kwvzf/uSGOy4ZEtGDkqi2yJSkYY8bXsi4HmNRMn6/F4mZkdxVjR1TlMGxOPc6K3Zkk7D7K/sxCxvYJ5tnrRjS4DSCssw9PXj2M31zYl67OWUqVOpN2NwzRx8eHnJwcgoODNTE0A2MMOTk5+PhoY6SrFZRW8M3+bFbsPsrqvVkcKyrHw00YEx3Er8dFc93onri7nf3/ab03gTob7S4pREZGkpaWRlZWlt2htBs+Pj5ERkbaHUa7VOUwfLXrCO/8kMq65BwqqgxdfD25cEAokwaFM6F/qP7CVy2q3SUFT09PoqOj7Q5DqXqVVlSxZFMar6xJJiWnmMhAX349LppJA8MY1TsQD3edlkzZo90lBaVas9zict76/iBvrE0hp6icYZFdeOHncVw2pFujqoaUam6aFJRqAYeOFfPatwdYvOEQxeVVXDgglFsn9OXcPkHa9qVaFU0KSrlQZkEpzyfsZ9H6QwBMHdGdWyf0YWA3vSuZap00KSjlAgWlFcxbk8yr3xygosrBrDG9uGNiX7p39bU7NKXqpUlBqWZUVlnF2+tSeWFVIseKypkyLILf/WwAUSHaLVS1DZoUlGoGVQ7D0i2H+ceX+zicW8L5/UJ44LKBDI3sYndoSp0VTQpKNVFZZRU3vPoDP6YcZ0iPzjwxcyjjY0LtDkupRtGkoFQTPZ+wnx9TjvPXq4Zy/eieuGnXUtWGaVJQqgk2px7npdVJXBsfyc/Pab3391CqoXTYpFKNVFJexf8u3kpEF18emTLY7nCUahZaUlCqkZ76Yi/J2UW8c/M5dPLR+YlU+6AlBaUa4fukHF7/7gCzx/ZmXL8Qu8NRqtloUlDqLBWWVXL/+1uJCvbjgckD7Q5HqWalSUF1eMYYvth5hH8l7Ce7sOyM2z/+6W4O55bw9DXD8fPSGljVvuj/aNWhpeYU8+iyHazaa91/4z9fJ3HrhL7cPD4af++f/nms3pvJwvWp3DahD/FRQS0drlIup0lBdUhllVW8/HUyL6xKxMNNeGTKYCbEhPDMV/t4dsU+3lp3kLsvjuH60T3xdN7bIK+4ggeWbCMmLIB7L+lv8zdQyjU0KagO59v92TyydAcHsou4YmgEj0wZXH3f45duGMWm1OM8sXwPj3y0g9e/PcD9lw5g8pBuzP14J9mF5bz6q9H4eLrb/C2Ucg1NCqrDyMwv5c+f7ubjren0DvZjwY1juKD/T6ejiOsVyHu3ncvKPZk8+fkefvPOJvqFBZCYWcjdk2J0PiPVrmlSUO1eaUUVb6xN4YWViZRVObjn4hhuv6Bvvb/2RYRJg8KZOCCMJZvSeParfQzv2ZX/uahfC0auVMvTpKDaLYfDsGxrOk99sZfDuSVcOCCUP14ZS/RZTGPt7iZcG9+TmXGROIypbl9Qqr2yJSmIyL3AzYABtgO/BvyA94AoIAW41hhz3I74VNv3fVIOf12+m+2H84jt3pmnrh7GeU0YZObuJrijE92p9q/Fk4KI9ADuAgYbY0pEZDFwPTAYSDDGPCEiDwIPAg+0dHyqbUvMLOBvy/eQsCeT7l18eOba4Uwf0UNnLlWqgeyqPvIAfEWkAquEkA48BEx0vr8AWI0mBdVABaUVPPn5HhauP4Sfpzu/v2wAN46L1l5CSp2lFk8KxpjDIvI0kAqUAF8aY74UkXBjTIZzmwwRCattfxG5FbgVoFcvnapYwba0XH67cDOHjhXzy3N7c9ekGIIDvO0OS6k2yY7qo0BgGhAN5AL/FZEbGrq/MWYeMA8gPj7euCJG1TY4HIbXvzvAk5/vISTAm/duG8toHWWsVJPYUX10MXDAGJMFICIfAOcBR0UkwllKiAAybYhNtRE5hWX87r9bWbU3i0sGh/PU1cPo6udld1hKtXl2JIVU4FwR8cOqPpoEbACKgNnAE87lUhtiU23A90k53PPeZo4XVfDY1Fh+NbY3ItqQrFRzsKNN4QcReR/YBFQCm7GqgwKAxSJyE1biuKalY1OtW2WVg+dXJvKvlfuJDvbn9Tmjie2uo4uVak629D4yxjwKPHra6jKsUoNSP3HoWDH/u3gr61OOMTMukj9Ni611FlOlVNPoX5Vq1YwxvL8xjcc+3gXAM9cOZ0ZcpM1RKdV+aVJQrVZOYRkPfbCdL3cdZUx0EP+4Zjg9g/zsDkupdk2TgmqVEnYf5YEl28gvqeQPlw/kpvP74K6jkpVyOU0KqsWUlFexLS2X0E7edO/qW+to46KySv7y6S4Wrj/EwG6dePvmcxjYrbMN0SrVMWlSUC2irLKKn7+6js2pudXrgvy96N7Vh4guvvTo6ktoJ2/e+/EQh44Xc/sFfbn3khi8PXSaCqVakiYF1SIe+3gXm1NzeWTKYAL9PMnIK+VwbgkZuSWk5hSzLjmHgtJKIgN9ee/WsYyJ1pHJStlBk4JyuUXrU3n3h1Ruv6AvN50fXed2BaUV+Hq646H3LFDKNpoUlEttTj3OH5fuZHxMCPdfOqDebTv5eLZQVEqpuuhPMuUyWQVl3PH2JsI6e/P89SO195BSbYCWFJRLVFQ5uPPdTeSWlLPkjvMI9NfJ6pRqCzQpKJf46/LdrD9wjOeuG6HzEynVhmj1kWp2H25OY/53Kdw4LprpI3vYHY5S6ixoUlDNasfhPB5csp1zooN46PKBdoejVN1KjkNhJhi9V1dNWn2kms2xonJuf3sjQf5evPCLODy1a2nbVV4M+7+AHUvg4Fro0hPCBkHowJPLLj3BrY5/44pS66Jbchx8ukCXRpYYC46CbyB4NLJNqrIcjqdAzn7I3u9cJlrL4hxrG+/OENwPQmIgOAZC+lnLoD7g1Qrm2jIGKkuhogQqiq1/m4pi8O4EwX2b/XCaFFSzKCmv4qYFP5JZUMbi28YSovdIbjhHFZTlWxcnt7MYwe2ogoIjkH8YHJXgHwb+IdZFuDE3Haosh6QEKxHsWQ4VRRAQDjE/s46TtAq2Ljy5vac/hA6Azt2hNA9KcqHkmJUIKopP/eyQAdDvYug3CXqfB56+tcdQmgcH1ljHSloJxw+AmyeEx0L3ERAxwlqGxZ6aKBwOyD0Imbvg6C7I3GktcxLBVJ3czj/MuvgPnGIlAncva5uc/ZDyHWx779R4uvaC0EEQNvDkMmRA8yaL0jxnokqskbwSrXN+IhFQS2kmdgZcM7/54nAS04aLTvHx8WbDhg12h9HhVVY5uO2tjazam8mLvxjFZUO62R1S61JRAoc3Qm4q5Kdbf+wFGSefFx61LlziBn7BJy/uAWHgH2o9PLyt7fMPQ95ha1lw5NQL3gnuXif3O/Hw6WJdiD39rAvaieeefoCB/V/B7mXWBco3EAZPgyEzofe4UxNVyXHI3ANZu08uCzPBpyv4BVn7nniceJ2fDokJkPItVJWBhw9EnQ99J1lJoiTXSgDJqyBtg/WdvAKsbaLOh6IsSN8CGVus+E58x7DB1oX9eApk7raS2AmBUVbiCBsIIf2tX/7BfcG3a/3/VuVFkJN0skSRtcd6ZO8HR4VzI4HA3laSCImpUcKIsf79akvIxkBeGmTvhax91jLbmQCKatx5WNytzw6OsUpXJ/6NPH3By7/Gv5svdImEiOH1f586iMhGY0x8re9pUlBNYYzhgSXbWLwhjb9MH8IN5/a2OyT7Oaqsi1jyKkheDYfWWxfDE3y6Wr+uO3WDTs6lX5B1wS3KgsIsa1mUCUXZUF5o7efhA517WBeLzj1Ofe7mYW1bc7/CTOt5YZb1GeVFtScRAK9OMPAKKxH0vRDcXTCQsLzYqopKXGE9cvbXeFOgRxz0uRD6XgSRo39aZWSMlQDSN1sJIn0LHEu2EkB4rJUkwmOtqi3vgOaNvarCOlbmbitJnFgeS4aq8pPb+XR1Jol+1r9L3iHIciaAmknLN/BksjpRXRUSA4HRja8qOwuaFJTLPPXFHl5YlcRdk2K475L+dodjn+Mp1q/t5NVw4Bsoc/6iDR8KfS6A6AusP/6Abmdf9VBebNUp+wY2rlqopqoKKzmcqJaoKLESVtjguqt0XOX4Qet8+XS2zo9fG5zvqqoS8lJPtlOcqPrJ3g+FR6zEENLfqmarXg6wSoI23le8vqSgbQqq0RasTeGFVUnMGtOTey+OsTucupXmw5HtVvVNcD8IH2wVxZtDYRas/htsfMP6Fd61F8ROP5kI/EOafgwvv+arw3b3tKpQzlSN0hICe8Oo2XZH0TTuHlaDdFAf4GenvldVab3fxpwxYhGZAiw3xjhaIB7VRny6LYO5H+/k4kHh/HnaEMTGXz2nKDgKR7ZBxlbncpvVWHkKsf6Iuw2xfsl3GwLdhlq/6hr6PSpKYd2L8M0z1i/u+Bth7G+cFwelaJMJARpWUrge+KeILAHmG2N2uzgm1cqtTcrm3ve2MKpXIP/++Uj7ZzXNOwwb58OWhZCfdnJ9YJTVEDfyBmvZtZdVtD+y42Ti2LX05Pb+odav+74XWfXqnbv/9FjGWL1zVjxmVRv0nwyX/AlCO3DVmWpXzpgUjDE3iEhnYBYwX0QMMB9YaIwpcHWAqnXZlZ7PbW9upHewH6/Ojq/17mktwhg48DWsfwX2fgbGYXV5HPsbKwF0G2r1uDld6ACrQfWE0nyrG+OR7VaDcPJq2PG+c9uBzgRxkdWN8sh2+OIPVk+ibkNh2jKrmkipdqTBDc0iEgLcANwD7Ab6Ac8bY/51VgcUGQDU7AzcB/gj8KZzfRSQAlxrjDle32dpQ7PrVFY5yMgrJfVYMQdzikk9VsyhY8WsTcrGx9OdJXecR/euLdwwCVaXxC0L4cdXrYY93yCI+6VVfRMY1fTPNwaO7rS6SCathNTvrUZeN0+rS2JAN5j0CAyfdXZjCpRqRZrU+0hErgRuBPoCbwELjDGZIuIH7DbGNLoPooi4A4eBc4A7gWPGmCdE5EEg0BjzQH37a1JoXpVVDh5ZuoO1STkcPl5CpePk/w1PdyEy0I/oEH8emjyQmPBOjTuIMWff68LhgEPrrIFT29+36vB7xMPomyH2KvD0aVwsDVFRYiWG5NVWd8Nzbmu+RmqlbNLU3kfXAM8aY9bUXGmMKRaRG5sY2yQgyRhzUESmAROd6xcAq4F6k4JqXvO+SWbh+kNcPCicK4ZG0DvYj55BfvQK8iOii2/T7odw7AB8eh8c+tEakNTPOXCpvobZrL3WCNNt/7Xq7z39rFGcY26G7iMbH8vZ8PQ9WYWkVAfQkKTwKJBx4oWI+ALhxpgUY0xCE49/PXBi3Hy4MSYDwBiTISJhTfxsdRb2HMnnua/2M3lIN178RVzz9SZyVMEPL8PKP1sjdgddaf3y3veZ9X5g9MnpD6LGW4OsdiyxkkHGVmufPhdaVTYDLm/+QUlKqVM0pPpoA3CeMabc+doL+M4YM7pJB7Y+Jx2INcYcFZFcY0zXGu8fN8YE1rLfrcCtAL169Rp18ODBpoShsG6IM/2F7ziSV8qX904guLnmLcrcA8v+B9J+tObPmfKsNTQfrKkEEhOska0p31hVQm6eVl9/47DmuBl2nTXCtlN488SjlAKaXn3kcSIhABhjyp0X9KaaDGwyxhx1vj4qIhHOUkIEkFnbTsaYecA8sNoUmiGODu/fKxPZmZ7Pf24Y1TwJobIcvn0W1jxlzeQ441UYevWpbQnBfa3HObdCZZlVekhaCe7e1rah9d/PWSnlGg1JClkiMtUYswzAWfef3QzHnsXJqiOAZcBs4AnncmltO6nmteNwHi+sSmT6iO7NM5Hd4Y2w9H+sbp5DrobJT555VK+HN/SZaD2UUrZqSFK4HXhHRP4NCHAI+FVTDursuXQJcFuN1U8Ai0XkJiAVq4FbuVBZZRX3Ld5CkL8Xj00d0vQPXP8KfPZ7q9vmrEUwYHLTP1Mp1aIaMngtCThXRAKw2iCaPGDNGFMMBJ+2LgerN5JqIc+t2M++o4XMnzOaLn5NnBVz7b/gy4etEb4zXq594JhSqtVr0OQcInIFEAv4nOiVYoz5kwvjUi62KfU4L3+dxHXxPblwYBM7eq15Clb+BQZPh5mvumbaZaVUizjjpDUi8h/gOuC3WNVH1wA6aX4bVlJexe8WbyWiiy8PTxl08g1HFXz7HOxa1rD71hpjJYOVf7F6Cs18TROCUm1cQ0oK5xljhonINmPMYyLyD+ADVwemGic1p5gfU47RO9iPvqEBBPr/tKPYU1/sJTm7iHduPodOPs6LuMMBH98Nm9+yXvcYBRfPhegJtR/IGPjqEavaKO5XMOU5nfZBqXagIUmh1LksFpHuQA4Q7bqQVGN9vuMIv/vvVgrLKqvXBfp50jc0gD6h/vQNDcDH0535aw/wy3N7M66fs1eQMfDZ/VZCGP+/1oCy1X+DBVdat0y8eC5EDDt5IIcDPn8A1s+D0bfA5L/XfQN3pVSb0pCk8LGIdAWeAjZh3UH6FVcGpc5OlcPwzFd7eWFVEsMju/DYtCEcKyojOauIpKxCkrKKWLknk8UbrGmlewX58eDkgdbOxsAX/2dNMHfeb+GiR6zxBEOvtnoTffMPeHk8DL0GLvw/6NobPrkbNr0JY/8HfvYXW+8gpZRqXvWOaBYRN+BcY8xa52tvwMcYk9dC8dVLJ8SD3OJy7lq0hTX7srguviePTYutczrrvOIKkrML6dHVl7DOPlZCSPgTfPsMjLnNGlNw+gW+JBe+ew7W/QccldaU0embYPzv4KKHNSEo1QY1dZbU740xY10SWRN19KSwKz2f297ewJG8Uh6bOoSfn9Pr7D7g67/Dqsdh1ByrTaC+C3x+Bnz9BGx+By54AC64vymhK6Vs1NRpLr4UkZnAB6ahN19QLvfR5sM8+ME2uvp68d5tY4nr9ZNpour37XNWQhj+c7ji2TP/4u8cAVf+Ey5/WnsYKdWONSQp3Af4A5UiUorVLdUYYzq7NDJVq8oqB48v383871IYEx3ECz+PI7TTWc5XtO4/sOJRa7K5af8+u0ZiTQhKtWsNGdHcyLupKFd4a91B5n+Xwq/HRfGHywfhWfP+yKX5VgNw2noQd3DzcD5qPC8vgq3vwsApcNXL2o1UKXWKMyYFEam1o/rpN91RrmeM4Z0fUhnRsyuPXhl78o38dPjhP7BhPpTlQ1Bf6z4EjkprQJqj8tTHkJkw/T/6q18p9RMNqT6q2aLoA4wBNgJ6K6oWtvHgcRIzC/n7TOeYgaO7rMFj2/9r3Ydg8HSrW2mPOFvjVEq1XQ2pPrqy5msR6Qn83WURqTotXH+IAG8PpnZJhLfvsW5Q4+ln3bR+7G+a58b1SqkOrUET4p0mDWiGeZbV2cgrqeDT7en8JWobPu/+FfxDrXEC8TeBX5Dd4Sml2omGtCn8C2sUM1gT6I0AtrowJlWLpVsOU1FRwZW5b1s3rf/15+DpY3dYSql2piElhZqjwyqBhcaY71wUj6qFMYaF6w9xW8g2vAtS4fK/aUJQSrlEQ5LC+0CpMaYKQETcRcTPeaMc1QK2peWxOyOPd0M+gtCBMOByu0NSSrVTDRm1lAD41njtC6xwTTiqNgvXpzLZcyuBhfth3D06I6lSymUaUlLwMcYUnnhhjCl03mNZtYDCskqWbT3M8oDl4NXLmr1UKaVcpCE/OYtEpLrju4iMAkpcF5Kq6eOt6Qyp2ElUyQ4Yd5cOOFNKuVRDSgr3AP8VkXTn6wis23OqFrBwfSqP+H+K8Q5FRt5gdzhKqXauIYPXfhSRgcAArMnw9hhjKlwemWJneh5Vh7cw2nsTXPBH8PQ9805KKdUEZ6w+EpE7AX9jzA5jzHYgQER+05SDikhXEXlfRPaIyG4RGSsiQSLylYjsdy7Pci7o9mfR+kPc6fkxxqsTjL7Z7nCUUh1AQ9oUbjHG5J54YYw5DtzSxOP+E/jcGDMQGA7sBh4EEowxMVg9nh5s4jHatJLyKjZv/pHL3H5AxtwCPl3sDkkp1QE0JCm4iZy8A4uIuANejT2giHQGJgCvARhjyp1JZxqwwLnZAmB6Y4/RHnyyLZ0bqj4Cdy84t0kFM6WUarCGJIUvgMUiMklELgIWAp814Zh9gCxgvohsFpFXRcQfCDfGZAA4l2FNOEab9+W6Tcz0+BaJ+xUEhNodjlKqg2hIUngAqzrnDuBOYBunDmY7Wx5AHPCSMWYkUMRZVBWJyK0iskFENmRlZTUhjNZr39ECzj2yEHcxyHm/tTscpVQHcsakYIxxAOuAZCAemITVBtBYaUCaMeYH5+v3sZLEURGJAHAuM+uIZ54xJt4YEx8a2j5/QS9du51Z7iupGDQDAnvbHY5SqgOps0uqiPQHrgdmATnAewDGmAubckBjzBEROSQiA4wxe7GSzC7nYzbwhHO5tCnHaUuMMWQWlJGSXcTBY8V03voaflIGE39nd2hKqQ6mvnEKe4BvgCuNMYkAInJvMx33t8A7IuKFVQL5NVapZbGI3ASkAtc007FanZV7jrI2MYeDx4o5mFNE5rHjRFSm00cy6CPp3OzxGblRl9I1bKDdoSqlOpj6ksJMrJLCKhH5HFiENXityYwxW7Cqok43qTk+vzXblHqcRW++xASPXUz2OkqUSSfYPRPcT25jgvoil8+1LUalVMdVZ1IwxnwIfOjsGTQduBcIF5GXgA+NMV+2TIjthzGGpR+8wzyvZzFeAUhIDARPhJAYCO5nLYP6IF7+doeqlOqgGjLNRRHwDlZ1TxBWtc6DgCaFs/TpxiRuPPZPCjr1ptM9P+i0FUqpVues7tFsjDkGvOx8qLNQUl5F7md/prdbJo6ZH2tCUEq1Snq3lhby0WfLub5yGZn9rsWtzwS7w1FKqVppUmgBR3MLGbbpEYo8uhA28+92h6OUUnXSpNAC1i98nFg5QPklT4Jvh5/8VSnVimlScLE9u7Zx8ZFX2d/1fELPudbucJRSql6aFFzIOByUfXQ3VeJOxM9fBGmWYR5KKeUymhRcaOvyeQwv38TuwfcSEKZzGCmlWj9NCi5SmnuUqA1/YZf7QOJm6hxGSqm2QZOCi6S+ezd+ppjSyc/i7u5+5h2UUqoV0KTgArnbPqN/5md8Hvhz4uLPszscpZRqME0Kza28CMfH95JoujPk+sfsjkYppc6KJoVmlv3pnwmqyODbAQ/Tp1uw3eEopdRZ0aTQjMyR7QRufZkPuIirpuuYBKVU23NWE+Kpejgc5C++kwoTQMmFj9LFz9PuiJRS6qxpSaGZVP34Gl2ObWWe701cO36Y3eEopVSjaEmhORQcoeqruXxfFcs5U2/H011zrVKqbdKrVzMo/+T3mMoyPuj+v1w0KNzucJRSqtE0KTTVvi/x2ruUf1dO55bplyA6v5FSqg3T6qOmKC+i4uN7OWh6cHzE7QyK6Gx3REop1SRaUmiKr5/EsyCNP5mbuevSIXZHo5RSTWZLSUFEUoACoAqoNMbEi0gQ8B4QBaQA1xpjjtsRX4Mc2YFZ+2/eq5zIOZOmEtbJx+6IlFKqyewsKVxojBlhjIl3vn4QSDDGxAAJztetk8OB+fhu8gjgDf8buen8aLsjUkqpZtGaqo+mAQuczxcA0+0L5Qw2LUAOb+DRsl/wm8tH4+Ops6AqpdoHu5KCAb4UkY0icqtzXbgxJgPAuQyzKbb6GYPj+xfYLjEc6nEFVw6LsDsipZRqNnb1PhpnjEkXkTDgKxHZ09AdnUnkVoBevXq5Kr66HVqPW85+3qy4lYevjNUuqEqpdsWWkoIxJt25zAQ+BMYAR0UkAsC5zKxj33nGmHhjTHxoaGhLhXzy+JvepBgfcqOnENcrsMWPr5RSrtTiSUFE/EWk04nnwM+AHcAyYLZzs9nA0paO7YzKCnDs+ICllWOZEt/P7miUUqrZ2VF9FA586Kx28QDeNcZ8LiI/AotF5CYgFbjGhtjqt+MD3CuLWeZ2Ea8P7mZ3NEop1exaPCkYY5KB4bWszwEmtXQ8Z8Ox6U2STSQRsePx9dIeR0qp9qc1dUlt3TL34HZ4AwsrL2BGXE+7o1FKKZfQpNBQm9+iEg++9ZvE2L56m02lVPukSaEhKstxbFnICkccE0cOxt1Nu6EqpdonTQoNse8z3EpyWFQ5kavietgdjVJKuYwmhYbY9BbZbiFkhY1jYDedHlsp1X5pUjiTvMOYpATeLT+f6XE2jKBWSqkWpEnhTLa8ixgHS6ouYOqI7nZHo5RSLqV3XquPw4HZ/BYb3YbSq18s4Z31nglKqfZNSwr1SfkGyT3Im6UTuGqkNjArpdo/TQr12fQmJe4BrHE/l0tjdVoLpVT7p0mhLiXHMbs/5qOqcVw4pBf+3lrTppRq/zQp1GXbf5GqMt4qu0CrjpRSHYb+/K3L5jc56BVDtucAxvULsTsapZRqEVpSqE36FjiynddLzmfaiO46rYVSqsPQpFCbXUtxiAcfVoxlulYdKaU6EE0KtUlKYLfHACLCIxgcodNaKKU6Dk0KpyvMgoytfFocy1VxPXDeIU4ppToETQqnS1oJwBrHMKbptBZKqQ5Gk8LpkhLIc+uCI3wYEV187Y5GKaValCaFmhwOTGICqyuHMr5/mN3RKKVUi9OkUNORbUhxtpUUYkLtjkYppVqcJoWaElcA8IPbcOKjAm0ORimlWp5tSUFE3EVks4h84nwdJCJfich+57Llr8pJK9nv1od+ffri4+ne4odXSim72VlSuBvYXeP1g0CCMSYGSHC+bjml+ZhDP/Bl+RAmxOi0FkqpjsmWpCAikcAVwKs1Vk8DFjifLwCmt2hQB9YgjkrWVA3nfE0KSqkOyq6SwnPA7wFHjXXhxpgMAOeyZbv/JCVQ4uZHqn8sA8I7teihlVKqtWjxpCAiU4BMY8zGRu5/q4hsEJENWVlZzROUMZjEFaxzxDK2f4SOYlZKdVh2lBTGAVNFJAVYBFwkIm8DR0UkAsC5zKxtZ2PMPGNMvDEmPjS0mbqN5iQhuakkVAxhgnZFVUp1YC2eFIwxDxljIo0xUcD1wEpjzA3AMmC2c7PZwNIWC8rZFfVrxzC9d4JSqkNrTeMUngAuEZH9wCXO1y0jKYEM9x506hZDaCfvFjusUkq1Nrbeec0YsxpY7XyeA0xq8SAqSjEp3/JV+XjG99dSglKqY2tNJQV7pH6PVBSzqmqYticopTo8TQpJCVSKJ1vcYxnVW6e2UEp1bJoUEhPY6jaIYdE9dGoLpVSH17GTQn46ZO7ii9JYxusoZqWU6uBJITEBgK8dw5nQX9sTlFKqYyeFpARy3UPIDehHTFiA3dEopZTtOm5ScFRhklaxusq6y5pObaGUUh05KRzehJTmsqJ8iLYnKKWUU8dNCkkJOHDjW8cQztepLZRSCrB5RLOtEleQ6NmfyOAeBAfo1BZKKQUdtaRQfAxzeCNflA5mvI5iVkqpah0zKRxLpsKrK6sqh2l7glJK1dAxk0JkPI8PWsoejwE6tYVSStXQMZMCsCbxGOf0CcHbQ6e2UEqpEzpkUjh0rJgD2UXanqCUUqfpkEmhtKKKS2PDdWoLpZQ6TYfskhoT3omXfxlvdxhKKdXqdMiSglJKqdppUlBKKVVNk4JSSqlqmhSUUkpV06SglFKqmiYFpZRS1TQpKKWUqqZJQSmlVDUxxtgdQ6OJSBZwsJ5NQoDsFgrnbGlsjaOxNY7G1jjtNbbexphap3Ro00nhTERkgzGmVQ5d1tgaR2NrHI2tcTpibFp9pJRSqpomBaWUUtXae1KYZ3cA9dDYGkdjaxyNrXE6XGztuk1BKaXU2WnvJQWllFJnoV0mBRG5TET2ikiiiDxodzw1iUiKiGwXkS0isqEVxPO6iGSKyI4a64JE5CsR2e9c2nIj6zpimysih53nb4uIXG5DXD1FZJWI7BaRnSJyt3O97eetnthaw3nzEZH1IrLVGdtjzvWt4bzVFZvt561GjO4isllEPnG+dsl5a3fVRyLiDuwDLgHSgB+BWcaYXbYG5iQiKUC8MaZV9H0WkQlAIfCmMWaIc93fgWPGmCecSTXQGPNAK4ltLlBojHm6peOpEVcEEGGM2SQinYCNwHRgDjaft3piuxb7z5sA/saYQhHxBL4F7gZmYP95qyu2y7D5vJ0gIvcB8UBnY8wUV/2dtseSwhgg0RiTbIwpBxYB02yOqdUyxqwBjp22ehqwwPl8AdZFpcXVEZvtjDEZxphNzucFwG6gB63gvNUTm+2MpdD50tP5MLSO81ZXbK2CiEQCVwCv1ljtkvPWHpNCD+BQjddptJI/CicDfCkiG0XkVruDqUO4MSYDrIsMEGZzPKf7HxHZ5qxesqVq6wQRiQJGAj/Qys7babFBKzhvziqQLUAm8JUxptWctzpig1Zw3oDngN8DjhrrXHLe2mNSkFrWtZqMD4wzxsQBk4E7nVUkquFeAvoCI4AM4B92BSIiAcAS4B5jTL5dcdSmlthaxXkzxlQZY0YAkcAYERliRxy1qSM228+biEwBMo0xG1vieO0xKaQBPWu8jgTSbYrlJ4wx6c5lJvAhVnVXa3PUWTd9oo460+Z4qhljjjr/eB3AK9h0/pz1zkuAd4wxHzhXt4rzVltsreW8nWCMyQVWY9XZt4rzdkLN2FrJeRsHTHW2Ry4CLhKRt3HReWuPSeFHIEZEokXEC7geWGZzTACIiL+z8Q8R8Qd+Buyofy9bLANmO5/PBpbaGMspTvwROF2FDefP2Sj5GrDbGPNMjbdsP291xdZKzluoiHR1PvcFLgb20DrOW62xtYbzZox5yBgTaYyJwrqerTTG3ICrzpsxpt09gMuxeiAlAf9ndzw14uoDbHU+draG2ICFWMXiCqxS1k1AMJAA7Hcug1pRbG8B24Ftzj+KCBviOh+rSnIbsMX5uLw1nLd6YmsN520YsNkZww7gj871reG81RWb7efttDgnAp+48ry1uy6pSimlGq89Vh8ppZRqJE0KSimlqmlSUEopVU2TglJKqWqaFJRSSlXTpKDUGYhIVY1ZMrdIM868KyJRUmMWWKXs5mF3AEq1ASXGmv5AqXZPSwpKNZJY98Z40jkP/3oR6edc31tEEpyTqCWISC/n+nAR+dA5Z/9WETnP+VHuIvKKcx7/L50japWyhSYFpc7M97Tqo+tqvJdvjBkD/BtrJkucz980xgwD3gGed65/HvjaGDMciMMa1Q4QA7xgjIkFcoGZLv02StVDRzQrdQYiUmiMCahlfQpwkTEm2TkJ3RFjTLCIZGNNh1DhXJ9hjAkRkSwg0hhTVuMzorCmaY5xvn4A8DTG/KUFvppSP6ElBaWaxtTxvK5talNW43kV2tanbKRJQammua7G8nvn87VYs1kC/ALr1o5gTVp2B1Tf0KVzSwWpVEPpLxKlzszXeUeuEz43xpzoluotIj9g/cCa5Vx3F/C6iNwPZAG/dq6/G5gnIjdhlQjuwJoFVqlWQ9sUlGokZ5tCvDEm2+5YlGouWn2klFKqmpYUlFJKVdOSglJKqWqaFJRSSlXTpKCUUqqaJgWllFLVNCkopZSqpklBKaVUtf8Hxrr+ieUYG+4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, NUM_EPOCHS+1), log_dict['train_acc_per_epoch'], label='Training')\n",
    "plt.plot(np.arange(1, NUM_EPOCHS+1), log_dict['valid_acc_per_epoch'], label='Validation')\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train ACC: 73.52%\n",
      "Validation ACC: 73.52%\n",
      "Test ACC: 72.11%\n"
     ]
    }
   ],
   "source": [
    "with torch.set_grad_enabled(False):\n",
    "    \n",
    "    train_acc = compute_accuracy(model=model,\n",
    "                                 data_loader=test_loader,\n",
    "                                 device=DEVICE)\n",
    "    \n",
    "    test_acc = compute_accuracy(model=model,\n",
    "                                data_loader=test_loader,\n",
    "                                device=DEVICE)\n",
    "    \n",
    "    valid_acc = compute_accuracy(model=model,\n",
    "                                 data_loader=valid_loader,\n",
    "                                 device=DEVICE)\n",
    "    \n",
    "\n",
    "print(f'Train ACC: {valid_acc:.2f}%')\n",
    "print(f'Validation ACC: {valid_acc:.2f}%')\n",
    "print(f'Test ACC: {test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sys        : 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51) \n",
      "[GCC 9.4.0]\n",
      "matplotlib : 3.3.4\n",
      "PIL        : 9.0.1\n",
      "torchvision: 0.11.2\n",
      "numpy      : 1.22.0\n",
      "torch      : 1.10.1\n",
      "pandas     : 1.4.1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%watermark -iv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "convnet-vgg16.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "371px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}