pytorch_ipynb/cnn/cnn-densenet121-cifar10.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UEBilEjLj5wY"
   },
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 536,
     "status": "ok",
     "timestamp": 1524974472601,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "GOzuY8Yvj5wb",
    "outputId": "c19362ce-f87a-4cc2-84cc-8d7b4b9e6007"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "\n",
      "CPython 3.7.3\n",
      "IPython 7.6.1\n",
      "\n",
      "torch 1.1.0\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rH4XmErYj5wm"
   },
   "source": [
    "# DenseNet-121 CIFAR-10 Image Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network in this notebook is an implementation of the DenseNet-121 [1] architecture on the MNIST digits dataset (http://yann.lecun.com/exdb/mnist/) to train a handwritten digit classifier.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following figure illustrates the main concept of DenseNet: within each \"dense\" block, each layer is connected with each previous layer -- the feature maps are concatenated.\n",
    "\n",
    "\n",
    "![](../images/densenet/densenet-fig-2.jpg)\n",
    "\n",
    "Note that this is somewhat related yet very different to ResNets. ResNets have skip connections approx. between every other layer (but don't connect all layers with each other). Also, ResNets skip connections work via addition\n",
    "\n",
    "$$\\mathbf{x}_{\\ell}=H_{\\ell}\\left(\\mathbf{X}_{\\ell-1}\\right)+\\mathbf{X}_{\\ell-1}$$,\n",
    "\n",
    "whereas $H_{\\ell}(\\cdot)$ can be a composite function  of operations such as Batch Normalization (BN), rectified linear units (ReLU), Pooling, or Convolution (Conv).\n",
    "\n",
    "In DenseNets, all the previous feature maps $\\mathbf{X}_{0}, \\dots, \\mathbf{X}_{\\ell}-1$ of a feature map $\\mathbf{X}_{\\ell}$ are concatenated:\n",
    "\n",
    "$$\\mathbf{x}_{\\ell}=H_{\\ell}\\left(\\left[\\mathbf{x}_{0}, \\mathbf{x}_{1}, \\ldots, \\mathbf{x}_{\\ell-1}\\right]\\right).$$\n",
    "\n",
    "Furthermore, in this particular notebook, we are considering the DenseNet-121, which is depicted below:\n",
    "\n",
    "\n",
    "\n",
    "![](../images/densenet/densenet-tab-1-dnet121.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**References**\n",
    "    \n",
    "- [1] Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708), http://openaccess.thecvf.com/content_cvpr_2017/html/Huang_Densely_Connected_Convolutional_CVPR_2017_paper.html\n",
    "\n",
    "- [2] http://yann.lecun.com/exdb/mnist/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MkoGLH_Tj5wn"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "ORj09gnrj5wp"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data.dataset import Subset\n",
    "\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I6hghKPxj5w0"
   },
   "source": [
    "## Model Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23936,
     "status": "ok",
     "timestamp": 1524974497505,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "NnT0sZIwj5wu",
    "outputId": "55aed925-d17e-4c6a-8c71-0d9b3bde5637"
   },
   "outputs": [],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Hyperparameters\n",
    "RANDOM_SEED = 1\n",
    "LEARNING_RATE = 0.001\n",
    "BATCH_SIZE = 128\n",
    "NUM_EPOCHS = 20\n",
    "\n",
    "# Architecture\n",
    "NUM_CLASSES = 10\n",
    "\n",
    "# Other\n",
    "DEVICE = \"cuda:0\"\n",
    "GRAYSCALE = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CIFAR-10 Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "train_indices = torch.arange(0, 48000)\n",
    "valid_indices = torch.arange(48000, 50000)\n",
    "\n",
    "\n",
    "train_and_valid = datasets.CIFAR10(root='data', \n",
    "                                   train=True, \n",
    "                                   transform=transforms.ToTensor(),\n",
    "                                   download=True)\n",
    "\n",
    "train_dataset = Subset(train_and_valid, train_indices)\n",
    "valid_dataset = Subset(train_and_valid, valid_indices)\n",
    "test_dataset = datasets.CIFAR10(root='data', \n",
    "                                train=False, \n",
    "                                transform=transforms.ToTensor(),\n",
    "                                download=False)\n",
    "\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          num_workers=4,\n",
    "                          shuffle=True)\n",
    "\n",
    "valid_loader = DataLoader(dataset=valid_dataset, \n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          num_workers=4,\n",
    "                          shuffle=False)\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE,\n",
    "                         num_workers=4,\n",
    "                         shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1 | Batch index: 0 | Batch size: 128\n",
      "Epoch: 2 | Batch index: 0 | Batch size: 128\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(DEVICE)\n",
    "torch.manual_seed(0)\n",
    "\n",
    "for epoch in range(2):\n",
    "\n",
    "    for batch_idx, (x, y) in enumerate(train_loader):\n",
    "        \n",
    "        print('Epoch:', epoch+1, end='')\n",
    "        print(' | Batch index:', batch_idx, end='')\n",
    "        print(' | Batch size:', y.size()[0])\n",
    "        \n",
    "        x = x.to(device)\n",
    "        y = y.to(device)\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([3, 0, 1, 3, 3, 5, 0, 4, 9, 4])\n",
      "tensor([1, 0, 4, 1, 8, 2, 0, 3, 5, 3])\n"
     ]
    }
   ],
   "source": [
    "# Check that shuffling works properly\n",
    "# i.e., label indices should be in random order.\n",
    "# Also, the label order should be different in the second\n",
    "# epoch.\n",
    "\n",
    "for images, labels in train_loader:  \n",
    "    pass\n",
    "print(labels[:10])\n",
    "\n",
    "for images, labels in train_loader:  \n",
    "    pass\n",
    "print(labels[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([5, 0, 3, 6, 8, 7, 9, 5, 6, 6])\n",
      "tensor([7, 5, 8, 0, 8, 2, 7, 0, 3, 5])\n"
     ]
    }
   ],
   "source": [
    "# Check that validation set and test sets are diverse\n",
    "# i.e., that they contain all classes\n",
    "\n",
    "for images, labels in valid_loader:  \n",
    "    pass\n",
    "print(labels[:10])\n",
    "\n",
    "for images, labels in test_loader:  \n",
    "    pass\n",
    "print(labels[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### MODEL\n",
    "##########################\n",
    "\n",
    "# The following code cell that implements the DenseNet-121 architecture \n",
    "# is a derivative of the code provided at \n",
    "# https://github.com/pytorch/vision/blob/master/torchvision/models/densenet.py\n",
    "\n",
    "import re\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.utils.checkpoint as cp\n",
    "from collections import OrderedDict\n",
    "\n",
    "\n",
    "\n",
    "def _bn_function_factory(norm, relu, conv):\n",
    "    def bn_function(*inputs):\n",
    "        concated_features = torch.cat(inputs, 1)\n",
    "        bottleneck_output = conv(relu(norm(concated_features)))\n",
    "        return bottleneck_output\n",
    "\n",
    "    return bn_function\n",
    "\n",
    "\n",
    "class _DenseLayer(nn.Sequential):\n",
    "    def __init__(self, num_input_features, growth_rate, bn_size, drop_rate, memory_efficient=False):\n",
    "        super(_DenseLayer, self).__init__()\n",
    "        self.add_module('norm1', nn.BatchNorm2d(num_input_features)),\n",
    "        self.add_module('relu1', nn.ReLU(inplace=True)),\n",
    "        self.add_module('conv1', nn.Conv2d(num_input_features, bn_size *\n",
    "                                           growth_rate, kernel_size=1, stride=1,\n",
    "                                           bias=False)),\n",
    "        self.add_module('norm2', nn.BatchNorm2d(bn_size * growth_rate)),\n",
    "        self.add_module('relu2', nn.ReLU(inplace=True)),\n",
    "        self.add_module('conv2', nn.Conv2d(bn_size * growth_rate, growth_rate,\n",
    "                                           kernel_size=3, stride=1, padding=1,\n",
    "                                           bias=False)),\n",
    "        self.drop_rate = drop_rate\n",
    "        self.memory_efficient = memory_efficient\n",
    "\n",
    "    def forward(self, *prev_features):\n",
    "        bn_function = _bn_function_factory(self.norm1, self.relu1, self.conv1)\n",
    "        if self.memory_efficient and any(prev_feature.requires_grad for prev_feature in prev_features):\n",
    "            bottleneck_output = cp.checkpoint(bn_function, *prev_features)\n",
    "        else:\n",
    "            bottleneck_output = bn_function(*prev_features)\n",
    "        new_features = self.conv2(self.relu2(self.norm2(bottleneck_output)))\n",
    "        if self.drop_rate > 0:\n",
    "            new_features = F.dropout(new_features, p=self.drop_rate,\n",
    "                                     training=self.training)\n",
    "        return new_features\n",
    "\n",
    "\n",
    "class _DenseBlock(nn.Module):\n",
    "    def __init__(self, num_layers, num_input_features, bn_size, growth_rate, drop_rate, memory_efficient=False):\n",
    "        super(_DenseBlock, self).__init__()\n",
    "        for i in range(num_layers):\n",
    "            layer = _DenseLayer(\n",
    "                num_input_features + i * growth_rate,\n",
    "                growth_rate=growth_rate,\n",
    "                bn_size=bn_size,\n",
    "                drop_rate=drop_rate,\n",
    "                memory_efficient=memory_efficient,\n",
    "            )\n",
    "            self.add_module('denselayer%d' % (i + 1), layer)\n",
    "\n",
    "    def forward(self, init_features):\n",
    "        features = [init_features]\n",
    "        for name, layer in self.named_children():\n",
    "            new_features = layer(*features)\n",
    "            features.append(new_features)\n",
    "        return torch.cat(features, 1)\n",
    "\n",
    "\n",
    "class _Transition(nn.Sequential):\n",
    "    def __init__(self, num_input_features, num_output_features):\n",
    "        super(_Transition, self).__init__()\n",
    "        self.add_module('norm', nn.BatchNorm2d(num_input_features))\n",
    "        self.add_module('relu', nn.ReLU(inplace=True))\n",
    "        self.add_module('conv', nn.Conv2d(num_input_features, num_output_features,\n",
    "                                          kernel_size=1, stride=1, bias=False))\n",
    "        self.add_module('pool', nn.AvgPool2d(kernel_size=2, stride=2))\n",
    "\n",
    "\n",
    "class DenseNet121(nn.Module):\n",
    "    r\"\"\"Densenet-BC model class, based on\n",
    "    `\"Densely Connected Convolutional Networks\" <https://arxiv.org/pdf/1608.06993.pdf>`_\n",
    "\n",
    "    Args:\n",
    "        growth_rate (int) - how many filters to add each layer (`k` in paper)\n",
    "        block_config (list of 4 ints) - how many layers in each pooling block\n",
    "        num_init_featuremaps (int) - the number of filters to learn in the first convolution layer\n",
    "        bn_size (int) - multiplicative factor for number of bottle neck layers\n",
    "          (i.e. bn_size * k features in the bottleneck layer)\n",
    "        drop_rate (float) - dropout rate after each dense layer\n",
    "        num_classes (int) - number of classification classes\n",
    "        memory_efficient (bool) - If True, uses checkpointing. Much more memory efficient,\n",
    "          but slower. Default: *False*. See `\"paper\" <https://arxiv.org/pdf/1707.06990.pdf>`_\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, growth_rate=32, block_config=(6, 12, 24, 16),\n",
    "                 num_init_featuremaps=64, bn_size=4, drop_rate=0, num_classes=1000, memory_efficient=False,\n",
    "                 grayscale=False):\n",
    "\n",
    "        super(DenseNet121, self).__init__()\n",
    "\n",
    "        # First convolution\n",
    "        if grayscale:\n",
    "            in_channels=1\n",
    "        else:\n",
    "            in_channels=3\n",
    "        \n",
    "        self.features = nn.Sequential(OrderedDict([\n",
    "            ('conv0', nn.Conv2d(in_channels=in_channels, out_channels=num_init_featuremaps,\n",
    "                                kernel_size=7, stride=2,\n",
    "                                padding=3, bias=False)), # bias is redundant when using batchnorm\n",
    "            ('norm0', nn.BatchNorm2d(num_features=num_init_featuremaps)),\n",
    "            ('relu0', nn.ReLU(inplace=True)),\n",
    "            ('pool0', nn.MaxPool2d(kernel_size=3, stride=2, padding=1)),\n",
    "        ]))\n",
    "\n",
    "        # Each denseblock\n",
    "        num_features = num_init_featuremaps\n",
    "        for i, num_layers in enumerate(block_config):\n",
    "            block = _DenseBlock(\n",
    "                num_layers=num_layers,\n",
    "                num_input_features=num_features,\n",
    "                bn_size=bn_size,\n",
    "                growth_rate=growth_rate,\n",
    "                drop_rate=drop_rate,\n",
    "                memory_efficient=memory_efficient\n",
    "            )\n",
    "            self.features.add_module('denseblock%d' % (i + 1), block)\n",
    "            num_features = num_features + num_layers * growth_rate\n",
    "            if i != len(block_config) - 1:\n",
    "                trans = _Transition(num_input_features=num_features,\n",
    "                                    num_output_features=num_features // 2)\n",
    "                self.features.add_module('transition%d' % (i + 1), trans)\n",
    "                num_features = num_features // 2\n",
    "\n",
    "        # Final batch norm\n",
    "        self.features.add_module('norm5', nn.BatchNorm2d(num_features))\n",
    "\n",
    "        # Linear layer\n",
    "        self.classifier = nn.Linear(num_features, num_classes)\n",
    "\n",
    "        # Official init from torch repo.\n",
    "        for m in self.modules():\n",
    "            if isinstance(m, nn.Conv2d):\n",
    "                nn.init.kaiming_normal_(m.weight)\n",
    "            elif isinstance(m, nn.BatchNorm2d):\n",
    "                nn.init.constant_(m.weight, 1)\n",
    "                nn.init.constant_(m.bias, 0)\n",
    "            elif isinstance(m, nn.Linear):\n",
    "                nn.init.constant_(m.bias, 0)\n",
    "\n",
    "    def forward(self, x):\n",
    "        features = self.features(x)\n",
    "        out = F.relu(features, inplace=True)\n",
    "        out = F.adaptive_avg_pool2d(out, (1, 1))\n",
    "        out = torch.flatten(out, 1)\n",
    "        logits = self.classifier(out)\n",
    "        probas = F.softmax(logits, dim=1)\n",
    "        return logits, probas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "_lza9t_uj5w1"
   },
   "outputs": [],
   "source": [
    "torch.manual_seed(RANDOM_SEED)\n",
    "\n",
    "model = DenseNet121(num_classes=NUM_CLASSES, grayscale=GRAYSCALE)\n",
    "model.to(DEVICE)\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RAodboScj5w6"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_acc(model, data_loader, device):\n",
    "    correct_pred, num_examples = 0, 0\n",
    "    model.eval()\n",
    "    for i, (features, targets) in enumerate(data_loader):\n",
    "            \n",
    "        features = features.to(device)\n",
    "        targets = targets.to(device)\n",
    "\n",
    "        logits, probas = model(features)\n",
    "        _, predicted_labels = torch.max(probas, 1)\n",
    "        num_examples += targets.size(0)\n",
    "        assert predicted_labels.size() == targets.size()\n",
    "        correct_pred += (predicted_labels == targets).sum()\n",
    "    return correct_pred.float()/num_examples * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 1547
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2384585,
     "status": "ok",
     "timestamp": 1524976888520,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "Dzh3ROmRj5w7",
    "outputId": "5f8fd8c9-b076-403a-b0b7-fd2d498b48d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/020 | Batch 000/375 | Cost: 2.4002\n",
      "Epoch: 001/020 | Batch 150/375 | Cost: 1.3511\n",
      "Epoch: 001/020 | Batch 300/375 | Cost: 1.3326\n",
      "Epoch: 001/020\n",
      "Train ACC: 61.52 | Validation ACC: 60.25\n",
      "Time elapsed: 0.99 min\n",
      "Epoch: 002/020 | Batch 000/375 | Cost: 1.1005\n",
      "Epoch: 002/020 | Batch 150/375 | Cost: 1.0096\n",
      "Epoch: 002/020 | Batch 300/375 | Cost: 1.2628\n",
      "Epoch: 002/020\n",
      "Train ACC: 66.19 | Validation ACC: 63.55\n",
      "Time elapsed: 1.90 min\n",
      "Epoch: 003/020 | Batch 000/375 | Cost: 0.6471\n",
      "Epoch: 003/020 | Batch 150/375 | Cost: 0.8107\n",
      "Epoch: 003/020 | Batch 300/375 | Cost: 0.8848\n",
      "Epoch: 003/020\n",
      "Train ACC: 73.32 | Validation ACC: 67.40\n",
      "Time elapsed: 2.79 min\n",
      "Epoch: 004/020 | Batch 000/375 | Cost: 0.6951\n",
      "Epoch: 004/020 | Batch 150/375 | Cost: 0.6852\n",
      "Epoch: 004/020 | Batch 300/375 | Cost: 0.8775\n",
      "Epoch: 004/020\n",
      "Train ACC: 76.38 | Validation ACC: 67.90\n",
      "Time elapsed: 3.65 min\n",
      "Epoch: 005/020 | Batch 000/375 | Cost: 0.4929\n",
      "Epoch: 005/020 | Batch 150/375 | Cost: 0.5942\n",
      "Epoch: 005/020 | Batch 300/375 | Cost: 0.6016\n",
      "Epoch: 005/020\n",
      "Train ACC: 76.70 | Validation ACC: 67.95\n",
      "Time elapsed: 4.53 min\n",
      "Epoch: 006/020 | Batch 000/375 | Cost: 0.5888\n",
      "Epoch: 006/020 | Batch 150/375 | Cost: 0.5500\n",
      "Epoch: 006/020 | Batch 300/375 | Cost: 0.4939\n",
      "Epoch: 006/020\n",
      "Train ACC: 84.21 | Validation ACC: 74.85\n",
      "Time elapsed: 5.51 min\n",
      "Epoch: 007/020 | Batch 000/375 | Cost: 0.3982\n",
      "Epoch: 007/020 | Batch 150/375 | Cost: 0.4272\n",
      "Epoch: 007/020 | Batch 300/375 | Cost: 0.3908\n",
      "Epoch: 007/020\n",
      "Train ACC: 87.29 | Validation ACC: 74.65\n",
      "Time elapsed: 6.43 min\n",
      "Epoch: 008/020 | Batch 000/375 | Cost: 0.4332\n",
      "Epoch: 008/020 | Batch 150/375 | Cost: 0.2335\n",
      "Epoch: 008/020 | Batch 300/375 | Cost: 0.3678\n",
      "Epoch: 008/020\n",
      "Train ACC: 84.10 | Validation ACC: 71.00\n",
      "Time elapsed: 7.35 min\n",
      "Epoch: 009/020 | Batch 000/375 | Cost: 0.2343\n",
      "Epoch: 009/020 | Batch 150/375 | Cost: 0.2704\n",
      "Epoch: 009/020 | Batch 300/375 | Cost: 0.3429\n",
      "Epoch: 009/020\n",
      "Train ACC: 88.51 | Validation ACC: 74.35\n",
      "Time elapsed: 8.26 min\n",
      "Epoch: 010/020 | Batch 000/375 | Cost: 0.1757\n",
      "Epoch: 010/020 | Batch 150/375 | Cost: 0.1748\n",
      "Epoch: 010/020 | Batch 300/375 | Cost: 0.4755\n",
      "Epoch: 010/020\n",
      "Train ACC: 92.77 | Validation ACC: 74.20\n",
      "Time elapsed: 9.27 min\n",
      "Epoch: 011/020 | Batch 000/375 | Cost: 0.2347\n",
      "Epoch: 011/020 | Batch 150/375 | Cost: 0.1618\n",
      "Epoch: 011/020 | Batch 300/375 | Cost: 0.3075\n",
      "Epoch: 011/020\n",
      "Train ACC: 90.78 | Validation ACC: 74.10\n",
      "Time elapsed: 10.24 min\n",
      "Epoch: 012/020 | Batch 000/375 | Cost: 0.1233\n",
      "Epoch: 012/020 | Batch 150/375 | Cost: 0.1385\n",
      "Epoch: 012/020 | Batch 300/375 | Cost: 0.2406\n",
      "Epoch: 012/020\n",
      "Train ACC: 93.26 | Validation ACC: 76.00\n",
      "Time elapsed: 11.15 min\n",
      "Epoch: 013/020 | Batch 000/375 | Cost: 0.0757\n",
      "Epoch: 013/020 | Batch 150/375 | Cost: 0.0658\n",
      "Epoch: 013/020 | Batch 300/375 | Cost: 0.2070\n",
      "Epoch: 013/020\n",
      "Train ACC: 91.64 | Validation ACC: 75.80\n",
      "Time elapsed: 12.13 min\n",
      "Epoch: 014/020 | Batch 000/375 | Cost: 0.1285\n",
      "Epoch: 014/020 | Batch 150/375 | Cost: 0.1468\n",
      "Epoch: 014/020 | Batch 300/375 | Cost: 0.1070\n",
      "Epoch: 014/020\n",
      "Train ACC: 95.18 | Validation ACC: 77.80\n",
      "Time elapsed: 13.10 min\n",
      "Epoch: 015/020 | Batch 000/375 | Cost: 0.1204\n",
      "Epoch: 015/020 | Batch 150/375 | Cost: 0.0461\n",
      "Epoch: 015/020 | Batch 300/375 | Cost: 0.1005\n",
      "Epoch: 015/020\n",
      "Train ACC: 93.26 | Validation ACC: 74.80\n",
      "Time elapsed: 14.10 min\n",
      "Epoch: 016/020 | Batch 000/375 | Cost: 0.0591\n",
      "Epoch: 016/020 | Batch 150/375 | Cost: 0.1024\n",
      "Epoch: 016/020 | Batch 300/375 | Cost: 0.0679\n",
      "Epoch: 016/020\n",
      "Train ACC: 95.66 | Validation ACC: 76.80\n",
      "Time elapsed: 15.02 min\n",
      "Epoch: 017/020 | Batch 000/375 | Cost: 0.0359\n",
      "Epoch: 017/020 | Batch 150/375 | Cost: 0.0615\n",
      "Epoch: 017/020 | Batch 300/375 | Cost: 0.0725\n",
      "Epoch: 017/020\n",
      "Train ACC: 92.06 | Validation ACC: 75.85\n",
      "Time elapsed: 16.01 min\n",
      "Epoch: 018/020 | Batch 000/375 | Cost: 0.0831\n",
      "Epoch: 018/020 | Batch 150/375 | Cost: 0.1023\n",
      "Epoch: 018/020 | Batch 300/375 | Cost: 0.0666\n",
      "Epoch: 018/020\n",
      "Train ACC: 94.32 | Validation ACC: 75.10\n",
      "Time elapsed: 17.01 min\n",
      "Epoch: 019/020 | Batch 000/375 | Cost: 0.0499\n",
      "Epoch: 019/020 | Batch 150/375 | Cost: 0.1068\n",
      "Epoch: 019/020 | Batch 300/375 | Cost: 0.0657\n",
      "Epoch: 019/020\n",
      "Train ACC: 96.89 | Validation ACC: 77.75\n",
      "Time elapsed: 17.99 min\n",
      "Epoch: 020/020 | Batch 000/375 | Cost: 0.0751\n",
      "Epoch: 020/020 | Batch 150/375 | Cost: 0.0423\n",
      "Epoch: 020/020 | Batch 300/375 | Cost: 0.0398\n",
      "Epoch: 020/020\n",
      "Train ACC: 94.66 | Validation ACC: 76.70\n",
      "Time elapsed: 18.90 min\n",
      "Total Training Time: 18.90 min\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "cost_list = []\n",
    "train_acc_list, valid_acc_list = [], []\n",
    "\n",
    "\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    \n",
    "    model.train()\n",
    "    for batch_idx, (features, targets) in enumerate(train_loader):\n",
    "        \n",
    "        features = features.to(DEVICE)\n",
    "        targets = targets.to(DEVICE)\n",
    "            \n",
    "        ### FORWARD AND BACK PROP\n",
    "        logits, probas = model(features)\n",
    "        cost = F.cross_entropy(logits, targets)\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        cost.backward()\n",
    "        \n",
    "        ### UPDATE MODEL PARAMETERS\n",
    "        optimizer.step()\n",
    "        \n",
    "        #################################################\n",
    "        ### CODE ONLY FOR LOGGING BEYOND THIS POINT\n",
    "        ################################################\n",
    "        cost_list.append(cost.item())\n",
    "        if not batch_idx % 150:\n",
    "            print (f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d} | '\n",
    "                   f'Batch {batch_idx:03d}/{len(train_loader):03d} |' \n",
    "                   f' Cost: {cost:.4f}')\n",
    "\n",
    "        \n",
    "\n",
    "    model.eval()\n",
    "    with torch.set_grad_enabled(False): # save memory during inference\n",
    "        \n",
    "        train_acc = compute_acc(model, train_loader, device=DEVICE)\n",
    "        valid_acc = compute_acc(model, valid_loader, device=DEVICE)\n",
    "        \n",
    "        print(f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d}\\n'\n",
    "              f'Train ACC: {train_acc:.2f} | Validation ACC: {valid_acc:.2f}')\n",
    "        \n",
    "        train_acc_list.append(train_acc)\n",
    "        valid_acc_list.append(valid_acc)\n",
    "        \n",
    "    elapsed = (time.time() - start_time)/60\n",
    "    print(f'Time elapsed: {elapsed:.2f} min')\n",
    "  \n",
    "elapsed = (time.time() - start_time)/60\n",
    "print(f'Total Training Time: {elapsed:.2f} min')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "paaeEQHQj5xC"
   },
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6514,
     "status": "ok",
     "timestamp": 1524976895054,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "gzQMWKq5j5xE",
    "outputId": "de7dc005-5eeb-4177-9f9f-d9b5d1358db9"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEJCAYAAACZjSCSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3hUxdfA8e9sSUJHmkgNTakJJfQiHQTsqBRRRMWGKCqKIoiIyE9RpCOKgKiAoiC+IB2kgwHpIDVAqCFACCFly7x/bLKk7G42ZUkg5/M8edi9d+69E9A9O+2M0lojhBAi7zLkdAWEEELkLAkEQgiRx0kgEEKIPE4CgRBC5HESCIQQIo+TQCCEEHmczwKBUqq8UmqtUuqgUmq/UuoNF2VaK6WilFK7En+G+6o+QgghXDP58N5W4G2t9U6lVCFgh1Jqpdb6QKpyG7TW3XxYDyGEEB74LBBorc8B5xJfRyulDgJlgdSBIENKlCihAwMDs15BIYTIQ3bs2HFJa13S1TlftgiclFKBQD1gm4vTTZVSu4GzwDta6/2e7hUYGEhoaGi211EIIe5kSqmT7s75PBAopQoCvwFvaq2vpTq9E6iotb6ulOoCLAKqubhHf6A/QIUKFXxcYyGEyFt8OmtIKWXGEQR+0lr/nvq81vqa1vp64uulgFkpVcJFuela6xCtdUjJki5bNkIIITLJl7OGFDADOKi1/spNmdKJ5VBKNUqsT6Sv6iSEECItX3YNNQf6AHuVUrsSj30AVADQWk8DugOvKKWsQCzQQ0s6VCF8xmKxEB4eTlxcXE5XRfhIQEAA5cqVw2w2e32NL2cNbQRUOmUmAZN8VQchRErh4eEUKlSIwMBAEhvj4g6itSYyMpLw8HAqVark9XWysliIPCQuLo7ixYtLELhDKaUoXrx4hlt8EgiEyGMkCNzZMvPvm2cCwX/no/lyxX9cuh6f01URQohcJc8EgqMXrzNxzVEuxyTkdFWEyNOUUvTp08f53mq1UrJkSbp1c2SaWbx4MWPGjPF4j7Nnz9K9e3cAZs2axYABAzJUh9GjR6dbpm/fvixYsCBD982MXbt2sXTpUp8/x5M8EwiSWkt2mZQkRI4qUKAA+/btIzY2FoCVK1dStmxZ5/mHHnqIIUOGeLxHmTJlsvQh7U0guFUkENxCSb1mEgeEyHkPPPAAS5YsAWDu3Ln07NnTeS75N/y+ffsycOBAmjVrRuXKlZ0f/mFhYdSuXdt5zenTp+ncuTP33XcfH3/8sfP4I488QoMGDahVqxbTp08HYMiQIcTGxlK3bl169+4NwA8//EBQUBDBwcEpWivr169P8+zUXF178uRJ2rVrR1BQEO3atePUqVMA/Prrr9SuXZvg4GBatWpFQkICw4cPZ/78+dStW5f58+dn7S82k25JrqHcIGkARQKBEA4f/7mfA2dTZ33JmpplCvPRg7XSLdejRw9GjhxJt27d2LNnD/369WPDhg0uy547d46NGzdy6NAhHnroIWeXUHLbt29n37595M+fn4YNG9K1a1dCQkL4/vvvKVasGLGxsTRs2JDHH3+cMWPGMGnSJHbtcixv2r9/P59++imbNm2iRIkSXL582etnu7t2wIABPPPMMzz77LN8//33DBw4kEWLFjFy5EiWL19O2bJluXr1Kn5+fowcOZLQ0FAmTcq5mfR5p0UgXUNC5BpBQUGEhYUxd+5cunTp4rHsI488gsFgoGbNmly4cMFlmQ4dOlC8eHHy5cvHY489xsaNGwGYMGECwcHBNGnShNOnT3PkyJE0165Zs4bu3btTooQju02xYsW8fra7a7ds2UKvXr0A6NOnj7M+zZs3p2/fvnz77bfYbDaPv/etlGdaBAaZMidECt58c/elhx56iHfeeYd169YRGek+s4y/v7/ztbvEA6mnTCqlWLduHatWrWLLli3kz5+f1q1bu5xfr7V2O+UyvWd7utZV/aZNm8a2bdtYsmQJdevWdbZKclreaREk/iktAiFyh379+jF8+HDq1KmT5XutXLmSy5cvExsby6JFi2jevDlRUVHcdddd5M+fn0OHDrF161ZnebPZjMViAaBdu3b88ssvzmCUvGsoPe6ubdasGfPmzQPgp59+okWLFgAcO3aMxo0bM3LkSEqUKMHp06cpVKgQ0dHRWf47yIo8EwgMib+pxAEhcody5crxxhtpdrDNlBYtWtCnTx/q1q3L448/TkhICJ07d8ZqtRIUFMSwYcNo0qSJs3z//v0JCgqid+/e1KpVi6FDh3L//fcTHBzMW2+95fVz3V07YcIEZs6cSVBQEHPmzGH8+PEADB48mDp16lC7dm1atWpFcHAwbdq04cCBAzk6WKxutxxvISEhOjMb06w9dJHnZv3DwlebUa/CXT6omRC538GDB6lRo0ZOV0P4mKt/Z6XUDq11iKvyeaZFkNQ3dHuFPSGE8L08EwgMMn1UCCFcyjOB4OaCMokEQgiRXJ4JBM4WQQ7XQwghcps8EwicC8rsEgqEECK5vBMIEv+UMCCEECnlnUCQ2CSQBWVC5Cyj0UjdunWpXbs2Dz74IFevXvXJc5o1a+aT+96J8lAgSHwhcUCIHJUvXz527drFvn37KFasGJMnT/bJczZv3uyT+2aH3JRnCPJQIJDBYiFyn6ZNm3LmzBkA1q1b59ycBhwZPGfNmgVAYGAgH330EfXr16dOnTocOnQIgBEjRtCvXz9at25N5cqVmTBhgvP6ggULOu/bunVrunfvTvXq1endu7dz9uDSpUupXr06LVq0YODAgSmenyQsLIyWLVtSv3596tev7wwwTz31VIp9BPr27ctvv/2GzWZj8ODBNGzYkKCgIL755htnPdq0aUOvXr2caTVcpckGmDFjBvfeey+tW7fmxRdfdKbljoiI4PHHH6dhw4Y0bNiQTZs2ZeFv/6Y8k3ROso8KkcpfQ+D83uy9Z+k68IDn3cWS2Gw2Vq9ezfPPP+9V+RIlSrBz506mTJnC2LFj+e677wA4dOgQa9euJTo6mvvuu49XXnkFs9mc4tp///2X/fv3U6ZMGZo3b86mTZsICQnhpZdeYv369VSqVCnFngjJlSpVipUrVxIQEMCRI0fo2bMnoaGh9OjRg/nz59OlSxcSEhJYvXo1U6dOZcaMGRQpUoR//vmH+Ph4mjdvTseOHYGb6bIrVaoE4DJNdnx8PJ988gk7d+6kUKFCtG3bluDgYADeeOMNBg0aRIsWLTh16hSdOnXi4MGDXv39eZJnAoEhaWWxxAEhclTSpjBhYWE0aNCADh06eHXdY489BkCDBg34/fffnce7du2Kv78//v7+lCpVigsXLlCuXLkU1zZq1Mh5LOnZBQsWpHLlys4P5Z49e6b4Vp7EYrEwYMAAdu3ahdFo5PDhw4Bjc52BAwcSHx/PsmXLaNWqFfny5WPFihXs2bPHuZFNVFQUR44cwc/Pj0aNGjmfB46cRAsXLgRwpsk+f/48999/vzOl9RNPPOF85qpVqzhw4IDz+mvXrhEdHU2hQoW8+jt0J88EgqR5Q9IiECKRl9/cs1vSGEFUVBTdunVj8uTJDBw4EJPJhN1ud5ZLnTI6KSW00WjEarWmOe7qnKcy3i4uHTduHHfffTe7d+/GbrcTEBAAQEBAAK1bt2b58uXMnz/f2aLQWjNx4kQ6deqU4j7r1q2jQIECKd67SpPtqV52u50tW7aQL18+r+rurTwzRqAk15AQuUqRIkWYMGECY8eOxWKxULFiRQ4cOEB8fDxRUVGsXr3ap8+vXr06x48fJywsDMBt5s+oqCjuueceDAYDc+bMSTHQ26NHD2bOnMmGDRucH/ydOnVi6tSpzjTXhw8fJiYmxuV9XaXJbtSoEX///TdXrlzBarXy22+/Oa/p2LFjip3Msms/gzwTCAwSCYTIderVq0dwcDDz5s2jfPnyPPnkk8700PXq1fPps/Ply8eUKVPo3LkzLVq04O6776ZIkSJpyr366qvMnj2bJk2acPjw4RTf6jt27Mj69etp3749fn5+ALzwwgvUrFmT+vXrU7t2bV566SWXrRR3abLLli3LBx98QOPGjWnfvj01a9Z01mvChAmEhoYSFBREzZo1mTZtWrb8XeSZNNS7T1/l4cmbmPFsCO1q3O2DmgmR+0ka6pSuX79OwYIF0Vrz2muvUa1aNQYNGpTT1XLWy2q18uijj9KvXz8effRRr6+XNNRuSPZRIURq3377LXXr1qVWrVpERUXx0ksv5XSVAMe02KRFd5UqVeKRRx7x6fPyzGCxTB8VQqQ2aNCgXNECSG3s2LG39Hl5pkUgQwRCONxu3cEiYzLz75t3AgFJXUPyP4HIuwICAoiMjJT/D+5QWmsiIyOdU1y9lee6hs5HxXkuKMQdrFy5coSHhxMREZHTVRE+EhAQkGZBXXryTCA4cckxj3fEnwfo27xSOqWFuDOZzeYUK1uFAB92DSmlyiul1iqlDiql9iul3nBRRimlJiiljiql9iil6vuqPqakHBNCCCFS8GWLwAq8rbXeqZQqBOxQSq3UWh9IVuYBoFriT2NgauKf2S6/X55p/AghRIb4rEWgtT6ntd6Z+DoaOAiUTVXsYeAH7bAVKKqUuscX9cnvb/TFbYUQ4rZ3S2YNKaUCgXrAtlSnygKnk70PJ22wQCnVXykVqpQKzewgV34/CQRCCOGKzwOBUqog8Bvwptb6WurTLi5JM69Naz1dax2itQ4pWbJkpuqRzyyBQAghXPFpIFBKmXEEgZ+01r+7KBIOlE/2vhxw1hd18TPlmSUTQgiRIb6cNaSAGcBBrfVXbootBp5JnD3UBIjSWp/zRX38jBIIhBDCFV9OpWkO9AH2KqWSkmZ/AFQA0FpPA5YCXYCjwA3gOV9VRloEQgjhms8CgdZ6I67HAJKX0cBrvqpDcmZpEQghhEt55tNRFpQJIYRreSYQGCUQCCGES3kmECglgUAIIVzJM4FACCGEaxIIhBAij5NAIIQQeZwEAiGEyOMkEAghRB4ngUAIIfK4PBkIVh24kNNVEEKIXCNPBoK3ftmVfiEhhMgj8mQguBZnzekqCCFErpEnA4EQQoibJBAIIUQeJ4FACCHyOAkEQgiRx0kgEEKIPE4CgRBC5HESCIQQIo+TQCCEEHlcng0E/52PzukqCCFErpBnA8GSPWex2XVOV0MIIXJcng0EE9YcZdB8yTkkhBB5NhAALN59NqerIIQQOS5PBwKAk5ExOV0FIYTIUXk+EDw4cWNOV0EIIXJUuoFAKWW8FRXJKZKSWgiR13nTIjiqlPpCKVXT57URQghxy3kTCIKAw8B3SqmtSqn+SqnCPq7XLaW1TCMVQuRd6QYCrXW01vpbrXUz4F3gI+CcUmq2Uqqqz2t4C5y4JAPGQoi8y6sxAqXUQ0qphcB44EugMvAnsNTH9ctW3/cNcXn89JXYW1wTIYTIPbzpGjoCPAx8obWup7X+Smt9QWu9AFjm2+plrzb3lXJ5vO/M7be4JkIIkXuYvCgTpLW+7uqE1nqgu4uUUt8D3YCLWuvaLs63Bv4ATiQe+l1rPdKL+mSaUsrlcRkiEELkZd60CEoppf5USl1SSl1USv2hlKrsxXWzgM7plNmgta6b+OPTICCEEMI1bwLBz8AvQGmgDPArMDe9i7TW64HLWaqdEEIIn/MmECit9RyttTXx50cguzpTmiqldiul/lJK1cqme7p2+Thsm04BZGBYCCGS8yYQrFVKDVFKBSqlKiql3gWWKKWKKaWKZeHZO4GKWutgYCKwyF3BxLULoUqp0IiIiMw97fw++GswFdUFt0XiLLbM3VsIIW5j3gwWP5X450upjvfD0TLwZrwgDa31tWSvlyqlpiilSmitL7koOx2YDhASEpK51kiRcgC0KBXHARexIHDIEgDGPRXMo/XKZeoRQghxO/JmQVklDz+ZCgIASqnSKnEaj1KqUWJdIjN7v3QlBoI+NTz/yov+ldTUQoi8Jd0WgVLKDLwCtEo8tA74RmttSee6uUBroIRSKhzHimQzgNZ6GtAdeEUpZQVigR7al7keCpQE/8IEXD0C3Ou2WNKuZVprrsdbuXAtnqqlCvqsWkIIkdO86RqaiuMDfEri+z6Jx17wdJHWumc65ycBk7x4fvZQCu4JpkDEHqCr22Ibj15i3MrD5PMzMuavQwCse6c1gSUK3KKKCiHEreVNIGiYOKCbZI1SarevKuRTZesTsGUKflhIcDROXBq/+gi1y97Mq3fperwEAiHEHcubWUM2pVSVpDeJi8luz+k1ZRtgsFuooU7mdE2EECLX8KZFMBjHFNLjgAIqAs/5tFa+UqY+AEGG4+y2eU6ceiPhZqyTDBRCiDuZx0CglDLgGMitBtyHIxAc0lrH34K6Zb8i5bAElCDYdpw56bRpjkdIamohRN7gsWtIa20HvtRax2ut92itd9+2QQBAKWJLBlFXHc3pmgghRK7hzRjBCqXU40lz/m93+aq3p6rhLFVVeE5XRQghcgVvAsFbOBLNxSulrimlopVS19K7KLcyBz+BXRl5xLgpp6sihBC5gjcriwtprQ1aaz+tdeHE97fvnsUFS3G6aGMeMW5CYffqEtmvQAhxJ/Nmq8rV3hy7nRjqPkU5dYkQdTinqyKEEDnObSBQSgUkZhctoZS6KynbqFIqEMe+BLet8k2fIE4F8KhxY05XRQghcpynFsFLwA6geuKfST9/AJN9XzUf8iuAqdaDdDVuxQ+PKZNcstjsTFh9hNiE23NdnRBCJOc2EGitx2utKwHvaK0rJ8s4GpyYJ+i2ZqrbgyLqBp0M/6RbduG/KWcY/RoazlcrDzNxzRFfVU8IIW4ZbwaLJyqlmimleimlnkn6uRWV86nKbThuL80LpqWkt3Z47vbTKb79J21gcyNVi2DVgQv0mL4FXyZRFUKI7ObNYPEcYCzQAmiY+BPi43r5nsHIt7auBBuOE6L+S7d42y/XpVum/5xQth6/jM2uWXngAmGXZHWyECL38ybXUAhQ06d7BeSQP2zN+cD0M08Z1xFqre6x7LmouHTvp5QCrdHAiz+EAhA2xn3KayGEyA28WVC2Dyjt64rkhBsE8KetKU+Y1lNdnUq3/OWYBCavPeq2Iylp6fWdFzKFEHcybwJBCeCAUmq5Umpx0o+vK3arfGPrBsBk83iM6WTXfnfBHr5Y/h//nLjs8ryrJBxaa9YfjsBul+gghMidvAkEI4BHgNHAl8l+bnszn2vISV2a4ZZnqWI4R3vDTo/lVx107HpvS+cr/wcL9zpfL9t3nme+387MzWFZrq8QQviCpwVl1QG01n8DW7XWfyf9ALdvBtJkDIlf4X+2tSNK5+dB42Yvr3N9XCV2Di3YcXO6adLYwunLN7JQUyGE8B1PLYKfk73ekurcFO4gVkzMt7Whs+EfipF+Pr294VHO1/FWG/O2n+LA2Ws3Bwm8dOFaHBNXH5HppkKIHOVp1pBy89rV+9tSsyrFna9/s7Wkv2kJjxo3MMPmeabP2WQziO77cFmmn//63H/ZfuIybWuUolaZIpm+jxBCZIWnFoF289rV+9uS2WigfoWiAPynKxCl8zPM/FO6g8ZJ9oRfzdLzY+KtANi9S4IqhBA+4SkQlFNKTVBKTUz2Oul92VtUP5+b0ruB8/VsW0cAXjL+6dW1O09lLRAkuTO2/BFC3K48dQ0NTvY6NNW51O9vW6WLBDhfj7N2Z6BpEd2N65lieyRb7u+p6SRDA0KI3MBtINBaz76VFckNNAaGW55lpHk296rTHNbls3zPi9HuVyRLHBBC5AberCPIU/7P1pR4baa3cVW23M+bb/3SNSSEyEkSCFK5TGGW20N41rSSu7yYSpoeTyuKZdqoECI3kEDgwhxrBwBeNy3K8r3OXI1Nt4y6M2bjCiFuU96kof5cKVVYKWVWSq1WSl1SSj19KyqXU+q36sI8a2ueNq7kHiKzdK+/9p1P8T4iOt7ZEjh0PhqQriEhRM7ypkXQUWt9DegGhAP3knJG0R2jeVXHArOQisWYZHsEP2XjceP6bLt/2KUYGn66im83HOdKTEK23VcIIbLCm0BgTvyzCzBXa+069eYdxGRQhOtSbLTVoodpLQayvuLLardz+ooj39DfhyNIsN28p7QIhBA5yZtA8KdS6hCODWpWK6VKAunv0nIbMxsNlLsrHz/Z2lNOXeJBg3fJ6Dy5Hmd1jgWkHiN+a/5uhi7cS5zFRuR1Rz6/wxei6Ttzu3NbTCGE8BVv9iweAjQFQrTWFiAGeDi965RS3yulLiql9rk5rxJXKh9VSu1RStXPaOWzy+fdgxj7RLDzvUaz8b22rLCH8K+9KuPMUwlU57L0DI37b/4Hzl3jp22nePKbLTQY5Zi2+uGifaz7L4Jdpz2vXl6x/zyBQ5Zw3osd1IQQwhVvBoufAKxaa5tS6kPgR6CMF/eeBXT2cP4BoFriT39gqhf39IknQ8rTvUG5NLN3bBh5w/IaBqV5zzQvS884eO7mVNTNxyIJv5I2LfWeZFlNvTV3u2NntQPnMn6tEEKAd11Dw7TW0UqpFkAnYDZefGhrrdcDnsYTHgZ+0A5bgaJKqXu8qfStMrjTfZzSd/OlpTsPGP9hr//zlOFSpu51+MJ1jkVcd75/fGrqzN4ZY7WlHLeQJQlCiMzyJhAkdVJ3BaZqrf8A/LLh2WWB08neh5PLktmZjY4WwuoST7PNXp1CKpbNAQP5yfwpBUh/fUBqw//Y71W5qBsWEqzuB6hPRd6g6tC/qDNieYbrIIQQqXkTCM4opb4BngSWKqX8vbwuPa56zF1+r1VK9VdKhSqlQiMiIrLh0a599GBNmlUpTsPAYimOB1cszlMJwxls6c9JeymaG/ezP+B5+nuZpTSjgkeucI4NnLmSNuAcOu/oZoqOszqPycwjIURmefOB/iSwHOistb4KFCN71hGEA8mzupUDzroqqLWerrUO0VqHlCxZMhse7Vq1uwvx84tNCDAbU51xfMr+amvN/Qlf84mlNwAfmOeyx/8FgtQxn9Xp7V93pzkmvUBCiOzkzayhG8AxoJNSagBQSmu9IhuevRh4JnH2UBMgSmudtak52Sxp8Dj1t+0Ztq60iB8PQGF1g8X+w2ht+PeW1UvGA4QQ2cmbWUNvAD8BpRJ/flRKve7FdXNx7HV8n1IqXCn1vFLqZaXUy4lFlgLHgaPAt8CrmfwdfOahumUoWzQfzzStmOZcuC5JYNzPdI4fA8A35nGobFh45kra5HSeI8GWY5G89tNOSWonhPCKN11DzwONtdbDtdbDgSbAi+ldpLXuqbW+R2tt1lqX01rP0FpP01pPSzyvtdavaa2raK3raK1z3WY3dxcOYNOQtlQqUcBtmUO6AqMtPfFXVrpnYzqK5JbuPZ9umX6zbv719Z25nSV7zxHvYcBZCCGSeBMIFKTYxNfGHbJ5vbcM6YzEfmvryi57Fb4wT6cgadcHZNWFa3Gs/e8iNYYtc+5znER5qJs0CIQQ3vAmEMwEtimlRiilRgBbgRk+rVUuk17U0xgYa30SgBdNS3xShy9X/EesxcbxiBj5gBdCZCtvBou/Ap7DsTjsCvCc1vprX1csN0mvRQCw0V6HlbYGvGFaSFGis/X5ltSLx9yUW7o3V421CyFuEx4DgVLKoJTap7XeqbWeoLUer7W+ddNjcglv5+h/b3Nk1MhqOorUkn/wrzhwnld/2umyXNLxpLGBGsOXcSoy+7uqhBB3Fo+BQGttB3YrpSrcovrkSp764ZPbYq/FfGtreprW8opxcbY935Zsu8uJa456LHvgbMrtNb9a+V+21UMIcWfyZozgHmB/4u5ki5N+fF2x29UnVsfmbe+Z51ESz5lDvfXFcu8/zLtM2JDivYctk4UQAvAuEHyMY3eykcCXyX6EC9fJT5v4L7FpxU9+n2bLpjYA+85cS7+QC4t3n8Uu0UAI4YHbQKCUqqqUaq61/jv5D44u6/BbV8Xbzwl9D59Y+3Cv4QwH/Z/z6bPWHLqYbhkJA0IITzy1CL4Gl9NfbiSeEx7MsnUCwF9Z+Nb8JQbslORqtrUQMiIziz4Onb/G9VRrFoQQdyZPgSBQa70n9cHEFcCBPqvRHUNRPW4mVm2gg3EHxwOe5p+AVzke8DSfmb4lP3EYsXErvq97esL/lh1i4NyUE8G01nT+egP9Zv3j24oJIXIFT4EgwMO5fNldkdzuzwEtGN6tpstzPRvdnFTVLejm3jpx+HNf/GzW2OpyVhfjmnb8tfU0reVAQD+OBfQhLKA3HQ2+/8C12uzO2UdL9pxzbm05dd0xFu9OmfQ1acHa9hOe9hUSQtwpPAWCf5RSaXIKKaWeB3b4rkq5U51yRejXohJHP32AHR+2T3GufoWiAPRrXokXWlZOcc6GkX6Wd2kWP4mg+BkExv3Miwlv8bX1MeK0GYDx5skU5jq+VHXoX3T+ej0Wm53Xft7JU9NT7pAWOGQJW49H+rQOQojcyeTh3JvAQqVUb25+8Ifg2J3sUV9XLLcyGQ0UK+DHfXcXAuC/C9GUKOhP2JiuAOw4eSXde6y0h7DSHsLX1u7UUCf5y/99Xjct4tPEqafZLSkL6ZGL17HaHK9dbXjzS+hpmlQuzqzNYT6phxAid3IbCLTWF4BmSqk2QO3Ew0u01mtuSc1yMaUUywe1wmqzs+HIJdpUL+U8V+OeQlQuUYDjl2K8utdBXZGVtgY8ZVzLWOuTxGfLLqApLd9/4Wb9hi8DwGrXrD54IUU5hSLOYmPk/x1wvM/gKPPVGwmsOXSRx+qXy1qFhRC3lDe5htZqrScm/uT5IJCcyWhIEQQA8vuZWPNO6wzdZ7atI4VVLOPMU7Kxdjf9fdj1FNPnZ6fM/P3bznCqD1vmfJ/R2UZvzt/FW7/s5liEb7u5hBDZKzv2HhZZtNFem3W2YLoYt/NuNucpAjAafJM1/OK1OP7Ydcb5PmkAOt4i+yAIcTuRQJArKF6yDALgVdNi2hpcJ5XLLG+yp7qS3oLkZ77fzhvzdnEtzpKp+wshcgcJBLlEPH4Ex00nXpuYZh5HE8OBbLt3drUI9p2J4nJMAqcvOzKanr3qGHCWFBZC3N4kEPjIN30aZPiaKArSIn4CF3Qx5vmNorKFczAAACAASURBVDTZM50zsy2C5HaeukK3iRup/8lKWn6+FkiZlfVidJxz/cG5qFjWHLrg6jZCiFxIAoGPdKpVmp6NKvC/x+tk6LoIijLU2g+ArQGvp1hfEEA8/iRQU4UxyTyeOuq4V8EiO1oEFxL7/11Ztu88jT5dzX8XHBlJnp8dmmIPZSFE7uZpHYHIos8ecwSB937bm6Hr1tuD+dDyHKPMM/nU/D3LbI2Y7DchTbluxm1YtJGuCaM5rMu7vV92dAyZje6/M2w4cilT9/xj1xnuK12I6qULZ7ZaQohsIIEgl/rR1oGa6iS9TGt40LgVgGidjzj8+D9bE0qqKAA6G7YzwLSIgZbX3d7ryo2sD+aaTe4DwZJMbpH5xrxdAM7FeEKInCGBIBf7xPo0RdV1ChDHK5Y3ueEi/dN7prm8ZPw/xqnunND3uLiLY31AVqXuXYq8Hk9UbOYDjAwwC5F7yBhBDhjxYE32jujIsjdbUtDffSyOJYBXLW/yrGWIyyAA8J21C/GYGWaa46vqAtBnxvYU79uMXZel+039+1iWrhdCZB8JBLdAz0YVaFypmPP9s80CKRRgpnrpwnzySK0s3TuSIsyztaGtcRddDFuzWlWvXYtLf6+CaA/rC6auy/5AcPFaHOFXbmT7fYW400kguAU+e6wOIx92pGvq2ah8immXj9Qty/+93iJL9x9j7el4jvm7bM9impXFYm/9stvtOV9setNo9Gpa/G9ttt9XiDudBIJb5L7ShTjy6QN89lhQiuNKKWqXLcKkXvXoUPPuTN07Hj+6xo+miLrBj36fUZaI7KgyAO8tSLM3kdeOe5lzSGvNj1tPEpUNg9pCiIyTQHALeZqC2S2oDBN71ktx7OcXG3t97/06kGGWvtRUJ1nv/yYlcZcO2zFIW0WdoZE6SHo7pP2173ymv71rL8eDtx6/zIeL9vHOAvctCCGE78isoVwkwGykbNF8nElM3dCsSokMXT/H1pGjuixz/T5ljt8YeiUM5TKFCVZHaWHYRyfjPwQZTmDXCoNyfEo/n/A2q+2eV0HHW2yZ+n1sySLBv6eucPpKLA8Fl0lTrtd3jrGN4xHXOR5xnR0nr9CkcnHKF8ufqecKITJGAkEu9e0zIZm6bou9Fj0ThjLL/Dnz/D5hvq01w8w/pSgTTT5CbffRzvgv75h+YU1CPbSHxuG8f05nqi5Jm+AAPDplM4DLQJAUL45FxND2y7+dx2V9gRC3hgSCXCZpX+FaZTK/2naLvRZvW15mnHmKMwh8ZulJhC7C7/ZWznLP25cyzPwjzQ372Wh3nwrji+X/ZaoeSS2b5K7EJHBXgezffEcIkXkyRpDLPNnQkSqiSD7HfsbFM/mh+X/2pjxjGcJGWy06xH/ON7YHUwQBgB9t7YnQhXnN+EfWKp0BzcZk795GsQk2/rfsEHGZ7L4SQvg4ECilOiul/lNKHVVKDXFxvq9SKkIptSvx5wVf1ud2MKh9NQ6PeoACiQvNKhTPfD/5FnstnrYM5Yh2vXVkPH5MtT5MU+MBHjBsy/RzMiLWYnOmr/bWzlNXCByyJMUspJ+3neL05RtM+/sYU9cdY86Wk85z2ttR6kRxFhsXr7lPqifEnc5ngUApZQQmAw8ANYGeSqmaLorO11rXTfz5zlf1uV0opfDzkNcnu/1g68AuexWm+o2nnWFHtt9/9cELnItK+cGf0VbBon8du6CtP+yYFnsjwcoHC/fSY/pW4qyOloDFfnNXtKGL9mXo/n1nbqfR6NUZukaIO4kvP3EaAUe11se11gnAPOBhHz7vjuTtRvD9mlfipfsrAzCpV710St9kxcSAxIR1M/y+pKUh/XUDVdQZRphmcZ86lW7Z52eH8ujkzV7Xx5OkhXhJaYqu3khwnvt82c1xjJ+3pV+v5LYev5z1yglxG/NlICgLJJ9uEp54LLXHlVJ7lFILlFLucynnUU83ruBVuQ+71uC9TtVZMagV3YLKsHhAc6+fEa5L0ShuMpd1Qeb4jaGyOuuynMLOK8bFrPB7l76mFUw2T8CfBJdlkzufxW6X1D09Kbp+JHedEFnmy0DgKg1+6v9t/wQCtdZBwCpgtssbKdVfKRWqlAqNiMi+VbO3A+Xl7mJKgcGguPfuQgAElSuaoedc5C7esbwMwBr/d1KsTlbY6Wtcxmq/d3jPPI/t9hp8YulNVcNZ3jH9kqHnZCellMQBIbKBLwNBOJD8G345IMVXTa11pNY6PvHtt4DLlU1a6+la6xCtdUjJkiV9UtncLGmHsVplCrN3REfn8ZKF/J2vXQWM2mUzNgV1jb0+zye8DcA3fuPww8LTxpUc8O/HCPMPXCM/gxJeoadlKDNsXZlnbc2zxuWUIXMb02SGxWYnzuIYD7geb+VaJlNhn7gUk6JrKTO01iz8N1xmLInbni8DwT9ANaVUJaWUH9ADWJy8gFIqeQL9h4CDPqzPbWvnhx14u8O9zOzbkEIBZufx9NoKpQq5Tl3tyWp7A15KeJPahjAOBzzLKPNMAN6zvMgjCZ+w0N7S+eQJ1sewY2Co+ccMPyczjlyM5sGJG2n46Srnscwudmszdh0PjN+Q4tjhxK02vfX34QgGzd+d6XUWQuQWPgsEWmsrMABYjuMD/het9X6l1Eil1EOJxQYqpfYrpXYDA4G+vqrP7axIfjOvt6tGqcKOD/Y/B7TwauVxZrYqXjygOcvtjfjN5siIGqP9qRf/DfNtbUgdes5SgsnWh+lq3E4PY/auD3Dlx62nOHQ+Yx/WnpxLtQ9zx3Hr2XnKXY6mtJJScWd1DESInObTeYpa66Va63u11lW01p8mHhuutV6c+Pp9rXUtrXWw1rqN1vqQL+tzp6hTrohXmUpNBs//vKvfvj9NCux77y7EikGteNvyCvfHf0Xt+BnE4e/mDjDN9hDb7NUZZppDaSK9+wUyaM7Wk+kXyiSLzZ7i/enL3u9nkBQWL16L47sNx4m3SheRuD3JyuLbWFC5Ih7PPxHieepplZIFqV025T2MiQPOQeWKclKX9piDCMCCibctL2PCzhS/8ZQgymW58uoCzxuXMND4Oz+aP+WtHBxkTq7a0L9SvM/IWrSkYZl/wq4waslBvlgmXUTi9iSB4DY2vkc9fn+1mdvkbOXucr8quXsD10HCmPjp1qKq95lPw3UpRll7U99wlLl+o9JMKe1m2MIG/0EMM//EW+YFtDDuZ6BpEV1v4Y5qv4ae5tL1eI5ejM7wymN3VKqusu82nsiW+3pjx8kr2fZ7CCGB4DZWwN9E/Qp3uT1ftVRBl8f/Htya0Y/eTDI37embk7UMiQML3i5kSzLH1pEPLc9RzXCGQaYFzuMtDXuY5DcRgCfjh9E0biJV4uawy16ZEebZXq1DyKqjF6MZvGAPIaNW0f6r9fx92P0U5Dfn7/L6vrM237oP/uRWHrjA41M381MGF84J4Y4EgjuYq8Hijx+qRcXiBVKksehcu3Sacl4uX0jhR1sHfra2pb9xCfXVYeqqo0wzj+OgvTxN4yayXdfgHMWxYWSMtRclVRTPGZd5vOfy/eczXpFkbHZN+6/Wpzh28Vq8m9Lu7Qm/yrFUO679E+b9wHJ2OpU4jnH0YvZuSyryLgkEd7DkawsGd7oPP5OBZ5pW9O7aTD7zU2tvzlKc3/1HMNdvFJd0EZ5JeJ9zFE9Rbqu9JittDXjL9CtByv1G9i/NyVz+oz4zHEn0Zm0OS3Nu7xnX4xiePDRpE+2S7ZWQneIsNqIysB4i6d9m/eEIJqw+4pM6ibxFAkEe8XDdMhwe9YDblcqbh7RlZt+GWX5ODPko3WcGAPGY6WN5nwhcr3J+1/IikRRhut9XlFMX05z/cNHeTNdjw5FLVP1gKQfPXUtzLquzkH7YEsbW49k3Q+rJb7YQ/PEKr8sntfSOX4rhq5WHvb7uuw3HORkZk9HqiTxAAoEAoEzRfLSpXsr53tvUFq6Yqrbm8fiP6Bj/Oae0+2muVyjMGwmvUZxrzDCPpRApp27+uDVlH3gA8Txo2MzTxpV4k2TIatdsOpq9q57/2HWG4X/sp8f07Bvo3hOesRZKZv5tomItjFpykF7f3pp04+L2IoEgjyhbNF+Gypcq5H7twNKBLQEomt/MH6+5Tm63Q9/HRdwPZCfZrmvwnOVdKqtzTDWPI4C0/ff5iGO06TsOBTzHRL9JjDLP5AXjUq9+j9SLxrLqjXneDyb7SmZitD0xZWtMgjWbayPuBBII8oDg8kUz/C0yaWMcV5LWqd1dKIDg8kXZ8G4bNrzbhpfvr8JHD7racsKzjfY6vG99gRbG/az0e5d8OD68C3KDp40rWeH3Hk8Z13LUXoZvrF1ZYWvAENNcmhr2Z/hZ3roeb+XZ77cTfsX7BWapXYuzcD3e+w9eb6eDpv6XfGF2qPOD3h174r0z384TdzIJBHe4Y6O7sPCVZlm+z4nPurg9V75YfsoXy8+QB6rzXPNKmbr/Atv9fGR5lvKGCNb7v0kD9R/L/IcwyjyTWPx4xjKE9glj+czam0GWVzmh72GSeYLPEt4t23eevw9HZKgPPrWgESuom07ff0yyQDHkNy/HRFIF9VUHL6Q72JwUJpRS2O2aG9IyuK1YbXa2n/DdvhkSCO5wRoNyrg3IClctCp3NSaBb9h7Kp5ZelFTX+M3/Y8Cx9qBjwudsst9c9xBDPvpb3sKMlWl+43yyFiGzi7VSZyK12jWV3l/CsMRd03acdGy7ue9MFIFDllDro+XOsvND00+gZ7dr/vdX2kwstnTqm3RaASP/7wA1hy8nwWr3eE3q5y7bdy7dlofIfoFDllB16F88+c0WQsN8EwwkEIgMu7dUIfo0qcjUp11mDc+09jXvZkfZp5lu7cpmW00eif+E7boGrjo0Tuh7eNPyGkGGE8z2+59XwaAEUc5up/QMXuDYqe33nWfokipLqSdT16WdCqu1Y6bS5ZgE5m53DIB3m7jR4322Ho9k4b/hALw8Z4dzOuyb83e57G4KGbUqzbEUdUgM2kopfkkMOAk27wPBrztO8/KPO/lpW9ZmXMUm2IiIztg6jrNXYzlwNu3sr7zoQibWwHhDAoFwq1FgMZfHDQbFJ4/UpkpJ1yuXvVW5RAGqlCyQ4li5u/Iz2tqbXpYPuYTnXEpr7PUZbOlPE8NBhpjmui1nwsps8xi2+7/KVv8BtDLszlA9D7iYgurORg+zlHafvur1fXpM38qg+Y56Ltt/ng1HLnHiUgyLd7vePc6TmHgr8Yl7OFyLs3AjIePJ8ZI+vF1lWj0WcZ0Ri/en21o4GRlDjeHLUqQR90azMWvoMsH7YCwyTgKBcKt4Qb80x15qVTnL9/3rjZZsH9qOP1NlPoWMz4j51daaOdb2PGdaTjND2k3rq6pw5vmN4n7jHpbaG3NWl+A781g6Gba7veeRDO5LkFxMvJUjF6JdDjKvOngBWxa6VtqMXZep62p9tNz5QZq8O+i/89Epxig8SdoMaPLaY2m6v178IZRZm8M4fsn9GgWtNfd/sc75fv/ZjC/q23cmikvXffONOLU4i40THn6fnJKFWd0eSSAQXgsb05X3u9Twquz7D1R3e48a9xSmVKEAlzOTMvPf+afW3oTZ72am+XNqqTDAseZgnt8nLPd7jxDDYYZbnmWAZSBPJQzjgK7IVPN4t8Ggw7j1Lo9749D5aDqMW0+L/61Nc+6nbadY+O8Zj9evPniBa3GZ23XNk+i4tB/4j0/dzIs/hHq8zmKzEzhkCZPWHnUem5L4OvJ6PE99s4XjEY4PTE8fUql/736z/vG26k7dJm6kawZbBvvPRnE5JuNjSO8u2EObsesyNOvrdiaBQLiVNMBYJJ/Zc0EXHq5b1qtyBT1MU/VWHP48Z3kXf2Vlif8HBKlj/OQ3miaGgyyw3U+9uGn8YOsEwDUK8HTCB+zRlZlgnszzxiUo0u8rr6VOMNg0z+UK6Oz0/OxQTkXebE0EDlni9bXJt9602OxYbXZi0+kG2nzM8wrpWBfbcEZcj2fRv2doMGoV27ycybIrVbeYxZa5llFG+8i7TtjIw5M9j8e4svmYo4svK7OrYhNs2T47y1fTfyUQCLf6tXBMBV35VqsMX+tqotLYJ4LTHJvWJ3sGnGe+3ZOeCUMBWOw/jHrqKGMsPXjP2p8rpNy7+Tr5eSlhEDvt1Rhm/omvzFPxtFK5kTrI734f8ZppMYv9PqS5IfOpL7xx5GLmuqYenbKZX0JPU/+TldQYtoyWn6/1qj/e0wwpV6esNs32DM5eMaRqLng7Kys7Zimdvhyb4WuSZsllJdN38McrqDl8efoFcwEJBMKtRpWKETama6b2Pk49ZXX5m61c7oFwT5F8KdJlN6vi/T4IST55uBaBJQowd/S79E8YxExrJzokfM4020Nur7lAMXpahjLJ+jCPGjfxivHPFOcVdh4wbONL8xTm+Y0iHj9eSHibGwTwk99nPGlM2/WTXZIGiTPqxKUYPly4j8sxCVjtmnNRcV51bczYeCJD02V/3RHOz25SYB+PuO5VP763T8vs57DWmpczmbAQbn6RyUogyMisrJyW9Xa5EC6k7vLx1L30+6vNuJLYj/tESDne/W2P18/ZNKRtivQZK+wNWWH3Nnme4kvrEwSq87xt+oWt9hr8q6tRiBtMM4+judGxcnm+tTVfWx/nHMXZFl+DKeav+dz8LflIYHZil1N6AojHihFrLvxfbtSSgwSYjTzdJGVm2qs3Eqg7cqXX93lr/i52h0fhZzRw+NMHAPj7cARXbySkSeGttSONtrs9M8AxOH5Xftf/3Rw6f41ShQIoViDthAaA8auPsCxVCvOjF6M5FxVHy2olPf4eMfFWZxeUPVUkuBZn4XqclTIZTNmSXWSwWNxWAsxGjo3uQgE/IwCli7hvVRQOMFOxuGMaaUZSYTxWv2yGcyilpjHwvuVFVNFyTDBPoonhAMv936W5cT+fWXrSOG4S71n7O9NoR5OffpZ3WWcL5mPzbIaafvQ4xlBThbHCbzC7/fuzxf91XjL+iQnfDUBm9lvoh4tSzriavv4YMzeFZegeuxOT5yXY7M55/89+v5035u1iw5GU02qjYi20/+pv1hy6kOY+F67FsXz/eU5cimHnqbRTbjcfu0TnrzfwoIe1GEdS7dVw6Xo87b9aT58Z29PdW3rI7ze7/pIHgs/+OkjQiBU0G7PG4/WhYZdp9+W6FMfiLDbnl52sWLo3a/tzuCOBQPiM0aBYO7g1iwe4TkznjR+fb+x8XTjg5rdpg4KvnqybpfolKVGiJMbuMyilrjDPbxRl1GXeSniZb2wPcoG0ayksmHjJMoij9jK8aFrKj+bPKEjq6aKat0y/sNT/A+41nGGxrSkH7RV43zyX781fUBLvNrXJTxyvG39nsGkenQzbMZLxNQCZMXrpIcZnYa+DLhM2eDXQ/fKcnWmOPTxpk8d9KJIyqJ65erPvP85i4+U5OwhLnPLZsWbKrLfJF9y9tyBli/O1n3YyOdmsqKSNf5Luu+VYJFPWHeWbv4+n+/sAdJ+2hWMRN6eeXo5J4Nnvt1PvE+9aV/+EXeZ/y9KuHgcytY7EG7mvnSruKKUKBWR4jKFkIX8iouNpW70ULaqVYNmbLbk7cbrpvR86NpsP/bCDy2vNRpXhGSlKAeUb8WDCp4QYDrPaVs9lAEguHj/aJ3zBcNMc+pmWsVh9SM8ExyK4R40b6WVcTX3DUc7qYnSLH83lxAHrp+xr+dg0i5X+7zLC8iyL7M1xNxfkWeNy3jf9TIC6OZ10j70Sgy0v8Z+ukKHf0Rta6yylH8+M1C2Y2h8tz9SUzS3HI1m2/zyxFhuz+zXC3+T+O+6iXWcpUdCfD7s5EiQu2XuOJXvP8VqbqkDK/E8fLtrH1uOXqVQi5cLHcSsPM6jDvV7V7f7P1xLt5e909OJ1npi2BYD3Oruegu0L0iIQuc6WIW15sWUlxjzuyC9UvXRh7irgl2J7TXd9w0ldTBmRNKMlpsi9/Gxrl24QuEkx0voMQywvUFZdYq3/22z0f4Ox5m8oQRSDLf1pFj/RGQQA5tva0CXhM47qsnztN4V3TfNJOySqecW4mI/Ns4kmP0MsL1A77jsGW/pTRkXyp99QBhgXZnvrYOWBtN00t8K1OItz8VZGg0Dnr9enWCT39+EIAocs4WI6aSy+25h2v+lXftyBxWZPsQXo3sTurtSLyzLSWkoeBAKHLHE5pddm16w9dJH2X93cBa/Xt1t5fOpmr5+TFRIIRK5jMhoY2rVmpmYrBXoRCIoX8KNp5eKsHOSYFhtU1pHKYmhX7xbLpTbP1pa+lveI1IU5pUvxQsLbtEr4ml9trXH1bf+4LsOTCcOZZ23Nq6bFTDBPIn9iDqTiRDHPbxTvmeexzNaQ++PHMc/Wluvk51dbazrEf84yeyPeMf/KHPNnFMN1+otK6hyvGhex0G84f/u9yf9M09Ptjhq/+giBQ5bc8l3MgkasoM3YdVxwkb4iPYfOR6foIkqSet2CN/7ad579qXIaxWQiHUd6FuwMT7NocMbG4zyXapHd5mOR7Dh5a/bFlq4hcVuZ3qeBx64fYzpfbZa/2YoyRQMoFOCYjbLw1WbUuMfxjT0rM0G22GvRMmG81+XtGBhifZHTuhSDzb9QW53gD1tznjKt5W6u8LGlDzNtnUkdSK5QmIGW19lir8lHph/43e8jPrY+w0Z7He437KadYSchhsNUMzhW8u6wV+OwLs8jxk08YNzGJ9Y+LLC1Qrv4Dpj0IXj/F+sIIJ4njevoatxGYWL4x16dcdbH06zJyE6NR6/O1HWrDlyg6t0pZx/9vtPzCm6AKzEJabqmrtzwfkBXa02l95fydod7eb1dNQCivVgVPmzRPv7ae46fX2zCqP87wHcbT/B0k+zv6ssIldl0uzklJCREh4Z6XhYv8q435/3Lol0pB9RebV2FKeuO0bJaCeYkG3x2JSMreZPr2aiCM7NoRvU3/smzphWUVZEctJfnfcuL7NJV072uqWE/X5i/oZy6OSMnSudnl70q2+3V+dPe1LlVaGV1ls/M39HYcIiNtlp8ZX2CnTptH7cZK08Z1/K26VfuUtfZb69INPlpYjhIpC7ESEsf/vAwrpGkJFdpbDhIAia22GsRTf6M/aXcZsLGdCUq1uL13tMlCvoT+mF7539vXYPuYcmec14/KzOUUju01iEuz0kgEHeSyzEJ1E82O6Nvs0BGPFSLg+euUb5Y/nRTWngKBH5GQ4pvkJ8+WpvH6pXjXFQs5Yvlp9rQvzJc3/LF8nH6ciwG7JTiCucpRkYSCZix8phxA2VVBEft5fjL3giLm4a+ws4zxpW8YfqNYuo606wPMtb6hHNtQx11nM/N31DDcJpt9upMsD7q3AeilgpjrHkaNQyn2GirxWfWXuzXaTchamHYywDTIhqpQxiU47MlQhdmkvVRlttCOJ84DdeV4kTR0RhKsDpGG+MuCnODfTqQjy3PsE9nPdlhbmI0KNa+3ZpWX2R8YaIvAoF0DYk7SuoFSLXKOLoykrp/suLgJ52p8oFjr+Tjo7s4V09XzkI67vn9m1KmaD4Chyzx+CHpSunCAZy/Fsd8WxuvymsMzLZ1YoGtFUNNP/Gy6U+aGfbxmbUXFdUFPjbNIpLCvJowkKX2xiQPSPt1IF0TRtPbuIq3TAtY5Decuba2TLd1oxjXaGQ4RAvDPlobd3NWF2Oi7VFW2epTWMXwjulXPjbP5mPzbNbaghlkeZWrFHLeuwjXed20kKeNqwhQFuK1iTBdmqX2xnQzbmWx3zB+sHVgtLU3CbhfmBikjtHVuJWmhgPcoy5zVRdkga0Vc21tuUb6Y0dGbPQ0ruEF41ISMHFal2Kq9UEO63JcI2sp11Oz2TWLdyfvvtL0NK5hla0+ER72+na3wC6rJBCIO0ry6Y/fPhNC+xqlMn2v+hWKcjLyBpGJC4GMBsU3fRpQtmi+bNn1LauWvdmS6DgrLT/P2LfKGPLxgfUFNtlrM8w8h7l+nwKwwVabgZYBbscB7BiYY+vIH7ZmDDHNo6dxDc+Ybra+zulifGnpzjTbQzdbJRo2JdSmijpLN8NWXjMtYpX/YGZZO/G3PZhqKpy3zAu4h0gW2lsy09qZ/boiSUFonLU7b5l+pa9pBS0Ne/nQ2o8t9lop6lVNhfOeaS7tjf8Sr03s0lVZbatHbUMY75vn0se0kmGW51hrr4ur1pYZK+0NOxho+p0ahtPssVciFn/aGf+lXeI9l9ob85O1HaH6Ppf3UNiprk5TQV3gqi7Ev7qqx6AFMHaFYxvU4kQx0jyTrsbtlOIq422Pe7zOF6RrSNxx5mw9yeQ1R9n6QbsMXztpzRHn/6AHR3bGYrcTNGIFD9ctw/ge9Txe665byc9k4I/XmvOAi53Otn/QjlKFA2j35boUi5C8cWx0F4wGlelxDYACxNLVuBUDml9t92PD6PW1FdQFWht2Ea3zs9FehwiKpntNDXWS/5mnE2S4OX3ziL0sgy0veRwXaWXYzSemmVQ0XGSnvSob7bW5ogvRwHCEBwzbuEEA061d+d72ADHcHPSvrw7zhfkbqhjOsc1enfHWxwi134cRG3XUCZoaDvC0aRUlVRSn7CX5zNqLv+yNAEUJoqhjOE5rwy4eNW6ksIrlkL08c21tuYE/pbhKO+NOyqhICnOD/OrmlNXD9rJMtj7MCnsIsbie/VZFneEx4wb6GpcTQAL/s/Zguq0bnroGi+Qzs/ujjun+PbsiYwRCZEDgkCWULZqPTUPaAo6UzkaV/t7PSR/IX3QPYtbmMOcsnHFPBfNI3bJ8vvw/WlUrSc9vtzqvServjYm3pti/2J3vngnhhR9CCS5flD9ec6zY7jNjW5oUDqkplTaBWkjFu9h56gqZSfA5sF01JmR65bGmJFG0NOzhPMXYbq+ebg6mQe3vZcqqfbxoXEJb47/UNzhWAkfp/Cy2NeNr6+NEutnRzoyVV0ypOgAADLxJREFUHsY1DDQtpKSKIl6bMKAxK8fU0PW2Ovxoa89qe323gTAfcTxo3MLzxr+4zxDuPH7QXp4DuiLROj977ZU5qCtQVZ3lbdMvVDRc5LoOYLk9hAP2QA7rchy2l6OiusCLpqV0MO7ArhVr7XUZbe3FMZ1+6vaKxfPz92DvugJTk0AgxC0wee1RzEZF/1ZVnFMLIe3gXlLAeKNdtRSrU/vN+oc1h9Lud/DJw7UY9sd+l/dK8n97zjLg53/d1q1O2SI8XLcMo5YcdB4LG9OVNmPXZWonrmOju1Dro2XOnct8LWxM1xQtn3uIpISKYq+uhLeD6/mI437DHoINx1DAdvt97LFXSXdL1JQ0ZYgkn4rnmi5ABEVcPl9hp6H6jydNf9PR8A+FVcq1Dtd0PqZbu/GLrTUXPYwJpPZ8i0oMS1wRnVESCITIAWeuxnL4QjRt7ks5TnH04nUuxyTQqJLrFczPJy4sWn3oIn2aVKT6PYUYunAfXeqUZkpv9/s3fLBwb4r00OsHt6F4QT+a/28N43vUo1W1EljtmkvX4zkREUOzqiU4FnGdz5cdYvl+z6uKNw1pS5zFRmyCjQCzgaqlHIO93nRLrXqrFe2/urnrW/sad7PqoPermAv5m9j7cScuXoujUSbXGuQkA3aKEU01QzhV1RmidAHW2YMzPADdpU5pJvSohym9xTJu5FggUEp1BsYDRuA7rfWYVOf9gR+ABkAk8JTWOszTPSUQiLzickwChQNMbA+7TK9vtzHqkdppUkWnFjhkCXflN7Pl/XYEmL3v7/9j1xnemLfL7Xl3LZE/d5/l9bnuWyJJ1/b/IZQVBy4QWDw/6xK7Nrwd21j11v3OdNUTVx+hebUSPDbF+9QLDQPv4soNS4rUEZmxeEBz/m/POaav9y75XHbL7LTRJDkSCJRSRuAw0AEIB/4BemqtDyQr8yoQpLV+WSnVA3hUa/2Up/tKIBB50X/no7n37oLpJoX799QVyhbNR6nCGU/PMWvTCa7csPB8y0pciUngzJVYen3nyPSZkQ+hpNWySZKujY6z4Gcy4G9yBKjJa48SE29lyrpjHu/n6tmbj11i67FIZm85SVSs59W8oR+2p0RBf9p/9TdHL16nV+MKbjfWmdm3IW2qO1pwybvwVh64wNI3WgIQb7Xx5Ddb2Z2JNBae9Gteie83Of7eRjxYkxF/HiCf2ejcLvR2DQRNgRFa606J798H0Fp/lqzM8sQyW5RSJuA8UFJ7qJQEAiFuD1diEjhzNZbaZb3rg99yLBKtNQt2hrP/zDX+uxDNg8FlmNjT82ytfWei6Ja4N8H4HnWd+2U/8/12mlQuxqutHbORwi7FMG7VYcY+EYzZaCDOYqP6sGUp7pX8w7bvzO2s+y/C5Qdw8jGgjx6syRMh5clvNnLmaiy/hp7myYbl+W3HGR6pV4ZxKw/ToWZpWt1bgjojXK88fqZpRUY+XJu3f9lN86rFeax+Oa7eSKBofj9i4q3k9zNmOTNsTgWC7kBnrfULie/7AI211gOSldmXWCY88f2xxDJup0BIIBAib7gSk0DBABPmTPaJZ0ScxYbJoFL0v9vtGrvWme6Td+V6vJVZm06QYLXTt3kljlyI5red4Yx+tE62PseVnFpZ7Cp8pY463pRBKdUf6A9QoULOJmcSQtwad7lJNe4LrsZTDAaFIQPpPrxR0N/EgLbVnO8bVy5O48oZW1HuC74MQeFA+WTvywGpt9dxlknsGioCXE59I631dK11iNY6pGRJz/uNCiGEyBhfBoJ/gGpKqUpKKb//b+/sY+SqyjD+e9LS8in9QtOIoW2CaDVKG0EqSBrFagkhxmjYamIjmiiKisaQVhIS/Qs1MWhiBKKoMaWiINg0aCFAJdbQUkq3tNbaVWpohLZRKX5EQ+H1j/OOnU53t9vuzOyB+/ySm3vOO2fOee6emX3nnnvv+wIDwJqONmuA5Vn+IPDQaNcHjDHGdJ+eLQ1FxCFJ1wLrKLeP3h4ROyR9FdgcEWuA7wM/ljREORMY6JUeY4wxw9PToHMRcR9wX4ftxrbyf4AP9VKDMcaY0XGqSmOMaTh2BMYY03DsCIwxpuHYERhjTMN52UUflXQA+PMJvn0WMHrg9onHGsdP7fqgfo216wNrPF7OiYhhH8R62TmC8SBp80iPWNeCNY6f2vVB/Rpr1wfW2E28NGSMMQ3HjsAYYxpO0xzBbRMtYAxY4/ipXR/Ur7F2fWCNXaNR1wiMMcYcTdPOCIwxxnTQGEcg6X2SdkkakrSiz2PfLml/JuJp2WZIekDS7txPT7skfTt1bpO0sO09y7P9bknLhxvrBPW9TtLDknZK2iHp8zVplHSypE2SBlPfV9I+V9LGHOvOjHKLpKlZH8rX57T1tTLtuyS9txv6OrROkvSEpLU1apS0R9KTkrZK2py2KuY5+50m6S5Jv8/P46LK9J2Xf7vW9ryk62rSeEJExCt+o0Q//SMwD5gCDALz+zj+pcBCYHub7evAiiyvAL6W5cuBX1KS9lwEbEz7DOBPuZ+e5eld0jcbWJjlMyi5pufXojHHOT3LJwEbc9yfAgNpvwW4JsufBm7J8gBwZ5bn59xPBebmZ2JSl+f6i8AdwNqsV6UR2APM6rBVMc/Z94+AT2R5CjCtJn0dWidR0uueU6vGMR/LRA3c14OERcC6tvpKYGWfNczhSEewC5id5dnArizfCizrbAcsA25tsx/RrstafwG8p0aNwKnAFuDtlAd1JnfOMSX0+aIsT8526pz39nZd0nY28CDwLmBtjlmbxj0c7QiqmGfgVcBT5LXL2vQNo3cJsKFmjWPdmrI09Frg6bb63rRNJK+JiGcAcv/qtI+ktS/HkEsUCyi/uqvRmEsuW4H9wAOUX8rPRcShYcb6v458/SAws5f6kpuB64GXsj6zQo0B3C/pcZUUsFDPPM8DDgA/yOW170k6rSJ9nQwAq7Ncq8Yx0RRHMKbcyJUwktaeH4Ok04G7gesi4vnRmo6gpWcaI+LFiDif8qv7QuCNo4zVd32SrgD2R8Tj7eZRxpuoeb44IhYCS4HPSLp0lLb91jiZsoT63YhYAPyLsswyEhP5XZkCXAn87FhNR9BS1f+kpjiCseRP7jf7JM0GyP3+tI+ktafHIOkkihNYFRE/r1EjQEQ8B6ynrLdOU8l13TnWSLmwe6nvYuBKSXuAn1CWh26uTCMR8Zfc7wfuoTjVWuZ5L7A3IjZm/S6KY6hFXztLgS0RsS/rNWocM01xBGPJn9xv2vM1L6esy7fsH827DS4CDuap5jpgiaTpeUfCkrSNG0mipA3dGRHfrE2jpLMkTcvyKcBlwE7gYUqu6+H0DZcLew0wkHfszAXOBTaNVx9ARKyMiLMjYg7l8/VQRHykJo2STpN0RqtMmZ/tVDLPEfEs8LSk89L0buB3tejrYBmHl4VaWmrTOHYm6uJEvzfK1fs/UNaWb+jz2KuBZ4AXKL8EPk5ZD34Q2J37GdlWwHdS55PA29r6uRoYyu1jXdR3CeW0dBuwNbfLa9EIvAV4IvVtB25M+zzKP8khyin61LSfnPWhfH1eW183pO5dwNIezfdiDt81VI3G1DKY247W96CWec5+zwc251zfS7mjphp92fepwF+BM9tsVWk83s1PFhtjTMNpytKQMcaYEbAjMMaYhmNHYIwxDceOwBhjGo4dgTHGNBw7AtNYJP0z93MkfbjLfX+5o/7bbvZvTDexIzCmBAQ8LkcgadIxmhzhCCLiHcepyZi+YUdgDNwEvDPjy38hA9x9Q9JjGUP+kwCSFqvkbbiD8nAQku7NAG47WkHcJN0EnJL9rUpb6+xD2fd2lbwAV7X1vV6HY/Gvyie+jek5k4/dxJhXPCuAL0XEFQD5D/1gRFwgaSqwQdL92fZC4M0R8VTWr46Iv2Xoi8ck3R0RKyRdGyVIXicfoDw9+1ZgVr7nkXxtAfAmSsyZDZT4Rb/p/uEacyQ+IzDmaJZQ4sNspYTjnkmJ+QOwqc0JAHxO0iDwKCWI2LmMziXA6ijRVPcBvwYuaOt7b0S8RAnzMacrR2PMMfAZgTFHI+CzEXFEEDBJiymhkdvrl1ESx/xb0npKDKFj9T0S/20rv4i/n6ZP+IzAGPgHJUVni3XANRmaG0mvz2idnZwJ/D2dwBsoobFbvNB6fwePAFfldYizKGlMuxJd1JgTxb84jCmRLg/lEs8PgW9RlmW25AXbA8D7h3nfr4BPSdpGiRT6aNtrtwHbJG2JEo66xT2UlJWDlIiv10fEs+lIjJkQHH3UGGMajpeGjDGm4dgRGGNMw7EjMMaYhmNHYIwxDceOwBhjGo4dgTHGNBw7AmOMaTh2BMYY03D+B6iHXXq2dqKuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_list, label='Minibatch cost')\n",
    "plt.plot(np.convolve(cost_list, \n",
    "                     np.ones(200,)/200, mode='valid'), \n",
    "         label='Running average')\n",
    "\n",
    "plt.ylabel('Cross Entropy')\n",
    "plt.xlabel('Iteration')\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd3gU1frA8e9JI4EEQu+9QwghhF4sKAoovQpIERF7v2K7er0/r72LKAgKiPSqIhZEAQVCAiGEZiihhhAIJSE9e35/zAZDSMgm2dlNsu/nefbZ3dkpb4blndkzZ96jtNYIIYRwHW7ODkAIIYRjSeIXQggXI4lfCCFcjCR+IYRwMZL4hRDCxXg4OwBbVKtWTTdq1MjZYQghRKkSHh5+TmtdPff0UpH4GzVqRFhYmLPDEEKIUkUpdSyv6dLUI4QQLkYSvxBCuBhJ/EII4WJKRRt/XjIyMjh58iSpqanODqXM8Pb2pl69enh6ejo7FCGEiUpt4j958iR+fn40atQIpZSzwyn1tNacP3+ekydP0rhxY2eHI4QwUalt6klNTaVq1aqS9O1EKUXVqlXlF5QQLqDUJn5Akr6dyf4UwjWU6sQvhBAlVWJqBvP+iuHClXRnh3IdSfxFdP78eYKCgggKCqJWrVrUrVv36vv0dNv+oSdNmsTBgwdvOM+MGTNYuHChPUIWQjjI6YspjPh8K6+s3cuAjzez8/gFZ4d0DVUaBmIJCQnRue/c3b9/P61bt3ZSRNd69dVX8fX15ZlnnrlmutYarTVubqXn+FqS9qsQpVHUqUtM/noHKelZ/OvOlszafITYi6k8d2crpvRq7NAmVaVUuNY6JPf00pORSolDhw4REBDAtGnTCA4OJjY2lqlTpxISEkLbtm157bXXrs7bs2dPIiIiyMzMxN/fn+nTp9O+fXu6devG2bNnAXjppZf48MMPr84/ffp0OnfuTMuWLfnrr78AuHLlCsOGDaN9+/aMGTOGkJAQIiIiHP/HC+HifjsQx8gvtuLhplj+YHfGd2vE94/2ok/rGry+bj/3zw/nUnKGs8Msvd05c/rPd3vZd/qyXdfZpk5FXrm7bZGW3bdvH1999RWff/45AG+++SZVqlQhMzOTW265heHDh9OmTZtrlrl06RI33XQTb775Jk899RRz585l+vTp161ba01oaChr167ltddeY/369XzyySfUqlWLFStWsHv3boKDg4sUtxCi6BZsjeGVtXtpU6cicyd0okZFbwAq+Xjy+biOfPVnDG/8uJ/+H29mxthggur7Oy1WOeM3QdOmTenUqdPV94sWLSI4OJjg4GD279/Pvn37rlvGx8eHfv36AdCxY0diYmLyXPfQoUOvm2fLli2MHj0agPbt29O2bdEOWEKIwrNYNK//sI+X1+zllpY1WDK129Wkn00pxeSejVk2rTsAIz7/i7lbjuKspvYyccZf1DNzs1SoUOHq6+joaD766CNCQ0Px9/dn3LhxefaV9/Lyuvra3d2dzMzMPNddrly56+YpDddphCiLUtKzeHJJBOv3nmFCt4b8++62uLvl34YfVN+fHx7ryTPLInnt+32EHk3greGBVPJx7N3ycsZvssuXL+Pn50fFihWJjY3lp59+svs2evbsydKlSwHYs2dPnr8ohChtNh48y7gvt3M4PsnZoeTpXFIaY2Zv46d9Z3j5rja8OvDGST+bf3kvZt/bkZcGtObX/XHc/ckW9py85ICI/2Fq4ldKPa6UilJK7VVKPWGd9qpS6pRSKsL66G9mDM4WHBxMmzZtCAgI4P7776dHjx5238ajjz7KqVOnCAwM5L333iMgIIBKlSrZfTtCOILWms9+P8Tkr3ew5dA57p8XViIuiOZ06GwSQz77kwNnLjNzbEfu61m43jpKKab0asKSB7qRmWVh2My/mL81xmG/3k3rzqmUCgAWA52BdGA98CAwFkjSWr9r67pKendOZ8vMzCQzMxNvb2+io6Pp27cv0dHReHgUviVP9qvIZrFoDsYl0qqWn8O6ICanZ/Ls8kh+iIzl7vZ1GN6xHlPm7aBrk6p8NbETHu7Ob6TYduQ8DywIx9Nd8eWETsW+SHvhSjpPLY1g48F4BrSrzZvD2uHnbZ+mn/y6c5rZxt8a2Ka1TrYG8AcwxMTtuaykpCT69OlDZmYmWmu++OKLIiV9IXJ69bu9zN96jJtbVuf1Ie2o6+9j6vZOJCQzdUE4B85cZnq/VjzQuwlKKf47KIDpK/fwv3UH+PfdbQpekYlW7zrFs8t306BKeb6e1Jn6VcoXe52VK3gxZ0InZm0+wjs/HWTv6UvMGBtM2zrm/Wo38/AZBfRWSlVVSpUH+gP1rZ89opSKVErNVUpVzmthpdRUpVSYUiosPj7exDBLP39/f8LDw9m9ezeRkZH07dvX2SGJUm7+1hjmbz1G7xbV2X4kgb7v/8H8rTFYLOa0EPx16BwDP93CqQvJfDWxE9Nuanr1V8bozg2Y2L0Rc/88ypIdx03ZfkG01nyyIZonlkTQsWFlVj7Ywy5JP5ubm2LaTU1ZPLUrqRkWhnz2Fwu3HzOt6ce0xK+13g+8BfyC0cyzG8gEZgJNgSAgFngvn+Vnaa1DtNYh1atfN1awEMIkm/6O5z/f7eO21jX4amInfn6yN8ENK/PvNXsZNWurXS+2aq2Zu+Uo4+eGUs23HGsf6cnNLWtcN99LA1rTq3k1XlodRejRBLtt3xbpmRaeXR7Je7/8zdAOdZk/uQuVypvTC6dToyr88FhPujapyouronh8cQRJaXn38CsOUxvMtNZztNbBWuveQAIQrbWO01pnaa0twGyMawBCiBLg0NkkHv52J81r+PLh6A64uynqVynP/MmdeXdEe/6OS6LfR5uZsfEQGVmWYm0rNSPrarfGPq1qsOrhHjSqViHPeT3c3fh0TDD1K5dn2jfhnEhILta2bXUpJYNJX4eyPPwkj/dpznsj2+PlYe51hqq+5fh6YieevaMlP+yJZevh83bfhtm9empYnxsAQ4FFSqnaOWYZgtEkJIRwsgtX0rlv3g7Kebjx5YQQfMv9c51IKcXwjvX45ane3Na6Bu/8dJBBn/5J1KmidUM8cymVUbO2sWLnSZ64rTmfj+t4zfbyUqm8J7MnhJCRZeH++WFcMeFMOKdj568w4vO/CD2awLsj2vPk7S0cdpHbzU3x8C3N2PDUTdzepqb912/3NV5rhVJqH/Ad8LDW+gLwtlJqj1IqErgFeNLkGIQQBUjPtDDtm3BiL6XyxfgQ6lXOu/26hp83n43tyOfjOhKflMagGX/y5o8HSM3IsnlbYTEJ3PXJFg7FJTJrfEeeuK0Fbjb0fwdoWt2XGfcE83dcIk8uiTDtmsP6qFju+ngLcZfTmDepM8M71jNlOwXJ7xdQcZnd1NNLa91Ga91ea73BOm281rqd1jpQaz1Qax1rZgxmufnmm6+7GevDDz/koYceyncZX19fAE6fPs3w4cPzXW/urqu5ffjhhyQn//NTt3///ly8eNHW0IW4htaal1dHsf1oAm8PC6Rjwzz7W1zjzoBa/PrkTQwPrsfnfxym30eb2X6k4CaJb7cfZ8zsbfiWc2f1wz3o27ZWoePt3aI6Lw1ow8/74nj/l78LvfyNZGRZ+O/3+5j2zU6a1PDlh8d60r1ZNbtuoyRwfqfYUmrMmDEsXrz4mmmLFy9mzJgxBS5bp04dli9fXuRt507869atw9/feQWfROk2Z8tRloSd4JFbmjG4Q12bl6tU3pO3hgeycEoXMi0WRs3axour9pCYev3NVumZFl5ctYcXVu2he9NqrHm4J81r+hU55kk9GjEqpD6fbjzEmohTRV5PTqcvpjDqi63M2XKUid0bseyBbvn+8intJPEX0fDhw/n+++9JS0sDICYmhtOnTxMUFESfPn0IDg6mXbt2rFmz5rplY2JiCAgIACAlJYXRo0cTGBjIqFGjSElJuTrfgw8+eLWc8yuvvALAxx9/zOnTp7nlllu45ZZbAGjUqBHnzp0D4P333ycgIICAgICr5ZxjYmJo3bo1999/P23btqVv377XbEe4rg3743h93X76t6vFU7e3KNI6ejSrxk9P9GZKz8YsCj1O3w828duBuKufxyemMfbLbSzcfpxpNzVl7sROxe4Vo5Tiv4MD6NSoMv9aHsnuE8X7xfvH3/EM+HgzB88k8uk9HXh1YFvTL+I6U9m4y+fH6XBmj33XWasd9Hsz34+rVq1K586dWb9+PYMGDWLx4sWMGjUKHx8fVq1aRcWKFTl37hxdu3Zl4MCB+V4UmjlzJuXLlycyMpLIyMhrSiq//vrrVKlShaysLPr06UNkZCSPPfYY77//Phs3bqRatWt/goaHh/PVV1+xfft2tNZ06dKFm266icqVKxMdHc2iRYuYPXs2I0eOZMWKFYwbN84++0qUSgfOXOaxRbsIqFOJ90YE2dzOnpfyXh68dFcbBgTW5rkVkUz+OoyB1jtvn1sRyYXkdD4e04GB7evYLX4vDzdmjuvIoE//ZOqCMNY+0pOauapiFiTLovno17/5ZOMhWtb047OxwTSp7mu3GEuqsntIc4CczT3ZzTxaa1544QUCAwO57bbbOHXqFHFxcfmuY9OmTVcTcGBgIIGBgVc/W7p0KcHBwXTo0IG9e/cWWHxty5YtDBkyhAoVKuDr68vQoUPZvHkzAI0bNyYoKAi4cdlnUXIs2XGcfh9tZnHocTKL2XUyt3NJadz3dRi+3h7MvjcEHy93u6y3Q4PKfP9oL564rTk/RsVy79xQ3JRi+bTudk362ar5luPLCSEkpmYydX5YoS4yxyemce/c7Xz82yGGB9dj1UM9XCLpQ1k547/BmbmZBg8ezFNPPcXOnTtJSUkhODiYr7/+mvj4eMLDw/H09KRRo0Z5lmHOKa9fA0ePHuXdd99lx44dVK5cmYkTJxa4nhvd5ZddzhmMks7S1FNyZVk0/1u3nzlbjlK1ghfTV+5hzpajPHdnK/q0rlHsLoWpGVk8sCCc81fSWPpAN2pVKtxZckG8PNx44rYW9G9XmzURp5jcozFVfcsVvGARta5dkQ9GBfHAgnCeWxHJh6OCCtxHoUcTeOTbnVxKyeDtYYGM7FT/hvOXNXLGXwy+vr7cfPPNTJ48+epF3UuXLlGjRg08PT3ZuHEjx44du+E6evfufXUw9aioKCIjIwGjnHOFChWoVKkScXFx/Pjjj1eX8fPzIzExMc91rV69muTkZK5cucKqVavo1auXvf5cl5OZZSE67vr9bKbE1AymzNvBnC1HmdSjEdtf6MPn44LJtGimzA9j1KxtRBSjPVtrzfMr9xB+7ALvjwwisJ55nQJa1PTj2TtamZr0s93RthbP9G3BmojTzPzjcL7zWSyaz/84zJjZ26hQzoNVD/VwuaQPZeWM34nGjBnD0KFDrzb5jB07lrvvvpuQkBCCgoJo1arVDZd/8MEHmTRpEoGBgQQFBdG5s3Ejc/v27enQoQNt27alSZMm15Rznjp1Kv369aN27dps3Ljx6vTg4GAmTpx4dR1TpkyhQ4cO0qxTRK9+t5dvth1ncFAdXh3YFv/yXgUvVAwnEpK5b94OjsRf4fUhAYzt0hCAOwNq06d1TRaHHuejDdEMnvEnAwJr8687WtKwauH6eX/2+2FW7TrF07cbZ+RlycO3NONgXBLv/HSQ5jX8rrvx6VJyBk8vi+DX/Wfp364Wbw0LtFsVzNLGtLLM9iRlmR1H9qth44GzTPp6Bx0bVmb3iYtUruDFm0Pb0ae1/e+iBKPpYdo34WRZNDPHBufbdzwpLZNZm44we9MRMrIsjOvakEdvbWbTWfX6qFimfbOTQUF1bGoOKY1SM7IY+cVWDp9NYsVD3WlVqyIAkScv8tDCncRdTuWF/q2Z2L1Rmfz7c8uvLLM09QiRy/mkNJ5dHkmrWn4snNKF1Q/3oGoFL+6bF8bTS3dzKcW+g4IsCzvB2C+34e/jyaqHut/whiHfch48dXsL/nj2ZkZ2qs+Cbce46Z3fmbHxECnp+V/YjDp1iSeX7Caovj9vDQsss0nP29OdWeNDqFDOgynzwjiflMaCrTEMn7kVi0Wz5IFuTOpRuEFTyiI54xfXcPX9qrXmgQXh/H4wnjWP9KB1beOMMT3Twie/RfPZ74ep7luON4a145Y8qkgWRpZF8/b6A3yx6Qg9mlXls3s6Frp/+6Gziby1/iC/7IujZsVyPHV7C4Z3rH/NEIBxl1MZ9OmfuClY/UgPavjZ92JuSRRx4iIjv9iKbzkPEq6kc3PL6nwwMojKFcxtritpyuQZf2k4aJUmsj9hWfhJft4XxzN3tLia9MHoqfJ035aseqg7ft4eTPpqB88tj+RyHnep2iIpLZMHFoTzxaYjjOvagK8ndS7STU3Navgx+94Qlk3rRh1/H55bsYd+Hxk3UGmtSUnP4v75YVxOzWDOxE4ukfTBGNT8neGBJKdn8kzfFsyd0Mnlkv6NlNoz/qNHj+Ln50fVqlVd/mebPWitOX/+PImJiTRu3NjZ4TjFiYRk7vxwEwF1K/Ht/V3zHTg7NSOLjzZE88Ufh6lV0Zu3hgfSq7ntY0acvJDMlHlh/B2XyCt3t2VC90Z2iV9rzfqoM7y1/gAx55Pp0rgKvuU8+O3gWWaNDzGlymNJl2XRNg2AXlbld8ZfahN/RkYGJ0+eLLBvu7Cdt7c39erVw9PT9Xo6ZFk0o77YysEzifz4RC+barTsOn6Bp5ft5kj8Fe7p0oAX+rcusLRw+LEEHlgQTlqmhRn3BNO7hf0HGcrIsrAo9Dgf/RrN+SvpPN+vFQ/c1NTu2xElX5lL/ELY04yNh3jnp4O8P7I9Q4NtL8GbmpHF+7/8zezNR6jr78PbwwPp3jTvi7Mrd55k+oo91Pb3Zs6ETjSrYe5doompGUSdukzXJlXkV7GLKpNt/ELYQ9SpS3zwy98MaFebIYWoTglGL5IX+rdm2QPd8HR3457Z23llTRTJ6f8MEmKxXsR9aulughv6s/qhHqYnfQA/b0+6NZWmUHE9SfzCpaVmZPHEkgiqVPDi9SEBRU6SIY2qsO6xXkzq0Yh5W49x54ebCT2awJW0TKZ9E85nvx9mTOf6zJ/cRS4yCqeTO3eFS3vzxwMcOpvE/Mmdi31nro+XO6/c3ZY72tbiX8sjGTVrK3Uq+RB7KYWX72rD5B6ucdOQKPnkjF+4rM3R8Xz9VwwTuzey60XWrk2q8uPjvRjftSHpWRbmTOjEfT3lpiFRcph6cVcp9ThwP6CA2VrrD5VSVYAlQCMgBhhpHYs3X3JxV9jbxeR07vhwE77lPPjhsV54e9qnLLEQJYnDL+4qpQIwkn5noD1wl1KqOTAd2KC1bg5ssL4XwmG01ry4OorzSel8NLqDJH3hcsxs6mkNbNNaJ2utM4E/gCHAIGCedZ55wGATYxDiOmsiTvNDZCxP3t6CgLqVnB2OEA5nZuKPAnorpaoqpcoD/YH6QE2tdSyA9TnPgidKqalKqTClVFh8fLyJYQpXcupiCi+viaJjw8pMk5uahIsyLfFrrfcDbwG/AOuB3UDmDRe6dvlZWusQrXVI9er2v7tRuB6LRfP00ggsFs0HI4Nc+lZ+4dpM7dWjtZ6jtQ7WWvcGEoBoIE4pVRvA+nzWzBiEyDZny1G2HUnglbvb0qBqwSUZhCirTE38Sqka1ucGwFBgEbAWmGCdZQKwxswYhADYH3uZd346SN82NRkRYntJBiHKIrNv4FqhlKoKZAAPa60vKKXeBJYqpe4DjgMjTI5BuLi0zCyeXBJBRR9P3hjaTvrTC5dnauLXWl830rfW+jzQx8ztCpHTez//zYEzicydGOKQgb+FKOnkzl1Rpm09fJ7Zm49wT5cG3NrK9erRC5EXSfyizLqcmsEzy3bTqGoFXhrgusNJCpGbFGkTZdZr3+3jzOVUlk/rRnkv+aoLkU3O+EWZtGF/HMvDT/LgTU3p0KCys8MRokSRxC/KnIvJ6UxfuYdWtfx4tE8zZ4cjRIkjv39FmfPq2r1cuJLOVxM7Uc5DCrAJkZuc8YsyZX3UGVZHnOaRW5tJATYh8iGJX5QZCVfSeWn1HtrUrsjDt0gTjxD5kcQvTPPXoXO8vPragcfN9O81UVxKyeC9ke3xdJevthD5kTZ+YYqth88z6esdpGVaiDl/hS8nhJja3v5DZCzfR8byTN8WtK5d0bTtCFEWyGmRsLvwYxe4b94OGlQpz0sDWrM5+hxPLdlNlsWcYT7PJaXx8poo2tWtJDX2hbCBnPELu9pz8hIT54ZSs6I3C6d0oUZFb7SG19ftp6KPB/8bYt8iaVprXl4dRVJqJu+NbI+HNPEIUSBJ/MJuDpy5zPi526no43k16QPc37sJF1PSmbHxMP7lvXjuzlZ22+ba3af5MeoMz93ZihY1/ey2XiHKMkn8wi4Oxycx7svteHu4s+j+rtTx97nm82f6tuRicgYzfz9MJR9PuzTJnL2cyr/X7KVDA3+m9m5S7PUJ4Sok8YtiO34+mbGztwOw8P4ueY5upZTitUEBXE7N5M0fD1DJx5MxnRsUeZtaa15YtYfUjCzeHdFehlEUohAk8YtiOX0xhXu+3EZqZhaLp3alaXXffOd1d1O8N6I9iakZvLBqDxW9PRkQWLtI21216xS/7j/LSwNa33CbQojryZUwUWRnL6dyz+xtXErOYMHkLrSqVXA3Si8PN2aO7UjHBpV5YskuNv0dX+jtnrmUyqtr9xLSsDKTejQuSuhCuDRJ/KJIzielMfbL7ZxNTOPryZ1oV8/28gg+Xu7MmdiJZjX8eGBBOOHHLti8rNaa51dGkp5l4R1p4hGiSMwebP1JpdRepVSUUmqRUspbKfW1UuqoUirC+ggyMwZhf5eSMxg/J5TjCcnMmdCJjg2rFHodlXw8mT+5MzUrlmPSV6EcOHPZpuWWhZ9k48F4nruzFY2rVSj0doUQJiZ+pVRd4DEgRGsdALgDo60fP6u1DrI+IsyKQdhfUlomE74K5dDZJGbdG0K3plWLvK7qfuVYcF8Xynt5GAeS88k3nP/0xRT++90+OjeuwoRujYq8XSFcndlNPR6Aj1LKAygPnDZ5e8JEKelZTP5qB3tOXeLTezpwU4vqxV5n/SrlWXBfZzKyLIyds424y6l5zqe15rkVkWRpzbvD2+MmTTxCFJlpiV9rfQp4FzgOxAKXtNY/Wz9+XSkVqZT6QClVLq/llVJTlVJhSqmw+PjCXwAU9pWakcXUBWGEHUvgw1FB9G1by27rbl7Tj68ndSYhKZ1754RyMTn9unkWhZ5gc/Q5nu/fOs/uokII25nZ1FMZGAQ0BuoAFZRS44DngVZAJ6AK8Fxey2utZ2mtQ7TWIdWrF//MUhRdeqaFhxfuZHP0Od4e3p6729ex+zaC6vsz+94Qjp67wqSvd3Al7Z+KnicSknn9h330aFaVscXo+y+EMJjZ1HMbcFRrHa+1zgBWAt211rHakAZ8BXQ2MQZRTJlZFp5YsosNB87yf4MDGN6xnmnb6t6sGp/c04HdJy4y7Ztw0jKzsFiMJh6At4YFShOPEHZgZuI/DnRVSpVXRlWuPsB+pVRtAOu0wUCUiTGIYrBYNM8uj2TdnjO8NKA147o2NH2bd7StxVvDAtkcfY4nFkcwf2sMfx0+z0t3taFeZWniEcIeTLtzV2u9XSm1HNgJZAK7gFnAj0qp6oACIoBpZsUgimfun0dZtesUz/RtwZRejquFMyKkPpdTM/nv9/v4MeoMvZpXY3Sn+g7bvhBlnaklG7TWrwCv5Jp8q5nbFPZhsWjmbz1Gl8ZVeOTW5g7f/n09G3MlLZOlYSd4a1igXUs5C+HqpFaPyNNfh89zPCGZp/u2cFoMj/VpzqO3NpOkL4SdSckGkadFO47jX96TO+zYbbMoJOkLYX+S+MV1ziel8fPeMwztUA9vT/PGyRVCOIckfnGdlTtPkZGlGdNZLqgKURZJ4hfX0FqzaMdxOjasTHMZylCIMkkSv7hG6NEEjsRfKdboWEKIkk0Sv7jG4h0n8PP2YEC7oo2MJYQo+STxi6suJqfzw55YBgfVxcdLLuoKUVZJ4hdXrdp1ivRMizTzCFHGFZj4lVKPWCttijJMa83i0BO0r1eJNnUKHjtXCFF62XLGXwvYoZRaqpS6U8kdNWXSzuMXORiXyGg52xeizCsw8WutXwKaA3OAiUC0Uup/SqmmJscmHGhx6HEqeLmbUmtfCFGy2NTGr7XWwBnrIxOoDCxXSr1tYmzCQRJTM/g+MpaBQXXwLSflm4Qo6wr8X66UegyYAJwDvsQYKD1DKeUGRAP/MjdEYbY1EadJychidCdp5hHCFdhyelcNGKq1PpZzotbaopS6y5ywhCMtCj1O69oVCaxXydmhCCEcwJamnnVAQvYbpZSfUqoLgNZ6v1mBCcfYc/ISe09f5p7O9aUSphAuwpbEPxNIyvH+inWaKAMW7TiOt6cbgzrUdXYoQggHsSXxK+vFXcBo4kEGcCkTrqRlsmbXKQa0q0NFb09nhyOEcBBbEv8RpdRjSilP6+Nx4IgtK1dKPamU2quUilJKLVJKeSulGiultiulopVSS5RSXsX7E0RRfR95mivpWdzTRcovC+FKbEn804DuwCngJNAFmFrQQkqpusBjQIjWOgBwB0YDbwEfaK2bAxeA+4oWuiiuRaEnaF7Dl+AGcmO2EK7Elhu4zmqtR2uta2ita2qt79Fan7Vx/R6Aj1LKAygPxGIMtr7c+vk8YHBRAhfFsz/2MhEnLjK6cwO5qCuEi7GlH783xll5W8A7e7rWevKNltNan1JKvQscB1KAn4Fw4KLWOtM620kgz6uKSqmpWH9ZNGgg/cvtbXHocbw83BgqF3WFcDm2NPUswKjXcwfwB1APSCxoIWtht0FAY6AOUAHol8esOo9paK1naa1DtNYh1atXtyFMYavUjCxW7TpFv4BaVK4gl1iEcDW2JP5mWuuXgSta63nAAKCdDcvdBhzVWsdrrTOAlRjXCvytTT9gHEROFyFuUQzr9sRyOTVT7tQVwkXZkvgzrM8XlVIBQCWgkQ3LHQe6KqXKWyt69gH2ARuB4dZ5JgBrChWxKLZFocdpXP+1JlAAACAASURBVK0CXZtUcXYoQggnsCXxz7I227wErMVI3m8VtJDWejvGRdydwB7rtmYBzwFPKaUOAVUxqn4KBzl0NpEdMRcY3Unu1BXCVd3w4q61ENtlrfUFYBPQpDAr11q/ArySa/IRoHNh1iPsZ3HoCTzdFcM61nN2KEIIJ7nhGb/1Lt1HHBSLMFlaZhYrdp7k9jY1qeZbztnhCCGcxJamnl+UUs8opeorpapkP0yPTNjdT3vjuJCcIWPqCuHibKm5k91f/+Ec0zSFbPYRzrc49Dj1q/jQo2k1Z4cihHCiAhO/1rqxIwIR5oo5d4W/Dp/nmb4tcHOTi7pCuDJb7ty9N6/pWuv59g9HmGXxjhO4uylGhEhBNiFcnS1NPZ1yvPbG6I+/E5DEX0qkZ1pYHn6CW1vVoGZF74IXEALg0ik4uA6C7wUP6QxQaFrDxWNQsS64l6yy57Y09Tya871SqhJGGQdRSmzYH8e5pHTGdJazfWEDrSHiW1g/HdIuQ/QvMGqBJP/CyMqEdc9A+FdQviq0HQLtRkD9LlAC7p8pyoAqyUBzewcizLNoxwlqV/LmphY1nB2KKOkux8L3T8Df66FBd2h2K/z2f7B0AoycDx5S26lA6cmw4j7j11LIZEi5CLu+gR1fgn8DCBgOgSOhRmunhWhLG/93/FNIzQ1oAyw1MyhhPycSktkcHc9jtzbHXS7qivxoDXuWwbpnITMV7ngDukwDNzfwqQw/PA3LJsKIryX530hyAnw7Ck7ugH7vQBfr0CVpiXDgB4hcCn9+CFveh5oBxq+AgGHg79hf47ac8b+b43UmcExrfdKkeISdLQs7AcDITtLMI/KRdBa+fxIOfA/1OsPgmVCt2T+fd5piHBjWPQPLJxnJv4S1WZcIF47BN8Pg4nHj11Gbgf98Vs4P2o82HklnYe8q4yDw6yvGo2EPaDcc2gyG8ubfJqVyDKeb9wxKNQZitdap1vc+QE2tdYzp0VmFhITosLAwR22uzMjMstDjrd9oXbsiX0+SKhkiD1ErjbP59Ctw64vQ7RFwc8973m2fw/rnoPVAGD7XMck/IxU8S0GHhNhIWDjc+LU0Zgk07GbbcglHYM9y4yBwPhrcPKHZbRA4Alr0A6/yxQpLKRWutQ7JPd2WM/5lGOWUs2VZp3XKe3bhDFprLiRnEHsphbjLqcReSiXq1GXiLqfx2iC5U1fkcuU8rHvaOPOsE2yc5ddodeNluk4DbYGfnocVU2DYHHAvymVCG6RchPXPw+5vjSaR1ncbjxptSsTF0Wsc+R0WjwPvSjB5bcH7MacqTeCmf0HvZyF2t9HcFrUC/v4RvHyh1V3Q8wm7Xw+w5V/NQ2udnv1Ga50uA6Q7VpZFE5+Ydk1SP3MplTO5XqdnWq5Zzk1B+/r+3NpKLuqKHPZ/b1zATbkIt74MPZ6wPYF3ewh0Fvz8kvHLYMgs+yf/6F9g7aNGk0jwvXDuEPz+Jvz+hpEoW99t/OqoE2xcg3CmyGWw+kGo1hzGLodKRRzRTimoE2Q8bn8NYrYYB4F9a6FzgUOcF5ot/2LxSqmBWuu1RnxqEHDO7pGI6yzYdoyZGw8Rl5hGluXaJjkvdzdqVfKmVkVvgur7X31du5I3NSsZz9V9y+Hh7uT/GKLkSE6AH5+DPUuhVjsYvxpqBRR+Pd0fNc78f/k3KDcY8kX+zUOFkXoZfnoBdi2A6q1g9EKo29H4LOmscXF0/3ewdQb8+RH41YHWdxkHggbdzfv1kRet4a9P4JeXoVEvGPUN+PjbZ91u7tDkJuPR/11TutHa0sbfFFiIMXwiGOPk3qu1PmT3aPLhim38C7bG8PKavXRuXIUujatcTezZz1UqeEk9fWG7v3+CtY9B8jno9Qz0fqb4bfSb34cN/4HAUUZTUXGS/+GNsOYRSDwN3R+Dm5/Pv20/5QL8/TPsXwuHNkBmCvhUgZb9jYNAk5vNvS5gscDPL8K2z4z++UO+KLH3OBS5jV9rfRhjJC1fjANFgePtiuJZsuM4L6/Zy22tazJzXDCectbuerSGiIVG/29PH/D2N84or3mufP20cn7XtoGnXISfXoSIb4z28XuWGM0J9tDrKePM/7f/Gmf+g2YUPvmnJRm/HMLmQNVmMPknqF9ARwSfytB+lPFIT4bDG4xfAvu/M/5OL19o3tc4CDS/3dgn9pKRCqunGddGuj4EfV93fnNTEdjSj/9/wNta64vW95WBp7XWL5kdnCtaufMk01fu4aYW1ZkxtoMkfVeUcAS+exyObjKSdVaG0UUw5aJxtquz8l9WuV97MLh0Eq7EQ6+n4abn7H9m2vsZI/lvfN1I/gM/tT0RxmyB1Q8Zf1vXh6HPy8ZBrjC8yv9z4TczHWI2GQeAAz/A3pXg7gX1OhnNMY17Qd2Qov8aSLkIi8fCsS3Q9/+MJq9Sypamnl1a6w65pu3UWgebGlkOrtLU833kaR5btIuuTaoyd2InvD3t0G4qSo+sTNj6qXER090Lbv8PBE+8NpFqDelJRhJKtR4Irr62vs9+nXrRSMY3vwD1Opob+8Y34I83ocN4uPvjGyf/9GSjiWj751C5MQz+DBp2z3/+orBkwYntxt2zRzfDmUjjAOXhnetA0NG2g+Hl00Yf/XPRMORzo899KVCc7pzuSqlyWus064p8gAL3lFKqJbAkx6QmwL8Bf+B+IN46/QWt9Tob4ijTftp7hscXRxDSsApfTgiRpO9qTkcYPVnORELLATDgXahY5/r5lDKaLsr5ASXoprybpxuJddPbxsHmrg/zTv7Htxm9YBKOGL1VbnsVvCrYPx43d+Ngkn1ASbkIx7cavzKObjIOrr//Dzx8jKalxr2gUW+o0+H6O5PP7odvhkPqJRi33LiGUMrZkvi/ATYopb6yvp8EzCtoIa31QSAIQCnlDpwCVlmX/0Br/e4NFncpvx2I45FvdxJYrxJzJ3WivJcDeycI50pPNpLQ1hlQoZpxx2frgSWvr3pBlIJbXjCaoTa/ZyTeAe//83dkpBg1f7bOMMoTTPgOGvd2XHw+/tCyn/EAo4fT8a3Gr4GYzUZsAJ7loUFXaNTTOBBkpsCSccYvhUnroHag42I2kS0Xd99WSkUCtwEKWA80LOR2+gCHtdbHpCfKtTZHxzPtm520qmXcXetbTpK+yzjyu9GWfyEGgicYTTs+lZ0dVdEpZdwXoC2w5QPjzL//u3Aq3DjLP/c3dJwEff9r3wuuRVG+CrQaYDzAOBDEbDEOAjFbYMNr/8xbtTmMWwGVC5v2Si5bs8wZwAKMBI4CKwq5ndHAohzvH7EO8BKGcaH4Qu4FlFJTgakADRqUzTtPtx05z/3zw2hSrQLzJ3emkk8Jq39y9oBRVrac3417k3iWL31nqM6UnGDcABWxEKo0hQnfG00NZYFS0OcVo439r48hbh+c2AZ+tWHcSmjWx9kR5q18FaO2TnZ9naR44yJu9kHZAfVzHCnfi7tKqRYYCXsMcB6jvf4ZrXWhDnvWu3xPA2211nFKqZoYN4Bp4L9Aba315Butoyxe3A0/lsD4OaHU9fdh0dSuVPMtgf2AF46EQ78Yr7Ul//ncPK8/GPhUNl7XbGP8x5EDg3Fhdu9K4yaq5ATo8bhxu35he7KUBlobB7etn0KHcXDH/4ySBsKhinJx9wCwGbg7+2YtpdSTRdh2P2Cn1joOIPvZur7ZwPdFWGeptvvERSbO3UHNit4snNKlZCb984ch+ie4abrRDTA9Me9eI3lNSzoL8QeN92mXjOeeTzj7L3KuSyeNYmh/rzcuII5fZdw9W1YpBXe8bhzcfKVkSElzo8Q/DOOMf6NSaj2wGKONv7DGkKOZRylVW2sda307BIgqwjpLrahTlxg/Zzv+FTz59v4u1CipQyGGzjLO5EMmGb0zvCsZj8K0c2oNyyfDr68a/dFb9DUt3DxFLjP6l7t7Gjf1lPMFLz+jF0k5X+s0vxyf5fHey9fo9+3hU7SSABaLcXPSr68av5r6vm7UuXdkeQFnkqRfIuX77dNarwJWKaUqAIOBJ4GaSqmZwCqt9c8FrVwpVR64HXggx+S3lVJBGE09Mbk+K9MOnklk/Jzt+Jbz4NspXaldqYT+xE+9DLsWGrej+9Uq+nqUMu7mPH/IGJHo/t+MYlaOcOhX4w7LGm2Mwl7pScZdosnHjEExst9npdm+TuVuNMt4lDMOBJ7eRm+P7Idn7tc+cHoXnAqDprfCXR9A5Uam/clC2MqWXj1XMGr1LFRKVQFGANOBAhO/1joZqJpr2viihVq6HTqbxNgvt+Hl4ca393elfpXi1dk21e5FRtNOl2nFX5dXeRj9Lcy6GRaNhikb7FfMKj+nI2DJvVC9NUz8Abwr5j9vVsa1B4L0pFzvrxg11rMfGalGF7/MNKOLYmaa8T4j1Wi3zz2fZwWjlkvgKLnOIUqMAu/cLQlK+8XdmHNXGPnFViwaFk/tSrMavs4OKX8WC3waYvRimPKr/dZ77C+Ydzc0ucWoF2OPao55uXAMvrzNOCu/7xeoWNuc7QhRCuR3cVcKwZjsREIy98zeRkaWhYVTupTspA9GwauEw/Y528+pYXfo/47RSyhnH2l7Sk4wbqvPSjNqo0vSFyJPLnKFyTnOXErlni+3kZSWybf3d6VlLSfftGKL7Z+Dby3j7lF7C5kMZ/YYg03XamffeicZKUZT0sXjcO/qwo2CJISLkTN+E72+bj/nk9JZcF8XAuqWgj7M8X8bF0U73Xd9vRJ7ufMtY9CMNQ8bFz7twZJlDAV4IhSGzrJ/wS8hyhhJ/CY5m5jK+qhYRndqQPv6Jl/MtJfQWUZVyI4TzduGh5dRj6ZCdaPEbdLZ4q1Pa1g/HQ58D3e+AW0H2ydOIcowSfwmWRJ6gowszbiupaTcROoliPgWAoaZ3/fat7oxrF5yAiwZb9RRL6o/PzIOWN0ega4P2i9GIcowSfwmyMyy8G3ocXo1r0aT6iX8Ym62XQsh4wp0cdBtFbXbw+AZRh2Xdc8YZ+6FFbkMfn3FOFjd/l/7xyhEGSWJ3wS/7j9L7KVUxnctJdX8LFkQ+gXU72qUE3CUgGHQ8ynYOQ92fFm4ZY/8blR8bNTLOt6rfJWFsJX8bzHBgm0x1PX3oU/rms4OxTbRvxhVCB11tp/TrS9B8zuMdvqjm21b5kyU0URUrTmM+qbEDnQtREklid/ODp1N4s9D57mnSwPc3UrJnZrbZ4JfHWPcUkdzc4dhs42yCssmGDdg3cilk7BwhFFDZ+wy8+8CFqIMksRvZ99sO4aXuxujOpWgYfFu5OwBo9mk031GMTNn8K4EoxcZY84uHmuUSchLykVjCLz0JGMIvEr1HBunEGWEJH47upKWyYrwk/RvV6tkllrOS+gX4F7O3C6ctqjWDIbPhbN7YfVD11/szUwzDgoJh40eQTXbOidOIcoASfx2tDriFIlpmYzvVkou6qZcgN2LIXCEMd6rszW/zRh8e99q2JxjSGaLBVY9YIyINHimY8dqFaIMksRvJ1prFmw9RpvaFQluUErGTd31DWQkQ+cSVBm7+2PQbqQx+PWBdca0X16GvauMLpv2LPMghIuSWj12EnbsAgfOJPLm0HaUigHlLVnGjU8Ne0DtQGdH8w+lYODHxsDcK6dC8L2wbYZRNK77o86OTogyQc747WT+1mP4eXswMKiOs0OxzcEfjYJmzujCWRBPH6OGv6ePkfRbDzTGbC0NB1QhSgE547eD7Lo847s2orxXKdml2z+HSvWh5QBnR5K3SnVh7FKIWgm3vGhe/X4hXFApyVIlW6mryxO3F2I2w23/Kdljv9bp4Ng7iYVwEaY19SilWiqlInI8LiulnlBKVVFK/aKUirY+l5IroXkrlXV5tn9hjAcbfK+zIxFCOIFpiV9rfVBrHaS1DgI6AsnAKozxejdorZsDG6zvS61SV5cnOQEil0LgSGN4RSGEy3HUxd0+wGGt9TFgEDDPOn0eUKoLqC/YFkOdSt7c2srkUsb2snOeMQh4SbyoK4RwCEcl/tHAIuvrmlrrWADrc54ZUyk1VSkVppQKi4+Pd1CYhZNdl2ds14Z4uJeCDlJZmRD6pVHRUu58FcJlmZ6tlFJewEBgWWGW01rP0lqHaK1Dqlevbk5wxfTNtmN4uqvSU5fn4A9w+aT9B1IXQpQqjjhN7Qfs1FrHWd/HKaVqA1ifizn2nnP8U5endumpy7P9C/BvAC37OTsSIYQTOSLxj+GfZh6AtcAE6+sJwBoHxGB3ayJOk5iWyb2lpS5PbCQc+xM6T5U+8UK4OFMTv1KqPHA7sDLH5DeB25VS0dbP3jQzBjNorZm/NaZ01eUJ/QI8y0OHcc6ORAjhZKbevaO1Tgaq5pp2HqOXT6mVXZfnjdJSl+fKOWN82g5jwaeUHKiEEKYpBV1RSp4F1ro8g0pLXZ7wryErrWRV4RRCOI0k/kKKT0zjx6hYRnSsXzrq8mRlwI450ORmqNHK2dEIIUoASfyFtDj0eOmqy7P/O0g8DV0edHYkQogSQhJ/IZTaujyVG0Pzvs6ORAhRQpSCtoqSI7suz6sDHXTXa1YGXIk3ni2Z1z6ycr7P/jzr2nmTzsKJbXDHG+Amx3ghhEESfyFk1+XpY++6PFmZcOEonN0P8Qf+eT4XbST14vD2N3rzCCGElSR+G2XX5Xn2jpZFr8tjyYILMf8k95wJPivtn/n8G0KN1kbzTOWG4F4O3DyMG6/cPa2vre/dcrx398jxmafxefmq4F3JLvtACFE2SOK3UXZdnpEhhajLk3gGdi/6J8mf+xsyU//5vFJ9qN4Kmt5qJPrqraB6S/CqYP8/QAghrCTx2yBnXZ7qfjbW5Um/AguGwNl9ULGukdQb9zaea7Q2Enw5P3MDF0KIPEjit0Gh6/JoDd8/aZzlj1sBzW4zN0AhhCgE6epRgOy6PK0LU5cnbC5ELoGbn5ekL4QocSTxFyDcWpfn3m4NbavLcyoc1k+HZrdD72fND1AIIQpJEn8B5hemLk9yAiydAL41Yegs6TsvhCiRJDPdQHZdnuEd6xVcl8digZX3Q1IcjJwnA5kLIUosubh7Awu2xpCRpRnf1YaLupvegUO/woD3oW5H02MTQoiikjP+fByJT+LzTUcYEFi74Lo8hzbA729A4GgImeyYAIUQoogk8efBYtE8v3IP3h5uvHJ3mxvPfPEErJhi9M2/6wMoDQOzCCFcmiT+PCwNO8H2owm80L81Nfy8858xMw2WTTAKo41cAF7lHRekEEIUkdlj7vorpZYrpQ4opfYrpboppV5VSp1SSkVYH/3NjKGwziam8r91++nSuAqjOhVQnuGnF43um4NnQLVmjglQCCGKyeyLux8B67XWw5VSXkB54A7gA631uyZvu0j+890+UjMt/K+g8XQjl8GO2dDtEWgzyHEBCiFEMZmW+JVSFYHewEQArXU6kF6SByffsD+OHyJjefr2FjS90QXds/vhu8egQTe47VVHhSeEEHZhZlNPEyAe+EoptUsp9aVSKrvs5CNKqUil1FylVJ51EJRSU5VSYUqpsPj4eBPDNCSlZfLS6iha1vTjgZua5j9jWiIsGQ9evjD8K6NMshBClCJmJn4PIBiYqbXuAFwBpgMzgaZAEBALvJfXwlrrWVrrEK11SPXq1U0M0/DuTwc5czmVN4a1w8sjn92iNax5BBIOw/C5ULG26XEJIYS9mZn4TwIntdbbre+XA8Fa6zitdZbW2gLMBjqbGINNdh6/wLytMdzbteGNC7Ft/xz2rYY+/4bGvRwWnxBC2JNpiV9rfQY4oZRqaZ3UB9inlMp5mjwEiDIrBltkZFl4fsUealX05tk7W+U/4/Ft8PNL0HIA9HjCcQEKIYSdmd2r51FgobVHzxFgEvCxUioI0EAM8IDJMdzQrE1HOBiXyOx7Q/Atl8/uSIqHZRONEbMGfyY3aQkhSjVTE7/WOgIIyTV5vJnbLIwj8Ul8tCGa/u1qcXubmnnPZMmCFZMh5QLc9wv4+Ds2SCGEsDOXLdKmteaFVXso5+HGq3e3zX/Gja/D0U0waAbUDnRcgEIIYRKXLdmwNOwE245YyzJUzKcsw4F1sPk96DAeOoxzbIBCCGESl0z88YlpvP7Dfjo3rsKokHzKMpwMg+WToXYQ9H/HsQEKIYSJXDLx/+e7vaRmWPjfkHa4ueVxofZcNCwcAX41Yewy8PRxfJBCCGESl0v8vx2I4/vIWB65tRnNauRRluFyLCwYCm7uMG4l+NZwfJBCCGEil7q4m5SWyUuromhR05dpeZVlSL0EC4dD8nmY9ANUvUHpBiGEKKVcKvG/+9NBYi+nsvye7teXZchIhcVjIf4A3LMU6nRwTpBCCGEyl0n8u6xlGcZ3bUjHhrnKMliyYNVUiNkMQ2dDsz5OiVEIIRzBJdr4M7IsPL9yDzX9vHn2jpbXfqg1rJ8O+9ZA3/+DwJHOCVIIIRzEJc74Z206woEzicwa3xE/71xllLe8D6GzjAFVuj/qnACFEMKByvwZ/9FzV/hoQzT9AmrRt22taz/ctRA2vAbtRsDt/3VOgEII4WBlOvFrrXlhpVGW4T8Dc5Vl+PtnWPsoNLkZBn0GbmV6VwghxFVlOtstCzvJ1iPneb5frrIMJ8Ng2QSoFQCjvgEPL+cFKYQQDlamE//FlHR6NKvK6E45yjJk35XrWwPGLodyfs4LUAghnKBMX9yd2rspU3o2+acsg9yVK4QQZTvxA/8k/dRLxpm+3JUrhHBxZT7xA5CZZr0rd7/clSuEcHllP/FbLLBS7soVQohspl7cVUr5K6WWK6UOKKX2K6W6KaWqKKV+UUpFW58rF7ymIrp6V+5quStXCCGszO7V8xGwXmvdCmgP7AemAxu01s2BDdb35vjzQwj9Qu7KFUKIHExL/EqpikBvYA6A1jpda30RGATMs842DxhsVgz4N4SgcXJXrhBC5GDmGX8TIB74Sim1Syn1pVKqAlBTax0LYH3Os0+lUmqqUipMKRUWHx9ftAgChsLgGXJXrhBC5GBmRvQAgoGZWusOwBUK0ayjtZ6ltQ7RWodUr17drBiFEMLlmJn4TwIntdbbre+XYxwI4pRStQGsz2dNjEEIIUQupiV+rfUZ4IRSKrsAfh9gH7AWmGCdNgFYY1YMQgghrmd2P/5HgYVKKS/gCDAJ42CzVCl1H3AcGGFyDEIIIXIwNfFrrSOAkDw+kruohBDCSaS7ixBCuBhJ/EII4WIk8QshhItRWmtnx1AgpVQ8cMzZceSjGnDO2UHcgMRXPBJf8Uh8xVecGBtqra+7EapUJP6STCkVprXO6wJ2iSDxFY/EVzwSX/GZEaM09QghhIuRxC+EEC5GEn/xzXJ2AAWQ+IpH4iseia/47B6jtPELIYSLkTN+IYRwMZL4hRDCxUjit4FSqr5SaqN13OC9SqnH85jnZqXUJaVUhPXxbwfHGKOU2mPddlgenyul1MdKqUNKqUilVLADY2uZY79EKKUuK6WeyDWPQ/efUmquUuqsUioqxzSbxoNWSk2wzhOtlJqQ1zwmxfeOdfzqSKXUKqWUfz7L3vC7YGJ8ryqlTuX4N+yfz7J3KqUOWr+Lpgy9mk98S3LEFqOUishnWUfsvzxzisO+g1preRTwAGoDwdbXfsDfQJtc89wMfO/EGGOAajf4vD/wI6CArsB2J8XpDpzBuLHEafsPY1jQYCAqx7S3genW19OBt/JYrgpGpdkqQGXr68oOiq8v4GF9/VZe8dnyXTAxvleBZ2z49z+MMUKfF7A79/8ls+LL9fl7wL+duP/yzCmO+g7KGb8NtNaxWuud1teJGIPG13VuVIU2CJivDdsA/+wBcRysD3BYa+3UO7G11puAhFyTbRkP+g7gF611gtb6AvALcKcj4tNa/6y1zrS+3QbUs/d2bZXP/rNFZ+CQ1vqI1jodWIyx3+3qRvEppRQwElhk7+3a6gY5xSHfQUn8haSUagR0ALbn8XE3pdRupdSPSqm2Dg0MNPCzUipcKTU1j8/rAidyvD+Jcw5eo8n/P5wz9x/YNh50SdmPkzF+weWloO+CmR6xNkXNzaeZoiTsv15AnNY6Op/PHbr/cuUUh3wHJfEXglLKF1gBPKG1vpzr450YzRftgU+A1Q4Or4fWOhjoBzyslOqd63OVxzIO7ctrHZBnILAsj4+dvf9sVRL244tAJrAwn1kK+i6YZSbQFAgCYjGaU3Jz+v4DxnDjs32H7b8Cckq+i+UxrVD7UBK/jZRSnhj/QAu11itzf661vqy1TrK+Xgd4KqWqOSo+rfVp6/NZYBXGT+qcTgL1c7yvB5x2THRX9QN2aq3jcn/g7P1nZct40E7dj9YLeXcBY7W1wTc3G74LptBax2mts7TWFmB2Ptt19v7zAIYCS/Kbx1H7L5+c4pDvoCR+G1jbBOcA+7XW7+czTy3rfCilOmPs2/MOiq+CUsov+zXGRcCoXLOtBe619u7pClzK/knpQPmeaTlz/+Vgy3jQPwF9lVKVrU0Zfa3TTKeUuhN4DhiotU7OZx5bvgtmxZfzmtGQfLa7A2iulGps/QU4GmO/O8ptwAGt9cm8PnTU/rtBTnHMd9DMK9dl5QH0xPgpFQlEWB/9gWnANOs8jwB7MXopbAO6OzC+Jtbt7rbG8KJ1es74FDADo0fFHiDEwfuwPEYir5RjmtP2H8YBKBbIwDiDug+oCmwAoq3PVazzhgBf5lh2MnDI+pjkwPgOYbTtZn8HP7fOWwdYd6PvgoPiW2D9bkViJLDaueOzvu+P0YvlsCPjs07/Ovs7l2NeZ+y//HKKQ76DUrJBCCFcjDT1CCGEi5HEL4QQLkYSvxBCuBhJ/EII4WIk8QshhIuRxC8EoJTKUtdWELVb1UilVKOcVSKFcDYPZwcgRAmRorUOcnYQQjiCnPELcQPW2uxvKaVCrY9m1ukNlVIbrAXJNiilGlin11RGrfzd1kd366rcTbyFFQAAAUZJREFUlVKzrbXXf1ZK+TjtjxIuTxK/EAafXE09o3J8dllr3Rn4FPjQOu1TjDLXgRjF0j62Tv8Y+EMbxeaCMe7+BGgOzNBatwUuAsNM/nuEyJfcuSsEoJRK0lr75jE9BrhVa33EWlTrjNa6qlLqHEZJggzr9FitdTWlVDxQT2udlmMdjTDqpze3vn8O8NRa/5/5f5kQ15MzfiEKpvN5nd88eUnL8ToLub4mnEgSvxAFG5Xjeav19V8YlSUBxgJbrK83AA8CKKXclVIVHRWkELaSsw4hDD7q2sG312uts7t0llNKbcc4URpjnfYYMFcp9SwQD0yyTn8cmKWUug/jzP5BjCqRQpQY0sYvxA1Y2/hDtNbnnB2LEPYiTT1CCOFi5IxfCCFcjJzxCyGEi5HEL4QQLkYSvxBCuBhJ/EII4WIk8QshhIv5fxcEGXYJVCorAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, NUM_EPOCHS+1), train_acc_list, label='Training')\n",
    "plt.plot(np.arange(1, NUM_EPOCHS+1), valid_acc_list, label='Validation')\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation ACC: 76.70%\n",
      "Test ACC: 74.97%\n"
     ]
    }
   ],
   "source": [
    "with torch.set_grad_enabled(False):\n",
    "    test_acc = compute_acc(model=model,\n",
    "                           data_loader=test_loader,\n",
    "                           device=DEVICE)\n",
    "    \n",
    "    valid_acc = compute_acc(model=model,\n",
    "                            data_loader=valid_loader,\n",
    "                            device=DEVICE)\n",
    "    \n",
    "\n",
    "print(f'Validation ACC: {valid_acc:.2f}%')\n",
    "print(f'Test ACC: {test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch       1.1.0\n",
      "matplotlib  3.1.0\n",
      "pandas      0.24.2\n",
      "torchvision 0.3.0\n",
      "numpy       1.16.4\n",
      "re          2.2.1\n",
      "PIL.Image   6.0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%watermark -iv"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "convnet-vgg16.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "371px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}