pytorch_ipynb/gnn/gnn-basic-graph-spectral-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "\n",
      "CPython 3.7.3\n",
      "IPython 7.6.1\n",
      "\n",
      "torch 1.2.0\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Runs on CPU or GPU (if available)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Graph Neural Network with Spectral Graph Convolution on MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementing a very basic graph neural network (GNN) using a spectral graph convolution. \n",
    "\n",
    "Here, the 28x28 image of a digit in MNIST represents the graph, where each pixel (i.e., cell in the grid) represents a particular node. The feature of that node is simply the pixel intensity in range [0, 1]. \n",
    "\n",
    "Here, the adjacency matrix of the pixels is basically just determined by their neighborhood pixels. Using a Gaussian filter, we connect pixels based on their Euclidean distance in the grid.\n",
    "\n",
    "In the related notebook, [./gnn-basic-1.ipynb](./gnn-basic-1.ipynb), we used this adjacency matrix $A$ to compute the output of a layer as \n",
    "\n",
    "$$X^{(l+1)}=A X^{(l)} W^{(l)}.$$\n",
    "\n",
    "Here, $A$ is the $N \\times N$ adjacency matrix, and $X$ is the $N \\times C$ feature matrix (a  2D coordinate array, where $N$ is the total number of pixels -- $28 \\times 28 = 784$ in MNIST). $W$ is the weight matrix of shape $N \\times P$, where $P$ would represent the number of classes if we have only a single hidden layer.\n",
    "\n",
    "In this notebook, we modify this code using spectral graph convolution, i.e.,\n",
    "\n",
    "$$X^{(l+1)}=V\\left(V^{T} X^{(l)} \\odot V^{T} W_{\\text {spectral }}^{(l)}\\right).$$\n",
    "\n",
    "Where $V$ are the eigenvectors of the graph Laplacian $L$, which we can compute from the adjacency matrix $A$. Here, $W_{\\text {spectral }}$ represents the trainable weights (filters).\n",
    "\n",
    "- Inspired by and based on Boris Knyazev's tutorial at https://towardsdatascience.com/tutorial-on-graph-neural-networks-for-computer-vision-and-beyond-part-2-be6d71d70f49."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "from scipy.spatial.distance import cdist\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data.dataset import Subset\n",
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Settings and Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Device\n",
    "DEVICE = torch.device(\"cuda:3\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Hyperparameters\n",
    "RANDOM_SEED = 1\n",
    "LEARNING_RATE = 0.05\n",
    "NUM_EPOCHS = 50\n",
    "BATCH_SIZE = 128\n",
    "IMG_SIZE = 28\n",
    "\n",
    "# Architecture\n",
    "NUM_CLASSES = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image batch dimensions: torch.Size([128, 1, 28, 28])\n",
      "Image label dimensions: torch.Size([128])\n"
     ]
    }
   ],
   "source": [
    "train_indices = torch.arange(0, 59000)\n",
    "valid_indices = torch.arange(59000, 60000)\n",
    "\n",
    "custom_transform = transforms.Compose([transforms.ToTensor()])\n",
    "\n",
    "\n",
    "train_and_valid = datasets.MNIST(root='data', \n",
    "                                 train=True, \n",
    "                                 transform=custom_transform,\n",
    "                                 download=True)\n",
    "\n",
    "test_dataset = datasets.MNIST(root='data', \n",
    "                              train=False, \n",
    "                              transform=custom_transform,\n",
    "                              download=True)\n",
    "\n",
    "train_dataset = Subset(train_and_valid, train_indices)\n",
    "valid_dataset = Subset(train_and_valid, valid_indices)\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          num_workers=4,\n",
    "                          shuffle=True)\n",
    "\n",
    "valid_loader = DataLoader(dataset=valid_dataset, \n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          num_workers=4,\n",
    "                          shuffle=False)\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE,\n",
    "                         num_workers=4,\n",
    "                         shuffle=False)\n",
    "\n",
    "# Checking the dataset\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.shape)\n",
    "    print('Image label dimensions:', labels.shape)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def precompute_adjacency_matrix(img_size):\n",
    "    col, row = np.meshgrid(np.arange(img_size), np.arange(img_size))\n",
    "    \n",
    "    # N = img_size^2\n",
    "    # construct 2D coordinate array (shape N x 2) and normalize\n",
    "    # in range [0, 1]\n",
    "    coord = np.stack((col, row), axis=2).reshape(-1, 2) / img_size\n",
    "\n",
    "    # compute pairwise distance matrix (N x N)\n",
    "    dist = cdist(coord, coord, metric='euclidean')\n",
    "    \n",
    "    # Apply Gaussian filter\n",
    "    sigma = 0.05 * np.pi\n",
    "    A = np.exp(- dist / sigma ** 2)\n",
    "    A[A < 0.01] = 0\n",
    "    A = torch.from_numpy(A).float()\n",
    "    \n",
    "    return A\n",
    "\n",
    "    \"\"\"\n",
    "    # Normalization as per (Kipf & Welling, ICLR 2017)\n",
    "    D = A.sum(1)  # nodes degree (N,)\n",
    "    D_hat = (D + 1e-5) ** (-0.5)\n",
    "    A_hat = D_hat.view(-1, 1) * A * D_hat.view(1, -1)  # N,N\n",
    "    \n",
    "    return A_hat\n",
    "    \"\"\"\n",
    "\n",
    "\n",
    "def get_graph_laplacian(A):\n",
    "    # From https://towardsdatascience.com/spectral-graph-convolution-\n",
    "    #   explained-and-implemented-step-by-step-2e495b57f801\n",
    "    #\n",
    "    # Computing the graph Laplacian\n",
    "    # A is an adjacency matrix of some graph G\n",
    "    N = A.shape[0] # number of nodes in a graph\n",
    "    D = np.sum(A, 0) # node degrees\n",
    "    D_hat = np.diag((D + 1e-5)**(-0.5)) # normalized node degrees\n",
    "    L = np.identity(N) - np.dot(D_hat, A).dot(D_hat) # Laplacian\n",
    "    return torch.from_numpy(L).float()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATQAAAD8CAYAAAD5TVjyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3df6xc5X3n8ffnXv+ipI75URAGXEi4gXiJYhzLdcSqSiAuhkWYP9JdrHZxJVSrC3QTEW1r1FW0m1VXpJVKi8SimpDGVNkQ4qbF8nrjJYao6irGmELc61/rGzcNFhQHML5piLF957t/nOdcD9dnzjkzc+aeH/N9oaOZOXPmzBld/NXznO/zfR6ZGc451wQjZV+Ac84VxQOac64xPKA55xrDA5pzrjE8oDnnGsMDmnOuMQYS0CStkXRI0oSkjYP4DudcfUn6qqRjksY7vC9Jj4QYslfS8jznLTygSRoFHgVuBZYC6yQtLfp7nHO19jVgTcr7twJjYdsAPJbnpINooa0EJszsiJmdAp4C1g7ge5xzNWVmfwu8nXLIWuBJi+wCFkm6LOu8c4q6wDaXA6+2vT4K/MrMgyRtIIq8nP8L+sTIu4s6nlDnLcB+frLzN0rgFQ/Opfopx980s1/q9fO3fPp8e+vtqVzHvrT3vX1A+z/aTWa2qYuvS4ojlwOvp31oEAFNCfvOiTbhx20CWKgL7Z3XL2LNkhXYmTPnfvrUKCMfG6M1fjD5C+fOY+S8BUxNTiZf0cgotPL9IZxrqu/aln/q5/NvvT3F7h1Lch07etnhk2a2oo+vyxVHZhpEl/MocGXb6yuA17I+tGbJCr7z4z3Jb7am0I9fY+T66xLfttOnaP38JJo7r+PnGRnNugTnXAoDWjn/K0BPcWQQAe1FYEzS1ZLmAXcBW7M+ZGfOcMviZex47ZXE96cmJ2ntP5wa1AAPas4NiGGctqlcWwG2AneHbOcq4ISZpXY3YQABzczOAPcDO4ADwNNmti/v5+OgpjkJveHWFK39hzl5+8rk7w5BbXThwuSTe7fTub4U1UKT9A3g+8C1ko5KukfS70j6nXDIduAIMAE8Dtyb5/oGcQ8NM9seLqgncffzlsXLzn2zNcWC7S8xcv11iffU7PQpWkQttTjAncPvqTnXNcOYKij5ZmbrMt434L5uz1vJSoH27mdaSy2r+5naUvPup3Nda2G5trJUMqDFikgUdORBzbmuGDCF5drKUomApvMWJAaXXImC8YOpQU1z53miwLmCeAstB/v5SUaWjnXsIqZ2P8ETBc7NAgNOm+XaylKJgAbQGj+ILVncscWU1f2MEwVJMsepgbfUnMtgObubQ9/ljLXGDzKydCyz++mJAudKYDCVcytLpQIaREHt5G2f6Lml5hUFzg1GVCmQbytL5QIawIJtu3O11JJ4RYFzgyKmcm5lqWRAg7Pdz54SBV5R4FzhoqSAcm1lqWxAg7ZEQQd5EgWdZI5TA2+pOdcmGofmLbRsUscu4PQ4sx4TBYyMZo5T80SBc/m0TLm2slQjoJkxct6C1KDmiQLnyuUttC7EkzN2CiyeKHCuXIaYYiTXVpbKBDTIDiyeKHCuXN7l7JKdPpXZ/fSKAudmnyFO2WiurSyVC2iQ3f30igLnZl80sHYk11aWSgY0yNdS80SBc7PLkwJ9mJqcnB5akWTBtt0dg0sRUw+lfbffU3PDxkxM2UiurSzVCWgZLZ60+1qDmnoo1rH76dyQaaFcW1kyA5qkr0o6Jmm8bd+Fkp6VdDg8XhD2S9IjkiYk7ZW0PPeVpHTjPFHgXPmipMCcXFtZ8rTQvgasmbFvI7DTzMaAneE1wK3AWNg2AI91dTUpQc0TBc6VqxFJATP7W+DtGbvXApvD883AnW37n7TILmCRpMu6uqKUe1OeKHCuXFOmXFtZeg2ll8aLfobHS8L+y4FX2447GvadQ9IGSXsk7TnNewlX1ltLzSsKnBuMYawUSArNifNXmtkmM1thZivmMv/cAzLuqYFXFDg321o2kmsrS6/f/EbclQyPx8L+o8CVbcddAbzW89UVkSjowBMFznUnKk5vZgttK7A+PF8PPNO2/+6Q7VwFnIi7pj3LSBSkjRXrd+qhPOPUPFHghoUhTttorq0seYZtfAP4PnCtpKOS7gEeAlZLOgysDq8BtgNHgAngceDeQq4yoxs3sEQBpCYKAE8UuKFhRuUH1mYOGDGzdR3eujnhWAPu6/eiOhoZTQxuU5OT0wsKx/e42i3YtpuR66+jtf/wOZ9vb6ndsnhZ4rnZ/7Po8+MHz3m/fTHjpO+eDmp+b83VXrmDZvOoTqVAHp4ocK40RvVbaPUKaOAVBc6VqKlJgXLVuaLAuZoy8k3u6BM89qKuFQXgLTVXS9EydnNybWWpb0CLZbTUOpmuKEhQVEVBR579dLXkCw0PXsY9tTgDmfjRMM6s50TB+MHUREHmEnnO1YjR3EqBaikrUQCeKHBDxVtos8UTBc4NlJkKbaFJWiPpUJg/cWPC+0skPS/p5TC/4m1Z52xOQANPFDg3QFFSoJjSJ0mjwKNEcyguBdZJWjrjsP8MPG1mNwB3Af8j67zNCmixCk895LWfrr4KXVNgJTBhZkfM7BTwFNF8iu0MiP/BfJAcE100M6BVuKJganLSu5+ulqKkQO5xaBfH8x2GbcOM0+WZO/G/AL8p6ShRnfjvZl1jMwMaVLaiADIK2sFbaq6yuqgUeDOe7zBsm2acKs/cieuAr5nZFcBtwF9KSo1ZzQ1okNriyRqnNp0oSJDV/Yxbap3kGqfmXMUUXCmQZ+7Ee4CnAczs+8AC4OK0kzY7oGWI5zNLTRTcvrJjiykrqI0uXJh6Ty1tjBzgLTVXOQUukvIiMCbpaknziG76b51xzI8Js/pI+ihRQPtJ2kmHJ6ANIFEA6UHNEwWuSczgdGsk15Z9LjsD3A/sAA4QZTP3SfqSpDvCYV8AflvSD4BvAL8VpijrqLyiq9mWMi9Z1pxmcUWBfvxaYlc1DmprlqzAzpw553vjRMGCbbsTv3sqtBQTu8He/XQVEXU5i2sDmdl2opv97fu+2PZ8P3BjN+ccnhYaeKLAuT55pUDVlFxR4JNEurrqcthGKYYvoEGpFQVe++nqq9jSp0HIs0jKlaGe6oCkfZI+F/ZfKOlZSYfD4wVhvyQ9Euqz9kpaPugf0bMSKgoKqf30oOZK0grrCmRtZckTSs8AXzCzjwKrgPtCzdVGYKeZjQE7w2uIarPGwrYBeKzwqy5Kxj21rCXy4qEZSfqdesgrClzVRFnO0VxbWTIDmpm9bmZ/H57/lCjFejlR3dXmcNhm4M7wfC3wpEV2AYviRYkrKaPFk9b9BAY29RB4osBVS+Om4JZ0FXAD8AJwabyIcHi8JByWp0YLSRviOq/TvNf9lRfJEwXO5dKELicAkj4A/BXweTNLqxvKU6OFmW2K67zmMj/vZQyOJwqcS9WYLKekuUTB7Otm9u2w+424Kxkej4X9eWq0qmvAiYKBTRLpQc3NgiZkOQU8ARwwsz9pe2srsD48Xw8807b/7pDtXAWciLumtZCjpdbxo2VOEulBzQ2YmThjI7m2suT55huBfw/cJOmVsN0GPASslnQYWB1eQ1TKcASYAB4H7i3+smdBSkstraB9NiaJ9KDmylL1LmdmLaeZ/R3J98UgVMLPON6A+/q8rvKl1H5OTU7mqv1sjR9MPHUc1G5ZvCzxe7NqP+PVpLz2082m+B5alQ1npUBeGd3PrHFq/Uw9tGDb7tSWWhxUnZtNVW+heUDLI6Mb1+/UQ4mJAkjtfsY6JgrAu5+uUI0bhza0PFHgHNCgcWgOTxS4oWYGZ1ojubayeEDrRgEVBYNaTQq8osANnnc5m6bPioJBTRLpFQVu0PweWpOVWPvpFQWuLGbKtZXFA1qvikgUdOCJAldVnhRoun4SBddf13uiIAzeTZI1Rs6DmuuFmd9Da76yEgWQmiiIeaLAFUdMtUZybWXxgFYETxS4IeH30IaJJwpcgzVmPjSXU4mTRHqiwA2cRffR8mxl8YA2CCWsJuUVBW42eJZzGGWsJgVeUeDqxzwpMMT6HKdmSxZ3fL+IREEqb6m5DrzLOexSup/xJJFJpseZ9ZooyBinFk8Smfzl3v10yTzLOezKShSAJwpcoaLWlwc0B54ocI1Q+2EbkhZI2i3pB5L2SfqvYf/Vkl6QdFjSNyXNC/vnh9cT4f2rBvsTasITBa4BmnAP7T3gJjP7OLAMWBOWp/sy8LCZjQHHgXvC8fcAx83sGuDhcJyDzKDmFQWuygzRao3k2sqS+c0W+Zfwcm7YDLgJ2BL2bwbuDM/XhteE928Oa3s6KKT20ysKXFks51aWvCunj0p6hWh19GeBHwLvmNmZcMhR4PLw/HLgVYDw/gngooRzbpC0R9Ke07zX36+oG68ocHVUcFJA0hpJh8LtqY0djvm3kvaH213/M+ucuQKamU2Z2TLgCmAl8NGkw+JrSHmv/ZybzGyFma2Yy/w8lzE0EtfbbDOdKEiQN1HQSdI6o+/jQW24FdREkzQKPArcCiwF1klaOuOYMeBB4EYz+1fA57PO21Vn18zeAb4HrAIWSYr7NlcAr4XnR4ErwwXNAT4IvN3N9wyVlHtqecap9ZooYGQ0NVGQOU7NDaUCW2grgQkzO2Jmp4CniG5Xtftt4FEzOx59tx3LOmmeLOcvSVoUnp8HfAY4ADwPfDYcth54JjzfGl4T3n8urKbukniiwNWEAa2Wcm3AxfEtpbBtmHG66VtTQfttq9hHgI9I+r+Sdklak3WNeVpolwHPS9oLvAg8a2bbgN8HHpA0QXSP7Ilw/BPARWH/A0Bi39i1qXOiwA0PA0z5NngzvqUUtk0zzpbn1tQcYAz4FLAO+ErcuOokeSrU9m8w2wvckLD/CFGzceb+k8CvZ53XzZCRKBhduJAWyfe4WuMHOXn7ShZsfynxPHFL7ZbFyxK/N04UtMYPJn53iyigdry/NjLqwW1IFNjXmr41FbTftmo/ZpeZnQb+UdIhogD3YqeTeqVA1XhFgauy4sZtvAiMhQH684C7iG5Xtfsb4NMAki4m6oIeSTupB7Sq8YoCV1n5EgJ5kgJhSNf9wA6ie/JPm9k+SV+SdEc4bAfwlqT9RPfs/5OZvZV2Xg9oVeSJAldVBY6sNbPtZvYRM/uwmf1h2PdFM9sanpuZPWBmS83sY2b2VNY5PaBVVUaiIG2Zutb4wY6fL2LqofieXsfrds1kYC3l2sriAa3KMoJDWksNGNjUQ4C31IaWcm7l8IBWBxVOFHjt55CpeDGnB7Q6qHCiIF4hvuN1u2bxgOYKUdFEAXj3c2h0N7C2FB7Q6sQrClzJmjDBo6uSuk49BN5Sa4KW8m0l8YBWV/0mChJ4osBlkeXbyuIBra4y7qlljVPrZ+qhuHa003d7oqCh8iYEPKC5nmS0eAaWKABPFAylnAkBTwq4npWcKPDazyHjLTQ3cCUmCrz2c8i0cm4l8YDWJCVUFPhqUkPEx6G5WeUVBW7APMvpZpdXFLhBaso9tLA258uStoXXV0t6QdJhSd8Ms04iaX54PRHev2owl+46ypEo6PjROFGQIG/30xMFrizdtNA+RzSzZOzLwMNmNgYcB+4J++8BjpvZNcDD4Tg323KsUZCaKLh9ZcegmBXUFmzbnXpPbWpy0ltqNdWILqekK4B/A3wlvBZwE7AlHLIZuDM8XxteE96/ORzvyjCARAFk3FOD1ERBzBMFNWM0pvTpT4Hf42xC9iLgnTAvOLx/Tb3p9fbC+yfC8e8jaUO8Zt9p3uvx8l2mHEM6On60zNpPD2rVVPd7aJJuB46Z2UvtuxMOtRzvnd1htiles28u83NdrOtDSkstrfvpq0m5dk3oct4I3CHpR0TLtd9E1GJbJCnub7SvqTe93l54/4PA2wVes+vFgCoKwFeTGip1b6GZ2YNmdoWZXUW0dt5zZvYbRMtKfTYcth54JjzfGl4T3n/OrMwZktw0ryhw/ap7QEvx+8ADkiaI7pE9EfY/AVwU9j8AbOzvEl3hBlxRMLBJIj2olSpvd7PqXc5pZvY9M7s9PD9iZivN7Boz+3Uzey/sPxleXxPeT13p2JWgiERBB0UkCjp/uQe10jUky+maqJ9EwfXX9Z4oyFj3U3PneaKgohrVQnMNU0CioKfaT/BEQV01+B6aa4I+EwWDqv30REEFNe0emmuwuq4m5UFtdnkLzdWCVxS4HNTKt5XFA5p7P68ocDXmAc29X1mJAq8oqAfvcrra6bP7aUsWd3w/T6Ig7btTx6mBt9QGyZMCrtZSWmppY8Wmx5n1mChgZDRznJonCkriLTRXWyXWfnqioKI8oLnaK2E1KU8UVI/wLKdrggqvJgWeKJg1Bd9Dk7RG0qGw/kjHSSwkfVaSSVqRdU4PaC4fryhwUFiXU9Io8ChwK7AUWCdpacJxvwj8R+CFPJfnAc11xysKhltx99BWAhNh1p5TRJPHrk047r8BfwRkpLcjHtBcdzxRMNS66HJeHK8ZErYNM041vfZI0L4uSfRd0g3AlWa2Le/1JU+F4FyWkdHE4NbeUotbTu0WbNvd8fPtLbVbFi9LPvf4JCPXX0dr/OA577ffz0v6br+nVoD8Gcw3zSztnlfq2iOSRoiWwfyt3N+It9Bcr3K0eNLuaw1q6qFYx+6n650VmuWcXnskaF+XBOAXgeuB74X1TFYBW7MSAx7QXO8ysp+eKGig4u6hvQiMSbpa0jyi9Uq2Tn+N2Qkzu9jMrgrrmewC7jCzDv9TRDyguf4MaDUpTxRUU1HDNsKavfcDO4ADwNNmtk/SlyTd0ev15V05/UeS/kHSK5L2hH0XSnpW0uHweEHYL0mPhLEleyUt7/XiXE14omB4FFgpYGbbzewjZvZhM/vDsO+LZrY14dhPZbXOoLsW2qfNbFnbjb6NwE4zGwN2cnZ1p1uBsbBtAB7r4jtcnXlFQbPlDWY1LX1aC2wOzzcDd7btf9Iiu4gWJL6sj+9xdeEVBY0mmjPbhgH/R9JLbeNJLjWz1wHC4yVhf+b4EgBJG+IxKqd5r7erd9VTRKKgA08UlK8pAe1GM1tO1J28T9KvphybOr5keofZJjNbYWYr5jI/52W4WshIFMRTACV+tM+ph7KWyLPTpzxR0I8mdDnN7LXweAz4a6KyhTfirmR4PBYOzxpf4oZBRjduYIkCSE0UAJ4o6EfdA5qk80OBKJLOB34NGCcaM7I+HLYeeCY83wrcHbKdq4ATcdfUDSFPFDRHzu5m1buclwJ/J+kHwG7gf5nZd4CHgNWSDgOrw2uA7cARYAJ4HLi38Kt29eGJgmapeAsts5bTzI4AH0/Y/xZwc8J+A+4r5OpcM8RBLSFIxPe0WpBYfzl9T2z/zxI/H3c/k2o/2xMFnWo/W6TUfkLH6x5WZU7emIdXCrjZUeeKAjetCV1O54pR14oC8Htq0PiBtc71JqOl1sl0oiBBUYmCjjxREPGA5twMGYmCPEvk9ZwoGD+YmijIXCJviDWpUsC5YpU19RB4RUEf1LJcW1k8oLnyeKKgXvwemnMZPFFQK97ldC6PClcUeO1nG2+hOZdDhSsKpiYnvfsZeAvNubwqukYBZBS0w/C01LyF5lwXUlo8WePUphMFCbK6n3FLrZNc49Sazgpd9WkgPKC5WolrP1MTBbev7NhiygpqowsXpt5TSxsjBzS6pebj0JzrxwASBZAe1DxRkMEs31YSD2iuujxRUDneQnOuH54oqA4fWOtcAUquKPBJIs/ypIBzRSixosBrP8/ygOZckUqoKCik9rMJQc3wpIBzhcq4p5a1RF48NCNJv1MPDUOioBFJAUmLJG2RdFDSAUmflHShpGclHQ6PF4RjJekRSROS9kpaPtif4IZORosnrfsJDGzqIRiCREFDkgJ/BnzHzK4jWjDlALAR2GlmY8DO8BqixYjHwrYBeKzQK3YOPFFQgkYMrJW0EPhV4AkAMztlZu8Aa4HN4bDNwJ3h+VrgSYvsAhbFCxI7VyhPFMwuyze5Y9UnePwQ8BPgLyS9LOkrYcHhS+MFhMPjJeH4y4FX2z5/NOx7H0kbJO2RtOc07/X1I9yQG3CiYGCTRNYyqOXcSpInoM0BlgOPmdkNwM84271MooR95/xEM9tkZivMbMVc5ue6WOcS5WipdfxomZNE1jCo1b7LSdTCOmpmL4TXW4gC3BtxVzI8Hms7/sq2z18BvFbM5TqXIqWlllbQPhuTRDYiqBnQsnxbSTIDmpn9M/CqpGvDrpuB/cBWYH3Ytx54JjzfCtwdsp2rgBNx19S5gSoiUdBBnnFqQ5EoaECXE+B3ga9L2gssA/478BCwWtJhYHV4DbAdOAJMAI8D9xZ6xc6lyeh+Zo1T62fqoQXbdqe21KYmJ9MTBTVQZJdT0hpJh8IQr3NuY0l6QNL+MPxrp6Rfzjpnwt3Oc5nZK8CKhLduTjjWgPvynNe5gRkZTQ1umjsvcdLGOCi19h9O/Hwc1NYsWYGdOXPO+3H3szV+sON3jy5c2HmyyozrLltRGUxJo8CjRI2ho8CLkraa2f62w14GVpjZu5L+A/BHwL9LO69XCrhm8kRB8YqdbWMlMGFmR8zsFPAU0ZCvs19n9ryZvRte7iK6H5/KA5prNk8UFCYaWGu5NuDieFhW2DbMOF2u4V1t7gH+d9Y1ekBzzVZAomBQk0RCDRMFrZwbvBkPywrbphlnyjW8C0DSbxLd8vrjrMvzgOaar8+KgkFNElnHioIuWmhZcg3vkvQZ4A+AO8wscwS+BzQ3PEqs/WxERUGx99BeBMYkXS1pHnAX0ZCvaZJuAP6cKJgdSzjHOTygueFRRKKgg+FIFBRXy2lmZ4D7gR1Ek108bWb7JH1J0h3hsD8GPgB8S9IrkrZ2ON20XMM2nGuUDkMj4kRB6+cnux7S0d5Su2XxssRzMz7ZcUhHe6IgcQ3QOKiVfW+twMkbzWw70bjV9n1fbHv+mW7P6S00N3zKShRAaqIgVtlEgfkU3M5VkycKeuNTcDtXYZ4o6E5Dajmda6YSJ4msY6JArVaurSwe0JyDUlaTql1FgdHNwNpSeEBzDjJXkwKvKBD5BtXmHFg7EB7QnIv1OU7Nlizu+H4RiYJUs9VS86SAczWT0v3U3HnpLbXrr+s9URB/PkE8j1vpiQIPaM7VTFmJAqh2osDvoTlXY54oOIdnOZ2rK08UzPzm+nc5JV0bCkPjbVLS5yVdKOlZSYfD4wXheEl6JMwTvlfS8sH/DOcGJCOoDVVFgVH/gGZmh8xsmZktAz4BvAv8NdHanDvNbAzYydm1Om8FxsK2AXhsEBfu3KwpYjWpplQUNOwe2s3AD83sn4jm/94c9m8G7gzP1wJPWmQXsChev9O52vKKAqDQCR4HotuAdhfwjfD80ni9zfB4Sdjf7VzhztVex1WcgulEQYK8iYJOEqcbaldkUKt7lzMWZpW8A/hW1qEJ+875hZI2xAsonCZzZl3nqiPlnlqecWq9JgoYGU1NFGSOU+uXGUy18m0l6aaFdivw92b2Rnj9RtyVDI/xFLm55go3s03xAgpzmd/9lTtXljonCvrVlBYasI6z3U2I5v9eH56vB55p2393yHauAk7EXVPnGqOuiYJ+NSGgSfoFohWOv922+yFgtaTD4b2Hwv7twBFgAngcuLewq3WuSuqaKOiVAS3Lt5UkV0Azs3fN7CIzO9G27y0zu9nMxsLj22G/mdl9ZvZhM/uYmXWu83CuCepYUdATA2vl20rilQLO9auuFQXdMhqVFHDOdTIsiYIm3ENzzuWQkSiIh1YkfnT8YMfPFzH1UOZYtbw8oDk3RDLGe6W11ICBTT1UjAYUpzvnelDBREHfDGi18m0l8YDm3CBUNFHQN2+hOTekKpoo6F2zSp+cc92qaEVBTwzMWrm2snhAc27QKlpR0JMmVAo45wrQb6IgwawnCvwemnMOyLynljVOrZ+ph1rjB3u+7LMXaZ7ldM61yZhscWCJgqJ4C8059z5lJQr6ZtjUVK6tLB7QnCtDiYsZ96wp0wc55wakhIqCvvj0Qc65jsqqKOiBAdayXFsektZIOhTW8N2Y8P58Sd8M778g6aqsc3pAc65sZVUUdMuKm+BR0ijwKNFaJUuBdZKWzjjsHuC4mV0DPAx8Oeu8HtCcq4IciYKOH40TBQmK7n4WmBRYCUyY2REzOwU8RbSmb7v2tX+3ADdLSlpVbtogUiFd+ynH/+W7tuVQ2dcxQBcDb5Z9EQPS5N8Gs/n70uJA1nRm/5D+9mi01Pcvd3U9M/yU4zu+a1suznn4AkntTcNNZrap7XXS+r2/MuMc08eY2RlJJ4CLSPl7VCKgAYfMbEXZFzEokvY09fc1+bdB839fN8xsTYGny7N+b641ftt5l9M5V4Y86/dOHyNpDvBB4O20k3pAc86V4UVgTNLVkuYBdxGt6duufe3fzwLPmaWXIVSly7kp+5Baa/Lva/Jvg+b/vlKEe2L3AzuAUeCrZrZP0peAPWa2FXgC+EtJE0Qts7uyzquMgOecc7XhXU7nXGN4QHPONUbpAS2r/KHqJF0p6XlJByTtk/S5sP9CSc9KOhweLwj7JemR8Hv3Slpe7i/IJmlU0suStoXXV4dSlMOhNGVe2N91qUrZJC2StEXSwfA3/GST/nbDptSAlrP8oerOAF8ws48Cq4D7wm/YCOw0szFgZ3gN0W8dC9sG4LHZv+SufQ440Pb6y8DD4bcdJypRgR5KVSrgz4DvmNl1wMeJfmeT/nbDxcxK24BPAjvaXj8IPFjmNRXwm54BVgOHgMvCvsuIBg8D/Dmwru346eOquBGND9oJ3ARsIxrs+CYwZ+bfkChj9cnwfE44TmX/hpTfthD4x5nX2JS/3TBuZXc5k8ofLi/pWvoWulg3AC8Al5rZ6wDh8ZJwWN1+858CvwfEFccXAe+Y2Znwuv3631eqAsSlKlX1IeAnwF+ELvVXJJ1Pc/52Q6fsgNZ1aUNVSfoA8FfA580srZq4Nr9Z0u3AMTN7qX13wqGW470qmgMsBx4zsxuAn3G2e5mkbr9v6JQd0PKUP1SepLlEwezrZvbtsPsNSZeF9y8DjoX9dfrNN6ViJcYAAAERSURBVAJ3SPoR0WwINxG12BaFUhR4//V3XapSsqPAUTN7IbzeQhTgmvC3G0plB7Q85Q+VFqYzeQI4YGZ/0vZWe9nGeqJ7a/H+u0PGbBVwIu7eVI2ZPWhmV5jZVUR/m+fM7DeA54lKUeDc39ZVqUqZzOyfgVclXRt23QzspwF/u6FV9k084Dbg/wE/BP6g7Ovp4fr/NVG3Yy/wSthuI7p3tBM4HB4vDMeLKLP7Q6JJX1aU/Rty/s5PAdvC8w8Bu4EJ4FvA/LB/QXg9Ed7/UNnXneN3LQP2hL/f3wAXNO1vN0yblz455xqj7C6nc84VxgOac64xPKA55xrDA5pzrjE8oDnnGsMDmnOuMTygOeca4/8DAvHK4/4wKNMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "A = precompute_adjacency_matrix(28)\n",
    "plt.imshow(A, vmin=0., vmax=1.)\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATQAAAD8CAYAAAD5TVjyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAbVUlEQVR4nO3df5Bd5X3f8fdHP1hvcbzih2EYIEGeXRyrnjGQHSKGTmuD2RXrDOIPnKJJhOxRo5kaUnvsNhVNx21pp4PTaYiZoTSy5VhQxxgTO2iIYoUIPGk7EUa2HMVClnXRboIGBxkQshuCkLTf/nGeKy67q71nd8/dc+65n5fnmXvPcx6d+xx2/J3z/DqPIgIzszpYUnYFzMyK4oBmZrXhgGZmteGAZma14YBmZrXhgGZmtdGRgCZpjaSDkhqSNnfiN8yse0n6kqSjkn5wlvOSdH+KIfskXZPnuoUHNElLgQeAm4FVwDpJq4r+HTPral8G1sxy/mZgKKVNwIN5LtqJJ7RrgUZEHI6IN4FHgLUd+B0z61IR8RfAq7MUWQs8FJndwApJl7S77rKiKtjiUuCFluMjwC9PLSRpE1nkRX3n/NI7T5zbgaqYWdPPOPZyRLx7vv9+9EPnxiuvns5V9rv7TuwH3mjJ2hIRW+bwczPFkUuBH8/2jzoR0DRD3rT1VenmtgC8S+fHBf/rXzO4fm8HqmNmAH8ej/3NQv79K6+e5js7fz5X2aWXHHojIoYX8HO54shUnWhyHgEubzm+DHix3T8aXL+XxsNXd6A6ZlaEACZz/q8A84ojnQhozwJDklZKOge4Hdie5x86qJlVVxCcjNO5UgG2A3ek0c7VwPGImLW5CR1ockbEKUl3ATuBpcCXImJ/3n/fDGpufppVT0FPX0j6KvBB4EJJR4D/ACwHiIj/CewAxoAG8Drw8TzX7UQfGhGxI1VoXhzUzKonCE4X9LqxiFjX5nwAd871upVdKeDmp1n1TBK5UlkqG9AgC2o/2rqQgRIzK0oAp4lcqSyVDmgAV27cw7E/GSq7GmaGn9AKcd5HDnF8x2DZ1TDraQGcjMiVytIVAQ1gYKzhoGZWosjZ3HSTMycHNbMSBZzOmcrSVQENHNTMypKtFMiXytJ1AQ0c1MzKIU7nTGXpyoAGDmpmiy0bFFCuVJauDWiQBTXPUzNbHNk8ND+hddSVG/fwDztXll0Ns54wGcqVytL1AQ2gf3TcQc2sw/yEtogc1Mw6KxCnWZIrlaU2AQ0c1Mw6zU3OReagZtYZgXgzluZKZaldQAMHNbNOyCbWLsmVylLLgAYOamad4EGBEvWPjvslkWYFiRCnY0muVJZaBzTIXhJ54s+uKLsaZrUwiXKlsrQNaJK+JOmopB+05J0v6UlJh9LneSlfku6X1JC0T9I1nax8Xn0jEw5qZguUDQosy5XKkucJ7cvAmil5m4FdETEE7ErHADcDQyltAh4sppoL56BmtjC1GBSIiL8AXp2SvRbYlr5vA25tyX8oMruBFZIuKaqyC+WgZrYwp0O5UlnmG0ovbm76mT4vSvmXAi+0lDuS8qaRtEnSHkl7TnJintWYOwc1s/npxZUCM4XmGd9fGRFbImI4IoaX01dwNWbnoGY2P5OxJFcqy3x/+aVmUzJ9Hk35R4DLW8pdBrw4/+p1Tt/IhOepmc1Btji9nk9o24EN6fsG4PGW/DvSaOdq4HizaVpF/aPjfkmkWU6BOBlLc6Wy5Jm28VXgL4H3SjoiaSNwL3CTpEPATekYYAdwGGgAXwA+0ZFaF2hgrOHmp1kOEVR+Ym3bCSMRse4sp26coWwAdy60Uout2afWNzJRdlXMKqzcSbN51H6lQF4eKDCbXVD9JzQHtBYOamazq+ugQG05qJnNLMj3cke/4LFiHNTMpsu2sVuWK5XFAe0s+kYmPKXD7G280XBXGxhrePKtWRLUd6VAz+gfHXfz0yzxE1oNuE/NLHtjbZFPaJLWSDqY3p+4eYbzPy/paUl70/sVx9pd0wEtJwc163XZoEAxS58kLQUeIHuH4ipgnaRVU4r9e+DRiLgauB34H+2u64A2Bw5q1tsK3VPgWqAREYcj4k3gEbL3KbYK4F3p+wA5XnThgDZHDmrWq7JBgdzz0C5svu8wpU1TLpfn3Yn/Efh1SUfI1on/Zrs6ljdhpIt57af1qjmsAng5IoZnOZ/n3YnrgC9HxH+XdB3wsKT3R8Tk2S7qJ7R56huZ8BZ51lMKXimQ592JG4FHASLiL4F3ABfOdlEHtAUYXL/X89SspxS4ScqzwJCklZLOIev03z6lzN+S3uoj6X1kAe0ns13UAW2BPE/NekUEnJxckiu1v1acAu4CdgIHyEYz90u6R9ItqdhngN+Q9FfAV4GPpVeUnZX70ArgPjXrBVmTs7hnoIjYQdbZ35r32ZbvzwHXz+WafkIriEc/rRd4pUAPcVCzOpvjtI1SOKAVzEHN6qvYpU+dkGeTlMvTeqoDkvZL+mTKP1/Sk5IOpc/zUr4k3Z/WZ+2TdE2nb6JqHNSsribTvgLtUlnyhNJTwGci4n3AauDOtOZqM7ArIoaAXekYsrVZQyltAh4svNZdwPPUrG6yUc6luVJZ2ga0iPhxRHwvff8Z2RDrpWTrrralYtuAW9P3tcBDkdkNrGhuStxrBtfv9ZOa1UbtXsEt6QrgauAZ4OLmJsLp86JULM8aLSRtaq7zOsmJude8S7j5aXVShyYnAJLeCfwR8KmI+OlsRWfImzYZLiK2RMRwRAwvpy9vNbqSg5rVQW1GOSUtJwtmX4mIb6Tsl5pNyfR5NOXnWaPVcxzUrA7qMMopYCtwICJ+t+XUdmBD+r4BeLwl/4402rkaON5smvY6BzXrZhHiVCzJlcqSZ+nT9cB64K8lfT/l/TvgXuBRSRvJFpF+NJ3bAYwBDeB14OOF1rjLeZmUdbMym5N5tA1oEfF/mLlfDNJK+CnlA7hzgfWqtb6RCf5h50r6R8fLropZbs0+tCrzSoGS9I+Oe56adZ1aDApYZ3iemnWT2s1Ds+J5oMC6SW3moVnnOKhZN4iAU5NLcqWyOKBVhIOadQM3OS03BzWrMveh2Zw5qFmVRShXKosDWgU156mZVY0HBWxe+kfHOb5jsOxqmJ0R4T40W4CBsYabn1Yh4vTkklypLA5oFec+NasS96HZgjmoWRXU5n1oVj4HNStdZP1oeVJZHNC6iIOalc2jnFYoBzUrS3hQwDqhb2TCUzqsFG5yWkcMjDU8+dYWnUc5rWP6R8fd/LRFkz19OaBZB7lPzRZT10/bkPQOSd+R9FeS9kv6Tyl/paRnJB2S9DVJ56T8vnTcSOev6OwtmIOaLZY69KGdAG6IiA8AVwFr0vZ0nwPui4gh4BiwMZXfCByLiEHgvlTOOsxBzTotEJOTS3KlsrT95cj8v3S4PKUAbgAeS/nbgFvT97XpmHT+xrS3p3WYg5p1WuRMZcm7c/rStCfnUeBJ4HngtYg4lYocAS5N3y8FXgBI548DF8xwzU2S9kjac5ITC7sLO8NBzTqm4EEBSWskHUzdU5vPUuZXJT2Xurv+sN01cwW0iDgdEVcBlwHXAu+bqVizDrOca73mlogYjojh5fTlqYbl1Dcy4S3yrDMKekSTtBR4ALgZWAWsk7RqSpkh4G7g+oj4x8Cn2l13To3diHgN+DawGlghqblR8WXAi+n7EeDyVKFlwADw6lx+xxbOW+RZJxT4hHYt0IiIwxHxJvAIWXdVq98AHoiIY9lvx9F2F80zyvluSSvS937gw8AB4GngtlRsA/B4+r49HZPOP5V2U7dF5uanFSmAyUnlSsCFzS6llDZNudyZrqmktduq6UrgSkn/V9JuSWva1XFZuwLAJcC29Ii4BHg0Ip6Q9BzwiKT/AuwFtqbyW4GHJTXInsxuz/Eb1iHNoNY3MlF2VazbBZB/jtnLETE8y/k8XVPLgCHgg2StwP8t6f2ppTijtgEtIvYB0zpkIuIw2WPj1Pw3gI+2u64tHgc1K0qBba0zXVNJa7dVa5ndEXESGJd0kCzAPXu2i3qlQI9w89MKUdy8jWeBoTRB/xyyltz2KWX+GPgQgKQLyZqgh2e7qANaD3FQs4XJNyCQZ1AgTem6C9hJ1if/aETsl3SPpFtSsZ3AK6l762ng30TEK7NdN08fmtWIm5+2IAUO70XEDmDHlLzPtnwP4NMp5eIntB7keWo2LwExqVypLA5oPcrz1Gx+lDOVwwGth7lPzeas4os5HdB6nIOazYkDmlWdg5rl0pxYmyeVxAHNAAc1y6cOL3i0HuGgZm1NKl8qiQOavU3fyIR3k7KzUuRLZXFAs2n6R8e976dNl3dAwAHNqmZgrOHmp02Rc0DAgwJWRe5Ts2n8hGbdzEHN3mYyZyqJA5q15aBmgOehWX04qBl4lNNqxEHNatOHlvbm3CvpiXS8UtIzkg5J+lp66ySS+tJxI52/ojNVtzL0jUx4SodV1lye0D5J9mbJps8B90XEEHAM2JjyNwLHImIQuC+VsxoZGGs4qPWoWjQ5JV0GfAT4YjoWcAPwWCqyDbg1fV+bjknnb0zlrUY8T60HBbVZ+vR7wG/x1oDsBcBr6b3g8PY99c7st5fOH0/l30bSpuaefSc5Mc/qW5ncp9aDur0PTdKvAEcj4rut2TMUjRzn3sqI2BIRwxExvJy+XJW16nFQ6y11aHJeD9wiaYJsu/YbyJ7YVkhqbrLSuqfemf320vkBsg2HraYc1HpItz+hRcTdEXFZRFxBtnfeUxHxa2TbSt2Wim0AHk/ft6dj0vmn0u4tVmMOaj2i2wPaLP4t8GlJDbI+sq0pfytwQcr/NLB5YVW0buGgVm95m5tlNjnntC9nRHwb+Hb6fhi4doYybwAfLaBu1oWa89QGxhplV8U6ocQRzDy8UsAKNzDW8Esia6rqT2gOaNYR/aPjbn7WUY370Mxm5T61mumCPjQHNOsoB7Wa8ROa9ToHtfrQZL5UFgc0WxQOarYYHNBs0Tio1YCbnGZv6RuZoPHw1WVXw+bDgwJm0w2u3+sntW7lJzSz6dz87FIOaGYzc1DrLsKjnGazclDrIgX3oUlaI+lg2n/krC+xkHSbpJA03O6aDmhWOge1LlJQk1PSUuAB4GZgFbBO0qoZyv0c8K+AZ/JUzwHNKsFBrUsU14d2LdCIiMMR8SbZy2PXzlDuPwO/A7yR56IOaFYZDmrVN4cm54XNPUNS2jTlUmf2Hkla9yXJfku6Grg8Ip7IW785vQ/NrNOa89QG1+8tuyo2k/wjmC9HxGx9XrPuPSJpCdk2mB/L/Yv4Cc0qyPPUKioKHeU8s/dI0rovCcDPAe8Hvp32M1kNbG83MOCAZpXk5mdFFdeH9iwwJGmlpHPI9ivZfuZnIo5HxIURcUXaz2Q3cEtE7Jntog5oVlkOatVT1LSNtGfvXcBO4ADwaETsl3SPpFvmW79cfWjpke9nwGngVEQMSzof+BpwBTAB/GpEHEu7pH8eGANeBz4WEd+bbwWttzWDWt/IRNlVMSh0FUBE7AB2TMn77FnKfjDPNefyhPahiLiqpaNvM7ArIoaAXby1u9PNwFBKm4AH5/AbZtP4Sa0i8jY3u3Tp01pgW/q+Dbi1Jf+hyOwm25D4kgX8jpmDWgWI+rxtI4A/k/TdlvkkF0fEjwHS50Upv+38EgBJm5pzVE5yYn61t57SNzLh3aRKVvWAlnce2vUR8aKki4AnJf1wlrKzzi85kxGxBdgC8C6dX+J/Ausm/aPj3vezTBX/f2quJ7SIeDF9HgW+SbZs4aVmUzJ9Hk3F280vMVuQgbGGm59l6fY+NEnnpgWiSDoXGAF+QDZnZEMqtgF4PH3fDtyhzGrgeLNpalYU96mVoOC3bXRCnibnxcA3s9kYLAP+MCK+JelZ4FFJG4G/BT6ayu8gm7LRIJu28fHCa22Gp3SUouJNzrYBLSIOAx+YIf8V4MYZ8gO4s5DambXhoLa4ynx5Yx5eKWBdz83PxVP1JqcDmtWCg9oiqPnEWrNK6RuZ4PiOwbKrUW8OaGaLZ2Cs4cm3HVKnlQJmXaN/dNzNzw7RZORKZXFAs1pyn1oHuA/NrDwOasVzk9OsRA5qBfMTmlm5HNSK4yc0swpwUCuIn9DMqqG5RZ7NUxS661NHOKBZTxlcv9fz1ObJ89DMKsjz1BYgIl8qiQOa9ST3qc2Pn9DMKspBbY48sdas2hzU5saDAmYV56CWnwOaWRdwUMsh8KCAWbfwPLX2ajEoIGmFpMck/VDSAUnXSTpf0pOSDqXP81JZSbpfUkPSPknXdPYWzIozuH6vn9RmU5NBgc8D34qIXyTbMOUAsBnYFRFDwK50DHAzMJTSJuDBQmts1mFufs6sFhNrJb0L+KfAVoCIeDMiXgPWAttSsW3Aren7WuChyOwGVjQ3JDbrFg5qM4h8L3es+gse3wP8BPgDSXslfTFtOHxxcwPh9HlRKn8p8ELLvz+S8t5G0iZJeyTtOcmJBd2EWSc4qM2gBk3OZcA1wIMRcTXw97zVvJyJZsibdosRsSUihiNieDl9uSprttgc1N6u65ucZE9YRyLimXT8GFmAe6nZlEyfR1vKX97y7y8DXiymumaLz0EtCWAy8qWStA1oEfF3wAuS3puybgSeA7YDG1LeBuDx9H07cEca7VwNHG82Tc26Vd/IhN/SAbVocgL8JvAVSfuAq4D/CtwL3CTpEHBTOgbYARwGGsAXgE8UWmOzkvSPjvf8PLUim5yS1kg6mKZ4TevGkvRpSc+l6V+7JP1Cu2suy/PDEfF9YHiGUzfOUDaAO/Nc16zbNOep9Y1MlF2VUhQ1gilpKfAA2cPQEeBZSdsj4rmWYnuB4Yh4XdK/BH4H+OezXdcrBczmqGf71Ip928a1QCMiDkfEm8AjZFO+3vq5iKcj4vV0uJusP35WDmhm89CLQS2bWBu5EnBhc1pWSpumXC7X9K4WG4E/bVfHXE1OM5uuGdR6qvmZ/00aL0fETN1UTbmmdwFI+nWyLq9/1u5H/YRmtgC99qQ2hye0dnJN75L0YeC3gVsiou0MfAc0swXqmaBWbB/as8CQpJWSzgFuJ5vydYakq4HfJwtmR2e4xjQOaGYF6I15asWt5YyIU8BdwE6yl108GhH7Jd0j6ZZU7L8B7wS+Lun7kraf5XJnuA/NrCD9o+Mc3zHIwFij7Kp0ToEvb4yIHWTzVlvzPtvy/cNzvaaf0MwKNDDWqG/zM/wKbrOeU+s+Nb+C26z31Dao1WQtp5nNUR2DmiYnc6WyOKCZdVCtglqQTazNk0rigGbWYXUJaiLfpNqcE2s7wgHNbBH0jUxwfMdg2dVYOA8KmBlkUzq6fvKtA5qZNfWPjndv89N9aGY2VTf3qXmU08ym6c6glrO5WeUmp6T3poWhzfRTSZ+SdL6kJyUdSp/npfKSdH96T/g+Sdd0/jbMuk/XBbWg+wNaRByMiKsi4irgl4DXgW+S7c25KyKGgF28tVfnzcBQSpuABztRcbM66LqgVrM+tBuB5yPib8je/70t5W8Dbk3f1wIPRWY3sKK5f6eZTddNQa1u89BuB76avl/c3G8zfV6U8uf6rnCzntc3MtEdW+R1e5OzKb1V8hbg6+2KzpA37Q4lbWpuoHCStm/WNau9wfV7qz1PLQJOT+ZLJZnLE9rNwPci4qV0/FKzKZk+m6/IzfWu8IjYEhHDETG8nL6519yshvpHx6sf1OrwhAas463mJmTv/96Qvm8AHm/JvyONdq4GjjebpmbWXqWDWh0CmqR/RLbD8Tdasu8FbpJ0KJ27N+XvAA4DDeALwCcKq61Zj6hkUAtgMvKlkuTaUyDtXnzBlLxXyEY9p5YN4M5CamfWw5pBrX90vOyqJAFR4pyMHLxSwKzCKvWkFtRqUMDMSlCtoFaDPjQzK1f/6Dg/2jpcdjUc0MysGFdu3FPySyJrsDjdzKpjYKxRXlALYHIyXyqJA5pZlyk3qPkJzcwKVk5Qq9fSJzOrkEUPagERk7lSWRzQzLrYoge1iq8UcEAz63IDYw2O/cnQ4vyY+9DMrNPO+8ihzs9Ti/Aop5ktjis37un8SyL9hGZmi2Vw/d4OBrUgTp/OlcrigGZWMx0Lal3w+iAHNLMa6lxQm8yXSuKAZlZTRQe1AGIycqU8JK2RdDDt4bt5hvN9kr6Wzj8j6Yp213RAM6uxQoNaRGFPaJKWAg+Q7VWyClgnadWUYhuBYxExCNwHfK7ddR3QzGpucP3ewqZ0FDgocC3QiIjDEfEm8AjZnr6tWvf+fQy4UdJMu8qdoShxiPVMJaSfAQfLrkcHXQi8XHYlOqTO9wb1ur9fiIh3z/cfS/oW2X+PPN4BvNFyvCUitrRc6zZgTUT8i3S8HvjliLirpcwPUpkj6fj5VOasf49cewosgoMRUYG313WGpD11vb863xvU//7mIiLWFHi5PPv35trjt5WbnGZWhjz7954pI2kZMAC8OttFHdDMrAzPAkOSVko6B7idbE/fVq17/94GPBVt+siq0uTc0r5IV6vz/dX53qD+91eKiDgl6S5gJ7AU+FJE7Jd0D7AnIrYDW4GHJTXInsxub3fdSgwKmJkVwU1OM6sNBzQzq43SA1q75Q9VJ+lySU9LOiBpv6RPpvzzJT0p6VD6PC/lS9L96X73Sbqm3DtoT9JSSXslPZGOV6alKIfS0pRzUv6cl6qUTdIKSY9J+mH6G15Xp79dryk1oOVc/lB1p4DPRMT7gNXAnekeNgO7ImII2JWOIbvXoZQ2AQ8ufpXn7JPAgZbjzwH3pXs7RrZEBeaxVKUCPg98KyJ+EfgA2X3W6W/XWyKitARcB+xsOb4buLvMOhVwT48DN5GtfLgk5V1CNnkY4PeBdS3lz5SrYiKbH7QLuAF4gmyy48vAsql/Q7IRq+vS92WpnMq+h1nu7V3A+NQ61uVv14up7CbnpcALLcdHUl5XSk2sq4FngIsj4scA6fOiVKzb7vn3gN8CmiuOLwBei4hT6bi1/mfuLZ0/nspX1XuAnwB/kJrUX5R0LvX52/WcsgPanJc2VJWkdwJ/BHwqIn46W9EZ8ip5z5J+BTgaEd9tzZ6haOQ4V0XLgGuAByPiauDveat5OZNuu7+eU3ZAy7P8ofIkLScLZl+JiG+k7JckXZLOXwIcTfnddM/XA7dImiB7G8INZE9sK9JSFHh7/ee8VKVkR4AjEfFMOn6MLMDV4W/Xk8oOaHmWP1Raep3JVuBARPxuy6nWZRsbyPrWmvl3pBGz1cDxZvOmaiLi7oi4LCKuIPvbPBURvwY8TbYUBabf25yWqpQpIv4OeEHSe1PWjcBz1OBv17PK7sQDxoAfAc8Dv112feZR/39C1uzYB3w/pTGyvqNdwKH0eX4qL7KR3eeBvwaGy76HnPf5QeCJ9P09wHeABvB1oC/lvyMdN9L595Rd7xz3dRWwJ/39/hg4r25/u15KXvpkZrVRdpPTzKwwDmhmVhsOaGZWGw5oZlYbDmhmVhsOaGZWGw5oZlYb/x+C1T0NgWO57gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "L = get_graph_laplacian(A.numpy())\n",
    "plt.imshow(L, vmin=0., vmax=1.)\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### MODEL\n",
    "##########################\n",
    "\n",
    "from scipy.sparse.linalg import eigsh\n",
    "        \n",
    "\n",
    "class GraphNet(nn.Module):\n",
    "    def __init__(self, img_size=28, num_filters=2, num_classes=10):\n",
    "        super(GraphNet, self).__init__()\n",
    "        \n",
    "        n_rows = img_size**2\n",
    "        self.fc = nn.Linear(n_rows*num_filters, num_classes, bias=False)\n",
    "\n",
    "        A = precompute_adjacency_matrix(img_size)\n",
    "        L = get_graph_laplacian(A.numpy())\n",
    "        Λ,V = eigsh(L.numpy(), k=20, which='SM') # eigen-decomposition (i.e. find Λ,V)\n",
    "\n",
    "        V = torch.from_numpy(V)\n",
    "        \n",
    "        # Weight matrix\n",
    "        W_spectral = nn.Parameter(torch.ones((img_size**2, num_filters))).float()\n",
    "        torch.nn.init.kaiming_uniform_(W_spectral)\n",
    "        \n",
    "        self.register_buffer('A', A)\n",
    "        self.register_buffer('L', L)\n",
    "        self.register_buffer('V', V)\n",
    "        self.register_buffer('W_spectral', W_spectral)\n",
    "\n",
    "        \n",
    "\n",
    "    def forward(self, x):\n",
    "        \n",
    "        B = x.size(0) # Batch size\n",
    "\n",
    "        ### Reshape eigenvectors\n",
    "        # from [H*W, 20] to [B, H*W, 20]\n",
    "        V_tensor = self.V.unsqueeze(0)\n",
    "        V_tensor = self.V.expand(B, -1, -1)\n",
    "        # from [H*W, 20] to [B, 20, H*W]\n",
    "        V_tensor_T = self.V.T.unsqueeze(0)\n",
    "        V_tensor_T = self.V.T.expand(B, -1, -1)\n",
    "        \n",
    "        ### Reshape inputs\n",
    "        # [B, C, H, W] => [B, H*W, 1]\n",
    "        x_reshape = x.view(B, -1, 1)\n",
    "        \n",
    "        ### Reshape spectral weights\n",
    "        # to size [128, H*W, F]\n",
    "        W_spectral_tensor = self.W_spectral.unsqueeze(0)\n",
    "        W_spectral_tensor = self.W_spectral.expand(B, -1, -1)\n",
    "        \n",
    "        ### Spectral convolution on graphs\n",
    "        # [B, 20, H*W] . [B, H*W, 1]  ==> [B, 20, 1]\n",
    "        X_hat = V_tensor_T.bmm(x_reshape) # 20×1 node features in the \"spectral\" domain\n",
    "        W_hat = V_tensor_T.bmm(W_spectral_tensor)  # 20×F filters in the \"spectral\" domain\n",
    "        Y = V_tensor.bmm(X_hat * W_hat)  # N×F result of convolution\n",
    "\n",
    "        ### Fully connected\n",
    "        logits = self.fc(Y.reshape(B, -1))\n",
    "        probas = F.softmax(logits, dim=1)\n",
    "        return logits, probas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(RANDOM_SEED)\n",
    "model = GraphNet(img_size=IMG_SIZE, num_classes=NUM_CLASSES)\n",
    "\n",
    "model = model.to(DEVICE)\n",
    "\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=LEARNING_RATE)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/050 | Batch 000/461 | Cost: 2.3133\n",
      "Epoch: 001/050 | Batch 150/461 | Cost: 1.1899\n",
      "Epoch: 001/050 | Batch 300/461 | Cost: 1.0481\n",
      "Epoch: 001/050 | Batch 450/461 | Cost: 0.9287\n",
      "Epoch: 001/050\n",
      "Train ACC: 73.79 | Validation ACC: 78.10\n",
      "Time elapsed: 0.07 min\n",
      "Epoch: 002/050 | Batch 000/461 | Cost: 0.8224\n",
      "Epoch: 002/050 | Batch 150/461 | Cost: 0.9684\n",
      "Epoch: 002/050 | Batch 300/461 | Cost: 0.6952\n",
      "Epoch: 002/050 | Batch 450/461 | Cost: 0.8158\n",
      "Epoch: 002/050\n",
      "Train ACC: 77.48 | Validation ACC: 82.20\n",
      "Time elapsed: 0.14 min\n",
      "Epoch: 003/050 | Batch 000/461 | Cost: 0.8203\n",
      "Epoch: 003/050 | Batch 150/461 | Cost: 0.8409\n",
      "Epoch: 003/050 | Batch 300/461 | Cost: 0.8602\n",
      "Epoch: 003/050 | Batch 450/461 | Cost: 0.7012\n",
      "Epoch: 003/050\n",
      "Train ACC: 78.55 | Validation ACC: 83.40\n",
      "Time elapsed: 0.21 min\n",
      "Epoch: 004/050 | Batch 000/461 | Cost: 0.7919\n",
      "Epoch: 004/050 | Batch 150/461 | Cost: 0.9010\n",
      "Epoch: 004/050 | Batch 300/461 | Cost: 0.6895\n",
      "Epoch: 004/050 | Batch 450/461 | Cost: 0.6981\n",
      "Epoch: 004/050\n",
      "Train ACC: 79.30 | Validation ACC: 84.10\n",
      "Time elapsed: 0.28 min\n",
      "Epoch: 005/050 | Batch 000/461 | Cost: 0.6080\n",
      "Epoch: 005/050 | Batch 150/461 | Cost: 0.6627\n",
      "Epoch: 005/050 | Batch 300/461 | Cost: 0.7620\n",
      "Epoch: 005/050 | Batch 450/461 | Cost: 0.8047\n",
      "Epoch: 005/050\n",
      "Train ACC: 79.66 | Validation ACC: 84.50\n",
      "Time elapsed: 0.35 min\n",
      "Epoch: 006/050 | Batch 000/461 | Cost: 0.5992\n",
      "Epoch: 006/050 | Batch 150/461 | Cost: 0.5546\n",
      "Epoch: 006/050 | Batch 300/461 | Cost: 0.6459\n",
      "Epoch: 006/050 | Batch 450/461 | Cost: 0.5968\n",
      "Epoch: 006/050\n",
      "Train ACC: 79.91 | Validation ACC: 85.10\n",
      "Time elapsed: 0.42 min\n",
      "Epoch: 007/050 | Batch 000/461 | Cost: 0.7909\n",
      "Epoch: 007/050 | Batch 150/461 | Cost: 0.6488\n",
      "Epoch: 007/050 | Batch 300/461 | Cost: 0.7580\n",
      "Epoch: 007/050 | Batch 450/461 | Cost: 0.5646\n",
      "Epoch: 007/050\n",
      "Train ACC: 80.50 | Validation ACC: 85.00\n",
      "Time elapsed: 0.48 min\n",
      "Epoch: 008/050 | Batch 000/461 | Cost: 0.6147\n",
      "Epoch: 008/050 | Batch 150/461 | Cost: 0.6998\n",
      "Epoch: 008/050 | Batch 300/461 | Cost: 0.5563\n",
      "Epoch: 008/050 | Batch 450/461 | Cost: 0.5611\n",
      "Epoch: 008/050\n",
      "Train ACC: 80.73 | Validation ACC: 85.60\n",
      "Time elapsed: 0.56 min\n",
      "Epoch: 009/050 | Batch 000/461 | Cost: 0.5629\n",
      "Epoch: 009/050 | Batch 150/461 | Cost: 0.6245\n",
      "Epoch: 009/050 | Batch 300/461 | Cost: 0.7393\n",
      "Epoch: 009/050 | Batch 450/461 | Cost: 0.6670\n",
      "Epoch: 009/050\n",
      "Train ACC: 81.09 | Validation ACC: 85.70\n",
      "Time elapsed: 0.62 min\n",
      "Epoch: 010/050 | Batch 000/461 | Cost: 0.6582\n",
      "Epoch: 010/050 | Batch 150/461 | Cost: 0.7550\n",
      "Epoch: 010/050 | Batch 300/461 | Cost: 0.7028\n",
      "Epoch: 010/050 | Batch 450/461 | Cost: 0.6558\n",
      "Epoch: 010/050\n",
      "Train ACC: 81.00 | Validation ACC: 85.70\n",
      "Time elapsed: 0.69 min\n",
      "Epoch: 011/050 | Batch 000/461 | Cost: 0.5472\n",
      "Epoch: 011/050 | Batch 150/461 | Cost: 0.6051\n",
      "Epoch: 011/050 | Batch 300/461 | Cost: 0.5875\n",
      "Epoch: 011/050 | Batch 450/461 | Cost: 0.4688\n",
      "Epoch: 011/050\n",
      "Train ACC: 81.50 | Validation ACC: 85.90\n",
      "Time elapsed: 0.76 min\n",
      "Epoch: 012/050 | Batch 000/461 | Cost: 0.5227\n",
      "Epoch: 012/050 | Batch 150/461 | Cost: 0.6252\n",
      "Epoch: 012/050 | Batch 300/461 | Cost: 0.6359\n",
      "Epoch: 012/050 | Batch 450/461 | Cost: 0.8590\n",
      "Epoch: 012/050\n",
      "Train ACC: 81.61 | Validation ACC: 86.50\n",
      "Time elapsed: 0.83 min\n",
      "Epoch: 013/050 | Batch 000/461 | Cost: 0.4933\n",
      "Epoch: 013/050 | Batch 150/461 | Cost: 0.5844\n",
      "Epoch: 013/050 | Batch 300/461 | Cost: 0.4684\n",
      "Epoch: 013/050 | Batch 450/461 | Cost: 0.5275\n",
      "Epoch: 013/050\n",
      "Train ACC: 81.79 | Validation ACC: 86.50\n",
      "Time elapsed: 0.90 min\n",
      "Epoch: 014/050 | Batch 000/461 | Cost: 0.6382\n",
      "Epoch: 014/050 | Batch 150/461 | Cost: 0.7612\n",
      "Epoch: 014/050 | Batch 300/461 | Cost: 0.5378\n",
      "Epoch: 014/050 | Batch 450/461 | Cost: 0.5651\n",
      "Epoch: 014/050\n",
      "Train ACC: 81.94 | Validation ACC: 86.50\n",
      "Time elapsed: 0.97 min\n",
      "Epoch: 015/050 | Batch 000/461 | Cost: 0.5122\n",
      "Epoch: 015/050 | Batch 150/461 | Cost: 0.6347\n",
      "Epoch: 015/050 | Batch 300/461 | Cost: 0.6239\n",
      "Epoch: 015/050 | Batch 450/461 | Cost: 0.6026\n",
      "Epoch: 015/050\n",
      "Train ACC: 82.01 | Validation ACC: 87.00\n",
      "Time elapsed: 1.03 min\n",
      "Epoch: 016/050 | Batch 000/461 | Cost: 0.6380\n",
      "Epoch: 016/050 | Batch 150/461 | Cost: 0.5865\n",
      "Epoch: 016/050 | Batch 300/461 | Cost: 0.3510\n",
      "Epoch: 016/050 | Batch 450/461 | Cost: 0.5859\n",
      "Epoch: 016/050\n",
      "Train ACC: 82.06 | Validation ACC: 86.50\n",
      "Time elapsed: 1.10 min\n",
      "Epoch: 017/050 | Batch 000/461 | Cost: 0.6827\n",
      "Epoch: 017/050 | Batch 150/461 | Cost: 0.6415\n",
      "Epoch: 017/050 | Batch 300/461 | Cost: 0.7186\n",
      "Epoch: 017/050 | Batch 450/461 | Cost: 0.6067\n",
      "Epoch: 017/050\n",
      "Train ACC: 82.41 | Validation ACC: 87.70\n",
      "Time elapsed: 1.17 min\n",
      "Epoch: 018/050 | Batch 000/461 | Cost: 0.7209\n",
      "Epoch: 018/050 | Batch 150/461 | Cost: 0.6981\n",
      "Epoch: 018/050 | Batch 300/461 | Cost: 0.6810\n",
      "Epoch: 018/050 | Batch 450/461 | Cost: 0.6180\n",
      "Epoch: 018/050\n",
      "Train ACC: 82.55 | Validation ACC: 87.50\n",
      "Time elapsed: 1.24 min\n",
      "Epoch: 019/050 | Batch 000/461 | Cost: 0.7285\n",
      "Epoch: 019/050 | Batch 150/461 | Cost: 0.7734\n",
      "Epoch: 019/050 | Batch 300/461 | Cost: 0.7189\n",
      "Epoch: 019/050 | Batch 450/461 | Cost: 0.5652\n",
      "Epoch: 019/050\n",
      "Train ACC: 82.46 | Validation ACC: 87.30\n",
      "Time elapsed: 1.31 min\n",
      "Epoch: 020/050 | Batch 000/461 | Cost: 0.7076\n",
      "Epoch: 020/050 | Batch 150/461 | Cost: 0.4096\n",
      "Epoch: 020/050 | Batch 300/461 | Cost: 0.7485\n",
      "Epoch: 020/050 | Batch 450/461 | Cost: 0.7334\n",
      "Epoch: 020/050\n",
      "Train ACC: 82.48 | Validation ACC: 87.30\n",
      "Time elapsed: 1.38 min\n",
      "Epoch: 021/050 | Batch 000/461 | Cost: 0.4686\n",
      "Epoch: 021/050 | Batch 150/461 | Cost: 0.6241\n",
      "Epoch: 021/050 | Batch 300/461 | Cost: 0.5736\n",
      "Epoch: 021/050 | Batch 450/461 | Cost: 0.4948\n",
      "Epoch: 021/050\n",
      "Train ACC: 82.67 | Validation ACC: 88.00\n",
      "Time elapsed: 1.45 min\n",
      "Epoch: 022/050 | Batch 000/461 | Cost: 0.4657\n",
      "Epoch: 022/050 | Batch 150/461 | Cost: 0.6718\n",
      "Epoch: 022/050 | Batch 300/461 | Cost: 0.6647\n",
      "Epoch: 022/050 | Batch 450/461 | Cost: 0.4913\n",
      "Epoch: 022/050\n",
      "Train ACC: 82.87 | Validation ACC: 87.90\n",
      "Time elapsed: 1.52 min\n",
      "Epoch: 023/050 | Batch 000/461 | Cost: 0.5567\n",
      "Epoch: 023/050 | Batch 150/461 | Cost: 0.4976\n",
      "Epoch: 023/050 | Batch 300/461 | Cost: 0.5911\n",
      "Epoch: 023/050 | Batch 450/461 | Cost: 0.4014\n",
      "Epoch: 023/050\n",
      "Train ACC: 82.91 | Validation ACC: 87.80\n",
      "Time elapsed: 1.59 min\n",
      "Epoch: 024/050 | Batch 000/461 | Cost: 0.5728\n",
      "Epoch: 024/050 | Batch 150/461 | Cost: 0.6313\n",
      "Epoch: 024/050 | Batch 300/461 | Cost: 0.5825\n",
      "Epoch: 024/050 | Batch 450/461 | Cost: 0.4720\n",
      "Epoch: 024/050\n",
      "Train ACC: 83.00 | Validation ACC: 87.90\n",
      "Time elapsed: 1.66 min\n",
      "Epoch: 025/050 | Batch 000/461 | Cost: 0.5128\n",
      "Epoch: 025/050 | Batch 150/461 | Cost: 0.4793\n",
      "Epoch: 025/050 | Batch 300/461 | Cost: 0.7191\n",
      "Epoch: 025/050 | Batch 450/461 | Cost: 0.5402\n",
      "Epoch: 025/050\n",
      "Train ACC: 83.12 | Validation ACC: 88.30\n",
      "Time elapsed: 1.72 min\n",
      "Epoch: 026/050 | Batch 000/461 | Cost: 0.4961\n",
      "Epoch: 026/050 | Batch 150/461 | Cost: 0.4546\n",
      "Epoch: 026/050 | Batch 300/461 | Cost: 0.5333\n",
      "Epoch: 026/050 | Batch 450/461 | Cost: 0.5073\n",
      "Epoch: 026/050\n",
      "Train ACC: 82.98 | Validation ACC: 87.90\n",
      "Time elapsed: 1.79 min\n",
      "Epoch: 027/050 | Batch 000/461 | Cost: 0.7034\n",
      "Epoch: 027/050 | Batch 150/461 | Cost: 0.5373\n",
      "Epoch: 027/050 | Batch 300/461 | Cost: 0.5158\n",
      "Epoch: 027/050 | Batch 450/461 | Cost: 0.5705\n",
      "Epoch: 027/050\n",
      "Train ACC: 83.15 | Validation ACC: 88.00\n",
      "Time elapsed: 1.86 min\n",
      "Epoch: 028/050 | Batch 000/461 | Cost: 0.4614\n",
      "Epoch: 028/050 | Batch 150/461 | Cost: 0.4124\n",
      "Epoch: 028/050 | Batch 300/461 | Cost: 0.7368\n",
      "Epoch: 028/050 | Batch 450/461 | Cost: 0.5744\n",
      "Epoch: 028/050\n",
      "Train ACC: 82.85 | Validation ACC: 87.60\n",
      "Time elapsed: 1.93 min\n",
      "Epoch: 029/050 | Batch 000/461 | Cost: 0.5026\n",
      "Epoch: 029/050 | Batch 150/461 | Cost: 0.6048\n",
      "Epoch: 029/050 | Batch 300/461 | Cost: 0.6400\n",
      "Epoch: 029/050 | Batch 450/461 | Cost: 0.4906\n",
      "Epoch: 029/050\n",
      "Train ACC: 83.26 | Validation ACC: 88.10\n",
      "Time elapsed: 2.00 min\n",
      "Epoch: 030/050 | Batch 000/461 | Cost: 0.6298\n",
      "Epoch: 030/050 | Batch 150/461 | Cost: 0.5472\n",
      "Epoch: 030/050 | Batch 300/461 | Cost: 0.5469\n",
      "Epoch: 030/050 | Batch 450/461 | Cost: 0.4819\n",
      "Epoch: 030/050\n",
      "Train ACC: 83.30 | Validation ACC: 88.70\n",
      "Time elapsed: 2.07 min\n",
      "Epoch: 031/050 | Batch 000/461 | Cost: 0.6101\n",
      "Epoch: 031/050 | Batch 150/461 | Cost: 0.5150\n",
      "Epoch: 031/050 | Batch 300/461 | Cost: 0.5505\n",
      "Epoch: 031/050 | Batch 450/461 | Cost: 0.5634\n",
      "Epoch: 031/050\n",
      "Train ACC: 83.28 | Validation ACC: 88.60\n",
      "Time elapsed: 2.13 min\n",
      "Epoch: 032/050 | Batch 000/461 | Cost: 0.5655\n",
      "Epoch: 032/050 | Batch 150/461 | Cost: 0.6567\n",
      "Epoch: 032/050 | Batch 300/461 | Cost: 0.5758\n",
      "Epoch: 032/050 | Batch 450/461 | Cost: 0.5306\n",
      "Epoch: 032/050\n",
      "Train ACC: 83.31 | Validation ACC: 88.20\n",
      "Time elapsed: 2.20 min\n",
      "Epoch: 033/050 | Batch 000/461 | Cost: 0.6677\n",
      "Epoch: 033/050 | Batch 150/461 | Cost: 0.7450\n",
      "Epoch: 033/050 | Batch 300/461 | Cost: 0.5538\n",
      "Epoch: 033/050 | Batch 450/461 | Cost: 0.5642\n",
      "Epoch: 033/050\n",
      "Train ACC: 83.33 | Validation ACC: 88.40\n",
      "Time elapsed: 2.27 min\n",
      "Epoch: 034/050 | Batch 000/461 | Cost: 0.6287\n",
      "Epoch: 034/050 | Batch 150/461 | Cost: 0.4752\n",
      "Epoch: 034/050 | Batch 300/461 | Cost: 0.5957\n",
      "Epoch: 034/050 | Batch 450/461 | Cost: 0.4531\n",
      "Epoch: 034/050\n",
      "Train ACC: 83.50 | Validation ACC: 88.70\n",
      "Time elapsed: 2.34 min\n",
      "Epoch: 035/050 | Batch 000/461 | Cost: 0.5368\n",
      "Epoch: 035/050 | Batch 150/461 | Cost: 0.5658\n",
      "Epoch: 035/050 | Batch 300/461 | Cost: 0.6598\n",
      "Epoch: 035/050 | Batch 450/461 | Cost: 0.5858\n",
      "Epoch: 035/050\n",
      "Train ACC: 83.59 | Validation ACC: 88.50\n",
      "Time elapsed: 2.41 min\n",
      "Epoch: 036/050 | Batch 000/461 | Cost: 0.5557\n",
      "Epoch: 036/050 | Batch 150/461 | Cost: 0.4680\n",
      "Epoch: 036/050 | Batch 300/461 | Cost: 0.4905\n",
      "Epoch: 036/050 | Batch 450/461 | Cost: 0.9074\n",
      "Epoch: 036/050\n",
      "Train ACC: 83.67 | Validation ACC: 88.50\n",
      "Time elapsed: 2.48 min\n",
      "Epoch: 037/050 | Batch 000/461 | Cost: 0.6120\n",
      "Epoch: 037/050 | Batch 150/461 | Cost: 0.4668\n",
      "Epoch: 037/050 | Batch 300/461 | Cost: 0.5836\n",
      "Epoch: 037/050 | Batch 450/461 | Cost: 0.4536\n",
      "Epoch: 037/050\n",
      "Train ACC: 83.35 | Validation ACC: 88.80\n",
      "Time elapsed: 2.55 min\n",
      "Epoch: 038/050 | Batch 000/461 | Cost: 0.5380\n",
      "Epoch: 038/050 | Batch 150/461 | Cost: 0.4491\n",
      "Epoch: 038/050 | Batch 300/461 | Cost: 0.4500\n",
      "Epoch: 038/050 | Batch 450/461 | Cost: 0.6041\n",
      "Epoch: 038/050\n",
      "Train ACC: 83.69 | Validation ACC: 88.80\n",
      "Time elapsed: 2.61 min\n",
      "Epoch: 039/050 | Batch 000/461 | Cost: 0.4863\n",
      "Epoch: 039/050 | Batch 150/461 | Cost: 0.5673\n",
      "Epoch: 039/050 | Batch 300/461 | Cost: 0.4037\n",
      "Epoch: 039/050 | Batch 450/461 | Cost: 0.6392\n",
      "Epoch: 039/050\n",
      "Train ACC: 83.71 | Validation ACC: 88.70\n",
      "Time elapsed: 2.68 min\n",
      "Epoch: 040/050 | Batch 000/461 | Cost: 0.6707\n",
      "Epoch: 040/050 | Batch 150/461 | Cost: 0.5601\n",
      "Epoch: 040/050 | Batch 300/461 | Cost: 0.5265\n",
      "Epoch: 040/050 | Batch 450/461 | Cost: 0.4867\n",
      "Epoch: 040/050\n",
      "Train ACC: 83.76 | Validation ACC: 88.90\n",
      "Time elapsed: 2.75 min\n",
      "Epoch: 041/050 | Batch 000/461 | Cost: 0.5379\n",
      "Epoch: 041/050 | Batch 150/461 | Cost: 0.4588\n",
      "Epoch: 041/050 | Batch 300/461 | Cost: 0.5684\n",
      "Epoch: 041/050 | Batch 450/461 | Cost: 0.5547\n",
      "Epoch: 041/050\n",
      "Train ACC: 83.75 | Validation ACC: 88.60\n",
      "Time elapsed: 2.82 min\n",
      "Epoch: 042/050 | Batch 000/461 | Cost: 0.5714\n",
      "Epoch: 042/050 | Batch 150/461 | Cost: 0.3863\n",
      "Epoch: 042/050 | Batch 300/461 | Cost: 0.5142\n",
      "Epoch: 042/050 | Batch 450/461 | Cost: 0.6219\n",
      "Epoch: 042/050\n",
      "Train ACC: 83.79 | Validation ACC: 89.20\n",
      "Time elapsed: 2.89 min\n",
      "Epoch: 043/050 | Batch 000/461 | Cost: 0.5385\n",
      "Epoch: 043/050 | Batch 150/461 | Cost: 0.4801\n",
      "Epoch: 043/050 | Batch 300/461 | Cost: 0.6064\n",
      "Epoch: 043/050 | Batch 450/461 | Cost: 0.4959\n",
      "Epoch: 043/050\n",
      "Train ACC: 83.89 | Validation ACC: 88.80\n",
      "Time elapsed: 2.96 min\n",
      "Epoch: 044/050 | Batch 000/461 | Cost: 0.6742\n",
      "Epoch: 044/050 | Batch 150/461 | Cost: 0.5746\n",
      "Epoch: 044/050 | Batch 300/461 | Cost: 0.6846\n",
      "Epoch: 044/050 | Batch 450/461 | Cost: 0.6283\n",
      "Epoch: 044/050\n",
      "Train ACC: 83.91 | Validation ACC: 89.00\n",
      "Time elapsed: 3.03 min\n",
      "Epoch: 045/050 | Batch 000/461 | Cost: 0.5646\n",
      "Epoch: 045/050 | Batch 150/461 | Cost: 0.3776\n",
      "Epoch: 045/050 | Batch 300/461 | Cost: 0.5457\n",
      "Epoch: 045/050 | Batch 450/461 | Cost: 0.4897\n",
      "Epoch: 045/050\n",
      "Train ACC: 83.87 | Validation ACC: 89.10\n",
      "Time elapsed: 3.10 min\n",
      "Epoch: 046/050 | Batch 000/461 | Cost: 0.5300\n",
      "Epoch: 046/050 | Batch 150/461 | Cost: 0.6787\n",
      "Epoch: 046/050 | Batch 300/461 | Cost: 0.4310\n",
      "Epoch: 046/050 | Batch 450/461 | Cost: 0.5758\n",
      "Epoch: 046/050\n",
      "Train ACC: 84.01 | Validation ACC: 89.10\n",
      "Time elapsed: 3.17 min\n",
      "Epoch: 047/050 | Batch 000/461 | Cost: 0.6111\n",
      "Epoch: 047/050 | Batch 150/461 | Cost: 0.5679\n",
      "Epoch: 047/050 | Batch 300/461 | Cost: 0.6306\n",
      "Epoch: 047/050 | Batch 450/461 | Cost: 0.7292\n",
      "Epoch: 047/050\n",
      "Train ACC: 84.03 | Validation ACC: 89.20\n",
      "Time elapsed: 3.24 min\n",
      "Epoch: 048/050 | Batch 000/461 | Cost: 0.5925\n",
      "Epoch: 048/050 | Batch 150/461 | Cost: 0.6623\n",
      "Epoch: 048/050 | Batch 300/461 | Cost: 0.4188\n",
      "Epoch: 048/050 | Batch 450/461 | Cost: 0.3433\n",
      "Epoch: 048/050\n",
      "Train ACC: 83.89 | Validation ACC: 89.10\n",
      "Time elapsed: 3.31 min\n",
      "Epoch: 049/050 | Batch 000/461 | Cost: 0.4881\n",
      "Epoch: 049/050 | Batch 150/461 | Cost: 0.5040\n",
      "Epoch: 049/050 | Batch 300/461 | Cost: 0.5655\n",
      "Epoch: 049/050 | Batch 450/461 | Cost: 0.5264\n",
      "Epoch: 049/050\n",
      "Train ACC: 83.83 | Validation ACC: 88.60\n",
      "Time elapsed: 3.38 min\n",
      "Epoch: 050/050 | Batch 000/461 | Cost: 0.5284\n",
      "Epoch: 050/050 | Batch 150/461 | Cost: 0.6253\n",
      "Epoch: 050/050 | Batch 300/461 | Cost: 0.3891\n",
      "Epoch: 050/050 | Batch 450/461 | Cost: 0.4316\n",
      "Epoch: 050/050\n",
      "Train ACC: 83.90 | Validation ACC: 88.70\n",
      "Time elapsed: 3.45 min\n",
      "Total Training Time: 3.45 min\n"
     ]
    }
   ],
   "source": [
    "def compute_acc(model, data_loader, device):\n",
    "    correct_pred, num_examples = 0, 0\n",
    "    for features, targets in data_loader:\n",
    "        features = features.to(device)\n",
    "        targets = targets.to(device)\n",
    "        logits, probas = model(features)\n",
    "        _, predicted_labels = torch.max(probas, 1)\n",
    "        num_examples += targets.size(0)\n",
    "        correct_pred += (predicted_labels == targets).sum()\n",
    "    return correct_pred.float()/num_examples * 100\n",
    "    \n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "cost_list = []\n",
    "train_acc_list, valid_acc_list = [], []\n",
    "\n",
    "\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    \n",
    "    model.train()\n",
    "    for batch_idx, (features, targets) in enumerate(train_loader):\n",
    "        \n",
    "        features = features.to(DEVICE)\n",
    "        targets = targets.to(DEVICE)\n",
    "            \n",
    "        ### FORWARD AND BACK PROP\n",
    "        logits, probas = model(features)\n",
    "        cost = F.cross_entropy(logits, targets)\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        cost.backward()\n",
    "        \n",
    "        ### UPDATE MODEL PARAMETERS\n",
    "        optimizer.step()\n",
    "        \n",
    "        #################################################\n",
    "        ### CODE ONLY FOR LOGGING BEYOND THIS POINT\n",
    "        ################################################\n",
    "        cost_list.append(cost.item())\n",
    "        if not batch_idx % 150:\n",
    "            print (f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d} | '\n",
    "                   f'Batch {batch_idx:03d}/{len(train_loader):03d} |' \n",
    "                   f' Cost: {cost:.4f}')\n",
    "\n",
    "        \n",
    "\n",
    "    model.eval()\n",
    "    with torch.set_grad_enabled(False): # save memory during inference\n",
    "        \n",
    "        train_acc = compute_acc(model, train_loader, device=DEVICE)\n",
    "        valid_acc = compute_acc(model, valid_loader, device=DEVICE)\n",
    "        \n",
    "        print(f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d}\\n'\n",
    "              f'Train ACC: {train_acc:.2f} | Validation ACC: {valid_acc:.2f}')\n",
    "        \n",
    "        train_acc_list.append(train_acc)\n",
    "        valid_acc_list.append(valid_acc)\n",
    "        \n",
    "    elapsed = (time.time() - start_time)/60\n",
    "    print(f'Time elapsed: {elapsed:.2f} min')\n",
    "  \n",
    "elapsed = (time.time() - start_time)/60\n",
    "print(f'Total Training Time: {elapsed:.2f} min')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3hUZdrA4d8zqfTeEQNIkRIQQpGOIlWxK8i6IiquiFhWXdaCCuqy1g9URFQUG+JaUbAAghQBCQpIkxokghBAQk1/vz+mZMqZkjKZEJ77unIxc+o7h5nznLeLMQallFLKmy3SCVBKKVU6aYBQSillSQOEUkopSxoglFJKWdIAoZRSylJ0pBNQnGrWrGkSEhIinQyllDpjrF279pAxppbVujIVIBISEkhOTo50MpRS6owhInv8rdMiJqWUUpY0QCillLKkAUIppZSlMlUHoZQqvOzsbFJTU8nIyIh0UlQYxMfH07BhQ2JiYkLeRwOEUgqA1NRUKlWqREJCAiIS6eSoYmSM4fDhw6SmptK4ceOQ99MiJqUUABkZGdSoUUODQxkkItSoUaPAuUMNEEopFw0OZVdh/m81QABTF23nh21pkU6GUkqVKhoggFeX7GTFjkORToZSZz0R4cYbb3S9z8nJoVatWlx66aUAzJ07l8mTJwc8xr59+7jmmmsAePvttxk7dmyB0vD0008H3WbkyJF8/PHHBTpuYaxbt4758+eH/Tz+aIAARCAvTydOUirSKlSowMaNGzl9+jQACxYsoEGDBq71Q4cOZfz48QGPUb9+/SLdvEMJECVFA0QpYBNBw4NSpcOgQYOYN28eALNnz2b48OGude45gpEjRzJu3Di6detGkyZNXEEhJSWFNm3auPbZu3cvAwcOpEWLFjzxxBOu5VdccQUdO3akdevWzJgxA4Dx48dz+vRp2rdvz4gRIwB45513SExMpF27dh65m6VLl/qc25vVvnv27OHiiy8mMTGRiy++mN9//x2A//3vf7Rp04Z27drRq1cvsrKymDBhAnPmzKF9+/bMmTOnaBe2EMLWzFVEzgHeAeoCecAMY8wUr21GAP9yvD0B3GGMWe9YlwIcB3KBHGNMUtjSCuTp1KtKuTzx5SY27ztWrMdsVb8yj13WOuh2w4YNY+LEiVx66aVs2LCBUaNGsWzZMstt9+/fz/Lly9m6dStDhw51FS25++mnn9i4cSPly5enU6dODBkyhKSkJGbOnEn16tU5ffo0nTp14uqrr2by5Mm8/PLLrFu3DoBNmzbx1FNPsWLFCmrWrMmRI0dCPre/fceOHcvf//53brrpJmbOnMm4ceP4/PPPmThxIt9++y0NGjTg6NGjxMbGMnHiRJKTk3n55ZdDvs7FKZw5iBzgn8aY84GuwJ0i0sprm91Ab2NMIjAJmOG1vq8xpn04gwMAAhoflCodEhMTSUlJYfbs2QwePDjgtldccQU2m41WrVpx4MABy20uueQSatSoQbly5bjqqqtYvnw5AFOnTqVdu3Z07dqVvXv3sn37dp99v//+e6655hpq1qwJQPXq1UM+t799V65cyQ033ADAjTfe6EpP9+7dGTlyJK+//jq5ubkBP3dJCVsOwhizH9jveH1cRLYADYDNbtv86LbLKqBhuNITiE2b9inlIZQn/XAaOnQo999/P0uWLOHw4cN+t4uLi3O9Nn6e8rybd4oIS5YsYeHChaxcuZLy5cvTp08fyz4Cxhi/zUODnTvQvlbpmz59OqtXr2bevHm0b9/elYuJpBKpgxCRBOACYHWAzW4BvnZ7b4DvRGStiIwOcOzRIpIsIslpaYVrqiqiRUxKlSajRo1iwoQJtG3btsjHWrBgAUeOHOH06dN8/vnndO/enfT0dKpVq0b58uXZunUrq1atcm0fExNDdnY2ABdffDEfffSRK0i5FzEF42/fbt268eGHHwLw/vvv06NHDwB27txJly5dmDhxIjVr1mTv3r1UqlSJ48ePF/kaFFbYA4SIVAQ+Ae4xxlgWaopIX+wB4l9ui7sbYzoAg7AXT/Wy2tcYM8MYk2SMSapVy3LOi6BsIlrEpFQp0rBhQ+6+++5iOVaPHj248cYbad++PVdffTVJSUkMHDiQnJwcEhMTefTRR+natatr+9GjR5OYmMiIESNo3bo1Dz/8ML1796Zdu3bcd999IZ/X375Tp07lrbfeIjExkXfffZcpU+xVsw888ABt27alTZs29OrVi3bt2tG3b182b94csUpq8ZctK5aDi8QAXwHfGmNe8LNNIvAZMMgYs83PNo8DJ4wxzwU6X1JSkinMhEEdJy1gYJu6PHVl0Z9WlDpTbdmyhfPPPz/SyVBhZPV/LCJr/dXzhi0HIfaCtTeBLQGCQyPgU+BG9+AgIhVEpJLzNdAf2BjGtGozV6WU8hLO0Vy7AzcCv4qIs7blIaARgDFmOjABqAFMc1TUOJuz1gE+cyyLBj4wxnwTroSK+K/gUkqps1U4WzEtx97FINA2twK3WizfBbQLU9J8CNrMVSmlvGlParSSWimlrGiAQJu5KqWUFQ0Q6FhMSillRQOEg+YglIq8qKgo2rdvT5s2bbjssss4evRoWM7TrVu3sBy3rNEAAdhsoFkIpSKvXLlyrFu3jo0bN1K9enVeeeWVsJznxx9/DL5RhJSWcZhAAwQAgmgOQqlS5sILL+SPP/4AYMmSJa5Jg8A+Iurbb78NQEJCAo899hgdOnSgbdu2bN26FYDHH3+cUaNG0adPH5o0acLUqVNd+1esWNF13D59+nDNNdfQsmVLRowY4WryPn/+fFq2bEmPHj0YN26cx/mdUlJS6NmzJx06dKBDhw6uwHP99dd7zOMwcuRIPvnkE3Jzc3nggQfo1KkTiYmJvPbaa6509O3blxtuuME1vIjVcOQAb775Js2bN6dPnz7cdtttruHP09LSuPrqq+nUqROdOnVixYoVRbj6duHsB3HGENEMhFIevh4Pf/5avMes2xYGBZ4Nzik3N5dFixZxyy23hLR9zZo1+fnnn5k2bRrPPfccb7zxBgBbt25l8eLFHD9+nBYtWnDHHXcQExPjse8vv/zCpk2bqF+/Pt27d2fFihUkJSVx++23s3TpUho3buwxJ4W72rVrs2DBAuLj49m+fTvDhw8nOTmZYcOGMWfOHAYPHkxWVhaLFi3i1Vdf5c0336RKlSqsWbOGzMxMunfvTv/+/YH8YckbN24MYDkceWZmJpMmTeLnn3+mUqVKXHTRRbRrZ+8RcPfdd3PvvffSo0cPfv/9dwYMGMCWLVtCun7+aIBAm7kqVVo4J+tJSUmhY8eOXHLJJSHtd9VVVwHQsWNHPv30U9fyIUOGEBcXR1xcHLVr1+bAgQM0bOg5aHTnzp1dy5znrlixIk2aNHHdrIcPH+7xFO+UnZ3N2LFjWbduHVFRUWzbZh8QYtCgQYwbN47MzEy++eYbevXqRbly5fjuu+/YsGGDa4Kh9PR0tm/fTmxsLJ07d3adD+xjNn322WcAruHI//zzT3r37u0aOvzaa691nXPhwoVs3uwaLJtjx45x/PhxKlWqFNI1tKIBAp0wSCkfIT7pFzdnHUR6ejqXXnopr7zyCuPGjSM6Opq8vDzXdt5DczuH3o6KiiInJ8dnudW6QNuEOrLCiy++SJ06dVi/fj15eXnEx8cDEB8fT58+ffj222+ZM2eOKwdijOGll15iwIABHsdZsmQJFSpU8HhvNRx5oHTl5eWxcuVKypUrF1LaQ6F1EGgRk1KlTZUqVZg6dSrPPfcc2dnZnHvuuWzevJnMzEzS09NZtGhRWM/fsmVLdu3aRUpKCoDfkVTT09OpV68eNpuNd99916OCediwYbz11lssW7bMFRAGDBjAq6++6hpOfNu2bZw8edLyuFbDkXfu3JkffviBv/76i5ycHD755BPXPv379/eYea445pPQAIFjsD7NQShVqlxwwQW0a9eODz/8kHPOOYfrrrvONQz3BRdcENZzlytXjmnTpjFw4EB69OhBnTp1qFKlis92Y8aMYdasWXTt2pVt27Z55AL69+/P0qVL6devH7GxsQDceuuttGrVig4dOtCmTRtuv/12y1yNv+HIGzRowEMPPUSXLl3o168frVq1cqVr6tSpJCcnk5iYSKtWrZg+fXqRr0NYh/suaYUd7rv/iz/QtFZFXv1bxzCkSqkzgw737enEiRNUrFgRYwx33nknzZo149577410slzpysnJ4corr2TUqFFceeWVIe1baob7PpNoM1ellLfXX3+d9u3b07p1a9LT07n99tsjnSTA3nzX2ZmwcePGXHHFFWE7l1ZS4xzuO9KpUEqVJvfee2+pyDF4e+65gPOmFSvNQWCvg8jTAKGU1sWVYYX5v9UAgXPSCv1hqLNbfHw8hw8f1iBRBhljOHz4sKsZbqjCVsQkIucA7wB1gTxghjFmitc2AkwBBgOngJHGmJ8d624CHnFs+qQxZla40mqzaRGTUg0bNiQ1NZW0tLRIJ0WFQXx8vE8nwWDCWQeRA/zTGPOzY37ptSKywBiz2W2bQUAzx18X4FWgi4hUBx4DkrA/2q8VkbnGmL/CkVCtpFYKYmJiPHryKhW2IiZjzH5nbsAYcxzYAjTw2uxy4B1jtwqoKiL1gAHAAmPMEUdQWAAMDFdabdpRTimlfJRIHYSIJAAXAKu9VjUA9rq9T3Us87fc6tijRSRZRJILnTXWSmqllPIR9gAhIhWBT4B7jDHHvFdb7GICLPddaMwMY0ySMSapVq1ahUqjTbT1hlJKeQtrgBCRGOzB4X1jzKcWm6QC57i9bwjsC7A8POlEK6mVUspb2AKEo4XSm8AWY8wLfjabC/xd7LoC6caY/cC3QH8RqSYi1YD+jmXhSitGayGUUspDOFsxdQduBH4VEeewgg8BjQCMMdOB+dibuO7A3sz1Zse6IyIyCVjj2G+iMeZIuBJq057USinlI2wBwhizHOu6BPdtDHCnn3UzgZlhSJoPbeaqlFK+tCc1OhaTUkpZ0QCBBgillLKiAQLHnNRaSa2UUh40QGDPQWhHOaWU8qQBAnsltXaUU0opTxogcNRBRDoRSilVymiAQCcMUkopKxogsHeU02ZMSinlSQME9t58moNQSilPGiDQZq5KKWVFAwSOZq55kU6FUkqVLhogABDNPyillJdwjuZ6xli45UCkk6CUUqWO5iCUUkpZ0gChlFLKUtiKmERkJnApcNAY08Zi/QPACLd0nA/UckwWlAIcB3KBHGNMUrjSqZRSylo4cxBvAwP9rTTGPGuMaW+MaQ/8G/jBa9a4vo71GhyUUioCwhYgjDFLgVCnCR0OzA5XWpRSShVcxOsgRKQ89pzGJ26LDfCdiKwVkdFB9h8tIskikpyWlhbOpCql1Fkl4gECuAxY4VW81N0Y0wEYBNwpIr387WyMmWGMSTLGJNWqVSvcaVVKqbNGaQgQw/AqXjLG7HP8exD4DOgcgXQppdRZLaIBQkSqAL2BL9yWVRCRSs7XQH9gY2RSqJRSZ69wNnOdDfQBaopIKvAYEANgjJnu2OxK4DtjzEm3XesAn4mIM30fGGO+CVc6lVJKWQtbgDDGDA9hm7exN4d1X7YLaBeeVAVND47ApJRSZ73SUAdRauicEEoplU8DhJs8nVVOKaVcNEC40QChlFL5NEAAd110HqDTUiullDsNEECFOHtdveYglFIqX9AAISJRJZGQSLI5Gi5pfFBKqXyh5CB2iMizItIq7KmJkNNZ9gmps3J0YmqllHIKJUAkAtuAN0RklWNwvMphTleJenHhNgBumbUmwilRSqnSI2iAMMYcN8a8bozpBjyIvUf0fhGZJSLnhT2FJejn349GOglKKVVqhFQHISJDReQzYArwPNAE+BKYH+b0KaWUipBQhtrYDiwGnjXG/Oi2/ONAw3ArpZQ6s4USIBKNMSesVhhjxhVzepRSSpUSoVRS1xaRL0XkkIgcFJEvRKRJ2FOmlFIqokIJEB8AHwF1gfrA/9D5o5VSqswLJUCIMeZdY0yO4+897HNGK6WUKsNCCRCLRWS8iCSIyLki8iAwT0Sqi0h1fzuJyExHkZTlbHAi0kdE0kVkneNvgtu6gSLym4jsEJHxBf9YSimliiqUSurrHf/e7rV8FPachL/6iLeBl4F3Ahx7mTHmUvcFjqE9XgEuAVKBNSIy1xizOYS0KqWUKiZBA4QxpnFhDmyMWSoiCYXYtTOwwzGzHCLyIXA5oAFCKaVKUCgd5WJEZJyIfOz4GysiMcV0/gtFZL2IfC0irR3LGgB73bZJdSzzl77RIpIsIslpaWmFSoRNZxlVSikfodRBvAp0BKY5/jo6lhXVz8C5xph2wEvA547lVrdrv5XixpgZxpgkY0xSrVq1CpWQmCgd9VwppbyFUgfRyXETd/peRNYX9cTGmGNur+eLyDQRqYk9x3CO26YNgX1FPV8g0TYhM5wnUEqpM1Aoj865ItLU+cbRSS63qCcWkboiIo7XnR1pOQysAZqJSGMRiQWGAXOLer5AbFrGpJRSPkLJQTyAvanrLuzFP+cCNwfbSURmA32AmiKSin0U2BgAY8x04BrgDhHJAU4Dw4wxBsgRkbHAt0AUMNMYs6mgH0wppVTRBAwQImLDfvNuBrTAHiC2GmOClsgYY4YHWf8y9mawVuvmU4Ijxd7Y9VymLdlZUqdTSqkzQsAAYYzJE5HnjTEXAhtKKE0l7twa5SOdBKWUKnVCqYP4TkSudtYXlEVl+KMppVShhVIHcR9QAXvdQAb2YiZjjCkz0472zllJK0ljs0mIdFKUUqrUCGXK0UrGGJsxJtYYU9nxvswEB4Dai+7m8qgVkU6GUkqVKqH0pF4UyrIzmYmKI47sSCdDKaVKFb9FTCISD5TH3ky1Gvk9nCtjnxei7IjWAKGUUt4C1UHcDtyDPRisJT9AHMM+2mrZER1PnGiAUEopd34DhDFmCjBFRO4yxrxUgmkqedFxxJEFgDFGWzUppRShDff9koh0AxLctzfGBJrn4YwibkVMeQaiND4opVTwACEi7wJNgXXkj8FkCDwR0BlFouOJ41Skk6GUUqVKKP0gkoBWjnGSyqboOOIkHYAjJ7OoVSkuwglSSqnIC6Un9UagbrgTElHR8a4ipjlrfo9wYpRSqnQIJQdRE9gsIj9B/rQJxpihYUtVSXOrg8jKLbsZJaWUKohQAsTj4U5ExEXHu1ox/bjjEPdd0jzCCVJKqcgL1FGupTFmqzHmBxGJcx/iW0S6lkzySkh0PLGSA0Dynr8inBillCodAtVBfOD2eqXXumlhSEvkRMcR78hBKKWUsgsUIMTPa6v3vjuLzBSRgyKy0c/6ESKywfH3o4i0c1uXIiK/isg6EUkOdq4iiy1PeZ2VWimlPAQKEMbPa6v3Vt4GBgZYvxvobYxJBCYBM7zW9zXGtDfGJIVwrqKJrUQ5ySKq6FNtK6VUmRGokrqhiEzFnltwvsbxvkGwAxtjlopIQoD1P7q9XQU0DJracImrCEAFMjhGBTJzcomLjopYcpRSqjQIFCAecHvtXcxT3MU+twBfu7032GeyM8Brxhjv3IWLiIwGRgM0atSocGeP9QwQ2bmGuFDadymlVBkWaLC+WSWRABHpiz1A9HBb3N0Ys09EagMLRGSrMWapn3TOwFE8lZSUVLhODM4chJwOrfBMKaXOAqH0pA4bEUkE3gAuN8Ycdi43xuxz/HsQ+AzoHNaExFYCoCIZABw+oRXWSikVsQAhIo2AT4EbjTHb3JZXEJFKztdAf+zDfYSPew4C+GFbWlhPp5RSZ4KwlbSLyGygD/YZ6VKBx4AYAGPMdGACUAOY5ph/IcfRYqkO8JljWTTwgTHmm3ClE3DVQThzEC8s2MbfL0wI6ymVUqq0C2W472eAJ4HTwDdAO+AeY8x7gfYzxgwPsv5W4FaL5bsc5yg5rlZM9hzE0VM6u5xSSoVSxNTfGHMMuBRIBZrj2cLpzOeog6ggGa5FK3YcwhjDn+kZLPntYKRSppRSERNKgIhx/DsYmG2MORLG9ERGnGcRE8CIN1bzzcY/ufyV5Yx8a02kUqaUUhETSh3ElyKyFXsR0xgRqQVud9KyIDqeTBNNZTnpsXjtnr84cExbNCmlzk5BcxDGmPHAhUCSMSYbOAlcHu6ElSgR0qUy1TjusfiN5bsjlCCllIq8oAFCRK7F3sIoV0QeAd4D6oc9ZSUsvnJNqsvx4BsqpdRZIpQ6iEeNMcdFpAcwAJgFvBreZJW8nPjqVNMAoZRSLqEECOcQp0OAV40xXwCx4UtSZGTHVaMaJyKdDKWUKjVCCRB/iMhrwHXAfBGJC3G/M0p2bFWqy7FIJ0MppUqNUG701wHfAgONMUeB6pS1fhBAVlx1qnJS54RQSimHUFoxnQJ2AgNEZCxQ2xjzXdhTVsLqNUjAJoYaaC5CKaUgtFZMdwPvA7Udf++JyF3hTlhJK1fd3jCrtvxluf7Tn1MxRscCV0qdPUIpYroF6GKMmWCMmQB0BW4Lb7IioIp9krwGcthy9X0frefbTQcs1z3zzVbWpJS9DuZKqbNbKAFCwKNgPtexrGypei4ADcX/uEu//G6du5i2ZCfXTl8ZlmQppVSkhBIg3gJWi8jjIvI49vmj3wxrqiKhXDWOmgokiHUuAeC1pbtIP60jvSpVVMcysvn98KlIJ0MFEXQsJmPMCyKyBPuUoALcbIz5JdwJK3EipJi6NJb9ATfLzM6FcjEBt1FKWZv45WbOrVGeWStT2JV2kpTJQyKdJBVAwByEiNhEZKMx5mdjzFRjzJSCBAcRmSkiB0XEckY4sZsqIjtEZIOIdHBbd5OIbHf83RT6Ryq83aYuCTb/OQiArzf+WRJJUapMmrliN4/N3cSutJPBN1YRFzBAGGPygPWO6UEL421gYID1g4Bmjr/ROIbwEJHq2Geg64J9PurHRKRaIdMQspS8utTnMHFk+d3msbmbQj5ebp5h4x/pxZE0th04rsVbSqkSFUodRD1gk4gsEpG5zr9QDm6MWQoEat5zOfCOsVsFVBWRetjHfFpgjDlijPkLWEDgQFMsdpu62MTQKEBFdUFMWbiNS19aXixBov+LS7l2+o/FkCqllApNKPNBPBHG8zcA9rq9T3Us87fch4iMxp77oFGjwmZ07FJMXQCayH62m4Z+t+vz7GJSHBVsnRtXdy3PzMkl/VQ2P+48TGZOLr86AsPB4xlAlSKlDWDbgdDHilqTcoS8PEOXJjU8lv/y+180r1OJCnFhm45cqTJn/q/7GfP+z6x9pB81KsZFOjklxu9dQkTOA+oYY37wWt4L+KOYzm/VXNYEWO670JgZwAyApKSkIvVkG331IHK/nEAr2x6+zevkd7sUt9YXP+3OzyCN/eAXFmwOXIcRSF6e4f3Ve7g26RziY6IKfRzA1ezWvRLw6Kksrpz2I/3Or80bN/n/fEopT7N+TAHsD2kXnkUBIlAR0/8BVuNfn3KsKw6pwDlu7xsC+wIsD6uEerXZYRrQVnYVav+iBAeALzfs49EvNjFl0fYiHcefjOw8AFfORqnSZtWuw6Qc0grs0iJQgEgwxmzwXmiMSQYSiun8c4G/O1ozdQXSjTH7sQ8O2F9Eqjkqp/s7loXdL3nnkWTbhpBXEqfzcDwjB4Cjp4q3MvqF737j/dV7ivWYSoXDsBmr6PPckkgnwy9jXZBRZgUKEPEB1pUL5eAiMhtYCbQQkVQRuUVE/iEi/3BsMh/YBewAXgfGABhjjgCTgDWOv4mOZWEVH2PjF9OMynKKxELmIopTXp5hwhcbPZ6oZv2YwtY/Czag4NTvd/DwZ/ktjXVIKaUKRsre2BEhCVRTuUZEbjPGvO6+UERuAdaGcnBjzPAg6w1wp591M4GZoZynuJxXuxI/5zUDoH9UMutzziuxc//y+1/k5HrmWjbvP8Y7K/eQnJI/xIezmW1hOhg5v+R5Bg6dyKRmKSlLPXIyi6837mdEl3ODbpt2PJNalQKnOysnj9joMjdliYogcVaLnmUPV4F+RfcAN4vIEhF53vH3A3ArcHfJJK/kOVsvXRW1vNiOuffIaU5l5fhdv+PgCa6c9iNPztsCFO5pJTMn9HksDp3IJOnJhaxJOcKBYxkBt91z+CS5eeH9Vdz94S88/NlGth0IPOXrNxv30+mphby+dBdZOdZFgOv3HqX5I1+zeGvxNFVWpd9rP+xkxY5DBd7vj6On6fvcEvannw66rZyd8cF/gDDGHDDGdMPezDXF8feEMeZCY0yZ7k68Lq8p9eQI0fi/qRfEY3M30WrCt+w5bC8qSjl0kslfb3UNH37kpL1jXk4Bb8T/+XoLCePnsXRbGi0e+Ya1e6wHE/Tn2ukr6fL0Ir/rdx86Se9nlzDVT6X59B92FvicVg6fsH9+fzd9J+e5npq/hce/tO6w6Nzmh21pAY/14MfrSXy8YNVaGdm5HD6RWaB9ziTvrdrDJ2tTQ9p224HjzFy+O8wpgk/Wprp+H/785+utjHhjdYGP/cHqPew+dDLkzwyBi2czsnOZuz7sbWl8zrn3SPjGtAplwqDFxpiXHH/fhy0lpcirOZcB0F52FOtxez+7BIA+zy1h+g872XGwaHNgv/aDvZ5kuePpKdliyPH1e4+6XvvLmGz8I92Vk1ix45Dr5up8slq1y3oI9Mlfb+XqVwN33juWkU1Gtj13M/un30kYP8/RL8S/RVsO+A1KTh+s/p3JX28NuE0gHyWnciwjtAeA9NPZbNl/jBFvrKbjkwsLfc7S7pHPN/LP/60PadshU5cx8avNYU3P3iOn+Of/1nPn+z+H9TyhCCVX//T8LYyb/Qsrd1r/XsJh7Ae/0POZxeSFKZevBbUWVua1BuDjuIlhPc8lLy4l/VS2z5fvz3TPG2iw//pAT0CXv7IiaDoufWk5Pf+7GIARb6zmppk/Afnlrscycli16zCLfyt4sU3i49/R/8WlAHyUbO/76P3E4/35b5mVzAsLtvkcS7w2nP7DzgKnpzBueH0Vg6Ysc+VOjDG8uXw3xzLO3qFPsnPDX9iS6chRuheD/pmeQcL4eXy/tWhNyt2F0mjD+VsI1IppneNh7HgJfi+c1yFc/xsaICwco4LrdR+wAysAACAASURBVHHPUb3bq433zW//5LPN91sPcsvba/jvN6E9IR8OkgUPRVZunkcRz9FTWa4b95b9xxg2YxU3v7WmUMf+3REQ8n+IhWsSEqmGJJv2ebYaW7r9EJO+2swTc8P7BB2KwycySS/mZtFOp7JySDseuSI1q6f2Dan2m/AHq/fy25/HXbnTQA6dyLQsvpQCfKNcdRB+7sSrdh1mQ6pn/6LMnFymLtrOoROZIaWzKMI126UGCD9GZ90LwC1R84v1uH292nivT01n9urffbZbtPUgy7YXvOItkKuDjOXkHryuCCHnUVins3L59OdU9h09zSG3Mn3v77jPLH1Bfs8///6X37qHv05mhZwNP3Qik6On/Afd01n2H3tpyEF0fHIh7SaGZ4r4q6b9SKenIl+kZvW/ln46iwH/t5QHPvbpquUj6cmF3DPHdxDqwj5YPT1/i0/d2+Z9vk3PZy5P4YUF20h6ciFDXy6eRi95ecYjGHjnqoubBggvbRvYx0xaktcegIdiZhPutguf/hJ45BKrr8Ahi8rSwyezWLbdf+Xs3iPBW2s4pRw+FfBY7g4ezwjpZum8ihPmbuS+j9bTbfL3JHmV6bt/+a+dvpI3l+9m0z77k1mwJ76rpv3oESAe+uxXEsbPo+cz33PBpAX8X4g91JOeXEj7iQuCbucvNT2f+Z4nvtzEJ2tT2R6kZZa7nNw8Hvtio2UdjTGGmct3W/6/FzdnUcnWP0NPezg4r6/V0/G+o/Zr9GWQSmHnQ8H8Xz3b1fywLY3ZP/k+mAVjgBlLdwWse3Om9rRbrqEg46j5k5mTS5OH5lsWv4aLBggvY/o0BSCLGL7PtQeJi23hqyQLpQnpSYsmst43VrB/cW9807fIKlTeDyOvLPYt439j2S7XDdup81OL6OuogP81Nd01U9iPOz1zQM4f+sFj/m9yp72y4pO+2syQqfanr9+PhD4EgzGGDxw5M2dg/G5TwRvfPfL5r1ZHD7jP3iOneWtFCv/833oucdS/hGLJb2nMWrmHRz7byI6Dx3nJLaBt2X+ciV9t5u4PfylUheRHyXtJP53Nxj/SgzaJLmjuMTnlCO+uTClwmoJxPh0bi2V/HA3tYSfXT9GLv+mD1+89GrC4JpSinIPHMlz1bcXpVKb9/+3dVb6jImgdRAlxv9C3Zf8TgDdjn49MYhz2lNDUjKFkVp+ct8V1w3Z3+GQW6/Ye5bKXl9Pr2cW8sngHN7ye3/Rwf/ppnzJap9S/7D/2d1am+D3vvqOnfZ4CATpMWsCDH/u2vNmR5vvEtv3giaDNX729t8r/U2ZRc/db9h/jz/QMV5qc373cPMN1r63i+QXbOJFpfzjY6fg8K3YcpslDBSv23PhHOg9+vIG/vbGaS19azuMFmNMkmJU7D3PN9JU8+kXwYwZ6GMrIzvWoSzmdleuWgyh8+vIcO9tC+L9aui2Ny19ZQe9nl7DPKwBZBSt37ssf/WITD368gfQAxZRWMrJzA/ZLsjp3uOvlNEB4cf8y5pI/omoDCnZjiaR/vBtSR3cfzo56ofCubAfPJ89nv/3NY93z3+Vni72/1M6JkP4XoDVWt8nWLayPnMzio2Tf/Vbs8G1qmJtnXC20iqKgN6zsXN8K0vRT2Qyasoyu/1nETTN/4kRmjuu6LNp60D61LflPrHfNLvwsv84K0u0H7UVG6/cW32CNw19f5Xr9v+S9AYPPG8t8h6/5X/Jeftp9hMFTlnnUpZw/4RvX9yLU8Y/eW7XH5wk/z3Hpo7wihHultXMPZ2OK34+c4tZZyR7bW+UcJn212Wf0A3feuZdguY/R764N2C/Jub/lUNdhykJogPDi/WW8JnMCAEOjVkYiOYXyTSGKUiB45zJ33pXtZcW3btduxBurLLdxfkOs6kSs6hzu+XCdfT9j+PyXP8jOzfMpSsvNMx45Euf67FzDmPeDB/zdh04yZ83v5OUZj5kHrVrvhKs44oGPN/C2Y1hsK/vTfZ+OH/h4A9e9tpJdFg8cewrYAeyRzzeyapdnw4Y81001/+Ku3fMX05YEbiLtzLmtSTnCm8t35zcYcbt4by7fzcIt9qbfoRRfWhUNuVtq8fvbcfAEe4+c4tOfU13jqYW7YtqdBggv3pE42bQA4F8xH55RuYjS5uMQe6sW90i2Vl61uDnk5Obxa2o6t7vlvqxyIZD/HcnKzfNpvmhV5zDv1/0cz8jmyw37uWfOOlo++o3PNt5DXDtLY254fZVl0Zp3h8OhLy/nX5/8yn+/3Uq7J74j/XQ2fxw9TfNHvubDNfbycOdw795PsuHsiZudm0fC+HlMXbS9wEVyL39vr4NxT26wQ9wz5xdOZvrW2WXl5rmKuJ7xaj7uPKZV+q6dvpJJbh0C8yxyBX8cPc3q3b6dVL2LJwvagW7ehv30e+EHej6zmPs+Wh/wwS9cMUMDhJcaFWO9lgh/GPusbCvi76Y8gXsBq+C8f2Tu/BUlBbNoS+gdp6z6l7R49BsuC7Ep4p0f2BstfL/1IG1DHK7j6flbXE1nvXMLYO/QaFWxGaglUeen8osjnEPFO3vXHzudzS5HvcW8Dfv9HmPu+n30fGZxyC3WjpzMYsrC0OcrceaEXl+6C1sB72LOoWecX5eE8fN4en7gYtADxzJ5+8cUDh7PYPFvBz1ySx+vtV9f75t5QXJUVl/dwvRxeGnRdhLGz+Oi55b47Vjn08zbgvMaWTWzLQ4aILx0a1rTZ1n3zJdcrzfHjyKmmMZoOludzCr+TkMTQqgk9efwicxCD0jo7FEc7Cn8VFZu0MYG324qvt7BX2/c7xrDKNB9+fnv7HVFv4XYpPXfn27gxYWBm1latbLyN01kIFbbWxVFWbn+Nd+Onf/65NeA/0+hdJzzHi9NJPTyf2Ps9W3700/zvKOp6q5DJ2n7+Hc+uZ7mD3/tt7hOsBcpdpiU3xT7CT9jkxWVTkwcorYZb/Br/K0AbI//O1dmPsEvplmEU6WcQm326C1h/DySzq1WpHN/se4P7nbUM/hjjL3M2umhT62azxafp+fn55JOeQVk9xuav6Dl3kT5j6OnGfrScr68q4fPsay8szKFkd0beyw7kZnjGjojVDvTTrrOX5DhK0TyG1F4F6dNCjB+1EOfBf8/se7XElqEWPv7X1z03BLLDnreObisAJXfh09mMertNUEHMSwOYc1BiMhAEflNRHaIyHiL9S+KyDrH3zYROeq2Ltdt3dxwptPb8M7n+Cw7TnkaZ7znev9Z3GOkxN/AZbbAvZNV6ZdcxBFpgwUHsDc1dbcogsOR/3bgOL/8/her3QZh9G7B5t5E+d456zh8Motuk78PqXf/fj9NNYNV0gZyi1erolCFcut+YcE2EsbP81i29y/rwPm8Tyc1cbV+CibteKbf3ttRtvxb8b1zgn+frFoRhkPYAoSIRAGvAIOAVsBwEWnlvo0x5l5jTHtjTHvgJeBTt9WnneuMMUPDlU4r/7kq0XJCHoONSzOf9Fj2UuzLJZUsdQZz9vUoLa6ZvpLrZ1i30vL2k0UFbCB7Dp1i0JRl7E8/XWzzcqxzG5U4mGe++S34RkGEWmz06pIdjHq7cMHL3YnM/BzSZ0FGVrDePzzF3uHMQXQGdhhjdhljsoAPgcsDbD8cmB3G9BSLjaYJCRnvc5lboFgU+88IpkidCQIVGURCOCeB+mbTn2zZf4zuk78PKXcVilA6ulnxvtF/tzn0ep5QKp/X++n8WVD3zgltmHV/imMoDyvhDBANAPdmGamOZT5E5FygMeDehCVeRJJFZJWIXBG+ZBaG8KtpQtuMNwBoatvPb3F/J44shNJ1I1AqUoozBjmb6BZUj/8Wfgobq+bIZ5twBgjLDn9+th0GfGyMcQ/ZjYwxScANwP+JSFPLk4iMdgSS5LS0ku2ncJzy3Jplzz3ESQ6/xY9kd/zfSIm/gfdinirRtCilfB0PcVIoZS2cASIVcK/tbQj4G3pxGF7FS8aYfY5/dwFLgAusdjTGzDDGJBljkmrVqlXUNHt4YECLoNsszOvI2Ky7fJb3iNrEqzEvFmt6lFKqJIUzQKwBmolIYxGJxR4EfFojiUgLoBqw0m1ZNRGJc7yuCXQHIj87ix9f5V1Ij8wpjMsaS0LGB3yS2xOAQVFraC7FP6qjUkqVhLD1gzDG5IjIWOBbIAqYaYzZJCITgWRjjDNYDAc+NJ4Nls8HXhORPOxBbLIxptQGCIBUU4tUY8/B/DP7DjJMLCOiF/Fd3L8st386ezjL89qy2SSUYCqVUip0Ye0oZ4yZD8z3WjbB6/3jFvv9CLQNZ9rC7eGcW+gflUwtsW7lYJ+IKL9ULSHjgxJKmVJKhUaH2gijTpmvukaDdbcst43PspT4G/gq9iEAypHBczHTqUZ4xldRSqlQ6FAbYZZsWlrnDhz9YqbHvMjAKPuYMW1sKaTE3+Da5JqopTTNeJeKnCadiiWRXKWUctEAEWH/yL4XsuG+6I8YF/25z/qd8Te6XjfPmEUWMTSSAyyNu9e1vFvGVPbhO8igUkoVhQaIANo0qALA9L914B/vhW9eaoAXcq5ja14jpsVO5e9Z/yLNVOXruH97bLMt/ibLfX+MH8eorPuxYXjDbXrUhIz3Cf+khEqpskpCmYT7TJGUlGSSk4s+Loq7fUdPU79qOZ/BvEpCRU5xmjjAMyfhNCn7bzwa857Pcm9f5Xbl0ijfcXd0RFqlyg6r8eNCISJrHZ2SfWgldRD1q5aL2LlPUJ5cosgligGZk13Lr8icSELGB7yZO9iyfmNLnudotFbBAfJHpK3OMVLib3D9fR77iGuba6OWuJY3lYIPIqaUOnNpDiJEk77azJvLd3NNx4YhT59ZkmLIIdurxDCGHLbH/931flzWWKYWcfTZ1hlvsin+FqblDOWZnGGu5dHksMPtXABNMt4jT59BlCoR4chBaB1EiGKj7Te66MIOKxlm3sHBucw7hzE3oxtN5Q92mgbEku1RrzEl50rSTUUmxLzrsc9fpiLVxD5a5Kb4WwAYEz2XMdFzmZVzCTdFL8DKrvi/eby/P/t2Ps7tTSzZ9LOtZVrsVNe6sVl38VXehcSSTR/bOsZGf87SvESey7meKHKxYSw/o1IqfDQHEaLJX29l+g87Gd65EbN/sk9GfmffpryyeGdYzldSEmQ/XWxbmZPb12P5hbZNzI59iuezr+Gl3KuoyCk2OmbUC6R9xmvEkMOa+DvDkt4uGS+zOn4s47LGMjevGwA/xo2lvtjnLLg68zEOUpVljlZejTPewxQgFxNNDjkaiNQZKBw5CA0QIdpx8ASDpy7jg1u7cM10+7BRr47owB3vh7d1U2lTjWOUI4sMYlkQ9wA1xD4FY2LG6xyjPM5WU0IebWU3V0StoLttIy1skSuW22equwKIu9MmllaZMzEIfW3reCv2Wde6LBPF2OxxLM9ryyniaSL7+D7ufiC/uXGC7EcAG3nsNPkj2UeRS3WO0ztqPR/n9nY7o/O3FkoutDCzOKuzmQaIIMIZINy9szIFEaFr4+pc8uLSsJ+vrOhp28C7sZN5Necy3s4ZAMABqgPw7+j36WXbwOK8C3gm53pAqM4xastRtppzGBG1iKdiZvo99uycvgyPXhyWdM/P7czgqJ8CbnN31himxE4L+ZjuOaAOso1P4x73u62zmLAeh0mjCnnYHHU7hsbyJ2mmCieJ56e4MdSSYwzK/A9bzLmAs24qCn/BJopccrH5XR9MO9nBZpOgxX+lgAaIIEoqQDjtOHicfi/4BoiUyUMY+vJyNhTTbFPKzkYe/WxrWZDXkfocZkX83YAz91LBcp8J0e8wKto+8csnuT1pI7sZmfUvbo7+htHRvk2XW2S8TTvZyUdxk8L3QQroh9xEBEOvqF9dy8Zn38rkmDf87nN+xkxuivqO8TEfupYNzZxEa1sKX+d25iiVeDj6PW6Lzh8qrVnGOzSSAyyKe8DymHdnjeGLvO4AXBe1hGdiXvdYb9XvpoPY53B2D4A9Mv+Pc+UAK/ICDbdmaC172GXqcpp4wB7sro9azFe5XTlKpQD7np00QARR8gHiBP1e+AGAns1qkpNrWLnrsAaIM0gs2QiGWnKUVFPba60hJX6E652zPqOVpPBO7GReyrmSWbkDuEC281ncY67trsmcwM+mOc0llW/ixgPwVs4APs7tzby4h/ym5eLMZxlgS2ZM9Bd0zpzGVVHLeDLmrWL9vOHWIuNtMomlCidYHz+6QPum5NWhuhxjbPY43on9r2v5f7OHsdPUY0as//lVduTV57KsJ8kgFoOwLm40VeWkz3YzcwYyMedGQFz1bM7WdgNtP/GXqcRvpqFPANoYN4qKkuF6vzK3FfPzOvNubn+3rQxzYx8h0babdhkz/A6PE08m82IfYrtpyH3Zd3DKEQD9M3wa+xj/zR7OanO+x3L3gKwBIohIBohnrk7kmo4NyTWGmCgbCzYf4LZ3Si4t6swSTQ7NJZVbo+fzdW5nFuRZ/j75LvYBmtvs/U+ey76Wl3Ov9Biv64KM6VSS0+w1tVyV8e7rn8wewRu5gz0Cnbensm/g4Rj/owkPyXzab2DrnfkCd0Z9wXXRP7iWDc58mvkBAmFZMzrr3oDBy2lg5mTXA4O7h7JvIZsono2Z4Vq2LLcNU3Ku4gTlPfaZknMlU3OucnWcvSZzAsmmJaABIqiSDhDHMrJJfPw7AP57dVuu79TItc4YQ4dJC/jrVHaJpUeVVYYqnCzQgI028qjMScuiGCGPcmRxijjcn0Br8xcGOEQVy5Zf1TjGccpzoW0zy/PaeGxTi6OsiR/js8/Fmc96VOA7nS97fIaS8ZaQ8YF9rnfJnza0ccZ7LIx9gKa2/QCszmvJurym3G5RXOhvBIHCejXnMhLkTwY5BtcsDdzHYdMAEURJBwiABz9ez0fJqUy+qi3DOjfyWLd612Gun1F8X1ClSjPvjpm3ZP2TRXkdi3hUw/sxT7PNNOSJnL8TuDLdcIHs4Bdznt/tGpDmqrv6R9Y9fJPX2dXibr+pQRpViSOLRnKQR6PfddX7OJt7uztHDriaUwMMz3qYtXnNySaKOLLZGn8zAHNzL2RolGvCTN7KGeD6LJvjbqa8ZLrWPZh9m0/dDkBixgwGRCV75DK6ZLzsauQBZ2CAEJGBwBTsM8q9YYyZ7LV+JPAs4BzD4WVjzBuOdTcBzjEfnjTGzAp2vkgEiH99vIE5yXstAwTAtgPHaVa7Io3/Pd9ib6XKngakkYeN/dSIdFJKRA3SSadCIfvPGOpyhD8trpWQxzmSxu+mjmtZX9svnCaOVXmtfLY/o3pSi0gU8ApwCZAKrBGRuRZTh84xxoz12rc68BiQhL0mZq1j37/Cld6i8hdmm9exZ/FrVozj0IlM+reqw3ebD5RcwpQqYX9QK9JJKFGHqVKEvcUyOAAYbB7BAWBx3gVFOFfBhXOgnM7ADmPMLmNMFvAhcHmI+w4AFhhjjjiCwgJgYJjSWSTiyMkGy4iVi7Vf6k4J1T2W/6N3Uy5q6d16RimlIi+cAaIBsNftfapjmberRWSDiHwsIs5hSEPdFxEZLSLJIpKclpZWHOkuEClg/yKr7SvFaycjpVTpE84AYXXr9H7O/hJIMMYkAgsBZz1DKPvaFxozwxiTZIxJqlWr5LO2Fza1tyBoWS9wxx3xU2lmMB65jw9u61JsaVNKqaIIZ4BIBdwnJmgI7HPfwBhz2BjjrMJ/HegY6r6lxdB29fn50Uvo0KhaSNvHx0T5LHOPfF0b28sja1eKY/1j+Z1wtk4KXMJ23yXNQzq/UkqFKpwBYg3QTEQai0gsMAyY676BiNRzezsU2OJ4/S3QX0SqiUg1oL9jWalUvUJs0G2evrItLetW4tqkhjwy5HzG9GnqWudsSTZlWHtsNuGru3ow/+6eVCkXQ6/m9lxRsGHGz61RvgifQCmlfIUtQBhjcoCx2G/sW4CPjDGbRGSiiAx1bDZORDaJyHpgHDDSse8RYBL2ILMGmOhYdsbq0awm39zTi7joKG7t2YRK8TE+24ijgqJNgyrUrGifanTWzZ3Y/Z/BrnX+JHlVfrubMqy95fJAzeI+HdONm7snBDynUqpsC+t0X8aY+caY5saYpsaYpxzLJhhj5jpe/9sY09oY084Y09cYs9Vt35nGmPMcf2fWgDQh6O3IGfRvVddvE1mwBw0RIcom3NuvOfPH9WTjEwNY+0g/Pr+zO9d0bAhAg6rl/N7wL2/fgDdvsjdzbndOVctt2jSo7PG+Q6NqTLjUs631hEtbsfs/g4N+tqs6WLYnUEqdYXQ+yAhpVb8yKZOH0PHcagzrZK9u6Xhu4HqMu/s1o1X9ylSMi6ZGxTjan1OVZ65O5Lcnfesnpo3o4PE+WDFYTJTvV8E71zKqR2NEhEsT6/ls665WxbhCd9opLQa0rhN8I6XKOA0QpUDPZrVImTyEBlXLFXhfm02Ii/at+B7cth7PXduOzo3tRU/RNvt/dXmvSvI4x1Sqz1/bjlb1PHMR7n559BLX6xevty6ycurgCHTrJlzCM9ckhvApSp+buiVE9PyjezWJ6PmVAg0QZdo1HRvy0e0XAvYipPv7N/epj/jkjm7c3rsJjWtWYP7dPdn+1CDLHEk1txyIVW4DYNIVbVjzcD8GtK4LQNXysZRzC0h3XXRekT9T45r2eR8aVC1H+VjfwOhPQgEq8WeOTKJzgDodgC6Nq3NPv2YhH7OgHhp8fvCNlAozDRBnCRFh7EXNqF3Zc+z5Ng2q8O9B57uKk2KibJY5Ev/Htf877qLzuLHrudSqFOexvqKjE+BNF57LP/u3CHisR4ZY3xQ/G9ON5nUq+my78fEBlts/MKCFR3+Sj26/kCEBisWa1fY8dv2q5YiOsnnkmqw4g5U/wYrigunTwn+/nsLkNlXZ1e/88BSJaoAoY7xvdlbaNKjst2VTQd3bz97/Is9PTXuf5rX479Vt+bfbE3HbBlVYP6E/n9/Z3bUsZfIQbu3ZhJTJQ9g8cQA/jr/I4ziNqttvxs4iMfDfi/3OvufRrWlNV9Nf76AVH2PjP1fZZzMb3LYuC+7rbVlnUi1AvU0oPehj/eS0gul+nr0vjDONVt69pTMPDgwccItT2wZFGW9IhZt3I5PiogGijPnyrh78+nj/gNt8dVdPLm8fWkujykGGAXF2zzB+2mKJCNd3auTqILjjqUF8fmd3qpSPob2fFlXlY6OpX7Uc5zmCnYjwwvXtmP63jsy4MYnezWvRO8DTtVMLx0CJ5WOjuPKC/M87sltjKls0M37YEcTOqWZdHFUpPtojl9Oibn7veffcwogujVzprlnRN8j4yyk5OXvW16tSjul/68jgtnV5ZMj5xDqC40ODW9KkVkVGdW8c8DjenMV9wzufQ8rkIbxyQ4eQ6ohSJg/hy7t60DeEaw6w7MG+BUqXgu/u7VWk/S9uqTkIFYL4mCjLPhaF9dPD/dj0hHVRDsDI7o25tmNDbu/d1O827qKjbEQF6fTn5KxjEKByfAwD29SlUY3yzBrVmfKx0ZZ9Qz4b0831+sXr2zP7tq7UqRzPebUrkTJ5CFsnDeTBAS2oW8Ve1NaiTv6T12297DmYCnH5QXH9Y/2ZN64HAO3Pqcr5jop8QWhZN3/f/o56F4B2De2BzybwyJD8psINqpbjvVu6cGvP0CugB7apy7QRHbm1ZxNXULvCEdy964K+uquHz/7uOcqHXYHJft2GJNbjuqRzfPbx562bO7uaSd/fvzldm+TX07gHvXOqB6/vqVUpjm/v6cXi+/u4ckzuROz9dwKNENCgajn+4fa9G3dx8dQJFbVo0B/3r/1Ht1/I6ocudr13jvpc0LHdnBqFqaOsBggVUHxMlMcN01vFuGievbad5RN5UY27yP6Db1LLf1n/mD5NmTs2v6jqArchTyrERXNhU8+bT3xMFDab0PHcanw2phtjg1ScVykXQ+v6VZj+tw68PDy/6bDzhzxrVGdmjers6g0PkOd4LQJXuOVczq9XiR7NavqcI5RiQTvHORznjrKJR9FYmwZVmDo8fzjoZ65O5BO3gOlMYSg3oReua2e5/PMx3dj9n8GMvagZz1/Xnt7Na7Hq3xdzS4/AuRn3wA32nGmLupVoXLMC79/a1eNzpEwewu7/DOHy9g0Yd3EzjxupuxXjL2L8oJau962CjIcWqlBmyPnv1f6L//wZ2Cb/IaJz4+rU8aoP/OquHiQ/3C/gMapXiGWyRdFjlXLF//sDDRAqwgJV9PZrVYeUyUMC5ogeHNiSxIZVWXhfL75wq9MIxQWNqoWcmxnYph5VysdQtbw9LU1r2W/qvZvXcnV6dHLWx9i87sRVy/sWN8XH2HyKF/wNHe8M1N7HdTe0XX1X0Vz7RlWpHB/D1R0a8sGtXVwHDuUTX9WhIZ+N6cYPD/TxWO7suAn2J/hZozpTt0q8T27Ou/jwgkbVPPri1KsSeiV7ncrxzBxp7+gZqOK+f6u6PHppK94a2cm17Ce34PL3C8/1u2+DquWoXyXe73p3KZOHeEwv7FTbq67rWkcn1n8NbMlPD13M/13vO5fDkvv78PE/nC0Nq1CjYv4xJl3e2mf7pQ/2pY5XOj/1Cr7FSceZVoXyyg0dXHNcFMWCe3v5reAuiPNqF8/TYzCt61fh/Vu7kJTg2anxgnPy3zvrY7xvmo8P9f3B2xw33HEXN2PlzkOsSfE/J9a7o7rwzab9rmFYnP41sKUrcEH+2F7Osz/vyA3sSDvhSJfnceeO7c7BY5n0aFaTlo9+k/+ZQhyA0mn2bV1dT7LO/9IhifVcA1n+/OglHM/I5qPkVK66oGC97es7Wm11SqjOkt88h/Vf+kBfoqMEm024pUdjVu06DNif0mtXjifaJuTkGR6/rDX39mvOpS8tJ+1EJgNa1+XL9fYxQD+/szurdh3mrtm/hJaFAH59vD9tHXPS73p6MKt3H2H46/lTDNepHO/T+GFktwSP0TovdwAACs5JREFU+rOEmhVI8HpI6tW8Fr2a1aRxTd+cZWyUzSfAhzpQaGFogFCFEqjZaEFEF7KlTyR1P8+3mMi9DLhxDfsP3rvlT0W3orp+59dh4ZYDrtZd913SnM2t6zJ46jJXDsDqHKN7+db13NHHuv7HOxAYVwmV54rEhvlP+zd3T2DHwROWxwvGvTjP2bjhH72a0rZh/nWoFB8TtDjKSsu6lVl8fx/OrV6eZ7/9zWNdsPL3NQ/343R2LjabUK1CLEsdlej3zFkHQIdGValVKc41IGaUTZg1qjO1K8XRpFYFdqWdZNCUZT7HrRQfQ89mNVm2/RA2m9C1SXXGD2rJgWMZvLUixbIoz+ohwds7ozoDsHaP58PCJ3d0IzbaFnRctuKkAUKpYtbtvJp8d28vV93CBY2qkuQ1jEqnhGos3HKANvXzK7pb1a/MB7d2oWNC0Z4InU/xUTbP4Gvc6kb8eeyy4DewULx4fXs++/mPYm1+GazfiZOz2NDZJLpahViqWawf6MhBPHONPYd1Sas63NKjMWP6NPUo6jk/wAgDb43sRHZufo7xH72b8n8Lt9nfh/ax/OrQqCr/vbotJzJzmfTVZpo66uLcjxvuIWE0QChVTD65o5urgtrZKgXgszG+dSO39WxCnxa1PZrKgj24FNX0v3Vk3q/7/d5QS+L5s2bFOG4L03AhUTbh8nb1/a7v2KgaY/o0DTpcypDEevRvPcjVGiw6ysajXgNUBhMdZcO7X6nzujcO0LgiFM4m4oBHrqtBNXtx2/39mzP2ovD15gcNEEoVm2CDLbqz2cQnOBSX2pXjudmij8R1nc5h3d6jxdYcNFJ2Ph14RGGbTXhwYMuA2zj5GzbGn1D6KwxtV59zqpfnAj/9fIqqaa2KLHuwb4n0ptcAodRZonxsNP83zLcljQquZsVYYqJsHjlDf0QkrBXHEFpfk+IQ1gAhIgOBKUAU8IYxZrLX+vuAW4EcIA0YZYzZ41iXC/zq2PR3Y8xQlFJl1vxxPVm9+3Ckk2FpTZD+CWVV2AKEiEQBrwCXYJ9jeo2IzDXGbHbb7BcgyRhzSkTuAJ4BrnesO22MKZ4Bg5RSpV6r+pVpVT88YwoVVUm2HCpNwtnGsDOwwxizyxiTBXwIXO6+gTFmsTHmlOPtKqBhGNOjlFKqAMIZIBoAe93epzqW+XML8LXb+3gRSRaRVSJyhb+dRGS0Y7vktLQ0f5sppZQqoHDWQVjlySz7KIrI34AkoLfb4kbGmH0i0gT4XkR+Ncbs9DmgMTOAGQBJSUnF0CdXKaUUhDcHkQq4DxXZENjnvZGI9AMeBoYaYzKdy40x+xz/7gKWANr8QimlSlA4A8QaoJmINBaRWGAYMNd9AxG5AHgNe3A46La8mojEOV7XBLoD7pXbSimlwixsRUzGmBwRGQt8i72Z60xjzCYRmQgkG2PmAs8CFYH/OVoJOJuzng+8JiJ52IPYZK/WT0oppcJMjL+xhc9ASUlJJjk5OdLJUEqpM4aIrDXGJFmtO/OG0lRKKVUiylQOQkTSgD2F3L0mcKgYk3Om0utgp9fBTq+DXVm+DucaYyxnYipTAaIoRCTZXzbrbKLXwU6vg51eB7uz9TpoEZNSSilLGiCUUkpZ0gCRb0akE1BK6HWw0+tgp9fB7qy8DloHoZRSypLmIJRSSlnSAKGUUsrSWR8gRGSgiPwmIjtEZHyk0xMOIpIiIr+KyDoRSXYsqy4iC0Rku+Pfao7lIiJTHddjg4h0cDvOTY7tt4vITZH6PKESkZkiclBENrotK7bPLSIdHdd1h2PfUjmrjJ/r8LiI/OH4TqwTkcFu6/7t+Ey/icgAt+WWvxXHeGurHddnjmPstVJHRM4RkcUiskVENonI3Y7lZ913ImTGmLP2D/sYUTuBJkAssB5oFel0heFzpgA1vZY9A4x3vB4P/NfxejD2eTkE6AqsdiyvDuxy/FvN8bpapD9bkM/dC+gAbAzH5wZ+Ai507PM1MCjSn7kA1+Fx4H6LbVs5fgdxQGPH7yMq0G8F+AgY5ng9Hbgj0p/Zz3WoB3RwvK4EbHN83rPuOxHq39megwg6610Zdjkwy/F6FnCF2/J3jN0qoKqI1AMGAAuMMUeMMX8BC4CBJZ3ogjDGLAWOeC0uls/tWFfZGLPS2O8M77gdq1Txcx38uRz40BiTaYzZDezA/jux/K04npAvAj527O9+TUsVY8x+Y8zPjtfHgS3YJzE7674ToTrbA0RBZ707UxngOxFZKyKjHcvqGGP2g/2HA9R2LPd3TcrKtSquz93A8dp7+ZlkrKPoZKazWIWCX4cawFFjTI7X8lJNRBKwzzGzGv1O+HW2B4iQZ707w3U3xnQABgF3ikivANv6uyZl/VoV9HOf6dfjVaAp0B7YDzzvWF7mr4OI/H97d/MiRxWFcfj3ksgYVMSvhbsoRAQFFRKJGiELmYW4UYSIQsS4UEFFNyGYfyAgCIJuFEGQMQuNUVeOIMZgJEwgzLQJfu+ycFSUGBUkxuPiniKV5nbPtHSmp9PvA0VXV1Xf7nunuk7XrZpzLwX2Ac9FxO/9Nq0su6DaYimTHiCWNerduIuzo/P9BOyndBcs5ikx+dgM2NSrTS6UthpWvU/kfPfysRARixFxJiL+BV6n7BMweDv8Qul6Wdu1fFWSdBElOMxExHu52PtED5MeIJYc9W7cSbpE0mXNPDANHKPUs7n74lHgg5z/ENied3BsBk7mafcsMK0y2t8VWc7sClZlWIZS71x3StLm7Iff3ipr1WsOiOl+yj4BpR0ekjQl6TpgA+XCa/W7kn3tnwIP5uvbbbqq5N/pDeCriHiptcr7RC+jvko+6olyp8K3lDs0do/685yH+l1PueNkATje1JHSd/wJ8F0+XpnLBbya7fElsLFV1g7KRcvvgcdGXbdl1H0vpfvkNOXX3ePDrDewkXJg/QF4hcxMsNqmHu3wVtazQzkQXtvafnfW6Rtad+H0+q7kPjaX7fMOMDXqOvdohy2ULp8OMJ/TvZO4Tyx3cqoNMzOrmvQuJjMz68EBwszMqhwgzMysygHCzMyqHCDMzKzKAcKsQtIf+bhe0sNDLvuFrudfDLN8s2FxgDDrbz0wUICQtGaJTc4JEBFx54CfyWxFOECY9bcHuDvHTHhe0hpJL0o6konungCQtDXHGnib8k9VSHo/EyQeb5IkStoDrMvyZnJZc7aiLPtYjimwrVX2AUnvSvpa0szYjzNgY2Ht0puYTbRdlHET7gPIA/3JiNgkaQo4JOnj3PZ24OYoabIBdkTEr5LWAUck7YuIXZKejohbK+/1ACV53i3A1fmag7nuNuAmSm6fQ8BdwOfDr67ZWT6DMBvMNCU/zzwlVfRVlHxFAHOt4ADwrKQF4DAludsG+tsC7I2SRG8R+AzY1Cr7RJTkevOUri+z88pnEGaDEfBMRJyTqFDSVuDPruf3AHdExF+SDgAXL6PsXv5uzZ/B311bAT6DMOvvFGV4ysYs8FSmjUbSDZklt9vlwG8ZHG6kDFnZON28vstBYFte57iGMlTo3FBqYfY/+FeIWX8d4J/sKnoTeJnSvXM0LxT/TH1YyY+AJyV1KFlRD7fWvQZ0JB2NiEday/dTxjNeoGQd3RkRP2aAMVtxzuZqZmZV7mIyM7MqBwgzM6tygDAzsyoHCDMzq3KAMDOzKgcIMzOrcoAwM7Oq/wDSyaUOjpDmJgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_list, label='Minibatch cost')\n",
    "plt.plot(np.convolve(cost_list, \n",
    "                     np.ones(200,)/200, mode='valid'), \n",
    "         label='Running average')\n",
    "\n",
    "plt.ylabel('Cross Entropy')\n",
    "plt.xlabel('Iteration')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXiU5bn48e+dfV/IAoQAYV8FhIioqCBWBRVQUaHirrTaU6vWtrbHX60e29rWqrXtsQcV3FBEgaLWXalgVSDsmxDWkARCFpKQPZM8vz+eCQTIMoS8mSRzf65rrpl5513uN4R7njyrGGNQSinlO/y8HYBSSqm2pYlfKaV8jCZ+pZTyMZr4lVLKx2jiV0opHxPg7QA8ER8fb1JSUrwdhlJKdShr167NM8YknLy9QyT+lJQU0tLSvB2GUkp1KCKyv6HtWtWjlFI+RhO/Ukr5GE38SinlYzpEHX9DqquryczMpKKiwtuhdBohISEkJycTGBjo7VCUUg7qsIk/MzOTyMhIUlJSEBFvh9PhGWPIz88nMzOTPn36eDscpZSDOmxVT0VFBXFxcZr0W4mIEBcXp39BKeUDOmziBzTptzL9eSrlGzpsVY9SSjVq+3sQ0wu6j2y7a1aVwsGNkL0BAkMgaTR0HQb+7a/NTBN/C+Xn5zNp0iQADh06hL+/PwkJdoDc6tWrCQoKavYct99+Ow8//DCDBg1qdJ+///3vxMTEcNNNN7VO4Ep1dluWwDu3g38wTH0ORs5s+bkqj0LeTqitOfWzmmrI3Q5Z6yF7HeR+B6b2xH38g6HbWdBjtP0i6DsBorq3PJ5WIh1hIZbU1FRz8sjd7du3M2TIEC9FdKLf/OY3RERE8NBDD52w3RiDMQY/v45To9aefq5KnbZDm+Gly2yy9Q+CfSvhgvth0q/Bz7/pY6srIGcLZK2ziTxrnU36NJMjw+JsUq9L7klnQ3XZ8XNkr7d/BVSXQnA03PouJI1qtVtuioisNcaknrxdS/ytbNeuXUyfPp3x48ezatUq3n//fR577DHWrVtHeXk5N954I7/+9a8BGD9+PH/7298YPnw48fHx/PCHP+TDDz8kLCyMZcuWkZiYyCOPPEJ8fDz3338/48ePZ/z48XzxxRcUFRUxf/58zj//fEpLS7nlllvYtWsXQ4cOJT09nRdffJFRo9rml0updqE0HxZ+H0Ji4IbXIKwLfPhz+M+ztjR+7QsQEnXiMbW1sOcLWPMSpH8CtS67PTzRJvLh10G34RAQ3MAFBeL62yqlhtrHuvSxx4P9i+HQJnjrZnhtOtz2L1sN5CWdIvE/9t5WtmUXt+o5hyZF8ejVLfuH2bZtG/Pnz+cf//gHAE8++SRdunTB5XIxceJEZsyYwdChQ084pqioiIsvvpgnn3ySBx98kHnz5vHwww+fcm5jDKtXr+bdd9/l8ccf56OPPuKvf/0r3bp1Y/HixWzcuJHRo0e3KG6l2h1XFWx/FzLT4Jw7IX5Aw/vVuOCd2+BoDtzxIUR2tduvegYSh8KHv7B/Ccx60ybksgJY/zqkzYMjeyE8Ac79IfQaZ0vsUT0aTuYt5edvz3vruzB/CrwyFW7/ABIar+Z1UqdI/O1Nv379OOecc469f/PNN3nppZdwuVxkZ2ezbdu2UxJ/aGgokydPBmDMmDGsXLmywXNfe+21x/bZt28fAF999RW/+MUvABg5ciTDhnmvJKEUAMbAnn9DcXbDn4cn2BJ1eHzDnxcegLXzYd2rUJoLCKS9BOMfgPEP2sbT+j55BPaugOn/gB5jTvxs7N32C2PRrfDCJdDvEtv4W1MJvc6HSx6BIVMhoPl2uTPWpS/c+t6JyT+un/PXPUmnSPwtLZk7JTw8/Njr9PR0/vKXv7B69WpiYmKYPXt2g33l6zcG+/v743K5Gjx3cHDwKft0hHYa1QqMsXXHQeHN73sm1zC1zdeHN6VgL3zwEOz6rPl9o3vaknBd/XhNlS2F7/zIfj7wClvS7zrcJvcv/wCb34Yrn4Z+E+0+G96AVc/DuHth1KyGr9N3Atz9ha0K2vkRnD3bfV4v5I74AXDLMnj5Snh1mk3+Mb1O3c8Y+8UZFnfqF90Z6hSJvz0rLi4mMjKSqKgoDh48yMcff8wVV1zRqtcYP348ixYt4sILL2Tz5s1s27atVc+v2onPH4evnobYlBMbE7uPhOCIMz9/Zhq8fz/k74ER18M5d9lGUk+5quDr52DFn8AvEK54EgZNaWBHA0WZttGzriF1+7vHPw5PsCX7MbedmBCvexFG3QT/etDWk591PQyfAe/dD30ugu/9T9PxxfWDe7+19fje7mLZdSjc8k945Wr7uO0DCAw9sWE5ex2U5MDN/zz+JddKNPE7bPTo0QwdOpThw4fTt29fLrjggla/xo9//GNuueUWRowYwejRoxk+fDjR0dGtfh3lRZvfsUl/wOW2oTFzDWxd4v5QbMl18h8hpQW/X+WF8PljkDYfIrvD4Cth40JY+zL0PNd+AQyd1kgDp9u+r+D9ByFvh933iichKqnx/WNTIGX88fdlBTbRuSqh/6WNX6vfRLjnG/uz+OoZW/qP6QUzXgZ/D9KZiPeTfp3uI2H2Ulvq/+sYcJUf/yx+oP0rJWl04+0aZ0C7c3YCLpcLl8tFSEgI6enpXHbZZaSnpxMQcPrf6/pzbYcOboSXLrdVIrcsO14XXZLr7iq4Dja9BYUZcOWfbUnZE8bAlsXw0S+hLM82bk78FQRHQvkR2PAmrHkRCnbb6oZh19jPTnZkH2xdahPwlD/DwMta686blpcO3/zNxp3YgX9nM9Nse0bcAPtXXPdRp/Y+aqHGunNq4u8ECgsLmTRpEi6XC2MMTz31FJdd1rL/fPpzbWdKcuGFibbefc6XEHHKKnpWeSG8cwfs/twmwst+23QJOC8dPvgZ7Fluv1CuerbhvuW1tbD3S9uwmv4ZmAYGMvkH2QbUi34OQWEtu0/lCO3H34nFxMSwdu1ab4ehWltNNbx9m+3VcsdHjSd9gNAY+P4i+PTX8O3fIXcHXD8fQmOP71NbY/uqr3nRNrwGR8GUpyD1jsYbc/38bPVKK9cxK+/SxK+Up2pc8On/g91fwLVznZ8H5uNfwf6v4Jq5tlTeHP8AuOJ3ttrj/QfgxUth1kIIibbdIte+DEUHbD3+hF/ahB+R6Ow9qHbJ0cQvIg8Ad2HHPG8GbgcuAP6EnRm0BLjNGLPLyTiUOmNlBbb0vfdLm0jnXQHTn4dh05253rrXYPVcOO+/YOSNp3fs6JvtiNK3ZsPcCbbBtLYa+lwMl/8OBk1uPw2cyiscm0RGRHoA9wGpxpjhgD8wE3geuMkYMwp4A3jEqRiUahW5O+HFSbD/a5j2d/jRGtuv/O1b4d9P2nrw1nRgje2y2HcCXPpYy87R+zyYsxz6T7L91X+0xo4aHTpVk75yvKonAAgVkWogDMjGlv7rmqyj3duUah3lhbZE3lrD7dM/tY2mAcFw2/t2SD/Y1+/dD//+PRzeZkv/jQ2sqir1fNBVzlZ443rbFXLGfM+6KDYmphfc8GrLj1edlmMlfmNMFvAUkAEcBIqMMZ9gq34+EJFM4GbgSadicNKECRP4+OOPT9j27LPPcu+99zZ6TESEHWSTnZ3NjBkzGj3vyT2YTvbss89SVlZ27P2UKVMoLCz0NPTOa82L8Ife8NRAeONG+PcfbOIuzTv9cxkDX/8N3rgBYnrD3cuPJ32wXwTT/xcuewK2vWurfooybZXQrs9hxVOw8Cb48xD4XRIsvguqyxu/HtgG2VemQkAo3LzUTjKmlAMcK/GLSCwwDegDFAJvi8hs4FpgijFmlYj8DHga+2Vw8vFzgDkAvXo1MJzZy2bNmsXChQu5/PLLj21buHAhf/rTn5o9NikpiXfeeafF13722WeZPXs2YWG269wHH3zQ4nN1Gnnp8PEjkDzWjtDMWgc7P+bYlLrRvaDH2cdHvJ7cV9oY2/BZN2IyYxUc+NbO4XLNPxousYvA+T+G+EGw+E54brSd/6VOl352QFVwlJ2GIH83zHyj4fnY83fbpC9+tkqmS99W/fEoVZ+TVT2XAnuNMbkAIrIE27A70hizyr3PW8BHDR1sjJkLzAXbj9/BOFtkxowZPPLII1RWVhIcHMy+ffvIzs5m1KhRTJo0iSNHjlBdXc0TTzzBtGnTTjh23759XHXVVWzZsoXy8nJuv/12tm3bxpAhQygvP14qvOeee1izZg3l5eXMmDGDxx57jOeee47s7GwmTpxIfHw8y5cvJyUlhbS0NOLj43n66aeZN28eAHfddRf3338/+/btY/LkyYwfP56vv/6aHj16sGzZMkJDQ9v0Z+aYmmpYMsfOZ3LjaxDZzW6vKHaviLT++DD4bcvcB4kdEZl0th2slLXODmICO91At+HwvcfhvB/bLo1NGXgZ3PkprPqHrV6p+2IJjTm+T/9JsPhu2yd/5oITJxI7st8m/dpqO12vAyM1larPycSfAYwTkTCgHJgEpAHXi8hAY8xO4HvA9jO+0ocP2wUYWlO3s2By47VQcXFxjB07lo8++ohp06axcOFCbrzxRkJDQ1m6dClRUVHk5eUxbtw4pk6d2uh6ts8//zxhYWFs2rSJTZs2nTCl8m9/+1u6dOlCTU0NkyZNYtOmTdx33308/fTTLF++nPj4E2c2XLt2LfPnz2fVqlUYYzj33HO5+OKLiY2NJT09nTfffJMXXniBG264gcWLFzN79uzW+Vl5KmebLY03NfS/JVb+2Sb2618+nvTBluj7XGgfdUrzT/wi2LvStgkMvPz4ZGFdG5t/vQmJg+HqZxv/fPCVcNen8OZMOzPjtL/DWTOgKMvO1VJ1FG59v2OPQFUdhmOJ312V8w6wDnAB67El+ExgsYjUAkeAO5yKwWl11T11iX/evHkYY/jVr37FihUr8PPzIysri5ycHLp169bgOVasWMF9990HwIgRIxgxYsSxzxYtWsTcuXNxuVwcPHiQbdu2nfD5yb766iuuueaaY7ODXnvttaxcuZKpU6fSp0+fYwuz1J/Suc3s+RJenQph8TD6Fki9veEZCU9X5lr48o9w1g12SoHmhMfBgEvto611HWbbCt662VYNZa+HHR/adoFbl0H3xv9tlWpNjvbqMcY8Cjx60ual7kfraaJk7qTp06fz4IMPHltda/To0bz88svk5uaydu1aAgMDSUlJaXAa5voa+mtg7969PPXUU6xZs4bY2Fhuu+22Zs/T1PQbddM5g53SuX6VUptY86IdRdprnF0R6atnbCn7nLug36Tmq1MaUlUGS+fYUv6U5ttW2oXweDvfzgc/tfPMBIbDzUtOnUNeKQd1nMVg26GIiAgmTJjAHXfcwaxZdh7woqIiEhMTCQwMZPny5ezfv7/Jc1x00UUsWLAAgC1btrBp0ybATuccHh5OdHQ0OTk5fPjhh8eOiYyM5OjRow2e65///CdlZWWUlpaydOlSLrzwwlP2a3NHc2DHB3ZK3ZkL4P7NcNFDtqplwQx4bhR89aythjkdnz0K+bts75r69entXUAQXP0czJh3YhdRpdqIJv4zNGvWLDZu3MjMmTMBuOmmm0hLSyM1NZUFCxYwePDgJo+/5557KCkpYcSIEfzxj39k7NixgF1J6+yzz2bYsGHccccdJ0znPGfOHCZPnszEiSfOnzJ69Ghuu+02xo4dy7nnnstdd93F2Wd7MNTfaetfs3Ogj7ndvo9OtqsePbDV9lWPTrZJ/OkhsOQHdgBTc5MH7v7Cjmw99x470KmjEbHrsfbQZTJV29PZOdUJWv3nWlsDfxkFXVLsknONObzdLni9caFt6Ox2lq0G6nMRcFJVmKsSXnNPEfyDL+0CFkqpU+jsnMo7dn8BRRnwvWamHkgcAlc+BZc+ahfXWP0ivPeTxvf3C4BZb2jSV6oFNPErZ6XNt0vpDb7Ks/2DI+2skWNut6tM5e9ueL/EwZ7NWKmUOkWHTvzGmEb7x6vT1+rVfkVZdmHrC+47vmqUp0Sg51j7UEq1qg7buBsSEkJ+fn7rJysfZYwhPz+fkJAQzw4oL7RL7jVl/Wt2xabRt55xfEqp1tNhS/zJyclkZmaSm5vr7VA6jZCQEJKTk5vfsfggvDzFPt/09okjY+vUuOziH/0ugS59Wj9YpVSLddjEHxgYSJ8+mlDaXEmuHYFbcthOHfzGjXYmyV7nnrjfrk+hOAuu6JCTryrVqXXYqh7lBWUF8Oo0KDxg13e9/QM7anbBDMg6ac3ftPkQ0c2u9qSUalc08SvPlBfavvP5u2DWm3a64chutm9+aCy8di0ctKOOKcywi3qPvllXe1KqHdLEr5pXedSW6nO2wo2vQ796I4aje9jkHxQBr023A7HWuVd9Gn2Ld+JVSjVJE79qWlUpLLjBzqtz/ct27vmTxfa2i4f4Bdp55de+DAMua53ZN5VSra7DNu6qFsheD8v+CyqLPT+mqtQuVHLdizCkiUFYcf1s8p8/xS5oknr7mcerlHKEJn5fUXLYrgFrDPS9+PSOHTIVBk9pfr+EQXYFqR0f2BK/Uqpd0sTvC1xVsOgW2yvnzk+cXfAjcbB9KKXaLU38vuCjX0DGN3DdS7rKk1LK2cZdEXlARLaKyBYReVNEQsT6rYjsFJHtInKfkzH4vLT5kDYPLviJXeNVKeXzHCvxi0gP4D5gqDGmXEQWATOxk6v3BAYbY2pFJNGpGHxexrfwwc/s0oaTTl4BUynlq5yu6gkAQkWkGggDsoEngO8bY2oBjDGHHY7BNxVl2UW9Y3rCjJfAz9/bESml2gnHqnqMMVnAU0AGcBAoMsZ8AvQDbhSRNBH5UEQGNHS8iMxx75OmE7GdpupyeGs2VJfBzDfsyFqllHJzLPGLSCwwDegDJAHhIjIbCAYq3MuBvQDMa+h4Y8xcY0yqMSY1ISHBqTA7l6IsWP47u9Rh9jq45v/sylZKKVWPk1U9lwJ7jTG5ACKyBDgfyAQWu/dZCsx3MIbOr7YW9n4Ja16EHR+CqYUB34Nx/zhxagWllHJzMvFnAONEJAwoByYBaUAxcAm2pH8xsNPBGDouY6Bgjx1tm70eKooa2sk24ObvgrA4OP/HdsRsbEpbR6uU6kAcS/zGmFUi8g6wDnAB64G5QCiwQEQeAEqAu5yKocPZvRz2rrDVNPWTfUAIhHZp+JjY3nDRz2HoNAj0cPUspZRPc7RXjzHmUeDkfoSVwJVOXrdDWv86LPsR+AVA4lAYdg0kjbYLiicO0emNlVKtRkfutgcH1sD7D0DfiXau+8BQb0eklOrEdFpmbzt6yHa9jEqCGfM06SulHKclfm9yVdqkX3kUbl4CYY3U4yulVCvSxO8txsC/fgqZa+CGV6HrMG9HpJTyEVrV4y1rXoT1r8FFP7M9cpRSqo1oid8p3/zdJveuw6HHaHcPnVEQEg37/gMfPQwDr4AJv/J2pEopH6OJ3wk7P4aP/9t2yzy0Cba/e/yzuAFQmguxfeDaueCnf3QppdqWJv7WlpcOi++CbsPhjk8gKMyufJW9DrLco3CDwu0atiHR3o5WKeWDNPG3popiWPh9O9hq5hs26YPtrdP/UvtQSikv08TfWmprYckcyN8NtyyDmF7ejkgppRqkib+1/Pv3sPNDmPxH6HOht6NRSqlGactia9j2Lqz4I4yaDWPneDsapZRqkib+M5WzDZb+EHqMgSv/DCLejkgppZqkif9MHFgNr06D4Ai4cYFOi6yU6hA08bfUhjfg5Stt18xb34Oo7t6OSCmlPKKNu6ertgY+exS+/iv0uQiuf0UnV1NKdSia+E9HRTEsvhPSP4Fz7oYrfq8LpCilOhxHq3pE5AER2SoiW0TkTREJqffZX0WkxMnrt6r83fDipbD7C7jyabjyKU36SqkOybESv4j0AO4DhhpjykVkETATeFlEUoEYp67d6kpyYd4VUFsNNy+1VTxKKZ9njKHSVUtZVQ1lVS4qqmsor6olKSaEuIhgb4fXKKeregKAUBGpBsKAbBHxB/4EfB+4xuHrnzlj4L377MLnc5brvPlK+bCi8mpW7Mzli+8OszI9l/zSKow5dT8RGJEcw8RBCVwyOJHhSdH4+bWfrt6OJX5jTJaIPAVkAOXAJ8aYT0TkJ8C7xpiD0kSfdxGZA8wB6NXLi9MfrH8ddnwAl/9Ok75SHdiR0iq2HSwmu7CcsqoaSipdlFa6jr2urTXERQQRFxFMXHgQ8ZHBxIcHE+AvfJWex+ff5bBm3xFqag1dwoO4eGACPWNDCQnyJyzQn7CgAEKC/AkJ8OO7Q0f54rvD/OXzdJ79LJ34iGAmDErg3D5dSIkPp3eXMBIig2koBxaUVrE7t4Tdh0vYnVvCzeNS6BUX1qo/CzENfV21xolFYoHFwI1AIfA2sASbzCcYY1wiUmKMiWjuXKmpqSYtLc2ROJt0ZB88fwEknQ23vKtTKCvVAZRVuThUVEH64RK2ZhezLbuYbdlFZBdVnLKvv58QHuRPeHAAfiLkl1ZSUV3b4HkHd4tk0pBELhnclVE9Y/D3oASfX1LJlztzWb4jlxU7cykqrz72WWigP726hNErLozo0ED255ey63AJR8qO7xMc4Mf/3TyGCYMSW/CTABFZa4xJPXm7k1U9lwJ7jTG57gCWAI8BocAu9zddmIjsMsb0dzCOlqmtgaX3gPjB9Oc16SvloCpXLaWVLo6UVXGouIJDRRUcLLLPh4orKCitIsjfj9Agf0ID/QkJ9Cc0yI8gf3+OlFWRU1xBTnEFh4srOVrpOnZeP4G+CRGkpnRhWFIUQ5Oi6N0lnPBgm+yDA/xOKHUbYyirqiGvpJK8kirySyopq6ohNSWW5NjTL3XHRQRz7ehkrh2djKumloyCMvYXlJGRX8b+/DIyCkrZl1dKUXk1KXHhXDG8G/0SIuiXGEH/hAh6xIQ6UkXkZOLPAMaJSBi2qmcS8LQx5q91O7hL/O0v6QN88zfI+Bqm/wNieno7GqXalaLyavbklrAnt5ScoxVMG9WDHjGhzR5XUuniqY93sGpvASWV1ZRW2mqWKlfDpezo0EC6R4fQJTwIV40h92gl5dU1lFfVUFFdQ6WrlpiwQLpGhTCoWyQXDkiga1QIiZHB9E0IZ3C3KEKD/D2+LxEhPDiA8OAAeseFe3ycJwL8/eibEEHfhGYrORznZB3/KhF5B1gHuID1wFynrteqDm2BL56AIVfDyJnejkYprykqr+a7g8VsP1jMjhxb57wnt5S8ksoT9vv7F7v4+RWDuXlc70ZLqKv25PPTtzeSXVjORQMTGBQaQUSITbIRQQFEhAQQHRpIt+gQukeH0i0q5LSStvKcY3X8ralN6/hdlTB3ol0e8d5vITyuba6rlJfV1hpW7S3gP7vy+O5QMdsPHiWrsPzY5zFhgbYaIiHcllzjw+mXGIGfCL9etoWV6XmM7hXDH64bwYCukceOq6iu4amPd/DSf/bSq0sYT98wkjG9dbR7W/BGHX/HtPy3cHgrfH+RJn3lE3bmHGXp+iyWrc8iu6gCfz+hb3w4Y3rHctO4XgzpHsWQblF0jWq4FwrAq3eMZcm6LP7nX9u48rmv+NHE/twzoR87Dh3lwUUbSD9cwuxxvfjl5CGEB2va8Tb9F6gvdwf85zkYfSsMvNzb0SjVKowxVNcYyqtr3AOMaiitcvHN7nyWrMti28Fi/P2EiwbE8/CUIVw6JJGwoNNLDSLCdWOSuXhQAo+9t41nPtvJkvWZZB0pJy4iiFfuGMvFAxMcukN1ujTx15c2z07DMOnX3o5E+bCswnL+k57HkbIqCsurKSyr4khpNUfKqiivriEkwJ/gQD9CA/2P9XIJ9PejtMpFcbmL4opqisurOVrhori8mtIqF7WN1OiOTI7mN1cP5aqRScS3wkjT+Ihg/jrrbKaPSuLx97dx1YjuPDZ1ONFhOr1Je6KJv05VGWx4E4ZMhfB4b0ejfNS3e/L5wWtrj/X3DvATYsKCiA0LJDYsiC7hQVRW13K0wnVCD5eqmloiggOICgkkKjSAXl3CiAoNJDIkgIjgAELqukC6u0GGBPgzqFukYz1MJg3pyqQhXR05tzpzmvjrbF0KlUWQeru3I1HtkDGGvXmlfLungNyjlVw0MJ6RyTFN9rHenVvCsg3ZrEzP5ZJBifzg4n4EBTQ+HmTJukx+sXgTvbqE8cbd59I7LpzwIP9G69WVailN/HXWzof4gdD7Am9HotoBYwz788v4Zk8+37ofOcXHuzA+89lO4iOCuXRIIpOGdGV8/3hCg/zJKa7gvY3ZLNuQzeasIkRgQGIEf/50J8s2ZvPb6cM5t2/cKdd69rN0/vJ5Ouf3i+P5m8Zo1YhyVLOJX0T+C1hgjDnSBvF4x6HNkLkGLv+9rpnrgypdNew6XML2g0dtn/VDxXx38Cj5pVUAJEQGc17fOMb1jeO8fnHEhgXy5c5cPt2Ww782HWThmgMEB/gxsGskW7KLMAbO6hHNI1cO4eqRSXSNCmH5d4f5f8u2cOPcb7l+TDK/mjKE2PAgKl01/HLxZpasz2LGmGR+d81ZTf5VoFRraLYfv4g8gZ1OeR0wD/jYtHHnf8f78b//oJ2M7aff6WpanVCVq5b/7Mojq7CcvJJK8kuqjj3nllRyoKAMl7v1MzjAj0HdIhnSLYqzkqM5r18cfePDG61uqXLVsnpvAZ9tz2FzVhEX9I9n2qgk+jVQd15eVcNfPk/nhZV7iA4N5KHLBrFsQxar9hbw0GUD+dHE/lqto1pVY/34PRrAJfa38TLgdiAVWAS8ZIzZ3dqBNsTRxF9ZAn8eDIOvhGv/z5lrKK84XFzBglUZvLE6g9yjx6tpYsICia+bgTEimJT4MAZ3i2JI9yj6xId7NPnWmdh+sJhfLd3M+oxCgvz9+NP1I5g2qoej11S+6YwGcBljjIgcAg5hp1+IBd4RkU+NMT9v3VDb2JbFUHUUUu/wdiSqFRhjWJdxhJe/3s+Hmw/iqjVMGJTAzeN6M7xHNF3Cgwj0925VypDuUSz+4fm8tymblLhwRvbsOGsSqc7Bkzr++4BbgTzgReBnxphqEfED0oGOnfjT5kHiUOg51tuRqDNwtKKa9zYeZMGq/WzNLiYyJHnoxJEAABiLSURBVIBbz0/h5nG9SYlv3cm2WoOfn2gpX3mNJyX+eOBaY8z++huNMbUicpUzYbWR7PVwcANM/pM26nqJMYbMI+VEhgQQExZ02seuyzjCwtUHeH/TQcqraxjcLZLfXjOca87ucdqjT5XyFZ78z/gAKKh7IyKR2HV0VxljtjsWWVtImw8BoTDiBm9H4lMqXTWs3lvAF98d5ovvDrM/vwyAHjGhDEuKYlhSNMN72Ocu4UGUV9VQVu2yz1U1lFfXsPFAIW+tOUD64RLCg/yZfnYSN57Ti5HJ0dpAqlQzPEn8zwOj670vbWBbx1NRDJvfgbOug1CtY3Waq6aWpeuz+Gx7Dl+l51FaVUNwgB/n9Yvj9vNTqHDVsiWriG3ZxXy6PafBdUxPNqpnDH+47iyuHJFEhE78pZTHPPnfIvW7b7qreDr+/7LNi6C6FMZoo67T9uWVcv9bG9hwoJCk6BCmn92DSwYncn6/+AbnWy+pdPHdwWK2ZBVxtMJFaJBdzzQs6PjcNEkxofRP9P6CFkp1RJ4k8D3uBt7n3e/vBfY4F1IbMAbSXoZuZ0GPjv2HS3tmjOHttEx+895WAvyEv846m6tGdG+2KiYiOIDUlC6kpuiYCqWc4Em/th8C5wNZQCZwLnbB9GaJyAMislVEtojImyISIiILRGSHe9s8EWn7selZ6yBns+3CqfXBjjhSWsU9r6/j54s3MSI5mo/uv4irRyZp/btS7UCzJX5jzGHsyN3TIiI9gPuwDcHlIrLIfZ4FwGz3bm8Ad3H8r4m2kbnGPg/u2J2S2quV6bk89PZGCkqr+OXkwdx9YV9HFoxWSrWMJ/34Q4A7gWFASN12Y4wnleMBQKiIVANhQLYx5pN6514NJJ9u0Ges6IDtzROuC0O0RKWrhgMF5RwoKCOzsJzswnKyjrifC8s5WFRB/8QIXrr1HIb3iPZ2uEqpk3hSx/8a8B1wOfA4cBPQbDdOY0yWiDwFZADlwCcnJf1A4GbgJy2I+8wUHYDoZK3maUZ1TS1p+46w7WAx+/JK2Zdfyt68UrILy09Y2CPAT+geE0KPmFDO6xfHwK6R3Hpeii6UrVQ75Uni72+MuV5EphljXhGRN4CPmztIRGKBaUAfoBB4W0RmG2Ned+/yv8AKY8zKRo6fg7stoVevXh6EeRqKMm3iV6fIK6lk+XeHWb7jMCt35nG00gVAZEgAfeLDGd0rlmtHJ5MSF0bvuDB6xISREBns+Pw2SqnW40nir3Y/F4rIcOx8PSkeHHcpsNcYkwsgIkuwjcSvi8ijQALwg8YONsbMBeaCnaTNg+t5rihT19StJ7uwnCXrMvl0+2E2ZRZiDHSNCubKEd25ZHAiY3rH0iU8SBtmleokPEn8c92l90eAd4EI4P95cFwGME5EwrBVPZOANBG5C1ttNMkYU9uysM+AqxJKciC6Z5tfuj2prqnl8+05LFxzgC935gJ2QNSDlw5k4uBEhiVFaaJXqpNqMvG7J2Irdi/CsgLo6+mJjTGrROQd7Dz+LmA9tgRfCuwHvnEnliXGmMdbFn4LFGXaZx+t6tmdW8KitAMsXptJXkkV3aJC+PHE/lyf2pOeXcK8HZ5Sqg00mfjdo3T/Czv//mkzxjwKPHo613ScDyX+4opqNmcWseFAIRsPFLIxs5Cc4kr8/YRJgxOZObYnFw9M1Pp5pXyMJ0n4UxF5CHgLW1oHwBhT0Pgh7dixxN95q3o+2XqIP3z0Hbtzj/1z0Sc+nPP6xjGqZwxTRnQnMTKkiTMopTozTxJ/XX/9H9XbZjiNap92pSgTEIhK8nYkjvhsWw73LlhH/8QIHrpsICN7xjCiR4wu3q2UOsaTkbt92iKQNlOUARFdISDY25G0uhU7c7l3wTqGJkXx+l3nEhWiyV4pdSpPRu7e0tB2Y8yrrR9OG+ikffi/3ZPPnNfS6JsQzqt3jNWkr5RqlCdVPefUex2C7Za5Dui4ib/bWd6OolWt3X+EO15eQ3JsGAvuOve0V7JSSvkWT6p6flz/vYhEY6dx6HiMsYl/0GRvR+Kx2lrD0vVZFJVXMywpiqFJUUTWK81vzizitnmrSYwM5o27ziUuovNVYSmlWldLulaWAQNaO5A2UZoHrooO06OnqKyan769gc+2Hz5he0pcGMOSohnYNZL5X+8lKjSQBXePIzFKe+oopZrnSR3/e9hePGDn7x9KC/v1e13RAfvcAer4N2UWcu+CdeQUV/Do1UOZclZ3tmYXsTWrmK3ZxWzMLORfmw/SPTqEN+8eR4+YUG+HrJTqIDwp8T9V77UL2G+MyXQoHmd1gD78xhheX5XB/7y3jfiIIBb94DzO7hULQNeoEC4Z3PXYvkVl1YQG+RMU4Ml6OkopZXmS+DOAg8aYCgARCRWRFGPMPkcjc0I7H7VbWunil0s28+7GbCYMSuCZG0YRG954Q632zVdKtYQnif9t7KyadWrc285pePd2rOgABIZDaKy3IzmBq6aWD7Yc4tlPd7Ivv5SHLhvIvRP666pVSilHeJL4A4wxVXVvjDFVItIx+wu2swVYKqpreGdtJnNX7CGjoIy+CeG8fue5nN8/3tuhKaU6MU8Sf66ITDXGvAsgItOAPGfDckhRJsR4v36/uKKa17/dz7yv9pFXUsnI5Gh+NWUMlw3tqqV8pZTjPEn8PwQWiMjf3O8zgQZH87Z7RZnQfaRXQ/hmtx1he7TCxYUD4rlnwijO6xunc98rpdqMJwO4dmMXVIkAxBhz1PmwHFBdDqW5Xm3Yzcgv454Fa92DrcZxVrIuRK6UanvN9gMUkd+JSIwxpsQYc1REYkXkibYIrlUVZdlnL3XlLKl0cderazAGXrr1HE36Simv8aQD+GRjTGHdG/dqXFOcC8khxwZvtX3ir601PPDWBnbnlvK/N40mJT68zWNQSqk6niR+fxE5NgGMiIQCHk0IIyIPiMhWEdkiIm+KSIiI9BGRVSKSLiJvtVkPIS/24X/ms518ui2HR64cwgXaY0cp5WWeJP7Xgc9F5E4RuRP4FHiluYNEpAdwH5BqjBkO+AMzgT8AzxhjBgBHgDtbGvxpKTqANxZgeX9TNn/9Yhc3pvbktvNT2vTaSinVkGYTvzHmj8ATwBDsPD0fAb09PH8AECoiAUAYcBC4BHjH/fkrwPTTjLllijIhsjv4t91o1y1ZRTz09kZSe8fy+PRh2nNHKdUueDrJyyGgFrgOOx//9uYOMMZkYef5ycAm/CJgLVBojHG5d8sEejR0vIjMEZE0EUnLzc31MMwmFB1o0z78OcUVzHk1jS5hQTw/ewzBAf5tdm2llGpKo905RWQgtmpmFpCPXWxdjDETPTmxiMQC04A+QCF2moeGJsI3DWzDGDMXmAuQmpra4D6npSgTks4+49M0p6C0ihdW7uGVr/dRawzv/PB8EiJ1jnylVPvRVD/+74CVwNXGmF1gG2tP49yXAnuNMbnuY5dg5/yJEZEAd6k/GchuUeSno7bWJv4hVzt2iYLSKl50J/yy6hquGpHETyYNoH9ihGPXVEqplmgq8V+HLfEvF5GPgIXA6VRSZ2AHfoUB5dgqojRgOTDDfb5bgWUtiPv0lOZCTZUjXTmP1Cvh1yX8+y7pz4Cuka1+LaWUag2NJn5jzFJgqYiEYxtgHwC6isjzwFJjzCdNndgYs0pE3sGuz+sC1mOrbv4FLHQPAlsPvNQqd9IUh7pyfrM7n/sWrievpFITvlKqw/BkyoZSYAF2vp4uwPXAw0CTid997KPAoydt3gOMPf1Qz0ArD96qrTU8/+Vu/vzJDlLiw3n59nMYlqQjcZVSHcNprblrjCkA/s/96DhaccnFI6VVPLBoA//ekcvVI5P4/bVnERHckqWLlVLKO3wjYxVlQlAkhJxZqXzt/iP8+I115JVU8T/ThzP73F7aN18p1eH4TuI/wwVYXv1mH4+/t43uMSEsvud8nWRNKdVh+UjiP7PBW9/uyefXy7ZyyeBEnrlhlK51q5Tq0Dwdudux1ZX4W6CiuoZfLtlMry5h/O37Z2vSV0p1eJ2/xF9VCmX5LU78z36Wzt68UhbcdS5hQZ3/x6WU6vw6f4n/DBZg2ZJVxAsr93BDarJOp6yU6jR8IPG3rA9/dU0tP39nE13Cg/jvKUMdCEwppbyj89ddtHDU7gsr97DtYDH/mD1a6/WVUp2Kb5T4xc/Oxe+h3bklPPtZOpOHd+OK4Z4fp5RSHYEPJP5MiEwCf8/+uKmtNfxy8WZCAvx4bNowh4NTSqm25xuJ/zT68C9YncHqfQU8ctVQEiNDHAxMKaW8wwcS/wGP6/dzj1byhw+/Y3z/eK4f0/aLsiulVFvo3Im/ttZ25/Qw8b/6zT5Kq1w8Nk3Xx1VKdV6dO/GX5EBttUeJv6zKxWvf7ueyoV3pl6CrZimlOq/OnfiPdeXs1eyub6dlUlhWzZyL+joclFJKeVcnT/yezcPvqqnlxa/2MKZ3LGN6d2mDwJRSynscS/wiMkhENtR7FIvI/SIySkS+dW9LExHnVuPyMPF/vDWHAwXl3H2hlvaVUp2fYyN3jTE7gFEAIuIPZAFLgReAx4wxH4rIFOCPwARHgijKhOBoCIlqKk7mrthNn/hwvje0qyNhKKVUe9JWUzZMAnYbY/aLiAHqMnE0kO3YVUfMhOSm/6BYvbeAjZlFPDF9OP5+2pNHKdX5tVXinwm86X59P/CxiDyFrWo6v6EDRGQOMAegV6/mG2cblDzGPpowd8UeuoQHMUP77SulfITjjbsiEgRMBd52b7oHeMAY0xN4AHipoeOMMXONManGmNSEhARHYtt1+Ciff3eYW87rTUigvyPXUEqp9qYtevVMBtYZY3Lc728Flrhfvw0417jbjBdW7CU4wI+bx/X2VghKKdXm2iLxz+J4NQ/YOv2L3a8vAdLbIIZTHC6uYOn6LK5PTSYuItgbISillFc4WscvImHA94Af1Nt8N/AXEQkAKnDX47e1V77ZR3VtLXeO1y6cSinf4mjiN8aUAXEnbfsKaLrF1WGllS5e/zaDy4d2o098uDdDUUqpNte5R+424tNtORSVV3PnhX28HYpSSrU5n0z8e/JKEYGRyTHeDkUppdqcTyb+AwVlJEWHEhTgk7evlPJxPpn5DhSU0bNLqLfDUEopr/DJxJ9RUEbP2DBvh6GUUl7hc4m/orqGw0cr6dVFE79Syjf5XOLPPFIGQE9N/EopH+VziT+jQBO/Usq3+VziP1BQDqBVPUopn+VziT+joIzQQH/iI4K8HYpSSnmFzyX+uq6cIrroilLKN/lc4teunEopX+dTid8YQ+aRcm3YVUr5NJ9K/EfKqimpdGniV0r5NJ9K/AfcXTm1R49Sypf5VOLP0MSvlFLOJX4RGSQiG+o9ikXkfvdnPxaRHSKyVUT+6FQMJ6tL/MmxOkGbUsp3ObYClzFmBzAKQET8gSxgqYhMBKYBI4wxlSKS6FQMJ8s8UkZ8RBDhwY4uPKaUUu1aW1X1TAJ2G2P2A/cATxpjKgGMMYfbKAYyCspI1q6cSikf11aJfybwpvv1QOBCEVklIl+KyDltFAMHCsq1fl8p5fMcT/wiEgRMBd52bwoAYoFxwM+ARdLAMFoRmSMiaSKSlpube8ZxuGpqySos1wVYlFI+ry1K/JOBdcaYHPf7TGCJsVYDtUD8yQcZY+YaY1KNMakJCQlnHMTBogpqao2W+JVSPq8tEv8sjlfzAPwTuARARAYCQUCe00Ec0OmYlVIKcDjxi0gY8D1gSb3N84C+IrIFWAjcaowxTsYB9ebh18ZdpZSPc7RfozGmDIg7aVsVMNvJ6zbkwJEyAvyE7tEhbX1ppZRqV3xm5G5GQTlJMaEE+PvMLSulVIN8JgseKCjThl2llMLHEr925VRKKR9J/KWVLvJLq7RHj1JK4SOJ/8ARnZVTKaXq+EbiLygHtCunUkqBjyR+nYdfKaWO84nEf6CgjIjgAGLCAr0dilJKeZ3PJP6eXcJoYC44pZTyOT6R+DMKyuilXTmVUgrwgcRvjOHAkTJt2FVKKbdOn/hzSyqpqK6lV5wmfqWUAh9I/NqVUymlTuQDiV/n4VdKqfo6feKv68OfHKuNu0opBT6Q+A8UlNE1KpiQQH9vh6KUUu1Cp0/8GTods1JKncCxxC8ig0RkQ71HsYjcX+/zh0TEiMgpC623pswj5dqwq5RS9Ti29KIxZgcwCkBE/IEsYKn7fU/sWrwZTl0foMpVS3ZRuTbsKqVUPW1V1TMJ2G2M2e9+/wzwc8DRRdazC8sxRnv0KKVUfW2V+GcCbwKIyFQgyxizsakDRGSOiKSJSFpubm6LLqqzciql1KkcT/wiEgRMBd4WkTDgv4FfN3ecMWauMSbVGJOakJDQomvXLcCiSy4qpdRxbVHinwysM8bkAP2APsBGEdkHJAPrRKSbExfOKCgjyN+PrpEhTpxeKaU6JMcad+uZhbuaxxizGUis+8Cd/FONMXlOXLhPXDjXnN0DPz+djlkppeo4mvjdVTvfA37g5HUaM3NsL2aO7eWNSyulVLvlaOI3xpQBcU18nuLk9ZVSSp2q04/cVUopdSJN/Eop5WM08SullI/RxK+UUj5GE79SSvkYTfxKKeVjNPErpZSPEWMcnSCzVYhILrC/md3iAUdGALdzet++Re/b95zJvfc2xpwy2VmHSPyeEJE0Y0yqt+Noa3rfvkXv2/c4ce9a1aOUUj5GE79SSvmYzpT453o7AC/R+/Ytet++p9XvvdPU8SullPJMZyrxK6WU8oAmfqWU8jEdPvGLyBUiskNEdonIw96Ox0kiMk9EDovIlnrbuojIpyKS7n6O9WaMThCRniKyXES2i8hWEfmJe3unvncRCRGR1SKy0X3fj7m39xGRVe77fsu9rnWnIyL+IrJeRN53v+/09y0i+0Rks4hsEJE097ZW/z3v0IlfRPyBv2PX9R0KzBKRod6NylEvA1ectO1h4HNjzADgc/f7zsYF/NQYMwQYB/zI/e/c2e+9ErjEGDMSGAVcISLjgD8Az7jv+whwpxdjdNJPgO313vvKfU80xoyq13e/1X/PO3TiB8YCu4wxe4wxVcBCYJqXY3KMMWYFUHDS5mnAK+7XrwDT2zSoNmCMOWiMWed+fRSbDHrQye/dWCXut4HuhwEuAd5xb+909w0gIsnAlcCL7veCD9x3I1r997yjJ/4ewIF67zPd23xJV2PMQbAJknqL2XdGIpICnA2swgfu3V3dsQE4DHwK7AYKjTEu9y6d9Xf+WeDnQK37fRy+cd8G+ERE1orIHPe2Vv89d3TN3TYgDWzT/qmdlIhEAIuB+40xxbYQ2LkZY2qAUSISAywFhjS0W9tG5SwRuQo4bIxZKyIT6jY3sGunum+3C4wx2SKSCHwqIt85cZGOXuLPBHrWe58MZHspFm/JEZHuAO7nw16OxxEiEohN+guMMUvcm33i3gGMMYXAv7FtHDEiUldo64y/8xcAU0VkH7b69hLsXwCd/b4xxmS7nw9jv+jH4sDveUdP/GuAAe7W/iBgJvCul2Nqa+8Ct7pf3wos82IsjnDX774EbDfGPF3vo0597yKS4C7pIyKhwKXY9o3lwAz3bp3uvo0xvzTGJBtjUrD/p78wxtxEJ79vEQkXkci618BlwBYc+D3v8CN3RWQKtjTgD8wzxvzWyyE5RkTeBCZgp2nNAR4F/gksAnoBGcD1xpiTG4A7NBEZD6wENnO8zvdX2Hr+TnvvIjIC25jnjy2kLTLGPC4ifbEl4S7AemC2MabSe5E6x13V85Ax5qrOft/u+1vqfhsAvGGM+a2IxNHKv+cdPvErpZQ6PR29qkcppdRp0sSvlFI+RhO/Ukr5GE38SinlYzTxK6WUj9HErxQgIjXuGRHrHq024ZuIpNSfUVUpb+voUzYo1VrKjTGjvB2EUm1BS/xKNcE9P/of3PPirxaR/u7tvUXkcxHZ5H7u5d7eVUSWuufQ3ygi57tP5S8iL7jn1f/EPRJXKa/QxK+UFXpSVc+N9T4rNsaMBf6GHSWO+/WrxpgRwALgOff254Av3XPojwa2urcPAP5ujBkGFALXOXw/SjVKR+4qBYhIiTEmooHt+7CLoexxTxR3yBgTJyJ5QHdjTLV7+0FjTLyI5ALJ9acScE8l/al7IQ1E5BdAoDHmCefvTKlTaYlfqeaZRl43tk9D6s8pU4O2rykv0sSvVPNurPf8jfv119iZIwFuAr5yv/4cuAeOLaIS1VZBKuUpLXUoZYW6V7qq85Expq5LZ7CIrMIWlGa5t90HzBORnwG5wO3u7T8B5orIndiS/T3AQcejV+o0aB2/Uk1w1/GnGmPyvB2LUq1Fq3qUUsrHaIlfKaV8jJb4lVLKx2jiV0opH6OJXymlfIwmfqWU8jGa+JVSysf8f1OUHWS4cA/PAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, NUM_EPOCHS+1), train_acc_list, label='Training')\n",
    "plt.plot(np.arange(1, NUM_EPOCHS+1), valid_acc_list, label='Validation')\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation ACC: 88.70%\n",
      "Test ACC: 84.55%\n"
     ]
    }
   ],
   "source": [
    "with torch.set_grad_enabled(False):\n",
    "    test_acc = compute_acc(model=model,\n",
    "                           data_loader=test_loader,\n",
    "                           device=DEVICE)\n",
    "    \n",
    "    valid_acc = compute_acc(model=model,\n",
    "                            data_loader=valid_loader,\n",
    "                            device=DEVICE)\n",
    "    \n",
    "\n",
    "print(f'Validation ACC: {valid_acc:.2f}%')\n",
    "print(f'Test ACC: {test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numpy       1.16.4\n",
      "torch       1.2.0\n",
      "matplotlib  3.1.0\n",
      "torchvision 0.4.0a0+6b959ee\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%watermark -iv"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}