practical3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MeDW4HuegASG"
   },
   "source": [
    "# DL Indaba Practical 3\n",
    "# Convolutional Neural Networks\n",
    "*Developed by Stephan Gouws, Avishkar Bhoopchand & Ulrich Paquet.*\n",
    "\n",
    "**Introduction**\n",
    "\n",
    "In this practical we will cover the basics of convolutional neural networks, or \"ConvNets\". ConvNets were invented in the late 1980s/early 1990s, and have had tremendous success especially with vision (although they have also been used to great success in speech processing pipelines, and more recently, for machine translation). \n",
    "\n",
    "We will work to build our mathematical and algorithmic intuition around the \"convolution\" operation. Then we will construct a deep feedforward convolutional model with which we can classify MNIST digits with over 99% accuracy (our best model yet!).\n",
    "\n",
    "**Learning objectives**\n",
    "\n",
    "Understand:\n",
    "* what a convolutional layer is & how it's different from a fully-connected layer (including the assumptions and trade-offs that are being made), \n",
    "* how and when to use convolutional layers (relate it to the assumptions the model makes),\n",
    "* how backpropagation works through convolutional layers.\n",
    "\n",
    "**What is expected of you:**\n",
    "\n",
    "* Read through the explanations and make sure you understand how to implement the convolutional forwards pass.\n",
    "* Do the same for the backwards phase.\n",
    "* Train a small model on MNIST.\n",
    "* At this point, flag a tutor and they will give you access to a GPU instance. Now use the hyperparameters provided to train a state-of-the-art ConvNet model on MNIST."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 153,
     "output_extras": [
      {
       "item_id": 4
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 13901,
     "status": "ok",
     "timestamp": 1504366824744,
     "user": {
      "displayName": "Avishkar Bhoopchand",
      "photoUrl": "//lh5.googleusercontent.com/-OainnMWSi6A/AAAAAAAAAAI/AAAAAAAAAOQ/eI7Z19q5v7E/s50-c-k-no/photo.jpg",
      "userId": "105781587643595215149"
     },
     "user_tz": -60
    },
    "id": "p1tyOwH9Hiwa",
    "outputId": "f05866a1-64be-4d1f-9dcd-589c72e6eeb2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# Import TensorFlow and some other libraries we'll be using.\n",
    "import datetime\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Import Matplotlib and set some defaults\n",
    "from matplotlib import pyplot as plt\n",
    "plt.ioff()\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# Download the MNIST dataset onto the local machine.\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "za-6kOBMQuWg"
   },
   "source": [
    "# ConvNet Architectures\n",
    "When modelling an image using a regular feed-forward network, we quickly find that the number of model parameters grows exponentially. For example, our 2 layer MNIST feed-forward model from the previous practical already had over 600 000 parameters!\n",
    "\n",
    "**QUESTION**: How many parameters would a feed-forward network require if it had 2 hidden layers with 512 and 256 neurons respectively, an output size of 10 and an input image of shape [32, 32, 3]?\n",
    "\n",
    "ConvNets address this model parameter issue by exploiting structure in the inputs to the network (in particular, by making the assumption that the input is a 3D volume, which applies to images for example). The two key differences between a ConvNet and a Feed-forward network are:\n",
    "* ConvNets have neurons that are arranged in 3 dimensions: width, height, depth (depth here means the depth of an activation volume, not the depth of a deep neural network!)\n",
    "* The neurons in each layer are only connected to a small region of the layer before it.\n",
    "\n",
    "**QUESTION**: Unfortunately there is no such thing as a free lunch. What do you think is the trade-off a ConvNet makes for the reduction in memory required by fewer parameters? \n",
    "\n",
    "Generally a ConvNet architecture is made up of different types of layers, the most common being **convolutional layers**, **pooling layers** and **fully connected layers** that we encountered in the last practical. \n",
    "\n",
    "ConvNet architectures were key to the tremendous success of deep learning in machine vision. In particular, the first deep learning model to win the ImageNet competition in 2012 was called AlexNet (after Alex Krizhevsky, one of its inventors). It had 5 convolutional layers followed by 3 fully connected layers. Later winners included GoogLeNet and ResNet which also used batch normalisation, a technique we will see in this practical. If you're curious, have a look at [this](https://medium.com/towards-data-science/neural-network-architectures-156e5bad51ba) link for a great summary of different ConvNet archiectures. \n",
    "\n",
    "We will start by implementing the forward and backward passes of these layers in Numpy to get a good sense for how they work. Afterwards, we will implement a full ConvNet classifier in TensorFlow that we will apply to the MNIST dataset. This model should give us the best test accuracy we've seen so far! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R3-Aje8WEyfA"
   },
   "source": [
    "## Convolutional Layers\n",
    "A convolutional layer maps an **input volume**\\* to an **output volume** through a set of **learnable filters**, which make up the parameters of the layer. Every filter is small spatially (along width and height), but extends through the full depth of the input volume. (Eg: A filter in the first layer of a ConvNet might have size [5, 5, 3]). During the forward pass, we convolve (\"slide\") each filter across the width and height of the input volume and compute dot products between the entries of the filter and the input at any position. As we slide the filter over the width and height of the input volume we will produce a 2-dimensional activation map that gives the responses of that filter at every spatial position. Each convolutional layer will have a set of filters, and each of them will produce a separate 2-dimensional activation map. We will stack these activation maps along the depth dimension to produce the output volume. \n",
    "\n",
    "\n",
    "The following diagram and animation illustrates these ideas, make sure you understand them!\n",
    "\n",
    "![Convolutional Layer](images/conv_layer.png)\n",
    "![Animated Convolutional Layer](images/conv_animation.png)\n",
    "\n",
    "\\* An input volume refers to a 3 dimensional input. For example, a colour image is often represented as a 3 dimensional tensor of shape `[width, height, channels]` where `channels` refers to the colour values. A common colour encoding is RGB which has a value between 0 and 256 for each of the red, green and blue channels.\n",
    "\n",
    "### What size is the output volume?\n",
    "The size of the output volume is controlled by the hyperparameters of the convolutional layer:\n",
    "* **Filter Size** (F) defines the width and height of the filters in the layer. Note that filters always have the same depth as the inputs to the layer.    \n",
    "* **Depth** (D) of the layer defines the number of filters in the layer. \n",
    "* **Stride** (S) defines the number of pixels by which we move the filter when \"sliding\" it along the input volume. Typically this value would be 1, but values of 2 and 3 are also sometimes used. \n",
    "* **Padding** (P) refers to the number of 0 pixels we add to the input volume along the width and height dimensions. This parameter is useful in that it gives us more control over the desired size of the output volume and in fact is often used to ensure that the output volume has the same width and height as the input volume. \n",
    "\n",
    "If the width of the input volume is $w$, the width of the output volume will be $(w−F+2P)/S+1$. (**QUESTION:** Why?). Similarly for the height ($h$). \n",
    "\n",
    "**QUESTION**: What is the final 3D shape of the output volume? \n",
    "\n",
    "### Implementing the forward pass\n",
    "The parameters of a convolutional layer, with padded input $X^{pad}$, are stored in a weight tensor, $W$ of shape $[F, F, I, D]$ and bias vector $b$ of shape $[D]$ where I is the depth of $X$. \n",
    "For each filter $d \\in [0,D)$ in our convolutional layer, the value of the output volume ($O$) at position $(i, j, d)$ is given by:\n",
    "\n",
    "\\begin{align}\n",
    "O_{ij}^d = b_{d} + \\sum_{a=0}^{F-1} \\sum_{b=0}^{F-1} \\sum_{c=0}^{I-1} W_{a, b, c, d} X^{pad}_{i+a, j+b, c}    && (1)\n",
    "\\end{align}\n",
    "\n",
    "Don't be put off by all the notation, it's actually quite simple, see if you can tie this formula to the explanation of the convolutional layer and diagrams you saw earlier. \n",
    "\n",
    "**QUESTION**: The formula above assumed a stride size of 1 for simplicity. Can you modify the formula to work with an arbitrary stride?\n",
    "\n",
    "Now let's implement the forward pass of a convolutional layer in Numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "nynzAwx2_nHl"
   },
   "outputs": [],
   "source": [
    "## IMPLEMENT-ME: ...\n",
    "\n",
    "# Conv layer forward pass\n",
    "def convolutional_forward(X, W, b, filter_size, depth, stride, padding):\n",
    "    # X has size [batch_size, input_width, input_height, input_depth]\n",
    "    # W has shape [filter_size, filter_size, input_depth, depth]\n",
    "    # b has shape [depth]\n",
    "    batch_size, input_width, input_height, input_depth = X.shape\n",
    "    \n",
    "    # Check that the weights are of the expected shape\n",
    "    assert W.shape == (filter_size, filter_size, input_depth, depth)\n",
    "    \n",
    "    # QUESTION: Calculate the width and height of the output\n",
    "    # output_width = ...\n",
    "    # output_height = ...\n",
    "    #\n",
    "    # ANSWER:\n",
    "    output_width = (input_width - filter_size + 2*padding) / stride + 1\n",
    "    output_height = (input_height - filter_size + 2*padding) / stride + 1\n",
    "    ####\n",
    "    \n",
    "    # Apply padding to the width and height dimensions of the input\n",
    "    X_padded = np.pad(X, ((0,0), (padding, padding), (padding, padding), (0,0)), 'constant')\n",
    "    \n",
    "    # Allocate the output Tensor\n",
    "    out = np.zeros((batch_size, output_width, output_height, depth))\n",
    "    \n",
    "    # NOTE: There is a more efficient way of doing a convolution, but this most \n",
    "    # clearly illustrates the idea.\n",
    "    \n",
    "    for i in range(output_width):    # Loop over the output width dimension\n",
    "        for j in range(output_height):    # Loop over the output height dimension\n",
    "            \n",
    "            # Select the current block in the input that the filter will be applied to \n",
    "            block_width_start = i * stride\n",
    "            block_width_end = block_width_start + filter_size\n",
    "\n",
    "            block_height_start = j * stride\n",
    "            block_height_end = block_height_start + filter_size\n",
    "\n",
    "            block = X_padded[:, block_width_start:block_width_end, block_height_start:block_height_end, :]\n",
    "            \n",
    "            for d in range(depth):    # Loop over the filters in the layer (output depth dimension)\n",
    "                \n",
    "                filter_weights = W[:, :, :, d]\n",
    "                # QUESTION: Apply the filter to the block over all inputs in the batch\n",
    "                # out[:, w, h, f] = ...\n",
    "                # HINT: Have a look at numpy's sum function and pay attention to the axis parameter\n",
    "                # ANSWER:\n",
    "                out[:, i, j, d] = np.sum(block * filter_weights, axis=(1,2,3)) + b[d]\n",
    "                ###\n",
    "            \n",
    "    return out\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "68fLAdyYFOGF"
   },
   "source": [
    "Let's test our layer on some dummy data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 51,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 232,
     "status": "ok",
     "timestamp": 1504358221278,
     "user": {
      "displayName": "Avishkar Bhoopchand",
      "photoUrl": "//lh5.googleusercontent.com/-OainnMWSi6A/AAAAAAAAAAI/AAAAAAAAAOQ/eI7Z19q5v7E/s50-c-k-no/photo.jpg",
      "userId": "105781587643595215149"
     },
     "user_tz": -60
    },
    "id": "uG7LunQ6FNnH",
    "outputId": "2d84bea9-1ebd-4e98-824e-7b4ea2c017a9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing convolutional_forward\n",
      "PASSED\n"
     ]
    }
   ],
   "source": [
    "### Hyperparameters\n",
    "batch_size = 2\n",
    "input_width = 4\n",
    "input_height = 4\n",
    "input_depth = 3\n",
    "filter_size = 4\n",
    "output_depth = 3\n",
    "stride = 2\n",
    "padding = 1\n",
    "###\n",
    "\n",
    "# Create a helper function that calculates the relative error between two arrays\n",
    "def relative_error(x, y):\n",
    "    \"\"\" returns relative error \"\"\"\n",
    "    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))\n",
    "\n",
    "# Define the shapes of the input and weights\n",
    "input_shape = (batch_size, input_width, input_height, input_depth)\n",
    "w_shape = (filter_size, filter_size, input_depth, output_depth)\n",
    "\n",
    "# Create the dummy input\n",
    "X = np.linspace(-0.1, 0.5, num=np.prod(input_shape)).reshape(input_shape)\n",
    "\n",
    "# Create the weights and biases\n",
    "W = np.linspace(-0.2, 0.3, num=np.prod(w_shape)).reshape(w_shape)\n",
    "b = np.linspace(-0.1, 0.2, num=output_depth)\n",
    "\n",
    "# Get the output of the convolutional layer\n",
    "out = convolutional_forward(X, W, b, filter_size, output_depth, stride, padding)\n",
    "\n",
    "correct_out = np.array(\n",
    "        [[[[8.72013250e-02, 2.37300699e-01, 3.87400074e-01],\n",
    "             [1.34245123e-01, 2.86133235e-01, 4.38021347e-01]],\n",
    "            [[8.21928598e-02, 2.39447184e-01, 3.96701509e-01],\n",
    "             [4.47552448e-04, 1.59490615e-01, 3.18533677e-01]]],\n",
    "         [[[1.11179021e+00, 1.29050939e+00, 1.46922856e+00],\n",
    "             [9.01255797e-01, 1.08176371e+00, 1.26227162e+00]],\n",
    "            [[7.64688995e-02, 2.62343025e-01, 4.48217151e-01],\n",
    "             [-2.62854619e-01, -7.51917556e-02, 1.12471108e-01]]]])\n",
    "\n",
    "# Compare your output to the \"correct\" ones \n",
    "# The difference should be around 2e-8 (or lower)\n",
    "\n",
    "print 'Testing convolutional_forward'\n",
    "diff = relative_error(out, correct_out)\n",
    "if diff <= 2e-8:\n",
    "    print 'PASSED'\n",
    "else:\n",
    "    print 'The difference of %s is too high, try again' % diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U-3eBUD6E_Ph"
   },
   "source": [
    "### The derivative of a convolutional layer\n",
    "Assume we have some final loss function L and by following the steps of backpropagation, have computed the derivative of this loss up to the output of our convolutional layer ($\\frac{\\partial L}{\\partial O}$ or `dout` in the code below). In order to update the parameters of our layer, we require the derivative of L with respect to the weights and biases of the convolutional layer ($\\frac{\\partial L}{\\partial W}$ and $\\frac{\\partial L}{\\partial b}$). We also require the derivative with respect to the inputs of the layer ($\\frac{\\partial L}{\\partial X}$) in order to propagate the error back to the preceding layers. Unfortunately calculating these derivatives can be a little fiddly due to having to keep track of multiple indices. The calculus is very basic though!\n",
    "\n",
    "We start with the easiest one, $\\frac{\\partial L}{\\partial b}$:\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial L}{\\partial b} &= \\frac{\\partial L}{\\partial O} \\frac{\\partial O}{\\partial b} && \\vartriangleright \\text{(Chain Rule)} \\\\\n",
    " &= \\frac{\\partial L}{\\partial O} \\mathbf{1} && \\vartriangleright (\\frac{\\partial O}{\\partial b} = 1 \\text{ from equation } (1))\n",
    "\\end{align}\n",
    "\n",
    "Now we tackle $\\frac{\\partial L}{\\partial W}$:\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial L}{\\partial W} &= \\frac{\\partial L}{\\partial O} \\frac{\\partial O}{\\partial W} && \\vartriangleright \\text{(Chain Rule)}\n",
    "\\end{align}\n",
    "\n",
    "Let's calculate this derivative with respect to a single point $W_{abcd}$ in our weight tensor ($O_w$ and $O_h$ are the output width and height respectively):\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial L}{\\partial W_{abcd}} &= \\sum_{i=0}^{O_w-1} \\sum_{j=0}^{O_h-1} \\frac{\\partial L}{\\partial O_{ij}^d} \\frac{\\partial O_{ij}^d}{\\partial W_{abcd}}\n",
    "\\end{align}\n",
    "\n",
    "**QUESTION**: Why do we sum over the outputs here? **HINT**: Think about how many times a particular weight gets used. \n",
    "\n",
    "Now, looking at equation $(1)$, we can easily calculate $\\frac{\\partial O_{ij}^d}{\\partial W_{abcd}}$ as: \n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial O_{ij}^d}{\\partial W_{abcd}} &= X^{pad}_{i+a, j+b, c}\n",
    "\\end{align}\n",
    "\n",
    "Which gives a final result of:\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial L}{\\partial W_{abcd}} &= \\sum_{i=0}^{O_w-1} \\sum_{j=0}^{O_h-1} \\frac{\\partial L}{\\partial O_{ij}^d} X^{pad}_{i+a, j+b, c}\n",
    "\\end{align}\n",
    "\n",
    "Finally, we need $\\frac{\\partial L}{\\partial X}$, the derivative of the loss with respect to the **input** of the layer. This is sometimes also called a \"delta\". Remember, that before doing the convolution, we applied padding to the input $X$ to get $X^{pad}$. It's easier to calculate the derivative with respect to $X^{pad}$, which appears in our convolution equation, and then remove the padding later on to arrive at the delta. Unfortunately we need to introduce some *more* indexing for the individual components of $X^{pad}$:\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial L}{\\partial X^{pad}_{mnc}} &= \\sum_{i=0}^{O_w-1} \\sum_{j=0}^{O_h-1} \\sum_{d=0}^{D-1} W_{m-i, n-j, c, d} \\frac{\\partial L}{\\partial O_{ij}^d} \n",
    "\\end{align}\n",
    "\n",
    "Where do the indices $m-i$ and $n-j$ come from? Notice in equation $(1)$ that the padded input $X^{pad}_{i+a, j+b, c}$ is multiplied by the weight $W_{abcd}$. Now, when we index $X^{pad}$ with $m$ and $n$, setting $m=i+a$ and $n=j+b$ gives us $a=m-i$ and $b=n-j$ for the indices of $W$! \n",
    "\n",
    "Phew! Spend a few minutes to understand these equations, particularly where the indices come from. Ask a tutor if you get stuck!\n",
    "\n",
    "Note: Did you notice that the delta, $\\frac{\\partial L}{\\partial X^{pad}_{mnc}}$ looks suspiciously like the convolutional forward equation with the inputs $X^{pad}$ replaced by $\\frac{\\partial L}{\\partial O_{ij}^d}$ and different indexing into the weights? In fact the delta is exactly that, the forward convolution applied to the incoming derivative, with the filters flipped along the width and height axes. \n",
    "\n",
    "Now let's implement this in Numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "pF56E8-AFHs7"
   },
   "outputs": [],
   "source": [
    "## IMPLEMENT-ME: ...\n",
    "\n",
    "def convolutional_backward(dout, X, W, b, filter_size, depth, stride, padding):\n",
    "    batch_size, input_width, input_height, input_depth = X.shape\n",
    "    \n",
    "    # Apply padding to the width and height dimensions of the input\n",
    "    X_padded = np.pad(X, ((0,0), (padding, padding), (padding, padding), (0,0)), 'constant')\n",
    "    \n",
    "    # Calculate the width and height of the forward pass output\n",
    "    output_width = (input_width - filter_size + 2*padding) / stride + 1\n",
    "    output_height = (input_height - filter_size + 2*padding) / stride + 1\n",
    "    \n",
    "    # Allocate output arrays\n",
    "    # QUESTION: What is the shape of dx? dw? db?\n",
    "    # ANSWER: ...\n",
    "    dx_padded = np.zeros_like(X_padded) \n",
    "    dw = np.zeros_like(W)\n",
    "    db = np.zeros_like(b)\n",
    "    \n",
    "    # QUESTION: Calculate db, the derivative of the final loss with respect to the bias term\n",
    "    # HINT: Have a look at the axis parameter of the np.sum function.\n",
    "    db = np.sum(dout, axis = (0, 1, 2))\n",
    "    \n",
    "    for i in range(output_width):\n",
    "        for j in range(output_height):\n",
    "            \n",
    "            # Select the current block in the input that the filter will be applied to \n",
    "            block_width_start = i*stride\n",
    "            block_width_end = block_width_start+filter_size\n",
    "\n",
    "            block_height_start = j*stride\n",
    "            block_height_end = block_height_start + filter_size\n",
    "\n",
    "            block = X_padded[:, block_width_start:block_width_end, block_height_start:block_height_end, :]\n",
    "            \n",
    "            for d in range(depth):\n",
    "                # QUESTION: Calculate dw[:,:,:,f], the derivative of the loss with respect to the weight parameters of the f'th filter.\n",
    "                # HINT: You can do this in a loop if you prefer, or use np.sum and \"None\" indexing to get your result to the correct \n",
    "                # shape to assign to dw[:,:,:,f], see (https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#numpy.newaxis)\n",
    "                dw[:,:,:,d] += np.sum(block*(dout[:,i,j,d])[:,None,None,None], axis=0)\n",
    "                \n",
    "            dx_padded[:,block_width_start:block_width_end, block_height_start:block_height_end, :] += np.einsum('ij,klmj->iklm', dout[:,i,j,:], W)\n",
    "            \n",
    "        # Now we remove the padding to arrive at dx\n",
    "        dx = dx_padded[:,padding:-padding, padding:-padding, :]\n",
    "        \n",
    "    return dx, dw, db\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QqzwcoBpjcZI"
   },
   "source": [
    "Finally, we test the backward pass using numerical gradient checking. This compares the gradients generated by our backward function, with a numerical approximation obtained by treating our forward function as a \"black box\". This gradient checking is a very important testing tool when building your own neural network components or back-propagation system! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 85,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 764,
     "status": "ok",
     "timestamp": 1503655297426,
     "user": {
      "displayName": "Avishkar Bhoopchand",
      "photoUrl": "//lh5.googleusercontent.com/-OainnMWSi6A/AAAAAAAAAAI/AAAAAAAAAOQ/eI7Z19q5v7E/s50-c-k-no/photo.jpg",
      "userId": "105781587643595215149"
     },
     "user_tz": -60
    },
    "id": "-L-Gl3B7jegR",
    "outputId": "eee96a03-5205-4c54-9ab6-5100b395fbd0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing conv_backward_naive function\n",
      "dx check: PASSED\n",
      "dw check: PASSED\n",
      "db check: PASSED\n"
     ]
    }
   ],
   "source": [
    "def eval_numerical_gradient_array(f, x, df, h=1e-5):\n",
    "        \"\"\"\n",
    "        Evaluate a numeric gradient for a function that accepts a numpy\n",
    "        array and returns a numpy array.\n",
    "        \"\"\"\n",
    "        \n",
    "        # QUESTION: Can you describe intuitively what this function is doing? \n",
    "        \n",
    "        grad = np.zeros_like(x)\n",
    "        it = np.nditer(x, flags=['multi_index'], op_flags=['readwrite'])\n",
    "        while not it.finished:\n",
    "            ix = it.multi_index\n",
    "\n",
    "            oldval = x[ix]\n",
    "            x[ix] = oldval + h\n",
    "            pos = f(x).copy()\n",
    "            x[ix] = oldval - h\n",
    "            neg = f(x).copy()\n",
    "            x[ix] = oldval\n",
    "\n",
    "            grad[ix] = np.sum((pos - neg) * df) / (2 * h)\n",
    "            it.iternext()\n",
    "        return grad\n",
    "\n",
    "np.random.seed(231)\n",
    "\n",
    "# Normally, backpropagation will have calculated a derivative of the final loss with respect to \n",
    "# the output of our layer. Since we're testing our layer in isolation here, we'll just pretend\n",
    "# and use a random value\n",
    "dout = np.random.randn(2, 2, 2, 3)\n",
    "\n",
    "dx_num = eval_numerical_gradient_array(lambda x: convolutional_forward(X, W, b, filter_size, output_depth, stride, padding), X, dout)\n",
    "dw_num = eval_numerical_gradient_array(lambda w: convolutional_forward(X, W, b, filter_size, output_depth, stride, padding), W, dout)\n",
    "db_num = eval_numerical_gradient_array(lambda b: convolutional_forward(X, W, b, filter_size, output_depth, stride, padding), b, dout)\n",
    "\n",
    "out = convolutional_forward(X, W, b, filter_size, output_depth, stride, padding)\n",
    "dx, dw, db = convolutional_backward(dout, X, W, b, filter_size, output_depth, stride, padding)\n",
    "\n",
    "# Your errors should be around 1e-8'\n",
    "print('Testing conv_backward_naive function')\n",
    "\n",
    "dx_diff = relative_error(dx, dx_num)\n",
    "if dx_diff < 1e-8:\n",
    "    print 'dx check: PASSED'\n",
    "else:\n",
    "    print 'The difference of %s on dx is too high, try again!' % dx_diff\n",
    "    \n",
    "dw_diff = relative_error(dw, dw_num)\n",
    "if dw_diff < 1e-8:\n",
    "    print 'dw check: PASSED'\n",
    "else:\n",
    "    print 'The difference of %s on dw is too high, try again!' % dw_diff\n",
    "    \n",
    "db_diff = relative_error(db, db_num)\n",
    "if db_diff < 1e-8:\n",
    "    print 'db check: PASSED'\n",
    "else:\n",
    "    print 'The difference of %s on db is too high, try again!' % db_diff\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bczw3pcaJxh_"
   },
   "source": [
    "## (Max) Pooling Layers\n",
    "The purpose of a pooling layer is to is to reduce the spatial size of the representation and therefore control the number of parameters in the network. A pooling layer has no trainable parameters itself. It applies some 2D aggegation operation (usually a MAX, but others like average may also be used) to regions of the input volume. This is done independently for each depth dimension of the input. For example, a 2x2 max pooling operation with a stride of 2, downsamples every depth slice of the input by 2 along both the width and height. \n",
    "\n",
    "The output volume of a pooling layer alwyas has the same depth as the input volume. The width and height are calcualted as follows:\n",
    "$(W−F)/S+1$ where W is the width/height of the "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MdEN_ur1Kv1x"
   },
   "source": [
    "### Implementing the forward pass\n",
    "We again implement this in Numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "QgowKDV6KzOJ"
   },
   "outputs": [],
   "source": [
    "def max_pool_forward(X, pool_size, stride):\n",
    "    batch_size, input_width, input_height, input_depth = X.shape\n",
    "    \n",
    "    # Calculate the output dimensions\n",
    "    output_width = (input_width - pool_size)/stride + 1\n",
    "    output_height = (input_height - pool_size)/stride + 1\n",
    "    \n",
    "    # Allocate the output array\n",
    "    out = np.zeros((batch_size, output_width, output_height, input_depth))\n",
    "    \n",
    "    # Select the current block in the input that the filter will be applied to \n",
    "    for w in range(output_width):\n",
    "        for h in range(output_height):\n",
    "            block_width_start = w*stride\n",
    "            block_width_end = block_width_start+pool_size\n",
    "\n",
    "            block_height_start = h*stride\n",
    "            block_height_end = block_height_start + pool_size\n",
    "\n",
    "            block = X[:, block_width_start:block_width_end, block_height_start:block_height_end, :]\n",
    "            ## IMPLEMENT-ME CANDIDATE\n",
    "            out[:,w,h,:] = np.max(block, axis=(1,2))\n",
    "            \n",
    "    return out\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TmIkFclN9Hxz"
   },
   "source": [
    "Now we can test the `max_pool_forward` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 51,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 380,
     "status": "ok",
     "timestamp": 1503655319186,
     "user": {
      "displayName": "Avishkar Bhoopchand",
      "photoUrl": "//lh5.googleusercontent.com/-OainnMWSi6A/AAAAAAAAAAI/AAAAAAAAAOQ/eI7Z19q5v7E/s50-c-k-no/photo.jpg",
      "userId": "105781587643595215149"
     },
     "user_tz": -60
    },
    "id": "9DMjg7hf9NoM",
    "outputId": "0bcd66eb-207f-45c2-80d1-36c4e0da9524"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing max_pool_forward function:\n",
      "PASSED\n"
     ]
    }
   ],
   "source": [
    "### Hyperparameters\n",
    "batch_size = 2\n",
    "input_width = 4\n",
    "input_height = 4\n",
    "input_depth = 3\n",
    "pool_size = 2\n",
    "stride = 2\n",
    "###\n",
    "\n",
    "input_shape = (batch_size, input_width, input_height, input_depth)\n",
    "X = np.linspace(-0.3, 0.4, num=np.prod(input_shape)).reshape(input_shape)\n",
    "\n",
    "out = max_pool_forward(X, pool_size, stride)\n",
    "\n",
    "correct_out = np.array([\n",
    "        [[[-0.18947368, -0.18210526, -0.17473684],\n",
    "            [-0.14526316, -0.13789474, -0.13052632]],\n",
    "         [[-0.01263158, -0.00526316, 0.00210526],\n",
    "            [0.03157895, 0.03894737, 0.04631579]]],\n",
    "        [[[0.16421053, 0.17157895, 0.17894737],\n",
    "            [0.20842105, 0.21578947, 0.22315789]],\n",
    "         [[0.34105263, 0.34842105, 0.35578947],\n",
    "            [0.38526316, 0.39263158, 0.4]]]])\n",
    "\n",
    "# Compare the output. The difference should be less than 1e-6.\n",
    "print('Testing max_pool_forward function:')\n",
    "diff = relative_error(out, correct_out)\n",
    "if diff < 1e-6:\n",
    "    print 'PASSED'\n",
    "else:\n",
    "    print 'The difference of %s is too high, try again!' % diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f09Zws5lK1W7"
   },
   "source": [
    "### The derivative of a max-pool layer\n",
    "\n",
    "The max-pooling layer has no learnable parameters of its own, so the only derivative of concern is that of the output of the layer with respect to the input for the purpose of backpropagating the error through the layer. This is easy to calculate as it only requires that we recalculate (or remember) which value in each block was the maximum. Since each output depends only on one value in some FxF block of the input, the gradients of the max-pool layer will be sparse. \n",
    "\n",
    "Let's implement the backward pass in Numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "n_thklIPLDoV"
   },
   "outputs": [],
   "source": [
    "def max_pool_backward(dout, X, max_pool_output, pool_size, stride):\n",
    "    batch_size, input_width, input_height, input_depth = X.shape\n",
    "    \n",
    "    # Calculate the output dimensions\n",
    "    output_width = (input_width - pool_size)/stride + 1\n",
    "    output_height = (input_height - pool_size)/stride + 1\n",
    "    \n",
    "    # QUESTION: What is the size of dx, the derivative with respect to x? \n",
    "    # Allocate an array to hold the derivative\n",
    "    dx = np.zeros_like(X)\n",
    "    \n",
    "    for w in range(output_width):\n",
    "        for h in range(output_height):\n",
    "            # Which block in the input did the value at the forward pass output come from?\n",
    "            block_width_start = w*stride\n",
    "            block_width_end = block_width_start+pool_size\n",
    "\n",
    "            block_height_start = h*stride\n",
    "            block_height_end = block_height_start + pool_size\n",
    "\n",
    "            block = X[:, block_width_start:block_width_end, block_height_start:block_height_end, :]\n",
    "            \n",
    "            # What was the maximum value\n",
    "            max_val = max_pool_output[:, w, h, :]\n",
    "            \n",
    "            # Which values in the input block resulted in the output?\n",
    "            responsible_values = block == max_val[:, None, None, :]\n",
    "            \n",
    "            # Add the contribution of the current block to the gradient\n",
    "            dx[:,block_width_start:block_width_end,block_height_start:block_height_end, :] += responsible_values * (dout[:,w,h,:])[:,None,None,:]\n",
    "            \n",
    "    return dx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "k4rjVDGy8H3S"
   },
   "source": [
    "And we again use numerical gradient checking to ensure that the backward function is correct: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 51,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 384,
     "status": "ok",
     "timestamp": 1503655384343,
     "user": {
      "displayName": "Avishkar Bhoopchand",
      "photoUrl": "//lh5.googleusercontent.com/-OainnMWSi6A/AAAAAAAAAAI/AAAAAAAAAOQ/eI7Z19q5v7E/s50-c-k-no/photo.jpg",
      "userId": "105781587643595215149"
     },
     "user_tz": -60
    },
    "id": "EOATBv5Q8M9R",
    "outputId": "271fc91a-4c78-4def-a2df-8bc1f29d10f7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing max_pool_backward function:\n",
      "PASSED\n"
     ]
    }
   ],
   "source": [
    "# Define a hypothetical derivative of the loss function with respect to the output of the max-pooling layer.\n",
    "dout = np.random.randn(batch_size, pool_size, pool_size, input_depth)\n",
    "\n",
    "dx_num = eval_numerical_gradient_array(lambda x: max_pool_forward(x, pool_size, stride), X, dout)\n",
    "out = max_pool_forward(X, pool_size, stride)\n",
    "dx = max_pool_backward(dout, X, out, pool_size, stride)\n",
    "\n",
    "# Your error should be less than 1e-12\n",
    "print('Testing max_pool_backward function:')\n",
    "diff = relative_error(dx, dx_num)\n",
    "if diff < 1e-12:\n",
    "    print 'PASSED'\n",
    "else:\n",
    "    print 'The diff of %s is too large, try again!' % diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "59ZgyinR66B-"
   },
   "source": [
    "## Optimisation - an exercise for later\n",
    "Our implementations of convolutional and max-pool layers were based on loops, which are easy to understand, but are slow and inefficient compared to a vectorised implementation exploiting matrix multiplications. The vectorised form is how these layers are actually implemented in practice and are also required to make efficient use of GPUs in frameworks that support it, like TensorFlow. As an exercise, once you fully understand how the layers work, try to rewrite the code such that the convolution and max-pool operations are each implemented in a **single matrix multiplication**. \n",
    "\n",
    "(HINT: Matlab has a function called \"im2col\" that rearranges blocks of an image into columns, you will need to achieve something similar using Numpy!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5CyCc4zdntUe"
   },
   "source": [
    "# Building a 2-layer ConvNet in TensorFlow\n",
    "\n",
    "Now that we understand the convolutional and max pool layers, let's switch back to TensorFlow and build a 2-layer ConvNet classifier that we can apply to MNIST. We reuse essentially the same classifier framework we used in Practical 2 as well as the training and plotting functions, but we have added support for 2 new forms of regularisation, **dropout** and **batch normalisation**. These are explained in more detail later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "gTABb64k3eiW"
   },
   "outputs": [],
   "source": [
    "class BaseSoftmaxClassifier(object):\n",
    "    def __init__(self, input_size, output_size):        \n",
    "        # Define the input placeholders. The \"None\" dimension means that the \n",
    "        # placeholder can take any number of images as the batch size. \n",
    "        self.x = tf.placeholder(tf.float32, [None, input_size])\n",
    "        self.y = tf.placeholder(tf.float32, [None, output_size])    \n",
    "        \n",
    "        # We add an additional input placeholder for Dropout regularisation\n",
    "        self.keep_prob = tf.placeholder(tf.float32, name=\"keep_prob\")\n",
    "        \n",
    "        # And one for bath norm regularisation\n",
    "        self.is_training = tf.placeholder(tf.bool, name=\"is_training\")\n",
    "        \n",
    "        self.input_size = input_size\n",
    "        self.output_size = output_size\n",
    "        \n",
    "        # You should override these in your build_model() function.\n",
    "        self.logits = None\n",
    "        self.predictions = None\n",
    "        self.loss = None\n",
    "        \n",
    "        self.build_model()\n",
    "        \n",
    "    def get_logits(self):\n",
    "        return self.logits\n",
    "    \n",
    "    def build_model(self):\n",
    "        # OVERRIDE THIS FOR YOUR PARTICULAR MODEL.\n",
    "        raise NotImplementedError(\"Subclasses should implement this function!\")\n",
    "        \n",
    "    def compute_loss(self):\n",
    "        \"\"\"All models share the same softmax cross-entropy loss.\"\"\"\n",
    "        assert self.logits is not None    # Ensure that logits has been created! \n",
    "        data_loss = tf.reduce_mean(\n",
    "                tf.nn.softmax_cross_entropy_with_logits(logits=self.logits, labels=self.y))\n",
    "            \n",
    "        return data_loss\n",
    "    \n",
    "    def accuracy(self):\n",
    "        # Calculate accuracy.\n",
    "        assert self.predictions is not None    # Ensure that pred has been created!\n",
    "        correct_prediction = tf.equal(tf.argmax(self.predictions, 1), tf.argmax(self.y, 1))\n",
    "        accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
    "        return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wk8Vo4mzGvz6"
   },
   "source": [
    "Lets also bring in the training and plotting routines we developed in Prac 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "EAgod2aDGvFn"
   },
   "outputs": [],
   "source": [
    "def train_tf_model(tf_model,                                     \n",
    "                   session,    # The active session.\n",
    "                   num_epochs,    # Max epochs/iterations to train for.\n",
    "                   batch_size=100,    # Number of examples per batch.\n",
    "                   keep_prob=1.0,    # (1. - dropout) probability, none by default.\n",
    "                   optimizer_fn=None,    # TODO(sgouws): more correct to call this optimizer_obj\n",
    "                   report_every=1, # Report training results every nr of epochs.\n",
    "                   eval_every=1,    # Evaluate on validation data every nr of epochs.\n",
    "                   stop_early=True,    # Use early stopping or not.\n",
    "                   verbose=True): \n",
    "\n",
    "    # Get the (symbolic) model input, output, loss and accuracy.\n",
    "    x, y = tf_model.x, tf_model.y\n",
    "    loss = tf_model.loss\n",
    "    accuracy = tf_model.accuracy()\n",
    "\n",
    "    # Compute the gradient of the loss with respect to the model parameters \n",
    "    # and create an op that will perform one parameter update using the specific\n",
    "    # optimizer's update rule in the direction of the gradients.\n",
    "    if optimizer_fn is None:\n",
    "        optimizer_fn = tf.train.AdamOptimizer(1e-4)\n",
    "        \n",
    "    # For batch normalisation: Ensure that the mean and variance tracking \n",
    "    # variables get updated at each training step\n",
    "    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n",
    "    with tf.control_dependencies(update_ops):\n",
    "        optimizer_step = optimizer_fn.minimize(loss)\n",
    "\n",
    "    # Get the op which, when executed, will initialize the variables.\n",
    "    init = tf.global_variables_initializer()\n",
    "    # Actually initialize the variables (run the op).\n",
    "    session.run(init)\n",
    "\n",
    "    # Save the training loss and accuracies on training and validation data.\n",
    "    train_costs = []\n",
    "    train_accs = []\n",
    "    val_costs = []\n",
    "    val_accs = []\n",
    "\n",
    "    mnist_train_data = mnist.train\n",
    "    \n",
    "    prev_c_eval = 1000000\n",
    "    \n",
    "    # Main training cycle.\n",
    "    for epoch in range(num_epochs):\n",
    "\n",
    "        avg_cost = 0.\n",
    "        avg_acc = 0.\n",
    "        total_batch = int(mnist.train.num_examples / batch_size)\n",
    "\n",
    "        # Loop over all batches.\n",
    "        for i in range(total_batch):\n",
    "            batch_x, batch_y = mnist_train_data.next_batch(batch_size)\n",
    "                        \n",
    "            # Run optimization op (backprop) and cost op (to get loss value),\n",
    "            # and compute the accuracy of the model.\n",
    "            feed_dict = {x: batch_x, y: batch_y, tf_model.keep_prob: keep_prob,\n",
    "                                    tf_model.is_training: True}\n",
    "            _, c, a = session.run(\n",
    "                    [optimizer_step, loss, accuracy], feed_dict=feed_dict)\n",
    "                        \n",
    "            # Compute average loss/accuracy\n",
    "            avg_cost += c / total_batch\n",
    "            avg_acc += a / total_batch            \n",
    "        \n",
    "        train_costs.append((epoch, avg_cost))\n",
    "        train_accs.append((epoch, avg_acc))\n",
    "\n",
    "        # Display logs per epoch step\n",
    "        if epoch % report_every == 0 and verbose:\n",
    "            print \"Epoch:\", '%04d' % (epoch+1), \"Training cost=\", \\\n",
    "                        \"{:.9f}\".format(avg_cost)\n",
    "                \n",
    "        if epoch % eval_every == 0:\n",
    "            val_x, val_y = mnist.validation.images, mnist.validation.labels            \n",
    "            \n",
    "            feed_dict = {x : val_x, y : val_y, tf_model.keep_prob: 1.0,\n",
    "                                    tf_model.is_training: False}\n",
    "            c_eval, a_eval = session.run([loss, accuracy], feed_dict=feed_dict)\n",
    "            \n",
    "            if verbose:\n",
    "                print \"Epoch:\", '%04d' % (epoch+1), \"Validation acc=\", \\\n",
    "                            \"{:.9f}\".format(a_eval)\n",
    "                \n",
    "            if c_eval >= prev_c_eval and stop_early:\n",
    "                print \"Validation loss stopped improving, stopping training early after %d epochs!\" % (epoch + 1)\n",
    "                break\n",
    "                \n",
    "            prev_c_eval = c_eval\n",
    "                \n",
    "            val_costs.append((epoch, c_eval))\n",
    "            val_accs.append((epoch, a_eval))\n",
    "    \n",
    "    print \"Optimization Finished!\"\n",
    "    return train_costs, train_accs, val_costs, val_accs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "wAN12x1cG4eJ"
   },
   "outputs": [],
   "source": [
    "# Helper functions to plot training progress.\n",
    "\n",
    "def my_plot(list_of_tuples):\n",
    "    \"\"\"Take a list of (epoch, value) and split these into lists of \n",
    "    epoch-only and value-only. Pass these to plot to make sure we\n",
    "    line up the values at the correct time-steps.\n",
    "    \"\"\"\n",
    "    plt.plot(*zip(*list_of_tuples))\n",
    "\n",
    "def plot_multi(values_lst, labels_lst, y_label, x_label='epoch'):\n",
    "    # Plot multiple curves.\n",
    "    assert len(values_lst) == len(labels_lst)\n",
    "    plt.subplot(2, 1, 2)\n",
    "    \n",
    "    for v in values_lst:\n",
    "        my_plot(v)\n",
    "    plt.legend(labels_lst, loc='upper left')\n",
    "    \n",
    "    plt.xlabel(x_label)\n",
    "    plt.ylabel(y_label)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4WOTTYeveZgS"
   },
   "source": [
    "Now define some helper functions to build a convolutional layer and a linear layer (this is mostly the same as the previous practical, but we use slightly different weight and bias initializations which seem to work better with ConvNets on MNIST). In terms of regularisation, we use **dropout** rather than the L2 regularisation from the previous practical. \n",
    "\n",
    "## Dropout\n",
    "Dropout is a neural-network regularisation technique that is applied during model training. At each training step, a proportion `(1-keep_prob)` of neurons the network are \"dropped out\" (their inputs and outputs are set to 0, effectively ignoring their contribution) while the remaining `keep_prob` fraction are \"let through\" (**Nit**: they're actually rescaled by `1/keep_prob` to ensure that the variance of the pre-activations at the next layer remains unchanged). This can be interpreted as there being actually $2^n$ different network architectures (where n is the number of neurons) while only one is being trained at each training step. At test time, we use the full network, where each neuron's contribution is weighted by `keep_prob`. This is effectively the average of all the network possibilities and therefore dropout can also be thought of as an ensemble technique. \n",
    "\n",
    "In our ConvNet architecture, the majority of neurons occur in the fully connected layer between the convolutional layers and the output. It is therefore this fully connected layer that we are most concerned about overfitting and this is where we apply dropout. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "OY2RwcMh4gEr"
   },
   "outputs": [],
   "source": [
    "def _convolutional_layer(inputs, filter_size, output_depth):\n",
    "    \"\"\"Build a convolutional layer with `output_depth` square \n",
    "    filters, each of size `filter_size` x `filter_size`.\"\"\"\n",
    "\n",
    "    input_features = inputs.shape[3] \n",
    "    \n",
    "    weights = tf.get_variable(\n",
    "            \"conv_weights\", \n",
    "            [filter_size, filter_size, input_features, output_depth], \n",
    "            dtype=tf.float32,\n",
    "            initializer=tf.truncated_normal_initializer(stddev=0.1))\n",
    "     \n",
    "    ## IMPLEMENT-ME CANDIDATE\n",
    "    conv = tf.nn.conv2d(inputs, weights, strides=[1, 1, 1, 1], padding='SAME')\n",
    "    return conv\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "HB6jzZd6-XLp"
   },
   "outputs": [],
   "source": [
    "def _dense_linear_layer(inputs, layer_name, input_size, output_size, weights_initializer):\n",
    "    \"\"\"\n",
    "    Builds a layer that takes a batch of inputs of size `input_size` and returns \n",
    "    a batch of outputs of size `output_size`.\n",
    "         \n",
    "    Args:\n",
    "        inputs: A `Tensor` of shape [batch_size, input_size].\n",
    "        layer_name: A string representing the name of the layer.\n",
    "        input_size: The size of the inputs\n",
    "        output_size: The size of the outputs\n",
    "        \n",
    "    Returns:\n",
    "        out, weights: tuple of layer outputs and weights.\n",
    "        \n",
    "    \"\"\"\n",
    "    # Name scopes allow us to logically group together related variables.\n",
    "    # Setting reuse=False avoids accidental reuse of variables between different runs.\n",
    "    with tf.variable_scope(layer_name, reuse=False):\n",
    "        # Create the weights for the layer\n",
    "        layer_weights = tf.get_variable(\"weights\",\n",
    "                                        shape=[input_size, output_size], \n",
    "                                        dtype=tf.float32, \n",
    "                                        initializer=weights_initializer)\n",
    "        # Create the biases for the layer\n",
    "        layer_bias = tf.get_variable(\"biases\", \n",
    "                                     shape=[output_size], \n",
    "                                     dtype=tf.float32, \n",
    "                                     initializer=tf.constant_initializer(0.1))\n",
    "        \n",
    "        outputs = tf.matmul(inputs, layer_weights) + layer_bias\n",
    "    \n",
    "    return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lTChH7A0h0sD"
   },
   "source": [
    "Now build the ConvNetClassifier, we make the number of convolutional layers and the filter sizes parameters so that you can easily experiment with different variations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "STEy0-DO4Cfm"
   },
   "outputs": [],
   "source": [
    "class ConvNetClassifier(BaseSoftmaxClassifier):\n",
    "    def __init__(self, \n",
    "                 input_size,     # The size of the input\n",
    "                 output_size,    # The size of the output\n",
    "                 filter_sizes=[], # The number of filters to use per convolutional layer\n",
    "                 output_depths=[], # The number of features to output per convolutional layer\n",
    "                 hidden_linear_size=512,    # The size of the hidden linear layer\n",
    "                 use_batch_norm=False,     # Flag indicating whether or not to use batch normalisation\n",
    "                 linear_weights_initializer=tf.truncated_normal_initializer(stddev=0.1)):        \n",
    "        \n",
    "        assert len(filter_sizes) == len(output_depths)\n",
    "        \n",
    "        self.filter_sizes = filter_sizes\n",
    "        self.output_depths = output_depths\n",
    "        self.linear_weights_initializer = linear_weights_initializer\n",
    "        self.use_batch_norm = use_batch_norm\n",
    "        self.hidden_linear_size = hidden_linear_size\n",
    "        \n",
    "        super(ConvNetClassifier, self).__init__(input_size, output_size)\n",
    "        \n",
    "        \n",
    "    def build_model(self):\n",
    "        # Architecture: INPUT - {CONV - RELU - POOL}*N - FC\n",
    "        \n",
    "        # Reshape the input to [batch_size, width, height, input_depth]\n",
    "        conv_input = tf.reshape(self.x, [-1, 28, 28, 1])    \n",
    "        \n",
    "        prev_inputs = conv_input\n",
    "        \n",
    "        # Create the CONV-RELU-POOL layers:\n",
    "        for layer_number, (layer_filter_size, layer_features) in enumerate(\n",
    "                zip(self.filter_sizes, self.output_depths)):\n",
    "            with tf.variable_scope(\"layer_{}\".format(layer_number), reuse=False):\n",
    "                # Create the convolution:\n",
    "                conv = _convolutional_layer(prev_inputs, layer_filter_size, layer_features)\n",
    "                \n",
    "                # Apply batch normalisation, if required\n",
    "                if self.use_batch_norm:\n",
    "                    conv = tf.contrib.layers.batch_norm(conv, center=True, scale=True, \n",
    "                                                        is_training=self.is_training)\n",
    "                \n",
    "                # Apply the RELU activation with a bias\n",
    "                bias = tf.get_variable(\"bias\", [layer_features], dtype=tf.float32, initializer=tf.constant_initializer(0.1))\n",
    "                relu = tf.nn.relu(conv + bias)\n",
    "                \n",
    "                # Apply max-pooling using patch-sizes of 2x2\n",
    "                pool = tf.nn.max_pool(relu, ksize=[1, 2, 2, 1],\n",
    "                                                            strides=[1, 2, 2, 1], \n",
    "                                                            padding='SAME')\n",
    "                \n",
    "                # QUESTION: What is the shape of the pool tensor?\n",
    "                # ANSWER: ...\n",
    "                \n",
    "                prev_inputs = pool\n",
    "        \n",
    "        \n",
    "        # QUESTION: What is the shape of prev_inputs now? \n",
    "        \n",
    "        # We need to flatten the last (non-batch) dimensions of the convolutional\n",
    "        # output in order to pass it to a fully-connected layer:\n",
    "        flattened = tf.contrib.layers.flatten(prev_inputs)\n",
    "        \n",
    "        # Create the fully-connected (linear) layer that maps the flattened inputs\n",
    "        # to `hidden_linear_size` hidden outputs\n",
    "        flat_size = flattened.shape[1]\n",
    "        fully_connected = _dense_linear_layer(\n",
    "                flattened, \"fully_connected\", flat_size, self.hidden_linear_size, self.linear_weights_initializer)\n",
    "        \n",
    "        # Apply batch normalisation, if required\n",
    "        if self.use_batch_norm:\n",
    "            fully_connected = tf.contrib.layers.batch_norm(\n",
    "                    fully_connected, center=True, scale=True, is_training=self.is_training)\n",
    "        \n",
    "        fc_relu = tf.nn.relu(fully_connected)\n",
    "        \n",
    "        fc_drop = tf.nn.dropout(fc_relu, self.keep_prob)\n",
    "        \n",
    "        # Now we map the `hidden_linear_size` outputs to the `output_size` logits, one for each possible digit class\n",
    "        logits = _dense_linear_layer(\n",
    "                fc_drop, \"logits\", self.hidden_linear_size, self.output_size, self.linear_weights_initializer)\n",
    "        \n",
    "        self.logits = logits\n",
    "        self.predictions = tf.nn.softmax(self.logits)\n",
    "        self.loss = self.compute_loss()\n",
    "        \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PZPkvle1RDfk"
   },
   "source": [
    "Finally a function that wraps up the training and evaluation of the model (the same as prac 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "rYuK9io1RJ2w"
   },
   "outputs": [],
   "source": [
    "def build_train_eval_and_plot(build_params, train_params, verbose=True):\n",
    "    tf.reset_default_graph()\n",
    "    m = ConvNetClassifier(**build_params)\n",
    "\n",
    "    with tf.Session() as sess:\n",
    "        # Train model on the MNIST dataset.\n",
    "     \n",
    "        train_losses, train_accs, val_losses, val_accs = train_tf_model(\n",
    "                m, \n",
    "                sess,\n",
    "                verbose=verbose,\n",
    "                **train_params) \n",
    "        \n",
    "        # Now evaluate it on the test set:\n",
    "    \n",
    "        accuracy_op = m.accuracy()    # Get the symbolic accuracy operation\n",
    "        # Calculate the accuracy using the test images and labels.\n",
    "        accuracy = accuracy_op.eval({m.x: mnist.test.images, \n",
    "                                     m.y: mnist.test.labels,\n",
    "                                     m.keep_prob: 1.0,\n",
    "                                     m.is_training: False})    \n",
    "        \n",
    "        if verbose: \n",
    "            print \"Accuracy on test set:\", accuracy\n",
    "            # Plot losses and accuracies.\n",
    "            plot_multi([train_losses, val_losses], ['train', 'val'], 'loss', 'epoch')\n",
    "            plot_multi([train_accs, val_accs], ['train', 'val'], 'accuracy', 'epoch')\n",
    "            \n",
    "        \n",
    "        ret = {'train_losses': train_losses, 'train_accs' : train_accs,\n",
    "               'val_losses' : val_losses, 'val_accs' : val_accs,\n",
    "               'test_acc' : accuracy}\n",
    "        \n",
    "        # Evaluate the final convolutional weights\n",
    "        conv_variables = [v for v in tf.trainable_variables() if \"conv_weights\" in v.name]\n",
    "        conv_weights = sess.run(conv_variables)\n",
    "        \n",
    "        return m, ret, conv_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "03Pbb466Bdrh"
   },
   "source": [
    "Now train and evaluate the ConvNet model on MNIST. \n",
    "\n",
    "**NOTE**: Hopefully you answered the question in the first section about the tradeoffs a ConvNet makes with \"extra computation\"! Unfortuntaly the VMs we're using are pretty low-powered, so we will train a very small ConvNet just to check that it works. Once you've got this small ConvNet working, chat to a tutor to get access to a machine with a GPU and run with the following configuration to get to the promised 99%+ accuracy on MNIST!\n",
    "\n",
    "```\n",
    "model_params = {\n",
    "        'input_size': 784,\n",
    "        'output_size': 10,\n",
    "        'filter_sizes': [5, 5],\n",
    "        'output_depths': [32, 64],\n",
    "        'hidden_linear_size': 1024,\n",
    "        'use_batch_norm': False\n",
    "}\n",
    "\n",
    "training_params = {\n",
    "        'keep_prob': 0.5,\n",
    "        'num_epochs': 20,\n",
    "        'batch_size': 50,\n",
    "        'stop_early': False,\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 1273,
     "output_extras": [
      {
       "item_id": 31
      },
      {
       "item_id": 32
      },
      {
       "item_id": 33
      },
      {
       "item_id": 34
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 290757,
     "status": "ok",
     "timestamp": 1504368810426,
     "user": {
      "displayName": "Avishkar Bhoopchand",
      "photoUrl": "//lh5.googleusercontent.com/-OainnMWSi6A/AAAAAAAAAAI/AAAAAAAAAOQ/eI7Z19q5v7E/s50-c-k-no/photo.jpg",
      "userId": "105781587643595215149"
     },
     "user_tz": -60
    },
    "id": "_KQq51S9Bc73",
    "outputId": "2141f1f9-c30c-47a2-9525-fb1e42834a85"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 Training cost= 1.261568776\n",
      "Epoch: 0001 Validation acc= 0.879800022\n",
      "Epoch: 0002 Training cost= 0.516203916\n",
      "Epoch: 0002 Validation acc= 0.913600028\n",
      "Epoch: 0003 Training cost= 0.391842912\n",
      "Epoch: 0003 Validation acc= 0.930599988\n",
      "Epoch: 0004 Training cost= 0.326444157\n",
      "Epoch: 0004 Validation acc= 0.943199992\n",
      "Epoch: 0005 Training cost= 0.287748122\n",
      "Epoch: 0005 Validation acc= 0.948599994\n",
      "Optimization Finished!\n",
      "Accuracy on test set: 0.9475\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAD2CAYAAAB4IJvdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcXFWd9/HPr7bek04v2RM6IQGyQBLSCQFHxUfZRBYX\nCPtOxGVGdB5HZkbH0XGU0WfGERUhQAQEWQbQQQUZxQHHMYF0IEDCloVAOlsv2XpJL9V9nj9udae6\nUt1dSbr6Vld9369Xvbrq3tOV3/EK+XLOPeeacw4RERER8U/A7wJEREREcp0CmYiIiIjPFMhERERE\nfKZAJiIiIuIzBTIRERERnymQiYiIiPhMgUxERETEZwpkIiIiIj5TIBMRERHxWcjvAg5XRUWFq6qq\n8rsMERERkUGtWbOmwTlXOVi7ERfIqqqqqKmp8bsMERERkUGZ2buptNOUpYiIiIjPFMhEREREfKZA\nJiIiIuKzEXcPWTKdnZ3U1tbS1tbmdylpl5+fz+TJkwmHw36XIiIiIkMkKwJZbW0tJSUlVFVVYWZ+\nl5M2zjkaGxupra1l2rRpfpcjIiIiQyQrpizb2tooLy/P6jAGYGaUl5fnxEigiIhILsmKQAYMWRiL\ndnWzpaGF9mjXkHzfUMv20CkiIpKLsiaQDZXObkdLR5TN9S20d2ZmKBMREZHsokCWoCAcZHpFMc7B\n5oYW2lIIZXv37uX2228/7D/rox/9KHv37j2SMkVERCSLKJAlURAJMr2yyAtl9YOHsv4CWTQaHfD3\nnnrqKUpLS4+qVhERERn5FMj6kR/2QpmZF8oOdPQfym655RY2bdrE/PnzWbRoEe9///s5//zzmT17\nNgAXXnghCxcuZM6cOSxfvrz396qqqmhoaGDLli3MmjWLG2+8kTlz5nDmmWdy4MCBtPdRREREMkNW\nbHsR7xu/Ws/r2/cP2fc555hQWsCyD0xnekURBZFD/ye79dZbWbduHWvXruW5557j3HPPZd26db1b\nU6xYsYKysjIOHDjAokWL+OQnP0l5eXmf79iwYQMPPfQQd911FxdffDGPP/44V1xxxZD1Q0RERDKX\nRsgGYWaUFoQJmrG5oYXWjoGnIQEWL17cZ5+w2267jXnz5rFkyRK2bt3Khg0bDvmdadOmMX/+fAAW\nLlzIli1bhqwPIiIiktmyboTs6+fNScv3dkS72NzQwjv1LVRVFFGU1///dEVFRb3vn3vuOX7/+9+z\ncuVKCgsLOf3005PuI5aXl9f7PhgMaspSREQkh6RthMzMVphZnZmt6+f85Wb2qpm9ZmZ/NrN56apl\nKERC3urLUDDAOw0ttLQfHCkrKSmhqakp6e/t27ePMWPGUFhYyJtvvsmqVauGq2QREREZIdI5ZXkv\ncPYA598BPuicOxH4J2D5AG0zQiQUYHplEeFYKGtu6wSgvLyc973vfcydO5cvf/nLfX7n7LPPJhqN\nMmvWLG655RaWLFniR+kiIiKSwcw5l74vN6sCfu2cmztIuzHAOufcpMG+s7q62tXU1PQ59sYbbzBr\n1qyjqPTwdHZ1805DCx3Rbo4pL6Qkf3gf9D3c/RUREZEjY2ZrnHPVg7XLlJv6rwee7u+kmS0zsxoz\nq6mvrx/GspILBwNMrygiEgqwpbGV/bGRMhEREZEj4XsgM7MP4QWyr/TXxjm33DlX7ZyrrqysHL7i\nBhCKhbL8UIB3G1vZf0ChTERERI6Mr4HMzE4C7gYucM41+lnLkQgFA0yrKKIg7IWyfQc6/C5JRERE\nRiDfApmZTQWeAK50zr3tVx1HqzeURYK813iAva0KZSIiInJ40rYPmZk9BJwOVJhZLfB1IAzgnLsD\n+AegHLjdzACiqdz0lomCAS+UbWloYevuVhwwpjDid1kiIiIyQqQtkDnnLh3k/A3ADen684dbMGBU\nVRTxbmMslDkoK1IoExERkcH5flN/NgkGjKryIorzQtTuaaWxpT1pu+Li4mGuTERERDKZAtkQC8RC\nWUl+mG17DtDYnDyUiYiIiPTIumdZ+uGWW25hypQpfO5znwPgm9/8BsFgkN/+7ll279mDdXfxnW//\nMxdccIHPlYqIiEgmyr5A9vQtsPO1of3O8SfCObf2e3rp0qXcfPPNvYHs0Ucf5ZlnnuEv/+qv2BcN\nsWXbTq658EzOP/98YgsYRERERHplXyDzwYIFC6irq2P79u3U19czZswYxo8fzxe/+EX++Mc/EnWw\nY/t21m98l7kzq/wuV0RERDJM9gWyAUay0umiiy7iscceY+fOnSxdupQHH3yQ+vp61qxZQygUYuox\nVWxt2E/luDZf6hMREZHMpZv6h8jSpUt5+OGHeeyxx7jooovYt28fY8eOJRwO89xzz1G79T1GF4TZ\ntb8NB6Tzoe4iIiIysmTfCJlP5syZQ1NTE5MmTWLChAlcfvnlnHfeeZx44olUV1dzwgknMGF0PuGi\nCM7Bzv1tjB+Vr3vKRERERIFsKL322sHFBBUVFaxcufKQNs45NtTWU9/UjnMwYbRCmYiISK5TIBtm\nZsbE0nzMoKHZC2XeZ4UyERGRXKVA5gMzi42M4Y2U4ZhUWqBQJiIikqOyJpA550ZUoDEz7x4yoC42\nfTl5zOChTIsBREREsk9WrLLMz8+nsbFxxIUVM2PcqHzGjcpnT2sHW/ccGLAPzjkaGxvJz88fxipF\nREQk3bJihGzy5MnU1tZSX1/vdylHrK2tk7cORNkaCTKmMNzvSFl+fj6TJ08e5upEREQknbIikIXD\nYaZNm+Z3GUftzuc38Z0n3+ScueP5wSULiISyYgBTREREBqG/8TPIpz94LF/72GyeXreTzz74Eu3R\nLr9LEhERkWGgQJZhrv+LaXzzgjn8/o1d3PSzNbR1KpSJiIhkOwWyDHTVqVV85xMn8tzb9dx4fw0H\nOhTKREREspkCWYa6dPFUvvvJk/jTxgauu3c1rR1Rv0sSERGRNFEgy2AXVU/h3y6exwvvNHLNitU0\ntyuUiYiIZCMFsgz38QWT+cElC1jz3h6uuucF9rd1+l2SiIiIDDEFshHgvHkT+dGlC3i1dh9X3vMi\n+1oVykRERLKJAtkIcc6JE/jJFQt5ffs+Lr9nFXtaOvwuSURERIaIAtkIcsbscSy/spq3dzVz2d0v\n0Njc7ndJIiIiMgTSFsjMbIWZ1ZnZun7Om5ndZmYbzexVMzs5XbVkkw+dMJa7r6pmc30zl961ivom\nhTIREZGRLp0jZPcCZw9w/hxgZuy1DPhJGmvJKh84rpKfXrOIrbsPcMnyldTtb/O7JBERETkKaQtk\nzrk/ArsHaHIBcL/zrAJKzWxCuurJNqfNqODeaxexY18bS5evYse+A36XJCIiIkfIz3vIJgFb4z7X\nxo5Jik6ZXs7Prl9MfVM7S+9cRe2eVr9LEhERkSMwIm7qN7NlZlZjZjX19fV+l5NRFh5TxgM3nMKe\n1g6W3rmKrbsVykREREYaPwPZNmBK3OfJsWOHcM4td85VO+eqKysrh6W4kWT+lFIeunEJze1RLr5z\nJVsaWvwuSURERA6Dn4HsSeCq2GrLJcA+59wOH+sZ0eZOGs1DNy6hPdrNxXeuZFN9s98liYiISIrS\nue3FQ8BK4HgzqzWz683sJjO7KdbkKWAzsBG4C/hsumrJFbMnjuKhG5fQ7RxL71zFhl1NfpckIiIi\nKTDnnN81HJbq6mpXU1PjdxkZbWNdE5fe9QLd3Y4HbjiFWRNG+V2SiIhITjKzNc656sHajYib+uXw\nzBhbwiPLlhAOBrj0rlWs27bP75JERERkAApkWWp6ZTGPfHoJRZEQl921ile27vW7JBEREemHAlkW\nO6a8iIeXLWF0YZgr7n6BNe/u8bskERERSUKBLMtNKSvkkWWnUlYc4ap7XmD1loEeniAiIiJ+UCDL\nARNLC3hk2amMG5XP1SteZOWmRr9LEhERkTgKZDli/Oh8Hv70EiaVFnDtvS/ypw0NfpckIiIiMQpk\nOWRsST4PLVtCVXkR1923mufeqvO7JBEREUGBLOdUFOfx8xuXMKOymGX3r+HZN3b5XZKIiEjOUyDL\nQWVFEX5+4ymcMKGEmx5Yw2/X7fS7JBERkZymQJajSgsjPHDDKcydNJrP/fwlfvOqHiMqIiLiFwWy\nHDYqP8z91y1mwZRS/vKhl/jPtdv8LklERCQnKZDluJL8MPddt5hFVWV88ZG1PL6m1u+SREREco4C\nmVCUF+Leaxdz2rEV/N/HXuGR1e/5XZKIiEhOUSATAAoiQe6+upoPzKzkK4+/xgOr3vW7JBERkZyh\nQCa98sNB7rxyIR8+YSxf/eU67v3fd/wuSUREJCcokEkf+eEgP7liIWfNGcc//up17v6fzX6XJCIi\nkvUUyOQQkVCAH112MueeOIFv/eYNbn9uo98liYiIZLWQ3wVIZgoHA/zgkvmEgsZ3f/sWnVHHFz4y\n0++yREREspICmfQrFAzwbxfPJxgwvv/7t4l2d/OlM47DzPwuTUREJKsokMmAggHje5+aRzgQ4Id/\n2Ehnl+MrZx+vUCYiIjKEFMhkUMGA8Z1PnEgoaNzx/CY6u7r56rmzFMpERESGiAKZpCQQML514VzC\nwQD3/Okdol3d/OP5cxTKREREhoACmaTMzPj6ebMJBYy7//QOnd2Ob10wl0BAoUxERORoKJDJYTEz\n/v7cWYRDAX7y3CaiXd185xMnEVQoExEROWIKZHLYzIy/Oet4wsEAtz27gWiX43sXzVMoExEROUJp\n3RjWzM42s7fMbKOZ3ZLk/Ggz+5WZvWJm683s2nTWI0PHzPjSGcfxpTOO44mXt3HzI2uJdnX7XZaI\niMiIlLYRMjMLAj8GzgBqgdVm9qRz7vW4Zp8DXnfOnWdmlcBbZvagc64jXXXJ0PqrD88kHAzwL799\nk2hXN7dduoBwUA+AEBERORzp/JtzMbDRObc5FrAeBi5IaOOAEvOW6hUDu4FoGmuSNPjM6cfy1XNn\n8fS6nXz2wZdoj3b5XZKIiMiIks5ANgnYGve5NnYs3o+AWcB24DXgC865Q+a9zGyZmdWYWU19fX26\n6pWjcMP7p/ON8+fwu9d38ZkHXqKtU6FMREQkVX7PLZ0FrAUmAvOBH5nZqMRGzrnlzrlq51x1ZWXl\ncNcoKbr6tCr++eNz+cObddx4f41CmYiISIrSGci2AVPiPk+OHYt3LfCE82wE3gFOSGNNkmaXn3IM\n3/3kSfxpYwPX3bua1g7NQIuIiAwmnYFsNTDTzKaZWQS4BHgyoc17wIcBzGwccDywOY01yTC4eNEU\n/vWieaza3Mg1P11NS7tCmYiIyEDSFsicc1Hg88AzwBvAo8659WZ2k5ndFGv2T8BpZvYa8CzwFedc\nQ7pqkuHziZMn8/2l81nz7h6uXvEiTW2dfpckIiKSscw553cNh6W6utrV1NT4XYak6Dev7uALD7/M\n3Emjue+6xYwuCPtdkoiIyLAxszXOuerB2qU0QmZmXzCzUea5x8xeMrMzj75MyXbnnjSBH19+Muu3\n7+OKu19gb6u2mBMREUmU6pTldc65/cCZwBjgSuDWtFUlWeWsOeO544qFvLWzicvueoHdLQplIiIi\n8VINZD0PKfwo8DPn3Pq4YyKD+vCscdx1dTWb6pu5dPkqGprb/S5JREQkY6QayNaY2X/hBbJnzKwE\n0IML5bB88LhKVlyziHd3t3DJ8lXU7W/zuyQREZGMkGogux64BVjknGsFwnh7iIkclvfNqODeaxez\nfe8BLlm+ip37FMpERERSDWSnAm855/aa2RXAV4F96StLstmS6eXcf91i6praWbp8Jdv2HvC7JBER\nEV+lGsh+ArSa2Tzgr4FNwP1pq0qyXnVVGfdfv5jdLR0svXMlW3e3+l2SiIiIb1INZFHnbVh2AfAj\n59yPgZL0lSW54OSpY3jwhlNoaouy9M6VvNvY4ndJIiIivkg1kDWZ2d/ibXfxGzML4N1HJnJUTppc\nyoM3nMKBzi4uvnMlm+qb/S5JRERk2KUayJYC7Xj7ke3Ee1D499JWleSUuZNG89CyJUS7HJcsX8WG\nXU1+lyQiIjKsUgpksRD2IDDazD4GtDnndA+ZDJkTxo/i4WVLcA4uWb6KN3fu97skERGRYZPqo5Mu\nBl4ELgIuBl4ws0+lszDJPTPHlfDIp5cQChqXLl/F+u1ayCsiIrkh1SnLv8fbg+xq59xVwGLga+kr\nS3LVsZXFPLLsVArCQS676wVeq1UoExGR7JdqIAs45+riPjcexu+KHJaqiiIe+fSpFOeFuOzuVbz8\n3h6/SxIREUmrVEPVb83sGTO7xsyuAX4DPJW+siTXTSkr5NGbTmVMYYQr73mRmi27/S5JREQkbVK9\nqf/LwHLgpNhruXPuK+ksTGRSaQGPfvpUKkvyuGrFi7ywudHvkkRERNIi5WlH59zjzrkvxV6/SGdR\nIj3Gj87nkWVLmDA6n6t/+iL/u7HB75JERESG3ICBzMyazGx/kleTmWlfAhkWY0fl8/CyUzmmrIjr\n7l3N82/X+12SiIjIkBowkDnnSpxzo5K8Spxzo4arSJHKkjweWraE6ZXF3HhfDX94c5ffJYmIiAwZ\nrZSUEaOsKMJDN57C8eNL+PTP1vBf63f6XZKIiMiQUCCTEaW0MMIDN5zCnImj+eyDL/HUazv8LklE\nROSoKZDJiDO6IMzPrl/MvCml/OVDL/PkK9v9LklEROSoKJDJiFSSH+a+6xaz8Jgx3PzwyzzxUq3f\nJYmIiBwxBTIZsYrzQtx77SKWTC/nr//jFR5dvdXvkkRERI5IWgOZmZ1tZm+Z2UYzu6WfNqeb2Voz\nW29mz6ezHsk+hZEQ91y9iL+YUcHfPP4qP3/hPb9LEhEROWxpC2RmFgR+DJwDzAYuNbPZCW1KgduB\n851zc4CL0lWPZK+CSJC7rqrmQ8dX8ne/eI37/rzF75JEREQOSzpHyBYDG51zm51zHcDDwAUJbS4D\nnnDOvQeQ8ABzkZTlh4PcceVCzpg9jq8/uZ67/2ez3yWJiIikLJ2BbBIQf1NPbexYvOOAMWb2nJmt\nMbOrkn2RmS0zsxozq6mv1y7tklxeKMiPLzuZc+aO51u/eYM7nt/kd0kiIiIp8fum/hCwEDgXOAv4\nmpkdl9jIObfcOVftnKuurKwc7hplBImEAvzw0gWcN28itz79Jj98doPfJYmIiAwqlMbv3gZMifs8\nOXYsXi3Q6JxrAVrM7I/APODtNNYlWS4UDPD9i+cRChj/+ru36ex2fPEjMzEzv0sTERFJKp2BbDUw\n08ym4QWxS/DuGYv3n8CPzCwERIBTgO+nsSbJEaFggP93kRfKbnt2A9Gubr581vEKZSIikpHSFsic\nc1Ez+zzwDBAEVjjn1pvZTbHzdzjn3jCz3wKvAt3A3c65demqSXJLMGD8yydPIhQMcPtzm+js6ubv\nPjpLoUxERDJOOkfIcM49BTyVcOyOhM/fA76XzjokdwUCxrc/Ppdw0Ljrf96hs8vx9fNmK5SJiEhG\nSWsgE8kEZsY3zp9DKBBgxf++Q7S7m2+eP5dAQKFMREQygwKZ5AQz42sfm0U4ZNz5/GaiXY5vf/xE\nhTIREckICmSSM8yMW84+gUgwwA//sJHOLsd3P3USQYUyERHxmQKZ5BQz46/PPJ5QIMD3f/820e5u\n/vWieYSCfm/JJyIiuUyBTHLSFz4yk1DQ+N4zbxHtcvz7JfMJK5SJiIhPFMgkZ33uQzMIB41vP/Um\n0e5ufnjpyURCCmUiIjL89LeP5LRlHziWr583m2fW7+IzD6yhPdrld0kiIpKDNEImOe/a900jFAzw\ntV+u45oVq7lwwURmjC1hxthiRheE/S5PRERygAKZCHDlkmOIBI1v/Op1Vm5u7D0+blQeM8YWMzMW\n0GaOLWbmuBLKiiI+VisiItlGgUwkZumiqXxq4RS27TnAhromNtQ1s2FXMxvrmviPmq20dBycziwr\nihwMaLGQNmNsMWNL8vQUABEROWwKZCJxggFjankhU8sL+fCscb3HnXPs2NcWC2lNbKxrZmNdM796\nZTv726K97UryQ7GQVsLMccUcGwtsE0cXaBNaERHplwKZSArMjImlBUwsLeCDx1X2HnfOUd/czsZd\nzWyIhbQNdU08++YuHqnZ2tuuMBJkxtji3tfMsSXMHFvMlLJCbUwrIiIKZCJHw8wYW5LP2JJ8TptR\n0efc7paO3pG0DXXeqNqfNzbyxEvbettEQgGOrSxOmP4s5pjyIu2LJiKSQxTIRNKkrCjC4mllLJ5W\n1uf4/rZONtXFjajtamLt1j386pXtvW1CAaOqoqg3pM0YV8KMymKmVxaRHw4Od1dERCTNFMgSOQdd\nnRDSKjpJj1H5YRZMHcOCqWP6HG/tiLK5vsVbULDLC2tv7WzimfU76XZem4DB1LJCZsTuUZtR6Y2o\nHVtZTFGe/nEWERmp9G/wRLs3w+2nwoR5MLnae02qhtKpoNVzkkaFkRBzJ41m7qTRfY63R7t4p6El\nNpp2cAr0+bfr6Oxyve0mlRb0CWnaS01EZORQIEsUDMPiG2HbGqhZAatu944XjYXJi2DyQu/nxAWQ\nV+JvrZIT8kJBThg/ihPGj+pzvLOrm/d2t/ZuzdGzTcfKTY20R7t72yXbS23G2GLKi/OGuysiItIP\nc84N3iqDVFdXu5qamuH5w7o6Ydc6qK2JvVbD7k3eOQtA5azYKNoi72fF8RDQjdjir65ud+heavXN\nbNzVNOBeaj3ToNpLTURk6JjZGudc9aDtFMgOU+tub/SsdrUX0rbVQNs+71zeKJh0sjfF2RPSiioG\n/j6RYZK4l9qmei+svb2rqd+91GaMLWbGOO2lJiJypBTIhkt3tzdqVrv64GvX6+BiIxFjqrxw1hPS\nxp+oBQOSUeL3UtsYC2k923Q0NHf0tuvdS62yJ6RpLzURkcEokPmpowW2r/VGz3pG0pp2eOeCeTDh\npIMjaFowIBlsT0vHISFtY10zO/a19baJhAJMryhi5riSuOlPby+1SEhT+CKS2xTIMs2+bQdH0Lat\nge0vQzT2l5oWDMgIk7iXWs/Kz627D/S2SdxL7djYNKj2UhORXKJAlukSFwxsq4HGjd45LRiQESp+\nL7X4bTq2NLYk3UttRtzTCbSXmohkIwWykah3wUBsqjNxwcDEBbGApgUDMrL0t5faOw0th+ylFh/S\ntJeaiIx0GRHIzOxs4AdAELjbOXdrP+0WASuBS5xzjw30nVkdyBIdsmCgBnat14IByRrJ9lLrmQKN\n30ttbEkeM8dpLzURGXl8D2RmFgTeBs4AaoHVwKXOudeTtPsd0AasUCAbREcL7Hilb0jTggHJMol7\nqW3suV9Ne6mJyAiTCYHsVOAfnXNnxT7/LYBz7jsJ7W4GOoFFwK8VyI5Az4KBbbH70bRgQLJU/F5q\n3khaU2wFaDP7DnT2tuvZS633CQXaS01EfJJqIEvnHbSTgK1xn2uBU+IbmNkk4OPAh/ACmRyJ0ZO8\n15wLvc9dnd7UZvzmtW/9xjvXZ8FAbKpTCwZkhDAzJpYWMLG0gA8eV9l73DlHQ3NHn8UEG+qa+MOb\n9TxaU9vbrjAS5NjK2Gha3BToVO2lJiI+83tJ078DX3HOdQ80vWBmy4BlAFOnTh2m0kawYBgmzvde\ni2/0jrXuhm0vHZzqfP2X8NJ93rk+CwZiU53Flf1/v0iGMTMqS/KoLMnjtGP7LnZJtpfays2NPPHy\ntt42kVCAqWWFVBRHKC/Oo6LI+1leHKG8KK/3eHlxhJK8kKZDRWTI+TplaWbvAD3/ZqsAWoFlzrlf\n9ve9mrIcIn0WDMRWdSYuGOh9BJQWDEj2aWrr7L03bVNdM+82trK7pYOGlnYamzv6TIHGiwQDXlCL\nhbXy4ggVxXmUx4W4itjxsqKI9lwTyXGZcA9ZCO+m/g8D2/Bu6r/MObe+n/b3onvI/NXRCjvW9g1p\nvQsGIjBhXmxVZ+x+NC0YkCzWEe1mT2sHDc1eQGuMBbWG5g4am9tpbPF+NjR7beJXhcYryQtRVhzp\nDWwVcUEucTRuTGFEU6ciWcb3e8icc1Ez+zzwDN62Fyucc+vN7KbY+TvS9WfLEYoUwjGnea8e+7b1\nfQRUzU9h1e3euaKxfe9F04IBySKRUIBxo/IZNyp/0LbOOVo7urzAFgtuPaEtPtBt3d3Ky+/tZXdL\ne+9GufHMoKxwkNG3uHPFmj4VyRraGFYOT/yCgW1rvJ+HPGFg4cGpTi0YEDlEd7dj74HO3hG2xpZ2\nb7q0J8j1GZFrZ39bNOn3REKBfu53Swh0senTvJCmT0WGm+9TlumiQJaBEhcMxD9hIFICk07WggGR\no9Ae7WJPS6c32tZyMLQljsY1NndQ39xOR3/Tp/mhuBG3vlOmZbFjPedLNX0qMiQUyMQ/vQsGag6G\ntAEXDMyFkHZcFxkKzjlaOroOjr4l3O8WH+h6RuaSTZ8GzNt4N/5+t/KiyMEVp/H3xBXnURQJavpU\nJAkFMskshywYqIGm7d45LRgQ8U1Xt2Nva0ff+91673/rG+gaWzpo6mf6NC8U6J0eTbbi9GCg80bj\nIiHdyiC5QYFMMl+fBQNrYk8YOOCdK6o8OM2pBQMiGaM92sXu2PRo0hWocdOoDc0ddHQlnz4d1TN9\nmrjqtM/9b9770QVhPWFBRiwFMhl5ehYM9DwCSgsGREY05xzN7dHe0OaNuCVfgdrY3MHu1g6S/ZUU\nDFhs+vTgCNtA+78VavpUMogCmWSH+AUDPUGtba93TgsGRLJKV7djT+vB0NaQcL9bQ8IChub25NOn\n+eFA3ycs9Lnfre8K1LKiCOGg/sNO0keBTLJT4oKBbTWwc10/CwaqY08Y0IIBkWzU1tnVd9VpwirU\nxpa+24d0diX/+644L0RRXpCivJD3PhKKvY87ltf3WHzb+N/PCwU0Oid9KJBJ7uhdMBC3gW3igoFJ\ncRvYasGASM5xztHUM30at/9bQ5P3mKyW9ijNHVFa2r1Xc3tX3Ptov09iSBQKWFyI6z/kxQe9pMci\n3u+HNHo34imQSW5LdcHApGpv2lMLBkRkAJ1d3bS2d/WGtub2Q8NbsmMtHfHHu3rfR5PtNZJEXijQ\n/wjdACFtmV7AAAANNUlEQVSvKC8YF/i8n4XhoBZH+MD3RyeJ+Gr0JO81+wLvc1cn1L3ed9uNt56K\nNTYYO9tbMDBhPpSMh8IKKKqAwjLIL9WImkiOCwcDjC4MMLowfNTf5ZyjPdrdN6R19BPo4oJez7Hd\nLR28t7u19/dbOqJJF0MkMoPCcOI0bDAh8PWM5h0a6BKPaXp2aGmETHJXz4KB+Gd19iwYiBcIQWF5\nLKSVx72viL0vj72PHSsog6D+W0dEhkd3t+NAZ/wo3cGRuPiglyzkxQfCnuNtnalNzwYDRlEkSaBL\nuPfOC3nJ7sfrG/KydXGFRshEBlNYBjM/4r0AnIN9tdDaAC2NsZ8N3s/WxoPHdrzq/ex5PFQy+aV9\nQ1phWdz7JMEuXDA8fRaRrBOI3bdWlBdi7BB8X7Srm5aOxPCWPOTFH+/5Wd/U3ifk9beYIlGkd3o2\nGLdYInnI652qjSQPeUWR0IibnlUgE+lhBqVTvFcqujq9Ubbe4NYYC24NfY/t3gxbX/Te96wGTRQu\nio20DTL61nMsf7SmUUUkLULBAKMLAowuOPrpWfA2E25JuM+uJ+T1OdZxaMjb29pB7Z7Wg21TnJ4F\nKOwzKpcsvAVZVFXGmXPGD0k/j5YCmciRCoahZJz3SkV3N7TvG3j0raUBWuqh/k3vfc9ChESBcFxY\nG2T0rbACCsZoGlVEfJEXCpIXClJWFDnq73LOm55NGuiSHPNG8w4e27m/rc/0bWeXUyATyTmBgBeM\nCsYAM1L7nY6WgUffeo7tWOu973ca1aCgdJDRt4RgF84fqp6LiAwJM6MwEqIwEoIhWByfSffRK5CJ\nZLJIkfcac0xq7bs6+wa3xNG31gZvmrVxE2x9ITaN2s8NvJHiQ0NaYVlCiIs7ljdK06giMqJk0ipR\nBTKRbBIMe9t2lKQ4BN/d7a0sHWj0rbUBmnfBrte999G2fv7syMGRt8FG3wrLvWOB4ND1XURkBFMg\nE8llgUAsLJVBxczB2zvnTaP2O/oWd2zbS95oXPtA06hjUht96zmmx2CJSJZSIBOR1JlBXrH3GlOV\n2u9EO2IjbgOtRt0NjRvhvVVwYPcg06gpjL71rFbNK9E0qoiMCApkIpJeoQiMmuC9UtEzjdonsCUZ\nkWvaAbvWee+72pN/VzByaEjrd0PfMV7gCxcoxInIsFMgE5HMEj+NynGDt3cOOpoPjrTFh7jEY3u2\nxKZR9/f/fRb0glleccLPkuTHe9+XJJyLfdY0q4ikQIFMREY2My8s5ZVA2bTUfifanrAadTcc2APt\nTV64a2+O/Yz73FIfO97k/ezuTO3PCoT7CWxF/Ye4gT5rPzmRrKR/skUk94TyYNRE73Wkou3eAoc+\nIa4pLswl+xx7te2H/dv7tunvKQ6H1J4fC3P9jdoNFuoS3mulq0hGUCATETkSoTzvVVh29N/lnLed\nSH+hrqMl+ahdz+fWRtjzbt/jpLjhZbhwkKnZktRH9SJFuv9O5AilNZCZ2dnAD4AgcLdz7taE85cD\nXwEMaAI+45x7JZ01iYhkHDNvMUG4AKg8+u9zDjpbBw5xyT73BL/mndAYFwo7W1LtyAD33xUd3tRs\nXrE3GqiAJzkibYHMzILAj4EzgFpgtZk96Zx7Pa7ZO8AHnXN7zOwcYDlwSrpqEhHJCWYHn/JAis9a\nHUh3lxfWDpmKbUkt8O3d2vd3+ntG6yH9CCafZj3cqdqeY1pgIRksnSNki4GNzrnNAGb2MHAB0BvI\nnHN/jmu/CpicxnpERORIBIKQP8p7DYWu6MH76VIaxWvpO5Xb0tD3c1dHiv3oZ4FFuDA2BZ3vbZUS\nyj84JR3Kg2DewfOHHIs7l/RYRKN8kpJ0BrJJwNa4z7UMPPp1PfB0GusREZFMEAx5D7svKB2a74t2\nJA91qQS+tv3QtMvbyy4a/2rrf3+7wxXsL9xFEkJgXBgM5vVzLCHspRoeg2EFwwyXETf1m9mH8ALZ\nX/RzfhmwDGDq1KnDWJmIiGS8UARCZUOzwCKec97oW09I64oLa0mPdRwMcocc6+j7uc+xdmjbl/B9\ncd+R6hYrA7KBR/MGDIrJQuARhkdt29KvdP4vsw2YEvd5cuxYH2Z2EnA3cI5zrjHZFznnluPdX0Z1\ndXWKS4dERESOgsWFGD91d/cNaocdAgcKhnHf19HirdhNFgy72qE7evR9seCRTfv2CYpHGx7zMnK7\nl3QGstXATDObhhfELgEui29gZlOBJ4ArnXNvp7EWERGRkSkQgEDPKlwfdUWPMhgmGx1MEgx7Rgv7\nfF/c96a6pctAAiEvpC28Bs7656P/viGQtkDmnIua2eeBZ/C2vVjhnFtvZjfFzt8B/ANQDtxu3tx2\n1DlXna6aRERE5AgFQ94rUuRfDc55I3Uph8D4ewKThMDJmRM5zLmRNQNYXV3tampq/C5DREREZFBm\ntiaVwabAcBQjIiIiIv1TIBMRERHxmQKZiIiIiM8UyERERER8pkAmIiIi4jMFMhERERGfKZCJiIiI\n+GzE7UNmZvXAu8PwR1UADcPw52Qi9T135XL/c7nvkNv9V99z13D0/xjnXOVgjUZcIBsuZlaTq08N\nUN9zs++Q2/3P5b5Dbvdffc/NvkNm9V9TliIiIiI+UyATERER8ZkCWf+W+12Aj9T33JXL/c/lvkNu\n9199z10Z03/dQyYiIiLiM42QiYiIiPgspwOZmZ1tZm+Z2UYzuyXJeTOz22LnXzWzk/2oM11S6P/p\nZrbPzNbGXv/gR53pYGYrzKzOzNb1cz5rr30Kfc/m6z7FzP7bzF43s/Vm9oUkbbLy2qfY92y+9vlm\n9qKZvRLr/zeStMnWa59K37P22gOYWdDMXjazXyc5lxnX3TmXky8gCGwCpgMR4BVgdkKbjwJPAwYs\nAV7wu+5h7v/pwK/9rjVN/f8AcDKwrp/z2XztB+t7Nl/3CcDJsfclwNu58s99in3P5mtvQHHsfRh4\nAViSI9c+lb5n7bWP9e9LwM+T9TFTrnsuj5AtBjY65zY75zqAh4ELEtpcANzvPKuAUjObMNyFpkkq\n/c9azrk/ArsHaJK11z6Fvmct59wO59xLsfdNwBvApIRmWXntU+x71opdz+bYx3DslXgTdbZe+1T6\nnrXMbDJwLnB3P00y4rrnciCbBGyN+1zLof9ySqXNSJVq306LDeE+bWZzhqe0jJDN1z4VWX/dzawK\nWIA3WhAv66/9AH2HLL72sWmrtUAd8DvnXM5c+xT6Dtl77f8d+Bugu5/zGXHdczmQyeBeAqY6504C\nfgj80ud6ZHhk/XU3s2LgceBm59x+v+sZToP0PauvvXOuyzk3H5gMLDazuX7XNFxS6HtWXnsz+xhQ\n55xb43ctg8nlQLYNmBL3eXLs2OG2GakG7Ztzbn/PMLdz7ikgbGYVw1eir7L52g8o26+7mYXxAsmD\nzrknkjTJ2ms/WN+z/dr3cM7tBf4bODvhVNZe+x799T2Lr/37gPPNbAverTn/x8weSGiTEdc9lwPZ\namCmmU0zswhwCfBkQpsngatiKzCWAPucczuGu9A0GbT/ZjbezCz2fjHe/18ah71Sf2TztR9QNl/3\nWL/uAd5wzv1bP82y8tqn0vcsv/aVZlYae18AnAG8mdAsW6/9oH3P1mvvnPtb59xk51wV3t9zf3DO\nXZHQLCOue2i4/8BM4ZyLmtnngWfwVhyucM6tN7ObYufvAJ7CW32xEWgFrvWr3qGWYv8/BXzGzKLA\nAeASF1uSMtKZ2UN4q4oqzKwW+Dreja5Zf+1T6HvWXne8/1q+Engtdj8NwN8BUyHrr30qfc/maz8B\nuM/Mgnhh41Hn3K9z5N/5qfQ9m6/9ITLxumunfhERERGf5fKUpYiIiEhGUCATERER8ZkCmYiIiIjP\nFMhEREREfKZAJiIiIuIzBTIRkRSZ2elm9mu/6xCR7KNAJiIiIuIzBTIRyTpmdoWZvWhma83sztiD\nlZvN7Ptmtt7MnjWzyljb+Wa2KvZQ5V+Y2ZjY8Rlm9nsze8XMXjKzY2NfX2xmj5nZm2b2YM/u5iIi\nR0OBTESyipnNApYC74s9TLkLuBwoAmqcc3OA5/GeUABwP/CV2EOVX4s7/iDwY+fcPOA0oOdRKguA\nm4HZwHS8HfBFRI5Kzj46SUSy1oeBhcDq2OBVAVAHdAOPxNo8ADxhZqOBUufc87Hj9wH/YWYlwCTn\n3C8AnHNtALHve9E5Vxv7vBaoAv6U/m6JSDZTIBORbGPAfc65v+1z0OxrCe2O9Llx7XHvu9C/R0Vk\nCGjKUkSyzbPAp8xsLICZlZnZMXj/vvtUrM1lwJ+cc/uAPWb2/tjxK4HnnXNNQK2ZXRj7jjwzKxzW\nXohITtF/2YlIVnHOvW5mXwX+y8wCQCfwOaAFWBw7V4d3nxnA1cAdscC1Gbg2dvxK4E4z+2bsOy4a\nxm6ISI4x54501F5EZOQws2bnXLHfdYiIJKMpSxERERGfaYRMRERExGcaIRMRERHxmQKZiIiIiM8U\nyERERER8pkAmIiIi4jMFMhERERGfKZCJiIiI+Oz/A9oJEyMS3JJMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa028e75c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmsAAAD2CAYAAACJK8BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VfWd//HXJ/tyQxICCbIJSlBcQSPQ6rQuY4u1bl0U\nl1o3qK1ObX+d/mpna6fTmXF+7W86+qsdBcUdsbY6Mk6r1daliyhBsSwiIKIElbCTBEKSez+/P+5J\nchOyXCA39+be9/PxyCP3nvM9N5+vpyXvfL/fc465OyIiIiKSmrKSXYCIiIiI9E5hTURERCSFKayJ\niIiIpDCFNREREZEUprAmIiIiksIU1kRERERSmMKaiIiISApTWBMRERFJYQprIiIiIiksJ9kFDKQR\nI0b4hAkTkl2GiIiISL+WLVu2zd1H9tcurcLahAkTqK2tTXYZIiIiIv0ys/fiaadpUBEREZEUprAm\nIiIiksIU1kRERERSWELXrJnZLOB2IBu4x91v67a/HFgAHA00A9e5+8pg30agAQgDbe5ecyg1tLa2\nUldXR3Nz8yH3YygoKChg7Nix5ObmJrsUERERGUAJC2tmlg3cCZwL1AFLzWyxu6+OafY3wHJ3v8TM\njg3anxOz/yx333Y4ddTV1VFSUsKECRMws8P5qJTl7mzfvp26ujomTpyY7HJERERkACVyGnQ6sN7d\nN7h7C7AIuKhbm+OA3wG4+xpggplVDWQRzc3NVFRUpG1QAzAzKioq0n70UEREJBMlchp0DLAp5n0d\nMKNbmzeBzwG/N7PpwJHAWGAL4MDzZhYG7nb3eT39EDObC8wFGD9+fI+FpHNQa5cJfRQRkQwUiUCk\nFcItEG4NvlqCbTHvw609tzuU7VlZcNGdye55h2TfZ+024HYzWw6sAN4gukYN4Ax332xmlcBzZrbG\n3V/u/gFBiJsHUFNT44NUt4iIyNATCfcRemK2d4SYFgi3db6OtB3+9p5+3gHbY47zcP/9OlSWDdm5\nkJ0X/Z4VvM4rTtzPPASJDGubgXEx78cG2zq4+x7gWgCLDg29C2wI9m0Ovteb2ZNEp1UPCGupbteu\nXSxcuJCvfe1rB3XcZz7zGRYuXEhZWVmCKhMRkUPmHgSRgxm96S3E9BOc+gxUcbZt3+6RxP03ycqJ\nBp2s3JgAFGzLzuvc3x6M8op72N5b+54+O+Z13NtjQ1kuZGUn7r/HAEpkWFsKVJvZRKIhbTZwRWwD\nMysD9gZr2m4AXnb3PWZWDGS5e0Pw+lPADxJYa8Ls2rWLn/3sZweEtba2NnJyev/P/6tf/SrRpYmI\npLe2FmhphP0Nwff21w3R1x3b9nS+jm3f0tR3+EqkrB6CR+zIT+z23ELIH9ZDUDmIEHS427NyolOH\nkhAJC2vu3mZmNwPPEr11xwJ3X2VmNwb77wKmAA+YmQOrgOuDw6uAJ4N1WDnAQnd/JlG1JtKtt97K\nO++8w9SpU8nNzaWgoIDy8nLWrFnD2rVrufjii9m0aRPNzc3ccsstzJ07F+h8dFZjYyPnnXceZ5xx\nBn/6058YM2YMTz31FIWFhUnumYjIAHOH1r0xoWlP11DV0hANUt1D1f7Y1zHbwvvj+7nZ+ZAfgrxQ\nNPTkh6BoBJQd2XUkqL/QNFDbs3JA65AlhrmnzzKvmpoa7/5s0LfeeospU6YA8I//vYrVH+wZ0J95\n3OhhfO+C43vdv3HjRj772c+ycuVKXnzxRc4//3xWrlzZcYuNHTt2MHz4cPbt28dpp53GSy+9REVF\nRZewNmnSJGpra5k6dSqXXnopF154IVddddUBPyu2ryIigyLcdmBIah+56ilAtb/vaVtLY/zTdHkh\nyC8Jvoc633fZVhL93tO22PbZuj+lJIeZLYvnPrLJvsAg40yfPr3LvdDuuOMOnnzySQA2bdrEunXr\nqKio6HLMxIkTmTp1KgCnnnoqGzduHLR6RSTNuENbc9dQ1VeA6m9b2774fm5W7oEBqqAMSsf2EKB6\nCVXtgSu3WFNuklEyKqz1NQI2WIqLO68wefHFF3n++ed55ZVXKCoq4swzz+zxXmn5+fkdr7Ozs9m3\nL85/HEUkPUQiwRqq3tZd9TFS1dO0YaQtvp+bW3RggBo2uo9QFUwj9jTalZPf/88TkR5lVFhLhpKS\nEhoaGnrct3v3bsrLyykqKmLNmjUsWbJkkKsTkYTpWNx+mOuu9jdAa1N8P9Oyew5QJaPiC1VdQlho\nyFwpJ5LuFNYSrKKigtNPP50TTjiBwsJCqqo6H9Awa9Ys7rrrLqZMmcIxxxzDzJkzk1ipiADRkLVv\nJzTvin7ft7OXkavuIazbtnivFswpiAlQJdFQFaqEvKNigldJ36GqfVtuoRami6ShjLrAIN1lUl9F\n+uQeDUz7dsK+mNDV/hUbxDr2B9/7HcWyHsJST6Gqr6AV81qL20Uyli4wEJGhL9wGzbt7CFhxhLC+\n1mVl50NheedX2ZFwxMnB+7LO7QVl0ff5pZ2BK7dIi9tFZFAprIlI4rXui3OUKzaI7YL9u/v+3Pxh\nXcNV6ZggYJV3++q2LVf3KRSRoUNhTUTiE4lE12T1Ocq1q+cQ1nbgVc4dLLtrkAqNgpHHHhi4uoew\ngtLo3dRFRNKc/qUTyTRtLUGQOphRrmBfXzcszS3qGqYqjo5vlCsvpEXxIiJ9UFgTGYrco88t7C9g\ndQlhwbaWxj4+2KIjVrFhqnxC/6NchWW6j5aISIIorIkkUyTcuYC+x4DVRwiLtPb+udl5XcNU6Tio\nOrGH0a1uoSu/VIvnRURSjMJaigmFQjQ29jXyISmptfkgR7naX/ezgD6vpGuoqoxjLVf7AnpNLYqI\n9CgScfa2hmlobqWxuY2G/W00NrfRGHxvCUe4auaRyS6zg8KaSG/2N8COd2Hnu7BzIzRt7X0xfV/P\nR7TsriNYxSOhorr/tVwFpboHl4hIjEjEaWqJhqqG5uhXY0fQau1hWzSItYeyju0tbfR1m9m8nCyF\ntUxy6623Mm7cOG666SYAvv/975OTk8MLL7zAzp07aW1t5Yc//CEXXXRRkivNQO6wd3tnINuxIfp6\nx4bo+6atXdvnFHYNU8MnQuG0/ke58ks0yiUiGS3cHrKaY4NWa9dQ1RG0Wjvet+/veL0/vufaFudl\nU1KQS6ggh1B+DiUFOYwaVkBJQQ6h/Oj2kvycLvtj94XyUysepVY1ifbrW+GjFQP7maNOhPNu63X3\nZZddxje+8Y2OsPbzn/+cZ599lq9//esMGzaMbdu2MXPmTC688EJMv9AHXiQCDR90DWHtoWznxuit\nKDoYDBsTDWHHnAflE2H4UdH35ROhYFiyeiEikhThiHeEpN5Grxo6AlVrR9uuQauVppZwXD8vlN8Z\nntpD0xGlBcH23JhQFRu0crtsK87LITsrvX6fZlZYS4Jp06ZRX1/PBx98wNatWykvL2fUqFF885vf\n5OWXXyYrK4vNmzezZcsWRo0alexyh6a2Fti96cCRsfZAFt7f2TYrF8rGR0PY+JnR7+UTo4Gs7EjI\nLUhaN0REBkp7yIodvWroMkp14Fqtho73ncfEE7LMIJTXGZ5CBTkMK8xlTFlht1B14OhVbPgqzssh\nK81C1kDJrLDWxwhYIn3xi1/kF7/4BR999BGXXXYZjzzyCFu3bmXZsmXk5uYyYcIEmpv7uGmoRG9T\n0X26sv317rqu9//KLYqGsBHVMPnTnSNjw4+C0rGQlZ28foiI9KEtHKFpf5iGYASre9DqdV1Wc2uX\n4LU33pCV33U6sLQwl7HlhdFtBwSt3I5t7ceUFORSlJutkJVgCQ1rZjYLuB3IBu5x99u67S8HFgBH\nA83Ade6+Mp5jh5LLLruMOXPmsG3bNl566SV+/vOfU1lZSW5uLi+88ALvvfdesktMPvfoQv3e1o81\nbunavnB4NISNmwEnzY6+bh8lC1VqjZiIDKq2cKTL9F9DTLDqa11WexBraI6+3tcaf8gaFhOeyory\nGDu8iJL8nB7XZZXEhK72fQpZQ0fCwpqZZQN3AucCdcBSM1vs7qtjmv0NsNzdLzGzY4P258R57JBx\n/PHH09DQwJgxYzjiiCO48sorueCCCzjxxBOpqanh2GOPTXaJgyMSgcaPelk/9u6Bt7EoGR0NYNXn\nHrh+rLAsOX0QkbTU3Bpm594Wdja1Rr/vbWHX3q4L4PfEXlHYJWi10tzax9M9AlntI1lByCopyKG8\nOI9xw4tiwtSBo1fd12UV5WVrjXOGSeTI2nRgvbtvADCzRcBFQGzgOg64DcDd15jZBDOrAo6K49gh\nZcWKzgsbRowYwSuvvNJjuyF/j7Vwa7f1Y+/GrB97t+szIrNyouvHyifC2Jqu68fKJ+hh2yJy0Nyd\nPc1t7Nrbws69rexsagnCV2uwLXZ757a+wlaW0SVghfJzqCjO48iK4i7beluXNSxYLF+Yq5AlhyaR\nYW0MsCnmfR0wo1ubN4HPAb83s+nAkcDYOI8FwMzmAnMBxo8fPyCFSz9a9kYX7reHsNhRsl2bwGOG\n8XMKO6coJ53Tbf3YOD2IW0R61RqOsKsjUAUjXk0HBq9de1vY0RQdCdu1r5VwpOcbaGUZlBbmUl6c\nR3lRHmPKCjh+9DDKi3IpK8pjeHFex+vyojzKiqKjWQpZkmzJ/k15G3C7mS0HVgBvAPFd3xtw93nA\nPICampo+bnEnB+WA9WMbO0NZw4dd2xaURUPYmFPhhC90XT9WMkrrx0QynLuzrzXcMaK1a2/nVGP7\ntOOu2OC1t4VdTdEF873Jy8mivCiX8iBYHTOqJAhZuUHQymN4cWfwKi/KZVhBrtZoyZCUyLC2GRgX\n835ssK2Du+8BrgWw6J8t7wIbgML+jj0Y7p72fxV5X7di7vmA6KL93taP7dvZtX1oVDSAHX1251Rl\n+yhZ0fCB64iIpLRIxNnT3Nox0rWr2zqvjlGvjhDWyo69LbS09T7NWJKfQ1lxZ8iaOKK4M2R1BK4g\nmAWjXxrtkkySyLC2FKg2s4lEg9Zs4IrYBmZWBux19xbgBuBld99jZv0eG6+CggK2b99ORUVF2v4f\n293Zvn07BQXd7hEWboM9dd1udRGzfqx1b2dby4aycdHwdfwlB64fyyse1D6JSOK1tEW6TDH2Pt3Y\nGbx27W2hl1lGsrOMssJcyoJgNba8iBPH5DK8OK8jcHUEr+LOqcbc7KzB7bjIEJOwsObubWZ2M/As\n0dtvLHD3VWZ2Y7D/LmAK8ICZObAKuL6vYw+ljrFjx1JXV8fWrVv7bzwUuUOkjQLfx9jGN2H12pj1\nY+9DJGYaIacgGrzKJ8JRZ8asH5sYXeiv51CKDEnuzt6Wnq9m7BjdajpwW1+P7snPyeoSsqaMGtYR\nwtpHt9rDVvtUZEmBbmoqkgh20NNnKaympsZra2uTXcbAa97dw/3Hgvd7PgBizmF+KQyf0HVkrGP9\n2BGQpb9gRVJZOOLs2dfTmq5Wul7J2DV4tYT7mGYsyOkjZHUd5WoPXoV5unm0SKKZ2TJ3r+mvXbIv\nMBCIjo41be19/dje7V3bF1dGA9jET3S9/9jwo6IPDk/T6V6RoWZ/W7gzZHUf8WrqtqA+aLd7Xyu9\n/Q2dnWVdphKPrChi6riyjvVePV3VWFaYS46mGUWGNIW1wRIJw57NPTy/cmP0e0vM/dUsK/pYpPKJ\nMOWCbuvHJkJ+KGndEMlE7tHnLPY0uhW7rmtXt9Gwvh75U5ib3Rm8inMZXVZIeVEuw4NF9uXdrmQs\nL86jJD8nbdfeikjvFNYGUtt+2PleL/cfex/CLZ1ts/M6149NOKPrdGXZeMjJS1o3RDLVzqYW1m5p\nYF19I+uC7+9sbWRHUwut4d6XjAwryOlY3zUylM/kypIDFtLHBrPyojwKcjXNKCLxUVg7WPsbenl+\n5cboA8Vj14/llUTXj1UdD8d+tmsgGzZaDxQXSZId3UPZlkbW1TewrbHzD6rivGwmVZVw+qQRVJYU\nHHDriPYwVqppRhFJMIW1gzH/bNi8rOu2ohHRAHbkxw9cP1ZUofVjIkm0vXE/a7c0sr6+gbVBIFu3\npZHtTZ2hLJSfw6TKEGcdU8nkqhImVYWYXFXC6NICTTmKSEpQWDsYJ3w+uoasPZSVT4CCYcmuSiSj\nuTvbg5Gy9fWN0RGzLY2sq49OX7YL5edQXRXinClBKKuMhrIjFMpEJMUprB2Mj92U7ApEMpa7s62x\npWMtWew05s69rR3tSoJQdu6UKqqrQlRXlTC5KsSoYQplIjI0KayJSEpxd7Y27o+Ojm1pYG19I+u3\nNLK2voFdsaGsIIfJVSV8+vhRVFeVUB2MlFUNy1coE5G0orAmIknh7mxt2N+xlix2bdnufZ2hbFgQ\nys474YiOQFZdFaKyRKFMRDKDwpqIJJS7U9+wP2YtWfT72i0N7GnufNxRaWEuk6tCnH9STCirDDFS\noUxEMpzCmogMCHdny579B9ynbF23UFZWlMvkyhIuOHl0RyibVBViZEihTESkJwprInJQ3J2P9jRH\npy9jR8vqG2mICWXlRblUV5Vw4dTRVFdGpy6rK0sYEcpTKBMROQgKayLSI3fnw93NXW+JESz2b9jf\nGcqGF+dRXRni4qljOgJZdVWIEaH8JFYvIpI+FNZEMpy780F7KNsSE8rqG2mMCWUVxXlUV4W4eNoY\nJleFmFQZvSVGhUKZiEhCKayJZIhIxPlg976Oacu1wY1j129poCnmgeMjQvlUV4b4/CljmFRVwuTK\nEJMqFcpERJJFYU0kzUQizuZd+2KuuozeEmNdfSN7Y0LZyJJoKPtizbiOu/lXV4YoL85LYvUiItKd\nwprIENUeytbG3NF/fTB9GRvKKkvyqa4KcWnNOKqD515OGqlQJiIyVCQ0rJnZLOB2IBu4x91v67a/\nFHgYGB/U8mN3vy/YtxFoAMJAm7vXJLJWkVQViTh1O/cdcEuM9fWN7GvtDGVVw/KprizhstPGUR2s\nJ5tUGaKsSKFMRGQoS1hYM7Ns4E7gXKAOWGpmi919dUyzm4DV7n6BmY0E3jazR9y9/enLZ7n7tkTV\nKJJKwhGnbufejjv6t68tW1/fSHNrpKPdqGEFVFeFuHz6+GCkLMSkkSWUFuUmsXoREUmURI6sTQfW\nu/sGADNbBFwExIY1B0osetOlELADaOv+QSLpJBxxNu3Y2+NI2f62zlB2RGkBkypDXDnjSKorow8k\nn1QZorRQoUxEJJMkMqyNATbFvK8DZnRr81NgMfABUAJc5u7tv60ceN7MwsDd7j6vpx9iZnOBuQDj\nx48fuOpFDlM44rwfhLKO+5RtaeSdrV1D2ejSAiZVlfCxoyqi9ykLQtmwAoUyERFJ/gUGnwaWA2cD\nRwPPmdnv3X0PcIa7bzazymD7Gnd/ufsHBCFuHkBNTY0PYu0iALSFI0Eo63wQ+br6aChriQllY8oK\nmVQZ4vRJFR03jp1UGaJEoUxERPqQyLC2GRgX835ssC3WtcBt7u7AejN7FzgWeM3dNwO4e72ZPUl0\nWvWAsCYyWNrCEd7bsbfjEUtrgynMDduaDghl1VUh/qJ6RMctMSZVhgjlJ/tvIxERGYoS+dtjKVBt\nZhOJhrTZwBXd2rwPnAP83syqgGOADWZWDGS5e0Pw+lPADxJYq0ivtjXu5+uPvkHtxp20hDtD2djy\nQqorQ3xy8siOUHa0QpmIiAywhP1Wcfc2M7sZeJborTsWuPsqM7sx2H8X8E/A/Wa2AjDgO+6+zcyO\nAp4MHvacAyx092cSVatIb+obmrly/qts2rmXa06fwOSq6C0xjh4ZolihTEREBoFFZyDTQ01NjdfW\n1ia7DEkTW/Y0c/n8JXy0u5kF15zGzKMqkl2SiIikETNbFs99ZDU0INKDj3ZHg1r9nmbuv3Y60ycO\nT3ZJIiKSoRTWRLr5YNc+Lp+/hO2NLTx4/XROPVJBTUREkkdhTSRG3c69XD5/CbuaWnno+ulMG1+e\n7JJERCTDKayJBDbt2MvseUtoaG7l4RtmcPK4smSXJCIiorAmAvDe9iaumP8qjfvbWDhnJieMKU12\nSSIiIoDCmgjvbmviivlLaG4Ns3DODI4fraAmIiKpQ2FNMto7Wxu5Yv4SWsPOwjkzmXLEsGSXJCIi\n0oXCmmSs9fUNXD7/VdydR+fM5JhRJckuSURE5AAKa5KR1m5p4Ir5SzAzFs2dyaRKBTUREUlNWcku\nQGSwvfXhHmbPW0KWgpqIiAwBGlmTjLLqg91cdc+r5Odk8+jcmUwcUZzskkRERPqkkTXJGCs37+aK\n+a9SmJvNY19RUBMRkaFBI2uSEd7ctIsv3fsqJQW5LJo7k3HDi5JdkoiISFziGlkzsyfM7Hwz00ic\nDDlvvL+Tq+59ldKiXB77ioKaiIgMLfGGr58BVwDrzOw2MzsmgTWJDJhl7+3gS/e+xvDiPB6b+zHG\nliuoiYjI0BJXWHP35939SuAUYCPwvJn9ycyuNbPcRBYocqiWbtzB1fe+xsiSfB6b+zFGlxUmuyQR\nEZGDFve0pplVANcANwBvALcTDW/PJaQykcOwZMN2vrzgNapKC1g0dyajSguSXZKIiMghiXfN2pPA\n74Ei4AJ3v9DdH3P3vwJCfRw3y8zeNrP1ZnZrD/tLzey/zexNM1tlZtfGe6xIb/60fhvX3Pcao8sK\nWTR3JlXDFNRERGToivdq0Dvc/YWedrh7TU/bzSwbuBM4F6gDlprZYndfHdPsJmC1u19gZiOBt83s\nESAcx7EiB/j9uq3c8EAtEyqKeWTODEaE8pNdkoiIyGGJdxr0ODMra39jZuVm9rV+jpkOrHf3De7e\nAiwCLurWxoESMzOiI3Q7gLY4jxXp4sW367n+gVomjihmoYKaiIikiXjD2hx339X+xt13AnP6OWYM\nsCnmfV2wLdZPgSnAB8AK4BZ3j8R5LABmNtfMas2sduvWrfH0RdLQC2vqmfvgMiaNDPHonJlUKKiJ\niEiaiDesZQejX0DHFGfeAPz8TwPLgdHAVOCnZjbsYD7A3ee5e42714wcOXIASpKh5vnVW5j7UC3H\njCph4ZwZlBcPxP80RUREUkO8Ye0Z4DEzO8fMzgEeDbb1ZTMwLub92GBbrGuBJzxqPfAucGycx4rw\n7KqP+OojyzhudCkP3zCDsiIFNRERSS/xhrXvAC8AXw2+fgv8736OWQpUm9lEM8sDZgOLu7V5HzgH\nwMyqgGOADXEeKxnu1ys+5KZHXueEMaU8dP10Sgt1yz8REUk/cV0NGqwj+8/gKy7u3mZmNwPPAtnA\nAndfZWY3BvvvAv4JuN/MVgAGfMfdtwH0dGz83ZJ09/SfP+CWRcuZNq6M+649jZICBTUREUlP5u79\nNzKrBv4VOA7ouGmVux+VuNIOXk1NjdfW1ia7DEmwp5Zv5puPLafmyOEsuPY0Qvnx3oFGREQkdZjZ\nst5ugRYr3mnQ+4iOqrUBZwEPAg8fenkih+aJ1+v45mPLmT5xOPdfp6AmIiLpL96wVujuvyU6Evee\nu38fOD9xZYkc6PHaTXzr8Tf52NEV3HfNdIryFNRERCT9xfvbbr+ZZQHrgrVkm+njMVMiA23Ra+/z\n3SdXcMakEcy/uoaC3OxklyQiIjIo4h1Zu4Xoc0G/DpwKXAV8OVFFicR65NX3uPWJFXyieqSCmoiI\nZJx+R9aCG+Be5u5/DTQSvTeayKB48JWN/MNTqzj72Er+86pTyM9RUBMRkczSb1hz97CZnTEYxYjE\nWvCHd/nB06s597gqfnrFNAU1ERHJSPGuWXvDzBYDjwNN7Rvd/YmEVCUZ757fb+CH//MWs44fxR2X\nTyMvJ94ZexERkfQSb1grALYDZ8dsc0BhTQbc3S+9w7/+eg3nn3gE/zF7KrnZCmoiIpK54n2Cgdap\nyaC484X1/OjZt7ng5NH85NKTyVFQExGRDBdXWDOz+4iOpHXh7tcNeEWSse747Tr+/bm1XDx1ND/+\nooKaiIgIxD8N+nTM6wLgEuCDgS9HMpG785Pn13HHb9fxuVPG8KMvnEx2liW7LBERkZQQ7zToL2Pf\nm9mjwB8SUpFkFHfn//5mLT99YT2X1ozlXz93koKaiIhIjEN9Xk81UDmQhUjmcXf+7Zm3ueuld7h8\n+jj++eITyVJQExER6SLeNWsNdF2z9hHwnYRUJBnB3fmXX73F/N+/y1Uzx/ODC09QUBMREelBvNOg\nJYkuRDKHu/ODp1dz3x83cs3HJ/C9C47DTEFNRESkJ3Fdbmdml5hZacz7MjO7OHFlSbpyd76/eBX3\n/XEj150+UUFNRESkH/HeG+F77r67/Y277wK+l5iSJF1FIs7fP7WSB155j7mfOIq//+wUBTUREZF+\nxBvWemoXz0PgZ5nZ22a23sxu7WH/t81sefC10szCZjY82LfRzFYE+2rjrFNSVCTi/O1/reDhJe/z\n1TOP5rvnHaugJiIiEod4rwatNbN/B+4M3t8ELOvrADPLDtqfC9QBS81ssbuvbm/j7j8CfhS0vwD4\nprvviPmYs9x9W5w1SooKR5xbf/lnHl9Wx1+dPYn/de5kBTUREZE4xTuy9ldAC/AYsAhoJhrY+jId\nWO/uG9y9JTjuoj7aXw48Gmc9MkSEI863f/Emjy+r4xt/Wc23PnWMgpqIiMhBiPdq0CbggGnMfowB\nNsW8rwNm9NTQzIqAWcDNsT8WeN7MwsDd7j6vl2PnAnMBxo8ff5AlSiK1hSN86/E3eWr5B3zr3Mn8\n1TnVyS5JRERkyIn3atDnzKws5n25mT07gHVcAPyx2xToGe4+FTgPuMnMPtHTge4+z91r3L1m5MiR\nA1iSHI62cIRvPLacp5Z/wLc/fYyCmoiIyCGKdxp0RHAFKADuvpP+n2CwGRgX835ssK0ns+k2Beru\nm4Pv9cCTRKdVZQhoDUf4+qI3ePrPH/Ld847lprMmJbskERGRISvesBYxs445RjObQNcnGvRkKVBt\nZhPNLI9oIFvcvVFw/7ZPAk/FbCs2s5L218CngJVx1ipJ1NIW4eaFr/OrFR/xd+dP4SufPDrZJYmI\niAxp8V4N+rfAH8zsJcCAvyBYJ9Ybd28zs5uBZ4FsYIG7rzKzG4P9dwVNLwF+E6yLa1cFPBksRM8B\nFrr7M3EivimAAAASJklEQVTWKkmyvy3MTY+8zvNv1fP9C47jmtMnJrskERGRIc/c+xsgCxqaVRIN\naG8AhUC9u7+cwNoOWk1NjdfW6pZsydDcGuZrj7zO79bU808XHc+XPjYh2SWJiIikNDNb5u41/bWL\n90HuNwC3EF13thyYCbwCnH04RUp6aG4N85WHlvHS2q38yyUncsUMXZUrIiIyUOJds3YLcBrwnruf\nBUwDdvV9iGSC5tYwcx6s5eV1W/k/nz9JQU1ERGSAxbtmrdndm80MM8t39zVmdkxCK5OUt68lzPUP\nLOWVDdv50RdO5gunjk12SSIiImkn3rBWF9xn7b+A58xsJ/Be4sqSVNe0v43rH1jKa+/u4N8vPZlL\npimoiYiIJEK8TzC4JHj5fTN7ASgFdHVmhmrc38Z19y2l9r0d/OSyqVw0dUyySxIREUlb8Y6sdXD3\nlxJRiAwNDc2tXHPfUpZv2sUdl0/jsyeNTnZJIiIiae2gw5pkrj3NrVx972us3Lybn14+jfNOPCLZ\nJYmIiKQ9hTWJy+69rVy94FVWf7iHn115Cp86flSySxIREckICmvSr117W7jq3ldZ+1Ejd111KudM\nqUp2SSIiIhlDYU36tLOphSvveZX1Wxu5+0unctaxlckuSUREJKMorEmvtjfu58p7XuXdbU3Mv7qG\nT04emeySREREMo7CmvRoa8N+rrxnCe/v2Mu9Xz6NM6pHJLskERGRjKSwJgeob2jmivmvsnnnPhZc\ncxofP1pBTUREJFkU1qSLLXuauXz+Ej7a3cz9157GjKMqkl2SiIhIRlNYkw4f7t7HFfNfpX5PMw9e\nN52aCcOTXZKIiEjGU1gTADbv2sfl85awo6mFB6+fwalHlie7JBEREUFhTYBNO/Zy+fwl7N7XykPX\nT2faeAU1ERGRVJGVyA83s1lm9raZrTezW3vY/20zWx58rTSzsJkNj+dYGRjvb9/L7HlL2LOvlUdu\nmKGgJiIikmISFtbMLBu4EzgPOA643MyOi23j7j9y96nuPhX4LvCSu++I51g5fBu3NTF73is0tbSx\ncM5MThpbluySREREpJtEjqxNB9a7+wZ3bwEWARf10f5y4NFDPFYO0rvbmpg9bwn7WsMsvGEmJ4wp\nTXZJIiIi0oNEhrUxwKaY93XBtgOYWREwC/jlIRw718xqzax269ath110JnhnayOX3f0KreEIj86d\nyXGjhyW7JBEREelFQtesHYQLgD+6+46DPdDd57l7jbvXjBypxyH1Z92WBi67ewkRh0VzZ3LsKAU1\nERGRVJbIsLYZGBfzfmywrSez6ZwCPdhjJU5vf9TA5fOXkGXRoFZdVZLskkRERKQfiQxrS4FqM5to\nZnlEA9ni7o3MrBT4JPDUwR4r8Xvrwz1cPn8J2VnGorkzmVQZSnZJIiIiEoeE3WfN3dvM7GbgWSAb\nWODuq8zsxmD/XUHTS4DfuHtTf8cmqtZ0t3Lzbq6691UKc7N5dM5MJowoTnZJIiIiEidz92TXMGBq\namq8trY22WWklBV10aAWys/h0TkzGV9RlOySREREBDCzZe5e01+7VLnAQBJg+aZdXHHPEkoKclg0\nV0FNRERkKNLjptLU6+/v5Mv3vkZ5cR4L58xgbLmCmoiIyFCksJaGajfu4Jr7ljIilMfCOTMZXVaY\n7JJERETkEGkaNM289u4Orl7wGpUl+Sya+zEFNRERkSFOI2tp5JV3tnPd/UsZXVbAo3NmUjmsINkl\niYiIyGFSWEsTf1y/jesfWMq48iIWzpnJyJL8ZJckIiIiA0DToGng5bVbue7+pUyoKObRuQpqIiIi\n6UQja0Pci2/XM/ehZRw9MsQjN8xgeHFesksSERGRAaSwNoT9bs0WbnzodSaPCvHw9TMoK1JQExER\nSTeaBh2inlu9ha88tIxjjyjhketnKqiJiIikKYW1IeiZlR/x1YeXcdzoUh66fgalRbnJLklEREQS\nRGFtiPmfP3/ITQtf56SxpTx0/XRKCxXURERE0pnWrA0hi9/8gG8+tpxTxpdx37XTCeXr9ImIiKQ7\njawNEf/1xma+segNTj2ynPsV1ERERDKGfuMPAb9cVse3f/EmMyZWcO81NRTl6bSJiIhkCo2spbif\n127ir3/xJh8/egQLrjlNQU1ERCTD6Dd/Cnv0tff57hMr+MTkkcz70qkU5GYnuyQREREZZAkdWTOz\nWWb2tpmtN7Nbe2lzppktN7NVZvZSzPaNZrYi2FebyDpT0cNL3uO7T6zgrGMU1ERERDJZwkbWzCwb\nuBM4F6gDlprZYndfHdOmDPgZMMvd3zezym4fc5a7b0tUjanqgT9t5HuLV/GXUyq588pTyM9RUBMR\nEclUiRxZmw6sd/cN7t4CLAIu6tbmCuAJd38fwN3rE1jPkHDvH97le4tX8anjqvjZlacqqImIiGS4\nRIa1McCmmPd1wbZYk4FyM3vRzJaZ2dUx+xx4Ptg+t7cfYmZzzazWzGq3bt06YMUnw/yXN/BPT6/m\nvBNGceeVp5CXo+s/REREMl2yLzDIAU4FzgEKgVfMbIm7rwXOcPfNwdToc2a2xt1f7v4B7j4PmAdQ\nU1Pjg1j7gPrPF9/h355Zw/knHcF/XDaV3GwFNREREUnsyNpmYFzM+7HBtlh1wLPu3hSsTXsZOBnA\n3TcH3+uBJ4lOq6aln/5uHf/2zBouPHk0tyuoiYiISIxEpoKlQLWZTTSzPGA2sLhbm6eAM8wsx8yK\ngBnAW2ZWbGYlAGZWDHwKWJnAWpPm9ufX8ePfrOVz08bwk8umkqOgJiIiIjESNg3q7m1mdjPwLJAN\nLHD3VWZ2Y7D/Lnd/y8yeAf4MRIB73H2lmR0FPGlm7TUudPdnElVrMrg7P3luLXf8bj1fOHUs//b5\nk8jOsmSXJSIiIinG3IfsMq8D1NTUeG1t6t+Szd358W/e5s4X3mH2aeP4l0tOJEtBTUREJKOY2TJ3\nr+mvXbIvMMg47s5tz6zh7pc2cMWM8fzwohMU1ERERKRXCmuDyN355/95i3v+8C5Xf+xI/vHC4wmm\nekVERER6pLA2SNydf/zv1dz/p41ce/oE/uGzxymoiYiISL8U1gZBJOJ8b/EqHlryHjecMZG/PX+K\ngpqIiIjERWEtwSIR5++eWsnCV9/nK588iltnHaugJiIiInFTWEugSMT57hMreKx2E18782i+/elj\nFNRERETkoCisJUg44nznl3/mF8vq+PrZk/jmuZMV1EREROSgKawlQDjifPvxN3nijc188y8nc8tf\nVie7JBERERmiFNYGWFs4wrcef5Onln/AX39qMjefraAmIiIih05hbQC1hiN847Hl/M+fP+Q7s47l\nq2ceneySREREZIhTWBsgreEIX3/0DX698iP+9jNTmPOJo5JdkoiIiKQBhbUB0NIW4eaFr/Ob1Vv4\nh88ex3VnTEx2SSIiIpImFNYO0/62MDc98jrPv1XPDy46nqs/NiHZJYmIiEgaUVg7DM2tYb768DJe\neHsrP7z4BK6aeWSySxIREZE0o7B2iJpbw8x9aBkvr93Kv37uRC6fPj7ZJYmIiEgaUlg7BPtawsx5\nsJY/vrON//P5k7j0tHHJLklERETSVFYiP9zMZpnZ22a23sxu7aXNmWa23MxWmdlLB3NsMuxtaeO6\n+5fyx3e28eMvnKygJiIiIgmVsJE1M8sG7gTOBeqApWa22N1Xx7QpA34GzHL3982sMt5jk6FpfzSo\nLd24g59cOpWLp41JZjkiIiKSARI5sjYdWO/uG9y9BVgEXNStzRXAE+7+PoC71x/EsYOqLRzh2vuW\nUvveTm6fPU1BTURERAZFIsPaGGBTzPu6YFusyUC5mb1oZsvM7OqDOHZQ5WRn8ekTRnHH7GlccPLo\nZJYiIiIiGSTZFxjkAKcC5wCFwCtmtuRgPsDM5gJzAcaPT+wVmdfrZrciIiIyyBI5srYZiF19PzbY\nFqsOeNbdm9x9G/AycHKcxwLg7vPcvcbda0aOHDlgxYuIiIikgkSGtaVAtZlNNLM8YDawuFubp4Az\nzCzHzIqAGcBbcR4rIiIikvYSNg3q7m1mdjPwLJANLHD3VWZ2Y7D/Lnd/y8yeAf4MRIB73H0lQE/H\nJqpWERERkVRl7p7sGgZMTU2N19bWJrsMERERkX6Z2TJ3r+mvXUJviisiIiIih0dhTURERCSFKayJ\niIiIpLC0WrNmZluB9xL8Y0YA2xL8M1JVJvcdMrv/mdx3yOz+q++ZK5P7P1h9P9Ld+73vWFqFtcFg\nZrXxLAZMR5ncd8js/mdy3yGz+6++Z2bfIbP7n2p91zSoiIiISApTWBMRERFJYQprB29esgtIokzu\nO2R2/zO575DZ/VffM1cm9z+l+q41ayIiIiIpTCNrIiIiIilMYa0XZjbLzN42s/VmdmsP+83M7gj2\n/9nMTklGnYkQR9/PNLPdZrY8+PqHZNSZCGa2wMzqzWxlL/vT+bz31/d0Pu/jzOwFM1ttZqvM7JYe\n2qTzuY+n/2l5/s2swMxeM7M3g77/Yw9t0vncx9P/tDz37cws28zeMLOne9iXGufe3fXV7Yvow+Pf\nAY4C8oA3geO6tfkM8GvAgJnAq8muexD7fibwdLJrTVD/PwGcAqzsZX9anvc4+57O5/0I4JTgdQmw\nNlP+P38Q/U/L8x+cz1DwOhd4FZiZQec+nv6n5bmP6d//Ahb21MdUOfcaWevZdGC9u29w9xZgEXBR\ntzYXAQ961BKgzMyOGOxCEyCevqctd38Z2NFHk3Q97/H0PW25+4fu/nrwugF4CxjTrVk6n/t4+p+W\ngvPZGLzNDb66L+ZO53MfT//TlpmNBc4H7umlSUqce4W1no0BNsW8r+PAf7jiaTMUxduvjwdDwr82\ns+MHp7SUkK7nPV5pf97NbAIwjegIQ6yMOPd99B/S9PwH02DLgXrgOXfPqHMfR/8hTc898B/A/wYi\nvexPiXOvsCaH4nVgvLufBPw/4L+SXI8MjrQ/72YWAn4JfMPd9yS7nsHWT//T9vy7e9jdpwJjgelm\ndkKyaxpMcfQ/Lc+9mX0WqHf3ZcmupT8Kaz3bDIyLeT822HawbYaifvvl7nvah83d/VdArpmNGLwS\nkypdz3u/0v28m1ku0aDyiLs/0UOTtD73/fU/3c8/gLvvAl4AZnXbldbnvl1v/U/jc386cKGZbSS6\n5OdsM3u4W5uUOPcKaz1bClSb2UQzywNmA4u7tVkMXB1cKTIT2O3uHw52oQnQb9/NbJSZWfB6OtH/\nHW0f9EqTI13Pe7/S+bwH/boXeMvd/72XZml77uPpf7qefzMbaWZlwetC4FxgTbdm6Xzu++1/up57\nd/+uu4919wlEf9f9zt2v6tYsJc59zmD/wKHA3dvM7GbgWaJXRy5w91VmdmOw/y7gV0SvElkP7AWu\nTVa9AynOvn8B+KqZtQH7gNkeXDYz1JnZo0SvfBphZnXA94guuE3r8w5x9T1tzzvRv7C/BKwI1u4A\n/A0wHtL/3BNf/9P1/B8BPGBm2URDyM/d/elM+Pc+EE//0/Xc9ygVz72eYCAiIiKSwjQNKiIiIpLC\nFNZEREREUpjCmoiIiEgKU1gTERERSWEKayIiIiIpTGFNROQwmdmZZvZ0susQkfSksCYiIiKSwhTW\nRCRjmNlVZvaamS03s7uDB1g3mtlPzGyVmf3WzEYGbaea2ZLg4dVPmll5sH2SmT1vZm+a2etmdnTw\n8SEz+4WZrTGzR9rv+C4icrgU1kQkI5jZFOAy4PTgodVh4EqgGKh19+OBl4g+uQHgQeA7wcOrV8Rs\nfwS4091PBj4OtD96ZhrwDeA44CiiTwUQETlsetyUiGSKc4BTgaXBoFchUA9EgMeCNg8DT5hZKVDm\n7i8F2x8AHjezEmCMuz8J4O7NAMHnvebudcH75cAE4A+J75aIpDuFNRHJFAY84O7f7bLR7O+7tTvU\nZ/Dtj3kdRv++isgA0TSoiGSK3wJfMLNKADMbbmZHEv138AtBmyuAP7j7bmCnmf1FsP1LwEvu3gDU\nmdnFwWfkm1nRoPZCRDKO/vITkYzg7qvN7O+A35hZFtAK3AQ0AdODffVE17UBfBm4KwhjG4Brg+1f\nAu42sx8En/HFQeyGiGQgcz/UEX8RkaHPzBrdPZTsOkREeqNpUBEREZEUppE1ERERkRSmkTURERGR\nFKawJiIiIpLCFNZEREREUpjCmoiIiEgKU1gTERERSWEKayIiIiIp7P8DjX3xtg9fqbYAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa02f871390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 43s, sys: 8.12 s, total: 2min 51s\n",
      "Wall time: 2min 52s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "# Create a ConvNet classifier with 2 CONV-RELU-POOL layers, with 7x7 filters, \n",
    "# 32 and 64 output features and a hidden linear layer size of 512.\n",
    "model_params = {\n",
    "        'input_size': 784,\n",
    "        'output_size': 10,\n",
    "        'filter_sizes': [5],\n",
    "        'output_depths': [4],\n",
    "        'hidden_linear_size': 128,\n",
    "        'use_batch_norm': False\n",
    "}\n",
    "\n",
    "training_params = {\n",
    "        'keep_prob': 0.5,\n",
    "        'num_epochs': 5,\n",
    "        'batch_size': 50,\n",
    "        'stop_early': False,\n",
    "}\n",
    "\n",
    "trained_model, training_results, conv_weights = build_train_eval_and_plot(\n",
    "        model_params, \n",
    "        training_params, \n",
    "        verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SC9-p4EuSSyc"
   },
   "source": [
    "The ConvNet classifier takes quite a long time to train, but gives a very respectable test accuracy of over **99%**! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vV1i8ilwElbp"
   },
   "source": [
    "## What has the network learned?\n",
    "Remember that a filter in a convolutional layer is used to multiply blocks in the input volume. Let's plot the weights of the first layer of the trained model. Darker pixels indicate that the particular filter reacts more strongly to those regions of the input blocks. Notice how each filter has learned to react differently to different patterns in the input. It's tricky to see in our tiny filters, but those in lower layers of ConvNets, particularly when applied to natural images, often function as simple Gabor filters or edge detectors, while filters in higher layers often react to more abstract shapes and concepts. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 272,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2702,
     "status": "ok",
     "timestamp": 1504370167320,
     "user": {
      "displayName": "Avishkar Bhoopchand",
      "photoUrl": "//lh5.googleusercontent.com/-OainnMWSi6A/AAAAAAAAAAI/AAAAAAAAAOQ/eI7Z19q5v7E/s50-c-k-no/photo.jpg",
      "userId": "105781587643595215149"
     },
     "user_tz": -60
    },
    "id": "Bcr3zmMnQkC-",
    "outputId": "535554e9-cf7a-47e2-d17a-e54e3ba5cfa1"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhUAAAHVCAYAAABG5AlXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGiZJREFUeJzt3W+o3gX9//H32/1z8y/ZKnOalqFY8NU4SSBBLQQrqTvd\nMKpblRXfSEGQvFdQNyIICSqQFL9QFEHeiOybGGl9pT92chZNC6SyLGM6c7q2lpvv342davXbdc61\nnfd1fT5nPR5wYGdeXLwYZ2+fXOfPsqoCAGC1Thp6AABwYhAVAEALUQEAtBAVAEALUQEAtBAVAEAL\nUQEAtBAVAEALUQEAtFg/iyfdvHlznX766bN46hZbtmwZesJEe/fuHXrCsjZs2DD0hIle/OIXDz1h\not/97nexe/fuHHoHK9u0aVON+Ubs27dv6AkTPffcc0NPWNYpp5wy9ISJDh06NPSEZe3fv//Jqtq6\n0uNmEhWnn356XHPNNbN46havec1rhp4w0Q9+8IOhJyxrzP/jvuGGG4aeMNEb3vCGoScwpS1btsSb\n3vSmoWdMtLi4OPSEif74xz8OPWFZCwsLQ0+YaM+ePUNPWNaOHTseneZxPv0BALQQFQBAC1EBALQQ\nFQBAC1EBALQQFQBAC1EBALQQFQBAC1EBALQQFQBAC1EBALQQFQBAC1EBALQQFQBAi6miIjOvysxf\nZeYjmfnRWY8C6OJ+wfysGBWZuS4iPhcRb46ISyLinZl5yayHAayW+wXzNc0rFZdHxCNV9euq+ltE\nfDUi3j7bWQAt3C+Yo2mi4pyI+P0R7z+29HsAY+d+wRy1faFmZl6bmYuZubh///6upwWYuSPv14ED\nB4aeA2vWNFHxh4g494j3ty393r+oqluqaqGqFjZv3ty1D2A1jvl+bdq0aW7j4EQzTVT8JCJemZkX\nZObGiLgmIr4x21kALdwvmKP1Kz2gqg5m5ocj4q6IWBcRt1XVzpkvA1gl9wvma8WoiIioqm9FxLdm\nvAWgnfsF8+MnagIALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAA\nLUQFANBCVAAALUQFANBCVAAALdbP4kn37t0bP/rRj2bx1C2ef/75oSdMtH///qEnLOvAgQNDT5jo\n8ccfH3rCRM8999zQE5jSxo0b46UvfenQMyZ69NFHh54w0fbt24eesKyXvexlQ0+YaNeuXUNPWNaO\nHTumepxXKgCAFqICAGghKgCAFqICAGghKgCAFqICAGghKgCAFqICAGghKgCAFqICAGghKgCAFqIC\nAGghKgCAFqICAGghKgCAFitGRWbelpm7MvMX8xgE0MkNg/mZ5pWK2yPiqhnvAJiV28MNg7lYMSqq\n6vsR8dQctgC0c8Ngftq+piIzr83MxcxcPHjwYNfTAszckfdr//79Q8+BNastKqrqlqpaqKqF9evX\ndz0twMwdeb82b9489BxYs3z3BwDQQlQAAC2m+ZbSr0TEDyPiosx8LDPfO/tZAD3cMJifFb/4oare\nOY8hALPghsH8+PQHANBCVAAALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAALUQF\nANBCVAAALUQFANBCVAAALVb8p8+Px759++L++++fxVO3+PSnPz30hIle8IIXDD1hWc8999zQEya6\n+OKLh54w0cknnzz0BKa0YcOGOOecc4aeMdF3vvOdoSdM9O1vf3voCcvasmXL0BMmetGLXjT0hGXd\neeedUz3OKxUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtR\nAQC0EBUAQAtRAQC0EBUAQAtRAQC0WDEqMvPczLwnMx/KzJ2Zed08hgGslvsF87V+isccjIgbquqB\nzDwtIn6amXdX1UMz3gawWu4XzNGKr1RU1eNV9cDSr5+NiIcj4pxZDwNYLfcL5muaVyr+ITPPj4jL\nIuLHR/lv10bEtS2rAJpNe7/OPPPMue6CE8nUX6iZmadGxNcj4vqqeubf/3tV3VJVC1W10DkQYLWO\n5X6dcsop8x8IJ4ipoiIzN8Thv5Bfrqo7ZjsJoI/7BfMzzXd/ZETcGhEPV9VnZj8JoIf7BfM1zSsV\nV0TEeyJie2Y+uPT2lhnvAujgfsEcrfiFmlV1X0TkHLYAtHK/YL78RE0AoIWoAABaiAoAoIWoAABa\niAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABarPhPnx+Ps88+O973\nvvfN4qlbXHjhhUNPmOimm24aesKyPvCBDww9YaJbb7116AkTPfnkk0NPYEovfOEL4/3vf//QMyZ6\n7LHHhp4w0emnnz70hGW94x3vGHrCRIuLi0NPaOGVCgCghagAAFqICgCghagAAFqICgCghagAAFqI\nCgCghagAAFqICgCghagAAFqICgCghagAAFqICgCghagAAFqICgCgxYpRkZknZ+b9mfmzzNyZmR+f\nxzCA1XK/YL7WT/GYAxGxvar2ZuaGiLgvM/+3qn40420Aq+V+wRytGBVVVRGxd+ndDUtvNctRAB3c\nL5ivqb6mIjPXZeaDEbErIu6uqh8f5THXZuZiZi7u27eveyfAcTnW+7V79+75j4QTxFRRUVWHqurS\niNgWEZdn5quP8phbqmqhqha2bNnSvRPguBzr/TrrrLPmPxJOEMf03R9V9XRE3BMRV81mDsBsuF8w\ne9N898fWzDxz6debI+LKiPjlrIcBrJb7BfM1zXd/nB0R/5OZ6+JwhHytqr4521kALdwvmKNpvvvj\n5xFx2Ry2ALRyv2C+/ERNAKCFqAAAWogKAKCFqAAAWogKAKCFqAAAWogKAKCFqAAAWogKAKCFqAAA\nWogKAKCFqAAAWogKAKCFqAAAWqz4T58fr5NOGm+vvPSlLx16wkQf+9jHhp6wrE2bNg09YaLXvva1\nQ0+Y6POf//zQE5jSM888E3fdddfQMya6+OKLh54w0Y033jj0hGWN+X6ddtppQ09oMd7/8wMAa4qo\nAABaiAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABa\niAoAoIWoAABaTB0VmbkuM3dk5jdnOQigm/sF83Esr1RcFxEPz2oIwAy5XzAHU0VFZm6LiLdGxBdn\nOwegl/sF8zPtKxU3R8SNEfH8pAdk5rWZuZiZi/v27WsZB9DgmO7Xs88+O79lcIJZMSoy8+qI2FVV\nP13ucVV1S1UtVNXCli1b2gYCHK/juV+nnXbanNbBiWeaVyquiIi3ZeZvI+KrEbE9M78001UAPdwv\nmKMVo6KqbqqqbVV1fkRcExHfrap3z3wZwCq5XzBffk4FANBi/bE8uKrujYh7Z7IEYIbcL5g9r1QA\nAC1EBQDQQlQAAC1EBQDQQlQAAC1EBQDQQlQAAC1EBQDQQlQAAC1EBQDQQlQAAC1EBQDQQlQAAC1E\nBQDQIquq/0kzn4iIR5ue7oUR8WTTc83CmPeNeVvEuPd1b3tZVW1tfD5mpPl+RfxnfZx3G/O+MW+L\nGOiGzSQqOmXmYlUtDL1jkjHvG/O2iHHvG/M21pYxfyyNeVvEuPeNeVvEcPt8+gMAaCEqAIAWayEq\nbhl6wArGvG/M2yLGvW/M21hbxvyxNOZtEePeN+ZtEQPtG/3XVAAAa8NaeKUCAFgDRh0VmXlVZv4q\nMx/JzI8OvedImXlbZu7KzF8MveXfZea5mXlPZj6UmTsz87qhN/1dZp6cmfdn5s+Wtn186E1Hk5nr\nMnNHZn5z6C2sTe7X8Rnz/YpYGzdsyPs12qjIzHUR8bmIeHNEXBIR78zMS4Zd9S9uj4irhh4xwcGI\nuKGqLomI10XEf4/oz+5ARGyvqv+KiEsj4qrMfN3Am47muoh4eOgRrE3u16qM+X5FrI0bNtj9Gm1U\nRMTlEfFIVf26qv4WEV+NiLcPvOkfqur7EfHU0DuOpqoer6oHln79bBz+4Dpn2FWH1WF7l97dsPQ2\nqi/sycxtEfHWiPji0FtYs9yv4zTm+xUx/hs29P0ac1ScExG/P+L9x2JEH1hrRWaeHxGXRcSPh13y\nT0svzT0YEbsi4u6qGs22JTdHxI0R8fzQQ1iz3K8GY7xfEaO/YYPerzFHBauUmadGxNcj4vqqembo\nPX9XVYeq6tKI2BYRl2fmq4fe9HeZeXVE7Kqqnw69Bf6TjfV+RYz3ho3hfo05Kv4QEece8f62pd9j\nCpm5IQ7/hfxyVd0x9J6jqaqnI+KeGNfndq+IiLdl5m/j8EvW2zPzS8NOYg1yv1ZhLdyviFHesMHv\n15ij4icR8crMvCAzN0bENRHxjYE3rQmZmRFxa0Q8XFWfGXrPkTJza2aeufTrzRFxZUT8cthV/1RV\nN1XVtqo6Pw5/zH23qt498CzWHvfrOI35fkWM+4aN4X6NNiqq6mBEfDgi7orDX6jztaraOeyqf8rM\nr0TEDyPiosx8LDPfO/SmI1wREe+Jw5X64NLbW4YeteTsiLgnM38ehw/v3VXl2zY5obhfqzLm+xXh\nhi3LT9QEAFqM9pUKAGBtERUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtR\nAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0\nWD+TJ12/vjZu3DiLp26xefPmoSdMtG/fvqEnLOuvf/3r0BMm2rBhw9ATJjp06FAcOnQoh97Bys46\n66w677zzhp4x0Z49e4aeMFFVDT1hWQcPHhx6wkRnnHHG0BOWtXPnzierautKj5tJVGzcuDEuuuii\nWTx1i1e96lVDT5hox44dQ09Y1kMPPTT0hIle8pKXDD1hoj/96U9DT2BK5513Xtx7771Dz5jozjvv\nHHrCRIcOHRp6wrJ27do19ISJrr766qEnLOviiy9+dJrH+fQHANBCVAAALUQFANBCVAAALUQFANBC\nVAAALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAALUQFANBiqqjIzKsy81eZ+Uhm\nfnTWowC6uF8wPytGRWaui4jPRcSbI+KSiHhnZl4y62EAq+V+wXxN80rF5RHxSFX9uqr+FhFfjYi3\nz3YWQAv3C+Zomqg4JyJ+f8T7jy393r/IzGszczEzFw8ePNi1D2A1jvl+7d69e27j4ETT9oWaVXVL\nVS1U1cL69eu7nhZg5o68X2edddbQc2DNmiYq/hAR5x7x/ral3wMYO/cL5miaqPhJRLwyMy/IzI0R\ncU1EfGO2swBauF8wRyt+nqKqDmbmhyPirohYFxG3VdXOmS8DWCX3C+Zrqi9+qKpvRcS3ZrwFoJ37\nBfPjJ2oCAC1EBQDQQlQAAC1EBQDQQlQAAC1EBQDQQlQAAC1EBQDQQlQAAC1EBQDQQlQAAC1EBQDQ\nQlQAAC1EBQDQYqp/+vxYnXzyyXHhhRfO4qlbXHDBBUNPmOhd73rX0BOWdd999w09YaLXv/71Q0+Y\n6CMf+cjQE5jS3r174//+7/+GnjHR008/PfSEiT71qU8NPWFZH/zgB4eeMNGmTZuGntDCKxUAQAtR\nAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0EBUAQAtRAQC0\nEBUAQIsVoyIzb8vMXZn5i3kMAujkhsH8TPNKxe0RcdWMdwDMyu3hhsFcrBgVVfX9iHhqDlsA2rlh\nMD9tX1ORmddm5mJmLh44cKDraQFm7sj7tWfPnqHnwJrVFhVVdUtVLVTVwqZNm7qeFmDmjrxfZ5xx\nxtBzYM3y3R8AQAtRAQC0mOZbSr8SET+MiIsy87HMfO/sZwH0cMNgftav9ICqeuc8hgDMghsG8+PT\nHwBAC1EBALQQFQBAC1EBALQQFQBAC1EBALQQFQBAC1EBALQQFQBAC1EBALQQFQBAC1EBALQQFQBA\nC1EBALRY8Z8+Px7nnXdefOELX5jFU7e49dZbh54w0R//+MehJyzr5S9/+dATJnrxi1889ISJ1q+f\nyV81ZuCMM86Iq6++eugZE332s58desJE11133dATlvWJT3xi6AkTfe973xt6QguvVAAALUQFANBC\nVAAALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAALUQFANBCVAAA\nLUQFANBixajIzHMz857MfCgzd2bmdfMYBrBa7hfM1/opHnMwIm6oqgcy87SI+Glm3l1VD814G8Bq\nuV8wRyu+UlFVj1fVA0u/fjYiHo6Ic2Y9DGC13C+Yr2P6morMPD8iLouIHx/lv12bmYuZubh79+6e\ndQBNpr1fTzzxxLynwQlj6qjIzFMj4usRcX1VPfPv/72qbqmqhapaOOusszo3AqzKsdyvrVu3zn8g\nnCCmiorM3BCH/0J+uarumO0kgD7uF8zPNN/9kRFxa0Q8XFWfmf0kgB7uF8zXNK9UXBER74mI7Zn5\n4NLbW2a8C6CD+wVztOK3lFbVfRGRc9gC0Mr9gvnyEzUBgBaiAgBoISoAgBaiAgBoISoAgBaiAgBo\nISoAgBaiAgBoISoAgBaiAgBoISoAgBaiAgBoISoAgBaiAgBoseI/fX48Dh06FM8888wsnrrFhz70\noaEnTHTSSePuvC1btgw9YaIx/9mN+c+N/19VDT1hot/85jdDT5joJS95ydATlvWKV7xi6AkTveMd\n7xh6wrLuuuuuqR433isMAKwpogIAaCEqAIAWogIAaCEqAIAWogIAaCEqAIAWogIAaCEqAIAWogIA\naCEqAIAWogIAaCEqAIAWogIAaCEqAIAWK0ZFZp6cmfdn5s8yc2dmfnwewwBWy/2C+Vo/xWMORMT2\nqtqbmRsi4r7M/N+q+tGMtwGslvsFc7RiVFRVRcTepXc3LL3VLEcBdHC/YL6m+pqKzFyXmQ9GxK6I\nuLuqfnyUx1ybmYuZufjUU0917wQ4Lsd6v5544on5j4QTxFRRUVWHqurSiNgWEZdn5quP8phbqmqh\nqhZe8IIXdO8EOC7Her+2bt06/5Fwgjim7/6oqqcj4p6IuGo2cwBmw/2C2Zvmuz+2ZuaZS7/eHBFX\nRsQvZz0MYLXcL5ivab774+yI+J/MXBeHI+RrVfXN2c4CaOF+wRxN890fP4+Iy+awBaCV+wXz5Sdq\nAgAtRAUA0EJUAAAtRAUA0EJUAAAtRAUA0EJUAAAtRAUA0EJUAAAtRAUA0EJUAAAtRAUA0EJUAAAt\nRAUA0GLFf/r8eOzZsyfuvPPOWTx1iyuvvHLoCRPt3r176AnL+vOf/zz0hIlOOmm8jbxnz56hJzCl\nxx9/PD75yU8OPWOim2++eegJE11//fVDT1jWG9/4xqEnTHTHHXcMPaHFeK8wALCmiAoAoIWoAABa\niAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABaiAoAoIWoAABaiAoA\noMXUUZGZ6zJzR2Z+c5aDALq5XzAfx/JKxXUR8fCshgDMkPsFczBVVGTmtoh4a0R8cbZzAHq5XzA/\n075ScXNE3BgRz096QGZem5mLmbm4d+/elnEADY7pfv3lL3+Z3zI4wawYFZl5dUTsqqqfLve4qrql\nqhaqauHUU09tGwhwvI7nfp1yyilzWgcnnmleqbgiIt6Wmb+NiK9GxPbM/NJMVwH0cL9gjlaMiqq6\nqaq2VdX5EXFNRHy3qt4982UAq+R+wXz5ORUAQIv1x/Lgqro3Iu6dyRKAGXK/YPa8UgEAtBAVAEAL\nUQEAtBAVAEALUQEAtBAVAEALUQEAtBAVAEALUQEAtBAVAEALUQEAtBAVAEALUQEAtBAVAECLrKr+\nJ818IiIebXq6F0bEk03PNQtj3jfmbRHj3te97WVVtbXx+ZiR5vsV8Z/1cd5tzPvGvC1ioBs2k6jo\nlJmLVbUw9I5JxrxvzNsixr1vzNtYW8b8sTTmbRHj3jfmbRHD7fPpDwCghagAAFqshai4ZegBKxjz\nvjFvixj3vjFvY20Z88fSmLdFjHvfmLdFDLRv9F9TAQCsDWvhlQoAYA0QFQBAi1FHRWZelZm/ysxH\nMvOjQ+85Umbelpm7MvMXQ2/5d5l5bmbek5kPZebOzLxu6E1/l5knZ+b9mfmzpW0fH3rT0WTmuszc\nkZnfHHoLa5P7dXzGfL8i1sYNG/J+jTYqMnNdRHwuIt4cEZdExDsz85JhV/2L2yPiqqFHTHAwIm6o\nqksi4nUR8d8j+rM7EBHbq+q/IuLSiLgqM1838KajuS4iHh56BGuT+7UqY75fEWvjhg12v0YbFRFx\neUQ8UlW/rqq/RcRXI+LtA2/6h6r6fkQ8NfSOo6mqx6vqgaVfPxuHP7jOGXbVYXXY3qV3Nyy9jeqr\nhTNzW0S8NSK+OPQW1iz36ziN+X5FjP+GDX2/xhwV50TE7494/7EY0QfWWpGZ50fEZRHx42GX/NPS\nS3MPRsSuiLi7qkazbcnNEXFjRDw/9BDWLPerwRjvV8Tob9ig92vMUcEqZeapEfH1iLi+qp4Zes/f\nVdWhqro0IrZFxOWZ+eqhN/1dZl4dEbuq6qdDb4H/ZGO9XxHjvWFjuF9jjoo/RMS5R7y/ben3mEJm\nbojDfyG/XFV3DL3naKrq6Yi4J8b1ud0rIuJtmfnbOPyS9fbM/NKwk1iD3K9VWAv3K2KUN2zw+zXm\nqPhJRLwyMy/IzI0RcU1EfGPgTWtCZmZE3BoRD1fVZ4bec6TM3JqZZy79enNEXBkRvxx21T9V1U1V\nta2qzo/DH3Pfrap3DzyLtcf9Ok5jvl8R475hY7hfo42KqjoYER+OiLvi8BfqfK2qdg676p8y8ysR\n8cOIuCgzH8vM9w696QhXRMR74nClPrj09pahRy05OyLuycyfx+HDe3dV+bZNTiju16qM+X5FuGHL\n8mO6AYAWo32lAgBYW0QFANBCVAAALUQFANBCVAAALUQFANBCVAAALf4fOfzt/FGqyMIAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa02b2f04d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "weights = conv_weights[0]\n",
    "_, _, _, out_depth = weights.shape\n",
    "grid_size = int(out_depth**0.5)\n",
    "\n",
    "fig = plt.figure()\n",
    "\n",
    "i = 1\n",
    "for r in range(grid_size):\n",
    "    for c in range(grid_size):\n",
    "        ax = fig.add_subplot(grid_size, grid_size, i)\n",
    "        ax.imshow(weights[:, :, 0, r*grid_size+c], cmap=\"Greys\")\n",
    "        i += 1\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vNjw-0cBcfah"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZPehk6-WXLvo"
   },
   "source": [
    "# Aside: The Effect of Random Initialization - RUN THIS ON A GPU INSTANCE ONLY!\n",
    "Initialization of model parameters matters! Here is a ConvNet with different, but seemingly sensible, initialization of the weights in the linear layer. Running this gives significantly worse results. Judging by the accuracy plot, it's possible that training this model long enough will get it to a simliar level as before, but it will take much longer. This shows that initialization of model parameters is a an important consideration, especially as models become more complex. In practice, there are a number of different initialization schemes to consider. In particular, [Xavier](http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization) tends to work well with ConvNets and is worth considering. We won't go into any details in this practical though."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 1273,
     "output_extras": [
      {
       "item_id": 41
      },
      {
       "item_id": 42
      },
      {
       "item_id": 43
      },
      {
       "item_id": 44
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1771121,
     "status": "ok",
     "timestamp": 1503659151747,
     "user": {
      "displayName": "Avishkar Bhoopchand",
      "photoUrl": "//lh5.googleusercontent.com/-OainnMWSi6A/AAAAAAAAAAI/AAAAAAAAAOQ/eI7Z19q5v7E/s50-c-k-no/photo.jpg",
      "userId": "105781587643595215149"
     },
     "user_tz": -60
    },
    "id": "grDD7jGUXXpF",
    "outputId": "eea0addd-c8c1-4ba5-ed86-197eafdb392f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 Training cost= 95.223685306\n",
      "Epoch: 0001 Validation acc= 0.323000014\n",
      "Epoch: 0002 Training cost= 4.522310467\n",
      "Epoch: 0002 Validation acc= 0.294400007\n",
      "Epoch: 0003 Training cost= 3.359841522\n",
      "Epoch: 0003 Validation acc= 0.372599989\n",
      "Epoch: 0004 Training cost= 2.702690359\n",
      "Epoch: 0004 Validation acc= 0.417400002\n",
      "Epoch: 0005 Training cost= 2.234992374\n",
      "Epoch: 0005 Validation acc= 0.439799994\n",
      "Epoch: 0006 Training cost= 1.954660964\n",
      "Epoch: 0006 Validation acc= 0.479999989\n",
      "Epoch: 0007 Training cost= 1.770798251\n",
      "Epoch: 0007 Validation acc= 0.509599984\n",
      "Epoch: 0008 Training cost= 1.644968104\n",
      "Epoch: 0008 Validation acc= 0.519200027\n",
      "Epoch: 0009 Training cost= 1.517993350\n",
      "Epoch: 0009 Validation acc= 0.584399998\n",
      "Epoch: 0010 Training cost= 1.381812039\n",
      "Epoch: 0010 Validation acc= 0.646399975\n",
      "Epoch: 0011 Training cost= 1.280346963\n",
      "Epoch: 0011 Validation acc= 0.657599986\n",
      "Epoch: 0012 Training cost= 1.202895438\n",
      "Epoch: 0012 Validation acc= 0.695400000\n",
      "Epoch: 0013 Training cost= 1.134407729\n",
      "Epoch: 0013 Validation acc= 0.711199999\n",
      "Epoch: 0014 Training cost= 1.077231023\n",
      "Epoch: 0014 Validation acc= 0.729200006\n",
      "Epoch: 0015 Training cost= 1.020781099\n",
      "Epoch: 0015 Validation acc= 0.731800020\n",
      "Epoch: 0016 Training cost= 0.986821358\n",
      "Epoch: 0016 Validation acc= 0.748600006\n",
      "Epoch: 0017 Training cost= 0.948568019\n",
      "Epoch: 0017 Validation acc= 0.760999978\n",
      "Epoch: 0018 Training cost= 0.907726714\n",
      "Epoch: 0018 Validation acc= 0.773599982\n",
      "Epoch: 0019 Training cost= 0.863210992\n",
      "Epoch: 0019 Validation acc= 0.780600011\n",
      "Epoch: 0020 Training cost= 0.832545544\n",
      "Epoch: 0020 Validation acc= 0.786599994\n",
      "Optimization Finished!\n",
      "Accuracy on test set: 0.7838\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmsAAAD+CAYAAABleEI9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X90VPWd//HXvTOZJBMSYJJJBhQVQeuPKnXXlkV71BIw\nu20RIj9sXY8W/WK756v89LDH2OqpVenuUo60nlrxqOx6aP2ihHq0WBQsuLZfoPgTt7Df7a4/GjQ/\nSUJ+kcnM3O8f8yMz+QETZHLvTJ4PD8ydO/d+7nsm1/Cae+/nfgzLsiwBAADAkUy7CwAAAMDwCGsA\nAAAORlgDAABwMMIaAACAgxHWAAAAHIywBgAA4GAZDWs1NTW66qqrNG/evMS89vZ23X777aqqqtId\nd9yhjo6OxGsPPfSQrr/+es2fP1+HDx/OZGkAAABZIaNh7cYbb9RTTz2VMm/Tpk2aNWuWdu7cqZkz\nZ+qJJ56QJO3du1effPKJXn31VT344IN64IEHMlkaAABAVshoWLvyyitVUlKSMm/37t2qrq6WJFVX\nV2v37t2J+QsWLJAkzZgxQx0dHWpubs5keQAAAI436tesHTt2TGVlZZIkv9+vY8eOSZIaGxsVCAQS\ny1VUVKihoWG0ywMAAHAUx3QwGGrUK8MwbKgEAADAOUY9rJWWliZObzY1Ncnn80mKHkmrr69PLFdf\nX6/y8vJTtsfQpgAAIJe5M72BgWFq9uzZqq2t1Z133qnt27ersrJSklRZWaktW7bo61//ut59912V\nlJQkTpeejGEYamrqOOVygCT5/cXsL0gL+wpGgv0F6fL7i0e8TkbD2po1a7R//361tbXpuuuu0913\n360777xTK1as0LZt2zR58mRt3LhRknTttddq7969mjt3rgoLC7Vu3bpMlgYAAJAVDCsHziPybQbp\n4tsv0sW+gpFgf0G6TufImmM6GAAAAGAwwhoAAICDEdYAAAAcjLAGAADgYFkf1hqPddtdAgAAQMZk\nfVi756dvOO7GuJ2dndq+/YURr7d27Up1dXVmoCIAAJCtsj6stXb0qq0zaHcZKTo6jmv79ucHzY9E\nIidd75//+VEVFY3LVFkAACALZXwEg9HQcKxbE4vz7S4j4Re/eEyffnpUt9/+93K5XPJ48lVcXKxP\nPvlYv/zlNt177z1qampUMNirxYu/rXnzFkiSFi++QU899ay6u7t1zz3LddllX9IHH7wnv79CP/7x\nT+TxeGx+ZwAAYLTlRFirP9ati86dOORrW1//s/54pPGMbu/LF5Vryezpw77+D/9wtz766H/09NNb\n9M47b2nt2lV69tmtCgQCkqSamgdUXFys3t5eLVt2q669drZKSkok9Q9cX1f3F/3wh+v0j/94n+6/\n/17t2fO6rr/+b8/o+wAAAM6XM2HNyS655NJEUJOkrVt/qX//972SpMbGRtXVfaJLLvmipP5r7yZN\nmqxp06KB8AtfuEj19Z+Oas0AAMAZcj6sLZk9/aRHwUZDQUFBYvqdd97S228f1KZNm+XxeHT33d9V\nMDj4mrvkU56m6RpyGQAAkPuyvoNBSZFH9S3OOrLm9XrV3R2taWBP1a6uThUXF8vj8ejjjz/Sf/zH\nB0O24bQergAAwB5Zf2TtLP84Hfn4mELhiNwuZ2TPkpLxuuyyGbrttm/J48mXz+dLvDZz5lX69a+3\n6ZZbluicc87VF794WdKa/desGYYhAAAAw8ryQzg//T/v6LUDn+ih/zVTk8uK7C4HDuf3F6upqcPu\nMpAF2FcwEuwvSJffXzzidZxxKOpzOMsfvS+Z0zsZAAAAnI7sD2vlhDUAAJC7sj+sxY+sOayTAQAA\nwJmQ9WEtUFok0zBU30pYAwAAuSfrw1qe21TZhAKOrAEAgJyU9WFNkgI+rzp7+tTZ02d3KQAAAGdU\nzoQ1KXs7Gcyde43dJQAAAIfKrbCWpadCuQEuAAAYTtaPYCD1h7UGh3QyePzxnykQmKTq6kWSpKef\n3iTDMPTuu++os7NDoVBIy5Z9T1/96rU2VwoAAJwuN8Ja6fBH1mr//LLeaTx0Rrd3RfllunH6N4d9\nfc6c67Vx408SYe3113dpw4af6aab/l5er1ft7W367neXEtYAAMAp5URYG1/kUYHH5Zhr1i644Atq\na2tTS0uzWltbVVJSotLSMm3cuF7vvfeuTNNQc3OTWluPaeJE36kbBAAAY1ZOhDXDMBTweVXX1KVI\nxJJp9l8DduP0b570KFimfO1rlfrd73appaVFlZXX69VXX1F7e7ueeWaLTNPU4sU3qLc3OOp1AQCA\n7JITHQyk6KnQUDii5uMn7C5FkjR79lzt3v2q9u59XV/7WqU6Ozs1caJPpmnq7bcPqr7+s8SylmXZ\nWCkAAHCy3AlrE53VI3Tq1PPV3d0tv79CPl+prr/+b3XkyJ90223f1s6dO3TuuVMTy9IbFAAADCcn\nToNK/Z0MGo51S9NKba4m6l//9bnE9PjxE/SLXzw95HKvvrp3tEoCAABZJneOrGX5jXEBAACGkjNh\nrWIiYQ0AAOSenAlr+R6XfCX5hDUAAJBTbAtrmzdv1je/+U3NmzdPa9asUTAYVF1dnZYsWaKqqiqt\nXr1aoVBoRG1WTPSqtaNXJ4IjWw8AAMCpbAlrDQ0NevbZZ1VbW6uXXnpJ4XBYv/nNb7R+/XotXbpU\nO3fuVHFxsV544YURtdvfyaAnE2UDAACMOtuOrEUiEfX09CgUCunEiRMqLy/X/v37VVVVJUmqrq7W\na6+9NqI26WQAAAByjS237qioqNDSpUt13XXXqbCwUFdffbUuueQSlZSUyDSj+TEQCKixsXFE7U4i\nrAEAgBxjS1g7fvy4du/erd/97ncqLi7WihUr9MYbbwxaLt2bxfr9xZKkS1wuSVJrVzAxDxiIfQPp\nYl/BSLC/IFNsCWt/+MMfNGXKFE2YMEGSNGfOHL3zzjs6fvy4IpGITNNUfX29ysvL02qvqakjOhGx\n5HaZ+vjT4/3zgCR+fzH7BtLCvoKRYH9Buk4n1NtyzdrkyZP13nvvqbe3V5Zlad++fbrgggs0c+ZM\n/fa3v5Ukbd++XZWVlSNq1zQNVfgKVd/azXibAAAgJ9gS1i6//HJVVVVpwYIFuuGGG2RZlpYsWaI1\na9bomWeeUVVVldrb27Vo0aIRtx3wedUbDKutM5iBygEAAEaXbWOD3nXXXbrrrrtS5k2ZMkXPP//8\n52o3uUfoxOL8z9UWAACA3XJmBIM4bt8BAABySe6GtRbCGgAAyH65F9bioxi0EtYAAED2y7mwVlSQ\np2JvHkfWAABATsi5sCZFT4U2tfeoLxSxuxQAAIDPJWfDmmVJjW0M6A4AALJbzoY1iU4GAAAg++V2\nWDvWZXMlAAAAn09uhrV4j9BjnAYFAADZLSfDmn9CoUzD4Ma4AAAg6+VkWHO7TPknFBDWAABA1svJ\nsCZFr1vr7OlTZ0+f3aUAAACctpwNaxX0CAUAADkgZ8NavJMBp0IBAEA2y9mwNslHWAMAANkvZ8Na\ngLAGAAByQM6GtZIijwrzXYQ1AACQ1XI2rBmGoYqJXjW2disSsewuBwAA4LTkbFiTop0MQmFLzcdP\n2F0KAADAacntsMbtOwAAQJYbG2GN69YAAECWIqwBAAA4WE6HtYqJ8dOgXTZXAgAAcHpyOqzle1zy\nleRzZA0AAGStnA5rUvRUaFtnUCeCIbtLAQAAGLExEdYkqeFYj82VAAAAjNyYCWufHeO6NQAAkH3G\nTFjjXmsAACAbjZ2wRicDAACQhXI+rPnGFyjPbXLNGgAAyEo5H9ZMw1DFxELVt3bLshjQHQAAZBfb\nwlpHR4eWL1+uv/u7v9M3vvENvffee2pvb9ftt9+uqqoq3XHHHero6Dgj2wr4vOoNhtXWGTwj7QEA\nAIwW28Laww8/rGuvvVavvPKKXnzxRZ1//vnatGmTZs2apZ07d2rmzJl64oknzsi2KnyMZAAAALKT\nLWGts7NTBw8e1MKFCyVJbrdbxcXF2r17t6qrqyVJ1dXV2rVr1xnZHp0MAABAtnLbsdG6ujpNnDhR\n9957r44cOaIvfvGLqqmpUUtLi8rKyiRJfr9fra2tZ2R7gdJ4WKOTAQAAyC62HFkLhUL605/+pJtv\nvlnbt29XYWGhNm3aJMMwMrK9SRxZAwAAWcqWI2uBQECBQECXXXaZJOn666/Xk08+qdLSUjU3N6us\nrExNTU3y+Xxptef3F59ymQnj8tXU3pPWssht7ANIF/sKRoL9BZliS1grKyvTpEmT9OGHH2rq1Kna\nt2+fpk+frunTp6u2tlZ33nmntm/frsrKyrTaa2o6da/R8gkF+q+j7fr0s3bluXP+jiUYht9fnNb+\nArCvYCTYX5Cu0wn1toQ1Sfr+97+ve+65R6FQSFOmTNG6desUDoe1cuVKbdu2TZMnT9bGjRvP2PYq\nfF79v7p2NbZ26yz/uDPWLgAAQCbZFtYuuugibdu2bdD8zZs3Z2R7yZ0MCGsAACBbjJnzgf237+Be\nawAAIHuMwbBGj1AAAJA9xkxY808olGkYhDUAAJBVxkxYc7tM+ScUqL6FsAYAALLHmAlrUvRUaNeJ\nkDq6GdAdAABkh7EV1mI9QhsYdgoAAGSJsRXWYp0MPqNHKAAAyBJjMqzRyQAAAGSLtMLajh071NnZ\nKUnauHGj7rjjDn3wwQcZLSwTEmGNTgYAACBLpBXWHn/8cY0bN07vv/++3nzzTS1YsEAPPfRQpms7\n40qKPCrMd3FkDQAAZI20wprbHR2V6ve//70WL16sefPmqbe3N6OFZYJhGAr4vGps7VEkYtldDgAA\nwCmlFdYMw9COHTu0Y8cOzZo1S5LU19eX0cIyJeDzKhyx1NxOj1AAAOB8aYW173//+3r55Ze1aNEi\nTZkyRR999JFmzpyZ6doygk4GAAAgm7jTWeiv/uqv9POf/zzx/LzzztMPfvCDjBWVSRVJnQwun2Zz\nMQAAAKeQ1pG1H//4x+ro6FAoFNLNN9+sL33pS3rxxRczXVtGcGQNAABkk7TC2h/+8AcVFxfrzTff\nVEVFhXbu3Kmnn34607VlRAVhDQAAZJER3RT3j3/8o+bOnauKigoZhpGpmjIqP8+l0pJ8whoAAMgK\naYW10tJSPfDAA3rllVd09dVXKxQKKRwOZ7q2jAn4vGrrDKqnN2R3KQAAACeVVlj7yU9+oqlTp2rD\nhg0aP3686uvrtXTp0kzXljEBX5EkqaGVo2sAAMDZ0gprPp9Pt9xyi4qKivTnP/9ZgUBAN954Y6Zr\ny5gKX6Ekhp0CAADOl9atOw4dOqTly5fL4/HIsiyFQiH97Gc/06WXXprp+jIiUEonAwAAkB3SCmsP\nP/ywHnnkkcToBfv27dOPfvQjPffccxktLlO4fQcAAMgWaZ0G7enpSQQ1Sfqbv/kb9fRk73BNvpIC\nedwmYQ0AADheWmGtsLBQ+/btSzw/cOCACgsLM1ZUppmGofKJXjUc65FlMaA7AABwrrROg9bU1GjF\nihXyeDySooO4//SnP81oYZkW8BWqrqlTrR298pUU2F0OAADAkNIKa5dffrleffVVffjhh7IsS+ef\nf77y8vIyXVtGJXcyIKwBAACnOmlYG3hd2pQpUyRJoVBIoVAoq0+FxjsZNBzr1iXn+WyuBgAAYGgn\nDWtXXHGFDMNIXNcVH2LKsiwZhqHDhw9nvsIMid8Y9zM6GQAAAAc7aVg7cuTIaNUx6gLxG+MS1gAA\ngIONaCD3XOItyFOJN49RDAAAgKON2bAmRa9ba2k/ob5Q9g5KDwAActvYDmulXlmSGluz9wa/AAAg\nt9ka1iKRiKqrq/W9731PklRXV6clS5aoqqpKq1evVigUyuj2450MuG4NAAA4la1h7d/+7d80bdq0\nxPP169dr6dKl2rlzp4qLi/XCCy9kdPuMEQoAAJzOtrBWX1+vvXv3avHixYl5+/btU1VVlSSpurpa\nr732WkZrqIj3CKWTAQAAcCjbwtojjzyitWvXJu7d1traqvHjx8s0oyUFAgE1NjZmtAb/hEK5TIMj\nawAAwLFsCWt79uxRWVmZLr744sQNdy3LGjSoejzIZYrbZapsQiFhDQAAOFZaY4OeaW+//bZef/11\n7d27V729verq6tIjjzyijo4ORSIRmaap+vp6lZeXp9We31982rWcGyjRgT/Vy1Po0fhx+afdDrLH\n59lfMLawr2Ak2F+QKbaEtdWrV2v16tWSpAMHDujpp5/W+vXrtXLlSv32t7/V17/+dW3fvl2VlZVp\ntdfU1HHatfjGeSRJ//Ffjbrg7Amn3Q6yg99f/Ln2F4wd7CsYCfYXpOt0Qr2j7rO2Zs0aPfPMM6qq\nqlJ7e7sWLVqU8W3SyQAAADiZLUfWkn3lK1/RV77yFUnSlClT9Pzzz4/q9rl9BwAAcDJHHVmzQ6CU\nG+MCAADnGvNhrcSbp8J8N2ENAAA40pgPa4ZhKODzqrG1R+FIxO5yAAAAUoz5sCZFr1sLRyw1t5+w\nuxQAAIAUhDVJAXqEAgAAhyKsiU4GAADAuQhr6r99RwNhDQAAOAxhTVLFxEIZ4sgaAABwHsKaJE+e\nS76SAn1GWAMAAA5DWIsJ+ArV3hlUT2/I7lIAAAASCGsxAR+dDAAAgPMQ1mICpXQyAAAAzkNYi2FA\ndwAA4ESEtRjCGgAAcCLCWszEknx53CajGAAAAEchrMWYhqHyiV7Vt3YrYll2lwMAACCJsJYiUOpV\nsC+ito5eu0sBAACQRFhLwXVrAADAaQhrSSYR1gAAgMMQ1pJUxMManQwAAIBDENaScBoUAAA4DWEt\nibfArZIiD2ENAAA4BmFtgIDPq5b2E+oLhe0uBQAAgLA2UMDnlSWpobXH7lIAAAAIawMF6GQAAAAc\nhLA2AJ0MAACAkxDWBgiUEtYAAIBzENYGKBtfIJdpqIGwBgAAHICwNoDbZco/oVD1x7plMaA7AACw\nGWFtCAGfV10nQuro6bO7FAAAMMYR1oZAj1AAAOAUhLUh0MkAAAA4hduOjdbX12vt2rVqbm6Wy+XS\n4sWLdeutt6q9vV2rVq3S0aNHdfbZZ+vRRx9VcXHxqNcXP7JGJwMAAGA3W46suVwu3XvvvdqxY4ee\ne+45bdmyRf/93/+tTZs2adasWdq5c6dmzpypJ554wo7yuNcaAABwDFvCmt/v18UXXyxJKioq0rRp\n09TQ0KDdu3erurpaklRdXa1du3bZUZ6KvXny5rsJawAAwHa2X7NWV1enI0eOaMaMGWppaVFZWZmk\naKBrbW21pSbDMFTh86qxtUfhSMSWGgAAACSbrlmL6+rq0vLly1VTU6OioiIZhnFa7fj9Z/66tvMm\nl+jDz44rYroU8I874+3DPpnYX5Cb2FcwEuwvyBTbwlooFNLy5cs1f/58zZkzR5JUWlqq5uZmlZWV\nqampST6fL622mpo6znh9E4o8kqQ//blJeeLmuLnC7y/OyP6C3MO+gpFgf0G6TifU23YatKamRtOn\nT9dtt92WmDd79mzV1tZKkrZv367Kykq7ytMkOhkAAAAHsOXI2ltvvaWXXnpJF154oRYsWCDDMLRq\n1SotW7ZMK1eu1LZt2zR58mRt3LjRjvIk0SMUAAA4gy1h7a//+q91+PDhIV/bvHnz6BYzjPKJhTLE\nKAYAAMBetvcGdSpPnku+kgKOrAEAAFsR1k4iUOpVe1dQPb0hu0sBAABjFGHtJLhuDQAA2I2wdhKE\nNQAAYDfC2kkkwhqdDAAAgE0IayfBkTUAAGA3wtpJTCzJl8dtEtYAAIBtCGsnYcYGdG9o7VbEYsgp\nAAAw+ghrpxDweRXsi6ito9fuUgAAwBhEWDuF+HVrn3EqFAAA2ICwdgr0CAUAAHYirJ1CoJQeoQAA\nwD6EtVOIH1lrIKwBAAAbENZOoTDfrfFFHo6sAQAAWxDW0hDwedXSfkLBvrDdpQAAgDGGsJaGCp9X\nlqTG1h67SwEAAGMMYS0NDDsFAADsQlhLAz1CAQCAXQhraZjEkTUAAGATwloayiYUyGUahDUAADDq\nCGtpcJmm/BMKVd/SLYsB3QEAwCgirKUp4POquzekju4+u0sBAABjCGEtTXQyAAAAdiCspYnbdwAA\nADsQ1tJEWAMAAHYgrKUpHtY+ru9QU1uPeoMMPQUAADLPbXcB2aLYm6dxhXk6/HGr/vEX/1eS5HGb\nKvZ6VOzNU0lR9LHY61GJN2m6KC/xPM/tsvldAACAbENYS5NhGPru/Ev1wf+0qKO7T8e7g+ro6lNH\nT1B1TV0K1Xecso0Cj2tQkIuGPY9KvHkpwW9cYZ7cLg58AgAw1mV9WPuso1En+iLyugtlGpkNN5ee\n59Ol5/kGzbcsSyeCYXV0B3W8u08d3cFooOuKPkbnBxMhr6X+hMKRU9+vrajArcJ8two8bhV4XMr3\nuFSQF3tMeR59vcDjUn6eKzbtVn7S83yPS6ZhZOJjAQAAGZT1YW3FjgckSYYMFboLVJTnlTfPq6I8\nr4rcRSrKK4xO5xWpyF0YfUxapsCVL+NzhhjDMFSYHw1W5RNPvbxlWeruDQ0Ic7GQ1xU7ahcLd929\nIR07fkIngmFFPucNeT15ZiLs5ee5VZCfFP7yogEvL8+Ux20qz20qz+1KmjblcbuGfd0Te+52GZ/7\n8wQAAP2yPqzNPv9qtXS0q7uvW1193erq61LriTaFrPQ6AJiGqSK3NzXk5XkT8+Lzx8UCn8f0yDRM\nuUxTphH94zJMmYZLpozEvJMFFsMwVFSQp6KCPE0qTe99WpalUDiiE8GweoNhnQiGdaIvaToYUm/K\n87B6+0JDLt/bF1ZnT49OBMM60wMyGFJ/uMtzKc9lJgU8V1Kwi/5xu0y5TCPx6HIZcplDzYvOd7sM\nueLzY6+5TTNlPZcraV2zf3m3y1BvX1ihcESmaXCkEQCQFQzLYeMnvfHGG3rkkUdkWZYWLlyoO++8\n85TrNDWlXi9mWZaCkT519XXFAlz/n+5Q96B5XaEudff1qKuvW5bOzMcRD239Yc6UqQHzzGjIG/b1\npOBnKHrEyow9GgMeTcMcZpnh5kcfLUsKR6RI2Io+RixFIkqaTn0eDseehy2Frdh6sXXD4UjsMTov\nFLYUiliKhC2FwtH50Z9PPCQZin7csedWUniyBs5LXXZwG/HpIdoYtJ3+adMwoj8HGTJNyTDM2Dwj\n9nMwUp6bhmI/N0Muw5BhxpYxzUQANI1ocIw/NwzFHg2ZRjSsR7eVOi++XGIdM/poyOifji0/sL3+\n5ZO3N1R7hgxpQHtJdSXqS319YJvJ7ZnJ7ZhDLKP+95s83+lh2e8vHvS7BRgO+wvS5fcXj3gdRx1Z\ni0Qi+tGPfqTNmzervLxcixYtUmVlpaZNmzaidgzDUL7Lo3yXR76CNM5LxrdvRXQidEKdw4W6vm4F\nI0FFrEjKn/Cg6egpy0hseuDrlhVRyAorEuobsE5/W44XvzwwzQ6urvQXtZUlKRz7c0baSwmSya8M\nDo6yJEWMYZYZZp3EdpKDaVLQTQm9/aHVGvB88HIDlx3wWkr7/fOs+LyUdoZfJv6fZMbCm6FodDZk\nyEwE2ujcaJiWlDQ/Kh4CDRmSEQvd6g+ysVf6A2NsWvHwmPS6EVvf43EpFIwkbSv+RUex2pSozYjV\nIMUDqZmoJ/5FSYltx79cKdFuPHzH60wO1kbsr/h04n0n2o/tDUb/tGEMmJ+0ruLTin82/c+VWGfg\n6/2fdeLzHLCshtz2gNcG1BNve1BNw9U3YL3ENoZoM/F/SLyWpJUTtah/Inl7/YsbSdNJiye1E1/f\n09mrzp6+oZcf0LaRtM3k+oykBpPfe/Kyifod/mUHZ5ajwtr777+vc889V2eddZYk6Rvf+IZ27949\n4rB2ukzDlDd22tNu/cHOkiVLlhWJPVqKxB4Tz5Nes2QpkvTa4PmR2DoDl4n0z5MlWVJEljREO9FH\nJdWlkywT6Z+2EnMSyyn2fhRvI6mGQW0OtZ2UtpRoO3k78fcQkaWC/Dz1nAjG1lP8XSh+gDm5diVN\nJ/5Oee9Jdcc/X2vgz0iSNXi9xPzEvOQ2+58n/k6pr79uWYmplPUjisQ+23D0uWX1v9fET9a5Xwos\nDci2dsnPcPvWgMe42LeFEYX9RJvGMMudYr346tZwIWCYGhIrnqqWIdpLqW2YdlOWGb69oes+xTaG\n+xwGtZX0fKhaTvJ5nrrGoQxX9xDbSdp2UqQesEwsLCZtPxH+rKTgOeT6hoZaIjl9pm5tQDAdoqX+\n3Dk4DA9YMnWZgetb8ZdT604Jv1ZqMcbgajV5XIX+99yvOToAOyqsNTQ0aNKkSYnnFRUVOnTokI0V\n2Sd+ChRnFqcqUg0V+vsfB8xLCvmR2BHiiOJHkGOBX5Gko8qR/vUUGbCcpUgkPKL1E6FcSoTdxHTi\nFSVC8IC5gwJv8peIlOnYOvn5bnX3BBOfT/z9x8Nv8heF1OfxNhNfd6LTsWCdaENWotb40fTkAN4f\n1BNVJ/3MYvMSryU9WqnP420OWm7AZ5X8+nCf8cAlU6ctDffZD9WCBr06zDaswa8N39bA7cSeOfff\n4Kww+BPOLccjboWta+U2nHv+x1FhzWGXzwE5L34aTobkyooT1aOHYJ+bUkK/lRoOh/sS0L/cgECb\nFILLSsepqSVpf7FS10ipIWX9pIWHWH6ofxdTj7DHHhNH2ZND8+Aj8/2fQdK7Tlp3uHaHk1rfcIF6\niGdW8itDt9//RWPgJ98/NfDzGfoL2oB1BhTh95bJbTr795+jwlogENCnn36aeN7Q0KDy8vJTrnc6\nF+th7GJ/QbrYVzAS4yYX2V0CcpSjzrNddtll+uSTT3T06FEFg0H95je/UWVlpd1lAQAA2MZRR9Zc\nLpd+8IMf6Pbbb5dlWVq0aNGodS4AAABwIsfdZw0AAAD9HHUaFAAAAKkIawAAAA5GWAMAAHAwR3Uw\nGKnTGUcUY9Ps2bM1btw4maYpt9utF154we6S4CA1NTXas2ePSktL9dJLL0mS2tvbtWrVKh09elRn\nn322Hn30URUXcyuPsW6ofeWxxx7T1q1bVVpaKklatWqVrrnmGjvLhEPU19dr7dq1am5ulsvl0uLF\ni3XrrbdnoWeSAAAFmklEQVSO+PdL1nYwiEQiqqqqShlHdMOGDfQexZAqKytVW1ur8ePH210KHOjg\nwYMqKirS2rVrE/8A/8u//IsmTJigZcuWadOmTTp+/LjuuecemyuF3YbaVx577DEVFRVp6dKlNlcH\np2lqalJzc7MuvvhidXV16cYbb9TPf/5z1dbWjuj3S9aeBk0eRzQvLy8xjigwFMuyFIk4dyxM2OvK\nK69USUlJyrzdu3erurpaklRdXa1du3bZURocZqh9RWIEHgzN7/fr4osvliQVFRVp2rRpamhoGPHv\nl6wNa0ONI9rY2GhjRXAywzB0xx13aOHChdq6davd5SALHDt2TGVlZZKiv3BbW1ttrghOtmXLFs2f\nP1/33XefOjoYpgyD1dXV6ciRI5oxY4ZaWlpG9Psla8Ma32IwEs8995xqa2v15JNPasuWLTp48KDd\nJQHIETfffLN27dqlF198UWVlZVq3bp3dJcFhurq6tHz5ctXU1KioqEiGYYxo/awNa6c7jijGJr/f\nL0ny+XyaO3euDh06ZHNFcLrS0lI1NzdLil534vP5bK4ITuXz+RL/+C5ZsoTfL0gRCoW0fPlyzZ8/\nX3PmzJE08t8vWRvWGEcU6erp6VFXV5ckqbu7W2+++aYuuOACm6uC0ww8Wj979mzV1tZKkrZv387v\nFyQM3FeampoS06+99pouvPDC0S4JDlZTU6Pp06frtttuS8wb6e+XrO0NKkVv3fHwww8nxhHl1h0Y\nyl/+8hfdddddMgxD4XBY8+bNY19BijVr1mj//v1qa2tTWVmZ7r77bs2ZM0crVqzQZ599psmTJ2vj\nxo1DXliOsWWofWX//v06fPiwTNPUWWedpQcffDBxPRLGtrfeeku33HKLLrzwQhmGIcMwtGrVKl1+\n+eVauXJl2r9fsjqsAQAA5LqsPQ0KAAAwFhDWAAAAHIywBgAA4GCENQAAAAcjrAEAADgYYQ0AAMDB\nCGsA8DkcOHBACxcutLsMADmMsAYAn9NIx/kDgJFw210AAGTK+++/r/Xr1yeGG1u+fLmmT5+uhQsX\nqrq6Wr///e8lSffff7+uvPJKSdKvf/1rPfXUUzJNU+ecc45++MMfJsbte+KJJ/Tyyy/LNE15vV79\n6le/khQd++/+++/Xu+++K9M0tWHDBp1//vk2vGMAOckCgBx0/Phxa8GCBVZTU5NlWZbV2NhoXXPN\nNdbhw4etL3zhC9aLL75oWZZlHThwwLrmmmusYDBo/ed//qf11a9+1WpubrYsy7IeffRRa+XKlZZl\nWVZtba110003WV1dXZZlWVZbW5tlWZa1f/9+69JLL7UOHz5sWZZlPf7449Y999wzqu8VQG7jyBqA\nnPT222+rrq5Oy5YtSwy87XK5FAqF5PF4dMMNN0iSvvzlL6ugoEAffvihDhw4oOuuu06lpaWSpG99\n61uaP3++JGnPnj369re/La/XK0kaP358YltTp07VRRddJEmaMWOG9uzZM1pvE8AYQFgDkLMuuugi\nPfvssynzjh49Omg5y7JkGEbiceD8+PRw8vPzE9PxQAgAZwodDADkpCuuuEIfffSR9u/fn5h36NAh\nWZalYDCol156SZJ08OBBBYNBTZ06VbNmzdLevXvV0tIiSdq6dauuuuoqSdLs2bP1q1/9KnH9W1tb\n2yi/IwBjFUfWAOSkkpISPf744/qnf/onrVu3TsFgUOecc47uu+8+TZgwQYcPH9aTTz4pSdqwYYPc\nbremT5+u1atX6zvf+Y5M09SUKVP04IMPSpIWLFigxsZG3XTTTXK5XBo3bpy2bNli51sEMEYY1smO\n7QNAjjl69KgWLlyoffv22V0KAKSF06AAxhzuiwYgm3BkDQAAwME4sgYAAOBghDUAAAAHI6wBAAA4\nGGENAADAwQhrAAAADkZYAwAAcLD/D2XS0bc/5tLyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x56181f90d890>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmgAAAD+CAYAAACOT/k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl4k9ed//23NkuyLNuSLXnFC2YxBhNICAESIJCFJGQt\nzsokaemEztLftNOnbSYzzTAz6dBJpj+ep5mZbtM2NFubNCFLMUmTQoITIJAAYTWb8YItW97kTfty\nP3/IMThAkIOFvHxf1+VLlnXf8tdwLH18zrnPUSmKoiCEEEIIIUYMdaILEEIIIYQQg0lAE0IIIYQY\nYSSgCSGEEEKMMBLQhBBCCCFGGAloQgghhBAjjAQ0IYQQQogRJu4Braqqiptuuolly5bxy1/+8qzH\nm5ubeeihh7jrrru444472Lp1a7xLEkIIIYQY0VTxXActEomwbNky1q9fj91up6KignXr1lFSUjJw\nzD//8z9TVlbGfffdR01NDY888ghbtmyJV0lCCCGEECNeXHvQ9u/fT2FhIXl5eeh0OpYvX87mzZsH\nHaNSqejr6wOgp6eHrKyseJYkhBBCCDHiaeP55E6nk5ycnIH7WVlZHDhwYNAx3/zmN1m1ahXPPfcc\nPp+PZ555Jp4lCSGEEEKMeHHtQYtl9LSyspIVK1awdetWfvGLX/C9730vniUJIYQQQox4cQ1o2dnZ\nOByOgftOpxO73T7omFdeeYWbb74ZgFmzZuH3++ns7PzC55XtQ4UQQggxlsV1iLO8vJyGhgaampqw\n2WxUVlaybt26Qcfk5uayfft27rrrLmpqaggEAlit1i98XpVKRVtbbzxLF2OIzWaW9iJiIm1FDIW0\nFxErm8085HPiGtA0Gg2PP/44q1atQlEUKioqKCkp4emnn6a8vJwlS5bw6KOP8oMf/ID169ejVqt5\n8skn41mSEEIIIcapiBLBF/Kj0+jQqeMagS5aXJfZiCf5q0XESv7KFbGStiKGQtrLpRcNWD68IR+e\nkA9vyIs35D39edDb/1j0NvrY6c99IT8KCun6NJ5Y8Bhq1aVZr3/E9aAJIYQQQpwpFAnhDnrxhDy4\ng9EPT9AzOGgNhKz+z/uDly/sG/L3M2gMGLUGrAYLRq0Bo9ZIUWrBJQtnX5YENCGEEEIMWUSJ4Al5\nBwLWZ2HLHfJ84dd8Yf+Qvs9noSrDaCFZa8SoNWLUGvo/Nww8btQZSf7sc230c4PWMOKD2PlIQBNC\nCCEEiqLQE+ilw9dJp6+rP1y5+2+9uEPuQcHLG/KhENssqSS1DpPORKYxg2RdMiZdMiatEZPOhEmX\nTLLWSLLOOChcGbVGDFr9qA1YF0sCmhBCCDFO+EI+Onwu2r0ddHg7afd1Rm+9nXT4XAQjwS88X6PS\nYNIlk6ZPJceUTUp/2BoIXbpkTNrPPjeRrDNi0iaj0+gu0U84dkhAE0IIIcaIcCSMy98VDVxnBbBO\n+oLuc56XrDWSbbKTYbCSabRiNVj6w9dnPVzR0KXXJKFSqS7xTzU+SUAbJn19fbz77tvcdVfFkM77\n/ve/zZo1P8RkSolTZUIIIcYKRVHoC7r7A1jHWQHM5e8mokTOOk+r0mA1WphgziPTmEGm0ToQxjIM\nVpJ1xgT8NOKLSEAbJr29Pbz22h/OCmiRSAS1+vzj50899f/FuzQhhBCjyOlhyGjo6hi4ddHu6yQQ\nDpzzvLSkVIpSC84KX5lGK2n61HE7l2u0koA2TH7+8//G4Whi1aqVaDQakpL0mM1mGhrqefHFV3ns\nse/S1tZKIODn7rvv57bb7gTg7rtv59e/fg6Px8N3v/t3lJfP4uDBfdhsWfzHf/xfkpKSEvyTCSGE\nGE7BSIhOn+us4PXZfXfQc87zDBoDNmMGmcYMMgyWQbdWg4Ukmec1pozJgPbylhN8fKR1WJ/zylI7\n9yyddN7H//qv/w91dSf5zW9eYO/e3Xz/+3/Pc8+9THZ2NgD/+I9rMJvN+P1+HnnkIRYvXkpqaipw\neiy/sfEU//qvP+LRR/+Jf/7nx3j//S3ceONNw/pzCCGEiK+IEqHL390/B+zMIBadiN/t7znn1Y+f\nDUMWmPPJMFrJNFjJMFrJMFjIMFoxaZNl/tc4MiYD2khQVjZ9IJwBvPzyi3zwwVYAWltbaWxsoKxs\nBpzxS5qTk0tJSTQETp1aSkuLAyGEECNPX8BNV0c7J5yNZ/WEuXxdhJXwWeeoUJGuT2NSejEZBisZ\nRkv/bXQYMjXJLMOQYsCYDGj3LJ30hb1dl4LBYBj4fO/e3ezZ8wm//OV6kpKS+D//5xsEAmfPIThz\nOFOt1pzzGCGEEJeOL+Sn2e3E4W6muc9Jk7sFR1/zea+GNOtSmGDOG+j1Ot0LZsVqSEej1lzin0CM\nVmMyoCVCcnIyHk903sDntzd1u/swm80kJSVRX1/HoUMHz/kco3RbVCGEGPXCkTBOTxsOdwvNfS39\nQayFDl/nWcdmGKwUpxVQYM0lWUkZ1BOm18i8YTE8JKANk9TUNMrLL+Phh+8jKUmP1WodeOyqqxbw\n+uuv8hd/cQ8FBYXMmFF+xpmn5xPI3AIhhIgvRVHo9HXhcDfj6GvB0R/EnJ62s4YlU3QmplgmkWfK\nJicli1xTDjmmLAxaPSCbpY9WwVAElQq0mpE9nKxSRmm3jfxSiFjJi6iIlbSVsaUv6I6GsDOCWLO7\n5ay9IJPUOnJSssk1ZZN7xm1qkvkLn1/ay8iiKAoefwhXr5+uXj+d/beuPj+u3tMffd4gGal6nvzr\nBagvUceIzfbFbelcpAdNCCHEiKcoCiElTCgSOuMjTEiJ3gYjAZzutoEg5nC30BMYHJ7UKjX2ZBu5\npmhv2GdhLMNokcn5I1w4EqG7L4Crrz90ffbxufuB0NmL9H5Gr9OQbtaTbzNRWmC5ZOHsy5KAJoQQ\nYth4Q16a+lpo6mvGE/T2B6hzharT94OREOHPh69z3B8Kiz6d6Rmlg3rFskx2dGp52xtpQuEI7d0+\nXD2+gd6urt4Anb0+uvrvd7sDfNF4X2qyjuyMZCwpeizm6Ed6/230awaMes2omkokLVUIIcSQKYqC\ny99FY6+Dxj4HjX3NNPY6zjmpPhZalQatWnv6Q6XBoNOf8TUNWlX/7cAxp+/r1DoyjJb+nrEsjFrZ\numikiUQUWru8NLX10dTmpqk9+uHs9BCOnDt9aTUq0lP0TMpLi4auFD3Wz4WvtBQ9Ou3Y6wGVgCaE\nEOILhSIhWtyt/UHMQVNvM419Djwh76DjUnQmSi2TyTfnkpeSgzkppT9EaftD1LnDlUY1uno2xBeL\nKAod3b7+ENZHU7sbR5sbR4eHUHjwEKQhSUNRtpnsjGSsZsNA79dnPWBmo27ctg0JaEIIIQZ4gh4a\n+5pp6u8Ra+xz0Ox2DrrCUYUKW3IGpdbJ5KfkDgSytKTUcftmOh4pioKr14+j3U1jfxhztLtxtHvw\nBwcPSSdp1eTZTORnmsi1mcjLTCEv04Q1VS9t5jwkoAkhxDgUXW7CFe0V6+0fouxz0OlzDTpOp9aR\nb86NBrH+MJZryh5YakKMDz3uAE1tfTS2u3G0uweGKL3+0KDjtBoV2VYTeTYTeZmnbzPTjKjVEsSG\nIu4BraqqirVr16IoCitWrGD16tWDHv/Rj37Ezp07UalUeDweXC4Xu3btindZCXfDDYt4992qRJch\nhBgHAuEALe7WaK/YZ8OUfc14Q75Bx5l1KUyzThkIYvkpudiTM+UKx3FCURR6PEFaOqLDkU1tfQO9\nY33e4KBj1SoVWVYj04ss5NlSBsKY3WJEo5b2MhziGtAikQhPPPEE69evx263U1FRwXXXXUdJScnA\nMY899tjA588//zzV1dXxLGnEkC5dIcRw84V8tHhaaXa30uJ20uJ20uxupdPnGrQ5twoV9mQbZdap\n/cOT0TCWph/6Wk1i9AlHIrR3+Wju8NDc4Y7edrpp6fDg9g3uEVMBtnQjk/PTyO0PYfmZKWRZk8fk\nxPyRJK4Bbf/+/RQWFpKXlwfA8uXL2bx586CAdqaNGzfyrW99K54lxc3PfvZfZGfncNddFQD85je/\nRKVS8emne+nr6yUUCvHII3/FNdcsTnClQojRzhP09AcxJy3u07cuf9dZx5qTUpicPpFsUxa5KVnk\np+SRm5ItWxKNA75AiJZOD83t0QAWDWSec141qVGrsFuMTJmQTk6GiZyMZPJsJnIyTOh1sn9oIsQ1\noDmdTnJycgbuZ2VlceDAgXMe63A4aGpqYt68eRf9fTec2Mje1nN/ny9rtr2cr0y69byPX3/9jfzk\nJ/93IKBt2fJn1q37L+69dyXJycl0d3fxjW98TQKaECJmfQF3NHx5nIN6xboDZ69en65Po9QymRxT\nFtkmO9n9tyk6UwIqF5eKoih09QVo6XDT3OkZ1Cvm6vWfdbxRr6Egy0xORnL/RzSM2dKNI37ro/Em\nrgFtKLtIVVZWsmzZslE79Dd58lS6urro6GjH5XKRmppKRkYmP/nJj9m371PUahXt7W24XJ1YLNYL\nP6EQYlxQFIWeQF90ONIT7QmLDk066Qu6zzrearBQljGVnOQssk1Z5JjsZJvssu7XGBcKR2jr8g4e\nluzw0NLpxus/exFfi1lPWZFlIIDlWJPJyTSRZkoate+z401cA1p2djYOh2PgvtPpxG63n/PYTZs2\nsWbNmpif+4v2tfqG7X7g/pifa7gsX34zH3/8Ie3t7dx55+3s2PEefr+HjRvfRK1Ws3TpUlJSdNhs\nZlSqL7c3l/hy5N9axOpStJWmnhbeq93B0fYaGnuacQc8gx5XocKekslU20TyU3OiH2k55JmzMOgM\nca9PxG442ouiKLi9Qdq6vNEPl5f2/tu2Lg/tXV7au31EPjcsqdWoyMlMYUJWCvl2M/n2FCbYzeTa\nTCQbdBddl0isuAa08vJyGhoaaGpqwmazUVlZybp168467uTJk/T09DBr1qyYn3skblA7b95innrq\nh3R3d/Pf//1LNm9+F6PRTEeHmz17PsHhcNDR4Uan6yUSUUbkzzAWyYbGIlbxbCuBcIC9rQfY5thF\nTXctEN0b0mbMYFLaRHKSPxuWzCIr2UaS5nNvsGHo7QrSS/Aczy4SIdb2EgyF6ez109njp7PHR0eP\nb+Dzzl4/HT0+/IFzb2WlUkF6ip6JOalkWwcPS2amG855xaS714e713eOZxOJMuI2S9doNDz++OOs\nWrUKRVGoqKigpKSEp59+mvLycpYsWQJEe8+WL18ez1IuieLiiXg8Hmy2LKzWDG688SYeffQ7PPzw\n/ZSWTqOwsHjgWOliFmJ8ONXrYLtjJx879w4sa1FqmcyC3LnMzCxD9/kgJkaViKLQ4w6cM3x19Aew\nHnfgvOebDFrs6UYyUg1YUvVkpBqwmvVYUw1kpBpINyfJshXjlEoZykSxEUR6RESspAdNxGq42oo3\n5OMT56dsd+ykobcJgLSkVObnXsn8nCvJNMo81NHE4wvS2uWl1RUdgnT2D0F2uQO0ubxfsI+kGuvn\nQ1fa6c+tqXoMSbJe/Hgw4nrQhBBivFAUhdqeerY5drHHuY9AJIhapaY8s4yrc+dSZp2KRi3LFYxE\nSn8v2Gch7Mwg1tblPWuRVoiuD2ZJNVCYbY6GLXN/EOsPXhmpBszJ43cfSXHxJKAJIcRF6Au42dWy\nm23NH9PidgKQYbCyIHcu83KuIF2fluAKBUQXZ+3s8dPaP/m+1eUd1Cv2+b0jIbo2WGa6kYm5qdjS\njdgtRuz9t5lpRnJz0qR3XsSNBDQhhBiiiBLhmKuG7Y5d7Gs7SEgJo1VpuMJ+GQty5zLFUiLbIyVA\nMBSmrcs3EL7aXF6cXZ7oVZHdvnMORep1mkHBy3bG51azQfaPFAkjAU0IIWLU5e/mo+bd7HDsot3X\nCUB2sp2rc+cyN/sKUpJkUdhLQVEU2rt9NDj7ONXaO3Db2ePnXLPBUow6CrPNA0HMlm4ky5KMzWIk\nVYYhxQglAU0IIb5AOBLmcOdRtjl2cajjCBElgk6tY172HK7Om0txaqG8wcdRKBzB0e6mwdlHw0AY\n68PrH7xnZKopiSkT0qMhzGLEbkkeCGPJBnmrE6OPtFohhDiHdm8nOxy72NH8Cd2BHgAKzHksyJ3L\nnKxZsnJ/HHh8QU619g2EsVPOPpra3YOGJlVAljWZ8olWCrLMFNhTmGBPIS1Fn7jChYgDCWhCCNEv\nGA6y27mP7Y5dHHEdB8CgMbAwbz5X585lgjkvwRWODYqi4Or1U++MhrCG1j4anL20dw9eXFWnVUdD\nWFbKQBjLt6WgT5KrYcXYJwFNCDGuKIqCN+Sl09eFy9+Fy9dFp6+LTp+LY9019Pr7AChJK2JB7lwu\nt88kSZOU4KpHr1A4Qkunh1POvmgg6w9jbt/gIcoUo47pRRYmfBbI7GayrEZZpFWMWxLQhBBjSjAS\nosvXPSh8ufyu/ttuXD4X/vC5V3Y361NYOmEhV+fOJduUdYkrH/38gTCn2qIBrMHZS72zj6Y2N6Fw\nZNBxdouRaYX9Ycwe7R1LT5FNvIU4kwQ0IcSooSgKfUE3nT5XNHz1h7AzP+8JnH9dqmStkUxjBhZ9\nOlZDOhZDOlZ9OumG6P3Jefl0dLgv4U80erl9QRpaoiEsGsZ6aen0cObeNFqNijxbykAIm9A/X8yo\nl7ceIS5EfkuEECNSQ28j+9sOnxG+XLj83YQioXMer1VpSDekMzl9IlaD5azwZdGnYdAavvB7qmU4\n7SyKotDVFxjUK3au+WKGJA2T89MpyEqhMMtMQZaZnIxktBr5NxXiy5CAJoQYUfzhABtP/on3Tn2I\ncsaqVuakFHJN2efs+bLoLZiTTLI47EVSFIW2Li8N/fPF6p3RZS0+v9m3OVnHjOLoVZSF2dE5Y7Z0\nI2oZohRi2EhAE0KMGEc6j/PikVfp8HViM2ZwZ8kt5KbkYNGnodPoEl3emBKORGju8ER7xVr65421\n9uL1D97yKCPVwOzJmdFesWwzhTJfTIhLQgKaECLh3EEPG45v5KOWT1Cr1NxQcC23FN9AkoSyYRFR\nFBpb+zjp6BkYpmxs6yMYOj15XwVkZyQzs8TcP0QZnTeWYpT/AyESQQKaECJhFEVhb9sBXj72Or2B\nPvJTclk5rYICc36iSxv1ut0BDtV2cLC2k8O1nfR4ggOPadQq8mymgblihVlm8u0mDEnyliDESCG/\njUKIhOjyd/PS0dfZ334InVrLHSU3c92ERWjUsgjplxEKRzjR2M3B2k4O1nbQ4OwbeCwtJYmrZ2Qz\nZUI6hdlmcjNNMnlfiBFOApoQ4pKKKBG2O3bx2olN+MI+JqdP5IHSFdiTbYkubdRxujwcPNnJodpO\nqhtc+APR+WNajYpphRZmTLQyoziDfJtJ5owJMcpIQBNCXDKtnjZePPIqx7tOYtAYuH/qV1iQO1eu\nvoyR1x/iSL1roJesrev0UhfZ1mRmFFuZMdHK1AkW2Q5JiFFOApoQIu7CkTCbG6qorHuXUCTEzMzp\n3Dv1TtL1aYkubUSLKAoNzl4OnuzkYG0nNU3dAxuHG/UarphiY/pEKzOKrGSmy+btQowlEtCEEHHV\n0NvIi9WvcKrPgVmXwj1ldzLbVi5DbufR3efnYG102PJQXSe9/ZP7VUBRjpkZxRnMmGhlYm6q7FMp\nxBgW94BWVVXF2rVrURSFFStWsHr16rOO2bRpE//zP/+DWq1m6tSp/PjHP453WUKIOAuEg2yqfZfN\np6qIKBHmZc/hK5NvxaRLTnRpI0ooHOF4YzcHazs4eLKTU62fm9xfns2M4gzKiiyYk2XTdiHGi7gG\ntEgkwhNPPMH69eux2+1UVFRw3XXXUVJSMnBMfX09v/rVr3jppZdISUmhs7MzniUJIS6BY64aXjzy\nCm3eDjIMVh4oXUGpdXKiyxoRIoqCo81NdYOLw7WdHGnowh88Pbm/rMgS7SUrtpInk/uFGLfiGtD2\n799PYWEheXl5ACxfvpzNmzcPCmgvv/wyDzzwACkpKQBYrdZ4liSEiCNP0MvrNZVsc+xChYqlExZy\n68Rl6DXjt+dHURSaOzwcaXBxpN7FkYYu+ryn1yTLyUhmenH0asupBenodTK5XwgR54DmdDrJyckZ\nuJ+VlcWBAwcGHVNXVwfA/fffj6Io/O3f/i0LFy6MZ1lCiDjY13aQl46+Rnegl1xTNiunVVCUWpDo\nsi45RVFwurwDgexoQxfdZ+xlaTHrmT89m9LCdKYVWshMk8n9QoizxTWgKYpywWPC4TANDQ288MIL\nOBwOVq5cSWVl5UCPmhBiZOv29/KHY6+zt+0AWpWGW4uXcUPhYrTq8XENkqIotHX7+nvHooHM1esf\neDzNlMRVZVmUFqRTWmjBnm6UYUshxAXF9RU0Ozsbh8MxcN/pdGK32wcdk5WVxezZs1Gr1eTn51Nc\nXExdXR0zZsz4wue22cxxqVmMTdJehp+iKLxfu4Nn972KO+BhamYJ37hyJfmpORc+eQSLpa20ujwc\nONHO/hPtHKhpp83lHXgsLSWJqy/LZeakTMpLMsm3p0ggG8PktUXES1wDWnl5OQ0NDTQ1NWGz2ais\nrGTdunWDjrn++uuprKzkzjvvpLOzk/r6eiZMmHDB525r641X2WKMsdnM0l6GWbu3gxePvMpR1wn0\nmiTumXInC/PmofarR/W/9fnaiqvXf8YcMtegBWJNBi1XTLFRWmihtCCd3MzBE/vb2/vOej4xNshr\ni4jVlwnycQ1oGo2Gxx9/nFWrVqEoChUVFZSUlPD0009TXl7OkiVLWLhwIdu2bWP58uVoNBq+//3v\nk5Ymi1cKMRKFI2Hea/yQjSffIRgJMj2jlPum3oXVYEl0acOq2x3gaH8gq27owtnpGXjMqNcya1Lm\nQCDLt6eglh4yIcQwUymxTBQbgeSvFhEr+St3eDT1NfN89R9o6G0kRWfi7sm3c0XWrDExfBdRFA7X\ndXLkVDd7j7bS3HE6kBmSNEyZkE5pgYXSwnQK7GbU6tH/M4uLJ68tIlYjrgdNCDH6hSNh/tywlcra\ndwkrYa7MupyKybeRkmRKdGkXrc8b5MP9zby/t4nWrug8Mr1Ow4xiK6WFFqYWpFOUbZYV+4UQl5wE\nNCHEeTndrTxb/TJ1PQ2kJpl5oHQF5ZlliS7rotW19LBldxM7q50EQxF0WjXXlOdw66ISrMlatBoJ\nZEKIxJKAJoQ4S0SJ8P6pD3nz5NsEIyHmZM3inil3juptmoKhMLuqW9myp4na5h4A7OlGrp2dxzUz\nc0gx6mTISggxYkhAE0IM0u7t4LnqlznRVUuKzsTDZfcz216e6LK+tNYuL+/vbeLD/c30eYOogFmT\nMllyeR7Ti60ywV8IMSJJQBNCANF1zT50fMSGE5UEwgFm2WZw39SvYE4afYtGRyIKB0528N7eJg7U\ndKAAKUYdt8wr5NpZuWSmy+r9QoiRTQKaEAKXr4vnq//AEddxjFojD5fdx5VZs0fdFZq9ngAf7m/m\nvb1NtHdH1yoryUtl6ex85pTa0WllbpkQYnSQgCbEOKYoCh+17OaVY2/iC/uYnlHKA6UrSNePnrUI\nFUWhtrmXLXsa2VXdSigcIUmrZtFlOSyZnU9htqz0LoQYfSSgCTFOdft7ePHIqxzsqMag0bOytIL5\nOVeOml4zfzDMrsNOtuxtor4lOrE/y5rM0tl5XF2eTbJBl+AKhRDiy5OAJsQ4oygKu1v38fLR13GH\nPEyxTOIvSu8mwzg6dgNwujy8t6eJbQeacftCqFQwe3ImS6/IZ1qhRSb9CyHGBAloQowjvYE+Xjr6\nGnvbDpCk1p3eQ1M1sudmRSIK+2raeW9PEwdrOwFINSVx64Iirp2VizXVkOAKhRBieElAE2Kc2Nd2\nkN8d2UBvsI+JaUU8OO0e7MmZiS7rC/V4Anywz8H7e5vo6PEDMDk/jaWX53PFVJssKCuEGLMkoAkx\nxnmCHl4+9iYfO/egVWu5a9Jylk5YOKJ7zfq8QV6rOskH+x2Ewgp6nYZrZ+exZHYeE+yjb9kPIYQY\nKgloQoxhhzqO8kL1H+gO9FBgzuehsnvJMWUluqzzikQUtu5zsGFrDW5fiCyLkevnTGD+9GySDfJy\nJYQYP+QVT4gxyBfyseHERrY5dqFRabi1eBk3Fl6LRq1JdGnndaKxm+ffPUqDsw9Dkob7lk5i6RX5\nMowphBiXJKAJMcYcc53g+eo/0OFzkZeSw4PT7mWCOTfRZZ1XV5+fP7xXw45DLQBcPSObimtLSEvR\nJ7gyIYRIHAloQowRgXCA12veYmvjNtQqNTcVLuXm4uvRqkfmr3koHGHz7kbe+LAWXyBMYZaZlTdO\nYVLe6FkkVwgh4mVkvnILIYbkZHcdzx1+mVZvO1nJdh4qu4ei1IJEl3Veh+o6efHdYzR3eDAZtDy0\nbCqLLstFrZY1zIQQAmIMaJWVlSxbtgytVvKcECNJMBxkY+07bG6oAmDphIXcNvEmkjQjcxX99m4v\nL205we6jbahUsGR2HnctmkiKcWTWK4QQiRJT4tq4cSNPPvkkK1as4L777iMra+ReBSbEWBSOhOkJ\n9NId6KHb30u3v4eeQA972w7S4naSabDyYNm9TEovTnSp5xQMhXlrZwObdtQTCEWYlJ/GyuunyD6Z\nQghxHjEFtJ/97Gc0NTXx+9//nhUrVnD55ZfzwAMPMG/evHjXJ8SYNjh49YevQA89/h66+r/W4++l\nL+hGQTnncyzKm88dJbdg0I68SfWKovDpiXZ+9+fjtHf7SDMl8fBNk5g3PWvU7PkphBCJoFIU5dyv\n+uexZ88evvOd79Dd3U1+fj5r1qxhzpw55z2+qqqKtWvXoigKK1asYPXq1YMef+2113jqqafIzs4G\nYOXKlVRUVFywjra23qGULcYxm818ydvLZ8Grq7+nq9vfQ3egtz+E9QwEMnfQc97gBaDXJJGmTyUt\nKZU0fSqpSeZB9zONVqyGkbmHZkunhxf/fIyDJzvRqFXcMGcCt11dhFE/cqdKJKKtiNFL2ouIlc02\n9NGCmF4pA4EAmzZt4ne/+x3hcJhvf/vb3HLLLezfv5/vf//7bNmy5ZznRSIRnnjiCdavX4/dbqei\nooLrrrsmjRE7AAAgAElEQVSOkpKSQcctX76cH/zgB0MuXoiRIBwJc7K7ngPthznmOoHL301f0P2F\n5xg0elL1ZnJMWQOBK1VvJv2zIKZPJS3JjEE7+vaY9AVC/HF7He/sOkU4ojC9yML9108hN9OU6NKE\nEGLUiCmgLV26lKuuuop/+Id/YPbs2QNfnzNnDvPnzz/vefv376ewsJC8vDwgGsQ2b958VkAbYiee\nEAnnC/k50nmM/e2HOdhRjTvoAUCn1mI1WMg1ZZOqH9zbldbf+5WalDoihyMvlqIo7Kpu5eX3TuDq\n9ZORauC+6yZz+ZRMGc4UQoghiimgbdiwAbvdfs7H/v3f//285zmdTnJycgbuZ2VlceDAgbOOe+ed\nd/jkk08oKiriscceGxjuFGIk6fb3cLC9mv3thzjiOkEoEgIgLSmVa/LmMTNzOlMsJehG6Lpj8XSq\ntY8X3z3G0VNdaDVqbr+6iJvnFaLXjdydC4QQYiSL6Z3k9ddf55577iE9PR0Al8vFq6++yl/+5V9+\n4Xmx9IwtXbqUW2+9FZ1Ox+9//3seffRRfvvb38ZSlhBxpSgKzW4nB9oPs7/9MHU9DQOP5ZqymWmb\nzszMMiaY80b0xuPx5PEFee2DWrbsaURRYPbkTO69bjL2dGOiSxNCiFEt5nXQzpzcb7FY2Lhx4wUD\nWnZ2Ng6HY+C+0+k8qycuLe30quH33HMPP/7xj2Mq/MtMuBPjV6ztJRwJc7S9hk+a9vOxYz/OvjYA\n1Co10+1TuDLvMubkzsSekhnPcke8SEThzx838Oymw3T3BcizmXjkznKuKB39S/DIa4sYCmkvIl5i\nCmjn6gkLh8MXPK+8vJyGhgaampqw2WxUVlaybt26Qce0tbVhs9kA2Lx5M5MmTYqlJLlyRsTsQlda\n+UJ+qjuPceBz88n0miRm22cyM7OM6RmlmHTJ0RO80OYdv+3vpKOHF949Sm1zL3qdhruvLeGGKyeg\n1ahH/e+lXJUnhkLai4hV3K7iLCoq4plnnuGrX/0qiqKwfv16CgouvI2MRqPh8ccfZ9WqVSiKQkVF\nBSUlJTz99NOUl5ezZMkSnnvuObZs2YJWqyUtLY0f/ehHQ/4hhBiqbn/PwNDl0TPmk6Xr01iYN5/y\nzLJxO5/sfLr7/GyoOskH+5sBmFeWxd1LJmExj70LHoQQItFiWgfN6XTyve99j71796JSqZg9ezZP\nPfVUQncUkL9aRKxsNjOtrT00u53sbz/M/vZD1PecGng8LyWH8swyZmaWUWDOlysOP6e9y8tbuxr4\nYF8zoXCEfFsKK2+YzNSCkbn+2sWQHhExFNJeRKy+TA/akBaq9XiiQz/JyclD/kbDTX4pxIVElAg1\nXXUcdx9nZ8Ne2n2dQHQ+2aT0iczMLKM8s4xMozXBlY5MjnY3mz6q56NDTiKKQmaagVvmF7JwZg4a\n9di8KELecMVQSHsRsYrbECdAb28vtbW1+P3+ga9deeWVQ/6GQsSbJ+hlZ8tuqpq20+ppBwbPJ5uR\nUUqyLvF/ZIxUtc09bNpRz55jbShAXqaJW+YXMneafcwGMyGEGGliCmibNm3iySefpKenB7vdTkND\nA6Wlpbz22mvxrk+ImDX2Oqhq2sHHLXsIRIJo1Vquyr6C66YswK7OkflkX0BRFI6d6mLjjnoO1UZ7\nGotzzNw6v4jLJmeilmFfIYS4pGJ6x/r5z3/Ohg0b+PrXv87rr7/Otm3b+NOf/hTv2oS4oFAkxKet\nB9jatIOT3XUAZBgsLMybz/ycK0lJMskwxBdQFIX9NR1U7qjnRFM3ANMKLSyfX8i0QovMxxNCiASJ\nKaBptVoyMjIGlta4+uqrY16vTIh4cPm6+NCxk22OnfQG+gAos05lUf58pmeUjtuFY2MViSh8crSV\nyh31nGqN/vvNmpTJ8vmFlOSlXeBsIYQQ8RZTQEtKSkJRFAoLC3nuuefIy8sbuGBAiEtFURSOuk7w\nQdMO9rcfJqJEMGqNLJ2wkIV587Enj+/FY2MRCkfYfrCFtz6qx+nyolJFl8u4ZV4h+faURJcnhBCi\nX0wB7Vvf+hZ9fX1897vf5V/+5V/o7e1lzZo18a5NCAC8IR87m3dT1bQDp6cVgAkpuSzKX8CcrFkk\naZISXOHI5w+Eqdrn4O1dDbh6/WjUKhZdlsvN8wrIssgFE0IIMdJcMKCFw2EaGhqYP38+ZrOZ9evX\nX4KyhABHXwtbm7azq2UPgXAArUrDlVmzWZS/gOLUApkfFQOPL8jmPU28+/Ep+rxBknRqbrxyAsvm\nFsgCs0IIMYJdMKBpNBpeeukl7r333ktRjxjnwpEwn7YdpKppOye6agGw6NO5qXApC3LnYk6SYbhY\n9LgDvPvJKbbsacTrD5Os13LbgiKun5OPOVl6HIUQYqSLaYjzqquu4u233+amm26Kdz1inOryd7Ot\nKTrpvzsQveKy1DKZRfkLmJFRikatSXCFo0NHt4+3dzVQtc9BMBQh1ZTErfOLuHZ2Hka9LDMihBCj\nRUyv2K+99hrPPPMMBoMBo9GIoiioVCp27NgR7/rEGKYoCse7TlLVuJ197Yf6J/0bWJJ/DQvz5pFl\nsie6xFGjucPNWx81sONQC+GIQkaqgZvnFXBNeQ5JOgm3Qggx2sQU0F599dV41yHGEV/Ix66WPVQ1\n7aDZ7QSi+2EuypvPldmXo5dJ/zGrb+ml8qN6dh9pRQFyMpK5ZV4hV5VlodXIUiNCCDFaxRTQ8vLy\n4l2HGOO6/T3UdNdxtPM4nzg/xRf2o1FpuMJ+GYvyF1CSViST/mMUDIXZfbSNrZ86OHqqC4DCbDO3\nzi9k9hSbrPovhBBjQEwBbd68eed885QhTnEuESWC09PGya46arrrqOmqHdioHCBdn8b1BYtZkHsV\nafqhbyA7Xjna3VTtc7DtQDNuXwiAsiILN11VwPQiqwRcIYQYQ4Y8xOn3+/njH/+IVisTjkVUMByk\nvreRk9111HTVUdtdjzt0eiFjo9bA9IxSJqYVUZJWyMS0Ipn0H6NAMMwnR1up+tTBscboVkypyTpu\nmVfIostysMsaZkIIMSapFEVRvsyJ99xzDy+//PJw1xMz2VsxcfqCbmq766np7yFr6DlFSAkPPJ5h\nsDAxrZiS9EJK0orJNtkTuvXSaNyLs6mtj62fOth+sAWPP9pbNr3YyuLLcpk1OVPml8XJaGwrInGk\nvYhY2WxDHy36Ut1gp06doqOj48ucKkYZRVFo93ZS011LTVcdJ7vraOlfzR9AhYp8cy4laUXRHrL0\nItL1spfjl+EPhvnkSCtbP3UMbFyeZkpi+eWFLLwsF3u6McEVCiGEuFSGPActEokQCoX4p3/6p7gW\nJhIjHAnT2OegpquWmu56arprBzYjB0jSJFFqmczE9CJK0oooSp2AQWtIYMWjX2Nrf2/ZoRa8/hAq\nYMZEK4svy+OySRnSWyaEEOPQkOegabVaMjMz0WhkDtFY4A35qO2uH5g/VtfTQCASHHg8LSmVy+0z\nB3rH8kw5Mn9sGPgDYXYdcVL1qYMaRw8AaSlJXHdFEYtm5pApvWVCCDGuxRTQ3G43+fn5JCdHJyR7\nPB6ampqYPHlyXIsT8dPu7WBzQxU7mj8mGInOcVKhIseUNdA7VpJWhNVgkasDh1GDs5et+xx8dKgF\nrz+MCphZksHiy3KZOSkDjVp6y4QQQsQY0P7hH/6Bl1566fRJWi2PPvooGzZsuOC5VVVVrF27FkVR\nWLFiBatXrz7ncW+//Tbf/va3efXVV5k+fXqM5YuhOtXr4N3699jTuh8FhQyDhSuyZvXPISskWSdX\nBQ43XyDEruro3LLa5mhvmcWs54Y5E7hmZg6ZadJbJoQQYrCYAlo4HEan0w3cT0pKIhwOf8EZUZFI\nhCeeeIL169djt9upqKjguuuuo6SkZNBxbreb559/nlmzZg2xfBGL6JZKNbxT/z7VnceA6Mr9NxRc\ny+X2mTJkGSf1Lad7y3yBMCoVXFaSweJZeZSXWKW3TAghxHnFFNC0Wi2nTp1iwoQJADQ0NMQ0B23/\n/v0UFhYO7ESwfPlyNm/efFZA+8lPfsIjjzzCr371q6HWL75ARImwr+0Q79a/T33vKQAmp0/khsIl\nlFmnyNBlHHj9IXZVO9n6qYO6lujl9xaznmVzC1g4MwdrqlxQIYQQ4sJiCmjf/OY3uf/++1m8eDEA\nW7du5Yc//OEFz3M6neTk5Azcz8rK4sCBA4OOqa6upqWlhcWLF0tAGybBSIhdzbv586mttHraUaHi\nMtsMbii4luK0gkSXNyb1eYO8vbOBzXsa8ff3ls2alMniWbmUT8xArZYwLIQQInYxBbQlS5bw/PPP\ns23bNgBWr15NYWHhBc+70Bq4iqKwdu1annzyyZjPEefnDXn5sGkn7536gO5ALxqVhgU5V3J9wWKy\nTPZElzcmef0h3v34FH/6uAGvP0xaShI3X1XANeXSWyaEEOLLiymgdXZ2kpuby8qVKwEIBoN0dnZi\ntVq/8Lzs7GwcDsfAfafTid1+Oii43W5OnDjBgw8+GF0Qtb2dv/mbv+FnP/vZBS8U+DKr8o5VLm83\nm45t4Z2aKrxBH0atgdtLb+CWKUuxGtMTXd6IMNztxRcIsWlbLa9sOUGvJ0CqKYkHlpVy84Ji9DqZ\n0zeayWuLGAppLyJeYtrq6e677+bZZ5/FaIxebebxePjqV796wa2ewuEwN910E+vXr8dms3H33Xez\nbt26s+agfebBBx/kscceo6ys7IKFy/Ya0Opp488NW9nZvJuQEsaclMLS/IVckzePZJ1cGfiZ4dyO\nJRiKULXPwcbtdXS7Axj1Wm66qoDrr8jHqJf9aUc72bpHDIW0FxGruG31FAgEBsIZQHJyMn6//4Ln\naTQaHn/8cVatWoWiKFRUVFBSUsLTTz9NeXk5S5YsGXS8SqWSIc4Y1Pec4t369/m07SAKCpnGDK4v\nWMy87CvQaXQXfgIxZKFwhO0HW/jjtlo6evzodRpuXVDIsrkFmAzyby6EEGJ4xfwn/5lDmh0dHUQi\nkZjOW7RoEYsWLRr0tb/7u78757HPPvtsrOWMO4qicKTzOO80vM8x1wkACsx53FC4hFm2GQndjHws\ni0QUdlY7eePDWlpdXnRaNcvmTuDmeYWkJiclujwhhBBjVEwB7cEHH+T+++/njjvuQFEU3nzzTR55\n5JF41yaI7o35adsB3q1/n1N90fl8pZbJ3FB4LVMtk2SpjDhRFIU9x9p4/YNamtrdaNQqllyex63z\ni7CY9YkuTwghxBgXU0CrqKigoKCA999/H4B///d/Z86cOfGsa9wLhIN81PwJmxu20u7rRIWKy+0z\nuaHgWgpS8xNd3pilKAoHTnbyWtVJ6p29qFRwTXkOt19dJPtjCiGEuGRiCmi9vb188MEHHD9+HJ/P\nx8GDBwEZkowHT9BDVdMO3j+1jd5gH1q1lmvy5nHdhEXYkzMTXd6YdqTexYYPTnKisRuAudPs3HFN\nMTkZpgRXJoQQYryJKaD94z/+IyUlJdTV1fGtb31L9suMk2OuGn598Hn6gm6MWgM3Fi7h2vxrSNPL\nZdzxVNPUzYaqk1TXuwCYPTmTOxdOZII9JcGVCSGEGK9iCmj19fX813/9F5s3b+bWW2/lxhtv5KGH\nHop3beOGoihsbdrOq8f/CMBtE29icf4CjFpZ6DSeGpy9vFZ1kn01HQDMKLZy16KJFOekJrgyIYQQ\n411MAS0pKXq1mk6no6uri7S0NDo7O+Na2HgRjIR4+ehrbG/+mBSdiUfKH2JSenGiyxrTHO1uXv+w\nlk+OtAIwJT+NuxZNZGqBJcGVCSGEEFExBbSioiK6urq47bbbuPfeezGbzTLEOQy6/b3874Fnqe2p\nZ4I5j9XlD2E1SEiIl9YuL298UMtHh1tQFCjOMXPXoolML7LK1bBCCCFGlJh2EjjTJ598Qm9vL4sW\nLUKjSdyWNqN99eb6nlP88sCzdPm7mZM1i5WlFSRpZF2teFDptKz/40E+3N9MOKKQbzNx18KJzJqc\nKcFMDCIrw4uhkPYiYhW3nQTOJMtrXLxdLXt44cgrhCNh7iy5hesLFktQiIMeT4CN2+vY+qmDYChC\nljWZO68p5sppdtTy7y2EEGIEk80DL6GIEuH1mk1sbqjCqDWwuvwhpmeUJrqsMScYCvPnTxrZuKMO\nrz+M3ZrMrfMKmT8jC41adlwQQggx8klAu0Q8QQ+/OfQi1Z3HyEq28Y2ZXyUr2ZbossaUiKKwq9rJ\nq++fpKPHh8mg5f7rJnP3jaV0udyJLk8IIYSImQS0S6DZ7eQX+9fT5u1gRkYpX51+P0atrEo/nI6d\n6uKlLcepbe5Fq1Fx09wCli8oxGTQodNKr5kQQojRRQJanB1oP8z6Q7/DF/ZzY+ESbpu4TDY2H0bO\nTg+vvF/D7mNtAFxZamfFtSXYZVsmIYQQo5gEtDhRFIU/1W9h48l30Kq1rJr+AFdkzUp0WWNGnzfI\nm9tqeW9PE+GIQkleKvcuncykvLRElyaEEEJcNAloceAPB3iu+mX2tu7Hok/nGzMfZoI5L9FljQnB\nUITNuxvZuL0Ojz9EZpqBu5dMYs5Um1wJK4QQYsyQgDbMOryd/OLAb2nqa6YkrZhHyh/EnCR7Ol4s\nRVH4+Egrr7xfQ3u3j2S9lnuXTmLp5fkyx0wIIcSYIwFtGB1z1fCrg8/hDnpYmDefism3oVXLP/HF\nOtHUzUtbjlPT1INGreKGORO47eoiUoy6RJcmhBBCxMWoTA8/3/UcxaaJTLNOHhFXQ35+s/P7pn6F\nhXnzElzV6Nfa5eWV92sG9sy8YqqNimtLyLIkJ7gyIYQQIr5GZUDbUrsd2I5apWZS+kTKM0qZkTkN\newLWFZPNzoef2xfkj9vq2Ly7kXBEoTgnlXuXTmLKhPRElyaEEEJcEkPei3MkONlZT9WJ3Rxsr6ah\nt3Hg61nJNmZkTGNG5jRK0orQqOO7V6hsdj68QuEI7+1p4s1ttbh9ITJSDVRcW3LRWzPJfnkiVtJW\nxFBIexGx+jJ7ccY9oFVVVbF27VoURWHFihWsXr160OO///3veeGFF9BoNJhMJv7t3/6NkpKSCz7v\nZ78U3f4eDnUc4WB7NdWu4wTCAQCMWgNl1qnMyJxGWcZUUnSmYf25ZLPz4aMoCnuOtfGH92todXkx\n6jXcOr+I6+fko9NefMiWF1ERK2krYiikvYhYjbiAFolEWLZsGevXr8dut1NRUcG6desGBTC3243J\nFA1PW7Zs4cUXX+RXv/rVBZ/7XL8UwXCQ410nOdhRzYH2ajp9LgBUqChOK6S8v3ctx5R1UUsynLnZ\n+R0lN8tm5xfhpKOHl7Yc53hjNxq1imtn5XH7NUWYk4cv7MqLqIiVtBUxFNJeRKy+TECL6xy0/fv3\nU1hYSF5edA2w5cuXs3nz5kEB7bNwBuDxeFBfxGbWOo2OsoyplGVM5e7Jd9DsdnKwo5qD7dWc7K7n\nZHcdb5x8C6vBwoyMaZRnTmNy+kR0mtiuBpTNzodPe5eXV7bWsKs6egHA7MmZVFxbQk7G8PZ0CiGE\nEKNRXAOa0+kkJydn4H5WVhYHDhw467gXXniB9evXEwqF+O1vfzss31ulUpGbkk1uSjY3Fi6hL+Dm\ncOdRDrZXc7jzKFVN26lq2k6SJolplsnMyJzG9IxS0vSp53w+2ex8eHh8QSp31PPuJ42EwhEKs83c\nu2QSpYUyd08IIYT4TFwDWqyjpytXrmTlypVUVlby05/+lP/4j/8Y9lpSkkzMzb6cudmXE46Eqemu\n42B7NQc7qtnXfoh97YcAKDDnMyNzGuUZ08g356JWqWWz82GgKAofHXbyuz8fp88bxJqqZ8WiEq6a\nnnVRFwAIIYQQY1FcA1p2djYOh2PgvtPpxG63n/f4W265hTVr1lzweXcdbmFuWfbF1ZY1i6unRPfG\nbO5tZY/jAHuaD3C49TgNvY1sqn0XiyGN8qxSPm7ahzfk485py7hvxu0XNQw7HrW6PPz0lX3sPtKK\nPknDgzdP447FJeh18b3K9kxfZvxfjE/SVsRQSHsR8RLXgFZeXk5DQwNNTU3YbDYqKytZt27doGPq\n6+spLCwE4L333qOoqOiCz/vD3+zkO/fMYnqxdVjq1GJkrnUuc61z8U71Ut15nIPt1RzqOEJV/U50\nat3AZucdHe5h+Z7jQURReH9vE394vwZ/IExZkYWHbyrFlm6kp8tzyeqQibwiVtJWxFBIexGxGnEX\nCWg0Gh5//HFWrVqFoihUVFRQUlLC008/TXl5OUuWLOH5559nx44d6HQ6UlNTefLJJy/8vGoV//vH\nQ/zrqrmkpeiHtWaj1sjl9plcbp9JRIlwqreJ1CQzFoMskjoUzR1u1r91hOON3STrtXztllKuKc+R\nq12FEEKIGIzKhWpf31rDr988yPQiC39/7yyZwzSChMIR/rSrgTc+rCMUjnDFFBsrb5xC+jAH6aGQ\nv3JFrKStiKGQ9iJiNeJ60OLljkUT+fhQM/trOnjro3qWzy9KdEkCqG/p5ZlN1TS09pFqSuIvbpjC\nnNLzzzkUQgghxLmNyoCmUqn4+vJp/MszH/NaVS1TJ1iYlJ+W6LLGrUAwzBvbavnTzlNEFIVrZuZw\n79JJmAyxrS8nhBBCiMFG7eWI5uQkVt9WhoLCL948SJ83mOiSxqWjDS7W/GYXb33UgDVVz/9z3yxW\n3TJNwpkQQghxEUZtQAOYWmDh9quL6ejxs/6tIzGvuyYuntcf4tk/HeXJF/fS6vJyw5wJPPH1q5he\nNDxX1gohhBDj2agc4jzTbQuKOFLvYs+xNrbsaeK6K/ITXdKY9+mJdp7701FcvX5yM0187eZSSvJk\niFkIIYQYLqM+oKnVKlbfPp01v9nFS1uOMzk/jYIsWTgwHno8AX735+PsPOxEo1Zx+9VFLJ9fhE47\nqjtihRBCiBFnTLyzWsx6vr58GqGwws/eOIQvEEp0SWOKoijsONTCD/53JzsPOynOSWXN167kzoUT\nJZwJIYQQcTDqe9A+c9mkTG68cgLvfHyK5985xl/eWpboksaEzh4fz/7pKPtrOkjSqbnvuslcf0U+\narWsPSeEEELEy5gJaAAV15ZwvLGL7QdbmFZo4erynESXNGp9fpumaYUWHr65FHu6bBIvhBBCxNuY\nGp/SatR8444ZGPUann/nGM2yb+aX0tzh5qkX9vD8O8fQqFR87ZZSvnvfLAlnQgghxCUypgIagD3d\nyMM3leIPhvnFG4cIhsKJLmnUCIUjVO6oY81vPuZYYzdXTLHxw0euYuHMXNlDUwghhLiExtQQ52fm\nTsvicJ2Lqn0OXt5Sw8obpyS6pBFPtmkSQgghRo4xGdAA7r9+MjVN3Wze08i0IguXT7EluqQRKRAM\n8+a2Ot7e2RDdpqk8h3uWTiLFKDsBCCGEEIkyZgOaXqfhr+6YzhO//YRnNlVTmGUmI82Q6LJGjIii\n8MmRVjZUnaTV5SUzzcDDN5UyvVh2AhBCCCESbcwGNIA8Wwr3Xz+Z3759lF+8eYjvPzAbrWbMTbsb\nEkVROHCygw1bT9LQ2odGreKGORP4yqKJ6JM0iS5PCCGEEIzxgAaw6LJcqutd7Kpu5Y0Pa1mxuCTR\nJSXM0QYXr1ad5ERjNypg/vQs7rimGLslOdGlCSGEEOIMYz6gqVQqHlpWSm1zD5t21FNaaBl3G3rX\ntfSwYetJDtZ2AjB7ciZ3LZpIvi0lwZUJIYQQ4lzGfEADSDZo+as7ZrD2ud387x8P86+r5pJmSkp0\nWXHX1O7m9aqT7D7WBkBZkYW7Fk2kJFc2NhdCCCFGsnER0ACKc1KpuLaEl7ac4FcbD/P391yGeoyu\n7dXW5eWND2vZcagFRYGS3FS+smgi08ZZz6EQQggxWo2bgAZww5UTqK53sb+mg7c+qmf5/KJElzSs\nuvr8/HF7HVWfOghHFPJtJr6yqITLJmXIQrNCCCHEKBL3gFZVVcXatWtRFIUVK1awevXqQY+vX7+e\nP/zhD2i1WqxWK2vXriUnJz57aKpVKr6+fBprfrOL16pqmVpgYVLe6B/u6/MGeeujejbvbiQQimC3\nGLlzYTFzp2WN2V5CIYQQYiyL65oTkUiEJ554gl//+tds3LiRyspKampqBh1TVlbGhg0beOONN7jx\nxht56qmn4lkS5uQkvnH7dBQUfvHGQdy+YFy/Xzx5/SHe3FbLoz/fzls7GzAZdTx801R++JdXMa8s\nW8KZEEIIMUrFNaDt37+fwsJC8vLy0Ol0LF++nM2bNw86Zu7cuej1egBmzZqF0+mMZ0kATC2wcNuC\nIjp6/KzfdARFUeL+PYdTMBTmnV0NPPrzHbz+QS0atZr7lk7iP74xj8Wz8sb9Wm9CCCHEaBfXIU6n\n0zlouDIrK4sDBw6c9/hXXnmFRYsWxbOkAbdfXczRhi52H2vjvb1NLL08/5J834sRCkfYdqCZN7fV\n4er1Y9RruHNhMTfMmYBRP66mEwohhBBjWlzf1YfSM/XGG29w6NAhnnvuuThWdJparWL17dNZ85td\n/H7zCSblpVGQZb4k33uoIorCrmonr39QS6vLi06r5uarCrh5XqHsmSmEEEKMQXENaNnZ2TgcjoH7\nTqcTu91+1nHbt2/nl7/8Jc8//zw6XWyBw2a7+DBls5n5zgOX82+/3sn/bqzm//37xSOqJ0pRFD4+\n7OS5t6qpa+5Bo1Zxy4Ii7rl+ChlpxkSXN6oMR3sR44O0FTEU0l5EvMQ1jZSXl9PQ0EBTUxM2m43K\nykrWrVs36JjDhw+zZs0afv3rX2OxWGJ+7ra23mGpschm4sYrJ/DOx6f4yYu7+fqtZcPyvBerut7F\nhq011Dh6UAELZmRz+zXF2NONRAKhYfv5xwObzSz/XiIm0lbEUEh7EbH6MkE+rgFNo9Hw+OOPs2rV\nKhRFoaKigpKSEp5++mnKy8tZsmQJ//mf/4nX6+Vb3/oWiqKQm5vLT3/603iWdZaKa0s4dqqLbQdb\nmHds9UEAAAo5SURBVFZkYcGM+CzzcSGhcITjjd1U7qjjcJ0LgCum2LhzYTF5si2TEEIIMW6olNF2\nCWO/4f6rpdXl4V+e+RhFgTVfu5Jsa/w3EPf6Q9Q4ujl2qpvjp7o42dxDMBQBYHqxla8smkhxTmrc\n6xjr5K9cEStpK2IopL2IWI24HrTRxG5J5uGbSvnFm4f4+esH+aeH5qDTDu9yFT3uAMcbuzh2qptj\njV2ccvYR6c/HKiDfnsKU/HTmlNqYWhD7cK8QQgghxpb/v737j426vuM4/rq7Qptef9n2CvQHpNCe\nxQ4bMtyiMZ2WEkgMlnIowozoDGR/YKVKWFISMksUjaZBQySUsLiZBtOYU1JYFqVZbTShXTGOZjkQ\nmUhboO0VrmUFvZV+9gd6sVAmZzjue9fn45/efb/f+977m3zyuVc/3x8fAtqP/PqeGfJ9c0Ht/zyn\n5r9/pd8ucf/sfRlj5B/+Vl/2BEKh7PyFy6H1CQ6b5ualyZ2fIXdBuory0pWcxB2ZAACAgHaDNZVu\nfdU3otajvbpnzl1a6Hbd0ufGjdFZ/6hO9gR0oiegk73Dunjpu9D6xOkO/aIwU8UFGXLnp6twVpqm\nT3NE6jAAAEAMI6BdJ3GaQ7+vKtX2P3fpT3/16Y8zUpWVnnTDdmNXx/XN+Uv6sjegkz3DOtkb0Oi3\nY6H1qcnT9Eu361ogK0hXQU6KHHae8A8AAH4aAW0S+a4Uraks1l/+dkJ7Wv6lP6xdqP+OjevU2RGd\n7Anoy56A/n12RMHvL+iXpOz0JJUVZctdkKHi/HTNzEyWjbkwAQDAz0BAu4nflOXKd/qi/nF8QHWN\nRzQ0/N2EC/rzXM7vT1dmyF2QobtSE6NbMAAAiBsEtJuw2Wxat6xEZwb+I3/gigpzU+XOz1Dx9yNk\nTi7oBwAAEUJA+z+SkxJU/7tfyRjDBf0AAOCOIaD9hNv9LDQAAICfQvoAAACwGAIaAACAxRDQAAAA\nLIaABgAAYDEENAAAAIshoAEAAFgMAQ0AAMBiCGgAAAAWQ0ADAACwGAIaAACAxRDQAAAALCbiAa29\nvV3Lli3T0qVL1djYeMP6rq4urVy5UqWlpfroo48iXQ4AAIDlRTSgjY+Pa/v27dq3b58OHjyoQ4cO\n6dSpUxO2yc3N1auvvqrly5dHshQAAICYkRDJnR87dkxz5sxRXl6eJOmRRx5Ra2ur5s2bF9omNzdX\nkmSz2SJZCgAAQMyI6Ahaf3+/Zs2aFXo/Y8YMDQwMRPIrAQAAYl5EA5oxJpK7BwAAiEsRPcU5c+ZM\nnT17NvS+v79fOTk5t2XfLlfqbdkPpgbaC24VbQXhoL0gUiI6grZgwQKdOXNGfX19CgaDOnTokBYv\nXnzT7RlxAwAAkGwmwqmovb1dL7/8sowxWrVqlTZs2KC33npLCxYs0MMPP6zu7m5t3LhRIyMjSkxM\nlMvlUktLSyRLAgAAsLSIBzQAAACEh5kEAAAALIaABgAAYDEENAAAAIuJ6GM2IqG9vV2vvPKKjDHy\neDzasGFDtEuCRVVUVCglJUV2u10JCQl6//33o10SLKSurk5tbW3KysoK3Zg0PDys2tpa9fX1KT8/\nXzt37lRqKo9RmOomayu7du1Sc3OzsrKyJEm1tbUqLy+PZpmwiPPnz2vLli3y+/1yOBx67LHH9NRT\nT4Xdv8TUTQLj4+NaunSp3nnnHeXk5GjVqlVqaGiYMHUU8IPFixfL6/UqPT092qXAgrq6uuR0OrVl\ny5bQj+7rr7+ujIwMrV+/Xo2NjRoZGdHmzZujXCmibbK2smvXLjmdTj3zzDNRrg5WMzg4KL/fr/nz\n52t0dFQrV67U22+/La/XG1b/ElOnOH88t+e0adNCc3sCkzHGaHx8PNplwKIWLVqktLS0CctaW1tV\nXV0tSaqurtbhw4ejURosZrK2IvHsTkzO5XJp/vz5kiSn06l58+apv78/7P4lpgIac3siHDabTc8+\n+6w8Ho+am5ujXQ5iwIULF5SdnS3pWid78eLFKFcEK2tqalJVVZW2bt2qS5cuRbscWFBvb6+OHz+u\nsrIyDQ0NhdW/xFRA478VhOO9996T1+vV3r171dTUpK6urmiXBCBOrF27VocPH9aBAweUnZ2tHTt2\nRLskWMzo6KhqampUV1cnp9Mpm80W1udjKqBFcm5PxB+XyyVJyszM1JIlS9Td3R3limB1WVlZ8vv9\nkq5dR5KZmRnlimBVmZmZoR/cxx9/nP4FE4yNjammpkZVVVWqrKyUFH7/ElMBLdy5PTF1XblyRaOj\no5Kky5cv69NPP1VxcXGUq4LVXD8qX1FRIa/XK0n64IMP6F8Qcn1bGRwcDL3++OOP5Xa773RJsLC6\nujoVFRVp3bp1oWXh9i8xdRenNPncnsD1enp6tHHjRtlsNl29elXLly+nrWCCF198UR0dHQoEAsrO\nztZzzz2nyspKPf/88zp37pxyc3P15ptvTnpxOKaWydpKR0eHfD6f7Ha78vLyVF9fH7q+CFPb0aNH\n9eSTT8rtdstms8lms6m2tlb33nuvNm3adMv9S8wFNAAAgHgXU6c4AQAApgICGgAAgMUQ0AAAACyG\ngAYAAGAxBDQAAACLIaABAABYDAENAMLU2dkpj8cT7TIAxDECGgD8DOHOqwcA4UiIdgEAcDsdO3ZM\nb7zxRmiqr5qaGhUVFcnj8ai6ulqfffaZJGnbtm1atGiRJOnDDz/Uvn37ZLfbNXv2bL300kuhefL2\n7NmjgwcPym63Kzk5Wfv375d0ba69bdu26YsvvpDdbldDQ4Pmzp0bhSMGEJcMAMSJkZERs2LFCjM4\nOGiMMWZgYMCUl5cbn89n7r77bnPgwAFjjDGdnZ2mvLzcBINBc+LECfPggw8av99vjDFm586dZtOm\nTcYYY7xer1m9erUZHR01xhgTCASMMcZ0dHSY0tJS4/P5jDHG7N6922zevPmOHiuA+MYIGoC48fnn\nn6u3t1fr168PTW7tcDg0Njam6dOn69FHH5Uk3XfffUpKStLXX3+tzs5OPfTQQ8rKypIkPfHEE6qq\nqpIktbW1ac2aNUpOTpYkpaenh76rsLBQJSUlkqSysjK1tbXdqcMEMAUQ0ADElZKSEr377rsTlvX1\n9d2wnTFGNpst9Pf65T+8vpnExMTQ6x9CIADcLtwkACBuLFy4UKdPn1ZHR0doWXd3t4wxCgaDamlp\nkSR1dXUpGAyqsLBQ999/vz755BMNDQ1Jkpqbm/XAAw9IkioqKrR///7Q9WyBQOAOHxGAqYoRNABx\nIy0tTbt379Zrr72mHTt2KBgMavbs2dq6dasyMjLk8/m0d+9eSVJDQ4MSEhJUVFSkF154QU8//bTs\ndrsKCgpUX18vSVqxYoUGBga0evVqORwOpaSkqKmpKZqHCGCKsJn/N4YPAHGgr69PHo9HR44ciXYp\nAHBLOMUJYErguWUAYgkjaAAAABbDCBoAAIDFENAAAAAshoAGAABgMQQ0AAAAiyGgAQAAWAwBDQAA\nwGL+B+8vfK7B7MooAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x561829a534d0>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3h 38min 22s, sys: 14min 55s, total: 3h 53min 18s\n",
      "Wall time: 29min 29s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "# Create a ConvNet classifier with 2 CONV-RELU-POOL layers, with filter sizes of \n",
    "# 5 and 5 and 32 and 64 output features.\n",
    "model_params = {\n",
    "        'input_size': 784,\n",
    "        'output_size': 10,\n",
    "        'filter_sizes': [5, 5],\n",
    "        'output_depths': [32, 64],\n",
    "        'hidden_linear_size': 1024,\n",
    "        'use_batch_norm': False\n",
    "        'linear_weights_initializer': tf.random_normal_initializer()\n",
    "}\n",
    "\n",
    "training_params = {\n",
    "        'keep_prob': 0.5,\n",
    "        'num_epochs': 5,\n",
    "        'batch_size': 50,\n",
    "        'stop_early': False,\n",
    "}\n",
    "\n",
    "trained_model, training_results = build_train_eval_and_plot(\n",
    "        model_params, \n",
    "        training_params, \n",
    "        verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KVuVQoUSogFA"
   },
   "source": [
    "## Batch Normalisation\n",
    "Batch normalisation (batch norm) is a more recent (2015) and arguably more powerful normalisation technique than dropout. It is based on the observation that machine learning models often perform better and train faster when their inputs are normalised to have 0 mean and unit variance. In multi-layered deep neural networks, the output of one layer becomes the input to the next. The insight behind batch norm is that each of these layer inputs can also be normalised. Batch norm has been shown to have numerous benefits including:\n",
    "* Networks tend to train faster\n",
    "* Allows higher learning rates to be used (further improving training speed).\n",
    "* Reduced sensitivity to weight initialisation.\n",
    "* Makes certain activation functions feasible in deep networks (When inputs have very large (absolute) expected values, certain activation functions become saturated (For example, the output of sigmoid is always to 1 for large inputs). Relu activations can also \"die out\" when the expected value of the input is a large negative value (why?). This results in wasted computation as these neurons become uninformative. Normalising the inputs to have 0 mean keeps these activation functions in the \"sensible\" parts of their domains.)\n",
    "\n",
    "### How does it work? \n",
    "To normalise some inputs X, ideally we would like to set\n",
    "$\\hat X = \\frac{X - E[X]}{\\sqrt{VAR[X]}}$\n",
    "but this requires knowledge of the population mean and variance statistics, which we don't know, at least during training. We therefore use the **sample mean** and **sample variance** of each batch encountered during training as unbiased estimates of these statistics. During testing, we use statistics gathered throughout training as better estimates of the population statistics. In addition to this, we would like the model to have some flexibility over the extent to which batch norm is applied, and this flexibility should be learned! In order to do this, we introduce two new trainable parameters, $\\gamma$ and $\\beta$ for each layer that batch norm is applied to. Suppose we have a batch of inputs to a layer, $B={x_1,...,x_m}$, we normalise these as follows: \n",
    "\n",
    "$\\mu_B = \\frac{1}{m} \\sum_{i=1}^{m} x_i$ &nbsp;&nbsp;&nbsp; (Batch mean)\n",
    "\n",
    "${\\sigma_B}^2 = \\frac{1}{m} \\sum_{i=1}^{m} (x_i - \\mu_B)^2$ &nbsp;&nbsp;&nbsp; (Batch variance)\n",
    " \n",
    "$\\hat x_i= \\frac{x_i - \\mu_B}{\\sqrt{{\\sigma_B}^2}}$ &nbsp;&nbsp;&nbsp; (Normalised)\n",
    "\n",
    "$y_i = \\gamma \\hat x_i + \\beta$ &nbsp;&nbsp;&nbsp; (Scale and shift)\n",
    "\n",
    "\n",
    "At test time, we normalise using the mean and variance computed over the entire training set:\n",
    "\n",
    "$E[x] = E_B[\\mu_B]$\n",
    "\n",
    "$VAR[x] = \\frac{m}{m-1}E_B[{\\sigma_B}^2]$\n",
    "\n",
    "$\\hat x = \\frac{x - E[x]}{\\sqrt{VAR[x]}}$\n",
    "\n",
    "$y = \\gamma \\hat x + \\beta$\n",
    "\n",
    "### Implementation Details\n",
    "Tracking the mean and variance over the training set can become a little fiddly. Many implementations also use a *moving average* of the batch mean and variance as estimates of the population mean and variance for use during testing. Luckily, TensorFlow provides batch norm out of the box in the form of the `tf.contrib.layers.batch_norm` function. \n",
    "\n",
    "Since the behaviour of batch norm changes during training and testing, we need to pass a placeholder input to the function that indicates which phase we are in. Furthermore, the batch norm function uses variable updates to track the moving average mean and variance. These values are not used during training and so TensorFlow's graph execution logic will not naturally run these updates when you run a training step. In order to get around this, the `batch_norm` function adds these update ops to a *graph collection* that we can access in our training function. The following code, which you will see in the `train_tf_model` function retrieves these ops and then adds a *control dependency* to the optimiser step. This effectively tells TensorFlow that the update_ops must be run before the optimizer_step can be run, ensuring that the estimates are updated whenever we do a training step. \n",
    "\n",
    "```\n",
    "update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n",
    "with tf.control_dependencies(update_ops):\n",
    "    optimizer_step = optimizer_fn.minimize(loss)\n",
    "```\n",
    "\n",
    "Further choices to consider when using batch norm are where to apply it (some apply it immediately before each activation function, some *after* the activation function), whether to apply it to all layers and whether or not to *share* the gamma and beta parameters over all layers or have separate values for each layer. . \n",
    "\n",
    "Have a look at the ConvNetClassifer class above to see what choices were made, try changing these and see what results you get! (See the [TensorFlow documentation](https://www.tensorflow.org/api_docs/python/tf/contrib/layers/batch_norm) for a list of even more parameters you can experiment with)\n",
    "\n",
    "Now, finally, let's switch batch norm on and see how our ConvNetClassifier performs. (Note: we shouldn't expect it to necessarily perform better than dropout as we are already close to the limits of how well we can classify MNIST with our relatively small ConvNet!)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 1103,
     "output_extras": [
      {
       "item_id": 30
      },
      {
       "item_id": 31
      },
      {
       "item_id": 32
      },
      {
       "item_id": 33
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 871282,
     "status": "ok",
     "timestamp": 1503578600521,
     "user": {
      "displayName": "Avishkar Bhoopchand",
      "photoUrl": "//lh5.googleusercontent.com/-OainnMWSi6A/AAAAAAAAAAI/AAAAAAAAAOQ/eI7Z19q5v7E/s50-c-k-no/photo.jpg",
      "userId": "105781587643595215149"
     },
     "user_tz": -60
    },
    "id": "yGv9t5hdoTWF",
    "outputId": "e5124d04-3688-44ac-dd9d-19f0c5556bc2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 Training cost= 0.316522977\n",
      "Epoch: 0001 Validation acc= 0.127800003\n",
      "Epoch: 0002 Training cost= 0.099678086\n",
      "Epoch: 0002 Validation acc= 0.174600005\n",
      "Epoch: 0003 Training cost= 0.062110167\n",
      "Epoch: 0003 Validation acc= 0.677200019\n",
      "Epoch: 0004 Training cost= 0.046187922\n",
      "Epoch: 0004 Validation acc= 0.949800014\n",
      "Epoch: 0005 Training cost= 0.036256776\n",
      "Epoch: 0005 Validation acc= 0.981000006\n",
      "Epoch: 0006 Training cost= 0.026549975\n",
      "Epoch: 0006 Validation acc= 0.984600008\n",
      "Epoch: 0007 Training cost= 0.021220099\n",
      "Epoch: 0007 Validation acc= 0.989600003\n",
      "Epoch: 0008 Training cost= 0.018451589\n",
      "Epoch: 0008 Validation acc= 0.991999984\n",
      "Epoch: 0009 Training cost= 0.015217619\n",
      "Epoch: 0009 Validation acc= 0.991800010\n",
      "Epoch: 0010 Training cost= 0.013219205\n",
      "Epoch: 0010 Validation acc= 0.989799976\n",
      "Epoch: 0011 Training cost= 0.010857335\n",
      "Epoch: 0011 Validation acc= 0.990999997\n",
      "Epoch: 0012 Training cost= 0.009771569\n",
      "Epoch: 0012 Validation acc= 0.991400003\n",
      "Epoch: 0013 Training cost= 0.008388710\n",
      "Epoch: 0013 Validation acc= 0.990199983\n",
      "Epoch: 0014 Training cost= 0.008493815\n",
      "Epoch: 0014 Validation acc= 0.990999997\n",
      "Epoch: 0015 Training cost= 0.006492311\n",
      "Epoch: 0015 Validation acc= 0.991199970\n",
      "Optimization Finished!\n",
      "Accuracy on test set: 0.9907\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmgAAAD+CAYAAACOT/k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xtc1HW+P/DXd25ch+E2gBp4gVAUsIuVl47X1NJcQCVP\nuZubu1lbR9puenTtXrpbm25uv7R2dUuPW6djoWtZUaDYWmpaiiKglncaboJyH2bm+/tjmOGOOMOX\n71xezx4sM9/bvOc9sL743j6CKIoiiIiIiMhlKOQugIiIiIjaYkAjIiIicjEMaEREREQuhgGNiIiI\nyMUwoBERERG5GAY0IiIiIhfTJwHNYrEgLS0NDz/8cId5mZmZGDNmDNLS0pCWloatW7f2RUlERERE\nLkvVFy+yadMmxMbGoqamptP5M2fOxIoVK/qiFCIiIiKXJ/keNIPBgNzcXKSnp3e5DO+VS0RERNRC\n8oC2cuVKLFmyBIIgdLlMVlYWUlJS8Nhjj8FgMEhdEhEREZFLkzSg7d69G+Hh4UhISOhyL9nkyZOR\nk5OD7du3Y8yYMVi6dKmUJRERERG5PEHKsThXr16Nf/3rX1AqlWhsbERtbS2mTp2KV199tdPlLRYL\nbr31Vhw8eLDb7Yqi2O0eOSIiIiJ3JmlAa+3AgQPYuHEj1q9f32Z6WVkZ9Ho9AODLL7/Ehg0b8MEH\nH1x1e2Vl1ZLU6Q30ei375wT2z3HsnXPYP+ewf45j75yj12uveZ0+uYqzvbVr1yIpKQmTJk3C5s2b\nkZOTA5VKBZ1Oh1WrVslREhEREZHL6LM9aL2NSd5x/EvIOeyf49g757B/zmH/HMfeOceRPWgcSYCI\niIjIxTCgEREREbkYBjQiIiIiF8OARkRERORiGNCIiIiIXAwDWi+pqalBZubWa15vyZLfo7a280Hk\niYiIyDsxoPWS6uoryMz8vw7TLRZLt+u9+upfEBAQKFVZRERE5IZkuVGtJ1q//k0UF1/EwoXzoVQq\nodH4QKvV4ty5s/jnPz/CsmVPoaysFEZjI9LT78WsWakAgPT0X2DDhs2oq6vDU09lICnpBhw7dgR6\nfST++MfXodFoZH5nRERE1Nc8MqB9mHMK3xWW9uo2bxkWgXsmx3U5/3e/W4wzZ37Cxo1b8MMPh7Bk\nyePYvPlDREVFAQCWL38OWq0WjY2NePDB+zFhwmQEBQUBaBlT9MKF83jhhVVYuvQPePbZZdi9OwfT\npt3Zq++DiIiIXJ9HBjQ51ZsaYDQ3YfjwEfZwBgAffvhPfP11LgCgtLQUFy6cw/DhiQBaBnLo168/\nYmOtIXDo0GEwGIr7tHYiIiJyDR4Z0O6ZHNft3q7eVlFfiW+LD6K8/hKWfP08FMUm+Pj62uf/8MMh\nfP/9QbzzzrvQaDRYvPghGI3GDttpfThToVB2ugwRERF5Po8MaFITRRHnay4ir+w4jpYfx4WaYpjq\nmlBfVwcfpQZlDRXQmVpaW1tbA61WC41Gg7NnzyA//1iX2yUiIiJiQOshk8WEk5U/Ia/cGsoqG6sA\nAEpBiYTQeCTHj0DWzRqceDsPVaYrECL629e97bax2LbtI/zyl/cgJmYgEhOTWm255Rw0QRBARERE\nJIhuutumrKxa8teoa6rH8YpC5JUfR35FERrMDQAAP5UfEsOGIVk/Agmh8fBTtRzOrDc1YMnXz+O6\nwH5YestjktfoCL1e2yf981Tsn+PYO+ewf85h/xzH3jlHr9de8zp9sgfNYrFgzpw5iIyMxPr169vM\nMxqNWLp0KfLz8xESEoI1a9agf//+XWxJehX1lThafhx55fk4WfUTLKL1PmZhviEY028UkvXDEasb\nDKVC2en6fipfXB88BEWVp1DVeBnBPrq+LJ+IiIg8QJ8EtE2bNiE2NhY1NR3vmL9161bodDpkZWVh\n586deO2117BmzZq+KAtA2/PJ8srzcbHmZ/u8GO11SA4fgWT9cPQPiOrxIcjE8AQUVZ5CfkUhxvW/\nTarSiYiIyENJHtAMBgNyc3Px8MMP4x//+EeH+dnZ2cjIyAAATJ8+HS+++KLUJbU6nywfeeXHUdV4\nGQCgEpQYHjoUSeHDkawf7vDer8SwBHx0cgeOlTOgERER0bWTPKCtXLkSS5YsQXV158euS0tL7fcL\nUyqVCAoKQlVVFYKDg3u1jrqmeuRXFCKvPB/HK4rQYG4EAPir/HBL5E1I1g/vcD6ZoyL8wxHpr0fh\npRNoMjdBrVQ7vU0iIiLyHpIGtN27dyM8PBwJCQnYv39/p8u0v0ZBFMUeHUrsyQl3ZbUVOHgxDweL\nj+B46UmYm88n0weEYXL/sRg1YCSG6eOg6uJ8MmfcEj0SnxR9hVLxZ9ygH9Hr23eWIycsUgv2z3Hs\nnXPYP+ewf45j7/qWpAHt+++/R05ODnJzc9HY2Ija2losWbIEr776qn2ZqKgoGAwGREZGwmw2o6am\nBjrd1Q8tdnY1iSiKOF99EXnlPT+frLKirhfeaUex/rEAvsK/f/weA1QxkryGo3g1jnPYP8exd85h\n/5zD/jmOvXOOy13F+cQTT+CJJ54AABw4cAAbN25sE84AYNKkScjMzMTIkSPx+eefY/To0df0Glc7\nnyxZPxxJ4Y6fT+aoWN0g+Kl8cayiAPeIKR32Ck6dOh5ffrmnT2siIiIi9yDLjWrXrl2LpKQkTJo0\nCenp6Xj66acxbdo0BAcHY/Xq1Vddv9ZYh+8MP3R7Ptnw0Hj49sL5ZI5SKqwB8VDpEfxcW4L+gVFt\n5vOmtERERNQVt7xR7b0fPmo/nyzMNxTJ+uFIDu/+/mRSW7fur4iK6oe0tLkAgI0b38HFWgP2fvc1\ntGIA/BV+ePDBh3H77RMAANOmTUBWVq4stXJXtXPYP8exd85h/5zD/jmOvXOOyx3ilMrQ8FgMCRzS\n5f3JPj71CX4oPdqrr3ljRBJmx93d5fw77piGN9543R7QcnK+wkt/+hOK4y7j+ohY/CbuPjz00AP2\ngEZERETUFbcMaM9PfsLlkvz11w9FVVUVKirKUVlZiaCgIMRExaD2zVLsPHEIR7R7UF5ehsrKSwgJ\nCZW7XCIiInJhbhnQrmZ23N3d7u2SyqRJU7Br11eoqKjAlCnTkJX1GfyafBD/u1vw68R78dri59HY\naOzzuoiIiMi9KOQuwJNMnjwV2dlZyM3NwaRJU1BTU4OYyGgICgFf7s2CwdBy2w83PPWPiIiI+ohH\n7kGTy+DBQ1BXVwe9PhKhoWGYNu1OZC/9AqcPnENxfy0GDhxkX5ZXcRIREVFX3PIqTqDzG9W6qv8t\n2oY9F7/B7298CNeHxMpdDq/GcRL75zj2zjnsn3PYP8exd85x5CpOHuLsA4nhCQCAoxUFMldCRERE\n7oABrQ/EBw+BRqHGsfJCuUshIiIiN8CA1gfUSjWGhcajpK4UpXXlcpdDRERELo4BrY8khg8DAORX\ncC8aERERdY8BrY+MCLMGtGPlPA+NiIiIuseA1keCfXSI0Q7AyaqfUG9qkLscIiIicmEMaH0oMSwB\nZtGMwksn5S6FiIiIXJikAc1oNCI9PR2pqamYNWsW3nzzzQ7LZGZmYsyYMUhLS0NaWhq2bt0qZUmy\nst1ug4c5iYiIqDuSjiSg0WiwadMm+Pn5wWw2495778X48eORnJzcZrmZM2dixYoVUpbiEqK1AxCk\n0eJYRQEsogUKgTswiYiIqCPJE4Kfnx8A6940k8nU6TJuOpjBNVMICiSGDUNNUy3OXrkgdzlERETk\noiQPaBaLBampqRg3bhzGjRvXYe8ZAGRlZSElJQWPPfYYDAaD1CXJyn6Yk6MKEBERURckD2gKhQLb\ntm3Dnj17cOTIEZw6darN/MmTJyMnJwfbt2/HmDFjsHTpUqlLktXQkOuhEpQ8D42IiIi61KeDpb/5\n5psICAjAAw880Ol8i8WCW2+9FQcPHuyrkmTxSu5fccRwHOtmrUSYf4jc5RAREZGLkfQigUuXLkGt\nVkOr1aKhoQHffvstFi1a1GaZsrIy6PV6AEB2djbi4uJ6tO2ysuper7evxAddjyOG48g9cRD/MWB0\nn7++Xq916/7Jjf1zHHvnHPbPOeyf49g75+j12mteR9KAVlZWhv/+7/+GxWKBxWLBjBkzMGHCBKxd\nuxZJSUmYNGkSNm/ejJycHKhUKuh0OqxatUrKklxCYlgC/g/bcay8QJaARkRERK6tTw9x9iZ3T/Iv\n738d5fUVePU/nodGqenT1+ZfQs5h/xzH3jmH/XMO++c49s45juxB4424ZJIYloAmiwknKn+UuxQi\nIiJyMQxoMrHdbuMob7dBRERE7TCgyWRwUAwCVP44Vl7gNTfqJSIiop5hQJOJUqHE8LChqGq8jIs1\nP8tdDhEREbkQBjQZcVQBIiIi6gwDmoyGh8ZDISg4qgARERG1wYAmI3+1P4boBuLMlfOoNtbIXQ4R\nERG5CAY0mSWGJUCEiPyKQrlLISIiIhfBgCazJNt5aDzMSURERM0Y0GQW6R+BcN9QFFw6AZPFJHc5\nRERE5AIY0GQmCAISwxPQYG7EqarTcpdDRERELoABzQXwdhtERETUGgOaC4gLHgIfpQZHOaoAERER\ngQHNJagVKiSExqO8vgKldWVyl0NEREQykzSgGY1GpKenIzU1FbNmzcKbb77Z6TKPP/44pk2bhnnz\n5qG4uFjKklxWYhgHTyciIiIrSQOaRqPBpk2bsG3bNmzbtg179uxBXl5em2W2bt0KnU6HrKwsLFiw\nAK+99pqUJbmsEeHDAPB2G0RERNQHhzj9/PwAWPeUmUwdbyORnZ2NtLQ0AMD06dPx7bffSl2SSwrS\naDEwKBo/Xj6DuqZ6ucshIiIiGUke0CwWC1JTUzFu3DiMGzcOycnJbeaXlpYiKioKAKBUKhEUFISq\nqiqpy3JJSWEJsIgWFFwqkrsUIiIikpHkAU2hUNgPbx45cgSnTp1qM7/9VYuiKEIQBKnLckm2220c\nLeewT0RERN5M1VcvFBgYiFtvvRVff/014uLi7NOjoqJgMBgQGRkJs9mMmpoa6HS6q25Pr9dKWa4s\nwsOHIuSYDgWVRQgLC4BCIV1+9sT+9SX2z3HsnXPYP+ewf45j7/qWpAHt0qVLUKvV0Gq1aGhowLff\nfotFixa1WWbSpEnIzMzEyJEj8fnnn2P06NE92nZZWbUUJctueMgw7C3ejwM/5iM2eJAkr6HXaz22\nf32B/XMce+cc9s857J/j2DvnOBJuJT3EWVZWhvvvvx8pKSlIT0/H7bffjgkTJmDt2rXYtWsXACA9\nPR2VlZWYNm0a3nvvPTz55JNSluTykjiqABERkdeTdA/a0KFDkZmZ2WF6RkaG/bFGo8Ebb7whZRlu\nZWhIHNQKFY6VFyAl9i65yyEiIiIZcCQBF6NRahAfEofiWgMq6ivlLoeIiIhkwIDmgmyjCuTzMCcR\nEZFXYkBzQYnNowpw2CciIiLvxIDmgkJ9QzAgsB9OVP6IRrNR7nKIiIiojzGguajEsASYLCYUXTop\ndylERETUxxjQXFQib7dBRETktRjQXNSgoGgEqgNwrLygw3BYRERE5NkY0FyUQlBgRNgwXDZW43zN\nRbnLISIioj7EgObC7Ic5y3mYk4iIyJswoLmwhNDroRAUOFZeKHcpRERE1IcY0FyYn8oPccFDcLb6\nPC43cpBaIiIib8GA5uKSwqw3rc2v4F40IiIib8GA5uJ4uw0iIiLvo5Jy4waDAUuWLEF5eTmUSiXS\n09Nx//33t1nmwIEDeOSRRxAdHQ0AmDp1Kh555BEpy3IrEf56RPiHo+DSCTRZTFArJP3IiIiIyAVI\n+q+9UqnEsmXLkJCQgNraWsyePRvjxo1DbGxsm+VGjRqF9evXS1mKW0sMS0DO+a9xqvInJITFy10O\nERERSUzSQ5x6vR4JCdZDdAEBAYiNjUVpaamUL+mRkpoPc3LwdCIiIu/QZ+egXbhwAYWFhUhOTu4w\n7/Dhw0hNTcWiRYtw6tSpvirJbcTqBsNX6ctRBYiIiLxEnwS02tpaZGRkYPny5QgICGgzb8SIEdi1\naxe2bduG+fPn49FHH+2LktyKUqHE8LB4VDRcgqGOeyCJiIg8nSBKvEvGZDLhoYcewvjx47FgwYKr\nLj958mR8/PHHCA4OlrIst7PnzH68uf9dzE9OQ0rCNLnLISIiIglJfkng8uXLERcX12U4Ky8vR3h4\nOAAgLy8PAHoUzsrKvOvGrdHqgRAgYN/ZwxgbPsapben1Wq/rX29i/xzH3jmH/XMO++c49s45er32\nmteRNKAdOnQIO3bsQHx8PFJTUyEIAh5//HEUFxdDEATMmzcPX3zxBd5//32oVCr4+vpizZo1Upbk\ntgI1ARisi8FPl8+gtqkOAWp/uUsiIiIiifQooO3cuRPjx49HYGAg3njjDeTl5eHxxx9HYmJit+vd\nfPPNKCjo/srD+fPnY/78+T2v2IslhiXgp8tncbyiCLdE3Sh3OURERCSRHl0ksG7dOgQGBiIvLw//\n/ve/kZqaipdfflnq2qgdjipARETkHXoU0FQq6462vXv3Ij09HbNmzUJjY6OkhVFH/QOiEOITjPyK\nIpgtZrnLISIiIon0KKAJgoCdO3di586dGDPGeoJ6U1OTpIVRR4IgICk8AfWmevx0+azc5RAREZFE\nehTQVqxYgU8++QRz585FdHQ0zpw5g9tuu03q2qgTPMxJRETk+Xp0kcBNN92Et956y/580KBBeOaZ\nZyQriroWHxwLjUKNY+UFSIubKXc5REREJIEe7UH74x//iOrqaphMJtx333244YYbsH37dqlro06o\nlWoMDb0ehrpSlNVVyF0OERERSaBHAe2bb76BVqvFv//9b0RGRuKLL77Axo0bpa6NupAUxsOcRERE\nnuyaxuL87rvvMHXqVERGRkIQBKlqoqsYET4MAHCsnAGNiIjIE/UooIWFheG5557DZ599hnHjxsFk\nMsFs5m0e5BLso0N0YH+crPoJDaYGucshIiKiXtajgPb6669j8ODBWL16NXQ6HQwGAx544AGpa6Nu\nJIYnwCyaUXjppNylEBERUS/rUUALDQ3FL3/5SwQEBODUqVOIiorC7Nmzpa6NumG73cZRnodGRETk\ncXp0m42jR48iIyMDGo0GoijCZDLhr3/9K0aMGCF1fdSFGO110KoDkV9eCItogUK4ptMJiYiIyIX1\nKKC98sorWLlypX0UgX379uGll17CBx98IGlx1DWFoMCI8GHY9/NBnKu+gEFBMXKXRERERL2kR7td\n6uvr7eEMAEaPHo36+nrJiqKesd9ug1dzEhEReZQeBTQ/Pz/s27fP/vzAgQPw8/O76noGgwH3338/\nZsyYgVmzZmHTpk2dLvfyyy9j2rRpSElJQUEBw0ZPDQu9HkpByYBGRETkYXp0iHP58uV47LHHoNFo\nAFgHSl+7du1V11MqlVi2bBkSEhJQW1uL2bNnY9y4cYiNjbUvk5ubi3PnziErKwtHjhzBc889hw8/\n/NDBt+NdfFW+uD54CAorT6Kq8TKCfXRyl0RERES9oEcBLTk5GVlZWTh9+jREUcSQIUOgVquvup5e\nr4derwcABAQEIDY2FqWlpW0CWnZ2NlJTUwEAI0eORHV1NcrLyxEeHu7I+/E6ieEJKKw8iWPlBbh9\nwGi5yyEiIqJe0O0hzvr6evuXyWRCdHQ0YmJiYDKZrvkctAsXLqCwsBDJycltppeWliIqKsr+PDIy\nEiUlJde0bW+WyGGfiIiIPE63e9BuvPFGCIIAURQBwD68kyiKEAShx+eL1dbWIiMjA8uXL0dAQECb\nebZtt8ZhpHpO7x+GKP8IFF46BaO5CRrl1fdsEhERkWvrNqAVFhY6/QImkwkZGRlISUnBHXfc0WF+\nZGQkDAaD/bnBYEBERMRVt6vXa52uzVPcGjMS/yr8EiWWYtwUldijddg/57B/jmPvnMP+OYf9cxx7\n17d6dA6aM5YvX464uDgsWLCg0/lTpkzBli1bMGPGDBw+fBhBQUE9Ov+srKy6t0t1W0P8YgF8ib0/\nfY9o9cCrLq/Xa9k/J7B/jmPvnMP+OYf9cxx75xxHwq2kAe3QoUPYsWMH4uPjkZqaCkEQ8Pjjj6O4\nuBiCIGDevHmYMGECcnNzMXXqVPj5+WHVqlVSluSRhugGwk/lh2PlBRDjU3mImIiIyM1JGtBuvvnm\nHp2n9uyzz0pZhsdTKpQYETYUB0sOo7jWgAGB/eQuiYiIiJzAARw9hO1qzqO8aS0REZHbY0DzEMPD\nhkKAwFEFiIiIPAADmocIUPtjiG4Qzlw5h2pjjdzlEBERkRMY0DxIUngCRIg4XlEkdylERETkBAY0\nD5IYzlEFiIiIPAEDmgeJ8o9AmG8ojlecgNlilrscIiIichADmgcRBAGJ4QloMDfgx8un5S6HiIiI\nHMSA5mGSeLsNIiIit8eA5mHiQoZAo9TwPDQiIiI3xoDmYdQKFRJC41FaV46SujK5yyEiIiIHMKB5\nINuoAvk8zElEROSWGNA80IiwYQCAoxWFMldCREREjmBA80A6Hy0GaqNxquon1Jvq5S6HiIiIrpGk\nAW358uUYO3YsZs2a1en8AwcOYNSoUUhLS0NaWhreeustKcvxKonhw2ARLSi4dFLuUoiIiOgaSRrQ\nZs+ejQ0bNnS7zKhRo5CZmYnMzEw88sgjUpbjVeyjCvA8NCIiIrcjaUAbNWoUgoKCpHwJ6kJ04ADo\nNEHIryiERbTIXQ4RERFdA9nPQTt8+DBSU1OxaNEinDp1Su5yPIZ1VIFhqGmqxZkr5+Uuh4iIiK6B\nrAFtxIgR2LVrF7Zt24b58+fj0UcflbMcj2O73QYPcxIREbkXlZwvHhAQYH88YcIEvPDCC6iqqkJw\ncPBV19XrtVKW5hHGhdyIjflbUFhVhN/o09vMY/+cw/45jr1zDvvnHPbPcexd35I8oImi2OW88vJy\nhIeHAwDy8vIAoEfhDADKyqqdL84LXB8Si+MVRSg6fw6hviEArL9k7J/j2D/HsXfOYf+cw/45jr1z\njiPhVtKA9uSTT2L//v2oqqrCxIkTsXjxYjQ1NUEQBMybNw9ffPEF3n//fahUKvj6+mLNmjVSluOV\nksIScLyiCMfKCzH+ujFyl0NEREQ9IGlAe/3117udP3/+fMyfP1/KErzeiLAEANtwrKKAAY2IiMhN\nyH4VJ0krzC8E/QOiUFR5Co1mo9zlEBERUQ8woHmBxPAEmCwmnKjkbUyIiIjcAQOaF0hqHlXgKG+3\nQURE5BYY0LzAoKAYBKj9cay8oNuraomIiMg1MKB5AYWgwIiwYbhsvIILNcVyl0NERERXwYDmJTiq\nABERkftgQPMSCaHxUAgKHK1gQCMiInJ1DGhewl/thzjdYJy9ch5VDVfkLoeIiIi6wYDmRRKbr+b8\nofiYzJUQERFRdxjQvIgtoO09d5BXcxIREbkwBjQvEumvR1zwYOSVFOCzM1/JXQ4RERF1gQHNy/w2\n8VfQB4Th09Nf4oDhe7nLISIiok4woHkZrSYQy8Y/Cj+VL7YU/B9OVv4kd0lERETUDgOaF7ouqB8e\nTLwfFoj429FNKKkrk7skIiIiakXSgLZ8+XKMHTsWs2bN6nKZl19+GdOmTUNKSgoKCniPrr4yNDQO\n9w2dg1pTHdYd2YgaY63cJREREVEzSQPa7NmzsWHDhi7n5+bm4ty5c8jKysKLL76I5557TspyqJ0x\n/W/B9IGTUVZfgXeOvocmi0nukoiIiAgSB7RRo0YhKCioy/nZ2dlITU0FAIwcORLV1dUoLy+XsiRq\n5+4h03BzxEj8ePkM/qfgQ95+g4iIyAXIeg5aaWkpoqKi7M8jIyNRUlIiY0XeRyEo8KuEezBENxAH\nSw7j09Nfyl0SERGR15M1oHW2t0YQBBkq8W5qpRqLkhYg3DcUn535Cvt/PiR3SURERF5NJeeLR0ZG\nwmAw2J8bDAZERET0aF29XitVWV6hff/00GKFdjH+kP0athRtxeCo/hgRES9Tda6PP3+OY++cw/45\nh/1zHHvXtyQPaN2d0zRlyhRs2bIFM2bMwOHDhxEUFITw8PAebbesrLq3SvQ6er220/6pEYDfjvgV\n3jz8d7z69Xo8dfOjiAroWWD2Jl31j66OvXMO++cc9s9x7J1zHAm3kga0J598Evv370dVVRUmTpyI\nxYsXo6mpCYIgYN68eZgwYQJyc3MxdepU+Pn5YdWqVVKWQz0QHxKL+cPmYlPB/2LdkY14atR/QasJ\nlLssIiIiryKIbnrZHpO843ryl9AnP32Bz85kY4huIDJuWAS1Ut1H1bk+/iXpOPbOOeyfc9g/x7F3\nznFkDxpHEqBOzRw8DaMib8BPl89ic8GHsIgWuUsiIiLyGgxo1ClBEPDLYemI1Q3CodIj+PSnLLlL\nIiIi8hoMaNQl2+039H5h+PxsDr4t/k7ukoiIiLwCAxp1K1ATgEdGLkSAyh//LPoIRZdOyV0SERGR\nx2NAo6uK8NdjUfICKCDgb8c2wVDL0R6IiIikxIBGPRIXPBjzE9JRb2rAW0c24oqRV/MQERFJhQGN\neuzWqJswY/BUVDRU4u2892A0N8ldEhERkUdiQKNrMmPQHbgl8iacuXIOm45/wNtvEBERSYABja6J\nIAiYnzAXccGD8UPZUez46Qu5SyIiIvI4DGh0zdQKFRYlLUCEfziyzu7C3uL9cpdERETkURjQyCEB\nan/8LnkhAtT++KAoEwWXTshdEhERkcdgQCOHRfiH46GkX0MBAX8/+j8orjHIXRIREZFHYEAjp8QG\nD8KvEu5Bg7kB6/L+gcuNvP0GERGRsxjQyGmjom7E3YOn41JDJd7OexdGs1HukoiIiNya5AFtz549\nuPPOOzF9+nS88847HeZnZmZizJgxSEtLQ1paGrZu3Sp1SSSBOwdNxm1RN+Ns9Xm8y9tvEBEROUUl\n5cYtFgteeuklvPvuu4iIiMDcuXMxZcoUxMbGtllu5syZWLFihZSlkMQEQcB9w+bgUkMljpQdw7Yf\nd2J23N1yl0VEROSWJN2DlpeXh4EDB2LAgAFQq9WYOXMmsrOzOywniqKUZVAfUSlUWJR0PyL99cg+\ntwdfX9zQUO10AAAUiElEQVQnd0lERERuSdKAVlJSgn79+tmfR0ZGorS0tMNyWVlZSElJwWOPPQaD\ngVcCujN/tT8eGbkQgeoAfHhiG45XFMldEhERkduR9BBnT/aMTZ48GXfffTfUajU++OADLF26FO+9\n995V19Prtb1RoteSsn96aPHf/o/ghV1rsDF/C16a8hRiggdI9npy4M+f49g757B/zmH/HMfe9S1J\nA1pUVBSKi4vtz0tKShAREdFmGZ1OZ398zz334M9//nOPtl1Wxts5OEqv10revxDo8auEediYvwWv\n7H4TT4/6L+h8giR9zb7SF/3zVOydc9g/57B/jmPvnONIuJX0EGdSUhLOnTuHixcvwmg04tNPP8WU\nKVPaLFNWVmZ/nJ2djbi4OClLoj50c+RIpAy5C5WNVVif9w808vYbREREPSLpHjSlUolnnnkGCxcu\nhCiKmDt3LmJjY7F27VokJSVh0qRJ2Lx5M3JycqBSqaDT6bBq1SopS6I+NnXgRJTVl+Obn7/Du/nv\n48GkX0Eh8PZ7RERE3RFEN72EkrtaHdfXu6rNFjP+35ENKKo8hcnR/4E518/qs9eWAnf1O469cw77\n5xz2z3HsnXNc7hAnEQAoFUr8NvFXiPKPQM75r7Hnwjdyl0REROTSGNCoT/ir/fC7kQuhVQfiwxPb\ncay8QO6SiIiIXJZbBrTDJ0pR29Akdxl0jcL9QvFQ8q+hUiixMX8LLlQXX30lIiIiLyTpRQJSeebt\nbyEAuC4iEEOjgzE0Jhjx0cHQ+mvkLo2uYrAuBguG34u/H9uMdXn/wNOj/gvBPrqrr0hERORFlM8/\n//zzchfhiKYmM86X1uDUxcv4rrAUn+8/h4OFpbhYVovGJjMC/dTw1bhl/pRcQIAP6urku+VFv4BI\naBRqHC47hpOVP2JU5I1QKdzns5K7f+6MvXMO++cc9s9x7J1zAgJ8rnkd9/lXsZX7pg/D1JsGoMlk\nwemfr6DoXCVOnK/CyYuXcbG8Frt+uAgAiAz1t+5ha97LFhrkK3PlZHNHzASU1Zdjb/EB/CP/n3go\neQFvv0FERNTMLQOajVqlQHy09fAmAJjMFpw1VKPofBWKzlXh5IUq7DlSjD1HrOc6het87YdDh8aE\nQK/zhSAIcr4FryUIAubFp6GivhLHKgrw0ckdSI9PkbssIiIil+DWAa09lVKB2AE6xA7QYcbogTBb\nLDhfWoOicy2Bbe9RA/YetQ7IHqL1wdAY6x62+OhgRIX6M7D1IaVCid8m/RKvH3oLuy/sRbhfGCZF\n3y53WURERLLzqIDWnlKhwKCoIAyKCsL0W2NgEUVcKK3BifNVKDpfhRPnq7AvvwT78ksAALoATfPe\nNWtg6x8eAAUDm6T8VH74XfJCvHbor/jo5A6E+4UiKXy43GURERHJyqtHEhBFEcUVdThxrtJ+WPRy\nbctJkIF+amtgaw5t1+kDoVC4f2BzxTtCn71yHmu+Xw9BEJAWOwNhfmEI8dEhxDcYfirXOnfQFfvn\nLtg757B/zmH/HMfeOceRkQS8OqC1J4oiSivrm8OaNbRdutJon+/vo8L11+kwNCYEQ2OCERMZCKXC\n/U5sd9VftMNlx/D3o5shou2PpK/SFyG+OoT4BNu/B/sG2wNciE8wNEp1n9Xpqv1zB+ydc9g/57B/\njmPvnMOA1stEUUT55QbrIdFzVSg6X4myqgb7fB+NEtcP0DWfxxaCQf20UCldP7C58i9acY0B56ov\noLLhMiobq1DZWIWq5sf1poYu1wtQ+7cJcNYQZwt1wQj2Ceq1W3m4cv9cHXvnHPbPOeyf49g75zgS\n0Dz6HDRnCYIAfbAf9MF+GJfUDwBw6UqD/Ry2onNVOHb6Eo6dvgQA0KisFylcpw+Ej0YBH7USGrUS\nPm2+FNBolO3mKaBRK3m+G4D+gVHoHxjV6bwGUwMqGy+jssEa3GwhzhbgSuvKcKGm69EJtJpAe2Br\n2fvWshcuSKOFUqGU6q0RERH1mOQBbc+ePVi5ciVEUcScOXOwaNGiNvONRiOWLl2K/Px8hISEYM2a\nNejfv7/UZTksNMgXo0dEYfQIa4i4XNNoDWvNFx0UnK1EwdlKh7atUSlaQpumObiprI81zUGuddjT\nNC+nUSlarWOb13aaO+zZuxpflS/6qXzRLyCy0/miKKLOVN8hwFU2XEZVYxUqG6pQXPMzzlVf6HR9\nhaBAkEbbdk9cqxAX7BMMrSZAyrdIREQEQOKAZrFY8NJLL+Hdd99FREQE5s6diylTpiA2Nta+zNat\nW6HT6ZCVlYWdO3fitddew5o1a6Qsq1fpAn1wa0Ikbk2whobqOiMqrjSg0WhGY5MFxiYzGpu/jE0W\n++PGJjOMRttjS6tlrF9Xao1obDKjyWTplTqVCgGa5j11vhoVIIpQKhVQKgQoFAJUzd+VCsE+3fZl\nnd48TSm0WkcBpVKAQmiZbltOYVvWPq/VNpXttmmbJlhfWyEACoX1uUKwLisIsD9WCNa9m7ZlhFbL\nB6j9EaD2x3XazkO+RbSgpqnWvtetJcRV2ffOna0+j9NXzna6vkJQwEepASBAKSigaP5q/bjtNKX9\nsdBuuautY33/Cijtz7tbp3k9W10KZYfl29fT9Xa6f00iIpKepAEtLy8PAwcOxIABAwAAM2fORHZ2\ndpuAlp2djYyMDADA9OnT8eKLL0pZkuS0/ppeHRPUYhHtwc0W5lqHPnvwM3ac1n65RqN1mtlsQZPJ\nArPFBLNFhNligdkswmxxy9MR27CGOOt3oTnMWUNcS7hrE+wEBRSKMCiEcAiCAJUCCBcAqBpgUdU3\nf9XBoqyHWVkHk7IOokKERbTALIgARIiCGYAJIiyAIDZf5GCBaP/POt1TKNB9uLMHTAhQCMrmwGhd\nRqNRw9RkhgBr/wVYA58gCBCavwNonm+dhuZlm6e2nW5d2bqubarQeq7tdaz/wf64/XZt27HNaz3f\n+rjlf1tey7a91lrPa1V122WEjuu0vFTb12r92v4lPqitbYAI6x5jESJspxHbHtv+g2j9KQREiCJa\nfh5F2zTRNte+DYton9IyDSJg327L67bZhn2a7bNs/XkKUEBo01/7Mh0+75bPSWH//GzrKFp9bu22\nY19H0WUNgiDAv0Rj7Z8oWn9DW/XQ9t0CS9v57ZaxiNbf6c7XbbuNztdtP73l9br6eRDazxE6Wa7d\nz2tn09tvqfXPWpt5QrvlAPj4qGA0mtt9Bm1/B1v/Ltk/LUd+H9v8Lrb9PW3/c4NW27P1HUDHz7iL\nz7OzzxCdfKZA1z8THT9TdPhs18x8FtdK0oBWUlKCfv362Z9HRkbi6NGjbZYpLS1FVJT1cKFSqURQ\nUBCqqqoQHBwsZWluQ6EQ4Oejgp9P731U3Z3sabEFNovY8mW2TrN0mCbalzfZHpvFltBnn29bp2Wa\nqfm5RWy9Tss2LaII0fZdtP7DYZ0O+/yWZZp/GW3zxVbbaLeu2GZd615eS7t1W7apgcWihoigXui6\n9RccQruv5mlCJ9MgWFrm9WhdS7v1W81rs76lm3kt04TW22w13dxpvRZAMHfzXlrVQ+TFWsUM+7PW\nEckWt1u0n9L6UWe/T2Kny5K02n+eLSG0/Z9nPSdpQOvJBaLtlxFFkXfzl5FCIUChUKLvblrh+sTm\nQGexWANcWFggSkuruwmJnYTG1sGw+Xnr+e1fw2Jp3g9nW69d4BTbbbd1ELX99QbbsiJapqF5L4n1\nj8Tmv/g6n2b/K1Bs9d02r/kxLLa9LrC/D9vrWntn+0vS+litVqLB2GR90mqPjK3PYqt/jERbQbDt\n07HOEGGBKKD5r1zbIq22B9s80fYEomB7ndav0LovrV+37fpt1rBtD62/t7xH+0ui5T3Zvwv2t9PM\nYu2PgJbXsa/fvH2x9dZs04TmCS3/PymKLfv2WhbuuBx6sFzLtjrZRpvHnS1nm93qjdpCuT2ct3+O\nVsHdNq/5cVfbar+9Vsu1/SOgk21AsNfesW8t89p+76JHtnkdlu98PTj8T3Vv6TzQdV5W6153sm6H\n/rdbr90fY51/Lq2Xb/+a7Wttt0779Vs20vGzBDp+Lq2W6/TnoMM67eZf6+d6z9UXaU/SgBYVFYXi\n4par6kpKShAREdFhGYPBgMjISJjNZtTU1ECn0111245cskot2D/nDIwOkbsEIiLyYJKe8ZuUlIRz\n587h4sWLMBqN+PTTTzFlypQ2y0yaNAmZmZkAgM8//xyjR4+WsiQiIiIilyf5jWr37NmDV155BaIo\nYu7cuVi0aBHWrl2LpKQkTJo0CUajEU8//TQKCgoQHByM1atX47rrrpOyJCIiIiKX5rYjCRARERF5\nKt7UiIiIiMjFMKARERERuRgGNCIiIiIX43YBbc+ePbjzzjsxffp0vPPOO3KX4zYMBgPuv/9+zJgx\nA7NmzcKmTZvkLsktWSwWpKWl4eGHH5a7FLdTXV2NjIwM3HXXXZg5cyaOHDkid0lu5d1338Xdd9+N\nWbNm4cknn4TRaJS7JJe2fPlyjB07FrNmzbJPu3z5MhYuXIjp06fjN7/5DaqrO79ht7frrHevvvoq\n7rrrLqSkpGDx4sWoqamRsULX1ln/bDZs2IBhw4ahqqrqqttxq4BmG9tzw4YN+OSTT/Dpp5/ixx9/\nlLsst6BUKrFs2TLs3LkTH3zwAbZs2cLeOWDTpk1thiqjnnvllVcwYcIEfPbZZ9i+fTv7eA1KSkqw\nefNmfPzxx9ixYwfMZjN27twpd1kubfbs2diwYUObae+88w7GjBmDL774ArfddhvefvttmapzbZ31\n7vbbb8enn36K7du3Y+DAgexdNzrrH2DdUfLNN9+gf//Ox4puz60CWuuxPdVqtX1sT7o6vV6PhIQE\nAEBAQABiY2NRWloqc1XuxWAwIDc3F+np6XKX4nZqampw8OBBzJkzBwCgUqkQGBgoc1XuxWKxoL6+\nHiaTCQ0NDR1u+k1tjRo1CkFBbYdpy87ORlpaGgAgLS0NX331lRylubzOejd27FgoFNbIcMMNN8Bg\nMMhRmlvorH8AsHLlSixZsqTH23GrgNbZ2J4MGdfuwoULKCwsRHJystyluBXbLxeHIrt2Fy5cQEhI\nCJYtW4a0tDQ888wzaGhokLsstxEZGYkHHngAEydOxPjx46HVajF27Fi5y3I7ly5dQnh4OADrH62V\nlZUyV+Setm7divHjx8tdhlvJyclBv379MHTo0B6v41YBjbdsc15tbS0yMjKwfPlyBAQEyF2O29i9\nezfCw8ORkJDAn0MHmEwmHD9+HPfddx8yMzPh6+vLc0ivwZUrV5CdnY1du3bh66+/Rl1dHXbs2CF3\nWeSF1q1bB7Va3en5VdS5hoYGrF+/HosXL7ZP68m/I24V0Hoytid1zWQyISMjAykpKbjjjjvkLset\nfP/998jJycGUKVPw5JNPYv/+/de0q9rbRUVFISoqCklJSQCA6dOn4/jx4zJX5T6++eYbREdHIzg4\nGEqlElOnTsUPP/wgd1luJywsDOXl5QCAsrIyhIaGylyRe8nMzERubi5ef/11uUtxK7YhL1NSUjB5\n8mSUlJRgzpw5qKio6HY9twpoPRnbk7q2fPlyxMXFYcGCBXKX4naeeOIJ7N69G9nZ2Vi9ejVuu+02\nvPrqq3KX5TbCw8PRr18/nD59GgCwb98+XiRwDfr3748jR46gsbERoiiyfz3Ufi/F5MmT8fHHHwOw\nhg3++9G19r3bs2cP/v73v2PdunXQaDQyVeU+WvcvPj4ee/fuRXZ2NnJychAZGYnMzEyEhYV1uw2V\n1EX2JqVSiWeeeQYLFy60j+3J/5PqmUOHDmHHjh2Ij49HamoqBEHA448/zvMIqM+sWLECTz31FEwm\nE6Kjo7Fq1Sq5S3IbycnJmD59OlJTU6FSqTB8+HDcc889cpfl0mx7uquqqjBx4kQsXrwYixYtwmOP\nPYaPPvoI/fv3xxtvvCF3mS6ps969/fbbaGpqwsKFCwEAI0eOxPPPPy9voS6qs/7ZLpACAEEQenSI\nk2NxEhEREbkYtzrESUREROQNGNCIiIiIXAwDGhEREZGLYUAjIiIicjEMaEREREQuhgGNiIiIyMUw\noBERXaMDBw60ua8REVFvY0AjInKAIAhyl0BEHsytRhIgIrqavLw8/PnPf0ZtbS0AICMjA3FxcZgz\nZw7S0tKwd+9eAMCzzz6LUaNGAQC2bduGDRs2QKFQICYmBi+88IJ9nMa3334bn3zyCRQKBfz9/fH+\n++8DsI5t++yzz+Lw4cNQKBRYvXo1hgwZIsM7JiKPJBIReYgrV66IqampYllZmSiKolhaWiqOHz9e\nLCgoEIcOHSpu375dFEVRPHDggDh+/HjRaDSKRUVF4u233y6Wl5eLoiiKf/nLX8Tf//73oiiK4scf\nfyzOmzdPrK2tFUVRFKuqqkRRFMX9+/eLI0aMEAsKCkRRFMV169aJTz31VJ++VyLybNyDRkQe4/vv\nv8eFCxfw4IMP2se6UyqVMJlM0Gg0+MUvfgEAuOWWW+Dr64vTp0/jwIEDmDhxon3g4v/8z/9ESkoK\nAGD37t2499574e/vDwDQ6XT21xo8eDCGDRsGwDou4e7du/vqbRKRF2BAIyKPMmzYMGzevLnNtIsX\nL3ZYThRF+6DFrc8na/1c7GaoYh8fH/tjWwgkIuotvEiAiDzGjTfeiDNnzmD//v32aUePHoUoijAa\njdixYwcA4ODBgzAajRg8eDDGjBmD3NxcVFRUAAA+/PBDjB07FgAwefJkvP/++/bz2aqqqvr4HRGR\nt+IeNCLyGEFBQVi3bh3+9Kc/YdWqVTAajYiJicEf/vAHBAcHo6CgAH/7298AAKtXr4ZKpUJcXBye\neOIJ/PrXv4ZCoUB0dDRefPFFAEBqaipKS0sxb948KJVKBAYGYsuWLXK+RSLyEoLY3T58IiIPcPHi\nRcyZMwf79u2TuxQioh7hIU4i8gq8bxkRuRPuQSMiIiJyMdyDRkRERORiGNCIiIiIXAwDGhEREZGL\nYUAjIiIicjEMaEREREQuhgGNiIiIyMX8f7YMl30Z943LAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x55d2c6dc4cd0>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmgAAAD+CAYAAACOT/k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VPW9P/D3mSX7QhJmJosQIJGE4LBUEKE2CKigEQET\nRaW21/SR2j4VvdXailLaUqFQ5bnS3y3ClUoF6lIFuRCs1IDE68IiS1AW2SMJmSRkIQvJZOZ8f3/M\nZJiQQCYzOTkzmffreTBn+Z6TTz5B8s5ZJSGEABERERH5DY3aBRARERFRewxoRERERH6GAY2IiIjI\nzzCgEREREfkZBjQiIiIiP8OARkRERORnFA1o8+fPx4QJEzB9+vRrjvnjH/+Iu+66CzNmzMDRo0eV\nLIeIiIgoICga0O6//36sWbPmmut37dqFkpISbN++HX/4wx+wcOFCJcshIiIiCgiKBrQxY8YgJibm\nmusLCwsxc+ZMAMDIkSNRX1+PqqoqJUsiIiIi8nuqXoNWUVGBxMRE17zJZILFYlGxIiIiIiL1qRrQ\nOnvLlCRJKlRCRERE5D90an5yk8mE8vJy13x5eTmMRmOX2wkhGOSIPCCEgCycH2UBWQgIAciycK27\nMi0gy7gy7Ta247RwbgfndteYFgJCbtufY99CxpV9tPuczvXuNbutv3q5EHDuGx1qssuyc5wMu7BD\nFgJ2IcMu2x3L4VgunMuFLEOG7NgPAIgrv0AK4fjj6imEa961vK32tmkIQEhtq66sc0671l21jWN/\nEuBcLoRjndy2tQwISBBCAiBBEoAQjnnXNCRAlpz7bfvovj+3aefnvLqGjus63+b6+3N8b/quzr44\n57JOfzy5jZc6WeZaJF0ZINwH82deINvyyoxub6N4QLveu9inTJmCDRs24J577sHBgwcRExOD/v37\nd7lPSZJQWVnfk2UGFYMhOqj7J8sCdlmGXRaOP3bnR+cyud0y53K7gF04lkdHh6G6pgl2WXbuS1zZ\nrt203GG53e6cFwJ2u+w2ffX2crt92Zwfr0y3X99WnytstYUarzokAEkGNDIgyZCcH68777ZMkjqb\ntzvnHYlDkgTQ9gcdp6XO1l29vsM6OD6HBOf6Lrj9HITWq0YFBiG5fZRcHyVXEHBb1sm05ByjuWre\nfR9tyyQ4w6nU1loBIQnXtHswcZTllnAlt+l24wVcf5Pdxgi0Hy/abd+29qppCe3SdmdTbePa/w3y\nl6Qpubp9dd9d66Qrc52Ol66xD+mq/bjNQZKg02pgt8vOXz7Qrv9Xeu0+d9V6cfUa9+5f/T1z//er\n4+fpsJ929VzN0+/k9cZdbytP/274WUB75plnsHv3btTW1uL222/Hk08+idbWVkiShNmzZ2PixInY\ntWsX7rzzToSHh2PJkiVKlkMKEkLAZhdotclotctotdkd0855m2u53G55q825zm3eEY7cApQr3FwV\neOyyK9y4Bxm73XEkpLOQJcvehpYe6ZIzTDjDizNYuAKPdGW95PZRoxWQtAJaDSDpBTQaAUlz5aNO\nI6CXHNNX9uX4IyQZkOyAxjEtYHcutzvnndOwu9YLSVatQ0DbjwcNNJLkjAUaaCSNY6mkhQYSJMl9\nmWNaAw0kqf28a1pybutcr5U0kJwf3ec1bUfmJee3q91BC4ErCaMthLjNt23YFgKc69v+xl09sv1+\nnVMdVraPEKGhOjQ1t7iODsoQEEJ2HqF0HAV0rRNtRwTdxsCxXLiNdxx9vDK2bVvhNt41xrneV+5B\nQuMKB44goHGtk65aB9f3vq3vmrawcdVH9/24ooYkIUSnQ6vN7tpfWy1Xvj/O/0rulV6ZblvnvtQ1\nSnLfw1VbSlL7sW4DJbeRjiORbd+z9h9l0RZinL+EuYKJfGXeub3s2k52HYHtfL9yu/21hSLXOvd5\nWTj+eZLcQqHkHhGvhLy2/rp6K7X9Xw3XOPdA6R4pXfts+1528rncQ6j7OrffANp/j646+Njxu3qN\nOanzdR2PZbp/rzv/PN6QxPUOcfmxYD4C5ClZCDS32NHU0oqmZhsut9jQ1GxDSHgIqmsaOwSjq4NT\nq02Gzd758s7GqUGrkRx/tBI0kgStVnNlmcYx71guQaeRoHFb7j7OsVwDrdZtW40GkkZGs6YWTVIN\nGlGNy6IOkk6Gze4MNY4fWZCv/ihkyLC3+6hmNHSnkTTQa3TQaXTQSY6PrnmNDjqNFnqNHjqN9hrr\nncskx1jHvN41ffW+3ZeZ+seipqbJLTy5BSrnD15evnBt/nL0+0qAawt3jhDn+NY5fxh3Grqg6vfY\nX/oXiNg73xgM0d3eRtVr0Oj6hBCw2mQ0NdvQ1NyKJmfAamq2OacdyxqbbbjsWmZDY3OrI4y12Dr5\njdx7Oq0EvU4DvVYDvU6D8FAdYiIc064/znU6befLuxqn0zr+dAhZ7vNugayn/qGXhYzq5lqUNVxA\nWWM5ShsuoKyhHBWXqyDLV4VPq+ODBAlajRY6SQutpIVGo3FOO8KKVtI6jtA4p3WSFlqNY5lj/JVt\ntRqNc3zbGOc4TSfL3Ofd9qfTaKGRtO3DlCtgaV0hSiOpd29QfEQ07I19+XxicNBIGkDq22eGidTG\ngKYwm112C1Ttg1ajc7otXDVeHb6abbB38yrbUL0WEWE69IsKRXL/SESE6hARpkNEqB4RYTqEh+rQ\nPyESLZet7QNSF4Gq7UhUX9DY2oSyhgsobSxHWUO5K5S12K3txoVpwzAoZgCSo5KQEpmI5KgkJEYY\ncUNiAqovNqkadIiIqG9jQOshLVY7Dp6swp6jFlhqLruCmLW1e6f+dFoJEWF6RIbpYegX7gxXOucy\nx3S482NkmN61vm2ZTtt1aAiWQ9Wtsg3ljRWuAFbW4DgyVme91G6cRtLAFGFAcmQiUqKSkByViOTI\nJMSH9ev0CJ1eq2c4IyIiRTGg+cBml/H1mWrsPmLBgROVrjAWGeYIT7FRoa5Q5X4UK8J9WZjebb0O\nIXqeNOiuq09PtgWxistVHS5o7hcai6yEDKREOoJYSlQSjBEG6DX8X4GIiPwHfyp1kywEvi2pxe6j\nFuw7VoHGZhsAwNgvHLdkmTAuy4SU/pEqV9l3dTw9WY6yxgudnJ4MdZyedB0VS0JypAkR+giVKici\nIvIcA5oHhBA4Z6nHl99YsPdYBWrqWwAAsVEhuHPMAIzLMsEQLeHjjz9CSv+8bu37ueeexsKFf0Rk\nZJQSpQeszk5PljWWo7alrt247p6eJCIiCgQMaNdx4WIjdh+xYPcRx3VlABARqkP2yCSMG2ZCxsA4\naDSOEHDhQhk2bfonZs1qH9BkWYZGc+3rlZYt+y/lvoAA02xrxqaTBThZdxYVTZU8PUlEREGLP9mu\nUn2pGXuOVmD3EQvOWRwX0ofoNLhlmBHjsky4aXAC9LqOgeu11/4fyspKkZ8/B1qtFiEhoYiOjkZJ\nyTn84x/v4/nnn0VlZQWs1hY88MDDmD59JgDggQfuw5o169DU1IRnn50Hs3kUvv76EAwGE/70p1cQ\nEhLSq1+/mj449SH+r2w3T08SEVHQ65MB7d0dJ7H3WIXH44UQaGm1o6XVDpv9ymMt9DoNQvVahOg1\niI8Ow+gbDdfcx89+9iTOnj2Nv/1tAw4c+ArPPfefWLfuXSQmJgIA5s9fiOjoaLS0tODxx3+EiRMn\nIyYmBu5PID5//jv8/vdL8Otfv4Df/vZ5fPLJDtx117TuNyAAnag5jU9Lv0BSpAm/GfsUdDwqRkRE\nQSxofwoKCFhbZbS0Ol5J1EanlRCq1yJUr/XpGqasrOGucAYA7777D3z66S4AQEVFBc6fL0FW1k1w\nf5VLUlIy0tLSAQAZGZkoLy/z+vMHEqu9Ff849h4kSPjhsAcYzoiIKOj1yZ+ED05Ox4OT0zssb7XJ\n+Pr0Rew+asHBE1WwOoNZqika47JMuGWYEfExYT1SQ1jYlf0cOPAV9u/fh9Wr1yIkJARPPvlTWK3W\nDtu4n87UaLSdjumLtp35NyouV2HygB9gUMxAtcshIiJSXZ8MaO5kWeB4SQ2+PGLBV8cr0dTieCyG\nKT4C45zXlSUl+P5YjIiICDQ1NQFwnDJ119jYgOjoaISEhODcubP45puvO91HgL4W1Scll86j8Lsi\n9A+Lx71DpqpdDhERkV/okwFNCIEzF+qx+4gFe45ZUNfgOBIVFx2KH4xMwrgsE1JN0T36GIaYmFiY\nzSPx4x8/hJCQUMTHx7vWjRs3AR988D5++MMHMXBgKm66yey25ZUagu2xEHbZjvXH/glZyHgkMw+h\n2uC5IYKIiOh6JBGgh206e1VRaZXjsRh7jlhQUet4LEZkmA5jMo24NcuEGwf06zPvk/SFv7zq6V9n\nd2DL6X9hQtItmDOse8+PU5O/9C8QsXe+Yf98w/55j73zjcEQ3e1tFD+CVlRUhMWLF0MIgdzcXMyd\nO7fd+rKyMsyfPx/V1dXo168f/vznP8NkMnm8/6q6y67HYnxX0QAACNFrcGuWCbdkmXDT4HiP3k9J\nvau8sQIfnvk3YkOiMSs9R+1yiIiI/IqiAU2WZSxatAhr166F0WhEXl4epkyZgrS0NNeYpUuXYtas\nWZgxYwZ2796NV155BcuWLbvufusaWrBj/3l8ecSCk+cdT5bXaiSMSu+PcVkmjErvj9AQvtPSX8lC\nxoZj/4RN2DE7435E6MPVLomIiMivKBrQiouLkZqaipSUFABATk4OCgsL2wW0U6dOYf78+QCAcePG\n4ec//3mX+/3R7z+CLAtIADIH9sOtwxPxvaEGRIXrFfk6qGcVlX6B03Xn8D3jCIw0DFe7HCIiIr+j\naECzWCxISkpyzZtMJhw+fLjdmMzMTGzfvh2PPvootm/fjqamJtTV1SE2Nvaa+80YGIeRQ+IxdpgJ\ncdGhitVPPe/i5RpsPvUhInUReGDoDLXLISIi8kuKBjRP7j947rnnsGjRImzatAljxoyByWSCVnv9\n05PLnvxBT5UYtLy5YNFXQgisLloLq92KueMeQVpKcq/X0FPU6F9fwd75hv3zDfvnPfaudyka0BIT\nE1FWduVp+BaLBUajsd0Yo9GIv/zlLwCApqYmbN++HVFRUV3um3eTeE+tu3F2X/gKh8qPICs+A5kR\nwwL2e8i7mbzH3vmG/fMN++c99s433oRbRW9vNJvNKCkpQWlpKaxWKwoKCjBlypR2Y2pqalxH2lat\nWoXc3FwlS/Ibd96ZrXYJveqStR7vnfhfhGpD8FDG/UH3zDciIqLuUDSgabVaLFiwAPn5+bj33nuR\nk5ODtLQ0rFixAjt37gQA7NmzB9OmTcO0adNQXV2NJ554QsmS/EawBZR3v92MJttlzEi7BwnhcWqX\nQ0RE5NcUfw5adnY2srPbHy2aN2+ea3rq1KmYOjXwX/GzcuVfkJiYhFmzHA9c/dvfVkOSJBw8eAAN\nDfWw2Wx4/PEncNttE1WutPcdrPwaByqKMSR2EH6Qcqva5RAREfm9Pvmqp40nt+JAxeGuB3bDaKMZ\n96ffe831d9xxF1599RVXQNux42MsX/4XzJ49BxEREairq8VPf/pY0AW0ptYmvHN8E3QaHX6YmQeN\nxIcGExERdaVPBjQ13HhjBmpra3HxYhVqamoQExODhIT+ePXVl3Ho0EFoNBKqqipRU1ONuLj4rnfY\nR2w6WYBL1nrcN2QaTJHGrjcgIiKivhnQ7k+/97pHu5QyadIU7Nz5MS5evIgpU+7C9u0foq6uDm+8\nsQEajQYPPHAfWlqsvV6XWo5Vn8DnF/bihqhk3DEwuI4cEhER+YLnm3rQ5Ml3orBwO3bt2oFJk6ag\noaEBcXHx0Gg02L9/H8rLL7jGBug76j3WYrfiH8feh0bSYM6wPGg1fPUWERGRpxjQetDgwUPQ1NQE\ng8GE+PgE3HXXNBw7dgQ//vHD+OijbUhNHewa29fv4tx6+iNcbK7GHQMnYmD0DWqXQ0REFFD65ClO\nNf3972+7pmNj++G11/7W6bjt23f1Vkm97kzdOez87v9gjOiPuwfdoXY5REREAYdH0KhHtco2rD/2\nHgQE5mQ+gBAtX2BPRETUXQxo1KO2n92B8kYLslPGI73f4K43ICIiog4Y0KjHlDZcwEfndiIutB/u\nS7tb7XKIiIgCFgMa9Qi7bMeGo+/BLux4KGMWwnVhapdEREQUsBjQqEfsPP9/OFf/Hcaavoeb+g9T\nuxwiIqKAxoBGPqtoqsLW09sRpY9E3o3T1S6HiIgo4DGgkU+EEPjHsffQKrfiwaEzEBUSqXZJRERE\nAU/xgFZUVIRp06Zh6tSpWL16dYf1Fy5cwI9+9CPMmjULM2bMwK5dfff5YH3R52V7cKL2NMz9s/A9\n40i1yyEiIuoTFH1QrSzLWLRoEdauXQuj0Yi8vDxMmTIFaWlprjErV67EPffcg4ceeginTp3C448/\njh07dihZFvWQ2pY6bDxZgDBtGB7KmNXn345ARETUWxQ9glZcXIzU1FSkpKRAr9cjJycHhYWF7cZI\nkoSGhgYAwKVLl2AymZQsiXqIEAJvH9+IZnsz7k/PQb/QWLVLIiIi6jMUPYJmsViQlJTkmjeZTDh8\n+HC7Mb/4xS+Qn5+PdevWobm5GW+88YaSJVEP2V9xCIerjmJovzRMSL5F7XKIiIj6FEWPoAkhuhxT\nUFCA3Nxc7Nq1C6tWrcKvfvUrJUuiHtBgbcS7326GXqPHw5m5PLVJRETUwxQ9gpaYmIiysjLXvMVi\ngdFobDfmvffew5o1awAAo0aNQktLC6qrqxEfH3/dfRsM0T1fcBDxpX9vf/keGlob8ejIXAxPDc7X\nOfHvn/fYO9+wf75h/7zH3vUuRQOa2WxGSUkJSktLYTAYUFBQgOXLl7cbk5ycjM8//xyzZs3CqVOn\nYLVauwxnAFBZWa9U2X2ewRDtdf++rjqKT8/tQWr0AIyNGxOU3wdf+hfs2DvfsH++Yf+8x975xptw\nq2hA02q1WLBgAfLz8yGEQF5eHtLS0rBixQqYzWZMmjQJv/71r/Hiiy9i7dq10Gg0WLp0qZIlkQ8u\n25rx9vFN0EgazBmWB61Gq3ZJREREfZIkPLlQzA8xyXvP29+E3jm+CUWlX+DuQXfg3iF3KVBZYOBv\nkt5j73zD/vmG/fMee+cbb46g8U0C5JGTtWdQVPoFEiNNmDpostrlEBER9WkMaNQlq70VG47+ExIk\n/DAzD3qNomfGiYiIgh4DGnXpw7Mfo+JyFW4f8H0Mjk1VuxwiIqI+jwGNrquk/jw+LtmFhLA4TB8y\nTe1yiIiIggIDGl2TXbZjw9H3IAsZj2TmIVQbonZJREREQYEBja7p45JdON9QhvFJY5EZf6Pa5RAR\nEQUNBjTqVHljBbad/RgxIdG4Pz1H7XKIiIiCCgMadSALGRuOvQebbMPsjFmI0EeoXRIREVFQYUCj\nDj4t/RKn685itMGMUYab1C6HiIgo6DCgUTsXL9dg86ltiNCF44GhM9Uuh4iIKCgxoJGLEAJvH9+I\nFrsVuTdOR2xo919NQURERL5jQCOXPeX7caT6OIbFD8W4xJvVLoeIiChoMaARAOCStR7vn9iCEG0I\nHs64H5IkqV0SERFR0GJAIwDAP7/djEZbE2YMuRsJ4fFql0NERBTUFH/rdVFRERYvXgwhBHJzczF3\n7tx265csWYLdu3dDkiQ0NTWhpqYGe/bsUboscnOo8mvsryjGkNhUZN8wXu1yiIiIgp6iAU2WZSxa\ntAhr166F0WhEXl4epkyZgrS0NNeY559/3jW9fv16HD16VMmS6CpNrZfxzvFN0ElazMnMg0biQVUi\nIiK1KfrTuLi4GKmpqUhJSYFer0dOTg4KCwuvOX7r1q249957lSyJrrLp5FbUWetx9+A7kBhpUrsc\nIiIigsIBzWKxICkpyTVvMplQUVHR6diysjKUlpbi1ltvVbIkcnOs+gQ+v7AXKVFJuHPg7WqXQ0RE\nRE6KBjQhhMdjCwoKMHXqVN492Eta7Fb849j7kCDhh5kPQKvRql0SEREROSl6DVpiYiLKyspc8xaL\nBUajsdOx27Ztw8KFCz3et8HAh6j6ovDCTlxsrsZ9mXfh5rRhapcTcPj3z3vsnW/YP9+wf95j73qX\nogHNbDajpKQEpaWlMBgMKCgowPLlyzuMO336NC5duoRRo0Z5vO/KyvqeLDWo1EiV2PbtDhjCEzDJ\nNJG97CaDIZo98xJ75xv2zzfsn/fYO994E24VDWharRYLFixAfn4+hBDIy8tDWloaVqxYAbPZjEmT\nJgFwHD3LyclRshRyapVteG3/eggIzMnMQ4hWr3ZJREREdBVJdOdCMT/CJO+dgtPbse3sx7gt5VY8\nnHG/2uUEJP4m6T32zjfsn2/YP++xd77x5ggaH3oVROpa6vHRuZ1ICI/DzLR71C6HiIiIroEBLYjs\nrzgEu7Djvsw7Ea4LU7scIiIiugYGtCCy13IAGkmD8QNvVrsUIiIiug4GtCBR0VSFc5e+Q0ZcOvqF\nxahdDhEREV0HA1qQ+MpyEAAw1jRa5UqIiIioKwxoQUAIgb2WA9BrdBhhGK52OURERNQFBrQgcL6h\nDJamStzUP4s3BxAREQUAjwJaQUEBbDab0rWQQvZaDgAAxpo8f1MDERERqcejgLZ161ZMnjwZr776\nKiwWi9I1UQ+ShYyvLIcQrgtDVkKm2uUQERGRBzwKaCtXrsRbb70Fm82G3NxczJs3D19++aXStVEP\nOFV7BrUtdRhtMEOvUfTNXkRERNRDPL4GLSUlBc888wxWrFiB4uJi/OxnP8P06dOxb98+JesjH+11\n3r05hndvEhERBQyPDqlYrVZs27YNb731Fux2O55++mncc889KC4uxnPPPYcdO3YoXSd5wSbbcKCi\nGLEh0bgxboja5RAREZGHPApokydPxrhx4/Cb3/wGo0dfORIzZswYjB8/XrHiyDdHq79Fk+0yJg/4\nATQSb9glIiIKFB4FtI0bN8JoNHa67qWXXrrutkVFRVi8eDGEEMjNzcXcuXM7jNm2bRv++7//GxqN\nBhkZGXj55Zc9KYu6sLfccffmGN69SUREFFA8OqzywQcfoLa21jVfU1OD119/vcvtZFnGokWLsGbN\nGmzduhUFBQU4depUuzHnzp3D66+/jnfeeQdbtmzB/Pnzu/klUGeabS0orjoCY3h/DIy+Qe1yiIiI\nqBs8fg5av379XPNxcXHYunVrl9sVFxcjNTUVKSkp0Ov1yMnJQWFhYbsx7777Lh555BFERUUBAOLj\n47tTP11DcdU3aJVbMcY0CpIkqV0OERERdYNHAU0I0WGZ3W7vcjuLxYKkpCTXvMlkQkVFRbsxZ8+e\nxZkzZ/Dwww/joYcewqeffupJSdSFfa67N3l6k4iIKNB4FNAGDRqEN954A0IIyLKMv/3tbxg4cGCX\n23UW7K5mt9tRUlKCDRs24OWXX8aLL76IhoYGT8qia6i3NuBo9bcYGJ0CU2Tn1w4SERGR//LoJoEX\nXngBv/rVr7B8+XJIkoTRo0dj2bJlXW6XmJiIsrIy17zFYulws4HJZMLo0aOh0Whwww03YPDgwTh7\n9ixuuumm6+7bYIj2pPSgdODkfshCxu1pt16zT+yfb9g/77F3vmH/fMP+eY+9610eBTSTyYQ333wT\nTU1NAICIiAiPdm42m1FSUoLS0lIYDAYUFBRg+fLl7cbccccdKCgowMyZM1FdXY1z585hwIABXe67\nsrLeoxqC0c6TX0KChIzIzE77ZDBEs38+YP+8x975hv3zDfvnPfbON96EW4/f/VNfX48zZ86gpaXF\ntWzs2LHX3Uar1WLBggXIz8+HEAJ5eXlIS0vDihUrYDabMWnSJPzgBz/AZ599hpycHGi1Wjz33HOI\njY3t9hdCDhcv1+BU3VkMjUtHv1D2kYiIKBBJwoMLxbZt24alS5fi0qVLMBqNKCkpQWZmJjZt2tQb\nNXaKSb5z28/txOZTH2JOZh4mJN/S6Rj+JuQb9s977J1v2D/fsH/eY+98480RNI9uEnjttdewceNG\npKam4qOPPsLrr78Os9nc7U9GyttnOQidpMUow/Wv4SMiIiL/5VFA0+l0SEhIcD1a4/vf/z4OHz6s\naGHUfWUN5ShtuIDhCZmI0Ht2nSARERH5H4+uQQsJCYEQAqmpqVi3bh1SUlJcNwyQ/9hrcb7aKXF0\nFyOJiIjIn3kU0J566ik0NDTg2Wefxe9+9zvU19dj4cKFStdG3SCEwD7LQYRqQ3BTwjC1yyEiIiIf\ndBnQ2h4kO378eERHR2Pt2rW9UBZ115lL51DdXINxiTcjRKtXuxwiIiLyQZfXoGm1Wrzzzju9UQv5\nYG85X+1ERETUV3h0k8C4cePwr3/9S+layEt22Y79FYcQpY9ERly62uUQERGRjzy6Bm3Tpk144403\nEBYWhvDwcAghIEkSvvjiC6XrIw8cqzmJhtZGTLxhArQardrlEBERkY88Cmjvv/++0nWQD/a13b1p\n4t2bREREfYFHAS0lJUXpOshLVrsVhyq/RkJYHAbHDFS7HCIiIuoBHgW0W2+9FZIkdVjOU5zqO1x1\nFC12K26/4bZOv0dEREQUeLp9irOlpQVbtmyBTufxe9ZJQfssvHuTiIior/HoLs6UlBTXnyFDhuCp\np57Crl27lK6NutDU2oRvLh5DSlQSkqMS1S6HiIiIeohHAe1q3333HS5evNjTtVA3Hag8DLuw8+gZ\nERFRH9Pta9BkWYbNZsMLL7zg0ScoKirC4sWLIYRAbm4u5s6d2279pk2bsGzZMiQmOo4AzZkzB3l5\ned35GoLWPufDaW82MqARERH1Jd2+Bk2n06F///7Qart+3pYsy1i0aBHWrl0Lo9GIvLw8TJkyBWlp\nae3G5eTk4MUXX+xm6cGttqUOJ2pPIy12EBLC49Quh4iIiHqQR6c4GxsbERcXh5SUFJhMJrS0tODE\niRNdbldcXIzU1FSkpKRAr9cjJycHhYWFHcYJIbpfeZD7ynIIAoLPPiMiIuqDPApov/nNb6DXX3kB\nt06nw69//esut7NYLEhKSnLNm0wmVFRUdBi3fft2zJgxA0899RTKy8s9KSno7bMcgEbS4HvGEWqX\nQkRERD3Ds5USAAAXi0lEQVTMo4Bmt9vbBbSQkBDY7fYut/PkyNjkyZOxY8cObN68GePHj/co+AU7\nS2MFSupLkRU/FFEhkWqXQ0RERD3Mo2vQdDodvvvuOwwYMAAAUFJS4tE1aImJiSgrK3PNWywWGI3G\ndmNiY2Nd0w8++CBefvlljwo3GKI9GtcX7bQ4HnEy+cbxXvchmPvXE9g/77F3vmH/fMP+eY+9610e\nBbRf/OIXePjhhzFx4kQAwK5du/DHP/6xy+3MZjNKSkpQWloKg8GAgoICLF++vN2YyspKGAwGAEBh\nYSHS09M9Kryyst6jcX2NEAJFp3cjRKNHasgQr/pgMEQHbf96AvvnPfbON+yfb9g/77F3vvEm3HoU\n0CZNmoT169fjs88+AwDMnTsXqampXW6n1WqxYMEC5OfnQwiBvLw8pKWlYcWKFTCbzZg0aRLWrVuH\nHTt2QKfTITY2FkuWLOn2FxFMSurPo+JyFcaYRiFMF6p2OURERKQASXhwoVh1dTWioqIQEhICAGht\nbUV9fT3i4+MVL/BagjXJv39iC3Z89ymeGPEfMPfP8mof/E3IN+yf99g737B/vmH/vMfe+cabI2ge\n3STw05/+tN1NAa2trXjiiSe6/cnIN7KQ8ZXlICJ1ERgWP1TtcoiIiEghHgU0q9WK8PBw13xERARa\nWloUK4o6923NKdRZ6zHaaIZOw5fVExER9VUev4uzurraNX3x4kXIsqxIQXRt+yyOVzvx4bRERER9\nm0eHYR599FE8/PDDmDFjBoQQ+N///V88/vjjStdGblrtrThYeRj9QmOR1m+Q2uUQERGRgjwKaHl5\neRg4cCA++eQTAMBLL72EMWPGKFkXXeWb6uO4bGvG95PHQSN5fOCTiIiIApBHAa2+vh6ffvopTpw4\ngebmZnz99dcAgDfffFPR4uiKfeUHAPD0JhERUTDw6FDM/PnzodVqcfbsWcyePRtarRYjRvAdkL3l\nsq0Zhy8eRWKEETdEJXW9AREREQU0jwLauXPn8PTTTyMsLAz33nsvVq1ahX379ildGzkdqvwaNtmG\nMabRkCRJ7XKIiIhIYR4FtLYH1Or1etTW1kKv17e7q5OUdeXuzVEqV0JERES9waNr0AYNGoTa2lpM\nnz4ds2fPRnR0NIYPH650bQTgkrUex6pPYFDMQBgiEtQuh4iIiHqBRwHt5ZdfBgA89thjMJvNqK+v\nR3Z2tqKFkcN+SzEEBI+eERERBZFuP46ej9foXfssByBBwveMI9UuhYiIiHoJH6jlx6ouX8SZSyXI\niEtHbGj3X7RKREREgUnxgFZUVIRp06Zh6tSpWL169TXH/etf/0JmZia++eYbpUsKGK6bAxL57DMi\nIqJgomhAk2UZixYtwpo1a7B161YUFBTg1KlTHcY1NjZi/fr1GDWK11m1EUJgb/kB6DQ6jDLwhgwi\nIqJgomhAKy4uRmpqKlJSUqDX65GTk4PCwsIO41599VU8/vjj0Ov1SpYTUEobLqC8qQLmhGEI14Wr\nXQ4RERH1IkUDmsViQVLSlSffm0wmVFRUtBtz9OhRlJeXY+LEiUqWEnB4epOIiCh4dfsuzu4QQnS5\nfvHixVi6dKnH2wQDWcjYZzmIcF0YhsdnqF0OERER9TJFA1piYiLKyspc8xaLBUaj0TXf2NiIkydP\n4tFHH4UQAlVVVfj5z3+OlStXdvkgXIOh797VeLTyBGpaajFp8AQkJ8Yr8jn6cv96A/vnPfbON+yf\nb9g/77F3vUvRgGY2m1FSUoLS0lIYDAYUFBRg+fLlrvVRUVH44osvXPOPPvoonn/+eWRlZXW578rK\nekVq9gf/Pv45AOCm2OGKfJ0GQ3Sf7p/S2D/vsXe+Yf98w/55j73zjTfhVtGAptVqsWDBAuTn50MI\ngby8PKSlpWHFihUwm82YNGlSu/GSJAX9KU6bbMMBSzFiQqIxNC5N7XKIiIhIBYoGNADIzs7u8Fqo\nefPmdTr2zTffVLocv3es+gQabU2YNOA2aCQ+R5iIiCgYMQH4mb2WAwCAsSbevUlERBSsGND8SLOt\nBcWV38AQnoCB0TeoXQ4RERGphAHNjxyuOgKr3IoxptGQJEntcoiIiEglDGh+ZJ/z9OYYE195RURE\nFMwY0PxEg7URR6q/xYDoFCRGGrvegIiIiPosBjQ/caCyGLKQefSMiIiIGND8xd7yg5Ag4WbjSLVL\nISIiIpUxoPmB6uYanKo7g/R+gxEX1k/tcoiIiEhlDGh+4CvLIQB89hkRERE5MKD5gb2WA9BKWowy\nmtUuhYiIiPwAA5rKyhrKUdpwAVkJGYjUR6hdDhEREfkBBjSVfWU5CAAYy7s3iYiIyIkBTUVCCOy1\nHESoNgTm/llql0NERER+ggFNRWcvleBiczVGGm5CiDZE7XKIiIjITyge0IqKijBt2jRMnToVq1ev\n7rD+7bffxvTp0zFz5kzMmTMHp06dUrokv7HXeXpzDO/eJCIiIjeKBjRZlrFo0SKsWbMGW7duRUFB\nQYcANn36dGzZsgUffPABfvKTn2DJkiVKluQ37LId+y2HEKWPRGZcutrlEBERkR9RNKAVFxcjNTUV\nKSkp0Ov1yMnJQWFhYbsxkZGRrummpiZoNMFx1vXbmlOob23A94wjodVo1S6HiIiI/IhOyZ1bLBYk\nJSW55k0mEw4fPtxh3IYNG7B27VrYbDb8/e9/V7Ikv7HXcgAAMDaRd28SERFRe4oerhJCeDRuzpw5\n+Pe//41nn30Wf/3rX5UsyS9Y7a04VPk1EsLiMDgmVe1yiIiIyM8oegQtMTERZWVlrnmLxQKj0XjN\n8ffccw8WLlzo0b4Nhmif61PLl9/tR7O9BdOG3g6jMUaVGgK5f/6A/fMee+cb9s837J/32LvepWhA\nM5vNKCkpQWlpKQwGAwoKCrB8+fJ2Y86dO4fUVMdRpJ07d2LQoEEe7buysr6ny+01hSe+AAAMjx6u\nytdhMEQHdP/Uxv55j73zDfvnG/bPe+ydb7wJt4oGNK1WiwULFiA/Px9CCOTl5SEtLQ0rVqyA2WzG\npEmTsH79enzxxRfQ6/WIiYnB0qVLlSxJdU2tTfim6iiSIxORHJWodjlERETkhxQNaACQnZ2N7Ozs\ndsvmzZvnmn7hhReULsGvHKz8GjZhx1g++4yIiIiuITieaeFH2h5Oe7NppMqVEBERkb9iQOtFtS11\nOFFzCkNiByEhPF7tcoiIiMhPMaD1ov2WQxAQGGvis8+IiIjo2hjQetFey0FoJA1GG0eoXQoRERH5\nMQa0XmJpqkRJ/Xlkxt+I6JAotcshIiIiP8aA1kv2OW8O4N2bRERE1BUGtF4ghMA+ywHoNXqM6J+l\ndjlERETk5xjQesF39aWoaKrCiP5ZCNOFqV0OERER+TkGtF6w13IAADCGd28SERGRBxjQFCYLGV9Z\nDiFCF46shAy1yyEiIqIAwICmsJO1p1FnvYTRxhHQaRR/sxYRERH1AQxoCttb3nb3Jk9vEhERkWcY\n0BTUKttwoPIw+oXGIq3fYLXLISIiogDBgKagIxeP47LtMm42jYRGYquJiIjIM4qnhqKiIkybNg1T\np07F6tWrO6xfu3YtcnJyMGPGDDz22GO4cOGC0iX1mn3Ouzf5cFoiIiLqDkUDmizLWLRoEdasWYOt\nW7eioKAAp06dajcmKysLGzduxObNm3HXXXdh2bJlSpbUa5ptzThcdQSmCCNuiEpWuxwiIiIKIIoG\ntOLiYqSmpiIlJQV6vR45OTkoLCxsN+aWW25BaGgoAGDUqFGwWCxKltRrDlV+g1bZhrGmUZAkSe1y\niIiIKIAoGtAsFguSkpJc8yaTCRUVFdcc/9577yE7O1vJknpN27s3b+bdm0RERNRNij6YSwjh8djN\nmzfjm2++wbp16zwabzBEe1uW4uqaL+FYzQmkxw/C8FT/vHvTn/sXCNg/77F3vmH/fMP+eY+9612K\nBrTExESUlZW55i0WC4xGY4dxn3/+OVavXo3169dDr9d7tO/Kyvoeq7On7Tr/BWQhY1TCCL+s02CI\n9su6AgX75z32zjfsn2/YP++xd77xJtwqeorTbDajpKQEpaWlsFqtKCgowJQpU9qNOXLkCBYuXIiV\nK1ciLi5OyXJ6zd7yA5Ag4XvGEWqXQkRERAFI0SNoWq0WCxYsQH5+PoQQyMvLQ1paGlasWAGz2YxJ\nkybhz3/+My5fvoynnnoKQggkJyfjr3/9q5JlKarq8kWcuXQOmXE3IjY0Ru1yiIiIKAAp/nLI7Ozs\nDhf+z5s3zzX9xhtvKF1Cr9pnOQQAGMObA4iIiMhLfHt3D2ixW3Gy9gyOV5/A7vKvoNPoMMp4k9pl\nERERUYBiQPOCXbbjXP15HK8+gWM1J3CmrgR2YQcA6DQ63D3oDoTrwlWukoiIiAIVA5oHhBAob6rA\nseoTOF5zAidqTqPZ3gIAkCBhQHQKMuNvREZcOobEDkKI1rM7UYmIiIg6w4B2DTXNtThecxLHqk/i\n25oTqLNeub3YGN4fY+JHIzPuRgyNS0OkPkLFSomIiKivYUBzamq9jBO1p3Cs+iSO15yEpenKGw+i\n9VEYYxqFjDjHUbKE8L7xOBAiIiLyT0Eb0FplG87UncXx6pM4VnMS5y59BwHHmw9CtCEYnpCJzLh0\nZMTfiOTIRL5Pk4iIiHpN0AQ0Wcg431CG484jZCdrz6BVbgUAaCQNBsemugLZoJgB0GmCpjVERETk\nZ/p0Cqm6fBHHqk/gWM1JfFtzEo2tTa51yZGJyIhPR2bcjUjvNxhhujAVKyUiIiK6ok8FtHprA751\nXth/vOYELjbXuNbFhfaDOSnLeWF/OmJD+dJXIiIi8k8BHdDcHxB7rOYEShsuuNaF68IxynATMpyn\nLY3h/XkdGREREQWEgAxo73+zDfvPf4PTdefaPSA2Iy4dGXHpyIy/EQOiU6CRFH0XPBEREZEiAjKg\nvfP1FtcDYtsCGR8QS0RERH1FQAa0X932BAxSIh8QS0RERH2S4ucAi4qKMG3aNEydOhWrV6/usH7f\nvn24//77MXz4cGzfvt2jfY5NGclwRkRERH2WogFNlmUsWrQIa9aswdatW1FQUIBTp061G5OcnIw/\n/elPmD59upKlEBEREQUMRU9xFhcXIzU1FSkpKQCAnJwcFBYWIi0tzTUmOTkZAHiHJREREZGTokfQ\nLBYLkpKSXPMmkwkVFRXX2YKIiIiIFA1oQggld09ERETUJyl6ijMxMRFlZWWueYvFAqPR2CP7Nhj4\nJgBfsH++Yf+8x975hv3zDfvnPfaudyl6BM1sNqOkpASlpaWwWq0oKCjAlClTrjmeR9yIiIiIAEko\nnIqKiorw0ksvQQiBvLw8zJ07FytWrIDZbMakSZNw+PBh/OIXv8ClS5cQGhoKg8GALVu2KFkSERER\nkV9TPKARERERUffwZZVEREREfoYBjYiIiMjPMKARERER+ZmAC2hdvduTOldeXo4f/ehHuOeeezB9\n+nS8+eabapcUkGRZxqxZs/DEE0+oXUrAqa+vx7x583D33XcjJycHhw4dUrukgLF27Vrce++9mD59\nOp555hlYrVa1S/Jr8+fPx4QJE9q9QrCurg75+fmYOnUqfvKTn6C+vl7FCv1bZ/1btmwZ7r77bsyY\nMQNPPvkkGhoaVKzQv3XWvzZr1qxBZmYmamtru9xPQAU0T97tSZ3TarV4/vnnsW3bNrz99tvYsGED\ne+eFN998s92ryshzL730EiZOnIgPP/wQmzdvZh89ZLFYsG7dOmzcuBFbtmyB3W7Htm3b1C7Lr91/\n//1Ys2ZNu2WrV6/G+PHj8dFHH2HcuHFYtWqVStX5v876d9ttt6GgoACbN29Gamoq+3cdnfUPcBwo\n+fzzz12vuOxKQAU093d76vV617s9qWsGgwHDhg0DAERGRiItLY2v3eqm8vJy7Nq1Cw888IDapQSc\nhoYG7Nu3D7m5uQAAnU6HqKgolasKHLIs4/Lly7DZbGhubu6xB373VWPGjEFMTEy7ZYWFhZg1axYA\nYNasWfj444/VKC0gdNa/CRMmQKNxRIZRo0ahvLxcjdICQmf9A4DFixfjueee83g/ARXQ+G7PnnH+\n/HkcO3YMI0aMULuUgNL2P5ckSWqXEnDOnz+PuLg4PP/885g1axYWLFiA5uZmtcsKCCaTCY899hhu\nv/12ZGdnIzo6GhMmTFC7rIBTXV2N/v37A3D8wlpTU6NyRYHrvffeQ3Z2ttplBJQdO3YgKSkJGRkZ\nHm8TUAGNj2zzXWNjI+bNm4f58+cjMjJS7XICxieffIL+/ftj2LBh/HvoBZvNhiNHjuCRRx7Bpk2b\nEBYWxmtIPXTp0iUUFhZi586d+PTTT9HU1MSHeZNqVq5cCb1e3+n1VdS55uZmvPbaa3jyySddyzz5\nORJQAU3Jd3sGA5vNhnnz5mHGjBm444471C4noOzfvx87duzAlClT8Mwzz2D37t3dOlQd7BITE5GY\nmAiz2QwAmDp1Ko4cOaJyVYHh888/x4ABA9CvXz9otVrceeedOHDggNplBZyEhARUVVUBACorKxEf\nH69yRYFn06ZN2LVrF1555RW1Swkoba+8nDFjBiZPngyLxYLc3FxcvHjxutsFVEDr7rs9qb358+cj\nPT0dP/7xj9UuJeD88pe/xCeffILCwkIsX74c48aNw7Jly9QuK2D0798fSUlJOHPmDADgyy+/5E0C\nHkpOTsahQ4fQ0tICIQR756Grj1BMnjwZGzduBOAIGvzZcX1X96+oqAivv/46Vq5ciZCQEJWqChzu\n/Rs6dCg+++wzFBYWYseOHTCZTNi0aRMSEhKuuw+d0kX2JK1WiwULFiA/P9/1bk/+Q+WZr776Clu2\nbMHQoUMxc+ZMSJKE//zP/+R1BNRrXnzxRTz77LOw2WwYMGAAlixZonZJAWHEiBGYOnUqZs6cCZ1O\nh6ysLDz44INql+XX2o5y19bW4vbbb8eTTz6JuXPn4qmnnsL777+P5ORkvPrqq2qX6bc669+qVavQ\n2tqK/Px8AMDIkSPxu9/9Tt1C/VRn/Wu7QQoAJEny6BQn38VJRERE5GcC6hQnERERUTBgQCMiIiLy\nMwxoRERERH6GAY2IiIjIzzCgEREREfkZBjQiIiIiP8OARkTUTXv27Gn3XCMiop7GgEZE5AVJktQu\ngYj6sIB6kwARUVeKi4vx8ssvo7GxEQAwb948pKenIzc3F7NmzcJnn30GAPjtb3+LMWPGAAA++OAD\nrFmzBhqNBgMHDsTvf/9717saV61aha1bt0Kj0SAiIgJvvfUWAMe7bX/729/i4MGD0Gg0WL58OYYM\nGaLCV0xEfZIgIuojLl26JGbOnCkqKyuFEEJUVFSI7OxscfToUZGRkSE2b94shBBiz549Ijs7W1it\nVnH8+HFx2223iaqqKiGEEP/1X/8lnn76aSGEEBs3bhSzZ88WjY2NQgghamtrhRBC7N69WwwfPlwc\nPXpUCCHEypUrxbPPPturXysR9W08gkZEfcb+/ftx/vx5PP7446533Wm1WthsNoSEhOC+++4DAIwd\nOxZhYWE4c+YM9uzZg9tvv9314uKHHnoIM2bMAAB88sknePjhhxEREQEAiI2NdX2uwYMHIzMzE4Dj\nvYSffPJJb32ZRBQEGNCIqE/JzMzEunXr2i0rLS3tME4I4Xppsfv1ZO7z4jqvKg4NDXVNt4VAIqKe\nwpsEiKjPGD16NM6ePYvdu3e7lh0+fBhCCFitVmzZsgUAsG/fPlitVgwePBjjx4/Hrl27cPHiRQDA\nu+++iwkTJgAAJk+ejLfeest1PVttbW0vf0VEFKx4BI2I+oyYmBisXLkSS5cuxZIlS2C1WjFw4EC8\n8MIL6NevH44ePYr/+Z//AQAsX74cOp0O6enp+OUvf4n/+I//gEajwYABA/CHP/wBADBz5kxUVFRg\n9uzZ0Gq1iIqKwoYNG9T8EokoSEjiesfwiYj6gNLSUuTm5uLLL79UuxQiIo/wFCcRBQU+t4yIAgmP\noBERERH5GR5BIyIiIvIzDGhEREREfoYBjYiIiMjPMKARERER+RkGNCIiIiI/w4BGRERE5Gf+P4mQ\n0F25DEnsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x55d2d3e517d0>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5h 23min 49s, sys: 16min 2s, total: 5h 39min 51s\n",
      "Wall time: 14min 30s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "## UNFORTUNATELY THIS WILL ALSO NOT WORK ON THE VM's, YOU'LL NEED TO GET A GPU INSTANCE TO RUN THIS!\n",
    "\n",
    "# Create a ConvNet classifier with 2 CONV-RELU-POOL layers, with filter sizes of \n",
    "# 5 and 5 and 32 and 64 output features.\n",
    "model_params = {\n",
    "        'input_size': 784,\n",
    "        'output_size': 10,\n",
    "        'filter_sizes': [5, 5],\n",
    "        'output_depths': [32, 64],\n",
    "        'use_batch_norm': True,\n",
    "}\n",
    "\n",
    "training_params = {\n",
    "        'keep_prob': 1.0,    # Switch off dropout\n",
    "        'num_epochs': 15,\n",
    "        'batch_size': 50,\n",
    "        'stop_early': False,\n",
    "}\n",
    "\n",
    "trained_model, training_results = build_train_eval_and_plot(\n",
    "        model_params, \n",
    "        training_params, \n",
    "        verbose=True\n",
    ")\n",
    "\n",
    "# QUESTION: Try experimenting with different archictures and hyperparameters and \n",
    "# see how well you can classify MNIST digits! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2SgtWksRT4hJ"
   },
   "source": [
    "# NB: Before you go (5min)\n",
    "\n",
    "Pair up with someone else and go through the questions in \"Learning Objectives\" at the top. Take turns explaining each of these to each other, and be sure to ask the tutors if you're both unsure!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lxEVtIbcovwl"
   },
   "source": [
    "# Additional resources\n",
    "\n",
    "* Chip Huyen's wonderful script: https://github.com/chiphuyen/stanford-tensorflow-tutorials/blob/master/examples/07_convnet_mnist.py\n",
    "* NVidia's ConvNet Tutorial: https://github.com/alrojo/tensorflow-tutorial/blob/master/lab2_CNN/lab2_CNN.ipynb\n",
    "* https://github.com/DeepLearningDTU/nvidia_deep_learning_summercamp_2016/blob/master/lab2/lab2_CNN.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5lJYoEZV32p_"
   },
   "source": [
    "# Feedback\n",
    "\n",
    "Please send any bugs and comments to dli-practicals@googlegroups.com."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "last_runtime": {
    "build_target": "//learning/deepmind/dm_python:dm_notebook",
    "kind": "private"
   },
   "name": "Prac 3: Deep ConvNets (Solution)",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}