GestureRecognitionTensorFlow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's this TensorFlow business?\n",
    "\n",
    "You've written a lot of code in this assignment to provide a whole host of neural network functionality. Dropout, Batch Norm, and 2D convolutions are some of the workhorses of deep learning in computer vision. You've also worked hard to make your code efficient and vectorized.\n",
    "\n",
    "For the last part of this assignment, though, we're going to leave behind your beautiful codebase and instead migrate to one of two popular deep learning frameworks: in this instance, TensorFlow (or PyTorch, if you switch over to that notebook)\n",
    "\n",
    "#### What is it?\n",
    "TensorFlow is a system for executing computational graphs over Tensor objects, with native support for performing backpropogation for its Variables. In it, we work with Tensors which are n-dimensional arrays analogous to the numpy ndarray.\n",
    "\n",
    "#### Why?\n",
    "\n",
    "* Our code will now run on GPUs! Much faster training. Writing your own modules to run on GPUs is beyond the scope of this class, unfortunately.\n",
    "* We want you to be ready to use one of these frameworks for your project so you can experiment more efficiently than if you were writing every feature you want to use by hand. \n",
    "* We want you to stand on the shoulders of giants! TensorFlow and PyTorch are both excellent frameworks that will make your lives a lot easier, and now that you understand their guts, you are free to use them :) \n",
    "* We want you to be exposed to the sort of deep learning code you might run into in academia or industry. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How will I learn TensorFlow?\n",
    "\n",
    "TensorFlow has many excellent tutorials available, including those from [Google themselves](https://www.tensorflow.org/get_started/get_started).\n",
    "\n",
    "Otherwise, this notebook will walk you through much of what you need to do to train models in TensorFlow. See the end of the notebook for some links to helpful tutorials if you want to learn more or need further clarification on topics that aren't fully explained here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Datasets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import math\n",
    "import timeit\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n",
      "done collecting training data\n",
      "done collecting testing data\n",
      "training\n",
      "(1574, 76, 66, 3)\n",
      "(1574,)\n",
      "testing\n",
      "(382, 76, 66, 3)\n",
      "(382,)\n",
      "num_training 1574\n",
      "num_training 1514\n",
      "num_validation 60\n",
      "num_test 100\n",
      "(1574, 76, 66, 3)\n",
      "<class 'numpy.float64'>\n",
      "X_train:  (1514, 76, 66, 3)\n",
      "y_train:  (1514,)\n",
      "X_val:  (60, 76, 66, 3)\n",
      "y_val:  (60,)\n",
      "X_test:  (100, 76, 66, 3)\n",
      "y_test:  (100,)\n"
     ]
    }
   ],
   "source": [
    "# from cs231n.data_utils import load_CIFAR10\n",
    "from cs231n.data_utils import get_MARCEL_data\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))\n",
    "\n",
    "data = get_MARCEL_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)\n",
    "\n",
    "X_train = data['X_train']\n",
    "y_train = data['y_train']\n",
    "X_val = data['X_val']\n",
    "y_val = data['y_val']\n",
    "X_test = data['X_test']\n",
    "y_test = data['y_test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Model\n",
    "\n",
    "### Some useful utilities\n",
    "\n",
    ". Remember that our image data is initially N x H x W x C, where:\n",
    "* N is the number of datapoints\n",
    "* H is the height of each image in pixels\n",
    "* W is the height of each image in pixels\n",
    "* C is the number of channels (usually 3: R, G, B)\n",
    "\n",
    "This is the right way to represent the data when we are doing something like a 2D convolution, which needs spatial understanding of where the pixels are relative to each other. When we input image data into fully connected affine layers, however, we want each data example to be represented by a single vector -- it's no longer useful to segregate the different channels, rows, and columns of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The example model itself\n",
    "\n",
    "The first step to training your own model is defining its architecture.\n",
    "\n",
    "Here's an example of a convolutional neural network defined in TensorFlow -- try to understand what each line is doing, remembering that each layer is composed upon the previous layer. We haven't trained anything yet - that'll come next - for now, we want you to understand how everything gets set up. \n",
    "\n",
    "In that example, you see 2D convolutional layers (Conv2d), ReLU activations, and fully-connected layers (Linear). You also see the Hinge loss function, and the Adam optimizer being used. \n",
    "\n",
    "Make sure you understand why the parameters of the Linear layer are 5408 and 10.\n",
    "\n",
    "### TensorFlow Details\n",
    "In TensorFlow, much like in our previous notebooks, we'll first specifically initialize our variables, and then our network model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# clear old variables\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# setup input (e.g. the data that changes every batch)\n",
    "# The first dim is None, and gets sets automatically based on batch size fed in\n",
    "X = tf.placeholder(tf.float32, [None, 76, 66, 3])\n",
    "y = tf.placeholder(tf.int64, [None])\n",
    "is_training = tf.placeholder(tf.bool)\n",
    "\n",
    "def simple_model(X,y):\n",
    "    # define our weights (e.g. init_two_layer_convnet)\n",
    "    \n",
    "    # setup variables\n",
    "    Wconv1 = tf.get_variable(\"Wconv1\", shape=[7, 7, 3, 32])\n",
    "    bconv1 = tf.get_variable(\"bconv1\", shape=[32])\n",
    "    W1 = tf.get_variable(\"W1\", shape=[33600, 10])\n",
    "    b1 = tf.get_variable(\"b1\", shape=[10])\n",
    "\n",
    "    # define our graph (e.g. two_layer_convnet)\n",
    "    a1 = tf.nn.conv2d(X, Wconv1, strides=[1,2,2,1], padding='VALID') + bconv1\n",
    "    h1 = tf.nn.relu(a1)\n",
    "    h1_flat = tf.reshape(h1,[-1,33600])\n",
    "    y_out = tf.matmul(h1_flat,W1) + b1\n",
    "    return y_out\n",
    "\n",
    "y_out = simple_model(X,y)\n",
    "\n",
    "# define our loss\n",
    "total_loss = tf.losses.hinge_loss(tf.one_hot(y,10),logits=y_out)\n",
    "mean_loss = tf.reduce_mean(total_loss)\n",
    "\n",
    "# define our optimizer\n",
    "optimizer = tf.train.AdamOptimizer(5e-4) # select optimizer and set learning rate\n",
    "train_step = optimizer.minimize(mean_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow supports many other layer types, loss functions, and optimizers - you will experiment with these next. Here's the official API documentation for these (if any of the parameters used above were unclear, this resource will also be helpful). \n",
    "\n",
    "* Layers, Activations, Loss functions : https://www.tensorflow.org/api_guides/python/nn\n",
    "* Optimizers: https://www.tensorflow.org/api_guides/python/train#Optimizers\n",
    "* BatchNorm: https://www.tensorflow.org/api_docs/python/tf/contrib/layers/batch_norm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the model on one epoch\n",
    "While we have defined a graph of operations above, in order to execute TensorFlow Graphs, by feeding them input data and computing the results, we first need to create a `tf.Session` object. A session encapsulates the control and state of the TensorFlow runtime. For more information, see the TensorFlow [Getting started](https://www.tensorflow.org/get_started/get_started) guide.\n",
    "\n",
    "Optionally we can also specify a device context such as `/cpu:0` or `/gpu:0`. For documentation on this behavior see [this TensorFlow guide](https://www.tensorflow.org/tutorials/using_gpu)\n",
    "\n",
    "You should see a validation loss of around 0.4 to 0.6 and an accuracy of 0.30 to 0.35 below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training\n",
      "Iteration 0: with minibatch training loss = 5.55 and accuracy of 0.2\n",
      "Epoch 1, Overall loss = 3.11 and accuracy of 0.529\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAHwCAYAAADjOch3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl4VPXd///ne7JCyEISDGFPAFFERbYguMStaq3aWtda\ntValrb27/Hp30bv93tred1urvbvYVluqtlitS2utWre6xQUQBFGUTULYSdgCE0KYrJ/fH3OCEYGE\nMGfOTPJ6XNdcmTlzZs47HCmvflZzziEiIiIiiSEUdAEiIiIi8iGFMxEREZEEonAmIiIikkAUzkRE\nREQSiMKZiIiISAJROBMRERFJIApnItKrmJkzs1FB1yEiciAKZyISGDNbY2Z7zKy+w+O3QdfVzszG\nmdnzZrbNzDpdFFLBT0RiQeFMRIJ2vnOuX4fHfwRdUAfNwKPAdUEXIiK9h8KZiCQkM/uCmc02s9+a\nWdjMlpvZGR3eH2RmT5pZrZlVmtkNHd5LMbP/MrNVZrbLzBaa2dAOX3+mma00s51m9jszs/3V4Jxb\n4Zy7F1hymL9LyMx+YGZrzWyLmd1vZrnee5lm9oCZbffqecvMijr8GVR5v8NqM7vycOoQkeSgcCYi\niawMWAUUArcA/zCzfO+9h4ENwCDgYuAnZna69963gCuATwI5wBeBhg7f+ylgMnAccClwtr+/Bl/w\nHqcBpUA/oL379hogFxgKFABfBvaYWRZwJ3Cucy4bmAa843OdIpIAFM5EJGj/9FqM2h83dHhvC/Ar\n51yzc+4RYAVwntcKNh34nnMu4px7B7gHuNr73PXAD7yWL+ece9c5t73D997mnNvpnFsHvAKM9/l3\nvBL4hXOuyjlXD9wMXG5mqUS7TguAUc65VufcQudcnfe5NmCcmfVxzlU75w6rBU9EkoPCmYgE7dPO\nubwOjz92eG+jc67jQPy1RFvKBgG1zrld+7w32Hs+lGiL24HUdHjeQLQly0+DiNbXbi2QChQBfwGe\nBx42s01mdruZpTnndgOXEW1Jqzazp83sKJ/rFJEEoHAmIols8D7jwYYBm7xHvpll7/PeRu/5emBk\nfErskk3A8A6vhwEtwGavVfCHzrmxRLsuP4XXAuice945dxZQDCwH/oiI9HgKZyKSyI4Avm5maWZ2\nCXA08Ixzbj0wB/ipN6D+OKIzKh/wPncP8D9mNtqijjOzgkO9uPfZTCDde51pZhmdfCzdO6/9kQI8\nBPx/ZlZiZv2AnwCPOOdazOw0MzvWO6+OaDdnm5kVmdmF3tizRqCeaDeniPRwqUEXICK93lNm1trh\n9QvOuc94z+cBo4FtwGbg4g5jx64Afk+0VWoHcItz7kXvvV8AGcC/iU4mWA60f+ehGA6s7vB6D9Eu\nyREH+cy+48JuAO4j2rX5GpBJtBvza977A73fYwjRAPYI0a7OAUQnNtwPOKKTAb7Sjd9BRJKMfXQ4\nh4hIYjCzLwDXO+dOCroWEZF4UremiIiISAJROBMRERFJIOrWFBEREUkgajkTERERSSAKZyIiIiIJ\nJKmX0igsLHQjRozw9Rq7d+8mKyvL12uIP3TvkpPuW3LSfUteunfxs3Dhwm3OuQGdnZfU4WzEiBEs\nWLDA12tUVFRQXl7u6zXEH7p3yUn3LTnpviUv3bv4MbO1nZ+lbk0RERGRhKJwJiIiIpJAFM5ERERE\nEojCmYiIiEgCUTgTERERSSAKZyIiIiIJROFMREREJIEonImIiIgkEIUzERERkQSicCYiIiKSQBTO\nRERERBKIwpmIiIhIAlE4ExEREUkgCmciIiIiCUThTERERCSBKJyJiIiIJBCFs4A454IuQURERBKQ\nwlkAKrfsYtwtz7OiZlfQpYiIiEiCUTgLwJJNdexuauWFpTVBlyIiIiIJRuEsADXhCACzK7cHXImI\niIgkGoWzAFR74Wzhuh1EmlsDrkZEREQSicJZANpbzppa2liwZkfA1YiIiEgiUTgLQHVdhInD+5Ma\nMmav2hZ0OSIiIpJAUoMuoDeqCe/h1CMHADBnlcadiYiIyId8azkzszFm9k6HR52ZfdPM8s3sBTNb\n6f3s751vZnanmVWa2WIzm+BXbUFqaW1j665GBub2YfrIAt7bsJPwnuagyxIREZEE4Vs4c86tcM6N\nd86NByYCDcDjwE3AS8650cBL3muAc4HR3mMGcLdftQVpa30jbQ6KczOZNqqQNgfzqtR6JiIiIlHx\nGnN2BrDKObcWuBCY5R2fBXzae34hcL+LehPIM7PiONUXN+0zNQfmZnLCsDwy00Lq2hQREZG94hXO\nLgce8p4XOeeqvec1QJH3fDCwvsNnNnjHepT2mZrFuZlkpKYweUQ+sys1KUBERESifJ8QYGbpwAXA\nzfu+55xzZnZIm0ya2Qyi3Z4UFRVRUVERizIPqL6+PqbXeH1NdHxZ5eKF1Cw3iq2J17c088/nXiYv\nU5NnYynW907iQ/ctOem+JS/du8QTj9ma5wJvO+c2e683m1mxc67a67bc4h3fCAzt8Lkh3rGPcM7N\nBGYCTJo0yZWXl/tWOEBFRQWxvMbsp5eSuWot551VjplRMCrMox+8AUVjKD+hxzUUBirW907iQ/ct\nOem+JS/du8QTj6aaK/iwSxPgSeAa7/k1wBMdjl/tzdqcCoQ7dH/2GNXhCMW5fTAzAMYOyiEnM5U5\nWu9MRERE8LnlzMyygLOAL3U4fBvwqJldB6wFLvWOPwN8EqgkOrPzWj9rC0pNOMLAnMy9r1NCxokj\nC5hduR3n3N7QJiIiIr2Tr+HMObcbKNjn2Haiszf3PdcBX/WznkRQUxdhyoj8jxybPqqQ55dsZl1t\nA8MLsgKqTERERBKBRqDHUVubY3NdhIG5mR85Pm1kIQCzK7WkhoiISG+ncBZH23c30dzqKN4nnI0c\nkEVRTob22RQRERGFs3hqX+OsKOej4czMmD6ykLmrttPWdkgri4iIiEgPo3AWR9XhPQAU5/b52HvT\nRhVSu7uJ5TW74l2WiIiIJBCFsziqqftw66Z9TRsZnTehJTVERER6N4WzOKoOR0hLMQqy0j/23qC8\nPpQUZmmfTRERkV5O4SyONocjFOVkEgrtfy2zaSMLmFe1nebWtjhXJiIiIolC4SyOorsDfLxLs930\nUYXsbmpl8YadcaxKREREEonCWRzV1EUYuJ/JAO1OLC3ATOudiYiI9GYKZ3HinKM6vIeBORkHPKd/\nVjpji3OYXalJASIiIr2VwlmchPc0E2luO2jLGUS7Nhet28meptY4VSYiIiKJROEsTqq9BWgPNuYM\nopMCmlrbWLC2Nh5liYiISIJROIuT9t0B9rfGWUeTR+STGjKNOxMREemlFM7ipH0B2s5azrIyUjlh\nWJ4WoxUREemlFM7ipDocIWQwoN+BJwS0mzaykPc2hgk3NMehMhEREUkkCmdxUhPewxHZmaSmdP5H\nPn1UIc7B3Cp1bYqIiPQ2CmdxUh2OUNRJl2a78UPz6JOWoq5NERGRXkjhLE5qwhGKc7oWztJTQ0wp\nydd6ZyIiIr2Qwlmc1IQjnc7U7Gj6qAJWbd29d5aniIiI9A4KZ3FQ39jCrsaWTmdqdjRtZCEAc6vU\neiYiItKbKJzFQVfXOOtobHEOeX3TtN6ZiIhIL6NwFgc1e3cHOPjWTR2FQsaJpQXMqdyGc86v0kRE\nRCTBKJzFQXV4D9D5ArT7mjaqkE3hCGu2N/hRloiIiCQghbM4aG85OyKn8wVoO5o+sgBAszZFRER6\nEYWzOKiui1CQlU5Gasohfa6kMIvi3EytdyYiItKLKJzFwaEuo9HOzJg2spC5q7bT1qZxZyIiIr2B\nwlkcVIcjhzzerN20kQXsaGhmWU1djKsSERGRRKRwFgeb67rXcgbRfTYB5mhJDRERkV5B4cxnkeZW\nanc3HdIyGh0NzM2kdEAWszXuTEREpFdQOPPZ5jpvAdou7qu5P9NHFjJ/dS1NLW2xKktEREQSlMKZ\nz6q7sTvAvqaPKqChqZV3N+yMVVkiIiKSoBTOfNadrZv2NbW0ADOtdyYiItIbKJz5bG/L2WF0a+b1\nTWfcoFxNChAREekFFM58trkuQk5mKlkZqYf1PdNGFrBo/Q4amlpiVJmIiIgkIoUzn1WH93R7pmZH\n00YV0tzqeGvNjhhUJSIiIolK4cxn3d0dYF+TR/QnLcWYo3FnIiIiPZrCmc+qw5HDGm/Wrm96KicM\n66/1zkRERHo4hTMfNbe2sbW+MSYtZxBd72zJpjp2NjTF5PtEREQk8Sic+WjLrkaco9v7au5r+qgC\nnIO5qzRrU0REpKdSOPNRTXgPcHhrnHV0/NA8stJT1LUpIiLSgymc+agm3AgQk9maAGkpIaaU5Gu9\nMxERkR5M4cxH1TFuOQOYNrKQqm279363iIiI9CwKZz6qCUfom55CTubhLUDb0bRRBQBqPRMREemh\nFM58VF0XXUbDzGL2nUcPzCE/K13jzkRERHoohTMfxWoB2o5CIePE0gLmVG7HORfT7xYREZHgKZz5\nyI9wBtGuzZq6CFXbdsf8u0VERCRYCmc+aWtzbK6LxGyNs46mjywE0FZOIiIiPZCv4czM8szs72a2\n3MyWmdmJZpZvZi+Y2UrvZ3/vXDOzO82s0swWm9kEP2vz27bdjbS0OQbGaBmNjoYX9GVwXh9ma1KA\niIhIj+N3y9mvgeecc0cBxwPLgJuAl5xzo4GXvNcA5wKjvccM4G6fa/NVTTgCQHEM9tXcl5lx4sgC\n5lZtp61N485ERER6Et/CmZnlAqcA9wI455qcczuBC4FZ3mmzgE97zy8E7ndRbwJ5ZlbsV31+q/bC\nmR9jziC6lVN4TzNLq+t8+X4REREJhp8tZyXAVuBPZrbIzO4xsyygyDlX7Z1TAxR5zwcD6zt8foN3\nLCnV+BzOpnnjzmZr3JmIiEiPErvVUff/3ROArznn5pnZr/mwCxMA55wzs0PqlzOzGUS7PSkqKqKi\noiJG5e5ffX19t64xb0UTqQaL35pDKIbrnHU0KMt46q2VjHHrOz+5F+ruvZNg6b4lJ9235KV7l3j8\nDGcbgA3OuXne678TDWebzazYOVftdVtu8d7fCAzt8Pkh3rGPcM7NBGYCTJo0yZWXl/tUflRFRQXd\nucY/axZR3H8Hp592WuyL8pwVfp9HF2xg2kmnkJ6qibf76u69k2DpviUn3bfkpXuXeHz7F905VwOs\nN7Mx3qEzgKXAk8A13rFrgCe8508CV3uzNqcC4Q7dn0mnOhyhOCf2MzU7mjaqkD3NrSxat8PX64iI\niEj8+NlyBvA14EEzSweqgGuJBsJHzew6YC1wqXfuM8AngUqgwTs3aW2ui3DckDxfrzG1tICQwexV\n2ykrLfD1WiIiIhIfvoYz59w7wKT9vHXGfs51wFf9rCdenHNUhyOcfYw/kwHa5fZJY9zgXOau2gZn\nHenrtURERCQ+NFDJBzsbmmlsafNtpmZH00YWsmjdTnY3tvh+LREREfGfwpkP9q5x5sMCtPuaPqqA\nljbH/DW1vl9LRERE/Kdw5oOauj2Af2ucdTRpeD7pKSHtsykiItJDKJz5oL3lrNiHfTX31Sc9hQnD\n87TPpoiISA+hcOaDmnCElJAxIDsjLtebPrKQpdV11O5uisv1RERExD8KZz6oCUc4IjuDlJA/OwPs\na9qo6FZOc1ep9UxERCTZKZz5oKYuEpfxZu2OH5JLv4xUZq/SuDMREZFkp3Dmg+pwhOI4hrPUlBBT\nSvLVciYiItIDKJz5oCYcoSgOy2h0NG1kAau37WbTzj1xva6IiIjElsJZjO2KNFPf2BLXljOA6d64\ns9laUkNERCSpKZzFWE37ArRxWEajozFF2RRkpTNHXZsiIiJJTeEsxj5c4yy+LWehkHHiyAJmV24j\nuk2piIiIJCOFsxirqYvf1k37mj6qkC27GqncUh/3a4uIiEhsKJzFWHu3ZrwnBACUjxlAekqIuypW\nxf3aIiIiEhsKZzFWHY5Q2C+D9NT4/9EW5/bh+pNLeHzRRt7SRugiIiJJSeEsxmrCexiYG59tm/bn\nP04fRXFuJrc8sYTWNo09ExERSTYKZzFWHY4wMCe+MzU76pueyvfPO5ql1XX8df66wOoQERGR7lE4\ni7GauvjuDrA/5x1bzImlBfz8+RXaDF1ERCTJKJzFUKS5lZ0NzXHdV3N/zIxbLziG+sYWfv7vFYHW\nIiIiIodG4SyGagJa42x/xgzM5uoTh/PQ/HW8tyEcdDkiIiLSRQpnMVS9d3eA4MMZwDfPPJKCrHT+\n+8n3adPkABERkaSgcBZDNXXRTceDWIB2f3L7pPG9c45i0bqd/GPRxqDLERERkS5QOIuhRGs5A/js\nhCGcMCyP255dRl2kOehyREREpBMKZzFUE46Q2yeNvumpQZeyVyhk/PCCY9i+u4lfv7gy6HJERESk\nEwpnMVQdDn4Zjf05bkgel08eyp/nrOGDzbuCLkdEREQOQuEshjbXRRKqS7Oj75x9FP0yUrn1ySU4\np8kBIiIiiUrhLIYSteUMID8rnW9/4kjmrNrOM+/VBF2OiIiIHIDCWYw0tbSxrb4x0K2bOvO5suEc\nXZzD/z69lIamlqDLERERkf1QOIuRLbsiOEegm553JiVk/OjCY6gOR7jrlVVBlyMiIiL7oXAWIzV7\nl9FI3JYzgMkj8vn0+EHMfK2KNdt2B12OiIiI7EPhLEaqE2jrps7c/MmjSUsx/udfS4MuRURERPah\ncBYjNQm4AO2BFOVk8vUzRvPS8i28vHxz0OWIiIhIBwpnMVJTFyErPYXsjMRZgPZgrp1eQumALH74\n1FIiza1BlyMiIiIehbMYqQlH1zgzs6BL6ZL01BC3nn8Ma7c3cO8bq4MuR0RERDwKZzFSHd5DcYJP\nBtjXKUcO4Oxjivjty5Vs2rkn6HJEREQEhbOYqQlHKMpJ/PFm+/rBeWNpc44fP7Ms6FJEREQEhbOY\naG1zbN7VmBQzNfc1NL8vXykfydOLq5lTuS3ockRERHo9hbMY2FbfSGubS4qZmvvz5VNHMqR/H259\nagnNrW1BlyMiItKrKZzFQDKtcbY/mWkp/PenxvLB5nrun7s26HJERER6NYWzGEimNc4O5KyxRZxy\n5AB+9cIHbN3VGHQ5IiIivZbCWQzUhKMzHZNttmZHZsYt548l0tLKz55bHnQ5IiIivZbCWQxU10VI\nTw3Rv29a0KUclpED+vHFk0r4+8INvL1uR9DliIiI9EoKZzFQE44wMCd5FqA9mK+dPpqinAxueWIJ\nrW0u6HJERER6HYWzGKj2dgfoCfplpPJfnzya9zaGeeSt9UGXIyIi0usonMVATTiStDM19+eC4wcx\npSSfO55fzs6GpqDLERER6VUUzg6Tc46aup7TcgbRyQE/vOAYwnua+b9/fxB0OSIiIr2Kwtlh2tHQ\nTFNLG8VJuHXTwRxdnMNVU4fz4Ly1LNkUDrocERGRXsPXcGZma8zsPTN7x8wWeMfyzewFM1vp/ezv\nHTczu9PMKs1ssZlN8LO2WKn2ltEYmMTLaBzIt84aQ17fdG59cgnOaXKAiIhIPMSj5ew059x459wk\n7/VNwEvOudHAS95rgHOB0d5jBnB3HGo7bDVJvjvAweT2TeN754zhrTU7eOKdTUGXIyIi0isE0a15\nITDLez4L+HSH4/e7qDeBPDMrDqC+Q1LdA3YHOJhLJg7l+CG5/PiZZeyKNAddjoiISI/ndzhzwL/N\nbKGZzfCOFTnnqr3nNUCR93ww0HHthg3esYRWE46QEjIK+2UEXYovQiHjhxeOY3t9I9f+6S3CDQpo\nIiIifkr1+ftPcs5tNLMjgBfM7CP7AjnnnJkd0mAmL+TNACgqKqKioiJmxe5PfX39Qa+xaEUjuenw\n+muv+lpH0L5yfAZ/eHcH5/3iBb49KZO8zMSfS9LZvZPEpPuWnHTfkpfuXeLxNZw55zZ6P7eY2ePA\nFGCzmRU756q9bsst3ukbgaEdPj7EO7bvd84EZgJMmjTJlZeX+/gbQEVFBQe7xh8r32REaivl5dN9\nrSNo5cC0SduYcf8C/m8xPHDdZIYXZAVd1kF1du8kMem+JSfdt+Sle5d4fGv+MLMsM8tufw58Angf\neBK4xjvtGuAJ7/mTwNXerM2pQLhD92fCii5A2/Nmau7P9FGF/PWGqdRHWvjs3XNZuqku6JJERER6\nHD/7poqAN8zsXWA+8LRz7jngNuAsM1sJnOm9BngGqAIqgT8CN/pYW0w453rU1k1dcfzQPP725Wmk\npRiXzZzL/NW1QZckIiLSo/jWremcqwKO38/x7cAZ+znugK/6VY8fdjW20NDU2iOX0TiYUUf047Gv\nTOOqe+dx1b3zuOvKCZxxdFHnHxQREZFOJf6o7gTWvsZZUQ/bHaArBuX14W9fnsZRA7OZ8ZeFPLZw\nQ9AliYiI9AgKZ4ehugcvQNsV+VnpPHjDVKaW5vOff3uXe99YHXRJIiIiSU/h7DDU7N26qXeGM4B+\nGanc94XJnHPMQP7nX0v5+fMrtNWTiIjIYVA4OwzV4QhmcER27w1nABmpKfzuyglcMWUov32lku//\n831a2xTQREREusPvRWh7tM11EQr7ZZCeqoybEjJ+8plj6d83nbsqVrGzoYlfXjaejNSUoEsTERFJ\nKgpnh6E6HOm14832x8z47jlH0b9vOj9+Zhl1exbwh6smkpWh/8xERES6Sk0+h6EmHOmVMzU7c8Mp\npfz8kuOZW7Wdz90zj9rdTUGXJCIikjQUzg6DWs4O7OKJQ/j95yeyrLqOS34/h0079wRdkoiISFJQ\nOOumhqYWwnuae/VMzc6cNbaI+784hS11jVx89xwqt9QHXZKIiEjCUzjrpppevsZZV00tLeChGVNp\nam3j0j/MZfGGnUGXJCIiktAUzrqppi4azgbm9I5Nzw/HuMG5/O3L0+ibnsIVM99kTuW2oEsSERFJ\nWApn3aSWs0NTUpjFY1+ZxpD+ffnCn97iufergy5JREQkISmcdVP71k0ac9Z1RTmZPPKlqYwbnMON\nD77Nw/PXBV2SiIhIwlE466aacIS8vmlkpmmR1UOR1zedB64v4+TRA7jpH+9xd8UqbfckIiLSgcJZ\nN1WHIwzUGmfd0jc9lT9ePYkLjh/Ez55bzj2va8N0ERGRdlq6vZtq6vZovNlhSE8N8avLxtPc2sZP\nn13GMYNzmDayMOiyREREAqeWs26qCUcYmKuZmocjFDLuuOR4Sgqz+NpfF1Ed1kK1IiIiCmfd0NTS\nxrb6JrWcxUC/jFT+cNVEIs2tfOWBt2lsaQ26JBERkUApnHXD5jrN1IylUUdkc8clx/PO+p3877+W\nBV2OiIhIoBTOuqF9AVq1nMXOJ48tZsYppfzlzbU8tnBD0OWIiIgERuGsG/aucabZmjH13bPHMLU0\nn/96/D2WbAoHXY6IiEggFM66ocYbuK5uzdhKTQnxmysm0L9vOl9+YCHhhuagSxIREYk7hbNuqA5H\n6JeRSnZmWtCl9DgDsjO46/MTqAlH+OYji2hr0wK1IiLSuyicdUN0GQ21mvllwrD+/Pf5x/DKiq3c\n+fLKoMsRERGJK4Wzbqipi2gygM8+XzaMiyYM5tcvreSV5VuCLkdERCRuFM66oUZbN/nOzPjxp4/l\nqIE5fOPhRazb3hB0SSIiInGhcHaIWlrb2LKrUS1ncdAnPYXff34CAF9+YCGRZi1QKyIiPZ/C2SHa\nVt9Ea5ujSOEsLoYXZPHry09gaXUd33/8fZzTBAEREenZFM4OUfv+j2o5i5/TjjqCb5wxmsfe3sCD\n89YFXY6IiIivFM4OUc3eBWi16Xk8feOM0ZSPGcAPn1rConU7gi5HRETENwpnh6h9dwC1nMVXKGT8\n6rLxDMzN5MYH32ZbfWPQJYmIiPhC4ewQba6LkJEaIq+vFqCNt7y+6dx95URqdzfxtb8uoqW1LeiS\nREREYk7h7BBVh6NrnJlZ0KX0SuMG5/LjzxzL3Krt3PHvFUGXIyIiEnMKZ4dIuwME7+KJQ7iybBh/\neLWKZ9+rDrocERGRmFI4O0TVdXu0AG0C+O/zxzJ+aB7f/tu7VG6pD7ocERGRmFE4OwRtbY7N4UYG\n5mqmZtAyUlO4+/MTyExL4csPLKS+sSXokkRERGKi03BmZt8wsxyLutfM3jazT8SjuERT29BEU2ub\nZmomiOLcPvzmihOo2lrP9/6+WAvUiohIj9CVlrMvOufqgE8A/YGrgNt8rSpB7V3jTOEsYUwbVch3\nzzmKp9+r5t43VgddjoiIyGHrSjhrn5b4SeAvzrklHY71KjVa4ywhfemUUs45ZiA/fXY5c1dtD7oc\nERGRw9KVcLbQzP5NNJw9b2bZQK9cYKq6Ti1nicjMuOOS4xhe0JevPfT23hAtIiKSjLoSzq4DbgIm\nO+cagDTgWl+rSlA14T2khozCrIygS5F9ZGem8YfPT6ShqZUbH1xIU0uv/P8PIiLSA3QlnJ0IrHDO\n7TSzzwM/AML+lpWYqsMRinIyCYV6Za9uwhtdlM0dFx/P2+t28uOnlwZdjoiISLd0JZzdDTSY2fHA\nfwKrgPt9rSpBaQHaxHfeccXccHIJs+auZdEWLa8hIiLJpyvhrMVF1yi4EPitc+53QLa/ZSUmhbPk\n8L1zjqKwXzrzaxTOREQk+XQlnO0ys5uJLqHxtJmFiI4761Wcc9F9NbU7QMJLTQkxpSSfFbVtWvtM\nRESSTlfC2WVAI9H1zmqAIcAdvlaVgOoiLexpblXLWZKYMiKf2ohjw449QZciIiJySDoNZ14gexDI\nNbNPARHnXK8bc/bhGmfauikZlJUWADBvdW3AlYiIiByarmzfdCkwH7gEuBSYZ2YXd/UCZpZiZovM\n7F/e6xIzm2dmlWb2iJmle8czvNeV3vsjuvML+aU6HG2BUctZchhTlE1WGsxfrUVpRUQkuXSlW/P7\nRNc4u8Y5dzUwBfh/h3CNbwDLOrz+GfBL59woYAfRddTwfu7wjv/SOy9haOum5BIKGUf2T1HLmYiI\nJJ2uhLOQc25Lh9fbu/g5zGwIcB5wj/fagNOBv3unzAI+7T2/0HuN9/4Z3vkJoTocwQyOyNYCtMli\nTP8U1m4AaahyAAAgAElEQVRv0I4BIiKSVFK7cM5zZvY88JD3+jLgmS5+/6+A7/Lh0hsFwE7nXPsa\nBxuAwd7zwcB6AOdci5mFvfO3dfxCM5sBzAAoKiqioqKii6V0T319PRUVFSxa0UhuujH79dd8vZ7E\nztDMRsD489NvMHVQV/5Tl0TQ/ndOkovuW/LSvUs8nf6L5Zz7jpl9FpjuHZrpnHu8s895kwe2OOcW\nmln54ZX5kXpmAjMBJk2a5MrLY/bV+1VRUUF5eTn3Vc1n+IAmystP8vV6EjutL79Cv4wm6voUUV5+\nbNDlSBe1/52T5KL7lrx07xJPl5oTnHOPAY8d4ndPBy4ws08CmUAO8Gsgz8xSvdazIcBG7/yNwFBg\ng5mlArlEu1ATwuZwhBGFfYMuQw5BSsiYOLw/8zXuTEREksgBx46Z2S4zq9vPY5eZ1XX2xc65m51z\nQ5xzI4DLgZedc1cCrwDtsz2vAZ7wnj/pvcZ7/2WXQCuIVof3aBmNJFRWmk/llnq21TcGXYqIiEiX\nHDCcOeeynXM5+3lkO+dyDuOa3wO+ZWaVRMeU3esdvxco8I5/C7jpMK4RU7sbW6iLtFCk3QGSTllJ\ndL2zt9R6JiIiSSIuo6SdcxVAhfe8iuhyHPueEyG6llrCqalrX4BW4SzZHDs4l8y0EPNW13LuscVB\nlyMiItKpLi2J0dtpjbPklZ4aYuLw/lrvTEREkobCWRdUh9VylsymjChgeU0d4YbmoEsRERHplMJZ\nF2z2ujU15iw5lZXm4xy8tUatZyIikvi6srfmRWa20szChzJbsyepDu8hPyudzLSUoEuRbhg/NI/0\nlBDztM+miIgkga5MCLgdON85t6zTM3uomnCEgWo1S1qZaSmMH5qncWciIpIUutKtubk3BzOIjjnT\nZIDkVlaaz/sbw9Q3tnR+soiISIAOtgjtRWZ2EbDAzB4xsyvaj3nHe40ahbOkN6UknzYHCzTuTERE\nEtzBujXP7/C8AfhEh9cO+IcvFSWYplbH9t1NFKtbM6lNHN6f1JAxf3Ut5WOOCLocERGRAzpgOHPO\nXRvPQhLVzsboDlJqOUtufdNTGTc4V+POREQk4XVltuYsM8vr8Lq/md3nb1mJY0ckGs60r2byKyvN\nZ/GGnexpag26FBERkQPqyoSA45xzO9tfOOd2ACf4V1JiaQ9najlLfmUl+TS3Ohat2xF0KSIiIgfU\nlXAWMrP+7S/MLJ847cmZCGob2wCFs55g0oh8Qoa6NkVEJKF1JWT9HzDXzP7mvb4E+Il/JSWWHRFH\ndkYq/TJ6TR7tsXIy0xg7KEeL0YqISELrtOXMOXc/cBGw2Xtc5B3rFWojTq1mPciUEQUsWreTxhaN\nOxMRkcTUlQkBf3HOLXXO/dZ7LDWzv8SjuESwQ+GsRykrzaexpY3FG8JBlyIiIrJfXRlzdkzHF2aW\nAkz0p5zEUxtxFCuc9RiTR+QDMK9KXZsiIpKYDrZDwM1mtgs4rsOG57uALcATcaswQC2tbYQbHQO1\njEaPkZ+VzpiibE0KEBGRhHXAcOac+6lzLhu4wzmX45zL9h4Fzrmb41hjYLbWN+JALWc9zJSSfBau\n3UFza1vQpYiIiHxMVyYE3OwtPDvFzE5pf8SjuKBVhyOAltHoacpK82loamXJprqgSxEREfmYrkwI\nuB54DXge+KH381Z/y0oMNe3hTPtq9ihTSjTuTEREEldXJgR8A5gMrHXOnUZ0d4CdB/9Iz3D6UUfw\n05P6UDogK+hSJIaOyM6ktDCL+Rp3JiIiCagr4SzinIsAmFmGc245MMbfshJDZloKxf1CZKSmBF2K\nxFhZaT7z19TS2uaCLkVEROQjuhLONngbn/8TeMHMngDW+luWiL+mlOSzK9LCsmqNOxMRkcTS6Z5E\nzrnPeE9vNbNXgFzgOV+rEvFZWUkBAPNX1zJucG7A1YiIiHyoKy1nmNkEM/s6cBywwTnX5G9ZIv4a\nlNeHIf37aJ9NERFJOF2ZrfnfwCygACgE/mRmP/C7MBG/lZUUMH91Lc5p3JmIiCSOrrScXQlMds7d\n4py7BZgKXOVvWSL+KyvNZ0dDMyu31AddioiIyF5dCWebgI4LfWUAG/0pRyR+ytrXO9OSGiIikkAO\ntrfmb8zsTiAMLDGzP5vZn4D36SXrnEnPNiy/LwNzMrUYrYiIJJSDzdZc4P1cCDze4XiFb9WIxJGZ\nMaUknzertuOcw8yCLklEROTA4cw5NyuehYgEoaw0nyff3cSa7Q2UFGonCBERCd4Bw5mZPeqcu9TM\n3gM+Np3NOXecr5WJxEFZh302Fc5ERCQRHKxb8xvez0/FoxCRIIwc0I/CfunMX13L5VOGBV2OiIjI\nQbs1q72f2qpJeqz2cWeasSkiIomiK4vQXmRmK80sbGZ1ZrbLzLQhofQYU0bks3HnHjbsaAi6FBER\nkS6tc3Y7cIFzLtc5l+Ocy3bO5fhdmEi8lJVG99mcV6XWMxERCV5Xwtlm59wy3ysRCciYomxy+6Qx\nX12bIiKSAA42IaDdAjN7BPgn0Nh+0Dn3D9+qEomjUMiYPCJfm6CLiEhC6Eo4ywEagE90OOYAhTPp\nMcpK8nlx2WY210Uoysns/AMiIiI+6TScOeeujUchIkEqK/1wn80Ljh8UcDUiItKbHWwR2u865243\ns9+w/0Vov+5rZSJxNLY4h34Zqcyr2q5wJiIigTpYy1n7JIAFBzlHpEdITQkxcXh/TQoQEZHAHWwR\n2qe8n9pjU3qFstJ8bn9uBdvrGynolxF0OSIi0kt1ZRHaSWb2uJm9bWaL2x/xKE4kntr32XxrjVrP\nREQkOF2Zrfkg8B3gPaDN33JEgnPs4Dwy00K8WVXLOeOKgy5HRER6qa6Es63OuSd9r0QkYOmpISYM\n07gzEREJVld2CLjFzO4xsyu8fTYvMrOLOvuQmWWa2Xwze9fMlpjZD73jJWY2z8wqzewRM0v3jmd4\nryu990cc1m8m0g1lJQUsq6kj3NAcdCkiItJLdSWcXQuMB84Bzvcen+rC5xqB051zx7d/3symAj8D\nfumcGwXsAK7zzr8O2OEd/6V3nkhclZXm4xwsWKvWMxERCUZXujUnO+fGHOoXO+ccUO+9TPMeDjgd\n+Jx3fBZwK3A3cKH3HODvwG/NzLzvEYmL8UPzSE8JMW91LWccXRR0OSIi0gt1peVsjpmN7c6Xm1mK\nmb0DbAFeAFYBO51zLd4pG4DB3vPBwHoA7/0wUNCd64p0V2ZaCuOH5jGvSvtsiohIMLrScjYVeMfM\nVhPtqjSiDWPHdfZB51wrMN7M8oDHgaMOp1gAM5sBzAAoKiqioqLicL/yoOrr632/hviju/euKNTE\n0xubefbFV+iTarEvTA5Kf+eSk+5b8tK9SzxdCWfnHO5FnHM7zewV4EQgz8xSvdaxIcBG77SNwFBg\ng5mlArnAx5ovnHMzgZkAkyZNcuXl5Ydb3kFVVFTg9zXEH929dymDt/JU1Xz6DhvHqUcOiH1hclD6\nO5ecdN+Sl+5d4um0W9M5t3Z/j84+Z2YDvBYzzKwPcBbRLaFeAS72TrsGeMJ7/qT3Gu/9lzXeTIIw\nYVh/UkLG/NXq2hQRkfjrSstZdxUDs8wshWgIfNQ59y8zWwo8bGb/CywC7vXOvxf4i5lVArXA5T7W\nJnJAWRmpHDs4l3lVmrEpIiLx51s4c84tBk7Yz/EqYMp+jkeAS/yqR+RQlJXmc98bq4k0t5KZlhJ0\nOSIi0ot0ZbamSK9TVpJPc6vj7XU7gi5FRER6GYUzkf2YNCIfM7SVk4iIxJ3Cmch+5GSmMbY4R+PO\nREQk7hTORA6grKSAt9ftoLGlNehSRESkF1E4EzmAKSX5NLa08d6GcNCliIhIL6JwJnIAU0ryAZin\ncWciIhJHCmciB5Cflc6YomyFMxERiSuFM5GDmFKSz8I1tbS0tgVdioiI9BIKZyIHUVaaz+6mVpZs\nqgu6FBER6SUUzkQO4sNxZ9pnU0RE4kPhTOQgjsjOpLQwS4vRiohI3CiciXRiSkk+81fX0trmgi5F\nRER6AYUzkU6UleZTF2lheY3GnYmIiP8UzkQ6UVZSAGifTRERiQ+FM5FODMrrw5D+fbTPpoiIxIXC\nmUgXlJUUMH9NLc5p3JmIiPhL4UykC8pK8qnd3UTllvqgSxERkR5O4UykC8pKtc+miIjEh8KZSBcM\ny+9LUU6GwpmIiPhO4UykC8wsOu5s9XaNOxMREV8pnIl0UVlpPpvrGlm7vSHoUkREpAdTOBPpojLt\nsykiInGgcCbSRSMH9GNofh8eXbBBXZsiIuIbhTORLjIzbji5lIVrd2i3ABER8Y3CmcghuHTSUAr7\npXNXxaqgSxERkR5K4UzkEGSmpXDt9BJe/WAr728MB12OiIj0QApnIofoqhOHk52Ryt2vqvVMRERi\nT+FM5BDlZKbx+ROH8+x71azetjvockREpIdROBPphi9OLyEtJcQf1HomIiIxpnAm0g0DsjO4dNJQ\nHnt7AzXhSNDliIhID6JwJtJNM04ppc3BPa9XBV2KiIj0IApnIt00NL8vFxw/iL/OX8eO3U1BlyMi\nIj2EwpnIYfhK+UgamlqZNXdN0KWIiEgPoXAmchiOLMrmzKOL+POcNexubAm6HBER6QEUzkQO042n\njWRnQzMPzV8XdCkiItIDKJyJHKYJw/pTVpLPPa+vprGlNehyREQkySmcicTAjaeNoqYuwj8XbQy6\nFBERSXIKZyIxcMroQo4ZlMPvX62itc0FXY6IiCQxhTORGDAzbiwfxeptu3l+SU3Q5YiISBJTOBOJ\nkXPGDaS0MIu7KipxTq1nIiLSPQpnIjGSEjK+dGop72+s4/WV24IuR0REkpTCmUgMfeaEIQzMyeSu\nisqgSxERkSSlcCYSQ+mpIa4/uYQ3q2p5e92OoMsREZEkpHAmEmNXTBlGXt807nplVdCliIhIElI4\nE4mxrIxUvjBtBC8u28yKml1BlyMiIklG4UzEB1+YNoK+6Sn8/lW1nomIyKFROBPxQV7fdD43ZRhP\nvruJ9bUNQZcjIiJJxLdwZmZDzewVM1tqZkvM7Bve8Xwze8HMVno/+3vHzczuNLNKM1tsZhP8qk0k\nHq4/uZSQwczXqoIuRUREkoifLWctwH8658YCU4GvmtlY4CbgJefcaOAl7zXAucBo7zEDuNvH2kR8\nNzA3k89OGMKjC9azdVdj0OWIiEiS8C2cOeeqnXNve893AcuAwcCFwCzvtFnAp73nFwL3u6g3gTwz\nK/arPpF4+NKpI2lubeO+2auDLkVERJJEXMacmdkI4ARgHlDknKv23qoBirzng4H1HT62wTsmkrRK\nCrM499hiHpi7lrpIc9DliIhIEkj1+wJm1g94DPimc67OzPa+55xzZnZImxCa2Qyi3Z4UFRVRUVER\nw2o/rr6+3vdriD8S5d5N6dfK040t/OjBV/jUyPSgy0l4iXLf5NDoviUv3bvE42s4M7M0osHsQefc\nP7zDm82s2DlX7XVbbvGObwSGdvj4EO/YRzjnZgIzASZNmuTKy8v9Kh+AiooK/L6G+COR7t1L2+ZT\nsSnM/159MplpKUGXk9AS6b5J1+m+JS/du8Tj52xNA+4FljnnftHhrSeBa7zn1wBPdDh+tTdrcyoQ\n7tD9KZLUbiwfybb6Jh5dsL7zk0VEpFfzc8zZdOAq4HQze8d7fBK4DTjLzFYCZ3qvAZ4BqoBK4I/A\njT7WJhJXZSX5TBiWxx9eraK5tS3ockREJIH51q3pnHsDsAO8fcZ+znfAV/2qRyRIZsaN5aO4/v4F\nPPXuJi6aMCTokkREJEFphwCRODn9qCMYU5TN3RWraGs7pHkwIiLSiyicicRJKGR8pXwkK7fU8+Ky\nzUGXIyIiCUrhTCSOPnVcMUPz+3BXxSqiPfkiIiIfpXAmEkepKSFmnDKSd9bvZG7V9qDLERGRBKRw\nJhJnl0wcQmG/DO6uWBV0KSIikoAUzkTiLDMthetOKuH1ldt4b0M46HJERCTBKJyJBODzU4eRnZnK\n3a9WBl2KiIgkGIUzkQBkZ6Zx9YnDefb9GlZtrQ+6HBERSSAKZyIBuXZ6CekpIf7wqsaeiYjIhxTO\nRAJS2C+DyycP5fFFG6kO7wm6HBERSRAKZyIBuuGUUpyDP762OuhSREQkQSiciQRoSP++XDB+EA/N\nX0ft7qagyxERkQSgcCYSsK+cOpI9za1c9oe5/O+/lvLvJTXsUFATEem1UoMuQKS3G12Uzc8+eyyP\nvb2R+99cyz1vRLs4jyzqx5SSfCaPyGdKST7FuX0CrlREROJB4UwkAVw2eRiXTR5GY0srizeEmb+6\nlvmra/nnok088OY6AIbm92HKiAKmlPRnSkkBIwr6YmYBVy4iIrGmcCaSQDJSU5g8Itpa9tXToKW1\njeU1u5i3upb5q7fzyootPPb2BgAGZGcwxWtVmzwin6MGZhMKKayJiCQ7hTORBJaaEmLc4FzGDc7l\nupNKcM6xams981fvYP7q7cxfXcvT71UDkJOZGg12JdHANm5QLumpGlYqIpJsFM5EkoiZMeqIbEYd\nkc3nyoYBsGFHA/NX1/LWmlrmra7lpeVbAMhMCzFhWH8uOH4QF08cQmqKgpqISDJQOBNJckP692VI\n/75cNGEIAFt3NbJgTS3z19Qyu3IbN/3jPWa+VsW3zx7DueMGapyaiEiCUzgT6WEGZGdw7rHFnHts\nMc45Xly2hTueX86ND77NcUNy+d45RzF9VGHQZYqIyAGon0OkBzMzzhpbxLPfOIU7Lj6ObbsaufKe\neVx17zze2xAOujwREdkPhTORXiAlZFwyaSgvf7ucH5x3NO9vDHP+b9/gq399m9XbdgddnoiIdKBw\nJtKLZKalcP3Jpbz63dP4+umjeGX5Fs76xat8//H32FIXCbo8ERFB4UykV8rJTONbnxjDq985jc+V\nDeORt9Zzyh2vcPtzywnvaQ66PBGRXk3hTKQXG5CdwY8uHMdL/3kqZx8zkLsqVnHK7a8w87VVRJpb\ngy5PRKRXUjgTEYYXZPHry0/g6a+fxPihefzkmeWc9vMKHnlrHS2tbUGXJyLSqyicichexwzKZdYX\np/DQDVMpysnke4+9x9m/eo3n3q/GORd0eSIivYLCmYh8zIkjC3j8xmn8/vMTAfjyA2/z6bvmMGfV\ntoArExHp+RTORGS/zIxzxg3k+W+ewu2fPY4tdRE+98d5XH3ffN7fqDXSRET8onAmIgeVmhLi0slD\neeXb5Xz/k0ezeMNOPvWbN/j1iyuDLk1EpEdSOBORLslMS+GGU0p59Tunce64gfzm5ZWs2lofdFki\nIj2OwpmIHJLcPmn86MJxZKSG+Nmzy4MuR0Skx1E4E5FDNiA7g6+Uj+TfSzczr2p70OWIiPQoCmci\n0i3XnVTKwJxMfvLMMtratMyGiEisKJyJSLf0SU/h22eP4d0NYZ5avCnockREegyFMxHptotOGMzY\n4hxuf26FtnsSEYkRhTMR6bZQyPjBeUezcece/jxnTdDliIj0CApnInJYpo0q5PSjjuB3L1dSu7sp\n6HJERJKewpmIHLabzz2KhuZW7nxJC9OKiBwuhTMROWyji7K5fPJQHnhzLVVamDYhVW7ZxbV/ms/J\nt7/MrDlraGppC7okETkAhTMRiYlvnnkkGakhbtPCtAllZ0MTtz65hLN/9ToL1u6gsF8Gtzy5hDN/\n8SpPvLNRy6CIJCCFMxGJCS1Mm1haWtuYNWcN5T+v4P65a7h88lAqvl3OP74yjT9dO5msjFS+8fA7\nnP/bN3j1g604p5AmkigUzkQkZrQwbWJ47YOtnPvr17nlySWMLc7h6a+fzI8/cywF/TIwM04bcwRP\nf+0kfnXZeOoizVxz33w+98d5vLt+Z9CliwgKZyISQ1qYNlirttZz3Z/f4ur75tPU2sbMqyby4PVl\nHF2c87FzQyHj0ycM5qVvlXPr+WP5YPMuLvzdbG58cKHGDYoELDXoAkSkZ7nohMHc98Zqbn9uBWcf\nM5DMtJSgS+rxwnuaufOllcyas4bMtBRuPvcovjB9BBmpnf/Zp6eG+ML0Ei6eNJR7Xq/ij69V8fyS\nzVw6aSjfPHM0RTmZcfgNRKQjtZyJSExpYdr4aWlt44E313Lazyu4b/ZqLpk0hFe+Xc6XTh3ZpWDW\nUb+MVL555pG8+t3TuGrqcP6+cD2n3vEKP3tuOeE9zT79BiKyP2o5E5GY67gw7aWThpKflR50ST3O\n7Mpt/OippazYvIuyknz++/yxHDMo97C/t7BfBrdecAxfnF7CL15Ywe9fXcVf563jxvKRXDNthFpC\nReLAt5YzM7vPzLaY2fsdjuWb2QtmttL72d87bmZ2p5lVmtliM5vgV10iEh+9YWHaJ9/dxM3/WMys\nOWtYuLaWhqYW36+5Zttubrh/AVfeM4/dTS3cfeUEHp4xNSbBrKNhBX351eUn8K+vncT4oXn89Nnl\nnPbzCh59az0trVojTcRPfrac/Rn4LXB/h2M3AS85524zs5u8198DzgVGe48y4G7vp4gkqY4L0159\n4nBKB/QLuqSYcc7xu1cq+fm/P6BPWgp7mtcDEDIYOaAf4wbncsygHMYNzmXsoBxyMtMO+5p1kWZ+\n+3Ilf5q9mvSUEN89ZwxfnF7ie0vWMYNymfXFKcxdtZ3bnlvOdx9bzMzXq/jO2WP4xNgizMzX64v0\nRr6FM+fca2Y2Yp/DFwLl3vNZQAXRcHYhcL+LLrTzppnlmVmxc67ar/pExH/fPPNI/rloI7c9u5yZ\nV08KupyYaGtz/M/TS/nT7DV85oTB3H7xcWyvb+L9jWHe3xTm/Y1h5q7azuOLNu79zIiCvowbnBt9\nDIoGt/5d7OptbXM8umA9//fvFWzf3cTFE4bwnbPHcEScB+qfOLKAf944jeeX1HD78yv40l8WMmFY\nHt875yjKSgviWotITxfvMWdFHQJXDVDkPR8MrO9w3gbvmMKZSBJrX5j25//+gHlV25P+H/Hm1ja+\n+/fFPL5oI1+cXsIPzjuaUMgYmJvJwNxMzhxbtPfcrbsaWbIpzJJNdby/Mcy7G3byr8Uf/k/a4Lw+\njBucw7hBuYwbEg1tA7IzPnK9N6u286OnlrK0uo7JI/rzpy9M4dghse2+PBRmxjnjijnz6CL+vnAD\nv3pxJZfNfJPTjzqCU/q34pxTS5pIDJifq0J7LWf/cs6N817vdM7ldXh/h3Ouv5n9C7jNOfeGd/wl\n4HvOuQX7+c4ZwAyAoqKiiQ8//LBv9QPU19fTr1/P6Y7pTXTvEkNjq+Om1/aQl2n8v6mZhDr5xztR\n71tjq+N37zSyeGsrF49O47zStEMOIvVNjnW72lhT18racBtr69qoafjwf4PzMozhOSFG5ITYUN/G\nws2tFGQal41JZ/LAlIQLPk2tjhfXNvOvqmYaWqAg0zimMIVxhSmMzU+hX3pi1Sv7l6h/53qi0047\nbaFzrtNuhHi3nG1u7640s2Jgi3d8IzC0w3lDvGMf45ybCcwEmDRpkisvL/exXKioqMDva4g/dO8S\nx668DXz7b++yq/+RXDh+8EHPTcT7trOhietmLeD9bQ389KJjuWLKsJh9965IM8uqd0W7Rb2u0aeq\n6slMS+HbnziS608uTegZkp8Avt/QzC8fq2Cz5fNG5TZe29CIGRw3JI9TRhdy8ugBnDAsj7QUrd6U\niBLx71xvF+9w9iRwDXCb9/OJDsf/w8weJjoRIKzxZiI9RzIvTFsTjnDNffNZvW03d105gXPGFcf0\n+7Mz05hSks+Ukvy9x/Y0tdLmHFkZybHaUW7fNMqHplFePpGW1jbe3RDm9ZVbeX3lNu6qWMVvXq6k\nX0YqJ44s2BvWhhf0TbiWQJFE4dvffDN7iOjg/0Iz2wDcQjSUPWpm1wFrgUu9058BPglUAg3AtX7V\nJSLx174w7efumcef56zhy6eODLqkLqnaWs9V985nZ0MTf752MtNGFcblun3Skye87is1JcTE4f2Z\nOLw/3zzzSMJ7mpm7ajuvrdzKax9s5YWlmwEYmt+Hk0cP4JTRAzhxZAG5fQ5/RqtIT+HnbM0rDvDW\nGfs51wFf9asWEQlesi1M+96GMF/403wAHp5xYqAD8ZNZbp80zhk3kHPGDcQ5x9rtDby+ciuvrdzG\nk+9s4q/z1pESMsYPzeNkr1Xt+CG5pKoLVHqx5GgzF5Ee4eZzj+KcX7/OnS+t5NYLjgm6nAOas2ob\nM+5fSG6fNP5y3ZQetUZbkMyMEYVZjCjM4qoTR9Dc2saidTv3hrVfv7SSX724kpzMVKaPiga1U8cM\nYHBen6BLF4krhTMRiZtkWJj2ufer+fpD7zCisC/3f7GMgbna+NsvaSmhvePt/vMTY9ixu4nZq7bx\n+gfbeG3lVp59v4aUkPHV00bxtdNHaUKB9Br6L11E4uqbZx5JRmqInz23POhSPuah+eu48cG3GTc4\nh0e/dKKCWZz1z0rnU8cN4mcXH8ecm07nxW+dygXHD+LOl1by6d/NZkXNrqBLFIkLhTMRiav2hWmf\nX7KZeVXbgy4H+HA7ppv/8R6nHDmAB6+fSl7fxB4T19OZGaOO6McvLxvP7z8/gZpwhPN/8wa/f3UV\nrW3+rc8pkggUzkQk7q47qZSBOZn85Jll/397dx5eRX3vcfz9TU5WkkAhJiCLrAqRJQgBFLTRooKl\nRQt1QWRRax8rtVat9Sltvbe1XqmtWKu3FgXEKrZSWkpvWbRoRFQWFcIiIAgopRAWkSSQhCy/+8cZ\n8BSMrOfMnPB5PU8eZubMmflOfs+Yj7P8ftT5/Ie2rs7x4D/W8sj89VydfzZPj+od129LNkSDurZg\n/vcv4dLOZ/Hw3HVc+/u32bJ7v99liUSNwpmIxFxaciL3Xnkexf/ax99X/tu3Oqpr67hnRjGTF21m\nbP+2PHptvp5rCqjsjBSeGtmLidf14IOSMgb/5g2ee3uL7+FeJBr0QoCI+MLvjmkrDtZyx/T3eHXd\nTidYDkEAABE7SURBVO694lzuuLSjOkUNODPjmp6t6Ne+GT+cuYqf/m0NL68pYcLw7g3mjU7nHKUV\nNZSUVbKztIqS0kpKyiqprK5jZN82MR/wXvyhcCYivojsmHbaW1v4dgw7pt13oJqbpy3jvY/38otr\nunJj33Nitm85dS0apzFtbAEvLt3Kg/94n0ETF/LTr+UxvFerwAZs5xxlVTXsLPVCV1klJV742lla\nxc6I+aqaus/dxvOLP+KR4d35SpfcGFcvsaZwJiK+OdQx7ROvbeSbMeqYtqS0klGTw8MxPTniAq7q\ndnqHY5LYMDNG9G3DgI7Z3DujmB/8eSXz15Tw0De6kpPpz9Wl3eVVvPXhHnbsq/gseJVVsbM0HLwq\nqmuP+k6j5ERys1LJyUqhZ5sm5GSmePOp5B6eTmHb3gru/OMKbpn2DqMuPIcfXdUlroZBkxOjcCYi\nvoplx7Sbd+/npslL2Lv/IFPHFtA/RsMxSfS0aZbOi7f1Y+qbm/nl/PVcOXEhD17dja92j03oLimt\nZN7qHcxdvZ2lmz/h0CNwaUmJ5GalkJOVSrdWTRgYEbRyMlMPf5ZxnOOndsrNZNYdF/HLeeuZvGgz\nizft4fEbetK5eVYUj078onAmIr46smPaaFm9LTwcU52DF2/rR/dWTaK2L4mtxATj1ovbU3jeWdz9\nUjF3TH+P+WvO5mdDz49KlyjbPq0IB7JV23n34704B51yMhh3WScu75JL2+x0MlJCp/0Wa0ookZ8M\nyePiTtncO2MlX3/iTcZf1YVRF54T2Nu5cnIUzkTEd3cNPJdZy7cxYd46bmh9ctuoqa2jvKqGssoa\nSiurKaus8X6q2V1exeMLNmo4pgauY04mM2+/iN8VfcjjCzaweNMeJgzrzqWdc0552x/vOcDc1duZ\ns3oHxVs/BaBLiyzuHngug7s1p2NO5inv43gVnpfDvLsu5gczinlg9hpe/2AXvxzeneyMlJjVINGl\ncCYivjvUMe2vXv6AdqEU0jbtOSJkVXvzn02XRQSw0spqDhw8+nmeSJ2bZzJ1bAEtGjeMt/rk8yUl\nJnDnVzpxWecc7nmpmLHPLuP6gtaM/2oXMlOTTmhbH+4qZ97qHcxZtZ01/y4FoHurxvxwUGcGd21O\n2+xG0TiE45KdkcKUMQVMe2sLD81dx6DH3uDX1/bgy+ee5VtNcvoonIlIINwyoD0vLPmYp4oreap4\n8VGfJ4cSyEoNkZmaRGZqiKzUJHKzUsmMWBb5WdZ/LA/xpfRkEhJ06+dM0bVlY2Z/tz8TX9nApIUf\n8saG3Tzyze5c1KH+5wydc2zYWc6cVduZu2oH60vCw0Vd0KYJ46/qwqCuzWndND1Wh3BMZsaY/u3o\n16EZd764nNFTlnLLgHbcN+g8UkJ6WSCeKZyJSCCkJScy/Vv9+PM/3+bCXvnhkJX2WbjSHxs5USmh\nRO4f3JnL88JX0UY8vYSx/dty35WdD48C4Zzj/e2lzF21gzmrt7Np137MoOCcpjzwtTwGdW0e+Kut\nnZtnMXvcAB6as5bJizbz1od7+O0N+TG91Sqnl8KZiARGu+xGFDQPMaCT3qKU06fXOU2Z872LmTB3\nHVPf3MLr63dxzxXnsXLbp8xdtYOPPzlAgkG/9s0Y278dV56f61t3HCcrNSmRnw3tyiWdzuK+mSsZ\n8ttF/GRIHiP6tNHLAnFI4UxERBq89OQQ/z20K1ec35wfzAi/0RlKMC7qmM3thR24Ii+XZg3ggfqB\nebnMa3Ux98woZvxfV/P6+l1MGNadL8WgD0E5fRTORETkjNG/Yzbzvn8J727ZS882TaLS1YbfcrJS\nmTa2D1Pe3MyEeesY9JuFTLw2n4vUr1/c0Ai/IiJyRslKTeLSzjkNMpgdkuD1/fbX7/SnUUqIGycv\n4eG56zhYz9BQEiwKZyIiIg1U15aN+b/vDuD6gjY89fqHDPvdW2zaVe53WXIMCmciIiINWHpyiP/5\nRjeeGtmLrXsPMOS3i3hp2Vacc36XJvXQM2ciIiJngEFdm9OjdWPu/lMx981cyesf7OKha7odtd7B\nmjrKKqsPj7hRVlnjTf/nssj5cq8z6PKq8Lq1tY6fDMnj2oKTHPLjDKdwJiIicoZo0TiN52/ty6SF\nm/j1y+tZuuUTGlk1LHuN8qrwKBzH81xaciiBzJQQGV4/hBkpIVo3TSczJTz//vZSfviXlaQmJ/L1\nHmfH4MgaFoUzERGRM0hignF7YQcu6tCM3yzYwO49e2h7dpPDQSscsJLI8IJWhjfqRuT8sTqFrjhY\ny+ipS7n7TytIT0pkYF5ujI6uYVA4ExEROQP1aN2EKWMKKCoqorCw52nddlpyIpNH92bkM0v4zvT3\nmDqmgP7qyuO46YUAEREROe0yU5OYdnMf2jVrxLeee4d3P9rrd0lxQ+FMREREoqJJejJ/uLUPuVmp\njJm6lNXb9vldUlxQOBMREZGoyclM5flb+5KVmsSoKUvZuLPM75ICT+FMREREoqplk/BboglmjHxm\nKVs/OeB3SYGmcCYiIiJR1y67Ec/f2ofKmlpGPLOYHfsq/S4psBTOREREJCY6N89i2tg+7N1fzcjJ\nS9hTXuV3SYGkcCYiIiIx06N1EyaP7s3WTw4waspS9lVU+11S4CiciYiISEz1bd+M39/Uiw9Kyrj5\n2WUcOFjjd0mBonAmIiIiMVd4Xg6PX9+T5R/v5bbn3qWyutbvkgJD4UxERER8MbhbCx4Z3oNFG3cz\nbvpyqmuPPa7nmUDhTERERHwzrFcrfj70fP65toR7ZxRTW+f8Lsl3GltTREREfHXThW0pr6plwrx1\npCcn8tA13TAzv8vyjcKZiIiI+O72wg7sr6rhidc20ig5xPivdjljA5rCmYiIiATCPVecS3lVDc8s\n2kxGaoi7Bp7rd0m+UDgTERGRQDAzfjokj/1VNTz2zw1kpIS49eL2fpcVcwpnIiIiEhgJCcbDw7pz\n4GAtD/5jLenJIUb0beN3WTGlcCYiIiKBkphgTLwun4rqWsbPWkWjlESG5rf0u6yYUVcaIiIiEjjJ\noQT+98YL6NuuKXe/VMzLa3b4XVLM6MqZiIiIBFJqUiLPjC5g5DNLGDd9OVPGFDCgU/bnrltTW0dF\ndS0VB2s54P1UVNd8Nn14eU14+vC64XWG5rfk8rzcGB/h51M4ExERkcDKSAkxbWwfrpv0Nt967h26\ntWzMAS90VUQEr4MnOLpAKMFIS04kPTmR9OQQAzp+fujzg8KZiIiIBFrj9CT+cEtffjxrFfsqqsnJ\nTA0Hq6RwuEpLDpF2ePpQ4AovT09OPPxZenLo8OdJicF9skvhTERERALvrMwUfn9Tb7/LiIlAxUYz\nG2Rm681so5nd73c9IiIiIrEWmHBmZonAk8BgIA+4wczy/K1KREREJLYCE86APsBG59wm59xB4I/A\nUJ9rEhEREYmpIIWzlsDWiPl/ectEREREzhhx90KAmd0G3AaQm5tLUVFRVPdXXl4e9X1IdKjt4pPa\nLT6p3eKX2i54ghTOtgGtI+Zbecv+g3NuEjAJoHfv3q6wsDCqRRUVFRHtfUh0qO3ik9otPqnd4pfa\nLniCdFtzGdDJzNqZWTJwPTDb55pEREREYiowV86cczVmNg6YDyQCU5xza3wuS0RERCSmAhPOAJxz\nc4A5ftchIiIi4pcg3dYUEREROeMpnImIiIgEiMKZiIiISIAonImIiIgEiMKZiIiISIAonImIiIgE\niMKZiIiISIAonImIiIgEiMKZiIiISICYc87vGk6ame0CPorybrKB3VHeh0SH2i4+qd3ik9otfqnt\nYucc59xZx1oprsNZLJjZO8653n7XISdObRef1G7xSe0Wv9R2waPbmiIiIiIBonAmIiIiEiAKZ8c2\nye8C5KSp7eKT2i0+qd3il9ouYPTMmYiIiEiA6MqZiIiISIAonH0BMxtkZuvNbKOZ3e93PXJ8zGyL\nma0ysxVm9o7f9Uj9zGyKme00s9URy5qa2StmtsH790t+1ihHq6fd/svMtnnn3Qozu8rPGuVoZtba\nzF4zs/fNbI2Zfc9brnMuYBTO6mFmicCTwGAgD7jBzPL8rUpOwKXOuXy9Hh54zwKDjlh2P7DAOdcJ\nWODNS7A8y9HtBjDRO+/ynXNzYlyTHFsNcI9zLg/oB9zh/V3TORcwCmf16wNsdM5tcs4dBP4IDPW5\nJpEGxTm3EPjkiMVDgWne9DTg6pgWJcdUT7tJwDnntjvn3vOmy4C1QEt0zgWOwln9WgJbI+b/5S2T\n4HPAy2b2rpnd5ncxcsJynXPbvekdQK6fxcgJGWdmK73bnro1FmBm1hboCSxB51zgKJxJQzTAOXcB\n4VvSd5jZJX4XJCfHhV8n1yvl8eF3QAcgH9gO/NrfcqQ+ZpYBzATucs6VRn6mcy4YFM7qtw1oHTHf\nylsmAeec2+b9uxP4K+Fb1BI/SsysBYD3706f65Hj4Jwrcc7VOufqgKfReRdIZpZEOJi94Jz7i7dY\n51zAKJzVbxnQyczamVkycD0w2+ea5BjMrJGZZR6aBq4AVn/xtyRgZgOjvenRwN98rEWO06E/7p5r\n0HkXOGZmwGRgrXPu0YiPdM4FjDqh/QLeq+CPAYnAFOfcL3wuSY7BzNoTvloGEAKmq92Cy8xeBAqB\nbKAEeACYBbwEtAE+Aq51zunh8wCpp90KCd/SdMAW4NsRzzFJAJjZAOANYBVQ5y3+EeHnznTOBYjC\nmYiIiEiA6LamiIiISIAonImIiIgEiMKZiIiISIAonImIiIgEiMKZiIiISIAonImI78zs62b2hYMt\nm9nZZvZnb3qMmT1xgvv40XGs86yZDT+R7Z5OZlZkZr392r+IBIPCmYj4zjk32zn38DHW+bdz7lSC\n0zHDWTwzs5DfNYjI6aFwJiJRY2ZtzWydd0XqAzN7wcwGmtmbZrbBzPp46x2+Euat+7iZvWVmmw5d\nyfK2FdnrfGvvStMGM3sgYp+zvEHv1xwa+N7MHgbSzGyFmb3gLRvlDdJdbGZ/iNjuJUfu+3OOaa2Z\nPe3t42UzS/M+O3zly8yyzWxLxPHNMrNXzGyLmY0zs7vNbLmZLTazphG7uMmrc3XE76eRN5j4Uu87\nQyO2O9vMXgUWnEpbiUhwKJyJSLR1JDwIdmfvZwQwALiX+q9mtfDWGQLUd0WtDzAM6A58M+J24M3O\nuV5Ab+BOM2vmnLsfqHDO5TvnbjSz84EfA5c553oA3zvBfXcCnnTOnQ986tVxLF2BbwAFwC+AA865\nnsDbwKiI9dKdc/nAd4Ap3rLxwKvOuT7ApcAj3vBkABcAw51zXz6OGkQkDiiciUi0bXbOrfIGxF4D\nLHDhoUlWAW3r+c4s51ydc+59ILeedV5xzu1xzlUAfyEcqCAcyIqBxUBrwkHqSJcBM5xzuwGOGKrm\nePa92Tm3wpt+9wuOI9Jrzrky59wuYB/wd2/5kb+HF72aFgJZZtaE8Bix95vZCqAISCU81A6Efw8a\nakekAdEzCiISbVUR03UR83XU/9+gyO9YPescOfacM7NCYCBwoXPugJkVEQ4yJ+J49h25Ti2Q5k3X\n8Nn/9B653+P9PRx1XF4dw5xz6yM/MLO+wP56ahSROKUrZyISry43s6be815XA28CjYG9XjDrDPSL\nWL/azJK86VcJ3wptBnDEM1+nYgvQy5s+2ZcXroPDg1Tvc87tA+YD3zUz8z7reYp1ikiAKZyJSLxa\nCswEVgIznXPvAPOAkJmtJfy82OKI9ScBK83sBefcGsLPfb3u3QJ99DTV9CvgdjNbDmSf5DYqve8/\nBdziLfs5kES4/jXevIg0UBZ+9ENEREREgkBXzkREREQCROFMREREJEAUzkREREQCROFMREREJEAU\nzkREREQCROFMREREJEAUzkREREQCROFMREREJED+Hyd55oFMjtQLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1189a3c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation\n",
      "Epoch 1, Overall loss = 3.7 and accuracy of 0.0667\n"
     ]
    }
   ],
   "source": [
    "def run_model(session, predict, loss_val, Xd, yd,\n",
    "              epochs=5, batch_size=64, print_every=100,\n",
    "              training=None, plot_losses=False):\n",
    "    # have tensorflow compute accuracy\n",
    "    correct_prediction = tf.equal(tf.argmax(predict,1), y)\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "    \n",
    "    # shuffle indicies\n",
    "    train_indicies = np.arange(Xd.shape[0])\n",
    "    np.random.shuffle(train_indicies)\n",
    "\n",
    "    training_now = training is not None\n",
    "    \n",
    "    # setting up variables we want to compute (and optimizing)\n",
    "    # if we have a training function, add that to things we compute\n",
    "    variables = [mean_loss,correct_prediction,accuracy]\n",
    "    if training_now:\n",
    "        variables[-1] = training\n",
    "    \n",
    "    # counter \n",
    "    iter_cnt = 0\n",
    "    for e in range(epochs):\n",
    "        # keep track of losses and accuracy\n",
    "        correct = 0\n",
    "        losses = []\n",
    "        # make sure we iterate over the dataset once\n",
    "        for i in range(int(math.ceil(Xd.shape[0]/batch_size))):\n",
    "            # generate indicies for the batch\n",
    "#             start_idx = (i*batch_size)%X_train.shape[0]\n",
    "            start_idx = (i*batch_size)%Xd.shape[0]\n",
    "            idx = train_indicies[start_idx:start_idx+batch_size]\n",
    "            \n",
    "            # create a feed dictionary for this batch\n",
    "            feed_dict = {X: Xd[idx,:],\n",
    "                         y: yd[idx],\n",
    "                         is_training: training_now }\n",
    "            # get batch size\n",
    "#             actual_batch_size = yd[i:i+batch_size].shape[0]\n",
    "            actual_batch_size = yd[idx].shape[0]\n",
    "            \n",
    "            # have tensorflow compute loss and correct predictions\n",
    "            # and (if given) perform a training step\n",
    "            loss, corr, _ = session.run(variables,feed_dict=feed_dict)\n",
    "            \n",
    "            # aggregate performance stats\n",
    "            losses.append(loss*actual_batch_size)\n",
    "            correct += np.sum(corr)\n",
    "            \n",
    "            # print every now and then\n",
    "            if training_now and (iter_cnt % print_every) == 0:\n",
    "                print(\"Iteration {0}: with minibatch training loss = {1:.3g} and accuracy of {2:.2g}\"\\\n",
    "                      .format(iter_cnt,loss,np.sum(corr)/actual_batch_size))\n",
    "            iter_cnt += 1\n",
    "        total_correct = correct/Xd.shape[0]\n",
    "        total_loss = np.sum(losses)/Xd.shape[0]\n",
    "        print(\"Epoch {2}, Overall loss = {0:.3g} and accuracy of {1:.3g}\"\\\n",
    "              .format(total_loss,total_correct,e+1))\n",
    "        if plot_losses:\n",
    "            plt.plot(losses)\n",
    "            plt.grid(True)\n",
    "            plt.title('Epoch {} Loss'.format(e+1))\n",
    "            plt.xlabel('minibatch number')\n",
    "            plt.ylabel('minibatch loss')\n",
    "            plt.show()\n",
    "    return total_loss,total_correct\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    with tf.device(\"/cpu:0\"): #\"/cpu:0\" or \"/gpu:0\" \n",
    "        sess.run(tf.global_variables_initializer())\n",
    "        print('Training')\n",
    "        run_model(sess,y_out,mean_loss,X_train,y_train,1,64,100,train_step,True)\n",
    "        print('Validation')\n",
    "        run_model(sess,y_out,mean_loss,X_val,y_val,1,64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a specific model\n",
    "\n",
    "In this section, we're going to specify a model for you to construct. The goal here isn't to get good performance (that'll be next), but instead to get comfortable with understanding the TensorFlow documentation and configuring your own model. \n",
    "\n",
    "Using the code provided above as guidance, and using the following TensorFlow documentation, specify a model with the following architecture:\n",
    "\n",
    "* 7x7 Convolutional Layer with 32 filters and stride of 1\n",
    "* ReLU Activation Layer\n",
    "* Spatial Batch Normalization Layer (trainable parameters, with scale and centering)\n",
    "* 2x2 Max Pooling layer with a stride of 2\n",
    "* Affine layer with 1024 output units\n",
    "* ReLU Activation Layer\n",
    "* Affine layer from 1024 input units to 10 outputs\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# clear old variables\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# define our input (e.g. the data that changes every batch)\n",
    "# The first dim is None, and gets sets automatically based on batch size fed in\n",
    "X = tf.placeholder(tf.float32, [None, 76, 66, 3])\n",
    "y = tf.placeholder(tf.int64, [None])\n",
    "is_training = tf.placeholder(tf.bool)\n",
    "\n",
    "# define model\n",
    "def complex_model(X,y,is_training):    \n",
    "    # setup variables\n",
    "    Wconv1 = tf.get_variable(\"Wconv1\", shape=[7, 7, 3, 32])\n",
    "    bconv1 = tf.get_variable(\"bconv1\", shape=[32])\n",
    "    W1 = tf.get_variable(\"W1\", shape=[33600, 1024])\n",
    "    b1 = tf.get_variable(\"b1\", shape=[1024])\n",
    "    W2 = tf.get_variable(\"W2\", shape=[1024, 10])\n",
    "    b2 = tf.get_variable(\"b2\", shape=[10])\n",
    "\n",
    "    # define our graph\n",
    "    a1 = tf.nn.conv2d(X, Wconv1, strides=[1,1,1,1], padding=\"VALID\") + bconv1\n",
    "    h1 = tf.nn.relu(a1)\n",
    "    h2 = tf.contrib.layers.batch_norm(h1, center=True, scale=True, is_training=is_training)\n",
    "    h3 = tf.layers.max_pooling2d(inputs=h2, pool_size=[2, 2], strides=2) \n",
    "    h3_flat = tf.reshape(h3,[-1,33600])\n",
    "    h4 = tf.matmul(h3_flat,W1) + b1\n",
    "    h5 = tf.nn.relu(h4)\n",
    "    h5_flat = tf.reshape(h5,[-1,1024])\n",
    "    y_out = tf.matmul(h5_flat,W2) + b2\n",
    "    return y_out\n",
    "\n",
    "\n",
    "y_out = complex_model(X,y,is_training)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make sure you're doing the right thing, use the following tool to check the dimensionality of your output (it should be 64 x 10, since our batches have size 64 and the output of the final affine layer should be 10, corresponding to our 10 classes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "233 ms ± 1.35 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n",
      "(64, 10)\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Now we're going to feed a random batch into the model \n",
    "# and make sure the output is the right size\n",
    "x = np.random.randn(64, 76, 66, 3)\n",
    "with tf.Session() as sess:\n",
    "    with tf.device(\"/cpu:0\"): #\"/cpu:0\" or \"/gpu:0\"\n",
    "        tf.global_variables_initializer().run()\n",
    "\n",
    "        ans = sess.run(y_out,feed_dict={X:x,is_training:True})\n",
    "        %timeit sess.run(y_out,feed_dict={X:x,is_training:True})\n",
    "        print(ans.shape)\n",
    "        print(np.array_equal(ans.shape, np.array([64, 10])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see the following from the run above \n",
    "\n",
    "`(64, 10)`\n",
    "\n",
    "`True`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPU!\n",
    "\n",
    "Now, we're going to try and start the model under the GPU device, the rest of the code stays unchanged and all our variables and operations will be computed using accelerated code paths. However, if there is no GPU, we get a Python exception and have to rebuild our graph. On a dual-core CPU, you might see around 50-80ms/batch running the above, while the Google Cloud GPUs (run below) should be around 2-5ms/batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "241 ms ± 6.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    with tf.Session() as sess:\n",
    "        with tf.device(\"/cpu:0\") as dev: #\"/cpu:0\" or \"/gpu:0\"\n",
    "            tf.global_variables_initializer().run()\n",
    "\n",
    "            ans = sess.run(y_out,feed_dict={X:x,is_training:True})\n",
    "            %timeit sess.run(y_out,feed_dict={X:x,is_training:True})\n",
    "except tf.errors.InvalidArgumentError:\n",
    "    print(\"no gpu found, please use Google Cloud if you want GPU acceleration\")    \n",
    "    # rebuild the graph\n",
    "    # trying to start a GPU throws an exception \n",
    "    # and also trashes the original graph\n",
    "    tf.reset_default_graph()\n",
    "    X = tf.placeholder(tf.float32, [None, 32, 32, 3])\n",
    "    y = tf.placeholder(tf.int64, [None])\n",
    "    is_training = tf.placeholder(tf.bool)\n",
    "    y_out = complex_model(X,y,is_training)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should observe that even a simple forward pass like this is significantly faster on the GPU. So for the rest of the assignment (and when you go train your models in assignment 3 and your project!), you should use GPU devices. However, with TensorFlow, the default device is a GPU if one is available, and a CPU otherwise, so we can skip the device specification from now on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model.\n",
    "\n",
    "Now that you've seen how to define a model and do a single forward pass of some data through it, let's  walk through how you'd actually train one whole epoch over your training data (using the complex_model you created provided above).\n",
    "\n",
    "Make sure you understand how each TensorFlow function used below corresponds to what you implemented in your custom neural network implementation.\n",
    "\n",
    "First, set up an **RMSprop optimizer** (using a 1e-3 learning rate) and a **cross-entropy loss** function. See the TensorFlow documentation for more information\n",
    "* Layers, Activations, Loss functions : https://www.tensorflow.org/api_guides/python/nn\n",
    "* Optimizers: https://www.tensorflow.org/api_guides/python/train#Optimizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Inputs\n",
    "#     y_out: is what your model computes\n",
    "#     y: is your TensorFlow variable with label information\n",
    "# Outputs\n",
    "#    mean_loss: a TensorFlow variable (scalar) with numerical loss\n",
    "#    optimizer: a TensorFlow optimizer\n",
    "# This should be ~3 lines of code!\n",
    "total_loss = tf.losses.softmax_cross_entropy(logits=y_out, onehot_labels=tf.one_hot(y,10))\n",
    "mean_loss = tf.reduce_mean(total_loss)\n",
    "optimizer = tf.train.RMSPropOptimizer(1e-3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# batch normalization in tensorflow requires this extra dependency\n",
    "extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n",
    "with tf.control_dependencies(extra_update_ops):\n",
    "    train_step = optimizer.minimize(mean_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model\n",
    "Below we'll create a session and train the model over one epoch. You should see a loss of 1.4 to 1.8 and an accuracy of 0.4 to 0.5. There will be some variation due to random seeds and differences in initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training\n",
      "Iteration 0: with minibatch training loss = 3.2 and accuracy of 0.12\n",
      "Epoch 1, Overall loss = 0.963 and accuracy of 0.706\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.9628788291463134, 0.70607661822985468)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "\n",
    "sess.run(tf.global_variables_initializer())\n",
    "print('Training')\n",
    "run_model(sess,y_out,mean_loss,X_train,y_train,1,64,100,train_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the accuracy of the model.\n",
    "\n",
    "Let's see the train and test code in action -- feel free to use these methods when evaluating the models you develop below. You should see a loss of 1.3 to 1.5 with an accuracy of 0.45 to 0.55."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation\n",
      "Epoch 1, Overall loss = 26.7 and accuracy of 0.0167\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(26.709161122639973, 0.016666666666666666)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Validation')\n",
    "run_model(sess,y_out,mean_loss,X_val,y_val,1,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a _great_ model on CIFAR-10!\n",
    "\n",
    "Now it's your job to experiment with architectures, hyperparameters, loss functions, and optimizers to train a model that achieves ** >= 70% accuracy on the validation set** of CIFAR-10. You can use the `run_model` function from above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Things you should try:\n",
    "- **Filter size**: Above we used 7x7; this makes pretty pictures but smaller filters may be more efficient\n",
    "- **Number of filters**: Above we used 32 filters. Do more or fewer do better?\n",
    "- **Pooling vs Strided Convolution**: Do you use max pooling or just stride convolutions?\n",
    "- **Batch normalization**: Try adding spatial batch normalization after convolution layers and vanilla batch normalization after affine layers. Do your networks train faster?\n",
    "- **Network architecture**: The network above has two layers of trainable parameters. Can you do better with a deep network? Good architectures to try include:\n",
    "    - [conv-relu-pool]xN -> [affine]xM -> [softmax or SVM]\n",
    "    - [conv-relu-conv-relu-pool]xN -> [affine]xM -> [softmax or SVM]\n",
    "    - [batchnorm-relu-conv]xN -> [affine]xM -> [softmax or SVM]\n",
    "- **Use TensorFlow Scope**: Use TensorFlow scope and/or [tf.layers](https://www.tensorflow.org/api_docs/python/tf/layers) to make it easier to write deeper networks. See [this tutorial](https://www.tensorflow.org/tutorials/layers) for making how to use `tf.layers`. \n",
    "- **Use Learning Rate Decay**: [As the notes point out](http://cs231n.github.io/neural-networks-3/#anneal), decaying the learning rate might help the model converge. Feel free to decay every epoch, when loss doesn't change over an entire epoch, or any other heuristic you find appropriate. See the [Tensorflow documentation](https://www.tensorflow.org/versions/master/api_guides/python/train#Decaying_the_learning_rate) for learning rate decay.\n",
    "- **Global Average Pooling**: Instead of flattening and then having multiple affine layers, perform convolutions until your image gets small (7x7 or so) and then perform an average pooling operation to get to a 1x1 image picture (1, 1 , Filter#), which is then reshaped into a (Filter#) vector. This is used in [Google's Inception Network](https://arxiv.org/abs/1512.00567) (See Table 1 for their architecture).\n",
    "- **Regularization**: Add l2 weight regularization, or perhaps use [Dropout as in the TensorFlow MNIST tutorial](https://www.tensorflow.org/get_started/mnist/pros)\n",
    "\n",
    "### Tips for training\n",
    "For each network architecture that you try, you should tune the learning rate and regularization strength. When doing this there are a couple important things to keep in mind:\n",
    "\n",
    "- If the parameters are working well, you should see improvement within a few hundred iterations\n",
    "- Remember the coarse-to-fine approach for hyperparameter tuning: start by testing a large range of hyperparameters for just a few training iterations to find the combinations of parameters that are working at all.\n",
    "- Once you have found some sets of parameters that seem to work, search more finely around these parameters. You may need to train for more epochs.\n",
    "- You should use the validation set for hyperparameter search, and we'll save the test set for evaluating your architecture on the best parameters as selected by the validation set.\n",
    "\n",
    "### Going above and beyond\n",
    "If you are feeling adventurous there are many other features you can implement to try and improve your performance. You are **not required** to implement any of these; however they would be good things to try for extra credit.\n",
    "\n",
    "- Alternative update steps: For the assignment we implemented SGD+momentum, RMSprop, and Adam; you could try alternatives like AdaGrad or AdaDelta.\n",
    "- Alternative activation functions such as leaky ReLU, parametric ReLU, ELU, or MaxOut.\n",
    "- Model ensembles\n",
    "- Data augmentation\n",
    "- New Architectures\n",
    "  - [ResNets](https://arxiv.org/abs/1512.03385) where the input from the previous layer is added to the output.\n",
    "  - [DenseNets](https://arxiv.org/abs/1608.06993) where inputs into previous layers are concatenated together.\n",
    "  - [This blog has an in-depth overview](https://chatbotslife.com/resnets-highwaynets-and-densenets-oh-my-9bb15918ee32)\n",
    "\n",
    "If you do decide to implement something extra, clearly describe it in the \"Extra Credit Description\" cell below.\n",
    "\n",
    "### What we expect\n",
    "At the very least, you should be able to train a ConvNet that gets at **>= 70% accuracy on the validation set**. This is just a lower bound - if you are careful it should be possible to get accuracies much higher than that! Extra credit points will be awarded for particularly high-scoring models or unique approaches.\n",
    "\n",
    "You should use the space below to experiment and train your network. The final cell in this notebook should contain the training and validation set accuracies for your final trained network.\n",
    "\n",
    "Have fun and happy training!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Feel free to play with this cell\n",
    "from classifiers import my_model\n",
    "\n",
    "# def my_model(X,y,is_training):\n",
    "#     # setup variables\n",
    "#     Wconv1 = tf.get_variable(\"Wconv1\", shape=[3, 3, 3, 64])\n",
    "#     bconv1 = tf.get_variable(\"bconv1\", shape=[64])\n",
    "#     Wconv2 = tf.get_variable(\"Wconv2\", shape=[5, 5, 64, 32])\n",
    "#     bconv2 = tf.get_variable(\"bconv2\", shape=[32])\n",
    "#     W1 = tf.get_variable(\"W1\", shape=[33600, 1024])\n",
    "#     b1 = tf.get_variable(\"b1\", shape=[1024])\n",
    "#     W2 = tf.get_variable(\"W2\", shape=[1024, 512])\n",
    "#     b2 = tf.get_variable(\"b2\", shape=[512])\n",
    "#     W3 = tf.get_variable(\"W3\", shape=[512, 128])\n",
    "#     b3 = tf.get_variable(\"b3\", shape=[128])\n",
    "#     W4 = tf.get_variable(\"W4\", shape=[128, 10])\n",
    "#     b4 = tf.get_variable(\"b4\", shape=[10])\n",
    "\n",
    "#     # first conv layer (with relu and bn)\n",
    "#     a1 = tf.nn.conv2d(X, Wconv1, strides=[1,1,1,1], padding=\"VALID\") + bconv1\n",
    "#     h1 = tf.nn.relu(a1)\n",
    "#     h2 = tf.contrib.layers.batch_norm(h1, center=True, scale=True, is_training=is_training, scope='bn')\n",
    "    \n",
    "#     # second conv layer (with only relu)\n",
    "#     a11 = tf.nn.conv2d(h2, Wconv2, strides=[1,1,1,1], padding=\"VALID\") + bconv2\n",
    "#     h11 = tf.nn.relu(a11)\n",
    "    \n",
    "#     # pool layer\n",
    "#     h3 = tf.layers.max_pooling2d(inputs=h11, pool_size=[2, 2], strides=2) \n",
    "    \n",
    "#     # three affine layers\n",
    "#     h3_flat = tf.reshape(h3,[-1,5408])\n",
    "#     h4 = tf.matmul(h3_flat,W1) + b1\n",
    "#     h5 = tf.nn.relu(h4)\n",
    "    \n",
    "#     h5_flat = tf.reshape(h5,[-1,1024])\n",
    "#     h6 = tf.matmul(h5_flat,W2) + b2\n",
    "#     h7 = tf.nn.relu(h6)\n",
    "    \n",
    "#     h7_flat = tf.reshape(h7,[-1,512])\n",
    "#     h8 = tf.matmul(h7_flat,W3) + b3\n",
    "#     h9 = tf.nn.relu(h8)\n",
    "    \n",
    "#     # output layer (softmax)\n",
    "#     h9_flat = tf.reshape(h9,[-1,128])\n",
    "#     y_out = tf.matmul(h9_flat,W4) + b4\n",
    "    \n",
    "#     return y_out\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, 76, 66, 3])\n",
    "y = tf.placeholder(tf.int64, [None])\n",
    "is_training = tf.placeholder(tf.bool)\n",
    "\n",
    "# softmax loss function with RMSProp optimizer\n",
    "y_out = my_model(X,y,is_training)\n",
    "total_loss = tf.losses.softmax_cross_entropy(logits=y_out, onehot_labels=tf.one_hot(y,10))\n",
    "mean_loss = tf.reduce_mean(total_loss)\n",
    "optimizer = tf.train.RMSPropOptimizer(1e-3)\n",
    "\n",
    "# batch normalization in tensorflow requires this extra dependency\n",
    "extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n",
    "with tf.control_dependencies(extra_update_ops):\n",
    "    train_step = optimizer.minimize(mean_loss)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training\n",
      "Iteration 0: with minibatch training loss = 2.7 and accuracy of 0.19\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-75-009f529fddbe>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mglobal_variables_initializer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Training'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0mrun_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_out\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mmean_loss\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m20\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m64\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtrain_step\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Validation'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0mrun_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_out\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mmean_loss\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mX_val\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_val\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m64\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-62-9452a39cdb1d>\u001b[0m in \u001b[0;36mrun_model\u001b[0;34m(session, predict, loss_val, Xd, yd, epochs, batch_size, print_every, training, plot_losses)\u001b[0m\n\u001b[1;32m     41\u001b[0m             \u001b[0;31m# have tensorflow compute loss and correct predictions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     42\u001b[0m             \u001b[0;31m# and (if given) perform a training step\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 43\u001b[0;31m             \u001b[0mloss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvariables\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mfeed_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfeed_dict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     44\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     45\u001b[0m             \u001b[0;31m# aggregate performance stats\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/benlimonchik/Desktop/assignment3/.env/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    776\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    777\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 778\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    779\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    780\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/benlimonchik/Desktop/assignment3/.env/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    980\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    981\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m--> 982\u001b[0;31m                              feed_dict_string, options, run_metadata)\n\u001b[0m\u001b[1;32m    983\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    984\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/benlimonchik/Desktop/assignment3/.env/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1030\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1031\u001b[0m       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,\n\u001b[0;32m-> 1032\u001b[0;31m                            target_list, options, run_metadata)\n\u001b[0m\u001b[1;32m   1033\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1034\u001b[0m       return self._do_call(_prun_fn, self._session, handle, feed_dict,\n",
      "\u001b[0;32m/Users/benlimonchik/Desktop/assignment3/.env/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1037\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1038\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1039\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1040\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1041\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/benlimonchik/Desktop/assignment3/.env/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1019\u001b[0m         return tf_session.TF_Run(session, options,\n\u001b[1;32m   1020\u001b[0m                                  \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1021\u001b[0;31m                                  status, run_metadata)\n\u001b[0m\u001b[1;32m   1022\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1023\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Feel free to play with this cell\n",
    "# This default code creates a session\n",
    "# and trains your model for 10 epochs\n",
    "# then prints the validation set accuracy\n",
    "sess = tf.Session()\n",
    "\n",
    "sess.run(tf.global_variables_initializer())\n",
    "print('Training')\n",
    "run_model(sess,y_out,mean_loss,X_train,y_train,20,64,100,train_step,True)\n",
    "print('Validation')\n",
    "run_model(sess,y_out,mean_loss,X_val,y_val,1,64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training\n",
      "Iteration 0: with minibatch training loss = 1.32 and accuracy of 0.56\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-76-840d44e9c2a4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m# We're looking for >= 70% accuracy on Validation\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Training'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mrun_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_out\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mmean_loss\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m64\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtrain_step\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Validation'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0mrun_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_out\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mmean_loss\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mX_val\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_val\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m64\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-62-9452a39cdb1d>\u001b[0m in \u001b[0;36mrun_model\u001b[0;34m(session, predict, loss_val, Xd, yd, epochs, batch_size, print_every, training, plot_losses)\u001b[0m\n\u001b[1;32m     41\u001b[0m             \u001b[0;31m# have tensorflow compute loss and correct predictions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     42\u001b[0m             \u001b[0;31m# and (if given) perform a training step\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 43\u001b[0;31m             \u001b[0mloss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvariables\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mfeed_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfeed_dict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     44\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     45\u001b[0m             \u001b[0;31m# aggregate performance stats\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/benlimonchik/Desktop/assignment3/.env/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    776\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    777\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 778\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    779\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    780\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/benlimonchik/Desktop/assignment3/.env/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    980\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    981\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m--> 982\u001b[0;31m                              feed_dict_string, options, run_metadata)\n\u001b[0m\u001b[1;32m    983\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    984\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/benlimonchik/Desktop/assignment3/.env/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1030\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1031\u001b[0m       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,\n\u001b[0;32m-> 1032\u001b[0;31m                            target_list, options, run_metadata)\n\u001b[0m\u001b[1;32m   1033\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1034\u001b[0m       return self._do_call(_prun_fn, self._session, handle, feed_dict,\n",
      "\u001b[0;32m/Users/benlimonchik/Desktop/assignment3/.env/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1037\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1038\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1039\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1040\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1041\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/benlimonchik/Desktop/assignment3/.env/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1019\u001b[0m         return tf_session.TF_Run(session, options,\n\u001b[1;32m   1020\u001b[0m                                  \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1021\u001b[0;31m                                  status, run_metadata)\n\u001b[0m\u001b[1;32m   1022\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1023\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Test your model here, and make sure \n",
    "# the output of this cell is the accuracy\n",
    "# of your best model on the training and val sets\n",
    "# We're looking for >= 70% accuracy on Validation\n",
    "print('Training')\n",
    "run_model(sess,y_out,mean_loss,X_train,y_train,5,64,100,train_step,True)\n",
    "print('Validation')\n",
    "run_model(sess,y_out,mean_loss,X_val,y_val,1,64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Describe what you did here\n",
    "In this cell you should also write an explanation of what you did, any additional features that you implemented, and any visualizations or graphs that you make in the process of training and evaluating your network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I designed a deep network of the following architecture: [conv-relu-bn-conv-relu-pool] -> [affine]x3 -> [softmax]. For the first convolutional layer I used a small filter size of 3x3 to capture low level features such as edges. Then, in the next convolutional layer I increased the filter size to 5x5 to capture more abstract features from the first conv layer (like colored blobs or small object parts). In addition I used less filters in the second convolutional layer. After the convolutional layers, I used a pooling layer to reduce the data size by 75% to farther manipulate it in a reasonable time. After the pooling layer, I used three affine layers to smoothly reduce the number of neurons to the final number of classes. I used a reduction factor of x4 for the affine layers. I used loss graphs and loss/accuracy reports to visualize and evaluate my model's performance and tune the different hyperparameters accordingly.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Set - Do this only once\n",
    "Now that we've gotten a result that we're happy with, we test our final model on the test set. This would be the score we would achieve on a competition. Think about how this compares to your validation set accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test\n",
      "Epoch 1, Overall loss = 3.46 and accuracy of 0.09\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(3.4570130920410156, 0.089999999999999997)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Test')\n",
    "run_model(sess,y_out,mean_loss,X_test,y_test,1,64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Going further with TensorFlow\n",
    "\n",
    "The next assignment will make heavy use of TensorFlow. You might also find it useful for your projects. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extra Credit Description\n",
    "If you implement any additional features for extra credit, clearly describe them here with pointers to any code in this or other files if applicable."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}