CNN Tutorial MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST Convolution Classifier Tutorial with Hidden Layer Visualization\n",
    "\n",
    "This notebook is intended as an introductory tutorial to convolution networks and the MNIST dataset. This network can be used to achieve good results on the Kaggle MNIST challenge, and the hidden layer visualization reveals parts of the network's decision-making process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and Processing Data:\n",
    "\n",
    "An increasingly prevalent step in image classification networks is preprocessing and batch normalization. For this dataset, we will limit our preprocessing to rescaling and normalizing the mean and standard deviation, but please check out my preprocessing tutorial for a more elaborate approach. \n",
    "\n",
    "To start, we'll import the MNIST data without one-hot encoding - we'll do that ourselves later. If you'd rather avoid one-hot encoding manually, just set `one_hot=True` and make the necessary adjustments. We will also load the data manually, even though just using the mnist.train.next_batch command can be easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from tensorflow.examples.tutorials.mnist import input_data # downloads MNIST images\n",
    "\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=False) \n",
    "\n",
    "data, labels = mnist.train.next_batch(50000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then divide the dataset into train and validation datasets, and preprocess them. We subtract the per-pixel mean (the average of each pixel over the entire training set, which sets the mean to zero, and the standard deviation, in this case, to about .25."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean: 0.000020, Std: 0.259339\n"
     ]
    }
   ],
   "source": [
    "data = data.reshape(50000, 28, 28, 1)\n",
    "\n",
    "temp_data = data - np.mean(data, axis=0)\n",
    "\n",
    "train_dataset = temp_data[:45000]\n",
    "train_labels = labels[:45000]\n",
    "\n",
    "valid_dataset = temp_data[45000:]\n",
    "valid_labels = labels[45000:]\n",
    "\n",
    "print(\"Mean: %f, Std: %f\" % (np.mean(train_dataset), np.std(train_dataset)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Verify data\n",
    "\n",
    "We can use the matplotlib.pyplot module to view some of the sample images and their corresponding labels. The data is encoded by defaut as a single `(784, 1)` Numpy array, so we need to manually reshape it in order to display the correct image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number: 9\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEptJREFUeJzt3X1sXfV5B/Dvc699E/yWxAQyYwJJ1JQuykSYvGwVbKV0\nUGCgQNpFRH1JpqgpU9fCWnWLskkL/2xoWsmYtFYyJSUMShlqENnGaIM7NWLqC4aleakHYakjEjl2\nIK+245fr++wPH1o3+Dy/m3vuvefaz/cjRbHvc8+9v1z7m3Pt5/x+P1FVEJE/mbQHQETpYPiJnGL4\niZxi+ImcYviJnGL4iZxi+ImcYviJnGL4iZyqq+qTNTRq/bzWaj4lkSvjZ08hPzwkxdw3UfhF5HYA\njwLIAvimqj5s3b9+XiuW/smXkzwlERl+8a1Hir5vyW/7RSQL4J8B3AFgBYD1IrKi1McjoupK8jP/\nagBvqeoRVR0D8B0Aa8ozLCKqtCThbwfw9pTPj0W3/RoR2Swi3SLSnR8eSvB0RFROFf9tv6p2qmqH\nqnbUNTRW+umIqEhJwn8cwOIpn18d3UZEM0CS8L8KYLmILBWRHID7AOwuz7CIqNJKbvWpal5E/gzA\n9zDZ6tuhqofKNjIiqqhEfX5VfRHAi2UaCxFVES/vJXKK4SdyiuEncorhJ3KK4SdyiuEncqqq8/kp\nRgU3TZKkj53mhk5FzUqPp0mOT/jcMwHP/EROMfxETjH8RE4x/EROMfxETjH8RE6x1VcOCdthwXZc\ngroUEj530uMNwVZc4NSkoVOXVQ88d/CfNQtagTzzEznF8BM5xfATOcXwEznF8BM5xfATOcXwEznF\nPn+xrF56wl55ZsKuS6CeGY8fQGbcPjY7VvpjA+HrCCyFwHffRM5uphdypR+vgecOjS14nUDoOoAa\nuE6AZ34ipxh+IqcYfiKnGH4ipxh+IqcYfiKnGH4ipxL1+UWkF8B5ABMA8qraUY5BpSLQq7f62cE+\nfD5QH7OfvG7EPr7uQvzxuUF7cGNNWbOe/+y7Zn3rB//TrGeMixy29dxtHrtiYb9Z/++eD5j1+f8T\nfyFAxnjNAABz7EZ86DoB2C+r/e1WpWsAynGRz0dV9Z0yPA4RVRHf9hM5lTT8CuBlEXlNRDaXY0BE\nVB1J3/bfpKrHReRKAHtE5H9Vde/UO0T/KWwGgLqWBQmfjojKJdGZX1WPR38PAHgewOpp7tOpqh2q\n2lHX0Jjk6YiojEoOv4g0ikjzex8DuA3AwXINjIgqK8nb/kUAnheR9x7n26r6UllGRUQVV3L4VfUI\ngOvLOJbKStDHB+xefnDO/Kj95PXDdj03aA8udzb+QoLBdnvS+28/sM+sNwcuMvjB2d806xljsYO1\nS35mHnt17pRZ/9Ite8z6n175qdha3XOXm8eGvmEmAs340Hx+Md5zJ9pa/BKw1UfkFMNP5BTDT+QU\nw0/kFMNP5BTDT+TU7Fm6O9TKq2CrLxuaknvBfuxQK2/OabuXmB2Kr19z/1Hz2EzghesbaTHrPe/+\nhlmvf6o1trbgpyfMY498+iqz/tefetasf+La+Fbif1y42Tx2ot4+LxbqzXLw+80sh5aCL1MrkGd+\nIqcYfiKnGH4ipxh+IqcYfiKnGH4ipxh+IqdmT58/JNQ7DdSt5bdDU3rrRuw+fv15e3ltGbePP/xA\nfNN5bYM9LXY0sBf1K2/ay2N/8J9GzXrmyBuxtdCXZMmuuWb9sZt+36z/7hW9sbXgcuqhbdNDg58B\neOYncorhJ3KK4SdyiuEncorhJ3KK4SdyiuEncmpm9fkr2FsNz/ePv0OoZxya758dsZvK/b9j73S0\n6fouewCGhqzdp7/sDbvXnhk8Z9Zlwbz44rj9wsnZQbM+sPcas45P9Np1Q7WWz54Wl+4mokpi+Imc\nYviJnGL4iZxi+ImcYviJnGL4iZwK9vlFZAeAuwAMqOrK6LZWAM8CWAKgF8A6VT1duWHWNusaAADI\n5O36O791mVn/0v27zPqIxs/nb87YmwY88vV1Zr29e8isF1rsscuF+MUOMsP29t9Q+3ULrZ3fljsT\nWxtsy5rHhrZV18BpM3SdQKrXEUSKOfM/AeD2i27bAqBLVZcD6Io+J6IZJBh+Vd0L4OLlYNYA2Bl9\nvBPAPWUeFxFVWKk/8y9S1b7o4xMAFpVpPERUJYl/4aeqCuOqexHZLCLdItKdH7Z/fiSi6ik1/P0i\n0gYA0d8DcXdU1U5V7VDVjroGe4IKEVVPqeHfDWBD9PEGAC+UZzhEVC3B8IvIMwB+BOA6ETkmIpsA\nPAzgVhE5DOAPo8+JaAYJ9vlVdX1M6WNlHkuY1Rut9DrqxuOLvaw+JNDnb1nbZ9ZDrF7+Qy990jx2\n6X67166ZUMPa/rdJwXhxrBoAZOxz02j7mFlf03QwtrZ2S3wNAO7e/hdmXQLr+ldrTn4SvMKPyCmG\nn8gphp/IKYafyCmGn8gphp/IqZm1dHcSCVsv1tLeoVZf7xr7/9i/u/aHZt2asgsA4xr/ZVz8fbsn\nlRmz69lhe/9xGbJbhWJM29W8vXT30U329uCPfuQJsz7X+Jrf2v1589jchcB04lzSb6hkh5cDz/xE\nTjH8RE4x/EROMfxETjH8RE4x/EROMfxETvnp84cEt+g2Dg30bG9c9aZZzwQuFDg13mTWdz5+8eLK\nv9LWf9Z+bmNpbQCQc/bSazpob6NdGIt//OFbVprHXv9HPWa9JWNfY7Dx8H2xtaZdzeax4w1mOSw0\nxdyqc4tuIqokhp/IKYafyCmGn8gphp/IKYafyCmGn8ip2dPnD/VGk/RdAcBoxY832E/+523fN+u9\n4wvNevfZa836VV0X76P6KzJuz9eX0+fMeuGMfZ1AYcTutZ+8/8OxtRs2HjCP/VCTvaT5lw/9sVlv\nfmxefLHFPNRcvwFA5ZeKrwKe+YmcYviJnGL4iZxi+ImcYviJnGL4iZxi+ImcCvb5RWQHgLsADKjq\nyui2bQA+B+BkdLetqvpipQZZlIR92eA220Z9rMXu89cHHrwxM2rWX+/6kFlfWjgdW5NRe75+YeEC\ns44r7fqRT9v161b3xtYa6+x/d3Ngvv7g0Fz7+AR7LSTt44euE6iFywSKOfM/AWC61SK2q+qq6E+6\nwSeiSxYMv6ruBRB/CRkRzUhJfub/oojsF5EdIhJ470hEtabU8H8DwDIAqwD0Afha3B1FZLOIdItI\nd37YXg+OiKqnpPCrar+qTqhqAcBjAFYb9+1U1Q5V7ahraCx1nERUZiWFX0Tapnx6L4CD5RkOEVVL\nMa2+ZwDcDGChiBwD8DcAbhaRVZjsWPQCsPc7JqKaEwy/qq6f5ubHKzCWVIX6slZ9wm43Y67Yc+qb\nMxfMul5n/67k2B2tsbVQP3vlWntt/Pk5e2z1w2fM+ttn5sfWVs0/Zh7bELj+IfRFE02vmx7ay6EW\n8Ao/IqcYfiKnGH4ipxh+IqcYfiKnGH4ip2bP0t0BwSmcCaZ4Nh23D94zZE/JvbnB3sL726u/adbP\nd8T3Gt+dsLf3PpE3lrcG8M64vZX1wIj9+MsvPxlbW1h/3jw2F2iR5scSfPuGWnFJ6zMAz/xETjH8\nRE4x/EROMfxETjH8RE4x/EROMfxETrnp8weX7g5M/7Rmj9aN2Mf+4+67zHrhbnvx4480vmHW50r8\n8tyh6cLPne4w64eet69RCF0/sW7jD+w7GI6MXmnW2/49Z9Y1Ez84DZz2gnX2+YlopmL4iZxi+Imc\nYviJnGL4iZxi+ImcYviJnJo1ff4kS28DSLTFd6jX3f7DvFl/7scfN+tPNd9p1jMT8YPLjtn/sKZf\nDJr1a/qPmvUj2+OXDQeAkUJ9bG1Oxt4+/KnnPmbWrzppb+E9uiD+uQt1dqO+kA008kOnzRlwHQDP\n/EROMfxETjH8RE4x/EROMfxETjH8RE4x/EROBfv8IrIYwJMAFmGy292pqo+KSCuAZwEsAdALYJ2q\nnq7cUJMJzr9O0JeVgt1Lz47ZFwLkztjXATT22v1wGY2vy8iYeawO2/P9j278gFn/+LKfmvVxzcbW\nzuYbzGNbewLr9s+Nf2wAyM+J/6Ialx8AADSQjNB8/9nS588D+IqqrgDwewC+ICIrAGwB0KWqywF0\nRZ8T0QwRDL+q9qnq69HH5wH0AGgHsAbAzuhuOwHcU6lBElH5XdLP/CKyBMANAH4CYJGq9kWlE5j8\nsYCIZoiiwy8iTQC+C+BBVT03taaqipir30Vks4h0i0h3fngo0WCJqHyKCr+I1GMy+E+r6q7o5n4R\naYvqbQAGpjtWVTtVtUNVO+oaGssxZiIqg2D4RUQAPA6gR1UfmVLaDWBD9PEGAC+Uf3hEVCnFTOm9\nEcBnABwQkX3RbVsBPAzgX0VkE4CjANZVZohlEmi9aGAKp2bj23maCT24Xc6M2a0+GR4N1OOnturQ\nsH3sPHsL7nkfPWHWC4Ee6rxsfCvxWy/dYh67+Izdphxttft14w3xY5sw2oAAYHQoJ+tJl/augVZg\nMPyq+grih2pPuCaimsUr/IicYviJnGL4iZxi+ImcYviJnGL4iZyaNUt3B/uqgf/mCoFXYmJOfC0/\n137y7Fz7ybMX7CfPZENNZeNCgsCxPV9tM+v3XvGqWW+fc8as73jzw7G1q7vsqcqhPv5Yk/26Txhf\nl9CU3kKgz18LffqkeOYncorhJ3KK4SdyiuEncorhJ3KK4SdyiuEncmrW9PmD8/UT9vnNXn5oe28E\nmsaSM8uhtQbqcvGDH1zWYh675ZZ/M+vzs/bSay0Ze5vs7N55sbWRy+0lzccb7X936PoKq5cfmq8f\n/H6aAfP1Q3jmJ3KK4SdyiuEncorhJ3KK4SdyiuEncorhJ3Jq9vT5Ewr1fQtG3zYfWLe/UG/XrfXl\nAaBugT247Gj8YgNf/dunzWNvu+yUWW/I2NcgPNjXYdbrz8dfBDEy3z73BLfRDq2tb9Ud9PFDeOYn\ncorhJ3KK4SdyiuEncorhJ3KK4SdyiuEncirY5xeRxQCeBLAIkzPXO1X1URHZBuBzAE5Gd92qqi9W\naqCJJezLWusBBPvRobUEcoF564HrAGQivvbQ9s+axz6UsJ8d3Ke+OcGxoVNTwrEleezZoJiLfPIA\nvqKqr4tIM4DXRGRPVNuuqv9QueERUaUEw6+qfQD6oo/Pi0gPgPZKD4yIKuuS3hiJyBIANwD4SXTT\nF0Vkv4jsEJEFMcdsFpFuEenOD9tLQhFR9RQdfhFpAvBdAA+q6jkA3wCwDMAqTL4z+Np0x6lqp6p2\nqGpHXUNjGYZMROVQVPhFpB6TwX9aVXcBgKr2q+qEqhYAPAZgdeWGSUTlFgy/iAiAxwH0qOojU26f\nur3rvQAOln94RFQpxfy2/0YAnwFwQET2RbdtBbBeRFZhsv3XC+DzFRnhDBCc/hmaehr6Lzi08ndw\n6fDSVXRqa9JptUk4aOWFFPPb/lcw/UtVuz19IgriFX5ETjH8RE4x/EROMfxETjH8RE4x/EROcenu\nYiXoC1e0X43ADuEVvAYgMfbaU8UzP5FTDD+RUww/kVMMP5FTDD+RUww/kVMMP5FTolq9RrCInARw\ndMpNCwG8U7UBXJpaHVutjgvg2EpVzrFdq6pXFHPHqob/fU8u0q2q9gbvKanVsdXquACOrVRpjY1v\n+4mcYviJnEo7/J0pP7+lVsdWq+MCOLZSpTK2VH/mJ6L0pH3mJ6KUpBJ+EbldRN4QkbdEZEsaY4gj\nIr0ickBE9olId8pj2SEiAyJycMptrSKyR0QOR39Pu01aSmPbJiLHo9dun4jcmdLYFovIf4nIz0Xk\nkIg8EN2e6mtnjCuV163qb/tFJAvgTQC3AjgG4FUA61X151UdSAwR6QXQoaqp94RF5A8ADAJ4UlVX\nRrf9PYBTqvpw9B/nAlX9yxoZ2zYAg2nv3BxtKNM2dWdpAPcA2IgUXztjXOuQwuuWxpl/NYC3VPWI\nqo4B+A6ANSmMo+ap6l4Apy66eQ2AndHHOzH5zVN1MWOrCarap6qvRx+fB/DeztKpvnbGuFKRRvjb\nAbw95fNjqK0tvxXAyyLymohsTnsw01gUbZsOACcALEpzMNMI7txcTRftLF0zr10pO16XG3/h9343\nqeoqAHcA+EL09rYm6eTPbLXUrilq5+ZqmWZn6V9K87Urdcfrcksj/McBLJ7y+dXRbTVBVY9Hfw8A\neB61t/tw/3ubpEZ/D6Q8nl+qpZ2bp9tZGjXw2tXSjtdphP9VAMtFZKmI5ADcB2B3CuN4HxFpjH4R\nAxFpBHAbam/34d0ANkQfbwDwQopj+TW1snNz3M7SSPm1q7kdr1W16n8A3InJ3/j/H4C/SmMMMeNa\nBuBn0Z9DaY8NwDOYfBs4jsnfjWwCcDmALgCHAbwMoLWGxvYvAA4A2I/JoLWlNLabMPmWfj+AfdGf\nO9N+7YxxpfK68Qo/Iqf4Cz8ipxh+IqcYfiKnGH4ipxh+IqcYfiKnGH4ipxh+Iqf+H5FwpoA9XVVK\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114ae8f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import random\n",
    "\n",
    "n = random.randint(0, len(valid_dataset))\n",
    "\n",
    "image = valid_dataset[n] #mnist.train.next_batch(1)\n",
    "print(\"Number: {}\".format(valid_labels[n]))\n",
    "\n",
    "plt.imshow(image.reshape(28, 28))\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Class Distribution:\n",
    "\n",
    "An important step when dealing with any sort of classification is to ensure that our data is relatively uniform across classes. We can visualize the class distribution using a histogram provided my the matplotlib module. You can see that the labels are actually not uniformly distributed, and we could prune the labels if we so desired."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADzJJREFUeJzt3V+MHWd9xvHvg02DgaYkytYKtql9YVo5liCNZblNVbW4\nbVyBcK4iI0GsKoov4rahQgKbm6oXllKpQjRSiWQBjSMolsUfxaIJrTFBVaUmYQNpjR2sWCTBdp14\noaKGXoQ6/HqxL/XJxtbuxrs79r7fj3R03vmdeee8ZxTnOfPOnNlUFZKkPr1h6AFIkoZjCEhSxwwB\nSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6tnToAUznhhtuqNWrVw89DEm6qjz11FM/rKqx\n6da74kNg9erVjI+PDz0MSbqqJHlhJus5HSRJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscM\nAUnq2IxCIMnzSY4keTrJeKtdn+RQkmfb83Uj6+9OciLJ8SS3jdRvads5keT+JJn7jyRJmqnZ/GL4\n96vqhyPLu4DDVXVfkl1t+WNJ1gHbgJuAtwNfT/LOqnoFeAC4G3gCeATYAjw6B5/jirJ61z8O9t7P\n3/fewd5b0tXncqaDtgL7WnsfcPtIfX9VvVxVzwEngI1JbgSurarHq6qAh0b6SJIGMNMQKCa/0T+V\nZEerLa+qM639IrC8tVcAJ0f6nmq1Fa09tS5JGshMp4N+p6pOJ/lV4FCS742+WFWVpOZqUC1odgC8\n4x3vmKvNSpKmmNGRQFWdbs9nga8AG4GX2hQP7flsW/00sGqk+8pWO93aU+sXe7+9VbWhqjaMjU17\nJ1RJ0us0bQgkeUuSX/5FG/gj4LvAQWB7W2078HBrHwS2JbkmyRpgLfBkmzo6l2RTuyrozpE+kqQB\nzGQ6aDnwlXY151LgH6rqa0m+BRxIchfwAnAHQFUdTXIAOAacB3a2K4MA7gEeBJYxeVXQorsySJKu\nJtOGQFV9H3jXReo/AjZfos8eYM9F6uPA+tkPU5I0H/zFsCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwB\nSeqYISBJHTMEJKljs/l7ApI6N9TfyvDvZMwfjwQkqWOGgCR1zOkgzQn/pKZ0dfJIQJI6ZghIUscM\nAUnqmCEgSR3zxPAiM+QJWklXn0UdAv4PUdLlWuw/kFvUISAtVn7B0VzxnIAkdcwjAUlXPI985o8h\noKveYp+zleaT00GS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOjbj\nEEiyJMl3kny1LV+f5FCSZ9vzdSPr7k5yIsnxJLeN1G9JcqS9dn+SzO3HkSTNxmzuHXQv8AxwbVve\nBRyuqvuS7GrLH0uyDtgG3AS8Hfh6kndW1SvAA8DdwBPAI8AW4NE5+STSAvOmZloMZnQkkGQl8F7g\n0yPlrcC+1t4H3D5S319VL1fVc8AJYGOSG4Frq+rxqirgoZE+kqQBzHQ66JPAR4Gfj9SWV9WZ1n4R\nWN7aK4CTI+udarUVrT21LkkayLQhkOR9wNmqeupS67Rv9jVXg0qyI8l4kvGJiYm52qwkaYqZHAnc\nCrw/yfPAfuA9ST4HvNSmeGjPZ9v6p4FVI/1Xttrp1p5af42q2ltVG6pqw9jY2Cw+jiRpNqYNgara\nXVUrq2o1kyd8v1FVHwQOAtvbatuBh1v7ILAtyTVJ1gBrgSfb1NG5JJvaVUF3jvSRJA3gcv6y2H3A\ngSR3AS8AdwBU1dEkB4BjwHlgZ7syCOAe4EFgGZNXBXllkCQNaFYhUFXfBL7Z2j8CNl9ivT3AnovU\nx4H1sx2kJGl++IthSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0z\nBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNA\nkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUsemDYEkb0ryZJJ/T3I0\nyV+1+vVJDiV5tj1fN9Jnd5ITSY4nuW2kfkuSI+21+5Nkfj6WJGkmZnIk8DLwnqp6F/BuYEuSTcAu\n4HBVrQUOt2WSrAO2ATcBW4BPJVnStvUAcDewtj22zOFnkSTN0rQhUJN+2hbf2B4FbAX2tfo+4PbW\n3grsr6qXq+o54ASwMcmNwLVV9XhVFfDQSB9J0gBmdE4gyZIkTwNngUNV9QSwvKrOtFVeBJa39grg\n5Ej3U622orWn1i/2fjuSjCcZn5iYmPGHkSTNzoxCoKpeqap3AyuZ/Fa/fsrrxeTRwZyoqr1VtaGq\nNoyNjc3VZiVJU8zq6qCq+jHwGJNz+S+1KR7a89m22mlg1Ui3la12urWn1iVJA5nJ1UFjSd7W2suA\nPwS+BxwEtrfVtgMPt/ZBYFuSa5KsYfIE8JNt6uhckk3tqqA7R/pIkgawdAbr3Ajsa1f4vAE4UFVf\nTfJvwIEkdwEvAHcAVNXRJAeAY8B5YGdVvdK2dQ/wILAMeLQ9JEkDmTYEquo/gJsvUv8RsPkSffYA\ney5SHwfWv7aHJGkI/mJYkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghI\nUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1\nzBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1LFpQyDJqiSPJTmW\n5GiSe1v9+iSHkjzbnq8b6bM7yYkkx5PcNlK/JcmR9tr9STI/H0uSNBMzORI4D3ykqtYBm4CdSdYB\nu4DDVbUWONyWaa9tA24CtgCfSrKkbesB4G5gbXtsmcPPIkmapWlDoKrOVNW3W/snwDPACmArsK+t\ntg+4vbW3Avur6uWqeg44AWxMciNwbVU9XlUFPDTSR5I0gFmdE0iyGrgZeAJYXlVn2ksvAstbewVw\ncqTbqVZb0dpT6xd7nx1JxpOMT0xMzGaIkqRZmHEIJHkr8CXgw1V1bvS19s2+5mpQVbW3qjZU1Yax\nsbG52qwkaYoZhUCSNzIZAJ+vqi+38kttiof2fLbVTwOrRrqvbLXTrT21LkkayEyuDgrwGeCZqvrE\nyEsHge2tvR14eKS+Lck1SdYweQL4yTZ1dC7JprbNO0f6SJIGsHQG69wKfAg4kuTpVvs4cB9wIMld\nwAvAHQBVdTTJAeAYk1cW7ayqV1q/e4AHgWXAo+0hSRrItCFQVf8KXOp6/s2X6LMH2HOR+jiwfjYD\nlCTNH38xLEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQ\nkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ\n6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSx6YNgSSfTXI2yXdHatcnOZTk2fZ83chr\nu5OcSHI8yW0j9VuSHGmv3Z8kc/9xJEmzMZMjgQeBLVNqu4DDVbUWONyWSbIO2Abc1Pp8KsmS1ucB\n4G5gbXtM3aYkaYFNGwJV9S/Af00pbwX2tfY+4PaR+v6qermqngNOABuT3AhcW1WPV1UBD430kSQN\n5PWeE1heVWda+0VgeWuvAE6OrHeq1Va09tS6JGlAl31iuH2zrzkYy/9LsiPJeJLxiYmJudy0JGnE\n6w2Bl9oUD+35bKufBlaNrLey1U639tT6RVXV3qraUFUbxsbGXucQJUnTeb0hcBDY3trbgYdH6tuS\nXJNkDZMngJ9sU0fnkmxqVwXdOdJHkjSQpdOtkOQLwO8BNyQ5BfwlcB9wIMldwAvAHQBVdTTJAeAY\ncB7YWVWvtE3dw+SVRsuAR9tDkjSgaUOgqj5wiZc2X2L9PcCei9THgfWzGp0kaV75i2FJ6pghIEkd\nMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFD\nQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQk\nqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxxY8BJJsSXI8yYkkuxb6/SVJFyxoCCRZAvwd8MfAOuAD\nSdYt5BgkSRcs9JHARuBEVX2/qn4G7Ae2LvAYJEnNQofACuDkyPKpVpMkDWDp0AO4mCQ7gB1t8adJ\njr/OTd0A/HBuRrUouD8ucF+8mvvjgitiX+SvL3sTvzaTlRY6BE4Dq0aWV7baq1TVXmDv5b5ZkvGq\n2nC521ks3B8XuC9ezf1xQW/7YqGng74FrE2yJskvAduAgws8BklSs6BHAlV1PsmfAv8ELAE+W1VH\nF3IMkqQLFvycQFU9AjyyQG932VNKi4z74wL3xau5Py7oal+kqoYegyRpIN42QpI6tihDwFtTXJBk\nVZLHkhxLcjTJvUOPaWhJliT5TpKvDj2WoSV5W5IvJvlekmeS/NbQYxpSkr9o/06+m+QLSd409Jjm\n26ILAW9N8RrngY9U1TpgE7Cz8/0BcC/wzNCDuEL8LfC1qvoN4F10vF+SrAD+HNhQVeuZvHhl27Cj\nmn+LLgTw1hSvUlVnqurbrf0TJv+Rd/sr7SQrgfcCnx56LENL8ivA7wKfAaiqn1XVj4cd1eCWAsuS\nLAXeDPznwOOZd4sxBLw1xSUkWQ3cDDwx7EgG9Ungo8DPhx7IFWANMAH8fZse+3SStww9qKFU1Wng\nb4AfAGeA/66qfx52VPNvMYaALiLJW4EvAR+uqnNDj2cISd4HnK2qp4YeyxViKfCbwANVdTPwP0C3\n59CSXMfkrMEa4O3AW5J8cNhRzb/FGAIzujVFT5K8kckA+HxVfXno8QzoVuD9SZ5ncprwPUk+N+yQ\nBnUKOFVVvzgy/CKTodCrPwCeq6qJqvpf4MvAbw88pnm3GEPAW1OMSBIm53yfqapPDD2eIVXV7qpa\nWVWrmfzv4htVtei/6V1KVb0InEzy6620GTg24JCG9gNgU5I3t383m+ngRPkVeRfRy+GtKV7jVuBD\nwJEkT7fax9svt6U/Az7fvjB9H/iTgcczmKp6IskXgW8zeVXdd+jg18P+YliSOrYYp4MkSTNkCEhS\nxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1LH/AzDZ/a5bynw+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114ae8b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.hist(train_labels)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Utilities:\n",
    "\n",
    "These are utilities for processing the data and handling the process of training the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-hot encode labels:\n",
    "\n",
    "We can use a very concise but somewhat sophisticated script to one-hot encode the labels. The inner statement in parenthesis uses Numpy broadcasting to convert the arange array, with shape `(10, )`, into an array of shape `(10, len(labels))`, and the labels array, with shape `(len(labels),)`, into an array with shape `(len(labels), 1)`. The `[:,None]` syntax creates a new axis with length 1. The two arrays are then compared elementwise, returning a boolean array which is `True` in the desired position. We then cast the boolean values as integers, to get our one-hot encoded labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def encode_labels(labels):\n",
    "    return (np.arange(10) == labels[:,None]).astype(np.uint8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  3 Encoded label:  [0 0 0 1 0 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "train_labels_encoded = encode_labels(train_labels)\n",
    "valid_labels_encoded = encode_labels(valid_labels)\n",
    "\n",
    "print(\"Label: \", train_labels[2], \"Encoded label: \", train_labels_encoded[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate batches from preprocessed data:\n",
    "\n",
    "This is pretty self explanatory. Given a dataset and the corresponding labels, it generates a random set of examples of the desired size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def next_batch(dataset, labels, batch_size):\n",
    "    indices = random.sample(range(len(dataset)), batch_size)\n",
    "    batch_data = dataset[indices]\n",
    "    batch_labels = labels[indices].reshape(batch_size,10)\n",
    "    return batch_data, batch_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate validation accuracy:\n",
    "\n",
    "The importance of a validation set in training neural networks cannot be overstated. Think about a trendline: high correlation with a linear trendline is statistically significant, but if you make the fit more and more elaborate, it's less and less meaningful. **It's easy to overfit data**, and a validation set which evaluates the accuracy impartially is essential. You can cut off training when validation accuracy stops improving, and use that information to make better models.\n",
    "\n",
    "This function evaluates the percent accuracy for a batch of a given size from the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate(size):\n",
    "    images_valid, labels_valid = next_batch(valid_dataset, valid_labels_encoded, size)\n",
    "    percent = model.evaluate(images_valid.reshape(size, vsize, hsize, num_channels), labels_valid)[1]*100\n",
    "    print(\"Your model is {} percent accurate!\".format(percent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Setup:\n",
    "\n",
    "We begin the training process by importing the relevant layers and activations from Keras. In Keras, the Sequential API is designed to be easy to use with linear neural networks, which most simple convolution networks are. However, more complicated networks, like the Google Inception architecture, are not at all sequential, and the functional API is more appropriate there. There are four models below, and you are welcome to run each of them, or only one at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, BatchNormalization, Conv2DTranspose, Reshape, Flatten, Conv2D, Dropout, GlobalAveragePooling2D\n",
    "from keras.optimizers import RMSprop, Adam, SGD, Nadam\n",
    "from keras.layers.advanced_activations import LeakyReLU\n",
    "from keras.initializers import TruncatedNormal\n",
    "\n",
    "import keras\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.gridspec as gridspec\n",
    "\n",
    "image_size = [28, 28, 1]\n",
    "\n",
    "batch_size = 32\n",
    "\n",
    "vsize = image_size[0]\n",
    "hsize = image_size[1]\n",
    "num_channels = image_size[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Fully Connected Model\n",
    "\n",
    "The simplest MNIST models use a series of fully connected layers to make predictions. These fully connected layers are just large matrix multiplications applied to the entire image, followed by non-linear activations. They are easy to use and their derivatives are simple, making them a good place to start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Flatten(input_shape=(vsize, hsize, num_channels)))\n",
    "model.add(Dense(512))\n",
    "model.add(Activation('relu')) # we can also specify the activation in the previous layer, i.e. Dense(512, activation='relu')\n",
    "model.add(Dense(256))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dense(64))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dense(10))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to specify the optimizer which Keras will use to train the model. We will use the classic Stochastic Gradient Descent optimizer, including a momentum parameter and using Nesterov predictive momentum. More information on these optimizers can be found in [this](http://ruder.io/optimizing-gradient-descent/index.html#adam) excellent guide. We compile the model, and use the summary method show the number of parameters in each layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_2 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 256)               131328    \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 64)                16448     \n",
      "_________________________________________________________________\n",
      "activation_9 (Activation)    (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 10)                650       \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 10)                0         \n",
      "=================================================================\n",
      "Total params: 550,346\n",
      "Trainable params: 550,346\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "optimizer = RMSprop(lr=0.0001, clipvalue=1.0, decay=3e-8)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "model.summary() # this will list the layers and number of parameters for each"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Model\n",
    "\n",
    "Here we actually train the model. For additional control, we perform this step manually, batch by batch. For better results, increase the number of iterations (`num_steps`), and possibly changee the optimizer and learning rate in the previous cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter:  0\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 12.5 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  2.31291\n",
      "model loss is  0.493307\n",
      "model loss is  0.299057\n",
      "model loss is  0.283493\n",
      "model loss is  0.202367\n",
      "Iter:  1000\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 96.875 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  0.280828\n",
      "model loss is  0.183525\n",
      "model loss is  0.289912\n",
      "model loss is  0.0912936\n",
      "model loss is  0.113893\n",
      "Iter:  2000\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 93.75 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  0.0933689\n"
     ]
    }
   ],
   "source": [
    "num_steps = 2001\n",
    "\n",
    "for i in range(num_steps):\n",
    "    if i % 1000 == 0:\n",
    "        print(\"Iter: \", i)\n",
    "        evaluate(batch_size)\n",
    "        print('\\n')\n",
    "    \n",
    "    images_train, labels_train = next_batch(train_dataset, train_labels_encoded, batch_size)\n",
    "\n",
    "    loss = model.train_on_batch(images_train.reshape(batch_size, vsize, hsize, num_channels), labels_train) \n",
    "    \n",
    "    if i % 200 == 0:\n",
    "        print(\"model loss is \", loss[0])   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That worked pretty well. But let's try something different. **Before continuing, please continue down to the \"Evaluate Model\" header if you want to make predictions for this model on a test dataset, like the Kaggle MNIST competition**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Small convolution model\n",
    "\n",
    "This is a small convolution model, which can be trained in a matter of minutes. This model uses a series of convolution layers, followed by ReLU activations, and Dropout to prevent overfitting. In the final step, GlobalAveragePooling is used, which takes the average of each layer, turning a spatially-large vector into a `1x1xn` prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convolution networks are a more natural way of dealing with image classification than fully connected networks, because they preserve spatial information about the input image. GlobalAveragePooling also has the advantage of preserving information separated out by the feature map, and encouraging the network to produce a feature map that describes the image. Convolution layers are also much smaller than fully connected layers, since they're only connected to a small region of the image at a time, and share weights across the whole image. Look how much more complex this network looks, even though it uses about a fifth as many parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1 (Conv2D)            (None, 28, 28, 1)         10        \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_1 (LeakyReLU)    (None, 28, 28, 1)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 28, 28, 28)        56        \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_2 (LeakyReLU)    (None, 28, 28, 28)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 28, 28, 28)        7084      \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_3 (LeakyReLU)    (None, 28, 28, 28)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 14, 14, 28)        7084      \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_4 (LeakyReLU)    (None, 14, 14, 28)        0         \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 14, 14, 28)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 14, 14, 56)        14168     \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_5 (LeakyReLU)    (None, 14, 14, 56)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 14, 14, 56)        28280     \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_6 (LeakyReLU)    (None, 14, 14, 56)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 7, 7, 56)          28280     \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_7 (LeakyReLU)    (None, 7, 7, 56)          0         \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 7, 7, 56)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_8 (Conv2D)            (None, 7, 7, 56)          28280     \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_8 (LeakyReLU)    (None, 7, 7, 56)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_9 (Conv2D)            (None, 7, 7, 56)          3192      \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_9 (LeakyReLU)    (None, 7, 7, 56)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_10 (Conv2D)           (None, 4, 4, 10)          570       \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_10 (LeakyReLU)   (None, 4, 4, 10)          0         \n",
      "_________________________________________________________________\n",
      "global_average_pooling2d_1 ( (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "activation_25 (Activation)   (None, 10)                0         \n",
      "=================================================================\n",
      "Total params: 117,004\n",
      "Trainable params: 117,004\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(1, 3, strides=1, padding = 'same', input_shape=(vsize, hsize, num_channels))) # for visualization\n",
    "model.add(LeakyReLU(0.2))\n",
    "\n",
    "model.add(Conv2D(28, 1, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(28, 3, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(28, 3, strides=2, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "model.add(Conv2D(56, 3, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(56, 3, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(56, 3, strides=2, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "model.add(Conv2D(56, 3, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(56, 1, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(10, 1, strides=2, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "\n",
    "model.add(GlobalAveragePooling2D())\n",
    "\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "optimizer = Adam(lr=0.0002) # here we use the Adam optimizer, a more elaborate cousin of SGD\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Model\n",
    "\n",
    "Here we actually train the model. For additional control, we perform this step manually, batch by batch. Note that while this model uses fewer parameters than the fully connected model, it takes much longer to train, since backpropogation is much more complicated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter:  0\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 3.125 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  2.30261\n",
      "model loss is  1.4226\n",
      "model loss is  1.0183\n",
      "model loss is  0.709499\n",
      "model loss is  0.661462\n",
      "Iter:  1000\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 75.0 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  0.436884\n",
      "model loss is  0.612463\n",
      "model loss is  0.513328\n",
      "model loss is  0.433636\n",
      "model loss is  0.280018\n",
      "Iter:  2000\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 96.875 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  0.326926\n",
      "model loss is  0.525506\n",
      "model loss is  0.577065\n",
      "model loss is  0.518575\n",
      "model loss is  0.530569\n",
      "Iter:  3000\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 90.625 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  0.163107\n",
      "model loss is  0.142122\n",
      "model loss is  0.151462\n",
      "model loss is  0.173525\n",
      "model loss is  0.0709266\n",
      "Iter:  4000\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 96.875 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  0.359121\n",
      "model loss is  0.339858\n",
      "model loss is  0.0282522\n",
      "model loss is  0.185371\n",
      "model loss is  0.321326\n"
     ]
    }
   ],
   "source": [
    "num_steps = 5000\n",
    "\n",
    "for i in range(num_steps):\n",
    "    if i % 500 == 0:\n",
    "        print(\"Iter: \", i)\n",
    "        evaluate(batch_size)\n",
    "        print('\\n')\n",
    "    \n",
    "    images_train, labels_train = next_batch(train_dataset, train_labels_encoded, batch_size)\n",
    "\n",
    "    loss = model.train_on_batch(images_train.reshape(batch_size, vsize, hsize, num_channels), labels_train) \n",
    "    \n",
    "    if i % 200 == 0:\n",
    "        print(\"model loss is \", loss[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Large convolution model\n",
    "\n",
    "This is a much larger convolution model, and it may take a while to train on a CPU. The structure is mostly the same, except the hidden layers are deeper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_11 (Conv2D)           (None, 28, 28, 1)         10        \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_11 (LeakyReLU)   (None, 28, 28, 1)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_12 (Conv2D)           (None, 28, 28, 84)        168       \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_12 (LeakyReLU)   (None, 28, 28, 84)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_13 (Conv2D)           (None, 28, 28, 84)        63588     \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_13 (LeakyReLU)   (None, 28, 28, 84)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_14 (Conv2D)           (None, 14, 14, 84)        63588     \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_14 (LeakyReLU)   (None, 14, 14, 84)        0         \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 14, 14, 84)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_15 (Conv2D)           (None, 14, 14, 168)       127176    \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_15 (LeakyReLU)   (None, 14, 14, 168)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_16 (Conv2D)           (None, 14, 14, 168)       254184    \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_16 (LeakyReLU)   (None, 14, 14, 168)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_17 (Conv2D)           (None, 7, 7, 168)         254184    \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_17 (LeakyReLU)   (None, 7, 7, 168)         0         \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 7, 7, 168)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_18 (Conv2D)           (None, 7, 7, 168)         254184    \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_18 (LeakyReLU)   (None, 7, 7, 168)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_19 (Conv2D)           (None, 7, 7, 168)         28392     \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_19 (LeakyReLU)   (None, 7, 7, 168)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_20 (Conv2D)           (None, 4, 4, 10)          1690      \n",
      "_________________________________________________________________\n",
      "leaky_re_lu_20 (LeakyReLU)   (None, 4, 4, 10)          0         \n",
      "_________________________________________________________________\n",
      "global_average_pooling2d_2 ( (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 10)                0         \n",
      "=================================================================\n",
      "Total params: 1,047,164\n",
      "Trainable params: 1,047,164\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(1, 3, strides=1, padding = 'same', input_shape=(vsize, hsize, num_channels))) # for visualization\n",
    "model.add(LeakyReLU(0.2))\n",
    "\n",
    "model.add(Conv2D(84, 1, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(84, 3, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(84, 3, strides=2, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "\n",
    "model.add(Dropout(0.3))\n",
    "\n",
    "model.add(Conv2D(168, 3, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(168, 3, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(168, 3, strides=2, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "\n",
    "model.add(Dropout(0.4))\n",
    "\n",
    "model.add(Conv2D(168, 3, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(168, 1, strides=1, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "model.add(Conv2D(10, 1, strides=2, padding = 'same'))\n",
    "model.add(LeakyReLU(0.2))\n",
    "\n",
    "model.add(GlobalAveragePooling2D())\n",
    "\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "optimizer = Adam(lr=0.0002) # here we use the Adam optimizer, a more elaborate cousin of SGD\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Model\n",
    "\n",
    "Here we actually train the model. For additional control, we perform this step manually, batch by batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter:  0\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 18.75 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  2.30287\n",
      "model loss is  0.790993\n",
      "model loss is  0.349134\n",
      "model loss is  0.454545\n",
      "model loss is  0.196152\n",
      "Iter:  1000\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 93.75 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  0.155478\n",
      "model loss is  0.371035\n",
      "model loss is  0.635919\n",
      "model loss is  0.025465\n",
      "model loss is  0.191232\n",
      "Iter:  2000\n",
      "32/32 [==============================] - 0s\n",
      "Your model is 100.0 percent accurate!\n",
      "\n",
      "\n",
      "model loss is  0.0811365\n",
      "model loss is  0.117066\n",
      "model loss is  0.246956\n",
      "model loss is  0.0768135\n",
      "model loss is  0.238888\n"
     ]
    }
   ],
   "source": [
    "num_steps = 3000\n",
    "\n",
    "for i in range(num_steps):\n",
    "    if i % 1000 == 0:\n",
    "        print(\"Iter: \", i)\n",
    "        evaluate(batch_size)\n",
    "        print('\\n')\n",
    "    \n",
    "    images_train, labels_train = next_batch(train_dataset, train_labels_encoded, batch_size)\n",
    "\n",
    "    loss = model.train_on_batch(images_train.reshape(batch_size, vsize, hsize, num_channels), labels_train) \n",
    "    \n",
    "    if i % 200 == 0:\n",
    "        print(\"model loss is \", loss[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Model\n",
    "\n",
    "Now that we've seen all three models in action, let's make some actual predictions. Here, you can load test data and save predictions for the classic [Kaggle competition](https://www.kaggle.com/c/digit-recognizer/data). If you want to participate in the Kaggle challenge with any of these datasets, please go to the link and save the test.csv file to the cloned directory. You can also use other test datasets here instead, if you want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll evaluate the accuracy of our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5000/5000 [==============================] - 28s    \n",
      "Your model is 96.92 percent accurate!\n"
     ]
    }
   ],
   "source": [
    "evaluate(5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load test dataset from CSV\n",
    "\n",
    "If you aren't using the Kaggle dataset, just skip this step and load your own in a new cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "test = np.genfromtxt('test.csv', delimiter=\",\", dtype=int)[1:,:].reshape(len(test), 28, 28, 1)\n",
    "\n",
    "test_dataset = test - np.mean(test, axis=0)\n",
    "\n",
    "print(\"Mean: %f, Std: %f\" % (np.mean(test_dataset), np.std(test_dataset)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate Predictions for Test Dataset\n",
    "\n",
    "This may take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predictions = model.predict(test_dataet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save predictions to file:\n",
    "\n",
    "This function will save your images to a csv file in the correct format for submission to the Kaggle competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def save_to_file(data, name):\n",
    "    index = np.linspace(1,len(data),len(data))\n",
    "    data = np.argmax(data, axis=1)\n",
    "    with open('{}.csv'.format(name), 'w') as csvfile:\n",
    "        writer = csv.writer(csvfile, delimiter=',')\n",
    "        writer.writerow(['ImageId', 'Label'])\n",
    "        for i in range(len(data)):\n",
    "            writer.writerow([index[i].astype(np.uint32), data[i]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "save_to_file(predictions, \"MNIST_predictions\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save model to file\n",
    "\n",
    "If you'd like to save the current model for future use, run this script. To save multiple models, change the file name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.save('saved_models/MNIST_model_2.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load pre-trained version of the model\n",
    "\n",
    "Keras allows us to load and save entire models, so running this command will load a pretrained model which you can continue training or use for testing purposes. I have included a pre-trained model in the github repository, but you can use your own models for the next step if you so desire. However, this is only compatible with the third model, so you may wish to load the pre-trained model here, or modify the following code to work with any model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.models import load_model \n",
    "\n",
    "model = load_model('saved_models/MNIST_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hidden Layer Visualization\n",
    "\n",
    "Before you proceed, make sure you have run one of the second two models most recently, or load one of those models above. We will be viewing hidden layers in the MNIST networks, which will tell us something about how the network is parsing the information.\n",
    "\n",
    "Our first step is to create a new network, with just a single layer identical to the first layer in the convolution networks above. We initialize the layer with weights from the above network, and view that output next to the corresponding input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visualization = Sequential()\n",
    "visualization.add(Conv2D(1, 3, strides=1, padding = 'same', input_shape=(vsize, hsize, num_channels), weights=model.layers[0].get_weights())) # for visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEUVJREFUeJzt3X9sVWWaB/DvYykFGcKloi0CUUyICVEppEGTwZWVHQIE\ngfnHwB8jm5ipJjPESfhj0f1jNTFRNzszMcZMLCsZXEZnVhmVBDIbIGtwjCEUg4Lojkg6AgEKakXk\nZ8uzf/Qw6WjP817ve37c9vl+EtLb+9y39+2BL+fe+5xzXlFVEJE/15Q9ASIqB8NP5BTDT+QUw0/k\nFMNP5BTDT+QUw0/kFMNP5BTDT+TUqCKfrFKpaGtra5FPSeTKiRMn0NvbK9U8Nir8IrIIwLMAGgD8\np6o+bT2+tbUVnZ2dMU9JRIaOjo6qH1vzy34RaQDwPIDFAGYCWCUiM2v9eURUrJj3/HMBHFLVw6p6\nCcDvASzPZlpElLeY8E8BcGTQ90eT+/6OiHSISJeIdPX29kY8HRFlKfdP+1W1U1XbVbW9Uqnk/XRE\nVKWY8B8DMG3Q91OT+4hoGIgJ/x4AM0RkuoiMBrASwJZspkVEeau51aeqfSLycwD/g4FW3wZV/TCz\nmRFRrqL6/Kq6DcC2jOZCRAXi4b1ETjH8RE4x/EROMfxETjH8RE4x/EROFXo+P9VGpKrTs0v52aHx\nR44cSa19/PHH5ti7777brI8ePdqsx/CwkhX3/EROMfxETjH8RE4x/EROMfxETjH8RE6x1VeAUDss\nz3po7DXXxP3//9prr5n1F154IbV2/vx5c+yjjz5q1hcvXmzWr1y5klqLbeWNhFYg9/xETjH8RE4x\n/EROMfxETjH8RE4x/EROMfxETrHPX6WYXnqo3tDQYNZDvfhRo9L/Gq0aEO61r1+/3qxv377drE+c\nOLGmGgDMmTPHrI8dO9asX758ObXW399vjg3VQ33+2HoRuOcncorhJ3KK4SdyiuEncorhJ3KK4Sdy\niuEnciqqzy8i3QC+BtAPoE9V27OYVBliznsP9eFDffzQJagbGxvNelNTU2pt2zZ7EeVQn/7cuXNm\n/a677jLrt99+e2pt6dKl5tgbbrjBrF+4cMGsX7x4seaxly5dMuuh4wCsawmEFHUMQBYH+fyjqp7O\n4OcQUYH4sp/IqdjwK4AdIrJXRDqymBARFSP2Zf88VT0mIjcA2C4iH6vqrsEPSP5T6ACAlpaWyKcj\noqxE7flV9VjytQfA6wDmDvGYTlVtV9X2SqUS83RElKGawy8i40Rk/NXbABYCOJDVxIgoXzEv+1sA\nvJ60yEYBeFlV/5TJrIgodzWHX1UPA5iV4VxyFXvOvdXLD50zH+rTjxkzxqyH5rZp06bU2u7du82x\noXPqFyxYYNZXrlxp1q+77rrUWqhXbp2PD4S3q3V8RezS5KHjAEJCv3sR2OojcorhJ3KK4SdyiuEn\ncorhJ3KK4Sdyys2lu2OXqrbaRrGtvL6+PrP+1ltvmfWurq7U2qRJk8yx69atM+szZ84066Htap2e\nGtrmMT8bsNtpoW0eajOGxodO6bV+t6JO6eWen8gphp/IKYafyCmGn8gphp/IKYafyCmGn8ipEdPn\nz/OU3VA9dEpvqB57eW3rtNxnnnnGHBu6PHY9nHqaJnRJdKseuyx67CnB9YB7fiKnGH4ipxh+IqcY\nfiKnGH4ipxh+IqcYfiKnRkyfP1bMcQChnvHmzZvN+tatW836TTfdZNafeOKJ1Fpzc7M5NnQJ6pjz\n0kP1UC89pKjz3mt57th6EbjnJ3KK4SdyiuEncorhJ3KK4SdyiuEncorhJ3Iq2OcXkQ0AlgLoUdXb\nkvuaAfwBwM0AugHcr6pf5jfN8ln96lCv+9VXXzXrEyZMMOsPP/ywWW9tbU2tha4/H6qH+tEx59SH\nxF633zpGIXT8Qqg+HPr4IdXs+X8LYNG37lsHYKeqzgCwM/meiIaRYPhVdReAL75193IAG5PbGwGs\nyHheRJSzWt/zt6jq8eT2CQAtGc2HiAoS/YGfDry5SX2DIyIdItIlIl29vb2xT0dEGak1/CdFZDIA\nJF970h6oqp2q2q6q7ZVKpcanI6Ks1Rr+LQBWJ7dXA3gzm+kQUVGC4ReRVwC8C+BWETkqIg8CeBrA\nj0TkEwD/lHxPRMNIsM+vqqtSSgsynkupQn1Z67z35557zhwbWst9xQq7WTJr1iyzbl1bP3ad+VCv\nPdTHjzk+IiTUi7e2S2g9gtDPHgl4hB+RUww/kVMMP5FTDD+RUww/kVMMP5FTbi7dHXuK5bvvvpta\n27Fjhzl22bJlZn3lypU1zekq63eLPSU3dvlxqx5q9cW242JafXmfklsPp/xyz0/kFMNP5BTDT+QU\nw0/kFMNP5BTDT+QUw0/klJs+f0io57xp06bUWqhXPnfuXLPe1NRk1mOElsE+c+aMWb/++uvN+ujR\no2t+/phLb2cxPk8xS5cXdQwA9/xETjH8RE4x/EROMfxETjH8RE4x/EROMfxETo2YPn+oNxp7mehP\nP/00tdbc3Bz13Hku99zZ2WnW9+/fb9ZDlw0PLS9unTd/8eJFc+wdd9xh1kPb5fTp06m1W2+91Rz7\n1VdfmfVrr73WrA8H3PMTOcXwEznF8BM5xfATOcXwEznF8BM5xfATORXs84vIBgBLAfSo6m3JfY8D\n+CmAU8nDHlPVbXlNsgih896tet59/JhjGM6ePWuOnTJlilk/deqUWe/p6THr1hLhvb295ti3337b\nrIfmZvX5rSXXAeCbb74x64sWLTLrDzzwgFkPrXdQhGr2/L8FMNRv+mtVbUv+DOvgE3kUDL+q7gLw\nRQFzIaICxbznXyMiH4jIBhGZmNmMiKgQtYb/NwBuAdAG4DiAX6Y9UEQ6RKRLRLpC7/GIqDg1hV9V\nT6pqv6peAbAeQOoVKlW1U1XbVbW9UqnUOk8iylhN4ReRyYO+/TGAA9lMh4iKUk2r7xUA8wFMEpGj\nAP4NwHwRaQOgALoBPJTjHIkoB8Hwq+qqIe5+MYe5RIm5TnpsPdSHf+edd8z6kiVLzHpoXQDrGIR1\n69aZY0P95lA/vK+vr+b6l19+aY49d+6cWd+zZ49Zf//992t+7m3b7O71yy+/bNZ37dpl1h955JHU\nWltbmzk2KzzCj8gphp/IKYafyCmGn8gphp/IKYafyKnyzyssSGyr78knn0ytbdy40Ry7e/dus/7G\nG2+Y9fvuu8+sNzY2mvUYoZ8dWqLbaoOOGzfOHBtqI06dOtWsW6fdhk7ZnT17tll/6qmnzPrhw4fN\n+tq1a1NrO3fuNMdmhXt+IqcYfiKnGH4ipxh+IqcYfiKnGH4ipxh+IqfY50+ELt19zz33pNa6u7vN\nsVu3bjXrzz//vFk/ePCgWZ8/f35qLdQLnzFjhlkPbZcyhU6ltnr5e/fuNceGjs0IneocMmfOnKjx\nWajfv1kiyhXDT+QUw0/kFMNP5BTDT+QUw0/kFMNP5BT7/BmMf+ghe9mC9vZ2sx7qKYd60vv27Uut\njRkzxhw7ffp0sx4a39zcbNatXnxoie1JkyaZ9QMH7LViPvvss9TaoUOHzLHnz58366HjH9asWWPW\nFy5caNaLwD0/kVMMP5FTDD+RUww/kVMMP5FTDD+RUww/kVPBPr+ITAPwEoAWAAqgU1WfFZFmAH8A\ncDOAbgD3q6q97nGOYvv4MUI933nz5pn1e++916x//vnnZr2npye1Floq+ujRo2a9t7c3qm5dm3/C\nhAnm2K6uLrM+fvx4s97S0pJaW7ZsmTl24sSJZn3WrFlmvb+/36xbxz+ErlOQlWr2/H0A1qrqTAB3\nAfiZiMwEsA7ATlWdAWBn8j0RDRPB8KvqcVV9L7n9NYCPAEwBsBzA1aVqNgJYkdckiSh73+s9v4jc\nDGA2gN0AWlT1eFI6gYG3BUQ0TFQdfhH5AYDNAH6hqmcG13TgTcqQb1REpENEukSkK/T+kIiKU1X4\nRaQRA8H/nar+Mbn7pIhMTuqTAQz5qZOqdqpqu6q2VyqVLOZMRBkIhl8GPkZ/EcBHqvqrQaUtAFYn\nt1cDeDP76RFRXqo5pfeHAH4CYL+IXD139DEATwP4bxF5EMBfAdyfzxSrE2qPXLlyxayHWjPW+NDP\nDgm1Cm+88UazPm3atNTanXfeaY69cOGCWQ8tkx3ablYLtqGhwRwbakOGlge3freLFy+aY2N/79C/\nx6LaeZZg+FX1zwDS/gYXZDsdIioKj/AjcorhJ3KK4SdyiuEncorhJ3KK4Sdyys2lu0N91VDf1lqS\nedQoezM2Njaa9dBxAnn2hMeOHWvWQ3ML9cOt7WYtoQ0ATU1NZj10eW2rl593H3844J6fyCmGn8gp\nhp/IKYafyCmGn8gphp/IKYafyKkR0+ePPZ8/1PfNU+wxCFY/PLZfHdpuofPirXPqQ336UN06hgCw\nf/e8z8cfDscBcM9P5BTDT+QUw0/kFMNP5BTDT+QUw0/kFMNP5NSI6fOH5HkcQOhnh44hCF07P3R9\neqseWhMgdrtcvny55nrMWCA8tzzXWhgOffwQ7vmJnGL4iZxi+ImcYviJnGL4iZxi+ImcYviJnAr2\n+UVkGoCXALQAUACdqvqsiDwO4KcATiUPfUxVt+U10bzF9LvzvpZA6DgAq5cvkra6enVifzdrfOx6\nBXmudzAS+vgh1Rzk0wdgraq+JyLjAewVke1J7deq+h/5TY+I8hIMv6oeB3A8uf21iHwEYEreEyOi\nfH2v9/wicjOA2QB2J3etEZEPRGSDiExMGdMhIl0i0tXb2xs1WSLKTtXhF5EfANgM4BeqegbAbwDc\nAqANA68MfjnUOFXtVNV2VW2vVCoZTJmIslBV+EWkEQPB/52q/hEAVPWkqvar6hUA6wHMzW+aRJS1\nYPhl4OPiFwF8pKq/GnT/5EEP+zGAA9lPj4jyUs2n/T8E8BMA+0VkX3LfYwBWiUgbBtp/3QAeymWG\ndSKm9RNz6ikQbtfFtvNi5HkJ69h2m4d2XYxqPu3/M4Ch/nUN254+EfEIPyK3GH4ipxh+IqcYfiKn\nGH4ipxh+IqfcXLo7TyNhueY0eZ8STOXhnp/IKYafyCmGn8gphp/IKYafyCmGn8gphp/IKSmyDysi\npwD8ddBdkwCcLmwC30+9zq1e5wVwbrXKcm43qer11Tyw0PB/58lFulS1vbQJGOp1bvU6L4Bzq1VZ\nc+PLfiKnGH4ip8oOf2fJz2+p17nV67wAzq1Wpcyt1Pf8RFSesvf8RFSSUsIvIotE5P9E5JCIrCtj\nDmlEpFtE9ovIPhHpKnkuG0SkR0QODLqvWUS2i8gnydchl0kraW6Pi8ixZNvtE5ElJc1tmoj8r4gc\nFJEPReSR5P5St50xr1K2W+Ev+0WkAcBfAPwIwFEAewCsUtWDhU4khYh0A2hX1dJ7wiLyDwDOAnhJ\nVW9L7vt3AF+o6tPJf5wTVfVf6mRujwM4W/bKzcmCMpMHrywNYAWAf0aJ286Y1/0oYbuVseefC+CQ\nqh5W1UsAfg9geQnzqHuqugvAF9+6ezmAjcntjRj4x1O4lLnVBVU9rqrvJbe/BnB1ZelSt50xr1KU\nEf4pAI4M+v4o6mvJbwWwQ0T2ikhH2ZMZQkuybDoAnADQUuZkhhBcublI31pZum62XS0rXmeNH/h9\n1zxVbQOwGMDPkpe3dUkH3rPVU7umqpWbizLEytJ/U+a2q3XF66yVEf5jAKYN+n5qcl9dUNVjydce\nAK+j/lYfPnl1kdTka0/J8/mbelq5eaiVpVEH266eVrwuI/x7AMwQkekiMhrASgBbSpjHd4jIuOSD\nGIjIOAALUX+rD28BsDq5vRrAmyXO5e/Uy8rNaStLo+RtV3crXqtq4X8ALMHAJ/6fAvjXMuaQMq9b\nALyf/Pmw7LkBeAUDLwMvY+CzkQcBXAdgJ4BPAOwA0FxHc/svAPsBfICBoE0uaW7zMPCS/gMA+5I/\nS8redsa8StluPMKPyCl+4EfkFMNP5BTDT+QUw0/kFMNP5BTDT+QUw0/kFMNP5NT/A1TJb5NtXbQD\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12d045da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEdxJREFUeJzt3V1sXdWVB/D/ypdDEoPxJP4gHzgBA0GBSSQTITWMOuq0\nolGlpBJCzUOVEajpQ6eaSn0YxDwMj2g0bcXDqFI6RA2jDu1ILSJCiBFEI0gDVJiQwaQB8oGbOCRx\ngjF2CPle8+CTygWftS5333PPcdb/J0W277rn3u1j/3Ntr7P3FlUFEcUzo+wBEFE5GH6ioBh+oqAY\nfqKgGH6ioBh+oqAYfqKgGH6ioBh+oqBmNfPJ2tratKurq5lPSRTKiRMnMDo6KrXcNyn8InI/gCcA\nzATwH6r6uHX/rq4ubNu2LeUpicjw0EMP1Xzfun/sF5GZAP4dwDcB3Algk4jcWe/jEVFzpfzOvxbA\nQVU9rKoXAPwawIbGDIuIipYS/sUAjk76eCi77S+IyBYR6ReR/tHR0YSnI6JGKvyv/aq6VVX7VLWv\nra2t6KcjohqlhP8YgKWTPl6S3UZE00BK+N8A0Csiy0VkDoDvANjRmGERUdHqbvWp6iUR+QcA/4OJ\nVt82Vd3XsJFdQ0RqartOS1euXDHrly9frqtWS/26664z6zNmFPdb7bWwAlZSn19VnwfwfIPGQkRN\nxMt7iYJi+ImCYviJgmL4iYJi+ImCYviJgmrqfP7prMq9+pSxecdevHjRrI+NjZn1gwcP5tbOnTtn\nHrtixQqz7vXx582bl1tL7dN75206XAfAV36ioBh+oqAYfqKgGH6ioBh+oqAYfqKg2OrLFNnK8x47\n9bmtlldqK29kZMSs79q1y6wfPXo0t3bbbbeZx1qtOgBob2836xcuXMitea241HrK17RZbUK+8hMF\nxfATBcXwEwXF8BMFxfATBcXwEwXF8BMFFabPn9pLt45P7eN7U1NnzbK/TNby2efPnzePPXz4sFnf\nvXu3Wf/ggw/M+vj4eG5t0aJF5rHeDk/eeZszZ05u7dKlS+ax3rLhnpRefbOmC/OVnygohp8oKIaf\nKCiGnygohp8oKIafKCiGnyiopD6/iAwCGAdwGcAlVe1rxKDqHEuhx1v11D69V/f6umfPns2tvfXW\nW+axg4ODZt1z6623mvXFixfn1u655x7z2AULFpj1uXPnmvWU6x+87wfvOgFPFZb2bsRFPn+rqqcb\n8DhE1ET8sZ8oqNTwK4CXRORNEdnSiAERUXOk/ti/TlWPiUgHgBdF5F1VfWXyHbL/FLYAQGdnZ+LT\nEVGjJL3yq+qx7O0wgGcArJ3iPltVtU9V+7yJGkTUPHWHX0Tmi0jr1fcBfAPAO40aGBEVK+XH/k4A\nz2QtkVkA/ktVX2jIqIiocHWHX1UPA/jrBo7FVeRW1Clz7mfPnm0e69W957bmxAPAq6++mlsbGhoy\nj21paTHr9957r1lfuXKlWbd+1WttbTWPtebjA/a6/IC/Bbgldd1+bz2AKvT52eojCorhJwqK4ScK\niuEnCorhJwqK4ScKikt311i3pt2mtvLOnDlj1r1tsA8dOpRb6+rqMo9dv369WV+yZIlZv/766826\nNe3Wa3d502Znzpxp1q3H91px3nNb04VreXzre4JbdBNRoRh+oqAYfqKgGH6ioBh+oqAYfqKgGH6i\noCrV5y9yyq7HW37b6il7fVmvj//yyy+bdW957WXLluXWHnjgAfPYjo4Osz5//nyz7i07bvF65Slf\nE8Aem3esV0+9boRTeomoNAw/UVAMP1FQDD9RUAw/UVAMP1FQDD9RUJXq85fJ6ylb9c8++8w81tsm\n+/333zfr1jbXALBhw4bcmtfH97a59vrRFy9eNOtWvzu1V55ST70uxFOFPr6Hr/xEQTH8REEx/ERB\nMfxEQTH8REEx/ERBMfxEQbl9fhHZBuBbAIZVdVV2WzuA3wDoATAI4EFV/bi4YaZL7eta/ewDBw6Y\nx77wwgtmvaenx6x7a+tb1wF48+1T169PuT7COza1V24dn7oFd5Fja5ZaXvl/CeD+z932CICdqtoL\nYGf2MRFNI274VfUVACOfu3kDgO3Z+9sBbGzwuIioYPX+zt+pqsez908A6GzQeIioSZL/4KcTv7zk\n/gIjIltEpF9E+kdHR1OfjogapN7wnxSRbgDI3g7n3VFVt6pqn6r2tbW11fl0RNRo9YZ/B4DN2fub\nATzbmOEQUbO44ReRpwG8BuB2ERkSkYcBPA7g6yJyAMDfZR8T0TTi9vlVdVNO6WsNHkupvH72+Ph4\nbu21114zj/X2sF+3bp1Z7+3tNespe72n7CNfSz2FN3Zv3X/rc/OOLbrPXwW8wo8oKIafKCiGnygo\nhp8oKIafKCiGnyioMEt3e62Zs2fPmvXdu3fn1rwpvffdd59ZX7NmjVlP2U7aa2GmtupSt7q2pLb6\nrHpqq+9awFd+oqAYfqKgGH6ioBh+oqAYfqKgGH6ioBh+oqAq1ef3eqtFTh8dGhoy6wMDA7m11tZW\n89i7777brHtTfmfPnm3WrWXFP/30U/PYTz75xKx7W3h7dWt57tTtv71rGM6fP59bO3funHmsx5sK\nPR3wlZ8oKIafKCiGnygohp8oKIafKCiGnygohp8oqEr1+YvkXSNw/Phxs37kyJHc2u23324ee8MN\nN5h1r4/vsXrW1rgBv1/d0dFh1r3rCFLm1Ht9/FOnTpl16xoG7/P2rn+YP3++Wfeu/bB2ryryepbJ\n+MpPFBTDTxQUw08UFMNPFBTDTxQUw08UFMNPFJTb5xeRbQC+BWBYVVdltz0G4HsArjZaH1XV54sa\nZCN4vVNvfrc1N9yqAf68dO+5vesAxsbGcmv79u0zj/WuQfD63XPmzDHr1udmzfUH/D7/mTNnzLq1\nF4O15Trgf794axF4+xVYX7POzk7z2JaWFrNeq1pe+X8J4P4pbv+Zqq7O/lU6+ET0RW74VfUVACNN\nGAsRNVHK7/w/FJG3RWSbiNzYsBERUVPUG/6fA1gBYDWA4wB+kndHEdkiIv0i0j86Olrn0xFRo9UV\nflU9qaqXVfUKgF8AWGvcd6uq9qlqnzWZgYiaq67wi0j3pA+/DeCdxgyHiJqlllbf0wC+CmChiAwB\n+BcAXxWR1QAUwCCA7xc4RiIqgBt+Vd00xc1PFjCWUnnzsxcsWJBb89bdHx4eNuu9vb1m3Zszb/Xa\nV69ebR7r9fEvXLhg1r158Vb99OnT5rEff/yxWf/www/NunXevM/b6sMDQFdXl1nv7u4263fddVdu\nbWTEbq55j10rXuFHFBTDTxQUw08UFMNPFBTDTxQUw08UFJfuzixcuNCs33HHHbk1b9nvQ4cOmfWb\nb77ZrHtTPGfNyv8y3nLLLeax3nRh67EBf/ltazqzNxXauxzcO+8HDx7Mre3Zs8c89vXXXzfrzz33\nnFlftWqVWbfOy8aNG81jG7U9OF/5iYJi+ImCYviJgmL4iYJi+ImCYviJgmL4iYK6Zvr8Xh/fq3vb\nbFtLNe/atcs81ttK+t133zXrnrlz5+bWvCWq582bZ9a9JahTlrj2+tXekubW5w3Yy5LfeKO97GR7\ne7tZ/+ijj8y6N43bun6iUX18D1/5iYJi+ImCYviJgmL4iYJi+ImCYviJgmL4iYKqVJ/f6xkX+dje\n8tsrV66s+7EHBgbM+nvvvWfWvaW7rfUAOjo6zGO9bbC9z827DsB6fG/rcm8bbGu+PmAv/e1t722t\n3wAAy5YtM+vLly836z09Pbk179qM1tZWs14rvvITBcXwEwXF8BMFxfATBcXwEwXF8BMFxfATBeX2\n+UVkKYCnAHQCUABbVfUJEWkH8BsAPQAGATyoqvaeytPYokWLcmve3HDrGgEAOHLkiFn3tqLev39/\nbs1bK6Ctrc2sW1uTA/b24B5v+29vXX7v+gdvq2uLt1aA18e/6aabzPqSJUtya945b5RaXvkvAfix\nqt4J4F4APxCROwE8AmCnqvYC2Jl9TETThBt+VT2uqnuy98cB7AewGMAGANuzu20HYG8zQkSV8qV+\n5xeRHgBrAPwBQKeqXv257AQmfi0gommi5vCLyAIAvwXwI1Udm1zTiYuwp7wQW0S2iEi/iPR7e68R\nUfPUFH4RmY2J4P9KVX+X3XxSRLqzejeAKVcsVNWtqtqnqn3eH5eIqHnc8MvEtK4nAexX1Z9OKu0A\nsDl7fzOAZxs/PCIqSi1Ter8C4LsABkRkb3bbowAeB/DfIvIwgD8BeLCYITaGNz3U22raOt5rzXjb\nf/f29pp1b+rr2NhYbs1bYtprh3ntOG/Kr7UFuLdFtzdd2Dvv1vLb3hRub1t0b8lzb+tz6/vJ+15t\nFDf8qvp7AHlf4a81djhE1Cy8wo8oKIafKCiGnygohp8oKIafKCiGnyioSi3dnSK1j+9ti2wtQe0t\nf+09dktLi1n3ppdaU4qtZb0B/7ykfm5W/ezZs+ax3hbd3nUC1vUR3ufl1b3vt2b16lPwlZ8oKIaf\nKCiGnygohp8oKIafKCiGnygohp8oqEr1+b3eqDV3PLXP7/V1rXntM2bY/4d6c95Ttya3PndvbKm8\n82r14r11ClLrKddmXAt9fA9f+YmCYviJgmL4iYJi+ImCYviJgmL4iYJi+ImCqlSfv0heX9br+6Y8\nduo1CN7a+dYa8V6fP3WdA29sKX1+77FT1hoou49fhesE+MpPFBTDTxQUw08UFMNPFBTDTxQUw08U\nFMNPFJTb5xeRpQCeAtAJQAFsVdUnROQxAN8DcCq766Oq+nxRAwXs3qg3Jz61r2r1lFN75d7687Nm\n2V8max/71POSug5CkefNO9763CL08T21XORzCcCPVXWPiLQCeFNEXsxqP1PVfytueERUFDf8qnoc\nwPHs/XER2Q9gcdEDI6Jifanf+UWkB8AaAH/IbvqhiLwtIttEZMo9o0Rki4j0i0j/6Oho0mCJqHFq\nDr+ILADwWwA/UtUxAD8HsALAakz8ZPCTqY5T1a2q2qeqfW1tbQ0YMhE1Qk3hF5HZmAj+r1T1dwCg\nqidV9bKqXgHwCwBrixsmETWaG36Z+HPxkwD2q+pPJ93ePelu3wbwTuOHR0RFqeWv/V8B8F0AAyKy\nN7vtUQCbRGQ1Jtp/gwC+X8gIa5Sy7Hctx6e0jbyWlDe1NXVp8BSpU19Tzltqu6zIdtt0aOV5avlr\n/+8BTPXdVWhPn4iKxSv8iIJi+ImCYviJgmL4iYJi+ImCYviJguLS3ZmUXnlqv7ro6chFShnbtfp5\nTRd85ScKiuEnCorhJwqK4ScKiuEnCorhJwqK4ScKSprZzxSRUwD+NOmmhQBON20AX05Vx1bVcQEc\nW70aObabVXVRLXdsavi/8OQi/araV9oADFUdW1XHBXBs9SprbPyxnygohp8oqLLDv7Xk57dUdWxV\nHRfAsdWrlLGV+js/EZWn7Fd+IipJKeEXkftF5D0ROSgij5QxhjwiMigiAyKyV0T6Sx7LNhEZFpF3\nJt3WLiIvisiB7O2U26SVNLbHRORYdu72isj6ksa2VET+V0T+KCL7ROQfs9tLPXfGuEo5b03/sV9E\nZgJ4H8DXAQwBeAPAJlX9Y1MHkkNEBgH0qWrpPWER+RsAZwA8paqrstv+FcCIqj6e/cd5o6r+U0XG\n9hiAM2Xv3JxtKNM9eWdpABsB/D1KPHfGuB5ECeetjFf+tQAOquphVb0A4NcANpQwjspT1VcAjHzu\n5g0Atmfvb8fEN0/T5YytElT1uKruyd4fB3B1Z+lSz50xrlKUEf7FAI5O+ngI1dryWwG8JCJvisiW\nsgczhc5s23QAOAGgs8zBTMHdubmZPrezdGXOXT07Xjca/+D3RetUdTWAbwL4QfbjbSXpxO9sVWrX\n1LRzc7NMsbP0n5V57urd8brRygj/MQBLJ328JLutElT1WPZ2GMAzqN7uwyevbpKavR0ueTx/VqWd\nm6faWRoVOHdV2vG6jPC/AaBXRJaLyBwA3wGwo4RxfIGIzM/+EAMRmQ/gG6je7sM7AGzO3t8M4NkS\nx/IXqrJzc97O0ij53FVux2tVbfo/AOsx8Rf/QwD+uYwx5IxrBYD/y/7tK3tsAJ7GxI+BFzHxt5GH\nAfwVgJ0ADgB4CUB7hcb2nwAGALyNiaB1lzS2dZj4kf5tAHuzf+vLPnfGuEo5b7zCjygo/sGPKCiG\nnygohp8oKIafKCiGnygohp8oKIafKCiGnyio/wdYS+Ia/cBCWQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12d10bf98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-29-a1fa6fdaf7af>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     10\u001b[0m     \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_output\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvsize\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhsize\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcmap\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'Greys'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m     \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     13\u001b[0m     \u001b[0mclear_output\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "import time\n",
    "from IPython.display import clear_output\n",
    "\n",
    "hidden_output = visualization.predict(train_dataset.reshape(len(train_dataset),vsize, hsize, num_channels))\n",
    "\n",
    "while True:\n",
    "    n = random.randint(0, len(train_dataset))\n",
    "    plt.imshow(train_dataset[n].reshape(vsize, hsize), cmap='Greys')\n",
    "    plt.show()\n",
    "    plt.imshow(hidden_output[n].reshape(vsize, hsize), cmap='Greys')\n",
    "    plt.show()\n",
    "    time.sleep(1)\n",
    "    clear_output(wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now we will view some of the deeper layers, although they will be less informative overall. **Note: this is designed to work with the second model. Please run that model, or change the depth of the second layer to 84**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visualization2 = Sequential()\n",
    "\n",
    "visualization2.add(Conv2D(1, 3, strides=1, padding = 'same', input_shape=(vsize, hsize, num_channels), weights=model.layers[0].get_weights())) # for visualization\n",
    "model.add(LeakyReLU(0.2))\n",
    "\n",
    "visualization2.add(Conv2D(84, 1, strides=1, padding = 'same', weights=model.layers[2].get_weights()))\n",
    "\n",
    "visualization2.add(Conv2D(84, 3, strides=1, padding = 'same', weights=model.layers[4].get_weights()))\n",
    "\n",
    "hidden_output = visualization2.predict(train_dataset[0:20].reshape(20,vsize, hsize, num_channels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE2pJREFUeJzt3V2IXOd5B/D/MzO7s5qVVtLqKxtJjeREpFVNI5dFDcSU\nFDfBNgE5FzXRRVDBRCmkoaG5qHEv6ktTGgdflIBSi8gldRKaGPvCtNiiYAJp8Np15K8kksU6lrT6\n/tjvna+nF3vsrO09z7OeMzNndp//D4Rm550z552z898zu89531dUFUQUTyHvDhBRPhh+oqAYfqKg\nGH6ioBh+oqAYfqKgGH6ioBh+oqAYfqKgSl3dWWVQ+zYOd3OXIajk3YPVR9boha21W9dRn51Z0Tsi\nU/hF5G4AjwEoAvg3VX3EenzfxmHseeDvs+ySlsHwf3RrNfzjjz+64se2/LFfRIoA/hXAPQD2Azgs\nIvtbfT4i6q4sv/MfBHBGVc+qahXAjwAcak+3iKjTsoR/J4B3lnx9LrnvfUTkqIiMichYfXYmw+6I\nqJ06/td+VT2mqqOqOlqqDHZ6d0S0QlnCfx7A7iVf70ruI6JVIEv4XwSwT0T2ikg/gK8AeKY93SKi\nTmu51KeqdRH5WwD/jcVS33FVfb1tPaPe0MkyYofLbWu1nNeu45apzq+qzwJ4tj1dIaJu4uW9REEx\n/ERBMfxEQTH8REEx/ERBMfxEQXV1PH9UmYfcetsb7er9eHeeO2vfM9Xam9me21yMytl2zV4jsATP\n/ERBMfxEQTH8REEx/ERBMfxEQTH8REHFKfU5pZtCw24Xo724YG9bH7DbtWi3N7ztjR/hWvBqWnZz\noWY/oFCzty/Nprc1++xtvb41ytm2t3iVvrVQCuSZnygohp8oKIafKCiGnygohp8oKIafKCiGnyio\nOHV+R8Gp1Vv1bLcO7xzlesUuGnu19mafsb1X63aGzXrXMFh1fACoV4xdO3X+xoB9XIqzzouzmtdA\nnT4rnvmJgmL4iYJi+ImCYviJgmL4iYJi+ImCYviJgspU5xeRcQBTABoA6qo62o5OdYI3/ro033rh\nt1q2683edQCeRn+G6wCcl1Ws2u2VCWffDbt9oZl+frGuAQCA4px3fYO9fU/r5NLnK9SOi3z+QlWv\ntuF5iKiL+LGfKKis4VcAz4vISyJytB0dIqLuyPqx/05VPS8i2wE8JyK/VtUXlj4g+aFwFABKQ5sz\n7o6I2iXTmV9Vzyf/XwbwFICDyzzmmKqOqupoqTKYZXdE1EYth19EBkVkw7u3AXwRwGvt6hgRdVaW\nj/07ADwlIu8+z3+o6n+1pVdE1HEth19VzwL4TBv70lHeUtXVIbvwWh1Kr2fXhuxB8eUb9s5LM07R\n1xtzb9TqCzW7Du+Nx++fsXc+t8V+bXPb0/dfvuasCVA3myEz9mtr9qU/v3eNgLeWgvd+cpc274H5\nBFjqIwqK4ScKiuEnCorhJwqK4ScKiuEnCirM1N1N55XWNji1lz3pNbFPbL1pbjr+zjazvXK632wv\nT9p9659Obx+4Ya89XpyzS3mNsn1+qO61a2LV4fT9F6r2tptOOzVOR31da20A0HCGaXvvJ2QsFXZD\nD3SBiPLA8BMFxfATBcXwEwXF8BMFxfATBcXwEwUVps4vdrkbhaozpNeoSe/ZcN3cdnp72W5/e6vZ\n7tX5K5fS1w/vvzZnbisz82Z7c71dEB/42JDZPr03ve/eEtye/in7myrGtOHeec8dkuuNwvaG/DpP\n3w088xMFxfATBcXwEwXF8BMFxfATBcXwEwXF8BMFtXbq/E7h1Fuiu2/Gbq9dTx9z/7tpexmyTevs\nWvu0vWv0T9rj2vuvpHe+MG3X8VGz58cuFOzzQ+WKXWsfuJz+FtOCM634gv26S7POxRsGa1pvAGjY\nl2ag6V4I0Pt45icKiuEnCorhJwqK4ScKiuEnCorhJwqK4ScKyq3zi8hxAF8CcFlVb0/uGwbwYwB7\nAIwDuF9Vb2TuTQcHOXt1fm/fxYX0uu6GvgVz210Ve17/t7Z/3N65x6jFa9lei1orA2Z7c729poA0\n7ANXNC4zqK03N0Wt4lxjcM5YmxxAcTb9eyZOIb9ZtI9bwz4sq8JKzvw/AHD3B+57EMBJVd0H4GTy\nNRGtIm74VfUFAB+cquYQgBPJ7RMA7mtzv4iow1r9nX+Hqk4kty8C2NGm/hBRl2T+g5+qKozfmEXk\nqIiMichYfda5gJ6IuqbV8F8SkREASP6/nPZAVT2mqqOqOlqqDLa4OyJqt1bD/wyAI8ntIwCebk93\niKhb3PCLyJMAfgHg0yJyTkQeAPAIgC+IyGkAf5l8TUSriFvnV9XDKU13tbTHXpiwfBlWPRoASlPp\nNePzUxvNbTeXZ832vpv2z2Bp2GPuVdL71thcMbdtDNhvAW/cuzTtb2j5Rnq7V+f31rCXeee4VNJr\n9dWN9uuuDtmvu+nU+VfDcH9e4UcUFMNPFBTDTxQUw08UFMNPFBTDTxTU2pm6OyOxZ4k2hwRvqdiX\nLc/UvWGx9r4LzrDZplHS8kpORadcVp6wy5S4eNVuxydTW+a228elbo82RmPIHpY7vy29fXqnfd5r\nOsnQ9BXbkwc47T2AZ36ioBh+oqAYfqKgGH6ioBh+oqAYfqKgGH6ioOLU+Z26a6FuP8AaNntrwS5I\nv3Vxm9m+83/tWvvAS2fN9sa1D86v+nvFij2k16NN+wKIwiZ7OPOtvenXINT+yL6G4NqIfR3A5Cft\n11bbmn5cS8ZQYwDov2FfIOEt6V5fZ7f3Ap75iYJi+ImCYviJgmL4iYJi+ImCYviJgmL4iYLqfp0/\ny5TGGcZIe0t0e+P5G5XWd64X7OsASnP2UtOy3l7pqLBzu7GxM/V23XnhNfsahPldm8z2WWP18ZEt\nt8xt/+wPx832u4beMNtPTu5PbfvPF0fNbQffcZbotqcSWBV45icKiuEnCorhJwqK4ScKiuEnCorh\nJwqK4ScKyq3zi8hxAF8CcFlVb0/uexjA1wBcSR72kKo+26lO/r4zGTZ15sZvOvOw1zam18PVmRy/\nNGP/jC1N2XV+LTvz229MHzw+ude+xmB+i9332RHn+oZdc3bztgupbZ8ZPm9ue/fGU2b7x4tTZnvN\nmFy/MGt/w2sbzGYUF+z2VTBt/4rO/D8AcPcy939XVQ8k/zoffCJqKzf8qvoCgPSpYohoVcryO/83\nReSUiBwXkc1t6xERdUWr4f8egNsAHAAwAeA7aQ8UkaMiMiYiY/VZZ+IzIuqalsKvqpdUtaGqTQDf\nB3DQeOwxVR1V1dFSxR6gQkTd01L4RWRkyZdfBvBae7pDRN2yklLfkwA+D2CriJwD8E8APi8iB7BY\n0RgH8PUO9pGIOsANv6oeXubuxzvQl45S5zPOwma73t2/PX2O+Xt22uPKL2y169ln79xqti807c4P\n9l1MbSs7awqUnGsUdg/Ydfw/GLxhtt+spl+DMD69xdz2af1Ts/03t3aY7WdO7Uptq0w4117YSwqg\naQ/3XxV4hR9RUAw/UVAMP1FQDD9RUAw/UVAMP1FQYZbo9qZaXhi2p7C+Y2Qite3Q0P+Z236qzx7g\neaVhT499vWkP6b3YGEptO73wMXPb8wv2sIybNXsZ7HOz9tTdv54wphV3yowXhtJfFwBcPTtstm98\nK/3cVpyzvyeNcpY55v2p4nthyC/P/ERBMfxEQTH8REEx/ERBMfxEQTH8REEx/ERBrZ06v7cEtzN1\nd6Fm13Wvzq1PbWs4c4qvL9gXGVxpTJvtz0//sdn+k/E7UttuvG3X8ctX7Sms+ybNZvRN2wd+62R6\n+9Qu+9xz9U/svkndWX68kb5vrw6flXMJQ0/gmZ8oKIafKCiGnygohp8oKIafKCiGnygohp8oqLVT\n53cU7CHzGLhmF2bPvzyS2vZXE39jbtu/rma2L0za1wGUL9jzRA+eS2/bddm+wKH/5rzZXpqx+y41\n+/m1mH5cF5zx+tWy/U2r2cP5UVufPm15oZat0L8a6vgenvmJgmL4iYJi+ImCYviJgmL4iYJi+ImC\nYviJgnLr/CKyG8ATAHZgcdT8MVV9TESGAfwYwB4A4wDuV1V7veasjNKsNz5b7Gn5UVyw24duprcV\nfmvX6Qs1u720YHe+ULM7X6ymtxcW7G0LVbtOL1W71i7z9nUAKKe/xYpVe9N6zZlrYMDuW9N4d6v9\n1C5vyXdnioeesJIzfx3At1V1P4DPAviGiOwH8CCAk6q6D8DJ5GsiWiXc8KvqhKq+nNyeAvAmgJ0A\nDgE4kTzsBID7OtVJImq/j/Q7v4jsAXAHgF8C2KGq765hdRGLvxYQ0Sqx4vCLyHoAPwXwLVV938xu\nqqpI+Y1cRI6KyJiIjNVnZzJ1lojaZ0XhF5E+LAb/h6r6s+TuSyIykrSPALi83LaqekxVR1V1tFQZ\nbEefiagN3PCLiAB4HMCbqvrokqZnABxJbh8B8HT7u0dEnbKSIb2fA/BVAK+KyCvJfQ8BeATAT0Tk\nAQBvA7g/c2+yjLL0tvXavVKgUZYqOcs9l7xym1Pqs0p5AFAw2ovzdjmsMGPXOGXWHvKLujcnevoS\n3/1T9utqztpvz9IWu1ZY25B+XEtzTi3OK9WtglKexw2/qv4c6S/1rvZ2h4i6hVf4EQXF8BMFxfAT\nBcXwEwXF8BMFxfATBRVm6u7MjFJ8oe7U6eed6wCm7WGxxTm7vTCdXqv36vQ6N2e31505z0vOW2hd\n+nDmvmn7GoHipP3cstXedX1j+nUEtRln+W/nug+vfTVM7c0zP1FQDD9RUAw/UVAMP1FQDD9RUAw/\nUVAMP1FQa6fO79RV3amWnXZrqmf3udWZmrvujNd36vxWLd+r4yNjHV/W27MzNTalt1eH7Fp7Y8i+\nDuCePW+Y7c8XP53aNj212dx23RX7DdW0V03PxrtGINvq4u/hmZ8oKIafKCiGnygohp8oKIafKCiG\nnygohp8oqDVT5/fGT4vzY86r2zaMVbbrdfvJizW7MNuoOrX0BbtzhXp656TovPCC3a7GeHwAqA0N\nmO3V4f7UtoVN9r437bhltu9bd8ls/0V5b3q/JrPV8VfDeH0Pz/xEQTH8REEx/ERBMfxEQTH8REEx\n/ERBMfxEQbl1fhHZDeAJADuwOJL4mKo+JiIPA/gagCvJQx9S1Wcz9aaD45ib9tBxl0p657zx/M2i\nvfN62X6CvkH721ScT6+1Fxr2QdOCfdAbA3bfapXW2+e22vseLNnj+cfn7Yn7r02lzyVQ22wfl/5b\ndt/E7pqvB64TWMlFPnUA31bVl0VkA4CXROS5pO27qvovneseEXWKG35VnQAwkdyeEpE3AezsdMeI\nqLM+0u/8IrIHwB0Afpnc9U0ROSUix0Vk2XmRROSoiIyJyFh9diZTZ4mofVYcfhFZD+CnAL6lqpMA\nvgfgNgAHsPjJ4DvLbaeqx1R1VFVHSxV7vjci6p4VhV9E+rAY/B+q6s8AQFUvqWpDVZsAvg/gYOe6\nSUTt5oZfRATA4wDeVNVHl9w/suRhXwbwWvu7R0SdspK/9n8OwFcBvCoiryT3PQTgsIgcwGIBbhzA\n1zvSw6Ws8kjG6Yy9UqBR6fNLfX12XadesdsXNnplp/TOi3NcvKGp1pTlANAs2U+gGQaNXz2zxWx/\n8nefNduLM+nfmL6ptV/K86zkr/0/x/IvJVtNn4hyxSv8iIJi+ImCYviJgmL4iYJi+ImCYviJgur+\n1N1tWl6466w6v1cLd2q+Tee7IGWnJm0dU+94e3V+r16dYWn0glNLr1xwzk3OfOy5Tq+9Ct7nPPMT\nBcXwEwXF8BMFxfATBcXwEwXF8BMFxfATBSWq3StIisgVAG8vuWsrgKtd68BH06t969V+Aexbq9rZ\nt0+o6raVPLCr4f/QzkXGVHU0tw4YerVvvdovgH1rVV5948d+oqAYfqKg8g7/sZz3b+nVvvVqvwD2\nrVW59C3X3/mJKD95n/mJKCe5hF9E7haR34jIGRF5MI8+pBGRcRF5VUReEZGxnPtyXEQui8hrS+4b\nFpHnROR08v+yy6Tl1LeHReR8cuxeEZF7c+rbbhH5HxF5Q0ReF5G/S+7P9dgZ/crluHX9Y7+IFAH8\nFsAXAJwD8CKAw6r6Rlc7kkJExgGMqmruNWER+XMA0wCeUNXbk/v+GcB1VX0k+cG5WVX/oUf69jCA\n6bxXbk4WlBlZurI0gPsA/DVyPHZGv+5HDsctjzP/QQBnVPWsqlYB/AjAoRz60fNU9QUA1z9w9yEA\nJ5LbJ7D45um6lL71BFWdUNWXk9tTAN5dWTrXY2f0Kxd5hH8ngHeWfH0OvbXktwJ4XkReEpGjeXdm\nGTuSZdMB4CKAHXl2Zhnuys3d9IGVpXvm2LWy4nW78Q9+H3anqh4AcA+AbyQfb3uSLv7O1kvlmhWt\n3Nwty6ws/Z48j12rK163Wx7hPw9g95KvdyX39QRVPZ/8fxnAU+i91YcvvbtIavL/5Zz7855eWrl5\nuZWl0QPHrpdWvM4j/C8C2Ccie0WkH8BXADyTQz8+REQGkz/EQEQGAXwRvbf68DMAjiS3jwB4Ose+\nvE+vrNyctrI0cj52Pbfitap2/R+Ae7H4F/+3APxjHn1I6ddtAH6V/Hs9774BeBKLHwNrWPzbyAMA\ntgA4CeA0gOcBDPdQ3/4dwKsATmExaCM59e1OLH6kPwXgleTfvXkfO6NfuRw3XuFHFBT/4EcUFMNP\nFBTDTxQUw08UFMNPFBTDTxQUw08UFMNPFNT/Awh97qPE2LtWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x139326e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEoFJREFUeJzt3W+MXOV1BvDnmfGsd73eDV4bHMeBYAsniqGtabc0UlBE\nlSYCmsTkQylui4xkxUhNUWmjNIh+CB/6AVUNCEUVrVOsmCqQREooqKGh4FZFiDRlQY75Y1IcYood\n48XYsXft/TM7c/phx9ECe8+7zJ2ZO7vn+UmWd+fMnfvu7Dx7Z/fc+740M4hIPKWiByAixVD4RYJS\n+EWCUvhFglL4RYJS+EWCUvhFglL4RYJS+EWCWtbRnfX1W2VwqJO7FAmlevoEZibOcCH3zRV+klcD\nuAdAGcA/mdmd3v0rg0O45I/+Ms8uRcRx8IG7Fnzfpt/2kywD+HsA1wDYDGAbyc3NPp6IdFae3/mv\nAHDQzF41s2kA3wawtTXDEpF2yxP+9QBen/P54cZtb0NyJ8kRkiO1iTM5dicirdT2v/ab2S4zGzaz\n4XJff7t3JyILlCf8RwBcOOfzDzZuE5FFIE/4nwGwieQGkj0AbgDwSGuGJSLt1nSrz8xmSP4ZgMcw\n2+rbbWYvtmxkItJWufr8ZvYogEdbNBYR6SCd3isSlMIvEpTCLxKUwi8SlMIvEpTCLxJUR6/nl+aw\nnYsq5XzsPGOzBV117u28wH0vATryiwSl8IsEpfCLBKXwiwSl8IsEpfCLBKVWXwck22GJeq7tU49d\n9++Qd+z+gyceOlUvJe7gHdpSrb6cY1sMdOQXCUrhFwlK4RcJSuEXCUrhFwlK4RcJSuEXCUp9/gVy\n+905+/SsJXrtNX/7UjW7Vq76j12eTjz2TGJsifMEPPVlfrO8VknUexKP79TriVe+lXOcQ7AA3XCe\ngI78IkEp/CJBKfwiQSn8IkEp/CJBKfwiQSn8IkHl6vOTPARgDEANwIyZDbdiUEVI9eLPfiD7Drd8\n1l+o+L5/+H23Xpnw912e8gdXmahn1paN+ycJVMadkwQAlCb8+vjGQbe+4a8OZNZ+dvdmd9uVv5hy\n69MDFbde7c8+ts30pc4h8J/z1DkKVnbL7nQBnToHoBUn+fyumR1vweOISAfpbb9IUHnDbwCeIPks\nyZ2tGJCIdEbet/1XmtkRkhcAeJzky2b25Nw7NH4o7ASAysCqnLsTkVbJdeQ3syON/0cBPATginnu\ns8vMhs1suNzXn2d3ItJCTYefZD/JgXMfA/g0gBdaNTARaa88b/vXAniI5LnHecDMftiSUYlI2zUd\nfjN7FcBvtHAsbdXO+edvWfWaW/+XrYfd+uQ/rnPrlfHsPj4A9JzKvih/2alJd1ueGnfrNu7X+1b4\nF9WfmMr+VW/0N/2G9iUP+CdAlKb8cxhKM8sza0w14lMT9ydeUHXmOw+gE9TqEwlK4RcJSuEXCUrh\nFwlK4RcJSuEXCSrO1N3Jpar9+sCr2bUvv3G5u+3NF/2XW//6zA1ufflJ/9LW8okz2cVTY+629TG/\nlVef8Ntt/NFP3PrLR7Kfm94Pn3a3LY0n2pRTM269Us4+ttVT04JX/ONi6pJepFp53uuxQ5f06sgv\nEpTCLxKUwi8SlMIvEpTCLxKUwi8SlMIvEtSS6fMnl8FO9vmbXyb7qWMb3W1vGnrarR//db8pvOGn\niX73WHaf36b8cwSQuPS0tGKFv33JP34MPd6bWVuz/Zi7bX3gff6uT5/169PZ3zQmlx53y0uCjvwi\nQSn8IkEp/CJBKfwiQSn8IkEp/CJBKfwiQS2ZPn9uOa73n6n5ffpf1rOnkAaAqSG/qczJ7Km5AQCW\nPXj2ZvfZAYCJZa5RSbxEnH0DwPJT2V/buj7/ev6XLr3IrQ/9tz/XgLvUdbuvmU89foeu2ffoyC8S\nlMIvEpTCLxKUwi8SlMIvEpTCLxKUwi8SVLLPT3I3gM8AGDWzyxq3DQH4DoCLARwCcL2ZnWzfMLvb\nyQOr3frkR/1eet+xxM/gRC8dy7OXybZe/xwD6/OX2Lay35DmjH+OwsqfZ68b8ObUSnfbkx/xn5dV\nz/ovX1uWvX3q67LUtyR12OyCPn7KQo783wRw9Ttuuw3AXjPbBGBv43MRWUSS4TezJwGceMfNWwHs\naXy8B8B1LR6XiLRZs7/zrzWzo42P3wCwtkXjEZEOyf0HPzMzOGfGk9xJcoTkSG3CWVNORDqq2fAf\nI7kOABr/j2bd0cx2mdmwmQ2X+/qb3J2ItFqz4X8EwPbGx9sBPNya4YhIpyTDT/JBAD8C8BGSh0nu\nAHAngE+RfAXA7zU+F5FFJNnnN7NtGaVPtngsXc2b979/0y/dbSfN7/OX/Wn5k3Prm9Pnr6/w+/z1\nXv8lwFpiroGqs6ABAJ7J/uLOVP1zDAaH33Tr9pC/fb0ne56FeiXV529vo96da6BDdIafSFAKv0hQ\nCr9IUAq/SFAKv0hQCr9IUGGm7k61VpKdF6fVVy757bCXJte79Que83t9ybZT2fkZnvjxXkq06koT\nVbfOM4nps536mWl/Ce7hC1536/9+4xa3XvpA9r7rR/w24er9bnlJ0JFfJCiFXyQohV8kKIVfJCiF\nXyQohV8kKIVfJKgl0+dP9vETs1+neEt03/rhve62P3zr19x6z7Hs6a3z4lTiktvJKf8Bxvyp16zm\nP/7pqy7J3jePu9vWEt/Uv7jmB259S+9r2dse+EN3W+xf45aTr6ecr7dO0JFfJCiFXyQohV8kKIVf\nJCiFXyQohV8kKIVfJKgl0+fPy+vjz9azG7dV85/Gemq2gMTU3Cj5P6Otkj1F9dgm/5r5mT5/38f9\nS+Zx/qX+9NrLy8cya4OVaXfbcqKZXob/TXurlr0E+NTj57vb9qaWRc/J+9I6Na23jvwiQSn8IkEp\n/CJBKfwiQSn8IkEp/CJBKfwiQSX7/CR3A/gMgFEzu6xx2x0AvgDgXJP3djN7tF2D7ITU9dknP5rd\nfP0t57pxAOhf418z/4vvrnLrT5/c6NYvHTyaWTtZXeFue/jseW69f7rPrVdr2ecYAMDy8kxmbWXF\nf176y3796weucuurHsju8w+aPw9BdUWq2d4Fa2zntJAj/zcBXD3P7Xeb2ZbGv0UdfJGIkuE3sycB\nnOjAWESkg/L8zn8Lyf0kd5P037eKSNdpNvz3AtgIYAuAowC+lnVHkjtJjpAcqU3488GJSOc0FX4z\nO2ZmNTOrA/gGgCuc++4ys2EzGy739Tc7ThFpsabCT3LdnE8/D+CF1gxHRDplIa2+BwFcBWANycMA\nvgrgKpJbMDtB8SEAN7dxjCLSBsnwm9m2eW6+rw1jaau886xPr/b7wp7ekr/G/crypFu/cuigWz9b\nW55ZOzjmX7f+8+Or3frUqH+eQOp56/+/7PMA3jrpb7zsppfd+sSR7D4+AKw9lX2OQXVQU1noDD+R\noBR+kaAUfpGgFH6RoBR+kaAUfpGglky/I28rLzV199qnsn9O3rH5c+62f7r+P/wHT3jg0G+79fGn\ns9t5Fz122t12w6nEKdens6feBgAkluhGObvVx54ef9d/3OvWV2046e8bA5mV1PTYVvLv0KnptdtJ\nR36RoBR+kaAUfpGgFH6RoBR+kaAUfpGgFH6RoJZMnz8p53kApeyrQzF67wZ3269yh1uvTPgnGZw3\n6k9hPVQdy6xxJrX2uN+wZqXi1tu7kLXvcxf5c8j82/s/kVkrT/kjT/bxl8DM3jryiwSl8IsEpfCL\nBKXwiwSl8IsEpfCLBKXwiwQVps9P8/u6qfkAvHpl3L+mvZzo45cnnZMIAJSq/vasZu+f0/5js+rX\nrepPO47pRN27Ln6Z//I7cPj9bn3buv9x6xNrso9t/W/4z6n5K4/DlsBhcwl8CSLSDIVfJCiFXyQo\nhV8kKIVfJCiFXyQohV8kqGSfn+SFAO4HsBazl2/vMrN7SA4B+A6AiwEcAnC9maUmUm+fnBeWJ6/f\n9s4TSFwyX5r2zwMoTST6/FPN99o5Oe1uamfO+vUpfy4B1BPnIPRmLx+Okn/sKb/uz9t//sf8NQmc\nlctRqyTm5U/1+RPzICyGef0XcuSfAfAlM9sM4GMAvkhyM4DbAOw1s00A9jY+F5FFIhl+MztqZs81\nPh4DcADAegBbAexp3G0PgOvaNUgRab339Ds/yYsBXA7gxwDWmtnRRukNzP5aICKLxILDT3IlgO8B\nuNXM3vbLlpkZMn7rJrmT5AjJkdpEYl04EemYBYWfZAWzwf+WmX2/cfMxkusa9XUARufb1sx2mdmw\nmQ2X+/pbMWYRaYFk+EkSwH0ADpjZXXNKjwDY3vh4O4CHWz88EWmXhVzS+3EANwJ4nuS+xm23A7gT\nwHdJ7gDwGoDr2zPEBUotuZxqzZQSUzmXne0T+05eLuxckgsAnPDbbXZ2IrNWd2oAYBOJet0ffMlr\n5QFgb3a7zgZWuNue97JbxvplOVp9/rBRTyUjddhMvh4T23dAMvxm9hSyv5RPtnY4ItIpOsNPJCiF\nXyQohV8kKIVfJCiFXyQohV8kqDBTd6d+zNX9lagxszy7MVvqS1yaOuVfH1qaTFw/ejbRFE704j3s\n6XHrpcT02hxY6dbrqwYzazOr+9xtp9/nf90fWubXB37nzex9/+sad1v3vA50R58+Lx35RYJS+EWC\nUvhFglL4RYJS+EWCUvhFglL4RYJaMn3+ZN819WMu0TOu9Wb30qvJ9Zr9p9noT1Hdk5jiurQ8+ySF\nUmLqbqTmOej1zwOYWelfGF89L7s+PeCf31Ce9M9feG3Gr39l02OZtb8p/4m7bfL1tAiu10/RkV8k\nKIVfJCiFXyQohV8kKIVfJCiFXyQohV8kqCXT50/Jex5A3VnSuZqY879W8R98ps+fTGD6PP/bVJ7K\nvi6etXxrl3tfNwDUehJ1Zx6E1LapXvpkYh3tLz/9B5m1wZyHvcXQx0/RkV8kKIVfJCiFXyQohV8k\nKIVfJCiFXyQohV8kqGSfn+SFAO4HsBaAAdhlZveQvAPAFwCcmxz9djN7tF0Dbbc85wFY6pr4vGsG\n9KXm7c8uMdHmz33deuJr8+pWytfn33HXrW59MEcvfin08VMWcpLPDIAvmdlzJAcAPEvy8UbtbjP7\nu/YNT0TaJRl+MzsK4Gjj4zGSBwCsb/fARKS93tPv/CQvBnA5gB83brqF5H6Su0muythmJ8kRkiO1\niTO5BisirbPg8JNcCeB7AG41s9MA7gWwEcAWzL4z+Np825nZLjMbNrPhcl9/C4YsIq2woPCTrGA2\n+N8ys+8DgJkdM7OamdUBfAPAFe0bpoi0WjL8JAngPgAHzOyuObevm3O3zwN4ofXDE5F2Wchf+z8O\n4EYAz5Pc17jtdgDbSG7BbPvvEICb2zLCLuG1flJdoXQ7LNEqTH2XvHZe6orevC2tdk5hnXNsEdp1\neSzkr/1PYf5vw6Lt6YuIzvATCUvhFwlK4RcJSuEXCUrhFwlK4RcJKszU3e2U97LY5OPn21xkXjry\niwSl8IsEpfCLBKXwiwSl8IsEpfCLBKXwiwRFs851kUm+CeC1OTetAXC8YwN4b7p1bN06LkBja1Yr\nx/YhMzt/IXfsaPjftXNyxMyGCxuAo1vH1q3jAjS2ZhU1Nr3tFwlK4RcJqujw7yp4/55uHVu3jgvQ\n2JpVyNgK/Z1fRIpT9JFfRApSSPhJXk3ypyQPkrytiDFkIXmI5PMk95EcKXgsu0mOknxhzm1DJB8n\n+Urj/3mXSStobHeQPNJ47vaRvLagsV1I8j9JvkTyRZJ/3ri90OfOGVchz1vH3/aTLAP4XwCfAnAY\nwDMAtpnZSx0dSAaShwAMm1nhPWGSnwAwDuB+M7uscdvfAjhhZnc2fnCuMrOvdMnY7gAwXvTKzY0F\nZdbNXVkawHUAbkKBz50zrutRwPNWxJH/CgAHzexVM5sG8G0AWwsYR9czsycBnHjHzVsB7Gl8vAez\nL56OyxhbVzCzo2b2XOPjMQDnVpYu9LlzxlWIIsK/HsDrcz4/jO5a8tsAPEHyWZI7ix7MPNY2lk0H\ngDcArC1yMPNIrtzcSe9YWbprnrtmVrxuNf3B792uNLMtAK4B8MXG29uuZLO/s3VTu2ZBKzd3yjwr\nS/9Kkc9dsytet1oR4T8C4MI5n3+wcVtXMLMjjf9HATyE7lt9+Ni5RVIb/48WPJ5f6aaVm+dbWRpd\n8Nx104rXRYT/GQCbSG4g2QPgBgCPFDCOdyHZ3/hDDEj2A/g0um/14UcAbG98vB3AwwWO5W26ZeXm\nrJWlUfBz13UrXptZx/8BuBazf/H/GYC/LmIMGePaCOAnjX8vFj02AA9i9m1gFbN/G9kBYDWAvQBe\nAfAEgKEuGts/A3gewH7MBm1dQWO7ErNv6fcD2Nf4d23Rz50zrkKeN53hJxKU/uAnEpTCLxKUwi8S\nlMIvEpTCLxKUwi8SlMIvEpTCLxLU/wNLV5xx9/4ZMAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x137e93278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "n = random.randint(0, 19) # which image\n",
    "m = random.randint(0, 83) #which layer\n",
    "\n",
    "plt.imshow(hidden_output[n,:,:,m])\n",
    "plt.show()\n",
    "\n",
    "plt.imshow(train_dataset[n].reshape(28, 28))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is fascinating, although not that enlightening. Stil it's nice to see something of what's going on. For more information about visualization techniques, check out [this paper](https://arxiv.org/pdf/1311.2901.pdf). We can go another layer deeper, after the image has been downsampled, and see how it sumarizes information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visualization3 = Sequential()\n",
    "\n",
    "visualization3.add(Conv2D(1, 3, strides=1, padding = 'same', input_shape=(vsize, hsize, num_channels), weights=model.layers[0].get_weights())) # for visualization\n",
    "model.add(LeakyReLU(0.2))\n",
    "\n",
    "visualization3.add(Conv2D(84, 1, strides=1, padding = 'same', weights=model.layers[2].get_weights()))\n",
    "\n",
    "visualization3.add(Conv2D(84, 3, strides=1, padding = 'same', weights=model.layers[4].get_weights()))\n",
    "\n",
    "visualization3.add(Conv2D(84, 3, strides=2, padding = 'same', weights=model.layers[6].get_weights()))\n",
    "\n",
    "hidden_output = visualization3.predict(train_dataset[0:20].reshape(20,vsize, hsize, num_channels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADjZJREFUeJzt3WusXOV1xvHnORfb5xh8g9YKNsVuSkxcBHVkESBN2sa0\ncoiFUdUPoNCaJlW+tA1JI6VGVIryrVIilEhNEyFCQhsLFDmkQSghOM4FRS0IY5DrW2xjCBhfsYNj\nbMfn4tUPM5ack/jS/e7ZZybr/5OsMzOe1+s943lm79mz31mOCAHIp2+yJwBgchB+ICnCDyRF+IGk\nCD+QFOEHkiL8QFKEH0iK8ANJDTRabHh6DM6Y02RJlJ7A6Vpm0Xu1e/TE19GjRzR28vhFPXKNhn9w\nxhz9/qp/arLkbwWfLhg7Xlq8bPjpgmdYFO6XuiTAkxn+gtq7/+P+i74vu/1AUoQfSKoo/LaX2/6p\n7V22V9c1KQCdVzn8tvslfVHSByQtlnSn7cV1TQxAZ5Vs+W+QtCsidkfEiKRHJa2sZ1oAOq0k/PMk\nvXbW9T3t2wD0gI4f8LP9UdsbbG8YP3m80+UAXKSS8L8u6cqzrs9v3/YrIuKBiFgaEUv7h6YXlANQ\np5LwPyfpatsLbU+RdIekx+uZFoBOq3z+VUSM2f4HSd+T1C/poYjYUtvMAHRU0em9EfEdSd+paS4A\nGsQZfkBShB9IivADSTW6pLeXlSwvLVmSK0kzX66+Lnd4z4mi2qcum1Y0/ueLBqvXnl1UWn2jkzNW\nKv8/bwJbfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFJp\nlvSWdIuVpPGh6q1TPV7W6nbgRPX1of0/O1BUe3jHqaLxR975h5XHjiz8ZVFtvzGl8tipPy/bLvaN\nVB9b3Fn5IrHlB5Ii/EBShB9IivADSZW06L7S9g9tb7W9xfY9dU4MQGeVHAMfk/TJiNho+1JJz9te\nFxFba5obgA6qvOWPiH0RsbF9+ZikbaJFN9AzannPb3uBpCWSnv0Nf0eLbqALFYff9iWSvinp4xHx\ni4l/T4tuoDsVhd/2oFrBXxMRj9UzJQBNKDnab0lfkbQtIu6vb0oAmlCy5X+PpL+W9H7bL7b/3FrT\nvAB0WOWP+iLiJ5LKVqwAmDSc4QckRfiBpHpqPX9Jm+zTU8tqn7q8+iLrgVkFi7slvXJF9TbX86Yv\nLKo9Y/32ovHH3jFWeey73/5KUe3n+q6qPHb8ZFlr8r7RgnfEDb2ZZssPJEX4gaQIP5AU4QeSIvxA\nUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IKs2S3vGp1VtsS1LfSPXiM348VFR7zvbq\nrar7nynroeI5s8vGF7Qnf2u0bB12f3/11uYlz7VekeBXBPCbEH4gKcIPJEX4gaTqaNfVb/sF20/U\nMSEAzahjy3+PWh16AfSQ0l598yV9UNKD9UwHQFNKt/yfl/QpSef8QJUW3UB3KmnUuULSwYh4/nz3\no0U30J1KG3XeZvsVSY+q1bDz67XMCkDHVQ5/RNwbEfMjYoGkOyT9ICLuqm1mADqKz/mBpGpZ2BMR\nP5L0ozr+LQDNYMsPJEX4gaR6aj3/ZJqxs/rr5NxvlJ0A6RmXVB771l9cX1R7ypujReOvfLL69yjs\n3l/WXnx0XvX24FOqfxVAS8nXR5R99cRFY8sPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8k\nRfiBpAg/kBThB5Ii/EBShB9IKs2S3tOFv+nRReOVxx77l0VFtS/9gzcrj719wU+Kaj+5951F42et\nnlJ57IKXqj/mkrTzby6rPrihZbWTiS0/kBThB5Ii/EBShB9IqrRR5yzba21vt73N9k11TQxAZ5Ue\n7f+CpCcj4q9sT5E0XMOcADSgcvhtz5T0Pkl3S1JEjEgaqWdaADqtZLd/oaRDkr5q+wXbD9r+tTa8\ntOgGulNJ+AckvUvSlyJiiaTjklZPvBMtuoHuVBL+PZL2RMSz7etr1XoxANADSlp075f0mu0z564u\nk7S1llkB6LjSo/3/KGlN+0j/bkl/Wz4lAE0oCn9EvChpaU1zAdAgzvADkiL8QFK9tZ7f1YeenlrW\nc/n3rjlQeeyfzd1RVPvd01+qPHb58Kmi2ldNfaNo/DdOvrfyWI+VreePwYJF+eMFTzap6LnaFLb8\nQFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kFRvrecv\nWJ49eKzsdW7v4ZmVxx6afWlR7eNDUyuP3TRytKj2Z55eWTT+HTueqzz28N+VdX8bn179+wD6T/YX\n1S55rjaFLT+QFOEHkiL8QFKlLbo/YXuL7c22H7E9ra6JAeisyuG3PU/SxyQtjYhrJfVLuqOuiQHo\nrNLd/gFJQ7YHJA1L2ls+JQBNKOnV97qkz0l6VdI+SUcj4qmJ96NFN9CdSnb7Z0taKWmhpCskTbd9\n18T70aIb6E4lu/23SHo5Ig5FxKikxyTdXM+0AHRaSfhflXSj7WHbVqtF97Z6pgWg00re8z8raa2k\njZL+t/1vPVDTvAB0WGmL7k9L+nRNcwHQIM7wA5Ii/EBSPbWkd7DgNIHffWGsqPbRPcOVx/542tuL\nao9G9dfof971l0W1r/n3snMzTqy4ofLYwzePFtXWqeqPW99YD/TYLsSWH0iK8ANJEX4gKcIPJEX4\ngaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5LqqfX8JW2Pp+09VlR66PnDlcce\nOVa2nv+/r1hSeez8LWVr4o9cN1Q0/tCyU5XH9vWX9bnuOzRYfexIUWmZFt0AuhXhB5Ii/EBSFwy/\n7YdsH7S9+azb5theZ3tn++fszk4TQN0uZsv/NUnLJ9y2WtL6iLha0vr2dQA95ILhj4inJR2ZcPNK\nSQ+3Lz8s6faa5wWgw6q+558bEfval/dLmnuuO9KiG+hOxQf8IiJ0nk/gadENdKeq4T9g+22S1P55\nsL4pAWhC1fA/LmlV+/IqSd+uZzoAmnIxH/U9Iul/JC2yvcf2RyT9q6Q/t71T0i3t6wB6yAXP7Y+I\nO8/xV8tqnguABnGGH5AU4QeS6qklvWPVu2Tr4E1lZyBftmlq5bFzNk48R+r/Z9rhmZXHvnF99WWt\nknRi8S+Lxrtgbav3V3/MJWnwrepttj1eVLpo+XlT2PIDSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKE\nH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUj21nr+k7fGp2dXXdkvSvvdW/9rxvpGyrywfmVF9\n7KnLCxemv1n2fQD9b1XfvgwcL/s/6xurPtani0r3BLb8QFKEH0iK8ANJVW3R/Vnb221vsv0t27M6\nO00AdavaonudpGsj4jpJOyTdW/O8AHRYpRbdEfFURJw5lvqMpPkdmBuADqrjPf+HJX23hn8HQIOK\nwm/7Pkljktac5z4ftb3B9obxk8dLygGoUeXw275b0gpJH4qIc55+ExEPRMTSiFjaP1R2sguA+lQ6\nw8/2ckmfkvQnEXGi3ikBaELVFt3/JulSSetsv2j7yx2eJ4CaVW3R/ZUOzAVAgzjDD0iK8ANJNb+k\nt6R1ccHYkuXAkhQFL5NjQ2W1S9pFTzvQX1a7cGlryfhJXVbbAy22S7HlB5Ii/EBShB9IivADSRF+\nICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaR8ni/erb+YfUjSz85zl8slvdHQ\ndKhN7d/G2ldFxO9czB0bDf+F2N4QEUupTW1qdx67/UBShB9IqtvC/wC1qU3tZnTVe34Azem2LT+A\nhnRF+G0vt/1T27tsr26w7pW2f2h7q+0ttu9pqvZZc+i3/YLtJxquO8v2WtvbbW+zfVODtT/Rfrw3\n237E9rQO13vI9kHbm8+6bY7tdbZ3tn/ObrD2Z9uP+ybb37I9qxO1L2TSw2+7X9IXJX1A0mJJd9pe\n3FD5MUmfjIjFkm6U9PcN1j7jHknbGq4pSV+Q9GREXCPp+qbmYHuepI9JWhoR10rql3RHh8t+TdLy\nCbetlrQ+Iq6WtL59vana6yRdGxHXSdoh6d4O1T6vSQ+/pBsk7YqI3RExIulRSSubKBwR+yJiY/vy\nMbUCMK+J2pJke76kD0p6sKma7bozJb1P7Z6LETESEW82OIUBSUO2ByQNS9rbyWIR8bSkIxNuXinp\n4fblhyXd3lTtiHgqIsbaV5+RNL8TtS+kG8I/T9JrZ13fowYDeIbtBZKWSHq2wbKfV6vVedO9aRZK\nOiTpq+23HA/ant5E4Yh4XdLnJL0qaZ+koxHxVBO1J5gbEfval/dLmjsJc5CkD0v67mQU7obwTzrb\nl0j6pqSPR8QvGqq5QtLBiHi+iXoTDEh6l6QvRcQSScfVud3eX9F+b71SrRegKyRNt31XE7XPJVof\neTX+sZft+9R667mm6dpSd4T/dUlXnnV9fvu2RtgeVCv4ayLisabqSnqPpNtsv6LWW5332/56Q7X3\nSNoTEWf2ctaq9WLQhFskvRwRhyJiVNJjkm5uqPbZDth+myS1fx5ssrjtuyWtkPShmKTP27sh/M9J\nutr2QttT1Dr483gThW1brfe92yLi/iZqnhER90bE/IhYoNbv/IOIaGQLGBH7Jb1me1H7pmWStjZR\nW63d/RttD7cf/2WanAOej0ta1b68StK3mypse7lab/dui4gTTdX9NREx6X8k3arWUc+XJN3XYN0/\nVmt3b5OkF9t/bp2E3/9PJT3RcM0/krSh/bv/l6TZDdb+jKTtkjZL+k9JUztc7xG1ji+MqrXX8xFJ\nl6l1lH+npO9LmtNg7V1qHec685z7ctPPuYjgDD8gq27Y7QcwCQg/kBThB5Ii/EBShB9IivADSRF+\nICnCDyT1f5TcD5S4U8riAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x137eb6080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEqFJREFUeJzt3XtwXOV5BvDn2dXKtmyDLWyMYzsIJ9wMKabRuE3iKWku\nlNB0DJ0G4k46bieDMy1Jk0JnSukfpf+0JJOQkpk2M6a4MYUQMgUGz9Rpi92Ey5SboNzNxSYG7NjY\nYGzLFySt9u0fWmcU0Hk/obO7Z6X3+c14LO275+ynlR6d1b7nOx/NDCIST6noAYhIMRR+kaAUfpGg\nFH6RoBR+kaAUfpGgFH6RoBR+kaAUfpGgOlr6YDNmWuXE7lY+pEgoQwf3o3rsCMdz31zhJ3kRgBsB\nlAH8i5ld792/cmI3PvSlq/I8pIg4tt96w7jvO+GX/STLAP4JwOcALAOwmuSyie5PRForz9/8KwBs\nM7NXzGwQwI8ArGrMsESk2fKEfxGA10d9vrN+268guZZkH8m+4aNHcjyciDRS09/tN7N1ZtZrZr3l\nrpnNfjgRGac84d8FYMmozxfXbxORSSBP+B8DcDrJ00h2AvgigI2NGZaINNuEW31mViX5VQD/hZFW\n33oze65hIxORpsrV5zezTQA2NWgsItJCOr1XJCiFXyQohV8kKIVfJCiFXyQohV8kqJbO55fmYJ5F\nlybzgk3jmrU+Nsux7VShI79IUAq/SFAKv0hQCr9IUAq/SFAKv0hQavW1QK5WHJBsx7HWpG3Hs71N\n/IszJvptiUNTql1nzvbMue+pQEd+kaAUfpGgFH6RoBR+kaAUfpGgFH6RoBR+kaDU52+AZB8/0Usv\nDSf2P+w/wHlrns2s/esHH3C3PVobdOsHE/X7ji1x63+95QuZtfmPlt1tmXheahW/PtyZXbMOv5Ff\n84eW+xyEdqAjv0hQCr9IUAq/SFAKv0hQCr9IUAq/SFAKv0hQufr8JHcA6AcwDKBqZr2NGFQRkr16\np57qR5eq/s5LfisdHQP+9ve//OHM2q/tWexuu3zBLre+ePoBtz6vctitf/2C/86s3Vi60N126Y/9\nJ3boBL8ZP9SVfWwbmuFuCkzzG/WWSk7qPIA2OOw24iSf3zazNxuwHxFpoTb4/SMiRcgbfgOwmeTj\nJNc2YkAi0hp5X/avNLNdJE8GcC/JF8zs/tF3qP9SWAsAldlzcz6ciDRKriO/me2q/78XwN0AVoxx\nn3Vm1mtmveWumXkeTkQaaMLhJzmT5OzjHwO4EED29DIRaSt5XvYvAHA3Ry6/3AHgh2b2nw0ZlYg0\n3YTDb2avADivgWMpVur69E7LuTTkb1we8PddOeZvXznsXxBg6brsWsch/7HfOjjHre9ZtNStv3qR\n3zD/6KdeyKx17vP79NN2v+3WywP+n5Gc60zoT73oTczHH07cwRLXGvDOK2nVtQDU6hMJSuEXCUrh\nFwlK4RcJSuEXCUrhFwkqzKW7U1N2U0tVe9Nyy6kpue/4D97Z709d7Tww5O//7aOZNR70p9xav18f\n+Mgpbr27d69b3z/QlVmb85K7KXig3613lPxjV62SXa9V/F7ccCVxae8O/3vKWqIV2AaH3TYYgogU\nQeEXCUrhFwlK4RcJSuEXCUrhFwlK4RcJasr0+fNcehtI9/n9Kb3+tqk+f8cRv89f7n/HrbM/u89v\nA/584q3fPMutn3P26259VsXf/wt3ZO//A5uyp/sCgHUkfjwH/Se+NJD9vJaG/H2XEsuiczKswZ2g\nI79IUAq/SFAKv0hQCr9IUAq/SFAKv0hQCr9IUFOmz5+U9zwAp57qCaeW6D68yLvENLD3T9wyLl72\nc/8OjuUdD7v1Simx/njCI2dlLx++cIl/rYDSmwfdujHRa/fKOdv0yTb/JDgNQEd+kaAUfpGgFH6R\noBR+kaAUfpGgFH6RoBR+kaCSfX6S6wF8HsBeMzu3fls3gDsA9ADYAeAyM/PXU57snFZ96loA/Yv8\npajX/vlGt75zsNutD1n2/k8sH3O3ffTtHrd+Qqe//W+c6J9jcMXK+zJr/7H0HHfbad/2zwOYvjux\nJkHZuW5/R+q6+jnOIUDrltnOYzxH/h8AuOhdt10DYIuZnQ5gS/1zEZlEkuE3s/sB7H/XzasAbKh/\nvAHAJQ0el4g02UT/5l9gZrvrH+8BsKBB4xGRFsn9hp+ZGZy/iEmuJdlHsm/46JG8DyciDTLR8L9B\nciEA1P/PXK3RzNaZWa+Z9Za7Zk7w4USk0SYa/o0A1tQ/XgPgnsYMR0RaJRl+krcDeAjAmSR3kvwy\ngOsBfJbkywA+U/9cRCaRZJ/fzFZnlD7d4LE0V9751zn6/H/wZ//j1ofM/zaUEw8w3Vk44KYHL3C3\n/fAPB936oen+OQoP/OEZbv13P/JMZu2j8/w1Aar/8Au3ft9dv+7WT3k0e02BWuInv+Z/2bm1w3kA\nOsNPJCiFXyQohV8kKIVfJCiFXyQohV8kqClz6e5U6yS5hHeCtz1r/s7v/sdPufWD2Ve3BgAMza+6\n9Xn/m/1tPP3F7OW7gfTYO/b5S3Cf/ff+8uEvnZy9RPf2P/WPPb+3LLtNCAArL/0/t/7Ua+dl1pJX\nJG+DVlyz6cgvEpTCLxKUwi8SlMIvEpTCLxKUwi8SlMIvEtSU6fPnlmMJbyZ6xnNe8i9/Pe9Rv1fO\nIb/PbxXn29jh/37nMX9KLw/75wlYv3/57PLO3Zm1Ex72p+TWzvab7Stm+5cNf6Dn/MzanO2Jedg5\nzwuZDHTkFwlK4RcJSuEXCUrhFwlK4RcJSuEXCUrhFwlKff7jEn1db/53ak58UupXsLPUNADQnMdP\n9PFx4JBbrh1M1N/xz1EAnV594mmbVvLPb6jQr7vXcEg8dvL6Dzm3b4fTCHTkFwlK4RcJSuEXCUrh\nFwlK4RcJSuEXCUrhFwkq2ecnuR7A5wHsNbNz67ddB+AKAPvqd7vWzDY1a5CtkOzV51ii2+11A7By\nYj3oxP5Zy74DB7OX7wYAG/Cvy19LbJ/Czs7M2kC3v22qz//EkR63PvtV53lJfr+n/oX7x3Pk/wGA\ni8a4/btmtrz+b1IHXySiZPjN7H4A+1swFhFpoTx/83+N5NMk15Oc27ARiUhLTDT83wewFMByALsB\nfCfrjiTXkuwj2Td89MgEH05EGm1C4TezN8xs2MxqAG4CsMK57zoz6zWz3nLXzImOU0QabELhJ7lw\n1KeXAni2McMRkVYZT6vvdgCfBDCP5E4AfwvgkySXY6QBtgPAV5o4RhFpgmT4zWz1GDff3ISxNFfO\n+dfenHlLvH6qVRLz8af734ZSOdFzHnIuNjCUOIdg2jT/sWcmTjJwzjEAAJ66KLM2cKa/nkFXyb8W\nwbYj8936jLeyn5fB2cWe3+b9vLXqFAOd4ScSlMIvEpTCLxKUwi8SlMIvEpTCLxJUmEt3J1t5qWm5\nXmum5PdmahW/zlqiHZfYP516ybusNwCaf9ZlqSPxIzLsr0++/fJ5mbXfP+chd9tKYkrv4w+e6dYX\nw9k+Nc06dVhMtOMmw4xgHflFglL4RYJS+EWCUvhFglL4RYJS+EWCUvhFgpoyff68Syofm+83Zqe9\nnV2rHE2dRJDqKeere0t4W6f/LWY1cYJD1e/j7/vMqW791JWv+ft3rN/6cX/fP/EvO17tyj5/In8f\nfxI08hN05BcJSuEXCUrhFwlK4RcJSuEXCUrhFwlK4RcJasr0+VNS5wF0ffxNt755+YbM2gXfutrd\ntvsFv1deGkpc/jrRiy85l+7moD8nHsP+vgdOy56PDwCly/e59cUzD2TW3hyc5W7b83f+8uDVuf6P\nr3cdhdQlFCxRTx42J8F8fx35RYJS+EWCUvhFglL4RYJS+EWCUvhFglL4RYJK9vlJLgFwC4AFGJkV\nv87MbiTZDeAOAD0AdgC4zMycWe/t7cDzJ7n1R846IbN2y1U3uNt+4da/cOun3ePPS+eA3+/mQHYv\nnwP+Mtf9553i1k/+y1fc+hy3Cuw8kn2PI/+cvXw3AMyadtitD8/wm/HDndnNdK8GALVEMpLXA5gE\nxvMlVAFcbWbLAPwmgCtJLgNwDYAtZnY6gC31z0VkkkiG38x2m9kT9Y/7AWwFsAjAKgDHT3vbAOCS\nZg1SRBrvfb14IdkD4HwAjwBYYGa766U9GPmzQEQmiXGHn+QsAHcC+IaZHRpdMzNDxlXySK4l2Uey\nb/jokVyDFZHGGVf4SVYwEvzbzOyu+s1vkFxYry8EsHesbc1snZn1mllvuctfFFJEWicZfpIEcDOA\nrWY2+m3tjQDW1D9eA+Cexg9PRJqFllrCmVwJ4AEAzwA4Pv/zWoz83f9jAB8E8CpGWn37vX3NOGWJ\nfehLV+Ud89jjTF09259Vi/KAv4NDS7Nrt132PXfbVwZPduvffOF33Prwfd1uffbr2V9c/xK/Hdaz\nym/lVWv+8eGlX/hv9Sy8szOzNmvbQXfboZO63PrA3IpbH5yVPfbqDHfT5LLqqSnBqcNqs6b0br/1\nBhzb8/q49p7s85vZg8ienfzp9zMwEWkfU+BUBRGZCIVfJCiFXyQohV8kKIVfJCiFXySoKXPp7mTf\nNPFrrtbh72DOi9nnAVz+kyvdbVd/7GG3/tUzfubWX+vxpxvvGciebnxwyG9oHxqc7tZ//tMet37G\nv/uXPLfO7OnKg/P8Mz4H5/g/nl4fHwCqzpc2Wfv4jaQjv0hQCr9IUAq/SFAKv0hQCr9IUAq/SFAK\nv0hQU6bPn5Tou6Yu1Vydnr2DeX1+U3jzI59w67d/zL/YwHlnv+rWn9q2JLPW8ZY/533pXf6l1XoO\nu5doQHWOfx7B4InZ8/mHZvvP21CX/03z+vgAYM65GxH6+Ck68osEpfCLBKXwiwSl8IsEpfCLBKXw\niwSl8IsEFabPn+rLMjXf32mXV0v+zktD/poA8x/ym857f3aaW1/srDlQqvrnELwz32+W1z7g9/FT\nS11Xpzm99uxTAEb2nZhzb4levVtP/DxMhT5+io78IkEp/CJBKfwiQSn8IkEp/CJBKfwiQSn8IkEl\n+/wklwC4BcACAAZgnZndSPI6AFcA2Fe/67VmtqlZA222POcB1FI948R5AN45BABQ9ZepB50BsOZv\nm+xnJ7+2VD17B+lt/XpqbF49Qh8/ZTwn+VQBXG1mT5CcDeBxkvfWa981s283b3gi0izJ8JvZbgC7\n6x/3k9wKYFGzByYizfW+/uYn2QPgfACP1G/6GsmnSa4nOTdjm7Uk+0j2DR/1LxklIq0z7vCTnAXg\nTgDfMLNDAL4PYCmA5Rh5ZfCdsbYzs3Vm1mtmveUuf202EWmdcYWfZAUjwb/NzO4CADN7w8yGzawG\n4CYAK5o3TBFptGT4SRLAzQC2mtkNo25fOOpulwJ4tvHDE5FmGc+7/Z8A8EcAniH5ZP22awGsJrkc\nI+2/HQC+0pQRtgm3NdTkdhnK/g7ozxhuqlytwpztNrXr8hnPu/0PYuxv06Tt6YuIzvATCUvhFwlK\n4RcJSuEXCUrhFwlK4RcJKsylu4uUd9pscv/5NpegdOQXCUrhFwlK4RcJSuEXCUrhFwlK4RcJSuEX\nCYpmresSk9wH4NVRN80D8GbLBvD+tOvY2nVcgMY2UY0c26lmNn88d2xp+N/z4GSfmfUWNgBHu46t\nXccFaGwTVdTY9LJfJCiFXySoosO/ruDH97Tr2Np1XIDGNlGFjK3Qv/lFpDhFH/lFpCCFhJ/kRSRf\nJLmN5DVFjCELyR0knyH5JMm+gseynuReks+Ouq2b5L0kX67/P+YyaQWN7TqSu+rP3ZMkLy5obEtI\n/pTk8ySfI/n1+u2FPnfOuAp53lr+sp9kGcBLAD4LYCeAxwCsNrPnWzqQDCR3AOg1s8J7wiR/C8Bh\nALeY2bn1274FYL+ZXV//xTnXzP6qTcZ2HYDDRa/cXF9QZuHolaUBXALgj1Hgc+eM6zIU8LwVceRf\nAWCbmb1iZoMAfgRgVQHjaHtmdj+A/e+6eRWADfWPN2Dkh6flMsbWFsxst5k9Uf+4H8DxlaULfe6c\ncRWiiPAvAvD6qM93or2W/DYAm0k+TnJt0YMZw4L6sukAsAfAgiIHM4bkys2t9K6VpdvmuZvIiteN\npjf83mulmS0H8DkAV9Zf3rYlG/mbrZ3aNeNaublVxlhZ+peKfO4muuJ1oxUR/l0Aloz6fHH9trZg\nZrvq/+8FcDfab/XhN44vklr/f2/B4/mldlq5eayVpdEGz107rXhdRPgfA3A6ydNIdgL4IoCNBYzj\nPUjOrL8RA5IzAVyI9lt9eCOANfWP1wC4p8Cx/Ip2Wbk5a2VpFPzctd2K12bW8n8ALsbIO/7bAfxN\nEWPIGNdSAE/V/z1X9NgA3I6Rl4FDGHlv5MsATgKwBcDLADYD6G6jsf0bgGcAPI2RoC0saGwrMfKS\n/mkAT9b/XVz0c+eMq5DnTWf4iQSlN/xEglL4RYJS+EWCUvhFglL4RYJS+EWCUvhFglL4RYL6f7EX\nqH7gijfLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11c267b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "n = random.randint(0, 19) # which image\n",
    "m = random.randint(0, 83) #which layer\n",
    "\n",
    "plt.imshow(hidden_output[n,:,:,m])\n",
    "plt.show()\n",
    "\n",
    "plt.imshow(train_dataset[n].reshape(28, 28))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hope this was useful, and helped you better understand how to implement convolution neural networks. Please let me know if you have any comments or ways of improving the notebook."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:custom_tensorflow]",
   "language": "python",
   "name": "conda-env-custom_tensorflow-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}