notebooks/lenet_in_tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Convolutional Neural Network in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we convert our [LeNet-5-inspired, MNIST-classifying, deep convolutional network](https://github.com/the-deep-learners/TensorFlow-LiveLessons/blob/master/notebooks/lenet_in_keras.ipynb) from Keras to TensorFlow (compare them side by side) following Aymeric Damien's [style](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/multilayer_perceptron.ipynb)."
   ]
  },

  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.random.seed(42)\n",
    "import tensorflow as tf\n",
    "tf.set_random_seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set neural network hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "epochs = 20\n",
    "batch_size = 128\n",
    "display_progress = 40 # after this many batches, output progress to screen\n",
    "wt_init = tf.contrib.layers.xavier_initializer() # weight initializer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set parameters for each layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# input layer: \n",
    "n_input = 784\n",
    "\n",
    "# first convolutional layer: \n",
    "n_conv_1 = 32\n",
    "k_conv_1 = 3 # k_size\n",
    "\n",
    "# second convolutional layer: \n",
    "n_conv_2 = 64\n",
    "k_conv_2 = 3\n",
    "\n",
    "# max pooling layer:\n",
    "pool_size = 2\n",
    "mp_layer_dropout = 0.25\n",
    "\n",
    "# dense layer: \n",
    "n_dense = 128\n",
    "dense_layer_dropout = 0.5\n",
    "\n",
    "# output layer: \n",
    "n_classes = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define placeholder Tensors for inputs and labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, [None, n_input])\n",
    "y = tf.placeholder(tf.float32, [None, n_classes])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define types of layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# dense layer with ReLU activation:\n",
    "def dense(x, W, b):\n",
    "    z = tf.add(tf.matmul(x, W), b)\n",
    "    a = tf.nn.relu(z)\n",
    "    return a\n",
    "\n",
    "# convolutional layer with ReLU activation:\n",
    "def conv2d(x, W, b, stride_length=1):\n",
    "    xW = tf.nn.conv2d(x, W, strides=[1, stride_length, stride_length, 1], padding='SAME')\n",
    "    z = tf.nn.bias_add(xW, b)\n",
    "    a = tf.nn.relu(z)\n",
    "    return a\n",
    "\n",
    "# max-pooling layer: \n",
    "def maxpooling2d(x, p_size):\n",
    "    return tf.nn.max_pool(x, \n",
    "                          ksize=[1, p_size, p_size, 1], \n",
    "                          strides=[1, p_size, p_size, 1], \n",
    "                          padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Design neural network architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def network(x, weights, biases, n_in, mp_psize, mp_dropout, dense_dropout):\n",
    "\n",
    "    # reshape linear MNIST pixel input into square image: \n",
    "    square_dimensions = int(np.sqrt(n_in))\n",
    "    square_x = tf.reshape(x, shape=[-1, square_dimensions, square_dimensions, 1])\n",
    "    \n",
    "    # convolutional and max-pooling layers:\n",
    "    conv_1 = conv2d(square_x, weights['W_c1'], biases['b_c1'])\n",
    "    conv_2 = conv2d(conv_1, weights['W_c2'], biases['b_c2'])\n",
    "    pool_1 = maxpooling2d(conv_2, mp_psize)\n",
    "    pool_1 = tf.nn.dropout(pool_1, 1-mp_dropout)\n",
    "    \n",
    "    # dense layer: \n",
    "    flat = tf.reshape(pool_1, [-1, weights['W_d1'].get_shape().as_list()[0]])\n",
    "    dense_1 = dense(flat, weights['W_d1'], biases['b_d1'])\n",
    "    dense_1 = tf.nn.dropout(dense_1, 1-dense_dropout)\n",
    "    \n",
    "    # output layer: \n",
    "    out_layer_z = tf.add(tf.matmul(dense_1, weights['W_out']), biases['b_out'])\n",
    "    \n",
    "    return out_layer_z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define dictionaries for storing weights and biases for each layer -- and initialize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bias_dict = {\n",
    "    'b_c1': tf.Variable(tf.zeros([n_conv_1])),\n",
    "    'b_c2': tf.Variable(tf.zeros([n_conv_2])),\n",
    "    'b_d1': tf.Variable(tf.zeros([n_dense])),\n",
    "    'b_out': tf.Variable(tf.zeros([n_classes]))\n",
    "}\n",
    "\n",
    "# calculate number of inputs to dense layer: \n",
    "full_square_length = np.sqrt(n_input)\n",
    "pooled_square_length = int(full_square_length / pool_size)\n",
    "dense_inputs = pooled_square_length**2 * n_conv_2\n",
    "\n",
    "weight_dict = {\n",
    "    'W_c1': tf.get_variable('W_c1', \n",
    "                            [k_conv_1, k_conv_1, 1, n_conv_1], initializer=wt_init),\n",
    "    'W_c2': tf.get_variable('W_c2', \n",
    "                            [k_conv_2, k_conv_2, n_conv_1, n_conv_2], initializer=wt_init),\n",
    "    'W_d1': tf.get_variable('W_d1', \n",
    "                            [dense_inputs, n_dense], initializer=wt_init),\n",
    "    'W_out': tf.get_variable('W_out', \n",
    "                             [n_dense, n_classes], initializer=wt_init)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predictions = network(x, weight_dict, bias_dict, n_input, \n",
    "                      pool_size, mp_layer_dropout, dense_layer_dropout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define model's loss and its optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=predictions, labels=y))\n",
    "optimizer = tf.train.AdamOptimizer().minimize(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define evaluation metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(predictions, 1), tf.argmax(y, 1))\n",
    "accuracy_pct = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create op for variable initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "initializer_op = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train the network in a session (identical to `intermediate_net_in_tensorflow.ipynb` except addition of `display_progress`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training for 20 epochs.\n",
      "Step 1 of 429 in epoch 1.\n",
      "Step 41 of 429 in epoch 1.\n",
      "Step 81 of 429 in epoch 1.\n",
      "Step 121 of 429 in epoch 1.\n",
      "Step 161 of 429 in epoch 1.\n",
      "Step 201 of 429 in epoch 1.\n",
      "Step 241 of 429 in epoch 1.\n",
      "Step 281 of 429 in epoch 1.\n",
      "Step 321 of 429 in epoch 1.\n",
      "Step 361 of 429 in epoch 1.\n",
      "Step 401 of 429 in epoch 1.\n",
      "Epoch 001: cost = 0.266, accuracy = 91.86%\n",
      "Step 1 of 429 in epoch 2.\n",
      "Step 41 of 429 in epoch 2.\n",
      "Step 81 of 429 in epoch 2.\n",
      "Step 121 of 429 in epoch 2.\n",
      "Step 161 of 429 in epoch 2.\n",
      "Step 201 of 429 in epoch 2.\n",
      "Step 241 of 429 in epoch 2.\n",
      "Step 281 of 429 in epoch 2.\n",
      "Step 321 of 429 in epoch 2.\n",
      "Step 361 of 429 in epoch 2.\n",
      "Step 401 of 429 in epoch 2.\n",
      "Epoch 002: cost = 0.095, accuracy = 97.19%\n",
      "Step 1 of 429 in epoch 3.\n",
      "Step 41 of 429 in epoch 3.\n",
      "Step 81 of 429 in epoch 3.\n",
      "Step 121 of 429 in epoch 3.\n",
      "Step 161 of 429 in epoch 3.\n",
      "Step 201 of 429 in epoch 3.\n",
      "Step 241 of 429 in epoch 3.\n",
      "Step 281 of 429 in epoch 3.\n",
      "Step 321 of 429 in epoch 3.\n",
      "Step 361 of 429 in epoch 3.\n",
      "Step 401 of 429 in epoch 3.\n",
      "Epoch 003: cost = 0.067, accuracy = 98.01%\n",
      "Step 1 of 429 in epoch 4.\n",
      "Step 41 of 429 in epoch 4.\n",
      "Step 81 of 429 in epoch 4.\n",
      "Step 121 of 429 in epoch 4.\n",
      "Step 161 of 429 in epoch 4.\n",
      "Step 201 of 429 in epoch 4.\n",
      "Step 241 of 429 in epoch 4.\n",
      "Step 281 of 429 in epoch 4.\n",
      "Step 321 of 429 in epoch 4.\n",
      "Step 361 of 429 in epoch 4.\n",
      "Step 401 of 429 in epoch 4.\n",
      "Epoch 004: cost = 0.055, accuracy = 98.30%\n",
      "Step 1 of 429 in epoch 5.\n",
      "Step 41 of 429 in epoch 5.\n",
      "Step 81 of 429 in epoch 5.\n",
      "Step 121 of 429 in epoch 5.\n",
      "Step 161 of 429 in epoch 5.\n",
      "Step 201 of 429 in epoch 5.\n",
      "Step 241 of 429 in epoch 5.\n",
      "Step 281 of 429 in epoch 5.\n",
      "Step 321 of 429 in epoch 5.\n",
      "Step 361 of 429 in epoch 5.\n",
      "Step 401 of 429 in epoch 5.\n",
      "Epoch 005: cost = 0.048, accuracy = 98.51%\n",
      "Step 1 of 429 in epoch 6.\n",
      "Step 41 of 429 in epoch 6.\n",
      "Step 81 of 429 in epoch 6.\n",
      "Step 121 of 429 in epoch 6.\n",
      "Step 161 of 429 in epoch 6.\n",
      "Step 201 of 429 in epoch 6.\n",
      "Step 241 of 429 in epoch 6.\n",
      "Step 281 of 429 in epoch 6.\n",
      "Step 321 of 429 in epoch 6.\n",
      "Step 361 of 429 in epoch 6.\n",
      "Step 401 of 429 in epoch 6.\n",
      "Epoch 006: cost = 0.041, accuracy = 98.71%\n",
      "Step 1 of 429 in epoch 7.\n",
      "Step 41 of 429 in epoch 7.\n",
      "Step 81 of 429 in epoch 7.\n",
      "Step 121 of 429 in epoch 7.\n",
      "Step 161 of 429 in epoch 7.\n",
      "Step 201 of 429 in epoch 7.\n",
      "Step 241 of 429 in epoch 7.\n",
      "Step 281 of 429 in epoch 7.\n",
      "Step 321 of 429 in epoch 7.\n",
      "Step 361 of 429 in epoch 7.\n",
      "Step 401 of 429 in epoch 7.\n",
      "Epoch 007: cost = 0.037, accuracy = 98.86%\n",
      "Step 1 of 429 in epoch 8.\n",
      "Step 41 of 429 in epoch 8.\n",
      "Step 81 of 429 in epoch 8.\n",
      "Step 121 of 429 in epoch 8.\n",
      "Step 161 of 429 in epoch 8.\n",
      "Step 201 of 429 in epoch 8.\n",
      "Step 241 of 429 in epoch 8.\n",
      "Step 281 of 429 in epoch 8.\n",
      "Step 321 of 429 in epoch 8.\n",
      "Step 361 of 429 in epoch 8.\n",
      "Step 401 of 429 in epoch 8.\n",
      "Epoch 008: cost = 0.031, accuracy = 99.00%\n",
      "Step 1 of 429 in epoch 9.\n",
      "Step 41 of 429 in epoch 9.\n",
      "Step 81 of 429 in epoch 9.\n",
      "Step 121 of 429 in epoch 9.\n",
      "Step 161 of 429 in epoch 9.\n",
      "Step 201 of 429 in epoch 9.\n",
      "Step 241 of 429 in epoch 9.\n",
      "Step 281 of 429 in epoch 9.\n",
      "Step 321 of 429 in epoch 9.\n",
      "Step 361 of 429 in epoch 9.\n",
      "Step 401 of 429 in epoch 9.\n",
      "Epoch 009: cost = 0.029, accuracy = 99.06%\n",
      "Step 1 of 429 in epoch 10.\n",
      "Step 41 of 429 in epoch 10.\n",
      "Step 81 of 429 in epoch 10.\n",
      "Step 121 of 429 in epoch 10.\n",
      "Step 161 of 429 in epoch 10.\n",
      "Step 201 of 429 in epoch 10.\n",
      "Step 241 of 429 in epoch 10.\n",
      "Step 281 of 429 in epoch 10.\n",
      "Step 321 of 429 in epoch 10.\n",
      "Step 361 of 429 in epoch 10.\n",
      "Step 401 of 429 in epoch 10.\n",
      "Epoch 010: cost = 0.025, accuracy = 99.20%\n",
      "Step 1 of 429 in epoch 11.\n",
      "Step 41 of 429 in epoch 11.\n",
      "Step 81 of 429 in epoch 11.\n",
      "Step 121 of 429 in epoch 11.\n",
      "Step 161 of 429 in epoch 11.\n",
      "Step 201 of 429 in epoch 11.\n",
      "Step 241 of 429 in epoch 11.\n",
      "Step 281 of 429 in epoch 11.\n",
      "Step 321 of 429 in epoch 11.\n",
      "Step 361 of 429 in epoch 11.\n",
      "Step 401 of 429 in epoch 11.\n",
      "Epoch 011: cost = 0.023, accuracy = 99.23%\n",
      "Step 1 of 429 in epoch 12.\n",
      "Step 41 of 429 in epoch 12.\n",
      "Step 81 of 429 in epoch 12.\n",
      "Step 121 of 429 in epoch 12.\n",
      "Step 161 of 429 in epoch 12.\n",
      "Step 201 of 429 in epoch 12.\n",
      "Step 241 of 429 in epoch 12.\n",
      "Step 281 of 429 in epoch 12.\n",
      "Step 321 of 429 in epoch 12.\n",
      "Step 361 of 429 in epoch 12.\n",
      "Step 401 of 429 in epoch 12.\n",
      "Epoch 012: cost = 0.022, accuracy = 99.28%\n",
      "Step 1 of 429 in epoch 13.\n",
      "Step 41 of 429 in epoch 13.\n",
      "Step 81 of 429 in epoch 13.\n",
      "Step 121 of 429 in epoch 13.\n",
      "Step 161 of 429 in epoch 13.\n",
      "Step 201 of 429 in epoch 13.\n",
      "Step 241 of 429 in epoch 13.\n",
      "Step 281 of 429 in epoch 13.\n",
      "Step 321 of 429 in epoch 13.\n",
      "Step 361 of 429 in epoch 13.\n",
      "Step 401 of 429 in epoch 13.\n",
      "Epoch 013: cost = 0.021, accuracy = 99.34%\n",
      "Step 1 of 429 in epoch 14.\n",
      "Step 41 of 429 in epoch 14.\n",
      "Step 81 of 429 in epoch 14.\n",
      "Step 121 of 429 in epoch 14.\n",
      "Step 161 of 429 in epoch 14.\n",
      "Step 201 of 429 in epoch 14.\n",
      "Step 241 of 429 in epoch 14.\n",
      "Step 281 of 429 in epoch 14.\n",
      "Step 321 of 429 in epoch 14.\n",
      "Step 361 of 429 in epoch 14.\n",
      "Step 401 of 429 in epoch 14.\n",
      "Epoch 014: cost = 0.019, accuracy = 99.37%\n",
      "Step 1 of 429 in epoch 15.\n",
      "Step 41 of 429 in epoch 15.\n",
      "Step 81 of 429 in epoch 15.\n",
      "Step 121 of 429 in epoch 15.\n",
      "Step 161 of 429 in epoch 15.\n",
      "Step 201 of 429 in epoch 15.\n",
      "Step 241 of 429 in epoch 15.\n",
      "Step 281 of 429 in epoch 15.\n",
      "Step 321 of 429 in epoch 15.\n",
      "Step 361 of 429 in epoch 15.\n",
      "Step 401 of 429 in epoch 15.\n",
      "Epoch 015: cost = 0.018, accuracy = 99.37%\n",
      "Step 1 of 429 in epoch 16.\n",
      "Step 41 of 429 in epoch 16.\n",
      "Step 81 of 429 in epoch 16.\n",
      "Step 121 of 429 in epoch 16.\n",
      "Step 161 of 429 in epoch 16.\n",
      "Step 201 of 429 in epoch 16.\n",
      "Step 241 of 429 in epoch 16.\n",
      "Step 281 of 429 in epoch 16.\n",
      "Step 321 of 429 in epoch 16.\n",
      "Step 361 of 429 in epoch 16.\n",
      "Step 401 of 429 in epoch 16.\n",
      "Epoch 016: cost = 0.016, accuracy = 99.43%\n",
      "Step 1 of 429 in epoch 17.\n",
      "Step 41 of 429 in epoch 17.\n",
      "Step 81 of 429 in epoch 17.\n",
      "Step 121 of 429 in epoch 17.\n",
      "Step 161 of 429 in epoch 17.\n",
      "Step 201 of 429 in epoch 17.\n",
      "Step 241 of 429 in epoch 17.\n",
      "Step 281 of 429 in epoch 17.\n",
      "Step 321 of 429 in epoch 17.\n",
      "Step 361 of 429 in epoch 17.\n",
      "Step 401 of 429 in epoch 17.\n",
      "Epoch 017: cost = 0.016, accuracy = 99.42%\n",
      "Step 1 of 429 in epoch 18.\n",
      "Step 41 of 429 in epoch 18.\n",
      "Step 81 of 429 in epoch 18.\n",
      "Step 121 of 429 in epoch 18.\n",
      "Step 161 of 429 in epoch 18.\n",
      "Step 201 of 429 in epoch 18.\n",
      "Step 241 of 429 in epoch 18.\n",
      "Step 281 of 429 in epoch 18.\n",
      "Step 321 of 429 in epoch 18.\n",
      "Step 361 of 429 in epoch 18.\n",
      "Step 401 of 429 in epoch 18.\n",
      "Epoch 018: cost = 0.014, accuracy = 99.52%\n",
      "Step 1 of 429 in epoch 19.\n",
      "Step 41 of 429 in epoch 19.\n",
      "Step 81 of 429 in epoch 19.\n",
      "Step 121 of 429 in epoch 19.\n",
      "Step 161 of 429 in epoch 19.\n",
      "Step 201 of 429 in epoch 19.\n",
      "Step 241 of 429 in epoch 19.\n",
      "Step 281 of 429 in epoch 19.\n",
      "Step 321 of 429 in epoch 19.\n",
      "Step 361 of 429 in epoch 19.\n",
      "Step 401 of 429 in epoch 19.\n",
      "Epoch 019: cost = 0.015, accuracy = 99.47%\n",
      "Step 1 of 429 in epoch 20.\n",
      "Step 41 of 429 in epoch 20.\n",
      "Step 81 of 429 in epoch 20.\n",
      "Step 121 of 429 in epoch 20.\n",
      "Step 161 of 429 in epoch 20.\n",
      "Step 201 of 429 in epoch 20.\n",
      "Step 241 of 429 in epoch 20.\n",
      "Step 281 of 429 in epoch 20.\n",
      "Step 321 of 429 in epoch 20.\n",
      "Step 361 of 429 in epoch 20.\n",
      "Step 401 of 429 in epoch 20.\n",
      "Epoch 020: cost = 0.015, accuracy = 99.50%\n",
      "Training Complete. Testing Model.\n",
      "\n",
      "Test Cost: 0.059\n",
      "Test Accuracy: 98.62%\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    session.run(initializer_op)\n",
    "    \n",
    "    print(\"Training for\", epochs, \"epochs.\")\n",
    "    \n",
    "    # loop over epochs: \n",
    "    for epoch in range(epochs):\n",
    "        \n",
    "        avg_cost = 0.0 # track cost to monitor performance during training\n",
    "        avg_accuracy_pct = 0.0\n",
    "        \n",
    "        # loop over all batches of the epoch:\n",
    "        n_batches = int(mnist.train.num_examples / batch_size)\n",
    "        for i in range(n_batches):\n",
    "\n",
    "            # to reassure you something's happening! \n",
    "            if i % display_progress == 0:\n",
    "                print(\"Step \", i+1, \" of \", n_batches, \" in epoch \", epoch+1, \".\", sep='')\n",
    "            \n",
    "            batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
    "            \n",
    "            # feed batch data to run optimization and fetching cost and accuracy: \n",
    "            _, batch_cost, batch_acc = session.run([optimizer, cost, accuracy_pct], \n",
    "                                                   feed_dict={x: batch_x, y: batch_y})\n",
    "            \n",
    "            # accumulate mean loss and accuracy over epoch: \n",
    "            avg_cost += batch_cost / n_batches\n",
    "            avg_accuracy_pct += batch_acc / n_batches\n",
    "            \n",
    "        # output logs at end of each epoch of training:\n",
    "        print(\"Epoch \", '%03d' % (epoch+1), \n",
    "              \": cost = \", '{:.3f}'.format(avg_cost), \n",
    "              \", accuracy = \", '{:.2f}'.format(avg_accuracy_pct), \"%\", \n",
    "              sep='')\n",
    "    \n",
    "    print(\"Training Complete. Testing Model.\\n\")\n",
    "    \n",
    "    test_cost = cost.eval({x: mnist.test.images, y: mnist.test.labels})\n",
    "    test_accuracy_pct = accuracy_pct.eval({x: mnist.test.images, y: mnist.test.labels})\n",
    "    \n",
    "    print(\"Test Cost:\", '{:.3f}'.format(test_cost))\n",
    "    print(\"Test Accuracy: \", '{:.2f}'.format(test_accuracy_pct), \"%\", sep='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compare with LeNet Keras results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Increase dropout probability(/ies) or add dropout to other conv layer? Stop earlier? Coming up in Lecture 5 :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### As an exercise, try converting our [AlexNet](https://github.com/the-deep-learners/TensorFlow-LiveLessons/blob/master/notebooks/alexnet_in_keras.ipynb) from Keras to TensorFlow following the same style as this LeNet-5 notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}