Chapter03/ch3_nb2_build_and_train_first_cnn_with_tf2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p style=\"border: 1px solid #e7692c; border-left: 15px solid #e7692c; padding: 10px; text-align:justify;\">\n",
    "    <strong style=\"color: #e7692c\">Tip.</strong> <a style=\"color: #000000;\" href=\"https://nbviewer.jupyter.org/github/PacktPublishing/Hands-On-Computer-Vision-with-TensorFlow-2/blob/master/Chapter03/ch3_nb2_build_and_train_first_cnn_with_tf2.ipynb\" title=\"View with Jupyter Online\">Click here to view this notebook on <code>nbviewer.jupyter.org</code></a>. \n",
    "    <br/>These notebooks are better read there, as Github default viewer ignores some of the formatting and interactive content.\n",
    "    </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<table style=\"font-size: 1em; padding: 0; margin: 0;\">\n",
    "    <tr style=\"vertical-align: top; padding: 0; margin: 0;background-color: #ffffff\">\n",
    "        <td style=\"vertical-align: top; padding: 0; margin: 0; padding-right: 15px;\">\n",
    "    <p style=\"background: #363636; color:#ffffff; text-align:justify; padding: 10px 25px;\">\n",
    "        <strong style=\"font-size: 1.0em;\"><span style=\"font-size: 1.2em;\"><span style=\"color: #e7692c;\">Hands-on</span> Computer Vision with TensorFlow 2</span><br/>by <em>Eliot Andres</em> & <em>Benjamin Planche</em> (Packt Pub.)</strong><br/><br/>\n",
    "        <strong>> Chapter 3: Modern Neural Networks</strong><br/>\n",
    "    </p>\n",
    "\n",
    "<h1 style=\"width: 100%; text-align: left; padding: 0px 25px;\"><small style=\"color: #e7692c;\">Notebook 2:</small><br/>Building and Training our First CNN<br/>with TensorFlow 2 and Keras</h1>\n",
    "<br/>\n",
    "<p style=\"border-left: 15px solid #363636; text-align:justify; padding: 0 10px;\">\n",
    "    This notebook presents how to implement the <strong><em>LeNet-5</em> architecture</strong> as presented in the book, and applies it to <strong>hand-written digit recognition</strong> (on the MNIST dataset).\n",
    "<br/>\n",
    "<p style=\"border-left: 15px solid #e7692c; padding: 0 10px; text-align:justify;\">\n",
    "    <strong style=\"color: #e7692c;\">Tip.</strong> The notebooks shared on this git repository illustrate some notions from the book \"<em><strong>Hands-on Computer Vision with TensorFlow 2</strong></em>\" written by Eliot Andres and Benjamin Planche, published by Packt. If you enjoyed the insights shared here, <a href=\"https://www.amazon.com/Hands-Computer-Vision-TensorFlow-processing/dp/1788830644\" title=\"Learn more about the book!\"><strong>please consider acquiring the book!</strong></a>\n",
    "<br/><br/>\n",
    "The book provides further guidance for those eager to learn about computer vision and to harness the power of TensorFlow 2 and Keras to build efficient recognition systems for object detection, segmentation, video processing, smartphone applications, and more.</p>\n",
    "        </td>\n",
    "        <td style=\"vertical-align: top; padding: 0; margin: 0; width: 280px;\">\n",
    "    <a href=\"https://www.amazon.com/Hands-Computer-Vision-TensorFlow-processing/dp/1788830644\" title=\"Learn more about the book!\" target=\"_blank\">\n",
    "        <img src=\"../banner_images/book_cover.png\" width=280>\n",
    "    </a>\n",
    "    <p style=\"background: #e7692c; color:#ffffff; padding: 10px; text-align:justify;\"><strong>Leverage deep learning to create powerful image processing apps with TensorFlow 2 and Keras. <br/></strong>Get the book for more insights!</p>\n",
    "    <ul style=\"height: 32px; white-space: nowrap; text-align: center; margin: 0px; padding: 0px; padding-top: 10px;\">\n",
    "    <li style=\"display: block;height: 100%;float: left;vertical-align: middle;margin: 0 25px 10px;padding: 0px;\">\n",
    "        <a href=\"https://www.amazon.com/Hands-Computer-Vision-TensorFlow-processing/dp/1788830644\" title=\"Get the book on Amazon (paperback or Kindle version)!\" target=\"_blank\">\n",
    "        <img style=\"vertical-align: middle; max-width: 72px; max-height: 32px;\" src=\"../banner_images/logo_amazon.png\" width=\"75px\">\n",
    "        </a>\n",
    "    </li>\n",
    "    <li style=\"display: inline-block;height: 100%;vertical-align: middle;float: right;margin: -5px 25px 10px;padding: 0px;\">\n",
    "        <a href=\"https://www.packtpub.com/application-development/hands-computer-vision-tensorflow-2\" title=\"Get your Packt book (paperback, PDF, ePUB, or MOBI version)!\" target=\"_blank\">\n",
    "        <img style=\"vertical-align: middle; max-width: 72px; max-height: 32px;\" src=\"../banner_images/logo_packt.png\" width=\"75px\">\n",
    "        </a>\n",
    "    </li>\n",
    "    </ul>\n",
    "        </td>\n",
    "        </tr>\n",
    "        </table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As presented in [Chapter 2](../ch2), we use Tensorflow and Keras helpers to load the commonly-used [MNIST](http://yann.lecun.com/exdb/mnist) training and testing datasets[$^1$](#ref) (Yann LeCun and Corinna Cortes hold all copyrights for this dataset). We also normalize the images (setting the pixel values from `[0, 255]` to `[0, 1]` and reshape them properly (as Tensorflow stores them as column-vectors):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_classes = 10\n",
    "img_rows, img_cols, img_ch = 28, 28, 1\n",
    "input_shape = (img_rows, img_cols, img_ch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "\n",
    "x_train = x_train.reshape(x_train.shape[0], *input_shape)\n",
    "x_test = x_test.reshape(x_test.shape[0], *input_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building and Training LeNet-5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have demonstrated how CNNs can be implemented different ways depending on the _level of parametrization_ versus _succinctness_ one needs. In this case, we will use the Keras API to showcase once again how straightforward it makes implementing and using neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instantiating Convolutional Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous notebook, we presented how one can perform convolutions on images. However, in neural networks, we want the convolutions' filters to be ***trainable***, and we may want to add a ***bias*** to the results and apply an ***activation function***.\n",
    "\n",
    "We thus need to wrap the convolution operation into a `Layer` object, similar to how the fully-connected layer we implemented in Chapter 1 was built around the matrix operations.\n",
    "\n",
    "TensorFlow 2/Keras provides its own `tf.keras.Layer` class we can extend. We demonstrate below how a simple convolution layer can be dfined this way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleConvolutionLayer(tf.keras.layers.Layer):\n",
    "\n",
    "    def __init__(self, num_kernels=32, kernel_size=(3, 3), strides=(1, 1), use_bias=True):\n",
    "        \"\"\"\n",
    "        Initialize the layer.\n",
    "        :param num_kernels:     Number of kernels for the convolution\n",
    "        :param kernel_size:     Kernel size (H x W)\n",
    "        :param strides:         Vertical and horizontal stride as list\n",
    "        :param use_bias:        Flag to add a bias after covolution / before activation\n",
    "        \"\"\"\n",
    "        # First, we have to call the `Layer` super __init__(), as it initializes hidden mechanisms:\n",
    "        super().__init__()  \n",
    "        # Then we assign the parameters:\n",
    "        self.num_kernels = num_kernels\n",
    "        self.kernel_size = kernel_size\n",
    "        self.strides = strides\n",
    "        self.use_bias = use_bias\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        \"\"\"\n",
    "        Build the layer, initializing its parameters according to the input shape.\n",
    "        This function will be internally called the first time the layer is used, though\n",
    "        it can also be manually called.\n",
    "        :param input_shape:     Input shape the layer will receive (e.g. B x H x W x C)\n",
    "        \"\"\"\n",
    "        # We are provided with the input shape here, so we know the number of input channels:\n",
    "        num_input_channels = input_shape[-1]  # assuming shape format BHWC\n",
    "\n",
    "        # Now we know how the shape of the tensor representing the kernels should be:\n",
    "        kernels_shape = (*self.kernel_size, num_input_channels, self.num_kernels)\n",
    "\n",
    "        # For this example, we initialize the filters with values picked from a Glorot distribution:\n",
    "        glorot_uni_initializer = tf.initializers.GlorotUniform()\n",
    "        self.kernels = self.add_weight(name='kernels',\n",
    "                                       shape=kernels_shape,\n",
    "                                       initializer=glorot_uni_initializer,\n",
    "                                       trainable=True)  # and we make the variable trainable.\n",
    "\n",
    "        if self.use_bias:  # If bias should be added, we initialize its variable too:\n",
    "            self.bias = self.add_weight(name='bias',\n",
    "                                        shape=(self.num_kernels,),\n",
    "                                        initializer='random_normal',  # e.g., using normal distribution.\n",
    "                                        trainable=True)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        \"\"\"\n",
    "        Call the layer and perform its operations on the input tensors\n",
    "        :param inputs:  Input tensor\n",
    "        :return:        Output tensor\n",
    "        \"\"\"\n",
    "        # We perform the convolution:\n",
    "        z = tf.nn.conv2d(inputs, self.kernels, strides=[1, *self.strides, 1], padding='VALID')\n",
    "\n",
    "        if self.use_bias:  # we add the bias if requested:\n",
    "            z = z + self.bias\n",
    "        # Finally, we apply the activation function (e.g. ReLU):\n",
    "        return tf.nn.relu(z)\n",
    "\n",
    "    def get_config(self):\n",
    "        \"\"\"\n",
    "        Helper function to define the layer and its parameters.\n",
    "        :return:        Dictionary containing the layer's configuration\n",
    "        \"\"\"\n",
    "        return {'num_kernels': self.num_kernels,\n",
    "                'kernel_size': self.kernel_size,\n",
    "                'strides': self.strides,\n",
    "                'use_bias': self.use_bias}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of TensorFlow mathematical operations (e.g. in `tf.maths` and `tf.nn`) already have their derivatives defined by the framework. \n",
    "Therefore, as long as a layer is composed of such operations, **we do not have to manually define its backward propagation**. TensorFlow will automatically cover this, which saves a lot of efforts!\n",
    "\n",
    "The convolutional layer we just implemented is therefore fully operational, and ready to be used inside a CNN, as we will demonstrate right away.\n",
    "\n",
    "***Note:*** As convolutional layers are the most basic component of CNNs, TensorFlow obviously provides its own `tf.keras.layers.Conv2D` class. The module `tf.keras.layers` contains a large variety of pre-implemented standard layers, which we recommend to use whenever possible (as they have more advacned interfaces and optimized operations). For the sake of demonstration, we will still stick to our own simpler `SimpleConvolutionLayer` for the rest of this notebook, while using other Keras pre-defined layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing LeNet-5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***LeNet-5***[$^2$](#ref) is a simple CNN composed of 7 layers (2 *conv*, 2 *max-pool*, 3 *FC* + 1 helper layer to flatten the feature maps before the *FC*). For more details, we invite our readers to go back to Chapter 3.\n",
    "\n",
    "Below, we thus present how one can extend the `tf.keras.Model` class to define a custom architecture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Model\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LeNet5(Model):\n",
    "    \n",
    "    def __init__(self, num_classes):\n",
    "        \"\"\"\n",
    "        Initialize the model.\n",
    "        :param num_classes:     Number of classes to predict from\n",
    "        \"\"\"\n",
    "        super(LeNet5, self).__init__()\n",
    "        # We instantiate the various layers composing LeNet-5:\n",
    "        # self.conv1 = SimpleConvolutionLayer(6, kernel_size=(5, 5))\n",
    "        # self.conv2 = SimpleConvolutionLayer(16, kernel_size=(5, 5))\n",
    "        # ... or using the existing and (recommended) Conv2D class:\n",
    "        self.conv1 = Conv2D(6, kernel_size=(5, 5), padding='same', activation='relu')\n",
    "        self.conv2 = Conv2D(16, kernel_size=(5, 5), activation='relu')\n",
    "        self.max_pool = MaxPooling2D(pool_size=(2, 2))\n",
    "        self.flatten = Flatten()\n",
    "        self.dense1 = Dense(120, activation='relu')\n",
    "        self.dense2 = Dense(84, activation='relu')\n",
    "        self.dense3 = Dense(num_classes, activation='softmax')\n",
    "        \n",
    "    def call(self, inputs):\n",
    "        \"\"\"\n",
    "        Call the layers and perform their operations on the input tensors\n",
    "        :param inputs:  Input tensor\n",
    "        :return:        Output tensor\n",
    "        \"\"\"\n",
    "        x = self.max_pool(self.conv1(inputs))        # 1st block\n",
    "        x = self.max_pool(self.conv2(x))             # 2nd block\n",
    "        x = self.flatten(x)\n",
    "        x = self.dense3(self.dense2(self.dense1(x))) # dense layers\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note:*** For those who already experimented with [PyTorch](https://pytorch.org), another machine learning framework, this functional object-oriented approach to building neural networks may seem familiar. It has been adopted by TensorFlow 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classiyfing MNIST with a CNN\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can instantiate and compile our model for digit classification. To train it for this task, we instantiate the optimizer (a simple _SGD_ one for this example) and define the loss (the _categorical cross-entropy_).\n",
    "\n",
    "***Note:*** The Keras loss named `\"sparse_categorical_crossentropy\"` performs the same cross-entropy operation as the `\"categorical_crossentropy\"`, but the former directly takes the ground-truth labels as inputs, while the latter requires the ground-truth labels to be _one-hotted_ before. Using the `\"sparse_...\"` loss thus saves us from manually having to transform the labels... "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LeNet5(num_classes)\n",
    "model.compile(optimizer='sgd', loss='sparse_categorical_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having sub-classed `tf.keras.Model`, our model has all its functionalities. For instance, we can call `model.summary()` to print a summary of its architecture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"le_net5_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              multiple                  156       \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            multiple                  2416      \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 multiple                  0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          multiple                  0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              multiple                  48120     \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              multiple                  10164     \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              multiple                  850       \n",
      "=================================================================\n",
      "Total params: 61,706\n",
      "Trainable params: 61,706\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# We can call `model.summary()` only if the model was built before. \n",
    "# It is normally done automatically at the first use of the network,\n",
    "# inferring the input shapes from the samples the network is given.\n",
    "# For instance, the command below would build the network (then use it for prediction):\n",
    "_ = model.predict(x_test[:10])\n",
    "\n",
    "# But we can build the model manually otherwise, providing the batched\n",
    "# input shape ourselves:\n",
    "batched_input_shape = tf.TensorShape((None, *input_shape))\n",
    "model.build(input_shape=batched_input_shape)\n",
    "\n",
    "# Method to visualize the architecture of the network:\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before launching the training, we also instantiate some Keras callbacks, i.e., utility functions automatically called at specific points during the training (before/after batch training, before/after a full epoch, etc.), in order to monitor it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "callbacks = [\n",
    "    # Callback to interrupt the training if the validation loss (`val_loss`) stops improving for over 3 epochs:\n",
    "    tf.keras.callbacks.EarlyStopping(patience=5, monitor='val_loss'),\n",
    "    # Callback to log the graph, losses and metrics into TensorBoard (saving log files in `./logs` directory):\n",
    "    tf.keras.callbacks.TensorBoard(log_dir='./logs', histogram_freq=1, write_graph=True)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(The Tensorboard callback allows us to monitor the training from Tensorboard. For that, open a console and launch the programm with the command \"`tensorboard --logdir=./logs`\". You can then access Tensorboard from a browser, via the URL \"[`localhost:6006`](localhost:6006)\".)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now pass everything to our model to train it:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/80\n",
      "60000/60000 - 15s - loss: 1.9806 - accuracy: 0.4209 - val_loss: 1.0401 - val_accuracy: 0.7739\n",
      "Epoch 2/80\n",
      "60000/60000 - 15s - loss: 0.5417 - accuracy: 0.8568 - val_loss: 0.3256 - val_accuracy: 0.9118\n",
      "Epoch 3/80\n",
      "60000/60000 - 15s - loss: 0.2981 - accuracy: 0.9130 - val_loss: 0.2364 - val_accuracy: 0.9314\n",
      "Epoch 4/80\n",
      "60000/60000 - 15s - loss: 0.2352 - accuracy: 0.9315 - val_loss: 0.2023 - val_accuracy: 0.9402\n",
      "Epoch 5/80\n",
      "60000/60000 - 15s - loss: 0.2002 - accuracy: 0.9417 - val_loss: 0.1754 - val_accuracy: 0.9471\n",
      "Epoch 6/80\n",
      "60000/60000 - 15s - loss: 0.1776 - accuracy: 0.9478 - val_loss: 0.1588 - val_accuracy: 0.9520\n",
      "Epoch 7/80\n",
      "60000/60000 - 15s - loss: 0.1617 - accuracy: 0.9523 - val_loss: 0.1412 - val_accuracy: 0.9587\n",
      "Epoch 8/80\n",
      "60000/60000 - 15s - loss: 0.1490 - accuracy: 0.9568 - val_loss: 0.1302 - val_accuracy: 0.9613\n",
      "Epoch 9/80\n",
      "60000/60000 - 14s - loss: 0.1388 - accuracy: 0.9588 - val_loss: 0.1215 - val_accuracy: 0.9630\n",
      "Epoch 10/80\n",
      "60000/60000 - 15s - loss: 0.1299 - accuracy: 0.9617 - val_loss: 0.1116 - val_accuracy: 0.9652\n",
      "Epoch 11/80\n",
      "60000/60000 - 15s - loss: 0.1230 - accuracy: 0.9635 - val_loss: 0.1185 - val_accuracy: 0.9639\n",
      "Epoch 12/80\n",
      "60000/60000 - 15s - loss: 0.1167 - accuracy: 0.9654 - val_loss: 0.1013 - val_accuracy: 0.9667\n",
      "Epoch 13/80\n",
      "60000/60000 - 15s - loss: 0.1111 - accuracy: 0.9666 - val_loss: 0.0994 - val_accuracy: 0.9691\n",
      "Epoch 14/80\n",
      "60000/60000 - 15s - loss: 0.1064 - accuracy: 0.9686 - val_loss: 0.0944 - val_accuracy: 0.9691\n",
      "Epoch 15/80\n",
      "60000/60000 - 15s - loss: 0.1020 - accuracy: 0.9690 - val_loss: 0.0938 - val_accuracy: 0.9710\n",
      "Epoch 16/80\n",
      "60000/60000 - 15s - loss: 0.0978 - accuracy: 0.9706 - val_loss: 0.0941 - val_accuracy: 0.9713\n",
      "Epoch 17/80\n",
      "60000/60000 - 15s - loss: 0.0943 - accuracy: 0.9714 - val_loss: 0.0857 - val_accuracy: 0.9731\n",
      "Epoch 18/80\n",
      "60000/60000 - 15s - loss: 0.0911 - accuracy: 0.9724 - val_loss: 0.0811 - val_accuracy: 0.9736\n",
      "Epoch 19/80\n",
      "60000/60000 - 15s - loss: 0.0880 - accuracy: 0.9734 - val_loss: 0.0813 - val_accuracy: 0.9736\n",
      "Epoch 20/80\n",
      "60000/60000 - 15s - loss: 0.0852 - accuracy: 0.9742 - val_loss: 0.0760 - val_accuracy: 0.9760\n",
      "Epoch 21/80\n",
      "60000/60000 - 15s - loss: 0.0830 - accuracy: 0.9745 - val_loss: 0.0791 - val_accuracy: 0.9739\n",
      "Epoch 22/80\n",
      "60000/60000 - 15s - loss: 0.0807 - accuracy: 0.9751 - val_loss: 0.0749 - val_accuracy: 0.9756\n",
      "Epoch 23/80\n",
      "60000/60000 - 15s - loss: 0.0781 - accuracy: 0.9764 - val_loss: 0.0716 - val_accuracy: 0.9772\n",
      "Epoch 24/80\n",
      "60000/60000 - 15s - loss: 0.0764 - accuracy: 0.9771 - val_loss: 0.0687 - val_accuracy: 0.9785\n",
      "Epoch 25/80\n",
      "60000/60000 - 15s - loss: 0.0747 - accuracy: 0.9777 - val_loss: 0.0730 - val_accuracy: 0.9770\n",
      "Epoch 26/80\n",
      "60000/60000 - 15s - loss: 0.0725 - accuracy: 0.9779 - val_loss: 0.0716 - val_accuracy: 0.9772\n",
      "Epoch 27/80\n",
      "60000/60000 - 15s - loss: 0.0709 - accuracy: 0.9784 - val_loss: 0.0650 - val_accuracy: 0.9789\n",
      "Epoch 28/80\n",
      "60000/60000 - 15s - loss: 0.0694 - accuracy: 0.9792 - val_loss: 0.0658 - val_accuracy: 0.9792\n",
      "Epoch 29/80\n",
      "60000/60000 - 14s - loss: 0.0682 - accuracy: 0.9792 - val_loss: 0.0645 - val_accuracy: 0.9791\n",
      "Epoch 30/80\n",
      "60000/60000 - 15s - loss: 0.0660 - accuracy: 0.9795 - val_loss: 0.0623 - val_accuracy: 0.9803\n",
      "Epoch 31/80\n",
      "60000/60000 - 15s - loss: 0.0652 - accuracy: 0.9800 - val_loss: 0.0645 - val_accuracy: 0.9790\n",
      "Epoch 32/80\n",
      "60000/60000 - 15s - loss: 0.0634 - accuracy: 0.9801 - val_loss: 0.0616 - val_accuracy: 0.9806\n",
      "Epoch 33/80\n",
      "60000/60000 - 14s - loss: 0.0620 - accuracy: 0.9810 - val_loss: 0.0642 - val_accuracy: 0.9801\n",
      "Epoch 34/80\n",
      "60000/60000 - 14s - loss: 0.0610 - accuracy: 0.9812 - val_loss: 0.0586 - val_accuracy: 0.9817\n",
      "Epoch 35/80\n",
      "60000/60000 - 15s - loss: 0.0602 - accuracy: 0.9815 - val_loss: 0.0633 - val_accuracy: 0.9792\n",
      "Epoch 36/80\n",
      "60000/60000 - 15s - loss: 0.0588 - accuracy: 0.9819 - val_loss: 0.0637 - val_accuracy: 0.9791\n",
      "Epoch 37/80\n",
      "60000/60000 - 14s - loss: 0.0580 - accuracy: 0.9823 - val_loss: 0.0586 - val_accuracy: 0.9819\n",
      "Epoch 38/80\n",
      "60000/60000 - 15s - loss: 0.0568 - accuracy: 0.9824 - val_loss: 0.0635 - val_accuracy: 0.9796\n",
      "Epoch 39/80\n",
      "60000/60000 - 15s - loss: 0.0558 - accuracy: 0.9827 - val_loss: 0.0590 - val_accuracy: 0.9818\n",
      "Epoch 40/80\n",
      "60000/60000 - 15s - loss: 0.0551 - accuracy: 0.9831 - val_loss: 0.0659 - val_accuracy: 0.9781\n",
      "Epoch 41/80\n",
      "60000/60000 - 15s - loss: 0.0540 - accuracy: 0.9837 - val_loss: 0.0586 - val_accuracy: 0.9808\n",
      "Epoch 42/80\n",
      "60000/60000 - 15s - loss: 0.0530 - accuracy: 0.9837 - val_loss: 0.0643 - val_accuracy: 0.9793\n",
      "Epoch 43/80\n",
      "60000/60000 - 15s - loss: 0.0521 - accuracy: 0.9841 - val_loss: 0.0543 - val_accuracy: 0.9832\n",
      "Epoch 44/80\n",
      "60000/60000 - 15s - loss: 0.0512 - accuracy: 0.9843 - val_loss: 0.0570 - val_accuracy: 0.9827\n",
      "Epoch 45/80\n",
      "60000/60000 - 15s - loss: 0.0506 - accuracy: 0.9842 - val_loss: 0.0525 - val_accuracy: 0.9842\n",
      "Epoch 46/80\n",
      "60000/60000 - 15s - loss: 0.0502 - accuracy: 0.9842 - val_loss: 0.0540 - val_accuracy: 0.9841\n",
      "Epoch 47/80\n",
      "60000/60000 - 15s - loss: 0.0489 - accuracy: 0.9853 - val_loss: 0.0558 - val_accuracy: 0.9816\n",
      "Epoch 48/80\n",
      "60000/60000 - 14s - loss: 0.0487 - accuracy: 0.9848 - val_loss: 0.0572 - val_accuracy: 0.9825\n",
      "Epoch 49/80\n",
      "60000/60000 - 14s - loss: 0.0477 - accuracy: 0.9855 - val_loss: 0.0528 - val_accuracy: 0.9841\n",
      "Epoch 50/80\n",
      "60000/60000 - 15s - loss: 0.0468 - accuracy: 0.9858 - val_loss: 0.0522 - val_accuracy: 0.9845\n",
      "Epoch 51/80\n",
      "60000/60000 - 15s - loss: 0.0463 - accuracy: 0.9858 - val_loss: 0.0595 - val_accuracy: 0.9798\n",
      "Epoch 52/80\n",
      "60000/60000 - 15s - loss: 0.0457 - accuracy: 0.9857 - val_loss: 0.0513 - val_accuracy: 0.9846\n",
      "Epoch 53/80\n",
      "60000/60000 - 15s - loss: 0.0450 - accuracy: 0.9861 - val_loss: 0.0499 - val_accuracy: 0.9853\n",
      "Epoch 54/80\n",
      "60000/60000 - 15s - loss: 0.0445 - accuracy: 0.9860 - val_loss: 0.0505 - val_accuracy: 0.9844\n",
      "Epoch 55/80\n",
      "60000/60000 - 16s - loss: 0.0437 - accuracy: 0.9862 - val_loss: 0.0507 - val_accuracy: 0.9850\n",
      "Epoch 56/80\n",
      "60000/60000 - 16s - loss: 0.0431 - accuracy: 0.9865 - val_loss: 0.0518 - val_accuracy: 0.9836\n",
      "Epoch 57/80\n",
      "60000/60000 - 16s - loss: 0.0430 - accuracy: 0.9865 - val_loss: 0.0504 - val_accuracy: 0.9843\n",
      "Epoch 58/80\n",
      "60000/60000 - 16s - loss: 0.0419 - accuracy: 0.9872 - val_loss: 0.0495 - val_accuracy: 0.9851\n",
      "Epoch 59/80\n",
      "60000/60000 - 16s - loss: 0.0415 - accuracy: 0.9874 - val_loss: 0.0502 - val_accuracy: 0.9842\n",
      "Epoch 60/80\n",
      "60000/60000 - 16s - loss: 0.0413 - accuracy: 0.9873 - val_loss: 0.0489 - val_accuracy: 0.9852\n",
      "Epoch 61/80\n",
      "60000/60000 - 16s - loss: 0.0409 - accuracy: 0.9870 - val_loss: 0.0531 - val_accuracy: 0.9838\n",
      "Epoch 62/80\n",
      "60000/60000 - 16s - loss: 0.0404 - accuracy: 0.9875 - val_loss: 0.0483 - val_accuracy: 0.9853\n",
      "Epoch 63/80\n",
      "60000/60000 - 16s - loss: 0.0396 - accuracy: 0.9876 - val_loss: 0.0490 - val_accuracy: 0.9859\n",
      "Epoch 64/80\n",
      "60000/60000 - 16s - loss: 0.0390 - accuracy: 0.9880 - val_loss: 0.0475 - val_accuracy: 0.9854\n",
      "Epoch 65/80\n",
      "60000/60000 - 16s - loss: 0.0386 - accuracy: 0.9883 - val_loss: 0.0498 - val_accuracy: 0.9837\n",
      "Epoch 66/80\n",
      "60000/60000 - 16s - loss: 0.0382 - accuracy: 0.9878 - val_loss: 0.0476 - val_accuracy: 0.9861\n",
      "Epoch 67/80\n",
      "60000/60000 - 15s - loss: 0.0374 - accuracy: 0.9885 - val_loss: 0.0515 - val_accuracy: 0.9853\n",
      "Epoch 68/80\n",
      "60000/60000 - 16s - loss: 0.0374 - accuracy: 0.9883 - val_loss: 0.0518 - val_accuracy: 0.9846\n",
      "Epoch 69/80\n",
      "60000/60000 - 16s - loss: 0.0364 - accuracy: 0.9887 - val_loss: 0.0482 - val_accuracy: 0.9845\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train,\n",
    "                    batch_size=32, epochs=80, validation_data=(x_test, y_test), \n",
    "                    verbose=2,  # change to `verbose=1` to get a progress bar\n",
    "                                # (we opt for `verbose=2` here to reduce the log size)\n",
    "                    callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Given a machine with recent GPU(s), this training is quite fast (~0.1ms/image in our case). The final accuracy we obtain on the validation dataset (**~98%!**) is also much better compared to our previous attempts with simpler networks. Indeed, the relative error has been approximately divided by 2, which is a significant improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref\"></a>\n",
    "#### References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. LeCun, Y., Cortes, C., Burges, C., 2010. MNIST handwritten digit database. AT&T Labs [Online]. Available: http://yann.lecun.com/exdb/mnist 2, 18.\n",
    "2. LeCun, Yann. \"*LeNet-5, convolutional neural networks.*\" [http://yann.lecun.com/exdb/lenet](http://yann.lecun.com/exdb/lenet) (2015): 20."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}