TensorFlow_MNIST_Exercises_All.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TensorFlow_MNIST_Exercises_All.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/van-dang/FEniCS-Colab/blob/master/TensorFlow_MNIST_Exercises_All.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "rwmQ557PloIs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Exercises\n",
        "\n",
        "There are several main adjustments you may try.\n",
        "\n",
        "It is useful to take note of the time it takes the algorithm to train. This will provide you with an additional level of understanding. That is also your first task.\n",
        "\n",
        "Using the code from the lecture as the basis, fiddle with the hyperparameters of the algorithm.\n",
        "\n",
        "1. The *width* (the hidden layer size) of the algorithm. Try a hidden layer size of 200. How does the validation accuracy of the model change? What about the time it took the algorithm to train? Can you find a hidden layer size that does better?\n",
        "\n",
        "2. The *depth* of the algorithm. Add another hidden layer to the algorithm. This is an extremely important exercise! How does the validation accuracy change? What about the time it took the algorithm to train? Hint: Be careful with the shapes of the weights and the biases.\n",
        "\n",
        "3. The *width and depth* of the algorithm. Add as many additional layers as you need to reach 5 hidden layers. Moreover, adjust the width of the algorithm as you find suitable. How does the validation accuracy change? What about the time it took the algorithm to train?\n",
        "\n",
        "4. Fiddle with the activation functions. Try applying sigmoid transformation to both layers. The sigmoid activation is given by the method: tf.nn.sigmoid()\n",
        "\n",
        "5. Fiddle with the activation functions. Try applying a ReLu to the first hidden layer and tanh to the second one. The tanh activation is given by the method: tf.nn.tanh()\n",
        "\n",
        "6. Adjust the batch size. Try a batch size of 1000. How does the required time change? What about the accuracy?\n",
        "\n",
        "7. Adjust the batch size. Try a batch size of 1. That's the SGD. How do the time and accuracy change? Is the result coherent with the theory?\n",
        "\n",
        "8. Adjust the learning rate. Try a value of 0.0001. Does it make a difference?\n",
        "\n",
        "9. Adjust the learning rate. Try a value of 0.02. Does it make a difference?\n",
        "\n",
        "10. Combine all the methods above and try to reach a validation accuracy of 98.5+ percent.\n",
        "\n",
        "Good luck!"
      ]
    },
    {
      "metadata": {
        "id": "oTviByY-loIu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Deep Neural Network for MNIST Classification\n",
        "\n",
        "We'll apply all the knowledge from the lectures in this section to write a deep neural network. The problem we've chosen is referred to as the \"Hello World\" for machine learning because for most students it is their first example. The dataset is called MNIST and refers to handwritten digit recognition. You can find more about it on Yann LeCun's website (Director of AI Research, Facebook). He is one of the pioneers of what we've been talking about and of more complex approaches that are widely used today, such as covolutional networks. The dataset provides 28x28 images of handwritten digits (1 per image) and the goal is to write an algorithm that detects which digit is written. Since there are only 10 digits, this is a classification problem with 10 classes. In order to exemplify what we've talked about in this section, we will build a network with 2 hidden layers between inputs and outputs."
      ]
    },
    {
      "metadata": {
        "id": "sejMtweYloIu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Import the relevant packages"
      ]
    },
    {
      "metadata": {
        "id": "nqfyqcOoloIv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from tensorflow.examples.tutorials.mnist import input_data\n",
        "\n",
        "# TensorFLow includes a data provider for MNIST that we'll use.\n",
        "# This function automatically downloads the MNIST dataset to the chosen directory. \n",
        "# The dataset is already split into training, validation, and test subsets. \n",
        "# Furthermore, it preprocess it into a particularly simple and useful format.\n",
        "# Every 28x28 image is flattened into a vector of length 28x28=784, where every value\n",
        "# corresponds to the intensity of the color of the corresponding pixel.\n",
        "# The samples are grayscale (but standardized from 0 to 1), so a value close to 0 is almost white and a value close to\n",
        "# 1 is almost purely black. This representation (flattening the image row by row into\n",
        "# a vector) is slightly naive but as you'll see it works surprisingly well.\n",
        "# Since this is a classification problem, our targets are categorical.\n",
        "# Recall from the lecture on that topic that one way to deal with that is to use one-hot encoding.\n",
        "# With it, the target for each individual sample is a vector of length 10\n",
        "# which has nine 0s and a single 1 at the position which corresponds to the correct answer.\n",
        "# For instance, if the true answer is \"1\", the target will be [0,0,0,1,0,0,0,0,0,0] (counting from 0).\n",
        "# Have in mind that the very first time you execute this command it might take a little while to run\n",
        "# because it has to download the whole dataset. Following commands only extract it so they're faster.\n",
        "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wUblFjx1loIy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Outline the model\n",
        "\n",
        "The whole code is in one cell, so you can simply rerun this cell (instead of the whole notebook) and train a new model.\n",
        "The tf.reset_default_graph() function takes care of clearing the old parameters. From there on, a completely new training starts."
      ]
    },
    {
      "metadata": {
        "id": "qHmOBuRNloIz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "input_size = 784\n",
        "output_size = 10\n",
        "# Use same hidden layer size for both hidden layers. Not a necessity.\n",
        "hidden_layer_size = 50\n",
        "\n",
        "# Reset any variables left in memory from previous runs.\n",
        "tf.reset_default_graph()\n",
        "\n",
        "# As in the previous example - declare placeholders where the data will be fed into.\n",
        "inputs = tf.placeholder(tf.float32, [None, input_size])\n",
        "targets = tf.placeholder(tf.float32, [None, output_size])\n",
        "\n",
        "# Weights and biases for the first linear combination between the inputs and the first hidden layer.\n",
        "# Use get_variable in order to make use of the default TensorFlow initializer which is Xavier.\n",
        "weights_1 = tf.get_variable(\"weights_1\", [input_size, hidden_layer_size])\n",
        "biases_1 = tf.get_variable(\"biases_1\", [hidden_layer_size])\n",
        "\n",
        "# Operation between the inputs and the first hidden layer.\n",
        "# We've chosen ReLu as our activation function. You can try playing with different non-linearities.\n",
        "outputs_1 = tf.nn.relu(tf.matmul(inputs, weights_1) + biases_1)\n",
        "\n",
        "# Weights and biases for the second linear combination.\n",
        "# This is between the first and second hidden layers.\n",
        "weights_2 = tf.get_variable(\"weights_2\", [hidden_layer_size, hidden_layer_size])\n",
        "biases_2 = tf.get_variable(\"biases_2\", [hidden_layer_size])\n",
        "\n",
        "# Operation between the first and the second hidden layers. Again, we use ReLu.\n",
        "outputs_2 = tf.nn.relu(tf.matmul(outputs_1, weights_2) + biases_2)\n",
        "\n",
        "# Weights and biases for the final linear combination.\n",
        "# That's between the second hidden layer and the output layer.\n",
        "weights_3 = tf.get_variable(\"weights_3\", [hidden_layer_size, output_size])\n",
        "biases_3 = tf.get_variable(\"biases_3\", [output_size])\n",
        "\n",
        "# Operation between the second hidden layer and the final output.\n",
        "# Notice we have not used an activation function because we'll use the trick to include it directly in \n",
        "# the loss function. This works for softmax and sigmoid with cross entropy.\n",
        "outputs = tf.matmul(outputs_2, weights_3) + biases_3\n",
        "\n",
        "# Calculate the loss function for every output/target pair.\n",
        "# The function used is the same as applying softmax to the last layer and then calculating cross entropy\n",
        "# with the function we've seen in the lectures. This function, however, combines them in a clever way, \n",
        "# which makes it both faster and more numerically stable (when dealing with very small numbers).\n",
        "# Logits here means: unscaled probabilities (so, the outputs, before they are scaled by the softmax)\n",
        "# Naturally, the labels are the targets.\n",
        "loss = tf.nn.softmax_cross_entropy_with_logits(logits=outputs, labels=targets)\n",
        "\n",
        "# Get the average loss\n",
        "mean_loss = tf.reduce_mean(loss)\n",
        "\n",
        "# Define the optimization step. Using adaptive optimizers such as Adam in TensorFlow\n",
        "# is as simple as that.\n",
        "optimize = tf.train.AdamOptimizer(learning_rate=0.001).minimize(mean_loss)\n",
        "\n",
        "# Get a 0 or 1 for every input in the batch indicating whether it output the correct answer out of the 10.\n",
        "out_equals_target = tf.equal(tf.argmax(outputs, 1), tf.argmax(targets, 1))\n",
        "\n",
        "# Get the average accuracy of the outputs.\n",
        "accuracy = tf.reduce_mean(tf.cast(out_equals_target, tf.float32))\n",
        "\n",
        "# Declare the session variable.\n",
        "sess = tf.InteractiveSession()\n",
        "\n",
        "# Initialize the variables. Default initializer is Xavier.\n",
        "initializer = tf.global_variables_initializer()\n",
        "sess.run(initializer)\n",
        "\n",
        "# Batching\n",
        "batch_size = 100\n",
        "\n",
        "# Calculate the number of batches per epoch for the training set.\n",
        "batches_number = mnist.train._num_examples // batch_size\n",
        "\n",
        "# Basic early stopping. Set a miximum number of epochs.\n",
        "max_epochs = 15\n",
        "\n",
        "# Keep track of the validation loss of the previous epoch.\n",
        "# If the validation loss becomes increasing, we want to trigger early stopping.\n",
        "# We initially set it at some arbitrarily high number to make sure we don't trigger it\n",
        "# at the first epoch\n",
        "prev_validation_loss = 9999999.\n",
        "\n",
        "import time\n",
        "start_time = time.time()\n",
        "\n",
        "# Create a loop for the epochs. Epoch_counter is a variable which automatically starts from 0.\n",
        "for epoch_counter in range(max_epochs):\n",
        "    \n",
        "    # Keep track of the sum of batch losses in the epoch.\n",
        "    curr_epoch_loss = 0.\n",
        "    \n",
        "    # Iterate over the batches in this epoch.\n",
        "    for batch_counter in range(batches_number):\n",
        "        \n",
        "        # Input batch and target batch are assigned values from the train dataset, given a batch size\n",
        "        input_batch, target_batch = mnist.train.next_batch(batch_size)\n",
        "        \n",
        "        # Run the optimization step and get the mean loss for this batch.\n",
        "        # Feed it with the inputs and the targets we just got from the train dataset\n",
        "        _, batch_loss = sess.run([optimize, mean_loss], \n",
        "            feed_dict={inputs: input_batch, targets: target_batch})\n",
        "        \n",
        "        # Increment the sum of batch losses.\n",
        "        curr_epoch_loss += batch_loss\n",
        "    \n",
        "    # So far curr_epoch_loss contained the sum of all batches inside the epoch\n",
        "    # We want to find the average batch losses over the whole epoch\n",
        "    # The average batch loss is a good proxy for the current epoch loss\n",
        "    curr_epoch_loss /= batches_number\n",
        "    \n",
        "    # At the end of each epoch, get the validation loss and accuracy\n",
        "    # Get the input batch and the target batch from the validation dataset\n",
        "    input_batch, target_batch = mnist.validation.next_batch(mnist.validation._num_examples)\n",
        "    \n",
        "    # Run without the optimization step (simply forward propagate)\n",
        "    validation_loss, validation_accuracy = sess.run([mean_loss, accuracy], \n",
        "        feed_dict={inputs: input_batch, targets: target_batch})\n",
        "    \n",
        "    # Print statistics for the current epoch\n",
        "    # Epoch counter + 1, because epoch_counter automatically starts from 0, instead of 1\n",
        "    # We format the losses with 3 digits after the dot\n",
        "    # We format the accuracy in percentages for easier interpretation\n",
        "    print('Epoch '+str(epoch_counter+1)+\n",
        "          '. Mean loss: '+'{0:.3f}'.format(curr_epoch_loss)+\n",
        "          '. Validation loss: '+'{0:.3f}'.format(validation_loss)+\n",
        "          '. Validation accuracy: '+'{0:.2f}'.format(validation_accuracy * 100.)+'%')\n",
        "    \n",
        "    # Trigger early stopping if validation loss begins increasing.\n",
        "    if validation_loss > prev_validation_loss:\n",
        "        break\n",
        "        \n",
        "    # Store this epoch's validation loss to be used as previous validation loss in the next iteration.\n",
        "    prev_validation_loss = validation_loss\n",
        "\n",
        "# Not essential, but it is nice to know when the algorithm stopped working in the output section, rather than check the kernel\n",
        "print('End of training.')\n",
        "\n",
        "#Add the time it took the algorithm to train\n",
        "print(\"Training time: %s seconds\" % (time.time() - start_time))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WincsKaoloI1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Test the model\n",
        "\n",
        "As we discussed in the lectures, after training on the training and validation sets, we test the final prediction power of our model by running it on the test dataset that the algorithm has not seen before.\n",
        "\n",
        "It is very important to realize that fiddling with the hyperparameters overfits the validation dataset. The test is the absolute final instance. You should not test before you are completely done with adjusting your model."
      ]
    },
    {
      "metadata": {
        "id": "JKAw2YaoloI2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "input_batch, target_batch = mnist.test.next_batch(mnist.test._num_examples)\n",
        "test_accuracy = sess.run([accuracy], \n",
        "    feed_dict={inputs: input_batch, targets: target_batch})\n",
        "\n",
        "# Test accuracy is a list with 1 value, so we want to extract the value from it, using x[0]\n",
        "# Uncomment the print to see how it looks before the manipulation\n",
        "# print (test_accuracy)\n",
        "test_accuracy_percent = test_accuracy[0] * 100.\n",
        "\n",
        "# Print the test accuracy formatted in percentages\n",
        "print('Test accuracy: '+'{0:.2f}'.format(test_accuracy_percent)+'%')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ocmSffwzloI4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Using the initial model and hyperparameters given in this notebook, the final test accuracy should be roughly between 97% and 98%. Each time the code is rerunned, we get a different accuracy as the batches are shuffled, the weights are initialized in a different way, etc.\n",
        "\n",
        "Finally, we have intentionally reached a suboptimal solution, so you can have space to build on it."
      ]
    }
  ]
}