TensorFlow_MNIST_All_Exercises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n",
    "\n",
    "There are several main adjustments you may try.\n",
    "\n",
    "Please pay attention to the time it takes for each epoch to conclude.\n",
    "\n",
    "Using the code from the lecture as the basis, fiddle with the hyperparameters of the algorithm.\n",
    "\n",
    "1. The *width* (the hidden layer size) of the algorithm. Try a hidden layer size of 200. How does the validation accuracy of the model change? What about the time it took the algorithm to train? Can you find a hidden layer size that does better?\n",
    "\n",
    "2. The *depth* of the algorithm. Add another hidden layer to the algorithm. This is an extremely important exercise! How does the validation accuracy change? What about the time it took the algorithm to train? Hint: Be careful with the shapes of the weights and the biases.\n",
    "\n",
    "3. The *width and depth* of the algorithm. Add as many additional layers as you need to reach 5 hidden layers. Moreover, adjust the width of the algorithm as you find suitable. How does the validation accuracy change? What about the time it took the algorithm to train?\n",
    "\n",
    "4. Fiddle with the activation functions. Try applying sigmoid transformation to both layers. The sigmoid activation is given by the string 'sigmoid'.\n",
    "\n",
    "5. Fiddle with the activation functions. Try applying a ReLu to the first hidden layer and tanh to the second one. The tanh activation is given by the string 'tanh'.\n",
    "\n",
    "6. Adjust the batch size. Try a batch size of 10000. How does the required time change? What about the accuracy?\n",
    "\n",
    "7. Adjust the batch size. Try a batch size of 1. That's the SGD. How do the time and accuracy change? Is the result coherent with the theory?\n",
    "\n",
    "8. Adjust the learning rate. Try a value of 0.0001. Does it make a difference?\n",
    "\n",
    "9. Adjust the learning rate. Try a value of 0.02. Does it make a difference?\n",
    "\n",
    "10. Combine all the methods above and try to reach a validation accuracy of 98.5+ percent.\n",
    "\n",
    "Good luck!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Neural Network for MNIST Classification\n",
    "\n",
    "We'll apply all the knowledge from the lectures in this section to write a deep neural network. The problem we've chosen is referred to as the \"Hello World\" of deep learning because for most students it is the first deep learning algorithm they see.\n",
    "\n",
    "The dataset is called MNIST and refers to handwritten digit recognition. You can find more about it on Yann LeCun's website (Director of AI Research, Facebook). He is one of the pioneers of what we've been talking about and of more complex approaches that are widely used today, such as covolutional neural networks (CNNs). \n",
    "\n",
    "The dataset provides 70,000 images (28x28 pixels) of handwritten digits (1 digit per image). \n",
    "\n",
    "The goal is to write an algorithm that detects which digit is written. Since there are only 10 digits (0, 1, 2, 3, 4, 5, 6, 7, 8, 9), this is a classification problem with 10 classes. \n",
    "\n",
    "Our goal would be to build a neural network with 2 hidden layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the relevant packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "# TensorFLow includes a data provider for MNIST that we'll use.\n",
    "# It comes with the tensorflow-datasets module, therefore, if you haven't please install the package using\n",
    "# pip install tensorflow-datasets \n",
    "# or\n",
    "# conda install tensorflow-datasets\n",
    "\n",
    "import tensorflow_datasets as tfds\n",
    "\n",
    "# these datasets will be stored in C:\\Users\\*USERNAME*\\tensorflow_datasets\\...\n",
    "# the first time you download a dataset, it is stored in the respective folder \n",
    "# every other time, it is automatically loading the copy on your computer "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "That's where we load and preprocess our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remember the comment from above\n",
    "# these datasets will be stored in C:\\Users\\*USERNAME*\\tensorflow_datasets\\...\n",
    "# the first time you download a dataset, it is stored in the respective folder \n",
    "# every other time, it is automatically loading the copy on your computer \n",
    "\n",
    "# tfds.load actually loads a dataset (or downloads and then loads if that's the first time you use it) \n",
    "# in our case, we are interesteed in the MNIST; the name of the dataset is the only mandatory argument\n",
    "# there are other arguments we can specify, which we can find useful\n",
    "# mnist_dataset = tfds.load(name='mnist', as_supervised=True)\n",
    "mnist_dataset, mnist_info = tfds.load(name='mnist', with_info=True, as_supervised=True)\n",
    "# with_info=True will also provide us with a tuple containing information about the version, features, number of samples\n",
    "# we will use this information a bit below and we will store it in mnist_info\n",
    "\n",
    "# as_supervised=True will load the dataset in a 2-tuple structure (input, target) \n",
    "# alternatively, as_supervised=False, would return a dictionary\n",
    "# obviously we prefer to have our inputs and targets separated \n",
    "\n",
    "# once we have loaded the dataset, we can easily extract the training and testing dataset with the built references\n",
    "mnist_train, mnist_test = mnist_dataset['train'], mnist_dataset['test']\n",
    "\n",
    "# by default, TF has training and testing datasets, but no validation sets\n",
    "# thus we must split it on our own\n",
    "\n",
    "# we start by defining the number of validation samples as a % of the train samples\n",
    "# this is also where we make use of mnist_info (we don't have to count the observations)\n",
    "num_validation_samples = 0.1 * mnist_info.splits['train'].num_examples\n",
    "# let's cast this number to an integer, as a float may cause an error along the way\n",
    "num_validation_samples = tf.cast(num_validation_samples, tf.int64)\n",
    "\n",
    "# let's also store the number of test samples in a dedicated variable (instead of using the mnist_info one)\n",
    "num_test_samples = mnist_info.splits['test'].num_examples\n",
    "# once more, we'd prefer an integer (rather than the default float)\n",
    "num_test_samples = tf.cast(num_test_samples, tf.int64)\n",
    "\n",
    "\n",
    "# normally, we would like to scale our data in some way to make the result more numerically stable\n",
    "# in this case we will simply prefer to have inputs between 0 and 1\n",
    "# let's define a function called: scale, that will take an MNIST image and its label\n",
    "def scale(image, label):\n",
    "    # we make sure the value is a float\n",
    "    image = tf.cast(image, tf.float32)\n",
    "    # since the possible values for the inputs are 0 to 255 (256 different shades of grey)\n",
    "    # if we divide each element by 255, we would get the desired result -> all elements will be between 0 and 1 \n",
    "    image /= 255.\n",
    "\n",
    "    return image, label\n",
    "\n",
    "\n",
    "# the method .map() allows us to apply a custom transformation to a given dataset\n",
    "# we have already decided that we will get the validation data from mnist_train, so \n",
    "scaled_train_and_validation_data = mnist_train.map(scale)\n",
    "\n",
    "# finally, we scale and batch the test data\n",
    "# we scale it so it has the same magnitude as the train and validation\n",
    "# there is no need to shuffle it, because we won't be training on the test data\n",
    "# there would be a single batch, equal to the size of the test data\n",
    "test_data = mnist_test.map(scale)\n",
    "\n",
    "\n",
    "# let's also shuffle the data\n",
    "\n",
    "BUFFER_SIZE = 10000\n",
    "# this BUFFER_SIZE parameter is here for cases when we're dealing with enormous datasets\n",
    "# then we can't shuffle the whole dataset in one go because we can't fit it all in memory\n",
    "# so instead TF only stores BUFFER_SIZE samples in memory at a time and shuffles them\n",
    "# if BUFFER_SIZE=1 => no shuffling will actually happen\n",
    "# if BUFFER_SIZE >= num samples => shuffling is uniform\n",
    "# BUFFER_SIZE in between - a computational optimization to approximate uniform shuffling\n",
    "\n",
    "# luckily for us, there is a shuffle method readily available and we just need to specify the buffer size\n",
    "shuffled_train_and_validation_data = scaled_train_and_validation_data.shuffle(BUFFER_SIZE)\n",
    "\n",
    "# once we have scaled and shuffled the data, we can proceed to actually extracting the train and validation\n",
    "# our validation data would be equal to 10% of the training set, which we've already calculated\n",
    "# we use the .take() method to take that many samples\n",
    "# finally, we create a batch with a batch size equal to the total number of validation samples\n",
    "validation_data = shuffled_train_and_validation_data.take(num_validation_samples)\n",
    "\n",
    "# similarly, the train_data is everything else, so we skip as many samples as there are in the validation dataset\n",
    "train_data = shuffled_train_and_validation_data.skip(num_validation_samples)\n",
    "\n",
    "# determine the batch size\n",
    "BATCH_SIZE = 100\n",
    "\n",
    "# we can also take advantage of the occasion to batch the train data\n",
    "# this would be very helpful when we train, as we would be able to iterate over the different batches\n",
    "train_data = train_data.batch(BATCH_SIZE)\n",
    "\n",
    "validation_data = validation_data.batch(num_validation_samples)\n",
    "\n",
    "# batch the test data\n",
    "test_data = test_data.batch(num_test_samples)\n",
    "\n",
    "\n",
    "# takes next batch (it is the only batch)\n",
    "# because as_supervized=True, we've got a 2-tuple structure\n",
    "validation_inputs, validation_targets = next(iter(validation_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outline the model\n",
    "When thinking about a deep learning algorithm, we mostly imagine building the model. So, let's do it :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_size = 784\n",
    "output_size = 10\n",
    "# Use same hidden layer size for both hidden layers. Not a necessity.\n",
    "hidden_layer_size = 50\n",
    "    \n",
    "# define how the model will look like\n",
    "model = tf.keras.Sequential([\n",
    "    \n",
    "    # the first layer (the input layer)\n",
    "    # each observation is 28x28x1 pixels, therefore it is a tensor of rank 3\n",
    "    # since we don't know CNNs yet, we don't know how to feed such input into our net, so we must flatten the images\n",
    "    # there is a convenient method 'Flatten' that simply takes our 28x28x1 tensor and orders it into a (None,) \n",
    "    # or (28x28x1,) = (784,) vector\n",
    "    # this allows us to actually create a feed forward neural network\n",
    "    tf.keras.layers.Flatten(input_shape=(28, 28, 1)), # input layer\n",
    "    \n",
    "    # tf.keras.layers.Dense is basically implementing: output = activation(dot(input, weight) + bias)\n",
    "    # it takes several arguments, but the most important ones for us are the hidden_layer_size and the activation function\n",
    "    tf.keras.layers.Dense(hidden_layer_size, activation='relu'), # 1st hidden layer\n",
    "    tf.keras.layers.Dense(hidden_layer_size, activation='relu'), # 2nd hidden layer\n",
    "    \n",
    "    # the final layer is no different, we just make sure to activate it with softmax\n",
    "    tf.keras.layers.Dense(output_size, activation='softmax') # output layer\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose the optimizer and the loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we define the optimizer we'd like to use, \n",
    "# the loss function, \n",
    "# and the metrics we are interested in obtaining at each iteration\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "That's where we train the model we have built."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "540/540 - 6s - loss: 0.3206 - accuracy: 0.9105 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00\n",
      "Epoch 2/5\n",
      "540/540 - 5s - loss: 0.1344 - accuracy: 0.9604 - val_loss: 0.1131 - val_accuracy: 0.9680\n",
      "Epoch 3/5\n",
      "540/540 - 5s - loss: 0.0969 - accuracy: 0.9710 - val_loss: 0.0972 - val_accuracy: 0.9745\n",
      "Epoch 4/5\n",
      "540/540 - 5s - loss: 0.0752 - accuracy: 0.9776 - val_loss: 0.0841 - val_accuracy: 0.9745\n",
      "Epoch 5/5\n",
      "540/540 - 5s - loss: 0.0609 - accuracy: 0.9812 - val_loss: 0.0681 - val_accuracy: 0.9790\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x229f22494a8>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# determine the maximum number of epochs\n",
    "NUM_EPOCHS = 5\n",
    "\n",
    "# we fit the model, specifying the\n",
    "# training data\n",
    "# the total number of epochs\n",
    "# and the validation data we just created ourselves in the format: (inputs,targets)\n",
    "model.fit(train_data, epochs=NUM_EPOCHS, validation_data=(validation_inputs, validation_targets), verbose =2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the model\n",
    "\n",
    "As we discussed in the lectures, after training on the training data and validating on the validation data, we test the final prediction power of our model by running it on the test dataset that the algorithm has NEVER seen before.\n",
    "\n",
    "It is very important to realize that fiddling with the hyperparameters overfits the validation dataset. \n",
    "\n",
    "The test is the absolute final instance. You should not test before you are completely done with adjusting your model.\n",
    "\n",
    "If you adjust your model after testing, you will start overfitting the test dataset, which will defeat its purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      1/Unknown - 1s 1s/step - loss: 0.0897 - accuracy: 0.97 - 1s 1s/step - loss: 0.0897 - accuracy: 0.9728"
     ]
    }
   ],
   "source": [
    "test_loss, test_accuracy = model.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.09. Test accuracy: 97.28%\n"
     ]
    }
   ],
   "source": [
    "# We can apply some nice formatting if we want to\n",
    "print('Test loss: {0:.2f}. Test accuracy: {1:.2f}%'.format(test_loss, test_accuracy*100.))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the initial model and hyperparameters given in this notebook, the final test accuracy should be roughly around 97%.\n",
    "\n",
    "Each time the code is rerun, we get a different accuracy as the batches are shuffled, the weights are initialized in a different way, etc.\n",
    "\n",
    "Finally, we have intentionally reached a suboptimal solution, so you can have space to build on it."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:py3-TF2.0]",
   "language": "python",
   "name": "conda-env-py3-TF2.0-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}