2-3-5-FashionMNISTConvolutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R6gHiH-I7uFa"
   },
   "source": [
    "# Improving Computer Vision Accuracy using Convolutions\n",
    "\n",
    "In previous lessons you saw how to do image recognition using a Deep Neural Network (DNN) containing three layers -- the input layer (in the shape of the data), the output layer (in the shape of the desired output) and a hidden layer. \n",
    "For convenience, here's the entire code again. Run it and take a note of the accuracy that is printed out at the end. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 665
    },
    "id": "xcsRtq9OLorS",
    "outputId": "c6ba16df-97a3-49d2-e6dd-e0c7def4745d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz\n",
      "32768/29515 [=================================] - 0s 0us/step\n",
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz\n",
      "26427392/26421880 [==============================] - 0s 0us/step\n",
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz\n",
      "8192/5148 [===============================================] - 0s 0us/step\n",
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz\n",
      "4423680/4422102 [==============================] - 0s 0us/step\n",
      "Epoch 1/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.5839 - accuracy: 0.7997 - val_loss: 0.4742 - val_accuracy: 0.8348\n",
      "Epoch 2/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.4258 - accuracy: 0.8510 - val_loss: 0.4388 - val_accuracy: 0.8475\n",
      "Epoch 3/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3965 - accuracy: 0.8596 - val_loss: 0.4402 - val_accuracy: 0.8460\n",
      "Epoch 4/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3787 - accuracy: 0.8664 - val_loss: 0.4270 - val_accuracy: 0.8511\n",
      "Epoch 5/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3643 - accuracy: 0.8706 - val_loss: 0.4093 - val_accuracy: 0.8571\n",
      "Epoch 6/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3565 - accuracy: 0.8731 - val_loss: 0.3998 - val_accuracy: 0.8585\n",
      "Epoch 7/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3473 - accuracy: 0.8757 - val_loss: 0.3955 - val_accuracy: 0.8599\n",
      "Epoch 8/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3430 - accuracy: 0.8773 - val_loss: 0.4003 - val_accuracy: 0.8561\n",
      "Epoch 9/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3360 - accuracy: 0.8801 - val_loss: 0.3938 - val_accuracy: 0.8637\n",
      "Epoch 10/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3313 - accuracy: 0.8810 - val_loss: 0.4017 - val_accuracy: 0.8553\n",
      "Epoch 11/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3267 - accuracy: 0.8820 - val_loss: 0.3882 - val_accuracy: 0.8644\n",
      "Epoch 12/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3230 - accuracy: 0.8844 - val_loss: 0.3831 - val_accuracy: 0.8668\n",
      "Epoch 13/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.3186 - accuracy: 0.8856 - val_loss: 0.4011 - val_accuracy: 0.8590\n",
      "Epoch 14/20\n",
      "1862/1875 [============================>.] - ETA: 0s - loss: 0.3157 - accuracy: 0.8861"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "mnist = tf.keras.datasets.fashion_mnist\n",
    "(training_images, training_labels), (val_images, val_labels) = mnist.load_data()\n",
    "training_images=training_images / 255.0\n",
    "val_images=val_images / 255.0\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Flatten(),\n",
    "  tf.keras.layers.Dense(20, activation=tf.nn.relu),\n",
    "  tf.keras.layers.Dense(10, activation=tf.nn.softmax)\n",
    "])\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(training_images, training_labels, validation_data=(val_images, val_labels), epochs=20)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zldEXSsF8Noz"
   },
   "source": [
    "Your accuracy is probably about 89% on training and 87% on validation...not bad...But how do you make that even better? One way is to use something called Convolutions. I'm not going to get into the details of Convolutions here, but the ultimate concept is that they narrow down the content of the image to focus on specific, distinct, features. \n",
    "\n",
    "If you've ever done image processing using a filter (like this: https://en.wikipedia.org/wiki/Kernel_(image_processing)) then convolutions will look very familiar.\n",
    "\n",
    "In short, you take an array (usually 3x3 or 5x5) and pass it over the image. By changing the underlying pixels based on the formula within that matrix, you can do things like edge detection. So, for example, if you look at the above link, you'll see a 3x3 that is defined for edge detection where the middle cell is 8, and all of its neighbors are -1. In this case, for each pixel, you would multiply its value by 8, then subtract the value of each neighbor. Do this for every pixel, and you'll end up with a new image that has the edges enhanced.\n",
    "\n",
    "This is perfect for computer vision, because often it's features like edges that distinguish one item for another. And once we move from raw image data to feature data, the amount of information needed is then much less...because you'll just train on the highlighted features.\n",
    "\n",
    "That's the concept of Convolutional Neural Networks. Add some layers to do convolution before you have the dense layers, and then the information going to the dense layers is more focussed, and possibly more accurate.\n",
    "\n",
    "Run the below code -- this is the same neural network as earlier, but this time with Convolutional layers added first. It will take longer, but look at the impact on the accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "C0tFgT1MMKi6",
    "outputId": "5850e18c-3f74-4413-b8ce-b061ff9e44df"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_2 (Conv2D)            (None, 26, 26, 64)        640       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 13, 13, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 11, 11, 64)        36928     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 5, 5, 64)          0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 1600)              0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 20)                32020     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                210       \n",
      "=================================================================\n",
      "Total params: 69,798\n",
      "Trainable params: 69,798\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.5161 - accuracy: 0.8153 - val_loss: 0.3862 - val_accuracy: 0.8601\n",
      "Epoch 2/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.3406 - accuracy: 0.8774 - val_loss: 0.3269 - val_accuracy: 0.8822\n",
      "Epoch 3/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.2945 - accuracy: 0.8950 - val_loss: 0.3115 - val_accuracy: 0.8885\n",
      "Epoch 4/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.2653 - accuracy: 0.9033 - val_loss: 0.3027 - val_accuracy: 0.8916\n",
      "Epoch 5/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.2414 - accuracy: 0.9111 - val_loss: 0.2707 - val_accuracy: 0.9010\n",
      "Epoch 6/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.2232 - accuracy: 0.9178 - val_loss: 0.2778 - val_accuracy: 0.8994\n",
      "Epoch 7/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.2068 - accuracy: 0.9240 - val_loss: 0.2646 - val_accuracy: 0.9027\n",
      "Epoch 8/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1931 - accuracy: 0.9282 - val_loss: 0.2711 - val_accuracy: 0.9016\n",
      "Epoch 9/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1837 - accuracy: 0.9315 - val_loss: 0.2655 - val_accuracy: 0.9066\n",
      "Epoch 10/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1673 - accuracy: 0.9381 - val_loss: 0.2618 - val_accuracy: 0.9089\n",
      "Epoch 11/20\n",
      "1875/1875 [==============================] - 7s 3ms/step - loss: 0.1575 - accuracy: 0.9424 - val_loss: 0.2808 - val_accuracy: 0.9017\n",
      "Epoch 12/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1491 - accuracy: 0.9439 - val_loss: 0.2711 - val_accuracy: 0.9112\n",
      "Epoch 13/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1385 - accuracy: 0.9484 - val_loss: 0.2902 - val_accuracy: 0.9060\n",
      "Epoch 14/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1320 - accuracy: 0.9517 - val_loss: 0.2748 - val_accuracy: 0.9069\n",
      "Epoch 15/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1245 - accuracy: 0.9531 - val_loss: 0.2988 - val_accuracy: 0.9075\n",
      "Epoch 16/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1172 - accuracy: 0.9565 - val_loss: 0.3046 - val_accuracy: 0.9069\n",
      "Epoch 17/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1108 - accuracy: 0.9583 - val_loss: 0.3028 - val_accuracy: 0.9113\n",
      "Epoch 18/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1043 - accuracy: 0.9612 - val_loss: 0.3380 - val_accuracy: 0.9104\n",
      "Epoch 19/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1026 - accuracy: 0.9617 - val_loss: 0.3316 - val_accuracy: 0.9072\n",
      "Epoch 20/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0933 - accuracy: 0.9657 - val_loss: 0.3219 - val_accuracy: 0.9109\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f7a10fc8748>"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "mnist = tf.keras.datasets.fashion_mnist\n",
    "(training_images, training_labels), (val_images, val_labels) = mnist.load_data()\n",
    "training_images=training_images.reshape(60000, 28, 28, 1)\n",
    "training_images=training_images / 255.0\n",
    "val_images=val_images.reshape(10000, 28, 28, 1)\n",
    "val_images=val_images/255.0\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "  tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "  tf.keras.layers.MaxPooling2D(2,2),\n",
    "  tf.keras.layers.Flatten(),\n",
    "  tf.keras.layers.Dense(20, activation='relu'),\n",
    "  tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model.summary()\n",
    "model.fit(training_images, training_labels, validation_data=(val_images, val_labels), epochs=20)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uRLfZ0jt-fQI"
   },
   "source": [
    "It's likely gone up to about 97% on the training data and 91% on the validation data. \n",
    "\n",
    "That's significant, and a step in the right direction!\n",
    "\n",
    "Try running it for more epochs -- say about 100, and explore the results! But while the results might seem really good, the validation results may actually go down, due to something called 'overfitting' which will be discussed later. \n",
    "\n",
    "(In a nutshell, 'overfitting' occurs when the network learns the data from the training set really well, but it's too specialised to only that data, and as a result is less effective at seeing *other* data. For example, if all your life you only saw red shoes, then when you see a red shoe you would be very good at identifying it, but blue suade shoes might confuse you...and you know you should never mess with my blue suede shoes.)\n",
    "\n",
    "Then, look at the code again, and see, step by step how the Convolutions were built:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RaLX5cgI_JDb"
   },
   "source": [
    "Step 1 is to gather the data. You'll notice that there's a bit of a change here in that the training data needed to be reshaped. That's because the first convolution expects a single tensor containing everything, so instead of 60,000 28x28x1 items in a list, we have a single 4D list that is 60,000x28x28x1, and the same for the validation images. If you don't do this, you'll get an error when training as the Convolutions do not recognize the shape. \n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "import tensorflow as tf\n",
    "mnist = tf.keras.datasets.fashion_mnist\n",
    "(training_images, training_labels), (val_images, val_labels) = mnist.load_data()\n",
    "training_images=training_images.reshape(60000, 28, 28, 1)\n",
    "training_images=training_images / 255.0\n",
    "val_images = val_images.reshape(10000, 28, 28, 1)\n",
    "val_images=val_images/255.0\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SS_W_INc_kJQ"
   },
   "source": [
    "Next is to define your model. Now instead of the input layer at the top, you're going to add a Convolution. The parameters are:\n",
    "\n",
    "1. The number of convolutions you want to generate. Purely arbitrary, but good to start with something in the order of 64\n",
    "2. The size of the Convolution, in this case a 3x3 grid\n",
    "3. The activation function to use -- in this case we'll use relu, which you might recall is the equivalent of returning x when x>0, else returning 0\n",
    "4. In the first layer, the shape of the input data.\n",
    "\n",
    "You'll follow the Convolution with a MaxPooling layer which is then designed to compress the image, while maintaining the content of the features that were highlighted by the convolution. By specifying (2,2) for the MaxPooling, the effect is to quarter the size of the image. Without going into too much detail here, the idea is that it creates a 2x2 array of pixels, and picks the biggest one, thus turning 4 pixels into 1. It repeats this across the image, and in so doing halves the number of horizontal, and halves the number of vertical pixels, effectively reducing the image to 25% of its original size.\n",
    "\n",
    "You can call model.summary() to see the size and shape of the network, and you'll notice that after every MaxPooling layer, the image size is reduced in this way. \n",
    "\n",
    "\n",
    "```\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RMorM6daADjA"
   },
   "source": [
    "Add another convolution\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "  tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "  tf.keras.layers.MaxPooling2D(2,2)\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b1-x-kZF4_tC"
   },
   "source": [
    "Now flatten the output. After this you'll just have the same DNN structure as the non convolutional version\n",
    "\n",
    "```\n",
    "  tf.keras.layers.Flatten(),\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qPtqR23uASjX"
   },
   "source": [
    "The same 20 dense layers, and 10 output layers as in the pre-convolution example:\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "  tf.keras.layers.Dense(20, activation='relu'),\n",
    "  tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C0GSsjUhAaSj"
   },
   "source": [
    "Now compile the model, call the fit method to do the training, and evaluate the loss and accuracy from the validation set.\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(training_images, training_labels, validation_data=(val_images, val_labels), epochs=20)\n",
    "```\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8KVPZqgHo5Ux"
   },
   "source": [
    "# Optional Additonal Exercises\n",
    "\n",
    "1. Try editing the convolutions. Change the 32s to either 16 or 64. Explore what impact this will have on accuracy and/or training time.\n",
    "\n",
    "2. Remove the final Convolution. What impact will this have on accuracy or training time?\n",
    "\n",
    "3. How about adding more Convolutions? What impact do you think this will have? Experiment with it.\n",
    "\n",
    "4. Remove all Convolutions but the first. What impact do you think this will have? Experiment with it. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Fashion-MNIST-Convolutions.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}