01-keras-introduction-mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras Sequential API for Multi-class Classification\n",
    "\n",
    "In this example notebook we are given the task of classifying images of handwritten digits using the MNIST dataset.\n",
    "\n",
    "For doing multi-class classification we have an output neuron for each of the possible classes, representing the probability of the input belonging to that class.\n",
    "The final predicted class is the one corresponding to the neuron which has the highest output value. \n",
    "\n",
    "We will build a neural network having 10 output neurons for each of the 10 digits.\n",
    "\n",
    "<img src=\"assets/mnist_cnn.png\"/>\n",
    "\n",
    "Based on:\n",
    "+ https://github.com/ADGEfficiency/teaching-monolith/blob/master/sequences/keras.ipynb\n",
    "+ https://github.com/sempwn/keras-intro\n",
    "+ https://github.com/keras-team/keras/blob/master/examples/mnist_cnn.py\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Recap of the steps to build a model and get predictions with Keras:\n",
    "1. Load Data\n",
    "2. Define Keras Model\n",
    "3. Compile Keras Model\n",
    "4. Fit Keras Model.\n",
    "5. Evaluate Keras Model\n",
    "6. Make Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:16:06.763587Z",
     "start_time": "2019-09-30T15:16:01.236854Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow.keras as keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras.datasets import mnist\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading in the MNIST dataset\n",
    "[MNIST](http://yann.lecun.com/exdb/mnist/) is a classic machine learning dataset (Le Cun post office) known as the \"hello world\" of classification problems.\n",
    "\n",
    "It consists of 70,000 28x28 pixel grayscale images of handwritten numbers, manually classified into the 10 digits, and split into a canonical training set and test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:16:07.106783Z",
     "start_time": "2019-09-30T15:16:06.763587Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 train samples, 10000 test samples\n",
      "training data shape:  (60000, 28, 28) (60000,)\n",
      "test data shape:  (10000, 28, 28) (10000,)\n"
     ]
    }
   ],
   "source": [
    "# input image dimensions\n",
    "(X_train, Y_train), (X_test, Y_test) = mnist.load_data()\n",
    "\n",
    "print('%d train samples, %d test samples'% (X_train.shape[0], X_test.shape[0]))\n",
    "print(\"training data shape: \", X_train.shape, Y_train.shape)\n",
    "print(\"test data shape: \", X_test.shape, Y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:16:07.263781Z",
     "start_time": "2019-09-30T15:16:07.106783Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAOSElEQVR4nO3dfYyU5bnH8d91sERCqwFZXmLJAZtN1JxYutkQIycNJ42NbEyQPzxCtMHEZKtCQmNNDuGYFPUfcnLaauKRhCqBo3UJpij8YSqK9YVEqwNyEETrC9hSCCwYKPiGLtf5Yx/MivvcM8zzzAt7fT/JZGaea+55rgz89pmZe2Zuc3cBGPn+qdUNAGgOwg4EQdiBIAg7EARhB4K4oJk7mzBhgk+bNq2ZuwRC2bdvn44cOWLD1QqF3cyuk/SgpFGSHnH3FanbT5s2TZVKpcguASR0d3fn1up+Gm9moyT9j6Q5kq6UtMDMrqz3/gA0VpHX7DMlve/uH7r7KUnrJM0tpy0AZSsS9ksl/W3I9f3Ztm8ws14zq5hZpb+/v8DuABRRJOzDvQnwrc/euvsqd+929+6Ojo4CuwNQRJGw75c0dcj170s6UKwdAI1SJOxvSOo0s+lmNlrSfEmbymkLQNnqnnpz96/MbLGkZzU49bba3XeX1hmAUhWaZ3f3ZyQ9U1IvABqIj8sCQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EERTl2wGhjp16lSy/uyzzybrL774Yt377uvrS9a7urqS9TvvvDNZ7+npOeeeGo0jOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EwTw7Cvnss8+S9XvvvTe3tm7duuTYjz76KFmfOHFisn799dfn1ubNm5ccu2HDhmT9scceS9bbcZ69UNjNbJ+kE5IGJH3l7t1lNAWgfGUc2f/N3Y+UcD8AGojX7EAQRcPukjab2TYz6x3uBmbWa2YVM6v09/cX3B2AehUN+yx375I0R9IiM/vx2Tdw91Xu3u3u3R0dHQV3B6BehcLu7gey88OSnpI0s4ymAJSv7rCb2Vgz+96Zy5J+KmlXWY0BKFeRd+MnSXrKzM7czxPu/sdSukLb2LhxY7J+zz33JOu7duX//R83blxy7F133ZWs33fffcn62LFjk/WURYsWJevV5unbUd1hd/cPJf2wxF4ANBBTb0AQhB0IgrADQRB2IAjCDgTBV1yD27lzZ7J+4403JuunT59O1h988MHc2u23354cO3r06GS9mtRXZCdPnpwce8UVVyTrW7duraunVuLIDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBMM8+wp04cSJZnzVrVrLu7sn69u3bk/WrrroqWU8ZGBhI1m+55ZZk/cknn8ytPf3008mxqZ+hlqTz8VeXOLIDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBDMs49wK1asSNZPnjyZrPf2Druq19eKzKNXU+2noqst+ZxyySWX1D32fMWRHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCYJ59BPj0009za319fYXu+/777y80/vjx47m1m266KTl28+bNhfb9yiuv5NauvvrqQvd9Pqp6ZDez1WZ22Mx2Ddk23syeM7P3svP0QtsAWq6Wp/FrJF131ralkra4e6ekLdl1AG2satjd/WVJH5+1ea6ktdnltZJuKLkvACWr9w26Se5+UJKy84l5NzSzXjOrmFmlv7+/zt0BKKrh78a7+yp373b37vPxR/qAkaLesB8ysymSlJ0fLq8lAI1Qb9g3SVqYXV4oaWM57QBolKrz7GbWJ2m2pAlmtl/SryStkLTezG6T9FdJ6UW80VCpNdK/+OKLQvd99OjRZH3s2LHJ+qJFi3Jrzz//fHLshRdemKw//vjjyXpXV1duzcySY0eiqmF39wU5pZ+U3AuABuLjskAQhB0IgrADQRB2IAjCDgTBV1xHgNT02ieffFLovtevX5+sP/DAA8n6sWPHcmvjx49Pjn3ttdeS9c7OzmQd38SRHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCYJ59BBgYGMitjRuX/uHf1E89S9Ly5cvraelrc+fOza098cQTybHVvuKKc8ORHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCYJ59BHjnnXdya6k5+FqMGTMmWX/44YeT9fnz5+fWmEdvLo7sQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAE8+zngb179ybr1157bW7t1KlThfY9Z86cZD01jy4xl95Oqh7ZzWy1mR02s11Dti03s7+b2Y7s1NPYNgEUVcvT+DWSrhtm+2/dfUZ2eqbctgCUrWrY3f1lSR83oRcADVTkDbrFZrYze5qf+0NnZtZrZhUzq/T39xfYHYAi6g37Skk/kDRD0kFJv867obuvcvdud+/u6Oioc3cAiqor7O5+yN0H3P20pN9JmlluWwDKVlfYzWzKkKvzJO3Kuy2A9lB1nt3M+iTNljTBzPZL+pWk2WY2Q5JL2ifp5w3sccR76aWXkvXUPLokTZ48Obd29913J8euWbMmWd+wYUOy/tBDDyXr1faP5qkadndfMMzmRxvQC4AG4uOyQBCEHQiCsANBEHYgCMIOBMFXXJtg9+7dyXq1r4maWbK+efPm3Nrll1+eHLtt27Zk/c0330zWP//882Qd7YMjOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EwTx7jb788svc2ttvv50c29XVlaxfcEH6n2HLli3JerW59JQ77rgjWe/r60vW33333br3jebiyA4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQTDPXqOjR4/m1mbMmJEcO2bMmGS92lz11KlTk/WUkydPJutLlixJ1keNGpWsV5unR/vgyA4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQTDPnqk2H93T01P3fb/wwgvJerV5dHdP1l9//fXc2s0335wc+8EHHyTrs2fPTtavueaaZB3to+qR3cymmtmfzGyPme02syXZ9vFm9pyZvZedj2t8uwDqVcvT+K8k/dLdr5B0taRFZnalpKWStrh7p6Qt2XUAbapq2N39oLtvzy6fkLRH0qWS5kpam91sraQbGtUkgOLO6Q06M5sm6UeS/ixpkrsflAb/IEiamDOm18wqZlbp7+8v1i2AutUcdjP7rqQ/SPqFu/+j1nHuvsrdu929u6Ojo54eAZSgprCb2Xc0GPTfu/uGbPMhM5uS1adIOtyYFgGUoerUmw2uF/yopD3u/pshpU2SFkpakZ1vbEiHTXLgwIFkvdrSxSkzZ85M1o8dO5asL1u2LFlfuXLlOfd0xq233pqsP/LII3XfN9pLLfPssyT9TNJbZrYj27ZMgyFfb2a3SfqrpBsb0yKAMlQNu7tvlWQ55Z+U2w6ARuHjskAQhB0IgrADQRB2IAjCDgTBV1wzkyZNStanT5+eW9u7d29y7GWXXZasHz9+PFmvNg8/ceKwn1SWJC1dmv5+0uLFi5P1aj8ljfMHR3YgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIJ59szFF1+crL/66qu5td7e3uTYTZs21dXTGZ2dncl6pVLJrV100UWF9o2RgyM7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgTBPHuNUt9337jxvP7JfATBkR0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgqgadjObamZ/MrM9ZrbbzJZk25eb2d/NbEd26ml8uwDqVcuHar6S9Et3325m35O0zcyey2q/dff/blx7AMpSy/rsByUdzC6fMLM9ki5tdGMAynVOr9nNbJqkH0n6c7ZpsZntNLPVZjYuZ0yvmVXMrNLf31+oWQD1qznsZvZdSX+Q9At3/4eklZJ+IGmGBo/8vx5unLuvcvdud+/u6OgooWUA9agp7Gb2HQ0G/ffuvkGS3P2Quw+4+2lJv5M0s3FtAiiqlnfjTdKjkva4+2+GbJ8y5GbzJO0qvz0AZanl3fhZkn4m6S0z25FtWyZpgZnNkOSS9kn6eUM6BFCKWt6N3yrJhik9U347ABqFT9ABQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCMHdv3s7M+iV9NGTTBElHmtbAuWnX3tq1L4ne6lVmb//s7sP+/ltTw/6tnZtV3L27ZQ0ktGtv7dqXRG/1alZvPI0HgiDsQBCtDvuqFu8/pV17a9e+JHqrV1N6a+lrdgDN0+ojO4AmIexAEC0Ju5ldZ2bvmtn7Zra0FT3kMbN9ZvZWtgx1pcW9rDazw2a2a8i28Wb2nJm9l50Pu8Zei3pri2W8E8uMt/Sxa/Xy501/zW5moyT9RdK1kvZLekPSAnd/u6mN5DCzfZK63b3lH8Awsx9LOinpf939X7Jt/yXpY3dfkf2hHOfu/9EmvS2XdLLVy3hnqxVNGbrMuKQbJN2qFj52ib7+XU143FpxZJ8p6X13/9DdT0laJ2luC/poe+7+sqSPz9o8V9La7PJaDf5nabqc3tqCux909+3Z5ROSziwz3tLHLtFXU7Qi7JdK+tuQ6/vVXuu9u6TNZrbNzHpb3cwwJrn7QWnwP4+kiS3u52xVl/FuprOWGW+bx66e5c+LakXYh1tKqp3m/2a5e5ekOZIWZU9XUZualvFulmGWGW8L9S5/XlQrwr5f0tQh178v6UAL+hiWux/Izg9LekrttxT1oTMr6Gbnh1vcz9faaRnv4ZYZVxs8dq1c/rwVYX9DUqeZTTez0ZLmS9rUgj6+xczGZm+cyMzGSvqp2m8p6k2SFmaXF0ra2MJevqFdlvHOW2ZcLX7sWr78ubs3/SSpR4PvyH8g6T9b0UNOX5dJ+r/stLvVvUnq0+DTui81+IzoNkmXSNoi6b3sfHwb9faYpLck7dRgsKa0qLd/1eBLw52SdmSnnlY/dom+mvK48XFZIAg+QQcEQdiBIAg7EARhB4Ig7EAQhB0IgrADQfw/cJ9KWHd1ZkgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# check one random data point\n",
    "idx = 7777\n",
    "print(Y_train[idx])\n",
    "plt.imshow(X_train[idx], cmap='Greys');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to pre-process the data for Keras:\n",
    "1. Ensure they are float32 for precision;\n",
    "+ Normalize the pixel values (which are naturally between 0 and 255) so that they are all between 0 and 1 instead;\n",
    "+ Flatten the image arrays from $n$x28x28 to $n$x784, so each row of the data is the full \"unrolled\" list of pixels;\n",
    "+ Convert class vectors to binary class matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('uint8')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure they are float32 for precision and normalize \n",
    "x_train = X_train.astype() / \n",
    "x_test = X_test.astype() /"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flatten the image arrays \n",
    "x_train = x_train.reshape()\n",
    "x_test = x_test.reshape()\n",
    "\n",
    "input_shape = # should be 28 * 28 = 784\n",
    "x_train.shape # should be n = 60000, 784"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert class vectors to binary class matrices\n",
    "num_classes = len(set(Y_train))\n",
    "print(num_classes)\n",
    "\n",
    "y_train = keras.utils.to_categorical()\n",
    "y_test = keras.utils.to_categorical()\n",
    "y_train[idx] # check the example number 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the model \n",
    "\n",
    "Add layers accordingly. \n",
    "The first layer in a Sequential model (and only the first, because following layers can do automatic shape inference) needs to receive information about its input shape, specified with the __input_dim__ parameter.\n",
    "\n",
    "+ How do we know the number of layers and their types?\n",
    "\n",
    "This is a very hard question. There are heuristics that we can use and often the best network structure is found through a process of trial and error experimentation. Generally, you need a network large enough to capture the structure of the problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:16:12.151721Z",
     "start_time": "2019-09-30T15:16:12.058122Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create multilayer neural network with more hidden layers.\n",
    "model = \n",
    "\n",
    "# We'll give it two layers of 100 neurons each, with sigmoid activations. \n",
    "model.add()\n",
    "model.add()\n",
    "\n",
    "# Then we will make the output layer go through a softmax activation, the standard for classification.\n",
    "model.add()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compiling the model\n",
    "Before training a model, you need to configure the learning process, which is done via the  `compile`  method. \n",
    "\n",
    "We should also check we've constructed layer by layer: layer order,  output shape of each layer and the  parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:16:22.985592Z",
     "start_time": "2019-09-30T15:16:22.890993Z"
    }
   },
   "outputs": [],
   "source": [
    "model.compile()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 100)               78500     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 100)               10100     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 89,610\n",
      "Trainable params: 89,610\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fitting the model\n",
    "\n",
    "The training process will run for a fixed number of iterations through of the rows in the dataset, that we must specify using the __epochs__ argument.\n",
    "\n",
    "We must also set the number of dataset rows that are considered before the model weights are updated within each epoch, called the batch size and set using the __batch_size__ argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:17:28.420682Z",
     "start_time": "2019-09-30T15:17:20.151779Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "batch_size = \n",
    "epochs = \n",
    "\n",
    "model.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:18:32.187126Z",
     "start_time": "2019-09-30T15:18:31.920929Z"
    }
   },
   "outputs": [],
   "source": [
    "model.evaluate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions\n",
    "Let's take a random digit example to find out how confident the model is at classifying the correct category\n",
    "\n",
    "By calling the __predict_proba()__ function, we will get the probability of the data instance belonging to each class.\n",
    "\n",
    "\n",
    "In the case of a multi-class classification problem like our example here, the softmax activation function is often used on the output layer and the likelihood of the observation for each class is returned as a vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:18:38.427250Z",
     "start_time": "2019-09-30T15:18:35.996081Z"
    }
   },
   "outputs": [],
   "source": [
    "probs = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Choosing a random data from test set and show probabilities for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:18:38.663648Z",
     "start_time": "2019-09-30T15:18:38.427250Z"
    }
   },
   "outputs": [],
   "source": [
    "i = np.random.randint(0,len(X_test))\n",
    "digit = X_test[i]\n",
    "\n",
    "plt.figure(figsize=(10,5));\n",
    "plt.subplot(1,2,1);\n",
    "plt.title('Example of digit: {}'.format(y_test[i]));\n",
    "plt.imshow(digit, cmap='gray');\n",
    "plt.axis('off');\n",
    "\n",
    "\n",
    "plt.subplot(1,2,2);\n",
    "plt.title('Probabilities for each digit class');\n",
    "plt.bar(np.arange(10), probs[i], align='center'); \n",
    "plt.xticks(np.arange(10),np.arange(10).astype(str));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrong predictions\n",
    "\n",
    "Let's look more closely at the predictions on the test data that weren't correct, using the __predict_classes()__ function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:18:39.269240Z",
     "start_time": "2019-09-30T15:18:39.027443Z"
    }
   },
   "outputs": [],
   "source": [
    "predictions = model.predict_classes(x_test, batch_size=32, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:18:39.535437Z",
     "start_time": "2019-09-30T15:18:39.504237Z"
    }
   },
   "outputs": [],
   "source": [
    "inds = np.arange(len(predictions))\n",
    "wrong_results = inds[Y_test!=predictions]\n",
    "len(wrong_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ Choosing a random wrong result from the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:18:42.100806Z",
     "start_time": "2019-09-30T15:18:41.817010Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "i = np.random.choice(wrong_results, size=1)[0]\n",
    "digit = X_test[i]\n",
    "\n",
    "plt.figure(figsize=(10,5));\n",
    "plt.subplot(1,2,1);\n",
    "plt.title('Example of digit: {}'.format(y_test[i]));\n",
    "plt.imshow(digit, cmap='gray');\n",
    "plt.axis('off');\n",
    "\n",
    "\n",
    "plt.subplot(1,2,2);\n",
    "plt.title('Probabilities for each digit class');\n",
    "plt.bar(np.arange(10), probs[i], align='center'); \n",
    "plt.xticks(np.arange(10),np.arange(10).astype(str));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ Plot histogram comparing wrong and correct classifications"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-30T15:18:52.418682Z",
     "start_time": "2019-09-30T15:18:52.151486Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "wrong_probs = np.array([probs[ind][digit] for ind,digit in zip(wrong_results, predictions[wrong_results])])\n",
    "all_probs =  np.array([probs[ind][digit] for ind,digit in zip(np.arange(len(predictions)),predictions)])\n",
    "\n",
    "plt.figure(figsize=(6,6));\n",
    "plt.hist(wrong_probs, alpha=0.5, density=True, label='wrongly-labeled');\n",
    "plt.hist(all_probs, alpha=0.5, density=True, label='all labels');\n",
    "plt.legend();\n",
    "plt.title('Comparison between wrongly and correctly classified labels');\n",
    "plt.xlabel('highest probability');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The current record for MNIST has 99.8% accuracy... So how can you achieve it? \n",
    "\n",
    "Convolutional neural networks have thus far proven to be the best at __computer vision tasks__.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}