05_deep_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/ascarrambad/ml-21-22/blob/main/05_deep_learning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y0a01a7_e-RB"
   },
   "source": [
    "# Machine Learning SP 2021/2022\n",
    "\n",
    "Prof. Cesare Alippi<br>\n",
    "Giorgia Adorni ([`giorgia.adorni@usi.ch`](mailto:giorgia.adorni@usi.ch))<br>\n",
    "Luca Butera ([`luca.butera@usi.ch`](mailto:luca.butera@usi.ch))<br>\n",
    "Matteo Riva ([`matteo.riva@usi.ch`](mailto:matteo.riva@usi.ch))\n",
    "\n",
    "---\n",
    "\n",
    "# Lab 05: Deep Learning\n",
    "\n",
    "In this lab, we are going to focus on some practical aspects of building deep neural networks, in particular CNNs. \n",
    "\n",
    "We will focus on two main tasks: \n",
    "\n",
    "1. Building a classifier for images of numerical digits;\n",
    "3. Building a classifier for hand gestures;\n",
    "\n",
    "Let's get started..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "30RKBWXz_xAE"
   },
   "source": [
    "## MNIST\n",
    "\n",
    "The **Modified National Institute of Standards and Technology database** is a large collection of handwritten digits that is widely used in machine learning as a benchmark for computer vision algorithms.   \n",
    "The dataset consists of 70000 images of handwritted digits. All images are 28 pixels by 28 pixels, in 8-bit grayscale (i.e., each pixel is represented by an integer value in the 0-255 range), and are equally divided into 10 classes.\n",
    "\n",
    "MNIST is usually considered as a multi-class classification problem, where the goal is to map each image to its corresponding class. \n",
    "\n",
    "Although nowadays MNIST is regarded as solved, machine learning practitioners like to joke that while it's true that if something works on MNIST, it may not work in the real world, it is also true that if it **doesn't** work on MNIST, it will surely not work in the real world.\n",
    "\n",
    "\n",
    "First let's import out libraries and define an helper function to plot images.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9fhvZP5SrZl4"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "\n",
    "def plot_sample(imgs, labels, nrows, ncols, resize=None, tograyscale=False):\n",
    "    # create a grid of images\n",
    "    fig, axs = plt.subplots(nrows, ncols, figsize=(4*ncols, 4*nrows))\n",
    "    # take a random sample of images\n",
    "    indices = np.random.choice(len(imgs), size=nrows*ncols, replace=False)\n",
    "    for ax, idx in zip(axs.reshape(-1), indices):\n",
    "        ax.axis('off')\n",
    "        # sample an image\n",
    "        ax.set_title(labels[idx])\n",
    "        im = imgs[idx]\n",
    "        if isinstance(im, np.ndarray):\n",
    "            im = Image.fromarray(im)  \n",
    "        if resize is not None:\n",
    "            im = im.resize(resize)\n",
    "        if tograyscale:\n",
    "            im = im.convert('L')\n",
    "        ax.imshow(im, cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-5CeWKF0szw2"
   },
   "source": [
    "Let's load the MNIST dataset and visualize a sample of digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "trKY2GuMtoBF"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import mnist\n",
    "\n",
    "# Load the data\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "plot_sample(x_train, y_train, 4, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GOftvishuRlz"
   },
   "source": [
    "## Dense classifier\n",
    "Now let's build a neural network with the tools that we have seen so far, i.e., using only dense layers.\n",
    "\n",
    "We start by pre-processing the images and reshaping them as vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IUXGWpOsuueT"
   },
   "outputs": [],
   "source": [
    "# Reshape to vectors\n",
    "x_train = x_train.reshape(-1, 28 * 28)  # shape: (60000, 784)\n",
    "x_test = x_test.reshape(-1, 28 * 28)    # shape: (10000, 784)\n",
    "\n",
    "# Normalize to 0-1 range\n",
    "x_train = x_train / 255.\n",
    "x_test = x_test / 255."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T-T8mzp9u72X"
   },
   "source": [
    "We also have to pre-process our targets in order to perform multi-class classification. We will use **one-hot encoding** to represent our numerical labels (0-9) as sparse binary vectors. For instance, the one-hot encoding of label 3 will be $[0, 0, 0, 1, 0 ,0 ,0, 0, 0, 0]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zD-HkP-evib0"
   },
   "outputs": [],
   "source": [
    "# Pre-process targets\n",
    "from tensorflow.keras import utils\n",
    "n_classes = 10\n",
    "y_train = utils.to_categorical(y_train, n_classes)\n",
    "y_test = utils.to_categorical(y_test, n_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QfoJd8iKvi2l"
   },
   "source": [
    "Now we build a neural classifier using the same tools that we saw in the previous lab. Remember that we reshaped our inputs to be vectors, so we are in the same familiar setting as always.\n",
    "\n",
    "However, this time we will be dealing with multi-class classification, which means that our output layer will have 10 possible outputs instead of a single one.\n",
    "Moreover, the sigmoid activation function that we used in our previous binary classifiers will be replaced by the normalized **softmax** function, which will give us a **probability distribution** over the possible labels:\n",
    "\n",
    "$$\n",
    "\\sigma(z)_i = \\frac{e^{z_i}}{\\sum_{j=1}^{K} e^{z_j}}\n",
    "$$\n",
    "\n",
    "where $K$ is the number of classes that we have. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iYsaysq4v_mO"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras import optimizers\n",
    "\n",
    "# Build model\n",
    "model = Sequential()\n",
    "model.add(Dense(128, activation='tanh', input_shape=x_train.shape[1:]))\n",
    "model.add(Dense(128, activation='tanh'))\n",
    "model.add(Dense(n_classes, activation='softmax'))\n",
    "\n",
    "# Store number of parameters of the model\n",
    "fcnn_params = model.count_params()\n",
    "\n",
    "# Compile model\n",
    "model.compile(optimizer=optimizers.Adam(learning_rate=0.001), \n",
    "              loss='categorical_crossentropy', \n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wz8ikRFXwKn7"
   },
   "source": [
    "We can now train and evaluate the model using Keras' `fit` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NUDsIDiRwP_S"
   },
   "outputs": [],
   "source": [
    "# Train model\n",
    "batch_size = 32\n",
    "epochs = 10\n",
    "model.fit(x_train, \n",
    "          y_train, \n",
    "          shuffle=True,  # True by default\n",
    "          batch_size=batch_size, \n",
    "          epochs=epochs, \n",
    "          validation_split=0.1)\n",
    "\n",
    "# Evaluate model\n",
    "scores = model.evaluate(x_test, y_test)\n",
    "print('Test loss: {} - Accuracy: {}'.format(*scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1xNoK7dVwYWB"
   },
   "source": [
    "Pretty good, hu? Let's look closer at some of the issues of this model:\n",
    "\n",
    "1. A lot of parameters for a pretty shallow net.\n",
    "2. It sees the image as a vector...\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sut8Ugzr-XPm"
   },
   "outputs": [],
   "source": [
    "# Take the test data and shift its contents to the right by p pixels\n",
    "p = 3\n",
    "x_test_roll = np.roll(x_test.reshape(-1, 28, 28), p, axis=-1)\n",
    "plt.subplot(121)\n",
    "plt.imshow(x_test[0].reshape(28, 28), cmap='gray')\n",
    "plt.subplot(122)\n",
    "plt.imshow(x_test_roll[0], cmap='gray')\n",
    "\n",
    "# Evaluate the model on the shifted data\n",
    "x_test_roll = x_test_roll.reshape(-1, 28 * 28)\n",
    "scores = model.evaluate(x_test_roll, y_test)\n",
    "print('Test loss: {} - Accuracy: {}'.format(*scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Lo0ou80gBDuS"
   },
   "source": [
    "## Convolutional neural networks\n",
    "\n",
    "CNNs were first introduced by Kunihiko Fukushima in 1980, and were later popularized by Y. LeCun, when he successfully applied backpropagation to train CNNs on MNIST.\n",
    "\n",
    "In CNNs, we use our **prior knowledge** about the problem (i.e., the data are images) to **regularize** the network, imposing that neurons **share** some weights (i.e., the convolutional kernels).\n",
    "\n",
    "![alt text](https://upload.wikimedia.org/wikipedia/commons/6/63/Typical_cnn.png)\n",
    "\n",
    "Let's re-build our classifier from scratch, using convolutional layers instead of fully connected ones. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UOD-ivnECrKZ"
   },
   "outputs": [],
   "source": [
    "# Load the data\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6di-USScCxa6"
   },
   "source": [
    "We still normalize the data to the 0-1 range, but this time we **do not reshape** the images into vectors. \n",
    "Instead, we add a **new dimension** which explicitly represents the different channels of our images. In the case of MNIST, we only have one 8-bit channel, so we only need to add a \"fake\" dimension at the end of our data in order to have a 4D tensor of shape `(None, 28, 28, 1)` (`None` stands for batch size). \n",
    "\n",
    "If we had RGB images, they would be represented as tensors of shape `(None, 28, 28, 3)`, where the last dimension represent the three colors (red, green, blue).\n",
    "\n",
    "We also one-hot encode the labels as we did before.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "x0cYp1r-DbzP"
   },
   "outputs": [],
   "source": [
    "# Normalize to 0-1 range\n",
    "x_train = x_train / 255.\n",
    "x_test = x_test / 255.\n",
    "\n",
    "# Add channels dimension\n",
    "x_train = x_train[..., None]\n",
    "x_test = x_test[..., None]\n",
    "\n",
    "# Pre-process targets\n",
    "n_classes = 10\n",
    "y_train = utils.to_categorical(y_train, n_classes)\n",
    "y_test = utils.to_categorical(y_test, n_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k9fPRpDhDd0v"
   },
   "source": [
    "Let's see how to build a ConvNet with Keras. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fFjHVNQwDrhE"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D, AveragePooling2D, Flatten, Dropout\n",
    "\n",
    "# Build model\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
    "model.add(MaxPooling2D(pool_size=(4, 4)))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "# Store number of parameters of the model\n",
    "cnn_params = model.count_params()\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer=optimizers.Adam(lr=0.001), \n",
    "              loss='categorical_crossentropy',  \n",
    "              metrics=['accuracy'])\n",
    "model.summary()\n",
    "\n",
    "print(\"\\nFCNN params: {:,} - CNN params: {:,}\".format(fcnn_params, cnn_params))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FUi12s7ID0mU"
   },
   "source": [
    "To train and evaluate the model, we do exactly as we did before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Qeak03BfD86z"
   },
   "outputs": [],
   "source": [
    "# Train model\n",
    "batch_size = 32\n",
    "epochs = 10\n",
    "model.fit(x_train, \n",
    "          y_train, \n",
    "          batch_size=batch_size, \n",
    "          epochs=epochs, \n",
    "          validation_split=0.1)\n",
    "\n",
    "# Evaluate model\n",
    "scores = model.evaluate(x_test, y_test)\n",
    "print('Test loss: {} - Accuracy: {}'.format(*scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rKwgS3cZJ85R"
   },
   "source": [
    "Now let's see if the CNN is actually more robust to translations w.r.t. the dense net. We can run the same test as before, by shifting images to the right and evaluating the performance on the shifted test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1YrDP_R0IlTz"
   },
   "outputs": [],
   "source": [
    "# Take the test data and shift its contents to the right by p pixels\n",
    "p = 3\n",
    "x_test_roll = np.roll(x_test, p, axis=2)\n",
    "plt.subplot(121)\n",
    "plt.imshow(x_test[0, ..., 0], cmap='gray')\n",
    "plt.subplot(122)\n",
    "plt.imshow(x_test_roll[0, ..., 0], cmap='gray');\n",
    "\n",
    "# Evaluate the model on the shifted data\n",
    "scores = model.evaluate(x_test_roll, y_test)\n",
    "print('Test loss: {} - Accuracy: {}'.format(*scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t0qYCKIirw0x"
   },
   "source": [
    "## Rock, Paper, Scissors\n",
    "\n",
    "(Inspired by the [lecture](https://github.com/alessandro-giusti/rock-paper-scissors) of Alessandro Giusti)\n",
    "\n",
    "In this part, we will try to solve a much more complex problem. We will use CNNs to classify hand gestures used to play rock, paper, scissors.\n",
    "\n",
    "We will use the data collected by students through a bot. Since these data are much more eterogenous compared to MNIST, we will need some preprocessing before feeding them to a model.\n",
    "\n",
    "Let's upload the data to Colab.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LywaOJonaFME"
   },
   "outputs": [],
   "source": [
    "!wget https://drive.switch.ch/index.php/s/L1vLauf3XgsdSIC/download -O data.zip  # https://drive.switch.ch/index.php/s/UuLqhcUVXcaq9L3/download bigger dataset from https://github.com/alessandro-giusti/rock-paper-scissors\n",
    "!unzip data.zip -d ./data > /dev/null 2>&1\n",
    "!apt-get install tree > /dev/null 2>&1\n",
    "!tree -d data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vcEmv6gmGA11"
   },
   "source": [
    "Ok, now we have the data. Let's define a function to load them and then let's try to see how they look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jtzqez7TZ95P"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "def load_imgs(path, folders):\n",
    "    imgs = []\n",
    "    labels = []\n",
    "    n_imgs = 0\n",
    "    for c in folders:\n",
    "        # iterate over all the files in the folder\n",
    "        for f in os.listdir(os.path.join(path, c)):\n",
    "            if not f.endswith('.jpg'):\n",
    "                continue\n",
    "            # load the image (here you might want to resize the img to save memory)\n",
    "            im = Image.open(os.path.join(path, c, f)).copy()\n",
    "            imgs.append(im)\n",
    "            labels.append(c)\n",
    "        print('Loaded {} images of class {}'.format(len(imgs) - n_imgs, c))\n",
    "        n_imgs = len(imgs)\n",
    "    print('Loaded {} images total.'.format(n_imgs))\n",
    "    return imgs, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Y-HEeKifZ95R"
   },
   "outputs": [],
   "source": [
    "imgs, labels = load_imgs('data', ['rock', 'paper', 'scissors'])\n",
    "# imgs, labels = load_imgs('data_big', ['rock', 'paper', 'scissors'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YdHe9dbqGU5N"
   },
   "source": [
    "We can use the function that we defined in the previous part to visualize the pictures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UK-V2cEhZ95V"
   },
   "outputs": [],
   "source": [
    "plot_sample(imgs, labels, 5, 5, resize=(64, 64), tograyscale=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rz9UwxtZGlsA"
   },
   "source": [
    "Before starting to work on them, we want to select the image size that we will use for training.\n",
    "\n",
    "Let's go back to the previous cell and change the image size until we find a good compromise between input size and sharpness of the image.\n",
    "\n",
    "Once done, we can build our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "m9A2V7v0Z95X"
   },
   "outputs": [],
   "source": [
    "# map class -> idx\n",
    "label_to_idx = {\n",
    "    'rock':0,\n",
    "    'paper':1,\n",
    "    'scissors':2\n",
    "}\n",
    "\n",
    "idx_to_label = {\n",
    "    0:'rock',\n",
    "    1:'paper',\n",
    "    2:'scissors'\n",
    "}\n",
    "\n",
    "def make_dataset(imgs, labels, label_map, img_size, rgb=True, keepdim=True):\n",
    "    x = []\n",
    "    y = []\n",
    "    n_classes = len(list(label_map.keys()))\n",
    "    for im, l in zip(imgs, labels):\n",
    "        # preprocess img\n",
    "        x_i = im.resize(img_size)\n",
    "        if not rgb:\n",
    "            x_i = x_i.convert('L')\n",
    "        x_i = np.asarray(x_i)\n",
    "        if not keepdim:\n",
    "            x_i = x_i.reshape(-1)\n",
    "        \n",
    "        # encode label\n",
    "        y_i = np.zeros(n_classes)\n",
    "        y_i[label_map[l]] = 1.\n",
    "        \n",
    "        x.append(x_i)\n",
    "        y.append(y_i)\n",
    "    return np.array(x).astype('float32'), np.array(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kl9l8sIDZ95Z"
   },
   "outputs": [],
   "source": [
    "x, y = make_dataset(imgs, labels, label_to_idx, (64,64), rgb=True, keepdim=True)\n",
    "print('x shape: {}, y shape:{}'.format(x.shape, y.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mQ4jORAoZ95b"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "np.random.seed(5674)\n",
    "\n",
    "train_idx, test_idx = train_test_split(np.arange(x.shape[0]), test_size=0.25, shuffle=True, stratify=y)\n",
    "train_idx, val_idx = train_test_split(train_idx, test_size=0.2, shuffle=True, stratify=y[train_idx])\n",
    "\n",
    "x_train, y_train = x[train_idx]/255., y[train_idx]\n",
    "x_val, y_val = x[val_idx]/255., y[val_idx]\n",
    "x_test, y_test = x[test_idx]/255., y[test_idx]\n",
    "\n",
    "print('Training, validation, test samples: {}, {}, {}'.format(len(x_train), len(x_val), len(x_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nZcXRODYZ95k"
   },
   "outputs": [],
   "source": [
    "# Define the network\n",
    "classifier = Sequential()\n",
    "classifier.add(Conv2D(16, (3,3), activation='relu', padding='same', input_shape=x_train.shape[1:]))\n",
    "classifier.add(MaxPooling2D((2,2)))\n",
    "classifier.add(Conv2D(16, (3,3), activation='relu', padding='same'))\n",
    "classifier.add(MaxPooling2D((2,2)))\n",
    "classifier.add(Conv2D(32, (3,3), activation='relu', padding='same'))\n",
    "classifier.add(AveragePooling2D((4, 4)))\n",
    "classifier.add(Flatten())\n",
    "classifier.add(Dense(128, activation='relu'))\n",
    "classifier.add(Dropout(0.25))\n",
    "classifier.add(Dense(3, activation='softmax'))\n",
    "\n",
    "classifier.compile(optimizer=optimizers.Adam(learning_rate=0.001),\n",
    "                   loss='categorical_crossentropy',                   \n",
    "                   metrics=['acc'],\n",
    "                  )\n",
    "classifier.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Q566macP-WjK"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "batch_size = 16\n",
    "\n",
    "es = EarlyStopping(monitor='val_loss', patience=50, restore_best_weights=True)\n",
    "\n",
    "history = classifier.fit(x_train, \n",
    "                         y_train, \n",
    "                         batch_size=batch_size, \n",
    "                         epochs=500, \n",
    "                         validation_data=(x_val, y_val),\n",
    "                         verbose=1,\n",
    "                         callbacks=[es])\n",
    "\n",
    "scores = classifier.evaluate(x_test, y_test)\n",
    "\n",
    "print('Test loss: {} - Accuracy: {}'.format(*scores))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RlaFuYuuAGo4"
   },
   "outputs": [],
   "source": [
    "def plot_history(history):\n",
    "    plt.figure(figsize=(15, 5))\n",
    "    plt.subplot(121)\n",
    "\n",
    "    # Auxiliary info and funcs\n",
    "    best_epoch = np.argmin(history.history['val_loss'])\n",
    "    epochs = history.epoch\n",
    "    smooth = lambda y: np.polyval(np.polyfit(epochs, y, deg=5), epochs)\n",
    "\n",
    "    # Plot training & validation accuracy values\n",
    "    plt.plot(smooth(history.history['loss']), c='C0', alpha=0.7, lw=3)\n",
    "    plt.plot(smooth(history.history['val_loss']), c='C1', alpha=0.7, lw=3)\n",
    "    plt.axvline(best_epoch, label='best_epoch', c='k', ls='--', alpha=0.3)\n",
    "    # Empirical values\n",
    "    plt.plot(history.history['loss'], label='train_loss', c='C0')\n",
    "    plt.plot(history.history['val_loss'], label='val_loss', c='C1')\n",
    "    plt.ylabel('loss')\n",
    "    plt.xlabel('epoch')\n",
    "    plt.legend()\n",
    "\n",
    "    plt.subplot(122)\n",
    "    # Plot training & validation accuracy values\n",
    "    plt.plot(smooth(history.history['acc']), c='C0', alpha=0.7, lw=3)\n",
    "    plt.plot(smooth(history.history['val_acc']), c='C1', alpha=0.7, lw=3)\n",
    "    plt.axvline(best_epoch, label='best_epoch', c='k', ls='--', alpha=0.3)\n",
    "    # Empirical values\n",
    "    plt.plot(history.history['acc'], label='train_accuracy', c='C0')\n",
    "    plt.plot(history.history['val_acc'], label='val_accuracy', c='C1')\n",
    "    plt.ylabel('accuracy')\n",
    "    plt.xlabel('epoch')\n",
    "    plt.legend()\n",
    "\n",
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zAUQC1brhzJ-"
   },
   "source": [
    "### Data augmentation\n",
    "\n",
    "We can manipolate the images to create synthetic data:\n",
    "\n",
    "Pros:\n",
    "- Train the network to be resilient to noise and perturbation in the image\n",
    "- More data...\n",
    "\n",
    "Cons:\n",
    "- ... not really"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MP2yTSW4Z95p"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "\n",
    "train_gen = ImageDataGenerator(width_shift_range=0.15,    # horizontal translation\n",
    "                               height_shift_range=0.15,   # vertical translation\n",
    "                               channel_shift_range=0.3,   # random channel shifts\n",
    "                               rotation_range=360,        # rotation\n",
    "                               zoom_range=0.3,            # zoom in/out randomly\n",
    "                               shear_range=15,            # deformation\n",
    "                              )\n",
    "\n",
    "val_gen = ImageDataGenerator()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0YrE4BAfiz24"
   },
   "source": [
    "Let's visualize the transformed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JceYXfAmZ95r"
   },
   "outputs": [],
   "source": [
    "def plot_gen_sample(gen, n_cols=5, n_rows=4):\n",
    "    fig, axs = plt.subplots(n_rows, n_cols, figsize=(4*n_cols, 4*n_rows))\n",
    "    batch = next(gen)[0]\n",
    "    for ax, im in zip(axs.reshape(-1), batch):\n",
    "        ax.axis('off')\n",
    "        ax.imshow(im)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "shEEz1-1Z95t"
   },
   "outputs": [],
   "source": [
    "nr, nc = 5, 5\n",
    "\n",
    "train_loader = train_gen.flow(x_train, y_train, batch_size=nr*nc)\n",
    "plot_gen_sample(train_loader, nr, nc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "asSihvEU509x"
   },
   "source": [
    "Now let's train the model again using the generators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qzL5RbNwZ95u"
   },
   "outputs": [],
   "source": [
    "# Define the network\n",
    "classifier = Sequential()\n",
    "classifier.add(Conv2D(16, (3,3), activation='relu', padding='same', input_shape=x_train.shape[1:]))\n",
    "classifier.add(MaxPooling2D((2,2)))\n",
    "classifier.add(Conv2D(16, (3,3), activation='relu', padding='same'))\n",
    "classifier.add(MaxPooling2D((2,2)))\n",
    "classifier.add(Conv2D(32, (3,3), activation='relu', padding='same'))\n",
    "classifier.add(AveragePooling2D((4,4)))\n",
    "classifier.add(Flatten())\n",
    "classifier.add(Dense(128, activation='relu'))\n",
    "classifier.add(Dropout(0.25))\n",
    "classifier.add(Dense(3, activation='softmax'))\n",
    "\n",
    "classifier.compile(optimizer=optimizers.Adam(learning_rate=0.001), \n",
    "                   loss='categorical_crossentropy',\n",
    "                   metrics=['acc'],\n",
    "                  )\n",
    "classifier.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VCIwmwG6Z95w"
   },
   "outputs": [],
   "source": [
    "batch_size = 16\n",
    "\n",
    "es = EarlyStopping(monitor='val_loss', patience=200, restore_best_weights=True)\n",
    "\n",
    "train_loader = train_gen.flow(x_train, y_train, batch_size=batch_size)\n",
    "val_loader = val_gen.flow(x_val, y_val, batch_size=x_val.shape[0])\n",
    "\n",
    "history = classifier.fit(train_loader,\n",
    "                         steps_per_epoch=x_train.shape[0]//batch_size,\n",
    "                         epochs=2000, \n",
    "                         validation_data=val_loader,\n",
    "                         validation_steps=1,\n",
    "                         callbacks=[es])\n",
    "\n",
    "scores = classifier.evaluate(x_test, y_test)\n",
    "\n",
    "print('Test loss: {} - Accuracy: {}'.format(*scores))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dw1bHM2RZ958"
   },
   "outputs": [],
   "source": [
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8GjaS1xKYCNz"
   },
   "source": [
    "Let's check the misclassified examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8g5UhcKSEoy4"
   },
   "outputs": [],
   "source": [
    "def plot_prediction(x, y, y_pred, class_map):\n",
    "    idxs = list(range(y.shape[1]))\n",
    "    for i in range(x.shape[0]):\n",
    "        plt.subplot(121)\n",
    "        plt.axis('off')\n",
    "        plt.imshow(x[i])\n",
    "        plt.subplot(122)\n",
    "        plt.bar(idxs, y_pred[i], color=['g' if i else 'r' for i in y[i]])\n",
    "        plt.ylim(0., 1.)\n",
    "        plt.xticks(idxs, [class_map[c] for c in idxs])\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BD_ao_FWZ_vK"
   },
   "outputs": [],
   "source": [
    "n_samples = 10\n",
    "\n",
    "# Choose among the misclassified test samples\n",
    "err = np.argmax(classifier.predict(x_test), axis=1) != np.argmax(y_test, axis=1)\n",
    "idxs = np.argwhere(err).ravel()\n",
    "\n",
    "# Choose from the whole test set\n",
    "# idxs = np.arange(len(x_test))\n",
    "\n",
    "idx = np.random.choice(idxs, n_samples, replace=False)\n",
    "\n",
    "y_pred = classifier.predict(x_test[idx])\n",
    "\n",
    "plot_prediction(x_test[idx], y_test[idx], y_pred, idx_to_label)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "machine_shape": "hm",
   "name": "05_deep_learning.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}