CNNIntro.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run clone_git_on_colab.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from course_settings import set_tf_nthreads\n",
    "set_tf_nthreads(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Convolutional neural networks\n",
    "\n",
    "## ImageNet and the \"Deep Learning\" Hype\n",
    "A large portion of the current hype around neural networks has been caused by successes in image recognition, driven by the [ImageNet](http://www.image-net.org/) challenge/dataset. Basically all successful algorithms for computer vision nowadays use in some way Convolutional neural networks in some way.\n",
    "\n",
    "![imagenet](figures/imagenet.webp)\n",
    "\n",
    "(Image from https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world/)\n",
    "\n",
    "## The convolution operation\n",
    "\n",
    "Convolutional neural networks make use of the convolution operation. They are mostly used for processing image data with 2D discrete convolutions:\n",
    "\n",
    "![convolution](figures/convolution.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Let's try it out\n",
    "Can you guess what the output image for convolutional kernel in the picture above will look like?\n",
    "\n",
    "It's the Sobel operator (https://en.wikipedia.org/wiki/Sobel_operator) that can be used for edge detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "sobel_x = np.array([\n",
    "    [-1, 0, 1],\n",
    "    [-2, 0, 2],\n",
    "    [-1, 0, 1]\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A nice library for loading images into python arrays is pillow (https://pillow.readthedocs.io/en/stable/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import PIL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grumpy_pil = PIL.Image.open(\"figures/grumpy.jpg\")\n",
    "grumpy_pil.thumbnail((200, 200))\n",
    "img_grumpy = np.array(grumpy_pil)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(img_grumpy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_grumpy_grey = img_grumpy.mean(axis=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "plt.imshow(img_grumpy_grey, cmap=\"Greys_r\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We could use `scipy.signal` to perform the convolution operation but to understand how the convolution works, let's quickly implement it by manually scanning over the image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def convolve(input_img, kernel):\n",
    "    # we will do a \"valid\" convolution\n",
    "    # that means the output will be 2 pixels smaller in both directions than the input\n",
    "    output_img = np.empty(shape=(input_img.shape[0]-2, input_img.shape[1]-2))\n",
    "    for j in range(output_img.shape[1]):\n",
    "        for i in range(output_img.shape[0]):\n",
    "            output_img[i][j] = np.sum(kernel * input_img[i:i+kernel.shape[0], j:j+kernel.shape[1]])\n",
    "    return output_img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "plt.imshow(convolve(img_grumpy_grey, sobel_x), cmap=\"Greys_r\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(abs(convolve(img_grumpy_grey, sobel_x)), cmap=\"Greys_r\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Vertical edges got highlighted! In a very simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "test_img = np.array([\n",
    "    [0, 0, 0, 0, 1, 1, 1, 1],\n",
    "    [0, 0, 0, 0, 1, 1, 1, 1],\n",
    "    [0, 0, 0, 0, 1, 1, 1, 1],\n",
    "    [0, 0, 0, 0, 1, 1, 1, 1],\n",
    "    [0, 0, 0, 0, 0, 0, 0, 0],\n",
    "    [0, 0, 0, 0, 0, 0, 0, 0],\n",
    "    [0, 0, 0, 0, 0, 0, 0, 0],\n",
    "    [0, 0, 0, 0, 0, 0, 0, 0],\n",
    "    [1, 1, 1, 1, 0, 0, 0, 0],\n",
    "    [1, 1, 1, 1, 0, 0, 0, 0],\n",
    "    [1, 1, 1, 1, 0, 0, 0, 0],\n",
    "    [1, 1, 1, 1, 0, 0, 0, 0],\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "convolve(test_img, sobel_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The transposed filter will highlight horizontal edges:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "sobel_y = sobel_x.T\n",
    "sobel_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "convolve(test_img, sobel_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "plt.imshow(convolve(img_grumpy_grey, sobel_y), cmap=\"Greys_r\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "And if we quadratically add the pictures above we get a nice highlighting of all edges:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "plt.imshow(\n",
    "    np.sqrt(\n",
    "        convolve(img_grumpy_grey, sobel_x)**2\n",
    "        + convolve(img_grumpy_grey, sobel_y)**2\n",
    "    ),\n",
    "    cmap=\"Greys_r\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also do this for all 3 color channels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = np.stack(\n",
    "    [\n",
    "        np.sqrt(\n",
    "            convolve(img_grumpy[..., i], sobel_x)**2\n",
    "            + convolve(img_grumpy[..., i], sobel_y)**2\n",
    "        )\n",
    "        for i in range(3)\n",
    "    ],\n",
    "    axis=-1\n",
    ")\n",
    "plt.imshow(img / img.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Application within a neural network\n",
    "\n",
    "You might imagine that such filters help a lot when processing images and e.g. trying to learn what they show. What a convolutional neural network (CNN) does is instead of using hand-designed filters have them as trainable parameters. We can then have layers with arbitrary many input features (each of them an image) and arbitrary many output features (each of them as well an image) by essentially sliding a neural network over them:\n",
    "\n",
    "![cnn_layer](figures/cnn_layer.png)\n",
    "\n",
    "The top row corresponds to the input features - before the first layer these are typically the 3 colors, red, green, blue and the bottom row corresponds to the output features. For each output feature, the neutral network will learn one convolutional kernel for each input feature. So the black lines in the graphic above correspond to the trainable weights.\n",
    "\n",
    "[animated version](https://homepages.physik.uni-muenchen.de/~Nikolai.Hartmann/cnn_anim.svg)\n",
    "\n",
    "In Addition to applying the filter one can (and typically will) also apply an activation function.\n",
    "\n",
    "The big advantage of convolutional layers for image processing is that the same weights are used at different places of the input image for processing. This allows to detect similar features, no matter where they occur in the image - a property sometimes referred to as \"translational invariance\".\n",
    "\n",
    "Below is another nice animated visualisation from the [CS231 course](http://cs231n.github.io/) -- here with a convolution of \"stride 2\", meaning the filters move in steps of 2 pixels over the image. The blue boxes are the inputs (padded with zeros), the red boxes are two filters and the two green boxes corresponds to the output for each of the two filters. (Note that the 3 channels are summed over for the output.)\n",
    "\n",
    "http://cs231n.github.io/assets/conv-demo/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Pooling layers\n",
    "\n",
    "In addition to convolutional layers, CNNs will typically perform some kind of downsampling (also called pooling or subsampling) in between. This has several reasons:\n",
    "\n",
    "- The region of the orginal image that the neural network can \"see\" will increase. This can help to make use of correlations between more distant areas within an image\n",
    "- The amount of computation decreases (smaller images further down in the network) - more depth and/or width of the network can be increased\n",
    "- Especially for classification problems the total NN output should be a few numbers, e.g. indicating in which category an image falls. Successively downsampling the image within the network will help to keep the number of parameters in the last layers small.\n",
    "\n",
    "Pooling typically takes the maximum, average or sum over a fixed sliding window. An example of \"Max\" pooling with a 2x2 window:\n",
    "\n",
    "![max_pooling](figures/Max_pooling.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Full CNN Architecture for image classification\n",
    "\n",
    "A typical CNN architecture for image classification consists of several convolutional layers with pooling layers in between and a simple fully-connected network as a last step:\n",
    "\n",
    "![max_pooling](figures/Typical_cnn.png)\n",
    "\n",
    "The fully connected network either has all output pixels of the last convolutional/pooling layer as input (\"flatten\") or uses the global average of each output feature of the last convolutional/pooling layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Let's try it out - CIFAR10\n",
    "\n",
    "We will use the [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset for trying CNNs. The dataset consists out of 60k 32x32 images, labelled for 10 categories. Our goal is to predict the category from processing the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import cifar10\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copy from /large_tmp/ if at CIP (otherwise keras will just download it from the web)\n",
    "import os\n",
    "from shutil import copyfile\n",
    "path_cifar10_cip = \"/large_tmp/LMU_DA_ML/cifar-10-batches-py.tar.gz\"\n",
    "path_cifar10_user = os.path.expanduser(\"~/.keras/datasets/cifar-10-batches-py.tar.gz\")\n",
    "if not os.path.exists(path_cifar10_user) and os.path.exists(path_cifar10_cip):\n",
    "    os.makedirs(os.path.expanduser(\"~/.keras/datasets\"), exist_ok=True)\n",
    "    copyfile(path_cifar10_cip, path_cifar10_user)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "`data` will contain 2 tuples of X, y for training (50k) and testing (10k) data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "data = cifar10.load_data()\n",
    "x_train, y_train = data[0]\n",
    "x_test, y_test = data[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Pictures are arranged as arrays with indices (x, y, color):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "x_train[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The target vector consists of label indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "y_train[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The labels are as follows (in that order)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "labels = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\", \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's look at a few random pictures of cats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def show_random_pictures(x):\n",
    "    pictures = x\n",
    "    rnd_idx = np.random.permutation(len(pictures))\n",
    "    fig, axs = plt.subplots(nrows=3, ncols=10, figsize=(20,6))\n",
    "    for i, ax in enumerate(axs.reshape(-1)):\n",
    "        ax.imshow(pictures[rnd_idx[i]])\n",
    "        ax.set_axis_off()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "show_random_pictures(x_train[y_train.reshape(-1) == labels.index('cat')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Define the NN\n",
    "Keras has all the components we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's build a model similar to [keras/examples/cifar10_cnn.py](https://github.com/keras-team/keras/blob/2.0.0/examples/cifar10_cnn.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    # lets start with 2 convolutional layers with kernel size 3, 32 output features each\n",
    "    Conv2D(32, 3, activation=\"relu\", input_shape=(32, 32, 3)),\n",
    "    Conv2D(32, 3, activation=\"relu\"),\n",
    "    # Max pooling (default window size is 2x2)\n",
    "    MaxPooling2D(),\n",
    "    # Add a 25% Dropout (randomly drops 25% of inputs during training)\n",
    "    Dropout(0.25),\n",
    "    # another block of 2 CNN layers with 64 output features each, followed by MaxPooling and Dropout\n",
    "    Conv2D(64, 3, activation=\"relu\"),\n",
    "    Conv2D(64, 3, activation=\"relu\"),\n",
    "    MaxPooling2D(),\n",
    "    Dropout(0.25),\n",
    "    # Flatten (reshape) all output pixels of all features into 1D array\n",
    "    Flatten(),\n",
    "    # add a fully connected final hidden layer with 512 neurons, followed by 50% dropout\n",
    "    Dense(512, activation=\"relu\"),\n",
    "    Dropout(0.5),\n",
    "    # 10 output neurons that are supposed to represent the 10 categories\n",
    "    # and output the probabilities of the image belonging to each category\n",
    "    Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We use the **softmax** activation function\n",
    "\n",
    "$$ \\sigma(\\textbf{z})_i = \\frac{e^{z_i}}{\\sum\\limits_{j=0}^{K} e^{z_j}} $$\n",
    "\n",
    "for $K=10$ output features. This will lead to output values that sum to 1 for all categories - indicating the probability to belong to a specific category.\n",
    "\n",
    "For a multi-classification problem the figure of merit to minimize is the categorical cross entropy\n",
    "\n",
    "$$ L = - \\frac{1}{N}\\sum\\limits_{i=0}^{N} \\sum\\limits_{k=0}^{K} y_{ik}\\log{\\hat{y}_{ik}} $$\n",
    "\n",
    "which is a generalized form of the cross entropy (you can easily check it's equivalent to the binary cross entropy for $K=2$)\n",
    "\n",
    "We will again use the *Adam* optimizer and tell keras to monitor the *accuracy* (fraction of correctly classified examples) during the training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "For preprocessing, we will simply divide by 255 (r, g, b values are between 0 and 255)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def preprocess(x):\n",
    "    return x / 255."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Since the NN will output 10 values, we \"one-hot-encode\" our target vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "y_train_onehot = keras.utils.to_categorical(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "y_train[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "y_train_onehot[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Now we are ready to start the training!\n",
    "\n",
    "But first we need to create a [tensorflow Dataset](https://www.tensorflow.org/guide/data) on CPU memory to avoid that the whole training data is put onto GPU (we only have 2GB of GPU memory available on the CIP nodes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.device('/device:CPU:0'):\n",
    "    ds = tf.data.Dataset.from_tensor_slices(\n",
    "        (\n",
    "            preprocess(x_train), # input\n",
    "            y_train_onehot, # target\n",
    "        )\n",
    "    )\n",
    "\n",
    "# split off 10% for validation\n",
    "n_train = int(len(ds) * 0.9)\n",
    "ds_train = ds.take(n_train)\n",
    "ds_val = ds.skip(n_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "history = model.fit(\n",
    "    # pass the dataset, let it be shuffled and batched\n",
    "    ds_train.shuffle(n_train).batch(64),\n",
    "    # number of iterations over training dataset\n",
    "    epochs=10,\n",
    "    validation_data=ds_val.batch(64),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "np.min(history.history['val_loss'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.max(history.history['val_acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We got around 70% accuracy on the validation sample! This is not perfect yet, but already quite impressive, given the relatively simple model and fast training. From the plots above we can see that the model is maybe not fully converged yet, so a few percent might be gained by continuing the training (you can try just executing the notebook cell above again).\n",
    "\n",
    "[Current state-of-the art neural networks](https://paperswithcode.com/sota/image-classification-on-cifar-10) reach accuracies of > 95% on CIFAR-10, so there is still a lot of room for optimization.\n",
    "\n",
    "But now, let's validate our score on the completely independent test sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "scores = model.predict(preprocess(x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "These scores are now the predicted probabilities for each label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "When we take the index of the highest probability, we get the \"best-guess\" predicted labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "predicted_labels = np.argmax(scores, axis=1)\n",
    "predicted_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "acc_test = (predicted_labels == y_test.reshape(-1)).mean()\n",
    "acc_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "For such multi class problems it is useful to plot a confusion matrix - telling us how often which label is confused with each of the other labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "plt.imshow(confusion_matrix(y_test, predicted_labels), plt.cm.Blues)\n",
    "plt.colorbar()\n",
    "plt.xticks(range(10), labels, rotation=90)\n",
    "plt.xlabel(\"True label\")\n",
    "plt.yticks(range(10), labels)\n",
    "plt.ylabel(\"Predicted label\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Our test sample has 10k pictures with 1k for each category, so the perfect confusion matrix would contain the value 1000 all over the diagonal. Overall we see that animals seem to be more difficult to distinguish than vehicles, and vehicles tend to be confused with other vehicles and animals with other animals.\n",
    "\n",
    "Lets look at a random sample of pictures that our network classifies as \"cats\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "show_random_pictures(x_test[predicted_labels == labels.index('cat')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Frogs seem to work rather well, but lets look in particular at images that are incorrectly classified as frogs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "show_random_pictures(x_test[(predicted_labels == labels.index('frog')) & (y_test.reshape(-1) != labels.index('frog'))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "For comparison, some actual frogs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "show_random_pictures(x_test[y_test.reshape(-1) == labels.index('frog')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's happening with birds that are confused with airplanes?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_random_pictures(x_test[(predicted_labels == labels.index('airplane')) & (y_test.reshape(-1) == labels.index('bird'))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Lets have a look at the actual predicted probabilites for a few examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def plot_probabilities(x, y, scores, index):\n",
    "    fig, ax = plt.subplots(figsize=(4, 2), nrows=1, ncols=2)\n",
    "    ax[1].imshow(x[index])\n",
    "    ax[1].set_title(labels[y[index][0]])\n",
    "    ax[0].barh(labels, scores[index])\n",
    "    ax[0].set_xlabel(\"pred. probability\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "for index in np.random.randint(0, len(x_test), 5):\n",
    "    plot_probabilities(x_test, y_test, scores, index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does our NN say to the grumpy cat image at the beginning of this tutorial? To find out, we must first scale it down to 32x32:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grumpy_32 = np.empty((32, 32, 3))\n",
    "# split into 32 ~equal parts and average\n",
    "for ix, x in enumerate(np.array_split(img_grumpy, 32)):\n",
    "    for iy, y in enumerate(np.array_split(x.mean(axis=0), 32)):\n",
    "        grumpy_32[ix][iy] = np.mean(y, axis=0)\n",
    "# normalize\n",
    "grumpy_32 /= grumpy_32.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(grumpy_32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_grumpy = model.predict(grumpy_32.reshape(-1, 32, 32, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_probabilities(grumpy_32.reshape(-1, 32, 32, 3), np.array([[labels.index('cat')]]), scores_grumpy, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save the trained model for reuse in a later notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"CNNIntro_model.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check for some images where the classification has failed if there were competing guesses by the network that were closer to the correct label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build an array of indices of falsely classified images:\n",
    "wrong_idxs = [idx for idx in range(len(y_test)) if predicted_labels[idx] != y_test.reshape(-1)[idx]]\n",
    "# (a possibly faster) numpy version would be:\n",
    "# wrong_idxs = np.where(predicted_labels != y_test.reshape(-1))[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pick a few images with wrong classification and plot scores:\n",
    "for i in range(5):\n",
    "    idx = np.random.choice(wrong_idxs)\n",
    "    plot_probabilities(x_test, y_test, scores, idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What has the network learned?\n",
    "* CNN very successful at recognizing objects in images (visual classification problems)\n",
    "    * sometimes with success rates close to humans\n",
    "    * question thus: what does the network learn? similar to human perception of images?\n",
    "* often not easy to reason how a certain classification process is done by a neural network\n",
    "    * big field of research\n",
    "    * will not cover this here but give an interesting warning / anecdotal finding\n",
    "* how can we trick a NN into thinking it sees something else than what the image shows?\n",
    "    * surprisingly (alarmingly?) easy\n",
    "* below: results from two papers from 2013 and 2014\n",
    "    * Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus, \"Intriguing properties of neural networks\", [arXiv:1312.6199 [cs.CV]][1]\n",
    "    * Anh Nguyen, Jason Yosinski, Jeff Clune, \"Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images\", [arXiv:1412.1897 [cs.CV]][2]\n",
    "    \n",
    "[1]: https://arxiv.org/abs/1312.6199\n",
    "[2]: https://arxiv.org/abs/1412.1897"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adversarial images\n",
    "* examples of images that have been tuned to look like an ostrich (*struthio camelus*) to the neural network:\n",
    "\n",
    "![figure 5 from 1312.6199][1]\n",
    "\n",
    "* in each block there are 3 images:\n",
    "    * the left images are the original images (correctly classified)\n",
    "    * the right images are the modified images (looking the same to the human eye)\n",
    "    * the middle images are the differences, magnified by a factor 10\n",
    "\n",
    "* reminder: this is a *struthio camelus* (image by MathKnight, clipped, CC BY-SA 4.0, [link][3]).\n",
    "![Struthio camelus, Chay Bar Yotvata, Israel][2]\n",
    "\n",
    "[1]: figures/1312.6199_Fig5_clip.png\n",
    "[2]: figures/struthio_camelus.jpg\n",
    "[3]: https://commons.wikimedia.org/w/index.php?curid=5010729"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* main findings of the paper:\n",
    "    * previously assumed that individual units (nodes) of the hidden layers have a semantic meaning -- seems not to be true <!--- not only activation of single node (= maximize scalar product with vector from \"natural basis\") but arbitrary direction (activation) gives similar images as output -->\n",
    "    * rather entire space of activations seems to contain bulk of semantic information (cf. figures 1 vs 2 and 3 vs 4 from paper)\n",
    "    * can design very small (imperceptible) perturbations to change the network's prediction for a previously correctly classified image to any other label (see above) <!--- i.e. \"smoothness assumption that underlies many kernel methods does not hold\" -->\n",
    "    * more interestingly perhaps, these \"adversarial\" images are robust in the sense that they also work to fool networks trained on different subsets of the training data into the same (wrong) belief\n",
    "    * paper proposes method to efficiently create these images (to e.g. include them in the training to improve the generalization of the predictions) <!--- \"hard-negative mining: in computer vision, consists of identifying training set examples (or portions thereof) whichare given low probabilities by the model, but which should be high probability instead\" -->\n",
    "* raises question whether the networks really learned \"useful\" properties of image classes that generalize well -- but then again they are successful on independent tests\n",
    "\n",
    "<!---\n",
    "* difference to change image is given by \"minimum distortion\" function D in the paper\n",
    "* properties cf. p.5: \n",
    "    1. exists and is small \n",
    "    2. cross model generalization\n",
    "    3. Cross training-set generalization\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evolved images\n",
    "* examples of images that have been designed so that given DNNs trained on image libraries are very sure (>99.6 %) to recognize familiar objects \n",
    "    * but images are unrecognizable to humans\n",
    "    * or what do you see?\n",
    "* upper two rows: noisy images (\"direct encoding\", this is not purely stochastic noise but has been evolved to fool the network)\n",
    "* lower two rows: images generated with a evolutionary algorithm (\"indirect encoding\" with higher probability to produce symmetric shapes and regular patterns, cf. the paper)\n",
    "\n",
    " ![figures 1 from 1412.1897 without labels][1] <!--- change to [2] to show labels -->\n",
    "\n",
    "You can try and see if you can fool the network we trained above in [this notebook][3].\n",
    "\n",
    "[1]: figures/1412.1897_Fig1_clip_nolabels.png\n",
    "[2]: figures/1412.1897_Fig1_clip.png\n",
    "[3]: CNNFoolingExercise.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "name": "CNNIntro.ipynb",
  "nbdime-conflicts": {
   "local_diff": [
    {
     "diff": [
      {
       "diff": [
        {
         "key": 0,
         "op": "addrange",
         "valuelist": [
          "3.6.8"
         ]
        },
        {
         "key": 0,
         "length": 1,
         "op": "removerange"
        }
       ],
       "key": "version",
       "op": "patch"
      }
     ],
     "key": "language_info",
     "op": "patch"
    }
   ],
   "remote_diff": [
    {
     "diff": [
      {
       "key": "version",
       "op": "remove"
      }
     ],
     "key": "language_info",
     "op": "patch"
    }
   ]
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}