ai4surgery_tf.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "ai4surgery.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "file_extension": ".py",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    },
    "mimetype": "text/x-python",
    "name": "python",
    "npconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": 3
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jcvrzXlgVuGW"
      },
      "source": [
        "\n",
        "\n",
        "Copyright (c) University of Strasbourg. All Rights Reserved."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/CAMMA-public/ai4surgery/blob/master/ai4surgery.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SDk8HCcPMa1U"
      },
      "source": [
        "<div>\n",
        "<a href=\"http://camma.u-strasbg.fr/\">\n",
        "<img src=\"https://raw.githubusercontent.com/CAMMA-public/ai4surgery/master/figs/data_camma_logo_tr.png\" width=\"100\" align=\"left\"/>\n",
        "</a>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WFEJzCSTOfHQ"
      },
      "source": [
        "## <h1><center>AI for surgery: Hands-on material using TensorFlow</center></h1>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BuWZ6rZA7CFg"
      },
      "source": [
        "# **Artificial Intelligence in Surgery: An AI Primer for Surgical Practice**\n",
        "\n",
        "**Neural Networks and Deep Learning**\n",
        "\n",
        "_Deepak Alapatt and Pietro Mascagni, Vinkle Srivastav, Nicolas Padoy_\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ft8v0VzAEmAj"
      },
      "source": [
        "# Introduction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oKubxMlzEmAj"
      },
      "source": [
        "This hands-on lab demonstrates how designing and training a neural network works in practice.  It is designed as a supplement to the **Neural Networks and Deep learning** chapter of the book **Artificial Intelligence in Surgery: An AI Primer for Surgical Practice**.\n",
        "\n",
        "Specifically, in this notebook we will develop and train a simple neural network for surgical tool detection in laparoscopic images. With the advent of advanced deep learning software libraries like tensorflow, many of the steps involved in working with neural networks have been abstracted out allowing for a much simpler process.\n",
        "\n",
        "To use the notebook, you will need to first log in to your google account."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eF-t8uFKEmAk"
      },
      "source": [
        "# Basics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ecLTFvYzEmAl"
      },
      "source": [
        "Below is a code cell that assigns the value 3 and 4 to variables *a* and *b*, respectively. Then the function *max* assigns the maximum value of *a* and *b* to a variable *c*. The values stored in all three variables are then printed to the screen."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UeVjCEbvEmAl"
      },
      "source": [
        "# This is a cell to code\n",
        "# Comments following a # can be added and won't affect the code flow\n",
        "\n",
        "# Assign the value 3 to the variable a and the value 4 to the variable 4\n",
        "a = 3     \n",
        "b = 4\n",
        "\n",
        "# Assign the maximum value of a and b to c                                                \n",
        "c = max(a, b)                     \n",
        "\n",
        "# Return the values of a, b and c                        \n",
        "print(a, b, c)                                            "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2RZxhj74EmAp"
      },
      "source": [
        "To run the code in the above cell, select it with a click and then either press the *play* button on the left of the cell, or use the shortcut \"Command/Ctrl+Enter\". To edit the code, just type directly into the cell and re-run the cell to see the result.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0OJVD1MtrvZP"
      },
      "source": [
        "# Imports"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gQMw9i4XEmAq"
      },
      "source": [
        "We import some libraries which contain functions that are useful for building and visualizing neural networks. For those unfamiliar with programming, just execute the cell below and move on to the next section."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xwrTPocArvZQ"
      },
      "source": [
        "# Install needed libraries\n",
        "\n",
        "!pip install -q tensorflow==2.2.0\n",
        "!pip install -q matplotlib==3.2.2\n",
        "!pip install -q numpy==1.18.5\n",
        "\n",
        "# Import the required libraries\n",
        "\n",
        "# Tensorflow contains functions needed to build and train neural networks\n",
        "import tensorflow as tf\n",
        "# Matlpotlib is a plotting library to help visualize our inputs and outputs data\n",
        "import matplotlib.pyplot as plt\n",
        "# Numpy is a library to simplify operations on matrices and other \n",
        "# mathematical objects\n",
        "import numpy as np\n",
        "# Random is a library to help generate random numbers, make random choices, etc\n",
        "import random\n",
        "\n",
        "print(\"Libraries successfully imported!\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GlISZeotrvZa"
      },
      "source": [
        "# Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L58da2K-rvZb"
      },
      "source": [
        "The neural network will take an input image containg a surgical tool tip and classify the image as showing either a **grasper**, a **hook**, a **clipper** or a **scissor**. For this purpose, we will use [Cholec-tinytools](https://s3.unistra.fr/camma_public/github/ai4surgery/cholec-tinytools.zip), a dataset of images of surgical tool tips generated from **Cholec80**. **Cholec80** is a widely used dataset containing 80 laparoscopic cholecystectomy videos annotated with surgical phases and tool presence released by the [CAMMA research group](http://camma.u-strasbg.fr/) (University of Strasbourg, France).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4u795DXhFfy3"
      },
      "source": [
        "![](https://raw.githubusercontent.com/CAMMA-public/ai4surgery/master/figs/dataset.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Yu5nmwQEmAv"
      },
      "source": [
        "# Download and extract the tool tip dataset from an online repository \n",
        "\n",
        "DATA_URL = (\n",
        "    'https://s3.unistra.fr/camma_public/github/ai4surgery/cholec-tinytools.zip'\n",
        ")\n",
        "path = tf.keras.utils.get_file('cholec-tinytools.zip', DATA_URL, extract=True)\n",
        "  \n",
        "#Stores the dataset in the variable \"path\"\n",
        "path = path.strip('.zip')                                     \n",
        "\n",
        "print(\"Dataset successfully extracted!\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k936i9dYF8mr"
      },
      "source": [
        "Note: The code above may take a few minutes to be executed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Q2f_4xTEmAz"
      },
      "source": [
        "# Define the dataset characteristics\n",
        "\n",
        "# ~60% of the dataset for training\n",
        "TRAINING_SET_SIZE = 1200      \n",
        "\n",
        "# ~10% of the dataset for validation\n",
        "VALIDATION_SET_SIZE = 200  \n",
        "\n",
        "# ~30% of the dataset for testing                              \n",
        "TEST_SET_SIZE = 599              \n",
        "\n",
        "# The names of surgical tools we want to classify\n",
        "CLASS_NAMES = ['grasper', 'hook', 'scissor', 'clipper']\n",
        "\n",
        "# The numbers of surgical tools (i.e. classes) we want to classify     \n",
        "NUMBER_CLASSES = 4                   \n",
        "\n",
        "# Each input image will have a resolution of 86x128 pixels                        \n",
        "IMAGE_SIZE = (86, 128)                        \n",
        "\n",
        "# RGB IMAGES are represented with 3 channels (red, blue, green)                \n",
        "CHANNELS = 3\n",
        "   \n",
        "# Number of images per batch                                               \n",
        "BATCH_SIZE = 16                                              \n",
        "\n",
        "print(\"Dataset characteristics defined!\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cry8dm8GJD40"
      },
      "source": [
        "Each pixel of an RGB image can be represented as a combination of red, green and blue values each between 0-255.\n",
        "![](https://raw.githubusercontent.com/CAMMA-public/ai4surgery/master/figs/rgb_decomp.png)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EDTFd_ihEmA2"
      },
      "source": [
        "# Preprocess the images in the dataset\n",
        "\n",
        "# The training images are scaled down (i.e. normalized) \n",
        "# to the range 0-1 by dividing by 255\n",
        "# To artificially augment the training set, we additionally use variants of  \n",
        "# each image generated by randomnly rotating it by 0-20 degrees\n",
        "train_preprocessing = tf.keras.preprocessing.image.ImageDataGenerator(\n",
        "    rescale=1./255, rotation_range=20\n",
        ")\n",
        "train_set = tf.keras.preprocessing.image.DirectoryIterator(\n",
        "    path +'/train',\n",
        "    image_data_generator=train_preprocessing,\n",
        "    batch_size=BATCH_SIZE,\n",
        "    target_size=IMAGE_SIZE,\n",
        "    classes=CLASS_NAMES,\n",
        "    shuffle=True,\n",
        ")\n",
        "\n",
        "# The validation and test images are similarly scaled down (i.e. normalized) \n",
        "# to the range 0-1 by dividing by 255\n",
        "validation_test_preprocessing = tf.keras.preprocessing.image.ImageDataGenerator(\n",
        "    rescale=1./255\n",
        ")\n",
        "validation_set = tf.keras.preprocessing.image.DirectoryIterator(\n",
        "    path +'/validation',\n",
        "    image_data_generator=validation_test_preprocessing,\n",
        "    batch_size=BATCH_SIZE,\n",
        "    classes=CLASS_NAMES,\n",
        "    target_size=IMAGE_SIZE\n",
        ")\n",
        "\n",
        "test_set = tf.keras.preprocessing.image.DirectoryIterator(\n",
        "    path +'/test',\n",
        "    image_data_generator=validation_test_preprocessing,\n",
        "    batch_size=BATCH_SIZE,\n",
        "    classes=CLASS_NAMES,\n",
        "    target_size=IMAGE_SIZE\n",
        ")\n",
        "\n",
        "print(\"\\n Dataset successfully preprocessed!\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pxbn7C-Tx2j3"
      },
      "source": [
        "Preview 4 random images from the training set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RaNhc_PWrvZg"
      },
      "source": [
        "# Define the figure using a matplotlib.pyplot function\n",
        "fig, axes = plt.subplots(1,4, figsize=(15, 15))               \n",
        "random_batch =  random.randint(0, len(train_set)-1)\n",
        "random_images, random_labels = train_set[random_batch]\n",
        "\n",
        "# For loops are used to iterate over a sequence, in this case each \n",
        "# image being plotted\n",
        "for image_number, axs in enumerate(axes):                     \n",
        "    img = random_images[image_number]\n",
        "    axs.imshow(img)\n",
        "    axs.axis(\"off\")\n",
        "    axs.set_title(CLASS_NAMES[np.argmax(random_labels[image_number])])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fkmjPZ_19YlC"
      },
      "source": [
        "Rerun the cell above to visualize the different appearances of the 4 types of tool tips in the dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8FCC_id6rvZj"
      },
      "source": [
        "# Convolutions\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6WVZg5L6GAO_"
      },
      "source": [
        "\n",
        "<div>\n",
        "<center>\n",
        "<img src=\"https://raw.githubusercontent.com/CAMMA-public/ai4surgery/master/figs/convolution.gif\" width=\"350\" /> </center>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xarV2SgH49X_"
      },
      "source": [
        "Here we will apply filters for edge detection to demonstrate the kind of information a single convolutional filter can extract from an image."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XBuxbqPu_rK4"
      },
      "source": [
        "# Read a sample image\n",
        "img = plt.imread(path + \"/train/hook/7013_16226.png\")\n",
        "\n",
        "# Convert the image to grayscale\n",
        "img_gray = tf.image.rgb_to_grayscale(img)\n",
        "\n",
        "# Visualize result \n",
        "fig = plt.figure(figsize=(15,5))\n",
        "\n",
        "# Display the original input and it's grayscale version \n",
        "fig.add_subplot(1, 2, 1); plt.imshow(img);\n",
        "plt.title(\"Original Input\"); plt.axis(\"off\")\n",
        "\n",
        "fig.add_subplot(1, 2, 2); plt.imshow(img_gray.numpy()[..., 0], 'gray');\n",
        "plt.title(\"Grayscale Input\"); plt.axis(\"off\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZXA7F9CFOUbg"
      },
      "source": [
        "Kernels for edge detection\n",
        "\n",
        "<div>\n",
        "<center>\n",
        "<img src=\"https://raw.githubusercontent.com/CAMMA-public/ai4surgery/master/figs/sobel_filter.png\" width=\"600\" /> </center>\n",
        "</div>\n",
        "\n",
        "\n",
        "The above kernels or filters contain [known weight values](https://en.wikipedia.org/wiki/Sobel_operator) for horizontal and vertical edges detection.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s7DlMuG6EmA_"
      },
      "source": [
        "# Define filters for edge detection\n",
        "\n",
        "# Horizontal filter\n",
        "horizontal_edge_detector = tf.constant(\n",
        "    [[1,2,1],[0,0,0],[-1,-2,-1]], dtype=tf.float32, shape=(3, 3, 1, 1)\n",
        ")\n",
        "\n",
        "# Vertical filter\n",
        "vertical_edge_detector = tf.constant(\n",
        "    [[1,0,-1],[2,0,-2],[1,0,-1]], dtype=tf.float32, shape=(3, 3, 1, 1)\n",
        ")\n",
        "\n",
        "# Display filters\n",
        "# We use \".numpy()\" to display the result as a matrix\n",
        "print('Horizontal filter')\n",
        "print(horizontal_edge_detector[:,:,0,0].numpy())\n",
        "\n",
        "print('Vertical filter')\n",
        "print(vertical_edge_detector[:,:,0,0].numpy())\n",
        "\n",
        "print(\"Edge detectors tensors successfully defined\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rYqfBKhCCa7D"
      },
      "source": [
        "Since most tensorflow functions are designed to take multiple batches of input images at a time, we first convert the input image to a 4 dimensional matrix as shown below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C6koJ5rpCVMN"
      },
      "source": [
        "# Expand to have 4 dimensional (4D) matrix\n",
        "# This is done because the tensorflow library expects the input to the \n",
        "# convolution to be in the shape Batch x Height x Width x Channels\n",
        "# In this case Batch = 1\n",
        "\n",
        "input_4d = tf.expand_dims(img_gray, 0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZLKHEzpTy2lJ"
      },
      "source": [
        "Apply a convolution for horizontal edge detection on the grayscale image"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QYxPPsSoviSE"
      },
      "source": [
        "# Convolution on the image with horizontal filter\n",
        "# The filter is shifted of 1 pixel (strides = 1) at every step, \n",
        "# zero padding is applied such the output will have the \"SAME\" spatial \n",
        "# dimensions as the input\n",
        "conv_h = tf.nn.conv2d(input=input_4d, filters=horizontal_edge_detector, strides=1, padding=\"SAME\") \n",
        "\n",
        "# The filter will output high absolute values (maybe positive or negative) \n",
        "# when it detects horizontal edges\n",
        "horizontal_edge_img = tf.abs(conv_h)\n",
        "\n",
        "# Visualize result \n",
        "fig = plt.figure(figsize=(15,5))\n",
        "\n",
        "# Display input\n",
        "fig.add_subplot(1, 2, 1); plt.imshow(input_4d.numpy()[0,:,:,0], 'gray');\n",
        "plt.title(\"Input Gray\"); plt.axis(\"off\")\n",
        "\n",
        "# Display detected horizontal edges\n",
        "fig.add_subplot(1, 2, 2); plt.imshow(horizontal_edge_img.numpy()[0,:,:,0], 'gray');\n",
        "plt.title(\"Horizontal edges\"); plt.axis(\"off\")\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LxYmz8rsy6nH"
      },
      "source": [
        "Similarly for vertical edge detection"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F9ztibIxDoJR"
      },
      "source": [
        "conv_v = tf.nn.conv2d(input=input_4d, filters=vertical_edge_detector, strides=1, padding=\"SAME\") \n",
        "vertical_edge_img = tf.abs(conv_v)\n",
        "\n",
        "# Visualize result \n",
        "fig = plt.figure(figsize=(15,5))\n",
        "\n",
        "# Display input\n",
        "fig.add_subplot(1, 2, 1); plt.imshow(input_4d.numpy()[0,:,:,0], 'gray');\n",
        "plt.title(\"Input Gray\"); plt.axis(\"off\")\n",
        "\n",
        "# Display detected vertical edges\n",
        "fig.add_subplot(1, 2, 2); plt.imshow(vertical_edge_img.numpy()[0,:,:,0], 'gray');\n",
        "plt.title(\"Vertical edges\"); plt.axis(\"off\")\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v_GDQoxLD69s"
      },
      "source": [
        "# The detected edges in the horizontal and vertical direction can then be \n",
        "# combined to identify all the edges in the image \n",
        "\n",
        "# Create a single output by adding the detected vertical and horizontal edges\n",
        "edge_detections = horizontal_edge_img + vertical_edge_img"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pgECaTp4O73d"
      },
      "source": [
        "# Visualize result \n",
        "fig = plt.figure(figsize=(30,5))\n",
        "\n",
        "# Display input\n",
        "fig.add_subplot(1, 5, 1); plt.imshow(input_4d.numpy()[0,:,:,0], 'gray');\n",
        "plt.title(\"Input Gray\"); plt.axis(\"off\")\n",
        "\n",
        "# Display detected horizontal edges\n",
        "fig.add_subplot(1, 5, 2); plt.imshow(horizontal_edge_img.numpy()[0,:,:,0], 'gray');\n",
        "plt.title(\"Horizontal edges\"); plt.axis(\"off\")\n",
        "\n",
        "# Display detected horizontal edges\n",
        "fig.add_subplot(1, 5, 3); plt.imshow(vertical_edge_img.numpy()[0,:,:,0], 'gray'); \n",
        "plt.title(\"Vertical edges\"); plt.axis(\"off\")\n",
        "\n",
        "# Display combined horizontal and vertical edges\n",
        "fig.add_subplot(1, 5, 4); plt.imshow(edge_detections.numpy()[0,:,:,0], 'gray'); \n",
        "plt.title(\"Combined\"); plt.axis(\"off\")\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EVZAhfhm7AlV"
      },
      "source": [
        "When building neural networks, filter weights will be learned during training to extract and aggregate information relevant to the target task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K-FHVvMgEmBC"
      },
      "source": [
        "## Convolution using `tf.keras`\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zoD3ImxIXCbT"
      },
      "source": [
        "Keras is a high-level interface library running on top of TensorFlow and other machine learning frameworks. It was designed to be more intuitive and user-friendly to enable fast experimentation with deep neural networks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VGjLb8f9cmcq"
      },
      "source": [
        "# Again, we expand the dimensions so that our input is in the shape\n",
        "# Batch x Height x Width x Channels\n",
        "\n",
        "input_4d = tf.expand_dims(img, 0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "16D9oRp5EmBC"
      },
      "source": [
        "# Use keras for convolution\n",
        "\n",
        "# Random convolution operation (i.e. untrained kernel weights)\n",
        "conv_img = tf.keras.layers.Conv2D(\n",
        "    filters=3, kernel_size=(3, 3),\n",
        "    strides=1, padding=\"SAME\",\n",
        "    input_shape=(1, 86, 128, 3))(input_4d)\n",
        "conv_img = conv_img.numpy()[0]\n",
        "\n",
        "# Normalize so that our output values lie within a valid range for display\n",
        "# i.e. 0-1\n",
        "min, max = conv_img.min(), conv_img.max()\n",
        "conv_img = (conv_img - min)/(max-min)\n",
        "\n",
        "# Visualize result\n",
        "fig = plt.figure(figsize=(15,5))\n",
        "\n",
        "# Display input\n",
        "fig.add_subplot(1, 2, 1); plt.imshow(img);plt.title(\"Input\"); \n",
        "plt.axis(\"off\")\n",
        "\n",
        "# Display convolution output\n",
        "fig.add_subplot(1, 2, 2); plt.imshow(conv_img);plt.title(\"Output\");\n",
        "plt.axis(\"off\")\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fViw60AgHe0W"
      },
      "source": [
        "Here the kernel weights are randomly initiated and not trained to extract any particular feature, hence the output changes at every execution. This demonstrates the kind of information that can get filtered out or focused on by a single convolution.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hoMk4UCGEmBF"
      },
      "source": [
        "## Pooling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCM-ew47EmZ8"
      },
      "source": [
        "\n",
        "<div>\n",
        "<center>\n",
        "<img src=\"https://raw.githubusercontent.com/CAMMA-public/ai4surgery/master/figs/pooling.png\" width=\"600\" /> </center>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RxViu5ikYz8K"
      },
      "source": [
        "Pooling layers apply mathematical operations such as averaging and maximum to reduce the size of the inputs. Such operations are applied using sliding filters, similar to the convolutional filters described above. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mzGS7OgKbw1b"
      },
      "source": [
        "We will now apply max pooling and average pooling to visualize how they differently affect an input. Feel free to explore other filter sizes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MgvnVTEMfDnF"
      },
      "source": [
        "# Experimenting with max pooling filter sizes\n",
        "\n",
        "# Change the value below to see how filter size affects the output\n",
        "filter_size = 5\n",
        "\n",
        "# Max Pooling using filter_size x filter_size (default: 5x5)\n",
        "# The filter is shifted with a stride of 2 in both the horizontal\n",
        "# and vertical directions.\n",
        "maxpooled_img =  tf.keras.layers.MaxPool2D(\n",
        "    pool_size=(filter_size, filter_size), strides=2)(input_4d)\n",
        "\n",
        "# Visualize the max pooling output\n",
        "fig = plt.figure()\n",
        "plt.imshow(maxpooled_img.numpy()[0]);\n",
        "plt.title(\"max pool {}\".format(filter_size)); plt.axis(\"off\")\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LbkiyQJdaBcY"
      },
      "source": [
        "# Experimenting with max pooling filter sizes\n",
        "\n",
        "# Max Pooling using 5x5 kernel, strides of 5\n",
        "maxpooled_img5x5 =  tf.keras.layers.MaxPool2D(\n",
        "    pool_size=(5, 5), strides=5)(input_4d)\n",
        "\n",
        "# Max Pooling using 10x10 kernel, strides of 10\n",
        "maxpooled_img10x10 = tf.keras.layers.MaxPool2D(\n",
        "    pool_size=(10, 10), strides=10)(input_4d)\n",
        "\n",
        "# Max Pooling using 20x20 kernel, strides of 20\n",
        "maxpooled_img20x20 =  tf.keras.layers.MaxPool2D(\n",
        "    pool_size=(20, 20), strides=20)(input_4d)\n",
        "\n",
        "# Visualize max pooling output\n",
        "fig = plt.figure(figsize=(10,5))\n",
        "\n",
        "fig.add_subplot(1, 3, 1); plt.imshow(maxpooled_img5x5.numpy()[0]);\n",
        "plt.title(\"max pool 5x5\")\n",
        "\n",
        "fig.add_subplot(1, 3, 2); plt.imshow(maxpooled_img10x10.numpy()[0])\n",
        "plt.title(\"max pool 10x10\")\n",
        "\n",
        "fig.add_subplot(1, 3, 3); plt.imshow(maxpooled_img20x20.numpy()[0]);\n",
        "plt.title(\"max pool 20x20\")\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7CM49a40aSdc"
      },
      "source": [
        "# Experimenting with average pooling filter sizes\n",
        "\n",
        "# Average Pooling using 5x5 kernel, stride of 5\n",
        "avgpooled_img5x5 = tf.keras.layers.AveragePooling2D(\n",
        "    pool_size=(5, 5), strides=5)(input_4d)\n",
        "\n",
        "# Average Pooling using 10x10 kernel, stride of 10\n",
        "avgpooled_img10x10 = tf.keras.layers.AveragePooling2D(\n",
        "    pool_size=(10, 10), strides=10)(input_4d)\n",
        "\n",
        "# Average Pooling using 20x20 kernel, strides of 20\n",
        "avgpooled_img20x20 = tf.keras.layers.AveragePooling2D(\n",
        "    pool_size=(20, 20), strides=20)(input_4d)\n",
        "\n",
        "# Visualize average pooling output\n",
        "fig = plt.figure(figsize=(10,5))\n",
        "\n",
        "fig.add_subplot(1, 3, 1); plt.imshow(avgpooled_img5x5.numpy()[0]);\n",
        "plt.title(\"avg pool 5x5\")\n",
        "\n",
        "fig.add_subplot(1, 3, 2); plt.imshow(avgpooled_img10x10.numpy()[0]);\n",
        "plt.title(\"avg pool 10x10\")\n",
        "\n",
        "fig.add_subplot(1, 3, 3); plt.imshow(avgpooled_img20x20.numpy()[0]);\n",
        "plt.title(\"avg pool 20x20\")\n",
        "\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "psLosmU_AEI3"
      },
      "source": [
        "\n",
        "In both average and max pooling, there is a trade-off between image(or feature) size and spatial detail."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QDRmojPfrvZn"
      },
      "source": [
        "# Tool Classification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WkJFamVUrvZo"
      },
      "source": [
        "It is now time to build a neural network for surgical tool classification. The classification model is defined below as a sequential stack of layers (picturized below)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_8S6fskZTOrS"
      },
      "source": [
        "\n",
        "<div>\n",
        "<center>\n",
        "<img src=\"https://raw.githubusercontent.com/CAMMA-public/ai4surgery/master/figs/cnn.png\" width=\"750\" /> </center>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0JOAiPWEf6aE"
      },
      "source": [
        "# Defining the neural network architecture\n",
        "model = tf.keras.Sequential()\n",
        "\n",
        "# Adding a convolution with 16 5x5 filters followed by a ReLU activation\n",
        "model.add(tf.keras.layers.Conv2D(\n",
        "    filters=16, kernel_size=5, activation=\"relu\", input_shape=(86, 128, 3))\n",
        ")\n",
        "# Adding max pooling over 5x5 patches of the previous layers output\n",
        "model.add(tf.keras.layers.MaxPooling2D(pool_size=(5,5)))     \n",
        "\n",
        "# Adding a second convolution with 32 3x3 filters followed by a ReLU activation                   \n",
        "model.add(tf.keras.layers.Conv2D(filters=32, kernel_size=3, activation=\"relu\")) \n",
        "\n",
        "# Adding a layer to flatten the multidimensional (HxWxC) input\n",
        "model.add(tf.keras.layers.Flatten())                           \n",
        "\n",
        "# Adding a fully connected layer with 4096 outputs followed by a ReLU activation                 \n",
        "model.add(tf.keras.layers.Dense(units=4096, activation=\"relu\"))                 \n",
        "\n",
        "# Adding a fully connected layer with 2048 outputs followed by a ReLU activation\n",
        "model.add(tf.keras.layers.Dense(units=2048, activation=\"relu\"))       \n",
        "\n",
        "# Adding a fully connected layer with 4 outputs followed by a softmax activation          \n",
        "model.add(tf.keras.layers.Dense(units=NUMBER_CLASSES, activation=\"softmax\"))    \n",
        "\n",
        "print(\"Neural network architecture successfully defined!\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ygIHhRKjgVVT"
      },
      "source": [
        "Notes:\n",
        "*   The output of the convolutional layer is a 3 dimensional HxWxC matrix for each element of the batch. This should be flattened to a single row of H*W*C elements before applying a fully connected layers.\n",
        "*   Fully connected layers are sometimes called dense layers because each output (here referred to as units) is densely connected to the previous layer.\n",
        "*   The model ends with a fully connected layer with 4 outputs and a softmax activation. This activation function is used to convert the output of the previous fully connected layer into a vector of 4 probabilities that sum to one, i.e. the probability of each surgical instrument to be in the image.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JAYs-kHdEmBL"
      },
      "source": [
        "# Defining the optimization method, a loss function and a metric\n",
        "\n",
        "# The optimization method is stochasitc gradient descent(SGD)\n",
        "opt = tf.keras.optimizers.SGD(learning_rate=0.01) \n",
        "\n",
        "# We specify the expected input shape to build our model \n",
        "# batch size x height x width x channels                   \n",
        "model.build([1, 86, 128, 3])\n",
        "\n",
        "# Categorical cross entropy is commonly used loss for classification problems\n",
        "model.compile(\n",
        "    optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy']\n",
        ")\n",
        "\n",
        "# model.summary generates a neat summary listing the number of parameters\n",
        "# in the entire network and per layer\n",
        "model.summary()                                                                     "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LeZ-VZeL3dL3"
      },
      "source": [
        "Note: **None** indicates that any batch size can be passed through the same network using the same architecture."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2L0L5SgqWmTb"
      },
      "source": [
        "Let's run the untrained network on a single image to see what the network input and output look like."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U2UilFbmWQdE"
      },
      "source": [
        "# Read image\n",
        "img = plt.imread(path + \"/train/hook/7013_16226.png\")\n",
        "print('Input:')\n",
        "plt.imshow(img)\n",
        "plt.show()\n",
        "\n",
        "# Expand to have 4 dimensional (4D) image tensor\n",
        "input_4d = tf.expand_dims(img, 0)\n",
        "\n",
        "# Uses the untrained model defined above to predict  \n",
        "prediction = model.predict(input_4d)                                   \n",
        "print('Output:')\n",
        "print(prediction[0])\n",
        "print(CLASS_NAMES)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCegTgYeXxYI"
      },
      "source": [
        "The network predicts 4 probability values corresponding to every considered class. Note that since we used a softmax activation in our last layer, the 4 probabilities sum to 1. Right now, the predictions are random; we will train the network to learn parameters to make better predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qHNjicVHEmBV"
      },
      "source": [
        "# Training our neural network\n",
        "# The model will iterate over the training data 15 times (i.e. epochs)\n",
        "history = model.fit(train_set, validation_data=validation_set, epochs=15)       \n",
        "\n",
        "# Plot the validation and test results for each training epoch\n",
        "plt.plot(history.history['accuracy'])\n",
        "plt.plot(history.history['val_accuracy'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'validation'])\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZim2t7msucO"
      },
      "source": [
        "The above graphic plots the neural network accuracy on the training set and on the validation set over training epochs. When the two lines converge training is effective, when those diverge (i.e. higher accuracy on the training set then on the validation set) the model is overfitting to the training data. An overfitted model will fail to generalize to unseen data (i.e. will perform poorly on test data). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L2V480xMEmBY"
      },
      "source": [
        " model.evaluate(test_set)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ceDPXGzIkY2"
      },
      "source": [
        "The **evaluate** function returns the model loss and accuracy (first and second number in squared brackets, respectively) on the test set. The goal is to get the loss close to 0 and accuracy close to 1 (100%)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lml-DCq28Ho9"
      },
      "source": [
        "Finally, we visualize the results on a few random images from the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IfUJHkI98RNr"
      },
      "source": [
        "# Define the figure using a matplotlib.pyplot function\n",
        "fig, axes = plt.subplots(1,4, figsize=(15, 15))               \n",
        "random_batch =  random.randint(0, len(test_set)-1)\n",
        "random_images, random_labels = test_set[random_batch]\n",
        "\n",
        "predictions = model.predict(random_images)\n",
        "\n",
        "# For loops are used to iterate over a sequence, in this case each \n",
        "# image being plotted\n",
        "for image_number, axs in enumerate(axes):                     \n",
        "    img = random_images[image_number]\n",
        "    axs.imshow(img)\n",
        "    axs.axis(\"off\")\n",
        "    prediction = CLASS_NAMES[np.argmax(predictions[image_number])]\n",
        "    label = CLASS_NAMES[np.argmax(random_labels[image_number])]\n",
        "    axs.set_title(\"Predicted Class {} \\n True Class {}\".format(prediction, label))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NSLhoxULBZk8"
      },
      "source": [
        "# Design your surgical tool classifier\n",
        "\n",
        "Here's a copy of the same model to play around with. Your aim shuld be to design a network architecture and pick its corresponding hyperparameters to maximize the validation accuracy.\n",
        "\n",
        "*   See what effect increasing and decreasing the number of epochs has on training\n",
        "*   Play around with the learning rate to see what's the optimal value\n",
        "*   Try changing the parameters (number of filters, kernel size, etc...)\n",
        "*   Try out different activation functions (\"tanh\", \"relu\", etc...)\n",
        "*   Try adding or removing layers\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K0KGwPh-BYkB"
      },
      "source": [
        "my_model = tf.keras.Sequential()\n",
        "my_model.add(\n",
        "    tf.keras.layers.Conv2D(filters=16, kernel_size=5, activation=\"relu\")\n",
        ")\n",
        "my_model.add(tf.keras.layers.MaxPooling2D(pool_size=(5,5)))\n",
        "my_model.add(\n",
        "    tf.keras.layers.Conv2D(filters=32, kernel_size=3, activation=\"relu\")\n",
        ")\n",
        "my_model.add(tf.keras.layers.Flatten())\n",
        "my_model.add(tf.keras.layers.Dense(units=4096, activation=\"relu\"))\n",
        "my_model.add(tf.keras.layers.Dense(units=2048, activation=\"relu\"))\n",
        "my_model.add(tf.keras.layers.Dense(units=NUMBER_CLASSES, activation=\"softmax\"))\n",
        "\n",
        "opt = tf.keras.optimizers.SGD(learning_rate=0.01)\n",
        "my_model.compile(\n",
        "    optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy']\n",
        ")\n",
        "\n",
        "history = my_model.fit(train_set, validation_data=validation_set, epochs=15)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yUwoKXjU0JFc"
      },
      "source": [
        "plt.plot(history.history['accuracy'])\n",
        "plt.plot(history.history['val_accuracy'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'validation'])\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xjoZ-vWiCaQ8"
      },
      "source": [
        "my_model.evaluate(test_set)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}