lecture_06_06_2022_convolutional_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Convolutional Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this lecture we will discuss deep learning for computer vision and major architecture called convolutional neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Computer vision tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/detection_segmentation.jpg\" height=\"1000\" width=\"1000\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Classifier methods\n",
    "<img src=\"images/cnn/cnn_classifier.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Fully convolutional model\n",
    "<img src=\"images/cnn/fully_conv.jpeg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/localization_classification.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Classification loss (SoftMax with Log likelihood) + Regression loss for each coordinat\n",
    "\n",
    "- Multi-task learning\n",
    "- Weighted sum of losses $w_1 C_1 + w_2 C_2$ ($w_1 C_1(W^{l_1}, b^{l_1}, W^{l_2}, b^{l_2} \\dots W^{L}, b^{L}) + w_2 C_2(w_1 C_1(W^{l_1}, b^{l_1}, W^{l_2}, b^{l_2} \\dots W^{L}, b^{L})$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Keypoint detection\n",
    "<img src=\"images/cnn/keypoint_detection.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/face_keypoints.jpeg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Object detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/object_detection.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Deep learning for computer vision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There was many approaches for image classification before 2012 but 2012 ImageNet contest winner AlexNet made a breakthrough"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Sometimes information loss might be a good idea"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As we remember from the regularization techniques, sometimes we used drouput, we loss information deliberately in order to avoid overfitting\n",
    "- Less information might make model invariant for changes\n",
    "- Less dimension means less computation power during the both training and inference time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How can we classify images (recognize objects in images)\n",
    "- Observe different features (color, edges etc) of image\n",
    "- Investigate different part of image as features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/image_parts_1.jpg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/image_parts_2.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the other hand, in order to generalize:\n",
    "- Make it size invariant\n",
    "- Location invariant\n",
    "- View invariant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/trans_invar_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How can we make model which: \n",
    "- select features and learns\n",
    "- will be invariant for object size, color, background etc changes\n",
    "- Will be not have enough\n",
    "- Will generalize well\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Weight shearing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First way to make model invariant for different (accptable) aspects is weights sharing\n",
    "<br>\n",
    "Instead of using different weight for each neuron let's repeat them values time after time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/weight_sharing_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How can we achieve weights sharing?\n",
    "Add restriction per layer to have copy of the weight or use other approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Convolutions on matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider matrix:\n",
    "$$\\begin{align} X &= \\begin{pmatrix}\n",
    "           x_{11}, x_{12}, \\dots, x_{1n} \\\\\n",
    "           x_{21}, x_{22}, \\dots, x_{2n} \\\\\n",
    "           \\vdots \\\\\n",
    "           x_{m1}, x_{n2}, \\dots, x_{mn} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "  <br>\n",
    "  and the smaller weights matrix:\n",
    "  $$\\begin{align} w &= \\begin{pmatrix}\n",
    "           w_{11}, w_{12}, \\dots, w_{1p} \\\\\n",
    "           w_{21}, w_{22}, \\dots, w_{2p} \\\\\n",
    "           \\vdots \\\\\n",
    "           w_{q1}, w_{q2}, \\dots, w_{qp} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then convolution per slice on step size $s$ which is called stride and on step $u$ in row direction and $v$ column direction, will be:\n",
    "$$\n",
    "\\sum_{i=1}^{p}\\sum_{j=1}^{q}w_{i, j}x_{i + s u, j + s v}\n",
    "$$\n",
    "where $u$ and $v$ are amount of steps for rows and columns respectively"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For instance weights matrix:\n",
    "  $$\\begin{align} w &= \\begin{pmatrix}\n",
    "           w_{11}, w_{12}, w_{13} \\\\\n",
    "           w_{21}, w_{22}, w_{23} \\\\\n",
    "           w_{31}, w_{32}, w_{33} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "<br>\n",
    "and let's slide this matrix on our original matrix $X$ with step $s$ and for each step calculate:\n",
    "<br>\n",
    "$$\n",
    "\\sum_{i=1}^{3}\\sum_{j=1}^{3}w_{i, j}x_{i + s u, j + s v}\n",
    "$$\n",
    "where $u$ and $v$ are amount of steps for rows and columns respectively"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For first step we'd have:\n",
    "$$\n",
    "\\sum_{i=1}^{3}\\sum_{j=1}^{3}w_{i, j}x_{i, j}\n",
    "$$\n",
    "<br>\n",
    "for second\n",
    "$$\n",
    "\\sum_{i=1}^{3}\\sum_{j=1}^{3}w_{i, j}x_{i + s, j + s}\n",
    "$$\n",
    "and etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If stride is one, for first step:\n",
    "$$\n",
    "\\sum_{i=1}^{3}\\sum_{j=1}^{3}w_{i, j}x_{i, j}\n",
    "$$\n",
    "<br>\n",
    "for second\n",
    "$$\n",
    "\\sum_{i=1}^{3}\\sum_{j=1}^{3}w_{i, j}x_{i + 1, j + 1}\n",
    "$$\n",
    "etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or for better understanding:\n",
    "<img src=\"images/cnn/convolution_1.gif\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we calculate sum over the Hadamard product per slide:\n",
    "<img src=\"images/cnn/convolution_2.gif\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Stride might be different for instance here is the convolution with tride two:\n",
    "<img src=\"images/cnn/convolution_3.gif\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Also in order of achieve desirable size at the output, we cat pad image with zeros, or other values:\n",
    "<img src=\"images/cnn/convolution_padding_1.gif\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What if want to have an input with depth for convolution:\n",
    "$$X \\in \\mathbb{R}^{h \\times w \\times c}$$\n",
    "<br>\n",
    "Our input has three dimensions: height, width and channels (depth), like RGB image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Our input is three dimensional tensor instead of matrix:\n",
    "<img src=\"images/cnn/tensor_3d_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then our weights matrix should also have the same dimensions and exactly the same channels:\n",
    "$$\n",
    "W \\in \\mathbb{R}^{k \\times p \\times c}\n",
    "$$\n",
    "For our instance:\n",
    "$$\n",
    "W \\in \\mathbb{R}^{3 \\times 3 \\times c}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We calculate convolution per slide with:\n",
    "$$\n",
    "\\sum_{i=1}^{3}\\sum_{j=1}^{3}\\sum_{k=1}^{c}w_{i, j, k}x_{i + s t, j + s p, k}\n",
    "$$\n",
    "<br>\n",
    "Note that convolution does not have strides for channels and always produces one two dimensional matrix, flattens channels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here is the example of convolution, where input has three channels:\n",
    "<img src=\"images/cnn/convolution_4.gif\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Terminology"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Sliding window weight matrix is called convolutional kernel or very often filter\n",
    "- Padding means adding border elements to image\n",
    "- Step size of sliding window, for filters is called stride\n",
    "- Output of the convolution (full output matrix) is called feature-map "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we have a hyperparameter alarm\n",
    "<img src=\"images/cnn/hp_alarm_1.jpg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How many additional parameters do we have?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Additional parameters:\n",
    "- Kernel / filter width\n",
    "- Kernel / filter height (but not depth / channels)\n",
    "- Stride\n",
    "- Padding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For instance if we take image and weights matrix which in convolutional neural networks is called filter:\n",
    "$$\\begin{align} w &= \\begin{pmatrix}\n",
    "       1, 0, -1 \\\\\n",
    "       1, 0, -1 \\\\\n",
    "       1, 0, -1 \\\\\n",
    "     \\end{pmatrix}\n",
    "\\end{align}$$\n",
    "and run it with stride one will get edges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "from pathlib import Path\n",
    "\n",
    "img_path = Path('images') / 'test.jpg'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "img_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### onvolutional neural networs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider input of our model is image with height, width and channels (which might be from 1 or 3 very often)\n",
    "Lets construct convolutional neural networks for each layer:\n",
    "- define number of convolutional filters (weights matrices) which will generate the same amount of feature maps\n",
    "- Kernel / filter width height\n",
    "- Stride\n",
    "- Padding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Did we note another hyperparameter?\n",
    "<img src=\"images/cnn/hp_alarm_1.jpg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The amount of feature maps\n",
    "<img src=\"images/cnn/convolutional_layer_1.jpeg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Did we forgot about something while we've been constructing the layer?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The activation function !!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "After each convolutional filter we'll activate output with activation function for instance ReLU and gate final feature map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Pooling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In order to make model more invariant for size and location of object and reduce dimension of model lets use pooling technique\n",
    "- maximum pooling\n",
    "- average pooling\n",
    "- minimum or other pooling (don't know if anybody use it)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Maximum pooling: \n",
    "<br>\n",
    "For each feature map:\n",
    "$$\\begin{align} F &= \\begin{pmatrix}\n",
    "       F_{11}, F_{12}, \\dots, F_{1p} \\\\\n",
    "       F_{21}, F_{22}, \\dots, F_{2p} \\\\\n",
    "       \\vdots \\\\\n",
    "       F_{q1}, F_{q2}, \\dots, F_{qp} \\\\\n",
    "     \\end{pmatrix}\n",
    "\\end{align}$$\n",
    "use the sliding window as we did for convolution and select maximum among elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Example of max-pooling\n",
    "<img src=\"images/cnn/pooling_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So here we have the following hyperparameters:\n",
    "- height of sliding window\n",
    "- width of sliding window\n",
    "- stride"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note that pooling layer does not flattens depth / channels. It works for each channel, on the matrix layer, so it does not reduce depth / channel dimensionality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note that pooling layer is not learnable layer, it does not have weights and biases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we can add max-pooling layer after the several convolution layers:\n",
    "<img src=\"images/cnn/cnn_2.png\" height=\"800\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Fully connected layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With convolutional and pooling layers we will end up with feature map, or even vector if we continue puting\n",
    "layers.\n",
    "<br>\n",
    "For the classification task we need classifier, for this we add several fully connected layers at the end of our model with output probability vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now convolutional neural network will look like:\n",
    "<img src=\"images/cnn/cnn_3.jpeg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or for the three dimensional objects:\n",
    "<img src=\"images/cnn/cnn_5.jpeg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We did not mention biases, for each layer, we can add bias after the convolution operation:\n",
    "<img src=\"images/cnn/cnn_bias_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we share biases as well.\n",
    "<br>\n",
    "For each convolutional kernel / filter we have one bias.\n",
    "<br>\n",
    "For instance if we have an feature maps $32 \\times 32 \\times 512 $ and kernel / filter $F = 3 \\times 3 \\times 512$ then we'll have a $3 \\times 3 \\times 512 + 1$ parameters with bias considered. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can build our model as lego adding convolutional, pooling and fully connected layers on to each other.\n",
    "But for the last layer we need to calculate dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For convolutional layer output dimensions would be:\n",
    "$$\n",
    "H_o = \\frac{H - F_h + 2 P}{S_h} + 1\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "W_o = \\frac{W - F_w + 2 P}{S_w} + 1\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Flatten vs global pooling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "At the end of the convolutional neural network, we need to convert / reshape our feature map in to the vector in order to use it as an input for the linear layers\n",
    "<br>\n",
    "Consider our feature maps are $F \\in \\mathbb{R}^{7 \\times 7 \\times 2048}$, so we have to reshape it to the $7 \\times 7 \\times 2048 = 100252$ dimensional vector and if we connect this vector with the 200 dimensional layer, we'll get $7 \\times 7 \\times \\times 500 = 50176000$ parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of this, let's put $7 \\times 7$ average or max pooling layer before the linear layer. So we'll get $F \\in \\mathbb{R}^{1 \\times 1 \\times 2048}$ dimensional output which can be reshaped (just remove additional dimensions) to the $2048$ dimensional vector and well have $2048 \\times 500 = 1024000$ weights for the linear layer with 500 dimensional output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Global average pooling\n",
    "<img src=\"images/cnn/gp_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In general global pooling means for the $F \\in \\mathbb{R}^{n \\times m \\times c}$ using the $n \\times m$ dimensional pooling which will output the $\\mathbb{R}^{1 \\times 1 \\times c}$ dimensional value which will be reshaped in the $\\mathbb{R}^{c}$ dimensional vector without any informational loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Global pooling:\n",
    "<img src=\"images/cnn/gp_2.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Sometimes concat pooling is used, just after global average and max pooling outputs are concatenated as a one vector with $2c$ dimension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Convolution as the linear layer with weights sharing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can assume convolution as the linear layer with the shared weights:\n",
    "- We can consider input image tensor as a reshaped tensor\n",
    "- convolutional filters / kernels as a weights for linear layer with activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note that pooling layer is not learnable and and we can leave it as it is, it does not take part in backpropagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Convolution as a linear layer with the shared weights:\n",
    "<img src=\"images/cnn/conv_as_linear_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or on the other hand:\n",
    "<img src=\"images/cnn/conv_as_linear_2.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Convolution as linear with forward and backward:\n",
    "<img src=\"images/cnn/conv_as_linear_3.jpg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we can use all the previous techniques as for linear layer:\n",
    "<img src=\"images/cnn/conv_as_linear_4.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can:\n",
    "- Backpropagarion\n",
    "- Use batch normalization\n",
    "- Weights normalization\n",
    "- Layer normalization\n",
    "- Optimization\n",
    "- Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Popular architectures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As we mentioned before, we can construct convolutional and linear layers as LEGO, there are several popular. We can stack convolutional, pooling blocks as well and use this blocks as building material for more complex models  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For instance:\n",
    "- Conv layer + Conv layer +  + batch normalization + Pool layer\n",
    "- Copy input run in through the previous layer and add original input at the and\n",
    "- Copy input run in Conv layers with different kernels (maybe some of them with pooling layer) and stack everything together at the output\n",
    "- Use the above block and add original input at the end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Different convolutional blocks:\n",
    "<img src=\"images/cnn/conv_blocks_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Residual blocks\n",
    "<img src=\"images/cnn/conv_blocks_2.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Inception blocks\n",
    "<img src=\"images/cnn/conv_blocks_3.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Inception-ResNet blocks\n",
    "<img src=\"images/cnn/conv_blocks_4.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note that the similar approach might be used for linear layers as well, for instance residual block"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### VGG-16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "VGG is the one of the easy architectures which still perform impressive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "VGG-16 architecture:\n",
    "<img src=\"images/cnn/vgg_16_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Layers and parameters:\n",
    "<img src=\"images/cnn/vgg_16_2.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from torchvision.models import vgg16\n",
    "\n",
    "model = vgg16(pretrained=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Calculate dimensions:\n",
    "$$\n",
    "W_o = \\frac{W - F_w + 2 P}{S_w} + 1\n",
    "$$\n",
    "<br>\n",
    "In our case:\n",
    "$$\n",
    "W_o = H_o = \\frac{W - 3 + 2 \\cdot 1}{1} + 1 = W\n",
    "$$\n",
    "The same shape\n",
    "For pooling layer:\n",
    "$$\n",
    "W_o = H_o = \\frac{W - 2 + 2 \\cdot 0}{2} + 1 = \\frac{W - 2}{2} + 1 = \\frac{W}{2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def cnn_dim(in_shape, ker_shape, stride, padding):\n",
    "    \"\"\"\n",
    "    Calculate output dimension of the convolutional layer\n",
    "    Args:\n",
    "       in_shape: input shape - height or width\n",
    "       ker_shape: kernel shape - height or width\n",
    "       stride: stride\n",
    "       padding: padding\n",
    "    \n",
    "    Returns:\n",
    "        output dimension of convolutional layer\n",
    "    \"\"\"\n",
    "    out_shape = (in_shape - ker_shape + 2 * padding) / stride + 1\n",
    "    out_shape = int(out_shape)\n",
    "    \n",
    "    return out_shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "out_dim = cnn_dim(224, 3, 1, 1)\n",
    "print(f'out_dim = {out_dim}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can manipulate size with kernel size, padding stride and above formula. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Inception"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Inception is the model consisting with inception building blocks:\n",
    "<img src=\"images/cnn/inception_arch_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here we should notice $a \\times 1$ convolution:\n",
    "<img src=\"images/cnn/inception_arch_2.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With $1 \\times \\ 1$ architecture we just reduce depth / channels of input to the $1$\n",
    "and instead of\n",
    "$$h \\times \\ w \\times c$$\n",
    "<br>\n",
    "we have\n",
    "$$h \\times \\ w \\times 1$$\n",
    "<br>\n",
    "or\n",
    "$$h \\times \\ w$$\n",
    "<br>\n",
    "which will reduce our calculations $c$ times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Global average pooling is used before the linear classifier layer:\n",
    "<img src=\"images/cnn/global_pooling_2.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Auxiliary outputs:\n",
    "<br>\n",
    "As we saw inception has a two auxiliary outputs with general classifier layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### ResNet architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ResNet consist of residual blocks:\n",
    "<img src=\"images/cnn/resnet_arch_1.png\" height=\"300\" width=\"300\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Residual blocks:\n",
    "<img src=\"images/cnn/resnet_arch_2.jpg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Different residual blocks:\n",
    "<img src=\"images/cnn/resnet_arch_3.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The output of the convolutional block might not be influential at the end of the model and too many blocks does not have an effect of the model performance:\n",
    "<img src=\"images/cnn/resnet_arch_4.jpg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are different ResNet models:\n",
    "- ResNet18\n",
    "- ResNet34\n",
    "- ResNet50\n",
    "- ResNet101\n",
    "- ResNet1001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Inception-resnet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Inception-ResNet is an enrichment of the inception blocks with residual connections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Inception-ResNet:\n",
    "<img src=\"images/cnn/inception_resnet_arch_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Inception-ResNet blocks:\n",
    "<img src=\"images/cnn/inception_resnet_arch_2.ppm\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### ResNeXt architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ResNext architecture:\n",
    "<img src=\"images/cnn/resnext_arch_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can add building blocks:\n",
    "<img src=\"images/cnn/resnext_arch_2.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are different types of ResNeXt architectures:\n",
    "- ResNeXt50_32X4d\n",
    "- ResNeXt50_64X4d\n",
    "- ResNeXt101_32X4d\n",
    "- ResNeXt101_64X4d\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Other architectures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- DenceNet\n",
    "- TraceNet\n",
    "- EfficientNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Choose of architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Depends on the:\n",
    "- task \n",
    "- coputational capacity for training\n",
    "- coputational capacity for test\n",
    "- data\n",
    "- etc\n",
    "<br>\n",
    "Use best practices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from torchvision.models import resnet50\n",
    "\n",
    "model = resnet50(pretrained=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "??model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from torchvision.models import resnext101_32x8d\n",
    "\n",
    "model = resnext101_32x8d(pretrained=False)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/questions.jpg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Thank you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}