Module 1. Introduction to Computer Vision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction-to-Computer-Vision\" data-toc-modified-id=\"Introduction-to-Computer-Vision-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction to Computer Vision</a></span><ul class=\"toc-item\"><li><span><a href=\"#Computer-Vision-Overview\" data-toc-modified-id=\"Computer-Vision-Overview-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Computer Vision Overview</a></span></li><li><span><a href=\"#Computer-Vision-Tasks\" data-toc-modified-id=\"Computer-Vision-Tasks-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Computer Vision Tasks</a></span></li><li><span><a href=\"#Computer-Vision-in-Practice\" data-toc-modified-id=\"Computer-Vision-in-Practice-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Computer Vision in Practice</a></span></li></ul></li><li><span><a href=\"#GluonCV-Toolkit\" data-toc-modified-id=\"GluonCV-Toolkit-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>GluonCV Toolkit</a></span><ul class=\"toc-item\"><li><span><a href=\"#GluonCV\" data-toc-modified-id=\"GluonCV-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>GluonCV</a></span></li><li><span><a href=\"#GluonCV-Model-Zoo\" data-toc-modified-id=\"GluonCV-Model-Zoo-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>GluonCV Model Zoo</a></span></li></ul></li><li><span><a href=\"#Apache-MXNet-Framework\" data-toc-modified-id=\"Apache-MXNet-Framework-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Apache MXNet Framework</a></span><ul class=\"toc-item\"><li><span><a href=\"#Apache-MXNet\" data-toc-modified-id=\"Apache-MXNet-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Apache MXNet</a></span></li><li><span><a href=\"#Imperative-vs-Symbolic\" data-toc-modified-id=\"Imperative-vs-Symbolic-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Imperative vs Symbolic</a></span><ul class=\"toc-item\"><li><span><a href=\"#Imperative-Programming\" data-toc-modified-id=\"Imperative-Programming-3.2.1\"><span class=\"toc-item-num\">3.2.1&nbsp;&nbsp;</span>Imperative Programming</a></span></li><li><span><a href=\"#Symbolic-Programming\" data-toc-modified-id=\"Symbolic-Programming-3.2.2\"><span class=\"toc-item-num\">3.2.2&nbsp;&nbsp;</span>Symbolic Programming</a></span></li></ul></li></ul></li><li><span><a href=\"#Summary\" data-toc-modified-id=\"Summary-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Summary</a></span></li><li><span><a href=\"#Quiz\" data-toc-modified-id=\"Quiz-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Quiz</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Key Concepts\n",
    "- Explain common computer vision tasks\n",
    "- Understand what tasks can be solved with GluonCV\n",
    "- Understand the benefits of Apache MXNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Computer Vision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computer Vision Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computer vision is a technological field to extract high level understanding from digital images/videos for various application. Computer vision is considered as a subfield of Artificial Intelligence (AI), because the goal is to automate tasks that the human visual system performs which involve the ability to recognize, describe, and interpret visual.\n",
    "\n",
    "Example: object detection from an image.\n",
    "\n",
    "<img src=\"assets/module1/object-detection-intro.png\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computer Vision Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "History:\n",
    "- David Hubel and Torsten Wiesel (1959): visual processing starts with a neuron\n",
    "- Kunihiko Fukushima: Neocognitron (now it is called as Convolutional Neural Network)\n",
    "- Yann LeCun: LeNet5 (apply Machine Learning technique to CNN)\n",
    "- Stop for a while, needs more robust hardware\n",
    "- 2012 was the resurgence of Computer Vision due to the availability of more powerful accelerators (GPU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computer vision tasks, from simple to complex:\n",
    "1. Image classification: **what** is the main object in the image? Limitation: multiple object classification.\n",
    "\n",
    "<img src=\"assets/module1/image-classification.png\" width=\"200\">\n",
    "\n",
    "2. Object detection: localize each object with a bounding box and classify it. Limitation: not every object is rectangular.\n",
    "\n",
    "<img src=\"assets/module1/object-detection.png\" width=\"200\">\n",
    "\n",
    "3. Semantic segmentation: seperate objects with its background with a mask. Limitation: objects will have the same mask.\n",
    "\n",
    "<img src=\"assets/module1/semantic-segmentation.png\" width=\"200\">\n",
    "\n",
    "4. Instance segmentation: generates masks for each objects. Limitation: needs a lot of manually labeled instance image.\n",
    "\n",
    "<img src=\"assets/module1/instance-segmentation.png\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computer Vision in Practice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some real-world applications of Computer Vision ([Reference](https://www.forbes.com/sites/bernardmarr/2019/04/08/7-amazing-examples-of-computer-and-machine-vision-in-practice)):\n",
    "\n",
    "1. **Self-driving cars**: Detect objects, lane markings, signs, and traffic signals by using multiple input from cameras, LIDAR, radar, and ultrasonic sensors.\n",
    "\n",
    "<img src=\"assets/module1/self-driving-cars.jpeg\" width=\"300\">\n",
    "\n",
    "2. **Google translate app**: Use optical character recognition (OCR) to see text language through phone's camera and overlay the translation by augmented reality (AR).\n",
    "\n",
    "<img src=\"assets/module1/google-translate.jpg\" width=\"300\">\n",
    "\n",
    "3. **Facial recognition**: used for police work, payment portals, security checkpoints at the airport.\n",
    "\n",
    "<img src=\"assets/module1/facial-recognition.jpg\" width=\"300\">\n",
    "\n",
    "4. **Healthcare**: enabling medical diagnostic methods using X-rays, mammography, and other image scans result to monitor patients' health condition.\n",
    "\n",
    "<img src=\"assets/module1/healthcare.png\" width=\"300\">\n",
    "\n",
    "5. **Sports tracking**: helping real-time play and strategy analysis, player performance and ratings, also track the brand sponsorship visibility in broadcasts.\n",
    "\n",
    "<img src=\"assets/module1/sport-performance.png\" width=\"300\">\n",
    "\n",
    "6. **Agriculture**: analyze harvested grain quality, or identify weeds for automatic spraying of herbicides.\n",
    "\n",
    "<img src=\"assets/module1/agriculture.jpeg\" width=\"300\">\n",
    "\n",
    "7. **Manufacturing**: monitor packaging and product quality so that defective products can be reduced. You can further check out my Kaggle Kernel for automatic product quality inspection of a submersible pump impeller using Convolutional Neural Network (CNN) in Keras: https://www.kaggle.com/tomythoven/casting-inspection-with-data-augmentation-cnn\n",
    "\n",
    "<img src=\"assets/module1/product-inspection.png\" width=\"300\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GluonCV Toolkit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GluonCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Problems with open source tools:\n",
    "- Limited choices of models\n",
    "- Incorrect implementation\n",
    "- Discontinue of maintenance\n",
    "- Gap between research and deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Knowledge Check**\n",
    "\n",
    "What common issues in training and deploying computer vision models are addressed by the introduction of the GluonCV toolkit?\n",
    "- [ ] Limited choice\n",
    "- [ ] Incorrect implementations\n",
    "- [ ] Discontinued support\n",
    "- [ ] All of the above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GluonCV created by AWS scientist to address above issues.\n",
    "- Model for multiple tasks: image classification, object detection, semantic segmentation, instance segmentation, and pose estimation.\n",
    "- Implementation with high accuracy.\n",
    "- Official maintenance and development.\n",
    "- Easy-to-use APIs for experimentation.\n",
    "\n",
    "Additional: https://gluon-cv.mxnet.io/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GluonCV Model Zoo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/module1/model-illustration.png\" width=\"400\">\n",
    "\n",
    "Model in Computer Vision = Convolutional Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Knowledge Check**\n",
    "\n",
    "Why does GluonCV have various models for the same computer vision task?\n",
    "- [ ] To prevent overfitting\n",
    "- [ ] To allow for tradeoffs between model accuracy and model complexity\n",
    "- [ ] To reduce model complexity\n",
    "- [ ] To increase computational cost and model accuracy for simple tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GluonCV model zoo: a chart to help us to choose the best pretrained model.\n",
    "\n",
    "<img src=\"assets/module1/model-zoo.png\" width=\"400\">\n",
    "\n",
    "Reference: https://gluon-cv.mxnet.io/model_zoo/classification.html\n",
    "\n",
    "- X-axis: speed\n",
    "- Y-axis: accuracy\n",
    "- Dot: model\n",
    "- Size: memory consumption"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apache MXNet Framework"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apache MXNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MXNet is a deep learning framework which provides software tool for building and training neural network models.\n",
    "\n",
    "Originated from Distributed Machine Learning Community (DMLC). Project from DMLC:\n",
    "- XGBoost\n",
    "- TVM (deep learning compiler)\n",
    "- MXNet\n",
    "\n",
    "Feature:\n",
    "- Multi-language support: Python, R, Perl, Java, Julia, Scala, Clojure, C++. \n",
    "- Rich ecosystem: model zoo, GluonCV, GluonNLP, Keras, MXBoard (visualize training process), model server (AWS), TensorRT (NVIDIA), TVM (deployment), ONNX (for interchangeable AI models)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Knowledge Check**\n",
    "\n",
    "What programming language is the MXNet backend engine implemented in?\n",
    "- [ ] Python\n",
    "- [ ] C++\n",
    "- [ ] Java\n",
    "- [ ] Scala"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imperative vs Symbolic\n",
    "Consider the following computational graph:\n",
    "\n",
    "<img src=\"assets/module1/comp-graph.png\" width=\"200\">\n",
    "\n",
    "We are going to compare two paradigms of deep learning programming interface:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Imperative Programming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]\n",
      "<NDArray 10 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "from mxnet import nd\n",
    "a = nd.ones(10)\n",
    "b = nd.ones(10) * 2\n",
    "c = b * a\n",
    "print(c)\n",
    "d = c + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Advantage:\n",
    "- Straightforward and flexible (easy to debug, by printing)\n",
    "- Use language natively (no new syntax)\n",
    "\n",
    "Disadvantage:\n",
    "- Hard to optimize (computational time, memory usage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Symbolic Programming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mxnet.symbol import Variable # for placeholder\n",
    "A = Variable('A')\n",
    "B = Variable('B')\n",
    "C = B * A\n",
    "D = C + 1\n",
    "# no execution above, only placeholder\n",
    "# f = compile(D)\n",
    "# d = f(A=nd.ones(10), B=nd.ones(10)*2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Advantage:\n",
    "- Optimized\n",
    "- Easy to serialize models (to be saved) so that can be deserialize across languages\n",
    "\n",
    "Disadvantage:\n",
    "- Hard to debug\n",
    "- Unsuitable for dynamic models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion**\n",
    "\n",
    "Why is debugging a computational graph easier in imperative programs than in symbolic programs?\n",
    "\n",
    "> Because we can print the results right after each computation step, to check whether the calculation is correct or wrong."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gluon API: hybrid programming interface (combination of imperative and symbolic programming). There are two stages:\n",
    "1. Develop model using imperative programming\n",
    "2. Optimize by convert into symbolic programming\n",
    "\n",
    "Convert imperative to symbolic by `net.hybridize()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Computer vision: \n",
    "    - Extracting high-level understanding from digital images.\n",
    "    - Example: classification, detection, segmentation.\n",
    "    - Using Convolutional Neural Network.\n",
    "\n",
    "2. GluonCV: \n",
    "    - Open-source computer vision toolkit from AWS.\n",
    "    - Has model zoo with pre-trained model.\n",
    "\n",
    "3. Apache MXNet: \n",
    "    - Deep learning framework, backend engine for GluonCV.\n",
    "    - Key: portable, efficient, scalable. \n",
    "    - Use hybrid programming (imperative + symbolic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Which of the following is not a computer vision task?\n",
    "    - [ ] Volumetric analysis\n",
    "    - [ ] Pose estimation\n",
    "    - [ ] Object detection\n",
    "    - [ ] Semantic segmentation\n",
    "\n",
    "2. Which deep learning framework is the GluonCV toolkit based on?\n",
    "    - [ ] Pytorch\n",
    "    - [ ] Apache MXNet\n",
    "    - [ ] Caffe\n",
    "    - [ ] Chainer\n",
    "\n",
    "3. Which of the following is untrue about the symbolic paradigm in deep learning frameworks?\n",
    "    - [ ] Symbolic programs do not need to be compiled before they can be executed\n",
    "    - [ ] Symbolic programs provide opportunities to optimize computational graphs\n",
    "    - [ ] Symbolic programs can be hard to debug when they throw an error\n",
    "    - [ ] Symbolic programs are often constructed with variable placeholders\n",
    "\n",
    "4. What command in the Gluon API of MXNet converts an imperative computational graph to a symbolic graph?\n",
    "    - [ ] .convert()\n",
    "    - [ ] .to_symbol()\n",
    "    - [ ] .hybridize()\n",
    "    - [ ] .optimize()\n",
    "\n",
    "5. What area of machine learning currently achieves State of the Art performance in computer vision tasks?\n",
    "    - [ ] Reinforcement Learning\n",
    "    - [ ] Metric Learning\n",
    "    - [ ] Similarity Learning\n",
    "    - [ ] Deep Learning\n",
    "\n",
    "6. What do image classification models predict?\n",
    "    - [ ] A cluster centroid for the class of objects in the image\n",
    "    - [ ] A hierarchy for objects in the image\n",
    "    - [ ] Another image that is similar to the input image\n",
    "    - [ ] A predefined label for the image\n",
    "\n",
    "7. Which computer vision tasks predicts pixel level masks for each distinct class of objects in the image?\n",
    "    - [ ] Object extraction\n",
    "    - [ ] Semantic Segmentation\n",
    "    - [ ] Instance Segmentation\n",
    "    - [ ] Super-resolution imaging\n",
    "\n",
    "8. What discovery by Hubel and Wiesel and implemented by Fukushima in the Neocognitron is crucial to the success of modern deep learning based computer vision systems?\n",
    "    - [ ] Vision is intimately tied to recognition and understanding\n",
    "    - [ ] Vision is achieved by convolution in the human brain\n",
    "    - [ ] Vision is hierarchical and local at each level\n",
    "    - [ ] Vision involves extensive feature engineering\n",
    "\n",
    "9. What exactly led to the resurgence of neural network models and deep learning for computer vision tasks in 2012?\n",
    "    - [ ] Availability of large datasets thanks to the internet\n",
    "    - [ ] More powerful computational software and resources\n",
    "    - [ ] Hardware accelerators like GPUs\n",
    "    - [ ] All of the above\n",
    "\n",
    "10. Which computer vision task is most appropriate for localizing appearances of barcodes in an image?\n",
    "    - [ ] Image classification\n",
    "    - [ ] Object Detection\n",
    "    - [ ] Semantic Segmentation\n",
    "    - [ ] Instance Segmentation"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "computer-vision",
   "language": "python",
   "name": "computer-vision"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "46px",
    "width": "160px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}