computer_vision_lecture5.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "computer-vision-lecture5.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyMjtkXtoLGPmUW4pZ0ijxz1",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/noahgift/edge-computer-vision/blob/master/computer_vision_lecture5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hat17x80OVRW",
        "colab_type": "text"
      },
      "source": [
        "## Edge Machine Learning\n",
        "\n",
        "![hero](https://user-images.githubusercontent.com/58792/45260894-08e67a00-b3a8-11e8-941f-e539cb10f8e1.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4h6NFsfUOaUc",
        "colab_type": "text"
      },
      "source": [
        "## Pragmatic AI Labs\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kaAAnCxZTerK",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://paiml.com/images/logo_with_slogan_white_background.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELi2P3zcBWu-",
        "colab_type": "text"
      },
      "source": [
        "This notebook was produced by [Pragmatic AI Labs](https://paiml.com/).  You can continue learning about these topics by:\n",
        "\n",
        "*   Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863917)\n",
        "*   Reading an online copy of [Pragmatic AI:Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
        "*  Watching video [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118) on Safari Books Online.\n",
        "* Purchasing video [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
        "*   Viewing more content at [noahgift.com](https://noahgift.com/)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jVxkbCuXOZbg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u491aIb_3cWE",
        "colab_type": "text"
      },
      "source": [
        "## Edge Machine Learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BPZbWOdb8QWe",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "*   Low Latency (Role of 5G?)\n",
        "*   Available Offline\n",
        "*   Privacy\n",
        "*   ASIC (Application Specific Integrated Circuits) play key role\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pvcq8U9VSsXX",
        "colab_type": "text"
      },
      "source": [
        "### Key Industry Players"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OfcSnJYhSycx",
        "colab_type": "text"
      },
      "source": [
        "All have \"inference\" based solutions\n",
        "\n",
        "*   AWS\n",
        "*   Google\n",
        "*   Apple\n",
        "*   Intel\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yylxGK6_3c7u",
        "colab_type": "text"
      },
      "source": [
        "**Sections**\n",
        "\n",
        "\n",
        "\n",
        "*   [Intel Movidius](https://colab.research.google.com/drive/1Q1UY2gK2qfx9aMPGEoRv7sqlzk7zbIJR#scrollTo=LpQ25u7y6wkU)\n",
        "*   [iOS CoreML](https://colab.research.google.com/drive/1Q1UY2gK2qfx9aMPGEoRv7sqlzk7zbIJR#scrollTo=2AjmMkHtDrLC)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LpQ25u7y6wkU",
        "colab_type": "text"
      },
      "source": [
        "### Intel Movidius"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ExLFWf165Vl",
        "colab_type": "text"
      },
      "source": [
        "#### Overview of Movidius\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h-W6djM9SoKk",
        "colab_type": "text"
      },
      "source": [
        "This is so cool!\n",
        "\n",
        "![stick](https://user-images.githubusercontent.com/58792/72656377-d9b32600-3968-11ea-93e4-0a5b969aab58.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tM7aVgGn78to",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "[Intel Movidius Neural Compute Stick](https://developer.movidius.com/)\n",
        "\n",
        "\"tiny fanless deep learning device for...AI programming at the edge\"\n",
        "\n",
        "\n",
        "![movidius workflow](https://user-images.githubusercontent.com/58792/45309910-b73e0c80-b4d9-11e8-8c72-e3aa17d61ab9.png)\n",
        "\n",
        "\n",
        "**How To**\n",
        "\n",
        "*  Using an Ubuntu VM is most straightforward path to install SDK\n",
        "\n",
        "![alt text](https://user-images.githubusercontent.com/58792/45322711-f5e5be00-b4fd-11e8-9a19-894d73450f70.png)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "**Resources:**\n",
        "\n",
        "*   [Neural Network Prototyping Video](https://www.youtube.com/watch?v=FDguqBh3V6g)\n",
        "*   [Getting Started Guide](https://developer.movidius.com/start)\n",
        "* [NCSDK](https://github.com/movidius/ncsdk)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qNRF0IP78A54",
        "colab_type": "text"
      },
      "source": [
        "#### Movidius Hello World"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j7ZKmDlyPH91",
        "colab_type": "text"
      },
      "source": [
        "Checkout sample applications and run hello world (hello_ncs_py):  https://github.com/movidius/ncappzoo/blob/master/README.md\n",
        "\n",
        "\n",
        "```shell\n",
        "\n",
        "parallels@parallels-vm:~/workspace/ncappzoo/apps/hello_ncs_py$ make run\n",
        "\n",
        "making run\n",
        "python3 hello_ncs.py;\n",
        "Error - no NCS devices detected, verify an NCS device is connected.\n",
        "parallels@parallels-vm:~/workspace/ncappzoo/apps/hello_ncs_py$ make run\n",
        "\n",
        "making run\n",
        "python3 hello_ncs.py;\n",
        "Hello NCS! Device opened normally.\n",
        "Goodbye NCS! Device closed normally.\n",
        "NCS device working.\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RlCe2zGx1hCa",
        "colab_type": "text"
      },
      "source": [
        "#### Movidius Real-time Image Classification\n",
        "\n",
        "Example using Caffe model from [ncappzoo Github repo](https://github.com/movidius/ncappzoo)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FkwHx8cS1kwm",
        "colab_type": "text"
      },
      "source": [
        "![Movidius](https://user-images.githubusercontent.com/58792/45392092-a1fcd700-b5da-11e8-85d3-35e792fd216f.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M2LH8STv8Q1c",
        "colab_type": "text"
      },
      "source": [
        "### ONNX"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SOMo-4on8S0X",
        "colab_type": "text"
      },
      "source": [
        "[Example of Coreml conversion](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2AjmMkHtDrLC",
        "colab_type": "text"
      },
      "source": [
        "### iOS CoreML"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W_r2CsFMDrLE",
        "colab_type": "text"
      },
      "source": [
        "#### Overview of CoreML\n",
        "\n",
        "\n",
        "![CoreML](https://docs-assets.developer.apple.com/published/7e05fb5a2e/4b0ecf58-a51a-4bfa-a361-eb77e59ed76e.png)\n",
        "\n",
        "[CoreML Docs](https://developer.apple.com/documentation/coreml)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BQorH943DrLF",
        "colab_type": "text"
      },
      "source": [
        "#### Key Concepts\n",
        "\n",
        "*  Optimized for on-device performance\n",
        "*  Minimal memory footprint\n",
        "*  Can convert trained models from third-party machine learning framework\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F-DcJbU177Vj",
        "colab_type": "text"
      },
      "source": [
        "#### Converting Trained Models\n",
        "\n",
        "Example of conversion process:\n",
        "\n",
        "*Install python package:*\n",
        "\n",
        "```python\n",
        "pip install -U coremltools\n",
        "```\n",
        "\n",
        "```python\n",
        "\n",
        "#import coremltools\n",
        "import coremltools\n",
        "\n",
        "#convert model\n",
        "coreml_model = coremltools.converters.caffe.convert('my_caffe_model.caffemodel')\n",
        "\n",
        "#save model to Core ML model format\n",
        "coremltools.utils.save_spec(coreml_model, 'my_model.mlmodel')\n",
        "```\n",
        "\n",
        "\n",
        "**Convertors**\n",
        "\n",
        "\n",
        "\n",
        "*   [MXNet Convertor](https://github.com/apache/incubator-mxnet/tree/master/tools/coreml)\n",
        "*   [TensorFlow Convertor](https://github.com/tf-coreml/tf-coreml)\n",
        "\n",
        "\n",
        "**Resources:**\n",
        "\n",
        "\n",
        "*   [Convert trained Models docs](https://developer.apple.com/documentation/coreml/converting_trained_models_to_core_ml)\n",
        "*   [coremltools](https://pypi.org/project/coremltools/)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i2sO5hiqzTNN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c243a0e6-3f08-4471-dbdf-c85d479f3399"
      },
      "source": [
        "!pip -q install -U coremltools"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     |████████████████████████████████| 3.4MB 4.7MB/s \n",
            "\u001b[?25h"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O_mtYudXzc29",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "ae8e27f1-ba44-4954-8cd5-0415d75a1a7c"
      },
      "source": [
        "import coremltools"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:TensorFlow version 2.3.0 detected. Last version known to be fully compatible is 1.14.0 .\n",
            "WARNING:root:Keras version 2.4.3 detected. Last version known to be fully compatible of Keras is 2.2.4 .\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XN4O6Q7Izj_y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#coremltools.converters.sklearn.convert"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v9oZIawL9q-3",
        "colab_type": "text"
      },
      "source": [
        "#### CoreML Image Classification Demo\n",
        "\n",
        "\n",
        "\n",
        "*   [Classifying Images with Vision and Core ML](https://developer.apple.com/documentation/vision/classifying_images_with_vision_and_core_ml)\n",
        "\n",
        "\n",
        "*Mobilenet.mlmodel (could have been converted)*\n",
        "\n",
        "![Mobilenet.mlmodel](https://user-images.githubusercontent.com/58792/45366919-e1083980-b594-11e8-9b4e-9900da6c452f.png)\n",
        "\n",
        "iOS Simulator Doing Image Classification\n",
        "\n",
        "![iOS Simulator](https://user-images.githubusercontent.com/58792/45366912-dea5df80-b594-11e8-8b67-81571f0f8b44.png)\n",
        "\n",
        "To use MXNet models:\n",
        "\n",
        "https://github.com/apache/incubator-mxnet/tree/master/tools/coreml\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mBv0Qgb5-i2l",
        "colab_type": "text"
      },
      "source": [
        "### Create ML"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CtdR2RNl-kW8",
        "colab_type": "text"
      },
      "source": [
        "Build AutoML https://developer.apple.com/documentation/vision/training_a_create_ml_model_to_classify_flowers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kxhaLibM-_PJ",
        "colab_type": "text"
      },
      "source": [
        "## TFHUB"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xs3U5qu3_Ci8",
        "colab_type": "text"
      },
      "source": [
        "https://tfhub.dev/ml-kit/collections/image-classification/1\n",
        "\n",
        "Comes in .tflite format\n",
        "\n",
        "* Convert to coreml?\n",
        "* Convert to ONNX\n",
        "* Deploy directly to coralhardware"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TCIsoHoDANZh",
        "colab_type": "text"
      },
      "source": [
        "## Additional Computer Devices and Workflows"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I27Sm3MtAP7z",
        "colab_type": "text"
      },
      "source": [
        "* Android from tflite:  https://cloud.google.com/vision/automl/docs/tflite-android-tutorial?hl=en_US\n",
        "* iOS from Google AutoML:  https://developer.apple.com/documentation/vision/classifying_images_with_vision_and_core_ml\n",
        "* Download from Google AutoML and put onto Coral AI:  https://coral.ai/docs/\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "luRBGVwZI_sm",
        "colab_type": "text"
      },
      "source": [
        "### AWS Deep Lense Examples"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cFHe-mWgJCc9",
        "colab_type": "text"
      },
      "source": [
        "![Screen Shot 2020-09-18 at 10 13 27 PM](https://user-images.githubusercontent.com/58792/93656938-43099e00-f9fc-11ea-9bc5-cc9f59cce31e.png)\n",
        "![Screen Shot 2020-09-18 at 10 13 45 PM](https://user-images.githubusercontent.com/58792/93656939-44d36180-f9fc-11ea-828e-9464ac0f2b83.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-W8GGJKZKdhH",
        "colab_type": "text"
      },
      "source": [
        "## Followup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CCcpIkfVKf0d",
        "colab_type": "text"
      },
      "source": [
        "* life long learning\n",
        "* qwiklabs\n",
        "* project management\n",
        "* communication\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "djgR1ScbkQZv",
        "colab_type": "text"
      },
      "source": [
        "## FAQ/Homework"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pra79jnAkUjo",
        "colab_type": "text"
      },
      "source": [
        "Do you need Apple to convert to CoreML?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "alPh481Fkgd9",
        "colab_type": "text"
      },
      "source": [
        "If a neural net can be trained in the cloud and then run on a phone in an edge scenario, then what is moved to the edge device? What gets transferred after training and placed on the device for use?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FNiX_Tf5k3PX",
        "colab_type": "text"
      },
      "source": [
        "Can you please point me to the best available models for agriculture using edge based ML?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZKF3Vh-Bl3e9",
        "colab_type": "text"
      },
      "source": [
        "What is stored in the file transferred from the cloud training model to the edge device. In other words, what is the nature of the model output that would allow functionality on the edge device? Is it a tensor with a legend? "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "omSTDQwaCftQ",
        "colab_type": "text"
      },
      "source": [
        "## References\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r6Nz9zVH8nWO",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "*   [Pragmatic AI: An Introduction to Cloud-Based Machine Learning-Physical Book](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863917)\n",
        "*   [Pragmatic AI: Pragmatic AI: An Introduction to Cloud-Based Machine Learning-SafariOnline Book](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
        "*  [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118)\n",
        "* [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
        "*  [noahgift.com](https://noahgift.com/)\n",
        "*  [paiml.com](https://paiml.com/)\n",
        "*  https://storage.googleapis.com/nexttpu/index.html\n",
        "* https://riscv.org/wp-content/uploads/2017/05/Mon0915-RISC-V-50-Years-Computer-Arch.pdf\n",
        "* https://software.intel.com/en-us/ai-academy/devcloud\n"
      ]
    }
  ]
}