Model_Maker_Object_Detection_Tutorial_detect_heads_using_Open_Images_modified_for_Maple_Syrup_Pi_Camera.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ricardodeazambuja/Maple-Syrup-Pi-Camera/blob/main/Model_Maker_Object_Detection_Tutorial_detect_heads_using_Open_Images_modified_for_Maple_Syrup_Pi_Camera.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h2q27gKz1H20"
      },
      "source": [
        "##### Copyright 2021 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TUfAcER1oUS6",
        "cellView": "form"
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IPeoG18CNt3I"
      },
      "source": [
        "######  Modified by [ricardodeazambuja.com](https://ricardodeazambuja.com)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gb7qyhNL1yWt"
      },
      "source": [
        "# Object Detection with TensorFlow Lite Model Maker - Modified!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sr3q-gvm3cI8"
      },
      "source": [
        "In the [previous notebook](https://github.com/ricardodeazambuja/Maple-Syrup-Pi-Camera/blob/main/Model_Maker_Object_Detection_Tutorial_modified_for_Maple_Syrup_Pi_Camera.ipynb), you learned how to use the [TensorFlow Lite Model Maker](https://www.tensorflow.org/lite/guide/model_maker) library to train a custom object detection model capable of detecting RPI Zero W within images on any device that can run TFLite models. The images were collected directly using Google Image Search, and annotated using the [VGG VIA Annotator](https://www.robots.ox.ac.uk/~vgg/software/via/via.html).\n",
        "\n",
        "This time, instead of using Google Image Search, we will not annotate anything because we will directly use the [Open Images dataset](https://storage.googleapis.com/openimages/web/index.html).\n",
        "\n",
        "The Model Maker library uses *transfer learning* to simplify the process of training a TensorFlow Lite model using a custom dataset. Retraining a TensorFlow Lite model with your own custom dataset reduces the amount of training data required and will shorten the training time."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bcLF2PKkSbV3"
      },
      "source": [
        "## Prerequisites\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2vvAObmTqglq"
      },
      "source": [
        "### Install the required packages\n",
        "Start by installing the required packages, including the Model Maker package from the [GitHub repo](https://github.com/tensorflow/examples/tree/master/tensorflow_examples/lite/model_maker) and the pycocotools library you'll use for evaluation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qhl8lqVamEty"
      },
      "source": [
        "!sudo apt -y install libportaudio2\n",
        "!pip install -q --use-deprecated=legacy-resolver tflite-model-maker==0.4.2 # I hope this will keep this notebook from breaking so often...\n",
        "!pip install -q pycocotools\n",
        "!pip install -q opencv-python-headless==4.1.2.30\n",
        "!pip uninstall -y tensorflow && pip install -q tensorflow==2.8.0 # I hope this will keep this notebook from breaking so often..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4xqAvcn2W1xJ"
      },
      "source": [
        "We will need colab_utils to convert the dataset, augment and split into TRAIN, VALIDATION and TEST."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qW1H_-UlLZLv"
      },
      "source": [
        "!pip install -q git+https://github.com/ricardodeazambuja/colab_utils.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0OiEnyLTiyz4"
      },
      "source": [
        "Let's install some tools to help us play with the [Open Images dataset](https://storage.googleapis.com/openimages/web/index.html). The tools can be found in [my fork of the Open Images repository](https://github.com/ricardodeazambuja/openimages)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GKf6PZ79itjI"
      },
      "source": [
        "!wget -nc -q https://raw.githubusercontent.com/ricardodeazambuja/openimages/main/downloader.py\n",
        "!wget -nc -q https://raw.githubusercontent.com/ricardodeazambuja/openimages/main/bbox_utils.py\n",
        "!pip install -q boto3"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l6lRhVK9Q_0U"
      },
      "source": [
        "Import the required packages."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XtxiUeZEiXpt"
      },
      "source": [
        "import numpy as np\n",
        "import os\n",
        "\n",
        "from tflite_model_maker.config import ExportFormat\n",
        "from tflite_model_maker import model_spec\n",
        "from tflite_model_maker import object_detector\n",
        "\n",
        "from tensorflow_examples.lite.model_maker.third_party.efficientdet.keras import train\n",
        "from tensorflow_examples.lite.model_maker.third_party.efficientdet.keras import train_lib\n",
        "\n",
        "import tensorflow as tf\n",
        "assert tf.__version__.startswith('2')\n",
        "\n",
        "tf.get_logger().setLevel('ERROR')\n",
        "from absl import logging\n",
        "logging.set_verbosity(logging.ERROR)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aW5aSYlVLmM0"
      },
      "source": [
        "from colab_utils import *"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Fix some problems with the original implementation..."
      ],
      "metadata": {
        "id": "31kG44JrMmnA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow_examples.lite.model_maker.third_party.efficientdet import inference\n",
        "from tensorflow_examples.lite.model_maker.third_party.efficientdet.keras import postprocess\n",
        "\n",
        "class DisplayCallback(tf.keras.callbacks.Callback):\n",
        "  \"\"\"Display inference result callback.\"\"\"\n",
        "\n",
        "  def __init__(self, sample_image, output_dir, update_freq=None):\n",
        "    super().__init__()\n",
        "    image_file = tf.io.read_file(sample_image)\n",
        "    self.sample_image = tf.expand_dims(\n",
        "        tf.image.decode_jpeg(image_file, channels=3), axis=0)\n",
        "    self.update_freq = update_freq\n",
        "    self.output_dir = output_dir\n",
        "\n",
        "  def set_model(self, model: tf.keras.Model):\n",
        "    self.model = model\n",
        "    self.config = model.config\n",
        "    self.width, self.height = self.config.image_size\n",
        "    self.sample_image = tf.image.resize(self.sample_image,\n",
        "                                        self.config.image_size,\n",
        "                                        preserve_aspect_ratio=False)\n",
        "    log_dir = os.path.join(self.config.model_dir, 'test_images')\n",
        "    self.file_writer = tf.summary.create_file_writer(log_dir)\n",
        "    self.min_score_thresh = self.config.nms_configs['score_thresh'] or 0.4\n",
        "    self.max_boxes_to_draw = self.config.nms_configs['max_output_size'] or 100\n",
        "\n",
        "  def on_train_batch_end(self, batch, logs=None):\n",
        "    if self.update_freq and batch % self.update_freq == 0:\n",
        "      self._draw_inference(batch)\n",
        "\n",
        "  def _draw_inference(self, step):\n",
        "    num_classes = (self.config.num_classes-1)\n",
        "    cls_outputs, box_outputs = self.model(self.sample_image, training=False)\n",
        "    detections = postprocess.generate_detections(self.config,\n",
        "                                                 cls_outputs,\n",
        "                                                 box_outputs,\n",
        "                                                 [1.0]*num_classes,\n",
        "                                                 list(range(num_classes)))\n",
        "\n",
        "    # detections => [id, x, y, w, h, score, class]\n",
        "    ids, xmin, ymin, w, h, scores, classes = detections[0].numpy().swapaxes(1,0)\n",
        "    # xmin /= self.width\n",
        "    # ymin /= self.height\n",
        "    # w /= self.width\n",
        "    # h /= self.height\n",
        "    boxes = np.vstack([xmin, ymin, w+xmin, h+ymin]).T\n",
        "    image = inference.visualize_image(\n",
        "        self.sample_image[0],\n",
        "        boxes,\n",
        "        classes.astype(int),\n",
        "        scores,\n",
        "        label_map=self.model.config.label_map,\n",
        "        min_score_thresh=self.min_score_thresh,\n",
        "        max_boxes_to_draw=self.max_boxes_to_draw)\n",
        "\n",
        "    with self.file_writer.as_default():\n",
        "      # tensorboard will do some weird normalization, but not if uint8\n",
        "      # https://github.com/tensorflow/tensorboard/issues/1113\n",
        "      tf.summary.image('Test image', tf.expand_dims(tf.cast(image, tf.uint8), axis=0))\n",
        "\n",
        "    return boxes"
      ],
      "metadata": {
        "id": "jgAJxpPJMllZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BRd13bfetO7B"
      },
      "source": [
        "### Prepare the dataset using Open Images\n",
        "\n",
        "\n",
        "We want to study how people move around a certain place (e.g. movement patterns so we can optimize the layout for social distancing). For that reason, we will create an object detector that detects human heads."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EuehjdM72Wn0"
      },
      "source": [
        "from bbox_utils import bboxUtils\n",
        "\n",
        "bu = bboxUtils()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FP9ddRDF26oO"
      },
      "source": [
        "First we check if there's such a label:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Bab5WBy2c7u"
      },
      "source": [
        "bu.search_by_name(\"head\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwgJQJ7X3BHn"
      },
      "source": [
        "Now we get the bboxes for training:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k69CXN-O2V6n"
      },
      "source": [
        "bboxes_train_all = bu.get_bboxes(\"train\", ['/m/04hgtk'])\n",
        "bboxes_train_all"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kvajvkBM3Mcb"
      },
      "source": [
        "And we filter to keep only photographs of real heads, not occluded or truncated, and definitely not from the inside:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TB1X5If921mn"
      },
      "source": [
        "bboxes_train = bboxes_train_all.loc[(bboxes_train_all['IsOccluded']=='0') & (bboxes_train_all['IsTruncated']=='0') & (bboxes_train_all['IsDepiction']=='0') & (bboxes_train_all['IsInside']=='0')]\n",
        "bboxes_train"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tX2luTIhZBo6"
      },
      "source": [
        "Before downloading the images, we want to make sure we have a list with unique file names"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Der1_wAiBdFa"
      },
      "source": [
        "img_ids_train = bu.get_unique_img_ids(bboxes_train)\n",
        "img_ids_train.shape[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ojQuoBZJB9oy"
      },
      "source": [
        "For this example we want to use a maximum of 500 images for training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LgyO8dr-DJBB"
      },
      "source": [
        "# Download images matching ImageIDs from \"train\" set to to directory \"imgs\"\n",
        "n_train_imgs = 500\n",
        "bu.get_images(img_ids_train[:n_train_imgs], \"train\", \"imgs\", num_processes=5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J0-SlgWIDcGe"
      },
      "source": [
        "# Confirming the images are where we expect...\n",
        "!ls -lah imgs | head"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0eYbqCR-ZTQ9"
      },
      "source": [
        "Filter the pandas dataframe to keep only the examples using the images we downloaded"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IlSKWDpnBf_i"
      },
      "source": [
        "bboxes_train_small = bboxes_train.loc[bboxes_train['ImageID'].isin(img_ids_train[:n_train_imgs])].copy()\n",
        "bboxes_train_small"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kT-vm4UEZlzf"
      },
      "source": [
        "As we will use the same CSV format as the [previous notebook](https://github.com/ricardodeazambuja/Maple-Syrup-Pi-Camera/blob/main/Model_Maker_Object_Detection_Tutorial_modified_for_Maple_Syrup_Pi_Camera.ipynb), it's necessary to add the split (TRAIN)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zem-kvnFCR2L"
      },
      "source": [
        "bboxes_train_small['Split'] = ['TRAIN']*bboxes_train_small.shape[0]\n",
        "bboxes_train_small = bboxes_train_small[['Split','ImageID','XMin','YMin','XMax','YMax']]\n",
        "bboxes_train_small"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JS7QuhOLaIrF"
      },
      "source": [
        "Now let's just to the same for the VALIDATION and TEST sets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a7YZyhn4DsTi"
      },
      "source": [
        "bboxes_val_all = bu.get_bboxes(\"validation\", ['/m/04hgtk'])\n",
        "bboxes_val = bboxes_val_all.loc[(bboxes_val_all['IsOccluded']=='0') & (bboxes_val_all['IsTruncated']=='0') & (bboxes_val_all['IsDepiction']=='0') & (bboxes_val_all['IsInside']=='0')]\n",
        "img_ids_val = bu.get_unique_img_ids(bboxes_val)\n",
        "print(f\"Total number of images filtered for validation: {img_ids_val.shape[0]}\")\n",
        "\n",
        "n_val_imgs = 64\n",
        "bu.get_images(img_ids_val[:n_val_imgs], \"validation\", \"imgs\", num_processes=5)\n",
        "bboxes_val_small = bboxes_val.loc[bboxes_val['ImageID'].isin(img_ids_val[:n_val_imgs])].copy()\n",
        "\n",
        "bboxes_val_small['Split'] = ['VALIDATION']*bboxes_val_small.shape[0]\n",
        "bboxes_val_small = bboxes_val_small[['Split','ImageID','XMin','YMin','XMax','YMax']]\n",
        "bboxes_val_small"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l5FksHPAFmTq"
      },
      "source": [
        "bboxes_test_all = bu.get_bboxes(\"test\", ['/m/04hgtk'])\n",
        "bboxes_test = bboxes_test_all.loc[(bboxes_test_all['IsOccluded']=='0') & (bboxes_test_all['IsTruncated']=='0') & (bboxes_test_all['IsDepiction']=='0') & (bboxes_test_all['IsInside']=='0')]\n",
        "img_ids_test = bu.get_unique_img_ids(bboxes_test_all)\n",
        "print(f\"Total number of images filtered for validation: {img_ids_test.shape[0]}\")\n",
        "\n",
        "n_test_imgs = 50\n",
        "bu.get_images(img_ids_test[:n_test_imgs], \"test\", \"imgs\", num_processes=5)\n",
        "bboxes_test_small = bboxes_test.loc[bboxes_test['ImageID'].isin(img_ids_test[:n_test_imgs])].copy()\n",
        "\n",
        "bboxes_test_small['Split'] = ['TEST']*bboxes_test_small.shape[0]\n",
        "bboxes_test_small = bboxes_test_small[['Split','ImageID','XMin','YMin','XMax','YMax']]\n",
        "bboxes_test_small"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "on8PzBAx2XZR"
      },
      "source": [
        "We will convert it to [CSV format](https://cloud.google.com/vision/automl/object-detection/docs/csv-format):\n",
        "```\n",
        "VALIDATION,imgs/image_1.jpg,human_head,0.16,0.17,,,0.70,0.69,,\n",
        "TRAIN,imgs/image_3.jpg,human_head,0.18,0.13,,,0.68,0.54,,\n",
        "TEST,imgs/image_10.jpg,human_head,0.18,0.22,,,0.71,0.56,,\n",
        "```\n",
        "\n",
        "* Each row corresponds to an object localized inside a larger image, with each object specifically designated as test, train, or validation data. You'll learn more about what that means in a later stage in this notebook.\n",
        "* Each row has a label: `rpi0` (because we only annotated for that!)\n",
        "* Bounding boxes are specified for each image using the top left and bottom right vertices.\n",
        "\n",
        "Where the bbox is specified as:  \n",
        "`x_relative_min, y_relative_min,,,x_relative_max,y_relative_max,,`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "thUjOSdUaWsd"
      },
      "source": [
        "Concatenating all datasets together"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XqMMJAx9Ms_w"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "result = pd.concat([bboxes_train_small,bboxes_val_small,bboxes_test_small])\n",
        "\n",
        "# Adding the label\n",
        "result['Label'] = ['human_head']*result.shape[0]\n",
        "\n",
        "# Adding the directory and file extension\n",
        "result['ImageID'] = result['ImageID'].apply(lambda f: \"imgs/\"+f+\".jpg\")\n",
        "\n",
        "# Making the dataframe just like the CSV format explained above\n",
        "result['Filler'] = ['']*result.shape[0]\n",
        "result[['Split','ImageID','Label','XMin','YMin','Filler','Filler','XMax','YMax','Filler','Filler']]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SwGiYhKPOXzz"
      },
      "source": [
        "result[['Split','ImageID','Label','XMin','YMin','Filler','Filler','XMax','YMax','Filler','Filler']].to_csv(\"full_dataset.csv\",header=False, index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pZL9jQIIOlvJ"
      },
      "source": [
        "# Let's confirm it worked...\n",
        "!head -n 15 full_dataset.csv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xushUyZXqP59"
      },
      "source": [
        "## Quickstart"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vn61LJ9QbOPi"
      },
      "source": [
        "There are six steps to training an object detection model:\n",
        "\n",
        "**Step 1. Choose an object detection model archiecture.**\n",
        "\n",
        "This tutorial uses the EfficientDet-Lite0 model. EfficientDet-Lite[0-4] are a family of mobile/IoT-friendly object detection models derived from the [EfficientDet](https://arxiv.org/abs/1911.09070) architecture.\n",
        "\n",
        "Here is the performance of each EfficientDet-Lite models compared to each others.\n",
        "\n",
        "| Model architecture | Size(MB)* | Latency(ms)** | Average Precision*** |\n",
        "|--------------------|-----------|---------------|----------------------|\n",
        "| EfficientDet-Lite0 | 4.4       | 37            | 25.69%               |\n",
        "| EfficientDet-Lite1 | 5.8       | 49            | 30.55%               |\n",
        "| EfficientDet-Lite2 | 7.2       | 69            | 33.97%               |\n",
        "| EfficientDet-Lite3 | 11.4      | 116           | 37.70%               |\n",
        "| EfficientDet-Lite4 | 19.9      | 260           | 41.96%               |\n",
        "\n",
        "<i> * Size of the integer quantized models. <br/>\n",
        "** Latency measured on Pixel 4 using 4 threads on CPU. <br/>\n",
        "*** Average Precision is the mAP (mean Average Precision) on the COCO 2017 validation dataset.\n",
        "</i>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CtdZ-JDwMimd"
      },
      "source": [
        "spec = model_spec.get('efficientdet_lite0')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(spec.config)"
      ],
      "metadata": {
        "id": "QkzG7KYXNW1o"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "spec.config['model_dir']"
      ],
      "metadata": {
        "id": "Kb631OqENbEJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s5U-A3tw6Y27"
      },
      "source": [
        "**Step 2. Load the dataset.**\n",
        "\n",
        "Model Maker will take input data in the CSV format. Use the `object_detector.DataLoader.from_csv` method to load the dataset and split them into the training, validation and test images.\n",
        "\n",
        "* Training images: These images are used to train the object detection model to recognize salad ingredients.\n",
        "* Validation images: These are images that the model didn't see during the training process. You'll use them to decide when you should stop the training, to avoid [overfitting](https://en.wikipedia.org/wiki/Overfitting).\n",
        "* Test images: These images are used to evaluate the final model performance.\n",
        "\n",
        "You can load the CSV file directly from Google Cloud Storage, but you don't need to keep your images on Google Cloud to use Model Maker. You can specify a local CSV file on your computer, and Model Maker will work just fine."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hGjp7PPUNKiK"
      },
      "source": [
        "# Total number of samples\n",
        "!cat full_dataset.csv | wc -l"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d3jQlqt3Wk1j"
      },
      "source": [
        "# Samples according to the split\n",
        "!cat full_dataset.csv | grep TRAIN | wc -l\n",
        "!cat full_dataset.csv | grep TEST | wc -l\n",
        "!cat full_dataset.csv | grep VALIDATION | wc -l"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GgecW-LF_goY"
      },
      "source": [
        "from PIL import Image\n",
        "\n",
        "with open(\"full_dataset.csv\", \"r\") as file:\n",
        "  ds = file.read()\n",
        "\n",
        "ann_number = 36\n",
        "\n",
        "l = ds.splitlines()[ann_number-1].split(\",\")\n",
        "print(l)\n",
        "drawbox(Image.open(l[1]).resize((800,600)), [float(i) for i in [l[3],l[4], l[7], l[8]]])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HD5BvzWe6YKa"
      },
      "source": [
        "train_data, validation_data, test_data = object_detector.DataLoader.from_csv('full_dataset.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pEcU-673SaUv"
      },
      "source": [
        "train_data.label_map"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2uZkLR6N6gDR"
      },
      "source": [
        "**Step 3. Train the TensorFlow model with the training data.**\n",
        "\n",
        "* The EfficientDet-Lite0 model uses `epochs = 50` by default, which means it will go through the training dataset 50 times. You can look at the validation accuracy during training and stop early to avoid overfitting.\n",
        "* Set `batch_size = 64` here so you will see that it takes 18 steps to go through the 1109 images in the training dataset.\n",
        "* Set `train_whole_model=True` to fine-tune the whole model instead of just training the head layer to improve accuracy. The trade-off is that it may take longer to train the model."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        " %load_ext tensorboard\n",
        "#%reload_ext tensorboard"
      ],
      "metadata": {
        "id": "Uxr76xLINp7r"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pkill tensorboard  # trust me, you will eventually need this..."
      ],
      "metadata": {
        "id": "PPt0wBDwNqXx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# from tensorboard import notebook\n",
        "# notebook.list() # View open TensorBoard instances\n",
        "#notebook.manager.start([\"serve\", \"--logdir\", f\"{model_dir}\"])  # starts a server using the model_dir\n",
        "#notebook.display(port=6006, height=1000)  # just create the visualization, without starting a new server"
      ],
      "metadata": {
        "id": "hO2LiVb9NsIJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# If you want to set a custom dir:\n",
        "# spec.config.model_dir = \"your_custom_dir\"\n",
        "model_dir = spec.config.model_dir\n",
        "model_dir"
      ],
      "metadata": {
        "id": "sXcDW46fNuXc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "ls $model_dir -la"
      ],
      "metadata": {
        "id": "YmLQZifyNyPh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "spec.config.sample_image = l[1]\n",
        "spec.config.sample_image"
      ],
      "metadata": {
        "id": "gA3VA53UN0KK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "spec.config.img_summary_steps = 1  # step size (in relation to batches)\n",
        "spec.config.img_summary_steps"
      ],
      "metadata": {
        "id": "ykZrEZiNN49o"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "spec.config.map_freq = 1  # AP eval frequency in epochs.\n",
        "spec.config.map_freq"
      ],
      "metadata": {
        "id": "6Iqn2HBOOGay"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "spec.config.verbose = 1 # shows when the checkpoints are really saved\n",
        "spec.config.verbose"
      ],
      "metadata": {
        "id": "dngguRRbOISS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# check https://github.com/tensorflow/examples/blob/37df0b9a2f7240831642cea4380e4f7ec33bee83/tensorflow_examples/lite/model_maker/third_party/efficientdet/aug/autoaugment.py\n",
        "spec.config.autoaugment_policy = 'v2'  # None or v0 to v3, it will take longer to train it...\n",
        "spec.config.autoaugment_policy"
      ],
      "metadata": {
        "id": "72063FCLOPpD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "batch_size = 64\n",
        "with spec.ds_strategy.scope():\n",
        "  object_detector_model = object_detector.create(train_data, model_spec=spec, batch_size=batch_size, train_whole_model=True, validation_data=validation_data, do_train=False)\n",
        "  train_ds, steps_per_epoch, _ = object_detector_model._get_dataset_and_steps(\n",
        "      train_data, batch_size, is_training=True)\n",
        "  validation_ds, validation_steps, val_json_file = object_detector_model._get_dataset_and_steps(\n",
        "      validation_data, batch_size, is_training=False)\n",
        "\n",
        "  config = object_detector_model.model.config\n",
        "  batch_size = config.batch_size\n",
        "  config.update(\n",
        "      dict(\n",
        "          steps_per_epoch=steps_per_epoch,\n",
        "          eval_samples=batch_size * validation_steps,\n",
        "          val_json_file=val_json_file,\n",
        "          batch_size=batch_size))\n",
        "  train.setup_model(object_detector_model.model, config)\n",
        "  train.init_experimental(config)"
      ],
      "metadata": {
        "id": "x7NfEzepORiT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(config)"
      ],
      "metadata": {
        "id": "m0F26_v5OVWy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "object_detector_model.model.summary()"
      ],
      "metadata": {
        "id": "lbGkRmR2OW96"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%tensorboard --logdir $model_dir --bind_all"
      ],
      "metadata": {
        "id": "Fs7lnz7ROZCq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# !tensorboard dev upload --logdir $model_dir \\\n",
        "#   --name \"Model Maker Object Detection Tutorial - modified for Maple Syrup Pi Camera\" \\\n",
        "#   --description \"Training results from https://github.com/ricardodeazambuja/Maple-Syrup-Pi-Camera/blob/main/Model_Maker_Object_Detection_Tutorial_modified_for_Maple_Syrup_Pi_Camera.ipynb\" \\\n",
        "#   --one_shot"
      ],
      "metadata": {
        "id": "B6cOmQP8Ojtb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Train the model for a certain number of epochs defined by `epochs`:     \n",
        "(if you want to train more epochs, just update the `initial_epoch`, add it to epochs, and run the cell again)"
      ],
      "metadata": {
        "id": "xW7VzSt9OgWt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#!rm -rf $model_dir/*  # this will delete all the saved stuff..."
      ],
      "metadata": {
        "id": "vihghooInuTS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "ls $model_dir"
      ],
      "metadata": {
        "id": "4MFHLT0FnvHP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "epochs = 50\n",
        "initial_epoch = 0 # it's always (last_epoch-1), so it starts at 0\n",
        "load_latest_checkpoint = False\n",
        "\n",
        "if load_latest_checkpoint:\n",
        "  object_detector_model.model.load_weights(tf.train.latest_checkpoint(model_dir))\n",
        "\n",
        "with spec.ds_strategy.scope():\n",
        "  train_lib.DisplayCallback = DisplayCallback # monkey patch with my alternative version...\n",
        "  keras_cb = train_lib.get_callbacks(config.as_dict(), validation_ds)\n",
        "  keras_cb[2].set_model(object_detector_model.model)\n",
        "  keras_cb[3].set_model(object_detector_model.model)\n",
        "\n",
        "  model = object_detector_model.model\n",
        "  model.fit(\n",
        "      train_ds,\n",
        "      epochs=epochs,\n",
        "      #initial_epoch=initial_epoch,\n",
        "      steps_per_epoch=steps_per_epoch,\n",
        "      callbacks=keras_cb,\n",
        "      validation_data=validation_ds,\n",
        "      validation_steps=validation_steps)"
      ],
      "metadata": {
        "id": "FlcpqSEbOcD6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# to load a specific epoch (choose from the results above that best fit your needs, check tensorboard to have a better view)\n",
        "# epoch = 10\n",
        "# _ = object_detector_model.model.load_weights(model_dir + f\"/ckpt-{epoch}\")\n",
        "# to load the latest\n",
        "# _ = object_detector_model.model.load_weights(tf.train.latest_checkpoint(model_dir))"
      ],
      "metadata": {
        "id": "KSM8LslCPY-z"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-BzCHLWJ6h7q"
      },
      "source": [
        "**Step 4. Evaluate the model with the test data.**\n",
        "\n",
        "After training the object detection model using the images in the training dataset, use the remaining ?? images in the test dataset to evaluate how the model performs against new data it has never seen before.\n",
        "\n",
        "As the default batch size is ??, it will take 1 step to go through the ?? images in the test dataset.\n",
        "\n",
        "The evaluation metrics are same as [COCO](https://cocodataset.org/#detection-eval)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fv4BlI3_gdUR"
      },
      "source": [
        "object_detector_model.evaluate(train_data) # let's check how it performs on the training data... just to make sure it learned something"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8xmnl6Yy7ARn"
      },
      "source": [
        "object_detector_model.evaluate(test_data) # ok, now using the unseen test data!"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CgCDMe0e6jlT"
      },
      "source": [
        "**Step 5.  Export as a TensorFlow Lite model.**\n",
        "\n",
        "Export the trained object detection model to the TensorFlow Lite format by specifying which folder you want to export the quantized model to. The default post-training quantization technique is full integer quantization."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hm_UULdW7A9T"
      },
      "source": [
        "object_detector_model.export(export_dir='.') # kind of slow..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZQpahAIBqBPp"
      },
      "source": [
        "**Step 6.  Evaluate the TensorFlow Lite model.**\n",
        "\n",
        "Several factors can affect the model accuracy when exporting to TFLite:\n",
        "* [Quantization](https://www.tensorflow.org/lite/performance/model_optimization) helps shrinking the model size by 4 times at the expense of some accuracy drop.\n",
        "* The original TensorFlow model uses per-class [non-max supression (NMS)](https://www.coursera.org/lecture/convolutional-neural-networks/non-max-suppression-dvrjH) for post-processing, while the TFLite model uses global NMS that's much faster but less accurate.\n",
        "Keras outputs maximum 100 detections while tflite outputs maximum 25 detections.\n",
        "\n",
        "Therefore you'll have to evaluate the exported TFLite model and compare its accuracy with the original TensorFlow model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RS3Ell_lqH4e"
      },
      "source": [
        "object_detector_model.evaluate_tflite('model.tflite', test_data) # the AP dropped a little bit after converting."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rVxaf3x_7OfB"
      },
      "source": [
        "You can download the TensorFlow Lite model file using the left sidebar of Colab. Right-click on the `model.tflite` file and choose `Download` to download it to your local computer.\n",
        "\n",
        "This model can be integrated into an Android or an iOS app using the [ObjectDetector API](https://www.tensorflow.org/lite/inference_with_metadata/task_library/object_detector) of the [TensorFlow Lite Task Library](https://www.tensorflow.org/lite/inference_with_metadata/task_library/overview).\n",
        "\n",
        "See the [TFLite Object Detection sample app](https://github.com/tensorflow/examples/blob/master/lite/examples/object_detection/android/lib_task_api/src/main/java/org/tensorflow/lite/examples/detection/tflite/TFLiteObjectDetectionAPIModel.java#L91) for more details on how the model is used in an working app.\n",
        "\n",
        "*Note: Android Studio Model Binding does not support object detection yet so please use the TensorFlow Lite Task Library.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "me6_RwPZqNhX"
      },
      "source": [
        "## (Optional) Test the TFLite model on your image\n",
        "\n",
        "You can test the trained TFLite model using images from the internet.\n",
        "* Replace the `INPUT_IMAGE_URL` below with your desired input image.\n",
        "* Adjust the `DETECTION_THRESHOLD` to change the sensitivity of the model. A lower threshold means the model will pickup more objects but there will also be more false detection. Meanwhile, a higher threshold means the model will only pickup objects that it has confidently detected.\n",
        "\n",
        "Although it requires some of boilerplate code to run the model in Python at this moment, integrating the model into a mobile app only requires a few lines of code."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XqS0rFCrqM1o",
        "cellView": "form"
      },
      "source": [
        "#@title Load the trained TFLite model and define some visualization functions\n",
        "\n",
        "import cv2\n",
        "\n",
        "from PIL import Image\n",
        "\n",
        "model_path = 'model.tflite'\n",
        "\n",
        "# Load the labels into a list\n",
        "classes = ['???'] * object_detector_model.model_spec.config.num_classes\n",
        "label_map = object_detector_model.model_spec.config.label_map\n",
        "for label_id, label_name in label_map.as_dict().items():\n",
        "  classes[label_id-1] = label_name\n",
        "\n",
        "# Define a list of colors for visualization\n",
        "COLORS = np.random.randint(0, 255, size=(len(classes), 3), dtype=np.uint8)\n",
        "\n",
        "def preprocess_image(image_path, input_size):\n",
        "  \"\"\"Preprocess the input image to feed to the TFLite model\"\"\"\n",
        "  img = tf.io.read_file(image_path)\n",
        "  img = tf.io.decode_image(img, channels=3)\n",
        "  img = tf.image.convert_image_dtype(img, tf.uint8)\n",
        "  original_image = img\n",
        "  resized_img = tf.image.resize(img, input_size)\n",
        "  resized_img = resized_img[tf.newaxis, :]\n",
        "  return resized_img, original_image\n",
        "\n",
        "\n",
        "def set_input_tensor(interpreter, image):\n",
        "  \"\"\"Set the input tensor.\"\"\"\n",
        "  tensor_index = interpreter.get_input_details()[0]['index']\n",
        "  input_tensor = interpreter.tensor(tensor_index)()[0]\n",
        "  input_tensor[:, :] = image\n",
        "\n",
        "\n",
        "def get_output_tensor(interpreter, index):\n",
        "  \"\"\"Retur the output tensor at the given index.\"\"\"\n",
        "  output_details = interpreter.get_output_details()[index]\n",
        "  tensor = np.squeeze(interpreter.get_tensor(output_details['index']))\n",
        "  return tensor\n",
        "\n",
        "\n",
        "def detect_objects(interpreter, image, threshold):\n",
        "  \"\"\"Returns a list of detection results, each a dictionary of object info.\"\"\"\n",
        "  # # Feed the input image to the model\n",
        "  # set_input_tensor(interpreter, image)\n",
        "  # interpreter.invoke()\n",
        "  # # Get all outputs from the model\n",
        "  # scores = get_output_tensor(interpreter, 0)\n",
        "  # boxes = get_output_tensor(interpreter, 1)\n",
        "  # count = int(get_output_tensor(interpreter, 2))\n",
        "  # classes = get_output_tensor(interpreter, 3)\n",
        "\n",
        "  signature_fn = interpreter.get_signature_runner()\n",
        "  # Feed the input image to the model\n",
        "  output = signature_fn(images=image.numpy().astype('uint8'))\n",
        "  # Get all outputs from the model\n",
        "  count = int(np.squeeze(output['output_0']))\n",
        "  scores = np.squeeze(output['output_1'])\n",
        "  classes = np.squeeze(output['output_2'])\n",
        "  boxes = np.squeeze(output['output_3'])\n",
        "\n",
        "  results = []\n",
        "  for i in range(count):\n",
        "    if scores[i] >= threshold:\n",
        "      result = {\n",
        "        'bounding_box': boxes[i],\n",
        "        'class_id': classes[i],\n",
        "        'score': scores[i]\n",
        "      }\n",
        "      results.append(result)\n",
        "  return results\n",
        "\n",
        "\n",
        "def run_odt_and_draw_results(image_path, interpreter, threshold=0.5):\n",
        "  \"\"\"Run object detection on the input image and draw the detection results\"\"\"\n",
        "  # Load the input shape required by the model\n",
        "  _, input_height, input_width, _ = interpreter.get_input_details()[0]['shape']\n",
        "\n",
        "  # Load the input image and preprocess it\n",
        "  preprocessed_image, original_image = preprocess_image(\n",
        "      image_path,\n",
        "      (input_height, input_width)\n",
        "    )\n",
        "\n",
        "  # Run object detection on the input image\n",
        "  results = detect_objects(interpreter, preprocessed_image, threshold=threshold)\n",
        "\n",
        "  # Plot the detection results on the input image\n",
        "  original_image_np = original_image.numpy().astype(np.uint8)\n",
        "  for obj in results:\n",
        "    # Convert the object bounding box from relative coordinates to absolute\n",
        "    # coordinates based on the original image resolution\n",
        "    ymin, xmin, ymax, xmax = obj['bounding_box']\n",
        "    xmin = int(xmin * original_image_np.shape[1])\n",
        "    xmax = int(xmax * original_image_np.shape[1])\n",
        "    ymin = int(ymin * original_image_np.shape[0])\n",
        "    ymax = int(ymax * original_image_np.shape[0])\n",
        "\n",
        "    # Find the class index of the current object\n",
        "    class_id = int(obj['class_id'])\n",
        "\n",
        "    # Draw the bounding box and label on the image\n",
        "    color = [int(c) for c in COLORS[class_id]]\n",
        "    cv2.rectangle(original_image_np, (xmin, ymin), (xmax, ymax), color, 2)\n",
        "    # Make adjustments to make the label visible for all objects\n",
        "    y = ymin - 15 if ymin - 15 > 15 else ymin + 15\n",
        "    label = \"{}: {:.0f}%\".format(classes[class_id], obj['score'] * 100)\n",
        "    cv2.putText(original_image_np, label, (xmin, y),\n",
        "        cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2)\n",
        "\n",
        "  # Return the final image\n",
        "  original_uint8 = original_image_np.astype(np.uint8)\n",
        "  return original_uint8"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8PqmKxKlE9Up"
      },
      "source": [
        "# https://image.cnbcfm.com/api/v1/image/106316521-1577799443873art19_2019mayeandkidslaughing.jpg?v=1577799781&w=740&h=416\n",
        "# https://media.npr.org/assets/img/2019/12/03/ap_080902152760-92f37e95e401b6576ccc475c4e75099faaf13ef7-s800-c85.jpg"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GkXtipXKqXp4",
        "cellView": "form"
      },
      "source": [
        "#@title Run object detection and show the detection results on unseen image\n",
        "\n",
        "INPUT_IMAGE_URL = \"https://media.npr.org/assets/img/2019/12/03/ap_080902152760-92f37e95e401b6576ccc475c4e75099faaf13ef7-s800-c85.jpg\" #@param {type:\"string\"}\n",
        "DETECTION_THRESHOLD = 0.5 #@param {type:\"number\"}\n",
        "\n",
        "TEMP_FILE = '/tmp/image.png'\n",
        "\n",
        "!wget -q -O $TEMP_FILE $INPUT_IMAGE_URL\n",
        "im = Image.open(TEMP_FILE)\n",
        "im.thumbnail((512, 512), Image.ANTIALIAS)\n",
        "im.save(TEMP_FILE, 'PNG')\n",
        "\n",
        "# Load the TFLite model\n",
        "interpreter = tf.lite.Interpreter(model_path=model_path)\n",
        "interpreter.allocate_tensors()\n",
        "\n",
        "# Run inference and draw detection result on the local copy of the original file\n",
        "detection_result_image = run_odt_and_draw_results(\n",
        "    TEMP_FILE,\n",
        "    interpreter,\n",
        "    threshold=DETECTION_THRESHOLD\n",
        ")\n",
        "\n",
        "# Show the detection result\n",
        "Image.fromarray(detection_result_image)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oxgWQyYOqZha"
      },
      "source": [
        "## (Optional) Compile For the Edge TPU\n",
        "\n",
        "Now that you have a quantized EfficientDet Lite model, it is possible to compile and deploy to a [Coral EdgeTPU](https://coral.ai/).\n",
        "\n",
        "**Step 1. Install the EdgeTPU Compiler**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Oy3QIn_YqaRP"
      },
      "source": [
        "! curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -\n",
        "\n",
        "! echo \"deb https://packages.cloud.google.com/apt coral-edgetpu-stable main\" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list\n",
        "\n",
        "! sudo apt-get update\n",
        "\n",
        "! sudo apt-get install edgetpu-compiler"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qRWewhqFqeL_"
      },
      "source": [
        "**Step 2. Select number of Edge TPUs, Compile**\n",
        "\n",
        "The EdgeTPU has 8MB of SRAM for caching model paramaters ([more info](https://coral.ai/docs/edgetpu/compiler/#parameter-data-caching)). This means that for models that are larger than 8MB, inference time will be increased in order to transfer over model paramaters. One way to avoid this is [Model Pipelining](https://coral.ai/docs/edgetpu/pipeline/) - splitting the model into segments that can have a dedicated EdgeTPU. This can significantly improve latency.\n",
        "\n",
        "The below table can be used as a reference for the number of Edge TPUs to use - the larger models will not compile for a single TPU as the intermediate tensors can't fit in on-chip memory.\n",
        "\n",
        "| Model architecture | Minimum TPUs | Recommended TPUs\n",
        "|--------------------|-------|-------|\n",
        "| EfficientDet-Lite0 | 1     | 1     |\n",
        "| EfficientDet-Lite1 | 1     | 1     |\n",
        "| EfficientDet-Lite2 | 1     | 2     |\n",
        "| EfficientDet-Lite3 | 2     | 2     |\n",
        "| EfficientDet-Lite4 | 2     | 3     |"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**NEW:** The edgetpu runtime version installed in the RPI image is 13, therefore you need to pass `-m 13` to the edgetpu_compiler  "
      ],
      "metadata": {
        "id": "-Wia0plJWXf9"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LZdonJGCqieU"
      },
      "source": [
        "NUMBER_OF_TPUS = 1\n",
        "\n",
        "!edgetpu_compiler -m 13 -s model.tflite --num_segments=$NUMBER_OF_TPUS"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-g6_KQXnqlTC"
      },
      "source": [
        "**Step 3. Download, Run Model**\n",
        "\n",
        "With the model(s) compiled, they can now be run on EdgeTPU(s) for object detection. First, download the compiled TensorFlow Lite model file using the left sidebar of Colab. Right-click on the `model_edgetpu.tflite` file and choose `Download` to download it to your local computer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VkQFz_qzqrrA"
      },
      "source": [
        "Now you can run the model in your preferred manner. Examples of detection include:\n",
        "* [pycoral detection](https://github.com/google-coral/pycoral/blob/master/examples/detect_image.py)\n",
        "* [Basic TFLite detection](https://github.com/google-coral/tflite/tree/master/python/examples/detection)\n",
        "* [Example Video Detection](https://github.com/google-coral/examples-camera)\n",
        "* [libcoral C++ API](https://github.com/google-coral/libcoral)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EoWiA_zX8rxE"
      },
      "source": [
        "## Advanced Usage\n",
        "\n",
        "This section covers advanced usage topics like adjusting the model and the training hyperparameters."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p79NHCx0xFqb"
      },
      "source": [
        "### Load the dataset\n",
        "\n",
        "#### Load your own data\n",
        "\n",
        "You can upload your own dataset to work through this tutorial. Upload your dataset by using the left sidebar in Colab.\n",
        "\n",
        "<img src=\"https://storage.googleapis.com/download.tensorflow.org/models/tflite/screenshots/model_maker_object_detection.png\" alt=\"Upload File\" width=\"1000\" hspace=\"0\">\n",
        "\n",
        "If you prefer not to upload your dataset to the cloud, you can also locally run the library by following the [guide](https://github.com/tensorflow/examples/tree/master/tensorflow_examples/lite/model_maker).\n",
        "\n",
        "#### Load your data with a different data format\n",
        "\n",
        "The Model Maker library also supports the `object_detector.DataLoader.from_pascal_voc` method to load data with [PASCAL VOC](https://towardsdatascience.com/coco-data-format-for-object-detection-a4c5eaf518c5#:~:text=Pascal%20VOC%20is%20an%20XML,for%20training%2C%20testing%20and%20validation) format. [makesense.ai](https://www.makesense.ai/) and [LabelImg](https://github.com/tzutalin/labelImg) are the tools that can annotate the image and save annotations as XML files in PASCAL VOC data format:\n",
        "```python\n",
        "object_detector.DataLoader.from_pascal_voc(image_dir, annotations_dir, label_map={1: \"person\", 2: \"notperson\"})\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E8VxPiOLy4Gv"
      },
      "source": [
        "### Customize the EfficientDet model hyperparameters\n",
        "\n",
        "The model and training pipline parameters you can adjust are:\n",
        "\n",
        "* `model_dir`: The location to save the model checkpoint files. If not set, a temporary directory will be used.\n",
        "* `steps_per_execution`: Number of steps per training execution.\n",
        "* `moving_average_decay`: Float. The decay to use for maintaining moving averages of the trained parameters.\n",
        "* `var_freeze_expr`: The regular expression to map the prefix name of variables to be frozen which means remaining the same during training. More specific, use `re.match(var_freeze_expr, variable_name)` in the codebase to map the variables to be frozen.\n",
        "* `tflite_max_detections`: integer, 25 by default. The max number of output detections in the TFLite model.\n",
        "* `strategy`:  A string specifying which distribution strategy to use. Accepted values are 'tpu', 'gpus', None. tpu' means to use TPUStrategy. 'gpus' mean to use MirroredStrategy for multi-gpus. If None, use TF default with OneDeviceStrategy.\n",
        "* `tpu`:  The Cloud TPU to use for training. This should be either the name used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 url.\n",
        "* `use_xla`: Use XLA even if strategy is not tpu. If strategy is tpu, always use XLA, and this flag has no effect.\n",
        "* `profile`: Enable profile mode.\n",
        "* `debug`: Enable debug mode.\n",
        "\n",
        "Other parameters that can be adjusted is shown in [hparams_config.py](https://github.com/google/automl/blob/df451765d467c5ed78bbdfd632810bc1014b123e/efficientdet/hparams_config.py#L170).\n",
        "\n",
        "\n",
        "For instance, you can set the `var_freeze_expr='efficientnet'` which freezes the variables with name prefix `efficientnet` (default is `'(efficientnet|fpn_cells|resample_p6)'`). This allows the model to freeze untrainable variables and keep their value the same through training.\n",
        "\n",
        "```python\n",
        "spec = model_spec.get('efficientdet-lite0')\n",
        "spec.config.var_freeze_expr = 'efficientnet'\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4J2qre1fwXsi"
      },
      "source": [
        "### Change the Model Architecture\n",
        "\n",
        "You can change the model architecture by changing the `model_spec`. For instance, change the `model_spec` to the EfficientDet-Lite4 model.\n",
        "\n",
        "```python\n",
        "spec = model_spec.get('efficientdet-lite4')\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LvQuy7RSDir3"
      },
      "source": [
        "### Tune the training hyperparameters\n",
        "\n",
        "The `create` function is the driver function that the Model Maker library uses to create models. The `model_spec` parameter defines the model specification. The `object_detector.EfficientDetSpec` class is currently supported. The `create` function comprises of the following steps:\n",
        "\n",
        "1. Creates the model for the object detection according to `model_spec`.\n",
        "2. Trains the model.  The default epochs and the default batch size are set by the `epochs` and `batch_size` variables in the `model_spec` object.\n",
        "You can also tune the training hyperparameters like `epochs` and `batch_size` that affect the model accuracy. For instance,\n",
        "\n",
        "*   `epochs`: Integer, 50 by default. More epochs could achieve better accuracy, but may lead to overfitting.\n",
        "*   `batch_size`: Integer, 64 by default. The number of samples to use in one training step.\n",
        "*   `train_whole_model`: Boolean, False by default. If true, train the whole model. Otherwise, only train the layers that do not match `var_freeze_expr`.\n",
        "\n",
        "For example, you can train with less epochs and only the head layer. You can increase the number of epochs for better results.\n",
        "\n",
        "```python\n",
        "model = object_detector.create(train_data, model_spec=spec, epochs=10, validation_data=validation_data)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3vPyZInPxJBT"
      },
      "source": [
        "### Export to different formats"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0xqNIcBM-4YR"
      },
      "source": [
        "The export formats can be one or a list of the following:\n",
        "\n",
        "*   `ExportFormat.TFLITE`\n",
        "*   `ExportFormat.LABEL`\n",
        "*   `ExportFormat.SAVED_MODEL`\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "enhsZhW3ApcX"
      },
      "source": [
        "By default, it exports only the TensorFlow Lite model file containing the model [metadata](https://www.tensorflow.org/lite/convert/metadata) so that you can later use in an on-device ML application. The label file is embedded in metadata.\n",
        "\n",
        "In many on-device ML application, the model size is an important factor. Therefore, it is recommended that you quantize the model to make it smaller and potentially run faster. As for EfficientDet-Lite models, full integer quantization  is used to quantize the model by default. Please refer to [Post-training quantization](https://www.tensorflow.org/lite/performance/post_training_quantization) for more detail.\n",
        "\n",
        "```python\n",
        "model.export(export_dir='.')\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RLGZs6InAnP5"
      },
      "source": [
        "You can also choose to export other files related to the model for better examination. For instance, exporting both the saved model and the label file as follows:\n",
        "```python\n",
        "model.export(export_dir='.', export_format=[ExportFormat.SAVED_MODEL, ExportFormat.LABEL])\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W5q_McchQ2C4"
      },
      "source": [
        "### Customize Post-training quantization on the TensorFlow Lite model\n",
        "\n",
        "[Post-training quantization](https://www.tensorflow.org/lite/performance/post_training_quantization) is a conversion technique that can reduce model size and inference latency, while also improving CPU and hardware accelerator inference speed, with a little degradation in model accuracy. Thus, it's widely used to optimize the model.\n",
        "\n",
        "Model Maker library applies a default post-training quantization techique when exporting the model. If you want to customize post-training quantization, Model Maker supports multiple post-training quantization options using [QuantizationConfig](https://www.tensorflow.org/lite/api_docs/python/tflite_model_maker/config/QuantizationConfig) as well. Let's take float16 quantization as an instance. First, define the quantization config.\n",
        "\n",
        "```python\n",
        "config = QuantizationConfig.for_float16()\n",
        "```\n",
        "\n",
        "\n",
        "Then we export the TensorFlow Lite model with such configuration.\n",
        "\n",
        "```python\n",
        "model.export(export_dir='.', tflite_filename='model_fp16.tflite', quantization_config=config)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HS4u77W5gnzQ"
      },
      "source": [
        "# Read more\n",
        "\n",
        "You can read our [object detection](https://www.tensorflow.org/lite/examples/object_detection/overview) example to learn technical details. For more information, please refer to:\n",
        "\n",
        "*   TensorFlow Lite Model Maker [guide](https://www.tensorflow.org/lite/guide/model_maker) and [API reference](https://www.tensorflow.org/lite/api_docs/python/tflite_model_maker).\n",
        "*   Task Library: [ObjectDetector](https://www.tensorflow.org/lite/inference_with_metadata/task_library/object_detector) for deployment.\n",
        "*   The end-to-end reference apps: [Android](https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android), [iOS](https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/ios), and [Raspberry PI](https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/raspberry_pi).\n",
        "\n"
      ]
    }
  ]
}