application_model_zoo/Example - Monkey detection in the wild.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/Tessellate-Imaging/Monk_Object_Detection/blob/master/application_model_zoo/Example%20-%20Monkey%20detection%20in%20the%20wild.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of contents\n",
    "\n",
    "\n",
    "## 1. Use trained model on Jetson Nano Board with board setup instructions\n",
    "\n",
    "\n",
    "\n",
    "## 2. Installation Instructions on cloud/local machine\n",
    "\n",
    "\n",
    "\n",
    "## 3. Use trained model to detect vehicles on road on cloud/local machines\n",
    "\n",
    "\n",
    "\n",
    "## 4. How to train using Tensorflow object detection API wrapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  More examples on \n",
    "  - Tensorflow object detection API 1.0 - https://github.com/Tessellate-Imaging/Monk_Object_Detection/tree/master/example_notebooks/12_tf_obj_1\n",
    "  - Tensorflow object detection API 2.0 - https://github.com/Tessellate-Imaging/Monk_Object_Detection/tree/master/example_notebooks/13_tf_obj_2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use trained model on Jetson Nano Board with board setup instructions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup your Jetson Nano Board\n",
    "\n",
    " - Step 1: Download Jetpack 4.3 SD Card Image https://developer.nvidia.com/jetpack-43-archive\n",
    " \n",
    " - Step 2: Write this image on SD Card. You may use https://www.balena.io/etcher/\n",
    " \n",
    " - Step 3: Attach your SD Ccard to Nano board and boot the system, and complete the installation steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install all the pre-requisite libraries on jetson nano\n",
    "\n",
    "Run these commands in the terminal\n",
    "\n",
    "\n",
    "### Update apt\n",
    " - sudo apt-get update\n",
    " - sudo apt-get upgrade\n",
    "\n",
    "\n",
    "### Install nano writer\n",
    " - sudo apt-get install nano\n",
    "\n",
    " \n",
    "### Install libraries required by OpenCV and Tensorflow\n",
    " - sudo apt-get install git cmake libatlas-base-dev gfortran libhdf5-serial-dev hdf5-tools nano locate libfreetype6-dev python3-setuptools protobuf-compiler libprotobuf-dev openssl libssl-dev libcurl4-openssl-dev cython3 libxml2-dev libxslt1-dev python3-pip\n",
    " - sudo apt-get install libopenblas-dev libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler libgflags-dev libgoogle-glog-dev liblmdb-dev \n",
    "\n",
    "\n",
    "### Install python 3 pip \n",
    " - sudo pip3 install virtualenv virtualenvwrapper\n",
    "\n",
    "\n",
    "### Add the following file to .bashrc\n",
    " - export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3\n",
    " - export WORKON_HOME=$HOME/.virtualenvs\n",
    " - export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv\n",
    " - source /usr/local/bin/virtualenvwrapper.sh\n",
    "\n",
    "\n",
    "### After adding run\n",
    " - source ~/.bashrc\n",
    "\n",
    "\n",
    "### Install Cmake \n",
    " - wget https://github.com/Kitware/CMake/releases/download/v3.18.1/cmake-3.18.1.tar.gz\n",
    " - tar -xvzf cmake-3.18.1.tar.gz\n",
    " - cd cmake-3.18.1 && cmake . && make -j3\n",
    " - sudo make install\n",
    "\n",
    "\n",
    "### Install further libraries \n",
    " - sudo apt install -y libjpeg-dev libpng-dev libtiff-dev libavcodec-dev libavformat-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libv4l-dev v4l-utils qv4l2 v4l2ucp libdc1394-22-dev libffi-dev libgtk-3-dev libatlas-base-dev gfortran\n",
    "\n",
    "\n",
    "### Add to ~/.bashrc\n",
    " - export PATH=/usr/local/cuda-10.0/bin${PATH:+:${PATH}}\n",
    " \n",
    " - export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64\\\n",
    "                         ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}\n",
    " \n",
    "\n",
    "### After adding run\n",
    " - source ~/.bashrc\n",
    "\n",
    "\n",
    "### Create a Virtual-Environment\n",
    " - mkvirtualenv -p /usr/bin/python3.6 tf_test1\n",
    "\n",
    "\n",
    "### Install all necessary python packages\n",
    " - pip install numpy==1.19.1\n",
    " - pip install scipy==1.5.1\n",
    " - pip install scikit-build protobuf cython -vvvv\n",
    " - pip install grpcio absl-py py-cpuinfo psutil portpicker six mock requests gast h5py astor termcolor protobuf keras-applications keras-preprocessing wrapt google-pasta -vvvv\n",
    " - pip install https://developer.download.nvidia.com/compute/redist/jp/v43/tensorflow-gpu/tensorflow_gpu-1.15.0+nv19.12-cp36-cp36m-linux_aarch64.whl -vvvv\n",
    " - pip install pandas pillow -vvvv\n",
    "\n",
    "\n",
    "### Install OpenCV\n",
    " - mkdir opencv && cd opencv\n",
    " - wget -O opencv.zip https://github.com/opencv/opencv/archive/4.1.2.zip\n",
    " - unzip opencv.zip\n",
    " - mv opencv-4.1.2 opencv\n",
    " - cd opencv && mkdir build && cd build\n",
    " - cmake -D CMAKE_BUILD_TYPE=RELEASE -D WITH_CUDA=OFF -D WITH_CUBLAS=OFF -D WITH_LIBV4L=ON -D BUILD_opencv_python3=ON -D BUILD_opencv_python2=OFF -D BUILD_opencv_java=OFF -D WITH_GSTREAMER=ON -D WITH_GTK=ON -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_EXAMPLES=OFF -D OPENCV_ENABLE_NONFREE=OFF ..\n",
    " - make -j3\n",
    " - sudo make install\n",
    " - cd ~/.virtualenvs/tf_test1/lib/python3.6/site-packages\n",
    " - ln -s /usr/local/lib/python3.6/site-packages/cv2/python-3.6/cv2.cpython-36m-aarch64-linux-gnu.so cv2.so\n",
    "\n",
    "\n",
    "### Clone and install Monk_Object Detection library\n",
    " - git clone https://github.com/Tessellate-Imaging/Monk_Object_Detection.git\n",
    " - cd Monk_Object_Detection/12_tf_obj_1/installation/\n",
    " - cd Monk_Object_Detection/12_tf_obj_1/installation\n",
    " - chmod +x install_nano.sh && ./install_nano.sh\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy Sample dataset to jetson nano"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download pretrained models to jetson nano"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1jsq7IZbVeK1uFaFZrNW8-m1MZ-QVZ5Cp' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1jsq7IZbVeK1uFaFZrNW8-m1MZ-QVZ5Cp\" -O obj_monkey_trained.zip && rm -rf /tmp/cookies.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! unzip -qq obj_monkey_trained.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! cp obj_monkey_trained/labelmap.txt ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference on Nano board (To Run only on Jetson Nano Board)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "sys.path.append(\"Monk_Object_Detection/12_tf_obj_1/lib/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from infer_detector import Infer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf = Infer();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model loading takes time on nano boards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.set_model_params('obj_monkey_trained/trt_fp16_dir/trt_graph.pb', \"obj_monkey_trained/classes.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls obj_monkey_trained/test/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Running for the first time builds the tensorRT engine for the model based on the plan saved in trt_fp16_dir folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Oputput will be saved as output.png\n",
    "scores, bboxes, labels = gtf.infer_on_image('obj_monkey_trained/test/img1.jpg', thresh=0.5, img_size=300);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run speed benchmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.benchmark_for_speed('obj_monkey_trained/test/1.jpg', img_size=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installation on cloud/local system\n",
    "\n",
    " - Run these commands\n",
    "     \n",
    "     - git clone https://github.com/Tessellate-Imaging/Monk_Object_Detection.git\n",
    "     \n",
    "     - cd Monk_Object_Detection/12_tf_obj_1/installation\n",
    "     \n",
    " - Select the right file and run\n",
    " \n",
    "     - chmod +x install_cuda10.sh && ./install_cuda10.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! git clone https://github.com/Tessellate-Imaging/Monk_Object_Detection.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only for COLAB\n",
    "# Switch to TF 1.0 version (Uncomment the following line)\n",
    "\n",
    "#%tensorflow_version 1.x\n",
    "\n",
    "# Now reset the runetime if prompted by colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check TF version\n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For colab use the command below\n",
    "! cd Monk_Object_Detection/12_tf_obj_1/installation && chmod +x install_colab.sh && ./install_colab.sh\n",
    "# Restart colab runtime now\n",
    "\n",
    "# For Local systems and cloud select the right CUDA version\n",
    "# ! cd Monk_Object_Detection/12_tf_obj_1/installation && chmod +x install_cuda10.sh && ./install_cuda10.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use already trained model for demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "sys.path.append(\"Monk_Object_Detection/12_tf_obj_1/lib/\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from infer_detector import Infer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf = Infer();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1jsq7IZbVeK1uFaFZrNW8-m1MZ-QVZ5Cp' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1jsq7IZbVeK1uFaFZrNW8-m1MZ-QVZ5Cp\" -O obj_monkey_trained.zip && rm -rf /tmp/cookies.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! unzip -qq obj_monkey_trained.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! cp obj_monkey_trained/labelmap.txt ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.set_model_params('obj_monkey_trained/export_dir/frozen_inference_graph.pb', \"obj_monkey_trained/classes.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores, bboxes, labels = gtf.infer_on_image(\"obj_monkey_trained/test/1.jpg\", \n",
    "                                            thresh=0.1, img_size=300, \n",
    "                                            bbox_thickness=3, text_size=2, text_thickness=4);\n",
    "from IPython.display import Image\n",
    "Image(filename='output.png', width=400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores, bboxes, labels = gtf.infer_on_image(\"obj_monkey_trained/test/2.jpg\", \n",
    "                                            thresh=0.3, img_size=300, \n",
    "                                            bbox_thickness=3, text_size=2, text_thickness=4);\n",
    "from IPython.display import Image\n",
    "Image(filename='output.png', width=400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores, bboxes, labels = gtf.infer_on_image(\"obj_monkey_trained/test/3.jpg\", \n",
    "                                            thresh=0.1, img_size=300, \n",
    "                                            bbox_thickness=3, text_size=2, text_thickness=4);\n",
    "from IPython.display import Image\n",
    "Image(filename='output.png', width=400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores, bboxes, labels = gtf.infer_on_image(\"obj_monkey_trained/test/4.jpg\", \n",
    "                                            thresh=0.1, img_size=300, \n",
    "                                            bbox_thickness=3, text_size=2, text_thickness=4);\n",
    "from IPython.display import Image\n",
    "Image(filename='output.png', width=400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores, bboxes, labels = gtf.infer_on_image(\"obj_monkey_trained/test/5.jpg\", \n",
    "                                            thresh=0.1, img_size=300, \n",
    "                                            bbox_thickness=3, text_size=2, text_thickness=4);\n",
    "from IPython.display import Image\n",
    "Image(filename='output.png', width=400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training on the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    " - Credits: https://storage.googleapis.com/openimages/web/index.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! git clone https://github.com/EscVM/OIDv4_ToolKit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cd to OIDv4_ToolKit and install requirements\n",
    "# pip install -r requirements.txtx\n",
    "\n",
    "# Then run the following command to download the monkey subset\n",
    "# python main.py downloader --classes Monkey --type_csv train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install pascal_voc_writer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_dir = \"OIDv4_ToolKit/OID/Dataset/train/Monkey\";\n",
    "label_dir = \"OIDv4_ToolKit/OID/Dataset/train/Monkey/Label\";"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir dataset\n",
    "! mkdir dataset/train\n",
    "! mkdir dataset/train/images\n",
    "! mkdir dataset/train/labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir dataset/val\n",
    "! mkdir dataset/val/images\n",
    "! mkdir dataset/val/labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import cv2\n",
    "from pascal_voc_writer import Writer\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_list = os.listdir(\"OIDv4_ToolKit/OID/Dataset/train/Monkey/Label\");\n",
    "\n",
    "for i in tqdm(range(len(label_list))):\n",
    "    img_name = label_list[i].split(\".\")[0] + \".jpg\";\n",
    "    cmd = \"cp OIDv4_ToolKit/OID/Dataset/train/Monkey/\" + img_name + \" dataset/train/images/\";\n",
    "    os.system(cmd)\n",
    "    f = open(\"OIDv4_ToolKit/OID/Dataset/train/Monkey/Label/\" + label_list[i]);\n",
    "    lines = f.readlines();\n",
    "    f.close();\n",
    "    \n",
    "    img = cv2.imread(\"dataset/train/images/\" + img_name)\n",
    "    h, w, c = img.shape;\n",
    "    writer = Writer(img_name, w, h)\n",
    "    for j in range(len(lines)):\n",
    "        tmp = lines[j].split(\" \");\n",
    "        label = tmp[0];\n",
    "        x1 = int(float(tmp[1]));\n",
    "        y1 = int(float(tmp[2]));\n",
    "        x2 = int(float(tmp[3]));\n",
    "        y2 = int(float(tmp[4]));\n",
    "        \n",
    "        writer.addObject(\"monkey\", x1, y1, x2, y2);\n",
    "    \n",
    "    writer.save(\"dataset/train/labels/\" + label_list[i].split(\".\")[0] + \".xml\")\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_list = os.listdir(\"OIDv4_ToolKit/OID/Dataset/validation/Monkey/Label\");\n",
    "\n",
    "for i in tqdm(range(len(label_list))):\n",
    "    img_name = label_list[i].split(\".\")[0] + \".jpg\";\n",
    "    cmd = \"cp OIDv4_ToolKit/OID/Dataset/validation/Monkey/\" + img_name + \" dataset/val/images/\";\n",
    "    os.system(cmd)\n",
    "    f = open(\"OIDv4_ToolKit/OID/Dataset/validation/Monkey/Label/\" + label_list[i]);\n",
    "    lines = f.readlines();\n",
    "    f.close();\n",
    "    \n",
    "    img = cv2.imread(\"dataset/val/images/\" + img_name)\n",
    "    h, w, c = img.shape;\n",
    "    writer = Writer(img_name, w, h)\n",
    "    for j in range(len(lines)):\n",
    "        tmp = lines[j].split(\" \");\n",
    "        label = tmp[0];\n",
    "        x1 = int(float(tmp[1]));\n",
    "        y1 = int(float(tmp[2]));\n",
    "        x2 = int(float(tmp[3]));\n",
    "        y2 = int(float(tmp[4]));\n",
    "        \n",
    "        writer.addObject(\"monkey\", x1, y1, x2, y2);\n",
    "    \n",
    "    writer.save(\"dataset/val/labels/\" + label_list[i].split(\".\")[0] + \".xml\")\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open(\"classes.txt\", 'w');\n",
    "f.write(\"monkey\\n\");\n",
    "f.close();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "sys.path.append(\"Monk_Object_Detection/12_tf_obj_1/lib/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from train_detector import Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf = Detector();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.list_models();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_img_dir = \"dataset/train/images/\";\n",
    "train_anno_dir = \"dataset/train/labels/\";\n",
    "class_list_file = \"classes.txt\";\n",
    "\n",
    "gtf.set_train_dataset(train_img_dir, train_anno_dir, class_list_file, batch_size=24)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_img_dir = \"dataset/val/images/\";\n",
    "val_anno_dir = \"dataset/val/labels/\";\n",
    "class_list_file = \"classes.txt\";\n",
    "\n",
    "gtf.set_val_dataset(val_img_dir, val_anno_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create tf record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.create_tfrecord(data_output_dir=\"data_tfrecord\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model and hyper params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.set_model_params(model_name=\"ssd_mobilenet_v1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.set_hyper_params(num_train_steps=100000, lr=0.03)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.export_params(output_directory=\"export_dir\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# (Optional) Optimize using TensorRT - Feature Not tested on colab\n",
    "# Available conversion types\n",
    "# - FP32\n",
    "# - FP16\n",
    "# - INT8 (Use int8 type only when your deployment and development platforms are similar. Else rebuild on deployment platform)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.TensorRT_Optimization_Params(conversion_type=\"FP16\", trt_dir=\"trt_fp16_dir\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training\n",
    "# tf.app.run() executes sys.exit() function hence cannot run in a jupyter notebook directory\n",
    "# Run in a terminal - python Monk_Object_Detection/12_tf_obj_1/lib/train.py\n",
    "# or\n",
    "# Run the following command from notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run Monk_Object_Detection/12_tf_obj_1/lib/train.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# exportaing trained model\n",
    "# tf.app.run() executes sys.exit() function hence cannot run in a jupyter notebook directory\n",
    "# Run in a terminal - python Monk_Object_Detection/12_tf_obj_1/lib/export.py\n",
    "# or\n",
    "# Run the following command from notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run Monk_Object_Detection/12_tf_obj_1/lib/export.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimizing For TensorRT - Feature Not tested on colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run Monk_Object_Detection/12_tf_obj_1/lib/optimize.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference on exported model (Unoptimized) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "sys.path.append(\"Monk_Object_Detection/12_tf_obj_1/lib/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from infer_detector import Infer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf = Infer();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.set_model_params('export_dir/frozen_inference_graph.pb', \"classes.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "img_list = os.listdir(\"dataset/val/images/\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores, bboxes, labels = gtf.infer_on_image(\"dataset/val/images/\"+img_list[0], \n",
    "                                            thresh=0.1, img_size=300, \n",
    "                                            bbox_thickness=3, text_size=2, text_thickness=4);\n",
    "from IPython.display import Image\n",
    "Image(filename='output.png', width=400) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores, bboxes, labels = gtf.infer_on_image(\"dataset/val/images/\"+img_list[10], \n",
    "                                            thresh=0.4, img_size=300, \n",
    "                                            bbox_thickness=3, text_size=2, text_thickness=4);\n",
    "from IPython.display import Image\n",
    "Image(filename='output.png', width=400) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.benchmark_for_speed(\"dataset/val/images/\"+img_list[10], img_size=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference on TensorRT optimized model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "sys.path.append(\"Monk_Object_Detection/12_tf_obj_1/lib/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from infer_detector import Infer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf = Infer();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.set_model_params('trt_fp16_dir/trt_graph.pb', \"classes.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "img_list = os.listdir(\"dataset/val/images/\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores, bboxes, labels = gtf.infer_on_image(\"dataset/val/images/\"+img_list[0], \n",
    "                                            thresh=0.1, img_size=300, \n",
    "                                            bbox_thickness=3, text_size=2, text_thickness=4);\n",
    "from IPython.display import Image\n",
    "Image(filename='output.png', width=400) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores, bboxes, labels = gtf.infer_on_image(\"dataset/val/images/\"+img_list[10], \n",
    "                                            thresh=0.4, img_size=300, \n",
    "                                            bbox_thickness=3, text_size=2, text_thickness=4);\n",
    "from IPython.display import Image\n",
    "Image(filename='output.png', width=400) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtf.benchmark_for_speed(\"dataset/val/images/\"+img_list[10], img_size=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}