Ex.02_SolarPanels/FasterRCNN_Tutorial_MeyerA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "FasterRCNN_Tutorial_MeyerA.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eBG7ZQ_pcEah",
        "colab_type": "text"
      },
      "source": [
        "# Object Detection Tutorial with Faster R-CNN Transfer Learning\n",
        "## Geopython 2019, Adrian F. Meyer\n",
        "Parts of this tutorial are based on the [Medium article](https://hackernoon.com/object-detection-in-google-colab-with-custom-dataset-5a7bb2b0e97e) by RomRoc, 2018.\n",
        "\n",
        "**In this Tutorial we will learn, how to use the Tensorflow Object Detection library, to detect solar panels on tiles of an aerial orthomosaic.**\n",
        "\n",
        "![alt text](https://wilddrone.ch/wp-content/uploads/2019/06/solardetector.jpg)\n",
        "\n",
        "The code, libraries and cloud environments used in this tutorial are currently available for free and are generally released open source.\n",
        "You will need a Google account to execute the Notebook in its entirety, because it is meant to be executed on the Google Colab platform.\n",
        "\n",
        "*Go to [Google Colab](https://colab.research.google.com/) and upload the notebook there. Make sure that you use Python 3 and GPU hardware acceleration as Runtime Environment.*\n",
        "\n",
        "The dataset provided is based on the publically available SwissImage orthomosaic by [SwissTopo](https://map.geo.admin.ch/?topic=swisstopo&lang=de&bgLayer=ch.swisstopo.swissimage). \n",
        "The images and annotations can be downloaded as Zip File (31 Mbyte) here:\n",
        "[https://drive.google.com/file/d/1i9RlEJTeB-KRauwhuMp-Vl13-Z3Yrj_y/view?usp=sharing](https://drive.google.com/file/d/1i9RlEJTeB-KRauwhuMp-Vl13-Z3Yrj_y/view?usp=sharing)\n",
        "\n",
        "\n",
        "\n",
        "![alt text](https://wilddrone.ch/wp-content/uploads/2019/06/Tiles.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KWJHCmFs_Vg6",
        "colab_type": "text"
      },
      "source": [
        "# Download Tensorflow Repo and Python Modules\n",
        "By executing the first code snippet you initialize your virtual linux-style machine. Use The little arrow \">\" in the top left corner to view the file system of your hosted system.\n",
        "You can use UNIX-style terminal commands by using the prefix % and elevated priviledge commands for installations with the prefix !."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rsxgHEoV8J56",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#make sure numpy is downgraded for compatibility reasons. \n",
        "#It will throw an error, which is not a problem, just click on the newly\n",
        "#appeared Button \"Restart Runtime\"\n",
        "!pip install numpy==1.17.4"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H9NdpuUj8MB9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd\n",
        "%tensorflow_version 1.x\n",
        "\n",
        "#make sure to be in /root and that tensorflow is running in version 1.15.2\n",
        "import tensorflow as tf\n",
        "print(tf.__version__)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hJ7NlRzJ8OKX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\"\"\"\n",
        "This allows you to check which GPU you have been allocated. Google offers free\n",
        "Tesla T4, Tesla K80, Tesla P100 (the P100 hax 1.6x more GFLOPs and 3x the memory bandwith than K80, the T4 is fairly slow).\n",
        "In theory you can restart the environment until you have the fast one. \n",
        "For testing and learning it doesn't really matter.\n",
        "\"\"\"\n",
        "  \n",
        "#We have to work with Tensorflow 1.15.2 for code compatibility reasons; by now TF v2 is available.\n",
        "from tensorflow.python.client import device_lib\n",
        "device_lib.list_local_devices()\n",
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2dyvc3xTxsy0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd\n",
        "  \n",
        "!git clone https://github.com/tensorflow/models.git\n",
        "  \n",
        "\"\"\"\n",
        "This repository contains a number of different models implemented in TensorFlow:\n",
        "The official models are a collection of example models that use TensorFlow's high-level APIs. They are intended to be well-maintained, tested, and kept up to date with the latest stable TensorFlow API. They should also be reasonably optimized for fast performance while still being easy to read. We especially recommend newer TensorFlow users to start here.\n",
        "The research models are a large collection of models implemented in TensorFlow by researchers. They are not officially supported or available in release branches; it is up to the individual researchers to maintain the models and/or provide support on issues and pull requests.\n",
        "The samples folder contains code snippets and smaller models that demonstrate features of TensorFlow, including code presented in various blog posts.\n",
        "\"\"\"  \n",
        "\n",
        "!apt-get install protobuf-compiler python-tk\n",
        "\n",
        "\"\"\"\n",
        "Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data; similar to JSON or XML.\n",
        "\"\"\"\n",
        "\n",
        "!pip install Cython contextlib2 pillow lxml matplotlib PyDrive\n",
        "\"\"\"\n",
        "These context modules are necessary python pachages. Especially Cython is important: It allows to call native C or C++ bindings from within python.\n",
        "\"\"\"\n",
        "\n",
        "!pip install pycocotools\n",
        "\"\"\"\n",
        "COCO is a large image dataset designed for object detection, segmentation, person keypoints detection, stuff segmentation, and caption generation. \n",
        "\"\"\"\n",
        "\n",
        "%cd ~/models/research\n",
        "!protoc object_detection/protos/*.proto --python_out=. #This initializes/compiles the Tensorflow Protobuf evnironment.\n",
        "\n",
        "import os\n",
        "os.environ['PYTHONPATH'] += ':/models/research/:/models/research/slim/'\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RVXThN8P_itE",
        "colab_type": "text"
      },
      "source": [
        "# Install Tensorflow on Virtual Machine"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Alxt_PF4yBae",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python setup.py build\n",
        "!python setup.py install > /dev/null\n",
        "\"\"\"\n",
        "This snippet builds and installs the Tensorflow API from the cloned git source.\n",
        "\"\"\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jr4Qf_v-SDeQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "THX1UucO0L5X",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd slim\n",
        "!pip install -e .\n",
        "\n",
        "%cd ..\n",
        "!python object_detection/builders/model_builder_test.py\n",
        "\"\"\"\n",
        "This tests if the installation was successful. The Tests should yield the output [ RUN  OK ]\n",
        "\"\"\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ceHKJMfzHdg9",
        "colab_type": "text"
      },
      "source": [
        "#Upload and Import Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CI9EJ13NBLVd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /datalab\n",
        "!wget http://2019.geopython.net/data/solar.zip\n",
        "\"\"\"\n",
        "We download the images, annotation files and independent test samples into the datalab folder.\n",
        "\"\"\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aaaI150MDVLq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /datalab\n",
        "!unzip solar.zip #Scroll through the unzip output to get an idea of the datalab folder content."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7LMyNhmeS-Yc",
        "colab_type": "text"
      },
      "source": [
        "You can download one of the XML files to check the structure of the PASCAL VOC Annotation format.\n",
        "Here is an example:\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "<annotation>\n",
        "\t<folder>sol2</folder>\n",
        "\t<filename>solar_10.JPG</filename>\n",
        "\t<path>C:\\temp\\sol2\\solar_10.JPG</path>\n",
        "\t<source>\n",
        "\t\t<database>Unknown</database>\n",
        "\t</source>\n",
        "\t<size>\n",
        "\t\t<width>901</width>\n",
        "\t\t<height>791</height>\n",
        "\t\t<depth>3</depth>\n",
        "\t</size>\n",
        "\t<segmented>0</segmented>\n",
        "\t<object>\n",
        "\t\t<name>solar</name>\n",
        "\t\t<pose>Unspecified</pose>\n",
        "\t\t<truncated>1</truncated>\n",
        "\t\t<difficult>0</difficult>\n",
        "\t\t<bndbox>\n",
        "\t\t\t<xmin>637</xmin>\n",
        "\t\t\t<ymin>152</ymin>\n",
        "\t\t\t<xmax>901</xmax>\n",
        "\t\t\t<ymax>591</ymax>\n",
        "\t\t</bndbox>\n",
        "\t</object>\n",
        "</annotation>\n",
        "\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "osh2Ty-vTejr",
        "colab_type": "text"
      },
      "source": [
        "We need to write our label name (in this case \"solar\") into a config file defining all detectable classes.\n",
        "In our case it is just on class.\n",
        "\n",
        "Then we iterate through all image files to extract the file names (paths are not relevant) we want to use for training and validation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6f41p5Nb1Uhl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd ..\n",
        "%cd /datalab\n",
        "\n",
        "!echo \"item { id: 1 name: 'solar'}\" > label_map.pbtxt\n",
        "\n",
        "image_files=os.listdir('images')\n",
        "im_files=[x.split('.')[0] for x in image_files]\n",
        "with open('annotations/trainval.txt', 'w') as text_file:\n",
        "  for row in im_files:\n",
        "    text_file.write(row + '\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jiE8dMrBMGTG",
        "colab_type": "text"
      },
      "source": [
        "#Data and Model Preparation\n",
        "## Generate Bounding Boxes on Images for RPN Network Training\n",
        "The same process need to be performed with the XML Annotation files.\n",
        "Additionally, we convert the images to PNG format for easier access."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r38LiZJ11nJ2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /datalab/annotations\n",
        "\n",
        "!mkdir trimaps\n",
        "\n",
        "from PIL import Image\n",
        "image = Image.new('RGB', (901, 791))\n",
        "\n",
        "for filename in os.listdir('xmls'):\n",
        "  filename = os.path.splitext(filename)[0]\n",
        "  image.save('trimaps/' + filename + '.png')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SlG7LCylMYB_",
        "colab_type": "text"
      },
      "source": [
        "##Generate Labelled Tensor Matrices (tf_records)\n",
        "The Tensorflow Record files contain the actual input data for the Machine Learning process in binary format.\n",
        "An API specific script can do the job for us.\n",
        "We use the famous \"pet faces model\" in our transfer learning process.\n",
        "We need to split the dataset at this point into training and validation data.\n",
        "70% of our data (148 of 212 images) will be used for training, the remaining 30% for validation (64 images)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XhwFAOny2OLc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /datalab\n",
        "\n",
        "!python ~/models/research/object_detection/dataset_tools/create_pet_tf_record.py --label_map_path=label_map.pbtxt --data_dir=. --output_dir=. --num_shards=1\n",
        "\n",
        "!mv pet_faces_train.record-00000-of-00001 tf_train.record\n",
        "\n",
        "!mv pet_faces_val.record-00000-of-00001 tf_val.record"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k0CoGNKoMiCr",
        "colab_type": "text"
      },
      "source": [
        "##Download the Model Checkpoint you want to use for Transfer Learning\n",
        "Many different COCO pretrained neural models can be used for bounding box related object detection with Tensorflow.\n",
        "They all have different advantages or disadvantages (e.g. inferencing speed, accuracy, easy to train, etc.).\n",
        "\n",
        "An overview can be found with the [TF Model Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Sj2b08bHfmZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /datalab\n",
        "!wget http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_v2_coco_2018_01_28.tar.gz\n",
        "  \n",
        "%cd /datalab\n",
        "!tar -xvzf faster_rcnn_inception_v2_coco_2018_01_28.tar.gz\n",
        "\n",
        "%cd /datalab\n",
        "!mv faster_rcnn_inception_v2_coco_2018_01_28 pretrained_model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qyk6YNdBMq8A",
        "colab_type": "text"
      },
      "source": [
        "##Configure the Paths and Training Parameters\n",
        "This specifies which files and model checkpoints should be used for the trainings process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GkNT0AWT6Q9K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /datalab\n",
        "\n",
        "import re\n",
        "\n",
        "#filename = '/datalab/pretrained_model/pipeline.config'\n",
        "filename = '/root/models/research/object_detection/samples/configs/faster_rcnn_inception_v2_pets.config'\n",
        "with open(filename) as f:\n",
        "  s = f.read()\n",
        "with open(filename, 'w') as f:\n",
        "  s = re.sub('PATH_TO_BE_CONFIGURED/model.ckpt', '/datalab/pretrained_model/model.ckpt', s)\n",
        "  s = re.sub('PATH_TO_BE_CONFIGURED/pet_faces_train.record-\\?\\?\\?\\?\\?-of-00010', '/datalab/tf_train.record', s)\n",
        "  s = re.sub('PATH_TO_BE_CONFIGURED/pet_faces_val.record-\\?\\?\\?\\?\\?-of-00010', '/datalab/tf_val.record', s)\n",
        "  s = re.sub('PATH_TO_BE_CONFIGURED/pet_label_map.pbtxt', '/datalab/label_map.pbtxt', s)\n",
        "  f.write(s)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aiJ6kHszMyeJ",
        "colab_type": "text"
      },
      "source": [
        "# Training on GPU\n",
        "The execution of this snippet might take a while. Normally the training of 3000 steps should take about 10 minutes (approx. 12 seconds for 100 steps). \n",
        "\n",
        "As a rough estimate, the loss value of Faster RCNN models should fall below 0.05 over a few thousand steps and then the training can be aborted. \n",
        "\n",
        "We configure automatic termination after 3'000 Steps, in productive trainings as much as 100'000-200'000 Steps can be neccesary.\n",
        "\n",
        "![alt text](https://cdn-images-1.medium.com/max/800/1*qGb5XNny5G8PrGQ5sejFvw.png)\n",
        "\n",
        "This graph shows the expected training progess of the model with the supervision tool \"Tensorboard\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jvGQxjNI6QNr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /datalab\n",
        "\n",
        "%cp -R /datalab /content\n",
        "\n",
        "!python ~/models/research/object_detection/model_main.py \\\n",
        "    --pipeline_config_path=/root/models/research/object_detection/samples/configs/faster_rcnn_inception_v2_pets.config \\\n",
        "    --model_dir=/datalab/trained \\\n",
        "    --train_dir=/datalab/trained \\\n",
        "    --logtostderr \\\n",
        "    --logdir=/datalab/trained \\\n",
        "    --num_train_steps=3000 \\\n",
        "    --num_eval_steps=500 \\\n",
        "    --max_evals=0 "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j6aIMnXJOm49",
        "colab_type": "text"
      },
      "source": [
        "# Export Inference Graph\n",
        "Inferencing means to apply the model to images which haven't been used for training.\n",
        "\n",
        "We reserved a few images to check if our model performs correctly.\n",
        "\n",
        "The frozen Inference Graph gets generated from the last model checkpoint and contains all elements of the model neccesary to perform inference (also on weaker hardware), but it cannot be used to continue training the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dsjkGdgTDmbh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /datalab\n",
        "\n",
        "lst = os.listdir('trained')\n",
        "lf = filter(lambda k: 'model.ckpt-' in k, lst)\n",
        "last_model = sorted(lf)[-1].replace('.meta', '')\n",
        "\n",
        "!python ~/models/research/object_detection/export_inference_graph.py \\\n",
        "    --input_type=image_tensor \\\n",
        "    --pipeline_config_path=/root/models/research/object_detection/samples/configs/faster_rcnn_inception_v2_pets.config \\\n",
        "    --output_directory=fine_tuned_model \\\n",
        "    --trained_checkpoint_prefix=trained/$last_model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gQub8i2wareJ",
        "colab_type": "text"
      },
      "source": [
        "##Alternative: Import a Finished Inference Graph\n",
        "Uncomment this section, if you want to download a finished inference graph if you could not finish the training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O9Emd73ISTk9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Alternative to training, you can download a fully trained model inference graph by uncommenting (#) the following lines\n",
        "\n",
        "#%cd /datalab\n",
        "#!wget http://2019.geopython.net/data/trained.tar.gz\n",
        "#%cd /datalab\n",
        "#!tar -xzf trained.tar.gz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nqAjy3fpOzlZ",
        "colab_type": "text"
      },
      "source": [
        "# Run Inference"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4tO_OCUXEvjr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /root/models/research/object_detection\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "import numpy as np\n",
        "import os\n",
        "import six.moves.urllib as urllib\n",
        "import sys\n",
        "import tarfile\n",
        "import tensorflow as tf\n",
        "import zipfile\n",
        "\n",
        "from collections import defaultdict\n",
        "from io import StringIO\n",
        "from matplotlib import pyplot as plt\n",
        "from PIL import Image\n",
        "\n",
        "# This is needed since the notebook is stored in the object_detection folder.\n",
        "sys.path.append(\"..\")\n",
        "from object_detection.utils import ops as utils_ops\n",
        "\n",
        "#if tf.__version__ < '1.4.0':\n",
        "#  raise ImportError('Please upgrade your tensorflow installation to v1.4.* or later!')\n",
        "  \n",
        "\n",
        "  \n",
        "  \n",
        "# This is needed to display the images.\n",
        "%matplotlib inline\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "from utils import label_map_util\n",
        "\n",
        "from utils import visualization_utils as vis_util\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# What model to download.\n",
        "# Path to frozen detection graph. This is the actual model that is used for the object detection.\n",
        "PATH_TO_CKPT = '/datalab/fine_tuned_model' + '/frozen_inference_graph.pb'\n",
        "\n",
        "# List of the strings that is used to add correct label for each box.\n",
        "PATH_TO_LABELS = os.path.join('/content/datalab', 'label_map.pbtxt')\n",
        "\n",
        "NUM_CLASSES = 37\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "detection_graph = tf.Graph()\n",
        "with detection_graph.as_default():\n",
        "  od_graph_def = tf.GraphDef()\n",
        "  with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:\n",
        "    serialized_graph = fid.read()\n",
        "    od_graph_def.ParseFromString(serialized_graph)\n",
        "    tf.import_graph_def(od_graph_def, name='')\n",
        "    \n",
        "    \n",
        "    \n",
        "    \n",
        "label_map = label_map_util.load_labelmap(PATH_TO_LABELS)\n",
        "categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)\n",
        "category_index = label_map_util.create_category_index(categories)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "def load_image_into_numpy_array(image):\n",
        "  (im_width, im_height) = image.size\n",
        "  return np.array(image.getdata()).reshape(\n",
        "      (im_height, im_width, 3)).astype(np.uint8)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# If you want to test the code with your images, just add path to the images to the TEST_IMAGE_PATHS.\n",
        "PATH_TO_TEST_IMAGES_DIR = '/datalab/testsamples/'\n",
        "TEST_IMAGE_PATHS = [ os.path.join(PATH_TO_TEST_IMAGES_DIR, 'image{}.JPG'.format(i)) for i in range(1, 4) ]\n",
        "\n",
        "# Size, in inches, of the output images.\n",
        "IMAGE_SIZE = (12, 8)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "def run_inference_for_single_image(image, graph):\n",
        "  with graph.as_default():\n",
        "    with tf.Session() as sess:\n",
        "      # Get handles to input and output tensors\n",
        "      ops = tf.get_default_graph().get_operations()\n",
        "      all_tensor_names = {output.name for op in ops for output in op.outputs}\n",
        "      tensor_dict = {}\n",
        "      for key in [\n",
        "          'num_detections', 'detection_boxes', 'detection_scores',\n",
        "          'detection_classes', 'detection_masks'\n",
        "      ]:\n",
        "        tensor_name = key + ':0'\n",
        "        if tensor_name in all_tensor_names:\n",
        "          tensor_dict[key] = tf.get_default_graph().get_tensor_by_name(\n",
        "              tensor_name)\n",
        "      if 'detection_masks' in tensor_dict:\n",
        "        # The following processing is only for single image\n",
        "        detection_boxes = tf.squeeze(tensor_dict['detection_boxes'], [0])\n",
        "        detection_masks = tf.squeeze(tensor_dict['detection_masks'], [0])\n",
        "        # Reframe is required to translate mask from box coordinates to image coordinates and fit the image size.\n",
        "        real_num_detection = tf.cast(tensor_dict['num_detections'][0], tf.int32)\n",
        "        detection_boxes = tf.slice(detection_boxes, [0, 0], [real_num_detection, -1])\n",
        "        detection_masks = tf.slice(detection_masks, [0, 0, 0], [real_num_detection, -1, -1])\n",
        "        detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(\n",
        "            detection_masks, detection_boxes, image.shape[0], image.shape[1])\n",
        "        detection_masks_reframed = tf.cast(\n",
        "            tf.greater(detection_masks_reframed, 0.5), tf.uint8)\n",
        "        # Follow the convention by adding back the batch dimension\n",
        "        tensor_dict['detection_masks'] = tf.expand_dims(\n",
        "            detection_masks_reframed, 0)\n",
        "      image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')\n",
        "\n",
        "      # Run inference\n",
        "      output_dict = sess.run(tensor_dict,\n",
        "                             feed_dict={image_tensor: np.expand_dims(image, 0)})\n",
        "\n",
        "      # all outputs are float32 numpy arrays, so convert types as appropriate\n",
        "      output_dict['num_detections'] = int(output_dict['num_detections'][0])\n",
        "      output_dict['detection_classes'] = output_dict[\n",
        "          'detection_classes'][0].astype(np.uint8)\n",
        "      output_dict['detection_boxes'] = output_dict['detection_boxes'][0]\n",
        "      output_dict['detection_scores'] = output_dict['detection_scores'][0]\n",
        "      if 'detection_masks' in output_dict:\n",
        "        output_dict['detection_masks'] = output_dict['detection_masks'][0]\n",
        "  return output_dict\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "for image_path in TEST_IMAGE_PATHS:\n",
        "  image = Image.open(image_path)\n",
        "  # the array based representation of the image will be used later in order to prepare the\n",
        "  # result image with boxes and labels on it.\n",
        "  image_np = load_image_into_numpy_array(image)\n",
        "  # Expand dimensions since the model expects images to have shape: [1, None, None, 3]\n",
        "  image_np_expanded = np.expand_dims(image_np, axis=0)\n",
        "  # Actual detection.\n",
        "  output_dict = run_inference_for_single_image(image_np, detection_graph)\n",
        "  # Visualization of the results of a detection.\n",
        "  vis_util.visualize_boxes_and_labels_on_image_array(\n",
        "      image_np,\n",
        "      output_dict['detection_boxes'],\n",
        "      output_dict['detection_classes'],\n",
        "      output_dict['detection_scores'],\n",
        "      category_index,\n",
        "      instance_masks=output_dict.get('detection_masks'),\n",
        "      use_normalized_coordinates=True,\n",
        "      line_thickness=8)\n",
        "  plt.figure(figsize=IMAGE_SIZE)\n",
        "  plt.imshow(image_np)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9cJTzq7If9lV",
        "colab_type": "text"
      },
      "source": [
        "## Run Inference on Additional Samples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5kOKL6IwbJwD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /root/models/research/object_detection\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "import numpy as np\n",
        "import os\n",
        "import six.moves.urllib as urllib\n",
        "import sys\n",
        "import tarfile\n",
        "import tensorflow as tf\n",
        "import zipfile\n",
        "\n",
        "from collections import defaultdict\n",
        "from io import StringIO\n",
        "from matplotlib import pyplot as plt\n",
        "from PIL import Image\n",
        "\n",
        "# This is needed since the notebook is stored in the object_detection folder.\n",
        "sys.path.append(\"..\")\n",
        "from object_detection.utils import ops as utils_ops\n",
        "\n",
        "#if tf.__version__ < '1.4.0':\n",
        "#  raise ImportError('Please upgrade your tensorflow installation to v1.4.* or later!')\n",
        "  \n",
        "\n",
        "  \n",
        "  \n",
        "# This is needed to display the images.\n",
        "%matplotlib inline\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "from utils import label_map_util\n",
        "\n",
        "from utils import visualization_utils as vis_util\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# What model to download.\n",
        "# Path to frozen detection graph. This is the actual model that is used for the object detection.\n",
        "PATH_TO_CKPT = '/datalab/fine_tuned_model' + '/frozen_inference_graph.pb'\n",
        "\n",
        "# List of the strings that is used to add correct label for each box.\n",
        "PATH_TO_LABELS = os.path.join('/content/datalab', 'label_map.pbtxt')\n",
        "\n",
        "NUM_CLASSES = 37\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "detection_graph = tf.Graph()\n",
        "with detection_graph.as_default():\n",
        "  od_graph_def = tf.GraphDef()\n",
        "  with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:\n",
        "    serialized_graph = fid.read()\n",
        "    od_graph_def.ParseFromString(serialized_graph)\n",
        "    tf.import_graph_def(od_graph_def, name='')\n",
        "    \n",
        "    \n",
        "    \n",
        "    \n",
        "label_map = label_map_util.load_labelmap(PATH_TO_LABELS)\n",
        "categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)\n",
        "category_index = label_map_util.create_category_index(categories)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "def load_image_into_numpy_array(image):\n",
        "  (im_width, im_height) = image.size\n",
        "  return np.array(image.getdata()).reshape(\n",
        "      (im_height, im_width, 3)).astype(np.uint8)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# If you want to test the code with your images, just add path to the images to the TEST_IMAGE_PATHS.\n",
        "PATH_TO_TEST_IMAGES_DIR = '/datalab/images/'\n",
        "TEST_IMAGE_PATHS = [ os.path.join(PATH_TO_TEST_IMAGES_DIR, 'solar_{}.JPG'.format(i)) for i in range(150, 166) ]\n",
        "\n",
        "# Size, in inches, of the output images.\n",
        "IMAGE_SIZE = (12, 8)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "def run_inference_for_single_image(image, graph):\n",
        "  with graph.as_default():\n",
        "    with tf.Session() as sess:\n",
        "      # Get handles to input and output tensors\n",
        "      ops = tf.get_default_graph().get_operations()\n",
        "      all_tensor_names = {output.name for op in ops for output in op.outputs}\n",
        "      tensor_dict = {}\n",
        "      for key in [\n",
        "          'num_detections', 'detection_boxes', 'detection_scores',\n",
        "          'detection_classes', 'detection_masks'\n",
        "      ]:\n",
        "        tensor_name = key + ':0'\n",
        "        if tensor_name in all_tensor_names:\n",
        "          tensor_dict[key] = tf.get_default_graph().get_tensor_by_name(\n",
        "              tensor_name)\n",
        "      if 'detection_masks' in tensor_dict:\n",
        "        # The following processing is only for single image\n",
        "        detection_boxes = tf.squeeze(tensor_dict['detection_boxes'], [0])\n",
        "        detection_masks = tf.squeeze(tensor_dict['detection_masks'], [0])\n",
        "        # Reframe is required to translate mask from box coordinates to image coordinates and fit the image size.\n",
        "        real_num_detection = tf.cast(tensor_dict['num_detections'][0], tf.int32)\n",
        "        detection_boxes = tf.slice(detection_boxes, [0, 0], [real_num_detection, -1])\n",
        "        detection_masks = tf.slice(detection_masks, [0, 0, 0], [real_num_detection, -1, -1])\n",
        "        detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(\n",
        "            detection_masks, detection_boxes, image.shape[0], image.shape[1])\n",
        "        detection_masks_reframed = tf.cast(\n",
        "            tf.greater(detection_masks_reframed, 0.5), tf.uint8)\n",
        "        # Follow the convention by adding back the batch dimension\n",
        "        tensor_dict['detection_masks'] = tf.expand_dims(\n",
        "            detection_masks_reframed, 0)\n",
        "      image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')\n",
        "\n",
        "      # Run inference\n",
        "      output_dict = sess.run(tensor_dict,\n",
        "                             feed_dict={image_tensor: np.expand_dims(image, 0)})\n",
        "\n",
        "      # all outputs are float32 numpy arrays, so convert types as appropriate\n",
        "      output_dict['num_detections'] = int(output_dict['num_detections'][0])\n",
        "      output_dict['detection_classes'] = output_dict[\n",
        "          'detection_classes'][0].astype(np.uint8)\n",
        "      output_dict['detection_boxes'] = output_dict['detection_boxes'][0]\n",
        "      output_dict['detection_scores'] = output_dict['detection_scores'][0]\n",
        "      if 'detection_masks' in output_dict:\n",
        "        output_dict['detection_masks'] = output_dict['detection_masks'][0]\n",
        "  return output_dict\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "for image_path in TEST_IMAGE_PATHS:\n",
        "  image = Image.open(image_path)\n",
        "  # the array based representation of the image will be used later in order to prepare the\n",
        "  # result image with boxes and labels on it.\n",
        "  image_np = load_image_into_numpy_array(image)\n",
        "  # Expand dimensions since the model expects images to have shape: [1, None, None, 3]\n",
        "  image_np_expanded = np.expand_dims(image_np, axis=0)\n",
        "  # Actual detection.\n",
        "  output_dict = run_inference_for_single_image(image_np, detection_graph)\n",
        "  # Visualization of the results of a detection.\n",
        "  vis_util.visualize_boxes_and_labels_on_image_array(\n",
        "      image_np,\n",
        "      output_dict['detection_boxes'],\n",
        "      output_dict['detection_classes'],\n",
        "      output_dict['detection_scores'],\n",
        "      category_index,\n",
        "      instance_masks=output_dict.get('detection_masks'),\n",
        "      use_normalized_coordinates=True,\n",
        "      line_thickness=8)\n",
        "  plt.figure(figsize=IMAGE_SIZE)\n",
        "  plt.imshow(image_np)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-JCChMpsfOz0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}