notebooks/SynthDet_Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SynthDet Evaluation\n",
    "\n",
    "In this notebook, we provide the FasterRCNN estimator performance and visualization for predictions on the test split of the [UnityGroceries-Real](https://github.com/Unity-Technologies/SynthDet/blob/master/docs/UnityGroceriesReal.md) data. We support performance metrics (mAP, mAP@IOU50, mAR@100) as well as bounding box predictions rendered on the original image. This would provide a better understanding of the given estimator.\n",
    "\n",
    "<!-- You can use this notebook by the following steps:\n",
    "- Specify the model path. Then, the notebook would load the checkpoints into a `FasterRCNN` estimator. The `FasterRCNN` can provide model predictions.\n",
    "- Provide the model performance metrics.\n",
    "- Can either specify or randomly select some cases for the visualization. -->\n",
    "\n",
    "\n",
    "## Settings\n",
    "\n",
    "- Point `data_root` below where you want to download the dataset. If you run this dataset inside a docker container, make sure you point this path to the directory where the external volume is mounted for data storage. \n",
    "\n",
    "- Specify a pre-trained `checkpoint_file` that you want to use. You can use one of the estimators that we provided. Optionally, you can use estimators you have trained using our pre-compiled Kubeflow pipeline (Please make sure the GCS credential is setup correctly)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify data_root\n",
    "data_path = \"/data\"\n",
    "\n",
    "# specify estimator path\n",
    "# 1. Real-trained estimator on 760 images\n",
    "# checkpoint_file = \"https://storage.googleapis.com/datasetinsights/models/Real-World/FasterRCNN.estimator\"\n",
    "\n",
    "# 2. Synth-trained estimator on 400K SynthDet dataset\n",
    "# checkpoint_file = \"https://storage.googleapis.com/datasetinsights/models/Synthetic/FasterRCNN.estimator\"\n",
    "\n",
    "# 3. Fine-tuned estimator (pre-trained on 400K Synthetic data and fine-tuned on 76 images)\n",
    "# checkpoint_file = \"https://storage.googleapis.com/datasetinsights/models/Synthetic-And-Real-World-76-images/FasterRCNN.estimator\"\n",
    "\n",
    "# 4. Fine-tuned estimator (pre-trained on 400K Synthetic data and fine-tuned on 380 images)\n",
    "# checkpoint_file = \"https://storage.googleapis.com/datasetinsights/models/Synthetic-And-Real-World-380-images/FasterRCNN.estimator\"\n",
    "\n",
    "# 5. Fine-tuned estimator (pre-trained on 400K Synthetic data and fine-tuned on 760 images)\n",
    "# This is the estimator that provide the best result.\n",
    "checkpoint_file = \"https://storage.googleapis.com/datasetinsights/models/Synthetic-And-Real-World-760-images/FasterRCNN.estimator\"\n",
    "\n",
    "# 6. Your estimator that was stored on GCS\n",
    "# checkpoint_file = \"gs://\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download UnityGroceries-Real Dataset\n",
    "\n",
    "This cell will download the public UnityGroceries-Real dataset to the location specified by data_path. You only need to run this cell once and assuming the dataset exists for the subsequence of the notebook execution. \n",
    "\n",
    "The [downloader](https://datasetinsights.readthedocs.io/en/latest/datasetinsights.io.html#datasetinsights.io.loader.create_loader) instantiates the dataset downloader after finding it with the source-uri provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasetinsights.io import create_downloader\n",
    "\n",
    "groceries_real_source_uri = \"https://storage.googleapis.com/datasetinsights/data/groceries/v3.zip\"\n",
    "downloader = create_downloader(source_uri=groceries_real_source_uri)\n",
    "downloader.download(source_uri=groceries_real_source_uri, output=data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load UnityGroceries-Real [test](https://github.com/Unity-Technologies/SynthDet/blob/master/docs/Readme.md) split which has 254 images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasetinsights.datasets import Dataset\n",
    "\n",
    "test_dataset = Dataset.create(\"GroceriesReal\", data_path=data_path, split=\"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Estimator\n",
    "An [estimator](https://datasetinsights.readthedocs.io/en/latest/datasetinsights.estimators.html#datasetinsights.estimators.base.Estimator) is a class of one modeling operation. It includes:\n",
    "\n",
    "1. input data and output data transformations (e.g. input image cropping, remove unused output labels…) when applicable. \n",
    "2. neural network graph (model) for either PyTorch or TensorFlow. \n",
    "3. procedures to execute model training and evaluation. <br>\n",
    "\n",
    "This cell will load an estimator specified in the variable name `checkpoint_file`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from yacs.config import CfgNode as CN\n",
    "from datasetinsights.estimators import create_estimator\n",
    "    \n",
    "config_yaml = \"\"\"\n",
    "    estimator: FasterRCNN\n",
    "    backbone: resnet50\n",
    "    num_classes: 64\n",
    "    task: object_detection\n",
    "    test:\n",
    "      batch_size: 8\n",
    "      dataset:\n",
    "        name: GroceriesReal\n",
    "        args:\n",
    "          version: v3\n",
    "          split: test\n",
    "    metrics:\n",
    "      mAP:\n",
    "        name: MeanAveragePrecisionAverageOverIOU\n",
    "      mAPIOU50:\n",
    "        name: MeanAveragePrecisionIOU50\n",
    "      mAR:\n",
    "        name: MeanAverageRecallAverageOverIOU\n",
    "    pretrained: False\n",
    "    pretrained_backbone: True\n",
    "    synchronize_metrics: True\n",
    "\"\"\"\n",
    "config = CN.load_cfg(config_yaml)\n",
    "estimator = create_estimator(\n",
    "    name=config.estimator,\n",
    "    config=config,\n",
    "    checkpoint_file=checkpoint_file,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction Visualization\n",
    "\n",
    "In order to improve visual inspection, we have color-coded bounding boxes predictions based on IOU value between prediction and ground truth bounding boxes. It is considered true positive if `IOU >= 0.5`. We only visualize prediction bounding box with `score >= 0.5`. \n",
    "\n",
    "- <font color='green'>Green boxes</font>: If the predicted bounding box can be matched to a ground truth bounding box. <br>\n",
    "- <font color='red'>Red boxes</font>: If the predicted bounding box can't be matched to a ground truth bounding box. <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasetinsights.stats.visualization import match_boxes, plot_bboxes, grid_plot\n",
    "\n",
    "def visualize_predictions(index=0):\n",
    "    \"\"\" Plot ground truth and prediction for one image.\n",
    "    \n",
    "    This method would plot two images: the ground truth is on the left;\n",
    "    the prediction from the loaded estimator is on the right.\n",
    "    \"\"\"\n",
    "    estimator.model.eval()\n",
    "    pil_image, gt_bboxes = test_dataset[index]\n",
    "    pred_bboxes = estimator.predict(pil_image, box_score_thresh=0.5)\n",
    "    colors = match_boxes(pred_bboxes, gt_bboxes)\n",
    "    gt_plot = plot_bboxes(pil_image, gt_bboxes, test_dataset.label_mappings)\n",
    "    pred_plot = plot_bboxes(pil_image, pred_bboxes, test_dataset.label_mappings, colors)\n",
    "    \n",
    "    titles = [\n",
    "        f\"ground truth for image index: {index}\",\n",
    "        f\"prediction for image index: {index}\",\n",
    "    ]\n",
    "    grid_plot([[gt_plot, pred_plot]], figsize=(7, 10), img_type=\"rgb\", titles=titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ipywidgets import interact\n",
    "\n",
    "# switch to evaluation mode.\n",
    "estimator_eval = estimator.model.eval()\n",
    "# Please select an index of a image.\n",
    "interact(visualize_predictions, index=list(range(len(test_dataset))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimator Performance\n",
    "This section calculates estimator performance on the UnityGroceries-Real Dataset test split. We report the following three evaluation metrics that are commonly used for object detection task:\n",
    "- [mAP](https://datasetinsights.readthedocs.io/en/latest/datasetinsights.evaluation_metrics.html#datasetinsights.evaluation_metrics.average_precision_2d.MeanAveragePrecisionAverageOverIOU): Average Precision average over all the labels and IOU thresholds = 0.5:0.95:0.05\n",
    "- [mAPIOU50](https://datasetinsights.readthedocs.io/en/latest/datasetinsights.evaluation_metrics.html#datasetinsights.evaluation_metrics.average_precision_2d.MeanAveragePrecisionIOU50): Mean Average Precision at IOU=50%.\n",
    "- [mAR](https://datasetinsights.readthedocs.io/en/latest/datasetinsights.evaluation_metrics.html#datasetinsights.evaluation_metrics.average_recall_2d.MeanAverageRecallAverageOverIOU): Average Recall average over all the labels and IOU thresholds = 0.5:0.95:0.05\n",
    "\n",
    "The next cell will take ~1 hour if you run model precision locally without GPU support. This will make predictions on the whole test set of 254 images. You can reduce computation time significantly on GPU with CUDA support."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is expected to take a while\n",
    "estimator.evaluate(data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "metric_names = [\"mAP\", \"mAPIOU50\", \"mAR\"]\n",
    "metrics = estimator.metrics\n",
    "df = pd.DataFrame({\n",
    "    \"Value\": [metrics[name].compute() for name in metric_names],\n",
    "})\n",
    "df.index = metric_names\n",
    "df"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}