ssd300_evaluation_Pascal_VOC.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SSD300 Pascal VOC Evaluation Tutorial\n",
    "\n",
    "This is a brief tutorial that goes over how to evaluate the SSD300 '07+12+COCO' model (i.e. the version trained on MS COCO and afterwards fine-tuned on Pascal VOC) on the Pascal VOC2007 test dataset using the official Matlab evaluation script that comes with the [`VOCdevkit`](http://host.robots.ox.ac.uk/pascal/VOC/voc2007/index.html).\n",
    "\n",
    "Of course the evaulation procedure described here is identical for SSD512, you just need to build a different model. Make sure that you build the model with the correct set of scaling factors to reproduce the official results. The models that were trained on MS COCO and fine-tuned on Pascal VOC need the MS COCO scaling factors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import backend as K\n",
    "from keras.models import load_model\n",
    "from keras.optimizers import Adam\n",
    "from scipy.misc import imread\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "from models.keras_ssd300 import ssd_300\n",
    "from keras_loss_function.keras_ssd_loss import SSDLoss\n",
    "from keras_layers.keras_layer_AnchorBoxes import AnchorBoxes\n",
    "from keras_layers.keras_layer_DecodeDetections import DecodeDetections\n",
    "from keras_layers.keras_layer_DecodeDetections2 import DecodeDetections2\n",
    "from keras_layers.keras_layer_L2Normalization import L2Normalization\n",
    "from data_generator.ssd_batch_generator import BatchGenerator\n",
    "from eval_utils.pascal_voc_utils import predict_all_to_txt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set the input image size for the model.\n",
    "img_height = 300\n",
    "img_width = 300"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load a trained SSD\n",
    "\n",
    "Either load a trained model or build a model and load trained weights into it. Since the HDF5 files I'm providing contain only the weights for the various SSD versions, not the complete models, you'll have to go with the latter option when using this implementation for the first time. You can then of course save the model and next time load the full model directly, without having to build it.\n",
    "\n",
    "You can find the download links to all the trained model weights in the README."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Build the model and load trained weights into it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 1: Build the Keras model\n",
    "\n",
    "K.clear_session() # Clear previous models from memory.\n",
    "\n",
    "model = ssd_300(image_size=(img_height, img_width, 3),\n",
    "                n_classes=20,\n",
    "                mode='inference',\n",
    "                l2_regularization=0.0005,\n",
    "                scales=[0.07, 0.15, 0.33, 0.51, 0.69, 0.87, 1.05], # The scales for Pascal VOC are [0.1, 0.2, 0.37, 0.54, 0.71, 0.88, 1.05]\n",
    "                aspect_ratios_per_layer=[[1.0, 2.0, 0.5],\n",
    "                                         [1.0, 2.0, 0.5, 3.0, 1.0/3.0],\n",
    "                                         [1.0, 2.0, 0.5, 3.0, 1.0/3.0],\n",
    "                                         [1.0, 2.0, 0.5, 3.0, 1.0/3.0],\n",
    "                                         [1.0, 2.0, 0.5],\n",
    "                                         [1.0, 2.0, 0.5]],\n",
    "                two_boxes_for_ar1=True,\n",
    "                steps=[8, 16, 32, 64, 100, 300],\n",
    "                offsets=[0.5, 0.5, 0.5, 0.5, 0.5, 0.5],\n",
    "                limit_boxes=False,\n",
    "                variances=[0.1, 0.1, 0.2, 0.2],\n",
    "                coords='centroids',\n",
    "                normalize_coords=True,\n",
    "                subtract_mean=[123, 117, 104],\n",
    "                swap_channels=True,\n",
    "                confidence_thresh=0.01,\n",
    "                iou_threshold=0.45,\n",
    "                top_k=200,\n",
    "                nms_max_output_size=400)\n",
    "\n",
    "# 2: Load the trained weights into the model.\n",
    "\n",
    "# TODO: Set the path of the trained weights.\n",
    "weights_path = 'path/to/trained/weights/VGG_VOC0712_SSD_300x300_ft_iter_120000.h5'\n",
    "\n",
    "model.load_weights(weights_path, by_name=True)\n",
    "\n",
    "# 3: Compile the model so that Keras won't complain the next time you load it.\n",
    "\n",
    "adam = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=5e-04)\n",
    "\n",
    "ssd_loss = SSDLoss(neg_pos_ratio=3, n_neg_min=0, alpha=1.0)\n",
    "\n",
    "model.compile(optimizer=adam, loss=ssd_loss.compute_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Load a trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: Set the path to the `.h5` file of the model to be loaded.\n",
    "model_path = 'path/to/trained/model.h5'\n",
    "\n",
    "# We need to create an SSDLoss object in order to pass that to the model loader.\n",
    "ssd_loss = SSDLoss(neg_pos_ratio=3, n_neg_min=0, alpha=1.0)\n",
    "\n",
    "K.clear_session() # Clear previous models from memory.\n",
    "\n",
    "model = load_model(model_path, custom_objects={'AnchorBoxes': AnchorBoxes,\n",
    "                                               'L2Normalization': L2Normalization,\n",
    "                                               'DecodeDetections': DecodeDetections,\n",
    "                                               'compute_loss': ssd_loss.compute_loss})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create a data generator for the evaluation dataset\n",
    "\n",
    "Instantiate a `BatchGenerator` that will serve the evaluation dataset during the prediction phase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dataset = BatchGenerator(box_output_format=['class_id', 'xmin', 'ymin', 'xmax', 'ymax'])\n",
    "\n",
    "# TODO: Set the paths to the dataset here.\n",
    "Pascal_VOC_dataset_images_dir = '../../datasets/VOCdevkit/VOC2007/JPEGImages/'\n",
    "Pascal_VOC_dataset_annotations_dir = '../../datasets/VOCdevkit/VOC2007/Annotations/'\n",
    "Pascal_VOC_dataset_image_set_filename = '../../datasets/VOCdevkit/VOC2007/ImageSets/Main/test.txt'\n",
    "\n",
    "dataset.parse_xml(images_dirs=[Pascal_VOC_dataset_images_dir],\n",
    "                  image_set_filenames=[Pascal_VOC_dataset_image_set_filename],\n",
    "                  annotations_dirs=[Pascal_VOC_dataset_annotations_dir],\n",
    "                  classes=classes,\n",
    "                  include_classes='all',\n",
    "                  exclude_truncated=False,\n",
    "                  exclude_difficult=False,\n",
    "                  ret=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Run the predictions over the evaluation dataset\n",
    "\n",
    "Now that we have instantiated a model and a data generator to serve the dataset, we can make predictions on the entire dataset and save those predictions in text files in the appropriate format in which the Pascal VOC evaluation server (or the evaluation Matlab script) needs them for the evaluation.\n",
    "\n",
    "Read the documenation to learn what the arguments mean, but the arguments as preset below reflect the parameters used in the evaluation of the original Caffe models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: Set the batch size.\n",
    "batch_size = 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of images in the evaluation dataset: 4952\n",
      "Producing results file: 100%|██████████| 619/619 [05:21<00:00,  2.11it/s]\n",
      "All results files saved.\n"
     ]
    }
   ],
   "source": [
    "predict_all_to_txt(model=model,\n",
    "                   img_height=img_height,\n",
    "                   img_width=img_width,\n",
    "                   batch_generator=dataset,\n",
    "                   batch_size=batch_size,\n",
    "                   batch_generator_mode='resize',\n",
    "                   classes=['background',\n",
    "                            'aeroplane', 'bicycle', 'bird', 'boat',\n",
    "                            'bottle', 'bus', 'car', 'cat',\n",
    "                            'chair', 'cow', 'diningtable', 'dog',\n",
    "                            'horse', 'motorbike', 'person', 'pottedplant',\n",
    "                            'sheep', 'sofa', 'train', 'tvmonitor'],\n",
    "                   out_file_prefix='ssd300_07+12_2007_test_eval/comp3_det_test_',\n",
    "                   model_mode='inference',\n",
    "                   confidence_thresh=0.01,\n",
    "                   iou_threshold=0.45,\n",
    "                   top_k=200,\n",
    "                   pred_coords='centroids',\n",
    "                   normalize_coords=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 4. Run the evaluation\n",
    "\n",
    "Now that we've generated the results text files for each class in the right format, we still have to evaluate the results.\n",
    "\n",
    "This repository doesn't provide its own Pascal VOC mAP evaluation module at the moment, so you'll have to use the official Matlab evaluation script that comes with the `VOCdevkit`. Note that the mAP computation formula differs for VOC2007 and VOC2012, and the `VOCdevkit` for each of these years comes with its own evaluation script that you should use for the respective dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}