docs/source/tutorials/detection_mistakes.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding Detection Mistakes with FiftyOne\n",
    "\n",
    "Annotations mistakes create an artificial ceiling on the performance of your models. However, finding these mistakes by hand is at least as arduous as the original annotation work! Enter FiftyOne.\n",
    "\n",
    "In this tutorial, we explore how FiftyOne can be used to help you find mistakes in your object detection annotations. To detect mistakes in classification datasets, check out [this tutorial](https://voxel51.com/docs/fiftyone/tutorials/classification_mistakes.html).\n",
    "\n",
    "We'll cover the following concepts:\n",
    "\n",
    "- Loading your existing dataset [into FiftyOne](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/index.html)\n",
    "- [Adding model predictions](https://voxel51.com/docs/fiftyone/recipes/adding_detections.html) to your dataset\n",
    "- Computing insights into your dataset relating to [possible label mistakes](https://voxel51.com/docs/fiftyone/user_guide/brain.html#label-mistakes)\n",
    "- Visualizing mistakes in the [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html)\n",
    "\n",
    "**So, what's the takeaway?**\n",
    "\n",
    "FiftyOne can help you find and correct label mistakes in your datasets, enabling you to curate higher quality datasets and, ultimately, train better models!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "If you haven't already, install FiftyOne:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install fiftyone"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to compute mistakenness, your dataset needs to have two [detections fields](https://voxel51.com/docs/fiftyone/user_guide/using_datasets.html#object-detection), one with your ground truth annotations and one with your model predictions.\n",
    "\n",
    "In this example, we'll load the [quickstart dataset](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/datasets.html#dataset-zoo-quickstart) from the FiftyOne Dataset Zoo, which has ground truth annotations and predictions from a [PyTorch Faster-RCNN model](https://github.com/pytorch/vision/blob/master/torchvision/models/detection/faster_rcnn.py) for a few samples from the COCO dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset already downloaded\n",
      "Loading 'quickstart'\n",
      " 100% |█████████████████| 200/200 [2.0s elapsed, 0s remaining, 99.8 samples/s]          \n",
      "Dataset 'quickstart' created\n"
     ]
    }
   ],
   "source": [
    "import fiftyone as fo\n",
    "import fiftyone.zoo as foz\n",
    "\n",
    "dataset = foz.load_zoo_dataset(\"quickstart\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:        quickstart\n",
      "Media type:  image\n",
      "Num samples: 200\n",
      "Persistent:  False\n",
      "Tags:        []\n",
      "Sample fields:\n",
      "    id:           fiftyone.core.fields.ObjectIdField\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    uniqueness:   fiftyone.core.fields.FloatField\n",
      "    predictions:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n"
     ]
    }
   ],
   "source": [
    "print(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Detection: {\n",
      "    'id': '5f452c60ef00e6374aad9394',\n",
      "    'attributes': {},\n",
      "    'tags': [],\n",
      "    'label': 'bird',\n",
      "    'bounding_box': [\n",
      "        0.22192673683166503,\n",
      "        0.06093006531397502,\n",
      "        0.4808845520019531,\n",
      "        0.8937615712483724,\n",
      "    ],\n",
      "    'mask': None,\n",
      "    'confidence': 0.9750854969024658,\n",
      "    'index': None,\n",
      "}>\n"
     ]
    }
   ],
   "source": [
    "# Print a sample ground truth detection\n",
    "sample = dataset.first()\n",
    "print(sample.predictions.detections[0])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by visualizing the dataset in the [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open the dataset in the App\n",
    "session = fo.launch_app(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Full Dataset](./images/detection_mistakes_full_dataset.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When working with FiftyOne datasets that contain a field with `Detections`, you can create a [patches view](https://voxel51.com/docs/fiftyone/user_guide/app.html#viewing-object-patches) both through Python and directly in the FiftyOne App to view each detection as a separate sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset:     quickstart\n",
      "Media type:  image\n",
      "Num patches: 1232\n",
      "Patch fields:\n",
      "    id:           fiftyone.core.fields.ObjectIdField\n",
      "    sample_id:    fiftyone.core.fields.ObjectIdField\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detection)\n",
      "View stages:\n",
      "    1. ToPatches(field='ground_truth', config=None)\n"
     ]
    }
   ],
   "source": [
    "patches_view = dataset.to_patches(\"ground_truth\")\n",
    "print(patches_view)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's open the App and click the [patches button](https://voxel51.com/docs/fiftyone/user_guide/app.html#viewing-object-patches), then select `ground_truth` to create the same view that we created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = fo.launch_app(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Ground Truth Patches](./images/detection_mistakes_gt_patches.gif)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute mistakenness\n",
    "\n",
    "Now we're ready to assess the mistakenness of the ground truth detections.\n",
    "\n",
    "We can do so by running the [compute_mistakenness()](https://voxel51.com/docs/fiftyone/api/fiftyone.brain.html#fiftyone.brain.compute_mistakenness) method from the FiftyOne Brain:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating detections...\n",
      " 100% |█████████████████| 200/200 [13.8s elapsed, 0s remaining, 9.9 samples/s]       \n",
      "Computing mistakenness...\n",
      " 100% |█████████████████| 200/200 [2.6s elapsed, 0s remaining, 52.2 samples/s]       \n",
      "Mistakenness computation complete\n"
     ]
    }
   ],
   "source": [
    "import fiftyone.brain as fob\n",
    "\n",
    "# Compute mistakenness of annotations in `ground_truth` field using \n",
    "# predictions from `predictions` field as point of reference\n",
    "fob.compute_mistakenness(dataset, \"predictions\", label_field=\"ground_truth\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above method populates a number of fields on the samples of our dataset as well as the ground truth and predicted objects:\n",
    "\n",
    "New ground truth object attributes (in `ground_truth` field):\n",
    "\n",
    "- `mistakenness` (float): A measure of the likelihood that a ground truth object's label is incorrect\n",
    "- `mistakenness_loc`: A measure of the likelihood that a ground truth object's localization (bounding box) is inaccurate\n",
    "- `possible_spurious`: Ground truth objects that were not matched with a predicted object and are deemed to be likely spurious annotations will have this attribute set to True\n",
    "\n",
    "New predicted object attributes (in `predictions` field):\n",
    "\n",
    "- `possible_missing`: If a highly confident prediction with no matching ground truth object is encountered, this attribute is set to True to indicate that it is a likely missing ground truth annotation\n",
    "\n",
    "Sample-level fields:\n",
    "\n",
    "- `mistakenness`: The maximum mistakenness of the ground truth objects in each sample\n",
    "- `possible_spurious`: The number of possible spurious ground truth objects in each sample\n",
    "- `possible_missing`: The number of possible missing ground truth objects in each sample"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing the results\n",
    "\n",
    "Let's use FiftyOne to investigate the results.\n",
    "\n",
    "First, let's show the samples with the most likely annotation mistakes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset:     quickstart\n",
      "Media type:  image\n",
      "Num samples: 200\n",
      "Sample fields:\n",
      "    id:                fiftyone.core.fields.ObjectIdField\n",
      "    filepath:          fiftyone.core.fields.StringField\n",
      "    tags:              fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:          fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)\n",
      "    ground_truth:      fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    uniqueness:        fiftyone.core.fields.FloatField\n",
      "    predictions:       fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    mistakenness:      fiftyone.core.fields.FloatField\n",
      "    possible_missing:  fiftyone.core.fields.IntField\n",
      "    possible_spurious: fiftyone.core.fields.IntField\n",
      "View stages:\n",
      "    1. SortBy(field_or_expr='mistakenness', reverse=True, create_index=True)\n"
     ]
    }
   ],
   "source": [
    "from fiftyone import ViewField as F\n",
    "\n",
    "# Sort by likelihood of mistake (most likely first)\n",
    "mistake_view = dataset.sort_by(\"mistakenness\", reverse=True)\n",
    "\n",
    "# Print some information about the view\n",
    "print(mistake_view)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Detection: {\n",
      "    'id': '5f452487ef00e6374aad2744',\n",
      "    'attributes': BaseDict({}),\n",
      "    'tags': BaseList([]),\n",
      "    'label': 'tv',\n",
      "    'bounding_box': BaseList([\n",
      "        0.002746666666666667,\n",
      "        0.36082,\n",
      "        0.24466666666666667,\n",
      "        0.3732,\n",
      "    ]),\n",
      "    'mask': None,\n",
      "    'confidence': None,\n",
      "    'index': None,\n",
      "    'area': 16273.3536,\n",
      "    'iscrowd': 0.0,\n",
      "    'mistakenness': 0.005771428346633911,\n",
      "    'mistakenness_loc': 0.16955941131917984,\n",
      "}>\n"
     ]
    }
   ],
   "source": [
    "# Inspect some samples and detections\n",
    "# This is the first detection of the first sample\n",
    "print(mistake_view.first().ground_truth.detections[0])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the App to visually inspect the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the samples we processed in rank order by the mistakenness\n",
    "session.view = mistake_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Mistakenness](./images/detection_mistakes_mistakenness.gif)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another useful query is to find all objects that have a high mistakenness, lets say > 0.95:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fiftyone import ViewField as F\n",
    "\n",
    "session.view = dataset.filter_labels(\"ground_truth\", F(\"mistakenness\") > 0.95)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![High Mistakenness](./images/detection_mistakes_above95.gif)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking through the results, we can see that many of these images have a bunch of predictions which actually look like they are correct, but no ground truth annotations. This is a common mistake in object detection datasets, where the annotator may have missed some objects in the image. On the other hand, there are some detections which are mislabeled, like the `cow` in the fifth image above which is predicted to be a horse.\n",
    "\n",
    "We can use a similar workflow to look at objects that may be localized poorly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.view = dataset.filter_labels(\"ground_truth\", F(\"mistakenness_loc\") > 0.85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Localization Mistakes](./images/detection_mistakes_localization.gif)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some of these examples, like the image of people on the beach, there is not necessarily highly mistaken localization, there are just a bunch of small, relatively overlapping objects. In other examples, such as the handbag in the second instance and the skis in the third instance, the localization is clearly off."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `possible_missing` field can also be useful to sort by to find instances of incorrect annotations.\n",
    "\n",
    "Similarly, `possible_spurious` can be used to find objects that the model detected that may have been missed by annotators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.view = dataset.match(F(\"possible_missing\") > 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Possible Missing](./images/detection_mistakes_possible_missing.gif)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example that showed up from this search is shown above. There is an `apple` that was not annotated that the model detected."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tagging and resolution\n",
    "\n",
    "Any label or collection of labels can be tagged at any time in the sample grid or expanded sample view. In the expanded sample view, individual samples can be selected by clicking on them in the media player. We can, for example, tag this `apple` prediction as `missing` and any other predictions without an associated ground truth detection.\n",
    "\n",
    "Labels with specific tags can then be selected with [select_labels()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html?highlight=select_labels#fiftyone.core.collections.SampleCollection.select_labels) stage and sent off to assist in improving the annotations with your annotation provided of choice. FiftyOne currently offers integrations for both [Labelbox](https://voxel51.com/docs/fiftyone/api/fiftyone.utils.labelbox.html) and [Scale](https://voxel51.com/docs/fiftyone/api/fiftyone.utils.scale.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A dataset can be filtered to only contain labels with certain tags\n",
    "# Helpful for isolating labels with issues and sending off to an annotation provider\n",
    "missing_ground_truth = dataset.select_labels(tags=\"missing\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**REMEMBER**: Since you are using model predictions to guide the mistakenness process, the better your model, the more accurate the mistakenness suggestions. Additionally, using logits of confidence scores will also provide better results. \n",
    "\n",
    "We used Faster-RCNN in this example which is quite a few years old. Using EfficientDet D7 provided much better results. For example, it was easily able to find this `snowboard` labeled as `skis`:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![skis](images/det_mistakenness_6.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.freeze() # screenshot the active App for sharing"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}