docs/source/tutorials/evaluate_detections.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Evaluating Object Detections with FiftyOne\n",
    "\n",
    "This walkthrough demonstrates how to use FiftyOne to perform hands-on evaluation of your detection model.\n",
    "\n",
    "It covers the following concepts:\n",
    "\n",
    "- Loading a dataset with ground truth labels [into FiftyOne](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/index.html)\n",
    "- [Adding model predictions](https://voxel51.com/docs/fiftyone/recipes/adding_detections.html) to your dataset\n",
    "- [Evaluating your model](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html#detections) using FiftyOne's evaluation API\n",
    "- Viewing the best and worst performing samples in your dataset\n",
    "\n",
    "**So, what's the takeaway?**\n",
    "\n",
    "Aggregate measures of performance like mAP don't give you the full picture of your detection model. In practice, the limiting factor on your model's performance is often data quality issues that you need to **see** to address. FiftyOne is designed to make it easy to do just that.\n",
    "\n",
    "Running the workflow presented here on your ML projects will help you to understand the current failure modes (edge cases) of your model and how to fix them, including:\n",
    "\n",
    "- Identifying scenarios that require additional training samples in order to boost your model's performance\n",
    "- Deciding whether your ground truth annotations have errors/weaknesses that need to be corrected before any subsequent model training will be profitable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "If you haven't already, install FiftyOne:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install fiftyone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we'll use an off-the-shelf [Faster R-CNN detection model](https://pytorch.org/docs/stable/torchvision/models.html#faster-r-cnn) provided by PyTorch. To use it, you'll need to install `torch` and `torchvision`, if necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install torch torchvision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you wanted to, you could download the pretrained model from the web and load it with `torchvision`. However, this model is also available via the [FiftyOne Model Zoo](https://docs.voxel51.com/user_guide/model_zoo/index.html), which makes our lives much easier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fiftyone.zoo as foz\n",
    "model = foz.load_zoo_model('faster-rcnn-resnet50-fpn-coco-torch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll perform our analysis on the validation split of the [COCO dataset](https://cocodataset.org/#home), which is conveniently available for download via the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/datasets.html#coco-2017).\n",
    "\n",
    "The snippet below will download the validation split and load it into FiftyOne."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading split 'validation' to '/Users/jacobmarks/fiftyone/coco-2017/validation' if necessary\n",
      "Found annotations at '/Users/jacobmarks/fiftyone/coco-2017/raw/instances_val2017.json'\n",
      "Images already downloaded\n",
      "Existing download of split 'validation' is sufficient\n",
      "Loading 'coco-2017' split 'validation'\n",
      " 100% |███████████████| 5000/5000 [14.7s elapsed, 0s remaining, 360.0 samples/s]      \n",
      "Dataset 'evaluate-detections-tutorial' created\n"
     ]
    }
   ],
   "source": [
    "import fiftyone as fo\n",
    "import fiftyone.zoo as foz\n",
    "\n",
    "dataset = foz.load_zoo_dataset(\n",
    "    \"coco-2017\",\n",
    "    split=\"validation\",\n",
    "    dataset_name=\"evaluate-detections-tutorial\",\n",
    ")\n",
    "dataset.persistent = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the dataset to see what we downloaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:        evaluate-detections-tutorial\n",
      "Media type:  image\n",
      "Num samples: 5000\n",
      "Persistent:  True\n",
      "Tags:        []\n",
      "Sample fields:\n",
      "    id:           fiftyone.core.fields.ObjectIdField\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n"
     ]
    }
   ],
   "source": [
    "# Print some information about the dataset\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Detection: {\n",
      "    'id': '66047a4705c1282f3e97c5e9',\n",
      "    'attributes': {},\n",
      "    'tags': [],\n",
      "    'label': 'potted plant',\n",
      "    'bounding_box': [\n",
      "        0.37028125,\n",
      "        0.3345305164319249,\n",
      "        0.038593749999999996,\n",
      "        0.16314553990610328,\n",
      "    ],\n",
      "    'mask': None,\n",
      "    'confidence': None,\n",
      "    'index': None,\n",
      "    'supercategory': 'furniture',\n",
      "    'iscrowd': 0,\n",
      "}>\n"
     ]
    }
   ],
   "source": [
    "# Print a ground truth detection\n",
    "sample = dataset.first()\n",
    "print(sample.ground_truth.detections[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the ground truth detections are stored in the `ground_truth` field of the samples.\n",
    "\n",
    "Before we go further, let's launch the [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html) and use the GUI to explore the dataset visually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = fo.launch_app(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Evaluate Detections Dataset](images/evaluate_detections_dataset.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add predictions to dataset\n",
    "\n",
    "Now let's generate some predictions to analyze.\n",
    "\n",
    "Because we loaded the model from the FiftyOne Model Zoo, it is a FiftyOne model object, which means we can apply it directly \n",
    "to our dataset (or any subset thereof) for inference using the sample collection's `apply_model()` method.\n",
    "\n",
    "The code below performs inference with the Faster R-CNN model on a randomly chosen subset of 100 samples from the dataset and stores the resulting predictions in a `faster_rcnn` field of the samples. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose a random subset of 100 samples to add predictions to\n",
    "predictions_view = dataset.take(100, seed=51)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 100% |█████████████████| 100/100 [1.4m elapsed, 0s remaining, 1.3 samples/s]      \n"
     ]
    }
   ],
   "source": [
    "predictions_view.apply_model(model, label_field=\"faster_rcnn\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load `predictions_view` in the App to visualize the predictions that we added:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.view = predictions_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Predictions View](images/evaluate_detections_prediction_view.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing detections\n",
    "\n",
    "Let's analyze the raw predictions we've added to our dataset in more detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing bounding boxes\n",
    "\n",
    "Let's start by loading the full dataset in the App:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resets the session; the entire dataset will now be shown\n",
    "session.view = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![All boxes](images/evaluate_detections_all_boxes.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only the 100 samples in `predictions_view` have predictions in their `faster_rcnn` field, so some of the samples we see above do not have predicted boxes.\n",
    "\n",
    "If we want to recover our predictions view, we can do this programmatically via `session.view = predictions_view`, or we can use the [view bar in the App](https://voxel51.com/docs/fiftyone/user_guide/app.html#using-the-view-bar) to accomplish the same thing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the view bar to create an `Exists(faster_rcnn, True)` stage\n",
    "# Now your view contains only the 100 samples with predictions in `faster_rcnn` field\n",
    "session.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Exists Filter](images/evaluate_detections_exists_filter.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each field of the samples are shown as togglable checkboxes on the left sidebar which can be used to control whether ground truth detections or predictions are rendered on the images.\n",
    "\n",
    "You can also click on an image to view the sample in more detail:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Click Image](images/evaluate_detections_click_image.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting samples of interest\n",
    "\n",
    "You can select images in the App by clicking on the checkbox when hovering over an image. Then, you can create a view that contains only those samples by clicking the orange checkmark with the number of selected samples in the top left corner of the sample grid and clicking `Only show selected samples`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Only show selected](images/evaluate_detections_only_show_selected.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's reset our session to show our `predictions_view`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.view = predictions_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confidence thresholding in the App\n",
    "\n",
    "From the App instance above, it looks like our detector is generating some spurious low-quality detections. Let's use the App to interactively filter the predictions by `confidence` to identify a reasonable confidence threshold for our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Click the down caret on the `faster_rcnn` field of Fields Sidebar\n",
    "# and apply a confidence threshold\n",
    "session.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Confidence Threshold](images/evaluate_detections_confidence_filter.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like a confidence threshold of 0.75 is a good choice for our model, but we'll confirm that quantitatively later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confidence thresholding in Python\n",
    "\n",
    "FiftyOne also provides the ability to [write expressions](https://voxel51.com/docs/fiftyone/user_guide/using_views.html#filtering) that match, filter, and sort detections based on their attributes. See [using DatasetViews](https://voxel51.com/docs/fiftyone/user_guide/using_views.html) for full details.\n",
    "\n",
    "For example, we can programmatically generate a view that contains only detections whose `confidence` is at least `0.75` as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fiftyone import ViewField as F\n",
    "\n",
    "# Only contains detections with confidence >= 0.75\n",
    "high_conf_view = predictions_view.filter_labels(\"faster_rcnn\", F(\"confidence\") > 0.75, only_matches=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the `only_matches=False` argument. When filtering labels, any samples that no longer contain labels would normally be removed from the view. However, this is not desired when performing evaluations since it can skew your results between views. We set `only_matches=False` so that all samples will be retained, even if some no longer contain labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset:     evaluate-detections-tutorial\n",
      "Media type:  image\n",
      "Num samples: 100\n",
      "Sample fields:\n",
      "    id:           fiftyone.core.fields.ObjectIdField\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    faster_rcnn:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "View stages:\n",
      "    1. Take(size=100, seed=51)\n",
      "    2. FilterLabels(field='faster_rcnn', filter={'$gt': ['$$this.confidence', 0.75]}, only_matches=False, trajectories=False)\n"
     ]
    }
   ],
   "source": [
    "# Print some information about the view\n",
    "print(high_conf_view)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Detection: {\n",
      "    'id': '66047b0f05c1282f3e986920',\n",
      "    'attributes': {},\n",
      "    'tags': [],\n",
      "    'label': 'airplane',\n",
      "    'bounding_box': [\n",
      "        0.5629980087280273,\n",
      "        0.7977214296832356,\n",
      "        0.03478360176086426,\n",
      "        0.1007584484383529,\n",
      "    ],\n",
      "    'mask': None,\n",
      "    'confidence': 0.9952868223190308,\n",
      "    'index': None,\n",
      "}>\n"
     ]
    }
   ],
   "source": [
    "# Print a prediction from the view to verify that its confidence is > 0.75\n",
    "sample = high_conf_view.first()\n",
    "print(sample.faster_rcnn.detections[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's load our view in the App to view the predictions that we programmatically selected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Load high confidence view in the App\n",
    "session.view = high_conf_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![High Confidence](images/evaluate_detections_high_conf_view.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing object patches\n",
    "\n",
    "There are multiple situations where it can be useful to visualize each object separately. For example, if a sample contains dozens of objects overlapping one another or if you want to look specifically for instances of a class of objects.\n",
    "\n",
    "In any case, the FiftyOne App provides a [patches view button](https://voxel51.com/docs/fiftyone/user_guide/app.html#viewing-object-patches) that allows you to take any `Detections` field in your dataset and visualize each object as an individual patch in the image grid. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Patches View](images/evaluate_detections_gt_patches.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate detections\n",
    "\n",
    "Now that we have samples with ground truth and predicted objects, let's use FiftyOne to evaluate the quality of the detections.\n",
    "\n",
    "FiftyOne provides a powerful [evaluation API](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html) that contains a collection of methods for performing evaluation of model predictions. Since we're working with object detections here, we'll use [detection evaluation](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html#detections)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running evaluation\n",
    "\n",
    "We can run evaluation on our samples via [evaluate_detections()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.evaluate_detections). Note that this method is available on both the `Dataset` and `DatasetView` classes, which means that we can run evaluation on our `high_conf_view` to assess the quality of only the high confidence predictions in our dataset.\n",
    "\n",
    "By default, this method will use the [COCO evaluation protocol](https://cocodataset.org/#detection-eval), plus some extra goodies that we will use later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating detections...\n",
      " 100% |█████████████████| 100/100 [1.8s elapsed, 0s remaining, 53.7 samples/s]         \n",
      "Performing IoU sweep...\n",
      " 100% |█████████████████| 100/100 [882.6ms elapsed, 0s remaining, 113.3 samples/s]      \n"
     ]
    }
   ],
   "source": [
    "# Evaluate the predictions in the `faster_rcnn` field of our `high_conf_view`\n",
    "# with respect to the objects in the `ground_truth` field\n",
    "results = high_conf_view.evaluate_detections(\n",
    "    \"faster_rcnn\",\n",
    "    gt_field=\"ground_truth\",\n",
    "    eval_key=\"eval\",\n",
    "    compute_mAP=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregate results\n",
    "\n",
    "The `results` object returned by the evaluation routine provides a number of convenient methods for analyzing our predictions.\n",
    "\n",
    "For example, let's print a classification report for the top-10 most common classes in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "               precision    recall  f1-score   support\n",
      "\n",
      "       person       0.89      0.80      0.84       263\n",
      "          car       0.72      0.56      0.63        55\n",
      "        chair       0.53      0.23      0.32        35\n",
      "         book       1.00      0.30      0.47        33\n",
      "       bottle       0.60      0.67      0.63         9\n",
      "          cup       0.93      0.81      0.87        16\n",
      " dining table       0.50      0.62      0.55        13\n",
      "traffic light       0.50      0.46      0.48        13\n",
      "         bowl       0.71      0.38      0.50        13\n",
      "      handbag       0.50      0.18      0.26        17\n",
      "\n",
      "    micro avg       0.81      0.64      0.72       467\n",
      "    macro avg       0.69      0.50      0.56       467\n",
      " weighted avg       0.81      0.64      0.70       467\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get the 10 most common classes in the dataset\n",
    "counts = dataset.count_values(\"ground_truth.detections.label\")\n",
    "classes_top10 = sorted(counts, key=counts.get, reverse=True)[:10]\n",
    "\n",
    "# Print a classification report for the top-10 classes\n",
    "results.print_report(classes=classes_top10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also compute the mean average-precision (mAP) of our detector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.3519380509318074\n"
     ]
    }
   ],
   "source": [
    "print(results.mAP())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since [evaluate_detections()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.evaluate_detections) uses the official [COCO evaluation protocol](https://cocodataset.org/#detection-eval), this mAP value will match what `pycocotools` would report.\n",
    "\n",
    "We can also view some precision-recall (PR) curves for specific classes of our model:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install ipywidgets to view the PR curves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install 'ipywidgets>=8,<9'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABCoAAAFoCAYAAACR/RoKAAAgAElEQVR4XuzdB3RVVfr38YeQkJCEQESkCErReQFBRlFB0dERRAcBpTiixGCjWAAdkKKICigCIg4qolgwBNARC4oo1TLiYIEREXVEAvylhWJCCGmE5F3PZm4mgYTcmnvK96zFQpJz9tn78xyQ+2PvfaoVFxcXCwcCCCCAAAIIIIAAAggggAACCCBgAYFqBBUWqAJdQAABBBBAAAEEEEAAAQQQQAABI0BQwYOAAAIIIIAAAggggAACCCCAAAKWESCosEwp6AgCCCCAAAIIIIAAAggggAACCBBU8AwggAACCCCAAAIIIIAAAggggIBlBAgqLFMKOoIAAggggAACCCCAAAIIIIAAAgQVPAMIIIAAAggggAACCCCAAAIIIGAZAYIKy5SCjiCAAAIIIIAAAggggAACCCCAAEEFzwACCCCAAAIIIIAAAggggAACCFhGgKDCMqWgIwgggAACCCCAAAIIIIAAAgggQFDBM4AAAggggAACCCCAAAIIIIAAApYRIKiwTCnoCAIIIIAAAggggAACCCCAAAIIEFTwDCCAAAIIIIAAAggggAACCCCAgGUECCosUwo6ggACCCCAAAIIIIAAAggggAACBBU8AwgggAACCCCAAAIIIIAAAgggYBkBggrLlIKOIIAAAggggAACCCCAAAIIIIAAQQXPAAIIIIAAAggggAACCCCAAAIIWEaAoMIypaAjCCCAAAIIIIAAAggggAACCCBAUMEzgAACCCCAAAIIIIAAAggggAAClhEgqLBMKegIAggggAACCCCAAAIIIIAAAggQVPAMIIAAAggggAACCCCAAAIIIICAZQQIKixTCjqCAAIIIIAAAggggAACCCCAAAIEFTwDCCCAAAIIIIAAAggggAACCCBgGQGCCsuUgo4ggAACCCCAAAIIIIAAAggggABBBc8AAggggAACCCCAAAIIIIAAAghYRoCgwjKloCMIuEvg1VdflZycHLn33nvdNXBGiwACCCCAAAIIIIAAAicVIKjgAUEAgSoVWLt2rbz88svy3XffSa9eveShhx6q0vtzMwQQQAABBBBAAAEEELC2AEGFtetD7xBwnMC+fftkx44dkpKSIqeeeipBheMqzIAQQAABBBBAAAEEEAhMgKAiMD+uRgABPwUmT54sRUVFBBV++nEZAggggAACCCCAAAJOFSCocGplGRcCFheoKKg4fPiwzJ8/v0zv4+LipH///hYfEd1DAAEEEEAAAQQQQACBYAgQVARDkTYQQMBngYqCiuzsbFmwYEGZ9mJjYyUpKUk0xMjPz/f5XlzgToH4+HgpKCgwPzgQ8EagevXqUrt2bW9O5RwEEEAAAQQQCKEAQUUIcWkaAQQqFvBn6QdBBU+ULwIEFb5oca4KEFTwHCCAAAIIIGANAYIKa9SBXiDgGoGjR4+avSmmTJkixcXFMmbMGPPhICIiolIDgopKiTihlABBBY+DrwIEFb6KcT4CCCCAAAKhESCoCI0rrSKAQAUC77zzjjzxxBNlvvvII49Ijx49KjUjqKiUiBMIKngGAhAgqAgAj0sRQAABBBAIogBBRRAxaQoBBEIrQFARWl+ntc6MCqdVNPTjIagIvTF3QAABBBBAwBsBggpvlDgHAQQsIUBQYYky2KYTBBW2KZVlOkpQYZlS0BEEEEAAAZcLEFS4/AFg+AjYSYCgwk7VCn9fCSrCXwO79YCgwm4Vo78IIIAAAk4VIKhwamUZFwIOFCCocGBRQzgkgooQ4jq0aYIKhxaWYSGAAAII2E6AoMJ2JaPDCLhXgKDCvbX3Z+QEFf6oufsaggp315/RI4AAAghYR4Cgwjq1oCcIIFCJAEEFj4gvAgQVvmhxrgoQVPAcIIAAAggEIvDbb79Jdna2tGrVKpBmQnZtcXGxVKtWLSjtFxUVSURERIVtlff9H374QerUqSONGzeutA8EFZUScQICCFhFgKDCKpWwRz8IKuxRJyv1kqDCStWgLwgggIC9BH7//Xe5/vrrZfTo0XLttdeWdH7Pnj1y9913y5lnnikzZswo+Xp6errcdddd5tf6/5+GDRtKnz595PLLLw/qwAsLC+X555+Xt99+27R75ZVXypgxYyQmJqbc+3z11VcyZcqUE7735ptvSlRUlGRmZsrEiRPls88+k8TERElKSpIBAwaY848ePSqvvPKKvPXWW5KRkSEdO3Y09/IEE//4xz9kzpw58s4770itWrVOOk6CiqA+BjSGAAKhFCCoCKWu89omqHBeTUM9IoKKUAvTPgIIIOBcgUcffdQMzvOzZ6Tz5s2TlJQU88FdP6g3b97cfGvHjh0m2Jg0aZKZZfDtt9/K3Llz5ZlnnpFLL700aFCLFi2SmTNnyrPPPiuxsbEmSNEwZPjw4eXe45NPPpFHHnlEpk6dWub7F110kZlBocHD/v37Zdy4cbJz507TjgYwl112mXzxxRfy4IMPGoM//OEPMn36dDlw4IC89tprJozRY+TIkVK/fn154IEHCCqCVmUaQgCBsAoQVISV33Y3J6iwXcnC3mGCirCXgA4ggAAC5sN6zZo1ZevWraJLBc466yy57777zId5PbZt22b+1X779u3ypz/9Sbp16yaNGjUSXXbx4osvyuDBg2Xx4sWiMxzGjx8vn376qfm1fmA+44wz5IYbbpB27drJkSNH5PXXXzcfruPi4kw7npkQ//znP+XHH380X9cP7k2bNpWbbrrJ9KW8Y8uWLXLjjTfKRx99JPXq1Stzit5Pf7z//vsmgBgyZEiZoOK9994rmXFwzz33SJMmTUwYEKzj1ltvlU6dOsnAgQNNkx9++KGZMfH5559XGFQ88cQTsmLFihO+r8tarrjiCpk9e7ZccMEF5vsTJkyQgoICE7jce++9ZuaIJ4TQ2STdu3eXhQsXytlnn11Sv759+8oHH3xgZpFUdDCjIlhPAO0ggEDIBQgqQk7sqBsQVDiqnFUyGIKKKmHmJgggYDGBRZ/tkwNZR6q8V30vP03qJkSecF8NJTQ8uPPOO83yAA0uOnfubD6879u3T/7yl7+Y0EA/fK9cuVI2bNhglhpoqKEfynU5ggYYnvBBlyaMGDFCmjVrJt99951ZvnDHHXfIk08+aT6s639rqKEhh37o1sAiNTXVzGzo2rWrCRdWrVolBw8elJdffrlcp6VLl4r+eO6558p8/+effzZLIz7++GPThl6v/63/v/HMqPAEFbp/xC233GICADU4/tBxvvvuu+Xe/5xzzjFhSHmHWmhg06VLF/PtTZs2maUaunRDjY4/NJjRoEFdatSoYUKd9u3bm70tNCTSkKF0IKNWy5cvN7NG+vXrZ2rlCUW0bR2PzqwovaRF66Q/NPQgqKjy33rcEAEEgi1AUBFsUWe3R1Dh7PqGYnQEFaFQpU0EELC6wM0Tf5RfduRWeTcXPtxazm5cs9yg4o9//KP5IKuHfijWZQsaBOgHff1Z91zQIz8/33xw1uUN+q/9es2yZcukbt265vtffvmlDBs2zMyc0A/zniMvL88EEI899ljJLAoNLnQWhX7g1g/fGpbozAE9PDMm9EN8eXsr6NIK/VDvmS3huc/f//53c61+3xOyaCCiH/w9QYWGKNHR0bJmzRoTHuhYdAbH8Yeev27dunLr1KBBA+nQocMJ39Pw48ILLyyznOTXX381gcKSJUtErzv+8AQitWvXNks7dEaKhkY6Nv2eBhirV6+WhIQEc6n2V5e36KwV3X9Cx6dLP3R5h4YiGt5MmzZN/vznP5fc6qmnnjIzZLTdig5mVFT5b0luiAAC/goQVPgr587rCCrcWfdARk1QEYge1yKAgF0FrDijonRQsXnzZjODQmci6Ad+/eCsH4JLHzoDQT9Ya1Chez2UDiR0NoF+sNb9GXQ2hn441r9TegIOTyigAYhnloUGFWvXri2ZIbF3714z06K8pR16r/vvv18uueSSMrMadBNL3bhS92rQ8eihe1ToLA3d38ETVOiMBf3Qrv3o1atXhW/EyMrKEu1HeYf+nae80EHP1RkVGhxoX/T46aefzMyNikKX49vXPut+FbrJps6o0CUupcOgBQsWmPBIzXTMGlpo4KKHLmNRMw1/WrduXdK0zpJJS0szM1gIKuz6Jwf9RgCBEgGCCh4GXwQIKnzR4lwVIKjgOUAAAQTCL6ChQ+mgQgMK/WCvsyNmzZplPuDrv8gff3iWfpQOKjzn6Af877//3vxrf9u2bc0sC10KoTMedAmJHvo9/QCub6Q4PqjwzIaoKKjQGQO65KT0DAENOnTPBs8bMfQeOkNBl6vobA3dkFI30yy9R8XJ9HXfjPLGrdfoGEaNGlXu5Tp7Qvfe0HBCD72/LqP55ptvvHpVqWdWis74yM3NNW4aNLRp08a0pyGGvsFEl3ccf+h5r776qlkaUvotI+qus0iOn4FS+npmVIT/9yI9QAABLwUIKryE4jQjQFDBg+CrAEGFr2KcjwACCARfQIMK3ZBS3yahG2pqCKCbZepsh/Xr18ugQYPk8ccfN3shHDp0yMyW0KUUnqUfpYMK/XCtX9f9EfSNFfov+LrsQz/w6+wLncmgH9r1lZv6NgxtU0MMX4MKDRs0mNA+eg6dyVFUVGQ2mfQcns0o9byWLVv6FFT4K61BgS7L0A1INSxQV5298fDDD5smdUaE7tXhWeairzFt0aKF6Z/uy/HQQw9JZGRkyfc1XDjllFPMvhe6WWZycrIx1BknutRENy3VnzX0eemll8wrWHVGTOlDAxwNaTz7ZpQ3NoIKfyvOdQggUOUCBBVVTm7rGxJU2Lp8Yek8QUVY2LkpAgggUEZAgwoNJHJycszXdTNG/bB/6qmnml9rKPD000+XfF/f5KH/Qq8fqo9f+vH111+b12F62urYsaMJJHRJgi5j0E0jNQzRQ8MM3bNC//7ga1Dhmc3hWU6h99MlF+W9alRf36kzE/TePXv2NCHC6aefHrKnQP/+rGGDzuLQQ5dgqJ/HU/uos0g8bwHR/UB0Tw/PoUtTJk6caMIiPdRNww6dHaKHjmHs2LFmk1K9l2fTTJ1hoiFF7969y4xNZ5Jcc801ZgNU3eC0ooOgImSPBA0jgECwBQgqgi3q7PYIKpxd31CMjqAiFKq0iQACCPgm4Fn6oXsh6J4H5W1e6fmXe10+UN73S99Rz9W3eugeFfra0+MPXdah7Xg2h/Stt8fO1nvoLAFdDqEfzq146B4X+kpWz0ajJ+ujzjrRQEFtde+P8g51U9Pj3xyiy0B0Y1ENKso7dIaMnlPRMhbPNQQVVnyK6BMCCJQrQFDBg+GLAEGFL1qcqwIEFTwHCCCAQPgFjt+jIvw98q4H//d//2dmD3je6uHdVe46S5fi6GyMDz/88IQNUY+XIKhw17PBaBGwtQBBha3LV+WdJ6iocnLb35CgwvYlZAAIIOAAAd3rQZclnHXWWbYbjS410WUfV1xxhe36XhUd1k01df+R8847r9LbEVRUSsQJCCBgFYF+j/0gI/rWkxYNo6zSJfphYQGCCgsXx6JdI6iwaGHoFgIIIICA6wQIKlxXcgaMgD0FiotFLhyyTuJiIuSpQY0IK+xZxirtNUFFlXI74mYEFY4oI4NAAAEEEHCAAEGFA4rIEBBwi8CWS+qWDLXaFb2kzsipbhk64/RDgKDCDzSXX0JQ4fIHgOEjgAACCFhGgKDCMqWgIwggUJlA6aBCz/3PtY9Ix7uSKruM77tUgKDCpYUPYNgEFQHgcSkCCCCAAAJBFCCoCCImTSGAQGgFdDPNA5m58s7jr0mP7540N1vyx7Fy7u03S7vmMaG9Oa3bToCgwnYlC3uHCSrCXgI6gAACCCCAgBEgqOBBQAAB2wiUfuvHjykLpeE/xpu+/xh3rtSIrCb1EyOl8cNPSUT9xrYZEx0NnQBBRehsndoyQYVTK8u4EEAAAQTsJkBQYbeK0V8EXCxw/OtJM6c9IMWfvVdGZE+b7tLqyRkuVmLoHgGCCp4FXwUIKnwV43wEEEDAngI//fST6N8TmjRpYrkBFBUVSUREhNf9KiwsFP3/V7Vq1by+5mQn5uXlyebNm41NnTp1gtKmP40QVPijxjUIIBAWgeODCu1E4cavTF/+9e0eafP2SPPfUbNWS/wZ1vsfT1jQXHxTggoXF9/PoRNU+AnHZQgggICNBL7++msZOXKkvPLKK3L22WeX9PzTTz+VmTNnSo8ePeS222474ev6hRo1asgf/vAHueWWW8pcG4zhZ2ZmysSJE+Wzzz6TxMRESUpKkgEDBlTY9OzZs+X999+XvXv3mnN69eplxhUdHV3mGm1Xv37FFVeYNo8/Fi1aJAsWLJB58+ZJXFycHD16VB566CHRwOKZZ54JxtD8aoOgwi82LkIAgXAIlBdUlO7HZ0PulXN3LJPfWl0r504L3x+s4bDhnicKEFTwVPgqQFDhqxjnI4AAAvYS0A/fN954owwcOFC6d+9epvPDhg0TnWmhx0cffSSRkZHmvz/88EOZMmWKPP3005KTkyOLFy+Wb775RvQD/mmnnRY0gDFjxsj+/ftl3LhxsnPnThk+fLjMmDFDLrvssnLvMWnSJOnYsaMJTnbt2iWjRo2Su+66S2666aaS8wsKCuTuu++W7777zoQeQ4cOLdPWF198Iffdd5/5mgY1+ncnPXSc1113nWifOnfuHLQx+tIQQYUvWpyLAAJhFagsqNj07RZp9Og1po/MqghrqSxxc4IKS5TBVp0gqLBVuegsAgg4VED/Rf/dd9+Vzz//3HxgbteunfmQrR/GX375ZTODICoqSi666CK59dZbzQyCf/7zn/Lrr7/KueeeK6tWrZLTTz9d+vfvf4LQ22+/LfPnzxf9ufRSCQ0IrrnmGnn++eflnnvuMT936NChJKjQmQUrVqwwvz5y5IhcfPHF8sQTT0jXrl2DUoXs7Gwz40FnSVxwwQWmzQkTJogGDRpIVHaoWe/eveWGG24omTVRXFwsjzzyiOjfn3WsZ555Zpmg4pdffpE777zTfE2DmNJBhd5PA5kXX3xRli5dWtntQ/J9goqQsNIoAgiEQqCyoELvueaeodJ6+8ey5fTLpMaYF0w34mIipEXDqFB0iTYtLEBQYeHiWLRrBBUWLQzdQgCBkApkvfuaFB44tnygKo/avW+T6qecOCNBl1/obAUNJ3SfBP3ArDMFdC+G9evXmxkEOjNCZzjoh3P9sJ2ammqWKWhAoTMQzjjjDPnrX/96wnB0aYV+7/glFW+88YYJL9566y2zTEL/DvHoo4+a63VGRemgQpdSdOnSxdz/T3/60wn30OUY2s/yjmuvvVYuvPDCE761bds26du3r5nJUa9ePfN9HdPy5cslJSWlwrJoaKLX6MyIrVu3yksvvWSWjegxZ84cs4xEv6ahh9p4ZlRo2KNBjo61RYsW0q9fvxOCikOHDsmf//xnWblyZVj2qiCoqMrfjdwLAQQCEvAmqEjfvF1q3N+lwvtEnHa66I/jj+otWktEXIJUi6sl0dfdGlA/udgaAgQV1qiDnXpBUGGnatFXBBAIlsBvAy6Xgs0/BKs5r9tpkvK51DjrnDLnawBx6aWXytixY6VPnz4l39OQQpdi6N8FdXmGftDWf+nX2RTTp083H+o//vhjee2118xsi4oO3aNBP6x7Zkt4ztMP6hoi6N4TOiNj9OjR5kO+7tngWfqh+zZkZGSYQENnMOi+DjExMSfcSpdZbN++vdwutGnTxgQDxx8bNmyQO+64Q1avXi0JCQnm2xrW6L4RGtRUdKjHiBEjZMuWLdK+fXvTbw0qNLzQIEX7qMGHLuHwBBU6S+X22283YYuGPDoTpbygQu+pQZAuRTn//PO9rmuwTiSoCJYk7SCAQMgFvAkqtBNrJ06RmM3fyJm/b/CrTzU695a4+6f4dS0XWUeAoMI6tbBLTwgq7FIp+okAAsEUsNKMCs/MAp3Z0KxZszLD9OyncNZZZ0nLli0lLS1NTjnlFDPbQYOKtWvXynPPPXdSGl1WsXDhwjIbYXo+qP/lL3+R+vXrS25urrz55ptmY0v9mgYVuoRC94PQEEFndOgSCw0xyjs0RMnKyir3e6eeemq5sxM0aNC9M5YtWyZ169Y112rIoGGMjq2yQ4OcwYMHm3FpKKH7SzRv3lxat25tLtXgQv9epBuFapCh5+j5euiyFw1fdJaJhjV6nefQZTBXX3219OzZs7IuBP37BBVBJ6VBBBAIlYC3QUXp+y9fly1PLdon9RMjZd6oJlKUvkOK9u48oYtH036UouxDkr/4NSnOyRbCilBVseraJaioOmun3ImgwimVZBwIIGBXAc+yCp0lcfnll5cZhv6rvy7r0A/Perz++uvy73//26egQmcI6MwInX3gOXQ/Cp1FocscPMe6devMB3sNPo5f+lGZre4zocFAeYeGA926dTvhW55xz507V3TWhR5Tp06V9PR0M2PEm2PatGlmJof2WUOO0mGJLiHRkEX34dCx63g9x4EDB8yeIMnJyaZvGgR5DjW///77T5iB4k1/Aj2HoCJQQa5HAIEqE/AnqNDOJU35TfZmFsrIvvWka/tjuxlXdBSl/SRZY242YUVEs5ZmOcjxR2TbDlKz/7AqGzc38k+AoMI/NzdfRVDh5uozdgQQsIqAZ4NM3T9B96jQD9m6SabOcNAZA/fee6/s2LFDnnzySTM7wZcZFQ8//LDZnFP3g9BDl3Doh3cNP66//voSAn2rh+6LoTMa9L9L71ERKqchQ4aYGSLjx4+XPXv2mOBAZz54gg39ur5lRMevm29qUKMzIBo3bmyWw+gYdFaELiE5/ii99OP471W09CM/P186depUZt+MUI29vHYJKqpSm3shgEBAAv4GFZ5ZFec2i5GnBjWstA+lw4ryTq5+zoWSMGVBpe1wQngFCCrC62/HuxNU2LFq9BkBBJwmoCGELrXQfRv00L0VdNaDfl2DBt0nIjY21ixR8DWo0GUUn3zyibzyyiumbZ05obMcjt8wUpdS6LKP2267zdxDZymE+u0XuuxFX0mqrybVQ5db6F4dnj03dHZD06ZNTUCjfyfWQEJDBs+h5+seFbpvRzCCCnXSOujbV8JxEFSEQ517IoCAXwL+BhXZuUVyy9Tf5HBekUwb2FDaNT9x46PjO1TeEpHiw1mSPelusxln7Vc/9WsMXFR1AgQVVWftlDsRVDilkowDAQScIKCzBvStFp63WOiYNEDYt2+f2SBSN9f09Th48KDZp0GXVeieE1Y8dHwaxFS0B0bpPuvfjXXZiM7EqFmzZtCGo8668agulSnv7SlBu9FJGiKoqApl7oEAAkER8Deo0JunrMyQ1FWZ0rxBlMwe3tjv/mR0P9tcm7hks99tcGHVCBBUVI2zk+5CUOGkajIWBBBAoHwBnRmhMyR06YTnVaBYlRVQH92cVPfM8CcQCoYnQUUwFGkDAQSqRCCQoKL0rIqrzo+XB2449o5qX4+Dt18uRXt3Se2Z70tE81a+Xs75VShAUFGF2A65FUGFQwrJMBBAAIGTCBQXF5tXmereDm3btsXqOAGdyaLLPnQvjzPOOCNsPgQVYaPnxggg4KtAIEGF3mvL7iMy8qVdZgmIv2FF1uib5eimb6TW5FTRTTU5rCtAUGHd2li1ZwQVVq0M/UIAAQQQcJsAQYXbKs54EbCxQKBBRTDCisNPj5KC1e9K7MAHJfq622ys6fyuE1Q4v8bBHiFBRbBFaQ8BBBBAAAH/BAgq/HPjKgQQCINAMIKKQMOK3PkzJW/hsxJz01BeURqGZ8CXWxJU+KLFuSpAUMFzgAACCCCAgDUECCqsUQd6gQACXggEK6jQW3leWar/PbJvPenaPt6LHogcWbvCvPkjqkNniX94tlfXcFJ4BAgqwuNu57sSVNi5evQdAQQQQMBJAgQVTqomY0HA4QLBDCr8DSsKN34lh8YmSfVzLpSEKQscLm7v4RFU2Lt+4eg9QUU41LknAggggAACJwoQVPBUIICAbQSCHVT4E1YUZ2dJZr/2Ui2ultR5c71t7NzYUYIKN1Y9sDETVATmx9UIIIAAAggES4CgIliStIMAAiEXCEVQ4U9YkdH9bDPWxCWbQz5mbuC/AEGF/3ZuvZKgwq2VZ9wIIIAAAlYTIKiwWkXoDwIIVCgQqqBCb5iyMkNSV2Wae+urS2/pkigNEiPL7UvW0B5ydOvPvKLU4s8qQYXFC2TB7hFUWLAodAkBBBBAwJUCBBWuLDuDRsCeAqEMKlRk1gcH5L0vswxOXEyEtGhYo1yopC+Gypm/b5DUS/4u2089z5wTXzNCenWqLe2ax9gT14G9JqhwYFFDPCSCihAD0zwCCCCAAAJeChBUeAnFaQggEH6BUAcVOsI9GYUyb2WGrFifXeGAB+yaJd0OvCdvnZYki+onlznv3GYx0vvS2hJfs/p/w46o8MO5tAcEFS4tfADDJqgIAI9LEUAAAQQQCKIAQUUQMWkKATcLFBUVSUREhNcEhYWFEhlZ/tKKihqpiqDCc28NLNIzCsvtSsLSWVLn4xck85q7JKvb3eacDWm58u6aLDmcV1ShwSWtY82rUHX2BUfoBQgqQm/stDsQVDitoowHAQQQQMCuAgQVdq0c/UbAIgKZmZkyceJE+eyzzyQxMVGSkpJkwIABFfZu8+bNMn/+fFm9erU5v0uXLjJ48GCpUaP8ZRalG6rKoOJkvBW9ojQ7t0jeWXNQNmzJNZcfzi+WtN0FZZrSJSUP3FBPNLTgCK0AQUVofZ3YOkGFE6vKmBBAAAEE7ChAUGHHqtFnBCwkMGbMGNm/f7+MGzdOdu7cKcOHD5cZM2bIZZdddkIvddbF1VdfLVdeeaUMHTrUXDdw4EC599575brrrqt0VFYPKk42AA0xHknZIxu35ZvTNKjo2r4WgUWlVff/BIIK/+3ceiVBhVsrz7gRQAABBKwmQFBhtYrQHwRsJJCdnS1XXOzBsq0AACAASURBVHGFzJ49Wy644ALT8wkTJkhBQYFMmjTphJHk5OTIn/70J5k6daoJK/R49NFHy/x8suFbJajQPvr7itJ31mTJ7CUHSoZZPzFS6teJlBaNakiDxCjp1SnBRk+AtbtKUGHt+lixdwQVVqwKfUIAAQQQcKMAQYUbq86YEQiSwLZt26Rv377y0UcfSb169Uyrqampsnz5cklJSSn3LhpMLFmyREaMGCFnn322jB07Vp588klp3769OV9nXehMi9KH7n1x6qmnigYdGoJY4TjQrYXpxin/+LdUi/ctXNDZFcvWHZK3/5kpezPL7oOhbxqZPvh09rEIQpHj4uLM83LkyJEgtEYTbhDQP2sSEnz7/ewGF8aIAAIIIIBAVQsQVFS1OPdDwEECGzZskDvuuMPsN+H5y/2iRYtk3rx5snjx4nJHumnTJrnrrrukefPm8sMPP5glIpMnT5aYmGOv9dSQ4pZbbilzrYYU2mZxcbH5YYVj1z09JX/Dv6Ths+9JzHmd/O7SLzvy5FBOofyyI1cWrtore34vkIZ1a8i0IS3kD4151anfsCJSrVo1c7lVnplAxsK1VSNw9OhRiYriTT1Vo81dEEAAAQQQqFiAoIKnAwEE/BbYsmWL3HjjjbJs2TKpW7euaWfBggWydOlSM7Pi+MOzVOTvf/+7dOrUSX7++WcZPXq0nHfeeSVLQE7WGSst/cgafbMc3fSNxI+bJVEdr/LbsPSFOtNi5JzdJRtwNm8QJe1a1JRO58SL/jdvC/GNmaUfvnlxtghLP3gKEEAAAQQQsIYAQYU16kAvELClgL7xQ9/aMXfuXGnTpo0Zg+4/kZ6eLtOnTz9hTGvXrjUbZ65atUpq165tvq+BxksvvSSff/55pQZWCipy58+UvIXPSsRpp5sfFR26LCR+3AuVjq30CbM+OCAr1mef8KpTfWPILV0SpTf7WHjlSVDhFRMnlRIgqOBxQAABBBBAwBoCBBXWqAO9QMC2AkOGDJFTTjlFxo8fL3v27JHk5GTRN4F069ZNDh06JPfcc4/5mgYa+lYQfbuHvo701ltvlby8PLnvvvvMXhV6TWWHFYOKyvqs36/9yicSUb+xN6eWOWdDWp6s2XRYtuzKl7Q9R0xwoZtvzhvVxOe23HgBQYUbqx7YmAkqAvPjagQQQAABBIIlQFARLEnaQcClArqhpr6SVEMIPXr27Gk2yNR13gcPHpTOnTubEEI33dRj5cqVsnDhQtm8ebP5tQYYGmZ4lo6cjNFKQUVxdpYc3frTSauek/p3szwk7r4npUaXPgE/IUlTfjObb74wrLG0aMg6+spACSoqE+L7xwsQVPBMIIAAAgggYA0Bggpr1IFeIGB7gX379klsbKzomxa8OXTTzFq1akl0dLQ3p5tzrBRUeNPp/MWvSc6cJ6TGlb0k7m9TvbnkpOfokpD3vsyS6y9JkLt7HNsThKNiAYIKng5fBQgqfBXjfAQQQAABBEIjQFARGldaRQCBEAjYLagoSt8hB+/4s1SLqyV13lwfsMiXP+bIo/PSzcaas4f7vpQk4A7YrAGCCpsVzALdJaiwQBHoAgIIIIAAAvr2tmLe28aDgAACNhGwW1ChrAdvv1yK9u6S2jPfl4jmrQKW7vXYdrNXRcqoJtIgMTLg9pzcAEGFk6sbmrERVITGlVYRQAABBBDwVYCgwlcxzkcAgbAJ2DGoOPz0KClY/a7EDnxQoq+7LWC7aW/tM28EGdK9Lm//qESToCLgx811DRBUuK7kDBgBBBBAwKICBBUWLQzdQgCBEwXsGFQUrHxbDj8zRqI6dJb4h2cHXNbl67LlqUX7pG3TaJk+uFHA7Tm5AYIKJ1c3NGMjqAiNK60igAACCCDgqwBBha9inI8AAmETsGNQ4dmnQtESlxx700kgR3ZukfSesN008c74MyW+ZkQgzTn6WoIKR5c3JIMjqAgJK40igAACCCDgswBBhc9kXIAAAuESsGNQoVZZQ3vI0a0/S63JqRLZtkPAfI+kpMu/fsqRS1rHSotG0dKueU2zwSahRVlagoqAHzXXNUBQ4bqSM2AEEEAAAYsKEFRYtDB0CwEEThSwa1CR89IkyX//danRubfEXNXHDKxavUYSUd+/N3d4ln8cLxQXEyHtmsdIp3Pi5Krz413/CBFUuP4R8BmAoMJnMi5AAAEEEEAgJAIEFSFhpVEEEAiFgF2DiiNrV0j2pLvLkET3HCCxg8b5xaTLP7bsLjA/9vx+RLbsype0PUfM20A8R/3ESDPjIr5mdfMlfUPIxa301+5ZKkJQ4dfj5eqLCCpcXX4GjwACCCBgIQGCCgsVg64ggMDJBewaVBRnZ8mhiUOODS7nkFkGEtGspdR+9oOglnxPRqF8n5YnKSszZG9m4Qlt64wLDS90xkX9xChp0TAqqPe3WmMEFVariPX7Q1Bh/RrRQwQQQAABdwgQVLijzowSAUcI2DWoOB4/o/vZ5kt13lgn1eITQlKbDWl5siEtt6TtDVtyZeO2/BPupXtbNDglyuxzoctFnDTjgqAiJI+WoxslqHB0eRkcAggggICNBAgqbFQsuoqA2wWcElRkjb5Zjm76Jmiba3r7XOiMi+XrDsmWXQWSnlkoabsLylyqocWjyQ3MMhEnHAQVTqhi1Y6BoKJqvbkbAggggAACFQkQVPBsIICAbQScElR4NteMuWmo1Ow/LKz+W3YfkfSMI2a5iAYXujzkqUGNHLEshKAirI+WLW9OUGHLstFpBBBAAAEHChBUOLCoDAkBpwo4JajwbK5Z/ZwLJWHKAkuUSzfofCRlT8nykHObxcgtXRLNW0TsehBU2LVy4es3QUX47LkzAggggAACpQUIKngeEEDANgJOCSp0c83Mfu2lWlwtqfPmekv5z/rggKxYn13yBhGdYdGiYQ1p0aiGXNI6zlbBBUGFpR4tW3SGoMIWZaKTCCCAAAIuECCocEGRGSICThFwSlCh9Th4++VStHeX1J75vkQ0b2WpEunsinfWHJTl67JPeHuIvvY0uUui2XjT6gdBhdUrZL3+EVRYryb0CAEEEEDAnQIEFe6sO6NGwJYCTgoqDj89SgpWvyuxAx+U6Otus2w9dAPO9IxC2bK7QN754mBJcGGHwIKgwrKPlWU7RlBh2dLQMQQQQAABlwkQVLis4AwXATsLOCmoyF/8muTMeUJqXNlL4v421TZl0VkWuvHm3sxC02cNLLq2ryW9Lkmw3KtNCSps81hZpqMEFZYpBR1BAAEEEHC5AEGFyx8Aho+AnQScFFQUpf0kB4f1lIjTTpfar35qpzKYvh4fWOjXdDnIXd3rWiawIKiw3WMV9g4TVIS9BHQAAQQQQAABI0BQwYOAAAK2EXBSUKHoGd3PNvZ13lgn1eITbFOH0h3dkJZnloT866cc82WdYfHoLQ0s8XpTggpbPlJh7TRBRVj5uTkCCCCAAAIlAgQVPAwIIGAbAacFFVmjb5ajm76RiGYtJSLuWFARP+4FW4YWupfFo/PSJW13gRmHvtr0ls51wvpsEVSEld+WNyeosGXZ6DQCCCCAgAMFCCocWFSGhIBTBZwWVOTOnyl5C58tU66Ym4ZKzf7DbFtC3b8idVWm6f+5zWLk0Vvqh20pCEGFbR+jsHWcoCJs9NwYAQQQQACBMgIEFTwQCCBgGwGnBRXF2VlydOtPxr8ofYccfmaMVIurJbVf+dSWsyo8D5IuB9HZFYfziiQuJkJ6X1pb2jWvaf67RcOoKnveCCqqjNoxNyKocEwpGQgCCCCAgM0FCCpsXkC6j4CbBJwWVBxfO89SELvPqtBxZecWySMpe2Tjtvwyw/S8JUQ33myQGBnSx5egIqS8jmycoMKRZWVQCCCAAAI2FCCosGHR6DICbhVwelBRuPErOTQ2yRGzKjzPqL4d5Ndd+bJlV76kZx4tea2pfl+Xhlx9QS25uFVsSJaHEFS49U8K/8dNUOG/HVcigAACCCAQTAGCimBq0hYCCIRUwOlBheI5aVZFeQ+DLgtZvu6QfPljjlka4jmOLQupYX751KCGQXmOCCqCwuiqRggqXFVuBosAAgggYGEBggoLF4euIYBAWQE3BBVOnFVR3nOsS0M0rFj2bdYJy0NeGNY4KHtZEFTwJ4ivAgQVvopxPgIIIIAAAqERIKgIjSutIoBACATcEFQom2dWRXTPARI7aFwIJK3XZOk9LUb2rSdd28cH3EmCioAJXdcAQYXrSs6AEUAAAQQsKkBQYdHC0C0EEDhRwC1BRVHaT3JwWE8DUHvm+xLRvJUrHgfPq02TOteR5C6JAY+ZoCJgQtc1QFDhupIzYAQQQAABiwoQVFi0MHQLAQTcG1ToyHNemiT5778ukW0uklpPznfF46D7VzwwZ7e0bRot0wc3CnjMBBUBE7quAYIK15WcASOAAAIIWFSAoMKihaFbCCDg7qCiODtLDt5+uRTnZEvswIckskVrA1KtXiOJqN/YkY/HnoxCSZ76m+jGmu8+cmbAYySoCJjQdQ0QVLiu5AwYAQQQQMCiAgQVFi0M3UIAAXcHFTr6/MWvSc6cJ8pARHXoLPEPz3bs49Hrse3mbSApo5pIg8TIgMZJUBEQnysvJqhwZdkZNAIIIICABQUIKixYFLqEAALlC7hlj4rSo8+eOESKsrPMl45u+kaqxdWSOm+ud+wjMuLFXeYtINMGNpR2zWMCGidBRUB8rryYoMKVZWfQCCCAAAIWFCCosGBR6BICCBBUlCeQNbSHHN36s9SanCqRbTs48jGZ9cEBee/LLAnGhpoEFY58REI6KIKKkPLSOAIIIIAAAl4LEFR4TcWJCCAQbgE3zqgobe7ZYDPmpqFSs/+wcJcjJPdfvi5bnlq0Ty5uFSuPJdcP6B4EFQHxufJiggpXlp1BI4AAAghYUICgwoJFoUsIIFC+gNuDiiNrV0j2pLslollLqf3sB458TLbsPiJ3zdwhzRtEyezhgW0aSlDhyEckpIMiqAgpL40jgAACCCDgtQBBhddUnIgAAuEWcHtQof4Z3c82ZajzxjqpFp8Q7pKE5P5dx2417S6f3Cyg9gkqAuJz5cUEFa4sO4NGAAEEELCgAEGFBYtClxBAoHwBggqRrNE3m00148fNkqiOVznyURkyc6ek7S4IeENNggpHPh4hHRRBRUh5aRwBBBBAAAGvBQgqvKbiRAQQCLcAQcX/Xlla48peEve3qeEuSUjuP+2tfbJifbaM7FtPuraP9/seBBV+07n2QoIK15aegSOAAAIIWEyAoMJiBaE7CCBQsQBBhUhR2k9ycFhPiTjtdKn96qeOfFxSVmZI6qpMuf6SBLm7R12/x0hQ4Teday8kqHBt6Rk4AggggIDFBAgqLFYQuoMAAgQVlT0DmX89T4pzsqX2K59IRP3ANpys7F7h+P6GtDx5YM5uqZ8YKcldEqV+YlRJN06rU10aJEZ61S2CCq+YOKmUAEEFjwMCCCCAAALWECCosEYd6AUCCHghwIyKY0iHnx4lBavfLRGrFldLqjdrdVLB6i1aS0Tcsc03a3TsIhHNT36+F+UI2SnZuUXSe8L2Ctv3BBj6CtP4mhEVnkdQEbISObZhggrHlpaBIYAAAgjYTICgwmYFo7sIuFmAoOJY9QtWvi2Hnxnj96OgwUbC5PmWDit0+Yce6RmFsuf3IyVjTc88KnszC0t+fUnrWGnXvKZc3Dr2hJkWBBV+PyKuvZCgwrWlZ+AIIIAAAhYTIKiwWEHoDgIIVCxAUFG+TVH6Dinau/Okj86R778y3z+ydoUc3fqzRDRracIKO77idPm6bFn2bZZs3JZfZsw60+LcZjHS6Zw483ODeglSUFBgfnAg4I0AQYU3SpyDAAIIIIBA6AUIKkJvzB0QQCBIAgQVgUMWZ2fJobH9S8KK6Iu7is6wiGzRutLGq7e5qNJzqvKEPRmF8n1anmxIy5Uvf8yRw3lFZW5/9ukx0qlNvHT4fzHSouH/9rmoyj5yL3sJEFTYq170FgEEEEDAuQIEFc6tLSNDwHECBBXBKWnpsMKXFjXQqNGlj8T0HGDJTTy37D5iQos1P2SfMNsiLiZC2jU/Ntuirc628HJDTl98ONf+AgQV9q8hI0AAAQQQcIYAQYUz6sgoEAi7QFFRkUREVLyxYUUd9OU6gorglVlfc5r/rxWmweLDWVK45ceTN55zyMzCKO+IHzdLojpeFbzOBaEl3ZDzl90iX/90UL74IbvMvhbafPMGUdKuRU2zv4Xuc8GBgAoQVPAcIIAAAgggYA0Bggpr1IFeIGBbgczMTJk4caJ89tlnkpiYKElJSTJgwICTjueXX36RF154QX744QfJyMiQRYsWSdOmTSs1IKiolCikJ2i4kfvea2afC309quew6uacpTfT9CwTWbPpsHy/Na/MMhHPW0SuOj8+pH40bn0Bggrr14geIoAAAgi4Q4Cgwh11ZpQIhExgzJgxsn//fhk3bpzs3LlThg8fLjNmzJDLLrus3Htu2LBB7rjjDunSpYv07NlTateuLWeeeaboh8rKDoKKyoSq/vueV6VacXPOk731Y8N/97bQjTk9bxHRwGLeqCZVj8gdLSNAUGGZUtARBBBAAAGXCxBUuPwBYPjOECguLpbc3NwTBhMbG9op7dnZ2XLFFVfI7Nmz5YILLjD3nzBhgnnLwqRJk8rFvfPOO004MX36dJ/xCSp8Jgv5BeVtzum5aczNQ0N+/5PdwNvXk2pY8cKSA2aWxbSBDc1eFhzuFCCocGfdGTUCCCCAgPUECCqsVxN6hIDXAvv27ZPXXntNVqxYYZZQHH+sXr1aEhISvG7P1xO3bdsmffv2lY8++kjq1atnLk9NTZXly5dLSkrKCc0dPHhQOnfuLN27d5eoqCgTrmjAob+OjIys9PYEFZUSheUEDSsO3n55meUg2pHEJZvD0h/PTb0NKvT8R1LS5V8/5cjIvvWka/vKZ/eEdWDcPGQCBBUho6VhBBBAAAEEfBIgqPCJi5MRsJaAzkpYuHChDBo0SBo2bHjCh/2rrrrKqwDA31F5lnGUDkR0v4l58+bJ4sWLT2h28+bNctNNN5lZGB07dpRDhw7J888/b/a0GDr02L++654Xjz32WJlr69SpI4888ogUFhaKbr7JYT2BI5t/kJwvPjYdO/zxG3I0faec9szbEv3HS8LWWQ2/9Hnx5pl5eWm6vPrRXrn9L6fJnd3qh63P3Di8Ajo7LTo6Oryd4O4IIIAAAgggIAQVPAQI2FhAg4jrrrtO7r333rCMYsuWLXLjjTfKsmXLpG7duqYPCxYskKVLl5qZFccfP//8s9lsc+XKlaLhgx4abMycOVM+//xz8+v8/Hz5+uuvy1yqHxwuuugi870jR46EZazc1HuBg1P/Jnkr35aEu8ZLzV53eH9hkM+MiYkx4Zb+qOzQN4OMe22ndDonXh6//fTKTuf7DhWoVq2axMXFOXR0DAsBBBBAAAH7CBBU2KdW9BSBEwSGDRsmjRo1Et3QMhyHzn7QTTHnzp0rbdq0MV2YOnWqpKenl7sHRVZWllx55ZVmWUjr1q3N+e+++648/vjjJpyo7PWmLP0IR5V9v2fu/JmSt/BZie45QGIHjfO9gSBd4cvSD91c84E5u81rS2cPbxykHtCM3QRY+mG3itFfBBBAAAGnChBUOLWyjMsVAmvXrpVRo0bJO++8I6eeempYxjxkyBA55ZRTZPz48bJnzx5JTk42wUm3bt3M0o577rnHfE0DDT109odOyZ88ebIcOHBAHnzwQWnQoIEJOCo7CCoqE7LG9ws3fiWHxiZJ9XMulIQpC8LWKV+CCu1k17FbTV+XT24Wtj5z4/AKEFSE15+7I4AAAggg4BEgqOBZQMDGAhoO6DKLio5Qb6ap99UNNfWVpPpqUj30laNjx441m2V6Ns/U4EI33dRDzxs9erToMhA9dDNNfUOIN0ELQYU9HlbdXDOzX3upFldL6ry5Pmyd9jWo6PXYdvPmj5RRTaRBYuWbu4ZtYNw4ZAIEFSGjpWEEEEAAAQR8EiCo8ImLkxGwlsCnn34qO3bsqLBTf/3rX6VGjRpV0ml9A4m+DtXb9d379+83Mys8e1V400mCCm+UrHFO5l/PM28Bqf3KJxJRPzxLKXwNKka8uEs2bsvnFaXWeITC0guCirCwc1MEEEAAAQROECCo4KFAAAHbCBBU2KZUkjX6Zjm66RupNTlVItt2CEvHfQ0qZn1wQN77MkuGdK8rvTuF7rW+YcHgpl4JEFR4xcRJCCCAAAIIhFyAoCLkxNwAgdAKFBQUmLdu6Ks/c3Nz5fTTTxd9G4j+7LSDoMI+Fc15aZLkv/+6xNw0VGr2HxaWjvsaVKSszJDUVZly/SUJcnePY2+x4XCXAEGFu+rNaBFAAAEErCtAUGHd2tAzBCoV0M0ob7/99pL9IXTpRU5OjrluypQp0rlz50rbsNMJBBX2qVbByrfl8DNjpMaVvSTub5VvlBqKkfkaVHje/NG2abRMH9woFF2iTYsLEFRYvEB0DwEEEEDANQIEFa4pNQN1ooC+1lNnUzz99NPStm1bsx+Fbm45c+ZMWbdunSxfvlxiYmIcM3SCCvuU0vPmj4hmLaX2sx+EpeO+BhV7MgoleepvUj8xUuaNahKWPnPT8AoQVITXn7sjgAACCCDgESCo4FlAwMYC+grQa665RoYNKzu1/pdffpGbb75ZUlJSpHXr1jYeYdmuE1TYq5QZ3c82HU5csjksHfc1qNBO8orSsJTKMjclqLBMKegIAggggIDLBQgqXP4AMHx7C/Tr10/atGkj48aNKzOQb7/9VoYMGUJQYe/y2r73B2+/XIr27pLaM9+XiOatqnw8/gQVQ2bulLTdBbz5o8qrZY0bElRYow70AgEEEEAAAYIKngEEbCzw3HPPydy5c+Whhx6SCy64QGrXri3fffedvPzyy7J792756KOPzCtAnXIwo8JelcyeOESOfLVK4sfNkqiOV1V55/0JKh5JSZd//ZQjj95SXy5pHVvlfeaG4RUgqAivP3dHAAEEEEDAI0BQwbOAgI0F8vLyZOTIkbJ27doyo0hMTJRp06bJH//4RxuP7sSuE1TYq5y582dK3sJnw/bmD3+CCs+bP5I615HkLon2Aqe3AQsQVARMSAMIIIAAAggERYCgIiiMNIJAeAU2bNggv/76a8nrSTt06CD6BhCnHQQV9qrokbUrJHvS3RJx2ukSc92tEtni2H4pEU1bSrX4hJAPxp+gYvm6bHlq0T65uFWsPJZcP+R95AbWEiCosFY96A0CCCCAgHsFCCrcW3tGjoDtBAgq7FWyorSf5OCwnhV2OtRLQvwJKjyvKG3eIEqeGtRI4mtG2Aud3gYkQFAREB8XI4AAAgggEDQBgoqgUdIQAlUjoHtQ6OtIp06davag+P777yu8sb6+1EkzKwgqquYZC+ZdCla+LUfTd0rx4Swp3PKjafropm/MzzrTImHm+yGbXeFPUKH98rz5Q//73GYx0umcOLm4daw0SHTOfi/BrLGT2iKocFI1GQsCCCCAgJ0FCCrsXD367kqBjRs3yvTp02Xy5MkmqNi0aVOFDpMmTZKaNWs6xomgwjGllKyhPeTo1p9Dun+Fv0GFLv9Ys+mw2VSz9KGzLNq1qCld2ydIi4ZRzikGIykRIKjgYUAAAQQQQMAaAgQV1qgDvUAAAS8ECCq8QLLJKYUbv5JDY5NMb2u/8olE1G8c9J77G1R4OpKdWyRf/pgjG9Jyzc+H84pK+lg/MbJktgVvBwl66cLWIEFF2Oi5MQIIIIAAAmUECCp4IBCwscDevXvll19+kfPPP98s8fjtt99k2bJlZhZFnz59JCYmxsajO7HrBBWOKqccfnqUFKx+VyLbXCSxScNLBle9zUVBGWigQcXxndCw4rstx0KLvZmFJd+Oi4mQds3/u0SkVSz7WgSleuFphKAiPO7cFQEEEEAAgeMFCCp4JhCwsYC+gvSLL76Qt99+WwoLC6VHjx6SkZFhRnTdddfJww8/bOPREVQ4qnjlDKY4O0sO3n65FOdkl/lunTfWBWXfimAHFaU7uWX3EVmzKduEFmm7C8r0X2dYtGtek30tbPgAE1TYsGh0GQEEEEDAkQIEFY4sK4Nyi0BycrJcdtllMnDgQFm5cqWMGTNGUlNTJTMzU0aNGiWrV6+WyEjnbADIjArnPdm62WbeirfNwIq2/mRCi1qTUyWybYeABxvKoKJ05/ZkFB5bIrIlt9x9LTq1iZerzo9nM86AKxr6BggqQm/MHRBAAAEEEPBGgKDCGyXOQcCiAr179xYNK66//nqZMWOGrFixQpYuXSp5eXly6aWXmtCiZcuWFu29790iqPDdzE5XZI2+2bwRJFivLa2qoKK0cUX7WujykN6X1pZelySwNMTCDyVBhYWLQ9cQQAABBFwlQFDhqnIzWKcJjBgxQo4ePSr33XefmVWhsyvGjx8v27Ztk759+8qiRYukadOmjhk2QYVjSlnuQHLnz5S8hc8G7U0g4Qgqjh+YzrTQN4isWH9seYtnE84Gp0SZGRYXs6eFpR5qggpLlYPOIIAAAgi4WICgwsXFZ+j2F1i/fr0MGjSoZCCeYOLZZ5+Vt956yywHqVGjhv0H+t8REFQ4ppQnDSqiew6Q2EHjAh6sFYIKzyA2pOXJC0sOnLCfxbSBDc1GnBzWECCosEYd6AUCCCCAAAIEFTwDCNhcYMuWLbJp0yY577zzpEmTJmY0CxculNNOO006d+5s89GV7T5BhaPKecJgPK8srX7OhZIwZUHAg7VSUFE6sEjPKJRl32bJxm35ktS5jiR3SQx4rDQQHAGCiuA40goCCCCAAAKBChBUBCrI9QggUGUCBBVVRh2WG3mCiohmLaX2sx8E3AcrBhWeQaWszJDUVZly/SUJcnePugGPlQaCI0BQERxHWkEAAQQQQCBQAYKKQAW5HoEqFtD9M188hAAAIABJREFUJz788EPp37+/bNy4UX777bcKe9CnTx+Jjo6u4h6G7nYEFaGztUrLGd3PNl1JXLI54C5ZOajQpSAPzNktbZtGy/TBjQIeKw0ER4CgIjiOtIIAAggggECgAgQVgQpyPQJVLLBmzRoZPny42Shz9uzZZh+Kio5PPvlEatWqVcU9DN3tCCpCZ2uVlj1BRZ031km1+ISAumXloGLL7iNy18wd0rxBlMwe3jigcXJx8AQIKoJnSUsIIIAAAggEIkBQEYge1yIQBoGioiLz+tGYmBiJiIgIQw/Cd0uCivDZV9WdPa8orTU5VSLbdgjotlYOKnRgXcduNeNbPrlZQOPk4uAJEFQEz5KWEEAAAQQQCESAoCIQPa5FIMwCv/zyiyxbtkyuv/76ko00tUtz586VunXrSo8ePcLcw+DenqAiuJ5WbM0TVMSPmyVRHa8KqItWDyp6PbZdDucVScqoJuZVpRzhFyCoCH8N6AECCCCAAAIqQFDBc4CAjQUmTJggP/74o6Smpkpk5P8+6PzjH/+QqVOnyhdffGFmXjjlIKhwSiUrHkfu/JmSt/BZiblpqNTsPyygAVs9qBjx4i7z5g9eURpQmYN6MUFFUDlpDAEEEEAAAb8FCCr8puNCBMIv0K9fP7n66qvltttuK9OZ/fv3yzXXXCNvvPGGnHXWWeHvaJB6QFARJEgLN+MJKqJ7DpDYQeMC6qnVg4ppb+2TFeuzZWTfetK1fXxAY+Xi4AgQVATHkVYQQAABBBAIVICgIlBBrkcgjALJycnSqlUrGTt2bJle/Pvf/5aBAweaDTebNm0axh4G99YEFcH1tGJrnleUVj/nQkmYsiCgLlo9qPC8ojSpcx1J7pIY0Fi5ODgCBBXBcaQVBBBAAAEEAhUgqAhUkOsRCKPAjBkzZP78+fLaa69J69atRf+SvW/fPpk4caJ89913smrVKomKigpjD4N7a4KK4HpasTVPUBHRrKXUfvaDgLpo9aBi+bpseWrRPrm4Vaw8llw/oLFycXAECCqC40grCCCAAAIIBCpAUBGoINcjEEaBrKws0eUfe/fuldjYWGnUqJH8+uuvpke6f0W3bt3C2Lvg35qgIvimVmzR84rSxCWbA+qe1YOKDWl58sCc3dK2abRMH9wooLFycXAECCqC40grCCCAAAIIBCpAUBGoINcjEGaB3Nxcs8Rj06ZNkpOTY5Z6/OUvfzFLQpx2EFQ4raLlj8cTVNR5Y51Ui0/we9BWDyqyc4uk94TtEhcTIe8+cqbf4+TC4AkQVATPkpYQQAABBBAIRICgIhA9rkXAQgJ5eXnmzR+l3/5hoe4FpSsEFUFhtHwjnleU1pqcKpFtO/jdX6sHFTqwrmO3mvEtn9zM73FyYfAECCqCZ0lLCCCAAAIIBCJAUBGIHtciEGaBwsJCmTt3rrz55puSkZEhjz32mFx77bVy//33m/0qnnrqqTD3MLi3J6gIrqdVW/MEFfHjZklUx6v87qYdgoohM3dK2u4CXlHqd5WDeyFBRXA9aQ0BBBBAAAF/BQgq/JXjOgQsIPDFF1/IfffdJ9ddd518++23MmjQIBNU6Caao0ePltWrV0tCgv9T5y0wxDJdIKiwWkVC0x/PK0pjbhoqNfsP8/smdggqRry4SzZuyyeo8LvKwb2QoCK4nrSGAAIIIICAvwIEFf7KcR0CFhDQkKJBgwYyZswYuffee83eFBpUpKenm59TU1OlZcuWFuhpcLpAUBEcR6u34gkqonsOkNhB4/zurh2CCl5R6nd5Q3IhQUVIWGkUAQQQQAABnwUIKnwm4wIErCOgMykGDBggvXv3LjeoeOutt6RZM+esfSeosM6zF8qeeF5RWv2cCyVhygK/b2WnoOL6SxLk7h51/R4rFwZHgKAiOI60ggACCCCAQKACBBWBCnI9AmEU0JkU+/fvl5deekmGDRtWMqNi9uzZ8vLLL8uaNWskOjo6jD0M7q0JKoLradXWPEFFtbhaUr3ZsbfXVG/eyufZFXYIKnhFqbWeQoIKa9WD3iCAAAIIuFeAoMK9tWfkDhD49ddfpV+/fnLGGWfIoUOHpF27dqIbbOreFffcc4/cdtttDhjl/4ZAUOGocp50MJ5XlHpO8md2hR2Cii27j8hdM3dI8wZRMnt4Y/cU2KIjJaiwaGHoFgIIIICA6wQIKlxXcgbsNIHNmzfLrFmzZN26dZKTkyNnnXWW3HjjjWaDzYiICEcNl6DCUeU86WB0VoXnODQ2yfxn4pLNPgHYIajQAfGKUp/KGtKTCSpCykvjCCCAAAIIeC1AUOE1FSciYD2Br7/+WrKysqRLly6mc0VFRY4LJ0qrE1RY7xmsih55ZlfUeWOdVIv3/i02dgkqej22XQ7nFUn9xEi5pHWsdG2fIC0aRlUFLfc4ToCggkcCAQQQQAABawgQVFijDvQCAb8EdI8KXfLx/PPP+3W93S4iqLBbxYLT36zRN8vRTd9IrcmpEtm2g9eN2iWoeGdNlrzzxUHZm1lYMjYNLc5tFiOdzokz4QVH1QgQVFSNM3dBAAEEEECgMgGCisqE+D4CFhbQDTM/+OADWbx4sYV7GbyuEVQEz9JOLWVPHCJHvlol8eNmSVTHq7zuul2CCs+AdL+KZd9myZc/5pQJLeJiIkxYQWjhden9PpGgwm86LkQAAQQQQCCoAgQVQeWkMQSqVuD333+X66+/XiZPniydOnWq2puH4W4EFWFAt8Atc+fPlLyFz0rMTUOlZv9hXvfIbkFF6YF5Qovvt+ZJ2u6Ckm95QotbuiRKg8RIry040TsBggrvnDgLAQQQQACBUAsQVIRamPYRCKHA+PHjZenSpRXeYfXq1ZKQ4P2a/hB2NShNE1QEhdF2jRSsfFsOPzNGalzZS+L+NtXr/ts5qCg9yD0ZhWaWxfJ1h0pCi6TOdSS5S6LXFpzonQBBhXdOnIUAAggggECoBQgqQi1M+wiEUODTTz+VHTt2VHiHv/71r1KjRo0Q9qBqmyaoqFpvq9xN3wCib/7w9RWlTgkqStchZWWGpK7KFIKK0DydBBWhcaVVBBBAAAEEfBUgqPBVjPMRQKBcgap44whBhTsfvuLsLMns116qxdWSOm+u9xrBiUHFhrQ8eWDObmnbNFqmD27ktQUneidAUOGdE2chgAACCCAQagGCilAL0z4CIRDIy8uT5557Tr788kszY+Lqq6+WpKQkiYqq+lcaZmZmysSJE+Wzzz6TxMRE048BAwZUOmoNNh5//HHR8OHJJ5+s9Hw9gaDCKyZHnuR5RWniks1ej4+gwmsqTvyvAEEFjwICCCCAAALWECCosEYd6AUCPgmMGzdOPv74Y+nYsaMcOXJE1q1bJ7fccosMHz7cp3aCcbK+InX//v2ifdq5c6fpw4wZM+Syyy47afMatMydO1datmwpqampXnWFoMIrJkee5M8rSp0YVGTnFknvCdtNjZdPbubIWodzUAQV4dTn3ggggAACCPxPgKCCpwEBmwnoDIYuXbrI3/72N7n55ptN71999VWZNWuW6J4V+uGsqo7s7Gy54oorZPbs2XLBBReY206YMEEKCgpk0qRJFXZDX6f64osvSs+ePeWLL74gqKiqgtn4Pv68otSJQYWWsOvYrQQVIXqWCSpCBEuzCCCAAAII+ChAUOEjGKcjEG6BzZs3y0033SRLliyRBg0amO6kp6fLtddeaz7w6wyFqjq2bdsmffv2lY8++kjq1atnbqt9WL58uaSkpJTbja+//lpGjhwpr7/+upkJ8t577xFUVFXBbHwff15R6tSgYsjMnebtH9MGNpR2zWNsXFXrdZ2gwno1oUcIIIAAAu4UIKhwZ90ZtY0FNmzYIHfccYd88sknUqtWLTOS/Px86dSpk5lVcdFFF1XZ6Dx9Kf0a1EWLFsm8efNEZ00cf6Slpcmtt94qTz/9tJmBoeceH1ToWP71r3+VuTQ6OlouvvhiM05d6sLhPoG85W/JwadGSkyXPlJ71NNeAcTExEhhYaH54aRj+PP/JxvScmXGXU3kvLNinTS0sI+lWrVqEhcXF/Z+0AEEEEAAAQTcLkBQ4fYngPHbTsATDnTv3l30A7we+kFMg4FLL71U6tevXzKm++67T2rWrBmyMW7ZskVuvPFGWbZsmdStW9fcZ8GCBbJ06dJyZ0noppk6i6Jr167m3I0bN8p3331n9tfo37+/xMbGimdzztKdrlOnjjz88MNmnLoJJ4f7BPK/+1L23tdHapzbUerPfNcrgMjISPO8OO2ZmbFol7z12QG5/S+nyZ3d/vf73SsUTjqpQHFxccmfq1AhgAACCCCAQPgECCrCZ8+dEfBL4KeffpIRI0ZUeq2+AWT+/Pkh3bPCs1+GborZpk0b06epU6eapSjTp08/oY/6ZhDtv+f44Ycf5D//+Y/06dNHkpOTTVBxsoPNNCstu2NP8OcVpU5d+pGyMkNSV2VKUuc6ktwl0bE1D8fAWPoRDnXuiQACCCCAwIkCBBU8FQggEJDAkCFD5JRTTpHx48fLnj17TOCgbwLp1q2bHDp0SO655x7zNd0A9PijvKUfBBUBlcPRF/v6ilKnBhVf/pgjj85Ll7ZNo2X64EaOrnlVD46goqrFuR8CCCCAAALlCxBU8GQggEBAArqhpr6SVF9Nqoe+yWPs2LGiMzoOHjwonTt3NsGFbrpJUBEQtesvzhraQ45u/VlqTU6VyLYdKvVwalCxIS1PHpizm6Ci0ifA9xMIKnw34woEEEAAAQRCIUBQEQpV2kTAhQL79u0zSzdCuREdSz9c+GCVGnLW6Jvl6KZvJH7cLInqeFWlGE4NKrJzi6T3hO1m/MsnN6vUgRO8FyCo8N6KMxFAAAEEEAilAEFFKHVpGwEEgipAUBFUTts15nlFacRpp4v+0CNu0DiJaN6q3LE4NajQwXYdu5WgIgRPMEFFCFBpEgEEEEAAAT8ECCr8QOMSBBAIjwBBRXjcrXLX/MWvSc6cJ8p0J7rnAIkdNM51QcWQmTslbXeBvDCssbRoGGWVEtm+HwQVti8hA0AAAQQQcIgAQYVDCskwEHCDAEGFG6pc8Rj1zR9Htx57a0xR+g45/MwYqRZXS+q8ud51QcWIF3fJxm35Mm1gQ2nXPMbdD0YQR09QEURMmkIAAQQQQCAAAYKKAPC4FAEEqlaAoKJqva1+N8/mmnH3PSk1uvQ5obtOXvox64MD8t6XWTKke13p3SnB6qWyTf8IKmxTKjqKAAIIIOBwAYIKhxeY4SHgJAGCCidVM/CxeJaCRHXoLPEPz3ZVUJGyMkNSV2VKUuc6ktwlMXBMWjACBBU8CAgggAACCFhDgKDCGnWgFwgg4IUAQYUXSC46RZeCZPZrb0Zc+5VPJKJ+4zKjd/KMii9/zJFH56XLxa1i5bHk+i6qemiHSlARWl9aRwABBBBAwFsBggpvpTgPAQTCLkBQEfYSWK4Dh58eJQWr35WYm4ZKzf7DXBNUbEjLkwfm7Ja2TaNl+uBGlquLXTtEUGHXytFvBBBAAAGnCRBUOK2ijAcBBwsQVDi4uH4O7cjaFZI96W7zutLoUvtUxNw8VJw8oyI7t0h6T9gucTER8u4jZ/qpx2XHCxBU8EwggAACCCBgDQGCCmvUgV4ggIAXAgQVXiC58JSDt18uRXt3lRl54pLNjg4qdLBdx241Y14+uZkLqx6aIRNUhMaVVhFAAAEEEPBVgKDCVzHORwCBsAkQVISN3tI31lkVhVuOvba0YNXbJrTQPSsSWrSUgoIC88OJx5CZOyVtd4Fc0jpWWjSKlviYCPOzHuc2O/Yzh28CBBW+eXE2AggggAACoRIgqAiVLO0igEDQBQgqgk7quAazRt8sRzd9I7Ump0qdizs7OqgY8eIu2bgt/6Q1PLdZjPl+uxY1zc9XnR8vDRIjHVf3YA2IoCJYkrSDAAIIIIBAYAIEFYH5cTUCCFShAEFFFWLb9FZuCip0n4otuwvkcF6R/LorX8yvd+XL4fxiM9OivIPXmZ78wSaosOlvfLqNAAIIIOA4AYIKx5WUASHgXAGCCufWNlgjy3lpkuS//7rEDnxQ6vYf6ugZFZWZlQ4y1mw6LCvWZ5sZFQ/cUK+yS137fYIK15aegSOAAAIIWEyAoMJiBaE7CCBQsQBBBU9HZQK582dK3sJnzetK6w1+0NVBRWkrXmda2ZNz7PsEFd45cRYCCCCAAAKhFiCoCLUw7SOAQNAECCqCRunYhggqyi/tlt1H5K6ZO6R+YqTMG9XEsfUPdGAEFYEKcj0CCCCAAALBESCoCI4jrSCAQBUIEFRUAbLNb1G48Ss5NDZJqp9zoTR6/n1mVJSqJ68zrfzhJqio3IgzEEAAAQQQqAoBgoqqUOYeCCAQFAGCiqAwOroRgoqKy9vrse1m482UUU1480cFTAQVjv7jgcEhgAACCNhIgKDCRsWiqwi4XYCgwu1PQOXjL0r7SQ4O6ykRp50up//jW2ZUlCLzvM502sCG0q75sdeWcpQVIKjgiUAAAQQQQMAaAgQV1qgDvUAAAS8ECCq8QOIUyeh+tlFo8ulugopSz8MjKenyr59y5NFb6sslrWN5UsoRIKjgsUAAAQQQQMAaAgQV1qgDvUAAAS8ECCq8QOIUgooKnoGUlRmSuipTkjrXkeQuiTwpBBU8AwgggAACCFhWgKDCsqWhYwggcLwAQQXPhDcCB2+/XIr27pIGc1ZI8Zl/MLMqOEQ8QcX1lyTI3T3qQkJQwTOAAAIIIICAZQUIKixbGjqGAAIEFTwD/ghkjb5Zjm76Rk575m2p1voCgor/Im5Iy5MH5uyWtk2jZfrgRv7QOv4aln44vsQMEAEEEEDAJgIEFTYpFN1EAAERZlTwFHgjQFBRvpInqGjeIEpmD2/sDaXrziGocF3JGTACCCCAgEUFCCosWhi6hQACJwoQVPBUeCOQO3+m5C18VhIGjJCa/Ycxo6IUWtexW82vlk9u5g2l684hqHBdyRkwAggggIBFBQgqLFoYuoUAAgQVPAP+CRBUVOxGUHHyZ4qgwr/fc1yFAAIIIIBAsAUIKoItSnsIIBAyAWZUhIzWUQ3nL35NcuY8IfF97pS4wQ8zo6JUdUe8uEs2bsuXaQMbSrvmMY6qezAGQ1ARDEXaQAABBBBAIHABgorADWkBAQSqSICgooqgbX6bwo1fyaGxSVLj3I6S+NSbBBUEFV4/0QQVXlNxIgIIIIAAAiEVIKgIKS+NI4BAMAUIKoKp6dy2CCoqru2sDw7Ie19mSVLnOpLcJdG5D4GfIyOo8BOOyxBAAAEEEAiyAEFFkEFpDgEEQidAUBE6Wye1XJS+Qw7e8WepXr+xnDpvDTMqShU3ZWWGpK7KJKio4IEnqHDSnwSMBQEEEEDAzgIEFXauHn1HwGUCBBUuK3gAw83ofra5uv7y7QQVpRy//DFHHp2XLm2bRsv0wY0CEHbmpQQVzqwro0IAAQQQsJ8AQYX9akaPEXCtAEGFa0vv88AJKson25CWJw/M2U1QUcETRVDh8281LkAAAQQQQCAkAgQVIWGlUQQQCIUAQUUoVJ3ZZtbQHnJ068+SOO0NkVbtnTlIP0aVnVskvSdsl7iYCHn3kTP9aMHZlxBUOLu+jA4BBBBAwD4CBBX2qRU9RcD1AgQVrn8EvAbIGn2zHN30DUFFOWJdx241X10+uZnXnm45kaDCLZVmnAgggAACVhcgqLB6hegfAgiUCBBU8DB4K+AJKuo8+pJUu+DP3l7mivOSpvwmezML5YVhjaVFwyhXjNnbQRJUeCvFeQgggAACCIRWgKAitL60jgACQRQgqAgipsObyp0/U/IWPitxSfdJjX73OHy0vg1vxIu7ZOO2fJk2sKG0ax7j28UOP5ugwuEFZngIIIAAArYRIKiwTanoKAIIEFTwDHgrQFBRsdS0t/bJivXZMqR7XendKcFbUlecR1DhijIzSAQQQAABGwgQVNigSHQRAQSOCRBU8CR4K1Cw8m05/MwYienSR2re96S3l7nivJSVGZK6KlOSOteR5C6Jrhizt4MkqPBWivMQQAABBBAIrQBBRWh9aR0BBIIoQFARREyHN1W48Ss5NDZJotp2kPjJqQ4frW/DW74uW55atE+uOj9eHrihnm8XO/xsggqHF5jhIYAAAgjYRoCgwjaloqMIIEBQwTPgrQBBRcVSG9Ly5IE5u6Vt02iZPriRt6SuOI+gwhVlZpAIIIAAAjYQIKiwQZHoIgIIHBMgqOBJ8FagODtLMvu1l2pxtaTOm+u9vcwV523ZfUTumrlD6idGyrxRTVwxZm8HSVDhrRTnIYAAAgggEFoBgorQ+tI6AggEUYCgIoiYLmgqo/vZZpSJSza7YLS+DbHr2K3mguWTm/l2ocPPJqhweIEZHgIIIICAbQQIKmxTKjqKAAIEFTwDvggQVFSs1eux7XI4r0jeGX+mxNeM8IXV0ecSVDi6vAwOAQQQQMBGAgQVNioWXUXA7QIEFW5/AnwbP0FFxV4jXtwlG7fly7SBDaVd8xjfYB18NkGFg4vL0BBAAAEEbCVAUGGrctFZBNwtQFDh7vr7OvqsO66Qo+k7pfYrn0hE/ca+Xu7o8z1BxaO31JdLWsc6eqy+DI6gwhctzkUAAQQQQCB0AgQVobOlZQQQCLIAQUWQQR3e3OEHb5GC79dKrcmpEtm2g8NH69vwUlZmSOqqTEnqXEeSuyT6drGDzyaocHBxGRoCCCCAgK0ECCpsVS46i4C7BQgq3F1/X0dPUFGxGEFF+TYEFb7+LuN8BBBAAAEEQiNAUBEaV1pFwHUCRUVFEhHh/aZ8R48eFf1Q4MtBUOGLFucSVFT8DGxIy5MH5uyWtk2jZfrgRjws/xUgqOBRQAABBBBAwBoCBBXWqAO9QMC2ApmZmTJx4kT57LPPJDExUZKSkmTAgAEVjmfatGmyfPlyycjIkHbt2sm1114rvXv39mr8BBVeMXHSfwXyJt8ruWuWSfy4WRLV8SpcSgkQVJT/OBBU8NsEAQQQQAABawgQVFijDvQCAdsKjBkzRvbv3y/jxo2TnTt3yvDhw2XGjBly2WWXlTsmPe+qq66SJk2ayNq1a+Xpp5+WRYsWSdOmTSs1IKiolIgTSgkcfWu2ZL0+XWJuGio1+w/DppRAdm6R9J6w3Xxl+eRm2PxXgKCCRwEBBBBAAAFrCBBUWKMO9AIBWwpkZ2fLFVdcIbNnz5YLLrjAjGHChAlSUFAgkyZN8mpMGlpUNgvD0xBBhVeknPRfAYKKkz8KXcduJag4joiggj8+EEAAAQQQsIYAQYU16kAvELClwLZt26Rv377y0UcfSb169cwYUlNTzdKOlJSUSseUnp5uln5MmTJFOnfuXOn5BBWVEnFCKQGCipM/DkNm7pS03QXywrDG0qJhFM+OiNk3p3bt2lgggAACCCCAQJgFCCrCXABuj4CdBTZs2CB33HGHrF69WhISEsxQdBnHvHnzZPHixScdWl5entx7772iP8+dO1ciIyPN+bqMpH///mWu1RBEA5Di4mLzgwMBbwQOvfWi7P/7OEm4YZDUHf64N5e46pzB03+Rf/96WF64/2xp/4d4V429osHqpsCeP4sAQQABBBBAAIHwCRBUhM+eOyNge4EtW7bIjTfeKMuWLZO6deua8SxYsECWLl1qgoWKjiNHjshDDz0k//nPf+Tll18umY2h5+sHhd9//73MpfqvnLpRZ05OjllWwoGANwKRmzdI+vDeEtnmIqk9daE3l7jqnOff3y/vrjkod/U4VfpcyiwCLb6+ucgTurrqYWCwCCCAAAIIWEyAoMJiBaE7CNhJQN/40aVLFzMjok2bNqbrU6dOFV3SMX369HKHovtaeDbgfO655+TUU0/1esgs/fCaihNFJOrX72XvfX2k+jkXSsKUBZgcJ5CyMkNSV2VKUuc6ktwlER+WfvAMIIAAAgggYBkBggrLlIKOIGBPgSFDhsgpp5wi48ePlz179khycrIJIrp16yaHDh2Se+65x3xNA43c3Fy57bbbzKwI3ZciPv7YdHOdMXHaaadVCkBQUSkRJ5QSIKg4+eOwfF22PLVon1zcKlYeS67Ps0NQwTOAAAIIIICAZQQIKixTCjqCgD0FdENNfSWpvppUj549e8rYsWMlKipKDh48aDbJ1OBCN93cu3evCTCOP3RZx4oVKyoFIKiolIgTCCq8fgY2pOXJA3N2S9um0TJ9cCOvr3PyiWym6eTqMjYEEEAAATsJEFTYqVr0FQELC+zbt09iY2MlLi4uZL0kqAgZrSMb1hk7v13R0IwtcclmR44xkEHtySiU5Km/Sf3ESJk3qkkgTTnmWoIKx5SSgSCAAAII2FyAoMLmBaT7CLhJgKDCTdUOfKwEFZUbdh271ZyUMqqJNEg89uYdNx8EFW6uPmNHAAEEELCSAEGFlapBXxBA4KQCBBU8IL4IEFRUrtXrse1yOK+ozInLJzer/EKHnkFQ4dDCMiwEEEAAAdsJEFTYrmR0GAH3ChBUuLf2/oycoKJytVkfHJAtu/IlPfOo7M0sNBcQVPCq1sqfHM5AAAEEEEAgtAIEFaH1pXUEEAiiAEFFEDFd0JQGFXuGXi9HNn4ltSanSmTbDi4Ytf9DHDJzp6TtLpAXhjWWFg2j/G/Ixlcyo8LGxaPrCCCAAAKOEiCocFQ5GQwCzhYgqHB2fYM9OoIK30RHvLhLNm7Ll2kDG0q75jG+XeyQswkqHFJIhoEAAgggYHsBggrbl5ABIOAeAYIK99Q6GCMlqPBNcdpb+2TF+mwZ2beedG0f79vFDjmboMIhhWQYCCCAAAK2FyCosH0JGQAC7hEgqHBPrYMxUoIK3xRTVmZI6qpMSepcR5K7JPp2sUPOJqhwSCEZBgIIIICA7QUIKmxfQgaAgHsECCrcU+tgjFSDiv3PPCi5770msQMflOjrbgtGs45t4501WTJ7yQG56vx4eeATJq4mAAAXJklEQVSGeo4d58kGRlDhyrIzaAQQQAABCwoQVFiwKHQJAQTKFyCo4MnwRUCDit9fniKHU5+RmJuGSs3+w3y53HXnbkjLkwfm7Ja2TaNl+uBGrhu/DpigwpVlZ9AIIIAAAhYUIKiwYFHoEgIIEFTwDAQuQFDhm6EnqGjeIEpmD2/s28UOOZugwiGFZBgIIIAAArYXIKiwfQkZAALuEWBGhXtqHYyRElT4rth17FZz0fLJzXy/2AFXEFQ4oIgMAQEEEEDAEQIEFY4oI4NAwB0CBBXuqHOwRqlBRdaS+XLwqZFS48peEve3qcFq2rHtEFRUl9q1azu2vgwMAQQQQAABuwgQVNilUvQTAQSEoIKHwBcBDSqyv/lMMh7oJ9XPuVASpizw5XJXnjvixV3y/9u7F+AqqjyP4z8CMSEJMSESREUeOpYrsLgrrs+UzBKhRGQniguOmSCFDFkLQUQcIhEcoHhJFhUcoUDEEBgfLDOUFBQpYBcfs1rqFhSFrmIwaniEYCVggCSQsHWaCeLAmnsvfW+6T3+7irIm6T59/p//car42X16V3m9nh/TRX17JgbOgCcqAtdyCkYAAQQQ8KgAQYVHG8O0EEDgfAGCClZFOAIEFeFonTmXoIInKsJfNVyBAAIIIICA+wIEFe6bMiICCERJgKAiSrCWDktQEX5j//DO9/rzX44qd0Ca8rLTwx/A51fwRIXPG8j0EUAAAQSsESCosKaVFIKA/QIEFfb32M0KTVBR991eVf3mDsVlXqlLV/yXm8NbOVbxlmqVbK0hqLCyuxSFAAIIIICAfwQIKvzTK2aKQOAFCCoCvwTCAjBBRUNDgyoHdnOuS9+wJ6zrg3hy6ae1WrC2Srf9XZJ+n9c5cAQ8URG4llMwAggggIBHBQgqPNoYpoUAAucLEFSwKsIRIKgIR+vMuTv31mnysgPq0z1BRWOvCH8An19BUOHzBjJ9BBBAAAFrBAgqrGklhSBgvwBBhf09drNCgorwNQ9Wn1Le/O/UOb2dVj3dNfwBfH4FQYXPG8j0EUAAAQSsESCosKaVFIKA/QIEFfb32M0Km4OKQ7/qpdPHa5X2xqdqk5Lq5i2sHGtgwddOXaVzelhZ388VRVARuJZTMAIIIICARwUIKjzaGKaFAALnCxBUsCrCEWgOKg5PHKbG3R+rw5wStetzSzhDBPLcnN9/o2N1TSp+uqsuT28XKAOCikC1m2IRQAABBDwsQFDh4eYwNQQQ+KkAQQUrIhwBgopwtH48d9LS/dpVXq/nx3RR356JkQ3i06sIKnzaOKaNAAIIIGCdAEGFdS2lIATsFSCosLe30aiMoCIy1eag4rnfdNbtNyRFNohPryKo8GnjmDYCCCCAgHUCBBXWtZSCELBXgKDC3t5GozKCishUi7dUq2RrjXIHpCkvOz2yQXx6FUGFTxvHtBFAAAEErBMgqLCupRSEgL0CBBX29jYalTUHFUdeW6C6Py5S4kOPq/3D46NxK6vGJKi41Kp+UgwCCCCAAAJ+FCCo8GPXmDMCARUgqAho4yMsm6AiMride+s0edkB9emeoKKxV0Q2iE+v4okKnzaOaSOAAAIIWCdAUGFdSykIAXsFCCrs7W00KiOoiEyVoIInKiJbOVyFAAIIIICAewIEFe5ZMhICCERZgKAiysCWDU9QEVlDa0806f4Z3yg5MU5/mt4tskF8ehVPVPi0cUwbAQQQQMA6AYIK61pKQQjYK0BQYW9vo1FZc1Bx/NP39ENB7nm3iMu8UuZPm5RUtet5g/P7+L+/xfln297/FI0p+WbMgQVfO3MtndPDN3N2Y6IEFW4oMgYCCCCAAAIXL0BQcfGGjIAAAjESIKiIEbQlt2kpqAilzLge1yu+zy1K+m1hKKdbc07uvO90qOaUXhl/la7pEm9NXS0VQlDRkhC/RwABBBBAIDYCBBWxceYuCCDgggBBhQuIARqiOahoaGi4YNVNlRVqOrRP5p+Nlft0+thRnSr7zDm3cffHZ69p2+tmpc5bEyA5adLS/dpVXq/nx3RR356JgamdoCIwraZQBBBAAAGPCxBUeLxBTA8BBH4UIKhgNYQj0FJQEcpY1UN+4ZyWvmFPKKdbc8704kr99+fH9dSwThp4U4o1dbVUCEFFS0L8HgEEEEAAgdgIEFTExpm7IICACwIEFS4gBmgIN4OKtDc+dfayCMpRvKVaJVtrlDsgTXnZ6UEpWwQVgWk1hSKAAAIIeFyAoMLjDWJ6CCDwowBBBashHAE3goqjv/u18xpIhzklatfnzEabQTjWfXBUSzZ8r1/dnqrH7ssIQslOjQQVgWk1hSKAAAIIeFyAoMLjDWJ6CCBAUMEaiEzAjaCidma+Tn60VclPzNUl2Q9ENhEfXrVzb50mLzugPt0TVDT2Ch9WENmUCSoic+MqBBBAAAEE3BYgqHBblPEQQCBqAjxRETVaKwd2I6g4sfol1f1xkRIfelztHx5vpdOFimoOKnpeHq8lE64KTN0EFYFpNYUigAACCHhcgKDC4w1ieggg8KMAQQWrIRwBN4KKhi3/oWMvTNEl/5yj5Cfnh3N73587sOBrp4bSOT18X0uoBRBUhCrFeQgggAACCERXgKAiur6MjgACLgoQVLiIGYCh3AgqTu36SD8U5CqInyglqAjAvySUiAACCCCAgEcFCCo82himhQAC5wsQVLAqwhFwI6hoqqzQkdG/VJvkDkp783/Cub3vz81/aZ/2HmjQ82O6qG/PRN/XE0oBPFERihLnIIAAAgggEH0BgoroG3MHBBBwSYCgwiXIgAzjRlBhqKqH/MIRS9+wJyByZ8qctHS/dpXXE1QEqusUiwACCCCAgDcECCq80QdmgQACIQgQVISAxClnBdwKKo4+fp8av/7fwH2i9A/vfK8//+Wo8odk6P47UgOxsniiIhBtpkgEEEAAAR8IEFT4oElMEQEEzggQVLASwhFwLaj43a/VuPvjwAUVxVuqVbK1RrkD0pSXnR4OvW/PJajwbeuYOAIIIICAZQIEFZY1lHIQsFmAoMLm7rpfm1tBRVA/UVr6aa0WrK3S3f+YoskPdnK/QR4ckaDCg01hSggggAACgRQgqAhk2ykaAX8KEFT4s2+tNWu3g4qEoSOV9NvC1ion5vfdubdOk5cdUJ/uCSoae0XM798aNySoaA117okAAggggMD5AgQVrAoEEPCNAEGFb1rliYm6FVQE9ROlZQdO6t9eqlDn9HZa9XRXT/Q02pMgqIi2MOMjgAACCCAQmgBBRWhOnIUAAi4LNDU1KS4uLqxRCSrC4gr8yW4FFU17P9eR8UMV1+N6XbronUC5Diz42qm3dE6PQNRNUBGINlMkAggggIAPBAgqfNAkpoiATQI1NTWaOXOmtm/frvT0dOXm5mrkyJEhlUhQERITJ/1VwK2gwgwX1E+UNgcV66Z1U0r78IJFPy5Eggo/do05I4AAAgjYKEBQYWNXqQkBDwtMmTJFhw8fVmFhofbt26cJEyZo4cKFysrKanHWBBUtEnHCOQJuBhU1//oPOn28Vpe++p+K63xVYJwnLd2vXeX1en5MF/XtmWh93QQV1reYAhFAAAEEfCJAUOGTRjFNBGwQqK2tVf/+/bVkyRL169fPKWnGjBlqaGjQrFmzWiyRoKJFIk6IUlBxNKCfKCWo4F8pBBBAAAEEEGgNAYKK1lDnnggEVKC8vFzDhg3Tpk2b1KnTmc8dlpSUqLS0VMXFxS2qEFS0SMQJUQoqjv3702rY9icljXlGCf8yKjDOxVuqVbK1RrkD0pSXnW593TxRYX2LKRABBBBAwCcCBBU+aRTTRMAGgZ07d2r06NHatm2bUlNTnZLWrl2rVatWaf369c7/NmHE6tWrf1JucnKyHn74YZ08eVKnTp2ygYIaYiBwySWXqLGx0flzscfRlUWqXbXwYofheo8LlGU9poHzZnp8lkwPAQQQQAAB+wUIKuzvMRUi4BmBsrIyDR8+XJs3b1ZGRoYzrzVr1mjjxo3OkxXmMK+HmJ+deyQlJTmbbnIggAACCCCAAAIIIICA/QIEFfb3mAoR8IyA+eJHdna2Vq5cqd69ezvzmj9/viorK1VUVPSz8zR7WPTq1Us5OTmeqYeJeFvgmWee0V133aVBgwZ5e6LMzjMCEydO1P333x/S5r6emTQTQQABBBBAwEIBggoLm0pJCHhZID8/Xx07dtS0adN08OBB5eXlyXwJZPDgwQQVXm6cD+dGUOHDprXylAkqWrkB3B4BBBBAAIG/ChBUsBQQQCCmAmZDTfNJUvNpUnMMHTpUBQUFio+PJ6iIaSfsvxlBhf09drtCggq3RRkPAQQQQACByAQIKiJz4yoEELhIgaqqKpm9J8xGmaEcvPoRihLnnCtAUMF6CFeAoCJcMc5HAAEEEEAgOgIEFdFxZVQEEHBZgKDCZdAADEdQEYAmu1wiQYXLoAyHAAIIIIBAhAIEFRHCcRkCCCCAAAIIIIAAAggggAACCLgvQFDhvikjIoAAAggggAACCCCAAAIIIIBAhAIEFRHCcRkCCCCAAAIIIIAAAggggAACCLgvQFDhvikjIoAAAggggAACCCCAAAIIIIBAhAIEFRHCcRkCCMRWoKmpSXFxcbG9KXfzhQBrwxdtYpIIIIAAAggggEDIAgQVIVNxIgIItIZATU2NZs6cqe3btys9PV25ubkaOXJka0yFe3pMINy1sWLFCm3YsOEnVfTq1ctZXxwIGAGzRo4fP65x48YBggACCCCAAAKtKEBQ0Yr43BoBBFoWmDJlig4fPqzCwkLt27dPEyZM0MKFC5WVldXyxZxhtUC4a6OoqEhffvmlRo0addYlJSVFvXv3ttqJ4loW+PDDD7V8+XLt2LFDOTk5mjp1assXcQYCCCCAAAIIRE2AoCJqtAyMAAIXK1BbW6v+/ftryZIl6tevnzPcjBkz1NDQoFmzZl3s8FzvY4FI1oYJKo4cOeKsIQ4EzhWoqqpSRUWFiouLddlllxFUsDwQQAABBBBoZQGCilZuALdHAIH/X6C8vFzDhg3Tpk2b1KlTJ+fEkpISlZaWOn+h4AiuQCRrwwQV5r+cm/ArNTXVeSqne/fuwUWk8vME5syZI7PnCU9UsDgQQAABBBBoXQGCitb15+4IIPAzAjt37tTo0aO1bds25y+W5li7dq1WrVql9evXYxdggUjWxrp16/TFF18oISFBu3fvlhlj8eLFuvXWWwMsSennChBUsB4QQAABBBDwhgBBhTf6wCwQQOACAmVlZRo+fLg2b96sjIwM54w1a9Zo48aNzpMVHMEVcGNtmD0u6uvrnT1POBAwAgQVrAMEEEAAAQS8IUBQ4Y0+MAsEELiAgPmqQ3Z2tlauXHl2w8P58+ersrJS5jF+juAKuLE2zNMUZvNEs4kiBwIEFawBBBBAAAEEvCNAUOGdXjATBBC4gEB+fr46duyoadOm6eDBg8rLy5P5L+GDBw/GK+ACLa0Ns2YyMzPPfmrSbMo6aNAgde3aVXv27NH48eM1YsQI5/UijmALNDY2OntTzJs3T6dPn3b+P6Zt27aKi4sLNgzVI4AAAggg0EoCBBWtBM9tEUAgNAGzaaL5JKn5NKk5hg4dqoKCAsXHx4c2AGdZK9DS2jAhhNksc+7cuY6BCbk+++yzsx5Dhgxx/kKamJhorRGFhSZg9i+ZPXv2T06ePn267rvvvtAG4CwEEEAAAQQQcFWAoMJVTgZDAIFoCZjPByYlJSk5OTlat2BcnwqEszbMZ02rq6udr8gQUPi04UwbAQQQQAABBKwXIKiwvsUUiAACCCCAAAIIIIAAAggggIB/BAgq/NMrZooAAggggAACCCCAAAIIIICA9QIEFda3mAIRQAABBBBAAAEEEEAAAQQQ8I8AQYV/esVMEUAAAQQQQAABBBBAAAEEELBegKDC+hZTIAIIIIAAAggggAACCCCAAAL+ESCo8E+vmCkCCCCAAAIIIIAAAggggAAC1gsQVFjfYgpEAAEEEEAAAQQQQAABBBBAwD8CBBX+6RUzRQABBBBAAAEEEEAAAQQQQMB6AYIK61tMgQgggAACCCCAAAIIIIAAAgj4R4Cgwj+9YqYIIIAAAggggAACCCCAAAIIWC9AUGF9iykQAQQQQAABBBBAAAEEEEAAAf8IEFT4p1fMFAEEEEAAAQQQQAABBBBAAAHrBQgqrG8xBSKAAAIIIIAAAggggAACCCDgHwGCCv/0ipkigAAC1gtUVFRo7969Tp1t2rRR+/btdd111yk1NTWmtc+dO1edO3fWqFGjVF9fr8cee0yjR4/W7bffHtN5cDMEEEAAAQQQQCCIAgQVQew6NSOAAAIeFXjjjTe0YMGC82Y3efJkDR8+PGazfuSRR3TNNdfo2Wef1YkTJ5SVlaUZM2Zo8ODBMZsDN0IAAQQQQAABBIIqQFAR1M5TNwIIIOBBgeag4v3331dCQoIOHTok83TDe++9p/Xr1+vKK6+MyawJKmLCzE0QQAABBBBAAIELChBUsDAQQAABBDwjcG5QkZiY6Mzrk08+UX5+vl588UXdcccdzs9++OEHvfzyy3r33XedMKNfv3568sknnddEmo+PPvpIK1as0Oeff6709HTddtttzjjmVY6pU6fqm2++UXV1tTIzM3Xvvfdq7NixateunXM5QYVnlgQTQQABBBBAAIEAChBUBLDplIwAAgh4VeBCQcW6des0e/ZslZaWqmPHjmpsbHT2jjhy5IjzOkiHDh309ttvq7y8XBs3blRKSorzBMbEiRN17bXX6sEHH3SCjdWrV2vhwoVKS0vT4sWLdfPNNzvjffXVV1q6dKnGjRvnBBQEFV5dHcwLAQQQQAABBIIiQFARlE5TJwIIIOADgeagwrzuYY6ysjItW7ZMQ4YM0XPPPef8bPv27Zo0aZJee+019enTx/nZnj179NBDDzn7W/Tv398JJxoaGpzXRZqP48eP6/Tp00pOTj77s2PHjjlPVUyfPt0JPF544QWCCh+sE6aIAAIIIIAAAnYLEFTY3V+qQwABBHwlcKHNNM0GlmYjy+Zj+fLlWrJkia6//vqzPzt16pTzZMRTTz2lnJwc5xWR3NxcPfHEE+fVb85duXKl8/TFt99+e/b3N910k/NkhTl49cNXy4bJIoAAAggggIBlAgQVljWUchBAAAE/C5z76ofZL8IED2ZjTRNO3HjjjU5p5rUNEzSYPSv+9ujWrZvzaod5qsLsR/Hoo4+ed84rr7yiV199VePHj3c+N3r55Zdr3rx5zl4XBBV+Xj3MHQEEEEAAAQRsESCosKWT1IEAAghYIPC3e1TU1tY6+1FUVlaqpKREV199tTZs2OC8BvLmm286nxC90HH33XerS5cuKi4u/smvzasfI0eOVGpqqhN4NB/Tpk1z7kFQYcEiogQEEEAAAQQQ8L0AQYXvW0gBCCCAgD0CF9pMc//+/RoxYoTz5Y7XX39d8fHxzh4U5qsg5ksfJrwwr3CYAMPsZXHnnXc65y1atEgPPPCA87OTJ086m2ma0GPbtm1au3at8zpJRkaGPvjgA+eJDV79sGcdUQkCCCCAAAII+FuAoMLf/WP2CCCAgFUCb731lubPn++87tH8eVJT4K5du5yQoW/fvs5TDxUVFTIbbppPlzYfZs8K86SF+dKH2YfChA/mT/Nxww03OGM3NTXJPEGxY8cO51e9e/d2zjebbJ77RIUZp7CwUCdOnFBWVpZmzpype+65xypvikEAAQQQQAABBLwoQFDhxa4wJwQQQACBkATq6uqcr3aYVznO/ZpH88XmU6ZVVVXO78xXPc49Dhw4oLZt2yozMzOke3ESAggggAACCCCAQGwECCpi48xdEEAAAQQQQAABBBBAAAEEEEAgBAGCihCQOAUBBBBAAAEEEEAAAQQQQAABBGIj8H8KTNSKAcqE/AAAAABJRU5ErkJggg==",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b2bf1a19f4b748d39ec822b60fb9a5b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FigureWidget({\n",
       "    'data': [{'customdata': array([0.99988198, 0.99976206, 0.99951147, 0.9986501 , 0.89960496, 0.89951644,\n",
       "                                   0.89939888, 0.89922802, 0.89912866, 0.89890523, 0.89880363, 0.89853175,\n",
       "                                   0.89828404, 0.89820041, 0.8979461 , 0.8976577 , 0.89749938, 0.89728618,\n",
       "                                   0.89644278, 0.8958758 , 0.89509976, 0.8889141 , 0.88526444, 0.87643276,\n",
       "                                   0.79657969, 0.79617561, 0.79541582, 0.79494855, 0.79427896, 0.79371219,\n",
       "                                   0.79325874, 0.79228693, 0.79010382, 0.78802435, 0.78699406, 0.78525752,\n",
       "                                   0.78358504, 0.78199387, 0.77871775, 0.7757645 , 0.77280099, 0.76468775,\n",
       "                                   0.75996894, 0.75730504, 0.67831692, 0.67567982, 0.67354833, 0.66996306,\n",
       "                                   0.66708532, 0.66099035, 0.65253912, 0.64910335, 0.6437327 , 0.6400404 ,\n",
       "                                   0.6362098 , 0.55842573, 0.55395973, 0.55032358, 0.54633195, 0.53838573,\n",
       "                                   0.5347954 , 0.53086147, 0.5256753 , 0.517045  , 0.51440163, 0.43472316,\n",
       "                                   0.42722535, 0.41919324, 0.41490045, 0.40448562, 0.32215623, 0.3179356 ,\n",
       "                                   0.31494951, 0.31069795, 0.30823869, 0.23063738, 0.15299265, 0.15200521,\n",
       "                                   0.07574205, 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                                   0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                                   0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                                   0.        , 0.        , 0.        , 0.        , 0.        ]),\n",
       "              'hovertemplate': ('<b>class: %{text}</b><br>recal' ... 'customdata:.3f}<extra></extra>'),\n",
       "              'line': {'color': '#3366CC'},\n",
       "              'mode': 'lines',\n",
       "              'name': 'person (AP = 0.509)',\n",
       "              'text': array(['person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person', 'person', 'person', 'person', 'person',\n",
       "                             'person', 'person', 'person'], dtype='<U6'),\n",
       "              'type': 'scatter',\n",
       "              'uid': '66ce9289-b6aa-4a54-bb82-0bc0460e85d7',\n",
       "              'x': array([0.  , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 , 0.11,\n",
       "                          0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2 , 0.21, 0.22, 0.23,\n",
       "                          0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3 , 0.31, 0.32, 0.33, 0.34, 0.35,\n",
       "                          0.36, 0.37, 0.38, 0.39, 0.4 , 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47,\n",
       "                          0.48, 0.49, 0.5 , 0.51, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57, 0.58, 0.59,\n",
       "                          0.6 , 0.61, 0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7 , 0.71,\n",
       "                          0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.8 , 0.81, 0.82, 0.83,\n",
       "                          0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.9 , 0.91, 0.92, 0.93, 0.94, 0.95,\n",
       "                          0.96, 0.97, 0.98, 0.99, 1.  ]),\n",
       "              'y': array([0.92      , 0.90888889, 0.90284238, 0.89852744, 0.88      , 0.88      ,\n",
       "                          0.85833333, 0.84762712, 0.83489985, 0.82650231, 0.82142118, 0.82142118,\n",
       "                          0.82142118, 0.82142118, 0.82142118, 0.81653111, 0.81653111, 0.81547709,\n",
       "                          0.80838064, 0.80431617, 0.80318612, 0.79497421, 0.7921321 , 0.79025332,\n",
       "                          0.75801627, 0.75730504, 0.7545687 , 0.75036704, 0.75036704, 0.74633333,\n",
       "                          0.74527706, 0.74300926, 0.73489046, 0.73043   , 0.72891309, 0.72255181,\n",
       "                          0.71933957, 0.71507423, 0.70769337, 0.70046499, 0.69900115, 0.69389888,\n",
       "                          0.69153571, 0.69054998, 0.63638141, 0.63585231, 0.63517771, 0.63110115,\n",
       "                          0.62859029, 0.62577531, 0.62185488, 0.62044935, 0.61628621, 0.61454057,\n",
       "                          0.60968873, 0.54427961, 0.54121914, 0.53782453, 0.53578699, 0.5319866 ,\n",
       "                          0.52940762, 0.52660187, 0.52270998, 0.51687176, 0.51557335, 0.4406486 ,\n",
       "                          0.43746655, 0.43406038, 0.43284696, 0.42778695, 0.34685383, 0.34561677,\n",
       "                          0.34448594, 0.3432135 , 0.34190356, 0.25871351, 0.17432327, 0.17403153,\n",
       "                          0.08731707, 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                          0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                          0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                          0.        , 0.        , 0.        , 0.        , 0.        ])},\n",
       "             {'customdata': array([0.9954716 , 0.99384798, 0.89484104, 0.89484104, 0.89227479, 0.89227479,\n",
       "                                   0.89119812, 0.89119812, 0.88875514, 0.88875514, 0.88697321, 0.88584394,\n",
       "                                   0.88584394, 0.78478823, 0.78478823, 0.78311113, 0.78311113, 0.77853746,\n",
       "                                   0.77853746, 0.77533967, 0.77533967, 0.76721376, 0.7543198 , 0.7543198 ,\n",
       "                                   0.67268419, 0.67268419, 0.67017357, 0.67017357, 0.66724778, 0.66724778,\n",
       "                                   0.66214673, 0.65545037, 0.65545037, 0.65282088, 0.65282088, 0.64337576,\n",
       "                                   0.64337576, 0.62889966, 0.62889966, 0.54726801, 0.54726801, 0.53973907,\n",
       "                                   0.52120453, 0.52120453, 0.43208969, 0.43208969, 0.34965264, 0.34965264,\n",
       "                                   0.33724424, 0.33724424, 0.24307428, 0.07925715, 0.07925715, 0.07671282,\n",
       "                                   0.07671282, 0.07531983, 0.07531983, 0.        , 0.        , 0.        ,\n",
       "                                   0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                                   0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                                   0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                                   0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                                   0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                                   0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                                   0.        , 0.        , 0.        , 0.        , 0.        ]),\n",
       "              'hovertemplate': ('<b>class: %{text}</b><br>recal' ... 'customdata:.3f}<extra></extra>'),\n",
       "              'line': {'color': '#DC3912'},\n",
       "              'mode': 'lines',\n",
       "              'name': 'car (AP = 0.344)',\n",
       "              'text': array(['car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car',\n",
       "                             'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car',\n",
       "                             'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car',\n",
       "                             'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car',\n",
       "                             'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car',\n",
       "                             'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car',\n",
       "                             'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car',\n",
       "                             'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car',\n",
       "                             'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car',\n",
       "                             'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car', 'car',\n",
       "                             'car'], dtype='<U3'),\n",
       "              'type': 'scatter',\n",
       "              'uid': '1397aa7c-2de6-4ffb-8933-314c318a25df',\n",
       "              'x': array([0.  , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 , 0.11,\n",
       "                          0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2 , 0.21, 0.22, 0.23,\n",
       "                          0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3 , 0.31, 0.32, 0.33, 0.34, 0.35,\n",
       "                          0.36, 0.37, 0.38, 0.39, 0.4 , 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47,\n",
       "                          0.48, 0.49, 0.5 , 0.51, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57, 0.58, 0.59,\n",
       "                          0.6 , 0.61, 0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7 , 0.71,\n",
       "                          0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.8 , 0.81, 0.82, 0.83,\n",
       "                          0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.9 , 0.91, 0.92, 0.93, 0.94, 0.95,\n",
       "                          0.96, 0.97, 0.98, 0.99, 1.  ]),\n",
       "              'y': array([0.91111111, 0.91111111, 0.9       , 0.9       , 0.88      , 0.88      ,\n",
       "                          0.88      , 0.88      , 0.87777778, 0.87777778, 0.86777778, 0.86777778,\n",
       "                          0.86777778, 0.79      , 0.79      , 0.79      , 0.79      , 0.75238095,\n",
       "                          0.75238095, 0.70827839, 0.70827839, 0.68805199, 0.67094633, 0.67094633,\n",
       "                          0.63432971, 0.63432971, 0.63432971, 0.63432971, 0.6331723 , 0.6331723 ,\n",
       "                          0.62206119, 0.62122507, 0.62122507, 0.62122507, 0.62122507, 0.61711367,\n",
       "                          0.61711367, 0.6049049 , 0.6049049 , 0.54814815, 0.54814815, 0.54597701,\n",
       "                          0.52910497, 0.52910497, 0.44904905, 0.44904905, 0.37854406, 0.37854406,\n",
       "                          0.3525132 , 0.3525132 , 0.24468468, 0.07837838, 0.07837838, 0.07692308,\n",
       "                          0.07692308, 0.07380952, 0.07380952, 0.        , 0.        , 0.        ,\n",
       "                          0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                          0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                          0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                          0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                          0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                          0.        , 0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "                          0.        , 0.        , 0.        , 0.        , 0.        ])}],\n",
       "    'layout': {'margin': {'b': 0, 'l': 0, 'r': 0, 't': 30},\n",
       "               'shapes': [{'line': {'dash': 'dash'}, 'type': 'line', 'x0': 0, 'x1': 1, 'y0': 1, 'y1': 0}],\n",
       "               'template': '...',\n",
       "               'xaxis': {'constrain': 'domain', 'range': [0, 1], 'title': {'text': 'Recall'}},\n",
       "               'yaxis': {'constrain': 'domain',\n",
       "                         'range': [0, 1],\n",
       "                         'scaleanchor': 'x',\n",
       "                         'scaleratio': 1,\n",
       "                         'title': {'text': 'Precision'}}}\n",
       "})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot = results.plot_pr_curves(classes=[\"person\", \"car\"])\n",
    "plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot.freeze()  # replaces interactive plot with static image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample-level analysis\n",
    "\n",
    "The evaluation routine also populated some new fields on our dataset that contain helpful information that we can use to evaluate our predictions at the sample-level.\n",
    "\n",
    "In particular, each sample now contains new fields:\n",
    "\n",
    "- `eval_tp`: the number of true positive (TP) predictions in the sample\n",
    "- `eval_fp`: the number of false positive (FP) predictions in the sample\n",
    "- `eval_fn`: the number of false negative (FN) predictions in the sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:        evaluate-detections-tutorial\n",
      "Media type:  image\n",
      "Num samples: 5000\n",
      "Persistent:  True\n",
      "Tags:        []\n",
      "Sample fields:\n",
      "    id:           fiftyone.core.fields.ObjectIdField\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    faster_rcnn:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    eval_tp:      fiftyone.core.fields.IntField\n",
      "    eval_fp:      fiftyone.core.fields.IntField\n",
      "    eval_fn:      fiftyone.core.fields.IntField\n"
     ]
    }
   ],
   "source": [
    "# Our dataset's schema now contains `eval_*` fields\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The individual predicted and ground truth objects also have fields populated on them describing the results of the matching process:\n",
    "\n",
    "- `eval`: whether the object is a TP/FP/FN\n",
    "- `eval_id`: the ID of the matching ground truth/predicted object, if any\n",
    "- `eval_iou`: the IoU between the matching objects, if any"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Detection: {\n",
      "    'id': '66047b0f05c1282f3e986920',\n",
      "    'attributes': {},\n",
      "    'tags': [],\n",
      "    'label': 'airplane',\n",
      "    'bounding_box': [\n",
      "        0.5629980087280273,\n",
      "        0.7977214296832356,\n",
      "        0.03478360176086426,\n",
      "        0.1007584484383529,\n",
      "    ],\n",
      "    'mask': None,\n",
      "    'confidence': 0.9952868223190308,\n",
      "    'index': None,\n",
      "    'eval_iou': 0.8995312552391188,\n",
      "    'eval_id': '66047a4b05c1282f3e97f285',\n",
      "    'eval': 'tp',\n",
      "}>\n"
     ]
    }
   ],
   "source": [
    "# Our detections have helpful evaluation data on them\n",
    "sample = high_conf_view.first()\n",
    "print(sample.faster_rcnn.detections[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These extra fields were added because we provided the ``eval_key`` parameter to [evaluate_detections()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.evaluate_detections). If we had omitted this parameter, then no information would have been recorded on our samples.\n",
    "\n",
    "Don't worry, if you forget what evaluations you've run, you can retrieve information about the evaluation later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['eval']\n"
     ]
    }
   ],
   "source": [
    "print(dataset.list_evaluations())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"key\": \"eval\",\n",
      "    \"version\": \"0.24.0\",\n",
      "    \"timestamp\": \"2024-03-27T22:10:32.599000\",\n",
      "    \"config\": {\n",
      "        \"cls\": \"fiftyone.utils.eval.coco.COCOEvaluationConfig\",\n",
      "        \"type\": \"detection\",\n",
      "        \"method\": \"coco\",\n",
      "        \"pred_field\": \"faster_rcnn\",\n",
      "        \"gt_field\": \"ground_truth\",\n",
      "        \"iou\": 0.5,\n",
      "        \"classwise\": true,\n",
      "        \"iscrowd\": \"iscrowd\",\n",
      "        \"use_masks\": false,\n",
      "        \"use_boxes\": false,\n",
      "        \"tolerance\": null,\n",
      "        \"compute_mAP\": true,\n",
      "        \"iou_threshs\": [\n",
      "            0.5,\n",
      "            0.55,\n",
      "            0.6,\n",
      "            0.65,\n",
      "            0.7,\n",
      "            0.75,\n",
      "            0.8,\n",
      "            0.85,\n",
      "            0.9,\n",
      "            0.95\n",
      "        ],\n",
      "        \"max_preds\": 100,\n",
      "        \"error_level\": 1\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(dataset.get_evaluation_info(\"eval\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can even load the view on which you ran an evaluation by calling the [load_evaluation_view()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html?highlight=load_evaluation_view#fiftyone.core.collections.SampleCollection.load_evaluation_view) method on the parent dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset:     evaluate-detections-tutorial\n",
      "Media type:  image\n",
      "Num samples: 100\n",
      "Sample fields:\n",
      "    id:           fiftyone.core.fields.ObjectIdField\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    faster_rcnn:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    eval_tp:      fiftyone.core.fields.IntField\n",
      "    eval_fp:      fiftyone.core.fields.IntField\n",
      "    eval_fn:      fiftyone.core.fields.IntField\n",
      "View stages:\n",
      "    1. Take(size=100, seed=51)\n",
      "    2. FilterLabels(field='faster_rcnn', filter={'$gt': ['$$this.confidence', 0.75]}, only_matches=False, trajectories=False)\n"
     ]
    }
   ],
   "source": [
    "# Load the view on which we ran the `eval` evaluation\n",
    "eval_view = dataset.load_evaluation_view(\"eval\")\n",
    "print(eval_view)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you can delete an evaluation from a dataset, including any information that was added to your samples, by calling [delete_evaluation()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.delete_evaluation)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation views\n",
    "\n",
    "So, now that we have a sense for the aggregate performance of our model, let's dive into sample-level analysis by creating an [evaluation view](https://voxel51.com/docs/fiftyone/user_guide/app.html#viewing-evaluation-patches).\n",
    "\n",
    "Any evaluation that you stored on your dataset can be used to generate an [evaluation view](https://voxel51.com/docs/fiftyone/user_guide/app.html#viewing-evaluation-patches) that is a patches view creating a sample for every true positive, false positive, and false negative in your dataset.\n",
    "Through this view, you can quickly filter and sort evaluated detections by their type (TP/FP/FN), evaluated IoU, and if they are matched to a crowd object.\n",
    "\n",
    "These evaluation views can be created through Python or directly in the App as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset:     evaluate-detections-tutorial\n",
      "Media type:  image\n",
      "Num patches: 37747\n",
      "Patch fields:\n",
      "    id:           fiftyone.core.fields.ObjectIdField\n",
      "    sample_id:    fiftyone.core.fields.ObjectIdField\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    faster_rcnn:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    crowd:        fiftyone.core.fields.BooleanField\n",
      "    type:         fiftyone.core.fields.StringField\n",
      "    iou:          fiftyone.core.fields.FloatField\n",
      "View stages:\n",
      "    1. ToEvaluationPatches(eval_key='eval', config=None)\n"
     ]
    }
   ],
   "source": [
    "eval_patches = dataset.to_evaluation_patches(\"eval\")\n",
    "print(eval_patches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.view = high_conf_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Evaluation Patches](images/evaluate_detections_eval_patches.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use this evaluation view to find individual false positive detections with a confidence of 0.85 or greater."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![FP Confidence Filter](images/evaluate_detections_high_conf_fp.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View the best-performing samples\n",
    "\n",
    "To dig in further, let's create a view that sorts by `eval_tp` so we can see the best-performing cases of our model (i.e., the samples with the most correct predictions):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Show samples with most true positives\n",
    "session.view = high_conf_view.sort_by(\"eval_tp\", reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Best Samples](images/evaluate_detections_eval_tp.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View the worst-performing samples\n",
    "\n",
    "Similarly, we can sort by the `eval_fp` field to see the worst-performing cases of our model (i.e., the samples with the most false positive predictions):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show samples with most false positives\n",
    "session.view = high_conf_view.sort_by(\"eval_fp\", reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Worst Samples](images/evaluate_detections_eval_fp.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering by bounding box area\n",
    "\n",
    "[Dataset views](https://voxel51.com/docs/fiftyone/user_guide/using_views.html) are extremely powerful. For example, let's look at how our model performed on small objects by creating a view that contains only predictions whose bounding box area is less than `32^2` pixels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute metadata so we can reference image height/width in our view\n",
    "dataset.compute_metadata()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Create an expression that will match objects whose bounding boxes have\n",
    "# area less than 32^2 pixels\n",
    "#\n",
    "# Bounding box format is [top-left-x, top-left-y, width, height]\n",
    "# with relative coordinates in [0, 1], so we multiply by image\n",
    "# dimensions to get pixel area\n",
    "#\n",
    "bbox_area = (\n",
    "    F(\"$metadata.width\") * F(\"bounding_box\")[2] *\n",
    "    F(\"$metadata.height\") * F(\"bounding_box\")[3]\n",
    ")\n",
    "small_boxes = bbox_area < 32 ** 2\n",
    "\n",
    "# Create a view that contains only small (and high confidence) predictions\n",
    "small_boxes_view = high_conf_view.filter_labels(\"faster_rcnn\", small_boxes)\n",
    "\n",
    "session.view = small_boxes_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Small Bboxes](images/evaluate_detections_small_bboxes.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can always re-run evaluation to see how our detector fairs on only small boxes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating detections...\n",
      " 100% |███████████████████| 34/34 [339.1ms elapsed, 0s remaining, 100.3 samples/s]     \n"
     ]
    }
   ],
   "source": [
    "# Create a view that contains only small GT and predicted boxes\n",
    "small_boxes_eval_view = (\n",
    "    high_conf_view\n",
    "    .filter_labels(\"ground_truth\", small_boxes, only_matches=False)\n",
    "    .filter_labels(\"faster_rcnn\", small_boxes, only_matches=False)\n",
    ")\n",
    "\n",
    "# Run evaluation\n",
    "small_boxes_results = small_boxes_eval_view.evaluate_detections(\n",
    "    \"faster_rcnn\",\n",
    "    gt_field=\"ground_truth\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "               precision    recall  f1-score   support\n",
      "\n",
      "       person       0.66      0.44      0.53        80\n",
      "          car       0.69      0.43      0.53        21\n",
      "        chair       0.00      0.00      0.00         4\n",
      "         book       0.00      0.00      0.00        20\n",
      "       bottle       0.25      1.00      0.40         1\n",
      "          cup       0.00      0.00      0.00         1\n",
      " dining table       0.00      0.00      0.00         2\n",
      "traffic light       0.56      0.33      0.42        15\n",
      "      handbag       0.00      0.00      0.00         7\n",
      "         boat       0.00      0.00      0.00         1\n",
      "\n",
      "    micro avg       0.61      0.33      0.43       152\n",
      "    macro avg       0.22      0.22      0.19       152\n",
      " weighted avg       0.50      0.33      0.39       152\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get the 10 most common small object classes\n",
    "small_counts = small_boxes_eval_view.count_values(\"ground_truth.detections.label\")\n",
    "classes_top10_small = sorted(small_counts, key=counts.get, reverse=True)[:10]\n",
    "\n",
    "# Print a classification report for the top-10 small object classes\n",
    "small_boxes_results.print_report(classes=classes_top10_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing detections in a crowd\n",
    "\n",
    "If you're familiar with the [COCO data format](https://cocodataset.org/#format-data), you'll know that the ground truth annotations have an `iscrowd = 0/1` attribute that indicates whether a box contains multiple instances of the same object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Detection: {\n",
      "    'id': '66047a4705c1282f3e97c5e9',\n",
      "    'attributes': {},\n",
      "    'tags': [],\n",
      "    'label': 'potted plant',\n",
      "    'bounding_box': [\n",
      "        0.37028125,\n",
      "        0.3345305164319249,\n",
      "        0.038593749999999996,\n",
      "        0.16314553990610328,\n",
      "    ],\n",
      "    'mask': None,\n",
      "    'confidence': None,\n",
      "    'index': None,\n",
      "    'supercategory': 'furniture',\n",
      "    'iscrowd': 0,\n",
      "}>\n"
     ]
    }
   ],
   "source": [
    "# View the `iscrowd` attribute on a ground truth object\n",
    "sample = dataset.first()\n",
    "print(sample.ground_truth.detections[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a view that contains only samples with at least one detection for which `iscrowd` is 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a view that contains only samples for which at least one detection has \n",
    "# its iscrowd attribute set to 1\n",
    "crowded_images_view = high_conf_view.match(\n",
    "    F(\"ground_truth.detections\").filter(F(\"iscrowd\") == 1).length() > 0\n",
    ")\n",
    "\n",
    "session.view = crowded_images_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Crowd Boxes](images/evaluate_detections_crowded.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More complex insights\n",
    "\n",
    "Let's combine our previous operations to form more complex queries that provide deeper insight into the quality of our detections.\n",
    "\n",
    "For example, let's sort our view of crowded images from the previous section in decreasing order of false positive counts, so that we can see samples that have many (allegedly) spurious predictions in images that are known to contain crowds of objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.view = crowded_images_view.sort_by(\"eval_fp\", reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Crowd FPs](images/evaluate_detections_crowded_fp.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the above view to another view that just sorts by false positive count, regardless of whether the image is crowded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "session.view = high_conf_view.sort_by(\"eval_fp\", reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![High Conf Eval FPs Sort](images/evaluate_detections_high_conf_eval_fp.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was one of the first images in the view. As we can see, while the evaluation is detecting $7$ false positives, all of the model's predictions seem accurate. It is just that the ground truth labels lumped a bunch of orange slices together into one box."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**See anything interesting?**\n",
    "\n",
    "What you find will likely be different because a random subset of samples were chosen. In our case, we find missing ground truth boxes for two of the laptop keyboards, a bottle, and even perhaps a cell phone. The model did not confidently predict many of the boxes in this image, but from a high-level, an example like this makes us consider the consequences of including complex or dense images in datasets. It will likely mean incorrect or incomplete ground truth annotations the annotators are not diligent! And that ultimately leads to confused models, and misinformed evaluations.\n",
    "\n",
    "This conclusion would have been nearly impossible to achieve without visually inspecting the individual samples in the dataset according to the variety of criteria that we considered in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.freeze()  # screenshot the active App for sharing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tagging and next steps\n",
    "\n",
    "In practice, the next step is to take action on the issues that we identified above. A natural first step is to *tag* the issues so they can be retrieved and dealt with later. FiftyOne provides support for [tagging samples and labels](https://voxel51.com/docs/fiftyone/user_guide/app.html#tags-and-tagging), both programmatically and via the App.\n",
    "\n",
    "In your App instance, try tagging the predictions with missing ground truth detections. You can do this by clicking on the boxes of the predictions of interest and using the tagging element in the top-right corner to assign a `possibly-missing` tag.\n",
    "\n",
    "Alternatively, we can programmatically tag a batch of labels by creating a view that contains the objects of interest and then applying [tag_labels()](https://voxel51.com/docs/fiftyone/user_guide/fiftyone.core.collections.html?highlight=tag_labels#fiftyone.core.collections.SampleCollection.tag_labels):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tag all highly confident false positives as \"possibly-missing\"\n",
    "(\n",
    "    high_conf_view\n",
    "        .filter_labels(\"faster_rcnn\", F(\"eval\") == \"fp\")\n",
    "        .select_fields(\"faster_rcnn\")\n",
    "        .tag_labels(\"possibly-missing\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These tagged labels could then be sent off to our annotation provider of choice for review and addition to the ground truth labels. FiftyOne currently offers integrations for [Scale AI](https://voxel51.com/docs/fiftyone/api/fiftyone.utils.scale.html), [Labelbox](https://voxel51.com/docs/fiftyone/api/fiftyone.utils.labelbox.html), and [CVAT](https://voxel51.com/docs/fiftyone/api/fiftyone.types.dataset_types.html?highlight=cvat#fiftyone.types.dataset_types.CVATImageDataset).\n",
    "\n",
    "For example, the snippet below exports the tagged labels and their source media to disk in CVAT format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export all labels with the `possibly-missing` tag in CVAT format\n",
    "(\n",
    "    dataset\n",
    "        .select_labels(tags=[\"possibly-missing\"])\n",
    "        .export(\"/path/for/export\", fo.types.CVATImageDataset)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this tutorial, we covered loading a dataset into FiftyOne and analyzing the performance of an out-of-the-box object detection model on the dataset.\n",
    "\n",
    "**So, what's the takeaway?**\n",
    "\n",
    "Aggregate evaluation results for an object detector are important, but they alone don't tell the whole story of a model's performance. It's critical to study the failure modes of your model so you can take the right actions to improve them.\n",
    "\n",
    "In this tutorial, we covered two types of analysis:\n",
    "\n",
    "- Analyzing the performance of your detector across different strata, like high confidence, small objects in crowded scenes\n",
    "- Inspecting the hardest samples in your dataset to diagnose the underlying issue, whether it be your detector or the ground truth annotations"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}