Object Detection/object_detection.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T3vCYmDG8XPN",
        "colab_type": "text"
      },
      "source": [
        "In this tutorial we will train a simple object detector for one class. We'd like to have a system which works on low-end devices in real-time. The provided code is simplified to show key steps in model creation, training and exporting to Lens Studio. If you'd like to obtain better precision then the first steps you could try are training for longer time, using higher resolution, increasing the 'width_mult' parameter in config (the model will become slower and bigger in this case so be careful with it).\n",
        "\n",
        "Current model configuration allows to achieve the lens execution time of ~27 msec on Iphone 6 with the model inference time of ~14-17 msec. The 2 times increase of input resolution for one side of the image increases the model inference time two times.\n",
        "\n",
        "If you'd like to use your own training pipeline and custom architecture then make sure that you have outputs of the model in the same format as the provided model. The example of conversion to onnx can be found in the last paragraph."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A6IYmh4d92cu",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5AznYqOMzQ0",
        "colab_type": "text"
      },
      "source": [
        "# Install libraries\n",
        "\n",
        "First, we need to prepare our work environment and install the necessary Python packages. If you're using Google Colab and get error message \"ERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\" then simply ignore it.\n",
        "\n",
        "We added strict version requirements for the packages for better reproducibility. Note that these versions of packages will replace already installed ones.\n",
        "\n",
        "### *You need to restart the kernel to use the installed packages.* (in Colab: Runtime->Restart runtime...)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "36XnOD0PGhEH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%pip install -q numpy opencv-python-headless \\\n",
        "    torch torchvision \\\n",
        "    albumentations tqdm"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q0ITyuv8omR9",
        "colab_type": "text"
      },
      "source": [
        "# Imports\n",
        "\n",
        "Google Colab already has these packages installed but you might need to download some of them for local execution"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QzjQZ5mT8ikz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import cv2\n",
        "import json\n",
        "import itertools\n",
        "import numpy as np\n",
        "import os\n",
        "import shutil\n",
        "import urllib.request\n",
        "import pandas as pd\n",
        "\n",
        "from collections import defaultdict\n",
        "from pathlib import Path\n",
        "from tqdm.notebook import tqdm\n",
        "from zipfile import ZipFile\n",
        "\n",
        "from matplotlib import pyplot as plt\n",
        "from PIL import Image\n",
        "\n",
        "import albumentations as albu\n",
        "import albumentations.augmentations.functional as AF\n",
        "\n",
        "import torch\n",
        "import torch.onnx as onnx\n",
        "import torch.nn.functional as F\n",
        "from torch import nn\n",
        "from torch.utils.data import DataLoader\n",
        "\n",
        "import torchvision\n",
        "from torchvision.models.mobilenetv2 import Conv2dNormActivation\n",
        "from torchvision.ops import box_iou, nms\n",
        "\n",
        "# Set random seeds for the libraries to obtain reproducible results\n",
        "RANDOM_SEED = 1337\n",
        "torch.manual_seed(RANDOM_SEED)\n",
        "torch.cuda.manual_seed(RANDOM_SEED)\n",
        "np.random.seed(RANDOM_SEED)\n",
        "\n",
        "# Flip values for slower training speed, but more determenistic results.\n",
        "torch.backends.cudnn.deterministic = False\n",
        "torch.backends.cudnn.benchmark = True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5fCRHPEheyWF",
        "colab_type": "text"
      },
      "source": [
        "It is recommened to train on GPU but you can run the model on CPU also"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QjZwwHj0ewc_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "DEVICE = torch.device('cpu')\n",
        "if torch.cuda.is_available():\n",
        "    DEVICE = torch.device('cuda:0')\n",
        "    torch.cuda.manual_seed(RANDOM_SEED)\n",
        "DEVICE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4DG5fLGNUDnP"
      },
      "source": [
        "\n",
        "# Global variables for training\n",
        "This training notebook uses COCO dataset: http://cocodataset.org/ The annotations in this dataset belong to the COCO Consortium and are licensed under a Creative Commons Attribution 4.0 License. http://cocodataset.org/#termsofuse Images are part of flickr and have corresponding licenses. To check license for each image please refer to the contents of http://images.cocodataset.org/annotations/image_info_unlabeled2017.zip\n",
        "\n",
        "# 1. Available COCO classes\n",
        "You can check available data here http://cocodataset.org/#explore\n",
        "\n",
        "COCO has 80 categories:\n",
        "\n",
        "person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, fire hydrant, stop sign, parking meter, bench, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket, bottle, wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake, chair, couch, potted plant, bed, dining table, toilet, tv, laptop, mouse, remote, keyboard, cell phone, microwave, oven, toaster, sink, refrigerator, book, clock, vase, scissors, teddy bear, hair drier, toothbrush"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "rRL-7EVJUCsm",
        "colab": {}
      },
      "source": [
        "# Following classes will be united into single category\n",
        "OBJECT_LABELS_UNION = ['car', 'truck', 'bus']\n",
        "# You can create your own categories like these:\n",
        "# OBJECT_LABELS_UNION = ['cat', 'dog']\n",
        "# OBJECT_LABELS_UNION = ['bird']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "pZCiq6wTQvK4",
        "colab": {}
      },
      "source": [
        "DATASET_PATH = Path('.')  # Path to the dataset\n",
        "DIR_TO_SAVE_RESULTS = Path('centernet_model')  # Model snapshots will be saved here\n",
        "os.makedirs(DIR_TO_SAVE_RESULTS, exist_ok=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JfHCARftVtRH",
        "colab": {}
      },
      "source": [
        "# This number can be different\n",
        "# The bigger number of epochs, longer training time, but better model quality\n",
        "# NUM_EPOCHS = 40 # => faster training \n",
        "# NUM_EPOCHS = 70\n",
        "\n",
        "NUM_EPOCHS = 100 # => better quality "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KsYoY5X10hN1",
        "colab_type": "text"
      },
      "source": [
        "Advanced constants"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nmx9RJqX0gqd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BATCH_SIZE = 32\n",
        "LEARNING_RATE = 1e-3\n",
        "NUM_WORKERS = 4  # Number of workers used in PyTorch dataloader\n",
        "\n",
        "INPUT_SIZE = (128, 256) # width & height\n",
        "FEATURE_MAP_SIZE_RATIO = 8 # Ratio between output&input size (e.g. 128 / 16 = 8)\n",
        "\n",
        "# Larger model might yield better results but will be slower.\n",
        "# width_mult is a coefficient which defines how many filters\n",
        "# to use from original mobilenet:\n",
        "MOBILENET_WIDTH_MULTIPLIER = 0.3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wRZdk1opOKZm"
      },
      "source": [
        "# 2. Download dataset (it might take some time)\n",
        "\n",
        "http://cocodataset.org/#home\n",
        "\n",
        "If you encounter message \"Disk is almost full\" on Google Colab then press \"ignore\" button. There is enough space to extract the archive and it will be deleted afterwards. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a_BxNY2QoYBx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def download_and_unpack_file(link, filename, unpack=True):\n",
        "    \"\"\" Download and unpack dataset's annotation files \"\"\"\n",
        "    if (DATASET_PATH / filename).exists():\n",
        "        print(\"{} already exists\".format(filename))\n",
        "        return\n",
        "    archname = link.split('/')[-1]\n",
        "    progress_bar = tqdm(desc=filename,\n",
        "                        dynamic_ncols=True, leave=False,\n",
        "                        mininterval=5, maxinterval=30,\n",
        "                        unit='KiB', unit_scale=True,\n",
        "                        unit_divisor=1024)\n",
        "    def update_progress(count, block_size, total_size):\n",
        "        if progress_bar.total is None:\n",
        "            progress_bar.reset(total_size)\n",
        "        progress_bar.update(count * block_size - progress_bar.n)\n",
        "    urllib.request.urlretrieve(link, archname, reporthook=update_progress)\n",
        "    urllib.request.urlcleanup()\n",
        "    progress_bar.close()\n",
        "    if unpack:\n",
        "        print(\"Unpacking the archive...\")\n",
        "        shutil.unpack_archive(archname, DATASET_PATH)\n",
        "        os.remove(archname)\n",
        "        print(\"Successfuly downloaded and extracted archive\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QkGY9Dkm6qrz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "os.listdir(\"./\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dDa7QyyhQ4uZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Train set is too big for colab so we will extract only files used in training. You can unpack whole dataset locally if you have enough free space\n",
        "download_and_unpack_file('http://images.cocodataset.org/zips/train2017.zip', 'train2017', unpack=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wgAM6ULkZF5V",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "download_and_unpack_file('http://images.cocodataset.org/annotations/annotations_trainval2017.zip', 'annotations')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "naFK1JknZIy5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "download_and_unpack_file('http://images.cocodataset.org/zips/val2017.zip', 'val2017')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fklGsd0to2ru",
        "colab_type": "text"
      },
      "source": [
        "# 3. Data class\n",
        "\n",
        "It provides the interface for image and annotations loading. It should be compatible with PyTorch's Dataloader class. The annotations are loaded into memory from json files while images are loaded from disc each time __getitem__() is called."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b-04kuy8Nt14",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Dataset(torch.utils.data.Dataset):\n",
        "    def __init__(self, phase, box_coder=None, transform=None):\n",
        "        super().__init__()\n",
        "        self.phase = phase\n",
        "        self.class_names = OBJECT_LABELS_UNION\n",
        "\n",
        "        self.dataset = torchvision.datasets.CocoDetection(\n",
        "            DATASET_PATH / (phase + '2017'),\n",
        "            annFile=DATASET_PATH / ('annotations/instances_'+phase+'2017.json'))\n",
        "        self.transform = transform\n",
        "        self.box_coder = box_coder\n",
        "\n",
        "        categories = self.dataset.coco.getCatIds()\n",
        "        categories = self.dataset.coco.loadCats(categories)\n",
        "        self.all_possible_classes = [category['name'] for category in categories]\n",
        "        self.negative_classes = set(self.all_possible_classes) - set(self.class_names)\n",
        "\n",
        "        self.filter_dataset()\n",
        "\n",
        "        if self.phase == \"train\":\n",
        "            self.extract_images()\n",
        "\n",
        "    def filter_dataset(self):\n",
        "        \"\"\" Leave only classes specified in the OBJECT_LABELS_UNION \"\"\"\n",
        "        filter_categories = self.dataset.coco.getCatIds(catNms=self.class_names)\n",
        "        \n",
        "        min_area = 500.\n",
        "        ann_ids = self.dataset.coco.getAnnIds(\n",
        "            catIds=filter_categories, areaRng=[min_area, float('inf')],\n",
        "            iscrowd=False)\n",
        "        im_ids = {self.dataset.coco.anns[ann_idx]['image_id'] for ann_idx in ann_ids}\n",
        "\n",
        "        if self.phase == \"train\":  # Use some part of remaining data to reduce the number of false positives\n",
        "            num_false_positives = len(im_ids) // 20 # 5 percent \n",
        "            cat_ids = self.dataset.coco.getCatIds(self.negative_classes)\n",
        "            for cls_id, neg_class in zip(cat_ids, self.negative_classes):\n",
        "                neg_subset = set(self.dataset.coco.getAnnIds(catIds=[cls_id], areaRng=[min_area, float('inf')], iscrowd=False))\n",
        "                neg_im_ids = {self.dataset.coco.anns[ann_idx]['image_id'] for ann_idx in neg_subset}\n",
        "                neg_im_count = num_false_positives // len(self.negative_classes)\n",
        "                if neg_class == 'person':\n",
        "                    neg_im_count *= 10\n",
        "                neg_im_count = min(neg_im_count, len(neg_im_ids - im_ids))\n",
        "                neg_subset = list(neg_im_ids - im_ids)[:neg_im_count]\n",
        "                im_ids.update(neg_subset)\n",
        "\n",
        "        im_ids = list(im_ids)\n",
        "        cat_ids = self.dataset.coco.getCatIds(catIds=filter_categories)\n",
        "\n",
        "        self.dataset.ids = sorted(im_ids)\n",
        "        self.dataset.coco.anns = {i: self.dataset.coco.anns[i] for i in ann_ids}\n",
        "        self.dataset.coco.imgs = {i: self.dataset.coco.imgs[i] for i in im_ids}\n",
        "        self.dataset.coco.cats = {i: self.dataset.coco.cats[i] for i in cat_ids}\n",
        "        imgToAnns, catToImgs = defaultdict(list), defaultdict(list)\n",
        "\n",
        "        for k, ann in self.dataset.coco.anns.items():\n",
        "            imgToAnns[ann['image_id']].append(ann)\n",
        "            catToImgs[ann['category_id']].append(ann['image_id'])\n",
        "        self.dataset.coco.imgToAnns = imgToAnns\n",
        "        self.dataset.coco.catToImgs = catToImgs\n",
        "\n",
        "    def extract_images(self):\n",
        "        \"\"\" Extract images which will be used \"\"\"\n",
        "        im_paths = []\n",
        "        for im_id in self.dataset.ids:\n",
        "            im_paths.append(self.dataset.coco.loadImgs(im_id)[0]['file_name'])\n",
        "\n",
        "        if not os.path.isdir('./train2017'):\n",
        "            os.mkdir('./train2017')\n",
        "\n",
        "        with ZipFile('./train2017.zip', 'r') as archive:\n",
        "            for image in tqdm(im_paths, dynamic_ncols=True, leave=False):\n",
        "                archive.extract('train2017/' + image, './')\n",
        "        return\n",
        "\n",
        "    def parse_annotation(self, annotation):\n",
        "        bboxes = []\n",
        "        for anno in annotation:\n",
        "            # Filter boxes with 0 area\n",
        "            if (anno['bbox'][2] < 1) or (anno['bbox'][3] < 1) or (anno['iscrowd'] and self.phase != 'val'):\n",
        "                continue\n",
        "            # Boxes in form x_left, y_top, w, h\n",
        "            bboxes.append(anno['bbox'])\n",
        "        \n",
        "        return {'bboxes': bboxes, 'labels': [1] * len(bboxes)}\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        image, annotation = self.dataset[index]\n",
        "    \n",
        "        annotations = self.parse_annotation(annotation)\n",
        "        annotations['image'] = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR)\n",
        "        if self.transform:\n",
        "            annotations = self.transform(annotations)\n",
        "       \n",
        "        if self.box_coder is None:\n",
        "            return annotations\n",
        "\n",
        "        annotations['bboxes'] = torch.tensor(annotations['bboxes'])\n",
        "        encoded_bboxes, encoded_labels = self.box_coder.encode(annotations['bboxes'])\n",
        "\n",
        "        annotations['bboxes'] = encoded_bboxes\n",
        "        annotations['labels'] = encoded_labels\n",
        "        return annotations\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.dataset)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oIXNzp35AlgB"
      },
      "source": [
        "# 4. Define data preprocessing and augmentations.\n",
        "\n",
        "For training we apply different noisy transformations as augmentations. Bounding boxes are adjusted accordingly.\n",
        "\n",
        "Lens Studio feeds the network camera input as images with values in range [0, 255]. So we will train the network with this input range without additional rescaling and normalization. You can add your own input normalization but make sure that you rescaled the weight of the first network's layer before exporting to onxx.\n",
        "\n",
        "It might be useful to tune min_area parameter in albu.BboxParams() if you are working with your own dataset.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ClD9OTUnUw4P",
        "colab": {}
      },
      "source": [
        "def train_transform(annotations):\n",
        "    image = annotations['image']\n",
        "    \n",
        "    size = (INPUT_SIZE[1], INPUT_SIZE[0]) # height, width\n",
        "    scale = min(size[0] / image.shape[0], size[1] / image.shape[1])\n",
        "    intermediate_size = int(image.shape[0] * scale), int(image.shape[1] * scale)\n",
        "    augmentation = albu.Compose(\n",
        "        [\n",
        "            albu.RandomSizedBBoxSafeCrop(*intermediate_size),\n",
        "            albu.HorizontalFlip(p=0.5),\n",
        "            albu.HueSaturationValue(p=0.5),\n",
        "            albu.RGBShift(p=0.5),\n",
        "            albu.RandomBrightnessContrast(p=0.5),\n",
        "            albu.MotionBlur(p=0.5),\n",
        "            albu.PadIfNeeded(*size)\n",
        "        ],\n",
        "        albu.BboxParams(format='coco', min_area=500.,\n",
        "                        min_visibility=0.3, label_fields=['labels'])\n",
        "    )\n",
        "\n",
        "    augmented = augmentation(**annotations)\n",
        "    augmented['image'] = augmented['image'].astype(\n",
        "        np.float32).transpose(2, 0, 1)\n",
        "    return augmented\n",
        "\n",
        "def validation_transform(annotations, with_bboxes=True):\n",
        "    bbox_params = None\n",
        "    if with_bboxes:\n",
        "        bbox_params = albu.BboxParams(format='coco', min_area=500.,\n",
        "                        min_visibility=0.3, label_fields=['labels'])\n",
        "    \n",
        "    image = annotations['image']\n",
        "    size = (INPUT_SIZE[1], INPUT_SIZE[0])\n",
        "    scale = min(size[0] / image.shape[0], size[1] / image.shape[1])\n",
        "    intermediate_size = [int(dim * scale) for dim in image.shape[:2]]\n",
        "    \n",
        "    augmentation = albu.Compose(\n",
        "        [\n",
        "            albu.Resize(*intermediate_size),\n",
        "            albu.PadIfNeeded(*size)\n",
        "        ],\n",
        "        bbox_params\n",
        "    )\n",
        "\n",
        "    augmented = augmentation(**annotations)\n",
        "    augmented['image'] = augmented['image'].astype(\n",
        "        np.float32).transpose(2, 0, 1)\n",
        "    augmented['scale'] = scale\n",
        "\n",
        "    augmented['in_size'] = image.shape[:2]\n",
        "    augmented['out_size'] = size\n",
        "    augmented['intermediate_size'] = intermediate_size\n",
        "    return augmented"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lvTwz86MWou9"
      },
      "source": [
        "# 5. Define box coder\n",
        "\n",
        "Box coder transforms annotations to the format suitable for training and allows to decode the ouputs of trained model. The ground truth bounding boxes should be trasformed into heatmaps compatible with the network ouputs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "v_9nTSADUw1o",
        "colab": {}
      },
      "source": [
        "class BoxCoder:\n",
        "    def __init__(self, image_size, ratio):\n",
        "        self.image_size = image_size\n",
        "        self.fw, self.fh = (i // ratio for i in image_size)\n",
        "        self.iw, self.ih = self.image_size\n",
        "\n",
        "    def encode(self, boxes):\n",
        "        \"\"\"Transforms the ground truth annotations into form of heatmaps\n",
        "        suitable for training of the object detector\"\"\"\n",
        "        cls_targets = torch.zeros((self.fh, self.fw))\n",
        "        loc_targets = torch.zeros((self.fh, self.fw, 4))\n",
        "\n",
        "        boxes_locations = []\n",
        "        if boxes.numel() > 0:\n",
        "            boxes[:, 2:] = boxes[:, :2] + boxes[:, 2:]\n",
        "            boxes /= torch.tensor([self.iw, self.ih, self.iw, self.ih])\n",
        "            box_center_xy = (boxes[:, :2] + boxes[:, 2:]) / 2\n",
        "            box_wh = (boxes[:, 2:] - boxes[:, :2])\n",
        "\n",
        "            mask = (box_center_xy[:, 0] >= 0) & (box_center_xy[:, 1] >= 0)\n",
        "            mask &= (box_center_xy[:, 0] < 1) & (box_center_xy[:, 1] < 1)\n",
        "\n",
        "            box_center_xy, box_wh = box_center_xy[mask], box_wh[mask]\n",
        "\n",
        "            for i, (xy, wh) in enumerate(zip(box_center_xy, box_wh)):\n",
        "                (x, y), (w, h) = xy, wh\n",
        "\n",
        "                ix, iy = int(x * self.fw), int(y * self.fh)\n",
        "                cx, cy = (ix + 0.5) / self.fw, (iy + 0.5) / self.fh\n",
        "\n",
        "                cls_targets[iy, ix] = 1\n",
        "                loc_targets[iy, ix] = torch.tensor([x - cx, y - cy, w, h])\n",
        "                boxes_locations.append((iy, ix))\n",
        "            # Activations are Gaussian-like curves\n",
        "            for iy, ix in boxes_locations:\n",
        "                for dx, dy in itertools.product(range(-1, 2), range(-1, 2)):\n",
        "                    if dx == dy == 0:\n",
        "                        continue\n",
        "                    nx = ix + dx\n",
        "                    ny = iy + dy\n",
        "                    if not 0 <= ny < self.fh or not 0 <= nx < self.fw:\n",
        "                        continue\n",
        "                    if cls_targets[ny, nx] == 0:\n",
        "                        cls_targets[ny, nx] = max(cls_targets[ny, nx],\n",
        "                                                  np.exp(-(abs(dx) + abs(dy))))\n",
        "\n",
        "        return loc_targets, cls_targets\n",
        "\n",
        "    def decode(self, loc_preds, cls_preds, score_thresh=0.5, nms_thresh=0.45,\n",
        "               normalized=False, max_detections=200):\n",
        "        \"\"\"\n",
        "        Decode predicted loc/cls back to real box locations and class labels\n",
        "        \"\"\"\n",
        "        boxes = []\n",
        "        labels = []\n",
        "        scores = []\n",
        "\n",
        "        cls_preds_thresh = cls_preds > score_thresh\n",
        "\n",
        "        for x, y in itertools.product(range(self.fw), range(self.fh)):\n",
        "            if not cls_preds_thresh[y, x]:\n",
        "                continue\n",
        "\n",
        "            box_params = loc_preds[y, x]\n",
        "            cx = (x + 0.5) / self.fw + box_params[0]\n",
        "            cy = (y + 0.5) / self.fh + box_params[1]\n",
        "            bw, bh = box_params[2:] * 0.5\n",
        "\n",
        "            boxes.append([cx - bw, cy - bh, cx + bw, cy + bh])\n",
        "            labels.append(1)\n",
        "            scores.append(cls_preds[y, x])\n",
        "\n",
        "        boxes = torch.tensor(boxes)\n",
        "        labels = torch.tensor(labels)\n",
        "        scores = torch.tensor(scores)\n",
        "        if boxes.numel() > 0:\n",
        "            if not normalized:\n",
        "                boxes *= torch.tensor([self.iw, self.ih, self.iw, self.ih])\n",
        "            \n",
        "            keep = nms(boxes, scores, nms_thresh)[:max_detections]\n",
        "            boxes = boxes[keep]\n",
        "            labels = labels[keep]\n",
        "            scores = scores[keep]\n",
        "            \n",
        "            boxes[:, 2:] = boxes[:, 2:] - boxes[:, :2]\n",
        "        return boxes, labels, scores"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2xcktWdpXDNM"
      },
      "source": [
        "# 6. Model definition\n",
        "This model is based on Mobilenet V2 https://arxiv.org/abs/1801.04381. This is a good starting place for general object detection model, however there were a range of new architectures optimized for mobile inference, including Mobilenet V3 https://arxiv.org/abs/1905.02244.\n",
        "\n",
        "This model runs on average ~17ms on iPhone 6.\n",
        "\n",
        "Our model will use pretrained weights of Mobilenet V2, however these weights assume the input to be RGB in range [0, 1] and input should be normalized. We will disregard this and just use pretrained weights as good initialization for our network. You can play around and check if using the network from scratch without pretrained weights helps you achieve better quality.\n",
        "\n",
        "# Modifications:\n",
        "* Relu6 is replaced with Relu. Make sure that you do the same for your custom architectures because CoreML doesn't support Relu6.\n",
        "* Last 2 blocks are removed.\n",
        "\n",
        "The detector is based on CenterNet approach (Objects as Points, https://arxiv.org/pdf/1904.07850.pdf). It has two heads: classification (heatmap with sigmoid activation) and location.\n",
        "\n",
        "# Important point regarding input and output ranges\n",
        "Lens studio feeds the network camera input as RGB images with values in range [0, 255]. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MGBXj2EyUwyl",
        "colab": {}
      },
      "source": [
        "def convert_layers(model):\n",
        "    \"\"\" Convert relu6 to relu for faster inference in libdnn \"\"\"\n",
        "    for name, module in reversed(model._modules.items()):\n",
        "        if len(list(module.children())) > 0:\n",
        "            model._modules[name] = convert_layers(model=module,)\n",
        "        if isinstance(module, nn.ReLU6):\n",
        "            model._modules[name] = nn.ReLU()\n",
        "    return model\n",
        "\n",
        "def _xavier_init_(m: nn.Module):\n",
        "    if isinstance(m, nn.Conv2d):\n",
        "        nn.init.xavier_uniform_(m.weight)\n",
        "\n",
        "class Detector(nn.Module):\n",
        "    def __init__(self, width_mult, box_transformer=None, test_transform=None):\n",
        "        super().__init__()\n",
        "        self.box_transformer = box_transformer\n",
        "        self.test_transform = test_transform\n",
        "        self.backbone = self.get_backbone(width_mult)\n",
        "\n",
        "        neck_dim = 160\n",
        "        self.neck = Conv2dNormActivation(self.backbone[-1].conv[-1].num_features,\n",
        "                               neck_dim, kernel_size=1)\n",
        "\n",
        "        self.smooth = nn.Conv2d(neck_dim, neck_dim, kernel_size=3, stride=1,\n",
        "                                padding=1, groups=neck_dim, bias=True)\n",
        "\n",
        "        self.cls_scores_out = nn.Sequential(\n",
        "            nn.Conv2d(neck_dim, 1, kernel_size=3, padding=1, bias=True),\n",
        "            nn.Sigmoid()\n",
        "        )\n",
        "        self.loc_out = nn.Conv2d(neck_dim, 4, kernel_size=3,\n",
        "                                 padding=1, bias=True)\n",
        "        self.to_convert = False\n",
        "\n",
        "        self.neck.apply(_xavier_init_)\n",
        "        self.cls_scores_out.apply(_xavier_init_)\n",
        "        self.loc_out.apply(_xavier_init_)\n",
        "\n",
        "        self = convert_layers(self)\n",
        "\n",
        "    def get_backbone(self, width_mult):\n",
        "        \"\"\"\n",
        "        Using mobilenet_v2 pretrained on imagenet from torchvision library\n",
        "        https://github.com/pytorch/vision/blob/master/torchvision/models/mobilenet.py\n",
        "        \"\"\"\n",
        "        model = torchvision.models.mobilenet_v2(width_mult=width_mult)\n",
        "        state_dict = torchvision.models.mobilenet_v2(pretrained=True).state_dict()\n",
        "\n",
        "        if width_mult != 1:\n",
        "            target_dict = model.state_dict()\n",
        "            for k in target_dict.keys():\n",
        "                if len(target_dict[k].size()) == 0:\n",
        "                    continue\n",
        "                state_dict[k] = state_dict[k][:target_dict[k].size(0)]\n",
        "                if len(state_dict[k].size()) > 1:\n",
        "                    state_dict[k] = state_dict[k][:, :target_dict[k].size(1)]\n",
        "\n",
        "        model.load_state_dict(state_dict)\n",
        "\n",
        "        return nn.Sequential(*(list(model.features.children())[:14]))\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.backbone(x)\n",
        "        x = self.neck(x)\n",
        "        x = self.smooth(F.interpolate(x, scale_factor=2))\n",
        "\n",
        "        cls_scores = self.cls_scores_out(x)\n",
        "        if self.to_convert:\n",
        "            return self.loc_out(x), cls_scores\n",
        "\n",
        "        cls_scores = torch.clamp(cls_scores, min=1e-4, max=1-1e-4)\n",
        "\n",
        "        return self.loc_out(x).permute((0, 2, 3, 1)), cls_scores.squeeze()\n",
        "\n",
        "    def set_conversion_mode(self, to_convert=False):\n",
        "        \"\"\" Set True for export to onnx in libdnn compatible format.\n",
        "        The reshape operation in heads might work incorrectly in lens studio\n",
        "        so it is ommited in the onnx graph.\n",
        "        \"\"\"\n",
        "        self.to_convert = to_convert\n",
        "\n",
        "    def load(self, model):\n",
        "        self.load_state_dict(torch.load(model, map_location=lambda storage, loc: storage))\n",
        "\n",
        "    def save(self, model_path):\n",
        "        torch.save(self.state_dict(), model_path)\n",
        "\n",
        "    def rescale_boxes(self, boxes, out_size, intermediate_size, scale):\n",
        "        \"\"\" Removes padding shift and rescales bounding boxes to original\n",
        "        input image size \"\"\"\n",
        "        outh, outw = out_size\n",
        "        rh, rw = intermediate_size\n",
        "\n",
        "        if rh < outh:\n",
        "            boxes[:, 1] -= ((outh - rh) / 2)\n",
        "            boxes[:, 1] = torch.clamp(boxes[:, 1], min=0, max=outh-1)\n",
        "        if rw < outw:\n",
        "            boxes[:, 0] -= ((outw - rw) / 2)\n",
        "            boxes[:, 0] = torch.clamp(boxes[:, 0], min=0, max=outw-1)\n",
        "\n",
        "        boxes /= scale\n",
        "        \n",
        "        return boxes\n",
        "\n",
        "    def predict(self, sample, score_threshold=0.5, nms_thresh=0.45):\n",
        "        \"\"\" sample (dict) {\"image\": cv2 BGR image} \"\"\"\n",
        "        sample = self.test_transform(sample, with_bboxes=False)\n",
        "        images = torch.tensor(sample[\"image\"]).unsqueeze(0)\n",
        "\n",
        "        with torch.no_grad():\n",
        "            boxes, cls = self.forward(images)\n",
        "\n",
        "        boxes, labels, probs = self.box_transformer.decode(boxes[0], cls, score_threshold, nms_thresh)\n",
        "        if len(boxes) == 0:\n",
        "            return torch.tensor([]), torch.tensor([]), torch.tensor([])\n",
        "\n",
        "        boxes = self.rescale_boxes(boxes, sample['out_size'], sample['intermediate_size'], sample['scale'])\n",
        "\n",
        "        return boxes, labels, probs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tYZtPd75asUX"
      },
      "source": [
        "# 7. Loss function\n",
        "\n",
        "It consists of two parts: localization (model estimates coordinates of objects' bounding boxes) and classification (predicts whether there is any object in the given region on heatmap)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iuOpJ2jicAqr",
        "colab": {}
      },
      "source": [
        "class Loss(nn.Module):\n",
        "    \"\"\" Focal loss is used for classification and L1-loss for regression. \"\"\"\n",
        "    def __init__(self):\n",
        "        super(Loss, self).__init__()\n",
        "\n",
        "    def cls_loss(self, pred, target, neg_weights=4, pos_weights=2):\n",
        "        pos_mask = (target == 1).float()\n",
        "        neg_mask = (target < 1).float()\n",
        "        pos_loss = -torch.log(pred) * torch.pow(1 - pred, pos_weights) * pos_mask\n",
        "        neg_loss = -torch.log(1 - pred) * torch.pow(pred, pos_weights) * torch.pow(1 - target, neg_weights) * neg_mask\n",
        "        num_pos = max(pos_mask.float().sum(), 1)\n",
        "        return (pos_loss.sum() + neg_loss.sum()) / num_pos\n",
        "\n",
        "    def forward(self, loc_preds, loc_targets, cls_preds, cls_targets):\n",
        "        batch_size = loc_preds.size(0)\n",
        "        cls_targets = cls_targets.view(batch_size, -1)\n",
        "        loc_targets = loc_targets.view(batch_size, -1, 4)\n",
        "        cls_preds = cls_preds.view(batch_size, -1)\n",
        "        loc_preds = loc_preds.view(batch_size, -1, 4)\n",
        "\n",
        "        pos = cls_targets > 0.99  # 0.5 in target means negative anchor\n",
        "        mask = pos.unsqueeze(2).expand_as(loc_preds)\n",
        "        \n",
        "        loc_loss = F.l1_loss(loc_preds[mask], loc_targets[mask], reduction='sum')\n",
        "        loc_loss /= pos.sum().item() + 1e-5\n",
        "\n",
        "        cls_loss = self.cls_loss(cls_preds, cls_targets)\n",
        "\n",
        "        loss = loc_loss + cls_loss\n",
        "        return loss, loc_loss, cls_loss"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ODB2FKVVaTwE"
      },
      "source": [
        "# 8. Create dataloaders\n",
        "\n",
        "PyTorch's dataloader allows to load multiple images in parallel processes for faster training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RMlhAClogwXc",
        "colab": {}
      },
      "source": [
        "box_coder = BoxCoder(INPUT_SIZE, FEATURE_MAP_SIZE_RATIO)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "9laVXWem8KHo",
        "scrolled": true,
        "colab": {}
      },
      "source": [
        "val_dataset = Dataset('val', box_coder, validation_transform)\n",
        "train_dataset = Dataset('train', box_coder, train_transform)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yOvxqR8f8dhI",
        "colab_type": "text"
      },
      "source": [
        "Create dataloaders for parallel loading of images"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_-P2U00y5OAv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_loader = torch.utils.data.DataLoader(\n",
        "    train_dataset,\n",
        "    batch_size=BATCH_SIZE,\n",
        "    shuffle=True,\n",
        "    num_workers=NUM_WORKERS,\n",
        "    worker_init_fn=lambda _: np.random.seed(),\n",
        "    drop_last=True\n",
        ")\n",
        "\n",
        "val_loader = torch.utils.data.DataLoader(\n",
        "    val_dataset,\n",
        "    batch_size=BATCH_SIZE // 4,\n",
        "    shuffle=False,\n",
        "    num_workers=NUM_WORKERS,\n",
        "    drop_last=False\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2H7q7zyGM9h2"
      },
      "source": [
        "# 9. Create and train the network\n",
        "\n",
        "Model snapshots will be saved each epoch to the DIR_TO_SAVE_RESULTS directory"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "jkvSk_QxBelG",
        "colab": {}
      },
      "source": [
        "model = Detector(MOBILENET_WIDTH_MULTIPLIER, box_transformer=box_coder,\n",
        "                 test_transform=validation_transform)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9BwuMrz5CrgS",
        "colab_type": "text"
      },
      "source": [
        "We'll also set up learning rate scheduler to drop learning rate if our network training platoes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7j1ho5GXs-Wd",
        "colab": {}
      },
      "source": [
        "optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE,\n",
        "                             weight_decay=1e-4)\n",
        "model.to(DEVICE)  # Move model to the device selected for training\n",
        "\n",
        "criterion = Loss()\n",
        "print(f\"Learning rate: {LEARNING_RATE}\")\n",
        "print(\"Using ReduceLROnPlateau scheduler.\")\n",
        "scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n",
        "    optimizer, 'min', factor=0.5, patience=5, verbose=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "z_26xSX3uLwf",
        "colab": {}
      },
      "source": [
        "print(\"Device used for training:\", DEVICE)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ia0E9m_hVUh6",
        "colab_type": "text"
      },
      "source": [
        "Define train, validation functions and train the network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xqX0uGZYTitH",
        "colab": {}
      },
      "source": [
        "def train(loader, net, criterion, optimizer, device):\n",
        "    net.train(True)\n",
        "    total_loss = 0.\n",
        "    total_regression_loss = 0.\n",
        "    total_classification_loss = 0.\n",
        "\n",
        "    progress_bar = tqdm(enumerate(loader), total=len(loader),\n",
        "                        dynamic_ncols=True, leave=False)\n",
        "\n",
        "    for i, data in progress_bar:\n",
        "        images = data['image'].to(device)\n",
        "        boxes = data['bboxes'].to(device)\n",
        "        labels = data['labels'].to(device)\n",
        "\n",
        "        optimizer.zero_grad()\n",
        "        locs, cls = net(images)\n",
        "        loss, regression_loss, classification_loss = criterion(locs, boxes,\n",
        "                                                               cls, labels)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "        total_loss += loss.item()\n",
        "        total_regression_loss += regression_loss.item()\n",
        "        total_classification_loss += classification_loss.item()\n",
        "        \n",
        "        progress_bar.set_description(\"loss {:.4f},  regression loss {:.4f}, classification loss {:.4f}\".format(\n",
        "            total_loss / (i+1), total_regression_loss / (i+1), total_classification_loss / (i+1)))\n",
        "\n",
        "def validate(loader, net, criterion, device):\n",
        "    net.eval()\n",
        "    total_loss = 0.\n",
        "    total_regression_loss = 0.\n",
        "    total_classification_loss = 0.\n",
        "    loader_len = len(loader)\n",
        "    for _, data in enumerate(loader):\n",
        "        images = data['image'].to(device)\n",
        "        boxes = data['bboxes'].to(device)\n",
        "        labels = data['labels'].to(device)\n",
        "\n",
        "        with torch.no_grad():\n",
        "            locs, cls = net(images)\n",
        "            loss, regression_loss, classification_loss = criterion(locs, boxes, cls, labels)\n",
        "\n",
        "        total_loss += loss.item()\n",
        "        total_regression_loss += regression_loss.item()\n",
        "        total_classification_loss += classification_loss.item()\n",
        "    return total_loss / loader_len, total_regression_loss / loader_len, total_classification_loss / loader_len\n",
        "# Start training\n",
        "for epoch in range(NUM_EPOCHS):\n",
        "    train(train_loader, model, criterion, optimizer,\n",
        "          device=DEVICE)\n",
        "    \n",
        "    val_loss, val_regression_loss, val_class_loss = validate(val_loader, model,\n",
        "                                                             criterion, DEVICE)\n",
        "    print(\"Epoch: {}, val loss {:.4f}, regression loss {:.4f}, classification loss {:.4f}\".format(\n",
        "          epoch, val_loss, val_regression_loss, val_class_loss))\n",
        "\n",
        "    scheduler.step(val_loss)\n",
        "    model_path = DIR_TO_SAVE_RESULTS / f\"e-{epoch}-{val_loss:.3f}.pth\"\n",
        "    model.save(model_path)\n",
        "    print(f\"Saved model {model_path}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8ro3qZQkpvg4"
      },
      "source": [
        "# 10. Check out the predictions of trained detector\n",
        "\n",
        "Set 'eval' mode for test inference"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "bglf8wgHs2CJ",
        "colab": {}
      },
      "source": [
        "# model.load(DIR_TO_SAVE_RESULTS / 'e-5-1.583.pth')  # Use to load saved model snapshot\n",
        "model.eval()\n",
        "model = model.to('cpu')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DMfD0XdNd_8n",
        "colab_type": "text"
      },
      "source": [
        "Use test image 'car_test_image.png'. Make sure you've uploaded it to the google colab environment. Try changing the score_threshold in model.predict() to see how the model predicts less confident detections."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "W1fzLsQGWGYA",
        "colab": {}
      },
      "source": [
        "sample = {'image': cv2.imread('./car_test_image.png')}  # Load the image in BGR format\n",
        "\n",
        "boxes, labels, probs = model.predict(sample, score_threshold=0.4)\n",
        "for i in range(boxes.size(0)):\n",
        "    box = boxes[i, :]\n",
        "    label = f\"{probs[i]:.2f}\"\n",
        "    cv2.rectangle(sample['image'], (box[0], box[1]),\n",
        "                  (box[0] + box[2], box[1] + box[3]), (255, 255, 0), 4)\n",
        "    cv2.putText(sample['image'], label, (box[0] + 20, box[1] + 40),\n",
        "                cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 0, 255),  2)\n",
        "plt.figure(figsize=(10, 10))\n",
        "plt.axis('off')\n",
        "plt.imshow(cv2.cvtColor(sample['image'], cv2.COLOR_BGR2RGB))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "i3-YzV9LhsUx"
      },
      "source": [
        "# 11. Export the model to onnx for furher usage in Lens Studio\n",
        "'det.onnx' file will be created in the DIR_TO_SAVE_RESULTS.\n",
        "\n",
        "\n",
        "LensStudio sends RGB image to the network input with values in range [0, 255]. Our network is already trained for that, but if you trained the network with other input range you might need to adjust it when you import your ONNX file.\n",
        "\n",
        "BatchNorm layers will be fused with convolution layers in studio so there is no need to do it in pytorch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "fczQK-MWh_Z7",
        "colab": {}
      },
      "source": [
        "onnx_model_path = DIR_TO_SAVE_RESULTS / 'det.onnx'\n",
        "dummy_input = torch.ones(1, 3, INPUT_SIZE[1], INPUT_SIZE[0],\n",
        "                         dtype=torch.float32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "29S6RiMJhh_g",
        "colab": {}
      },
      "source": [
        "# model.load(DIR_TO_SAVE_RESULTS / 'e-75-1.091.pth')  # Use to load saved model snapshot\n",
        "model.to('cpu')\n",
        "\n",
        "model.set_conversion_mode(to_convert=True)\n",
        "model = model.eval()\n",
        "\n",
        "input_names = ['data']\n",
        "output_names = ['loc', 'cls']\n",
        "\n",
        "onnx.export(model, dummy_input, onnx_model_path, verbose=False,\n",
        "            input_names=input_names, output_names=output_names,\n",
        "            keep_initializers_as_inputs=True, opset_version=11)\n",
        "\n",
        "model.set_conversion_mode(to_convert=False)\n",
        "print(\"Successfully saved model as {}\".format(onnx_model_path))"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "object_detection.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}