Object_Detection_TF1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZaoLlAKrK41G"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Picsell-ia/training/blob/master/Object_Detection_TF1.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6t3am_CDKjWU"
   },
   "source": [
    "# Object-detection made easy\n",
    "We will learn how to easily train an object detection model from a list of pre-trained models with the dataset you created on the picsell-IA platform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prerequisites\n",
    "\n",
    "We assume that you have a working python3.6+ installation on your computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 641
    },
    "colab_type": "code",
    "id": "bS6pLrgacUg7",
    "outputId": "2e1551d5-e572-49e9-ffd0-a324b80bdc06",
    "tags": []
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/Picsell-ia/training/\n",
    "%cd training\n",
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XhntBjQuKjWV"
   },
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 164
    },
    "colab_type": "code",
    "id": "ZeMjhE6jKjWY",
    "outputId": "bbfc2840-bc5c-4cc6-e693-bd95b4c84b66",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"slim\")\n",
    "from picsellia import Client\n",
    "import picsell_utils\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MYlIZgv9KjWe"
   },
   "source": [
    "# Setting up your Picsell client\n",
    "\n",
    "\n",
    "We need to connect with the Picsell platform to retrieve all of your project data. \n",
    "You can specify in the ``checkout_project`` method a ``png_dir`` to set the path where your images are, if it's not specified you'll need to download the images from the platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5g9FPMCdKjWg",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "api_token = \"api_token\"\n",
    "project_token = \"project_token\"\n",
    "model_name = \"model_name\"\n",
    "\n",
    "clt = Client(api_token=api_token)\n",
    "clt.checkout_project(project_token=project_token)\n",
    "clt.checkout_network(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UPNfFH5_KjWl"
   },
   "source": [
    "Once the client is initialized we can start the pre-processing of our data.\n",
    "\n",
    "# Data pre-processing\n",
    "\n",
    "Most of the pre-processing is done by the Tensorflow object-detection API. However we do need to put our data in a TFRecord format for the API to understand it and to create a protobuf config file describing all the parameters of our model, input pipeline, evaluation and training configuration ...\n",
    "\n",
    "Hopefully everything is handled by the picsell-utils package. \n",
    "\n",
    "## Downloading data\n",
    "\n",
    "We need the annotations and images on our machine. We also need a label map, mapping the labels names to a label ID that the Tensorflow object-detection API can comprehend.\n",
    "When we checked out the network the annotations were downloaded and saved and the label map was generated.\n",
    "We simply need to run ``dl_pictures()`` to download the images from the platform if you didn't specified ``png_dir`` when checking out the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 550
    },
    "colab_type": "code",
    "id": "dUq-cDBVKjWm",
    "outputId": "cc07db90-dc7f-4bfe-b23d-b8263471e90c",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "clt.train_test_split() #to remove\n",
    "clt.dl_pictures()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TFRecord files creation\n",
    "\n",
    "Tensorflow needs the data in a TFRecord format, a memory efficient format storing data as a sequence of binary records. We generate two .record files, one for the training and one for the evaluation.\n",
    "We set the annotation type to rectangle so we can extract the bounding boxes and their label from the annotations and store them inside the records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "annotation_type = \"rectangle\"\n",
    "picsell_utils.create_record_files(label_path=clt.label_path, record_dir=clt.record_dir,\n",
    "                         tfExample_generator=clt.tf_vars_generator, annotation_type=annotation_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Sv6vKXsNKjWs"
   },
   "source": [
    "## Editing the configuration file\n",
    "\n",
    "Every trainable object-detection model downloaded from the Picsell.ia hub is provided with the protobuf configuration file used to train it. We want to edit this file and set some variables specific to our project. \n",
    "\n",
    "Most of those variables are provided by the SDK, but it's still up to you to set up some of them. \n",
    "If you want to dwell inside the configuration file to have more control over your model, you can still open it up with after that and go change some settings like the learning rate decay strategy. But it's not the purpose of this guide.\n",
    "The argument ``incremental_or_transfer`` is here to specify if we should load all variables from the checkpoint, in the case you want to resume a training or if we should re-init the unfrozen variables to start the transfer learning process. In short, use ``incremental`` if you want to resume a training or ``transfer`` for a new training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 109
    },
    "colab_type": "code",
    "id": "DOgkvSDwKjWu",
    "outputId": "c7f4aeed-ac53-4df8-d91d-e51ebb29ec83"
   },
   "outputs": [],
   "source": [
    "#The number of steps you want your model to be trained on\n",
    "nb_steps = 100\n",
    "#Your batch size, highly depending of your hardware and model architecture\n",
    "#Some models will return errors while training with a really low batch size\n",
    "batch_size = 16\n",
    "#The learning rate used, it can be left to None to use the previous one. \n",
    "learning_rate = None\n",
    "\n",
    "picsell_utils.edit_config(model_selected=clt.model_selected, \n",
    "            config_output_dir=clt.config_dir,\n",
    "            record_dir=clt.record_dir, \n",
    "            label_map_path=clt.label_path, \n",
    "            num_steps=nb_steps,\n",
    "            batch_size=batch_size, \n",
    "            learning_rate=learning_rate,\n",
    "            annotation_type=annotation_type,\n",
    "            eval_number=len(clt.eval_list),\n",
    "            incremental_or_transfer=\"transfer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7TIdFiqvKjW0"
   },
   "source": [
    "## Training\n",
    "\n",
    "Now that the input pipeline is built we can finally launch the training. To do this we can use the wrapper function train from picsell_utils , we simply specify where we will save the checkpoint and where the configuration file is located."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "eYkGBEvnKjW1",
    "outputId": "33501fe1-bbae-4961-8d4f-4435c9ece68d",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "picsell_utils.train(ckpt_dir=clt.checkpoint_dir, \n",
    "                     conf_dir=clt.config_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SJZiDV6aKjW5"
   },
   "source": [
    "Once the training is done we want to send the training logs to the Picsell.ia platform. In this case we just want to display the TotalLoss and learning rate. You can print all the dictionnary keys with dict_log.keys() if you want to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QpKJL3l6KjW6",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "dict_log = picsell_utils.tfevents_to_dict(path='Sample Project/Sample SSD Model/0/checkpoint')\n",
    "dict_log = {\"TotalLoss\": dict_log[\"Losses/TotalLoss\"], \"LearningRate\": dict_log['LearningRate/LearningRate/learning_rate']}\n",
    "clt.send_logs(dict_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating\n",
    "\n",
    "Next is the evaluation phase, we can launch the evaluation with the evaluate method of the picsell_utils module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "metrics = picsell_utils.evaluate(clt.metrics_dir, clt.config_dir, clt.checkpoint_dir)\n",
    "clt.send_metrics(metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can send the checkpoints to the Picsell.ia platform now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clt.send_checkpoints()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DhXEFt43KjW9"
   },
   "source": [
    "# Exporting and infering\n",
    "\n",
    "Now that the training is done and that we checked the performance of our model through the metrics returned by the evaluation we may want to export and use this model.\n",
    "\n",
    "## Exporting the model\n",
    "\n",
    "To export our model as a saved_model.pb we use the export_infer_graph function while specifying the right paths. We can send it to the platform to be used on the playground for live inference. The label map also needs to be sent to the platform, however it was done when checking out the network, at the start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sfT1MFkIKjW-",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "picsell_utils.export_infer_graph(ckpt_dir=clt.checkpoint_dir, \n",
    "                       exported_model_dir=clt.exported_model_dir, \n",
    "                       pipeline_config_path=clt.config_dir)\n",
    "\n",
    "clt.send_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1UrcdpZyKjXD"
   },
   "source": [
    "## Inference\n",
    "\n",
    "Lastly we want to use the model on some images so we can send the results to the platform and see them on the dashboard.\n",
    "You can either specify a list of paths of images or the ``record_dir`` attribute from the client, if using a list of paths of images you need to set ``from_tfrecords`` to ``False``. Here we used the record file for inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ke4Jh6J9KjXE",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "picsell_utils.infer(clt.record_dir, exported_model_dir=clt.exported_model_dir, \n",
    "      label_map_path=clt.label_path, results_dir=clt.results_dir, min_score_thresh=0.5, from_tfrecords=True)\n",
    "clt.send_results()"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "notebookOBDTC.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.6.10 64-bit ('tensorflow_p36': conda)",
   "language": "python",
   "name": "python_defaultSpec_1595409265137"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}