TAO_Steps_for_Yolo_v4_tiny.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dMuP1BR8WGD3"
      },
      "source": [
        "## Steps that I used to perform model training, evaluating and inferencing the model using Tao toolkit on AWS EC2 cloud setup\n",
        "\n",
        "\n",
        "\n",
        "It is expected that the TAO toolkit,the related dependecies and pre-trained models are already installed on the setup, instructions are given in below document:\n",
        "\n",
        "#### TAO Quick start guide\n",
        "https://docs.nvidia.com/tao/tao-toolkit/text/tao_toolkit_quick_start_guide.html \n",
        "\n",
        "\n",
        "#### Yolo_v4_tiny model\n",
        "\n",
        "https://docs.nvidia.com/tao/tao-toolkit/text/object_detection/yolo_v4.html\n",
        "\n",
        "\n",
        "## The below steps were performed in a similar manner for all the three approaches."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y6OvXJquYdC5"
      },
      "source": [
        "### Step 1 TAO dependencies\n",
        "\n",
        "It is expected that the TAO toolkit and the related dependecies are already installed on the setup, instructions are given in below document:\n",
        "https://docs.nvidia.com/tao/tao-toolkit/text/tao_toolkit_quick_start_guide.html "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c2bu53jzW_kv"
      },
      "source": [
        "### Step 2 Pre-processing data-set\n",
        "\n",
        "Create below directory structure to store data and configuration files\n",
        "\n",
        "USER_EXPERIMENT_DIR=/workspace/tao-experiments/ppe_detector_approach/yolo_v4_tiny\n",
        "\n",
        "DATA_DOWNLOAD_DIR=/workspace/tao-experiments/ppe_detector_approach/yolo_v4_tiny/data\n",
        "\n",
        "SPECS_DIR=/workspace/tao-experiments/ppe_detector_approach/yolo_v4_tiny/specs\n",
        "\n",
        "TRAINING_DIR=/workspace/tao-experiments/ppe_detector_approach/yolo_v4_tiny/training\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n8Iym2AsZbry"
      },
      "source": [
        "### Step 3 Data directory\n",
        "\n",
        "DATA_DOWNLOAD_DIR, this directory will contain below subfolder:\n",
        "\n",
        "1) train:\n",
        "\n",
        "- This folder contains images and labels directory which will be used for model training\n",
        "\n",
        "2) test:\n",
        "\n",
        "- This folder contains images and labels directory which will be used for model evaluation on test dataset\n",
        "\n",
        "3) val:\n",
        "\n",
        "- This folder contains images and labels directory which will be used for model evaluation on validation dataset\n",
        "\n",
        "4) tfrecords:\n",
        "\n",
        "- This folder contains tfrecord files for train, test and validation dataset\n",
        "\n",
        "\n",
        "\n",
        "SPECS_DIR=/workspace/tao-experiments/ppe_detector_approach/yolo_v4_tiny/specs\n",
        "\n",
        "SPECS_DIR, this directory will contain below files:\n",
        "\n",
        "- training.cfg\n",
        "\n",
        "- inference.cfg\n",
        "\n",
        "- train_convert.cfg\n",
        "\n",
        "- test_convert.cfg\n",
        "\n",
        "- val_convert.cfg\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4LTec-DbbuTa"
      },
      "source": [
        "### Step 3 Tfrecords generate\n",
        "\n",
        "Create train_convert.cfg, test_convert.cfg and val_convert.cfg files for TAO to identify folders of dataset that needs to be converted to tfrecords\n",
        "\n",
        "Configuration file for dataset convert: https://docs.nvidia.com/tao/tao-toolkit/text/object_detection/yolo_v4_tiny.html#preparing-the-input-data-structure\n",
        "\n",
        "Sample command to generate training data-set tfrecord:\n",
        ">>  tao yolo_v4 dataset_convert -d /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/specs/train_convert.cfg -o /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/data/tfrecords/train/ -v\n",
        "\n",
        "\n",
        "Similarly for validation dataset:\n",
        ">>  tao yolo_v4 dataset_convert -d /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/specs/val_convert.cfg -o /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/data/tfrecords/val/ -v\n",
        "\n",
        "Similarly for Test dataset:\n",
        ">>  tao yolo_v4 dataset_convert -d /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/specs/test_convert.cfg -o /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/data/tfrecords/test/ -v\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P14c3Rp9dDco"
      },
      "source": [
        "### Step 4: Create training configuration file\n",
        "\n",
        "Training configuration document: https://docs.nvidia.com/tao/tao-toolkit/text/object_detection/yolo_v4_tiny.html#creating-a-configuration-file\n",
        "\n",
        "training.cfg file can be found under specs directory\n",
        "\n",
        "SPECS_DIR=/workspace/tao-experiments/ppe_detector_approach/yolo_v4_tiny/specs\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_PwT5Gaao-Hd"
      },
      "source": [
        "### Step 5: Generating Yolo anchor shapes\n",
        "\n",
        "Document: https://docs.nvidia.com/tao/tao-toolkit/text/object_detection/yolo_v4.html#generate-anchor-shape\n",
        "\n",
        "Sample command:\n",
        ">> tao yolo_v4_tiny kmeans -l /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/data/training/labels -i /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/data/training/images -x 960 -y 544\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q0nzkOEweBVF"
      },
      "source": [
        "### Step 5 Model training\n",
        "\n",
        "Training model: https://docs.nvidia.com/tao/tao-toolkit/text/object_detection/yolo_v4.html#training-the-model\n",
        "\n",
        "\n",
        "Sample command used for running model training:\n",
        ">> tao yolo_v4_tiny train -e /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/specs/training.cfg -r /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/ppe_training/ -k tlt_encode --gpus 4 --log_file /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/ppe_training/training_output.log\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "PedA9Vtae_XO"
      },
      "outputs": [],
      "source": [
        "# Output of above model training after 100/100 epochs:\n",
        "# Start to calculate AP for each class\n",
        "# *******************************\n",
        "# helmet        AP    0.70701\n",
        "# person        AP    0.55276\n",
        "# vest          AP    0.61022\n",
        "#               mAP   0.62333\n",
        "# *******************************"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oO2xvvwpgTuB"
      },
      "source": [
        "At the end of training, tao will generate model.tlt file that can be used for model evaluation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FDXwdsBbfOfC"
      },
      "source": [
        "### Step 6 Model Evaluation\n",
        "\n",
        "Model evaluation config: https://docs.nvidia.com/tao/tao-toolkit/text/object_detection/yolo_v4.html#evaluating-the-model\n",
        "\n",
        "Sample command to run model evaluation:\n",
        ">> tao yolo_v4_tiny evaluate -e /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/specs/training.cfg -m /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/ppe_training/weights/model.tlt -k tlt_encode\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "4nLX9SgKgmEN"
      },
      "outputs": [],
      "source": [
        "# Evaluate Model (Val dataset):\n",
        "# Start to calculate AP for each class\n",
        "# *******************************\n",
        "# helmet        AP    0.70701\n",
        "# person        AP    0.55276\n",
        "# vest          AP    0.61022\n",
        "#               mAP   0.62333\n",
        "# *******************************\n",
        "\n",
        "# Evaluate on Test dataset:\n",
        "# Start to calculate AP for each class\n",
        "# *******************************\n",
        "# helmet        AP    0.62435\n",
        "# person        AP    0.56971\n",
        "# vest          AP    0.61299\n",
        "#               mAP   0.60235\n",
        "# *******************************\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y55yO_Rrem_n"
      },
      "source": [
        "### Step 7 Model Inference\n",
        "\n",
        "Document: https://docs.nvidia.com/tao/tao-toolkit/text/object_detection/yolo_v4.html#running-inference-on-a-yolov4-model\n",
        "\n",
        "\n",
        "Sample command to generate inference files using trained model:\n",
        "\n",
        "For validation dataste\n",
        ">> tao detectnet_v2 inference -e /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/specs/inference.cfg -i /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/data/val/images/ -o /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/data/val/labels/ -k tlt_encode\n",
        "\n",
        "For test dataset:\n",
        ">> tao detectnet_v2 inference -e /workspace/tao-experimentsppe_detector_approach_1/yolo_v4_tiny/specs/inference.cfg -i /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/data/test/images/ -o /workspace/tao-experiments/ppe_detector_approach_1/yolo_v4_tiny/data/test/labels/ -k tlt_encode\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y7fUZrnXun8I"
      },
      "source": [
        "The further steps for export the model to run on Deepstream were not performed with Yolo_v4_tiny model. For the demonstration, only detectNet_v2 model was used."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WiDqvx4zsRkc"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1As8SBJrV7k3"
      },
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "name": "TAO_Yolo_v4_tiny.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.9.13 64-bit (windows store)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.13"
    },
    "vscode": {
      "interpreter": {
        "hash": "ab3afec5226c55a4579eb186438f17270aad9a4e84b31655f304cdfdaa08c6c2"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}