experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7mGmQbAO5pQb"
   },
   "source": [
    "# Setup\n",
    "\n",
    "Clone GitHub [repository](https://github.com/ultralytics/yolov5), install [dependencies](https://github.com/ultralytics/yolov5/blob/master/requirements.txt) and check PyTorch and GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wbvMlHd_QwMG",
    "outputId": "e8225db4-e61d-4640-8b1f-8bfce3331cea"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import utils\n",
    "display = utils.notebook_init()  # checks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4JnkELT0cIJg"
   },
   "source": [
    "# 1. Detect\n",
    "\n",
    "`detect.py` runs YOLOv5 inference on a variety of sources, downloading models automatically from the [latest YOLOv5 release](https://github.com/ultralytics/yolov5/releases), and saving results to `runs/detect`. Example inference sources are:\n",
    "\n",
    "```shell\n",
    "python detect.py --source 0  # webcam\n",
    "                          img.jpg  # image\n",
    "                          vid.mp4  # video\n",
    "                          screen  # screenshot\n",
    "                          path/  # directory\n",
    "                         'path/*.jpg'  # glob\n",
    "                         'https://youtu.be/LNwODJXcvt4'  # YouTube\n",
    "                         'rtsp://example.com/media.mp4'  # RTSP, RTMP, HTTP stream\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect Unquantized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 detect.py --cfg models/lpyolo.yaml --weights experiment_models/lpyolo.pt --img 640 --conf 0.25 --source data/images/zidane.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect Quantized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zR9ZbuQCH7FX",
    "outputId": "284ef04b-1596-412f-88f6-948828dd2b49"
   },
   "outputs": [],
   "source": [
    "!python3 detect.py --cfg models/lpyolo_quant.yaml --weights experiment_models/lpyolo_W4A4.pt --img 640 --conf 0.25 --source data/images/zidane.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0eq1SMWl6Sfn"
   },
   "source": [
    "# 2. Validate\n",
    "Validate a model's accuracy on the [COCO](https://cocodataset.org/#home) dataset's `val` or `test` splits. Models are downloaded automatically from the [latest YOLOv5 release](https://github.com/ultralytics/yolov5/releases). To show results by class use the `--verbose` flag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate Unquantized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 val.py --weights experiment_models/lpyolo_coco_anchors_unnorm.pt --data VOC.yaml --img 640 --half"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate Quantized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "X58w8JLpMnjH",
    "outputId": "3e234e05-ee8b-4ad1-b1a4-f6a55d5e4f3d"
   },
   "outputs": [],
   "source": [
    "!python3 val.py --weights experiment_models/lpyolo_W4A4.pt --data coco128.yaml --img 640 --half"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Unquantized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 train.py --img 640 --batch 64 --epochs 300 --data coco128.yaml --weights '' --cache --cfg models/lpyolo.yaml --classes 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Quantized (QAT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1NcFxRcFdJ_O",
    "outputId": "bbeeea2b-04fc-4185-aa64-258690495b5a"
   },
   "outputs": [],
   "source": [
    "!python3 train.py --img 640 --batch 32 --epochs 50 --data coco128.yaml --weights experiment_models/lpyolo.pt --cache --cfg models/lpyolo_quant.yaml --classes 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference Pretrained Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "model_pre = torch.hub.load('.',\n",
    "                      'custom',\n",
    "                      'experiment_models/lpyolo.pt',\n",
    "                       source='local',\n",
    "                       classes = 7,\n",
    "                       cfg = \"models/lpyolo.yaml\",\n",
    "                       force_reload=True\n",
    "                      )\n",
    "\n",
    "im = '/home/omar/datasets/coco128/images/train2017/000000579736.jpg'  # file, Path, PIL.Image, OpenCV, nparray, list\n",
    "results = model_pre(im)  # inference\n",
    "results.show()  # or .show(), .save(), .crop(), .pandas(), etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference Quantized Model (QAT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GMusP4OAxFu6"
   },
   "outputs": [],
   "source": [
    "# YOLOv5 PyTorch HUB Inference (DetectionModels only)\n",
    "import torch\n",
    "\n",
    "model_q = torch.hub.load('.',\n",
    "                       'custom',\n",
    "                       'experiment_models/lpyolo_W4A4.pt',\n",
    "                       source='local',\n",
    "                       classes = 7,\n",
    "                       force_reload=True,\n",
    "                       cfg = \"models/lpyolo_quant.yaml\",\n",
    "                      )\n",
    "im = 'https://ultralytics.com/images/bus.jpg'\n",
    "#im = '/home/omar/datasets/coco128/images/train2017/000000000338.jpg'  # file, Path, PIL.Image, OpenCV, nparray, list\n",
    "results = model_q(im)  # inference\n",
    "results.show()  # or .show(), .save(), .crop(), .pandas(), etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from brevitas.export import export_brevitas_onnx\n",
    "import torch\n",
    "\n",
    "IN_CH = 384\n",
    "OUT_CH = 640\n",
    "BATCH_SIZE = 1\n",
    "\n",
    "model_no_detect = torch.nn.Sequential(*[model_q.model.model.model[i] for i in range(15)])\n",
    "\n",
    "path = 'experiment_models/lpyolo_W4A4.onnx'\n",
    "inp = torch.randn(BATCH_SIZE,3, IN_CH, OUT_CH).cuda()\n",
    "\n",
    "detection_model = model_no_detect\n",
    "detection_model.cuda()\n",
    "detection_model.eval()\n",
    "\n",
    "exported_model = export_brevitas_onnx(detection_model, inp, path)\n",
    "\n",
    "detect_module = model_q.model.model.model[15]\n",
    "torch.save(detect_module.state_dict(), 'experiment_models/detect_module.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running model with detached detect head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "from models.yolo import Detect\n",
    "from utils.general import non_max_suppression\n",
    "from utils.dataloaders import letterbox\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "nc = 7\n",
    "anchors = np.array([[10, 13, 16, 30, 33, 23], [81, 82, 135, 169, 344, 319], [116, 90, 156, 198, 373, 326]])\n",
    "detect_head = Detect(nc, anchors,ch=[36,36,36],do_quant=False,inplace=True)\n",
    "detect_head.load_state_dict(torch.load(\"experiment_models/detect_module.pt\"))\n",
    "detect_head.cuda()\n",
    "detect_head.eval()\n",
    "detect_head.training = False\n",
    "detect_head.stride = [32,32,32]\n",
    "\n",
    "\n",
    "image = cv2.imread(\"data/images/zidane.jpg\")\n",
    "image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "image = letterbox(image, (384,640), auto=False)[0]\n",
    "\n",
    "image = image.astype(np.uint8)\n",
    "image = np.expand_dims(image, 0)\n",
    "image = np.transpose(image, (0,3,1,2))\n",
    "image = torch.from_numpy(image).to(\"cuda\").float()\n",
    "\n",
    "model_no_detect.eval().cuda()\n",
    "model_no_detect_out = model_no_detect(image).value.detach().to(\"cuda\")\n",
    "\n",
    "pred = detect_head([model_no_detect_out,model_no_detect_out,model_no_detect_out])\n",
    "print(pred[0].shape)\n",
    "pred = pred[0].detach()\n",
    "\n",
    "detections = non_max_suppression(pred,conf_thres = 0.30, iou_thres=0.45)\n",
    "print(detections)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test Detect seperated model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "detect_head.m[0](model_no_detect_out).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_q.model.model.model[15].m[0].weight.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_no_detect[0].conv.out_channels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_q.model.model.model[15].conv.int_weight()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_q.model.model.model[15].m[0].quant_bias()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clean Coco Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "def count_files_in_directory(directory_path):\n",
    "    # Get the list of files in the directory\n",
    "    files = os.listdir(directory_path)\n",
    "\n",
    "    # Count the number of files\n",
    "    file_count = len(files)\n",
    "\n",
    "    return file_count\n",
    "\n",
    "\n",
    "train_img_path = \"../datasets/coco128/images/train2017\"  \n",
    "train_label_path = \"../datasets/coco128/labels/train2017\" \n",
    "train_img_dir = os.listdir(train_img_path)\n",
    "train_label_dir = os.listdir(train_label_path)\n",
    "\n",
    "val_img_path = \"../datasets/coco128/images/val2017\"  \n",
    "val_label_path = \"../datasets/coco128/labels/val2017\" \n",
    "val_img_dir = os.listdir(val_img_path)\n",
    "val_label_dir = os.listdir(val_label_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_img_len = count_files_in_directory(train_img_path)\n",
    "train_label_len = count_files_in_directory(train_label_path)\n",
    "\n",
    "val_img_len = count_files_in_directory(val_img_path)\n",
    "val_label_len = count_files_in_directory(val_label_path)\n",
    "\n",
    "print(train_img_len, train_label_len)\n",
    "print(val_img_len, val_label_len)\n",
    "\n",
    "train_img_files = [img_path.replace(\".jpg\",\".txt\") for img_path in train_img_dir]\n",
    "\n",
    "train_imgs_to_delete = list(set(train_img_files) ^ set(train_label_dir))\n",
    "train_imgs_to_delete = [img_path.replace(\".txt\",\".jpg\") for img_path in train_imgs_to_delete]\n",
    "\n",
    "for train_img_to_delete in train_imgs_to_delete:\n",
    "    path = train_img_path + \"/\" + train_img_to_delete\n",
    "    if os.path.exists(path):\n",
    "        os.remove(path)\n",
    "    \n",
    "print(len(os.listdir(train_img_path)))\n",
    "\n",
    "val_img_files = [img_path.replace(\".jpg\",\".txt\") for img_path in val_img_dir]\n",
    "\n",
    "val_imgs_to_delete = list(set(val_img_files) ^ set(val_label_dir))\n",
    "val_imgs_to_delete = [img_path.replace(\".txt\",\".jpg\") for img_path in val_imgs_to_delete]\n",
    "\n",
    "for val_img_to_delete in val_imgs_to_delete:\n",
    "    path = val_img_path + \"/\" + val_img_to_delete\n",
    "    if os.path.exists(path):\n",
    "        os.remove(path)\n",
    "    \n",
    "print(len(os.listdir(val_img_path)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "idx = 2\n",
    "\n",
    "image_path = val_img_dir[idx]\n",
    "print(image_path)\n",
    "print(train_img_path)\n",
    "\n",
    "img = cv2.imread(val_img_path + \"/\" + image_path)\n",
    "img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
    "img = cv2.resize(img,(380,640))\n",
    "\n",
    "plt.imshow(img)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_path = val_label_path + \"/000000177934.txt\"\n",
    "with open(label_path, 'r') as file:\n",
    "    # Read the content of the file\n",
    "    file_content = file.read()\n",
    "\n",
    "    # Print the content\n",
    "    print(file_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clean VOC data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "\n",
    "directory_path = \"../datasets/VOC/labels/*\"\n",
    "label_files = glob.glob(f'{directory_path}/*.txt')\n",
    "\n",
    "\n",
    "count = 0\n",
    "for label_file in label_files:\n",
    "  with open(label_file, 'r') as file:\n",
    "    # Read the entire content of the file\n",
    "    file_content = file.read()\n",
    "    if not file_content:\n",
    "      count += 1\n",
    "      print(f\"label file {label_file} is empty\")\n",
    "\n",
    "      train_file = label_file.replace(\"/labels/\",\"/images/\")\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "YOLOv5 Tutorial",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}