Video Object Detection and Tracking with YOLOv3 (OpenCV).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Video Object Detection and Tracking with YOLOv3 (OpenCV library)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will show step by step how to detect and track objects in a video (i.e. a real-time video from a highway camera).   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Libraries\n",
    "\n",
    "First step, we need to import OpenCV, Numpy, Matplotlib libraries and Centroid Tracker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2 \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import Video\n",
    "from centroidtracker import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video\n",
    "\n",
    "Let's have a look at the video.\n",
    "\n",
    "Link (click on \"download\" and watch it):   [https://github.com/buropas/Object_Detection/blob/main/test.mp4](https://github.com/buropas/Object_Detection/blob/main/test.mp4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"test.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Take a look at the input video\n",
    "Video(\"test.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Model Configuration and Pre-trained Weights\n",
    "\n",
    "Yolov3 pre-trained weights can be downloaded here: https://pjreddie.com/media/files/yolov3.weights.\n",
    "\n",
    "Let's load Yolov3 model configuration and pre-trained weights by reading \"yolov3.cfg\" and yolov3.weights\".\n",
    "We also need to extract class names from the file \"coco.names\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "YOLOv3 LOADED SUCCESSFULLY\n"
     ]
    }
   ],
   "source": [
    "## Loading network configuration and pre-trained weights \n",
    "net = cv2.dnn.readNet(\"yolov3.weights\", \"yolov3.cfg\") \n",
    "\n",
    "## Save all the class names in a list  (80 CLASSES)\n",
    "with open(\"coco.names\", \"r\") as f:     \n",
    "    classes = [word.strip() for word in f.readlines()] \n",
    "    \n",
    "## Get layer names of the network \n",
    "layer_names = net.getLayerNames() \n",
    "\n",
    "## Determine the output layer names from the YOLO model  \n",
    "# (net.getUnconnectedOutLayers() gives the index position of the layers)\n",
    "output_layers = [layer_names[i[0] - 1] for i in net.getUnconnectedOutLayers()] \n",
    "\n",
    "print(\"YOLOv3 LOADED SUCCESSFULLY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is all we need in order to load the model configuration and pre-trained weights.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Capturing video \n",
    "Next step is to load the video and define the VideoWriter object in order to save our final video with detected objects. The output video will be saved as \"output.avi\".\n",
    "\n",
    "In the VideoWriter object we specify:\n",
    "- the output file name (output.avi), \n",
    "- the FourCC code (a 4-byte code used to specify the video codec), \n",
    "- the number of frames per second (fps), \n",
    "- the frame size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Loading video \n",
    "filename = \"test.mp4\"                         # filename\n",
    "cap = cv2.VideoCapture(filename)              # loading video\n",
    "\n",
    "# We get the resolution of our video (width and height) and we convert from float to integer\n",
    "frame_width = int(cap.get(3))\n",
    "frame_height = int(cap.get(4))\n",
    "\n",
    "# We create VideoWriter object and define the codec. The output is stored in 'output.avi' file.\n",
    "out_video = cv2.VideoWriter(\"output_track.avi\",                                # output name\n",
    "                            cv2.VideoWriter_fourcc('M', 'J', 'P', 'G'),  # 4-byte code used to specify the video codec\n",
    "                                                                         # (we pass MJPG)\n",
    "                            15,                                          # number of frames per second (fps) \n",
    "                            (frame_width, frame_height)                  # frame size\n",
    "                            )\n",
    "\n",
    "# set font and color of text and bounding boxes\n",
    "font = cv2.FONT_HERSHEY_PLAIN     # font\n",
    "color_g = (0, 255, 0)             # green color\n",
    "color_r = (0, 0, 255)             # red color"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Object Detection + Object Tracking\n",
    "\n",
    "The goal is to perform object detection and tracking only in a specific region (RoI - Region of Interest) of the video: we want to detect and track the incoming vehicles in the right part of the video.     \n",
    "So, the idea is that we process the video frame by frame.   \n",
    "For each frame:\n",
    "- we select the region of interest, preprocess it and pass it as input into the network,\n",
    "- we obtain detected objects in the region of interest as output of our network,\n",
    "- we discard objects detected with a confidence score lower than a specific threshold and we perform Non-maximum Suppression (NMS), which is a technique to filter the predictions,\n",
    "- we show detected objects with bounding boxes and classes,\n",
    "- furthermore, we track each detected object using a Centroid-based tracking algorithm.   \n",
    " The algorithm assigns a unique ID to the detected object and stores the centroid of the bounding box coordinates for that object.   \n",
    " Then, the algorithm updates IDs list when new vehicles are detected or old ones are removed and updates centroid coordinates as the vehicles move around frames. \n",
    "\n",
    "At the end, we save the video with detected and tracked objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Centroid-based Tracking\n",
    "\n",
    "Given an initial set of bounding box coordinates for each detected objects, Object Tracking consists in assigning a unique ID for each of the initial detections and then tracking each of the objects as they move around frames in a video, maintaining the assignment of unique IDs.\n",
    "\n",
    "In this project, we perform Object Tracking using a Centroid-based tracking algorithm.\n",
    "\n",
    "The Centroid Tracking algorithm requires a set of bounding box produced by the object detector for each detected object in every single frame.\n",
    "\n",
    "Once we have the bounding box coordinates we must compute the “centroid”, i.e. the x, y coordinates of the center point in the bounding box.\n",
    "\n",
    "\n",
    "The Centroid Tracking algorithm is a tracking algorithm  that relies on the Euclidean distance between existing object centroids (i.e., objects the centroid tracker has already seen before) and new object centroids between subsequent frames in a video.\n",
    "\n",
    "The \"maxDisappeared\" hyperparameter refers to the number of consecutive frames an object is allowed to be marked as “lost/disappeared” until the object is deregistered.   In the case that an object leaves the frame and does not come back for \"maxDisappeared\" frames, a new object ID would be assigned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate CentroidTracker for object tracking \n",
    "ct = CentroidTracker(maxDisappeared=8)      # max 8 frames for the object to be lost without being deregistered"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "while cap.isOpened():     # while the capture is correctly initialized...\n",
    "\n",
    "    # We process the video frame-by-frame\n",
    "    \n",
    "    ret, img = cap.read()           # we read each frame (img) from the video\n",
    "                                    # we also retrieve ret, which is a boolean value. \n",
    "                                    # ret is True if the frame is read correctly\n",
    "    \n",
    "    if ret == True:    # if the frame is read correctly, go on...\n",
    "        \n",
    "    \n",
    "        ## EXTRACT REGION OF INTEREST(ROI)\n",
    "        roi = img[210:, 450:]           # consider only a slice in pixels of the entire frame \n",
    "\n",
    "        height, width, _ = roi.shape    # retrieve height and width from the region of interest\n",
    "                                        # (we need height and width to build bounding boxes later)\n",
    "\n",
    "        ## IMAGE PREPROCESSING\n",
    "        # The cv2.dnn.blobFromImage function returns a blob which is our input image after\n",
    "        # scaling by a scale factor, and channel swapping.\n",
    "        # The input image that we need to pass to the Yolo algorithm must be 416x416\n",
    "        blob = cv2.dnn.blobFromImage(roi, 1/255.0, (416,416), (0,0,0), swapRB=True, crop= False)\n",
    "\n",
    "        ######################\n",
    "        ## OBJECT DETECTION ##\n",
    "        ######################\n",
    "        \n",
    "        net.setInput(blob)                      # set blob as input to the network\n",
    "        outs = net.forward(output_layers)       # runs a forward pass to compute the network output  \n",
    "\n",
    "\n",
    "        boxes = []\n",
    "        confidences = []\n",
    "        class_ids = []\n",
    "\n",
    "        for out in outs:            # for each output...\n",
    "            for detection in out:               # for each detection...\n",
    "                scores = detection[5:]             # array with 80 scores (1 score for each class)\n",
    "                class_id = np.argmax(scores)       # take the id of the maximum score\n",
    "                confidence = scores[class_id]      # confidence of the class with the maximum score\n",
    "\n",
    "                if confidence > 0.5:    # if the confidence of the detected object is above the threshold\n",
    "                                        # we start to create the bounding box...\n",
    "                    # Object detected\n",
    "                    center_x = int(detection[0] * width)             # x of the center point\n",
    "                    center_y = int(detection[1] * height)            # y of the center point\n",
    "                    w = int(detection[2] * width)                    # width of the detected object\n",
    "                    h = int(detection[3] * height)                   # height of the detected object\n",
    "\n",
    "                    # Rectangle coordinates\n",
    "                    x = int(center_x - w / 2)                         # x of the top left point\n",
    "                    y = int(center_y - h / 2)                         # y of the top left point\n",
    "\n",
    "                    boxes.append([x, y, w, h])\n",
    "                    confidences.append(float(confidence))\n",
    "                    class_ids.append(class_id)\n",
    "\n",
    "\n",
    "        ## NMS - NON-MAXIMUM SUPPRESSION\n",
    "        # We use NMS function in opencv to perform Non-maximum Suppression  \n",
    "        # The function performs non maximum suppression, given boxes and corresponding confidence scores\n",
    "        # We give it score threshold and nms threshold as arguments:\n",
    "        # score_threshold: keep only boxes with a confidence score higher than the threshold\n",
    "        # nms threshold: threshold used in non maximum suppression (IoU)\n",
    "        # The function returns indices of bounding boxes survived after NMS.\n",
    "        indexes = cv2.dnn.NMSBoxes(boxes, confidences, score_threshold = 0.5, nms_threshold = 0.3)\n",
    "        \n",
    "        \n",
    "        ## Draw bounding boxes of the final detected objects \n",
    "        final_boxes = []\n",
    "        for i in range(len(boxes)):              # for each box...\n",
    "            if i in indexes:                     # if the bounding box has survived after NMS...  \n",
    "        \n",
    "                x, y, w, h = boxes[i]                                      # bounding box coordinates\n",
    "                label = str(classes[class_ids[i]])                         # class label\n",
    "                cv2.rectangle(roi, (x, y), (x + w, y + h), color_g, 2)     # drawing rectangular bounding box\n",
    "                                                                # (x,y) is the top left corner of the box\n",
    "                                                                # (x + w, y + h) is the bottom right corner of the box\n",
    "                cv2.putText(roi, label, (x, y - 5), font, 1, color_r, 2)   # class of the detected object \n",
    "                final_boxes.append([x, y, x+w, y+h])            # append bounding box to the final boxes list\n",
    "\n",
    "        \n",
    "        #####################\n",
    "        ## OBJECT TRACKING ##\n",
    "        #####################\n",
    "        \n",
    "        objects = ct.update(final_boxes)   # we pass the list with final boxes to the tracker in order to have \n",
    "                                           # a unique id for each detection and its specific centroid coordinates\n",
    "\n",
    "        ## Draw the ID of the detected and tracked object\n",
    "        for (objectID, centroid) in objects.items():    # for each tracked object...\n",
    "            text = str(objectID +1)                                # unique id of the tracked object\n",
    "            cv2.putText(roi, text, (centroid[0], centroid[1] -30), font, 2, color_g, 2),   # text\n",
    "            #cv2.circle(roi, (centroid[0], centroid[1]), 4, (0, 255, 0), -1) # draw the centroid of tracked object\n",
    "            \n",
    "    \n",
    "        cv2.imshow(\"out\", roi)      # display the current frame with detected objects  \n",
    "\n",
    "        out_video.write(img)        # the frame is saved for the final video\n",
    "\n",
    "        key = cv2.waitKey(1)        # wait 1 millisecond between each frame\n",
    "        if key == 27:               # if exit button, break and close\n",
    "            break\n",
    "    \n",
    "    \n",
    "    else:   # if the frame is not read correctly, break...\n",
    "        break\n",
    "    \n",
    "# Release everything when job is finished\n",
    "cap.release()\n",
    "out_video.release()\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's have a look at the final result..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video with Detected and Tracked Objects\n",
    "\n",
    "Link to Output Video (click on \"download\" and just watch or download it):   \n",
    "\n",
    "[https://github.com/buropas/Object_Detection/blob/main/output_track.avi](https://github.com/buropas/Object_Detection/blob/main/output_track.avi)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}