Video Object Detection with YOLOv3 (OpenCV).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Video Object Detection with YOLOv3 (OpenCV library)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will show step by step how to detect objects in a video (i.e. a real-time video from a highway camera).   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Libraries\n",
    "\n",
    "First step, we need to import OpenCV, Numpy and Matplotlib libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2 \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import Video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video\n",
    "\n",
    "Let's have a look at the video.\n",
    "\n",
    "Link (click on \"download\" and watch it):   [https://github.com/buropas/Object_Detection/blob/main/test.mp4](https://github.com/buropas/Object_Detection/blob/main/test.mp4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"test.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Take a look at the input video\n",
    "Video(\"test.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Model Configuration and Pre-trained Weights\n",
    "\n",
    "Yolov3 pre-trained weights can be downloaded here: https://pjreddie.com/media/files/yolov3.weights.\n",
    "\n",
    "Let's load Yolov3 model configuration and pre-trained weights by reading \"yolov3.cfg\" and yolov3.weights\".\n",
    "We also need to extract class names from the file \"coco.names\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "YOLOv3 LOADED SUCCESSFULLY\n"
     ]
    }
   ],
   "source": [
    "## Loading network configuration and pre-trained weights \n",
    "net = cv2.dnn.readNet(\"yolov3.weights\", \"/home/pasquale/OBJ_DETECTION/darknet/cfg/yolov3.cfg\") \n",
    "\n",
    "## Save all the class names in a list  (80 CLASSES)\n",
    "with open(\"/home/pasquale/OBJ_DETECTION/darknet/data/coco.names\", \"r\") as f:     \n",
    "    classes = [word.strip() for word in f.readlines()] \n",
    "    \n",
    "## Get layer names of the network \n",
    "layer_names = net.getLayerNames() \n",
    "\n",
    "## Determine the output layer names from the YOLO model  \n",
    "# (net.getUnconnectedOutLayers() gives the index position of the layers)\n",
    "output_layers = [layer_names[i[0] - 1] for i in net.getUnconnectedOutLayers()] \n",
    "\n",
    "print(\"YOLOv3 LOADED SUCCESSFULLY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is all we need in order to load the model configuration and pre-trained weights.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Capturing video \n",
    "Next step is to load the video and define the VideoWriter object in order to save our final video with detected objects. The output video will be saved as \"output.avi\".\n",
    "\n",
    "In the VideoWriter object we specify:\n",
    "- the output file name (output.avi), \n",
    "- the FourCC code (a 4-byte code used to specify the video codec), \n",
    "- the number of frames per second (fps), \n",
    "- the frame size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Loading video \n",
    "filename = \"test.mp4\"                         # filename\n",
    "cap = cv2.VideoCapture(filename)              # loading video\n",
    "\n",
    "# We get the resolution of our video (width and height) and we convert from float to integer\n",
    "frame_width = int(cap.get(3))\n",
    "frame_height = int(cap.get(4))\n",
    "\n",
    "# We create VideoWriter object and define the codec. The output is stored in 'output.avi' file.\n",
    "out_video = cv2.VideoWriter(\"output.avi\",                                # output name\n",
    "                            cv2.VideoWriter_fourcc('M', 'J', 'P', 'G'),  # 4-byte code used to specify the video codec\n",
    "                                                                         # (we pass MJPG)\n",
    "                            15,                                          # number of frames per second (fps) \n",
    "                            (frame_width, frame_height)                  # frame size\n",
    "                            )\n",
    "\n",
    "# set font and color of text and bounding boxes\n",
    "font = cv2.FONT_HERSHEY_PLAIN     # font\n",
    "color = (0, 255, 0)               # green color"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Object Detection\n",
    "\n",
    "The goal is to perform object detection only in a specific region (RoI - Region of Interest) of the video: we want to detect the incoming vehicles in the right part of the video.     \n",
    "So, the idea is that we process the video frame by frame.   \n",
    "For each frame:\n",
    "- we select the region of interest, preprocess it and pass it as input into the network,\n",
    "- we obtain detected objects in the region of interest as output of our network,\n",
    "- we discard objects detected with a confidence score lower than a specific threshold and we perform Non-maximum Suppression (NMS), which is a technique to filter the predictions,\n",
    "- we show detected objects with bounding boxes, classes and confidence scores.      \n",
    "\n",
    "At the end, we save the video with detected objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "while cap.isOpened():     # while the capture is correctly initialized...\n",
    "\n",
    "    # We process the video frame-by-frame\n",
    "    \n",
    "    ret, img = cap.read()           # we read each frame (img) from the video\n",
    "                                    # we also retrieve ret, which is a boolean value. \n",
    "                                    # ret is True if the frame is read correctly\n",
    "    \n",
    "    if ret == True:    # if the frame is read correctly, go on...\n",
    "        \n",
    "    \n",
    "        ## EXTRACT REGION OF INTEREST(ROI)\n",
    "        roi = img[120:, 450:]           # consider only a slice in pixels of the entire frame \n",
    "\n",
    "        height, width, _ = roi.shape    # retrieve height and width from the region of interest\n",
    "                                        # (we need height and width to build bounding boxes later)\n",
    "\n",
    "        ## IMAGE PREPROCESSING\n",
    "        # The cv2.dnn.blobFromImage function returns a blob which is our input image after\n",
    "        # scaling by a scale factor, and channel swapping.\n",
    "        # The input image that we need to pass to the Yolo algorithm must be 416x416\n",
    "        blob = cv2.dnn.blobFromImage(roi, 1/255.0, (416,416), (0,0,0), swapRB=True, crop= False)\n",
    "\n",
    "        ## OBJECT DETECTION\n",
    "        net.setInput(blob)                      # set blob as input to the network\n",
    "        outs = net.forward(output_layers)       # runs a forward pass to compute the network output  \n",
    "\n",
    "\n",
    "        boxes = []\n",
    "        confidences = []\n",
    "        class_ids = []\n",
    "\n",
    "        for out in outs:            # for each output...\n",
    "            for detection in out:               # for each detection...\n",
    "                scores = detection[5:]             # array with 80 scores (1 score for each class)\n",
    "                class_id = np.argmax(scores)       # take the id of the maximum score\n",
    "                confidence = scores[class_id]      # confidence of the class with the maximum score\n",
    "\n",
    "                if confidence > 0.5:    # if the confidence of the detected object is above the threshold\n",
    "                                        # we start to create the bounding box...\n",
    "                    # Object detected\n",
    "                    center_x = int(detection[0] * width)             # x of the center point\n",
    "                    center_y = int(detection[1] * height)            # y of the center point\n",
    "                    w = int(detection[2] * width)                    # width of the detected object\n",
    "                    h = int(detection[3] * height)                   # height of the detected object\n",
    "\n",
    "                    # Rectangle coordinates\n",
    "                    x = int(center_x - w / 2)                         # x of the top left point\n",
    "                    y = int(center_y - h / 2)                         # y of the top left point\n",
    "\n",
    "                    boxes.append([x, y, w, h])\n",
    "                    confidences.append(float(confidence))\n",
    "                    class_ids.append(class_id)\n",
    "\n",
    "\n",
    "        ## NMS - NON-MAXIMUM SUPPRESSION\n",
    "        # We use NMS function in opencv to perform Non-maximum Suppression  \n",
    "        # The function performs non maximum suppression, given boxes and corresponding confidence scores\n",
    "        # We give it score threshold and nms threshold as arguments:\n",
    "        # score_threshold: keep only boxes with a confidence score higher than the threshold\n",
    "        # nms threshold: threshold used in non maximum suppression (IoU)\n",
    "        # The function returns indices of bounding boxes survived after NMS.\n",
    "        indexes = cv2.dnn.NMSBoxes(boxes, confidences, score_threshold = 0.5, nms_threshold = 0.4)\n",
    "\n",
    "\n",
    "        ## DRAW IMAGE WITH BOUNDING BOXES, CLASSES AND CONFIDENCE SCORES\n",
    "        for i in range(len(boxes)):              # for each box...\n",
    "            if i in indexes:                     # if the bounding box has survived after NMS... \n",
    "\n",
    "                x,y,w,h = boxes[i]                                      # bounding box coordinates\n",
    "                label = str(classes[class_ids[i]])                      # class label\n",
    "                confidence = str(round(confidences[i], 2))              # confidence score\n",
    "                cv2.rectangle(roi, (x, y), (x + w, y + h), color, 2)    # drawing rectangular bounding box\n",
    "                                                                # (x,y) is the top left corner of the box\n",
    "                                                                # (x + w, y + h) is the bottom right corner of the box\n",
    "                cv2.putText(roi, label + \" \" + confidence, (x, y - 5), font, 0.8, color, 2)   # text of the box \n",
    "            \n",
    "        cv2.imshow(\"out\", img)      # display the current frame with detected objects  \n",
    "\n",
    "        out_video.write(img)        # the frame is saved for the final video\n",
    "\n",
    "        key = cv2.waitKey(1)        # wait 1 millisecond between each frame\n",
    "        if key == 27:               # if exit button, break and close\n",
    "            break\n",
    "    \n",
    "    \n",
    "    else:   # if the frame is not read correctly, break...\n",
    "        break\n",
    "    \n",
    "# Release everything when job is finished\n",
    "cap.release()\n",
    "out_video.release()\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's have a look at the final result..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video with Detected Objects\n",
    "\n",
    "Link to Output Video (click on \"download\" and just watch or download it):   \n",
    "\n",
    "[https://github.com/buropas/Object_Detection/blob/main/output.avi](https://github.com/buropas/Object_Detection/blob/main/output.avi)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}