Self-driving-car

This project includes two tasks :

1. Predict the speed of a car from a video, captured from dashboard cam. ( a challenge released by Comma.ai )

2. Real time "Car" (object) detection using state of the art "SSD (Single Shot MultiBox Detector)"

Sample Output

1) Predict the speed of a car from a video,

Challenge Description from Comma.ai

""Basically, your goal is to predict the speed of a car from a video. data/train.mp4 is a video of driving containing 20400 frames. Video is shot at 20 fps. data/train.txt contains the speed of the car at each frame, one speed on each line.

data/test.mp4 is a different driving video containing 10798 frames. Video is shot at 20 fps. Your deliverable is test.txt

We will evaluate your test.txt using mean squared error. <10 is good. <5 is better. <3 is heart""

Dataset can be downloaded from Comma.ai speed challenge

Methodology and Model used

First Attempt

The first impression was to use the images/frames (converted from the video) and feed into some CNN models to map with the numbers (speed) . However, after a few trials, it was clear that single images can't be used to predict speed as single images have no features that can be related with speed.

Second Attempt

The second attempt( and a successful one :) ) was to estimate the speed using two frames. I used optical Optical flow to get the pattern of apparent motion of objects between two consecutive frames caused by the movemement of object. Clearly the movement of objects is more when the speed is high and less when the speed is low. Hence, Optical flow seems to be the perfect choice in order to predict speed.

Architecture used

The architecture used is Nvidia's "End to End Learning for Self-Driving Cars archtecture". I have incorporated a few changes in the architecture to improve the results. The implementation and training is done using Pytorch framework.

Results

MSE ~3.0 on training dataset and 4.0 on validation

Plot of actual speed vs predicted speed

2) Real time object detection using MobileNetSSD (Single Shot MultiBox Detector)**

A caffe implementation of MobileNet-SSD detection network, with pretrained weights for object (car) detection. OpenCV comes up with deep neural network module(DNN), which helps to use pretrained model from deep learning framework such Tensorflow, Caffe, Torch.

For the purpose of object tetection, the pre trained model is used "Mobile-SSD

Training and Validation of Task 1- Prediction of the speed

Please refer the comma.ipynb file for the traning and validation.

Play the output in opencv

1. Clone or download the comma_play.py and train_predict.txt
2. Get the training datasets from Comma.ai speed challenge. Specifically - train.mp4 and train.txt
3. Make sure numpy, argparse, and cv2 are installed.
4. Go to the same folder in commandline and run "python comma_play.py --video train.mp4"