Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
190 lines (138 sloc) 6.2 KB

Udacity Self Driving Car Challenge 2 - Using Deep Learning to Predict Steering Angles

2nd place solution for the Udacity Self Driving Car Challenge 2.

The overall goal of the Udacity Self-Driving Car Challenge is to build an open-source self-driving vehicle. This goal was broken into different challenges. In Challenge 2 your program had to predict steering commands from camera imagery.


The first phase of the challenge was based on driving data from El Camino Real (small curves, mostly straight driving), and the second phase was based on driving data from San Mateo to Half Moon Bay (curvy and highway driving).

The phase 1 data set has PNG images. The phase 2 data set is in the ROSBAG format. We used rwightman's Udacity Reader docker tool to convert the images into PNGs.

Even though the imagery is from three different cameras (left, center, right), we only used the center images.

The code assumes the following directory structure for data:

- models

- round1
-- train
--- 1 
---- center
----- 1477431483393873340.png
----- ...
----- 1477431802438024821.png
--- 2
---- center
----- 1477429515920298589.png
----- ...
----- 1477429548524716563.png
--- 21
---- center
----- 1477436619607286002.png
----- ...
----- 1477436971856447647.png
-- test
--- center
---- 1477439402646429224.png
---- ....
---- 1477439732692922522.png

- round2
-- train
--- center
---- 1479424215880976321.jpg
---- ...
---- 1479426572343447996.jpg
-- test
--- center
---- 1479425441182877835.jpg
---- ...
---- 1479425721881751009.jpg

Change data_path value in to point to this data directory.


The raw images are of size 640 x 480. In our final model, we resized the images to 256 x 192, converted from RGB color format to grayscale, computed lag 1 differences between frames and used 2 consecutive differenced images. For example, at time t we used [x_{t} - x_{t-1}, x_{t-1} - x_{t-2}] as input where x corresponds to the grayscale image. No future frames were used to predict the current steering angle.

To pre-process phase 2 training data, run:


To pre-process phase 2 test data, run:


These pre-processing scripts convert image sets to numpy arrays.


Our final model consisted of 3 streams that we merged at the final layer. Two of the streams were inspired by the NVIDIA's self-driving car paper, and one of the streams was inspired by’s steering model.

We were surprised that many tricks that work well with classification networks did not transfer over to this regression problem. For example, we did not use dropout, batch normalization or VGG-style 3x3 filters. It was hard to get the model to predict something other than the average value. Of course, there might have been a problem in the hyperparameter selection.

To train different models, run:


You can change these parameters in the file:

  • --data - alias for pre-processed data. There are multiple ways to pre-process the data (how many consecutive frames to use, image size, etc). This parameter value gives us information what data set to use.
  • --num_channels - number of channels the data has. For example, if you use 4 consecutive frames, then num_channels must be 4.
  • --img_height - image height in pixels, default is 192.
  • --img_width - image width in pixels, default is 256.
  • --model - model definition file, see for different models.
  • --test_part - which part of the data to use as validation set.
  • --batch_size - minibatch size, default is 32.
  • --num_epoch - number epochs to train, default is 10.
  • --data_path - folder path to pre-processed numpy arrays.

To predict steering angles from test data, run:


Online predictions and reproducing our results

We have included model checkpoint and 10 sample images from the test set to show how to get real-time predictions.

When you run


you should get the following output:

[ 0.00233838]

Inspecting the model

  • Automatic report

To get an automatic report with predicted steering angle distributions and error visualizations, run the following R script:

Rscript render_reports.R

You might want to change variables submission_filename, img_path, output_filename in the render_reports.R file.

  • Visualizing predicted steering angles

To visualize model predictions on test data, run:


White circle shows the true angle, black circle shows the predicted angle. You might need to change the variable VisualizeConfig in to point to the location of phase 2 images.

These visualizations can help us understand the weaknesses of the model. For example, human steering movements are smoother on straight road while the model zig-zags.

Pointers and Acknowledgements