Skip to content
Makefile C++ CMake Python Jupyter Notebook Shell
Branch: master
Clone or download
NeilNie trying to address issue #15
Changed the CMakeLists file for the mapping package in an attempt to address issue #15.
Latest commit 0562db7 Oct 11, 2019


Build Status codebeat badge Documentation Status License: MIT Donate

Welcome! This is an open source self-driving development platform aimed for rapid prototyping, deep learning, and robotics research. The system currently runs on a modified electric golf cart. Here are our goals:


Research and develop a deep learning-driven self-driving car. The vehicle should be able to achieve level 4 autonomy within a geofenced area.

The modules in this project.

  1. End-to-End Steering (Behavioral cloning)
  2. Semantic Segmentation
  3. Object Detection πŸš™
  4. Drive by Wire (DBW)
  5. CARLA simulator integration
  6. ZED stereoscopic vision system
  7. Mapping with rtabamp
  8. Path planning with ROS nav stack.
  9. Localization, pose tracking, and odom with ZED and rtabmap.

For the full documentation of the development process, please visit my website:

Table of Content

Try it out

Before you jump in, let me describe the hardware requirement for this project. A webcam is the minimum requirment. At this point, you can only run the whole system on the actual self-driving vehicle. ROS will throw warnings (even errors) at you if you don't have the hardware connected to your Linux machine. If you don't have access to the hardware setup, don't worry πŸ‘‡

  • The best way is to download and play back the ROS bags. [coming soon...]
  • You can tryout individual packages and nodes, and might find them helpful for your own projects.
  • You can also tryout the CARLA simulator. (Maybe even improve the current system.)

To compile the project:

  1. Make sure that you have ROS installed on your computer. (I am using ROS Melodic)
  2. Make sure you have all the dependencies installed.
Clone & Compile
  1. Clone the repository. $ git clone
  2. $ cd self-driving-golf-cart/ros
  3. $ catkin_make
  4. $ source devel/setup.bash
Launch ZED cameras
  • $ roslaunch zed_wrapper zed.launch (no rviz)
  • $ roslaunch zed_display display.launch (with rviz)
Launch the Navigation Stack
  • $ roslaunch path_planning rtab_mapping_navigation.launch


πŸš™ Bon Voyage πŸ˜€

About ROS

This project uses ROS. For more information on ROS, nodes, topics and others please refer to the ROS README.


(πŸ— Construction Zone 🚧)

Building a self-driving car is hard. Not everyone has access to expensive hardware. I am currently trying to integrate this project with the CARLA self-driving simulator. If you are interested in CARLA, please refer to this documentation. The ROS system in this project can partially run on the CARLA simulator.

If you want to try out the simulator, please refer to the documentation here.


Autopilot & End-to-End Behavioral Cloning

The autopilot system, found here in the autopilot node, uses deep learning to predict the steering commands and acceleration commands for the vehicle, only using data collected by the front facing camera.

What's Behavioral Cloning

In 2016, NVIDIA proposed a novel deep learning approach allowed their car to accurately perform real-time end-to-end steering command prediction. Around the same time, Udacity held a challenge that asked researchers to create the best end-to-end steering prediction model. Our goal is to further the work in behavioral cloning for self-driving vehicles.


NVIDIA's paper used a convolutional neural network with a single frame input. I believe that the single-frame-input CNN doesn't provide any temporal information which is critical in self-driving. This is the motive behind choosing the i3d architecture, which is rich in spacial-temporal information.

The input of the network is a 3d convolutional block, with the shape of n * weight * height * 3. n is the length of the input sequence. A flatten layer and a dense layer are added to the back of the network for the purpose of this regression problem.


Here is a video demo of deep learning model running on the autonomous golf cart.


Semantic Segmentation

The cart understands its surrounding through semantic segmentation, which is a technique in computer that classifies each pixel in an image into different categories. The vehicle can also make decisions based on the segmentic segmentation results. The cart can change its speed based on the proximity to nearby obstacles.


We deployed the ENet architecture for segmentation. ENet is design to work well in realtime applications. For more information, please visit the paper. We used the CityScape dataset for training and the python code for training and inferencing are located in the ./src/segmentation/scripts directory.


The Navigation Stack

The self-driving vehicle uses a modified version of the ROS navigation stack. The flowchart above illustrate the mapping and path planning process. First, I create a detailed map of the environment with rtabmap_ros. With that global map, I use the localization feature of rtabmap_ros and the odom feature of the zed camera system to localize and plan paths.


rtabmap (realtime appearance based mapping) allows me to construct a global map of the environment. For more information on the mapping package, please check out this .launch file.


Path Planning

The project uses the move_base node from the navigation stack. The image below shows the costmap (in blue and purple), and the global occupancy grid (in black and gray). move_base also plans the local and global path. Global paths are shown in green and yellow below. You can find the yaml files here.


Vehicle Motion Control

The move base node publishes /cmd_vel commands, which are processed and sent directly to the vehicle.


Contact / Info

If you are interested in the detailed development process of this project, you can visit Neil's blog at to find out more about it. Neil will make sure to keep you posted about all of the latest development on the club.



Neil (Yongyang) Nie | Email | Github | Website | Linkedin


Michael Meng | Email | Github

You can’t perform that action at this time.