Welcome! This is an open source self-driving development platform aimed for rapid prototyping, deep learning, and robotics research. The system currently runs on a modified electric golf cart. Here are our goals:
Research and develop a deep learning-driven self-driving car. The vehicle should be able to achieve level 4 autonomy within a geofenced area.
The modules in this project.
- End-to-End Steering (Behavioral cloning)
- Semantic Segmentation
- Object Detection
- Drive by Wire (DBW)
- CARLA simulator integration
- ZED stereoscopic vision system
- Mapping with rtabamp
- Path planning with ROS nav stack.
- Localization, pose tracking, and odom with ZED and rtabmap.
For the full documentation of the development process, please visit my website: www.neilnie.com
Table of Content
- Try it out
- About ROS
- Autopilot & End-to-End Behavioral Cloning
- Semantic Segmentation
- The Navigation Stack
Try it out
Before you jump in, let me describe the hardware requirement for this project. A webcam is the minimum requirment. At this point, you can only run the whole system on the actual self-driving vehicle. ROS will throw warnings (even errors) at you if you don't have the hardware connected to your Linux machine. If you don't have access to the hardware setup, don't worry
- The best way is to download and play back the ROS bags. [coming soon...]
- You can tryout individual packages and nodes, and might find them helpful for your own projects.
- You can also tryout the CARLA simulator. (Maybe even improve the current system.)
To compile the project:
- Make sure that you have ROS installed on your computer. (I am using ROS Melodic)
- Make sure you have all the dependencies installed.
Clone & Compile
- Clone the repository.
$ git clone https://github.com/sigmaai/self-driving-golf-cart.git
$ cd self-driving-golf-cart/ros
$ source devel/setup.bash
Launch ZED cameras
$ roslaunch zed_wrapper zed.launch(no rviz)
$ roslaunch zed_display display.launch(with rviz)
Launch the Navigation Stack
$ roslaunch path_planning rtab_mapping_navigation.launch
This project uses ROS. For more information on ROS, nodes, topics and others please refer to the ROS README.
Building a self-driving car is hard. Not everyone has access to expensive hardware. I am currently trying to integrate this project with the CARLA self-driving simulator. If you are interested in CARLA, please refer to this documentation. The ROS system in this project can partially run on the CARLA simulator.
If you want to try out the simulator, please refer to the documentation here.
Autopilot & End-to-End Behavioral Cloning
The autopilot system, found here in the autopilot node, uses deep learning to predict the steering commands and acceleration commands for the vehicle, only using data collected by the front facing camera.
What's Behavioral Cloning
In 2016, NVIDIA proposed a novel deep learning approach allowed their car to accurately perform real-time end-to-end steering command prediction. Around the same time, Udacity held a challenge that asked researchers to create the best end-to-end steering prediction model. Our goal is to further the work in behavioral cloning for self-driving vehicles.
NVIDIA's paper used a convolutional neural network with a single frame input. I believe that the single-frame-input CNN doesn't provide any temporal information which is critical in self-driving. This is the motive behind choosing the i3d architecture, which is rich in spacial-temporal information.
The input of the network is a 3d convolutional block, with the shape of
n * weight * height * 3.
n is the length of the input sequence. A flatten layer and a dense layer are added to the back of the network for the purpose of this regression problem.
Here is a video demo of deep learning model running on the autonomous golf cart.
The cart understands its surrounding through semantic segmentation, which is a technique in computer that classifies each pixel in an image into different categories. The vehicle can also make decisions based on the segmentic segmentation results. The cart can change its speed based on the proximity to nearby obstacles.
We deployed the ENet architecture for segmentation. ENet is design to work well in realtime applications. For more information, please visit the paper. We used the CityScape dataset for training and the python code for training and inferencing are located in the
The Navigation Stack
The self-driving vehicle uses a modified version of the ROS navigation stack. The flowchart above illustrate the mapping and path planning process. First, I create a detailed map of the environment with
rtabmap_ros. With that global map, I use the localization feature of
rtabmap_ros and the odom feature of the zed camera system to localize and plan paths.
rtabmap (realtime appearance based mapping) allows me to construct a global map of the environment. For more information on the mapping package, please check out this
The project uses the
move_base node from the navigation stack. The image below shows the costmap (in blue and purple), and the global occupancy grid (in black and gray).
move_base also plans the local and global path. Global paths are shown in green and yellow below. You can find the
yaml files here.
Vehicle Motion Control
The move base node publishes
/cmd_vel commands, which are processed and sent directly to the vehicle.
Contact / Info
If you are interested in the detailed development process of this project, you can visit Neil's blog at neilnie.com to find out more about it. Neil will make sure to keep you posted about all of the latest development on the club.