OpenCV Python Neural Network Autonomous RC Car
Switch branches/tags
Nothing to show
Clone or download
Latest commit b7392f0 Aug 10, 2018


Python3 + OpenCV3

See self-driving in action

This project builds a self-driving RC car using Raspberry Pi, Arduino and open source software. Raspberry Pi collects inputs from a camera module and an ultrasonic sensor, and sends data to a computer wirelessly. The computer processes input images and sensor data for object detection (stop sign and traffic light) and collision avoidance respectively. A neural network model runs on computer and makes predictions for steering based on input images. Predictions are then sent to the Arduino for RC car control.

Setting up environment with Anaconda

  1. Install miniconda on your computer

  2. Create auto-rccar environment with all necessary libraries for this project
    conda env create -f environment.yml

  3. Activate auto-rccar environment
    source activate auto-rccar

  To exit, simply close the terminal window. More info about managing Anaconda environment, please see here.

About the files

test/ RC car control with keyboard video streaming from Pi to computer sensor data streaming from Pi to computer
        data_test.npz: sample data
        train_predict_test.ipynb: a jupyter notebook that goes through neural network model in OpenCV3

raspberryPi/ stream video frames in jpeg format to the host computer send distance data measured by sensor to the host computer

    rc_keyboard_control.ino: control RC car controller

        trained cascade classifiers
        images for calibration, captured by pi camera pi camera calibration collect images in grayscale, data saved as *.npz neural network model model training and validation helper classes/functions for receive data from raspberry pi and drive the RC car based on model prediction

    trafic signal sketch contributed by @geek111

How to drive

  1. Testing: Flash rc_keyboard_control.ino to Arduino and run to drive the RC car with keyboard. Run on computer and then run on raspberry pi to test video streaming. Similarly, and can be used for sensor data streaming testing.

  2. Pi Camera calibration (optional): Take multiple chess board images using pi camera module at various angles and put them into chess_board folder, run and returned parameters from the camera matrix will be used in

  3. Collect training/validation data: First run and then run on raspberry pi. Press arrow keys to drive the RC car, press q to exit. Frames are saved only when there is a key press action. Once exit, data will be saved into newly created training_data folder.

  4. Neural network training: Run to train a neural network model. Please feel free to tune the model architecture/parameters to achieve a better result. After training, model will be saved into newly created saved_model folder.

  5. Cascade classifiers training (optional): Trained stop sign and traffic light classifiers are included in the cascade_xml folder, if you are interested in training your own classifiers, please refer to OpenCV doc and this great tutorial.

  6. Self-driving in action: First run to start the server on the computer, and then run and on raspberry pi.

中文文档 (感谢zhaoying9105)