Autorace provides hardware and example codes to achieve vision-based autonomous racing on RC-Cars. It is developed by RAM-LAB to support the 1st autonomous RC-Car racing competition in Hong Kong University of Science and Technology (HKUST). The competition data is Feb 26, 2021 (tentative, due to COVID-19).
Event Collaborators: School of Engineering, Robotics Institue (RI), Robotics and Multiperception Lab (RAM-LAB), Intelligent Autonomous Driving Center, Entrepreneurship Center
Keywords: autonomous racing, visual navigation, artifical intelligence, deep learning.
Maintaners: Peide Cai <pcaiaa@connect.ust.hk> Supervisor: Prof. Ming Liu <eelium@ust.hk>
If you meet with any problems, feel free to create an issue to let me and other participants know, then we can solve it together.
If you like the project, give it a star ⭐. It means a lot to the people maintaining it 🧙
- Features
- Build a RC-Car
- System Installation
- Start Your Journey of Self-Driving
- Notes
- Other Useful Toturials
- Credits
-
Coding language: Python3
-
Deep learning framework: PyTorch 1.6
-
On-board computer: Jetson Nano B1
-
Complete pipeline of data collection, model training and testing
-
Easy to use and DIY
-
What can you achieve 🤔?
- Build a small but powerful RC-Car that can drive itself.
- Record driving data (camera images, control actions) by teleoperating the RC-Car.
- Train different AI autopilots to autonomously drive your car on the track as fast as possible.
- Autonomous collision avoidance around different obstacles.
The RC-Car is named JetRacer, a high speed AI racing robot powered by Jetson Nano.
Some parts of the user guides on software in the above wiki (provided by the vendor) are out-of-date (you may meet different errors during tests). Please follow this repository for system and software installation in the following sections, and just take the above wiki as a reference.
Slides and Assemble Video: Link
This is for data collection and model deployment, covered in our 2nd workshop with slides and videos.
NVIDIA Jetson Nano lets you bring incredible new capabilities to millions of small, power-efficient AI systems. It is also the perfect tool to start learning about AI and robotics in real-world settings, with ready-to-try projects and the support of an active and passionate developer community.
To use this mainboard, we need to install a software development kit on it, which is named JetPack 👇
Based on the introduction in https://developer.nvidia.com/jetpack-sdk-44-archive
JetPack SDK is the most comprehensive solution for building AI applications. It includes the latest Linux Driver Package (L4T) with Linux operating system named Ubuntu (version: 18.04) and CUDA-X accelerated libraries and APIs for Deep Learning, Computer Vision, Accelerated Computing and Multimedia. It also includes samples, documentation, and developer tools for both host computer and developer kit.
In this project we use JetPack 4.4.
JetPack installation is quite simple: Flash the image to a microSD card -> Connect Nano to a display and boot the system (Ubuntu 18.04) -> Finish initialization. Step-by-step instructions are from Page 4 - Page 23 in this slides from workshop#2. Then you will enter the Ubuntu 18.04 system, which looks like the following:
Remember do not upgrade the system after installation, even the system reminds you of that. Because some of our library dependencies rely on the current version of jetpack, which is 4.4.
After installing the Ubuntu 18.04 system with JetPack4.4, we provide a system configuration named JetCard to make it easy to get started with AI on Jetson Nano. Simply execute the provided script and all external library dependencies will be installed automatically.
JetCard comes pre-loaded with:
- A Jupyter Lab server that starts on boot for easy web programming
- A script to display the Jetson Nano's IP address, CPU & GPU usage, battery life, charging status, etc.
- The popular deep learning frameworks PyTorch (version: 1.6)
- A Python3 library to drive RC-Cars Donkeycar
- Other development tools such as Oh My Zsh, virtualenv, torch2trt
After configuring your system using JetCard, you can get started prototyping AI projects from your web browser in Python3.
After you install JetPack 4.4 on the SD card, boot the system (Ubuntu 18.04) and open a terminal by pressing Ctrl+Alt+T, and then do the followings to use JetCard to quickly configure your system:
$ cd ~
$ git clone https://github.com/caipeide/jetcard
$ cd jetcard
$ sh ./install.sh <password>
The whole installation will cost about 40 min. After that the script will ask you to reboot. Take a look at the ip address shown on the display (10.79.157.13 in this case).
Now you can disconnect the HDMI port, keyboard and mouse on the jetson nano and start remote development: Open a browser on your own laptop and enter 10.79.157.13:8888 for remote connection to and development on the car. (the <password> you set earlier will be asked to enter the jupyterlab)
Finally, we clone this project to the RC-Car:
$ cd ~/projects/donkeycar
$ pip install -e .
$ cd ~
$ git clone https://github.com/caipeide/autorace
To ensure that the Jetson Nano doesn't draw more current than the battery pack can supply, place the Jetson Nano in 5W mode by calling the following command.
- You need to launch a new Terminal and enter following commands to select 5W power mode:
$ sudo nvpmodel -m1- Check if mode is correct (or take a look at the car display):
$ sudo nvpmodel -qm1: 5W power mode, m2: 10W power mode (MAXN)
A host PC is needed for model training
- Nvidia GPU (at least with 6 GB frame buffer), e.g., RTX 2060, GTX 1080.
- Ubuntu 18.04 system should be installed.
For participants who do not have a NVIDIA graphics card (GPU) in their computer, they can apply for using our server to train their models, and skip the following section 2.2. In this way, the laptop will be only used for remote connection to the RC-Car, which is doable on any system including Windows, MacOS and Ubuntu.
The following instructions refer to steps from 2 Ways to Install Nvidia Driver on Ubuntu 18.04 (GUI & Command Line). Here we simply choose to use graphical user interface (GUI) for installing the Nvidia driver.
First, go to system settings > details and check what graphics card your computer is using. By default, your integrated graphics card (Intel HD Graphics) is being used.
Then open softare & updates program from you application menu. Click the additional drivers tab. You can see what driver is being used for Nvidia card. If you can not see nvidia-driver-xxx in the list, open a terminal and do sudo apt update, then re-open this window, and available drivers can be updated.
As you can see many driver versions are available for the GeForce RTX 2080 Ti card on our server. Here we use the nvidia-driver-440 and it works fine for our model training. There might be some other drivers for your particular Nvidia card. Click Apply Changes button to install the driver.
After it’s installed, reboot your computer for the change to take effect. After that, go to system settings > details, you will see Ubuntu is using Nvidia graphics card.
You can open a terminal and do nvidia-smi to check the running information of your GPU:
We provide a script install_host.sh for you to quickly configure your host PC (or server account) with all necessary dependencies for model training.
By executing the script, the following packages will be automatically installed:
- Miniconda: A package manager that helps you find and install packages. Then a new conda environement named
autoracewill be automatically created - OpenCV: An open source computer vision and machine learning software library
- Matplotlib: A comprehensive library for creating static, animated, and interactive visualizations in Python
- Other tools such as Pytorch 1.6 and Donkeycar
After configuring your system, you can get started transmitting data between your host PC (or server account) and your RC-Car (via ssh), and using the collected dataset to train your own self-driving car (will be introduced in the next section Train a Self-driving Car)
- If you are using your own PC, open a terminal and do the first three steps first in the following (they are already installed on the server)
$ sudo apt update
$ sudo apt install openssh-server
$ sudo apt install git- Then you can install the other dependencies.
If you are using the server, connect to it with
sshfirst in a new terminal on your PC (or on the RC-Car with jupyterlab), for example, If you are using Ubuntu, open a new terminal (Ctrl+Alt+T) andssh -p <port_number> <server_account_name>@<server_ip_address>. Then do the followings in the terminal for host configuration.
$ cd ~
$ git clone https://github.com/caipeide/autorace
$ cd autorace
$ sh ./install_host.sh
$ source ~/.bashrc
$ conda create -n autorace python=3.6 -y
$ conda activate autorace
$ sh ./install_host_continue.shAll of the car's settings are in the config.py and myconfig.py scripts. config.py stores the default values for all parameterss, and you can adjust these settings in myconfig.py by uncommenting related lines and changing their values. When the main program manage.py starts, it will first read config.py and then override the variables with values you set in myconfig.py.
You can edit this file using two methods:
- If you are in a terminal, you can use nano to edit files:
nano ~/autorace/myconfig.py- If you perfer GUI, you can directly edit files in Jupyter Lab. Just double-click a file, make changes, and
Ctrl+Sto save.
Here uncomment means deleting the # symbol at the beginning of lines. In most editors, you can toggle between the comment and uncomment status by pressing Ctrl+/.
Make sure your car is off the ground to prevent a runaway situation.
- Turn on your car and the motor, and connect to the car via Jupter Lab on your PC.
- Open a terminal and run
donkey calibrate --channel 1 --bus=1 - Enter
370when prompted for a PWM value. Then you should hear your ESC beep indicating that it's calibrated. If not, adjust this value a little, or else the motor won't work. - Enter 400 and you should see your cars wheels start to go forward.
- Keep trying different values until you've found a reasonable max speed and remember this PWM value. Remember do not set the max speed too high, otherwise, the front wheel of the car will be easily damaged when crashed at fence. For repairing, refer to this.
Reverse on RC cars is a little tricky because the ESC must receive a reverse pulse, zero pulse, reverse pulse to start to go backwards. To calibrate a reverse PWM setting...
- Enter the reverse value, for example
330, then the zero throttle value you find above, then the reverse value again. - Enter values +/- 10 of the reverse value to find a reasonable reverse speed. Remember this reverse PWM value.
Enter these values in myconfig.py script as THROTTLE_FORWARD_PWM, THROTTLE_STOPPED_PWM, and THROTTLE_REVERSE_PWM.
- Open a terminal and run
donkey calibrate --channel 0 --bus=1 - Enter 320 and you should see the wheels on your car move slightly. If not enter 400 or 300.
- Next enter values +/- 10 from your starting value to find the PWM setting that makes your car turn all the way left and all the way right. Remember these values.
- Enter these values in
myconfig.pyscript asSTEERING_RIGHT_PWMandSTEERING_LEFT_PWM. Note that the default vaules areSTEERING_LEFT_PWM = 460,STEERING_RIGHT_PWM = 290, saved inconfig.py.
Note: You need to make sure that the value (STEERING_LEFT_PWM + STEERING_RIGHT_PWM)*0.5 can make the front wheels face straight ahead, then the car can drive in a straight line with steering = 0.
Now that you have your car roughly calibrated you can try driving it to verify that it drives as expected. Here's how to fine tune your car's calibration.
- Start your car by running
python manage.py drivein this folder. - Go to
<car_ip_address>:8887in a browser (Tested on Chrome). The following is what you will see: camera video streams, adjustable control values and driving modes. Now you can pressJorLon keyboard to adjust the car steering to left or right, and useIandKto adjust the throttle values.
- Now set the steering to zero and press
Ia few times to get the car to go forward. If it goes straight, that is good and you can move on to the next section; if not, adjust your values ofSTEERING_LEFT_PWMandSTEERING_RIGHT_PWMuntil a straight driving trajectory can be achieved.
Note: Too small throttle values, e.g., 0.1, may not be enough to drive the car. You need to increase it a bit.
Only for basic testing. If you want to collect data in a more flexible way, use the joystick introduced in the next section Driving with Physical Joystick Controller
This controller provides a UI window accessible at <car_ip_address>:8887. After you run python manage.py drive this module will be loaded.
-
Pilot mode - Choose this if the pilot should control the angle and/or throttle. It has three options:
local_anglewill only let the pilot model control the angle of the car.local_pilotwill let the pilot model control both the angle and throttle.user_modewhere you have full control over the car using keyboard shortcuts or mouse.
Notes
- Switches to pilot modes will only take effect if you load your neural network model when starting the
manage.py, which will be introduced in Model Testing. - Clicking with mouse on the right blue area (with texts "Click/touch to use joystic") will switch from
pilotmodes touser_mode. You can use this function to quickly stop your self-driving car.
-
Recording - Press record data to start recording images, steering angels and throttle values. By default the data will be automatically recorded if throttle is not zero in
user_mode. If the car is in pilot mode, you can use this function to manually record some self-driving data. Note do not use the data from a pilot mode to train you netowrk. Training details will be covered in Model Training. -
Throttle mode - Option to set the throttle as constant. This is used in races if you have a pilot that will steer but doesn't control throttle.
-
Max throttle - Select the maximum throttle for
user_mode.
R: toggle recordingI: increase throttle by 0.05K: decrease throttle by 0.05J: turn leftL: turn right
Note: Throttle range and steering range are both [-1.0, 1.0]. Throttle values that are less than 0 indicate reversing.
- Recommended for data collection, much more flexible to operate the RC-Car than using web controller.
- By default, no UI interface is published in this mode. However, you can set
USE_FPV = Trueinmyconfig.pyto monitor the camera video streams. The published FPV images are accessible in<car_ip_address>:8890
- Plug the USB receiver of the Joystick controller into Jetson Nano, then start the program (before that, make sure your car motor is powered on):
$ cd ~/autorace
$ python manage.py drive --jsOptionally, if you want joystick use to be sticky and don't want to add the
--jseach time, modify yourmyconfig.pyso thatUSE_JOYSTICK_AS_DEFAULT = True
- The joystick controls are shown as follows, they will also be printed on your sceen when the program starts.
+------------------+--------------------------+
| control | action |
+------------------+--------------------------+
| a_button | toggle_mode |
| b_button | toggle_manual_recording |
| x_button | erase_last_N_records |
| y_button | emergency_stop |
| right_shoulder | increase_max_throttle |
| left_shoulder | decrease_max_throttle |
| options | toggle_constant_throttle |
| left_stick_horz | set_steering |
| right_stick_vert | set_throttle |
| right_trigger | constant_rage_mode |
| left_trigger | constant_gentle_mode |
+------------------+--------------------------+
- Explanations on the operations.
left_stick_horz: Left analog stick - Left and right to adjust steeringright_stick_vert: Right analog stick - Forward to increase forward throttle, and Backward to increase reverse throttle. No operation for zero throttle.toggle_mode: Switches modes - "User, Local Angle, Local(angle and throttle)"toggle_manual_recording: Toggle recording of all data even if your car stops with zero throttle. This is disabled by default because a more suitable method auto record on throttle is enabled by default, which means that whenever the throttle is not zero, driving data will be recorded - as long as you are in user mode. The data will be saved in folderdata/tub_xx_xx_xx/erase_last_N_records: You don't want to use bad data to train you network, such as collisions with walls. This function can erase the data in the last 100 frames to keep your dataset clean.increase_max_throttle: the max_throttle for joystick control is set to 0.5 by default (when right analog stick is pushed to the front). Press right shoulder to increase the max_throttle byPER_THROTTLE_STEPif you need more speed.PER_THROTTLE_STEPis 0.05 by default, you can change this inmyconfig.py.decrease_max_throttle: Simmilar to the above. Press left shoulder to decrease the max_throttle byPER_THROTTLE_STEP.toggle_constant_throttle: Toggle constant throttle. Sets to max throttle.constant_rage_mode: Start a constant throttle mode with very a large throttleRAGE_THROTTLE = 0.75. This can be useful at long and straight tracks, where you can quickly accelerate your car. The throttle value can be adjusted inmyconfig.pyconstant_gentle_mode: Start a constant throttle mode with a low throttleGENTLE_THROTTLE = 0.45. This can be useful when RC-Car enters a sharp corner at high speeds, where you need to quickly slow the speed down but not stop. This throttle can also be adjusted inmyconfig.py
- Place some obstacles on the track, and practice driving around the track a couple times. Considering you may not drive well at first, you can set
AUTO_RECORD_ON_THROTTLE = Falseinmyconfig.pyto disable recording data automatically. - When you're confident you can drive 10 laps without mistake, restart the python
mange.pyprocess to create a new tub session. SetAUTO_RECORD_ON_THROTTLE = True. The joystick will auto record with any non-zero throttle. - If you crash or run off the track, loosen the throttle immediately to stop recording. Then tap the X button to erase the last 5 seconds of records.
- After you've collected 10-20 laps of good data (5-20k images) you can stop your car with
Ctrl+Cin the terminal session for your car. - The data you've collected is in the
data/folder in the most recent tub folder.
After you finish training a car can that can drive solely on the track based on the following procedure, you should return to this part and cooperate with another team to collect data on wheel-to-wheel driving (two cars running on the track). These data will be crucial for your model to learn how to behave in the presence of another car, such as overtaking and slowing down to avoid collisions. Otherwise, it is likely to perform poorly in the main race.
- To increase the robustness of your model, you should place different obstacles (colors, types) randomly on diffeent locations on the track during data collection.
DRIVE_LOOP_HZset inmyconfig.pyis the max frequency that the drive loop should run. The actual frequency may be less than this if there are many blocking parts, e.g., your designed AI model is too complex to run quickly.- You can choose whether or not to add extra control noise in
user_mode. If the valueCONTROL_NOISEis set toTrueinmyconfig.py(default value isFalse), random action noises on steering angle and throttle will be added during your tele-operation. This can help you to collect more divrese data that the car recovers from off-center and off-orientation mistakes. Based on these your trained agent can be more "intelligent". We also provide two scalarsTHROTTLE_NOISEandANGLE_NOISEto adjust the level of noise. Note with this module acctivated, the data collection process will be difficult.
Training a deep neural network on Jetson Nano can be painful (quite slow 🐌). Therefore, we will use more powerful PCs for faster training. The first step is to copy the collected data from your RC-Car to the host PC.
- If you are using your own computer for model training, open a new terminal on your host PC and use
rsyncto copy your carsdata/folder.
$ cd ~/autorace
$ rsync -rv --progress --partial <car_account_name>@<car_ip_address>:~/autorace/data ./- If you use the server for model training, open a new terminal on the RC-Car, then do the followings instead.
$ cd ~/autorace
$ rsync -rv -e 'ssh -p <port_number>' --progress --partial ./data <server_account_name>@<server_ip_address>:~/autorace/Now you can check the new data in the folder ~/autorace/data/ on your host PC:
$ ssh -p <port_number> <server_account_name>@<server_ip_address> # if you use the server, connect to it via ssh first on your RC-Car
$ ls ~/autorace/data/The copied data folders will be printed. The following is an example.
In the same terminal you can now run the training script on the latest tub by passing the path to that tub as an argument. For example,
$ cd ~/autorace
$ python manage.py train --model models/resnet18.pth --type resnet18 --tub data/tub_1_20-12-12/,data/tub_2_20-12-12The trained model will be saved in ~/autorace/models/resnet18.pth. Optionally you can pass no arguments for the tub, and then all tubs will be used in the default data/ folder.
$ python manage.py train --model models/resnet18.pth --type resnet18We provide three basic model types for training: linear, rnn and resnet18. Click these hyperlinks for more details. The following image shows the model architecture from Nvidia Self-Driving Car in 2016, which uses a series of convolutional layers and fully connected layers to learn driving behaviors.
You can change these model architectures in ai_drive_models.py, where linear -> LinearModel, rnn -> RNNModel and resnet18 -> LinearResModel. Note the rnn model runs slowly on the RC-Car, thus it may not be suitable for racing.
Other training parameters such as batchsize, learning rate, etc., can be configured in myconfig.py. Their default values should work well in most cases.
During training you can run nvidia-smi in a terminal to check GPU power consumption and memory usage. The following shows a training demo.
After the training is finished, you can view how the training loss and validation loss decrease in models/loss_plot_resnet18.png.
In previous step we managed to get a model trained on the data. Now is time to move the model back to RC-Car, so we can use it for testing it if it will drive itself.
- If you use your own computer for model training, open a new terminal on your host PC:
$ cd ~/autorace
$ rsync -rv --progress --partial ./models/resnet18.pth <car_account_name>@<car_ip_address>:~/autorace/models/- If you use the server for model training, open a new terminal on the RC-Car, then do the followings instead.
$ cd ~/autorace
$ rsync -rv -e 'ssh -p <port_number>' --progress --partial <server_account_name>@<server_ip_address>:~/autorace/models/resnet18.pth ./models/The resnet18 model trained above runs about 150 ms/frame (6.7 Hz) on the car, which can cause problems if your car drives fast (not quickly enough to make a decision to take turns -> collision). Therefore, we will accelerate the model as follows on the RC-Car:
$ cd ~/autorace
$ python accel_model.py --model models/resnet18.pth --type resnet18The process takes about 1 ~ 2 min, and the accelerated model will be saved in models/resnet18_trt.pth. The inference speed of this model is about 50 ms/frame (20 Hz).
Notes:
- You may have some problems if you try to acclerate a rnn model, which seems to be the limitation of the
torch2trtlibrary we use. Unfortunately, I have no solution on that. - Add
--halfto the above command if you want to save memory usage of the neural network. Then the accelerated model will be using FP16 rather than FP32, allowing deployment of larger networks.
Ensure to place the car on the track and power on its motor so that it is ready to drive.
$ python manage.py drive --model models/resnet18_trt.pth --trt --type resnet18When all modules are ready (1 ~ 2 min to warm up), the program will notice you to press ENTER to start. Then the car should start to drive on it's own. Congratulations!
Notes:
- Add
--halfto the above command if you use that during model acceleration.
- The ip address of the car may change automatically (it will change only once after each boot). If your JupyterLab losts connection, check if the car's ip is changed. If so, using the new ip to refresh the browser.
- If you collect lots of data, for example, more than 1000 images in a tub under
data/folder, DO NOT try to open the tub folder and view images through JupyerLab, because it will cost much time to load the files and the GUI may freeze. You can transfer the data to your laptop to view. - Do not directly turn off the power when system is running. Open a terminal and do
sudo shutdownfirst to shutdown the OS system, then turn off the power. - If the RC-Car crashes into the track fence at a high speed, the front wheels are likely to get stuck at their drive rods. Then you have to take off the two drive rods and reinstall them, which is a little troublesome. It is suggested to simply removing the drive rod of the two front wheels, then this problem can be solved. This video demonstrates how to fix the problem. Other hardware modifications are not allowed.
- You are free to DIY your own algorithms to drive the car for competition, but the following functions in this repo should be kept:
- Press
ENTERto start the car immediately when using autopilot mode
- Press
- If you think coding with Jupyter Lab is boring because it lacks code navigation functins like
go to definationandcode suggestions, you can choose VSCode for development.
Visual Studio Code has a high productivity code editor which, when combined with programming language services, gives you the power of an IDE and the speed of a text editor. In this topic, we'll first describe VS Code's language intelligence features (suggestions, parameter hints, smart code navigation) and then show the power of the core text editor.
After installation, you can install the Remote - SSH extension within VSCode to let your editor connected to your RC-Car. The following is a video demo showing how.
- https://github.com/MorvanZhou/PyTorch-Tutorial
- https://github.com/yunjey/pytorch-tutorial
- https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html
- Donkeycar: Open source hardware and software platform to build a small scale self driving car.
- JetRacer: An autonomous AI racecar using NVIDIA Jetson Nano.
- Jetcard: An SD card image for web programming AI projects with NVIDIA Jetson Nano.
- torch2trt: An easy to use PyTorch to TensorRT converter for model acceleration. Real-time computing performance is important for our high-speed driving occasions.
- Getting Started with Jetson Nano Developer Kit