Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment
Michigan Traffic Lab, University of Michigan
In this repository, we provide an implementation of our research project "Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment". More details can be found in the paper.
Users can use this repo to evaluate AV safety performance on a three-lane highway environment.
The structure of the code is as follows:
NADE_main.py: the main file to run vehicle testing simulations and generate raw testing data;NADE_core.py: the main algorithm of NADE;CAV_agent: the AV agents used in this study that trained by deep reinforcement learning;highway_env: the highway environment simulator;global_val.py: the parameters and settings used by the NADE algorithm;config: the folder where to put config files;data: the folder carries the static data (e.g., trained AV agent model) used in the repository;plot.ipynb: the file to analyze testing results and plot figures.
The simulation platform is developed based on an open-source highway simulator highway-env.
For more details of the highway-env, please refer to highway-env.
- Create new conda environment
You are recommended to create a new Conda environment to install the project
conda create -n NADE python=3.7
conda activate NADE- Pytorch installation
conda install pytorch==1.4.0 torchvision==0.5.0 cpuonly -c pytorch- Clone this repo
git clone https://github.com/michigan-traffic-lab/NADE.git
cd NADE- Install all required packages
- install Pytorch according to official guidance
- install other dependency
pip install -r requirements.txtThe data (AV2 agent and NDE behavior models, etc) can be downloaded here.
Unzip the file and put them into the Data folder, the file structure should look like:
Repo-path/Data/
|__ AV2_AGENT
|______ ...
|__ CALCULATE_CHALLENGE
|______ ...
|__ NDD_DATA
|______ ...
|__ PLOT_RESULTS
|______ ...
As the testing result generation is time-consuming,
for the convenience of usage, we provided example testing results using AV2 agent. The results
are stored in Data/PLOT_RESULTS/EXAMPLE_NADE_TESTING_RESULTS folder. It includes around
500,000 testing episodes, and they are stored in 50 files (each contains around 10,000 episodes).
Each row is a testing episode result, and the data format is:
Episode id, Test result(1: crash, 3: no crash), weight list (likelihood ratio) of each timestep, maximum criticality of each timestep, sampled action naturalistic probability of each timestep, sampled action importance function probability list of each timestep.
Use plot.ipynb to analyze results and generate crash rate and relative half-width figures. The
plotted figures will be saved in result folder by default.
If you want to generate new testing result, config your desired settings in configs.yml (e.g.,
NADE epsilon parameter etc.,) and run the following:
python NADE_main.py --experiment-name your-experiment-name --folder-idx 1The testing results will be saved in folder Log by default.
Note: The folder-idx denote the worker id of the running experiment, you can
run multiple experiments (e.g., folder-idx from 1 to 16) simultaneously for
parallel computing.
After generating the testing results, you can use plot.ipynb to analyze it.
Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Xintao Yan (xintaoy@umich.edu)
Haowei Sun (haoweis@umich.edu)
Shuo Feng (fshuo@umich.edu)
This project is licensed under the [PolyForm Noncommercial License 1.0.0]. Please refer to LICENSE for more details.
Henry Liu (henryliu@umich.edu)