ComOpT: COMbination and OPtimization for Testing autonomous driving systems

ComOpT is a research tool for coverage-driven and constraint-driven testing of autonomous driving systems.

License

AGPL v3.0

(Please know the implication of LICENSE before using or contributing to the code)

Developers (alphabetical by family name)

• Yuhang Chen
• Chih-Hong Cheng
• Changwen Li
• Tiantian Sun
• Rongjie Yan

Apollo, LGSVL and Python API package

ComOpT has been tested with the following software configuration:

• Apollo autonomous driving stack - master taken on 20210414
• LGSVL version - 2021.01
• LGSVL PythonAPI version - 2021.01
• Python >= 3.8

To run Apollo (master version taken on 20210414) with SVL Simulator, please follow the instructions detailed here.

Installing additional software

• Install Python requirements.

$ python -m pip install -r scripts/requirements.txt --user

• Install Neo4j and enable the service (only needed when scripts/output/ folder does not contain the map database). Details available at how to install neo4j on ubuntu 20.04

The user should set up the environment variables (see next session).

Environment variables

The following environment variables are only needed when scripts/output/ folder does not contain the map database:

• NEO4J_ENDPOINT: for setting up the connection (e.g., neo4j://localhost:7687)
• NEO4J_USERNAME: for accessing the graph database
• NEO4J_PW: for accessing the graph database

Usage

A. Coverage-driven scenario generation and testing

1. Launch SVL simulator and start simulation in "API Only" mode.

2. Launch Apollo and set the vehicle and map information.

3. To reproduce our experiment, run

   $ python scripts/ad_tester.py scripts/ad_tester_output
   

   To generate scenarios from the beginning, run

   $ python scripts/ad_tester.py scripts/new_folder_name
   

   Type y when asking whether generating abstract and concrete scenarios or not.

B. Scenario replaying

1. Launch SVL simulator and start simulation in "API Only" mode.

2. Launch Apollo and set the vehicle and map information.

3. To replay the demonstration scenario, scenario_0_0 for example, run

   $ python scripts/scenario_runner.py map_info/scenario_files/scenario_0_0.json
   

C. Map parsing

This step is only needed when scripts/output/ folder does not contain the map database.

1. Launch SVL simulator and start simulation in "API Only" mode.

2. Update the configuration file located at scripts/comopt/config/maps.json.

3. Enter scripts/comopt/map/tools/ and run

   $ python svl_transform.py MAP_NAME
   

   MAP_NAME is the same as the key value in the configuration file.

4. Enter scripts/ and run

   $ python map_parse.py map-name -u
   

   -u means run parsing ignoring buffer.

   map-name is the same as the folder name in scripts/comopt/data/map/.

   (eg: scripts/comopt/data/map/san_francisco means the map-name is san_francisco).

   Please ensure the environment variables are correctly configured to connect to the graph database

Scenario demos

A two-minute teaser video highlighting the techniques and some of the undesired scenarios discovered by ComOpT: https://www.youtube.com/playlist?list=PL6ii4xJXGd8L3bn9pWuchuKBPb0R5JgJ7

The simulation execution reports are located in execution_reports/. And the scenario description files are located in map_info/sceanrio_files/, which can be replayed by using scripts/scenario_runner.py.

There are 15 equivalence classes of scenarios (abstract scenarios) generated by ComOpT in the experiment. For each class of scenarios, there are 3 demonstration concrete scenarios generated by either random sampling or perturbation.

The simulation execution results of concrete scenarios in the videos are listed as follows. The "Scenario ID" in "X_Y" format represents the Yth demonstration concrete scenario of the Xth equivalence class.

Classification		Scenairo ID
Normal sceanrios		0_0, 7_2, 13_0, 14_2
Stagnant	Stuck in traffic (stuck by traffic participants or infrastructures)	0_1, 5_0, 8_2, 10_0, 10_1, 12_1
	No planning output	1_1, 3_1, 4_0, 4_1, 5_2, 12_2, 14_0, 14_1
	Do not move with planning output	1_2, 3_2, 6_1
Collision	Collision during lane change	0_2, 9_2
	Collision near junctions	9_0, 11_1, 13_1
	Collision caused by untimely launch of planning	4_2, 9_1, 10_2
Off course	Out of road	2_1, 3_0, 8_1
	To the lane in the opposite direction	3_0
	Not following the planning, but within the legal route	7_0, 11_0, 12_1
Others	Ignoring red lights (contains ignoring unlabelled traffic lights)	1_0, 5_1, 1_2
	Unnecessary hesitated moving	1_0, 2_0, 6_0
	Abnormal or uncomfortable driving	2_2, 3_2, 12_0, 13_1