Skip to content

ARiSE-Lab/deepTest

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

DeepTest: Automated testing of deep-neural-network-driven autonomous cars

DeepTest is a systematic testing tool for automatically detecting erroneous behaviors of DNN-driven vehicles that can potentially lead to fatal crashes.

Install Required Packages

OS: Ubuntu 16.04
Read through and run ./install.sh

Code Directories

models/

  • Reproducing Udacity self-driving car Challenge2[1] results for Rambo, Chauffeur and Epoch models.
  • Models and weights files are required to run these scripts.
  • For Rambo model, Keras backend should be changed to theano.

testgen/

  • Generate synthetic images, calculate cumulative coverage and record predicted outputs.
  • ncoverage.py: define NCoverage class for computing neuron coverage.

guided/

  • Combine different transformations and leverage neuron coverage to guide the search.
  • Re-runing the script will continue the search instead of starting from the beginning except deleting all pkl files.

metamorphictesting/

Models and Saved Weights

  • Rambo [2]
  • Chauffeur [3]
  • Epoch [4]

Datasets

Generate hmb3 dataset from HMB_3.bag

mkdir hmb3
python generate_hmb3.py --bagfile HMB_3.bag

The following commands are to install python-rosbag api.

sudo apt-get install python-rosbag
sudo apt-get install python-genmsg
sudo apt-get install python-genpy
sudo apt-get install python-rosgraph-msgs

Reproducing experiments

Dataset directory structure:

./Dataset/
├── hmb3/
        └── hmb3_steering.csv
        └── images
└── Ch2_001/
        └── center/
                 └── images
        └── CH2_final_evaluation.csv

Evaluating models' accuracy on existing test data

cd models/
python epoch_reproduce.py --dataset DATA_PATH/Dataset/
python chauffeur_reproduce.py --dataset DATA_PATH/Dataset/
python rambo_reproduce.py --dataset DATA_PATH/Dataset/

Generate synthetic images and compute cumulative neuron coverage

cd testgen/
./epoch_testgen_driver.sh 'DATA_PATH/Dataset/'
./chauffeur_testgen_driver.sh 'DATA_PATH/Dataset/'
python rambo_testgen_coverage.py --dataset DATA_PATH/Dataset/

Combine transformations and synthesize images by guided search

cd guided/
mkdir new/
rm -rf *.pkl
python epoch_guided.py --dataset DATA_PATH/Dataset/
python chauffeur_guided.py --dataset DATA_PATH/Dataset/
python rambo_guided.py --dataset DATA_PATH/Dataset/

Detected erroneous behaviors

https://deeplearningtest.github.io/deepTest/

Citation

If you find DeepTest useful for your research, please cite the following paper:

@article{tian2017deeptest,
  title={DeepTest: Automated testing of deep-neural-network-driven autonomous cars},
  author={Tian, Yuchi and Pei, Kexin and Jana, Suman and Ray, Baishakhi},
  journal={arXiv preprint arXiv:1708.08559},
  year={2017}
}

References

  1. Udacity self driving car challenge 2.
    https://github.com/udacity/self-driving-car/tree/master/challenges/challenge-2. (2016).
  2. Rambo model.
    https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/rambo. (2016).
  3. Chauffeur model.
    https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/chauffeur. (2016).
  4. Epoch model.
    https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/cg23. (2016).

About

A systematic testing tool for automatically detecting erroneous behaviors of DNN-driven vehicles

Resources

Stars

Watchers

Forks

Packages

No packages published