DeepXplore: Systematic DNN testing (SOSP'17)
See the SOSP'17 paper DeepXplore: Automated Whitebox Testing of Deep Learning Systems for more details.
The code should be run using python 2.7.12, Tensorflow 1.3.0, Keras 2.0.8, PIL, h5py, and opencv-python
sudo pip install tensorflow
if you have gpu,
pip install tensorflow-gpu
pip install keras
To set Keras backend to be tensorflow (two options):
1. Modify ~/.keras/keras.json by setting "backend": "tensorflow" 2. KERAS_BACKEND=tensorflow python gen_diff.py
pip install Pillow
pip install h5py
pip install opencv-python
Install from here.
- MNIST - MNIST dataset.
- ImageNet - ImageNet dataset.
- Driving - Udacity self-driving car dataset.
- PDF - Benign/malicious PDFs captured from VirusTotal/Contagio/Google provided by Mimicus.
- Drebin - Drebin Android malware dataset.
In every directory
The trained weights are provided in each directory (if required). Drebin's weights are not part of this repo as they are too large to be hosted on GitHub. Download from here and put them in ./Drebin/.
Note that as DeepXplore use randomness for its exploration, you should fix the seed of the random number generator if you want deterministic and reproducable results. An example is shown below.
import numpy as np import random random.seed(1) np.random.seed(1)
How to test your own DNN models.