Benchmarking and Visualization Tool for Adversarial Machine Learning
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
robustness Added MagNet in robustness. Nov 29, 2017


The goal of this project:

  • Several datasets ready to use: MNIST, CIFAR-10, ImageNet-ILSVRC and more.
  • Pre-trained state-of-the-art models to attack. [See details].
  • Existing attacking methods: FGSM, BIM, JSMA, Deepfool, Universal Perturbations, Carlini/Wagner-L2/Li/L0 and more. [See details].
  • Visualization of adversarial examples.
  • Existing defense methods as baseline.

The code was developed on Python 2, but should be runnable on Python 3 with tiny modifications.

Please follow the instructions to reproduce the Feature Squeezing results.

1. Install dependencies.

pip install -r requirements_cpu.txt

If you are going to run the code on GPU, install this list instead:

pip install -r requirements_gpu.txt

2. Fetch submodules.

git submodule update --init --recursive

3. Download pre-trained models.

mkdir downloads; curl -sL | tar xzv -C downloads

4. Usage of python

usage: python [-h] [--dataset_name DATASET_NAME] [--model_name MODEL_NAME]
               [--select [SELECT]] [--noselect] [--nb_examples NB_EXAMPLES]
               [--balance_sampling [BALANCE_SAMPLING]] [--nobalance_sampling]
               [--test_mode [TEST_MODE]] [--notest_mode] [--attacks ATTACKS]
               [--clip CLIP] [--visualize [VISUALIZE]] [--novisualize]
               [--robustness ROBUSTNESS] [--detection DETECTION]
               [--detection_train_test_mode [DETECTION_TRAIN_TEST_MODE]]
               [--nodetection_train_test_mode] [--result_folder RESULT_FOLDER]
               [--verbose [VERBOSE]] [--noverbose]

optional arguments:
  -h, --help            show this help message and exit
  --dataset_name DATASET_NAME
                        Supported: MNIST, CIFAR-10, ImageNet.
  --model_name MODEL_NAME
                        Supported: cleverhans, cleverhans_adv_trained and
                        carlini for MNIST; carlini and DenseNet for CIFAR-10;
                        ResNet50, VGG19, Inceptionv3 and MobileNet for
  --select [SELECT]     Select correctly classified examples for the
  --nb_examples NB_EXAMPLES
                        The number of examples selected for attacks.
  --balance_sampling [BALANCE_SAMPLING]
                        Select the same number of examples for each class.
  --test_mode [TEST_MODE]
                        Only select one sample for each class.
  --attacks ATTACKS     Attack name and parameters in URL style, separated by
  --clip CLIP           L-infinity clip on the adversarial perturbations.
  --visualize [VISUALIZE]
                        Output the image examples for each attack, enabled by
  --robustness ROBUSTNESS
                        Supported: FeatureSqueezing.
  --detection DETECTION
                        Supported: feature_squeezing.
  --detection_train_test_mode [DETECTION_TRAIN_TEST_MODE]
                        Split into train/test datasets.
  --result_folder RESULT_FOLDER
                        The output folder for results.
  --verbose [VERBOSE]   Stdout level. The hidden content will be saved to log
                        files anyway.

5. Example.

python --dataset_name MNIST --model_name carlini \
--nb_examples 2000 --balance_sampling \
--attacks "FGSM?eps=0.1;" \
--robustness "none;FeatureSqueezing?squeezer=bit_depth_1;" \
--detection "FeatureSqueezing?squeezers=bit_depth_1,median_filter_2_2&distance_measure=l1&fpr=0.05;"

Cite this work

You are encouraged to cite the following paper if you use EvadeML-Zoo for academic research.

  title={{Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks}},
  author={Xu, Weilin and Evans, David and Qi, Yanjun},
  booktitle={Proceedings of the 2018 Network and Distributed Systems Security Symposium (NDSS)},