Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
attacks
README.md
__init__.py
utils.py

README.md

Adversarial Machine Learning

Note: This section takes reference from the tf cleverhans library

Adversarial Machine Learning is the study of attacks and defenses that can be used to easily fool Machine Learning models and defend Machine Models against such attacks.

Attacks

The attacks implemented in this section can be found here: https://github.com/iArunava/scratchai/README.md

Benchmarks

This section performs benchmarking of all the attacks and defences implemented here.

The benchmarks reproduced here uses the ILSVC2012 Imagenet test set. The columns with Acc@n indicate the top-n accuracy and columns with w/o indicates the accuracy without the attack and the ones with w indicate the accuracy with the attack.

NA Indicates that it has not been measured yet.

Attack Lenet Alexnet VGG16 VGG19 Resnet18
Noise
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST 0.984 1.0 0.9858 1.0
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST 0.9907 1.0 0.9908 1.0
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Semantic
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST 0.233 0.645 0.986 1.0
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST 0.278 0.612 0.99 1.0
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Fast Gradient Sign Method
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST 0.509 0.993 0.986 1.0
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST 0.831 0.99 0.99 1.0
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Projected Gradient Descent
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST 0.187 0.982 0.986 1.0
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST 0.667 0.9984 0.99 1.0
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
DeepFool
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST 0.012 1.0 0.9858 1.0
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST 0.0084 1.0 0.9908 1.0
ILSVRC2012 0.11 0.77 0.54 0.78
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 NA NA NA NA
Dataset Acc@1 w/ Acc@5 w/ Acc@1 w/o Acc@5 w/o
MNIST NA NA NA NA
ILSVRC2012 0.10 0.88 0.69 0.88

All the above benchmarks are done using the following code:

>>> from scratchai import *
>>> net = nets.lenet_mnist() # Get the network of choice (pretrained on the dataset)
>>> attacks.benchmark_atk(attacks.PGD, net, dset='mnist', bs=16, topk=(1, 5))

[INFO] Setting bs to 16.
[INFO] Setting trf to Compose(
          Resize(size=(256, 256), interpolation=PIL.Image.BILINEAR)
          CenterCrop(size=(224, 224))
          ToTensor()
          Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
          ).
[INFO] Setting dset to mnist.
[INFO] Setting root to ./.
[INFO] Setting topk to (1, 5).
[INFO] Setting dfunc to <class 'torchvision.datasets.folder.ImageFolder'>.
[INFO] Setting download to True.
[INFO] Net Frozen!
100%|_______________________________________________________| 625/625 [00:18<00:00, 34.71it/s]

Attack Summary on lenet with pgd attack:
---------------------------------------------
Top 1 original accuracy is 0.9858
Top 5 original accuracy is 1.0

-----------------------------------
Top 1 adversarial accuracy is 0.1874
Top 5 adversarial accuracy is 0.9818

You can’t perform that action at this time.