Revisiting DeepFool: generalization and improvement

Official PyTorch implementation of "Revisiting DeepFool: generalization and improvement"

Abstract

Deep neural networks have been known to be vulnerable to adversarial examples, which are inputs that are modified slightly to fool the network into making incorrect predictions. This has led to a significant amount of research on evaluating the robustness of these networks against such perturbations. One particularly important robustness metric is the robustness to minimal ℓ2 adversarial perturbations. However, existing methods for evaluating this robustness metric are either computationally expensive or not very accurate. In this paper, we introduce a new family of adversarial attacks that strike a balance between effectiveness and computational efficiency. Our proposed attacks are generalizations of the well-known DeepFool (DF) attack, while they remain simple to understand and implement. We demonstrate that our attacks outperform existing methods in terms of both effectiveness and computational efficiency. Our proposed attacks are also suitable for evaluating the robustness of large models and can be used to perform adversarial training (AT) to achieve state-of-the-art robustness to minimal ℓ2 adversarial perturbations.

Illustration of SuperDeepFool

Repository Structure

└── superdeepfool         # Package folder
    ├── utils             # utils function
    ├──attacks            # Source code for attacks
      ├── attack          # Attack class
      ├── DeepFool        # DeepFool
      ├── SuperDeepFool   # SuperDeepFool
└── curvature           # source code of curvature
    └── curvature       # curvature analysis
└── Dockerfile        # Docker scripts
└── main.py           # main scripts

Running in Docker

Dockerfile Documentation

This Dockerfile is used to build a Docker image that runs a Python application. The application is stored in the /app directory within the container, and the Docker image is based on the official Python 3.9 slim-buster image.

Instructions

The following instructions are executed in order when the Dockerfile is built:

1. The working directory is set to /app using the WORKDIR command.
2. All files in the current directory are copied to the /app directory in the container using the COPY command.
3. The packages specified in the requirements.txt file are installed using the RUN command and the pip install command. The --no-cache-dir flag is used to prevent the package manager from caching package installation files.
4. Port 80 is exposed using the EXPOSE command. This tells Docker that the container will listen on port 80 when it is run.
5. An environment variable named NAME is defined using the ENV command. This variable can be accessed by the application running in the container.
6. The CMD command is used to specify the command that should be run when the container is launched. In this case, the python app.py command is run to start the Python application.

Building the Docker Image

To build the Docker image, navigate to the directory containing the Dockerfile and run the following command:

docker build -t <image-name> .

Replace <image-name> with the desired name for the Docker image.

Running the Docker Container

To run the Docker container, use the following command:

docker run --gpus all -it  NAME=<name> <image-name>

Replace <name> with the desired value for the NAME environment variable, and replace <image-name> with the name of the Docker image you built in the previous step. This command will run the container and enable access to all available GPUs on the host machine.

Running with python

To use this Python project, follow the instructions below:

1. Clone the repository
2. Open the terminal or command prompt and navigate to the cloned repository
3. Install the package with `pip install -e .`
4. Run the command "python main.py" to start the project
5. All results will be stored in the "results" folder

Results

Below, we provide the results with SDF. We test our algorithms on deep convolutional neural network architectures trained on MNIST, CIFAR-10, and ImageNet datasets. For the evaluation, we use all the MNIST test dataset, while for CIFAR-10 and ImageNet we use 1000 samples randomly chosen from their corresponding test datasets. For MNIST, we use a robust model called IBP. For CIFAR-10, we use three models: an adversarially trained PreActResNet-18, a regularly trained Wide ResNet 28-10 (WRN-28 − 10) and LeNet. These models are obtainable via the RobustBench. On ImageNet, we test the attacks on two ResNet-50 (RN-50) models: one regularly trained and one ℓ2 adversarially trained, obtainable through the robustness library.

Comparison with DeepFool in terms of mean and the median of ℓ₂-norm of perturbations

Attack	Mean-ℓ2	Median-ℓ2	Grads
DF	0.17	0.15	14
SDF	0.11	0.10	32

Comparison with DeepFool in terms of orthogonality

Attack	Lenet	ResNet-18	WRN-28-10
DF	0.89	0.14	0.21
SDF	0.92	0.72	0.80

ImageNet ResNet-50

Attack	FR	Median-ℓ2	Grads
ALMA	100	0.10	100
DDN	99.9	0.17	1000
FAB	99.3	0.10	900
FMN	99.3	0.10	1000
C&W	100	0.21	82667
SDF	100	0.09	37

Adversarial Training

we evaluate the performance of a model adversarially trained using SDF against minimum-norm attacks and AutoAttack. Our experiments provide valuable insights into the effectiveness of adversarial training with SDF and sheds light on its potential applications in building more robust models. We adversarially train a WRN-28-10 on CIFAR-10. Similar to the procedure followed Madry et al,, we restrict ℓ2-norms of perturbation to 2.6 and set the maximum number of iterations for SDF to 6. We train the model on clean examples for the first 200 epochs, and we then fine-tune it with SDF generated adversarial examples for 60 more epochs. Our model reaches a test accuracy of 90.8%.

Curvature analysis

For part of our experiments, we analyzed the curvature of the models. We've simplified the code from Efficient Training of Low-Curvature Neural Networks paper and modified it to be easier to use with RobustBench. Please refer to CURE and Efficient Training of Low-Curvature Neural Networks for a more detailed analysis. For use our code please cd to curvature folder and run this command :

python curvature.py

Implementations

In terms of implementation, we use Pytorch . Foolbox and Torchattcks libraries are used to implement adversarial attacks. Our code follows a similar pattern to Torchattcks, and we plan to integrate it with Torchattcks in the near future. This will enable us to leverage the benefits of Torchattcks and enhance the capabilities of our code for adversarial attacks in deep learning models.