CVPR 2023 | Demonstration | Quickstart | Usage | Join Leaderboard | Citation
This repository contains code for the CVPR 2023 Paper "Towards Compositional Adversarial Robustness: Generalizing Adversarial Training to Composite Semantic Perturbations." The research demo and the leaderboard of composite adversarial robustness can be found at CARBEN: Composite Adversarial Robustness Benchmark, which earlier appeared in IJCAI 2022 Demo Track.
Authors: Lei Hsiung, Yun-Yun Tsai, Pin-Yu Chen, and Tsung-Yi Ho.
The adversarial attacks have been widely explored in Neural Network (NN). However, previous studies have sought to
create bounded perturbations in a metric manner. Most such work has focused on
Figure 1. The Flow of the Composite Adversarial Attacks.
We combined the
In this README, we are going to show how to execute our code and derive the experimental results in our paper.
- cuda 11.3
- python 3.7.16
- numpy 1.21.6
- pytorch 1.12.0
- torchvision 0.13.0
- kornia 0.6.3
- requests
Composite-adv can be downloaded as a GitHub repository or a package, the code including training (GAT) and robustness evaluation (CAA) phases.
- Install Python 3.
- Use
Composite-Adv
as a package.pip install git+https://github.com/IBM/composite-adv.git
- Use
Composite-Adv
as a repository.git clone https://github.com/IBM/composite-adv.git cd composite_adv pip install -r requirements.txt
Please use composite_adv.utilities.make_model()
to load GAT pre-trained models.
Two Architectures Available: ResNet-50 (
Link | Link | Link | Link | Link | Link |
Link | Link | Link | Link | Link | Link |
One Architecture Available: ResNet-50 (
Link | Link | Link | Link |
One Architectures Available: Wide-ResNet-34-10 (
Link | Link | Link | Link |
This section demonstrate how to launch CAA and also using GAT to derive a robust model.
In getting_started.ipynb, we provide a step-by-step demonstration showing how to launch our composite adversarial attack (CAA). We use the CIFAR-10 dataset for demonstration, while other datasets could be executed similarly.
Evaluate robust accuracy / attack success rate of the model
python evaluate_model.py \
--arch ARCHITECTURE --checkpoint PATH_TO_MODEL \
--dataset DATASET_NAME --dataset-path DATASET_PATH \
--message MESSAGE_TO_PRINT_IN_CSV \
--batch-size BATCH_SIZE --output RESULT.csv \
"CompositeAttack(model, enabled_attack=(0,1,5), order_schedule='random', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(0,1,5), order_schedule='scheduled', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(3,4,5), order_schedule='random', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(3,4,5), order_schedule='scheduled', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(0,2,5), order_schedule='random', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(0,2,5), order_schedule='scheduled', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(0,1,2,3,4), order_schedule='random', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(0,1,2,3,4), order_schedule='scheduled', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(0,1,2,3,4,5), order_schedule='random', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(0,1,2,3,4,5), order_schedule='scheduled', inner_iter_num=10, dataset='DATASET_NAME')"
python evaluate_model.py \
--arch ARCHITECTURE --checkpoint PATH_TO_MODEL \
--dataset DATASET_NAME --dataset-path DATASET_PATH \
--message MESSAGE_TO_PRINT_IN_CSV \
--batch-size BATCH_SIZE --output RESULT.csv \
"NoAttack()" \
"CompositeAttack(model, enabled_attack=(0,), order_schedule='fixed', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(1,), order_schedule='fixed', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(2,), order_schedule='fixed', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(3,), order_schedule='fixed', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(4,), order_schedule='fixed', inner_iter_num=10, dataset='DATASET_NAME')" \
"CompositeAttack(model, enabled_attack=(5,), order_schedule='fixed', inner_iter_num=20, dataset='DATASET_NAME')" \
"AutoLinfAttack(model, 'DATASET_NAME')"
python train_cifar10.py \
--batch-size BATCH_SIZE --epochs 150 --arch ARCHITECTURE \
--checkpoint PATH_TO_MODEL_FOR_RESUMING.pt \
--mode TRAINING_OBJECTIVE --order random --enable 0,1,2,3,4,5 \
--model-dir DIR_TO_SAVE_EPOCH/ \
--log_filename TRAINING_LOG.csv
python train_cifar10.py \
--dist-backend 'nccl' --multiprocessing-distributed \
--batch-size BATCH_SIZE --epochs 150 --arch ARCHITECTURE \
--checkpoint PATH_TO_MODEL_FOR_RESUMING.pt \
--mode TRAINING_OBJECTIVE --order random --enable 0,1,2,3,4,5 \
--model-dir DIR_TO_SAVE_EPOCH/ \
--log_filename TRAINING_LOG.csv
python train_imagenet.py \
--dist-backend 'nccl' --multiprocessing-distributed \
--batch-size BATCH_SIZE --epochs 150 --arch ARCHITECTURE \
--checkpoint PATH_TO_MODEL_FOR_RESUMING.pt --stat-dict TYPE_OF_CHECKPOINT \
--mode TRAINING_OBJECTIVE --order random --enable 0,1,2,3,4,5 \
--model-dir DIR_TO_SAVE_EPOCH \
--log_filename TRAINING_LOG.csv
python train_svhn.py \
--batch-size BATCH_SIZE --epochs 150 --arch ARCHITECTURE \
--checkpoint PATH_TO_MODEL_FOR_RESUMING.pt \
--mode TRAINING_OBJECTIVE --order random --enable 0,1,2,3,4,5 \
--model-dir DIR_TO_SAVE_EPOCH/ \
--log_filename TRAINING_LOG.csv
We maintain the leaderboards to track the progress of the Compositional Adversarial Robustness. Specifically, we focus on "white-box" scenarios in which the attacker has all knowledge of the models. We have provided similar entries to those in the RobustBench leaderboard, and hereby solicit model submissions to compete against composite perturbations in our leaderboard. If you would like to submit your model, please follow the instructions in CARBEN-Leaderboard.ipynb to evaluate your model. After the robustness assessment is completed, please fill in the Google Form, and we will update the leaderboard after confirmation.
If you find this helpful for your research, please cite our papers as follows:
@inproceedings{hsiung2023caa,
title={{Towards Compositional Adversarial Robustness: Generalizing Adversarial Training to Composite Semantic Perturbations}},
author={Lei Hsiung and Yun-Yun Tsai and Pin-Yu Chen and Tsung-Yi Ho},
booktitle={{IEEE/CVF} Conference on Computer Vision and Pattern Recognition, {CVPR}},
publisher={{IEEE}},
year={2023},
month={June}
}
@inproceedings{hsiung2022carben,
title={{CARBEN: Composite Adversarial Robustness Benchmark}},
author={Lei Hsiung and Yun-Yun Tsai and Pin-Yu Chen and Tsung-Yi Ho},
booktitle={Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, {IJCAI-22}},
publisher={International Joint Conferences on Artificial Intelligence Organization},
year={2022},
month={July}
}