AdvEncoder

The implementation of our ICCV 2023 paper "Downstream-agnostic Adversarial Examples"

Abstract

Self-supervised learning usually uses a large amount of unlabeled data to pre-train an encoder which can be used as a general-purpose feature extractor, such that downstream users only need to perform fine-tuning operations to enjoy the benefit of ``large model". Despite this promising prospect, the security of pre-trained encoder has not been thoroughly investigated yet, especially when the pre-trained encoder is publicly available for commercial use.

In this paper, we propose AdvEncoder, the first framework for generating downstream-agnostic universal adversarial examples based on the pre-trained encoder. AdvEncoder aims to construct a universal adversarial perturbation or patch for a set of natural images that can fool all the downstream tasks inheriting the victim pre-trained encoder. Unlike traditional adversarial example works, the pre-trained encoder only outputs feature vectors rather than classification labels. Therefore, we first exploit the high frequency component information of the image to guide the generation of adversarial examples. Then we design a generative attack framework to construct adversarial perturbations/patches by learning the distribution of the attack surrogate dataset to improve their attack success rates and transferability. Our results show that an attacker can successfully attack downstream tasks without knowing either the pre-training dataset or the downstream dataset. We also tailor four defenses for pre-trained encoders, the results of which further prove the attack ability of AdvEncoder.

Latest Update

2023/8/24 We have released the official implementation code.

Setup

• Get code

git clone https://github.com/CGCL-codes/AdvEncoder.git

• Build environment

cd AdvEncoder
# use anaconda to build environment 
conda create -n AdvEncoder python=3.8
conda activate AdvEncoder
# install packages
pip install -r requirements.txt

• The final project should be like this:

  AdvEncoder
  └- utils
      └- predict
  └- model
      └- adv_gan
  └- data
  └- victims
      └- cifar10 (pre-training dataset)
        └- simclr
           └- simclr-cifar10-b30xch14-ep=999.ckpt
           └- clean_model
  └- output
  └- ...

• Download Victim Pre-trained Encoders

  • All of our pre-trained encoders were obtained from the solo-learn repository, and some missing pre-trained encoders were trained by us based on their code.
  • Please move the downloaded pre-trained encoder into /victims/[pre-dataset]/[method].

CIFAR-10

Method	Backbone	Epochs	Acc@1	Acc@5	Checkpoint
Barlow Twins	ResNet18	1000	92.10	99.73	Link
BYOL	ResNet18	1000	92.58	99.79	Link
DeepCluster V2	ResNet18	1000	88.85	99.58	Link
DINO	ResNet18	1000	89.52	99.71	Link
MoCo V2+	ResNet18	1000	92.94	99.79	Link
MoCo V3	ResNet18	1000	93.10	99.80	Link
NNCLR	ResNet18	1000	91.88	99.78	Link
ReSSL	ResNet18	1000	90.63	99.62	Link
SimCLR	ResNet18	1000	90.74	99.75	Link
SupCon	ResNet18	1000	93.82	99.65	Link
SwAV	ResNet18	1000	89.17	99.68	Link
VIbCReg	ResNet18	1000	91.18	99.74	Link
VICReg	ResNet18	1000	92.07	99.74	Link
W-MSE	ResNet18	1000	88.67	99.68	Link

ImageNet-100

Method	Backbone	Epochs	Acc@1	Acc@5	Checkpoint
Barlow Twins	ResNet18	400	80.38	95.28	Link
BYOL	ResNet18	400	80.16	95.02	Link
DeepCluster V2	ResNet18	400	75.36	93.22	Link
DINO	ResNet18	400	74.84	92.92	Link
MoCo V2+	ResNet18	400	78.20	95.50	Link
MoCo V3	ResNet18	400	80.36	95.18	Link
NNCLR	ResNet18	400	79.80	95.28	Link
ReSSL	ResNet18	400	76.92	94.20	Link
SimCLR	ResNet18	400	77.64	94.06	Link
SupCon	ResNet18	400	84.40	95.72	Link
SwAV	ResNet18	400	74.04	92.70	Link
VIbCReg	ResNet18	400	79.86	94.98	Link
VICReg	ResNet18	400	79.22	95.06	Link
W-MSE	ResNet18	400	67.60	90.94	Link

Quick Start

• Train AdvEnoder-PER

python gan_per_attack.py   # results saved in /output/[pre-dataset]/uap_results/gan_per

• Train AdvEnoder-PAT

python gan_pat_attack.py  # results saved in /output/[pre-dataset]/uap_results/gan_patch

• Train downstream classifiter

python train_down_classifier.py # clean models saved in /victims/[pre-dataset]/[victim-encoder]/clean_model

• Test performance of AdvEncoder

python test_down_classifier.py # results saved in /output/[pre-dataset]/log/down_test

Acknowledge

We greatly appreciate the immense contribution of the solo-learn team in the field of self-supervised learning by open-sourcing their repository.

BibTeX

If you find AdvEncoder both interesting and helpful, please consider citing us in your research or publications:

@inproceedings{zhou2023advencoder,
  title={Downstream-agnostic Adversarial Examples},
  author={Zhou, Ziqi and Hu, Shengshan and Zhao, Ruizhi and Wang, Qian and Zhang, Leo Yu and Hou, Junhui and Jin, Hai },
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV'23)},
  year={2023}
}