Code repository for paper 《Proxy Robustness in Vision Language Models is Effortlessly Transferable》
As a pivotal technique for improving the defense of deep models, adversarial robustness transfer via distillation has demonstrated remarkable success in conventional image classification tasks. However, this paradigm encounters critical challenges when applied to vision-language models (VLM) (e.g., CLIP): constructing adversarially robust teacher for large-scale multi-modal models demands prohibitively high computational resources. We bridge this gap by revealing an interesting phenomenon: vanilla CLIP (without adversarial training) exhibits intrinsic defensive capabilities against adversarial examples generated by another CLIP with different architectures. We formally define this as proxy adversarial robustness, and naturally propose a Heterogeneous Proxy Transfer (HPT) framework that establishes cross-architectural robustness distillation channels between CLIP variants, effortlessly enabling the VLM robustness transfer from proxy to target models. Yet, such proxy transfer paradigm easily induces severe overfitting, leading to a sharp degradation in zero-shot natural generalization. To resolve that, we design Generalization-Pivot Decoupling (GPD) by leveraging the difference in learning rate scheduling. This decouples the proxy transfer process into a generalization-anchored warm-up that maintains generalization and a generalization-pulled HPT that promotes adversarial robustness, to achieve an equilibrium between natural generalization and adversarial robustness. Extensive experiments on 15 zero-shot datasets demonstrate the effectiveness of our HPT-GPD method.
Replace the files in the replace folder to the source code in your environmet:
replace anaconda3/envs/zsrobust/lib/python3.9/site-packages/clip/clip.py and anaconda3/envs/zsrobust/lib/python3.9/site-packages/clip/model.py with clip.py and model.py in the replace folder respectively.
replace the anaconda3/envs/zsrobust/lib/python3.9/site-packages/torchvision/datasets with the files in replace/torchvision.datasets
for updated dataset loader
Our implementation is based on 《Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness》 (CVPR 2024).