Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer

Yuwen Tan¹  Qinhao Zhou¹  Xiang Xiang¹  Ke Wang²  Yuchuan Wu²  Yongbin Li²

¹School of Artificial Intelligence and Automation, Huazhong University of Science and Technology 

²Alibaba Group 

The code repository for "Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer" in PyTorch.

News

[02/2023]🎉 Our paper has been accepted by CVPR2024.

[03/2023] 🌟 arXiv paper has been released.

[04/2024] 🌟 The code repository has been released.

Abstract

Class-incremental learning (CIL) aims to enable models to continuously learn new classes while overcoming catastrophic forgetting. The introduction of pre-trained models has brought new tuning paradigms to CIL. In this paper, we revisit different parameter-efficient tuning (PET) methods within the context of continual learning. We observe that adapter tuning demonstrates superiority over prompt-based methods, even without parameter expansion in each learning session. Motivated by this, we propose incrementally tuning the shared adapter without imposing parameter update constraints, enhancing the learning capacity of the backbone. Additionally, we employ feature sampling from stored prototypes to retrain a unified classifier, further improving its performance. We estimate the semantic shift of old prototypes without access to past samples and update stored prototypes session by session. Our proposed method eliminates model expansion and avoids retaining any image samples. It surpasses previous pre-trained model-based CIL methods and demonstrates remarkable continual learning capabilities. Experimental results on five CIL benchmarks validate the effectiveness of our approach, achieving state-of-the-art (SOTA) performance.

Pipeline

We use adapter without parameter limitation as our baseline, compared with other PETuning method, we find adapter performs best in balancing the performance of old and new classes. We further train the classifier by sampling features with Gaussian samples, which improves the performance of the incremental process. During the construction of the distribution, we apply semantic bias correction to the prototype of each feature within each class.

Results

The following table shows the main results of our proposed method and other SOTA methods. Please note that there might be slight variations in results based on the type and quantity of NVIDIA GPUs.

Requirements

Dependencies

1. torch 1.11.0
2. torchvision 0.12.0
3. timm 0.6.12

Datasets

We provide the processed datasets as follows:

• CIFAR100: will be automatically downloaded by the code.
• CUB200 ImageNet-R ImageNet-A VTAB: Reference Revisiting

These subsets are sampled from the original datasets. Please note that we do not have the right to distribute these datasets. If the distribution violates the license, I shall provide the filenames instead.

You need to modify the path of the datasets in ./data/data.py according to your own path.

Training Scripts

Please follow the settings in the exps folder to prepare your json files, and then run:

python main.py --config ./exps/[configname].json

for imageneta:
python main.py --config ./exps/adapter_imageneta.json
for imagenetr:
python main.py --config ./exps/adapter_imagenetr.json
for cifar224:
python main.py --config ./exps/adapter_cifar224.json
for cub200:
python main.py --config ./exps/adapter_cub.json

Citation

If you find this useful in your research, please consider citing:

@article{tan2024semantically,
 title={Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer},
 author={Tan, Yuwen and Zhou, Qinhao and Xiang, Xiang and Wang, Ke and Wu, Yuchuan and Li, Yongbin},
 journal={arXiv preprint arXiv:2403.19979},
 year={2024}
}

Acknowledgment

This repo is based on RevisitingCIL and PyCIL.

The implemenations of parameter-efficient tuning methods are based on VPT, AdaptFormer, and SSF.

Thanks for their wonderful work!!!

Correspondence

If you have any question about this project, please contact xex@hust.edu.cn