SSF for Efficient Model Tuning

This repo is the official implementation of our NeurIPS2022 paper "Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning" (arXiv).

Usage

Install

• Clone this repo:

git clone https://github.com/dongzelian/SSF.git
cd SSF

• Create a conda virtual environment and activate it:

conda create -n ssf python=3.7 -y
conda activate ssf

• Install CUDA==10.1 with cudnn7 following the official installation instructions
• Install PyTorch==1.7.1 and torchvision==0.8.2 with CUDA==10.1:

conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 -c pytorch

• Install timm==0.6.5:

pip install timm==0.6.5

• Install other requirements:

pip install -r requirements.txt

Data preparation

• FGVC & vtab-1k

You can follow VPT to download them.

Since the original vtab dataset is processed with tensorflow scripts and the processing of some datasets is tricky, we also upload the extracted vtab-1k dataset in onedrive for your convenience. You can download from here and then use them with our vtab.py directly. (Note that the license is in vtab dataset).

• CIFAR-100

wget https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz

• For ImageNet-1K, download it from http://image-net.org/, and move validation images to labeled sub-folders. The file structure should look like:

  $ tree data
  imagenet
  ├── train
  │   ├── class1
  │   │   ├── img1.jpeg
  │   │   ├── img2.jpeg
  │   │   └── ...
  │   ├── class2
  │   │   ├── img3.jpeg
  │   │   └── ...
  │   └── ...
  └── val
      ├── class1
      │   ├── img4.jpeg
      │   ├── img5.jpeg
      │   └── ...
      ├── class2
      │   ├── img6.jpeg
      │   └── ...
      └── ...
  

• Robustness & OOD datasets

Prepare ImageNet-A, ImageNet-R and ImageNet-C for evaluation.

Pre-trained model preparation

• For pre-trained ViT-B/16, Swin-B, and ConvNext-B models on ImageNet-21K, the model weights will be automatically downloaded when you fine-tune a pre-trained model via SSF. You can also manually download them from ViT,Swin Transformer, and ConvNext.

• For pre-trained AS-MLP-B model on ImageNet-1K, you can manually download them from AS-MLP.

Fine-tuning a pre-trained model via SSF

To fine-tune a pre-trained ViT model via SSF on CIFAR-100 or ImageNet-1K, run:

bash train_scripts/vit/cifar_100/train_ssf_2.sh

or

bash train_scripts/vit/imagenet_1k/train_ssf_2.sh

You can also find the similar scripts for Swin, ConvNext, and AS-MLP models. You can easily reproduce our results. Enjoy!

Robustness & OOD

To evaluate the performance of fine-tuned model via SSF on Robustness & OOD, run:

bash train_scripts/vit/imagenet_a(r, c)/eval_ssf.sh

Citation

If this project is helpful for you, you can cite our paper:

@InProceedings{Lian_2022_SSF,
  title={Scaling \& Shifting Your Features: A New Baseline for Efficient Model Tuning},
  author={Lian, Dongze and Zhou, Daquan and Feng, Jiashi and Wang, Xinchao},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2022}
}

Acknowledgement

The code is built upon timm. The processing of the vtab-1k dataset refers to vpt, vtab github repo, and NOAH.