Tuning Pre-trained Model via Moment Probing

This repo is the official implementation of ICCV 2023 paper "Tuning Pre-trained Model via Moment Probing", created by Mingze Gao, Qilong Wang and Zhenyi Lin.

Introduction

In this paper, we propose a Moment Probing (MP) method, which performs a linear classifier on powerful representations characterized by feature distribution to finetune pretrained vision backbones efficiently. A partially shared module is proposed to learn two recalibrating parameters (PSRP), resulting in a complete finetuning framework MP+.

Usage

Data preparation

• CIFAR-100

wget https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz

• For ImageNet-1K, download it from http://image-net.org/, and move validation images to labeled sub-folders. The file structure should look like:

  $ tree data
  imagenet
  ├── train
  │   ├── class1
  │   │   ├── img1.jpeg
  │   │   ├── img2.jpeg
  │   │   └── ...
  │   ├── class2
  │   │   ├── img3.jpeg
  │   │   └── ...
  │   └── ...
  └── val
      ├── class1
      │   ├── img4.jpeg
      │   ├── img5.jpeg
      │   └── ...
      ├── class2
      │   ├── img6.jpeg
      │   └── ...
      └── ...
  

• Robustness & OOD datasets

Prepare ImageNet-A, ImageNet-R and ImageNet-C for evaluation.

Pre-trained model preparation

• For pre-trained ViT-B/16, Swin-B, and ConvNext-B models on ImageNet-21K, the model weights will be automatically downloaded when you fine-tune a pre-trained model. You can also manually download them from ViT,Swin Transformer, and ConvNext.

• For pre-trained AS-MLP-B model on ImageNet-1K, you can manually download them from AS-MLP.

Tuning Pre-trained Model via Moment Probing

To fine-tune a pre-trained ViT model via MP or MP+ on ImageNet-1K, run:

bash train_scripts/vit/imagenet_1k/train_mp.sh
bash train_scripts/vit/imagenet_1k/train_mp+.sh

Following the scripts we provided, you can easily reproduce our results.

Robustness & OOD

To evaluate the performance of fine-tuned model via MP or MP+ on Robustness & OOD, run:

bash train_scripts/vit/imagenet_a(r, c)/eval_mp.sh

checkpoint list (ViT-B/16 SUP)

Method	Acc@1(%)	#Params.(M)	log	Checkpoint
Linear Probing (LP)	82.04	0.77	log	params
Full Finetuning (FT)	83.58	86.57	log	params
Moment Probing (MP)	83.15	3.65	log	params
Moment Probing+ (MP+)	83.62	4.10	log	params

TODO

• Release training codes and related checkpoints of ViT on ImageNet.
• Release other model training codes and checkpoints.
• Release the code on other datasets (e.g. FGVC, Inat2017).

Citation

@inproceedings{gao2023iccv,
  title={Tuning Pre-trained Model via Moment Probing},
  author={Mingze Gao and Qilong Wang and Zhenyi Lin and Pengfei Zhu and Qinghua Hu and Jingbo Zhou},
  booktitle = {ICCV},
  year={2023}
}

Acknowledgement

Our code are built following DeiT, Swin Transformer, ConvNeXt, CLIP, SSF , thanks for their excellent work