Modulate Your Spectrum in Self-Supervised Learning

This is an official implementation of INTL.

@article{weng2024modulate,
  title={Modulate Your Spectrum in Self-Supervised Learning},
  author={Weng, Xi and Ni, Yunhao and Song, Tengwei and Luo, Jie and Anwer, Rao Muhammad and Khan, Salman and Khan, Fahad Shahbaz and Huang, Lei},
  journal={International Conference on Learning Representations (ICLR)},
  year={2024}
}

Requirements

• Install PyTorch (pytorch.org)
• Install wandb for Logging (wandb.ai)

Experiments on Standard SSL Benchmark

The code includes experiments in section 5.

Evaluation on ImageNet

Our pretrained ResNet-50 INTL (using multi-crop and EMA):

epochs	bs	top-1 acc	download
100	256	73.5%	script	ResNet-50	full checkpoint	lincls checkpoint
200	256	75.2%	script	ResNet-50	full checkpoint	lincls checkpoint
400	256	76.1%	script	ResNet-50	full checkpoint	lincls checkpoint
800	256	76.6%	script	ResNet-50	full checkpoint	lincls checkpoint

Our pretrained ResNet-50 INTL (without multi-crop or EMA):

epochs	bs	top-1 acc	download
800	256	73.1%	script	ResNet-50	full checkpoint	lincls checkpoint

You can choose to download either the weights of the pretrained ResNet-50 network or the full checkpoint, which also contains the weights of the projection and the state of the optimizer.

Evaluation on small and medium size datasets.

The datasets include CIFAR-10, CIFAR-100 and ImageNet-100.

The results and unsupervised pretraining scripts are shown in the following table:

CIFAR-10				CIFAR-100				ImageNet-100
top-1	5-nn	top-5	script	top-1	5-nn	top-5	script	top-1	5-nn	top-5	script
92.60	90.03	99.80	script	70.88	61.90	92.13	script	81.68	73.46	95.42	script

Transferring to Object Detection

Same as MoCo for object detection transfer, please see ./detection.

Transfer learning results of INTL (200-epochs pretrained on ImageNet):

downstream task	$AP_{50}$	$AP$	$AP_{75}$	ckpt	log
COCO detection	$61.2_{±0.08}$	$41.2_{±0.12}$	$44.7_{±0.19}$	coco_ckpt	coco_log
COCO instance seg.	$57.8_{±0.04}$	$35.7_{±0.05}$	$38.1_{±0.12}$	coco_ckpt	coco_log