Masked Image Modeling with Local Multi-Scale Reconstruction

PyTorch implementation of
Masked Image Modeling with Local Multi-Scale Reconstruction
CVPR 2023 Highlight (Top 2.6%)

Masked Image Modeling (MIM) achieves outstanding success in self-supervised representation learning. Unfortunately, MIM models typically have huge computational burden and slow learning process, which is an inevitable obstacle for their industrial applications. Although the lower layers play the key role in MIM, existing MIM models conduct reconstruction task only at the top layer of encoder. The lower layers are not explicitly guided and the interaction among their patches is only used for calculating new activations. Considering the reconstruction task requires non-trivial inter-patch interactions to reason target signals, we apply it to multiple local layers including lower and upper layers. Further, since the multiple layers expect to learn the information of different scales, we design local multi-scale reconstruction, where the lower and upper layers reconstruct fine-scale and coarse-scale supervision signals respectively. This design not only accelerates the representation learning process by explicitly guiding multiple layers, but also facilitates multi-scale semantical understanding to the input. Extensive experiments show that with significantly less pre-training burden, our model achieves comparable or better performance on classification, detection and segmentation tasks than existing MIM models.

Pre-Trained Models

Backbone	#Params	Target	GPU Hours/Ep.	PT Epoch	PT Resolution	PT log/ckpt	Top-1 (%)
ViT-B	86M	HOG	0.7	1600	224x224	log/ckpt	84.0
ViT-L	307M	HOG	1.0	800	224x224	log/ckpt	85.8
Swin-B	88M	Pixel	1.0	400	224x224	log/ckpt	84.0
Swin-B	88M	HOG	1.1	400	224x224	log/ckpt	84.1
Swin-L	197M	HOG	1.6	800	224x224	log/ckpt	85.6

The pre-training and fine-tuning instruction can be found in ViT, Swin and semantic_segmentation.

Citation

If you find this project useful in your research, please consider cite:

@inproceedings{wang2023masked,
  title={Masked Image Modeling with Local Multi-Scale Reconstruction},
  author={Wang, Haoqing and Tang, Yehui and Wang, Yunhe and Guo, Jianyuan and Deng, Zhi-Hong and Han, Kai},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={2122--2131},
  year={2023}
}

Acknowledgement

This code is built upon the implementation from MAE, GreenMIM, MMSeg and BEiT.