[PyTorch] DINO: self-DIstillation with NO labels

PyTorch implementation of "Emerging Properties in Self-Supervised Vision Transformers"

Concept

Concept ot the DINO [1].

Results

Summary

	Student ($x_1$)	Teacher ($x_2$)
Before (mismatch)
After (matched)

Detail (in Training)

Epoch	Student ($x_1$)	Teacher ($x_2$)
0
1
30
150
300

Requirements

• PyTorch 2.0.1

Reference

[1] Mathilde Caron, et al. "Emerging Properties in Self-Supervised Vision Transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.