Imagenet Pretrained Weights

[RaivoKoot]

Thanks for the work! In their paper they say
For all our experiments, we adopt the “Base” ViT model architecture (Dosovitskiy et al., 2020) pretrained on ImageNet.

I know that you said the official weights trained on kinetics and such are not officially released yet. However, I am not interested in those but am actually in need of the initial weights of the network just based on ViT Imagenet pretraining. I need to train this implementation of yours starting from those. From what it looks like, you don't have weights for this implementation that come from imagenet pretraining, do you?

[jwohlwend]

+1

[MohamedOsman1998]

+1

[tcapelle]

I am not sure it is possible, but ViT weights are available on timm and others.

[RaivoKoot]

For now, I found this repository that includes ViT initialization (https://github.com/m-bain/video-transformers).

[tcapelle]

You can even download a pretrained ViT from huggingface: https://huggingface.co/google/vit-large-patch16-224