How to use a non-timm pretrained image tower with a HF pretrained text tower?

[bryant1410]

I see that in #236 you folks used an OpenCLIP-pretrained image tower (ViT-H/14) along with an HF pretrained text tower (xlm-roberta-large). How do I reproduce using that config (xlm-roberta-large-ViT-H-14)? What flags did you pass for it? Because it seems that with --pretrained-image I can only pass a timm model in the config, otherwise I hit this assert:

open_clip/src/open_clip/factory.py

Line 177 in fb72f4d

assert False, 'pretrained image towers currently only supported for timm models'

If I try with --pretrained $NAME instead of --pretrained-image, then it's gonna look for pretrained checkpoints for the model name (xlm-roberta-large-ViT-H-14), then look for the $NAME weights, where the only one for that config is not ViT-H/14 but the one obtained after using it with XLM-RoBERTa. Still, if that weight was correct, using --pretrained would override the pretrained text tower weights that were already loaded, if I understand correctly.

What am I missing?

[bryant1410]

BTW, it'd be great if, for each trained model, you could share the exact version of the repo (commit hash I guess) and the command used to launch it (esp. to see which flags were used).

[bryant1410]

I guess this is the current answer? #237 (comment) Not sure.

Also, I guess another workaround is to use the HF version and load it as a timm model.

[rwightman]

@bryant1410 there's no built in support to load existing CLIP trained weights for image tower into a new model with a different text encoder. There were some PR's to add support but they didn't get merged due to being combined with lots of other changes or got lost in the shuffle, would be good to add at some point.

To do this right now you'd have to hack some code to load the image weights, the timm pretrained flag just passes down to timm to load pretrained (imagenet) weights.