Resize embeddings and vocab

[ambiSk]

I finetuned text encoder of CLIP and added some additional tokens to it, I would like to know is there a way to load the checkpoint with greater embedding size?

[mitchellnw]

not totally sure if this will work but worth trying making a new model config and changing vocab_size there? e.g., make a copy of this https://github.com/mlfoundations/open_clip/blob/main/src/open_clip/model_configs/ViT-B-16.json called ViT-B-16-bigvocab then use --model ViT-B-16-bigvocab.

[rwightman]

don't think resizing existing models is practical, but definitely possible to make new ones, or us HF pretrained models that have larger vocab / context length