You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi - I'm having trouble with prefix_size mismatch on my own finetuned model with CLIP features using ViT B-32. I'm learning just the transforming mapping and no gpt-2 finetuning, using the commands given in the readme.
My understanding is that CLIP with ViT B-32 uses prefix_size = 512 while ResNet encoders use prefix_size = 640, e.g. Radford et al Visual Transformers Appendix F, and also here in train.py:
model = ClipCaptionPrefix(prefix_length, clip_length=40, prefix_size=640,
num_layers=8, mapping_type='transformer')
This worked for me for your pretrained coco model, but not my finetuned model, where I get a dimensionality mismatch between 512/640. Can you help me out here? Should I be using prefix_size = 640 or 512 for training/inference?
Thank you!
Stella
Ps: FYI there are a few typos in the commands in the Readme.md, where num_layres should be num_layers.
The text was updated successfully, but these errors were encountered:
Hi - I'm having trouble with prefix_size mismatch on my own finetuned model with CLIP features using ViT B-32. I'm learning just the transforming mapping and no gpt-2 finetuning, using the commands given in the readme.
My understanding is that CLIP with ViT B-32 uses
prefix_size = 512
while ResNet encoders useprefix_size = 640
, e.g. Radford et al Visual Transformers Appendix F, and also here intrain.py
:CLIP_prefix_caption/train.py
Line 355 in 1ad805a
However, in the prediction script, which I've essentially copied from the transformer notebook (https://github.com/rmokady/CLIP_prefix_caption/blob/main/notebooks/transformer_inference.ipynb)
the prefix size is set to 640.
This worked for me for your pretrained coco model, but not my finetuned model, where I get a dimensionality mismatch between 512/640. Can you help me out here? Should I be using prefix_size = 640 or 512 for training/inference?
Thank you!
Stella
Ps: FYI there are a few typos in the commands in the
Readme.md
, wherenum_layres
should benum_layers
.The text was updated successfully, but these errors were encountered: