The downloaded pretrained weights cannot be imported correctly using the jupyter notebook.

[cenjinglun]

I have downloaded the pretrain weights from Google Drive. But it seems there's something wrong with it. It's not working because of mismatching!

https://drive.google.com/file/d/14pXWwB4Zm82rsDdvbGguLfx9F8aM7ovT/view?usp=sharing
https://drive.google.com/file/d/1IdaBtMSvtyzF0ByVaBHtvM0JYSXRExRX/view?usp=sharing

How can I do for this?

[rmokady]

Hi @cenjinglun,
It seems that you have a mismatch between the pretrained model and the configuration. Note that these models are using with MLP mapping network and not transformer.

To my opinion, the best way to make sure the pretrained model and configuration is matched, is using the colab notebook we provide.

I'll also will try to make it more clear in the readme.

[cenjinglun]

Thank you very much for pointing out my mistakes.

I want to run it on local, so I downloaded the weights mentioned above. However, because I am not so familiar with the code, I only replaced the model_path in clip_prefix_captioning_inference.ipynb.

model = ClipCaptionPrefix(prefix_length, clip_length=40, prefix_size=640, num_layers=8, mapping_type='transformer')
model_path = 'path/to/coco_weight.pt'       # what I add
model.load_state_dict(torch.load(model_path, map_location=CPU))

If possible, could you tell me the where shoule I load these weights?

[rmokady]

You can use the exact code appear in the notebook:
model = ClipCaptionModel(prefix_length) model.load_state_dict(torch.load(model_path, map_location=CPU))
Of course you need you use also other cells

Looking at your code, it seems you used the transformer notebook instead
Please try the clip_prefix_captioning_inference.ipynb notebook

[cenjinglun]

Ok, it works.

Thanks.