BLIP safetensors file does not work

[aliencaocao]

Is this even tested? Or something went wrong?

EDIT: it works only if you use the pytorch bins and index file from the omniparser HF repo, not if you use safetensors with the index file from original model's HF repo.

[yadong-lu]

It should work if you download the files from the HF repo. Feel free to update if it doesn't work for you.

[aliencaocao]

Yes only the pt files work, the safetensor files in the HF repo don't work due to lack of index json, and if I borrow the index json from the original model, it loads the weight wrongly.
So there fix would be either to remove the safetensors from your HF repo, or add your version of the index.json.
This also causes #8, when the weights are indeed there but it is missing the index json.

[yadong-lu]

Yes only the pt files work, the safetensor files in the HF repo don't work due to lack of index json, and if I borrow the index json from the original model, it loads the weight wrongly. So there fix would be either to remove the safetensors from your HF repo, or add your version of the index.json. This also causes #8, when the weights are indeed there but it is missing the index json.

which caption model you are referring to? I just to download a clean copy of weights from current hf repo: https://huggingface.co/microsoft/OmniParser/tree/main, and didn't encounter any problem of loading for both finetuning blip2 and florence.

Also I checked the output using blip2, the result makes sense to me:

[aliencaocao]

If you only download the safetensors and not .pt file. They are duplicates and only one of them is needed. Else, you would be downloading 2x the weights at 14gb

[abrichr]

@aliencaocao I believe this is confirmed working with huggingface_hub weights downloaded via download.py in #52:

(Also extended to return bounding boxes.)

[aliencaocao]

Yes if you download using huggingface_hub it downloads the duplicated weight files, which includes the correct file, and also the not working file that i have been referring to.

[yadong-lu]

@aliencaocao
I checked the commit history, it seems safetensors are added automatically by HF in this commit (which is strange): https://huggingface.co/microsoft/OmniParser/commit/43bce5b3ddbbfce86b49fa08ae02dd5b55b41fc0.

I loaded this safetensor file and the output is nonsense. I will remove the safetensor file from HF for now and make it default to use .bin file for model weights. Thanks for noting this.

[aliencaocao]

Yep that would clear the confusion for new users. Thanks