[BUG] ViT SAM trained models - Pretrained model achieve near 0 accuracy

[IdoGalil]

Hey!,
Testing vit_base_patch32_sam_224 and vit_base_patch16_sam_224 loaded with pretrained weights, I noticed a near 0 accuracy on the ImageNet validation set.

There is even an undeterminstic behavior with predicted labels between sequential runs. Debugging the code, as far as I can tell, it seems the pretrained weights were loaded properly, however (in terms of calling all the correct and necessary methods).

To Reproduce
Creating the model with:
timm_factory.create_model('vit_base_patch16_sam_224'), pretrained=True)
Extracting the proper transform (using config and create_transform(**config)) and running on ImageNet validation set.

Thanks!

[rwightman]

@IdoGalil they work, try with validate.py, there is a step btw grabbing the model pretraine cfg and calling create transform,

[IdoGalil]

I tried using validate.py, still getting near 0 accuracy:
validate.py ../../ImageNet --model vit_base_patch16_sam_224

Any other model works fine - and I've tested all of your other published models. Here is mnasnet_100 for example working great when using validate.py:

As for your comment, sorry if I'm missing something basic, but which step should be taken between grabbing the model (besides of course using eval() etc...) and creating the transform for the data? I've done only those steps with successful results for all of your other models

[trytolose]

I think I have the same issue with vit_base_patch16 and swin_base_patch_4_window12_384 - they don't evaluate properly on my dataset after finetuning. For example, efficientnets work well.

[IdoGalil]

I think I have the same issue with vit_base_patch16 and swin_base_patch_4_window12_384 - they don't evaluate properly on my dataset after finetuning. For example, efficientnets work well.

There is a clear distinction between our issues though: I haven't made any adjustments or fine-tuning to the models I've mentioned, I simply evaluated them with their pretrained weights on ImageNet with no modifications, a task on which they should definitely work properly

[trytolose]

I think I have the same issue with vit_base_patch16 and swin_base_patch_4_window12_384 - they don't evaluate properly on my dataset after finetuning. For example, efficientnets work well.

There is a clear distinction between our issues though: I haven't made any adjustments or fine-tuning to the models I've mentioned, I simply evaluated them with their pretrained weights on ImageNet with no modifications, a task on which they should definitely work properly

You're right, I tried Swin repo from Microsoft, got the same behavior. Looks like I have bugs in my code.

[rwightman]

@IdoGalil this was super confusing but I just had a though, I'm linking weights that I don't control (.npz files from the original source, they might have changed them), in my test I'd be running from my model cache ... removed from my cache and the re-downloaded weights don't work :/

EDIT: I figured it out, they removed the representation layer and used the same URL for the updated weights, set representation_size=0 in the model args, I'll fix it in the next day or so

[IdoGalil]

Works great, thanks!