pretrained weights from `self_supervised_pretraining` has extra key

[yiheng-wang-nv]

I downloaded the pretrained weights mentioned in https://github.com/Project-MONAI/tutorials/blob/master/self_supervised_pretraining/README.md and tried to use it in ssl_finetuning_train.py, but met an error:

Unexpected key(s) in state_dict: "patch_embedding.cls_token".

I compared the keys between the download weight file vitautoenc_weights.pt and the class ViTAutoEnc and found that patch_embedding.cls_token is not in the latest implementation of ViTAutoEnc network. Maybe the pretrained weight is trained in an old version of the network.

Here should the pretrained weight be updated, or the ssl_finetuning_train.py should be modified to load weights correctly?

[yiheng-wang-nv]

Hi @finalelement , could you please help to check it? Thanks!

[wyli]

It is a breaking change Project-MONAI/MONAI#3475 let's raise this issue on Monday, we may need to update the demo to only support 0.8.0, or have a workaround to load with mismatched model variables, or retrain it.

[wyli]

Hi @yiheng-wang-nv, perhaps we can have a similar weight dict modification like this line?

tutorials/self_supervised_pretraining/ssl_finetuning_train.py

Line 199 in d4e3f20

vit_weights.pop('conv3d_transpose.weight')

[yiheng-wang-nv]

Hi @wyli , if users do pretraining by themselves with the new API, the weights will not have this layer. Therefore, can we do sth like here?

[wyli]

sure, could you help create a pull request based on this approach?