Error in Loading Weights When Fine-tuning

[KingJamesSong]

Hi,

I want to perform fine-tuning on the HMDB51 dataset using the pre-trained models on K400.

I use the script 'train_hmdb51_TCP_TSN_R50.sh' and 'train_hmdb51_TCP_TEA_R50.sh' to load the weights from two pre-trained models 'K400_TCP_TSN_R50_8f.pth.tar' and 'K400_TCP_TEA_R50_8f.pth.tar', respectively.

However, both scripts have error in unexpected keys.

For TEA, the error says: "Unexpected key(s) in state_dict: "module.base_model.iSQRT.layer_reduce1.0.weight", "module.base_model.iSQRT.layer_reduce1.1.weight", "module.base_model.iSQRT.layer_reduce1.1.bias", "module.base_model.iSQRT.layer_reduce1.1.running_mean", "module.base_model.iSQRT.layer_reduce1.1.running_var", "module.base_model.iSQRT.layer_reduce1.1.num_batches_tracked", "module.base_model.iSQRT.layer_reduce2.0.weight", "module.base_model.iSQRT.layer_reduce2.1.weight", "module.base_model.iSQRT.layer_reduce2.1.bias", "module.base_model.iSQRT.layer_reduce2.1.running_mean", "module.base_model.iSQRT.layer_reduce2.1.running_var", "module.base_model.iSQRT.layer_reduce2.1.num_batches_tracked", "module.base_model.iSQRT.att_module.conv_1.weight", "module.base_model.iSQRT.att_module.conv_1.bias", "module.base_model.iSQRT.att_module.conv_2.weight", "module.base_model.iSQRT.att_module.conv_2.bias", "module.base_model.iSQRT.att_module.conv_1d.weight", "module.base_model.iSQRT.att_module.conv_1d.bias", "module.base_model.iSQRT.att_module.sp_att.conv_theta.weight", "module.base_model.iSQRT.att_module.sp_att.conv_theta.bias", "module.base_model.iSQRT.att_module.sp_att.conv_phi.weight", "module.base_model.iSQRT.att_module.sp_att.conv_phi.bias", "module.base_model.iSQRT.att_module.sp_att.conv_g.weight", "module.base_model.iSQRT.att_module.sp_att.conv_g.bias", "module.base_model.iSQRT.att_module.sp_att.norm.weight", "module.base_model.iSQRT.att_module.sp_att.norm.bias", "module.base_model.iSQRT.att_module.sp_att.norm.running_mean", "module.base_model.iSQRT.att_module.sp_att.norm.running_var", "module.base_model.iSQRT.att_module.sp_att.norm.num_batches_tracked"."

For TSN, the error is "Unexpected key(s) in state_dict: "module.base_model.layer4.iSQRT.layer_reduce1.weight", "module.base_model.layer4.iSQRT.layer_reduce_bn1.weight", "module.base_model.layer4.iSQRT.layer_reduce_bn1.bias", "module.base_model.layer4.iSQRT.layer_reduce_bn1.running_mean", "module.base_model.layer4.iSQRT.layer_reduce_bn1.running_var", "module.base_model.layer4.iSQRT.layer_reduce_bn1.num_batches_tracked", "module.base_model.layer4.iSQRT.layer_reduce2.weight", "module.base_model.layer4.iSQRT.layer_reduce_bn2.weight", "module.base_model.layer4.iSQRT.layer_reduce_bn2.bias", "module.base_model.layer4.iSQRT.layer_reduce_bn2.running_mean", "module.base_model.layer4.iSQRT.layer_reduce_bn2.running_var", "module.base_model.layer4.iSQRT.layer_reduce_bn2.num_batches_tracked", "module.base_model.layer4.iSQRT.att_module.conv_1.weight", "module.base_model.layer4.iSQRT.att_module.conv_1.bias", "module.base_model.layer4.iSQRT.att_module.conv_2.weight", "module.base_model.layer4.iSQRT.att_module.conv_2.bias", "module.base_model.layer4.iSQRT.att_module.conv_1d.weight", "module.base_model.layer4.iSQRT.att_module.conv_1d.bias", "module.base_model.layer4.iSQRT.att_module.sp_att.conv_theta.weight", "module.base_model.layer4.iSQRT.att_module.sp_att.conv_theta.bias", "module.base_model.layer4.iSQRT.att_module.sp_att.conv_phi.weight", "module.base_model.layer4.iSQRT.att_module.sp_att.conv_phi.bias", "module.base_model.layer4.iSQRT.att_module.sp_att.conv_g.weight", "module.base_model.layer4.iSQRT.att_module.sp_att.conv_g.bias", "module.base_model.layer4.iSQRT.att_module.sp_att.norm.weight", "module.base_model.layer4.iSQRT.att_module.sp_att.norm.bias", "module.base_model.layer4.iSQRT.att_module.sp_att.norm.running_mean","

I think something goes wrong with the key mappings.

Can you have a look at the codes and help to fix the issue?

Thanks in advance!

[ZilinGao]

@KingJamesSong Hi~

Thanks for your attention to our work!

It is a bug about not setting the args.keys_old to True. And it has been fixed now.

Thanks for your feedback~

[KingJamesSong]

Hi,
Thanks for your update.
Now I can load the weights and fine-tune on HMDB51, but I found the result is much lower than reported in the paper. :(
I use the script "train_hmdb51_TCP_TEA_R50.sh" for training and "test_hmdb51_TCP_TEA_R50.sh" for testing.
After fine-tuning, I can get the top-1 accuracy of about 64%, which is significantly smaller than the 76% reported in the paper. Is it related to the test setting or anything else?
Could
Thanks!

[ZilinGao]

@KingJamesSong
Hello,
I'm sorry for hearing that.
Did you change the default settings in the script, e.g. bs or lr ? If the bs is changed, the lr should be updated linearly as well~