re-fix norm compatibility in scale discriminator

[kan-bayashi]

What?

• Fix the norm compatibility issue of the scale discriminator in fine-tuining
• Continue from fix norm compatibility in scale discriminator #5240

Why?

When using load_pretrained_model, the state_dict of destination is updated with that of source.

espnet/espnet2/torch_utils/load_pretrained_model.py

Line 114 in 161e4bb

dst_state.update(src_state)

Therefore, when there is key mismatch, the final state dict includes keys included in both destination and source dict.
As a result, the state_dict at this point includes weight from source dict and weight_g / weight_v from destination dict:

(Pdb) state_dict.keys()
dict_keys(['discriminator.msd.discriminators.0.layers.0.0.bias', 'discriminator.msd.discriminators.0.layers.0.0.weight_orig', 'discriminator.msd.discriminators.0.layers.0.0.weight_u', 'discriminator.msd.discriminators.0.layers.0.0.weight_v', 'discriminator.msd.discriminators.0.layers.1.0.bias', 'discriminator.msd.discriminators.0.layers.1.0.weight_orig', 'discriminator.msd.discriminators.0.layers.1.0.weight_u', 'discriminator.msd.discriminators.0.layers.1.0.weight_v', 'discriminator.msd.discriminators.0.layers.2.0.bias', 'discriminator.msd.discriminators.0.layers.2.0.weight_orig', 'discriminator.msd.discriminators.0.layers.2.0.weight_u', 'discriminator.msd.discriminators.0.layers.2.0.weight_v', 'discriminator.msd.discriminators.0.layers.3.0.bias', 'discriminator.msd.discriminators.0.layers.3.0.weight_orig', 'discriminator.msd.discriminators.0.layers.3.0.weight_u', 'discriminator.msd.discriminators.0.layers.3.0.weight_v', 'discriminator.msd.discriminators.0.layers.4.0.bias', 'discriminator.msd.discriminators.0.layers.4.0.weight_orig', 'discriminator.msd.discriminators.0.layers.4.0.weight_u', 'discriminator.msd.discriminators.0.layers.4.0.weight_v', 'discriminator.msd.discriminators.0.layers.5.0.bias', 'discriminator.msd.discriminators.0.layers.5.0.weight_orig', 'discriminator.msd.discriminators.0.layers.5.0.weight_u', 'discriminator.msd.discriminators.0.layers.5.0.weight_v', 'discriminator.msd.discriminators.0.layers.6.0.bias', 'discriminator.msd.discriminators.0.layers.6.0.weight_orig', 'discriminator.msd.discriminators.0.layers.6.0.weight_u', 'discriminator.msd.discriminators.0.layers.6.0.weight_v', 'discriminator.msd.discriminators.0.layers.7.bias', 'discriminator.msd.discriminators.0.layers.7.weight_orig', 'discriminator.msd.discriminators.0.layers.7.weight_u', 'discriminator.msd.discriminators.0.layers.7.weight_v', 'discriminator.msd.discriminators.0.layers.0.0.weight', 'discriminator.msd.discriminators.0.layers.1.0.weight', 'discriminator.msd.discriminators.0.layers.2.0.weight', 'discriminator.msd.discriminators.0.layers.3.0.weight', 'discriminator.msd.discriminators.0.layers.4.0.weight', 'discriminator.msd.discriminators.0.layers.5.0.weight', 'discriminator.msd.discriminators.0.layers.6.0.weight', 'discriminator.msd.discriminators.0.layers.7.weight'])

As a result, the loading is failed. This PR fixes this issue.

Related issues

• A bug in HiFiGANScaleDiscriminator #4595
• Mismatched keys in pretrained JETS model #5237
• Unexpected key(s) in state_dict when running Japanese TTS Training （日本語TTS学習でエラーが出ました） #5239

[codecov]

Codecov Report

Merging #5249 (b685366) into master (b5a88e9) will increase coverage by 3.61%.
The diff coverage is 25.00%.

@@            Coverage Diff             @@
##           master    #5249      +/-   ##
==========================================
+ Coverage   71.37%   74.99%   +3.61%     
==========================================
  Files         655      655              
  Lines       58546    58552       +6     
==========================================
+ Hits        41789    43909    +2120     
+ Misses      16757    14643    -2114     

Flag	Coverage Δ
test_integration_espnet1	66.24% <ø> (+<0.01%)	⬆️
test_integration_espnet2	47.64% <25.00%> (?)
test_python	65.27% <0.00%> (-0.01%)	⬇️
test_utils	23.27% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
espnet2/gan_tts/hifigan/hifigan.py	82.42% <25.00%> (-0.84%)	⬇️

... and 62 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more