An issue about the default initialize methods.

[LiChenda]

Hello, I'm training some custom models with ESPNet2. And I find that the default initialize method checking the bias parameters of NN by the p.dim() function. This check is not precise (some parameters my also make p.dim() == 1 be True ) and set them to zero may lead to abnormal training. (I found one of my module always output zeros and the training may dead with ReLU. After removing the following lines, the training becomes OK. )

espnet/espnet2/torch_utils/initialize.py

Lines 77 to 80 in 96bd746

	# bias init
	for p in model.parameters():
	if p.dim() == 1:
	p.data.zero_()

[sw005320]

@LiChenda, thanks for the report.
Can you tell me more examples of when some parameters also make p.dim() == 1 be True?
How about making your treatment an option?

[LiChenda]

Some parameters like the weight $\gamma$ in BatchNorm, the weight $\alpha$ in PReLU, and some custom parameters defined with torch.nn.Parameter() may make p.dim() == 1 be True.
One possible solution might be updating these lines https://github.com/espnet/espnet/blob/96bd74641ceb463096067223d0734f70bddd8def/espnet2/torch_utils/initialize.py#L77-L80
with:

for name, p in model.named_parameters(): 
    if 'bias' in name:
        p.data.zero_() 

[LiChenda]

BTW, this function is called by amlost all the tasks in ESPnet2, I'm not sure if these updating keeps the previously trained model reproducible.

[sw005320]

for name, p in model.named_parameters(): 
    if 'bias' in name:
        p.data.zero_() 

This sounds good to me.
Yes, we can make this an option and make default false in some timing.
Meanwhile, we can conduct some tests and make default true.
Can you make a PR?

[LiChenda]

Sure!

[popcornell]

for name, p in model.named_parameters(): 
    if 'bias' in name:
        p.data.zero_() 

maybe make it case insensitive too just to be sure.
This is difficult to implement in a scalable way. Current solution thus makes sense.

Maybe raising a warning also could be useful so user knows that the bias is set to zero. But warning should be likely raised once (e.g. for each layer of that class) otherwise will be too verbose maybe.

[b-flo]

BTW, this function is called by amlost all the tasks in ESPnet2, I'm not sure if these updating keeps the previously trained model reproducible.

It should be noted that args.init is set to None by default for all tasks and most configs don't use initialization.
We currently have 1026 configs (!!). 172 configs use either chainer or xavier_uniform initialization, from which 110 are for the ASR task and 59 for the ENH task.
From what I see, it doesn't seem to impact config we usually use, at least for ASR.

[LiChenda]

BTW, this function is called by amlost all the tasks in ESPnet2, I'm not sure if these updating keeps the previously trained model reproducible.

It should be noted that args.init is set to None by default for all tasks and most configs don't use initialization. We currently have 1026 configs (!!). 172 configs use either chainer or xavier_uniform initialization, from which 110 are for the ASR task and 59 for the ENH task. From what I see, it doesn't seem to impact config we usually use, at least for ASR.

Thanks for pointing it out. I just made a PR for this issue, see #4574 .

[LiChenda]

The PR is merged, so I close this issue.

[kan-bayashi]

I found this issue has an impact on the most of TTS models since TTS modules is initialized using this method as a default.

espnet/espnet2/tts/fastspeech2/fastspeech2.py

Lines 470 to 475 in 14fcb2d

	# initialize parameters
	self._reset_parameters(
	init_type=init_type,
	init_enc_alpha=init_enc_alpha,
	init_dec_alpha=init_dec_alpha,
	)

espnet/espnet2/tts/fastspeech2/fastspeech2.py

Lines 828 to 829 in 14fcb2d

	if init_type != "pytorch":
	initialize(self, init_type)

The model including BatchNorm2d or BatchNorm1d modules should be improved with this changes.
(I'm not sure what is happened before fixing this issue...)