Why add 0 loss to the original loss?

[ghost]

mmnas/search_vqa.py

Lines 285 to 288 in 552e29e

	# for avoid backward the unused params
	loss += 0 * sum(p.sum() for p in net.module.alpha_prob_parameters())
	loss += 0 * sum(p.sum() for p in net.module.alpha_gate_parameters())
	loss += 0 * sum(p.sum() for p in net.module.net_parameters())

What is this part of the codes aimed at?
I'll appreciate it if anyone could explain that.

[cuiyuhao1996]

It is a simple trick for distributed training ("distributeddataparallel') with NCCL. If you use NCCL for GPU communication in Pytorch, the unused parameters can result in some errors. However, it would not affect the multi-thread method "dataparallel ".

[ghost]

@cuiyuhao1996 I got it. Thanks a lot.