Gradient of first layer updated twice?

[zhuchen03]

With Line 99 and Line 102 of https://github.com/a1600012888/YOPO-You-Only-Propagate-Once/blob/master/experiments/CIFAR10/pre-res18.yopo-5-3/training_function.py#L99, it seems that you are trying to reset the gradient of the first layer to its original value before BP through the whole network. However, this is ineffective, since Line 99 only keeps a reference to net.conv1.weight.grad with wgrad. As a result, Line 102 does not change the value of net.conv1.weight.grad after BP. Are these two lines just redundant or are they bugs?

[a1600012888]

Yes, you are right, thanks for pointing this out. These two lines are redundant.

We are computing the co-state ( at line 110) and the gradients of all the parameters at the same time using the same BP.

During our experiment stage, we also wondered whether it is necessary not to accumulate the gradients of the parameters in the first layer, so we add that two lines of code. But at that time, I did not realize that these two lines of code did not make a change.

Thanks for taking out time to read our code.