I think `optimizer.zero_grad()` should go before `loss.backward()`

[geopapa11]

I think optimizer.zero_grad() should go before loss.backward() to ensure that gradients are zeroed out before the calculation of the gradients that takes place inside loss.backward().

tutorials/beginner_source/basics/quickstart_tutorial.py

Line 157 in 30e14df

optimizer.zero_grad()

[jmarintur]

Hi @geopapa11, indeed, optimizer.zero_grad() should go before loss.backward(). I've just created a branch to modify it in my fork. Please let me know if you want to be the one creating the PR instead.

[geopapa11]

Thanks @jmarintur, I asked the person who made the corresponding change in this commit about a year ago to make sure I am not missing anything.
geopapa11@be6e863

[jmarintur]

Hi @geopapa11 , I quickly jumped into it. I thought about possible scenarios this would not be ideal. For a simple training loop, as it is, should work too.

[jmarintur]

I was looking at more issues in this repo and I found that this was already mentioned in issue#2693. In the comments, they pointed out that the accepted answer is to run first the optimizer.zero_grad() , then loss.backward(), and finally optimizer.step().

@svekars, for consistency, shouldn't we change it to the aforementioned order?

[albanD]

As discussed in #2800 I think we do want to keep the current order to ensure minimal memory use which is a generally important concern these days.

Thanks for bringing this up though!