LR too high for gradient accumulation

[marii-moe]

We were not dividing by the number of batches to accumulate, so this effectively increasing the learning rate. Added test to make sure this is fixed. I think this thread was lost when fastai/fastai2 got moved to fastai/fastai: fastai/fastai2#194

fixes #3023

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[jph00]

Nice one!

[muellerzr]

I know this is fixed and merged now, but I'm hesitant to use it. I was seeing better results with the old implementation of the high LR and got worse results here. Do you have any advice @marii-moe as to how I should adapt my old LR to work with this new adjustment?

[marii-moe]

I know this is fixed and merged now, but I'm hesitant to use it. I was seeing better results with the old implementation of the high LR and got worse results here. Do you have any advice @marii-moe as to how I should adapt my old LR to work with this new adjustment?

I am not familiar with your particular example, but you should get approximately the same results if you set your learning rate like so:
new_lr = old_lr*n_acc/bs