New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix reported training loss when using AMP #17

Merged

mallamanis merged 1 commit into master from dev/miallama/fix-training-loss-scaling

Jul 25, 2021

Contributor

mallamanis commented Jul 25, 2021

Scaling has to happen after reporting the loss to be accurate.
Also avoid GPU->CPU transfer of the minibatch loss during each iteration.


          Fix reported training loss when using AMP

2ba4358

mallamanis self-assigned this

mallamanis merged commit 6c55894 into master

mallamanis deleted the dev/miallama/fix-training-loss-scaling branch

July 25, 2021 19:09

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet