-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MAE finetune train loss nan #42
Comments
Hey @CodingMice , It might be due to the amp.autocast() . Disable it via amp.autocast(enabled=False) solves my problem. |
It would be great if more context is provided here. There could be multiple ways the Loss goes to NaN, and amp can indeed be one of them. |
FYI - This PyTorch issue thread with a long history could be a hint... And here's troubleshooting for the issue (also suggested in the thread): Anyway, a quick fix would be as commented by @Jeff-LiangF. |
I've faced the same issue. |
how to set the value of gradient clipping? 0.1? |
info : Loss is nan, stopping training.
The text was updated successfully, but these errors were encountered: