Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Promote NativeMixedPrecision to default MixedPrecision #3127

Closed
jph00 opened this issue Jan 6, 2021 · 0 comments
Closed

Promote NativeMixedPrecision to default MixedPrecision #3127

jph00 opened this issue Jan 6, 2021 · 0 comments
Labels

Comments

@jph00
Copy link
Member

jph00 commented Jan 6, 2021

The NativeMixedPrecision has been available in fastai since v2's release, which uses torch.cuda.amp to handle mixed precision training. MixedPrecision, on the other hand, previously used a mix of fastai's own code and code from NVIDIA and Pytorch.

We've now improved and carefully tested NativeMixedPrecision, and believe it is now generally faster and more reliable than the fastai version, and we've also checked it works well when combined with other callbacks (including gradient clipping, gradient accumulation, and distributed training).

Therefore, now when you use the MixedPrecision callback, you're actually getting the native version. To get the fastai version, you should instead use NonNativeMixedPrecision.

Similarly, if you use the Learner.to_fp16 method, you'll get the native version of the callback. Use Learner.to_non_native_fp16 for the fastai version.

If you find any issues where the old callback works better than the new one, please tell us (with a reproducible example) so we can fix it!

@jph00 jph00 added the breaking label Jan 6, 2021
@jph00 jph00 closed this as completed in c56afb8 Jan 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant