Why does Lightning AI Fabric gives warning to training with precision = "bf16-mixed" instead of "bf16" #18263
-
|
It gives the following warning: UserWarning: bf16 is supported for historical reasons but its usage is discouraged. Please set your precision to bf16-mixed instead! I have seen many articles training the model in pure bf16 and getting similar accuracy. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
Hi, previously, bf16 was a shorthand notation for bf16-mixed. To clearly distinguish between mixed precision and pure bf16 we introduced the notion of |
Beta Was this translation helpful? Give feedback.
-
|
Oh got it, thanks for the clarification. |
Beta Was this translation helpful? Give feedback.
-
|
During training, it would have been better if the warning about the 'bf16 notation' being deprecated was more clearly stated, advising to choose either 'bf16-mixed' or 'bf16-true'. I was under the misconception that 'bf16-true' was the one deprecated. I just realized this now. |
Beta Was this translation helpful? Give feedback.
Hi, previously, bf16 was a shorthand notation for bf16-mixed. To clearly distinguish between mixed precision and pure bf16 we introduced the notion of
bf16-mixedandbf16-truewhile justbf16still is a deprecated alias forbf16-mixed