[docs] Warn that GradScaler can scale under 1

[janeyx99]

Completes action item 1 in #99640

Stack from ghstack (oldest at bottom):

• -> [docs] Warn that GradScaler can scale under 1 #101569

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/101569

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a9ed77c:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[janeyx99]

@stas00 @crcrpar Would appreciate your reviews on phrasing!

[stas00]

Suggested change

	AMP/fp16 may not be for every model! For example, most bf16-pretrained models cannot operate in
	the fp16 numerical range of max 65k and will cause gradients to overflow instead of underflow. In
	this case, the scale factor may decrease under 1 as an attempt to bring gradients to a number
	representable in fp16. While one may expect the scale to always be above 1, our GradScaler does
	NOT make this guarantee to maintain performance. If you encounter NaNs in your loss or gradients
	when running with AMP or fp16, verify your model is compatible.
	AMP/fp16 may not work for every model! For example, most bf16-pretrained models cannot operate in
	the fp16 numerical range of max 64k and will cause gradients to overflow instead of underflow. In
	this case, the scale factor may decrease under 1 as an attempt to bring gradients to a number
	representable in fp16 dynamic range. While one may expect the scale to always be above 1, our GradScaler does
	NOT make this guarantee to maintain performance. If you encounter NaNs in your loss or gradients
	when running with AMP/fp16, verify your model is compatible.

[janeyx99]

Thanks for the comments! I thought the max for fp16 is 65.5k or something, no?

[stas00]

The maximum representable value is (2−2−10) × 215 = 65504

which is 64K (65504/1024)

[janeyx99]

Ah, I see the distinction was regarding the K vs k. I'm just going to use the actual number for maximal clarity.

[stas00]

Suggested change

	For performance reasons, the scale factor is not guaranteed to be above 1. If the
	scale falls below 1 and/or you are seeing NaNs in your gradients or loss, something
	is likely wrong. For example, bf16-pretrained models are often incompatible with
	AMP/fp16 due to differing numerical ranges.
	For performance reasons, the scale factor is not guaranteed to be above 1. If the
	scale falls below 1 and/or you are seeing NaNs in your gradients or loss, something
	is likely wrong. For example, bf16-pretrained models are often incompatible with
	AMP/fp16 due to differing dynamic ranges.

[stas00]

why is this a performance reason? I'd call it "numerical stability reasons", no?

[janeyx99]

Because adding a check would incur a device sync per step call, and device syncs are expensive

[janeyx99]

This is the current reason why we don't, but I intentionally did not mark this PR as one that would "fix" the issue as I'd like to leave that one open for more thoughts from the community if it comes up more often

[stas00]

oh, I understand what you meant now.

but if this is not synced how will a user know that it fell below 1? I'm not suggesting to have the overhead, just trying to understand the explanation.

[janeyx99]

Oh, like if they check the scale directly as a part of debugging, though I think people would notice the NaNs first

[ngimel]

add here that for performance reasons we are not checking scale factor value to avoid synchronization, so it is not guaranteed to be above 1.

[janeyx99]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here