do not scale gradient in bf16 mode

[kashif]

GradScaler only makes sense in float16 mode and not for bf16 on the GPU. Removed block of code which did this for bf16.

[sgugger]

Let's just wait a little bit for the conversation in the internal thread to be concluded before merging, but otherwise LGTM!
Thanks for making the fix!

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[kashif]

would be great if @stas00 can also have a look 🙇🏽

[stas00]

I confirm.

bf16 has an almost the same dynamic range as fp32, so no scaling is needed.

In case you want to do another PR, @kashif - the same needs to be fixed in HF Trainer

https://github.com/huggingface/transformers/blob/98d88b23f54e5a23e741833f1e973fdf600cc2c5/src/transformers/trainer.py#L573

probably it should be:

                if self.amp_dtype == torch.float16:
                    self.do_grad_scaling = True

[kashif]

sure @stas00 let me get the Trainer fixed