Add Bfloat16 scalar support to gloo backend

[anko-intel]

There was missing support for bfloat scalars. When I use gloo backend
torch.distributed.init_process_group(backend='gloo')
and run
torch.nn.parallel.DistributedDataParallel(model)
and model has Bfloat16 features I receive following error:
RuntimeError: Invalid scalar type

This change fix this issue.
c10::BFloat16 defines conversions from/to float, so calculations are made on float for bfloat.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/113557

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit abb0a4c with merge base 115da02 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: anko-intel / name: Andrzej Kotłowski (c2a0233, abb0a4c)

[facebook-github-bot]

@XilunWu has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[XilunWu]

Thanks for adding BFloat16 support to gloo PG. I'm not sure if this is the best way to support BFloat16 because we do have float16 support in gloo (Half i.e. gloo:;float16).

[jgong5]

Thanks for adding BFloat16 support to gloo PG. I'm not sure if this is the best way to support BFloat16 because we do have float16 support in gloo (Half i.e. gloo:;float16).

@XilunWu are you suggesting to add gloo::bfloat16 to gloo instead of using c10::bfloat16?

[jgong5]

Please add UT.

[XilunWu]

@jgong5 That seems to be a reasonable call, right? But to quickly unblock, we can merge this PR once the windows part is fixed, and add bfloat16 to gloo later.

[jgong5]

@jgong5 That seems to be a reasonable call, right? But to quickly unblock, we can merge this PR once the windows part is fixed, and add bfloat16 to gloo later.

Yes, that sounds reasonable.

[XilunWu]

Just a reminder that the windows part has issue. Need to fix.

[anko-intel]

Please add UT.

I added 2 test cases for bfloat. I think this will be enough and will not increase too much the time needed for testing.

[jgong5]

LGTM as long as CI passes.

[anko-intel]

@XilunWu can we go forward with this change?

[facebook-github-bot]

@XilunWu has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[XilunWu]

LGTM! Really appreciate the effort of adding BFloat16 scalar support!

[XilunWu]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here