[15/N] Add allreduce_coalesced custom op with CPU/CUDA implementations

[H-Huang]

Stack from ghstack:

• Allow Process Group to support multiple backends #88330 Allow Process Group to support multiple backends and move PG specfic implementations to backend class
• Remove ProcessGroupRoundRobin #87088 Remove ProcessGroupRoundRobin
• [17/N] Add _reduce_scatter_base custom op with CPU/CUDA implementation #88903 [17/N] Add _reduce_scatter_base custom op with CPU/CUDA implementation
• [16/N] Add _allgather_base custom op with CPU/CUDA implementation #88889 [16/N] Add _allgather_base custom op with CPU/CUDA implementation
• [15/N] Add allreduce_coalesced custom op with CPU/CUDA implementations #88846 [15/N] Add allreduce_coalesced custom op with CPU/CUDA implementations
• [14/N] Refactor _new_process_group_helper() to remove repeated code #88351 [14/N] Refactor _new_process_group_helper() to remove repeated code

Context

#86225

Differential Revision: D41227740

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88846

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit af4b406:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[kwen2501]

LGTM. Note that the API allreduce_coalesced could be removed later.

[H-Huang]

@H-Huang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[H-Huang]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here