[ROCm] Support building PyTorch with GLOO on Windows

[jammm]

Helps fix imports for transformers lib which seems to import torch.distributed. ROCm/TheRock#1202
It may or may not help in proper WIndows support for torch.distributed (since RCCL isn't on Windows atm), but it does let us build pytorch w/ USE_DISTRIBUTED=ON, which helps fix import errors like the above, so users on their single GPU systems can run comfyui without facing import errors.

Tested on Windows w/ PyTorch main branch via. TheRock.

[Nem404]

@jammm sorry if I’m being impatient, but since this PR was approved, could it be merged?

[jammm]

@jammm sorry if I’m being impatient, but since this PR was approved, could it be merged?

It's not up to me to merge this unfortunately. The gloo maintainers need to merge it. It's my first PR here actually, so not sure of the exact flow.

[Nem404]

It's not up to me to merge this unfortunately. The gloo maintainers need to merge it. It's my first PR here actually, so not sure of the exact flow.

Oh, I didn’t know that. Np, then we’ll wait for them.
I just thought anything is welcome that could help with: ROCm/TheRock#1202

[d4l3k]

LGTM, sorry for the delay!

If you're on the PyTorch slack feel free to ping me directly there for faster reviews

[d4l3k]

@jammm @Nem404 we don't currently have any ROCm CI builds for Gloo. If you're interested in contributing one that would make it a lot easier to prevent regressions like this in the future

[ScottTodd]

@jammm @Nem404 we don't currently have any ROCm CI builds for Gloo. If you're interested in contributing one that would make it a lot easier to prevent regressions like this in the future

We're working on contributing Windows ROCm CI builds with pytorch/pytorch#159520 and could enable USE_GLOO there too (I have it disabled in the scripts I'm starting at pytorch/pytorch#160776 right now).

[d4l3k]

@ScottTodd that sounds reasonable to me but we should also add something to this repo. PyTorch pins a commit from Gloo so it's always trailing. We don't run PT CI on Gloo repo, only run PT CI when bumping the Gloo pinned commit

[Nem404]

Eyy good to see things evolving here