[NCCL][PyTorch] Fix test-fusible-ops-file-rendezvous-bug

[nWEIdia]

world_group() used a single hardcoded init_method="file:///tmp/rdzv",
shared across every world_size this test parametrizes over ([device_count(), 1, 2]), and neither world_group() nor test_distributed_fuser_ops ever calls destroy_process_group() or removes the file afterwards. A world_size=N run's leftover FileStore content can then corrupt a differently-shaped world_size=M run that reuses the same path in the same pytest session, surfacing as a confusing NCCL bootstrap failure:

torch.distributed.DistBackendError: NCCL error ...
ncclOsSocketPollConnect: connect to ... Connection refused,
exceeded error retry count after 35 attempts

instead of a clear rendezvous error. Confirmed 100% deterministic: running the full file fresh (no pre-existing /tmp/rdzv) still fails test_distributed_fuser_ops[2] every time, because the [4] parametrization (which runs first) leaves /tmp/rdzv behind for [2] to trip over.

This was previously masked by older bundled NCCL (2.30.7), which apparently tolerated the stale/mismatched FileStore well enough to still succeed; a newer NCCL (2.31.2) surfaces it as a hard failure. See the investigation writeup for the full comparison:

Fix: key the rendezvous path by world_size
(file:///tmp/rdzv_test_fusible_ops_{world_size}), and defensively remove any pre-existing file at that path in test_distributed_fuser_ops before launching each subprocess job, to also cover a leftover file from an earlier crashed run of the same world_size.

Verified on a GB200 node
NCCL 2.31.2, the container that previously failed 2 of 3 parametrizations): all 3 world_size parametrizations now pass, including two runs of the full file back to back with no manual cleanup in between.

Authored-By: Claude Sonnet 5 noreply@anthropic.com

[greptile-apps]

Greptile Summary

The PR replaces shared FileStore rendezvous files with invocation-unique paths whose lifetimes cover each distributed subprocess.

• Creates a temporary rendezvous directory for every fusible-ops test invocation.
• Propagates its path to torchrun or mpirun children through NVTE_TEST_RDZV_PATH.
• Applies the isolation to both standard and userbuffers distributed tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
tests/pytorch/distributed/test_fusible_ops.py	Uses a per-invocation temporary FileStore path and waits for all launched ranks before cleaning it up, resolving the previously reported concurrent-path collision.
tests/pytorch/distributed/test_fusible_ops_with_userbuffers.py	Applies the same invocation-scoped rendezvous isolation to both torchrun and MPI userbuffers launches.

_{Reviews (2): Last reviewed commit: "Use unique rendezvous files in fusible o..." | Re-trigger Greptile}

[ptrendx]

/te-ci pytorch L1 L0

[nWEIdia]

cc @xiaofanl-nvidia per recommendation:

the NCCL version bump that bootstrap retry behavior measurably tightened between
2.30.7 and 2.31.2 in a way that will surface this whole class of "stale rendezvous file" bug more
aggressively across any test/user code with the same latent pattern

Just in case this behavior change (bootstrap retry) is not intended.

[timmoon10]

LGTM, good catch.