Skip to content

v0.2.0-alpha: CPU-validated, opt-in policies not validated on GPUs

Pre-release
Pre-release

Choose a tag to compare

@DavidCzartoryski DavidCzartoryski released this 10 Oct 14:52
9f00c23

CPU-validated, GPU validation pending. Everything in this alpha is tested on CPU. None of it has run on GPUs yet, and the new opt-in policies and the DDP comm hook in particular are not validated on GPUs. It is not on PyPI.

uv pip install "gpubridge @ git+https://github.com/DavidCzartoryski/gpubridge@v0.2.0-alpha"

Requires Python 3.10+ and PyTorch 2.3+. On GPU nodes, install the CUDA or ROCm build of PyTorch first.

What's new since v0.1.0-alpha

  • More collectives: ReduceOp.AVG, MAX and MIN for all_reduce, plus all_gather_into_tensor, reduce_scatter_tensor and async_op=True on every collective.
  • The island step of the default policy (reduce-bridge-broadcast) is now a reduce onto the island leader instead of an island all_reduce. This is the only change to the default path.
  • init() fails on every rank, with the same message, when LOCAL_RANK names a GPU the process can't see, instead of hanging.
  • gpubridge.ddp_comm_hook for DistributedDataParallel, so DDP gradients take the islands and the bridge (comm hook only, not a custom backend; not validated on GPUs).
  • Opt-in collective policies, off by default and never picked by auto, selectable with --policy in scripts/gpu/check.py and bench_all_reduce.py (not validated on GPUs):
    • pipelined-reduce-bridge-broadcast: the leaders' bridge step in chunks through pinned host memory.
    • sharded-bridge: k ranks per island share the bridge step over k links.
    • auto-tuned: picks a policy by message size from thresholds measured on the cluster.
  • BridgeTransport.create_groups for policies that use several bridge links.
  • Collective observers (gpubridge.observe): a record of every collective and the phases its policy marks.
  • GPU validation kit for Slurm clusters: a PROJECT_DIR setting, a configurable ROCm build, per-island NCCL/RCCL baselines, and a mixed heterogeneous-job path.
  • Docs: docs/ROADMAP.md holds three designs that wait on GPU results (bridge-only compression, an RDMA bridge transport, a torch.distributed backend for FSDP2), each "design only, not implemented". docs/PRIOR_ART.md covers the 2026 mixed-vendor systems (two called HetCCL, and Joint Training on AMD and NVIDIA GPUs).

How it compares

Those 2026 systems move cross-vendor data GPU to GPU over RDMA and report much higher cross-vendor bandwidth than a Gloo-based bridge through host memory, which is what gpubridge uses. The opt-in policies don't close that gap. gpubridge's case is that it runs on stock PyTorch builds over any network, today, as open source.

Validated (on CPU)

  • 331 tests in CPU simulation (330 pass, 1 skipped): in CI on Python 3.10 and 3.12 for the release commit, and locally on PyTorch 2.14.1 and 2.3.1.
  • Every collective, reduction and dtype matches flat-gloo bit for bit in every tested layout and policy, including uneven and three-island layouts for the opt-in policies.
  • The pipelined bridge stays correct when each staging copy completes only once something waits for it.
  • The sharded bridge carries 1/k of the tensor on each of its k links.
  • Through ddp_comm_hook, DDP gradients equal the exact average under five policies, and a failed bucket makes backward() raise on every rank.
  • ruff and shellcheck are clean.
  • Not rerun for this release: the v0.1.0-alpha check that CUDA and ROCm builds of PyTorch join one job (Docker, no GPUs).

Not yet validated

Anything on GPUs, including every opt-in policy and the DDP hook. The CPU simulation of sharded-bridge is 1.3x to 1.8x slower than the default policy on one machine, so whether it pays off on real links is open. HARDWARE_VALIDATION.md lists each open item (12 to 22 are new in 0.2) and the runbook maps it to a script.