Repository navigation
v0.2.0-alpha: CPU-validated, opt-in policies not validated on GPUs
Pre-releaseCPU-validated, GPU validation pending. Everything in this alpha is tested on CPU. None of it has run on GPUs yet, and the new opt-in policies and the DDP comm hook in particular are not validated on GPUs. It is not on PyPI.
uv pip install "gpubridge @ git+https://github.com/DavidCzartoryski/gpubridge@v0.2.0-alpha"Requires Python 3.10+ and PyTorch 2.3+. On GPU nodes, install the CUDA or ROCm build of PyTorch first.
What's new since v0.1.0-alpha
- More collectives:
ReduceOp.AVG,MAXandMINforall_reduce, plusall_gather_into_tensor,reduce_scatter_tensorandasync_op=Trueon every collective. - The island step of the default policy (
reduce-bridge-broadcast) is now a reduce onto the island leader instead of an island all_reduce. This is the only change to the default path. init()fails on every rank, with the same message, whenLOCAL_RANKnames a GPU the process can't see, instead of hanging.gpubridge.ddp_comm_hookforDistributedDataParallel, so DDP gradients take the islands and the bridge (comm hook only, not a custom backend; not validated on GPUs).- Opt-in collective policies, off by default and never picked by
auto, selectable with--policyinscripts/gpu/check.pyandbench_all_reduce.py(not validated on GPUs):pipelined-reduce-bridge-broadcast: the leaders' bridge step in chunks through pinned host memory.sharded-bridge: k ranks per island share the bridge step over k links.auto-tuned: picks a policy by message size from thresholds measured on the cluster.
BridgeTransport.create_groupsfor policies that use several bridge links.- Collective observers (
gpubridge.observe): a record of every collective and the phases its policy marks. - GPU validation kit for Slurm clusters: a
PROJECT_DIRsetting, a configurable ROCm build, per-island NCCL/RCCL baselines, and a mixed heterogeneous-job path. - Docs: docs/ROADMAP.md holds three designs that wait on GPU results (bridge-only compression, an RDMA bridge transport, a torch.distributed backend for FSDP2), each "design only, not implemented". docs/PRIOR_ART.md covers the 2026 mixed-vendor systems (two called HetCCL, and Joint Training on AMD and NVIDIA GPUs).
How it compares
Those 2026 systems move cross-vendor data GPU to GPU over RDMA and report much higher cross-vendor bandwidth than a Gloo-based bridge through host memory, which is what gpubridge uses. The opt-in policies don't close that gap. gpubridge's case is that it runs on stock PyTorch builds over any network, today, as open source.
Validated (on CPU)
- 331 tests in CPU simulation (330 pass, 1 skipped): in CI on Python 3.10 and 3.12 for the release commit, and locally on PyTorch 2.14.1 and 2.3.1.
- Every collective, reduction and dtype matches
flat-gloobit for bit in every tested layout and policy, including uneven and three-island layouts for the opt-in policies. - The pipelined bridge stays correct when each staging copy completes only once something waits for it.
- The sharded bridge carries 1/k of the tensor on each of its k links.
- Through
ddp_comm_hook, DDP gradients equal the exact average under five policies, and a failed bucket makesbackward()raise on every rank. - ruff and shellcheck are clean.
- Not rerun for this release: the v0.1.0-alpha check that CUDA and ROCm builds of PyTorch join one job (Docker, no GPUs).
Not yet validated
Anything on GPUs, including every opt-in policy and the DDP hook. The CPU simulation of sharded-bridge is 1.3x to 1.8x slower than the default policy on one machine, so whether it pays off on real links is open. HARDWARE_VALIDATION.md lists each open item (12 to 22 are new in 0.2) and the runbook maps it to a script.