v0.4.0-alpha: CPU-validated, not run on GPUs

CPU-validated. Nothing in this release has run on a GPU or on RDMA hardware. It is not on PyPI.

uv pip install "gpubridge @ git+https://github.com/DavidCzartoryski/gpubridge@v0.4.0-alpha"

Requires Python 3.10+ and PyTorch 2.8+. On GPU nodes, install the CUDA or ROCm build of PyTorch first.

What's new since v0.3.0-alpha

All of it is opt-in or experimental, and none of it is validated on GPUs.

• gpubridge as a torch.distributed backend (experimental). dist.init_process_group(backend="gpubridge") runs DDP and FSDP2 collectives through gpubridge across vendor islands. Before torch 2.14, add import gpubridge; on 2.14 and later, a lazy entry point registers the backend. Nothing hooks import torch. Unsupported calls (point-to-point, all-to-all, reduce, other reductions) are refused before any communication. Design: docs/TORCH_BACKEND.md (#38).
• Subgroups and HSDP on the backend (experimental). new_group and multi-dimensional DeviceMesh groups get their own topology: native-only within one vendor, bridged across vendors. HSDP with shard groups set to the vendor islands keeps all-gathers and reduce-scatters inside each vendor and sends one replicate all_reduce per step over the bridge. Before patching a private torch function, subgroup creation checks that it is what gpubridge expects; if not, it refuses with an error naming the torch version (#39, #40).
• Opt-in bridge compression, GPUBRIDGE_BRIDGE_COMPRESSION=bf16|fp8. It is off by default, and with it on, results are no longer bit-exact against the uncompressed sum. Triton kernels do the work, with a PyTorch fallback; every rank still ends with the same bits. Design and error bounds: docs/COMPRESSION.md (#37).
• An opt-in UCX bridge transport, init(bridge="ucx"). It was tested only on CPU, over UCX's TCP transport, and was not run on InfiniBand/RoCE or with GPUDirect RDMA. Design and open questions: docs/RDMA_TRANSPORT.md (#35).
• Fixed: the two-node job tests no longer hang on machines whose own hostname doesn't resolve (#36).

See CHANGELOG.md for the full list.

Validated (on CPU)

CI ran the full test suite in CPU simulation on Python 3.12 with the newest torch and on Python 3.10 with torch 2.8.0, plus the UCX bridge tests over TCP. Nothing in this release has run on GPUs.

Not yet validated

Anything on GPUs or RDMA hardware. HARDWARE_VALIDATION.md lists each open item, and the runbook maps each one to a script.

v0.3.0-alpha: CPU-validated, GPU validation kit ready

CPU-validated, with the GPU validation kit ready for a first GPU run. Everything in this alpha is tested on CPU. Nothing has run on a GPU yet; the opt-in policies, the DDP comm hook and the new opt-in pinned staging are not validated on GPUs. It is not on PyPI.

uv pip install "gpubridge @ git+https://github.com/DavidCzartoryski/gpubridge@v0.3.0-alpha"

Requires Python 3.10+ and PyTorch 2.8+ (was 2.3+). On GPU nodes, install the CUDA or ROCm build of PyTorch first.

Breaking changes

• PyTorch 2.8 or later is required: tearing down a Gloo group could deadlock on older releases (fixed upstream in 2.8).
• init() refuses a layout that puts two ranks of one island on the same GPU, on every rank with the same message. A GPU that reports neither a UUID nor a PCI address leaves its island unchecked, with a warning.
• Work.is_completed() turns true only once the collective's GPU work is done, as with NCCL.

What's new since v0.2.0-alpha

• gpubridge.in_flight(): the collective each thread is inside, with its phase, for debugging hangs.
• GPUBRIDGE_PINNED_STAGING=1 (opt-in, off by default): the default policy's leaders stage the bridge step through one pinned host buffer, ordered with CUDA events, so one GPU allocation can compare it with pageable copies.
• A fix in the pipelined policy: a pinned slot could be reused by the next collective before its last copy back had run.
• GPU-only correctness tests (pytest -m gpu tests/test_gpu.py), bit for bit against flat-gloo.
• The GPU validation kit, ready for a first run:
  • every job script runs end to end on CPU against a fake Slurm cluster, with hard time limits;
  • a ladder from one GPU to a real mixed job;
  • a network preflight that picks each node's Gloo interface;
  • CUDA-event benchmarks with named baselines;
  • full run metadata with an environment-variable allowlist;
  • validation_tables.py, which turns results into the HARDWARE_VALIDATION.md tables, including predicted against measured bridge time.
• A release workflow: every pull request and version tag builds the sdist and wheel and checks them with twine, and a tag must match the package version. Publishing to PyPI stays off.

See CHANGELOG.md for the full list with pull requests.

Validated (on CPU)

• 458 tests in CPU simulation: in CI on the tagged commit, 451 passed and 7 skipped (the GPU-only tests) on Python 3.12 with the newest torch, and 431 passed and 7 skipped on Python 3.10 with torch 2.8.0 (the job tests run on 3.12 only).
• Every collective, reduction and dtype matches flat-gloo bit for bit in every tested layout and policy, with and without pinned staging.
• The pinned staging and pipelined bridge stay correct when each copy completes only once something waits for it.

Not yet validated

Anything on GPUs. HARDWARE_VALIDATION.md lists each open item (23 to 25 are new in 0.3) and the runbook maps it to a script.

v0.2.0-alpha: CPU-validated, opt-in policies not validated on GPUs

CPU-validated, GPU validation pending. Everything in this alpha is tested on CPU. None of it has run on GPUs yet, and the new opt-in policies and the DDP comm hook in particular are not validated on GPUs. It is not on PyPI.

uv pip install "gpubridge @ git+https://github.com/DavidCzartoryski/gpubridge@v0.2.0-alpha"

Requires Python 3.10+ and PyTorch 2.3+. On GPU nodes, install the CUDA or ROCm build of PyTorch first.

What's new since v0.1.0-alpha

• More collectives: ReduceOp.AVG, MAX and MIN for all_reduce, plus all_gather_into_tensor, reduce_scatter_tensor and async_op=True on every collective.
• The island step of the default policy (reduce-bridge-broadcast) is now a reduce onto the island leader instead of an island all_reduce. This is the only change to the default path.
• init() fails on every rank, with the same message, when LOCAL_RANK names a GPU the process can't see, instead of hanging.
• gpubridge.ddp_comm_hook for DistributedDataParallel, so DDP gradients take the islands and the bridge (comm hook only, not a custom backend; not validated on GPUs).
• Opt-in collective policies, off by default and never picked by auto, selectable with --policy in scripts/gpu/check.py and bench_all_reduce.py (not validated on GPUs):
  • pipelined-reduce-bridge-broadcast: the leaders' bridge step in chunks through pinned host memory.
  • sharded-bridge: k ranks per island share the bridge step over k links.
  • auto-tuned: picks a policy by message size from thresholds measured on the cluster.
• BridgeTransport.create_groups for policies that use several bridge links.
• Collective observers (gpubridge.observe): a record of every collective and the phases its policy marks.
• GPU validation kit for Slurm clusters: a PROJECT_DIR setting, a configurable ROCm build, per-island NCCL/RCCL baselines, and a mixed heterogeneous-job path.
• Docs: docs/ROADMAP.md holds three designs that wait on GPU results (bridge-only compression, an RDMA bridge transport, a torch.distributed backend for FSDP2), each "design only, not implemented". docs/PRIOR_ART.md covers the 2026 mixed-vendor systems (two called HetCCL, and Joint Training on AMD and NVIDIA GPUs).

How it compares

Those 2026 systems move cross-vendor data GPU to GPU over RDMA and report much higher cross-vendor bandwidth than a Gloo-based bridge through host memory, which is what gpubridge uses. The opt-in policies don't close that gap. gpubridge's case is that it runs on stock PyTorch builds over any network, today, as open source.

Validated (on CPU)

• 331 tests in CPU simulation (330 pass, 1 skipped): in CI on Python 3.10 and 3.12 for the release commit, and locally on PyTorch 2.14.1 and 2.3.1.
• Every collective, reduction and dtype matches flat-gloo bit for bit in every tested layout and policy, including uneven and three-island layouts for the opt-in policies.
• The pipelined bridge stays correct when each staging copy completes only once something waits for it.
• The sharded bridge carries 1/k of the tensor on each of its k links.
• Through ddp_comm_hook, DDP gradients equal the exact average under five policies, and a failed bucket makes backward() raise on every rank.
• ruff and shellcheck are clean.
• Not rerun for this release: the v0.1.0-alpha check that CUDA and ROCm builds of PyTorch join one job (Docker, no GPUs).

Not yet validated

Anything on GPUs, including every opt-in policy and the DDP hook. The CPU simulation of sharded-bridge is 1.3x to 1.8x slower than the default policy on one machine, so whether it pays off on real links is open. HARDWARE_VALIDATION.md lists each open item (12 to 22 are new in 0.2) and the runbook maps it to a script.

v0.1.0-alpha: CPU-validated, GPU validation pending

CPU-validated, GPU validation pending. This alpha marks the point where every orchestration path is tested on CPU and the CPU side of a mixed CUDA/ROCm job has been shown to work with real PyTorch builds. It has not yet run on GPUs. It is not on PyPI.

uv pip install "gpubridge @ git+https://github.com/DavidCzartoryski/gpubridge@v0.1.0-alpha"

Requires Python 3.10+ and PyTorch 2.3+. On GPU nodes, install the CUDA or ROCm build of PyTorch first.

What's in it

• gpubridge.init(), all_reduce (SUM), broadcast and barrier over one job that mixes NVIDIA and AMD ranks: an NCCL or RCCL island per vendor, island leaders joined by a CPU Gloo bridge.
• Collective policies: auto (the default), reduce-bridge-broadcast, native-only and flat-gloo, plus hooks for size-based selection.
• A swappable bridge transport, with Gloo as the default.
• Discovery shares every rank's probe results, so a bad setup fails init() on every rank with the same message instead of hanging.
• An up-front dtype check: float16, bfloat16, float32, float64, int8, uint8, int32, int64.
• Simulation mode (GPUBRIDGE_VENDOR), CPU-only mode, and a test-only split-test mode that is flagged in every log and result.
• A GPU validation kit in scripts/gpu (Slurm, a single AMD machine, and a two-machine mixed job) with a runbook.

Validated (on CPU)

• 181 tests in CPU simulation, in CI on Python 3.10 and 3.12.
• CUDA and ROCm builds of PyTorch join one job and complete discovery, all_reduce and broadcast, with either build hosting the store: matched 2.9.1 and 2.14.1 pairs, and mixed releases from 2.9.1 to 2.14.1 (Docker, no GPUs).
• Gloo all_reduce and broadcast for every supported dtype on PyTorch 2.3.1, 2.6.0, 2.9.1 and 2.14.1.

Not yet validated

Anything on GPUs: NCCL and RCCL themselves, device selection on AMD nodes, stream ordering in the bridge step, Gloo between real hosts, watchdog timeouts, barrier() on ROCm, and bridge throughput. HARDWARE_VALIDATION.md lists each open item and the runbook maps it to a script.