Skip to content

v0.3.0-alpha: CPU-validated, GPU validation kit ready

Pre-release
Pre-release

Choose a tag to compare

@DavidCzartoryski DavidCzartoryski released this 12 Oct 15:56
· 432 commits to main since this release
e84d662

CPU-validated, with the GPU validation kit ready for a first GPU run. Everything in this alpha is tested on CPU. Nothing has run on a GPU yet; the opt-in policies, the DDP comm hook and the new opt-in pinned staging are not validated on GPUs. It is not on PyPI.

uv pip install "gpubridge @ git+https://github.com/DavidCzartoryski/gpubridge@v0.3.0-alpha"

Requires Python 3.10+ and PyTorch 2.8+ (was 2.3+). On GPU nodes, install the CUDA or ROCm build of PyTorch first.

Breaking changes

  • PyTorch 2.8 or later is required: tearing down a Gloo group could deadlock on older releases (fixed upstream in 2.8).
  • init() refuses a layout that puts two ranks of one island on the same GPU, on every rank with the same message. A GPU that reports neither a UUID nor a PCI address leaves its island unchecked, with a warning.
  • Work.is_completed() turns true only once the collective's GPU work is done, as with NCCL.

What's new since v0.2.0-alpha

  • gpubridge.in_flight(): the collective each thread is inside, with its phase, for debugging hangs.
  • GPUBRIDGE_PINNED_STAGING=1 (opt-in, off by default): the default policy's leaders stage the bridge step through one pinned host buffer, ordered with CUDA events, so one GPU allocation can compare it with pageable copies.
  • A fix in the pipelined policy: a pinned slot could be reused by the next collective before its last copy back had run.
  • GPU-only correctness tests (pytest -m gpu tests/test_gpu.py), bit for bit against flat-gloo.
  • The GPU validation kit, ready for a first run:
    • every job script runs end to end on CPU against a fake Slurm cluster, with hard time limits;
    • a ladder from one GPU to a real mixed job;
    • a network preflight that picks each node's Gloo interface;
    • CUDA-event benchmarks with named baselines;
    • full run metadata with an environment-variable allowlist;
    • validation_tables.py, which turns results into the HARDWARE_VALIDATION.md tables, including predicted against measured bridge time.
  • A release workflow: every pull request and version tag builds the sdist and wheel and checks them with twine, and a tag must match the package version. Publishing to PyPI stays off.

See CHANGELOG.md for the full list with pull requests.

Validated (on CPU)

  • 458 tests in CPU simulation: in CI on the tagged commit, 451 passed and 7 skipped (the GPU-only tests) on Python 3.12 with the newest torch, and 431 passed and 7 skipped on Python 3.10 with torch 2.8.0 (the job tests run on 3.12 only).
  • Every collective, reduction and dtype matches flat-gloo bit for bit in every tested layout and policy, with and without pinned staging.
  • The pinned staging and pipelined bridge stay correct when each copy completes only once something waits for it.

Not yet validated

Anything on GPUs. HARDWARE_VALIDATION.md lists each open item (23 to 25 are new in 0.3) and the runbook maps it to a script.