Repository navigation
v0.3.0-alpha: CPU-validated, GPU validation kit ready
Pre-release
Pre-release
·
432 commits
to main
since this release
CPU-validated, with the GPU validation kit ready for a first GPU run. Everything in this alpha is tested on CPU. Nothing has run on a GPU yet; the opt-in policies, the DDP comm hook and the new opt-in pinned staging are not validated on GPUs. It is not on PyPI.
uv pip install "gpubridge @ git+https://github.com/DavidCzartoryski/gpubridge@v0.3.0-alpha"Requires Python 3.10+ and PyTorch 2.8+ (was 2.3+). On GPU nodes, install the CUDA or ROCm build of PyTorch first.
Breaking changes
- PyTorch 2.8 or later is required: tearing down a Gloo group could deadlock on older releases (fixed upstream in 2.8).
init()refuses a layout that puts two ranks of one island on the same GPU, on every rank with the same message. A GPU that reports neither a UUID nor a PCI address leaves its island unchecked, with a warning.Work.is_completed()turns true only once the collective's GPU work is done, as with NCCL.
What's new since v0.2.0-alpha
gpubridge.in_flight(): the collective each thread is inside, with its phase, for debugging hangs.GPUBRIDGE_PINNED_STAGING=1(opt-in, off by default): the default policy's leaders stage the bridge step through one pinned host buffer, ordered with CUDA events, so one GPU allocation can compare it with pageable copies.- A fix in the pipelined policy: a pinned slot could be reused by the next collective before its last copy back had run.
- GPU-only correctness tests (
pytest -m gpu tests/test_gpu.py), bit for bit againstflat-gloo. - The GPU validation kit, ready for a first run:
- every job script runs end to end on CPU against a fake Slurm cluster, with hard time limits;
- a ladder from one GPU to a real mixed job;
- a network preflight that picks each node's Gloo interface;
- CUDA-event benchmarks with named baselines;
- full run metadata with an environment-variable allowlist;
validation_tables.py, which turns results into the HARDWARE_VALIDATION.md tables, including predicted against measured bridge time.
- A release workflow: every pull request and version tag builds the sdist and wheel and checks them with twine, and a tag must match the package version. Publishing to PyPI stays off.
See CHANGELOG.md for the full list with pull requests.
Validated (on CPU)
- 458 tests in CPU simulation: in CI on the tagged commit, 451 passed and 7 skipped (the GPU-only tests) on Python 3.12 with the newest torch, and 431 passed and 7 skipped on Python 3.10 with torch 2.8.0 (the job tests run on 3.12 only).
- Every collective, reduction and dtype matches
flat-gloobit for bit in every tested layout and policy, with and without pinned staging. - The pinned staging and pipelined bridge stay correct when each copy completes only once something waits for it.
Not yet validated
Anything on GPUs. HARDWARE_VALIDATION.md lists each open item (23 to 25 are new in 0.3) and the runbook maps it to a script.