Skip to content

MoRI v1.2.5

Latest

Choose a tag to compare

@jhchouuu jhchouuu released this 06 Nov 08:59
· 74 commits to main since this release
41c5257

MoRI v1.2.5

This release covers the 105 commits since v1.2.4.

The published wheels are now built on manylinux_2_28. They install on any distro with glibc 2.28 or newer, and they carry their own gRPC, so no part of mori needs gRPC on the host.

The other major themes:

  • EPv2 gets the AsyncLL path, LL128 small-token kernels and EP16/EP32 tuning on gfx1250, and plans that report the buffer sizes they need.
  • CCO adds a family of fused GEMM + collective operators.
  • CCO SDMA gains 2D transfers, wait and broadcast packets (gfx950 and gfx1250), and a complete device API for FlyDSL and Triton.
  • GDA gains doorbell coalescing.
  • MORI-IO adds a libfabric backend for Slingshot/CXI and recovers from a failed RDMA QP.
  • UMBP reworks its SSD tier and its eviction.

Highlights

  • Portable wheels. Wheels are built on manylinux_2_28 (#852): one wheel per Python version, for any glibc 2.28+ distro (RHEL 8/9, Ubuntu 20.04 and later).

    • UMBP's gRPC is now its own extension module (#751), so mori.ops, mori.io and mori.shmem no longer need gRPC at all. This removes the v1.2.4 known issue (ImportError: libgrpc++.so.1).
    • The wheel vendors UMBP's gRPC, together with the third-party license texts.
    • The wheels carry gfx1250 code alongside gfx942 and gfx950.
  • EPv2.

    • AsyncLL dispatch/combine on cco, including cross-node and gfx125x (#768).
    • On gfx1250: LL128 small-token dispatch/combine, plus EP16/EP32 dispatch tuning (#843).
    • Plans report the arena and staging sizes they need (info and describe, #844).
    • Routing handles can own their buffers (snapshot=True, #733).
  • Fused GEMM + collective operators on CCO:

    • GEMM + all-gather (#807);
    • all-gather fused into the fp8 GEMM that consumes it (#809);
    • GEMM + all-to-all (#695);
    • all-to-all fused into the fp8 GEMM that consumes it (#868);
    • GEMM + reduce-scatter for sequence parallelism (#802).

    A guide to these operators is in #913.

  • CCO SDMA and GDA.

    • 2D put/get (#805).
    • POLL_REGMEM wait and COPY broadcast packets on gfx950 (#814) and gfx1250 (#880).
    • A fused wait+put post (#866).
    • The full ccoSdma API is exposed to FlyDSL and Triton (#792).
    • GDA doorbell coalescing, with compile-time doorbell modes (#895).
  • MORI-IO.

    • Multi-NIC OFI/libfabric backend for Slingshot/CXI (#674).
    • A failed RDMA QP is replaced instead of failing every later transfer (#831).
    • Async NIC-event fault injection and port-down fail-fast (#815).
    • Simulated RDMA/TCP/XGMI backends that need no GPU or NIC (#867).

Features

  • feat(ep/v2): AsyncLL dispatch/combine on cco, with cross-node and gfx125x support (#768); plans report the arena and staging sizes they need (#844); a dispatch plan gets a default staging buffer that its caller can replace (#764); module __device__ staging is replaced by a shared buffer (#703)
  • feat(ep/v2, gfx1250): LL128 small-token dispatch/combine, plus EP16/EP32 dispatch tuning (#843)
  • feat(ep/v1): opt-in MORI_EP_DISABLE_RDMA_ATOMICS for InterNodeV1/V1LL (#736)
  • feat(cco): fused GEMM + collective operators: gemm_ag (#807), ag_gemm (#809), gemm_a2a (#695), a2a_gemm (#868), GEMM + reduce-scatter (#802)
  • feat(cco): 2D SDMA put/get via the LINEAR_LARGE_SUB_WINDOW packet (#805)
  • feat(cco): SDMA POLL_REGMEM wait and COPY broadcast packets, gfx950 (#814) and gfx1250 (#880); fused SDMA wait+put post (#866)
  • feat(cco): expose ccoSdma's full put/get API to FlyDSL and Triton (#792); cco_sdma_put_ptr_rsig__thread and cco_sdma_signal_slot bitcode (#786)
  • feat(cco): GDA doorbell coalescing and compile-time doorbell modes (#895)
  • feat(io): multi-NIC OFI/libfabric backend for Slingshot/CXI (#674); async NIC-event fault injection and port-down fail-fast (#815); simulated RDMA/TCP/XGMI backends (#867)
  • feat(rdma): register Ionic CQ/SQ/RQ rings and the atomic ibuf via dma-buf (#649)
  • feat(umbp):
    • BatchPrefetch stages a queued request's keys in local DRAM (#823);
    • large host DRAM buffers are registered as several NIC memory regions (#810);
    • remote writes are placed on the writer's NUMA node (#855);
    • a local ranged GET off SSD reads only its extent (#781);
    • SSD demote/promote skip staging (#776);
    • eviction converges onto one path, and nodes evict locally by default (#741);
    • masterless /metrics (#683);
    • per-medium byte counters for tier moves (#837);
    • hugepage-backed ranged scratch arenas (#830).
  • feat(umbp), more:
    • an SSD backend can keep small objects off it (min_object_size, #824);
    • the master routes small objects away from tiers they are under the floor of (#829);
    • the standalone server's ranged scratch is laid out by NUMA node (#856);
    • simulated storage and local/peer KV transfers (#862);
    • a warning when a NIC memory region outgrows what an AINIC translates (#896);
    • opt-in DRAM residency reporting to SGLang's KV Indexer (#851).
  • examples and POCs:
    • cco-sdma all-gather with start/wait (async mode) (#798);
    • Mega-MoE GEMM2 fused with an SDMA combine, gfx950 (#859);
    • example 11 gets an 8-wave GEMM2 option, an exact send/landing layout and a P2P fallback (#889).

Performance

  • perf(gemm-ar): mxfp8 with row-major scales staged through LDS (#806); load mxfp8 scales one K step ahead at K >= 4096 (#857); opt-in optimization paths (#748)
  • perf(ep/asyncll): fuse put-signal and drop drains (#801)
  • perf(EPv2/gfx1250): dispatch rows for EP4, hidden 2048, topk 8 (#881)
  • perf(umbp):
    • transfer plans are built incrementally (#758);
    • unbound DRAM tiers are prefaulted in parallel across NUMA nodes (#718);
    • the default DRAM eviction band is now 0.95/0.9 (#832);
    • SSD extent staging is freed when its hold drops, with a next-fit span search (#838);
    • SSD batch reads get their own io_uring rings per drive (#901);
    • local readers of the same extent share one SSD staging span (#902).

Bug Fixes

  • fix(anvil/sdma):
    • topology is read from sysfs, and libhsakmt gets its own node id, which fixes SDMA queue setup in containers that were given only some of the host's GPUs (#841);
    • SDMA queues are keyed on the KFD node id instead of the HIP device ordinal (#623);
    • on gfx1250, loopback is spread over the peer-link engines when there is no CPU link (#766).
  • fix(cco): two SDMA device-post deadlocks (#711); never fill the SDMA ring completely (#819); clear the HIP last-error after a failed fabric handle probe (#761)
  • fix(ep/v2): AsyncLL fused-receive deadlock under two streams, plus early notification by default on gfx950 (#875); each AsyncLL op gets its own SDMA queue per peer (#892)
  • fix(ep/v2): three gfx1250 combine slow paths at topk 9 (#886); clamp the untuned FlyDSL combine block count to the flag slots (#876); routing handles get their own buffers on request (#733)
  • fix(ep): close two InterNodeV1/V1LL dispatch/combine races, and add stress_accuracy (#796); fail fast for cross-node kernels on the CCO backend (#894)
  • fix(io): register host memory with ibv_reg_mr alone, never dma-buf (#890); a failed notification post waits for the transfer's data WRs (#836); XGMI IPC handle device ordinal (#803)
  • fix(bootstrap): retry SendMessage's ConnectSocket for cross-node reliability (#771)
  • fix(topology): load the GPU topology one comm at a time (#813)
  • fix(gemm-ar): drain the peeled tail's producers (#785)
  • fix(rdma): name the likely causes in fatal MR registration errors (#821)
  • fix(umbp):
    • SSD tier:
      • the SSD tier no longer runs out of fds (#754);
      • it gives disk back one segment at a time (#775);
      • a single-flight SSD read joins any in-flight read that covers its extent (#827);
      • SSD staging no longer fails local loads (#762);
      • an SSD Clear keeps the staging spans that local readers still hold (#788);
      • an in-place SSD read counts as an access (#793);
      • SSD keys are counted without snapshotting them (#808);
      • puts that a full SSD staging arena refuses are counted (#822).
    • Standalone server:
      • it keeps serving when its GPU-registered DRAM pool is migrated (#756);
      • it creates its HIP queues before it serves (#800);
      • its gather streams are pooled, so it stops leaking them (#799).
    • Eviction:
      • eviction demotions are retried instead of dropping the only copy (#763);
      • a local eviction round no longer rescans its queued demotions per batch (#783).
    • Correctness and reporting:
      • local clear is acknowledged when no master is configured (#769);
      • the backend commit happens outside the pool lock (#804);
      • a resolve counts only the reader's slice of the bytes (#835);
      • a remote put of a key that another writer holds counts as stored (#820);
      • MoriIoEngine turns MORI IO notifications off (#840);
      • false put-failure noise is gone (#777).

Build / CI / Tests / Docs

  • build(wheel): build the published wheels on manylinux_2_28 (#852)
  • build: UMBP's pybind is its own module (#751); PCI topology is read from sysfs, and the libpci dependency is dropped (#724)
  • build(flydsl): FlyDSL 0.4.0 in the dev image and examples (#817). FlyDSL stays an unpinned optional extra.
  • ci: the nightly version is based on the numeric part of the last release tag (#847); a dispatch can run the UMBP hicache smoke test (#834); fix the flaky test_engine (EADDRINUSE) (#853)
  • refactor(umbp): one incremental plan builder shared by every transfer engine (#774); track each SSD resolve read in one struct (#903); remove dead code (#778); port test_peer_pool's FailingEngine (#790)
  • docs: a guide to the fused GEMM + collective operators (#913); UMBP blog in the news section (#828); acknowledge hipFile in the GDS transfer engine (#782)

Wheels: cp310 and cp312, both manylinux_2_28_x86_64 (glibc 2.28 or newer), published to PyPI as amd-mori==1.2.5.

Full changelog: v1.2.4...v1.2.5