chore(cco): sync dev/cco with main + adapt GDA to collapsed-CQ#410
Merged
Conversation
## Motivation MORI must support both CCQE and non-CCQE ionic deployments from a single binary. The previous build-time IONIC_CCQE macro forced a recompile per host and broke when libionic.so and mori_application.so were out of sync. This change moves the decision to runtime: dlsym ionic_dv_create_cq_ex at CQ creation time — if the symbol is present and firmware build ≥ 58, use CCQE; otherwise fall back to ibv_create_cq_ex. A new environment variable MORI_DISABLE_IONIC_CCQE=1 overrides the runtime detection and forces the non-CCQE path even on hardware that supports it. ## Issues resolved Removed the build-time IONIC_CCQE macro and replaced it with a runtime capability check (IsCcqeSupported). Fixed a bug in the ionic GDA implementation where only the non-CCQE poll-CQ path was implemented; removed the now-redundant duplicate poll-CQ code. Co-authored-by: Qizhou Zhang <qizzhang@amd.com>
#339) - Fix SIOF by wrapping the registry in a Meyer's singleton (function-local static) - Add GetGpuStatesAddrProviderCount() diagnostic getter exposing live registry size - Add static_link_siof_test regression test that fails fast if SIOF regresses - Add missing <cstring> include in anvil.cpp for host-compiler builds
…#362) - Honor MORI_RDMA_TC / MORI_RDMA_SL on DEVX QPs (set DSCP + eth_prio). - Default UDP sport to fixed LID|0xC000; MORI_MLX5_ENABLE_UDP_SPORT=1 to rotate per-qpId for ECMP spread. - Fix teardown order + free leaked CQ/DBR buffers. - Clamp log_rra_max / log_sra_max to HCA per-QP cap.
…ent (#346) Motivation Add a deploy-mori skill that automates end-to-end MORI deployment inside a Docker container on AINIC (Pensando DSC) hosts. Technical Details New skill at .claude/skills/deploy-mori/SKILL.md or .cursor/skills/deploy-mori/SKILL.md covering container startup, base package installation, ainic bundle detection, rdma-core build, nicctl install, and MORI source install. Co-authored-by: Qizhou Zhang <qizzhang@smci355-ccs-aus-n06-21.prov.aus.ccs.cpe.ice.amd.com>
…node-v1 (#366) - Fix mlx5-RoCE internode-v1 silent corruption: combine overwrote dispatch's in-flight send source `staging`; dispatch now uses a dedicated `dispatchStaging` buffer. - Add mlx5 collapsed CQ (cc=1/oi=1): track completions via `CQE[0].wqe_counter`; final per-pe quiet waits for live `postIdx`, recycle gate keeps a snapshot. - Allocate GPU control structures (CQ/QP/doorbell/atomic ibuf) uncached; device-scope fences in the CQ drain. - Drop dead `outstandingWqe` writes on mlx5/psd.
…tecture (#257) Co-authored-by: TianDi101 <ditian12@amd.com> Co-authored-by: Niko Ma <nima@amd.com>
* fix(ep): guard all dispatch kernels against out-of-range expert id
Every dispatch kernel derives the destination rank as
pe = indices[i] / numExpertPerRank and requires 0 <= pe < worldSize. This is
guarded only by a debug assert (e.g. internode_v1.cpp:387/:493) which the
default Release build compiles out (NDEBUG). An out-of-range expert id (e.g. an
EPLB physical id >= worldSize*numExpertPerRank, while numExpertPerRank still
reflects the logical layout) then produces a garbage pe and an out-of-bounds
device access (GetAs(pe) / atomicAdd(counter+pe) / SendBufSlotOffset(...,pe,..))
-> HSA page fault (observed on wide-ep=16). The sibling LocalExpertCount kernel
already range-checks the id (ep_local_expert_count.hpp); the dispatch paths did
not.
Device side: drop an out-of-range pe instead of indexing OOB, in every dispatch
kernel variant (KernelType IntraNode/InterNode/InterNodeV1/InterNodeV1LL/AsyncLL):
- internode_v1.cpp (InterNodeV1 + LL recv): fold into shouldSkip
- intranode.hpp (IntraNode) and low_latency_async.cpp SendCopy (AsyncLL):
pe is warp-uniform -> coherent early skip via the dedup sentinel
- internode.hpp (InterNode) and low_latency_async.cpp SlotAssign: pe is
per-lane -> fold into the dedup null branch to keep __match_any_sync/__shfl
warp-coherent (no early continue)
Host side: add validate_dispatch_indices(), called once per dispatch (gated by
MORI_EP_VALIDATE_INDICES, default on), which fails loud with a clear error for
any out-of-range id, covering all kernel types in one place. Set the env to 0 to
skip the min/max reduction once the layout is known-good; the device guards
still prevent the fault.
Adds CPU regression tests (index math, per-kernel skip models, host validator).
* fix(ep): make host index validator opt-in and graph-capture-safe
validate_dispatch_indices() called int(indices.min()/max()), forcing a
GPU->CPU sync. Under HIP/CUDA graph capture (used by the dispatch/combine tests
and real workloads) that is illegal and aborts capture with
"HIP error: operation not permitted when stream is capturing", which failed the
native intranode/internode GPU tests (the eager tests passed).
- Default the validator OFF (opt-in via MORI_EP_VALIDATE_INDICES=1); the
always-on device-side guards remain the real crash protection.
- Also skip the check while a stream is capturing, so enabling it can never
break graph capture.
* fix(ep): drop host-side index validator and Python model tests (review)
Per review on #363: keep the change focused on the always-on device-side
destPe guard. Remove validate_dispatch_indices() and the
MORI_EP_VALIDATE_INDICES opt-in path (it was default-off, so it did not protect
normal dispatch, and its int(indices.min()/max()) GPU->CPU sync needed
graph-capture handling), and drop the two Python tests that modeled kernel
predicates instead of exercising the real dispatch path.
Net change is now only the device-side out-of-range guard in
internode_v1 / intranode / internode / low_latency_async.
…n shared runners (#368) * Initial plan * Fix UMBP client metrics test port allocation flake * Harden UMBP metrics test port selection --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
- Enable UMBP by default while keeping SPDK opt-in via BUILD_UMBP_SPDK. - Move UMBP Python tests under tests/python/umbp and add MORI-side UMBP coverage. - add UMBP python tests to nightly test-wheel job Co-authored-by: jhchouuu <jiahzhou@amd.com>
* Enhance sentinel handling in dispatch and combine operations - Implement logic to skip processing for negative expert IDs in both intra-node and inter-node dispatch functions. - Update the test suite to support various patterns of sentinel injection, ensuring that dispatched and combined outputs correctly handle these cases. - Introduce new parameters in test data generation to control sentinel behavior, improving test coverage for edge cases. * Implement routing handle for cached-mode dispatch - Introduce `EpDispatchRoutingHandle` class to manage per-call routing snapshots for mode-1 and mode-2 dispatch. - Enhance `EpDispatchCombineArgsRaw` to support routing-related pointers and replay mode. - Update dispatch and combine operations to utilize the new routing handle, allowing for improved routing management and replay correctness. - Add tests to validate the functionality of the routing handle, ensuring compatibility with existing dispatch mechanisms and correctness in multi-layer scenarios. * Fix comments and revert internode.hpp * Fix comments and remove changes to LL kernels. * Refactor dispatch combine tests and routing handle comparisons - Simplify test data generation documentation in `dispatch_combine_test_utils.py`. - Remove deprecated test for TP-replicated routing with sentinels in `test_dispatch_combine_intranode.py`. - Rename functions for clarity in `test_dispatch_combine_routing_handle.py`, ensuring consistency in naming conventions. - Update assertions to reflect changes in routing handle comparisons, enhancing test reliability. * Apply black formatting * Address copilot comments - Implement internode`test_sentinel_dispatch_combine` to validate the handling of sentinels in dispatch operations. - Update command options to include `test_sentinel` for executing sentinel tests. - Enhance routing pointer validation in `EpDispatchCombineRoutingPtrs` to ensure all required pointers are non-null. - Add tests for DeepEP-style -1 sentinel handling in `test_dispatch_combine_internode_v1.py`, ensuring robustness in multi-node scenarios. * Formatting * Use original source indices instead of dispatched indices * Add routing handle tests to CI and nightly workflows * Fix formatting * Address PR comments * Fix formatting * Fix formatting * Fix formatting dispatch combine ll * Merge clean up
…373) - Add ibv_shim.cpp for loading libibverbs at runtime - MORI_IBVERBS_LIB env var for optiaoal runtime libibverbs loading
* feat(rdma): add rail affinity support for endpoint connection --------- Signed-off-by: staryxchen <staryxchen@tencent.com>
- Change default qpPerTransfer = 4 to avoid AINIC resource overflow - Add logs for rdma transport implementation
* perf(umbp/dram): multi-threaded batch read + AVX2 NT CopyBlock Override DRAMTier::ReadBatchIntoPtr with a parallel CopyBlock path (UMBP_DRAM_READ_THREADS threads, default 8, capped to hardware concurrency) and advertise batch_read so LocalStorageManager dispatches the batched L3->L2 read instead of the serial per-key default. This breaks the single-core memcpy ceiling on cold DRAM. Large blocks (>=256KiB, the real cold KV-page regime) use AVX2 non-temporal stores (~1.5x over glibc memcpy on Zen4; NT avoids the dst read-for-ownership); tiny blocks fall back to memcpy. UMBP_DRAM_NT_COPY=0 disables NT. Holds the tier's single mutex for the whole batch so slot offsets stay valid during the parallel copy (no use-after-free vs Write/Evict/Clear); per-block copies still run in parallel within the batch. Ported from perf/dram-parallel-read (6710038 + 3b98bd0). Co-authored-by: Cursor <cursoragent@cursor.com> * perf(umbp/dram): multi-threaded batch write + AVX2 NT store on PUT path Symmetric counterpart of the committed batch-read optimization. Adds DRAMTier::BatchWrite (serial slot alloc + parallel non-temporal CopyBlock), LocalStorageManager::WriteBatchFromPtr, and batches StandaloneClient::BatchPut into a single tier write so payload copies parallelize, breaking the single-core memcpy ceiling on the L2->L3 backup path. Fan-out via UMBP_DRAM_WRITE_THREADS (default 8). Co-authored-by: Cursor <cursoragent@cursor.com> * perf(umbp/dram): multi-threaded + AVX2 NT copy on distributed local PUT/GET The distributed self-target (local) path copied KV pages with a single-threaded std::memcpy and never benefited from the DRAMTier multi-thread/AVX2 work: that optimization parallelizes within a call's pages, but in distributed mode one key maps to exactly one page (master page_size == KV block size), so it never engaged (verified: T=1 == T=8, both ~23 GiB/s). Parallelize across the many keys of one BatchPut/BatchGet instead - different threads handle different keys. PeerDramAllocator serializes Allocate/Commit/ Resolve internally (mutex); the per-key memcpy runs lock-free into distinct slots. Per-key copy uses AVX2 non-temporal stores (LocalCopyBlock, ported from DRAMTier). Thread count via UMBP_DRAM_{READ,WRITE}_THREADS; NT toggle via UMBP_DRAM_NT_COPY=0. Also lowers default thread count 8 -> 4 (DRAMTier + here). Measured (skyriver07, DP8 distributed case3, per-rank local-copy GiB/s): PUT local: T=1 23.1 -> T=4 54.8 -> T=8 57.2 (2.4x at default 4) GET local: T=1 19.9 -> T=4 46.2 -> T=8 47.1 (2.3x at default 4) Correctness: test_umbp_e2e (Put/Get round-trip + memcmp) PASS at threads=1/4/8. * style(umbp): apply clang-format (Google, clang-format 20) Reformat the touched UMBP DRAM-copy files to satisfy the repo pre-commit clang-format hook (mirrors-clang-format v20.1.8, BasedOnStyle: Google, ColumnLimit 100). No functional change. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: 123 <123@amd.com>
…-affinity knobs (#382)
… PSN (#395) - Add BnxtCollapsedCqDrain (one-CQE mode): reconstruct doneIdx from CQE[0], single drainer per QP; mirrors mlx5. - Drop WorkQueueHandle::outstandingWqe[65536] (−512 KB/WQ on GPU); reconstruct completions arithmetically. - Fix bnxt PSN accounting for mixed per-lane transfer sizes (warp-wide sum + exclusive prefix sum).
…amework streams (#402) Signed-off-by: Chengze Fan <fancz2002@gmail.com>
…405) Tidy up includes in core/ and application/ and split host code from device code. core/ no longer drags host-only headers (verbs.h / mlx5dv.h / hsakmt) into device builds. - core/ no longer includes application/; shared bits moved down into core/ - device-safe WcStatus mirror + vendored mlx5/bnxt/ionic ABI (parity-guarded) keep verbs/mlx5dv out of device TUs - removed dead Torch bootstrap, host RDMA primitives, duplicate warp/lane helpers - host-only *_dv.h moved to application/; core/utils.hpp -> core/utils/utils.hpp - bnxt WQE segments written as single 128-bit stores (perf)
Inherit the core/application include-hygiene + host/device separation refactor (#405). Conflicts resolved: - src/application/CMakeLists.txt: drop the BUILD_TORCH_BOOTSTRAP block (Torch bootstrap removed on main) - docker/Dockerfile.dev: BASE_IMAGE -> rocm7.2.4 (from main), keeping ubuntu24.04/py3.12 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sink RdmaEndpointDevice / RdmaDeviceVendorId from application to core (core::RdmaEndpointDevice — a device POD over core's WQ/CQ/Ibuf handles). shmem and cco now use it directly; cco's GDA device path includes only rdma_device.hpp (no application headers). application keeps the RdmaDeviceVendorId alias for its transport code. Adapt cco GDA completion to the merged main's collapsed-CQ model (#395): drop outstandingWqe[]/OUTSTANDING_TABLE_SIZE/recordOutstandingWqe; reconstruct the completed WQE from the CQE counter in quietUntil (PollCqOnce/PollCq per provider, mirroring shmem); MLX5_SEND_WQE_SHIFT -> core::MORI_MLX5_SEND_WQE_SHIFT. Verified on BNXT: build green; cco GDA (put/get/flush/signal/counter/barrier/ modes/multi_context) + LSA tests pass; p2p IBGDA benchmarks at NIC line rate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
6 tasks
jhchouuu
added a commit
that referenced
this pull request
Jul 6, 2026
Resolve the squash-sync divergence: dev/cco was previously kept current with main via a squash (#410) rather than a merge, so main's later commits conflicted against the stale snapshot. All 57 conflicts were in non-cco areas (src/umbp, src/io, src/application, tests, docs, CI, tuning_configs); resolved by taking main's current version for every one. src/cco (the module) merged cleanly and is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge latest
mainintodev/ccoand adapt cco's GDA layer to the changes itbrings (notably #395 bnxt collapsed-CQ + #405 include-hygiene refactor).
New work on top of inheriting main (review focus):
Verified on BNXT: build green; cco GDA + LSA tests pass; p2p IBGDA benchmarks at NIC line rate.