MoE CUDA Graph Reuse

MoE CUDA graph reuse

Grouped MoE execution captures and replays CUDA graphs only after the current graph has a complete, stable expert plan. The captured launch topology can then be reused across decode steps while slot admission and mapped expert addresses remain valid.

Important

CUDA REQUIRED: This feature captures and replays CUDA graphs. It does not affect another backend's graph mechanism.

Safety boundary

A graph key includes graph identity, execution namespace, row semantics, candidate generation, and relevant shape/capability state. Cache-plan coverage is certified before capture. LoRA changes, manifest replacement, ownership changes, failed admission, phase changes, and incompatible graph mutation invalidate or decline reuse.

The cache retains compatible backing across prompt/decode transitions so a prompt does not discard a useful decode graph unnecessarily. It also refreshes graph identity after mutation and orders sibling-prefetch reads before any slot can be repurposed.

Capture is an optimization, not a correctness requirement on moe-cache: an ineligible ordinary graph uses the established cached path. Required grouped speculative work fails closed if its required plan cannot be prepared.

Opt-in decode boundary updates

--decode-boundary-overlap, disabled by default, permits compatible grouped graph changes to update the existing executable when capture coverage is complete and resource fingerprints and ownership still match. Captured F32 strided copies use the existing copy kernel so changing row counts can be updated. An update failure re-instantiates the executable; an invalid resource or execution contract takes the established fallback.

Use it with --decode-overlap to also overlap eligible CPU graph preparation with the preceding decode. The old graph description is destroyed after launching the new executable. Buffer growth still synchronizes. Without this option, the previous boundary behavior remains active.

Available on moe-cache. The option originated in 9bae00957 before the branch collapse. See decode boundary overlap for enable/disable instructions, profiling limits, and the unresolved long-output parity difference.

Branch implementations

Branch	Capture integration	Plan/resource lifetime
moe-cache	ggml-cuda.cu	moe-cache.cu

PR 56 reviews the architecture. Speculative MoE graph variants covers drafting-specific retention.

Feature index