Grouped MoE Decode

Grouped MoE decode

Grouped decode admits an expert once across its gate, up, down, scale, and auxiliary banks. Every slot-bound bank then uses the same logical slot.

Important

CUDA REQUIRED: Planning, device admission/remapping, mapped MMQ/MMVQ dispatch, and graph capture are CUDA-only.

Eligibility

The fast path requires a certified target graph, known row semantics, a complete candidate manifest, supported CUDA kernels, and enough slots for all routed experts. It supports one row or independent parallel one-token rows.

The planner reads layout metadata, not model names. It supports separate gate/up/down, fused gate-up/down, and ungated up/down bank lists. NVFP4 scales remain slot-bound; eligible small scalar banks can stay resident. Unsupported cases use the established cached path.

The ungated two-bank path has exact legacy-vs-grouped direct, capture, replay, and dynamic-route coverage. See the Nemotron NP1 result for a profiled real-model run.

moe-cache can mark speculative work as required-grouped, in which case an unsupported plan fails closed rather than silently changing execution tier.

Replacement policy

Grouped decode uses decaying frequency with LRU tie-breaking. It ages history every 16 planning steps and resets it with the cache. Routing IDs and arithmetic do not change.

Leave GGML_CUDA_MOE_FREQUENCY unset or set it to 1 for frequency-aware eviction. Set 0 before context creation for LRU. More slots or better retention can reduce transfers, not GPU math or lazy PLE reads.

Read the public implementation commit and the Flash Next reproduction for a measured use of the default policy.

Branch implementations

Branch	Planning and resources	Dispatch
moe-cache	moe-cache.cu, moe-cache.cuh	ggml-cuda.cu, mmq.cu

Design/review history: PR 56.

Feature index - verify the selected path