Grouped MoE Drafting

Grouped MoE drafting

moe-cache runs eligible draft, MTP, and target-verification graphs through the grouped expert cache. Upstream llama.cpp still defines the drafting algorithms.

Important

CUDA REQUIRED: Grouped MoE drafting requires the CUDA expert cache. Ordinary upstream drafting keeps its normal backend support.

Important

START WITH --load-mode none: Add a positive --moe-expert-cache-host-pinned-mb budget when full pinning fails. Test mapped sources separately because page faults can dominate.

Covered work

• Integrated draft-mtp can execute MTP rows and target verification through grouped cached MoE plans.
• Model-backed draft-simple, draft-eagle3, draft-dflash, and draft-dspark can group target verification. A separate MoE draft model can use its own cache size.
• N-gram modes keep their normal lookup algorithm; only the resulting target verification batch is adapted.
• Speculative batches wider than direct slot capacity can use a host-staged grouped strategy instead of silently changing arithmetic.

Graph certificates and target/draft/MTP namespaces protect ownership. A speculative MoE cache request stops on an incomplete manifest, invalid row structure, unsupported kernel, stale owner, or failed resource preparation instead of silently changing arithmetic.

Implementation

• Speculative setup and row semantics: common/speculative.cpp
• Runtime certificate production: src/llama-context.cpp
• Server verification lifecycle: tools/server/server-context.cpp
• CUDA required-grouped planning: ggml/src/ggml-cuda/moe-cache.cu

Primary implementation commit: d659bcefa. Use the upstream speculative guide for the algorithms themselves.

Feature index - MoE Cache overview