Speculative MoE Graph Variants

Speculative MoE graph variants

Speculative decoding alternates among target decode, draft/MTP steps, verification spans, acceptance outcomes, and parallel row counts. These shapes can each have a valid grouped CUDA graph. Retaining only one graph would repeatedly evict a useful variant whenever the loop changed shape.

Important

CUDA REQUIRED: The retained variants on this page are grouped CUDA graphs for cached MoE execution.

Variant identity

Each retained entry is keyed by the graph and execution properties that affect safety: target/draft/MTP namespace, row semantics, graph identity, candidate generation, shape, and resource authority. A matching variant can be reused; a stale manifest, changed ownership, or incompatible resource plan invalidates it.

Retention is bounded by the backend's graph cache policy. It does not capture arbitrary unlimited shapes and does not make an unsupported graph eligible. Required-grouped speculative execution still fails closed when no valid plan can be built.

This feature complements MoE CUDA graph reuse: the base feature proves and replays a graph, while this extension keeps several speculative shapes alive at once.

Implementation

• Variant cache and grouped resources: ggml/src/ggml-cuda/moe-cache.cu
• Capture/replay selection: ggml/src/ggml-cuda/ggml-cuda.cu
• Coverage: tests/test-moe-cache.cpp

The extension was introduced on the former drafting branch in 15e501cb0 and now ships on moe-cache.

Feature index - Grouped MoE drafting