Independent Draft MoE Cache

Independent draft MoE cache

A separately loaded MoE draft model can use a different expert-cache slot count from the target. This avoids forcing a small draft model to reserve the target's large cache and avoids forcing a large target to use the draft's smaller budget.

Important

CUDA REQUIRED: Both target and draft expert caches are CUDA allocations.

Use --load-mode none for the allocated-host performance baseline and follow the host-source and partial-pinning guide. Integrated MTP uses the target model's cache rather than this independent draft cache.

Control

--spec-draft-moe-expert-cache-size N sets slots per cached expert tensor for the separate draft model. If omitted, the draft inherits --moe-expert-cache-size. An explicit zero disables caching for the draft only. Integrated MTP uses target weights and therefore uses the target cache rather than allocating this independent draft pool.

Target and draft pools are additive with model weights, KV, recurrent state, workspace, staging, and CUDA graph resources. The best default for a small draft model that fits entirely in VRAM is usually a zero draft cache with normal full-GPU placement.

The option does not select a draft model and does not alter target cache slots. Normal draft-model loading options remain documented upstream.

Implementation

• CLI and environment parsing: common/arg.cpp
• Separate draft model parameters: common/common.h
• Model construction: common/speculative.cpp
• Parser coverage: tests/test-arg-parser.cpp

The feature was introduced on the former drafting branch in e3371bf74 and now ships on moe-cache.

Feature index - MoE Cache flags