MoE Candidate Manifests

MoE candidate manifests

Candidate manifests describe the routed FFN groups that a backend may optimize. They replace inference from tensor names at execution time with an explicit, typed inventory built by the llama model layer.

Contract

A manifest identifies each logical expert group, its routed expert count, tensor roles, physical banks, quantization/layout metadata, and ownership lifetime. The runtime publishes a complete replace-all snapshot before scheduler reserve and refreshes it after changes such as LoRA application. Snapshot arrays are borrowed only for the registration call; tensor pointers in an accepted snapshot must remain valid until replacement.

The CUDA backend validates the entire group before granting grouped authority. Unknown roles, missing banks, contradictory geometry, stale tensor identities, unsupported block scales, or incomplete group coverage make the candidate ineligible. This fail-closed boundary is what keeps the grouped planner model-agnostic without pretending every GGUF layout is already supported.

V2 manifests add the broader typed bank descriptions needed for separate gate/up/down, fused gate-up/down, and ungated up/down projections, BF16, Q4 families, NVFP4, large block-scale banks, and eligible auxiliary tensors. The feature does not change GGUF tensor layout.

Important

The manifest is generic metadata, but its ABI is private to the fork implementation. Its only shipped consumer is the CUDA MoE cache.

Branch implementations

Branch	Internal ABI	Producer and consumer
moe-cache	ggml/src/ggml-backend-moe.h	src/llama-model.cpp, moe-cache.cu

Design/review history: PR 56.

Feature index - Grouped MoE decode