MoE Cache Ownership and Parallelism

MoE cache ownership and parallelism

MoE cache resources belong to the CUDA backend context that executes their layer. Plans, slots, copy streams, events, graph resources, and candidate generations therefore share one explicit owner instead of process-global mutable state.

Important

CUDA REQUIRED: The ownership model on this page governs CUDA expert caches.

Parallel requests

Independent one-token target rows can share a grouped plan when their combined routed expert set fits the configured slots. Admission is committed only after the complete plan is ready. In-flight leases protect slots until every consumer has finished, and cancellation/release clears request ownership without freeing resources still used by another call.

Process-wide diagnostic counters are not strict per-request attribution. They may be printed and reset at a server timing boundary while other work contributes to the same interval.

Multi-GPU behavior

Normal layer-split placement can give each owning CUDA device its own cache resources. The cache buffer is not a CUDA split-buffer implementation: row- or tensor-split sharding of one cached expert tensor is not supported. Use a single GPU or layer split when relying on the cache.

This resource model also localizes tensor-override provenance and refreshes candidate manifests after LoRA changes. Active LoRA remains ineligible for grouped authority where the effective cached expert inventory cannot be proven.

Branch implementations

Branch	Implementation	Lifecycle coverage
moe-cache	moe-cache.cu, ggml-cuda.cu	tests/test-moe-cache.cpp, tools/server/tests/unit/test_completion.py

Design/review history: PR 56.

Feature index