MoE Cache Diagnostics

MoE cache diagnostics

Use the cache reports to prove which path ran and where it spent time.

Important

CUDA REQUIRED: These reports describe the CUDA MoE cache. Remove --experimental-logs from normal production runs.

Controls and reports

Add --experimental-logs for detailed reports. Basic hit/miss/eviction summaries print without it.

• moe-grouped-decode: registered/covered groups, plan compiles/reuses, completed calls, host-staged calls, fallbacks, and errors.
• moe-cache-phase: prefill/decode operations, staging, route readback, cache activity, transfer bytes, and timing.
• moe-cache-experts: unique experts, reuse distance, and hot tensors.
• moe-cache-host: host budget, directly registered complete groups, and reserved staging.
• moe-cache-mm: sampled mmap page residency when mapped sources are used.

Counters aggregate across the process and reset at request-timing or unload boundaries. Under parallel load, one block does not belong exclusively to the request that triggered it.

covered > 0 and calls > 0 prove grouped execution. covered=0 means the cache used its established path. Compare H2D bytes only across runs that process the same tokens and do equivalent CPU work.

Branch implementations

Branch	Backend counters	Server reporting
moe-cache	moe-cache.cu	tools/server/server-context.cpp

Feature index - diagnostic interpretation