THE PREFILL GAP IS PROBABLY THE KV DTYPE: the oracle runs kv_cache_dtype=fp8_e4m3 (confirmed at runtime), we store bf16 — it moves HALF the KV bytes we do

[localai-bot]

Found auditing vllm.ai "Anatomy of vLLM" against our tree. This is the first hypothesis that explains every measured property of the gap.

Runtime-confirmed, not inferred

The pinned oracle's own startup config, from server logs captured today on the 27B and 35B:

kv_cache_dtype=fp8_e4m3

vLLM resolves kv_cache_dtype="auto" from the checkpoint: resolve_kv_cache_dtype_string (vllm/utils/torch_utils.py:374-392) -> get_kv_cache_quant_algo_string (:310-350) reads kv_cache_quant_algo from the ModelOpt quantization_config. nvidia/Qwen3.6-27B-NVFP4 and nvidia/Qwen3.6-35B-A3B-NVFP4 both declare kv_cache_quant_algo: FP8 (our own record already noted this: .agents/benchmark-record.md:17077, :17141).

Our side:

// include/vllm/v1/kv_cache_dtype.h:28-32
inline vt::DType ResolveKvCacheDType() {
  const char* kv_f32_env = std::getenv("VT_KV_CACHE_F32");
  return (kv_f32_env != nullptr && kv_f32_env[0] == '1') ? vt::DType::kF32
                                                         : vt::DType::kBF16;
}

docs/STATUS.md:115: "No model can run with an fp8 KV cache yet." ParseCacheDType exists with zero production callers. The CUDA store and the fp8 paged-attention read are named W2-W5 in .agents/specs/fp8-kv-cache.md:26-28.

vLLM moves half the KV bytes we do, on the gate models, in the canonical grid.

It matches every property of the measured gap

observed	explained by fp8-vs-bf16 KV?
The deficit is PREFILL; decode is ~0.99x	Yes. Every prefilled token pays a reshape_and_cache K+V write, and prefill's causal attention re-reads that cache. Prefill touches ~1024x more KV bytes per request than a decode step.
The ratio is FLAT across c1-c32	Yes. It is a per-token cost, invariant in concurrency. This is the property that ruled out launch-count levers, which would shrink as a fraction of a growing step.
The gap is ~5%, not ~20%	Yes. Diluted by the hybrid: only 16 of 64 layers use paged KV on the 27B (48 are GDN with recurrent state).
Decode-dominant harness reads 0.9883x	Yes. One decode step writes 1 token of KV; the asymmetry barely registers.

No other candidate this campaign examined fits all four. The launch-count levers are contradicted by the flatness; GDN prefill is kernel-for-kernel identical to vLLM's on GB10; the cuDNN "epilogue fusion" premise was refuted (we are more fused there).

Status: this is a MIRROR gap, not a denominator defect

vLLM is running its correct production configuration. We are the ones missing a capability. AGENTS.md: "When vLLM defines behavior, mirror every applicable mode, default, error, and edge case."

This is distinct from #414 (--language-model-only), which IS a denominator defect where we handicapped the oracle. Both are real; they run in opposite directions.

Owed

1. Read the checkpoints' hf_quant_config.json and confirm kv_cache_quant_algo: FP8 directly. The auditor could not (checkpoints not mounted on that box) and inferred it from the code path plus our own record. The runtime log is strong evidence, but read the file before scheduling work.
2. Measure it before building it. A reshape_and_cache + paged-read byte-traffic estimate at ISL 1024 over 16 full-attention layers, against the ~5% deficit. This is arithmetic and needs no GPU.
3. If it sizes, W2-W5 of .agents/specs/fp8-kv-cache.md becomes the parity row: CUDA fp8 store, fp8 paged-attention read, half-sized KV blocks, checkpoint-scale threading, --kv-cache-dtype.
4. Token-exactness is the hard constraint. An fp8 KV cache changes stored precision, so SACRED 27B 235/235 and 35B 315/315 must be re-established, and vLLM's own per-tensor k/v scale handling (BaseKVCacheMethod) must be mirrored exactly rather than approximated.

Second finding from the same audit, unrelated but cheap

src/vllm/v1/engine/output_processor.cpp:261-262 deep-copies prompt (std::string) and prompt_token_ids (std::vector<int32_t>) into every RequestOutput, once per emitted token. Upstream assigns references (vllm/v1/engine/output_processor.py:369-372). At a 1024-token prompt that is ~4 KB memcpy + a string allocation per generated token per request. Flat per-request tax, trivially fixable, worth its own small row.