Canonical Host KV Quantization

Canonical host-KV quantization

CPU and accelerator quantizers can produce different byte representations for the same standard quantized KV type. Both representations may be locally valid, but changing the numerical owner when KV placement changes can alter later model output. This feature keeps accelerator-executed host KV on the accelerator's quantization convention.

How it works

For a supported standard quantized host store, new F32 rows are converted to the persistent cache type on the accelerator, transferred back to host memory, and copied as same-type bytes into the host KV allocation. A bounded graph-owned staging buffer carries only the new rows. This staging traffic is distinct from the much larger history traffic used by attention.

No user flag controls this path. KV type, placement, and backend capability select the route. Unsupported combinations fail during construction or keep their established route.

Note

Backend scope: This is not generally CUDA-only, but it requires a supported accelerator conversion and transfer route. The current qualification is deepest for CUDA standard Q8 KV. KVarN and other specialized formats retain their own contracts.

Branch implementations

Branch	Implementation
beellama/main	src/llama-kv-cache.cpp, src/llama-kv-cache.h
llama/main	src/llama-kv-cache.cpp, src/llama-kv-cache.h
llama/dev	src/llama-kv-cache.cpp, src/llama-kv-cache.h
moe-cache	src/llama-kv-cache.cpp, src/llama-kv-cache.h

Design history: PR 30, preserve accelerator quantization for host KV stores.

Feature index - Pinned host KV