Repository navigation
kvserve v1.0.0: paged-KV LLM serving engine simulator
kvserve is a from-scratch, executable model of a vLLM/SGLang-class inference engine. It includes a PagedAttention block manager with refcounted automatic prefix caching, Orca continuous batching, Sarathi chunked prefill, recompute preemption, a GQA-aware roofline step-cost model, and three workload generators: Poisson, multi-tenant shared prompts and multi-turn agent sessions.
Results (simulated Llama-3-8B on one H100)
- Continuous vs static batching at 30 rps: 2.4× token throughput and 246× lower p99 TTFT (0.12 s vs 29.3 s). Goodput is 21.8 vs 0.41 req/s.
- Automatic prefix caching: 93.8% block hit rate skips 93% of prefill compute on shared system prompts. On multi-turn agent sessions the hit rate is 82% and 81% of prefill is skipped.
- Chunked prefill, long prompts: p99 TPOT is 3.8× lower (17 ms vs 64 ms). The Sarathi trade-off on median TTFT is reported.
- Memory pressure: recompute preemption completes every request. Block-manager invariants are asserted after every step, and no blocks leak.
- C++20 chained block-hash kernel: about 870× faster than the NumPy reference twin, with bit-exact output.
Quality gates (green Ubuntu CI)
- Mechanism-level tests and unit tests: refcounting, LRU eviction, duplicate-content blocks, OutOfBlocks atomicity, invariant corruption detection, roofline regimes.
- Hypothesis bit-exactness and chain-continuation properties for the hash kernel.
mypy --strict, ruff, 99.6% branch coverage, C++ built with-Werror.- CI re-runs the benchmark. No mocks.