Skip to content

kvserve v1.0.0 — Paged-KV LLM Serving Engine Simulator

Latest

Choose a tag to compare

@nagarjunak-pixel nagarjunak-pixel released this 18 Nov 11:22

kvserve v1.0.0: paged-KV LLM serving engine simulator

kvserve is a from-scratch, executable model of a vLLM/SGLang-class inference engine. It includes a PagedAttention block manager with refcounted automatic prefix caching, Orca continuous batching, Sarathi chunked prefill, recompute preemption, a GQA-aware roofline step-cost model, and three workload generators: Poisson, multi-tenant shared prompts and multi-turn agent sessions.

Results (simulated Llama-3-8B on one H100)

  • Continuous vs static batching at 30 rps: 2.4× token throughput and 246× lower p99 TTFT (0.12 s vs 29.3 s). Goodput is 21.8 vs 0.41 req/s.
  • Automatic prefix caching: 93.8% block hit rate skips 93% of prefill compute on shared system prompts. On multi-turn agent sessions the hit rate is 82% and 81% of prefill is skipped.
  • Chunked prefill, long prompts: p99 TPOT is 3.8× lower (17 ms vs 64 ms). The Sarathi trade-off on median TTFT is reported.
  • Memory pressure: recompute preemption completes every request. Block-manager invariants are asserted after every step, and no blocks leak.
  • C++20 chained block-hash kernel: about 870× faster than the NumPy reference twin, with bit-exact output.

Quality gates (green Ubuntu CI)

  • Mechanism-level tests and unit tests: refcounting, LRU eviction, duplicate-content blocks, OutOfBlocks atomicity, invariant corruption detection, roofline regimes.
  • Hypothesis bit-exactness and chain-continuation properties for the hash kernel.
  • mypy --strict, ruff, 99.6% branch coverage, C++ built with -Werror.
  • CI re-runs the benchmark. No mocks.