Skip to content

v0.19.1: gpt-oss-20b decodes 349.7 to 391.2 tok/s; speculation stops losing on GGUF K-quants

Choose a tag to compare

View all tags
@kekzl kekzl released this 16 Jul 22:15
· 710 commits to main since this release
v0.19.1
f7119ad
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • Gemma-4 per-layer attention routing (#1042): the SWA majority (5:1) now rides FA2 f16-QK per-layer; hd=512 global layers keep the faster materialized cuBLAS path, backed by a new fused WMMA FMHA hd=512 instantiation as the O(n)-memory fallback and a sliced-cuBLAS path at S-matrix overflow (3.4-3.9x faster than the fused kernel at Skv 8k/16k). The global chunk clamp is lifted for heterogeneous models — full 2048-row prefill chunks at any context (was ~190-row chunks at 64k). Gemma-4-12B pp16384 prefill +5.3% end-to-end.
  • gemm.fp8_attn_proj FP8 decode sidecar (#984): per-row-scale FP8 copies of full-precision attention projections, decode-only. gpt-oss-20b decode 349.7 → 391.2 tok/s (+12%), turning the llama.cpp statistical tie into a +13-19% lead.
  • Spec-decode verify GEMMs on the NVFP4 overlay (#998): verify chunks no longer pay full K-quant dequant per step — Qwen3-14B Q6_K decode at ctx 2048: 91.9 → 153.2 tok/s (+67%).
  • Tool calling, stage 2 (#1002): JSON-Schema $defs hoisting, strict argument enforcement via a dedicated TOOL_ARGS grammar gate, parallel tool calls, and Llama-3 forced tool-call arguments.
  • Long-context hygiene (#1022): NIAH retrieval gate past 16k, SWA-KV sizing validated (opt-in), and single-chunk TTFT baselines pinned for pp8192 through pp65536.
  • KV-pool exhaustion at decode is now diagnosable (#1042): the reject-newest cancel logs block numbers and remedies, admission warns when a prompt leaves less than one KV block of decode headroom, and imp_decode_step returns IMP_ERROR_CANCELLED instead of a bare "internal error".

Also in this release

  • Perf gate now measures speculation-OFF decode; decode/prefill baselines re-pinned accordingly, chunked-prefill baseline fixed.
  • gemm.nvfp4_lm_head moved to a per-model auto net rule (#982).
  • Dependency bumps: CUTLASS v4.6.1, cpp-httplib v0.50.1 (three upstream security fixes).
  • Prefill CUDA-graph continuation-chunk geometry fix (#981) — quantized-KV models no longer waste a forward per chunk.

Full details: CHANGELOG · Diff: v0.19.0...v0.19.1

Assets 2
Loading