docs(FEATURES): Vulkan is at parity with llama.cpp, not behind it (#310) - #311
Merged
Conversation
`docs/FEATURES.md` said Qwen3.6-27B decodes **4.24 tok/s vs llama.cpp's 4.35**. That is the pre-#191 number: it claimed we were BEHIND when the binding measurement is parity MET, and it contradicted the three other public surfaces, all of which were reviewed and landed saying 4.36: docs/STATUS.md decode 4.36/4.35 MET docs/BENCHMARKS.md decode 4.36 vs 4.35, MET (7 clean legs) README.md decodes 4.36 vs llama.cpp Vulkan 4.35 tok/s Corrected, and it keeps saying the pass is NARROW: the binding figure is the median of 7 clean legs at TPOT 228.66-230.25 ms, and the 0.69% leg spread IS the noise floor. The detail stays in BENCHMARKS and the record rather than being restated here. Also adds the load-phase result the paragraph was missing beside its memory one (`ENG-LOAD-DIRECT-UPLOAD`, #150): a load is 1.54x faster warm, alongside the already-recorded 100.8 -> 53.4 GiB peak RSS. WHY THE PARAGRAPH IS SHORTER THAN THE SUM OF ITS PARTS. The page budgets prose at 20 paragraphs and 700 chars each, and my first correction failed both. The paragraph-count failure was the interesting one: `_prose_paragraphs` treats any line starting with `*` as a list item, and on main the wrap happened to put `**prefill 21.5x**` at the start of a line -- so the checker had been reading ONE paragraph as TWO, and the page was inside its 700-char budget only by accident. Rewrapping honestly merged them and exposed 891 real chars. Rather than restore the accidental split or spend the count, the paragraph was tightened to 677 (`6/6 prompts / 96/96 tokens vs the vLLM 0.25.0 oracle` -> `6/6 prompts vs the vLLM 0.25.0 oracle`; `all nine of that model's ops` -> `every op of that model`), which is what the checker asks for: the numbers belong in BENCHMARKS, and this page explains what works. DELIBERATELY NOT DONE: moving Vulkan out of STATUS's "Not supported yet" section. Running a 27B at llama.cpp parity is a real milestone, but 25 native ops of 112 with quant/MoE/MLA at zero is not "supported", and the entry already states precisely what does and does not work. Gates: check-public-doc-tables, check-doc-checkpoint and check-agent-record all green. No source file changes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
This was referenced Aug 10, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs(FEATURES): Vulkan is at parity with llama.cpp, not behind it (#310)
docs/FEATURES.mdsaid Qwen3.6-27B decodes 4.24 tok/s vs llama.cpp's 4.35.That is the pre-#191 number: it claimed we were BEHIND when the binding
measurement is parity MET, and it contradicted the three other public surfaces,
all of which were reviewed and landed saying 4.36:
docs/STATUS.md decode 4.36/4.35 MET
docs/BENCHMARKS.md decode 4.36 vs 4.35, MET (7 clean legs)
README.md decodes 4.36 vs llama.cpp Vulkan 4.35 tok/s
Corrected, and it keeps saying the pass is NARROW: the binding figure is the
median of 7 clean legs at TPOT 228.66-230.25 ms, and the 0.69% leg spread IS the
noise floor. The detail stays in BENCHMARKS and the record rather than being
restated here.
Also adds the load-phase result the paragraph was missing beside its memory one
(
ENG-LOAD-DIRECT-UPLOAD, #150): a load is 1.54x faster warm, alongside thealready-recorded 100.8 -> 53.4 GiB peak RSS.
WHY THE PARAGRAPH IS SHORTER THAN THE SUM OF ITS PARTS. The page budgets prose
at 20 paragraphs and 700 chars each, and my first correction failed both. The
paragraph-count failure was the interesting one:
_prose_paragraphstreats anyline starting with
*as a list item, and on main the wrap happened to put**prefill 21.5x**at the start of a line -- so the checker had been reading ONEparagraph as TWO, and the page was inside its 700-char budget only by accident.
Rewrapping honestly merged them and exposed 891 real chars. Rather than restore
the accidental split or spend the count, the paragraph was tightened to 677
(
6/6 prompts / 96/96 tokens vs the vLLM 0.25.0 oracle->6/6 prompts vs the vLLM 0.25.0 oracle;all nine of that model's ops->every op of that model),which is what the checker asks for: the numbers belong in BENCHMARKS, and this
page explains what works.
DELIBERATELY NOT DONE: moving Vulkan out of STATUS's "Not supported yet"
section. Running a 27B at llama.cpp parity is a real milestone, but 25 native
ops of 112 with quant/MoE/MLA at zero is not "supported", and the entry already
states precisely what does and does not work.
Gates: check-public-doc-tables, check-doc-checkpoint and check-agent-record all
green. No source file changes.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude-Code:claude-opus-5 [Claude Code]