docs(readme): stop claiming faster than vLLM, the curve is a tie - #29
Merged
Conversation
The README's headline claims contradicted its own Performance section. The
table has always been honest ("treat those as ties", "only c1 is clearly
outside the noise"), but the badge, the heading, the hero caption and the
"Fast." bullet all said we beat vLLM outright.
That is the claim Hacker News pushed back on when the LocalAI post went up
(item 49125065), and correctly: 0.7% to 1.7% against a 0.5% noise band is not
a speed win. The LocalAI blog post now says "Those are ties". This makes the
README agree with it.
- badge: "token-exact + faster at every concurrency" -> "token-exact + same
throughput"
- heading: "Performance: faster than vLLM" -> "Performance", with the two
anchor references updated (README badge link, docs/BENCHMARKS.md)
- hero GIF alt and caption: "ahead at every concurrency" / "vllm.cpp finishes
first" -> identical output, finishing a hair ahead, with the margin inside
the noise band at five of six concurrencies
- "Fast." bullet: "we beat vLLM at all six concurrencies" -> "we match vLLM's
throughput", keeping the per-point breakdown
- dropped "A tie is still a good result for a 66 MiB binary against a mature
CUDA stack", which grades our own result in the sentence that reports it
No measurement changed. The llama.cpp prefill (1.18x) and MLX-LM prefill
results are real wins and are stated as before, as is the memory result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README's headline claims contradicted its own Performance section.
The table below the fold has always been honest — "treat those as ties", "only c1, at 4.5%, is clearly outside the noise". But everything above it said we beat vLLM outright: the badge read "token-exact + faster at every concurrency", the section was titled "Performance: faster than vLLM", and the hero GIF caption said "vllm.cpp finishes first".
That is exactly the claim Hacker News pushed back on when the LocalAI post went up, and the pushback was right — 0.7% to 1.7% against a 0.5% run-to-run noise band is not a speed win. The LocalAI blog post now says "Those are ties" (mudler/LocalAI#11324, merged). This makes the README agree with it.
Changes
token-exact + faster at every concurrencytoken-exact + same throughputPerformance: faster than vLLMPerformanceahead at every concurrencyidentical output at every concurrencyvllm.cpp finishes firstFast.bulletwe beat vLLM at all six concurrencieswe match vLLM's throughput, per-point breakdown keptA tie is still a good result for a 66 MiB binary against a mature CUDA stackThat last one grades our own result in the sentence that reports it. Pre-chewing the numbers is what made a skeptical reader look harder at them on HN, not less hard.
Not changed
9.1 GiB, already stated 15 lines earlier.Same tokens as vLLM. Same throughput. 140x less to install.tagline was already consistent, so it stands.Both anchor references to the renamed heading are updated (the badge link and
docs/BENCHMARKS.md:144);grepconfirms no dangling#performance-faster-than-vllmremains, and the new shields.io URL returns 200.