Skip to content

docs(readme): stop claiming faster than vLLM, the curve is a tie - #29

Merged
mudler merged 1 commit into
mainfrom
docs/readme-honest-throughput
Aug 3, 2026
Merged

docs(readme): stop claiming faster than vLLM, the curve is a tie#29
mudler merged 1 commit into
mainfrom
docs/readme-honest-throughput

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

The README's headline claims contradicted its own Performance section.

The table below the fold has always been honest — "treat those as ties", "only c1, at 4.5%, is clearly outside the noise". But everything above it said we beat vLLM outright: the badge read "token-exact + faster at every concurrency", the section was titled "Performance: faster than vLLM", and the hero GIF caption said "vllm.cpp finishes first".

That is exactly the claim Hacker News pushed back on when the LocalAI post went up, and the pushback was right — 0.7% to 1.7% against a 0.5% run-to-run noise band is not a speed win. The LocalAI blog post now says "Those are ties" (mudler/LocalAI#11324, merged). This makes the README agree with it.

Changes

Where Before After
badge token-exact + faster at every concurrency token-exact + same throughput
section heading Performance: faster than vLLM Performance
GIF alt text ahead at every concurrency identical output at every concurrency
GIF caption vllm.cpp finishes first identical output, finishing a hair ahead, margin inside the noise band at five of six
Fast. bullet we beat vLLM at all six concurrencies we match vLLM's throughput, per-point breakdown kept
after the table A tie is still a good result for a 66 MiB binary against a mature CUDA stack dropped

That last one grades our own result in the sentence that reports it. Pre-chewing the numbers is what made a skeptical reader look harder at them on HN, not less hard.

Not changed

  • No measurement. Every figure is identical (verified by diffing numeric tokens); the only "new" number is 9.1 GiB, already stated 15 lines earlier.
  • The real wins stay stated as they were: 1.18x llama.cpp prefill on the same GGUF, ahead of MLX-LM on prefill, and 24.88 GiB vs 28.18 GiB peak host memory, which is a clean result and still called one.
  • The Same tokens as vLLM. Same throughput. 140x less to install. tagline was already consistent, so it stands.

Both anchor references to the renamed heading are updated (the badge link and docs/BENCHMARKS.md:144); grep confirms no dangling #performance-faster-than-vllm remains, and the new shields.io URL returns 200.

The README's headline claims contradicted its own Performance section. The
table has always been honest ("treat those as ties", "only c1 is clearly
outside the noise"), but the badge, the heading, the hero caption and the
"Fast." bullet all said we beat vLLM outright.

That is the claim Hacker News pushed back on when the LocalAI post went up
(item 49125065), and correctly: 0.7% to 1.7% against a 0.5% noise band is not
a speed win. The LocalAI blog post now says "Those are ties". This makes the
README agree with it.

- badge: "token-exact + faster at every concurrency" -> "token-exact + same
  throughput"
- heading: "Performance: faster than vLLM" -> "Performance", with the two
  anchor references updated (README badge link, docs/BENCHMARKS.md)
- hero GIF alt and caption: "ahead at every concurrency" / "vllm.cpp finishes
  first" -> identical output, finishing a hair ahead, with the margin inside
  the noise band at five of six concurrencies
- "Fast." bullet: "we beat vLLM at all six concurrencies" -> "we match vLLM's
  throughput", keeping the per-point breakdown
- dropped "A tie is still a good result for a 66 MiB binary against a mature
  CUDA stack", which grades our own result in the sentence that reports it

No measurement changed. The llama.cpp prefill (1.18x) and MLX-LM prefill
results are real wins and are stated as before, as is the memory result.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mudler
mudler merged commit 8ff28ff into main Aug 3, 2026
6 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants