Row: SERVE-METRICS
Our OpenAI server exposes no speculative-decoding acceptance metric. vLLM at the
parity pin 5559679229bc961848b121ccdeaa8fa5d79bec98 exports four, including
vllm:spec_decode_num_accepted_tokens_total.
On our side the counters exist but are unreachable over HTTP.
spec_drafts_proposed() and spec_drafts_accepted() are GpuRunner accessors,
and their only non-test readers are in examples/bench/bench_core.h. So the
acceptance rate is obtainable from the bench binary and from nowhere else.
Why this blocks work rather than being cosmetic
Acceptance is the mechanism metric for speculative decoding. Throughput alone
cannot separate "the drafter proposed well" from "the target forward is fast",
and those call for opposite fixes. Two live comparisons are distorted by the gap:
- The benchmark in
docs/benchmarks/qwen38-27b-exl3-gb10.md reports acceptance
only because it runs through vllm-bench. The same measurement through the
server, which is how anyone would actually serve the model, reports none.
- A head-to-head against
MiaAI-Lab/exllamav3 has to drive both engines over
HTTP so one client times both. Their server reports usage; ours reports no
acceptance, so the comparison is asymmetric for a reason that is plumbing.
vllm-bench is also the only harness that can read the counters, and it
hardcodes ignore_eos = true (#2759),
so no configuration of this tree can report acceptance for an EOS-terminated
generation — which is what every published comparison measures.
Ask
Mirror vLLM's speculative-decoding metrics on the OpenAI server. Match the
upstream names and semantics rather than inventing our own, and gate them the
way the other exported metrics are gated.
Found while preparing the head-to-head in
#2761.
Refs #2495
Refs #2759
Row:
SERVE-METRICSOur OpenAI server exposes no speculative-decoding acceptance metric. vLLM at the
parity pin
5559679229bc961848b121ccdeaa8fa5d79bec98exports four, includingvllm:spec_decode_num_accepted_tokens_total.On our side the counters exist but are unreachable over HTTP.
spec_drafts_proposed()andspec_drafts_accepted()areGpuRunneraccessors,and their only non-test readers are in
examples/bench/bench_core.h. So theacceptance rate is obtainable from the bench binary and from nowhere else.
Why this blocks work rather than being cosmetic
Acceptance is the mechanism metric for speculative decoding. Throughput alone
cannot separate "the drafter proposed well" from "the target forward is fast",
and those call for opposite fixes. Two live comparisons are distorted by the gap:
docs/benchmarks/qwen38-27b-exl3-gb10.mdreports acceptanceonly because it runs through
vllm-bench. The same measurement through theserver, which is how anyone would actually serve the model, reports none.
MiaAI-Lab/exllamav3has to drive both engines overHTTP so one client times both. Their server reports usage; ours reports no
acceptance, so the comparison is asymmetric for a reason that is plumbing.
vllm-benchis also the only harness that can read the counters, and ithardcodes
ignore_eos = true(#2759),so no configuration of this tree can report acceptance for an EOS-terminated
generation — which is what every published comparison measures.
Ask
Mirror vLLM's speculative-decoding metrics on the OpenAI server. Match the
upstream names and semantics rather than inventing our own, and gate them the
way the other exported metrics are gated.
Found while preparing the head-to-head in
#2761.
Refs #2495
Refs #2759