Skip to content

The OpenAI server exports no speculative-decoding acceptance metric, so acceptance is readable only from vllm-bench #2770

Description

@localai-org-maint-bot

Row: SERVE-METRICS

Our OpenAI server exposes no speculative-decoding acceptance metric. vLLM at the
parity pin 5559679229bc961848b121ccdeaa8fa5d79bec98 exports four, including
vllm:spec_decode_num_accepted_tokens_total.

On our side the counters exist but are unreachable over HTTP.
spec_drafts_proposed() and spec_drafts_accepted() are GpuRunner accessors,
and their only non-test readers are in examples/bench/bench_core.h. So the
acceptance rate is obtainable from the bench binary and from nowhere else.

Why this blocks work rather than being cosmetic

Acceptance is the mechanism metric for speculative decoding. Throughput alone
cannot separate "the drafter proposed well" from "the target forward is fast",
and those call for opposite fixes. Two live comparisons are distorted by the gap:

  • The benchmark in docs/benchmarks/qwen38-27b-exl3-gb10.md reports acceptance
    only because it runs through vllm-bench. The same measurement through the
    server, which is how anyone would actually serve the model, reports none.
  • A head-to-head against MiaAI-Lab/exllamav3 has to drive both engines over
    HTTP so one client times both. Their server reports usage; ours reports no
    acceptance, so the comparison is asymmetric for a reason that is plumbing.

vllm-bench is also the only harness that can read the counters, and it
hardcodes ignore_eos = true (#2759),
so no configuration of this tree can report acceptance for an EOS-terminated
generation
— which is what every published comparison measures.

Ask

Mirror vLLM's speculative-decoding metrics on the OpenAI server. Match the
upstream names and semantics rather than inventing our own, and gate them the
way the other exported metrics are gated.

Found while preparing the head-to-head in
#2761.

Refs #2495
Refs #2759

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions