vllm-bench hardcodes ignore_eos and has no chat template, so a chat-completions comparator's protocol cannot be matched

[localai-org-maint-bot]

Row: BENCH-QWEN38-27B-SOTA

No harness in this tree can reproduce a chat-completions benchmark protocol: vllm-bench forces ignore_eos and applies no chat template, and the server applies both but reports no acceptance counters.

pangoleen/qwen3.8-27b-dgx-spark-dflash2 is the comparator for BENCH-QWEN38-27B-SOTA. Its bench/ctxsweep.py:138-141 posts to /v1/chat/completions with chat_template_kwargs {enable_thinking: false, preserve_thinking: false}, max_tokens 512 and no ignore_eos anywhere in bench/. Across the 25 rows of data/ctxsweep-recommended.csv, out_tokens is min 170, median 269, max 333: not one of 25 reaches the cap, so every generation they publish terminated on EOS.

What we have, and what each one cannot do.

	honours EOS	chat template	acceptance counters
vllm-bench (examples/bench/)	no — examples/bench/bench_core.h:558 sets sp.ignore_eos = true unconditionally; no BenchConfig field, no CLI flag in examples/bench/main.cpp:72-125, no environment variable	no — raw prompt strings only	yes — bench_core.h:933-939 from runner().spec_drafts_proposed()/accepted()
vllm-cli	yes (engine default ignore_eos = false)	no	no
vllm-server	yes — protocol.cpp:706, default false	yes — server_main.cpp:1401-1425	no
tools/bench/online_gate.py	no, and it enforces the absence: _parse_client_command_log raises HarnessError unless --ignore-eos and --skip-chat-template each appear exactly once and --endpoint is /v1/completions	no, same enforcement	no

The engine side is not the problem. include/vllm/sampling_params.h:193 defaults ignore_eos = false and src/vllm/v1/core/sched/utils.cpp:26-30 mirrors vLLM's _eos_token_id gate. The counters exist at include/vllm/v1/worker/gpu/runner.h:269-270. What is missing is any single client that puts the three together.

Why this is a measurement problem and not a convenience one. Suppressing EOS decodes past the point where the model would have stopped, into lower-entropy continuation, and lower-entropy text is easier for a drafter to predict. That can inflate accepted-tokens-per-pass, and it inflates it on OUR side of the comparison, so it flatters us. Our published 4.06 accepted per step (docs/benchmarks/qwen38-27b-exl3-gb10.md) was taken with every request forced to exactly 128 tokens; how much of it is the workload and how much is the forced length is currently unmeasurable. Their median generation is 269 tokens against our 128, which mildly flatters them in the other direction by amortizing per-request overhead over more steady-state decode.

Three separable pieces.

1. BenchConfig gains an ignore_eos field and --ignore-eos / --no-ignore-eos, defaulting to the current true so no landed figure moves. bench_core.h:558 reads it. One field, one flag, one line.
2. A chat-template render before EncodeWithSpecialTokens on the bench admission path, reusing MakeChatTemplatePromptFn which the server already builds.
3. A speculative-decoding metric family on /metrics, so a server-driven sweep can read acceptance the way ctxsweep.py reads sglang:spec_accept_length and vllm:spec_decode_num_accepted_tokens_total. Today the only server-side signal is the opt-in stderr trace VT_SPEC_TRACE=1 (src/vllm/v1/worker/gpu/runner.cpp:3182-3206), and parsing a log is not a metric.

Note that (3) also unblocks tools/bench/c8_leg_runner.py:28-31, which already records that scripts/dgx-online-serving.sh "cannot express a speculative workload yet".

Until at least (1) and (2) land, any comparison against their ladder must print observed output lengths beside every rate and name the stopping rule, which is what .agents/specs/bench-qwen38-27b-nvfp4-matched.md §4 axis (3) requires. Owed under that spec's ## Owed.