[Benchmark] Measure real decode speed when the server batches stream chunks

[kevincheng2]

Motivation

A server-side reasoning / tool-call parser only emits an SSE chunk once it has user-visible text, so decode steps whose text is still buffered are invisible to the client while their tokens are already counted in usage.completion_tokens. Taking TTFT from the first delta-carrying chunk then charges those tokens to TTFT and shrinks the decode window, which inflates the reported decode speed.

Modifications

• count a chunk whose delta is empty but whose usage.completion_tokens grew as a real token arrival, so TTFT/ITL see every generated token
• credit the tokens that only show up in the trailing usage chunk (e.g. tool-call closing markers) to ITL, so the numerator and the denominator cover the same tokens
• derive per-request decode speed from len(itl)/sum(itl); the previous (output_tokens - 1) form assumes the first chunk carries exactly one token, which is false under speculative decoding
• drop the "TPOT < 1ms is unreliable" filter and its duplicated report block: with the window measured correctly there is nothing to filter

This only touches the benchmark client. No engine, kernel or model code is changed.

Usage or Command

No new flags. Run the benchmark as before:

python benchmarks/benchmark_serving.py \
    --backend openai-chat \
    --model <model> \
    --endpoint /v1/chat/completions \
    --dataset-name EBChat \
    --dataset-path <dataset> \
    --num-prompts 100 \
    --max-concurrency 16

The decode-speed report is now a single section, 解码速度(tok/s, 首token之后), plus Requests in decode stats. The previous 解码速度 (过滤TPOT<1ms) block and the s_decode_filtered_* / n_decode_filtered / n_decode_reliable keys in the JSON result are removed, since the window is now measured correctly and there is nothing to filter.

Accuracy Tests

Not applicable — this PR does not affect model outputs. It changes only how the benchmark client attributes timestamps to tokens.

Measurement correctness was verified against a synthetic SSE stream served by a local aiohttp server, driving the real async_request_eb_openai_chat_completions. Ground truth: 22 tokens, one every 20ms (50 tok/s, real TTFT 20ms), where the first 8 decode steps are swallowed by a parser and 2 tokens appear only in the trailing usage chunk.

• server emits a usage chunk per buffered step: TTFT 24.3ms, len(itl)=21, decode 49.55 tok/s
• server does not: TTFT 183.2ms, previous formula 79.85 tok/s (60% too high), this PR's formula 49.43 tok/s

So the reported decode speed matches ground truth in both cases, and the remaining TTFT error in the second case requires the server to emit the per-step usage chunk.

Checklist

• Add at least a tag in the PR title. ([Benchmark])
• Format your code, run pre-commit before commit.
• Add unit tests. — the benchmark scripts under benchmarks/ have no unit test harness in this repo; correctness was instead verified end-to-end against a synthetic SSE server as described above.
• Provide accuracy results. — not applicable, see above.
• If the current PR is submitting to the release branch... — targets develop.

[kevincheng2]

/re-run all-failed