[Bug] bench_serving TPOT incorrect when using sglang-oai-chat backend

[Alexhaoge]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

When using sglang-oai-chat backend, online serving benchmark sglang.bench_serving will produce an incorrect tpot, if the inference ends early and model output length is shorter than the user-specified output length argument like --random-output-len, --sharegpt-output-len, etc.

Reproduction

1. Launch a sglang server with any LLM model. It is easier to reproduce the issue with a chat model capable to end the answer properly under OpenAI chat completion API.

python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --tp 8 --mem-fraction-static 0.9 --trust-remote-code

2. Use sglang.bench_serving with --backend sglang-oai-chat and sharegpt datasets. Set the output length large enough, for example 4096.

python3 -m sglang.bench_serving --backend sglang-oai-chat \
    --model /model/DeepSeek-V3 \
    --dataset-name sharegpt \
    --dataset-path /home/ShareGPT_V3_unfiltered_cleaned_split.json  \
    --sharegpt-output-len 4096 \
    --max-concurrency 1 --num-prompt 1 --flush-cache \

3. Get the generated token lenght from console log, and TPOT from result file sglang-oai-chat_xxxx_1_sharegpt.jsonl. Mean TPOT will be much smaller than mean ITL(1.57ms vs 11.72ms), which is apparently incorrect since the test only has one requests. Moreover, Total generated tokens will always be the same as the specified output length in step2, but Total generated tokens (retokenized) will be much smaller, indicating the inference ends early and the actual output length is much smaller.

============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 1
Benchmark duration (s): 7.75
Total input tokens: 16
Total generated tokens: 4096
Total generated tokens (retokenized): 550
Request throughput (req/s): 0.13
Input token throughput (tok/s): 2.06
Output token throughput (tok/s): 528.58
Total token throughput (tok/s): 530.64
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7725.69
Median E2E Latency (ms): 7725.69
---------------Time to First Token----------------
Mean TTFT (ms): 1276.67
Median TTFT (ms): 1276.67
P99 TTFT (ms): 1276.67
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.72
Median ITL (ms): 11.73
P95 ITL (ms): 11.83
P99 ITL (ms): 11.92
Max ITL (ms): 12.54
==================================================

sglang-oai-chat_0922_1_sharegpt.jsonl

{"backend": "sglang-oai-chat", "dataset_name": "sharegpt", "request_rate": Infinity, "max_concurrency": 1, "sharegpt_output_len": 4096, "random_input_len": 1024, "random_output_len": 1024, "random_range_ratio": 0.0, "duration": 7.749096982181072, "completed": 1, "total_input_tokens": 16, "total_output_tokens": 4096, "total_output_tokens_retokenized": 550, "request_throughput": 0.12904729445243548, "input_throughput": 2.0647567112389678, "output_throughput": 528.5777180771757, "mean_e2e_latency_ms": 7725.692559964955, "median_e2e_latency_ms": 7725.692559964955, "std_e2e_latency_ms": 0.0, "p99_e2e_latency_ms": 7725.692559964955, "mean_ttft_ms": 1276.6728261485696, "median_ttft_ms": 1276.6728261485696, "std_ttft_ms": 0.0, "p99_ttft_ms": 1276.6728261485696, "mean_tpot_ms": 1.5748521938501552, "median_tpot_ms": 1.5748521938501552, "std_tpot_ms": 0.0, "p99_tpot_ms": 1.5748521938501552, "mean_itl_ms": 11.72394382314441, "median_itl_ms": 11.73052005469799, "std_itl_ms": 0.0957150902932436, "p95_itl_ms": 11.826230585575104, "p99_itl_ms": 11.921554170548914, "concurrency": 0.9969797226347876, "accept_length": null}

Environment

python3 -m sglang.check_env
/usr/local/lib/python3.12/dist-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
Python: 3.12.11 (main, Jun 4 2025, 08:56:18) [GCC 11.4.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H20
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.6, V12.6.68
CUDA Driver Version: 550.127.08
PyTorch: 2.8.0+cu126
sglang: 0.5.3rc0
sgl_kernel: 0.3.11
flashinfer_python: 0.4.0rc1
triton: 3.4.0
transformers: 4.56.1
torchao: 0.9.0+cu126
numpy: 2.3.2
aiohttp: 3.12.15
fastapi: 0.116.1
hf_transfer: 0.1.9
huggingface_hub: 0.34.4
interegular: 0.3.3
modelscope: 1.29.2
orjson: 3.11.3
outlines: 0.1.11
packaging: 25.0
psutil: 7.0.0
pydantic: 2.11.7
python-multipart: 0.0.20
pyzmq: 27.0.2
uvicorn: 0.35.0
uvloop: 0.21.0
vllm: Module Not Found
xgrammar: 0.1.24
openai: 1.99.1
tiktoken: 0.11.0
anthropic: 0.66.0
litellm: Module Not Found
decord: 0.6.0

[Alexhaoge]

Looking into benchmark_serving.py, I notice that the payload for OpenAI chat completion requests does not include ignore_eos or include_usage which can be the main reason for this issue.

sglang/python/sglang/bench_serving.py

Lines 317 to 324 in 2101d93

	payload = {
	"model": request_func_input.model,
	"messages": messages,
	"temperature": 0.0,
	"max_tokens": request_func_input.output_len,
	"stream": not args.disable_stream,
	**request_func_input.extra_request_body,
	}

Currently the output length of each request is accquired by usage in the last chunk of the chat completion stream reponse. If the key does not exist it fallbacks to the output length user specified.

sglang/python/sglang/bench_serving.py

Lines 388 to 390 in 2101d93

	output_len = (data.get("usage") or {}).get(
	"completion_tokens", output_len
	)

In such case, the usage key will always be empty since include_usage is not passed. Because ignore_eos is not passed, the inference may ends early leading to incorrect output length and tpot.

sglang and sglang-oai backends have similar logic, but they have ignore_eos in the payload so the actual output length is usually very close to user-specified length so these two backend will not have similar issues unless --disable-ignore-eos.

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.