[Benchmark] Share one ClientSession across benchmark requests

[kevincheng2]

Motivation

async_request_eb_openai_chat_completions already accepts an optional session, but benchmark_serving never passes one, so every request opens a new aiohttp.ClientSession and therefore a new TCP connection. On long runs (--num-prompts in the tens of thousands) the accumulated TIME_WAIT sockets can exhaust local ports and perturb the latency being measured.

Modifications

Create one ClientSession with a bounded connection pool in benchmark() and hand it to every request when the backend request function accepts a session parameter, detected via inspect.signature. Backends without that parameter are called exactly as before. The session is closed after asyncio.gather completes.

This only touches the benchmark client. No engine, kernel or model code is changed.

Usage or Command

No new flags. Run the benchmark as before:

python benchmarks/benchmark_serving.py \
    --backend openai-chat \
    --model <model> \
    --endpoint /v1/chat/completions \
    --dataset-name EBChat \
    --dataset-path <dataset> \
    --num-prompts 20000 \
    --max-concurrency 64

The connection pool is sized at max_concurrency * 2 (unbounded when --max-concurrency is not given). Socket accumulation can be observed with ss -tan state time-wait | wc -l during a long run.

Accuracy Tests

Not applicable — this PR does not affect model outputs. It changes only connection reuse in the benchmark client.

pre-commit run passes on the changed file. Backends whose request function has no session parameter keep their previous call signature, so their behavior is unchanged by construction.

Checklist

• Add at least a tag in the PR title. ([Benchmark])
• Format your code, run pre-commit before commit.
• Add unit tests. — the benchmark scripts under benchmarks/ have no unit test harness in this repo; the change is guarded by an inspect.signature check so unsupported backends are called exactly as before.
• Provide accuracy results. — not applicable, see above.
• If the current PR is submitting to the release branch... — targets develop.

[kevincheng2]

/re-run all-failed

[kevincheng2]

Closing this — the connection reuse change is not needed for now. The decode-speed measurement fix is tracked separately in #8113.