Support benchmarking an existing/remote inference endpoint (BYO endpoint)

[aistackdev]

Summary

Investigated whether InferenceX could benchmark an existing/remote inference endpoint instead of always launching the server on the GPU runner itself (context: analysis of run 29940925355, the GLM-5.2 W4AFP8 H200 DRAM-KV-offload/HiCache sweep). Today the whole pipeline assumes "launch server, then benchmark it, on the same box" — the client always talks to http://0.0.0.0:$PORT.

Current blocker

benchmarks/benchmark_lib.sh hardcodes http://0.0.0.0:$port in three places: run_benchmark_serving, run_lm_eval, and (the one that actually matters for the agentic path the analyzed run used) build_replay_cmd's --url.

Status: implemented on feat/remote-bench-endpoint, not yet merged

• 1. Shared library change — --base-url/REMOTE_BASE_URL override added to run_benchmark_serving, run_lm_eval, and build_replay_cmd in benchmarks/benchmark_lib.sh. Defaults to today's behavior when unset.
• 2. The -remote-bench recipe — benchmarks/single_node/agentic/glm5.2_fp4_sglang-remote-bench.sh. Required pre-flight check on REMOTE_BASE_URL/REMOTE_GPU_TELEMETRY_URL/REMOTE_ENGINE_METRICS_URL, optional REMOTE_RESET_URL cache-reset hook, self-reported RUNNER_TYPE override, same artifact schema as local recipes. One file per model+precision+framework (not per hardware, since remote-bench never launches a local server — hw-specific tuning doesn't apply).
• 3. Runner / config registration — cluster:remote-bench label registered on a real controller box (bench-client_01, no GPU), added to configs/runners.yaml (no hardware: entry needed), new runners/launch_bench-client.sh dispatcher (no docker/GPU mounts, unlike every other launcher — also has to point RESULT_DIR/INFMAX_CONTAINER_WORKSPACE at the real host paths since there's no bind mount). Wired up via new optional inputs on benchmark-tmpl.yml (remote-base-url etc., all empty-default/backward-compatible) plus a new dedicated workflow_dispatch entrypoint (remote-bench.yml) — production sweeps dispatch via perf-changelog.yaml/validation.py's closed schema, which we left untouched rather than extending it for per-run endpoint values.

Prior art

No existing issue or PR covers this. Closest adjacent asks upstream: #1045 (Kubernetes-native runner for distributed benchmarking, llm-d) and #1507 (engine startup time polluting GPU metric capture) — both still assume we control the server, just orchestrated differently.

Next steps

Smoke-test a live dispatch against a real remote endpoint, then open the PR.

[aistackdev]

Gap found during smoke-testing: no way to capture remote engine logs

Local recipes get server.log for free (the launcher redirects the locally-spawned process's stdout/stderr into it, uploaded as server_logs_<name>). Remote-bench has no equivalent — there's no local process to redirect, so engine-side log lines never reach the job at all.

Concretely hit this while debugging the smoke test against sglang-vanilla: this line only showed up via kubectl logs, not anywhere in the GH Actions job or artifacts:

[2026-07-23 09:42:13] Default fused-MoE Triton config {...} needs more shared memory than the device provides (101376 bytes); degraded to {...}. Consider contributing a tuned config for this device (see benchmark/kernels/fused_moe_triton).

Same blind spot applies to the detokenizer-hang crash log we also only found via kubectl logs — from inside a remote-bench job there's currently no way to see why a run failed on the server side, only that it did (ServerDisconnectedError client-side).

Proposed fix, additive, same shape as the existing required endpoints: an optional REMOTE_LOG_URL the operator exposes (e.g. tailing the engine's log file over HTTP, same pattern as gpu_metrics_url//metrics in the /discover contract) that the -remote-bench recipe curls at the end of the run and saves into $RESULT_DIR (e.g. results/remote_engine.log), so it rides along in the existing agentic_<name> artifact upload — no new upload step needed. Optional rather than required: unlike GPU/engine metrics, a run can still produce a valid throughput result without logs, they're just harder to debug after the fact.

Not implementing this now — flagging so it's not lost, and can fold into #2's recipe changes as a follow-up once we agree on the shape of REMOTE_LOG_URL.