feat: support benchmarking remote/existing inference endpoints (BYO endpoint)#27
Merged
Merged
Conversation
…ndpoint) Adds a REMOTE_BASE_URL override to the three hardcoded http://0.0.0.0:$port paths in benchmark_lib.sh (run_benchmark_serving, run_lm_eval, build_replay_cmd), a new hw-agnostic *-remote-bench.sh recipe convention that benchmarks an externally-managed endpoint instead of launching a local server, and the runner/workflow plumbing to dispatch it (cluster:remote-bench label, launch_bench-client.sh, remote-bench.yml workflow_dispatch entrypoint). All additive; no existing recipe, workflow input, or config schema changes behavior when unset. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Smoke-tested against a live sglang-vanilla endpoint and hit a reproducible silent 100%-GPU hang, always on the same trace, always after a partial chunked-prefill chunk. Root cause: without a client-side context cap, aiperf replayed trace turns up to 255,999 tokens against a model deployed with only a 131,072-token context, relying entirely on server-side --allow-auto-truncate - which triggers the hang. Capping via --max-context-length (aiperf already supports this; local recipes never needed it since they hardcode --context-length for the server they launch themselves) avoids the pathological case entirely - confirmed with a full successful end-to-end run afterward. Required, not optional, since every remote target has a real context limit an operator can self-report, same as the other required remote-bench inputs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Documents the full workflow for bringing up a new remote-bench recipe: required env-var contract, why config behind the endpoint still needs self-reporting even though InferenceX doesn't control it, what file to write (one per model+precision+framework, not per hardware - the body is server-agnostic), the debug loop for pre-merge validation, and what a real ingest-able run produces. Also fixes remote-bench.yml: `image` was hardcoded to a placeholder string that flows straight into the ingested artifact's `image` field; `dp-attn` was hardcoded false. Both are now real, self-reported inputs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2 tasks
aistackdev
pushed a commit
that referenced
this pull request
Jul 24, 2026
…ndpoint) Adds remote-bench: a dedicated workflow_dispatch entrypoint and recipe convention (*-remote-bench.sh) for benchmarking an already-running, externally-managed inference endpoint instead of launching a server on the GPU runner. Includes the REMOTE_BASE_URL override across benchmark_lib.sh's serving/eval/replay call sites, the cluster:remote-bench runner registration, and the create-remote-bench / sync-remote-bench-with-main skills for bringing up new model configs and landing future changes onto main without dragging in unrelated vng-benchmark commits. Background: issue #26, PR #27.
This was referenced Jul 24, 2026
aistackdev
pushed a commit
that referenced
this pull request
Jul 24, 2026
…ndpoint) Adds remote-bench: benchmark an already-running, externally-managed inference endpoint instead of launching a server on the GPU runner. - benchmark_lib.sh: REMOTE_BASE_URL override in build_replay_cmd so aiperf points at an external endpoint instead of localhost. - benchmark-tmpl.yml: 6 optional remote-* workflow_call inputs/env vars, consumed only by *-remote-bench.sh recipes. - remote-bench-e2e.yml: single entrypoint, takes a JSON array of configs (one element for a smoke test, many for a full CCU-ladder sweep), matrix-fans to benchmark-tmpl.yml, then aggregates via collect-results + calc-success-rate. - Two model-agnostic *-remote-bench.sh recipes (dsv2lite fp8, glm5.2 fp4), the cluster:remote-bench runner label + non-GPU controller launcher (bench-client_01). - create-remote-bench skill: how to bring up a new recipe, including checking the endpoint's KV pool capacity to size a concurrency ladder, and the REMOTE_MAX_CONTEXT_LENGTH incident (a per-request cap is required, and even the model's nominal context window isn't automatically a safe value on small/dev GPUs). - sync-upstream + fork-changelog skills: since fork commits now live directly on main instead of a separate customization branch, syncing upstream is a real merge — fork-changelog lists every fork-only edit and why, sync-upstream is the procedure that checks against it. Background: issue #26/#28, PR #27/#30/#31/#34 (squashed into this single commit).
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #26. Adds support for benchmarking an already-running, externally-managed inference endpoint instead of always launching the server on the GPU runner itself.
benchmark_lib.sh:REMOTE_BASE_URL/--base-urloverride added torun_benchmark_serving,run_lm_eval, andbuild_replay_cmd(the hardcode actually used by the agentic path). Defaults to today's behavior when unset — no existing recipe is affected.*-remote-bench.shrecipe convention (glm5.2_fp4_sglang-remote-bench.sh,dsv2lite_fp8_sglang-remote-bench.sh): required pre-flight check onREMOTE_BASE_URL/REMOTE_GPU_TELEMETRY_URL/REMOTE_ENGINE_METRICS_URL/REMOTE_MAX_CONTEXT_LENGTH, optionalREMOTE_RESET_URLcache-reset hook, self-reportedRUNNER_TYPEoverride for ingest, same artifact schema as local recipes.cluster:remote-benchlabel on a real non-GPU controller box (bench-client_01),runners/launch_bench-client.sh(no docker/GPU mounts, unlike every other launcher), 6 new optional inputs onbenchmark-tmpl.yml(empty-default, backward compatible), and a newremote-bench.ymlworkflow_dispatch entrypoint for ad-hoc dispatch (production sweeps still go throughperf-changelog.yaml, left untouched).All changes are additive; existing recipes, workflows, and config schemas are unaffected when the new inputs/env vars are unset.
Test plan
bash -nsyntax check on all new/modified shell scriptsrun_benchmark_serving/run_lm_eval— none pass--base-urltodayconfigs/runners.yamlagainstRunnerConfig(validation.py) and all touched workflow filesdsv2lite_fp8_sglang-remote-bench.shend-to-end via SSH onbench-client_01against a live SGLang endpoint (sglang-vanilla, DeepSeek-Coder-V2-Lite-Instruct-FP8). First attempt hit a reproducible silent 100%-GPU hang mid-run (root-caused to aiperf replaying trace turns up to 255,999 tokens against a model deployed with only a 131,072-token context, relying on server-side--allow-auto-truncate). Added theREMOTE_MAX_CONTEXT_LENGTHrequirement in response; rerun completed a full successful profiling pass end-to-end (real throughput/latency metrics, CSV/JSON exports, server-metrics capture,replay_rc=0).remote-bench.ymlworkflow_dispatch once merged (workflow_dispatch requires the workflow to exist on the base branch first)Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com