You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The harness checks the on-disk path, which no longer exists:
scripts/dgx-online-serving.sh:293 [[ -x ${build_dir}/examples/server ]] || {
scripts/dgx-online-serving.sh:294 echo "provenance-recorded build did not produce examples/server"
scripts/dgx-online-serving.sh:382,424,693 same path
tools/bench/online_gate.py:3596 "server": build_dir / "examples" / "server"
tools/bench/online_gate.py:3756 server_path = build_dir / "examples" / "server"
Verified on a clean sm_121a Release build of bc6e3d72: examples/ contains vllm-server, vllm-bench, vllm-cli, and no server.
Introduced by 1a02ab4f feat(release): package canonical server (W6), which set
the release output name without updating the harness that consumes it.
Why this matters beyond a broken path
--execute is how a 27B/35B online-serving ratio becomes an accepted
measurement. While it aborts, throughput numbers for the gate models can only
come from ad-hoc harnesses that do not enforce the checks this one does — the
pinned revision, the oracle version and dependency inventory, cache-drop proof,
one-model/one-lock, and the recorded run recipe. That is the difference between
a number and evidence.
Fix
Point the harness at the built artifact and add a guard so the two cannot drift
again: assert the path the harness expects equals the OUTPUT_NAME declared in examples/CMakeLists.txt, rather than hardcoding either spelling in two places.
A red test must fail on today's tree before the fix.
tools/bench/online_gate.py has model keys 27 -> unsloth/Qwen3.6-27B-NVFP4
and 35 -> nvidia/Qwen3.6-35B-A3B-NVFP4. There is no key for nvidia/Qwen3.6-27B-NVFP4, which is the 27B the campaign targets, so that
model has no reproducible canonical recipe at all. Worth adding in the same
area of the code.
The canonical online-serving gate cannot run against current
main. It abortsbefore starting a server.
examples/CMakeLists.txtdeclares the CMake targetserverbut sets itsoutput name to
vllm-server:The harness checks the on-disk path, which no longer exists:
Verified on a clean
sm_121aRelease build ofbc6e3d72:examples/containsvllm-server,vllm-bench,vllm-cli, and noserver.Introduced by
1a02ab4f feat(release): package canonical server (W6), which setthe release output name without updating the harness that consumes it.
Why this matters beyond a broken path
--executeis how a 27B/35B online-serving ratio becomes an acceptedmeasurement. While it aborts, throughput numbers for the gate models can only
come from ad-hoc harnesses that do not enforce the checks this one does — the
pinned revision, the oracle version and dependency inventory, cache-drop proof,
one-model/one-lock, and the recorded run recipe. That is the difference between
a number and evidence.
Fix
Point the harness at the built artifact and add a guard so the two cannot drift
again: assert the path the harness expects equals the
OUTPUT_NAMEdeclared inexamples/CMakeLists.txt, rather than hardcoding either spelling in two places.A red test must fail on today's tree before the fix.
Related
tools/bench/online_gate.pyhas model keys27->unsloth/Qwen3.6-27B-NVFP4and
35->nvidia/Qwen3.6-35B-A3B-NVFP4. There is no key fornvidia/Qwen3.6-27B-NVFP4, which is the 27B the campaign targets, so thatmodel has no reproducible canonical recipe at all. Worth adding in the same
area of the code.