[Power] feat: extend dcgm energy lanes to gb dsv4/qwen3.5 fp4 / 扩展 dcgm 能耗采集到 gb dsv4 与 qwen3.5 fp4 - #2507
[Power] feat: extend dcgm energy lanes to gb dsv4/qwen3.5 fp4 / 扩展 dcgm 能耗采集到 gb dsv4 与 qwen3.5 fp4#2507edwingao28 wants to merge 4 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
…gm 能耗采集到 gb dsv4 与 qwen3.5 fp4 recipe 侧:5 个 DSV4 8k1k 点位与 qwen3.5-fp4 gb300 lowlat 配方加 telemetry 块(gb300 用 19401 端口)。launcher 侧:gb200 dsv4 分支在 power 时改克隆 pinned producer;gb300 新增仅 power 生效的 dsv4 路由(非 power 路径保持原样);precision 门换成 framework 门(dynamo-sglang 之外仍然快速失败)。契约测试同步:recipe 白名单 8 项 + 逐配方 telemetry 断言。
8e2ca1c to
3ad142d
Compare
|
Claude finished @edwingao28's task in 4m 58s —— View job Review of PR #2507
Found 1 blocking issue (inline comment on Everything else checked out clean:
Note: this sandbox's permissions blocked running |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3ad142d752
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # dynamo-sglang lanes run on (fp8 validated end-to-end, fp4 recipes | ||
| # parse-verified against the pin); other frameworks clone diverging refs | ||
| # (aflowers branch, sa-submission), so fail fast for them instead. | ||
| if [[ "$USES_DCGM_POWER" == "1" && "$FRAMEWORK" != "dynamo-sglang" ]]; then |
There was a problem hiding this comment.
When the new GB300 DSV4 or Qwen3.5 FP4 recipes enable telemetry, this relaxed gate routes them to the pinned producer, but both models resolve to compute-node-only /scratch/models paths (launch_gb300-nv.sh:44-54,83-87) while the submission block still adds --no-preflight only for agentic, GLM5.1, or Qwen3.5 FP8 runs (launch_gb300-nv.sh:420-437). The launcher explicitly documents that this preflight stats the model path from the login pod, so these new FP4 power jobs abort before sbatch; extend the skip condition to these power cases or retain the FP8 gate. 中文：新增的 GB300 DSV4 和 Qwen3.5 FP4 配方启用 telemetry 后会切换到带模型路径预检的 producer pin，但模型只存在于计算节点的 /scratch/models，当前 --no-preflight 条件又未覆盖这些 FP4 通道，因此任务会在提交 Slurm 作业前失败；应为这些能耗通道跳过预检，或暂时保留 FP8 精度限制。
AGENTS.md reference: AGENTS.md:L152-L152
Useful? React with 👍 / 👎.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| elif [[ $FRAMEWORK == "dynamo-sglang" && $MODEL_PREFIX == "dsv4" && "$USES_DCGM_POWER" == "1" ]]; then | ||
| # Note (wenyao): only the power lane routes here — non-power dsv4 sglang | ||
| # keeps the generic fallthrough untouched. The overlay is required: the | ||
| # telemetry-tagged DSV4 recipes are version-controlled in-repo only, | ||
| # neither upstream ref ships them. | ||
| git clone "$POWER_SRT_SLURM_URL" "$SRT_REPO_DIR" | ||
| cd "$SRT_REPO_DIR" | ||
| git checkout "$POWER_SRT_SLURM_PIN" || exit 1 | ||
| # The power lane must run the exact pinned producer SHA, never a moving branch. | ||
| test "$(git rev-parse HEAD)" = "$POWER_SRT_SLURM_PIN" || { echo "Error: srt-slurm HEAD does not match POWER_SRT_SLURM_PIN=$POWER_SRT_SLURM_PIN" >&2; exit 1; } | ||
| git rev-parse HEAD > "$GITHUB_WORKSPACE/power-producer-sha.txt" | ||
| mkdir -p recipes/sglang/deepseek-v4 | ||
| cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4" recipes/sglang/deepseek-v4 |
There was a problem hiding this comment.
🔴 BLOCKING: The two new gb300 power lanes (dsv4-fp4 and qwen3.5-fp4) route onto the pinned producer without also getting --no-preflight, so srtctl apply will fail before sbatch is ever called.
Why it matters: The pin follows the srt-slurm main lineage, which carries srtctl's pre-submit model-path stat (this launcher's own comment at the --no-preflight block says v1.0.25 "has the preflight"; sa-submission-q2-2026, where non-power fp4 lanes ran until now, "has none"). Both new lanes resolve model.path to compute-node-only NVMe that the GHA runner pod can't stat:
- dsv4 fp4 →
/scratch/models/DeepSeek-V4-Pro(lines 44–55 warn preflight "may fail" for exactly this path) - qwen3.5 fp4 →
/scratch/models/Qwen3.5-397B-A17B-NVFP4(the agentic branch comment at lines 188–198 states this path makes "the Path.is_dir() preflight ... fail before sbatch is ever called")
The --no-preflight condition at line ~436 only covers IS_AGENTIC, glm5.1, and qwen3.5 && fp8 — the fp8 power lane is protected, the two new fp4 power lanes are not.
Fix: gate --no-preflight on the power lane too (every gb300 power recipe resolves weights on compute-node /scratch; qwen3.5-fp8 already takes the flag, so this only adds the two lanes that need it):
if [[ "$IS_AGENTIC" == "1" || "$MODEL_PREFIX" == "glm5.1" || ( "$MODEL_PREFIX" == "qwen3.5" && "$PRECISION" == "fp8" ) || "$USES_DCGM_POWER" == "1" ]]; then
SRTCTL_APPLY_ARGS+=(--no-preflight)
fi(Note: I couldn't clone the pin from this sandbox to confirm the preflight exists at 6fc1bed — if your gb300 canary run passes preflight, this is moot; but the PR notes canaries haven't run yet, and the in-repo evidence above points at a submit-time failure.)
|
|
||
| # Note (wenyao): the producer pin descends from the fp8 srt-slurm lineage | ||
| # (cargo/maturin bootstrap); a non-fp8 power recipe would silently clone the | ||
| # wrong lineage, so fail fast instead. | ||
| if [[ "$USES_DCGM_POWER" == "1" && "$PRECISION" != "fp8" ]]; then | ||
| echo "Error: dcgm-power lanes are only validated for PRECISION=fp8, got: $PRECISION" >&2 | ||
| # Note (wenyao): the producer pin follows the srt-slurm main lineage that the | ||
| # dynamo-sglang lanes run on (fp8 validated end-to-end, fp4 recipes | ||
| # parse-verified against the pin); other frameworks clone diverging refs | ||
| # (aflowers branch, sa-submission), so fail fast for them instead. | ||
| if [[ "$USES_DCGM_POWER" == "1" && "$FRAMEWORK" != "dynamo-sglang" ]]; then | ||
| echo "Error: dcgm-power lanes are only validated for FRAMEWORK=dynamo-sglang, got: $FRAMEWORK" >&2 |
There was a problem hiding this comment.
🟡 AGENTS.md:152 still says dcgm-power lanes are validated for PRECISION=fp8 only, but this PR replaces that fp8 fail-fast in both launch_gb200-nv.sh and launch_gb300-nv.sh with a FRAMEWORK=dynamo-sglang gate and ships six new fp4 dcgm-power recipes (5 DSV4 fp4 + 1 qwen3.5 fp4). That doc sentence is now factually false and should be updated in this PR to describe the new framework-based gate.
Extended reasoning...
AGENTS.md:152 is the canonical guidance for the exact area this PR touches: "Power lanes: a recipe telemetry: block with provider: dcgm-power enables official energy collection for that config. ... dcgm-power lanes are validated for PRECISION=fp8 only." That last sentence describes the fail-fast that used to gate the dcgm-power lane.
This PR removes exactly that fail-fast. In both runners/launch_gb200-nv.sh (lines 308-314) and runners/launch_gb300-nv.sh, the check changes from:
if [[ "$USES_DCGM_POWER" == "1" && "$PRECISION" != "fp8" ]]; thento:
if [[ "$USES_DCGM_POWER" == "1" && "$FRAMEWORK" != "dynamo-sglang" ]]; thenand the PR adds six new fp4 recipes with an enabled telemetry: {provider: dcgm-power} block: five DSV4 8k1k fp4 recipes (gb200 c1/c512/c256, gb300 c1/c1024) and the qwen3.5-fp4 gb300 low-latency ladder recipe. utils/test_gb200_power_official_contract.py's POWER_RECIPES dict confirms this — it now lists these fp4 recipes alongside the original fp8 ones, and test_exactly_the_declared_recipes_opt_into_dcgm_power / test_every_power_recipe_declares_the_same_telemetry_contract assert they all carry the same enabled telemetry contract.
So after this PR merges, AGENTS.md:152's claim "dcgm-power lanes are validated for PRECISION=fp8 only" is simply wrong — fp4 dcgm-power lanes exist and are validated by the same contract tests. A contributor or coding agent who reads AGENTS.md before touching a power lane (which is exactly the workflow this doc section is written for — it's the "single source of truth" pointer for the producer pin and contract tests) would be misled into thinking fp4 power recipes are unsupported or need special justification, when the code and tests already treat them as first-class.
Step-by-step proof of staleness:
- Before this PR:
launch_gb200-nv.shgates power lanes with"$PRECISION" != "fp8"— matches AGENTS.md:152 exactly. - This PR's diff on
runners/launch_gb200-nv.sh(and the identical block inlaunch_gb300-nv.sh) replaces that condition with"$FRAMEWORK" != "dynamo-sglang"— precision is no longer part of the gate at all. - This PR adds
telemetry: {enabled: true, provider: dcgm-power, ...}to 6 new recipes, all of which declareprecision: "fp4"in theirmodel:block (e.g.disagg-gb200-1p1d-dep8-dep16-6-c512.yaml,8k1k_stp_lowlat_0.yaml). utils/test_gb200_power_official_contract.py::POWER_RECIPESnow enumerates these fp4 recipes andtest_every_power_recipe_declares_the_same_telemetry_contractpasses for them — i.e., CI itself now certifies fp4 dcgm-power lanes as valid.AGENTS.md:152was never touched by this diff and still reads "validated forPRECISION=fp8only," which directly contradicts steps 2-4.
The fix is small: update the sentence to describe the new framework gate, e.g. "dcgm-power lanes are validated for FRAMEWORK=dynamo-sglang only." This is a documentation-only fix with no runtime impact — nothing breaks if it's merged as-is, but the doc should be corrected in the same PR since it directly describes the mechanism this PR changes, and letting it drift makes AGENTS.md a less reliable source of truth for the next contributor or agent working in this exact area.
… lanes / 为 node-local fp4 能耗 lane 加 no-preflight 与 numa1 模型路径 两个 canary 都死在同一处:模型在 compute-node 本地盘(gb200 /mnt/numa1、gb300 /scratch),login 侧 runner stat 不到,srtctl preflight 必然误报。沿用 repo 已有豁免机制(agentic/glm5.1/qwen3.5-fp8 同款),范围限定 USES_DCGM_POWER=1;gb200 power lane 的 MODEL_PATH 指向 agentic 与 llm-d 实测在用的 /mnt/numa1 检查点,共享的 lustre 路径保持不动。
…dia-master 中已失效的 gb300 dsv4 镜像 pin launcher 的 import_squash 用的是 master yaml 的 image 字段,0520 nightly 已被 Docker Hub GC;与配方一致换成仍在线的 0721(B300 DSV4 家族同款)。仅动 dsv4-fp4-gb300-dynamo-sglang 一个 key,mtp key 不碰。
1 participant
Extends the dcgm-power measurement lanes (built in #2323/#2437/#2456) to the fp4 recipes the scale campaign needs: five DSV4 8k1k Pareto points (gb200 c1/c512/c256, gb300 c1/c1024) and the qwen3.5-fp4 gb300 low-latency ladder. Recipes opt in through the same telemetry block the fp8 lanes use; gb300 recipes take port 19401 for the known cluster-exporter collision.
Launcher changes, kept surgical:
dynamo-sglang && dsv4branch clones the pinned producer when the resolved recipe carries an enabled dcgm-power block, and keeps its currentNVIDIA/srt-slurm:mainclone otherwise. The recipe overlay is unchanged.dynamo-sglangonly; any other framework still exits early instead of silently cloning a diverging ref (aflowers branch / sa-submission).Safety checks done before opening:
6fc1bed) config loader — none of them use sa-submission-only schema fields (warmup_req_rate, spread placement, request-plane options).sa_bench_tokenizers.sglang_deepseek_v4(used by the gb300 DSV4 recipes) is present at the pin.mainlineage the pin descends from; the qwen3.5 fp4 recipe pins dynamo by version (pip path), which the pin supports.bash -n.Scope: the config keys already exist; dispatches for validation use exact-key test-config. One master-yaml line changes: the
dsv4-fp4-gb300-dynamo-sglangimage pin, whose nightly (20260520) has been garbage-collected from Docker Hub — bumped to the live20260721nightly the B300 DSV4 family already runs (recipe container fields bumped to match). Heads-up for maintainers: several other multinode pins (gb200DSV420260528, gb300 qwen3.5-fp420260624) are also gone from the Hub and currently survive only on per-cluster squash caches.Canary validation (both from this branch,
require-power=true):20260721image (fresh arm64 pull, no cache).The first canary attempt failed at srtctl preflight ("model alias resolved to a path unavailable") and produced the second commit: these fp4 models live on compute-node-local NVMe (gb200
/mnt/numa1, gb300/scratch), which the login-side runner cannot stat — the same situation the existing agentic/glm5.1/qwen3.5-fp8--no-preflightexemptions cover. The power lanes join that list, scoped toUSES_DCGM_POWER=1; the gb200 power lane also pointsMODEL_PATHat the/mnt/numa1checkpoint the agentic path already loads from (the shared lustre alias stays untouched for the external-cluster staging).中文:把能耗采集扩展到 scale campaign 需要的 fp4 配方(DSV4 五个点位 + qwen3.5-fp4 gb300 低延迟梯子)。launcher 改动保持外科式:gb200 dsv4 分支仅在 power 时改克隆 pinned producer;gb300 新增仅 power 生效的 dsv4 路由,非 power 路径逐字节不变;fp8 精度门换成 framework 门。六个配方已在 pin 的 loader 下解析通过,契约测试 16/16 绿。canary 链接补充后转正式 review。