Skip to content

chore(agentx): clean up outdated non-MTP recipes - #2531

Merged
cquil11 merged 1 commit into
mainfrom
agent/cleanup-dsv4-non-mtp-agentx
Aug 7, 2026
Merged

chore(agentx): clean up outdated non-MTP recipes#2531
cquil11 merged 1 commit into
mainfrom
agent/cleanup-dsv4-non-mtp-agentx

Conversation

@cquil11

@cquil11 cquil11 commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Remove seven DeepSeek-V4 non-MTP AgentX master-config entries and the seven benchmark scripts actually executed by the eight source attempts purged from the app.
  • Remove the GLM-5.2 B300 non-MTP AgentX master-config entry and its SGLang script, shared by the two B300 source runs purged from the app.
  • Refresh comments only inside the surviving MTP AgentX recipes so they do not point at deleted sibling scripts.
  • Preserve DeepSeek-V4/GLM-5.2 MTP recipes, fixed-sequence coverage, and unrelated H200/GB200/GB300 recipes.

The GLM-5.2 MI325X and H200 Dynamo-SGLang e2e/exploration cases in app PR #694 have no corresponding live script or master-config entry on upstream main, so no additional upstream files exist to remove for those attempts.

Source-run mapping

  • DeepSeek-V4 attempts 28911223583/3, 28955639528/3, 29376853679/1, 29413860950/3, 29445892486/2, 29486959583/2, 29706766201/5, and 29706772949/3 mapped to the seven removed DSV4 scripts.
  • GLM-5.2 B300 attempts 29651235293/1 and 29682242847/1 both mapped to benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh.
  • The mapping was taken from each GitHub Actions attempt's executed BENCH_SCRIPT, BENCHMARK_SCRIPT, or explicit benchmark command rather than inferred from its run title.

Companion InferenceX-app purge PRs

Why / impact

Reason: Non MTP which we aren't focusing on for AgentX as well as outdated AgentX harness.

The companion app PRs purge the existing database attempts and prevent re-ingest. This upstream cleanup prevents future sweeps from regenerating the same obsolete source configurations.

Verification

  • python -m pytest utils/matrix_logic/ -q — 224 passed.
  • Generated the remaining DSV4 and GLM-5.2 AgentX matrix from both active master YAML files successfully.
  • bash -n passed for every modified surviving shell script.
  • Confirmed the eight removed master keys are absent repo-wide and no executable configuration dispatches a deleted AgentX script.
  • git diff --check passed.

@cquil11
cquil11 marked this pull request as ready for review August 7, 2026 21:43
@cquil11
cquil11 requested a review from a team August 7, 2026 21:43
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@claude

claude Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Claude finished @cquil11's task in 4m 2s —— View job


Review of PR #2531

  • Gather context and inspect diff vs origin/main
  • Verify no dangling references to deleted scripts/config keys
  • Validate master-config edits (image accessibility, model prefixes)
  • Check perf-changelog.yaml requirement
  • Review surviving script/launcher comment edits
  • Post review feedback

Summary

One blocking issue, posted inline: both master configs were modified (8 config keys removed) but perf-changelog.yaml was not updated — repo policy requires a changelog entry appended to the end of the file whenever configs/amd-master.yaml or configs/nvidia-master.yaml change.

Everything else checks out:

  • No dangling references. All remaining mentions of the deleted config keys are historical perf-changelog.yaml entries (expected to stay). References like dsv4_fp4_b300_sglang.sh and dsv4_fp4_mi355x_vllm.sh in surviving files point at the still-existing fixed_seq_len/ siblings, not the deleted agentic/ scripts.
  • Config deletions are clean. Pure removals of the 7 DSV4 non-MTP entries and glm5.2-fp4-b300-sglang-agentic; no new images or keys introduced. The orphan-comment risk around the deleted dsv4-fp4-b300-sglang-agentic-hicache entry resolves correctly — the "GB200 DeepSeek-V4 disaggregated AgentX frontier" comment block now sits directly above the GB200 entry it describes.
  • Surviving script and launcher edits are comment-only (no functional changes), and the refreshed comments accurately stop referencing deleted scripts. The updated launcher examples (dsv4_fp4_b200_vllm_mtp.sh, qwen3.5_fp4_b300_sglang_mtp.sh) name scripts that exist.
  • The surviving agentic MTP scripts use the AIPerf trace-replay rig (run_agentic_replay_and_write_outputs), so the benchmark_client --use-chat-template requirement doesn't apply, and they were untouched functionally anyway.

Note: I could not run python -m pytest utils/matrix_logic/ -q to independently verify the author's claim of 224 passed — the command requires approval not granted to this workflow (--allowedTools would need to permit it). Structural review of the YAML deletions showed no issues.

@cquil11 cquil11 changed the title chore(agentx): clean up outdated non-MTP recipes / 清理过时的非 MTP 配方 chore(agentx): clean up outdated non-MTP recipes Aug 7, 2026
@cquil11
cquil11 force-pushed the agent/cleanup-dsv4-non-mtp-agentx branch from 1b3b4c1 to 5ec74f9 Compare August 7, 2026 21:46
Comment thread configs/nvidia-master.yaml
Delete only the DeepSeek-V4 and GLM-5.2 AgentX configs and scripts tied to purged source runs. Preserve MTP, fixed-sequence, and unrelated platform coverage. [skip-sweep]
@cquil11
cquil11 force-pushed the agent/cleanup-dsv4-non-mtp-agentx branch from 5ec74f9 to 26b5c87 Compare August 7, 2026 21:48
@cquil11
cquil11 merged commit de493d8 into main Aug 7, 2026
4 checks passed
@cquil11
cquil11 deleted the agent/cleanup-dsv4-non-mtp-agentx branch August 7, 2026 21:50
Comment on lines 945 to 950
- { tp: 8, ep: 8, conc-start: 128, conc-end: 256 }
- { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 1024 }

dsv4-fp4-b200-vllm-agentic:
image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:b200-dgxc
precision: fp4
framework: vllm
multinode: false
scenarios:
agentic-coding:
- dram-utilization: 0.80
search-space:
# Pure TP at low concurrency.
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 6, 8] }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 12, 16] }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [8, 12, 16] }
# DEP
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }

dsv4-fp4-b200-vllm-agentic-mtp:
image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec
model: deepseek-ai/DeepSeek-V4-Pro

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MODELS.md still says the DeepSeek-V4-Pro and GLM-5.2 agentic non-MTP retirement is 'not yet enacted' and that 'the non-MTP arm still runs' (lines 35, 142, 144), but this PR is exactly that enactment — it deletes the non-MTP scripts/configs. Please update the Deprecation Notice status and the two model-matrix rows to reflect that the retirement is now enacted, and consider archiving the removed scripts/configs under deprecated/ per the #2493/#2527 convention this doc credits, instead of deleting them outright.

Extended reasoning...

MODELS.md's Deprecation Notice explicitly tracks the DeepSeek-V4-Pro and GLM-5.2 agentic non-MTP retirements as a pending, not-yet-enacted item:

  • Line 15: "The speculative-decoding A/B retirements in the second table are not yet enacted."
  • Line 29/32: the A/B table lists DeepSeek-V4-Pro ... Agentic coding, non-MTP and GLM-5.2 ... Agentic coding, non-MTP as the deprecated arms.
  • Line 35: "Status: not yet enacted. Every non-spec-decode agentic arm above still runs. ... would drop all AMD and all SGLang agentic coverage for DeepSeek-V4-Pro and Qwen3.5 ... This round runs once the replacement arms exist."
  • Lines 142/144: the model support matrix rows for GLM-5.2 and DeepSeek-V4-Pro both say "Agentic coding (MTP-only pending — the non-MTP arm still runs; see the Deprecation Notice)".

This PR is precisely the enactment the doc describes as pending. It deletes dsv4_fp4_{b200,b300}_{sglang,vllm}.sh, dsv4_fp4_mi355x_{sglang,vllm}.sh, dsv4_fp4_mi355x_sglang-disagg.sh, and glm5.2_fp4_b300_sglang.sh, and removes the corresponding master-config keys from both configs/amd-master.yaml and configs/nvidia-master.yaml (dsv4-fp4-b200-vllm-agentic, dsv4-fp4-b300-vllm-agentic, dsv4-fp4-mi355x-vllm-agentic, dsv4-fp4-mi355x-sglang-agentic-hicache, dsv4-fp4-mi355x-sglang-disagg-agentic-hicache, dsv4-fp4-b200-sglang-agentic-hicache, dsv4-fp4-b300-sglang-agentic-hicache, glm5.2-fp4-b300-sglang-agentic). Yet MODELS.md is not in the PR's 18 changed files.

The code path that triggers the contradiction is simple: after this PR merges, a reader of MODELS.md sees "not yet enacted" / "the non-MTP arm still runs" for DeepSeek-V4-Pro and GLM-5.2, while the actual master configs and scripts confirm the non-MTP arm no longer exists at all. Nothing in the existing repo tooling (schema validation, pytest utils/matrix_logic/, etc.) cross-checks MODELS.md prose against the master YAMLs, so there is no mechanism that would have caught this before merge — it is a purely textual/documentation gap that only a human diff-read (or this review) surfaces.

Impact is limited to documentation accuracy: no benchmark, CI, or runtime behavior is affected, since MODELS.md is not consumed by any script in this repo (confirmed by its role as a maintainer-facing tracking doc, per the file's own description). But it is the canonical ledger the team uses to reason about what is and isn't safe to remove next, and the precedent PRs (#2493, #2527) both updated this exact section as part of their own enactment — so a maintainer reviewing this PR would very likely expect the same here, and future contributors reading MODELS.md would be misled into thinking the non-MTP arm still exists on AMD/SGLang for these two models.

Concrete proof, step by step:

  1. Before this PR: configs/nvidia-master.yaml contains dsv4-fp4-b200-vllm-agentic (non-MTP) and dsv4-fp4-b200-vllm-agentic-mtp (MTP) side by side; MODELS.md:35 says the non-MTP arm "still runs."
  2. This PR's diff (configs/nvidia-master.yaml lines 945-950 in the pre-image) deletes the dsv4-fp4-b200-vllm-agentic block entirely, leaving only dsv4-fp4-b200-vllm-agentic-mtp.
  3. The same pattern repeats for dsv4-fp4-b300-vllm-agentic, the three dsv4-fp4-mi355x-*-agentic* AMD entries, the two dsv4-fp4-*-sglang-agentic-hicache NVIDIA entries, and glm5.2-fp4-b300-sglang-agentic.
  4. After merge, grepping the repo for dsv4-fp4-b200-vllm-agentic: (non-MTP) returns nothing, while MODELS.md:35 and :144 still assert that arm "still runs."
  5. This is a direct, verifiable factual contradiction between the doc and the repo state it purports to describe.

Fix: update the Deprecation Notice's status line (currently "not yet enacted") to reflect that the DeepSeek-V4-Pro and GLM-5.2 rows of the A/B table are now enacted (Qwen3.5, MiniMax-M3, and Kimi-K3 rows presumably remain pending), and update lines 142/144 to drop the "non-MTP arm still runs" caveat for these two models. Following the archival convention from #2493/#2527 — moving the deleted scripts/configs into sibling deprecated/ directories and configs/deprecated/*.yaml instead of deleting them outright — would also preserve the audit trail those precedent PRs established, though that's a secondary nice-to-have relative to the doc-accuracy fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant