[AMD] [WIP] [AGENTX] GLM-5.2 MTP5 tune: raise acc-len and drop hicache at low conc / GLM-5.2 MTP5 调优：提升接受率目标并在低并发下关闭 hicache

[giovanniguastiamd]

Summary / 摘要

Tuning follow-up to #2488 (GLM-5.2 FP4 MI355X SGLang Agentic MTP).

Changes / 变更内容

1. MTP depth increase (script)

• --speculative-num-steps: 3 → 5

• --speculative-num-draft-tokens: 4 → 6

• SGLANG_SIMULATE_ACC_LEN: 2.99 → 3.61

  Higher measured acceptance rate on the agentic-coding corpus justifies deeper drafting; expected throughput gain ~15%.

2. TP8 low-concurrency arm: remove hicache (yaml)

• kv-offloading: dram, kv-offload-backend: {name: hicache} → kv-offloading: none for the TP8/EP8 conc [1,2,4] arm.

  At conc ≤ 4 the model weights + KV for active sessions fit within HBM without offloading; bypassing HiCache eliminates the L2 DRAM tier overhead and reduces TTFT/TPOT. The TP4 arm (with HiCache) still covers conc [1,2,4], so the Pareto front picks the better arm per concurrency.

---

摘要

对 #2488（GLM-5.2 FP4 MI355X SGLang Agentic MTP）的调优跟进。

1. 加深 MTP 草稿步数：speculative-num-steps 3→5，num-draft-tokens 4→6，SGLANG_SIMULATE_ACC_LEN 2.99→3.61。在 agentic-coding 语料上测得更高接受率，预计吞吐提升约 15%。

2. TP8 低并发臂去除 hicache：conc [1,2,4] 的 TP8/EP8 臂从 kv-offloading=dram+hicache 改为 kv-offloading=none，降低低负载下的首字延迟。TP4 臂（含 HiCache）仍覆盖相同并发点，由 SA 选取最优 Pareto 点。

Test plan / 测试计划

• Full sweep on MI355X cluster passes all concurrency points
• TP4+hicache and TP8+no-offload both produce valid results for conc [1,2,4]
• Throughput at higher concurrency (c8–c16) shows improvement vs baseline ([AMD] [AGENTX] GLM-5.2 FP4 MI355X SGLang Agentic MTP #2488)

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31574993434
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31574993434

[claude]

🟡 The appended perf-changelog.yaml entry sets pr-link to .../pull/PLACEHOLDER, which isn't a value utils/validate_perf_changelog.py recognizes (only XXX, .../pull/XXX, or the canonical .../pull/2570 link are accepted for a new PR entry). This won't fail this PR's own CI, but it will abort utils/merge_with_reuse.sh's canonicalization step at merge time. Please replace PLACEHOLDER with XXX (or 2570).

Extended reasoning...

The newly appended entry at the tail of perf-changelog.yaml (lines 5824-5827) sets:

pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER

utils/validate_perf_changelog.py defines the accepted values for a new entry's pr-link via validate_added_pr_link():

PR_LINK_PLACEHOLDERS = {
    "XXX",
    "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX",
}
...
def validate_added_pr_link(link, pr_number):
    ...
    expected = f"https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}"
    if link not in PR_LINK_PLACEHOLDERS and link != expected:
        raise ChangelogValidationError(...)

.../pull/PLACEHOLDER is neither a recognized placeholder nor the canonical .../pull/2570 link for this PR, so this check would reject it — it's clearly an author slip where a template placeholder token (PLACEHOLDER) was used instead of the repo's actual convention (XXX).

Where this actually bites, precisely: validate_added_pr_link is only reached through compare_entries(), and compare_entries() is only called from utils/prepare_perf_changelog_merge.py, not from the PR's own check-changelog CI job. That job (run-sweep.yml → python3 utils/validate_perf_changelog.py) drives main() → validate_matrix_compatible_change() → validate_generated_config() → process_changelog.py, and ChangelogEntry.pr_link in matrix_logic/validation.py is a plain, format-unvalidated str. So this PR's own CI will pass with PLACEHOLDER in place.

The real failure surfaces at merge time: utils/merge_with_reuse.sh runs prepare_perf_changelog_merge.py canonicalize --pr-number "$PR" ... (line 187), which calls compare_entries(base, head, pr_number=2570) → validate_added_pr_link("https://.../pull/PLACEHOLDER", 2570). Since PLACEHOLDER is neither in PR_LINK_PLACEHOLDERS nor equal to the expected .../pull/2570, this raises ChangelogValidationError, prepare_perf_changelog_merge.py's main() catches it and returns 1, and under merge_with_reuse.sh's set -euo pipefail the script aborts before it can canonicalize the link and proceed to squash-merge.

Step-by-step proof:

1. PR [AMD] [WIP] [AGENTX] GLM-5.2 MTP5 tune: raise acc-len and drop hicache at low conc / GLM-5.2 MTP5 调优：提升接受率目标并在低并发下关闭 hicache #2570 merges normally through GitHub's own merge button — check-changelog never calls compare_entries, so it's green despite the bad placeholder.
2. Someone (or automation) later runs utils/merge_with_reuse.sh 2570 to merge with sweep-reuse.
3. That script merges origin/main into the PR branch, then runs prepare_perf_changelog_merge.py canonicalize --pr-number 2570.
4. Inside, compare_entries(base_entries, head_entries, pr_number=2570) iterates the new addition and calls validate_added_pr_link("https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER", 2570).
5. expected = ".../pull/2570"; "PLACEHOLDER" link is not in {"XXX", ".../pull/XXX"} and doesn't equal expected, so ChangelogValidationError("new PR entry must use '.../pull/2570' or an XXX placeholder; found '.../pull/PLACEHOLDER'") is raised.
6. prepare_perf_changelog_merge.py's main() catches ChangelogValidationError and returns 1 (exit code 1).
7. merge_with_reuse.sh runs under set -euo pipefail, so the non-zero exit aborts the merge script right there — the changelog is never canonicalized and the PR is never squash-merged via this path.

Fix: replace PLACEHOLDER with XXX (matching the repo's existing convention, e.g. the entries just above it use real PR numbers already merged, and prior new entries use XXX before canonicalization) or with the literal PR number 2570.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31575062972
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31575062972

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31575566543
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31575566543