[AMD] [AGENTX] GLM-5.2: EP=1 interactivity + c12 throughput tuning on MI355X - #2777
[AMD] [AGENTX] GLM-5.2: EP=1 interactivity + c12 throughput tuning on MI355X#2777giovanniguastiamd wants to merge 4 commits into
Conversation
…e tuning - Switch TP8 arm from EP=8 to EP=1: eliminates MoE all-to-all, ITL p50 -5% at c4 (6.95→7.3 ms), P90 interactivity +5% (110.5 vs 105 tok/s/user). Extend TP8 sweep to conc [4, 6, 8, 10]. - Raise HiCache ratio 1.5→2.5, write policy write_through→write_through_selective: sustains throughput through c12 (+12% tok/s/GPU vs c10 baseline). Trim TP4 sweep to [1, 2, 4, 8, 10, 12]. - Double MAX_RUNNING_REQUESTS to 2×CONC for MTP draft+verify headroom; cap CUDA_GRAPH_MAX_BS at 64 (SGLang auto-interpolates the bs list). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
seungrokj
added
AMD
full-sweep-fail-fast
agentx
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # Per-arm L2 ratio (sizing rationale below) applies to both backends unless | ||
| # overridden via HICACHE_RATIO. TP arm (182.7 GB/rank device pool): the | ||
| # working set oversubscribes the device pool ~3x at conc 32, so the host | ||
| # tier is what carries the radix hits - ratio 1.5 (~2.9 TB pinned incl. | ||
| # sidecars) validates through the conc-24 long-context storm for the | ||
| # mooncake arm. The DP-attention arm (159.4 GB/rank) only runs at conc >= | ||
| # 32, where each DP rank's ~8 sessions nearly fit in its own device pool | ||
| # (~1.5-1.6M of 1.7M tokens at conc 64) and the host tier just absorbs | ||
| # overflow - ratio 1.5 boots but the host OOM killer takes the server | ||
| # mid-storm at conc 48, so it runs ratio 0.5 (~1.2 TB pinned, ~1.8 TB of | ||
| # load headroom) at negligible hit-rate cost. The hicache-only arm has no | ||
| # L3 to fall back on, so these ratios are unvalidated there - override with | ||
| # HICACHE_RATIO if the host OOMs or hit-rate is poor. | ||
| # agentic-coding corpus saturates any fixed DRAM pool at conc ≥ 10; ratio 2.5 | ||
| # (~4.8 TB pinned) sustains throughput through conc 12 and leaves headroom for | ||
| # the mooncake arm's conc-24 long-context storm. The DP-attention arm | ||
| # (159.4 GB/rank) only runs at conc >= 32, where the host tier just absorbs | ||
| # overflow - ratio 0.5 (~1.2 TB pinned, ~1.8 TB of load headroom) at negligible | ||
| # hit-rate cost (ratio 1.5 OOMs the host mid-storm at conc 48). | ||
| CACHE_ARGS=() | ||
| if agentic_kv_offload_enabled; then | ||
| if [ "$DP_ATTENTION" = "true" ]; then | ||
| HICACHE_RATIO="${HICACHE_RATIO:-0.5}" | ||
| else | ||
| HICACHE_RATIO="${HICACHE_RATIO:-1.5}" | ||
| # ratio=2.5 (vs the former 1.5): the agentic-coding corpus fills any | ||
| # fixed DRAM pool at conc ≥ 10; a larger host tier delays saturation | ||
| # and keeps throughput stable through conc 12. | ||
| HICACHE_RATIO="${HICACHE_RATIO:-2.5}" |
There was a problem hiding this comment.
🔴 HICACHE_RATIO default raised from 1.5 to 2.5 for the TP4/EP4 arm, and the script's own sizing comment puts the resulting pinned host memory at ~4.8 TB. configs/runners.yaml (lines 281-283) lists cluster:mi355x-amds available-cpu-dram-mib as 3,095,781 (~3.0-3.25 TB total node DRAM), so the new default requests more pinned host memory than physically exists on the node, before even accounting for the 0.8 dram-utilization headroom or other processes. HICACHE_RATIO is a bare env-var default (line 89) with no runtime check against TOTAL_CPU_DRAM_GB (validated only for positivity in benchmark_lib.sh, never used to size the ratio) or against the node's hardware capacity in runners.yaml, so nothing stops this mismatch from reaching the server launch.
Extended reasoning...
Running the TP4/EP4 arm at conc>=8-12 (its own comment says the corpus saturates the pool at conc>=10) makes HiCache try to pin close to the ~4.8 TB implied by ratio=2.5, exceeding the node's ~3-3.25 TB of DRAM; the host OOM killer kills the sglang server mid-run, the same failure mode the file's own comments describe happening to the DP-attention arm at ratio 1.5. At the prior ratio (1.5, ~2.9 TB per the old comment) the request fit under the node's real capacity; the new default does not. A correct fix would derive/clamp the pinned-memory target from TOTAL_CPU_DRAM_GB (already computed per-node from dram-utilization and GPU-count fraction) rather than hardcoding a ratio disconnected from actual host capacity.
Verification: normal. benchmarks/single_node/agentic/glm5.2_fp4_mi355x_sglang_mtp.sh:89 raises the TP4/EP4 arm default HICACHE_RATIO from 1.5 to 2.5, passed unclamped to --hicache-ratio (line 101). The merged code's own sizing comment states ratio 2.5 pins ~4.8 TB of host memory (lines 75-76: "ratio 2.5 (~4.8 TB pinned)"). configs/runners.yaml:281-282 lists cluster:mi355x-amds available-cpu-dram-mib:…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33157243353 |
Sweep validation shows TP8/EP=1 at c6 (P90=91.3) and c8 (P90=72.7) are dominated by the TP4/EP4/HiCache arm at those concurrency points. Keeping only c4 (best interactivity anchor) and c10 (crossover bound). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
giovanniguastiamd
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33182007357 |
…ew bullet, fix pr-link 2769→2777 The append-only validator rejects modifications to existing description bullets. Restore the first bullet to its original wording (with [4,6,8,10] sweep text) and append a new fourth bullet describing the c6/c8 trim. Also update pr-link to 2777. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33182931834 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33185778937 |
2 participants
Summary / 摘要
Tuning follow-up to #2570 (GLM-5.2 FP4 MI355X SGLang Agentic MTP).
Changes / 变更内容
1. TP8 arm: EP=8 → EP=1 (script + yaml)
GLM-5.2 is a MoE model. With EP=8 every forward pass introduces an all-to-all collective between 8 EP ranks to route tokens to the correct experts. At low concurrency (c4, effective batch ≈ 1–2 requests) this collective is overhead on essentially unshared traffic.
Switching to EP=1 (no expert parallelism, all experts replicated across TP=8 ranks) eliminates the all-to-all entirely:
The TP8 concurrency sweep is extended to [4, 6, 8, 10] to cover the mid-range crossover region (previously only [1, 2, 4]).
2. HiCache tuning: ratio 1.5 → 2.5, write_through_selective (script)
The agentic-coding corpus (ISL p50 ≈ 90k tokens) saturates any fixed DRAM KV pool at conc ≥ 10. Raising the host-DRAM ratio to 2.5× delays saturation and keeps throughput stable through conc 12:
write_through_selective(introduced in #2679) skips DRAM writes for non-reusable KV blocks, reducing host-bus traffic without impacting cache hit rate.The TP4 concurrency list is trimmed to [1, 2, 4, 8, 10, 12]; conc 16 is removed (untested and beyond the validated range).
3. MAX_RUNNING_REQUESTS 1×CONC → 2×CONC; CUDA_GRAPH_MAX_BS capped at 64 (script)
With MTP num-steps=5 the draft+verify phase transiently batches more tokens than CONC sessions. Doubling the in-flight slot budget prevents scheduler stalls under burst. SGLang auto-interpolates a batch-size list [1..max_bs], so capping
cuda-graph-max-bsat 64 keeps graph-capture memory bounded without losing coverage.摘要
对 #2570（GLM-5.2 FP4 MI355X SGLang Agentic MTP）的调优跟进。
1. TP8 臂：EP=8 → EP=1：GLM-5.2 是 MoE 模型，EP=8 在每次前向计算时引入 all-to-all 集合通信。低并发（c4）下该通信几乎无法分摊，EP=1 消除了这一开销：c4 ITL p50 7.3→6.95 ms（−5%），P90 交互性 105→110.5 tok/s/user（+5%）。TP8 并发扫描扩展至 [4, 6, 8, 10]。
2. HiCache 调优：ratio 1.5→2.5，write policy 改为 write_through_selective。agentic-coding 语料在 conc ≥ 10 时会填满任何固定 DRAM 池；更大的 host 端为 c12 提供吞吐支撑（+12% tok/s/GPU）。TP4 并发列表缩减为 [1, 2, 4, 8, 10, 12]。
3. MAX_RUNNING_REQUESTS 1×→2×CONC：MTP draft+verify 阶段瞬时 batch 超过 CONC，加大 in-flight slot 可防止调度器阻塞。CUDA_GRAPH_MAX_BS 上限设为 64（SGLang 已自动插值 bs 列表）。
Test plan / 测试计划
Note
Low Risk
Benchmark recipe and cluster YAML only; no production serving paths or security-sensitive logic.
Overview
Follow-up tuning for glm5.2-fp4-mi355x-sglang-agentic-mtp on MI355X: launch script defaults and
amd-master.yamlsearch space align with sweep validation on agentic-coding.The TP4 + HiCache arm raises default HiCache ratio from 1.5 to 2.5 and switches write policy to
write_through_selectiveso long-context load does not saturate host DRAM through conc 12; concurrency list drops untested 16 and keeps [1, 2, 4, 8, 10, 12].The TP8 comparison arm changes from EP=8 to EP=1 (no MoE all-to-all at low batch) and narrows concurrency to [4, 10] after mid-range points proved dominated by TP4/EP4.
glm5.2_fp4_mi355x_sglang_mtp.shdoublesMAX_RUNNING_REQUESTSto 2×CONC for MTP draft+verify burst headroom and capsCUDA_GRAPH_MAX_BSat 64 while SGLang interpolates batch sizes up to that cap.perf-changelog.yamlrecords the same rationale and links PR #2777.Reviewed by Cursor Bugbot for commit 7e622dd. Bugbot is set up for automated code reviews on this repo. Configure here.