[AMD][AgentX] DSv4 MI355X agentic disagg: add high-concurrency dots using DEP8#2308
[AMD][AgentX] DSv4 MI355X agentic disagg: add high-concurrency dots using DEP8#2308ichbinblau wants to merge 5 commits into
Conversation
…fill cudagraph for DSv4 agentic disagg Co-authored-by: Cursor <cursoragent@cursor.com>
…x router policy env var/name Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
ichbinblau
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
ichbinblau
added
AMD
agentx
|
hi @ichbinblau any chance u can priority doing disagg mtp for agentic |
Hi @functionstackx, this PR fixes the SGL router settings issues for high-concurrency use cases. I'll open another PR for the MTP changes. |
…ic-hicache Resolves review comment: the image bump (v0.5.14 -> v0.5.15.post1), re-enabled TP8/EP8/DPA hicache arm, and HiCache sizing/NCCL/router fixes need a changelog entry to trigger the benchmark sweep on merge. 中文：为 dsv4-fp4-mi355x-sglang-disagg-agentic-hicache 添加 perf-changelog.yaml 条目。解决评审意见：镶像升级（v0.5.14 -> v0.5.15.post1）、重新启用的 TP8/EP8/DPA hicache 分支，以及 HiCache 容量/NCCL/路由策略修复，需要一条 changelog 条目才能在合并时触发基准测试扫描。 Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29989948268 |
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29990310046 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29992400324 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29992400324 |
2 participants
Summary
Follow-on fixes for the DSv4 FP4 MI355X SGLang PD-disaggregated agentic benchmark (
dsv4-fp4-mi355x-sglang-disagg-agentic-hicache), rebased on top of currentmain(which already absorbed the bulk of this branch's earlier history via #2170). Two logical changes:What's changed
HiCache sizing (ratio vs size)
dsv4_fp4_mi355x_sglang-disagg.sh: setHICACHE_RATIO=4andFORCE_HICACHE_RATIO=1so this recipe explicitly opts into ratio-based sizing.job.slurm: forwardFORCE_HICACHE_RATIOinto the container's HiCache env file (hicache_mc_<job>.env) alongside the otherHICACHE_*vars — it was previously silently dropped, so the override never reached the server.NCCL / decode cuda-graph crash workaround
models.yaml: disable prefill cuda-graph (disable_cuda_graph: true) forDeepSeek-V4-Pro-AgentXas a complementary mitigation, drop--enable-two-batch-overlapfromdp_flags, lower prefillmem_fraction_static0.8→0.72 and no-dpmax_running_requests128→64.lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710→v0.5.15.post1-rocm720-mi35x-20260719.RDMA QoS/DCQCN + GPU-sanity pre-flight gates (
job.slurm)helpers/rdma_check.sh: validates NIC QoS (PFC/DSCP) and DCQCN configuration on every allocated node viasrunbefore any container/GPU time is spent. A misconfigured NIC otherwise only shows up much later as unexplained tail latency or throughput variance under MoRI cross-node KV transfer. Override withSKIP_RDMA_CHECK=1.helpers/gpu_sanity.sh: after container pre-clean/teardown, fails fast (naming the offending process) if a bare, non-containerized process is still holding GPU VRAM — instead of surfacing as an opaque OOM during model load ~30 minutes later.Router policy fix + DPA (ep=8/dp-attn) hicache arm
server_sglang.sh(agentic branch only — DI branch untouched): fixes a bug where the script readROUTER_PREFILL_POLICYbut recipe scripts exportPREFILL_ROUTER_POLICY, so the recipe's policy override was silently ignored and the router always fell back to the hardcoded default. Also switches the default prefill router policy fromcache_aware→consistent_hashing, adds--dp-awareto the router flags, and comments out the separate--decode-policyoverride (superseded by--dp-aware).configs/amd-master.yaml: bumpdsv4-fp4-mi355x-sglang-disagg-agentic-hicacheimage tov0.5.15.post1-rocm720-mi35x-20260719; re-enable (previously commented out) the TP8/EP8/DPA (ep=8, dp-attn=true) hicache search-space arm atconc-list: [64, 96, 128].trace_replay.sh: whenPREFILL_ENABLE_DP=true, exportAIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID=trueso aiperf's HTTP routing key is derived from the correlation ID under DP-attention.Test plan
bash -non all touched shell scripts (env.sh,job.slurm,server_sglang.sh,setup_deps.sh,helpers/gpu_sanity.sh,helpers/rdma_check.sh) — syntax OKconfigs/amd-master.yaml,benchmarks/multi_node/amd_utils/models.yaml— YAML parses cleanlyorigin/mainto confirm no unintended regression in theIS_AGENTIC/DI router split or the router-readiness canary (wait_or_die) added in a prior PRe2e Test - dsv4-fp4-mi355x-sglang-disagg-agentic-hicache(multi-node, mi355x-amds) — new TP8/EP8/DPA arm at conc 64/96/128