[AMD][AgentX] Update DSv4 MI355X vLLM MTP flags on the 2026-08-17 ROCm nightly / [AMD][AgentX] 更新 2026-08-17 ROCm nightly 上的 DSv4 MI355X vLLM MTP 标志 - #2590
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
9702fd6 to
b816427
Compare
jiacao-amd
changed the title
jiacao-amd
changed the title
…d complete the DP-attention curve Repin to the 08-09 nightly (f8d03e77) so the config can start at all: the previously pinned tag no longer serves this recipe, and the 08-12 nightly (3ee2df30) memory-faults during the profile run. Two arms: pure TP8 through its measured knee at 48, and DP-attention with the experts left TP-sharded (ep 1) extended from a single conc-64 point to 32/48/64/96. No EP8 arm -- EP8 only pays off with the MegaMoE backend, which needs mori.ir.flydsl, absent from this image. Recipe side, all DP-attention-conditional: cap --max-num-seqs at CONC rather than 2*CONC (the limit is per scheduler and DP-attention runs one per rank), set --max-num-batched-tokens 8192 with --prefill-schedule-interval 8 and --long-prefill-token-threshold 16384, and raise --gpu-memory-utilization from 0.8 to 0.86. 重新钉到 08-09 nightly（f8d03e77），配置才能起得来：原先钉的 tag 已经跑不动这个 recipe，而 08-12 nightly（3ee2df30）在 profile run 阶段会 memory fault。 两条 arm：纯 TP8 跑到实测拐点 48，以及专家仍按 TP 切分（ep 1）的 DP-attention arm，从原来单点 conc 64 扩展到 32/48/64/96。不加 EP8 arm —— EP8 只有配 MegaMoE 后端才划算，而它需要 mori.ir.flydsl，该镜像里没有。 recipe 侧改动均只在 DP-attention 下生效：--max-num-seqs 从 2*CONC 收到 CONC （该上限是每个 scheduler 生效的，而 DP-attention 每个 rank 一个 scheduler）， 设置 --max-num-batched-tokens 8192 配合 --prefill-schedule-interval 8 与 --long-prefill-token-threshold 16384，并把 --gpu-memory-utilization 从 0.8 提到 0.86。
b816427 to
1bbc5c4
Compare
DSv4-Pro is a mixed checkpoint: MXFP4 routed experts with an FP8 shared expert. vllm/models/deepseek_v4/amd/model.py gates the fused shared-expert path on VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS, which defaults to False, so the checked-in recipe was not running the configuration the validated manual runs used. The aiter side (aiter/fused_moe_dp_shared_expert.py) is present on the pinned base. The flag is mutually exclusive with expert parallelism -- _fuse_shared_experts_enabled() returns False when enable_expert_parallel is set -- which is consistent with both arms here running ep 1. DSv4-Pro 是混合精度检查点:routed experts 为 MXFP4,shared expert 为 FP8。 vllm/models/deepseek_v4/amd/model.py 通过 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS 控制 fused shared-expert 路径,该开关默认为 False,因此签入的 recipe 与已验证的 手工运行并非同一配置。aiter 侧(aiter/fused_moe_dp_shared_expert.py)在所钉基座中存在。 该开关与专家并行互斥 —— enable_expert_parallel 开启时 _fuse_shared_experts_enabled() 返回 False —— 这与本 PR 两条 arm 均为 ep 1 一致。
The gpu-memory-utilization bullet cited 138.85 GiB per rank, which was measured on a different build than the pinned nightly; the figure for this topology on a pre-#51473 base is 160.66 GiB with 34.17 GiB of KV left at 0.86. The max-num-batched-tokens comparison was taken on the expert-parallel MegaMoE topology, not on the dp-attn arm, so it motivates the value rather than establishing it. Also correct the dp-attn description: dp-attn true maps to --tensor-parallel-size 1 --data-parallel-size 8, so the weights are replicated per rank rather than TP-sharded. 将 gmu 与 MNBT 的依据回溯到其真实来源的运行。138.85 GiB 来自另一构建； 本拓扑在 pre-#51473 基座上的实测为每 rank 160.66 GiB、0.86 下仅剩 34.17 GiB KV。 MNBT 对比取自专家并行 MegaMoE 拓扑而非本 arm，只作选值依据。 同时更正 dp-attn 描述：其对应 TP=1 + DP=8，权重按 rank 复制而非 TP 切分。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @jiacao-amd's task in 3m 27s —— View job PR Review
LGTM - no blocking issues found Checks performed:
Note: the inferencemax-repos MCP server was not available in this run, so I could not verify against vLLM source that the ROCm 08-09 nightly accepts |
jiacao-amd
changed the title
jiacao-amd
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — the one inline finding is a nit (an inert flag value, not a functional break), and the rest of the repin/config changes are well-reasoned and self-contained.
Extended reasoning...
Overview
This PR repins the dsv4-fp4-mi355x-vllm-agentic-mtp recipe to a new ROCm nightly and extends the DP-attention concurrency ladder. Changes are confined to a benchmark launch script, the corresponding amd-master.yaml search-space entry, and an append-only perf-changelog.yaml entry — no production serving code, auth, or shared infrastructure is touched.
Security risks
None. This is benchmark/tuning configuration for an internal sweep harness; it does not touch request handling, credentials, or externally reachable code paths.
Level of scrutiny
Low-to-moderate is appropriate here: it's a data/config change (image pin bump, concurrency list, and a handful of vLLM CLI flags gated behind DP_ATTENTION=true) rather than a logic change to shared code. The one inline finding (a --long-prefill-token-threshold value that can never bind under the accompanying --max-num-batched-tokens budget) is flagged as a Nit by the bug hunter — it doesn't crash or break the run, it just leaves a piece of the stated 'chunk long prefills' rationale as dead configuration. The sweep itself will still execute and produce valid throughput numbers for the arm.
Other factors
The PR description is thorough and grounds each flag change (max-num-seqs capping, gpu-memory-utilization bump, aiter fused shared-expert flag) in specific measurements or code references. The PR is still in draft pending sweep results per its own description, and full-sweep-enabled is applied appropriately given the image/topology changes. No unresolved reviewer threads exist in the timeline.
seungrokj
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31676544794 |
|
/reuse-sweep-run |
中文：将 DSv4 MI355X vLLM AgentX MTP 配方重新固定到 2026-08-17 ROCm nightly，并显式配置 TP8 本地复现实验所需的 INT4 quick-reduce、FSE、FULL_AND_PIECEWISE 和 no-Gluon/no-probe 标志。
中文：将 PR #2590 分支同步到最新 main，并按 perf-changelog.yaml 规则重新追加本 PR 的基准测试触发条目。
jiacao-amd
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32003969222 |
Pin max-num-batched-tokens to 8192 for both pure TP8 and DP-attention so the pinned nightly retains the validated KV-cache capacity. Keep only concurrency 64 for the DP-attention arm and remove the Gluon/sparse-probe environment cleanup. Update the performance changelog with the initialization A/B result.\n\n中文：为纯 TP8 和 DP-attention 显式固定 max-num-batched-tokens=8192，使当前 nightly 保持已验证的 KV cache 容量。DP-attention 仅保留并发 64，并移除 Gluon/sparse probe 环境清理。同步在性能变更日志中记录初始化 A/B 结果。
Remove the explicit 1200-second execute-model timeout because the pinned image default is sufficient for the validated AgentX recipe. 中文：移除显式设置的 1200 秒 execute-model 超时，已验证的 AgentX 配方直接使用固定镜像的默认值即可。
Merge the latest origin/main and resolve the perf-changelog conflict by preserving main verbatim and appending PR 2590's entry at the tail. 中文：合并最新 origin/main，并按 append-only 规则解决 perf-changelog 冲突：完整保留 main 内容，将 PR 2590 的条目重新追加到文件末尾。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32003969222 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32082871496 |
|
/reuse-sweep-run |
chunfangamd
left a comment
chunfangamd
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. Link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/32082871496
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. Value: 95.75%. Link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/32082871496
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
Signed: @chunfangamd
❌❌❌ REJECTED ❌❌❌@chunfangamd — blocking: the sign-off contains no upstream recipe link. This PR changes a single-node vLLM recipe, so the additional detail section must link the merged ✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a listed owner of |
❌❌❌ REJECTED ❌❌❌@chunfangamd — blocking: the sign-off's additional detail section is still the empty template placeholder ("insert any additional info here") and contains no upstream recipe link. This PR changes a single-node vLLM recipe, so the sign-off must link the merged ✅ Check 0 (CODEOWNER): PASS — |
|
Opened the matching upstream recipes PR so this InferenceX change can satisfy Check 3 (single-node vLLM recipe documented upstream): It adds an opt-in Agentic pill plus copy-paste MI355X TP8 and DP-attention ( 中文：已开对应的上游 vLLM recipes PR，便于本 InferenceX 改动满足 Check 3（单节点 vLLM 配方需有上游文档）：vllm-project/recipes#824 。其中新增了可选的 Agentic 开关，以及本 PR 已验证的 MI355X 纯 TP8 与 DP-attention（ |
|
/stage-results 32082871496 |
|
@functionstackx staged run 32082871496: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-18~r32082871496 This run remains available across future |
|
/reuse-sweep-run |
|
/reuse-sweep-run |
6 participants
Summary
Update
dsv4-fp4-mi355x-vllm-agentic-mtpto the 2026-08-17 vLLM ROCm nightly, extend the pure TP8 AgentX ladder to concurrency 56 and 64, and make the runtime flags used by the validated local runs explicit in the checked-in recipe.This uses the current immutable vLLM ROCm nightly:
vllm/vllm-openai-rocm:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0sha256:4b403315f9a0d9b0ed4221dfc7ea04538701b4de0ce283c0e2cee622b217ff89nightlylast pushed:2026-08-17T05:32:09Z311b3513af33bc29b4acb2fde2e9313e5e9966a0The pinned nightly includes:
Search-space changes (
configs/amd-master.yaml)1, 4, 8, 16, 32, 40, 48extended to1, 4, 8, 16, 32, 40, 48, 56, 64.64.Ten benchmark points total: nine on pure TP8, one on DP-attention.
Recipe changes (
benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh)Applied to both arms:
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4(previously commented out).VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1. DSv4-Pro is a mixed checkpoint (MXFP4 routed experts, FP8 shared expert) and vLLM gates the fused shared-expert path on this flag, which defaults to off, so the checked-in recipe was not running the configuration the validated manual runs used. The flag is mutually exclusive with expert parallelism inside vLLM, which is consistent with both arms runningep 1.--gpu-memory-utilizationfrom0.8to0.86.--max-num-batched-tokens 8192instead of the nightly default of 16384. On the TP8 initialization check this raised GPU KV-cache capacity from 4,730,981 to 8,524,228 tokens and reduced peak activation memory from 11.44 GiB to 8.9 GiB.Applied to the DP-attention arm only:
--max-num-seqsatCONCrather than2*CONC. The limit is per scheduler and DP-attention runs one scheduler per rank. The pure TP8 arm keeps the existing2*CONCheadroom for AgentX subagent fan-out.--prefill-schedule-interval 8and--long-prefill-token-threshold 16384.Note on the open review thread: with
--max-num-batched-tokens 8192now pinned for both arms,--long-prefill-token-threshold 16384sits above the token budget and therefore never binds. It is inert rather than harmful, and the DP-attention point was measured with it present.--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}'is unchanged; it was already in the checked-in recipe.Validation
Full sweep on this branch: run 32082871496, conclusion
successon head4fb6de5, with all ten configured points green (pure TP8 c1/c4/c8/c16/c32/c40/c48/c56/c64 and DP-attention c64) plus theagentic eval ... c64 | eval-onlyjob.The only change since that run is the
perf-changelog.yamlmerge withorigin/main; the benchmark-affecting diff is unchanged, so reuse of that sweep was authorized with/reuse-sweep-run.Search-space generation was checked locally with:
/home/jiacao/.local/bin/uv run --with 'pydantic>=2' --with pyyaml \ python3 utils/matrix_logic/generate_sweep_configs.py full-sweep \ --config-files configs/amd-master.yaml \ --model-prefix dsv4 \ --framework vllm \ --runner-type cluster:mi355x-amds \ --min-conc 1 --max-conc 96 --no-evalsAcceptance-length methodology is unchanged by this PR: throughput runs pin
num_speculative_tokens 3tosynthetic_acceptance_length 2.49, the committed thinking-on golden value for DeepSeek-V4-Pro at draft length 3 ingolden_al_distribution/dsv4_mtp.yaml, while eval-only runs drop synthetic acceptance and use real target verification.Note: Docker socket access is not available on the login node used for this edit, so the image check was done via Docker Hub metadata plus upstream vLLM merge ancestry rather than by importing and grepping the container filesystem locally.
中文说明
将
dsv4-fp4-mi355x-vllm-agentic-mtp更新到 2026-08-17 vLLM ROCm nightly，把纯 TP8 的 AgentX 并发阶梯扩展到 56 和 64，并在签入的配方中显式写入已验证本机运行所使用的 runtime flags。本 PR 固定到当前不可变的 vLLM ROCm nightly 镜像：
vllm/vllm-openai-rocm:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0sha256:4b403315f9a0d9b0ed4221dfc7ea04538701b4de0ce283c0e2cee622b217ff89nightly最后推送时间：2026-08-17T05:32:09Z311b3513af33bc29b4acb2fde2e9313e5e9966a0该 nightly 已包含：
中文搜索空间变更（
configs/amd-master.yaml）1、4、8、16、32、40、48扩展为1、4、8、16、32、40、48、56、64。64不变。合计十个基准测试点：纯 TP8 九个，DP-attention 一个。
中文配方变更（
benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh）两条 arm 均生效：
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4（此前被注释掉）。VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1。DSv4-Pro 是混合精度检查点（routed experts 为 MXFP4，shared expert 为 FP8），vLLM 通过该开关控制 fused shared-expert 路径且默认关闭，因此签入的配方与已验证的手工运行并非同一配置。该开关在 vLLM 内部与专家并行互斥，这与两条 arm 均为ep 1一致。--gpu-memory-utilization从0.8提高到0.86。--max-num-batched-tokens 8192，替代该 nightly 的默认值 16384。在 TP8 初始化检查中，GPU KV cache 容量从 4,730,981 提升到 8,524,228 tokens，峰值激活内存从 11.44 GiB 降到 8.9 GiB。仅 DP-attention arm 生效：
--max-num-seqs从2*CONC收到CONC。该上限按 scheduler 生效，而 DP-attention 每个 rank 一个 scheduler。纯 TP8 arm 保留原有的2*CONC余量以容纳 AgentX 子智能体扇出。--prefill-schedule-interval 8与--long-prefill-token-threshold 16384。关于未决审阅意见的说明：由于两条 arm 现在都固定了
--max-num-batched-tokens 8192，--long-prefill-token-threshold 16384高于 token 预算，因此不会生效。它是惰性的而非有害的，且 DP-attention 数据点就是在带该标志的情况下实测的。--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}'未改动，签入配方中原本就有。中文验证
本分支完整 sweep：run 32082871496，在 head
4fb6de5上结论为success，配置的十个点全部通过（纯 TP8 c1/c4/c8/c16/c32/c40/c48/c56/c64 与 DP-attention c64），并包含agentic eval ... c64 | eval-only任务。此后的唯一改动是
perf-changelog.yaml与origin/main的合并，影响基准测试的 diff 未变，因此已通过/reuse-sweep-run授权复用该 sweep。搜索空间生成在本地用以下命令核对：
/home/jiacao/.local/bin/uv run --with 'pydantic>=2' --with pyyaml \ python3 utils/matrix_logic/generate_sweep_configs.py full-sweep \ --config-files configs/amd-master.yaml \ --model-prefix dsv4 \ --framework vllm \ --runner-type cluster:mi355x-amds \ --min-conc 1 --max-conc 96 --no-evals本 PR 未改动接受长度方法学：吞吐运行将
num_speculative_tokens 3固定到synthetic_acceptance_length 2.49，即golden_al_distribution/dsv4_mtp.yaml中 DeepSeek-V4-Pro 在 draft 长度 3 下的 thinking-on golden 值；eval-only 运行则去掉合成接受，使用真实的 target 校验。说明：本次编辑所在的 login node 没有 Docker socket 权限，因此镜像内容核验是通过 Docker Hub 元数据和上游 vLLM merge ancestry 完成的；没有在本地导入容器后 grep 容器文件系统。