[AgentX] Retune B200 vLLM MTP aggregate / 调优 B200 vLLM MTP 聚合配置 - #2621
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
875f175 to
f8596cd
Compare
ivanium
added
agentx
|
Claude finished @ivanium's task in 2m 33s —— View job Review: LGTM — no blocking issues found
Checks performed:
One non-blocking observation: the DEP8 conc ladder steps by 32 ( Runtime benchmark results are pending per the PR body; the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — this is a straightforward retune of the existing B200 DEP8/TP8 MTP recipe (config values, cudagraph capture-size handling, and an image bump), consistent with the established pattern from the B300 and Kimi-K3 sibling scripts. The one finding is a duplication nit that doesn't affect correctness.
Extended reasoning...
Overview
The PR retunes dsv4_fp4_b200_vllm_mtp.sh for updated AIPerf concurrency semantics, splits the TP8 pure-GPU-resident tier out of the sweep in favor of TP8 SimpleCPU + DEP8 SimpleCPU tiers, bumps the vLLM nightly image, and updates configs/nvidia-master.yaml / perf-changelog.yaml accordingly. The script changes replace the old single --max-cudagraph-capture-size token cap with an explicit per-multiple cudagraph_capture_sizes list (matching vLLM's rounding/dedup behavior), add a DP-attention scheduler-budget divisibility guard, and split MAX_NUM_SEQS calculation between DP-attention and non-DP-attention modes to avoid over-allocating the sequence budget per DP rank.
Security risks
None. This is benchmark tooling (bash scripts + YAML config) with no auth, network-exposed surface, or user-supplied input beyond environment variables set by the CI harness itself.
Level of scrutiny
Low-to-medium. This is a recipe-tuning PR in a benchmark-only script, following an established pattern already present in dsv4_fp4_b300_vllm_mtp.sh (divisibility guard, capture-size loop) and kimik3_fp4_b300_vllm_mtp.sh (capture-size loop). I verified the divisibility guard holds for every concurrency value in the new DEP8 list (all are multiples of 4, satisfying 2*CONC % TP == 0 for TP=8), and that the capture-size loop produces a non-empty, correctly bounded list for both the smallest (CONC=1, TP8) and largest (CONC=196, DEP8) configured points.
Other factors
The only issue raised is a nit about triplicated capture-size/divisibility logic across three sibling scripts, which is a pure refactor opportunity with no behavior change — it doesn't block this PR. No outstanding reviewer comments exist on the thread, and no prior review from me is present to reconcile against.
| # configured sizes up to multiples of (1+N) and deduplicates them; a plain | ||
| # 1..MAX_NUM_SEQS list would cover only MAX_NUM_SEQS/(1+N) decode sequences. | ||
| NUM_SPEC_TOKENS=3 | ||
| TOKENS_PER_SEQ=$((1 + NUM_SPEC_TOKENS)) | ||
| # Throughput pins synthetic MTP acceptance to the dsv4-pro golden AL (thinking_on, | ||
| # num_speculative_tokens=3, golden_al_distribution/dsv4_mtp.yaml). The EVAL_ONLY | ||
| # accuracy run uses real target verification instead -- synthetic acceptance | ||
| # bypasses verification and corrupts the SWE-bench eval (0.0000 score). | ||
| SYNTHETIC_ACCEPT_LEN=2.49 | ||
| if [ "${EVAL_ONLY:-false}" = "true" ]; then | ||
| SPEC_CONFIG="{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS}" | ||
| else | ||
| SPEC_CONFIG="{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"rejection_sample_method\": \"synthetic\", \"synthetic_acceptance_length\": $SYNTHETIC_ACCEPT_LEN}" | ||
| SPEC_CONFIG="{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"rejection_sample_method\": \"synthetic\", \"synthetic_acceptance_length\": 2.49}" | ||
| fi | ||
| TOKENS_PER_SEQ=$((1 + NUM_SPEC_TOKENS)) | ||
| MAX_CUDAGRAPH_CAPTURE_SIZE=$((MAX_NUM_SEQS * TOKENS_PER_SEQ)) | ||
| CUDA_GRAPH_CAPTURE_SIZES="" | ||
| for ((num_seqs = 1; num_seqs <= MAX_NUM_SEQS; num_seqs++)); do | ||
| if [ -n "$CUDA_GRAPH_CAPTURE_SIZES" ]; then | ||
| CUDA_GRAPH_CAPTURE_SIZES+="," | ||
| fi | ||
| CUDA_GRAPH_CAPTURE_SIZES+="$((num_seqs * TOKENS_PER_SEQ))" | ||
| done | ||
| COMPILATION_CONFIG="{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[${CUDA_GRAPH_CAPTURE_SIZES}],\"mode\":0}" | ||
|
|
There was a problem hiding this comment.
🟡 This PR copies two non-trivial blocks verbatim from dsv4_fp4_b300_vllm_mtp.sh: the DP-attention 2*CONC % TP divisibility check (lines 45-49, including the exact error string) and the MAX_NUM_SEQS/CUDA_GRAPH_CAPTURE_SIZES token-multiple loop (lines 245-266). The capture-size loop is a third copy of the one already duplicated in kimik3_fp4_b300_vllm_mtp.sh (~line 207). Since benchmark_lib.sh already centralizes shared agentic-recipe helpers, consider extracting mtp_cudagraph_capture_sizes(max_num_seqs, tokens_per_seq) and require_dp_attention_seq_budget_divisible(conc, tp) there so a future change to the rounding/dedup algorithm doesn't require lockstep edits across three files.
Extended reasoning...
What/where: dsv4_fp4_b200_vllm_mtp.sh (this PR) introduces two blocks that are verbatim (or near-verbatim) copies of logic already present in dsv4_fp4_b300_vllm_mtp.sh:
-
The DP-attention scheduler-budget divisibility guard at lines 45-49:
if [ "$DP_ATTENTION" = "true" ] && [ $((2 * CONC % TP)) -ne 0 ]; then echo "Error: DEP requires 2*CONC divisible by TP, got CONC='$CONC' and TP='$TP'" >&2 exit 1 fi
This is byte-for-byte identical (including the error string) to
dsv4_fp4_b300_vllm_mtp.sh:47-50. -
The
MAX_NUM_SEQSsplit (2*CONC/TPunder DP-attention vs.2*CONCotherwise) plus theCUDA_GRAPH_CAPTURE_SIZESfor-loop that builds a comma-joined list ofnum_seqs * TOKENS_PER_SEQmultiples (lines 245-266). This is identical todsv4_fp4_b300_vllm_mtp.sh:254-286, and the capture-size loop body specifically is duplicated a third time inkimik3_fp4_b300_vllm_mtp.sh:207-214.
Why it matters: benchmark_lib.sh already exists precisely to hold shared agentic-recipe logic — it currently centralizes require_agentic_kv_offload_none, require_agentic_kv_offload_backend, and wait_for_server_ready. The capture-size loop is pure mechanical string-building (not a tuning knob itself — MAX_NUM_SEQS and TOKENS_PER_SEQ remain inline inputs), so it is a strong candidate for the same treatment. With three independent copies, the comments describing the algorithm have already drifted slightly between files (this PR's comment differs in wording from b300's and kimik3's), which is a concrete symptom of the maintenance hazard: a future correction to the rounding/dedup behavior called out in the comments (vLLM rounds configured capture sizes up to multiples of TOKENS_PER_SEQ and dedups them) would require editing three files in lockstep, and it would be easy to update one and miss another.
Proof of triplication (step-by-step):
grep -n \"2 \* CONC % TP\" benchmarks/single_node/agentic/*.sh\"matches bothdsv4_fp4_b200_vllm_mtp.sh:45(this PR) anddsv4_fp4_b300_vllm_mtp.sh:47, with identical error text.grep -n \"CUDA_GRAPH_CAPTURE_SIZES\" benchmarks/single_node/agentic/*.sh\"matches the same for-loop construct indsv4_fp4_b200_vllm_mtp.sh,dsv4_fp4_b300_vllm_mtp.sh, andkimik3_fp4_b300_vllm_mtp.sh, each buildingnum_seqs=1..MAX_NUM_SEQSmultiplied byTOKENS_PER_SEQinto a comma-joined string.- Diffing the three loop bodies shows they are functionally and near-textually identical, differing only in surrounding comment prose.
Suggested fix: add two helpers to benchmark_lib.sh:
require_dp_attention_seq_budget_divisible(conc, tp)— encapsulates the guard and error message.mtp_cudagraph_capture_sizes(max_num_seqs, tokens_per_seq)— builds and echoes the comma-joined capture-size list.
Then call these from all three scripts instead of inlining the logic. This is a pure refactor with no behavior change, so it does not block merging this PR — flagging as a nit for a follow-up cleanup.
将 B200 vLLM MTP AgentX 聚合配置切换为 TP8 和 DEP8 SimpleCPUOffload，并按新的会话树并发语义重新调优并发、长预填充阈值和 GPU 显存利用率。
f8596cd to
889ae25
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31913109698 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31923897249 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31924116400 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31924116400 |
|
/stage-results 31924116400 |
|
/reuse-sweep-run 31924116400 |
|
@cquil11 staging run 31924116400 failed. Inspect the staging workflow. |
|
/stage-results 31924116400 |
|
@cquil11 staged run 31924116400: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-16~r31924116400 This run remains available across future |
2 participants
Summary / 摘要
Retune B200 DeepSeek-V4-Pro vLLM AgentX MTP for the updated AIPerf concurrency semantics, using SimpleCPU only: TP8
[1, 4, 8, 16, 24]and DEP8[32, 64, 96, 128, 160, 196].Update to vLLM
426e59f.针对新版 AIPerf 的并发语义重新调优 B200 DeepSeek-V4-Pro vLLM AgentX MTP，仅使用 SimpleCPU：TP8 为
[1, 4, 8, 16, 24]，DEP8 为[32, 64, 96, 128, 160, 196]。更新至 vLLM
426e59f。Validation / 验证
Bash syntax and YAML parsing passed.
Exact config-key matrix generation produced all 11 intended points.
utils/matrix_logic: 231 tests passed.Performance changelog validation and
git diff --checkpassed.Runtime benchmark results are pending.
Bash 语法与 YAML 解析通过。
指定配置键的矩阵生成包含全部 11 个预期测试点。
utils/matrix_logic：231 项测试通过。性能变更日志校验与
git diff --check通过。运行时基准结果待补充。