feat(glm52-agentx): add compact B300 Dynamo+TRT-LLM recipes / 新增紧凑布局的 B300 Dynamo+TRT-LLM 配方 - #2789
feat(glm52-agentx): add compact B300 Dynamo+TRT-LLM recipes / 新增紧凑布局的 B300 Dynamo+TRT-LLM 配方#2789Ankur-singh wants to merge 6 commits into
Conversation
新增 GLM-5.2 B300 AgentX 紧凑布局配方。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
在性能变更日志中补充 PR #2789 链接。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33422910978 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33422910978 |
仅保留 8p2d 主配置条目，并在 srt-slurm 配方中排除指定的 B300 节点。
更新性能变更日志，说明 GLM-5.2 B300 实验仅启用 8p2d 配置并排除不可用节点。
同步 origin/main，并保留 PR #2789 的性能变更日志条目。
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d4571f7. Configure here.
| DYN_TOKENIZER_CACHE: '1' | ||
| DYN_TOKENIZER_CACHE_BYTES: '8000000000' | ||
| ETCD_LEASE_TTL: '120' | ||
| type: dynamo |
There was a problem hiding this comment.
Selected recipe omits frontend placement
High Severity
The enabled 8p2d recipe and 6p1d omit orchestrator_placement: first_decode, unlike the other compact recipes. The eval rewrite only replaces that exact field with head, so it is a no-op here. Throughput then uses the srt-slurm default instead of first-decode, and eval loopback on the allocation head may miss the frontend.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit d4571f7. Configure here.
| sbatch_directives: | ||
| exclude: b300-001,b300-002,b300-004,b300-005,b300-009,b300-010,b300-012,b300-013,b300-014,b300-015,b300-018 | ||
| srun_options: | ||
| cpu-bind: verbose,mask_cpu:0x0000000000000000ffffffffffffffff0000000000000000ffffffffffffffff,0x0000000000000000ffffffffffffffff0000000000000000ffffffffffffffff,0x0000000000000000ffffffffffffffff0000000000000000ffffffffffffffff,0x0000000000000000ffffffffffffffff0000000000000000ffffffffffffffff,0xffffffffffffffff0000000000000000ffffffffffffffff0000000000000000,0xffffffffffffffff0000000000000000ffffffffffffffff0000000000000000,0xffffffffffffffff0000000000000000ffffffffffffffff0000000000000000,0xffffffffffffffff0000000000000000ffffffffffffffff0000000000000000 |
There was a problem hiding this comment.
CPU masks break compact worker packing
Medium Severity
Every compact recipe passes one 8-entry mask_cpu list. A 4-GPU worker is a 4-task srun, so Slurm applies only the first four masks. The second prefill worker on a node (physical GPUs 4-7) is therefore bound to the GPU 0-3 NUMA socket. HCA pinning uses the physical GPU, so NIC and CPU affinity disagree on half of the compact prefill ranks.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit d4571f7. Configure here.
| elif [[ $MODEL_PREFIX == "glm5.2" && $PRECISION == "fp4" && $FRAMEWORK == "dynamo-trt" ]]; then | ||
| export SERVED_MODEL_NAME="nvidia/GLM-5.2-NVFP4" | ||
| export MODEL_PATH="/scratch/models/GLM-5.2-NVFP4" | ||
| export SRT_SLURM_MODEL_PREFIX="nvidia/GLM-5.2-NVFP4" |
There was a problem hiding this comment.
Launcher served name mismatches recipes
Low Severity
The B300 launcher exports SERVED_MODEL_NAME as nvidia/GLM-5.2-NVFP4, while every new recipe and the working GB300 launcher use GLM-5.2-NVFP4. If the host export reaches the client or eval wait, the chat request uses a different wire name than the frontend and warmup 404s.
Reviewed by Cursor Bugbot for commit d4571f7. Configure here.
Restore the c1, c20, c30, c60, and c227 compact topology points and update the performance changelog to describe the full six-point curve. 中文：恢复 c1、c20、c30、c60 和 c227 紧凑拓扑点，并更新性能变更日志以描述完整的六点曲线。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33569232827 |
1 participant
Description
Adds six GLM-5.2 NVFP4 AgentX srt-slurm recipes on B300 using Dynamo + TensorRT-LLM disaggregated serving with compact worker packing. The master config now selects the complete compact curve at concurrency 1, 20, 30, 60, 227, and 233, spanning the 1p1d, 1p4d, 3p4d, 6p1d, and 8p2d recipes. The recipes exclude b300-001,b300-002,b300-004,b300-005,b300-009,b300-010,b300-012,b300-013,b300-014,b300-015,b300-018 from their Slurm allocations.
中文说明
新增六个 GLM-5.2 NVFP4 AgentX srt-slurm 配方，在 B300 上使用 Dynamo + TensorRT-LLM 解耦式服务和紧凑工作进程布局。主配置现已启用并发 1、20、30、60、227 和 233 的完整紧凑曲线，涵盖 1p1d、1p4d、3p4d、6p1d 和 8p2d 配方。这些配方在 Slurm 分配时排除 b300-001,b300-002,b300-004,b300-005,b300-009,b300-010,b300-012,b300-013,b300-014,b300-015,b300-018。
Related Issue
N/A
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.Note
Medium Risk
Changes multinode benchmark orchestration, UCX/fabric pinning, and srt-slurm integration rather than application logic, but misconfiguration could break AgentX runs or skew cluster networking performance.
Overview
Adds GLM-5.2 NVFP4 AgentX coverage on B300 via six new srt-slurm disaggregated Dynamo + TensorRT-LLM recipes (compact worker packing, NIXL KV transfer, MTP speculative decode), spanning 1p1d through 8p2d layouts at concurrencies 1, 20, 30, 60, 227, and 233. Recipes set
SRT_FABRIC_MODE(bia_faithfulorsymmetric), host-cache sizing, decode batch/CUDA-graph shapes, and Slurm node exclude lists for bad B300 hosts.b300_ctx_hca_pin.shpins prefill context ranks to per-GPU InfiniBand HCA pairs (with ACTIVE-port filtering); decode and non-matching launches stay unpinned.launch_b300-nv.shwires this path: pinned srt-slurm revision, recipe copy into the clone, container UCX (skips hostucx-no-udmount), preamble mount +default_bash_preamble, agentic AIPERF/HF hub mounts,--no-preflightfor/scratchmodels, eval orchestrator on head, optional stripping of forced MTP acceptance tokens,RESULT_FILENAMEoverride, andmeta_env.jsonfor eval score validation.agentic_srt.shnow setsAIPERF_SERVER_URLfromSRT_FRONTEND_HOST/SRT_FRONTEND_PORTwhen present and uses that base for frontend metrics drain polling instead of hardcoded localhost.configs/nvidia-master.yamlregistersglm5.2-fp4-b300-dynamo-trt-agentic-mtp-compactwith search-space entries pointing at each recipe;perf-changelog.yamldocuments the new config key.Reviewed by Cursor Bugbot for commit dd9c653. Bugbot is set up for automated code reviews on this repo. Configure here.