Add Qwen3.5 FP8 B300 AgentX MTP - #2515
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Claude finished @cquil11's task in 2m 54s —— View job PR Review
LGTM - no blocking issues found Verified during review:
One observation (non-blocking): the synthetic acceptance length 3.39 is reused from the FP4 recipes. This matches every other Qwen3.5 MTP recipe in the repo and is documented in the changelog, but if the FP8 target model's measured AL differs meaningfully, the throughput numbers will be skewed accordingly — worth confirming against a real-acceptance run if that hasn't been done. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — this mirrors the existing FP4 AgentX MTP recipe closely and follows established patterns.
Extended reasoning...
Overview
Adds a new B300 benchmark scenario (qwen3.5-fp8-b300-sglang-agentic-mtp) combining Qwen3.5-397B-A17B FP8 with SGLang native NEXTN MTP, plus its config entry and changelog note. Three files: a new benchmark shell script, a configs/nvidia-master.yaml addition, and a perf-changelog.yaml entry.
Security risks
None. This is benchmark infrastructure — no auth, no user input handling, no network-facing code beyond internal server launch flags. The script only downloads a model checkpoint and launches a local SGLang server.
Level of scrutiny
Low-to-moderate. This is a config/benchmark-recipe addition, not production logic. I diffed the new script against the existing qwen3.5-fp4-b300-sglang-agentic-mtp script it's explicitly mirroring, and the only differences are precision-specific (fp8 quantization flag vs. fp4/cutlass backend) plus one cosmetic stream-interval variable-vs-literal difference — no functional divergence. The config's search-space grid (TP4 and TP2/EP2 with HiCache arm) is byte-for-byte identical to the FP4 scenario's grid, matching the PR's stated intent to mirror it.
Other factors
The bug hunting system raised a concern that the mirrored TP2/EP2 grid might OOM for FP8's larger weights, but that was investigated and refuted. The PR includes validation steps (shellcheck syntax, config test-generation, full pytest suite) and confirms the FP8 checkpoint exists on cluster storage. This isn't CODEOWNER-restricted critical-path code, and the change is additive (new scenario key) with no modification to existing scenarios.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31150341648 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31152055079 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31152055079 |
|
/stage-results 31152055079 |
|
@cquil11 staged run 31152055079: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-07~r31152055079 This run remains available across future @cquil11 已将运行 31152055079 发布到预发布环境：https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-07~r31152055079 后续的 |
|
/reuse-sweep-run 31152055079 |
…-agentx-mtp # Conflicts: # perf-changelog.yaml
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
❌❌❌ REJECTED ❌❌❌@cquil11 — blocking: the PR's kernel configuration does not match the published SGLang cookbook recipe for Qwen3.5 FP8 on B300. The cookbook pins ✅ Check 0 (CODEOWNER): PASS — |
2 participants
Summary
Why
InferenceX has Qwen3.5 FP8 AgentX and fixed-sequence FP8 MTP coverage on B300, but no scenario combining AgentX with FP8 MTP. This adds that missing benchmark path alongside the existing FP4 AgentX MTP scenario.
Validation
bash -n benchmarks/single_node/agentic/qwen3.5_fp8_b300_sglang_mtp.shpython utils/matrix_logic/generate_sweep_configs.py test-config --config-files configs/nvidia-master.yaml --config-keys qwen3.5-fp8-b300-sglang-agentic-mtppython -m pytest utils/matrix_logic/ -q(224 passed)