[AMD] Kimi-K2.5 MXFP4 ATOMESH disagg config #2172
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
seungrokj
changed the title
seungrokj
added
AMD
full-sweep-fail-fast
evals-only
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - config-keys: | ||
| - kimik2.5-fp4-mi355x-atom-disagg | ||
| description: | ||
| - "Add Kimi-K2.5-MXFP4 entry to models_atom.yaml for ATOM disaggregated inference" | ||
| - "Set env: ATOM_DISABLE_MMAP=true, AITER_QUICK_REDUCE_QUANTIZATION=INT4, AITER_MXFP4_INTERMEDIATE=1, OMP_NUM_THREADS=1" | ||
| - "Add scheduler_delay_factor support to models_atom.yaml schema and server_atom.sh; set to 1 for Kimi-K2.5" |
There was a problem hiding this comment.
🔴 The new perf-changelog entry at lines 4737-4742 has two blocking issues that will hard-fail this PR's own sweep CI before any GPU work runs: (1) missing the required pr-link field — every neighboring entry has one, and ChangelogEntry in utils/matrix_logic/validation.py:714 declares pr_link as required with extra="forbid", so ChangelogEntry.model_validate in utils/process_changelog.py:158 will raise a Pydantic ValidationError; (2) the referenced config-keys: [kimik2.5-fp4-mi355x-atom-disagg] does not exist in configs/amd-master.yaml — only the non-disagg kimik2.5-fp4-mi355x-atom (line 811) is defined, so get_config_keys_from_master in utils/process_changelog.py:101-102 will raise ValueError: Config key '\''kimik2.5-fp4-mi355x-atom-disagg'\'' not found in master configs. Fix by appending pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2172 and either adding the kimik2.5-fp4-mi355x-atom-disagg entry to configs/amd-master.yaml or updating the changelog to reference an existing key.
Extended reasoning...
What the bug is
The new perf-changelog entry appended at perf-changelog.yaml:4737-4742 has two independent defects that both cause utils/process_changelog.py (invoked from .github/workflows/run-sweep.yml:242 on every labeled PR) to raise before the sweep does any real work.
Issue 1 — missing required pr-link
utils/matrix_logic/validation.py:708-714 defines:
class ChangelogEntry(BaseModel):
model_config = ConfigDict(extra="forbid", populate_by_name=True)
config_keys: List[str] = Field(alias="config-keys")
description: List[str]
pr_link: str = Field(alias="pr-link") # no default → requiredutils/process_changelog.py:158 runs ChangelogEntry.model_validate(entry_data) on every entry parsed out of the diff. The new entry has no pr-link: line, unlike every neighboring entry (e.g. lines 4710, 4718, 4726, 4735), so validation raises a Pydantic ValidationError: pr-link Field required.
Issue 2 — config-key does not exist in master config
configs/amd-master.yaml contains only kimik2.5-fp4-mi355x-atom (line 811, framework: atom, non-disagg). It has no kimik2.5-fp4-mi355x-atom-disagg key, even though sibling -atom-disagg keys exist for other models (dsv4-fp4-mi355x-atom-disagg L2344, minimaxm3-fp8-mi355x-atom-disagg L2651, minimaxm3-fp4-mi355x-atom-disagg L2721).
utils/process_changelog.py:87-105 get_config_keys_from_master raises ValueError('Config key "{k}" not found in master configs.') for any non-wildcard key not present in the loaded master config. The key here has no *, so it hits that branch and raises.
Step-by-step proof
- PR is labeled
full-sweep-fail-fast(per the recipe-reminder bot on this PR). .github/workflows/run-sweep.yml:242invokespython utils/process_changelog.py.process_changelog.pycallsget_added_lines→yaml.safe_loadon the added entries.- For the appended entry, it calls
ChangelogEntry.model_validate({'config-keys': ['kimik2.5-fp4-mi355x-atom-disagg'], 'description': [...]})— nopr-linkkey. - Pydantic raises
ValidationError: pr-link Field required. - Even if
pr-linkwere present, the next callget_config_keys_from_master(['kimik2.5-fp4-mi355x-atom-disagg'], master_config)would then raiseValueError: Config key 'kimik2.5-fp4-mi355x-atom-disagg' not found in master configs.— the key falls into theelifbranch at line 101 because it has no*wildcard and is not present in the loaded YAML. - The workflow step exits non-zero and the entire sweep is skipped.
Why existing code doesn't prevent it
Both checks are strict-by-design: extra="forbid" combined with an aliased required field, and get_config_keys_from_master explicitly raises rather than warning. This is intentional — the changelog is authoritative and stale/invalid keys should fail loud. Nothing in the PR touches configs/amd-master.yaml to define the referenced -atom-disagg variant, and nothing added pr-link to the new entry.
Impact
The PR's own full-sweep CI (and any future main-branch sweep processing this changelog line) hard-fails at the changelog validation step, before any GPU allocation. The new Kimi-K2.5 disagg config never gets exercised end-to-end.
How to fix
Two small changes to the entry / master config:
- Add
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2172under the new entry (matches every neighbor). - Either add a
kimik2.5-fp4-mi355x-atom-disaggentry toconfigs/amd-master.yamlalongside the models_atom.yaml / server_atom.sh changes (framework:atom-disagg, model:amd/Kimi-K2.5-MXFP4, multinode scenarios like the sibling MiniMax entries), or update the changelog line to reference an existing key such askimik2.5-fp4-mi355x-atomif the disagg config is intentionally deferred.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29216632132 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29216770759 |
seungrokj
added
full-sweep-enabled
and removed
full-sweep-fail-fast
evals-only
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29217338760 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29217426499 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29217426499 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29218534481 |
…tor support Add Kimi-K2.5-MXFP4 entry to models_atom.yaml with env vars and kv_cache_dtype from the single-node script. Implement scheduler_delay_factor as a new YAML-driven server_atom.sh parameter so models can set --scheduler-delay-factor per-model. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rt, and perf-changelog Add Kimi-K2.5-MXFP4 to models_atom.yaml for ATOM PD-disagg with env vars from single-node script. Implement scheduler_delay_factor as a new YAML-driven parameter in server_atom.sh. Update perf-changelog. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add atom-disagg branch to engine-specific extra flags in bench.sh, setting --trust-remote-code --tokenizer and MTP flags (--dsv4 or --use-chat-template) matching the vllm-disagg and sglang patterns. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6372d52 to
ffe3e10
Compare
…ogies Replace previous 1P2D/2P1D/1P1D scenarios with 2P1D, 1P4D, and 1P2D topologies for 8k/1k ISL/OSL. Trim concurrency lists for faster sweeps. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29298070441 |
Remove 1P2D topology, keep 1P1D with extended concurrency list up to 1024. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29303906040 |
…sagg Update image to atomesh_202607121715. Add 1P2D 8k/1k scenario alongside existing 1P1D with same concurrency sweep up to 1024. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29311834365 |
Add explicit max_num_seqs=256 to match upstream, remove 1P2D topology, and cap concurrency at 256. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29335987128 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29378817052 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29394272484 |
Scale decode to 2 workers across 2 nodes (TP8 each, DECODE_NODES=2). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Run both topologies: 1P1D at high concurrency [32,64,128,256] and 1P2D at low concurrency [1,2,4,8,16,32]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29401578319 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29405845383 |
Move conc=16 from 1P2D to 1P1D; move conc=32 out of 1P2D. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29406333352 |
|
/reuse-sweep-run |
|
kimik2.5 atom PD recipe at ROCm/ATOM#1575 |
chunfangamd
left a comment
chunfangamd
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. Link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29406333352
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. Range: 97.35% - 97.50%, Link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29406333352
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- This PR is not for the agentic workloads (For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.)
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
ROCm/ATOM#1575
Signed: @chunfangamd
chunfangamd
left a comment
chunfangamd
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. Link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29406333352
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. Range: 97.35% - 97.50%, Link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29406333352
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- This PR is not for the agentic workloads (For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.)
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
ROCm/ATOM#1575
Signed: @chunfangamd
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
Oseltamivir
left a comment
Oseltamivir
left a comment
There was a problem hiding this comment.
lgtm, no patches, dram rule enforced, atom recipe merged, pr checklist done
generate_sweep_configs.py no longer hardcodes total_cpu_dram_gb = 0 for multinode. It folds multinode into agentic_dram_offload_gb, sizing from the prefill worker's per-node GPU footprint (worker_gpus_per_node), still min(node_DRAM, 3TB) × utilization × gpus_used/gpus_per_node. For this 1P1D TP8 config that's ≈ 2400 GB/node.
total_cpu_dram_gb is now a required field on MultiNodeAgenticMatrixEntry
4 participants
Summary
Kimi-K2.5-MXFP4entry tomodels_atom.yamlfor ATOM PD-disaggregated inference with env vars (ATOM_DISABLE_MMAP,AITER_QUICK_REDUCE_QUANTIZATION=INT4,AITER_MXFP4_INTERMEDIATE=1,OMP_NUM_THREADS=1) and--kv_cache_dtype fp8scheduler_delay_factoras a new YAML-driven parameter inserver_atom.sh(YAML > env var > unset), set to1for Kimi-K2.5kimik2.5-fp4-mi355x-atom-disaggTest plan
kimik2.5-fp4-mi355x-atom-disaggCI sweep launches correctly with the newmodels_atom.yamlentry--scheduler-delay-factor 1appears in prefill and decode server launch commandsscheduler_delay_factorin YAML are unaffected (flag not injected)🤖 Generated with Claude Code