fix: recover PR 2380 ingest via sweep reuse - #2418
Conversation
… rest
Reduce the flag alignment to its one surviving piece and rebuild on main.
The full upstream Blackwell/NVIDIA block killed the engine core on every
completed concurrency -- c1/c2/c4/c8 across both KV arms in run 30378755933 --
roughly 13-16 min in, during agentic warmup:
vllm/model_executor/layers/fused_moe/runner/shared_experts.py:165
assert self._output[self._output_idx] is None
AssertionError (all 8 TP ranks simultaneously)
K3 declares num_shared_experts 2 and this image resolves its MoE backend to
FLASHINFER_TRTLLM_MXFP4_MXFP8 / TrtLlmMxfp4ExpertsMonolithic, so that reads as an
image-side invariant violation rather than a bad configuration value. The same
crash hit the non-DSpark recipe, so it is not spec-decoding specific.
Retained: --attention-config mla_prefill_backend=TRTLLM_RAGGED
Reverted: gpu-memory-utilization 0.95 -> 0.90
max-num-seqs 32 -> 2*CONC
max-model-len 1000000 -> 1048576
max-num-batched-tokens 32768 -> dropped (8192 default)
--no-enable-flashinfer-autotune -> dropped
VLLM_USE_V2_MODEL_RUNNER=1 -> dropped (V2 is on by default anyway)
cudagraph capture consequently enumerates 2*CONC token-multiples of (1+7)=8
again, instead of a fixed 32-entry ladder to 256.
Rebased onto main rather than amending, since main had advanced four commits and
the changelog diff had started reading main's tail as deletions.
…rk recipe
The upstream recipe sets this on BOTH its blackwell and nvidia paths
(recipes.vllm.ai/moonshotai/Kimi-K3), and this repo has never set it. It
defaults to 0.
It matters because of where it lands: vllm/models/kimi_k3/nvidia/model.py:549
threads it into LatentMoERunner as
runner_args={"enable_k3_latent_moe_tail_fusion": ...}, and that is the same
runner whose shared-experts output buffer asserted when the wider flag alignment
was tried:
vllm/model_executor/layers/fused_moe/runner/shared_experts.py:165
assert self._output[self._output_idx] is None (all 8 TP ranks)
So with it unset we have been running a MoE tail path upstream never exercises,
on every K3 run here including the green ones. That is a better explanation for
the crash than any of the six flags reverted earlier.
Enabled on its own so its effect is isolated. If this holds, the remaining
upstream deltas can be re-added in order: gpu-memory-utilization 0.95,
max-num-seqs 32 (spec arm), --no-enable-flashinfer-autotune,
VLLM_USE_V2_MODEL_RUNNER=1.
Not re-adding --max-num-batched-tokens 32768: re-reading the recipe YAML shows
that value appears nowhere for blackwell (AMD is 4096, PD prefill 16384, PD
decode 32), so it was introduced in error. --max-model-len is likewise 1048576
upstream, not 1000000, and already correct here.
…olden AL
Adopt the AgentX-prescribed measurement methodology for the DSpark arm, per
golden_al_distribution/README.md: "a submission may choose any supported draft
length, but it may not substitute a different acceptance target", because
"InferenceX is evaluating inference-system performance, not the ability to
fine-tune a benchmark-specific speculative head". The README also states up front
that AL is workload-dependent, so pinning the committed acceptance is what makes
submissions comparable -- running the acceptance this corpus happens to produce
would be the non-conforming choice.
throughput: rejection_sample_method synthetic, synthetic_acceptance_length 2.51
(committed golden AL at K=2, probabilistic curve)
EVAL_ONLY: rejection_sample_method block (real verification)
The eval split matters: synthetic acceptance commits drafts regardless of target
logits, so text is wrong and SWE-bench scores 0.0000. This follows
dsv4_fp4_b300_vllm_mtp.sh; kimik2.5_fp4_b300_mtp.sh omits the split.
Level 2, not 7. Draft length IS ours to choose, and verify width is a real system
cost synthetic acceptance does not remove -- at K=7 the target verifies 8
positions per step, which loses to no-speculation once the GPU saturates even at
the forced golden AL of 3.84. With cost ~ 1 + m(K-1) + f: K=2 clears break-even
across the m range (~1.7x low conc, ~1.2x saturated), K=3 is a wash at saturation
(~0.95x), K=7 is negative (~0.54x). cudagraph capture therefore enumerates
multiples of (1+2)=3.
Recorded in the script and config headers, because the throughput numbers will
otherwise be misread: the draft's REAL acceptance on this corpus is 1.16-2.01
with 13-41% position-1 acceptance. Its native window is 32k (rope yarn factor 32
over original_max_position_embeddings 32768, stretched to the declared 1M), and
the identical image, draft and spec config reach AL 4.18 / 0.826 position-1 on
short-context work. The synthetic numbers measure the system at a prescribed
acceptance; they are not evidence the draft predicts well at 100k+ context.
|
/reuse-sweep-run 30425131777 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional findings (outside current diff — PR may have been updated during review):
-
🔴
perf-changelog.yaml:1— This recovery PR only contains the empty step-4 bootstrap commit (d526a8b) — steps 5 and 6 of the recover-failed-ingest process were never applied.perf-changelog.yamlhas no appended recovery entry, and there's no two-parent commit-tree carrier attaching source headed0fe56dinto this PR's commit history. Becauseutils/find_reusable_sweep_run.py:validate_reusable_runrequires the source head SHA to be present in the PR's commit list, merging as-is will make the/reuse-sweep-run 30425131777reuse gate fail again with the same setup error, and no ingest will happen.Extended reasoning...
What's wrong: The PR description claims this recovers the failed ingest for PR #2380 by reusing artifacts from pinned source run
30425131777(headed0fe56d). But the actual commit history shows only one commit was made —d526a8b ("chore: prepare PR 2380 ingest recovery")— and it is a single-parent, empty (tree-unchanged) commit relative to its parentddeb02e. That's exactly the step-4 "bootstrap" commit described in.claude/commands/recover-failed-ingest.md. Steps 5 and 6 of that same process were never executed.Missing step 5 (changelog append):
perf-changelog.yaml's last entry is still the pre-existing PR #2139 entry — there is no new recovery entry referencing PR #2418 or run30425131777. Without this append, there is no new sweep-matrix scope for the ingest pipeline to pick up; the changelog is what tells the ingest process which points to import.Missing step 6 (ancestry carrier): There is no two-parent
commit-treecommit anywhere in the repo's history that attaches source headed0fe56das a second parent of the recovery branch. This is not a cosmetic omission —utils/find_reusable_sweep_run.py'svalidate_reusable_run(around lines 233-275) explicitly requires the reuse source's head SHA to appear inpr_commit_shas(), which is populated directly from GitHub's/pulls/{pr}/commitsendpoint. Ifed0fe56dis not part of PR #2418's commit list, that endpoint won't return it, and the same "Reusable source run ... head ... is not in PR #2418's commit list" error that caused the original failure will be raised again during the push-to-main setup job.Why nothing catches this today: There's no CI check that verifies the PR's commit graph actually contains the pinned reuse source's SHA before merge — that verification only happens at
setuptime on the push-to-main run, i.e. after merge. So this PR can pass any pre-merge checks and still fail identically to the original PR #2380 ingest failure once it lands on main.Concrete walkthrough:
git rev-list --parents -n 1 d526a8b→ single parentddeb02e;git diff d526a8b^ d526a8b→ empty. Confirms this is only the step-4 empty bootstrap commit.grep -n "pull/2418\|30425131777" perf-changelog.yaml→ no matches; last entry in the file is still PR #2139. Confirms step 5 (changelog append) never happened.git log --all→ed0fe56ddoes not appear as an ancestor or parent anywhere; it is not even a resolvable object in this repo's history. Confirms step 6 (attach commit) never happened.- On merge, main's push-to-main workflow triggers
Run Sweep, which callsfind_reusable_sweep_run.py.validate_reusable_runcallspr_commit_shas(2418), which hits/pulls/2418/commits. Sinceed0fe56dwas never added to this branch, it's absent from that list, sovalidate_reusable_runraises the same "not in PR #2418's commit list" error,setupfails, and no GPU/reuse ingest occurs — identical failure mode to the original problem this PR set out to fix.
Fix: Complete steps 5 and 6 of the recover-failed-ingest process — append the recovery entry to
perf-changelog.yamlfor the recovered sweep points, and create the two-parentgit commit-treecarrier commit that attachesed0fe56das a second parent so it lands in PR #2418's commit list. Without both, merging this PR is a no-op with respect to its stated purpose.
Ingest recovery audit — completeRecovered: Failed target
Source
Semantics checks (source SHA → PR #2380 head
|
* fix(kimik3): keep only TRTLLM_RAGGED in the DSpark recipe, revert the rest
Reduce the flag alignment to its one surviving piece and rebuild on main.
The full upstream Blackwell/NVIDIA block killed the engine core on every
completed concurrency -- c1/c2/c4/c8 across both KV arms in run 30378755933 --
roughly 13-16 min in, during agentic warmup:
vllm/model_executor/layers/fused_moe/runner/shared_experts.py:165
assert self._output[self._output_idx] is None
AssertionError (all 8 TP ranks simultaneously)
K3 declares num_shared_experts 2 and this image resolves its MoE backend to
FLASHINFER_TRTLLM_MXFP4_MXFP8 / TrtLlmMxfp4ExpertsMonolithic, so that reads as an
image-side invariant violation rather than a bad configuration value. The same
crash hit the non-DSpark recipe, so it is not spec-decoding specific.
Retained: --attention-config mla_prefill_backend=TRTLLM_RAGGED
Reverted: gpu-memory-utilization 0.95 -> 0.90
max-num-seqs 32 -> 2*CONC
max-model-len 1000000 -> 1048576
max-num-batched-tokens 32768 -> dropped (8192 default)
--no-enable-flashinfer-autotune -> dropped
VLLM_USE_V2_MODEL_RUNNER=1 -> dropped (V2 is on by default anyway)
cudagraph capture consequently enumerates 2*CONC token-multiples of (1+7)=8
again, instead of a fixed 32-entry ladder to 256.
Rebased onto main rather than amending, since main had advanced four commits and
the changelog diff had started reading main's tail as deletions.
* fix(kimik3): enable VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION on the DSpark recipe
The upstream recipe sets this on BOTH its blackwell and nvidia paths
(recipes.vllm.ai/moonshotai/Kimi-K3), and this repo has never set it. It
defaults to 0.
It matters because of where it lands: vllm/models/kimi_k3/nvidia/model.py:549
threads it into LatentMoERunner as
runner_args={"enable_k3_latent_moe_tail_fusion": ...}, and that is the same
runner whose shared-experts output buffer asserted when the wider flag alignment
was tried:
vllm/model_executor/layers/fused_moe/runner/shared_experts.py:165
assert self._output[self._output_idx] is None (all 8 TP ranks)
So with it unset we have been running a MoE tail path upstream never exercises,
on every K3 run here including the green ones. That is a better explanation for
the crash than any of the six flags reverted earlier.
Enabled on its own so its effect is isolated. If this holds, the remaining
upstream deltas can be re-added in order: gpu-memory-utilization 0.95,
max-num-seqs 32 (spec arm), --no-enable-flashinfer-autotune,
VLLM_USE_V2_MODEL_RUNNER=1.
Not re-adding --max-num-batched-tokens 32768: re-reading the recipe YAML shows
that value appears nowhere for blackwell (AMD is 4096, PD prefill 16384, PD
decode 32), so it was introduced in error. --max-model-len is likewise 1048576
upstream, not 1000000, and already correct here.
* feat(kimik3): DSpark at level 2 with synthetic acceptance pinned to golden AL
Adopt the AgentX-prescribed measurement methodology for the DSpark arm, per
golden_al_distribution/README.md: "a submission may choose any supported draft
length, but it may not substitute a different acceptance target", because
"InferenceX is evaluating inference-system performance, not the ability to
fine-tune a benchmark-specific speculative head". The README also states up front
that AL is workload-dependent, so pinning the committed acceptance is what makes
submissions comparable -- running the acceptance this corpus happens to produce
would be the non-conforming choice.
throughput: rejection_sample_method synthetic, synthetic_acceptance_length 2.51
(committed golden AL at K=2, probabilistic curve)
EVAL_ONLY: rejection_sample_method block (real verification)
The eval split matters: synthetic acceptance commits drafts regardless of target
logits, so text is wrong and SWE-bench scores 0.0000. This follows
dsv4_fp4_b300_vllm_mtp.sh; kimik2.5_fp4_b300_mtp.sh omits the split.
Level 2, not 7. Draft length IS ours to choose, and verify width is a real system
cost synthetic acceptance does not remove -- at K=7 the target verifies 8
positions per step, which loses to no-speculation once the GPU saturates even at
the forced golden AL of 3.84. With cost ~ 1 + m(K-1) + f: K=2 clears break-even
across the m range (~1.7x low conc, ~1.2x saturated), K=3 is a wash at saturation
(~0.95x), K=7 is negative (~0.54x). cudagraph capture therefore enumerates
multiples of (1+2)=3.
Recorded in the script and config headers, because the throughput numbers will
otherwise be misread: the draft's REAL acceptance on this corpus is 1.16-2.01
with 13-41% position-1 acceptance. Its native window is 32k (rope yarn factor 32
over original_max_position_embeddings 32768, stretched to the declared 1M), and
the identical image, draft and spec config reach AL 4.18 / 0.826 position-1 on
short-context work. The synthetic numbers measure the system at a prescribed
acceptance; they are not evidence the draft predicts well at 100k+ context.
* chore: prepare PR 2380 ingest recovery
* fix: recover PR 2380 ingest
1 participant
Recover the missing official ingest for PR #2380 (
kimik3-fp4-b300-vllm-agentic-dspark).The push-to-main
Run Sweepfor merge commit5ba0b0b(run 30507045519) failed insetup: PR #2380 carried/reuse-sweep-runauthorization, but everysuccesspull_requestrun on that branch was a no-op reuse-gate run with zero result artifacts, sofind_reusable_sweep_run.pycould not auto-resolve a source. No GPU jobs ran and no ingest was dispatched.This PR reuses the artifacts already produced by pinned source run 30425131777 (attempt 3, head
ed0fe56d), which holds 11bmk_agentic_*result artifacts pluseval_results_all. No benchmarks are rerun.