fix(ENG-EXPERT-STREAM): fall back to host BF16 when FP8-native device arm refuses - #3141
Merged
Merged
Conversation
… arm refuses The per-expert dispatch chain in RunGemma4Moe had an else-if branch for `fp8_native && ex.is_fp8` that called EnsureGemma4Fp8NativeOnDevice. When that returned false (every non-ROCm build, including CPU), the branch set neither fused_mix nor y: the expert computation was skipped entirely, and the accumulator received whatever the thread-local y scratch held. Without TSan the scratch contained non-zero garbage from a prior allocation, so the test passed by accident. Under TSan the allocator zero-initialises fresh pages, y was all zero, and REQUIRE(AnyNonZero(want)) failed (issue #2909). Add an else inside the fp8_native branch that performs the same host BF16 dequant + ExpertGeGLUHost fallback the `!fp8_native` sibling below already uses. The fix is minimal: it does not change the dispatch priority, the device path, or any arm that succeeds. Closes #2909 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki]
mudler
added a commit
that referenced
this pull request
Sep 11, 2026
Fixes the windows-msvc and role-discipline failures that blocked all open PRs since 57220ba. The macos-metal-mlx, sanitize-cpu address, and sanitize-cpu thread failures were already fixed by #3136, #3138, and #3141. windows-msvc: kv_cache_utils.cpp used fopen with "re" mode which MSVC does not support; build-windows-release.ps1 needed a dumpbin path fix role-discipline: commit 96c5e47 reached main without a row/ branch. Advances the enforcement floor from e1b5df1 to 96c5e47, the third advance, forgiving that one unrepairable violation. Re-pins the floor test assertion in lockstep and adds the third-advance section to the spec intake-triage: the spec change surfaced two untriaged _intake records (#558, #664) whose issue numbers appear in the forgiven-commits table. Both are already fixed by the tree; triaged #558 to BENCH-ORACLE-PIN-RECONCILE and #664 to GATE-WINDOWS-PORTABILITY-TARGET-SCOPE so the agent-record gate passes Paths: product (src/vllm/v1/core/kv_cache_utils.cpp), script (scripts/build-windows-release.ps1), checker data (scripts/ci-enforcement-floor.txt), test (tests/scripts/test_check_commit_trailers.py), document (.agents/specs/ci-enforcement-floor.md), record (.agents/issues/BENCH-ORACLE-PIN-RECONCILE/ISSUE-GH-558.md, .agents/issues/GATE-WINDOWS-PORTABILITY-TARGET-SCOPE/ISSUE-GH-664.md) Closes #3135 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki]
mudler
added a commit
that referenced
this pull request
Sep 11, 2026
…ation Commit 96c5e47 reached main without a row/ branch, freezing the agent-record gate: every PR based on current main fails it. Advances the enforcement floor from e1b5df1 to 96c5e47, the third advance, forgiving that one unrepairable violation. Re-pins the floor test assertion in lockstep and adds the third-advance section to the spec. The spec change surfaced two untriaged _intake records (#558, #664) whose issue numbers appear in the forgiven-commits table. Both are already fixed by the tree; triaged #558 to BENCH-ORACLE-PIN-RECONCILE and #664 to GATE-WINDOWS-PORTABILITY-TARGET-SCOPE so the agent-record gate passes. The windows-msvc-cpu job fails at test_openai_api_server.exe (STATUS_STACK_BUFFER_OVERRUN) but is not a required check: PR #3097 merged while it was red. The macos, sanitize-address, and sanitize- thread failures named in #3135 were fixed by #3136, #3138, and #3141. This commit resolves the last blocking failure. Paths: checker data (scripts/ci-enforcement-floor.txt), test (tests/scripts/test_check_commit_trailers.py), document (.agents/specs/ci-enforcement-floor.md), record (.agents/issues/BENCH-ORACLE-PIN-RECONCILE/ISSUE-GH-558.md, .agents/issues/GATE-WINDOWS-PORTABILITY-TARGET-SCOPE/ISSUE-GH-664.md) Closes #3135 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki]
localai-org-maint-bot
added a commit
that referenced
this pull request
Sep 12, 2026
…3142) Commit 96c5e47 reached main without a row/ branch, freezing the agent-record gate: every PR based on current main fails it. Advances the enforcement floor from e1b5df1 to 96c5e47, the third advance, forgiving that one unrepairable violation. Re-pins the floor test assertion in lockstep and adds the third-advance section to the spec. The spec change surfaced two untriaged _intake records (#558, #664) whose issue numbers appear in the forgiven-commits table. Both are already fixed by the tree; triaged #558 to BENCH-ORACLE-PIN-RECONCILE and #664 to GATE-WINDOWS-PORTABILITY-TARGET-SCOPE so the agent-record gate passes. The windows-msvc-cpu job fails at test_openai_api_server.exe (STATUS_STACK_BUFFER_OVERRUN) but is not a required check: PR #3097 merged while it was red. The macos, sanitize-address, and sanitize-thread failures named in #3135 were fixed by #3136, #3138, and #3141. This commit resolves the last blocking failure. Paths: checker data (scripts/ci-enforcement-floor.txt), test (tests/scripts/test_check_commit_trailers.py), document (.agents/specs/ci-enforcement-floor.md), record (.agents/issues/BENCH-ORACLE-PIN-RECONCILE/ISSUE-GH-558.md, .agents/issues/GATE-WINDOWS-PORTABILITY-TARGET-SCOPE/ISSUE-GH-664.md) Closes #3135 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The per-expert dispatch chain in RunGemma4Moe had an else-if branch for
fp8_native && ex.is_fp8that called EnsureGemma4Fp8NativeOnDevice. Whenthat returned false (every non-ROCm build, including CPU), the branch set
neither fused_mix nor y: the expert computation was skipped entirely, and
the accumulator received whatever the thread-local y scratch held.
Without TSan the scratch contained non-zero garbage from a prior allocation,
so the test passed by accident. Under TSan the allocator zero-initialises
fresh pages, y was all zero, and REQUIRE(AnyNonZero(want)) failed (issue
#2909).
Add an else inside the fp8_native branch that performs the same host BF16
dequant + ExpertGeGLUHost fallback the
!fp8_nativesibling below alreadyuses. The fix is minimal: it does not change the dispatch priority, the
device path, or any arm that succeeds.
Closes #2909
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]