feat(serving): GDN uniform-offload guard — artifact data forbids partial -ngl (#238) - #2108
Merged
Merged
Conversation
…the per-layer KV-head array forbid partial -ngl (#238, 5090 issue 3) Partial layer offload CRASHES GDN/SSM hybrids: the fused recurrent op cannot span CPU/GPU buffers (node->buffer->buft assertion, registered live from the 5090 against Kimi-K3). The legal serving shapes for such a model are all-resident-on-GPU (expert paging is fine — that's the expert axis, not the layer axis) or route-to-grid. Detection is the artifact's own self-description, never an arch-name match (#70): the per-layer attention.head_count_kv ARRAY containing zeros IS the model declaring recurrent layers. gguf_keys grows the canonical array-form reader (scalar-vs-array is itself the signal — a scalar is a uniform GQA model and must not set the flag), and MoeArchProfile carries uniform_offload_required for every launcher / placement consumer. BigMama's device_fit wire and any future -ngl emitter consume the flag and refuse partial with the remedy named. Test pins both directions: K3-shaped array-with-zeros → required; qwen3moe-shaped scalar → not. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BigMama's registered 5090 issue 3. Partial layer offload crashes GDN/SSM hybrids (fused op can't span CPU/GPU —
node->buffer->buftassertion). Legal shapes: all-resident-on-GPU (expert paging fine — different axis) or route-to-grid.Detection is pure artifact data, never arch-name matching (#70): zeros in the per-layer
attention.head_count_kvarray are the model's own declaration of recurrent layers.gguf_keysgrows the canonical array-form reader (scalar-vs-array is the signal);MoeArchProfile.uniform_offload_requiredcarries it to every launcher/placement consumer — her device_fit wire consumes the flag and refuses partial-nglwith the remedy named.Test pins both directions: K3-shaped array-with-zeros → required; qwen3moe scalar → not. 4/4 in the mod.
Completes my fix side of all three registered 5090 issues (issue 1 = her wire, I review; issue 2 = #2107 merged).
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo