fix(vision): capture the functional slice_update return in DeepStack and MRoPE-section loops#666
Merged
Merged
Conversation
…and MRoPE-section loops mlxcel_core::slice_update wraps the functional mlx::core::slice_update, which returns a new array and never mutates its src; four loops called it as a bare statement and silently lost every write (issue #650, the same class as the Granite 4 Vision blindness fixed by inject_at_positions). In qwen3_vl.rs and qwen3_vl_moe.rs the DeepStack injection loop dropped the whole per-layer visual refinement (the models ran on the main embeddings alone); the fix captures the return and, mirroring the Granite fix, collapses the contiguous single-image run to one slice_update with a per-row fallback for disjoint runs. In qwen2_vl.rs and paddleocr_vl.rs the apply_mrope section assembly dropped the H and W section writes, so every head dim kept the temporal frequencies and image tokens carried no intra-image spatial position information; the fix captures the return. Four co-located regression tests pin the semantics (deepstack_process must add visual embeds at masked positions; apply_mrope must place per-axis frequencies into their sections). Real-checkpoint validation: qwen2-vl-2b-4bit, qwen3-vl-4b-instruct-4bit, and qwen3-vl-30b-a3b-4bit all localize a left-red/right-blue split image correctly through the CLI; paddleocr_vl shares the identical fixed function with qwen2_vl and is covered by its regression test (no local checkpoint).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mlxcel_core::slice_updateis functional (wrapsmlx::core::slice_update, which returns a new array and never mutatessrc). Four loops called it as a bare statement and silently lost every write — the same class as the Granite 4 Vision blindness fixed byinject_at_positionsin PR #649.src/models/qwen3_vl.rs(DeepStack loop)inputs_embeds)src/models/qwen3_vl_moe.rs(DeepStack loop)src/models/qwen2_vl.rs(apply_mrope)src/models/paddleocr_vl.rs(apply_mrope)Fix
Capture the return. The DeepStack loops mirror the Granite fix: a contiguous single-image run collapses to one
slice_update, disjoint runs fall back to per-row with the return captured. Theapply_mropesites capture the return of each section write.Validation
deepstack_processmust add visual embeds at masked positions (qwen3_vl, qwen3_vl_moe);apply_mropemust place per-axis frequencies into their sections (qwen2_vl, paddleocr_vl). All four fail on the pre-fix code.qwen2-vl-2b-4bit: "The color on the left side of the image is red, and the color on the right side is blue."qwen3-vl-4b-instruct-4bit: "far left is red ... far right is blue."qwen3-vl-30b-a3b-4bit(MoE): "the color on the left side is red, and the color on the right side is blue."paddleocr_vlshares the identical fixed function withqwen2_vland is covered by its regression test (no local checkpoint on the validation machine).cargo fmt, andcargo clippy --all-targets -- -D warningsare clean.Closes #650