Skip to content

fix(deepseek_v32): add causal-mask fallback so maskless prefill is not bidirectional#667

Merged
inureyes merged 1 commit into
mainfrom
fix/issue-619-dsv32-causal-mask
Jul 5, 2026
Merged

fix(deepseek_v32): add causal-mask fallback so maskless prefill is not bidirectional#667
inureyes merged 1 commit into
mainfrom
fix/issue-619-dsv32-causal-mask

Conversation

@inureyes

@inureyes inureyes commented Jul 5, 2026

Copy link
Copy Markdown
Member

Summary

MLAAttention::forward in src/models/deepseek_v32.rs applied causality to the attention scores only when the caller supplied a mask. Both standard generation paths pass mask == None for prefill (the offline CLI text prefill and the VLM embeddings prefill), so a multi-token prefill attended bidirectionally and every layer wrote future-contaminated K/V into the cache. This is the class PR #618 root-caused and fixed on the deepseek_v3 backbone, which explicitly flagged deepseek_v32 as carrying the same structure. Two DSA-specific facets were exposed as well: the lightning indexer's top_indices received the same None (top-k selection could pick future keys), and apply_sparse_prefill_mask's documented invariant ("pe_scores already carries the causal mask") was false.

Fix

Mirrors deepseek_v3 post-#618: capture the pre-update live length (live_before = cache.live_len()) next to the rope offset, build create_causal_mask(l, live_before) once for any maskless multi-token forward, and route that effective mask to BOTH consumers, the indexer top-k selection and the additive pe_scores mask. Decode (l == 1) stays maskless: every cached or gathered position is causally valid.

Validation

  • Fail-then-pass: the dense prefill causality test fails on the pre-fix code (position 0 of a 2-token maskless prefill differs from the single-token forward by 9e-3) and passes after the fix.
  • The sparse-indexer variant pins the kv_len > index_topk prefill path post-fix; the existing indexer_top_indices_respects_causal_mask test pins that the additive mask steers top-k selection.
  • A decode-after-prefill test pins the maskless l == 1 step; a sanitize test locks that only the MTP trailer at layer_idx == num_hidden_layers is stripped (the fix(models/deepseek-v3): restore last layer, causal prefill, f16 clip #618-class off-by-one, which deepseek_v32 does not have).
  • 49 deepseek-family unit tests, cargo fmt, and cargo clippy --all-targets -- -D warnings are clean.
  • Real-checkpoint limitation, stated plainly: every public deepseek_v32-backbone checkpoint exceeds the 128 GB validation machine (DeepSeek-V3.2 ~350 GB at 4-bit; GLM-5 via glm_moe_dsa ~200 GB at 4-bit), so end-to-end generation could not be re-validated here. The fix is a byte-for-byte mirror of the fix(models/deepseek-v3): restore last layer, causal prefill, f16 clip #618 pattern that WAS validated on real hardware via the deepseek_v3 backbone (Kimi-VL), with the fail-then-pass unit proof standing in for this backbone.

Closes #619

…t bidirectional

MLAAttention::forward applied causality only when the caller supplied a mask, and both standard generation paths pass mask == None for prefill (the offline CLI text prefill and the VLM embeddings prefill), so a multi-token prefill attended bidirectionally and wrote future-contaminated K/V into the cache; the DSA facets were also exposed (the lightning indexer's top-k could select future keys, and apply_sparse_prefill_mask's "pe_scores already carries the causal mask" invariant was false). This is the same class PR #618 fixed on the deepseek_v3 backbone and explicitly flagged here. The fix mirrors deepseek_v3: capture the pre-update live length (live_before) next to the rope offset, build create_causal_mask(l, live_before) once for any maskless multi-token forward, and route that effective mask to BOTH consumers, the indexer top_indices selection and the additive pe_scores mask; decode (l == 1) stays maskless since every cached position is causally valid. Four regression tests: the dense prefill causality test fails before the fix (position 0 contaminated by position 1, max diff 9e-3) and passes after; the sparse-indexer variant pins the kv_len > index_topk path; a decode-after-prefill test pins the maskless decode step; a sanitize test locks that only the MTP trailer at layer_idx == num_hidden_layers is stripped (the #618-class off-by-one). Real-checkpoint validation is not feasible on the 128 GB validation machine: every public deepseek_v32-backbone checkpoint (DeepSeek-V3.2, GLM-5 via glm_moe_dsa) exceeds it; the fix is a byte-for-byte mirror of the #618 pattern that was validated on real hardware via the deepseek_v3 backbone, with the fail-then-pass unit proof standing in here.
@inureyes inureyes added type:bug Bug fixes, error corrections, or issue resolutions area:models Model architectures, weights, loading, metadata priority:high High priority labels Jul 5, 2026
@inureyes inureyes merged commit 26b08bb into main Jul 5, 2026
5 checks passed
@inureyes inureyes deleted the fix/issue-619-dsv32-causal-mask branch July 5, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:models Model architectures, weights, loading, metadata priority:high High priority type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(deepseek_v32): add causal-mask fallback so prefill is not bidirectional (same class as #618)

1 participant