Qwen3.5-4B: DeltaNet conv1d unstable on short prompts — long context works fine

[unamedkr]

Root Cause Found

Qwen3.5-4B's DeltaNet layers produce garbage on short prompts (< ~50 tokens) but work correctly on long prompts (150+ tokens with document context).

Evidence

# SHORT prompt — FAILS (even first call, fresh state)
"What is 2+2?" → "The answer to **"  # wrong

# LONG prompt — WORKS (4/4 correct)
"Document: Acme reported 847M...\nQuestion: What was revenue?" → "847 million" ✅

Hypothesis

DeltaNet's conv1d layer (conv_width=4) needs sufficient input tokens to initialize its recurrent state (conv_state buffer). With short prompts, the conv buffer contains uninitialized/zero values that produce unstable attention patterns.

Impact on RLV

The coherence check prompt ("Is this answered? YES/NO") is short (~40 tokens) and triggers this bug, causing:

• False UNSURE verdicts
• Unnecessary retries (3x slower)
• "empty response from server" when the model generates only template tokens

Proposed Fix

Option A: Pad short prompts with a warm-up prefix:

if (n_prompt < 50) {
    // Prepend neutral tokens to warm up conv_state
    prepend("The following is a question and answer.\n");
}

Option B: Initialize conv_state with a learned warm-up pass:

// Run a dummy forward pass with padding tokens before real generation
for (int i = 0; i < conv_width; i++)
    tq_forward(model, state, PAD_TOKEN, i);

Option C: Zero-initialize conv_state explicitly before each generate call (already done by calloc, but verify the state isn't being reused).

Environment

• Model: unsloth/Qwen3.5-4B-GGUF (Q4_K_M, 2.6GB)
• quant.h: latest main (e12fcbd DeltaNet fix)
• OS: macOS 15 (Apple M3, 16GB)

[unamedkr]

Root cause narrowed: NOT double-quantization

Tested with TQ_NO_Q4=1 (no Q4 conversion at all, pure GGUF on-the-fly dequant). Same failure:

TQ_NO_Q4=1: "What is 2+2?" → "The answer to **" (FAIL)

Also tested skipping DeltaNet Q4 re-quantization specifically (keeping FP32 weights). Segfault due to memory management, but the underlying issue persists.

Confirmed NOT the cause:

• ❌ Q4 double-quantization
• ❌ Memory corruption from Q4 conversion
• ❌ conv_state initialization (calloc zeros are correct for left-pad)

Remaining suspects:

1. DeltaNet delta_state accumulation: the recurrent state S_t = decay * S_{t-1} + beta * (K^T @ V) may diverge with short sequences because the decay factor hasn't stabilized
2. L2 normalization on tiny vectors: short prompts produce sparse Q/K vectors that L2-normalize to large values
3. The FP32 dequant of GGUF type 13 (Q4_K_M) loses critical DeltaNet-specific weight patterns: Q4_K_M uses super-block structure, dequant to FP32 is exact, but the original quantization may have lost DeltaNet-sensitive weights

Next step:

Compare Qwen3.5-4B outputs with llama.cpp (which also uses GGUF) on the same short prompts. If llama.cpp also fails → GGUF quantization problem. If llama.cpp succeeds → quant.cpp DeltaNet implementation bug.

[unamedkr]

Deep analysis: DeltaNet forward pass code review

Reviewed the complete deltanet_forward (lines 13776-14100 in quant.h):

Verified correct:

• conv1d implementation (causal_conv1d_silu_batch): correct left-padding with zeros
• Delta state update: S = decayS + beta(K^T@V) — matches paper
• L2 normalization of Q/K — correct per-head normalization
• Output gate (swish(z) * output) — correct
• Group norm (RMSNorm per head) — correct

Weight loading verified:

• DeltaNet QKV loaded from blk.N.attn_qkv.weight (Q5_K, type 13) → FP32 dequant ✅
• DeltaNet gate from blk.N.attn_gate.weight → FP32 dequant ✅
• DeltaNet norm from blk.N.ssm_norm.weight → FP32 dequant ✅
• Then all re-quantized to Q4 at load time (tq_quantize_weights_q4)

Q5_K → FP32 → Q4 double quantization confirmed NOT the cause:

• TQ_NO_Q4=1 (no Q4 conversion) produces same failure
• The GGUF on-the-fly path uses original Q5_K weights directly

Remaining hypothesis:

The DeltaNet recurrent state needs multiple tokens to "warm up" before
producing meaningful output. With only 9 prompt tokens ("Hi"), the
state hasn't accumulated enough information. With 150+ tokens (document
context), the state is well-conditioned.

This may be a fundamental characteristic of DeltaNet architecture
rather than an implementation bug — linear attention models need
more context to build useful state compared to full self-attention.

Recommendation:

For RLV with Qwen3.5-4B, ensure all LLM calls have sufficient
context (≥50 tokens) by prepending document context to short prompts
like the coherence check.