Qwen3 GPU inference produces garbage output (architecture not supported)

[noahgift]

Summary

Qwen3 models (e.g., Qwen3-4B) produce garbage output during GPU inference via apr run. Both FP16 and Q4K quantized models produce random token sequences. Qwen2.5 models work correctly with the same inference path.

Reproduction

apr import /path/to/qwen3-4b/ --arch qwen3 --quantize q4k -o qwen3-4b-q4k.apr
apr run qwen3-4b-q4k.apr --prompt "def fibonacci(n):" --max-tokens 32 --json --chat
# Output: gibberish tokens (e.g., "obufæīĢæľīçļĦ...")

Same result with FP16 (no quantization), confirming the issue is in the inference architecture, not the import pipeline.

Root Cause Analysis

Qwen3 config differences from Qwen2:

• attention_bias: false (Qwen2 has QKV biases)
• head_dim: 128 (explicit, not inferred)
• model_type: "qwen3" (not "qwen2")
• Qwen3ForCausalLM architecture class

The realizar GPU inference path likely:

1. Doesn't recognize qwen3 as a supported architecture
2. Falls through to a generic path that incorrectly applies QKV biases
3. Or uses wrong attention pattern (Qwen3 may use different RoPE or attention layout)

Environment

• gx10: NVIDIA GB10, sm_121 (Blackwell), CUDA 13.0
• Import pipeline: streaming sharded import with Per-layer F32 dequantization for CPU inference (32B OOM on 119GB) #478 fixes (quantization + tokenizer + weight tying all verified working)
• Qwen3-4B: 36 layers, hidden=2560, heads=32, kv_heads=8, intermediate=9728

Expected Behavior

Qwen3 models should produce coherent text, matching Qwen2.5 quality for equivalent parameter counts.

Impact

Blocks Qwen3-4B HumanEval evaluation. Currently falling back to Qwen2.5-Coder-7B-Instruct (proven 85.37% pass@1).

[noahgift]

Fix Applied: realizar@a82b9d9

Root Cause: Qwen3-4B has head_dim=128 but hidden_size/num_heads = 2560/32 = 80. All forward paths and APR loaders were hardcoding head_dim = hidden_dim / num_heads, causing Q dim to be 2560 instead of the correct 4096. This produced buffer overflows and garbage output.

Changes (6 files):

1. loader_apr_quantized.rs + loading.rs: Infer explicit_head_dim from Q proj tensor shape; use config.q_dim() for Q/O weight loading
2. cuda.rs: Both forward_cuda and forward_single_cuda_with_cache use config.head_dim()/q_dim()/kv_dim()
3. cuda_layer_weights.rs + clone_layer_weights.rs: Same q_dim fixes for cloned-weights GPU path
4. forward_cached.rs: GQA expansion uses config.head_dim()

Also includes (prior commit df23c21):

• QK norm weight loading from APR format (both HF and GGUF tensor names)
• Per-head RMSNorm application in all 4 forward paths (CPU cached, GPU batch, GPU cached, GPU cloned)

Verification: 15064 tests pass, 0 failures.

Next: Build on gx10, test Qwen3-4B inference coherence.

[noahgift]

Fixed in realizar commits 016bcb9, 0284c3e. Root cause: 25+ occurrences of hidden_dim/num_heads across all attention paths computed head_dim=80 instead of 128 (Qwen3 has explicit_head_dim != hidden_dim/num_heads). Verified: Qwen3-4B Q4K produces coherent output on gx10 CPU inference.