apr serve: matmul_fused.rs:211 panics with 'index out of bounds: len 0' on Qwen3-Coder-30B-MoE F32 weight

[noahgift]

Bug

Inference panics in fused_matmul_f32 at crates/aprender-serve/src/gguf/inference/matmul_fused.rs:211 when serving Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf. Multiple rayon worker threads panic simultaneously:

thread '<unnamed>' (3854672) panicked at crates/aprender-serve/src/gguf/inference/matmul_fused.rs:211:54:
index out of bounds: the len is 0 but the index is 56311808

thread '<unnamed>' (3854680) panicked at crates/aprender-serve/src/gguf/inference/matmul_fused.rs:211:54:
index out of bounds: the len is 0 but the index is 55238656

(Similar panics on 5+ other worker threads, indices 55-56M range.)

Client side observes the symptom as curl: (52) Empty reply from server or, for apr code / apr serve callers, Error: driver error: network error: apr serve: error sending request for url (http://127.0.0.1:N/v1/chat/completions).

Reproducer

apr from latest main (squash ff9d0c996 or b50b7cf21):

apr serve run /path/to/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf --port 19900 --host 127.0.0.1 &
# wait for ready (apr serve ready (4.0s))
curl -sS -X POST http://127.0.0.1:19900/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"hi"}],"max_tokens":10}'
# → curl: (52) Empty reply from server; serve panics on first inference call

Root cause hypothesis

Looking at the matmul kernel:

// crates/aprender-serve/src/gguf/inference/matmul_fused.rs:101-104
if weight.qtype == GGUF_TYPE_F32 {
    return Ok(self.fused_matmul_f32(input, &weight.data, in_dim, out_dim, seq_len));
}

fused_matmul_f32 indexes into data: &[u8] with base = row * in_dim * 4. The panic says data.len() == 0 but the computed index reaches ~56M. So weight.data is empty for a tensor marked GGUF_TYPE_F32.

Qwen3-MoE uses per-expert FFN weights: ffn_up_exps, ffn_gate_exps, ffn_down_exps stored as 3D tensors of shape (n_experts, hidden_dim, ff_dim). My hypothesis: the GGUF loader is registering the parent MoE-tensor as F32 with empty data while the actual data lives in per-expert slices that the matmul caller isn't aware of.

This is distinct from the GPU-side issues #1582 and #1583 (M-GPU-MOE-2.x and M-GPU-MOE-3) — those track GPU throughput + parity tests; this is a CPU apr serve correctness bug.

Empirical evidence

paiml/claude-code-parity-apr M260 dispatched the calibration-and-scale bench against this model; ALL 15 student-side dispatches failed with the same panic. Previous failure mode was the 30s startup-readiness timeout (fixed by #1782); after that fix landed, the next-layer bug surfaced — this one.

Immediate mitigations

1. Defensive guard in fused_matmul_f32 — check data.is_empty() and the size invariant data.len() >= row * in_dim * 4 + remainder BEFORE indexing. Return a clear RealizarError::InvalidShape with the tensor name + expected vs actual byte count. Turns the cryptic panic into actionable diagnostics for the next investigator.
2. Don't recommend Qwen3-Coder-30B with CPU apr serve in docs until the MoE F32 path is wired.

Long-term fix

Wire the Qwen3-MoE per-expert weight slicing through OwnedQuantizedTensor / matmul caller so the F32 path receives the correct expert slice for the currently-routed token.

Cross-references

• paiml/claude-code-parity-apr M260 calibration evidence: evidence/calibration-and-scale/scores.json
• aprender PR fix(orchestrate): #1781 apr serve startup-ready timeout — configurable + size-aware #1782 (timeout fix that surfaced this downstream bug)
• aprender#1583 / M-GPU-MOE-2.x — wgpu helpers + integration + parity test for qwen3-moe-forward-gpu-v1 #1582 — sibling Qwen3-MoE GPU work

[noahgift]

Verified fixed by #1790 on lambda-vector RTX 4090, apr 0.34.0 (main HEAD 2a95205):

$ apr bench /home/noah/models/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf \
    --fast --max-tokens 32 --warmup 1 --iterations 3 --json

{
  "model": "/home/noah/models/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf",
  "tokens_per_second": 1.4,
  "total_tokens": 3,
  "iterations": 3,
  "mean_time_ms": 709.58,
  "passed": false
}
error: Validation failed: Throughput 1.4 tok/s below minimum (spec H12)

No matmul_fused.rs:211 panic — bench completes 3 iterations and exits cleanly on the throughput-validation gate (separate concern: 1.4 tok/s is the perf side, tracked under M-GPU-MOE-3 #1583 PR-3h/PR-4 multi-week scope).

The defensive guard at the entry-point of fused_matmul (#1790) catches the empty/undersized weight case and returns gracefully instead of indexing past len=0. apr bench MoE GGUF dispatch path (#1751) also stable.

Closing as resolved.

[noahgift]

Scope + contract + Option A implementation: #1806

I scoped the work end-to-end and shipped Option A (clean architectural guard) in #1806. Full design at:

• Scope doc: docs/specifications/qwen3-moe-serve-dispatch-fix.md
• Contract: contracts/qwen3-moe-serve-dispatch-v1.yaml

Root cause (5-whys)

1. apr serve chat-completions handler at cuda_chat_backend.rs:564 calls Arc<Model>::generate()
2. → Model::forward() → dense FFN matmul on ffn_up.weight
3. Qwen3-MoE stores per-expert weights under ffn_up_exps.weight; the dense name has zero-byte data
4. aprender#1790's defensive guard catches this + returns RealizarError::InvalidShape
5. The MoE-aware path exists at infer/inference_result.rs:225 but is only wired into apr run CLI, NOT the HTTP handler

Three engineering options

Option	Description	Effort	Discharges
A (this PR)	Detection + clear-error short-circuit at handler entry	1-2 hours	V1_002 only
B (follow-up PR)	Wire infer::run_inference into chat handler; reuse CLI MoE dispatch	1-2 days	V1_001 + V1_003
C (architectural)	Push MoE branching into Arc<Model>::generate() / Model::forward()	3-5 days	V1_001 + V1_003 + cleaner separation

I recommend the staged approach: ship Option A now, defer Option B to a separate PR after CCPA-companion validates the clean error class. Option C is a future architectural cleanup.

What #1806 ships

• guard_qwen3_moe_dispatch() at cuda_chat_backend.rs — early-return with NOT_IMPLEMENTED for qwen3_moe arch
• is_qwen3_moe_arch() testable predicate
• 5 unit tests covering canonical + HuggingFace class names + lowercase variants + dense-arch negatives + unknown-arch negatives
• Contract YAML registering 4 falsification gates (V1_001..V1_004); this PR discharges V1_002
• Scope doc documenting the 3 options + companion-side integration plan

Companion-side impact

paiml/claude-code-parity-apr is SUSPENDED at M280 pending this fix. After #1806 merges, the companion can re-dispatch Phase 6 against Qwen3-Coder-30B-MoE + observe a clean moe_dispatch_not_implemented driver-error class instead of the previous opaque matmul/InvalidShape class. Meaningful CCPA measurement (compliance_cost_ratio > 0) still requires Option B.

Will leave this issue open until Option B lands.