Fix RoPE convention for Qwen3-MoE and other NEOX-family architectures

[pekkah]

Summary

Fixes Issue #6 — Qwen3-Coder-30B-A3B (and any other Qwen/Phi/Gemma/Falcon model) generated <|endoftext|> or <|im_end|> as the first token on many short prompts. Originally suspected as a Q4_K_M quantization artifact, the actual cause was a RoPE convention mismatch.

SharpInference was applying LLaMA-style interleaved RoPE (pairs (2i, 2i+1)) to all architectures. Qwen2/Qwen3 (and Phi, Gemma, Falcon, etc.) require NEOX-style rotation (pairs offset by headDim/2). The wrong rotation produced subtly-incorrect attention output that compounded layer-by-layer until the residual landed in a degenerate region where the LM head predicted EOT with high confidence.

Changes

• ModelHyperparams.IsNeoxRope — dispatched per architecture string from general.architecture GGUF metadata. NEOX list mirrors llama.cpp/src/llama-model.cpp:llama_model_rope_type() (full enumeration: 60+ archs).
• SimdKernels.ApplyRoPECachedNeox — new SIMD (AVX/FMA) kernel for NEOX rotation.
• Shaders.RoPENeox — new Vulkan compute shader for NEOX rotation.
• IComputeBackend.RoPE — added optional bool neox = false parameter; CpuBackend, VulkanBackend, CudaBackend updated.
• ForwardPass.ApplyRope() helper — dispatches between interleaved and NEOX based on _hp.IsNeoxRope. All RoPE call sites updated. Same dispatch added to GpuForwardPass and HybridForwardPass (both GPU and CPU layers).
• Tests — split CpuBackendTests.RoPE_PreservesNorm into two tests (interleaved and NEOX). The existing test was silently validating NEOX semantics because CpuBackend.RoPE was NEOX-only while the engine used interleaved everywhere — that inconsistency is now resolved.
• Diagnostic instrumentation — added SHARPI_TRACE_NORMS and SHARPI_TRACE_ROUTERS env-var-gated logging used to localize this bug. Kept for future debugging; zero overhead when disabled.
• Design doc — replaced the incorrect "Q4_K_M quirk" attribution in Phase 11 with the actual root cause and fix description.

Why is the convention picked automatically?

GGUF has no dedicated rope-type metadata key. general.architecture is the only signal, and llama.cpp itself hardcodes the convention per architecture. We mirror their full mapping. Special rope variants (MROPE for Qwen2VL, IMROPE for Qwen3VL family, conditional GLM4) are explicitly noted as not yet supported and would need their own dispatch + kernels.

Test plan

• Qwen3-Coder-30B-A3B previously-failing WGSL/PBR shader prompt — produces clean output
• Qwen3-Coder-30B-A3B Python fibonacci (V18 from bisection) — produces clean output
• Qwen3-Coder-30B-A3B + TurboQuant (--tq) — produces clean output
• SmolLM2 (LLaMA arch, interleaved RoPE — control) — unchanged on CPU, GPU at 164 t/s decode
• Full test suite: 206/208 pass; 2 failures are pre-existing partial-Llama-3.1-70B file issues unrelated to RoPE
• New RoPE_Neox_PreservesNorm test added; existing RoPE_PreservesNorm reframed for the default interleaved convention
• Not yet covered: GPU/Vulkan path on a NEOX-family model (Qwen3 8B etc.) — MoE GPU is gated by issue Hybrid GPU+CPU path broken for MoE models (GpuMoeFfn) #2, and no Qwen3 8B dense file is checked in locally. The new RoPENeox Vulkan shader compiles and dispatches correctly but has not been end-to-end exercised against a reference.

🤖 Generated with Claude Code