perf(cuda): batched-trunk prefill for the pure-attention layer-split hybrid (CudaHybridForwardPass / Qwen3-Coder-30B)

[pekkah]

Follow-up to #119 (PR #120). #119 extended the batched-prefill trunk to the dense and GPU-SLRU configs of CudaHybridGdnForwardPass (the all-GPU-trunk GDN-hybrid class). But Qwen3-Coder-30B-A3B — named in #119 — is a pure-attention MoE (no GDN layers) routed through a different class, CudaHybridForwardPass (the -g N layer-split hybrid: N attention layers + MoE on GPU, the rest on CPU). That class has no batched-prefill path at all — it prefills token-by-token via its own Forward.

So the Coder-30B CUDA row (and any other model on CudaHybridForwardPass) still pays per-token prefill: per-position attention launches + per-token MoE.

Scope

• Give CudaHybridForwardPass a batched prompt-prefill analogous to CudaHybridGdnForwardPass's: GEMM-batched attention projections (q/k/v/o, norms, RoPE) + batched KV-append + batched-query SDPA (shared-scores ≤4096 + wave >4096, reusing the perf(engine,cpu,cuda): remaining GDN-hybrid prefill headroom after #111/#112 (N-input MoE dots + per-position recurrence/SDPA batching) #114-B/perf(cuda): wave-based batched-query SDPA for >4096-token GDN-hybrid prefill (#114-B follow-up) #118 kernels) over the N prompt tokens, with the MoE/FFN stage swapped per the layer-split placement (GPU layers batched; CPU-offloaded layers per-token or CPU-batched).
• The attention kernels (llm_full_seq_attention(_global)(_bf16), llm_kv_append_batched, MatMulBatched GEMM-N, batched norms/RoPE) already exist and are backend-shared, so this is mostly host wiring + a parity oracle.
• Keep the _faulted non-transactional latch + bit-exactness contract.

Tests

• BatchedPrefill_BitwiseMatchesSequential for a CudaHybridForwardPass model (Qwen3-Coder-30B-A3B CUDA, -g -1), single + multi-chunk + >4096.

Related / out of scope

• Vulkan and CPU-only GDN-hybrid backends still prefill per-token (no batched kernels) — would need their own batched implementations; track separately if wanted.

[pekkah]

Closing as completed — the in-scope deliverables shipped and are green on master:

• CudaHybridForwardPass.PrefillBatchedTrunk gives the pure-attention layer-split hybrid (Qwen3-Coder-30B-A3B) a batched attention trunk (RmsNormBatched → MatMulBatched q/k/v → batched QK-norm/RoPE → KvAppendBatched → AttentionBatched/Wave → batched o-proj + residual), per the scope, with the FFN/MoE stage per-token and CPU layers per-token over the N rows.
• The requested parity oracle exists and passes for all three cases: BatchedPrefill_BitwiseMatchesSequential_Coder (single-chunk), ..._MultiChunk_... (startPos>0), and ..._Over4096_... (wave SDPA). Plus the _faulted latch + bit-exactness contract.
• perf(cuda): batched-trunk prefill falls back to per-token when the embedding is CPU-resident — exactly the low-VRAM config (#123 follow-up) #218 (PR perf(cuda): batched-trunk prefill for CPU-resident embedding (#218) #226) extended this to the CPU-resident-embedding (low-VRAM / 12 GB) config, which was previously gated out — bringing that config to ~parity with the GPU-embed batched throughput.

The "related / out of scope" items (Vulkan and CPU-only GDN-hybrid per-token prefill) remain tracked separately. Test-hardening follow-up for the non-transactional _faulted latch is now #227.