Carnice prefill toward llama.cpp 676 t/s — per-layer rewrite levers (op-offload follow-up to #387)

[pekkah]

Follow-up to #387, which took Carnice (Qwen3.6-35B-A3B-MTP, CPU-MoE) prefill from ~130 → 185.9 t/s (+46%) via the GPU op-offload. This issue records the measured ceiling analysis and the remaining levers to approach llama.cpp's 676 t/s pp512, so future attempts don't re-derive it or retry dead ends.

Baseline (RTX 4070 Ti + Ryzen 9 7900X, same hardware as llama.cpp)

llama-bench -m Carnice.gguf -ngl 99 -ncmoe 99 -fa on:

• default (op-offload on): 676 t/s pp512 / 56.9 tg128
• -nopo 1 (MoE matmul forced on CPU): 282 t/s → so ~2.4× of llama.cpp's lead is GPU op-offload, ~2.2× is a faster CPU kernel.

After #387: op-offload (SHARPI_MOE_GPU_PREFILL=1) = 185.9 t/s (vs our CPU path 127.6). Still ~3.6× under 676.

Why 676 needs a per-layer rewrite, not incremental levers

The op-offload is near its per-layer-serial floor: the residual stream is sequential across the 40 layers (layer L's combine must finish before L+1's trunk reads it). Per-token breakdown @ ~245 tokens (~1700ms total, warm/steady-state):

phase	ms	where
trunk (attn/GDN)	336	GPU, batched
router	376	HOST: matvec + softmax + top-k + shexp-gate, per token per layer
routed-MoE	~720	GPU op-offload; ~600 is the serial floor
combine	~15	host

llama.cpp does ~1.5 ms/tok; we do ~7 ms/tok. Closing it needs the per-layer compute ~3.6× faster.

Remaining levers (ranked)

1. Router-on-GPU (biggest single, 376ms ≈ 22%). Move the router matvec (norm[N×embDim] × routerW[numExperts×embDim]) + softmax + top-k + shexp-gate onto the GPU. Hard part: the expert bucketing (expStart/expTokI/expTokK CSR) feeds the op-offload's per-expert grouping — it must become GPU-side (or the selection round-trips to host). PrefillBatchedCpuMoe router loop ~CudaHybridGdnForwardPass.cs:2425-2442.
2. De-serialize the per-layer residual pipeline. The L→L+1 residual dependency serializes the GPU work; only the weight DMA prefetch overlaps. Investigate whether trunk(L+1) can start before MoE(L)'s combine (it can't read the residual, but its norm/QKV could prefetch), or a software-pipelined multi-layer schedule.
3. Grouped GEMM for the routed experts. Today it's 256 separate per-expert GEMM-N launches/layer (gemm ~256ms). A single grouped/segmented GEMM over all experts (MegaBlocks-style) amortizes launch + improves SM occupancy.
4. CPU decode kernel. Decode (tg) stays on CPU (op-offload is prefill-only); llama.cpp's ggml-cpu-zen4 is ~2.2× our int8 dots there. perf(cuda): GPU op-offload of MoE prefill (default-on, +46%) — close the Carnice gap to llama.cpp #387's VNNI helper is Q3_K-only and argmax-stable; extending VNNI coverage / a repacked tiled CPU GEMM is the decode lever.

Dead ends — do NOT retry (measured, regressed)

• GPU-side combine (move (routed+shared*scale)+resid to GPU, skip the per-layer routed download): regressed (185.9→139.8). The download was NOT the cost — the per-layer residual-serialization sync is inherent; removing the download just moved it. (perf(cuda): GPU op-offload of MoE prefill (default-on, +46%) — close the Carnice gap to llama.cpp #387 reverted it.)
• Async prefetch via UploadBackgroundRawInto: regressed (+560ms) — that path always stages mmap→pinned synchronously. Use a cudaMallocHost buffer + direct cudaMemcpyAsync instead (what perf(cuda): GPU op-offload of MoE prefill (default-on, +46%) — close the Carnice gap to llama.cpp #387 does).
• cudaHostRegister on the file-backed expert mmap: gives async overlap but only ~13 GB/s (not full PCIe). Use cudaMallocHost (copy once at load) → ~26 GB/s. (perf(cuda): GPU op-offload of MoE prefill (default-on, +46%) — close the Carnice gap to llama.cpp #387.)

Measurement discipline (important — easy to get wrong)

• The op-offload's one-time setup (~14 GB cudaMallocHost + copy + GPU-scratch alloc) was moved to load in perf(cuda): GPU op-offload of MoE prefill (default-on, +46%) — close the Carnice gap to llama.cpp #387 (eager EnsureGpuOffloadScratch in the ctor) so a single-turn bench reflects steady-state. Before that, cold one-shot was setup-polluted (79–98 t/s vs steady 186).
• Always warm-bench; [gpu-offload] per-layer buckets exclude alloc/JIT, routedMoE (in [batched-prefill]) includes them on cold runs.
• The new NVRTC kernels compile as one unit (any ForceEagerJit entry compiles all), so JIT is at load.

Key references

• Op-offload: CudaHybridGdnForwardPass.BatchedRoutedExpertsGpuOffload, PrefillBatchedCpuMoe.
• Q3_K GPU kernel: llm_matvec_q3k_gemm_n (CudaTextKernels.cs), dispatch CudaBackend.MatMulBatched.
• Gather/scatter: llm_gather_rows/llm_scatter_rows.
• Pinned buffer: EnsureExpertWeightsPinned, UploadRawIntoAsyncDirect, AllocatePinnedHost.
• CPU int8: SimdKernels.DotQ3K_Q8KS / DotQ4K_Q8KS, native native/cpu_vnni/q8k_vnni.c, Q8VnniInterop.

🤖 Generated with Claude Code

[pekkah]

Prefill chunking ↔ op-offload upload interaction (and which paths reach it)

The op-offload (BatchedRoutedExpertsGpuOffload) re-uploads each used expert's weights to the GPU once per prefill chunk, so the chunk size sets the upload count: bigger chunk → fewer uploads → bigger win; small chunks risk going upload-bound (the upload is the floor).

Which paths actually reach the op-offload:

• CLI single-turn (forward.Prefill over the whole prompt) → one batched call (measured: a 1984-token prompt is a single [batched-prefill] N=1984), so experts upload once per prompt. Best case, and more upload-efficient than llama.cpp. This is where +20% @ ~2K and +46% @ short were measured.
• Library / single-user server (InferenceEngine.GenerateAsync → PrefillChunked) chunks at SHARPI_PREFILL_CHUNK (default 512) → op-offload re-uploads per 512, amortized over 512 not the whole prompt. Still a win, smaller margin.
• Continuous batching (multi-user server) → does not reach it: CudaHybridGdnForwardPass is IForwardPass, not IBatchedForwardPass, so the GDN-hybrid MoE models can't run on ContinuousBatchingEngine. ⇒ default-on op-offload carries no CB regression risk.

llama.cpp reference: -ub/n_ubatch = 512 (physical compute batch, fixed — not prompt-adaptive), -b/n_batch = 2048 (logical). Under op-offload it likewise re-uploads experts per 512-token ubatch, so the CLI's one-shot chunk is actually a better amortization than llama.cpp's.

Optimization to track: upload each layer's experts once per prompt and reuse across the prompt's chunks (rather than per-chunk), so the chunked InferenceEngine path — and any future GDN-hybrid CB support — amortizes the upload over the whole prompt like the CLI path, instead of going upload-bound at small chunk sizes.

🤖 Generated with Claude Code