perf(cuda): Carnice decode toward llama.cpp ~58 t/s — GPU trunk decode kernels are the lever (6.17ms vs ~1ms/tok)

[pekkah]

Summary

Apples-to-apples on the same hardware + config (-ngl 99 -ncmoe 99, RTX 4070 Ti + 7900X), CUDA llama.cpp does Carnice (Qwen3.6-35B-A3B-MTP, GDN-hybrid CPU-MoE) decode at 58.3 tg128 vs our 31.9 (1.83×). This issue tracks the one real lever.

Decomposition (per token, measured; see branch feat/decode-cuda-graph-trunk)

Using SHARPI_DECODE_TRUNK_ONLY (skip the per-layer CPU-MoE handoff to time the pure GPU trunk) + SHARPI_DECODE_CUDA_GRAPH:

component	ours	notes
GPU trunk execution	6.17 ms	trunk-only + graph-on (40/40 layers captured) = pure GPU exec = 154 µs/layer
trunk CPU-launch overhead	~2.7 ms	eliminated by CUDA graph, but masked by per-layer syncs in full decode
coordination (40 blocking Download syncs + transfers + router)	~3.7 ms	capped — see below
CPU experts (routed dots)	~15.6 ms	RAM-bound (~70 GB/s ≈ DDR5 limit), equal to llama.cpp, not a lever

Why the trunk kernels are the only real lever: the trunk_L → experts_L → trunk_{L+1} chain is strictly serial, so the GPU is unavoidably idle during the CPU experts and the host must wait for the GPU trunk every layer regardless of sync mechanism. A perfect-coordination decode still floors at trunk 6.17 + experts 15.6 + router 2 ≈ 23.8 ms (~42 t/s) — yet llama.cpp does 17.2 ms, below our theoretical best case. So its GPU trunk is ~1 ms vs our 6.17 ms (~6× faster per layer); coordination/graph/sync are all marginal.

Lever: faster GPU trunk decode kernels (6.17 → ~1 ms target)

Even with CUDA graphs (launch latency gone) the trunk is 154 µs/layer of genuine GPU execution vs llama.cpp's ~25 µs/layer → the kernels themselves are slow for single-token work. Candidate work, likely in priority order for Carnice (30 GDN + 10 attention layers):

• Fused / higher-occupancy GDN recurrence decode — 30 of 40 layers; the per-head sequential recurrence (GdnRecurrenceDecode + GdnConv1dDecode) likely has low single-token occupancy. Biggest single target.
• Flash-attention decode for the 10 attention layers (vs separate scores/softmax/AV kernels).
• Better single-token Q4_K/Q6_K trunk matvecs (projections) — verify the dp4a/Q8_1 decode path is optimal at 1 row.
• Profile per-kernel-type first (GDN vs attn vs projection) to confirm where the 6.17 ms goes before optimizing.

Method note

SHARPI_DECODE_TRUNK_ONLY (used to isolate the trunk; produces garbage output so it was kept out of shipped code) can be re-added locally for profiling. The corrected understanding came from running llama-bench apples-to-apples (C:\p\llama.cpp-cuda) + reading ggml-cuda.cu — the local CPU-only llama.cpp builds get 21–23 tg128, which we already beat.

Already shipped / ruled out

• Adaptive MTP: +14% default decode (perf(engine): adaptive MTP — fall back to plain decode when acceptance is low (#388) #398, merged).
• CUDA-graph trunk: byte-identical, +3% (opt-in, branch feat/decode-cuda-graph-trunk) — small because launch overhead is small.
• Router-on-GPU-decode: regressed (-3.5%), rejected.
• Coordination/event-based sync: capped at ~2-3 ms by the serial dependency — not worth a rework.