CUDA Graph decode: device-side position to remove short-context regression (follow-up to #136)

[pekkah]

✅ RESOLVED (2026-06-14)

CUDA-graph decode is now default-on for Gemma 4 (SHARPI_CUDA_GRAPH != "0"), measuring +9–10% at both low and ~1K context. The short-context regression this issue targeted is gone — from the #137 launch fusions + dp4a decode matvec, plus the finding (#142) that the original "27 t/s at depth 0" cliff was a --verbose-prompt full-vocab-sort bench artifact (real matvecs sit at ~90% HBM, no clock cliff). The proposed device-side-int position was not needed: per-token cost is one cuGraphLaunch + a handful of node-param updates scoped to the position-varying nodes (TrackPositionNode / GraphPosKind). Acceptance (no regression, default-on) met. Closing.

---

Follow-up to #136 / PR #139

CUDA Graph decode (PR #139, SHARPI_CUDA_GRAPH, default off) replays the captured Gemma 4
decode region but updates ~168 kernel nodes per token via cuGraphExecKernelNodeSetParams
(one per position-varying op: RoPE q/k, KvAppend, Attention, per layer) before each
cuGraphLaunch. That per-token host overhead is why the win is context-dependent:

Context	graph OFF	graph ON	Δ
short prompt (~20 tok)	27.0	24.4	−9.7%
~1K ctx	42–43	45	+5–6%

(gemma-4-E4B-it-Q8_0, RTX 4070 Ti, interleaved/warm.) When per-token GPU work is small the
SetParams overhead exceeds the launch-collapse savings → net slower. This regression is the
only reason the feature ships default-off.

Proposed fix: device-side position

Store position in a single device int that the five position-varying kernels read, instead
of passing it (and seqLen/window bounds) by value. Then each token costs 1 small
cuMemcpyHtoDAsync + 1 cuGraphLaunch — no per-node SetParams at all. Expected to:

• remove the short-context regression (host overhead becomes O(1)/token, not O(layers)),
• likely widen the long-context win,
• make graphs a net win across contexts → candidate to flip the default to on.

Cost / risk

• Edit 5 NVRTC kernels (rope, rope_neox_with_factors, kv_append, attention,
  attention_swa) to take const int* for position-derived args and compute the derived
  quantities (seqLen = *pos + 1, SWA window bounds) in-kernel.
• These kernels are shared with the non-graph / prefill paths, so the by-value entry must be
  preserved (or a uniform indirection adopted) — bit-parity risk against the direct-launch
  path. The existing Gemma4CudaGraphParityTests bit-identical oracles will catch any drift.
• The win is still capped by the memory-bandwidth ceiling (~504 GB/s; decode reads 8 GB of
  Q8_0 weights/token), so don't expect more than low-double-digit %.

Acceptance

• No short-context decode regression vs direct launches (within noise).
• ≥ the current +5–6% at ~1K ctx.
• All graph bit-parity oracles stay green.
• If both hold, evaluate flipping SHARPI_CUDA_GRAPH to default-on.

[pekkah]

Deprioritizing this. A head-to-head vs llama.cpp (b9529 cuda-12.4) on the same RTX 4070 Ti /
gemma-4-E4B-it-Q8_0 shows our decode is ~1.8× behind even at 1K ctx (42–45 vs 78 t/s,
flat for llama.cpp) — see #142. Device-side position would, at best, recover the short-ctx
regression and a few % more, which doesn't close a 1.8× gap.

Per "don't optimize a path that won't reach competitive speed," the decode effort belongs in
#142 (fused flash-attention + tighter dequant-dot matvecs), and the dominant gap is prefill
GEMM (#141, ~70×). This stays as a possible secondary lever for #142's low-context clock
cliff (it does attack the launch-gap idle), but only after the structural kernel work — not on
its own.