The default cuBLAS prefill attention has per-call overhead (handle setup, ~70
batched-GEMM + softmax launches/layer, N×N score-buffer traffic) that dominates
at small sequence length. genie now dispatches the fused F32 query-tiled kernel
below a ~768-token crossover (cuBLAS's tensor-core GEMMs win at long context).
Prefill tok/s (cuBLAS-only → length-dispatch), Gemma 4 E2B on Orin
| length | before | after | Δ |
|---|---|---|---|
| ~76 | 224 | 461 | 2.05× |
| ~236 | 475 | 690 | 1.45× |
| ~547 | 587 | 632 | 1.08× |
| ~1547 | 640 | 639 | — |
| ~9098 | 537 | 537 | — |
Typical 50–300-token interactive prompts prefill ~1.4–2× faster (TTFT); long
context unchanged. vs llama.cpp real cold prompt-eval (llama-simple), genie
short-prefill is 2–7× faster (llama pays a ~1–2 s warmup genie does not).
JLLM_ATTN_SHORT_TILED=0 reverts.
Full Changelog: v1.3.0...v1.3.1