Repository navigation
Tempo9 1.2.1
Engine update only; the CLI and the Swift packages are unchanged apart from the engine they link.
Long-context prefill on Gemma 4 (and every hd256 / hd512 attention model)
The paged NAX attention kernel now splits the head dimension across simdgroups. Gemma 4's global attention layers (head_dim 512, one KV head) previously held a 32-tile accumulator per simdgroup and ran at about 3 TFLOPS, which was 90% of a 96k-token prefill. Measured on one M5 Pro 24 GB, same GGUF, same prompt (85% of the context, 32-token answer):
| model | prompt | 1.2.0 | 1.2.1 | llama.cpp 0.7.0 |
|---|---|---|---|---|
| Gemma 4 E2B | 24k tokens | 7.1 s | 4.6 s | 10.6 s |
| Gemma 4 E2B | 48k tokens | 22.8 s | 13.4 s | 28.3 s |
| Gemma 4 E2B | 96k tokens | 77.8 s | 43.4 s | 77.7 s |
| Gemma 4 E4B | 96k tokens | 105 s | 68 s | 100 s |
| Qwen3.5-9B | 28k tokens | 32 s | 28.5 s |
Greedy output is byte-identical to 1.2.0 on Gemma 4 E2B, E4B and Qwen3.5-9B. Off-switches for A/B: AS_METAL_FA_MLX_HD512_DS=1, AS_METAL_FA_MLX_HD256_DS=1.
Full write-up: https://www.thinkspread.com/blog/gemma4-e-tempo9-vs-llamacpp/