Skip to content

v0.1.0-alpha.3

v0.1.0-alpha.3 — Path B: layer-major batched prefill
@ai-hpc ai-hpc tagged this 14 May 17:47 
Validated on Jetson Orin Nano Super 8 GB, Qwen3-4B Q4_K_M, 25 W MAXN SUPER.

| | alpha.2 | alpha.3 | Δ |
|---|---|---|---|
| Prefill | 8.2 tok/s | 15.4 tok/s | +88% (1.88×) |
| TTFT | 2200 ms | 1181 ms | −47% |
| Decode | 7.5 tok/s | 7.5 tok/s | unchanged |
| Output | reference | bit-identical | ✓ |

Shipped: gemm_quant_batched (Q4_K/Q5_K/Q6_K), flash_attention_prefill_batched,
Engine::transformer_prefill, transformer_layer split into attn_compute /
attn_block / ffn_block, ScratchPool::mark / rewind_to, GenStats::ttft_ms,
JLLM_BATCHED_PREFILL env var (default on; =0 to opt out).

Series: PRs #13–#17 (kernels + wiring), #18 (default flip), #20 (docs).
Issue: #12 (closed).

Decode is still per-token at 7.5 tok/s (~13% of LPDDR5 peak). Tracked
for follow-up as Path C in #19.
Assets 2
Loading