v0.1.0-alpha.12 — Path I: INT8 KV cache (default, ~50 % KV memory saved)

Path I — INT8 KV cache (now default)

`--int8-kv` now works directly (was warned-and-ignored in alpha.11) and is the default for both CLI and embedded callers. `--fp16-kv` opts back into full-precision KV.

The flag had been a half-implemented stub since alpha.2 — Path I (six phases I1–I6) wired the per-head scale machinery end-to-end and validated quality on Jetson Orin Nano Super 8 GB before flipping the default.

Headline result

	FP16 KV (alpha.11)	INT8 KV (alpha.12 default)	Δ
KV pool memory	144 MB	74 MB (72 MB body + 2.25 MB per-head scales)	−49 %
Decode tok/s (33-tok prompt + 40 decode)	10.0	9.9	within noise
Prefill (33 tok)	857 ms / 38.5 tok/s	867 ms / 38.0 tok/s	+10 ms (scale-lookup overhead)
TTFT	867 ms	877 ms	+10 ms
Output	reference	sensibly-identical, FP16-ULP-bounded word-level drift	✓

The real win is memory headroom for longer contexts (8 K, 16 K) and concurrent voice/HA on the 8 GB Jetson. Throughput is essentially unchanged at chat-typical contexts because attention is only ~2 % of decode wall at N≈40; the INT8 savings on attention's KV-read bandwidth compound at longer contexts.

Path I series

PR	Phase	What
#63	I1 — per-head scale storage	New 2.25 MB scales region in KVCachePool; INT8-mode-only allocation
#64	I2 — fp16_to_int8 wiring + kernel bug fixes	per-head conversion + scale capture; fixed uninit row_max and nullptr scale_out crash on the formerly-stub path
#65	I3 — per-position scales through flash_attention + remove broken fallback	attention dequant reads k_scales[kv_pos * n_kv_heads + kv_head]; removed the kv_int8 → N-sequential-decode fallback that made alpha.11 hang on medium prompts
#66 (closed, superseded)	I5 — JLLM_INT8_KV_EXPERIMENTAL env gate	engineer-only switch for the I5 quality eval; obsolete once default flipped
#68	I6 — default flip + Path F interop guard	this release

Full umbrella + audit: #62.

Quality eval (the pivot, I5)

Single-prompt sensibility check at temp=0:

Run	Output (key phrase)
FP16 reference	"...high performance, low power consumption, and advanced AI capabilities, making it suitable for edge computing and real-time applications."
INT8 short	"...high performance, low power consumption, and advanced AI capabilities, making it suitable for edge computing applications requiring real-time processing and efficiency."
INT8 long (200-tok decode)	Full coherent paragraph about Jetson Orin Nano Super's hardware/software fit for smart-home LLM serving — comparable structure to the FP16 long-prompt baseline

Identical opening (24 tokens character-identical to FP16), then minor word-level paraphrase of the same point. Coherent, on-topic, no garbage / repetition / off-topic drift even at non-trivial context. v1-track will broaden to multiple prompts + perplexity holdout.

Path F (#45) interop guard

The on-disk format from F4a (#50, v2) doesn't yet carry the per-head INT8 scales that attention needs at hydrate time. Saving INT8 bytes without the scales would silently produce garbage on the next turn. alpha.12 skips Path F save when `kv_int8` is active, with a one-time stderr note. Use `--fp16-kv` if you need persistent KV today.

Path F format v3 (persist scales region) tracked separately as #67 — the follow-up that closes this gap and lets INT8 KV + persistent KV coexist for multi-turn warm-start.

vs llama.cpp (unchanged from alpha.11)

	Prefill (pp18)	Decode (tg64)
llama.cpp `-ngl 22`	17.97 ± 0.65 tok/s	6.33 ± 0.25 tok/s
genie-ai-runtime alpha.12	38.0 tok/s	9.9 tok/s
Δ	+111 %	+57 %

Cumulative since alpha.2

	alpha.2	alpha.12	Δ
Prefill (33-tok cold)	8.2 tok/s	38.0 tok/s	+363 %
Decode	7.5 tok/s	9.9 tok/s	+32 %
Cold TTFT	2200 ms	877 ms	−60 %
Warm-turn TTFT (Path F)	n/a	444 ms (with --fp16-kv)	new in alpha.9
KV pool memory	n/a	74 MB (−49 % vs alpha.11)	new in alpha.12
Cache safety (Path F5)	n/a	1 GB LRU cap	new in alpha.10

Not in this release

• Path F format v3 — persist INT8 scales (#67). The follow-up that restores INT8 KV + persistent KV multi-turn interop. Will land separately.
• Quality eval on a corpus, not just one prompt. alpha.12 ships on the single-prompt sensibility check (alpha-track bar); v1-track will broaden to multiple prompts + perplexity holdout.
• Cosmetic post-hydrate prefill-log over-reporting (Path F4b artifact).