Mirai S on 12 GB: bundle 20261014

mirai-s-ada, first release (bundle-20261006)

Mirai S 27B (Mirai Labs' Qwen3.8-27B-S, Qwen3.8-27B in their Mirai S trellis codec, in alesha-pro's GGUF conversion and llama.cpp port,
11.17 GB) served on an RTX 4070 12 GB with:

• the model's full 262,144-token window at q8_0 KV (tiered cache: ~44k positions in VRAM, the rest in pinned
  system RAM, bit-identical to an all-VRAM cache; alesha-pro's reference fork fits 64k at q8_0 on the same card),
• MTP speculative decoding from the GGUF's own draft block at every depth (outputs identical to drafting off):
  75.8 / 71.4 / 40.3 / 16.6 / 10.3 tok/s at 0 / 16k / 60k / 120k / 180k of context (the fork: 40 / 38 / 33.5 at 0 / 16k / 60k),
• prefill 1,090 tok/s on a 16.8k-token prompt (one int8 activation plane for the FFN matmuls of prompt tokens at
  KL 0.00028, a packed 1-bit attention mask, level-decode chunking; 2048 micro-batch mode 1,148),
• harness-proofing: apps that send effort: "high" or tiny output caps get answers instead of template errors,
• the optional layer (exact API cards, an API check that now flags names looked up on classes and fresh
  instances, a sandboxed Python tool).

Quality, measured with plans and gates frozen before results (docs/REPORT.md):

• HumanEval 164, greedy, tests executed in a sandbox: 158 at medium (budget 20k or 8k: identical), 158 at
  effort "low" for 0.71x the tokens, 154 thinking off.
• Long exact-work suite, 37 tasks, paired, same seeds: raw 18, behind the layer 28 (12 rescues, 2 losses).
  Coding family over two seed sets (24): raw 7, layer 12 at medium (tar 1 -> 5 of 8 is the API-check family;
  MIME 0 of 8 in both arms). At effort "low" (seeds 1-4): raw 6, layer 8 of 12; MIME raw 4 of 4.
• Recommended: defaults for chat and anything behind the layer; reasoning_effort: "low" for coding agents that
  run their own tools against the raw server (launch with MIRAI_EFFORT_ALLOWED=low,medium).

What did not work, kept in the record: overlapping the level decode behind the GEMM (the int8 GEMM already runs at
the card's tensor peak), a round-countdown note for tool loops (read and ignored), removing the layer's
"run it on the example" sentence (7 -> 4 of 12 without it: it stays), and every layer variant on the MIME task
(0 of 4 at every setting; only the raw server at effort "low" passes it, 4 of 4).

Install: unzip mirai-s-bundle-win-x64.zip into the repo, put the GGUF in models\, run start-server.ps1.
Binaries: sm_89 (RTX 40), CUDA 13 runtime included, NVIDIA driver only. Engine: professorpalmer/llama.cpp-ada-mirai
(PrismML's llama.cpp fork + the serving patches, also open as PrismML PRs #319-#323, + Mirai Labs' codec as alesha-pro ported it to ggml in
alesha-pro/llama.cpp-mirai-s and verified greedy token-for-token against it).

Credits: Mirai Labs (the model, its quantization and the Mirai S codec), Qwen (the base model, Qwen3.8-27B), alesha-pro
(GGUF conversion, the llama.cpp port of the codec, the reference fork), PrismML (the llama.cpp fork), sudoingX (planar
activations, batch-invariant mode, PrismML PR #218). Not affiliated with any of them.

bundle-20261006b (same day)

• Engine f11c75618: a drafter graph that borrows the Mirai head (DFlash, DSpark) now finds the codec's tensors through
  the target model; the earlier binaries aborted at drafter load.
• MIRAI_SPEC_TYPE=dflash: ggml-org's DFlash drafter for Qwen3.8-27B at draft 3, +13% decode at depth 0 and +8.5% at
  16k with identical outputs, for ~590 MiB of VRAM; a mode, the MTP block stays the default (receipt:
  receipts/mirai-port/dflash_probe.log).

bundle-20261006c (same day, evening)

• VRAM line 45,312 -> 57,856 positions on this card at the same 800 MiB margin, from two measured levers
  (receipts/mirai-port/mtp_q4_probe.log, docs/REPORT.md 6e): one transient CUDA pool for the target and draft
  contexts with a 256-token draft micro-batch (188 MiB; MIRAI_SHARED_POOL=0 reverts), and tooling/requant_mtp.py,
  which writes a copy of the GGUF with only the MTP draft block at Q4_0 (204 MiB; the launcher prefers the copy when
  it is in models\). Greedy identity 3/3 in every arm and 5/5 on the restarted product, decode unchanged
  (76.5 / 71.9 tok/s at 0 / 16k), acceptance 77.7% vs 78.2%. Decode by depth on the new line: 76.6 / 72.6 / 61.7 / 19.5 / 11.4 tok/s at
  0 / 16k / 60k / 120k / 180k (68.8 at 32k, 66.4 at 48k); 60k was 40.3 when it sat past the line.
• The launcher accepts the 10.96 GB copy (its size guard was set for the 11.17 GB published file).
• Housekeeping found while packaging: the sandbox canary and the FFN-gate driver carried absolute paths from the
  development machine; bench/lib.sh now derives the repo root from its own location.
• Not changed: K/V stays q8_0 (a q4_0 cache is a knob until its quality on this model is measured); the MTP block
  stays the default drafter (DFlash remains the MIRAI_SPEC_TYPE=dflash mode).
• The launcher supervises the server. One 180k-token request ended llama-server with an assert-style fail-fast
  once on the new defaults and did not reproduce in four replays (docs/REPORT.md 6e). An aborted server is now
  restarted (MIRAI_RESTARTS, default 3 within ten minutes) and MIRAI_STDERR_FILE keeps its raw stderr so the
  next assert leaves its text; tooling\stop.ps1 stops it without a restart.
• Engine be00bbdc1: alesha-pro's control-vector projection mode (--cvec-mode project, off unless passed; greedy
  identity against the fork's dump re-checked here 5/5 without the flag) plus a bounds guard on its per-layer flag.
• Linux: start-server.sh and the Ubuntu 22.04 / RTX 3090 check (build line, identity 5/5, decode by depth) from
  alesha-pro's pull request; the layer on Linux is not wired.
• Removing refusals, optional: alesha-pro's refusal-direction control vector and his measurements, in
  docs/CONTROL-VECTOR.md with a short README pointer; off unless the file is passed, not measured on the suite here.

bundle-20261007 (fixes from the first user reports on the sister Bonsai serve)

• Engine 9b28b6057: a JSON schema with an empty anyOf / oneOf / type list no longer fails the request with
  "failed to parse grammar"; the empty keyword is dropped and the rest of the schema converted (reported by Milor123
  against the Bonsai serve, which shares this code). Upstream llama.cpp rejects such schemas with a clear error since
  its September rewrite; the fork this engine is built on has not synced it yet.
• Layer: a response cut by the token limit is no longer run as a sandbox tool call (it is handed back with
  finish_reason: length); the "run the program on the example" sentence is only added when the request asks for
  code, so agents that offer run tools on every turn no longer get it on turns without a program.
• Launcher: warns when the GPU is already busy before the server starts. An app working on the card (found with
  KDE Connect) time-slices the GPU and costs MTP drafting about a third of its speed even when it uses no VRAM.

bundle-20261008 (lookup drafting, keep-alive, clearer launcher messages)

Same model file. New engine binaries (0aad5de56, sm_89) and a new launcher and layer.

• Lookup drafting, on by default. A lookup drafter drafts text that is already in the context, up to 32 tokens
  at a time, in front of the model's own draft block. When the answer copies the context (file rewrites, edit calls,
  quoted logs), decode is much faster. On new text, the speed is the same. The output is the same in every arm we ran
  (receipts/lookup_ab_mirai.jsonl):

  RTX 4070, tok/s	Before, 4k	Now, 4k	Before, 130k	Now, 130k
  Rewrite a 150-line file	86.5	265.6	20.1	84.9
  One edit tool call	85.1	103.8	19.8	24.9
  New text	75.6	75.7	17.4	17.3

  At 130k, most of the cache is in system RAM and each step costs more, so one long accepted draft saves more.

  MIRAI_LOOKUP=0 turns it off; MIRAI_LOOKUP_N sets the limit (32 is best for edit calls). Engine option:
  --spec-lookup-n-max N, a separate draft limit for the lookup drafters, so the draft block keeps its own small
  draft size. Idea credit: syv-ai/HyperQwen, which drafts out of the prompt for the same reason on vLLM.

• SSE keep-alive in the layer. While the server sends nothing (a long prefill), the layer sends a : keep-alive
  comment every 15 seconds, so proxies and tunnels do not close the stream. Clients ignore it.

• Launcher messages. At start, the launcher lists every MIRAI_*, LLAMA_ARG_* and GGML_* variable that is set
  in the window, and it says when drafting is off and why. It turns the engine's automatic fit off, because it sizes
  the VRAM itself (the "failed to fit params" warning is gone).

• Linux build: the quick start uses -DCMAKE_CUDA_ARCHITECTURES=native and lets cmake find CUDA (issue #3, from
  Gotoro).

bundle-20261014 (sparse reads past the VRAM line, faster long prefill)

Same model file. New engine binaries (engine commit 784986b41).

• Sparse host-tail reads, opt-in: MIRAI_SPARSE=1. Past the VRAM line, a decode step normally reads every K/V
  row in system RAM over PCIe. With sparse reads, each attention op reads the RAM pages whose stored int8 bounds give
  the highest possible score for the current query (at least 12,288 positions or 1/8 of the RAM part, plus the last
  1,024 positions) and skips the rest. RTX 4070, 262k window: mean decode at 192k 11.7 -> 16.2-17.0 tok/s, 128k
  about +20 %. Exact code copies and fact lookups (66 items at 160k / 200k / 245k) gave the same answers as the full
  read. A thinner budget (1/16 of the RAM part) was faster (20.9 tok/s at 192k) but got one digit of one fact wrong at
  245k, so the launcher uses 1/8. The bounds...