goinfer v0.19.0 — the local UI grew into a real chat app

serve -web used to be a page you could chat on. It is now the way most people will want to use
goinfer: conversations that survive a reload, model output that renders properly, a model you can
find, fetch, load and unload without touching a terminal. Alongside it, the server learned to take
work as jobs and batches, and DeepSeek and Kimi joined the models that run fully resident on CUDA.

Nothing here requires action

No flags were removed or renamed, no formats changed, and nothing needs rebuilding. Three things may
simply start working, or start warning you honestly, where they quietly did not:

• gpt-oss now gives you its answer. Its replies were stopping at the end of the thinking
  channel and never reaching the final one, so you got the reasoning and no response. It now stops
  on gpt-oss's own tokens, taken from the checkpoint's generation_config.json.
• MoE models on CUDA and Metal stay on the GPU more often. The GGUF loaders were quantizing the
  router, which made residency decline and fall back to CPU without saying why.
• Loading a large .gguf now warns you honestly before it runs your machine low on memory.
  The pre-flight check was pricing only the final resident weights, but a plain (non-streamed)
  .gguf load keeps the whole source file mapped in memory for the entire build alongside the
  weights it's building from it — on a 12 GB checkpoint, that's a real ~12 GB the old estimate
  missed. It's accounted for now, so the refusal or warning you get actually matches what your
  machine is about to do.

The browser UI

Start it the same way, and off by default as before:

goinfer-serve -web -model ~/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf

Reading a reply. Markdown renders — code blocks, lists, tables, links — with a copy button on
every code block and every message. A reasoning model's thinking is folded into a collapsed section
above the answer, so copying gives you the answer and not the deliberation. Each reply is labelled
with the model that wrote it and the compute path it ran on, and a divider marks where you switched.

Keeping your work. Conversations survive a reload, including one interrupted mid-answer. They
live in a Chats list you can open, rename and delete; a title starts as your first message and the
model is asked once, in the background, for a shorter one. You can export a conversation as
Markdown or JSON. If you reload while a reply is still being written, the page reconnects to it
rather than losing it.

Steering it. A system prompt box, kept across chats. The full set of sampling controls the API
already accepted — top_p, top_k, seed, stop sequences, penalties. Regenerate a reply, edit one
of your own messages and resend from there, or delete an exchange. A context meter shows roughly
how much of the model's window the conversation is using, warns at 80% and 95%, and explains in
plain words when a message no longer fits instead of returning a code.

Finding and running a model. Search HuggingFace from the repo box. Every file in the listing
carries a coarse fits / tight / won't fit tag for the machine you are on, before a multi-gigabyte
download rather than after. A finished pull offers to load the model then and there. You can see
what is currently resident and unload it to make room for something else.

Working comfortably. Dark mode, with contrast measured against WCAG AA on every surface in both
themes rather than eyeballed. A phone layout that wraps instead of scrolling sideways. An optional
Enter-sends setting, ↑ to edit your last message, Esc to stop. Errors that say what happened and
what to do about it — a full queue, a halted server and a wrong API key no longer read alike.

Attach an image on a vision model, or text and source files on any model — read, fenced and
prepended to your message, with a file refused by name rather than mangled if it is not valid UTF-8
or is over the cap. An HTML code block gets a Preview button, rendered in a sandboxed iframe on an
opaque origin with a strict CSP, so a previewed page cannot reach this page's storage or your API
key, or make a network request.

It is still asset-free and works offline — the page is now a small directory of files rather than
one, but nothing is fetched from the network to render it.

The server takes queued work now

• Fair admission. Requests wait in an explicit queue rather than on a mutex, so a cancelled
  request leaves immediately and a full queue returns a number you can act on.
• Jobs. POST /v1/jobs hands back an id; GET /v1/jobs/{id}/events replays from the beginning
  and then continues live, so a client that disconnects can pick the answer back up. Optionally
  journalled to disk with -job-dir.
• Batches. OpenAI's /v1/files + /v1/batches and Anthropic's Message Batches, both over the
  same job store — so an existing SDK's batch round-trip works against a local model.

One generation at a time per model, as before. This is about not losing work, not about
concurrency.

DeepSeek and Kimi run resident on CUDA

Multi-head Latent Attention is implemented on CUDA — the compressed-KV latent cache, the q-LoRA
bottleneck, the absorbed per-head matvecs and decoupled RoPE. deepseek_v2, deepseek_v3 and
kimi_k2 are now eligible for full CUDA residency rather than falling back.

Gemma 4 dense also gained residency on WebGPU and Metal, with batched-prefill parity and deeper
context support in Metal attention.

A faster forward pass

The shared kernel library moved to aikit v1.44.0, with the recurrent paths given stack scratch
instead of per-step allocation, fused copy-and-normalize, and bulk KV ring reads. A pipelined
KV-only path skips the LM head where the logits are not needed.

All of it preserves exact accumulation order and is bit-identical to what it replaced, with one
exception recorded in the source: the RMSNorm and LayerNorm reductions now sum in four parallel
partials, which reorders float addition. That stays inside the ≥ 1−1e-4 parity bar every family
test enforces.

Smaller things you might notice

• /v1/models and /health now publish context_window and vision per model — the real token
  limit a request is held to, from the function that enforces it, and whether image parts are
  accepted, from the check that otherwise refuses them.
• The startup banner tells the truth about context. It used to print "backend default" when
  -ctx was unset, and echo a requested -ctx even above the model's own maximum. It now prints
  the limit requests are actually held to and what set it.
• --exact-prefill reaches CUDA, which it never did.
• The fit guard prices hybrid, MLA, GPTQ and vision checkpoints correctly, so "will it fit" answers
  are right for more of the model zoo.
• Ministral 3 renders with its own chat template; SmolLM3 and Olmo 3 decline rather than guess.

Plus a long pass of smaller corrections across the decoder, the CUDA and Metal backends, the
tokenizer and the demos.

New to how any of this works?

There is an eleven-chapter primer in the repo, written for someone who knows Go and does not know
machine learning: An inference primer for Go engineers.
Each chapter ends in a measured number from this repo. It is the best starting point if you want to
understand what the engine is doing rather than only run it.

Getting it

Binaries for six platforms are attached to this release, for both goinfer-serve (each with the
backend that platform can actually use) and goinfer-chat, with checksums — plus a model-embedded
goinfer-chat-0.5b per platform if you want a single file that needs nothing else. The 1.5B
embedded tier is opt-in per release, since it is about 10 GB of assets.

go install github.com/townsendmerino/goinfer/cmd/serve@v0.19.0

goinfer is still pre-1.0: the forward-pass and quantization numerics are parity-gated against
HuggingFace and are the stable contract, and docs/api-tiers.md says which
surfaces v1.0 will bind. Until then, any surface may still change.

goinfer v0.18.0 — admin API (cancel/halt/socket), default-ON fit + fast prefill

Changed

• Three default-ON flips that change greedy output on serve/goinfer-chat, none previously
  recorded here (M-53):
  • Metal's f16-MMA batched prefill is now default ON above 512 tokens (§3.2 gate, 2026-09-09).
    Opt out with GOINFER_METAL_BATCHED_PREFILL=0 or --exact-prefill. --metal-fast-prefill is
    now a no-op (deprecated, kept for compatibility — see Added).
  • Metal's fused flash-attention prefill kernel (attention_prefill_fused) is now default ON
    (§3 gate, 2026-09-10), replacing the exact scalar kernel wherever hd%8==0 && hd<=128. Opt
    out with GOINFER_METAL_FUSED_ATTENTION=0 or --exact-prefill.
  • --fit is now default ON (docs/task-gpu-paths-2026-09.md, task-fit-to-hardware.md Phase
    2): an unpinned CUDA context now sizes to real free VRAM headroom instead of a flat 4096
    positions (commonly up to 8192, cuda/resident.go's fitDefaultCtx, or higher); a dense
    .gguf that won't fit resident RAM on CPU gets one automatic --stream-weights retry instead
    of a bare refusal. Never overrides an explicitly-set -ctx/-quant/--moe-cache-slots/
    --stream-weights. Opt out entirely with --fit=off.

Added

• K1, cancel-by-id (docs/task-halt-2026-09.md). A process-wide registry of in-flight
  generations, keyed by the id every response already carried. GET /admin/generations lists
  them; POST /admin/generations/{id}/cancel {"reason":...} stops one from outside the request
  that started it — the token loop exits at its next context check (already there; nothing in
  decoder/ changed), and the response reports it loudly: finish_reason/stop_reason
"cancelled" plus one more named event/field carrying the reason, so a client cannot mistake
  it for a natural stop. Non-streaming responses get a 499 JSON error instead of a 200. Served
  behind the existing -allow-admin gate.

• K2, global halt with no restart (docs/task-halt-2026-09.md). POST /admin/halt {"reason":...} stops every inference route with a 503 {"error":"halted","reason":...}
  before it would take an inflight slot, cancels every in-flight generation (K1's registry), and
  blocks until they have actually stopped (bounded 30s) before responding — the response itself
  carries the measured quiesced_in_ms. POST /admin/resume clears it; the model stays loaded
  either way, so resume is instant. GET /health gains halted/halt_reason/halt_at. Two
  non-HTTP triggers: -halt-file <path> (polled every 250ms — touch halts, rm resumes, no
  HTTP client needed) and SIGUSR1/SIGUSR2. -halt-exit-code N makes any halt exit the
  process with N once quiescence is reached, for a supervisor whose restart policy must not
  undo a deliberate halt. All off by default; existing behavior is unchanged until one is set.

• K5, the admin socket (docs/task-halt-2026-09.md). -admin-socket <path> serves /admin/*
  (load/unload, K1's generations list/cancel, K2's halt/resume, and a new GET /admin/status)
  on a Unix socket instead of the TCP listener — mode 0600, unlinked and recreated fresh at
  start, no -api-key check at all (the socket's file permissions are the auth). When set,
  /admin/* is not registered on the TCP listener at all (a request there 404s, not 403s — the
  surface is not even advertised); -allow-admin keeps its TCP-only meaning otherwise. Default
  suggested path /run/goinfer/admin.sock (~/Library/Application Support/goinfer/admin.sock
  on macOS). Control it with the same binary: serve status|ls|cancel <id> [reason]|halt [reason]|resume, dispatched the same way pull/check already are. Off by default; existing
  behavior is unchanged until it's set.

• --exact-prefill — forces bit-exact prompt ingestion on all backends, disabling both the CPU
  f32-attention fast path and Metal's fast/fused prefill kernels in one flag.

• fit subcommand (fit [-measure]) — reports what --fit would size a model to without
  actually serving it.

Deprecated

• --metal-fast-prefill — superseded by the default-ON behavior above; now a no-op, kept only so
  existing invocations don't fail to parse.

Fixed

• cuda/testdata/glue.ptx (the embedded, driver-JIT'd kernel blob every CUDA resident load
  compiles) was stale by three commits — every CUDA resident build, on any machine, from any
  commit since 7357856 (Cohere/Command-R's G5 row) has been silently declining to CPU.
  7357856 added layernorm_quant to cuda/glue.cu and wired it into cuda/backend.go's
  pipeline table, but never committed the regenerated .ptx alongside it — so BuildResident
  failed at cuModuleGetFunction with CUDA_ERROR_NOT_FOUND for layernorm_quant and declined
  the whole model to the CPU/staged path, unconditionally, for every family, not just Cohere
  (layernorm_quant loads eagerly for every resident model). Two further already-committed
  optimizations to glue.cu (glu_quant/rmsnorm_quant's warp-shuffle maxabs, RoPE's YaRN
  mscale) were caught in the same gap and never actually shipped either. Found while bringing up
  an unrelated feature on a fresh CUDA build — the first one attempted against a full checkout
  since the staleness landed. Fixed by regenerating via cuda/build_ptx.sh glue (no
  toolchain-pinning concern per cuda/testdata/REGEN.md — unlike moe.ptx, glue.ptx is not one
  of the version-pinned audited artifacts). Verified: layernorm_quant now present in the shipped
  PTX, and the full cuda test suite passes (117/0/105, real GPU) with the resident path actually
  reached rather than silently declined.

goinfer v0.17.2 — MaxPositions fix, live-verified memory guard, pull/constrain fixes

The third cold-user run (macbook-arm64, M1 Pro / 16 GB, run 2b against v0.17.1 — a targeted
continuation after run 2's own opencode leg tripped the swap-safety stop rule) and the fixes it
found. docs/task-first-hour.md "Batch 3".

The headline is not the memory guard everyone was watching — it is that Config.MaxPositions was
silently unset for 16 of the 18 GGUF architectures this project supports, which had left the
pre-existing context-length safety checks silently inert for those families for however long
they have shipped, and would have left this release's new guard just as inert on arrival had it
gone unnoticed. Both memory-guard fixes below shipped, went to a live Mac re-run, and were found
to still swap — twice — for a reason narrower each time; the third re-run confirmed the exact
scenario that produced the original swap no longer does, because the guard now refuses honestly
instead. All three live re-runs are recorded, not smoothed over, in the doc above.

Fixed

• Config.MaxPositions (the model's own context ceiling) was never populated for 16 of 18 GGUF
  architecture families — only Phi-3's builder set it, apparently by accident. This silently
  disabled the pre-existing per-request context-length checks for nearly every family this
  project supports, invisibly, for as long as they have shipped: every existing test fixture
  pinned MaxPositions by hand, so the gap was invisible to the whole prior test suite by
  construction. Fixed for all 16, with a new regression gate driven through the real
  architecture-dispatch table that itself caught two further, independent bugs while being
  written (a wrong metadata key in the Llama config builder; a missing required field for the two
  Mamba-hybrid families).
• A resident model's memory guard priced against a fixed fraction of TOTAL machine RAM, not
  what was actually free — so a real, shared machine (a browser and an IDE open, ordinary desktop
  load) could swap hard on a load or a request the guard had rated comfortably within budget.
  Found on a live Mac, twice: the request-time admission check (new this release) and,
  separately, the pre-existing load-time guard and its banner line both had the identical bug.
  Both now price against currently-available memory instead, read live rather than cached. A
  third live re-run confirmed the fix: the exact model/machine pair that swapped now refuses to
  load at all under real desktop memory pressure — cleanly, in under five seconds, with zero
  swapping — because the guard finally measures what is actually true. GOINFER_NO_FIT_GUARD=1
  remains the escape hatch for a machine the guard is wrong about.
• pull rejected the hf:owner/repo:quant reference syntax that --model itself accepts,
  and a quant that exists only as a split (multi-shard) GGUF file matched no candidate at all,
  so the dedicated "this is a split file" refusal was unreachable. Both fixed; the split-file
  refusal now names every shard and, when one exists, the nearest single-file quant that would
  work today.
• constrain.JSONSchema — the compiler for the one schema surface an API caller controls
  directly (response_format) — silently accepted trailing garbage after a valid schema, so
  {"type":"number"}0 compiled as though the schema were just the object. Found by continuing to
  fuzz past where a scheduled CI run had stopped; a second, unrelated bug in the fuzz test's own
  oracle (an inconsistent JSON-number decode that flagged a correct compile as wrong) was found
  and fixed in the same pass.
• The README's Mac cold-start comparison and a go get error message were both a release
  behind. Both now cite the real v0.17.1 numbers/behavior instead of the v0.16.0-era ones they
  still carried.

Added

• docs/integrations/opencode.md — the README has named opencode as a real-agent target
  since the previous batch; no recipe existed anywhere. States plainly that no run in this
  project has yet completed a full opencode tool-call turn end to end, for two different reasons
  on two different machines, rather than implying success nothing has measured.

Full findings, gates, and all three live Mac re-runs:
docs/task-first-hour.md,
docs/measurements/cold-user-2026-09-07-macbook-arm64.md.

goinfer v0.17.1 — the second cold-user run, and its fixes

[v0.17.1] — 2026-09-07

The second cold-user run (nobara-pc, Ryzen 3700X + RTX 2070 SUPER 8 GB, against v0.17.0) and the
fixes it found. Same ritual as v0.17.0's: a stranger, a published tag, no access to the tree —
docs/task-first-hour.md "Batch 2".

Two of the seven findings turned out to need a second look after the first fix shipped internally:
R9's first pass found CUDA's int4 staged path was CPU-only and wrongly generalized that int8/f32
"reach the backend" on every backend — they do not on CUDA or Metal, which have no staged GPU path
at any quant. R7's first pass, unable to verify a download for the checkpoint the run named,
substituted a different real one rather than fabricate a digest — which was the right instinct,
but the original checkpoint turned out to have a real download after all, findable on Hugging Face
directly rather than by grepping this tree's own test fixtures. Both are corrected here, before
release rather than after.

Fixed

• goinfer-chat --version was unanswerable; the released goinfer-serve's --version reported
  a VCS pseudo-version instead of its tag. goinfer-chat gained the same --version/version
  dispatch and unknown-positional error goinfer-serve already had; both binaries' release assets
  now have their version injected via -ldflags -X at build time (the release workflow's ephemeral
  submodule checkout, with R2-follow-on's go mod edit -replace applied, was enough on its own to
  make the VCS stamp read "modified" even on an exact tagged tree). Embedded-tier binaries
  (goinfer-chat-0.5b/-1.5b) now report the quant they were actually baked at, instead of
  --help's unrelated -quant flag default. Verified end to end against real binary builds,
  including the actual edited build-embed.sh run against a real checkpoint.
• CUDA and Metal have no staged (non-resident) GPU decode path at any quant — the "cuda-staged"/
  "metal-staged" banner labels named a path that was never real. Neither backend implements
  decoder.QuantBackend, and each one's own Backend.MatmulBT is a bare CPU call with no device
  dispatch, so a model that is not resident-eligible on cuda/metal was always running fully on CPU
  regardless of quant — confirmed on real hardware, VRAM sampled at 1 Hz through a full request,
  unchanged from idle. The banner now reports this the same way BackendReport() already reports a
  backend that failed to build in the first place (requested cuda → running on cpu: <reason>, and cuda has no staged decode path), matching what docs/hardware-matrix.md's generated table
  already only ever claims (resident or CPU, never a third state). WebGPU keeps a real staged
  path (the only backend that has one) for f32/int8/int8int8, with its own int4 gap noted
  separately. --backend's help text on both binaries states the rule plainly.
• A registry-recommended checkpoint (granite-4.0-h-tiny) loaded with a tokenizer pre-tokenizer
  decline on every pull. Measured the real HF tokenizer.json's Split regex (byte-identical to
  an already-implemented shape, different digit cap and merge-handling than its nearest sibling)
  and added the missing case. A new registry gate reads each entry's committed GGUF-header fixture
  and fails on any pre-tokenizer decline.
• serve check's minimal one-tool schema passed against a server that a real agent then broke
  under its own larger tool schema. Added a second, harness-scale tools row (a dozen tools with
  nested parameters, shaped like a real agent's) that reports a checkpoint too small to hold up as
  a SKIP naming the reason, not a false green. The registry gained a measured tools: column,
  shown in goinfer-chat models, populated from real runs where time allowed and marked honestly
  not yet measured otherwise — never guessed.
• README numbers lacked provenance/network context, and a citation could silently rot. The
  cold-start figure now states its download size and measured network speed; the steady-state
  hedge names the specific packaging defect (R2's Metal-less Mac asset) that produced the old
  comparison instead of reading as an engine result. A new readme-smoke step asserts every
  docs/-relative link the README cites resolves to a real file.

Added

• The recommendation registry gains two real 20-35B-class MoE checkpoints — gpt-oss-20b
  (OpenAI, real-oracle parity, validated resident on an 8 GB card via -moe-cache-experts) and
  gemma-4-26b-a4b (Google's own QAT q4_0 GGUF — the checkpoint docs/benchmarks.md §B4/§B4.1
  already has the most measurements on at this size). Neither the README's prior size-class example
  nor the curated models list had anything a cold user could actually download at this scale.
  Both sha256/bytes are Hugging Face's own git-LFS digest, cross-verified against real local copies
  of each file already on this box. A new readme-smoke marker resolves every README-named model
  reference (registry short name, demo: tier, or owner/repo) against the registry or Hugging
  Face directly, metadata only — currently red for both new entries against the published module,
  by design, until this tag lands.
• docs/task-fit-to-hardware.md now states explicitly that its planner covers slot/context
  placement, not per-layer CPU/GPU placement, and that real hybrid layer placement — the gap a peer
  comparison exposed at this same size class — is a separate, larger, unscheduled item.

Full findings, gates and the run report: docs/task-first-hour.md,
docs/measurements/cold-user-2026-09-06-nobara-pc.md.

goinfer v0.17.0 — seven families, and the first-hour fixes

Seven new model families, and the fixes from the first time somebody who had never seen goinfer
was handed the release and asked to use it.

The families first, because that is most of the diff: Qwen3-MoE (Qwen3-30B-A3B and the Coder
variant — the most-run local MoE of the year, and a hole this project should not have had), dense
Granite 4.2, Ministral 3, SmolLM3, Olmo 3, Olmo Hybrid, and Bailing Hybrid
(Ling 3.0 — the first checkpoint here with Kimi Delta Attention). Each is parity-gated against a
tiny oracle built from the real modeling code, with the departures from its nearest sibling
written down rather than assumed; the real-checkpoint runs for the two 30B-class ones are queued
on the Linux box and the tiers say so.

Then the part we would rather state here than have found: v0.16.0's downloadable Mac and Linux
binaries had no GPU backend in them, and the Mac one had no way to tell you so. A cold user on a
16 GB M1 Pro measured goinfer at 37.9 tok/s against Ollama's 82.3 and had every reason to believe
that was the engine. It was the packaging — the assets were built from an entrypoint that imports
no backend — and the gate written to catch it then found that every release's Mac and Linux
binaries had also been built against the previous release's engine. Both are fixed in the
workflow and asserted on every asset from now on; if you downloaded a v0.16.0 binary, please
download this one. goinfer-serve --version now tells you which backends a binary carries, the
load banner names the backend that is actually executing, and a model that will not fit in RAM is
refused before it is loaded, with the flag that would have made it fit. The protocol that found
all of this is in the repo and is now part of the release pre-flight.

Added

• Qwen3-MoE as a new family (qwen3_moe; Qwen3-30B-A3B / Qwen3-Coder-30B-A3B-Instruct): qwen3's
  QK-norm dense attention with the FFN replaced on every layer by a sparse MoE, and — unlike its
  qwen2_moe sibling — no always-on shared expert, confirmed against the real released config.json
  and a real GGUF's tensor list. Pure composition of two already-shipped adapters: no new forward
  path, so the family rides the generic uniform-layer dispatch (canBatchN/specRollbackSafe answer
  correctly for free) and is resident-admitted on cuda/metal/webgpu from day one, same backends as
  qwen2_moe and qwen3. GGUF support included (general.architecture == "qwen3moe", verified
  against a real file's header via HTTP Range, not assumed). Parity-gated against a tiny oracle
  (100.0% / 1.00000); real-checkpoint T3 is a follow-up (docs/task-families-2026-09.md).

• Dense Granite 4.2 (3B/8B/30B) as a new family (granite; distinct from the existing
  granitemoehybrid Granite-4.0-H). A plain llama skeleton — confirmed byte-identical tensor names
  to llama by instantiating GraniteForCausalLM directly — plus three of Granite's four scalar
  multipliers, all already generic on the shared Architecture descriptor (embedding/attention/
  logits scale); residual_multiplier is rejected unless 1.0, the only value any released 4.2 size
  ships. Reuses llamaTensorSchema verbatim — no new tensor schema. GGUF support included
  (general.architecture == "granite", verified against a real file). Resident-admitted on
  cuda/metal/webgpu from day one (empty feature profile — every scalar that varies from identity on
  a real checkpoint is either baked into the generic attention scale or checked to be 1.0).
  Parity-gated against a tiny oracle with non-trivial multipliers (100.0% / 0.9999999999999).

• Ministral 3 as a new family (mistral3/ministral3; 3B/8B/14B): Mistral's GQA skeleton
  (tensor names byte-identical, reused verbatim) plus two real deltas found by checking the
  release rather than assuming a config alias: no sliding window at all on any released size, and
  YaRN RoPE with an extra field, llama_4_scaling_beta, that scales the query by
  1 + beta·ln(1 + floor(pos/original_max_position_embeddings)) after RoPE, on every layer —
  Llama 4's own attention-temperature-tuning formula, generalized here into two new generic
  Architecture fields (AttnTempBeta/AttnTempOrigMaxPos, 0 = off) and wired into both generic
  forward paths (sequential decode and batched prefill/verify), proven to agree bit-for-bit. A new
  FeatAttnTemp resident-admission flag keeps this CPU-only until a GPU backend implements it —
  no backend declares it, so cuda/metal/webgpu all correctly decline rather than silently dropping
  the scale. Parity-gated against a tiny oracle whose prompt is deliberately longer than its
  original_max_position_embeddings, so the new mechanism is actually exercised, not identity
  (100.0% / 0.9999999999999605).

• SmolLM3-3B as a new family (smollm3): a plain llama-shaped dense GQA model with per-layer
  NoPE on every 4th layer via no_rope_layers — a field whose VALUES are the opposite of what its
  name suggests (1 = has RoPE, 0 = NoPE), verified against the real modeling_smollm3.py rather
  than assumed from the name; getting this backwards would silently flip which 9 of 36 layers are
  NoPE with correct shapes and plausible-but-wrong logits, no crash. Reuses the Config field and
  boolean convention llama4_text already established for the same JSON key, and the same
  layerNoPE Architecture hook cohere2 already populates — no new mechanism, just composed
  onto a third family. Tensor names byte-identical to llama (llamaTensorSchema reused verbatim).
  CPU-only (FeatNoPE is undeclared on every resident backend, same as cohere2). Parity-gated
  against a tiny oracle at the release's own every-4th-layer pattern (100.0% / 0.9999999999999544).

• Olmo 3 as a new family (olmo3; Ai2, 7B/32B): two real departures from every other family
  here, both verified against the real modeling_olmo3.py/configuration_olmo3.py rather than
  assumed. NormPlacement gains a fourth value, NormPostOnly — no pre-norm at all; only the
  attention/MLP sublayer OUTPUT is normalized before the residual add. QK-norm applies to the WHOLE
  projected q/k vector, not per head — a new QKNormWhole flag reuses the existing rmsNorm kernel
  with rows=1, dim=numHeads*headDim instead of rows=numHeads, dim=headDim, so no new math (and
  fixed a latent bug the addition surfaced: the existing FeatQKNorm derivation would have also
  incorrectly required the per-head kernel for a whole-vector family). YaRN scaling applies only to
  full-attention layers, sliding layers stay at plain RoPE — the same local/global RoPE split
  Mellum's own gate already implements, reused with the sliding-layer scaling left unset. Parity-
  gated against a tiny oracle whose sliding/full split and RoPE tables are actually exercised
  (prompt length exceeds the fixture's sliding window): 100.0% / 0.9999999999997883.

• Olmo Hybrid as a new family (olmo_hybrid; Ai2, 7B): the sibling MoE-free DeltaNet+softmax
  hybrid — qwen3_5's Gated DeltaNet on 3-of-4 layers, olmo3's own full-attention shape on the rest.
  Its norm placement differs BY LAYER KIND within one model (full-attention layers: NormPostOnly;
  DeltaNet layers: plain NormPre2) — the first family here where that varies, so Architecture
  gains NormPlacementLinear, a per-layer override keyed on the same layerIsLinear hook that
  already selects the mixer (nil for every other family, unaffected). Every other departure is a
  parameterization of the shared DeltaNet code, not new math: linear_allow_neg_eigval doubles the
  write-gate beta after the sigmoid; q/k/v projections AND the depthwise conv are separate tensors
  rather than qwen3_5's pre-concatenated ones (the conv split's true q/k/v-boundary layout was only
  confirmed by fetching the real checkpoint's safetensors header directly — both the source and a
  local re-save produced different, wrong splits); the output gated-RMSNorm is named o_norm/
  o_proj with a hardcoded 1e-5 epsilon independent of the model's own rms_norm_eps. The released
  checkpoint's rope_parameters is {"rope_theta": null} — no RoPE anywhere, on any layer — handled
  by a new NoPositionEncoding flag naming this as a fourth legitimate "no RoPE table" case in the
  existing position-information guard (alongside GPT-2/Nemotron-H/MLA). Parity-gated against a tiny
  oracle whose fixture reproduces the release's actual tensor layout: 100.0% / 0.9999999999998704.

• Bailing Hybrid as a new family (bailing_hybrid; inclusionAI, Ling 3.0): DeepSeek-style
  Multi-head Latent Attention alternating with Kimi Delta Attention (KDA) every layer_group_size
  layers being MLA, over a DeepSeekMoE FFN. MLA and the MoE router compose deepseekArchitecture's
  existing code, parameterized for two real naming departures (both mixers are self.attention,
  not self.self_attn; MLA's output projection is self.dense, not o_proj) plus an optional
  per-head sigmoid attention-output gate riding the same mechanism Laguna's own gate already ships
  (sigmoid where Laguna's is softplus — a real, checked difference, not an assumption). KDA is the
  one genuinely new primitive: a delta-rule recurrence structurally identical to Gated DeltaNet but
  with a PER-CHANNEL decay (one value per row of the state matrix, where Gated DeltaNet's is a
  single scalar per head) — proven against fla-org/flash-linear-attention's actual reference
  implementation, not the HF modeling file's opaque Triton-kernel call (this was proven ahead of
  time as a standalone rehearsal; see the "Fixed" entry below for what shipping it as a real family
  found). layer_types is not a config.json field for this family at all — the MLA/KDA pattern is
  computed from layer_group_size, replicated exactly from the real decoder layer's own ...

goinfer v0.16.0

This release is two things at once. The headline is prefill: CUDA prompt ingestion moves to tensor
cores and is on by default above 512 tokens, MoE prefill runs expert-major, and the CPU path gets
head fan-out, a fused schedule and aikit's register-blocked int4 tile — so the prefill deficit
against Ollama, the repo's largest open gap, narrows on every backend. Underneath that is a
whole-repo audit (docs/audit-2026-09-02.md) and its review (docs/review-2026-09-04.md)
worked through in a fortnight, plus the first pieces of the
onboarding work: a pull command, a browser UI, a serve check doctor and a startup banner. Two
defaults in this release change output at temperature 0 on long prompts; both are called out below
with their opt-outs.

Added

• LFM2 / LFM2.5 as an experimental family (lfm2; gated short convolution on most layers, GQA +
  QK-norm on the rest). CPU-only — no backend implements the short-conv feature yet — and
  parity-gated against a tiny oracle (100.0% / 1.00000). The bring-up found two silent bugs the
  forward hid (a zeroed NormEps read from the wrong JSON key, an absent AttnScale), which is why
  resolveArchitecture now runs a validateResolved() chokepoint for every family, written or not
  yet. The first cut also fell through seven family-dispatch lists (any prompt of two or more tokens
  panicked) and serialized without its conv weights; both were caught by the audit before the tag
  and are fixed here (audit-2026-09-02.md C-01/C-02/C-03).

• pull — fetch a model without leaving the tool. goinfer-chat pull <owner/repo>[:quant] and
  serve pull fetch a GGUF from HuggingFace, sha256-verified from the tree API, resumable; demo:
  shortcuts carry pinned digests, and a cached demo: ref resolves offline before any network call.
  --model accepts hf: and demo: refs directly. pull -embed bakes a pulled model into a single
  static binary. The package is exported as pull for embedders. HF repo names are validated
  against an allow-list rather than a slash count.

• serve -web — a local browser UI for chat and model pulls, served from the same binary. The
  root page is reachable without the API key so a browser can load it; the pull/list routes behind
  it are same-origin-checked and authenticated like the rest of the API.

• serve check — drives a running server the way an agent harness would (chat, streaming,
  tool call, structured output) and reports each surface; the structured-output check verifies the
  returned value, not only that JSON parsed, and the summary names what it skipped.

• The startup banner now reports the state a harness otherwise has to discover — backend,
  residency and prefill path (with its chunk width), and the declines that change what the server
  can do, including a pre-tokenizer shape this build cannot walk (PreTokenizerDecline).

• gpt-oss decodes GPU-resident on CUDA. The resident gate (G7) ran for the first time and
  failed: CUDA never applied gpt-oss's per-expert down-projection bias, and under expert caching the
  bias table was indexed by slot id rather than expert id. With gemv_w4a8_moe_wacc_bias the parity
  cosine went 0.750 → 0.9993 and the gate is green.

• Release binaries are attached to the GitHub Release (the README's download link previously
  pointed at nothing), the embedded 1.5B tier ships, and NOTICE now lists the Qwen2.5-Coder
  weights those binaries embed, with their licence (audit-2026-09-02.md M-33).

• Peer benchmarking: scripts/bench_peer.py gains llama.cpp as a third engine (with --fit
  left to place MoE layers rather than a forced -ngl 99) and MLX on the Mac; a new
  scripts/bench_peer_prefill.py measures TTFT-derived prefill rate with unique prefixes per
  request so a peer's prompt cache cannot stand in for its prefill. The redone matrix is scoped in
  docs/task-peer-benchmarks.md; its first pass is in docs/benchmarks.md.

• MTP / NextN self-draft head loader (decoder/mtp.go) — reads the head that every existing
  load path skips, by two detection routes because the formats disagree (GGUF declares a count in
  arch-prefixed metadata; the safetensors Qwen checkpoints declare nothing and are discoverable only
  by tensor presence). Measurement adapter only: nothing is wired into serving, the router or any
  generation path.

Changed

• CUDA prompt prefill runs on tensor cores, and is ON BY DEFAULT for prompts of 512 tokens or
  more. Two new kernels — attn_fused (FlashAttention-style attention, mma.sync m16n8k8) and
  gemm_w4a8_mma (int4×int8 GEMM with group scales, mma.sync m8n8k16) — replace the batched
  decode-shaped kernels for prompt ingestion. End-to-end prefill 3.91× faster at a 3900-token
  prompt on a 1.5B int4 (5.451 s → 1.393 s) and 4.10× at 512; against Ollama v0.32.5 the
  overhead-free marginal gap at depth narrows from 12.1× to 3.16× (1.5B) and 14.5× to 1.89×
  (0.5B), and goinfer is now faster to first token at every swept depth on the 0.5B.

  These kernels are NOT bit-identical to decode — the fused attention uses f16 K/V with an
  online-rescaled softmax, and the GEMM re-associates the cross-group float sum. They ship as a
  default only because docs/task-prefill-gap.md §3's fidelity gate passed: both arms scored
  against a CPU reference with f32 weights and f32 activations, teacher-forced on the reference's
  own continuation, over 10 prose prompts per cell on two models. At depth the fast path is
  measurably CLOSER to that reference than the exact path it replaces (hard flips 7 vs 10 at
  K=1024, 8 vs 12 at K=3900). It fails at K=256, which is why the 512-token floor exists and
  why it is 512 — a K=512 reference was generated specifically so the floor rests on a measured
  cell rather than an interpolation.

  GOINFER_CUDA_FAST_PREFILL=0 restores the previous behaviour completely; =attn / =gemm
  select one lever. The exact path stays bit-identical to the M=1 decode kernels and is what
  spec-decode verify and the parity gates run regardless of this setting.

  If you have prefill numbers from before this change, they measured the exact path. In
  particular TestPrefillTTFT's "batched" column now times the fast path at K ≥ 512 without the
  test having changed; set =0 to reproduce the older rows.

• --cpu-fast-attention is now ON BY DEFAULT, floored at 512 prompt tokens, with
  --cpu-exact-prefill as the opt-out. Prompt attention on the CPU backend runs in f32 unless
  you ask otherwise; the opt-out wins if both flags are passed. This changes output. Above the
  floor, prefill is no longer bit-identical to decode, and a long-prompt response can differ from
  what the same build produced before, at temperature 0.

  • The floor is the part that makes it safe to default. Attention cost grows with the square
    of the prompt, so the saving grows with length while the divergence does not — an eight-token
    prompt was measured diverging at the third generated token while buying nothing measurable
    (1.15× at 512, 1.43× at 2048, 2.28× at 8192). Below 512 tokens the exact kernel runs
    regardless, which is why every pre-existing forward golden still passes untouched.
  • It is not reproducible across architectures. The same prompt on the same checkout produces
    a different first token on arm64 than on amd64 — the compiler fuses multiply-add on one and not
    the other, and f32 has no wider accumulator to absorb it. --cpu-exact-prefill is the way to
    get a byte-comparable transcript between machines, and exists for that reason rather than as a
    courtesy.
  • Decode is unaffected either way, and speculative verify is structurally excluded (it passes the
    exact kernel as a parameter, not a runtime check).

• Prefill attention now uses a FUSED (FlashAttention-style) schedule under --cpu-fast-attention,
  and THIS CHANGES OUTPUT. The score block is kept resident and folded into the output
  accumulator with a running max and running sum, instead of materialising a kt × nKeys matrix
  and making three passes over it. A long prompt can now produce a different response than the
  same build produced before, at temperature 0 — the committed long-prompt goldens are
  regenerated on both architectures.

  • What it buys: +8.0% end-to-end (dense 1.5B, K=4096, paired and interleaved). The kernel
    win is much larger — 1.69–1.73× over a whole prefill's tiles — but A3's head fan-out already
    took most of what attention had to give, leaving it ~18% of this prefill.
  • This is a deliberate trade and it is a close one. Eight percent for a user-visible output
    change is a worse ratio than the flip that introduced this flag (1.43–2.28×). It ships because
    the operator chose it after the measurement was presented, not because the number argued for
    itself.
  • The divergence is small relative to what the flag already accepts: measured on one
    checkpoint at one depth, acc64 vs f32-materialized is cosine 0.998283 and acc64 vs f32-fused is
    0.998262 — an increment of ~2e-5. It rides --cpu-fast-attention rather than adding a second
    user-facing flag for that reason. GOINFER_FUSED_ATTENTION=0 restores the materialized path.
  • Declines, rather than approximating, for acc64 (whose bit-identity it would break) and for tree
    attention. See docs/measurements/p19-fused-attention-2026-09-01.md.

• --cpu-fast-attention now covers Mixture-of-Experts architectures. The flag previously
  refused MoE outright, on the argument that an f32 QKᵀ reassociation flips a top-k expert at a
  near-tie and cascades. That argument had never been measured on a MoE — no MoE appears in the
  A3 kernel-ratio record, and both G24 divergence tests load the dense ...

goinfer v0.15.0 — long prompts stopped being the slow part

Act on this first

If you built a .giw bundle for gpt-oss on v0.14.0, rebuild it. The writer accepted gpt-oss and
then dropped its per-head attention sinks, producing a CRC-valid, sink-free bundle that generated
confidently wrong text — no error at write, load, or run time. Fixed in .giw v6, where the tail is
written unconditionally rather than arch-gated, because arch-gating is exactly how the sinks went
missing. Applies to bundles from prequant or serve --stream-weights.

Nothing else in this release requires action.

Long prompts got much faster on CPU

If you feed goinfer a big system prompt, a long document, or an agent transcript, this is the
release you want. Prompt processing on Apple Silicon:

prompt	before	after
1,520 tokens	89.7 s	33.8 s (2.65×)
3,020 tokens	333.3 s	101.6 s (3.28×)

Attention now runs its heads in parallel during prompt processing, where it had been running on a
single core while the rest of the machine sat idle. Measured on an M1 Pro, dense 1.5B, and
bit-identical to the old path — same output, less time.

Very long prompts got a second fix: the per-core scratch used to grow with the square of the
prompt, so past ~8k tokens the parallelism switched itself off to avoid eating gigabytes. It now
works in tiles, so an 8k prompt uses ~16 MB per core instead of 272 MB, and the speedup survives.

Optional, for another 2.28× at 8k: --cpu-fast-attention. Off by default and not
bit-identical — measured cosine 0.9976 against the exact path, so a long-prompt answer can differ
from the default even at temperature 0. Decode is untouched, speculative decoding never uses it,
and it is refused for MoE models outright. The flag's --help carries all of that; the trade is
yours to make knowingly.

Agent and tool-calling fixes

Several of these were found by pointing a real agent harness at goinfer and watching it fail.

• role: "developer" now means system on the OpenAI-compatible routes. Before, it matched no
  case and fell through to a user turn — so a harness sending its system prompt that way had its
  entire agent scaffold delivered as the user's first message, and the result read like a bad model
  rather than a mangled request. Newer OpenAI APIs use this role, and at least one harness sends it
  to any endpoint it does not recognise, which is every goinfer deployment.
• Streaming tool calls no longer go silent. A tool-capable generation must be buffered before a
  call can be parsed, which meant stream: true sent nothing at all until it finished — measured 28
  minutes on one long agent prompt, against clients that give up after 5. Keep-alives now flow
  throughout, and on ChatML/Qwen, Mellum2 and Gemma 4 the prose streams incrementally as it is
  produced.
• An abandoned request now stops. Closing a connection mid-prompt used to leave the server
  processing it to completion — and a client that retries stacked one of those per attempt. We
  measured 47 minutes of CPU burning with nothing connected.
• /v1/messages validates roles. The Anthropic API accepts only user and assistant; anything
  else is now a clean 400 naming the offending role. Previously any typo'd or invented role was
  quietly folded into the conversation as a user turn.

Bigger models, smaller cards

• Qwen3.6-35B-A3B decodes GPU-resident on an 8 GB card — ~20 GB of int4 experts streamed against
  8 GB of VRAM.
• Gated-DeltaNet is GPU-resident on WebGPU and CUDA — qwen3_5 dense, its MoE sibling and
  qwen3_next now resolve to a real runner instead of declining. The CUDA port measured 15.9× over
  CPU decode when it landed on 2026-08-20; that figure is on the pre-2026-08-25 driver stack and has
  not been re-anchored since the box moved to 595.91.07 / Nobara 44, so read it as "large,
  measured then" rather than as a current number. Correctness has been re-verified across the
  bump — 24/24 PTX byte-identical.
• Qwen3.8 GGUF loader — 55.6 GB bf16 down to 16.5 GB on disk (3.4×). A 1.69× decode
  difference against the safetensors path was measured too, but only on a box with 46.8 GB of 62 GB
  already resident; its own record says not to assume it transfers, so it is not claimed here.
• .giw v6 represents every registered family. The list of things that could not be serialized
  is now empty.

Use it with an agent harness

There is now a tested recipe for running a fully local agent stack — DeepSeek Harness driving
goinfer, no cloud, no Python. It was written from an end-to-end run rather than from the docs, so it
tells you the parts that actually bite: the published npx install hangs, the provider config is a
namespaced dict rather than a list, and the model bar is "can answer from a tool result", which is
stricter than a context window. See the README.

Honest limits

• No continuous batching; one request at a time per model.
• Native GPU residency is dense-only.
• Vision prefill is slow on CPU.
• Prompt processing is superlinear in prompt length — roughly n^1.85 — so a 32k prompt is not four
  times a 16k one. The parallelism above helps the constant, not the curve.
• The CPU attention speedup is Apple Silicon-measured; other CPUs should benefit similarly but are
  not yet measured.
• Several CUDA throughput figures elsewhere in the docs predate the 2026-08-25 driver/distro upgrade
  and are marked STALE rather than carried forward. Parity across that bump was re-established; the
  throughput legs that bench_peer.py does not drive have not been re-run.

Provenance

Every number here is a real measurement with a machine, a commit and a date behind it — see
docs/benchmarks.md for the methodology and docs/measurements/ for the raw runs. Where a claim
was withdrawn, the withdrawal is in the tree too.

v0.14.0 — six model families, gpt-oss on Metal, and a go-installable tag

Six new model families, gpt-oss on Metal, and the v1.0 API tiers declared. 188 commits since
v0.13.0. The headline for consumers is smaller and more boring than any of that: the tagged
submodules are replace-free, so go install github.com/townsendmerino/goinfer/...@v0.14.0
works from outside a checkout for the first time (v0.13.0's gpu/cuda/metal tags each carried
replace … => ../, which go install pkg@version applies because that module becomes the main
module — and ../ does not exist).

Added

Model families

• Qwen3-Next (qwen3_next) — the 80B-A3B Gated-DeltaNet/softmax hybrid, gated by a real-weight
  layer-slice oracle at cosine 1.00000000.
• Nemotron 3 Nano (nemotron_h MoE) — T3 real oracle cosine 0.997668, continuation exact,
  plus a real Q4_K_M GGUF gate (coherent, 0.843 distinct-trigram).
• Laguna (poolside) XS-2.1 / XS.2 / M.1 — safetensors and GGUF (llama.cpp's own arch),
  tiny goldens at cosine 1.000000 for all three generations, a real 33B-A3B gate, and a
  real-weight slice oracle at cosine 1.00000000. Softplus attention output gating and per-layer
  query-head counts are new axes; the real gate found two per-layer-head bugs a tiny fixture
  could not.
• InternLM2 (adapter: renamed tensors + a GROUPED fused wqkv de-interleave) and InternLM3
  (a llama ALIAS — its dynamic-NTK rope is in-window identity).
• Qwen3.8 (qwen3_5) — the dense member of the DeltaNet/softmax hybrid; see below.

gpt-oss

• safetensors/MXFP4 loader for gpt-oss-20b and -120b, gated against the GGUF path
  (argmax identical, cosine 0.999121).
• The harmony chat template — without it the family was unreachable through chat.Detect.
• GPU residency on Metal (G10): SHIPPED and admitted end-to-end — attention sinks, the
  clamped-SwiGLU MoE expert kernel, and a custom router. The CUDA kernels landed too, without
  touching the audited PTX.

Elsewhere

• GPT-2 GPU residency on Metal (G9) — shipped, admitted end-to-end.
• Block-drafting speculation as a production API: serve --drafter, BlockSpec.GenerateStream,
  a resident block trunk, batched hidden-state capture, and PrefillLastNArgmax (batching the
  verify's LM head).
• Metal batched prefill behind --metal-fast-prefill (P11) — a 3.9–4.6× TTFT lever, opt-in
  because it is not bit-exact.
• serve: optional native TLS (-tls-cert/-tls-key), and a hard failure when -addr is
  non-loopback with no -api-key — the unauthenticated-by-default posture is now enforced, not
  merely documented.
• The v1.0 API tier declaration (docs/api-tiers.md, signed off 2026-08-18) and the
  apidiff gate that enforces it (scripts/apidiff_check.sh, wired into CI). v0.13.0 → this
  release is clean: zero incompatible changes to any Hard-tier name.
• docs/release-1.0-gate.md — v1.0 as a decision against criteria, each line naming its
  evidence.

Changed

• aikit v1.17.1 → v1.21.0 (root module).
• GPT-2's "gelu" now runs the exact-erf form. It had been silently running the tanh
  approximation — a different function, not a different rounding.

Fixed

• Qwen3.8's two pos>0 bugs, the Laguna per-layer-head pair, and the batched-prefill path that
  never applied the attention gate (it read as a plausible cosine 0.957, not as a failure).
• multimodal: oversized Qwen vision inputs are rejected before the pixels are decoded, not
  after.
• The w4a8 decode-parity gate now RUNS — its matched int8 .giw half had never been built, so
  it had skipped at every tag since 2026-08-12. Built from the same source GGUF as the int4 half;
  16/16 greedy-token agreement on first invocation.
• The parity-manifest emitter no longer promotes on evidence it does not have. EMIT_MANIFEST=1
  used to write status: validated beside method: tiny-golden for four families, and mangled
  real-model-oracle into a name no tier rule recognises. Status is now DERIVED from the method,
  the method is a closed vocabulary, and a source census over all 18 call sites runs in plain CI —
  where the emitter itself never does, which is why the typo survived for months.
• Two tautological CUDA gates: the expert-cache bit-exactness tests never checked the cache was
  engaged (and allocSlots really does clear it silently), and the greedy fast-path test never
  checked ResidentGreedy. Both would have compared a path to itself and reported success. Guarded
  and mutation-verified on the hardware.

Measured and NOT shipped

Recorded because a negative result costs the same to obtain and is worth as much:

• CPU block drafting is a loss — 0.75× on dense, and break-even (8.89 tok/round) exceeds the
  ceiling (8.00). Implemented, measured, unwired.
• DFlash pairing on CPU MoE: 0.82× — works, lossless, and still a loss. Do not ship.
• Metal's batched small-M verify kernel: bit-identity holds, the ceiling does not (NO-GO).
• --drafter is a loss on the served default. The published 1.60×/1.50× figures rested on a
  wrong baseline; re-measured, code is 1.44× and math 1.58× while chat is 0.61× unguarded.
  Corrected in place rather than quietly dropped.

Parity and evidence

• Mellum2 real-weight slice oracle (G11) — the CPU half at cosine 1.00000000 on both the
  sequential and batched-prefill paths, and the Metal resident half at 9/12 argmax-exact,
  cosine 0.972006. This closes the one open correctness gap G10 opened by admitting Mellum to
  Metal's resident path with no end-to-end validation there.
• B13's standing reds are closed. TestSerializeWeightsTo_matchesBuffer was already green.
  TestQwen35GGUF_vsSafetensors was reclassified on measured mechanism, not on a moved floor:
  the router is bit-identical between containers, every transform-bearing tensor sits at a uniform
  Q8_0 noise floor, the per-layer divergence curve has no step, and the two containers pick
  different top-8 experts in 779 of 3200 decisions — quant noise at a routing decision boundary,
  not a loader defect. The gate now floors the MEAN with a measured min.

Qwen3.8 (qwen3_5), in full

• The dense member of the Gated-DeltaNet/softmax hybrid family.
  Alibaba's Qwen3.8-27B (2026-08-14, Apache 2.0) is the same 3:1 hybrid goinfer already runs
  as qwen3_5_moe and qwen3_next, with a plain SwiGLU where they have a router. The
  checkpoint is multimodal; this is the text decoder only — the vision tower
  (model.visual.*, 333 tensors) and the MTP head (mtp.*) are never requested.

  The structural change in the forward is one FFN branch; everything else — the DeltaNet
  step, the gated softmax attention, the hybrid cache, the sequential prefill — is the
  existing path untouched. Registered as both qwen3_5 and qwen3_5_text (the released
  config nests the text dims under text_config and states each spelling at a different
  level).

  Three things were read off the released checkpoint rather than inherited by resemblance,
  and each would have been a silent wrong answer:

  • head_dim is 256 at hidden_size 5120 with 24 heads, so nH·hd = 6144 ≠ hidden.
    Deriving head_dim (or the query projection width) from hidden is wrong for this family.
  • attn_output_gate is true, so q_proj is double width (query ‖ gate, 12288 rows).
  • The DeltaNet projections ship as in_proj_qkv / in_proj_z / in_proj_a / in_proj_b
    — qkv fused, z separate — which is neither qwen3_next's fused pair
    (in_proj_qkvz + in_proj_ba) nor four fully-separate tensors. The existing split
    reader is the right one; the index is what says so.

  The config carries mrope_section [11, 11, 10] with mrope_interleaved: true. For
  text input this reduces exactly to standard partial RoPE — position_ids arrive 2-D
  and are expanded to three identical components, so the interleaved overwrite is a no-op —
  which is why no m-RoPE code was added. Image input is a follow-on, not a silent gap.

  Parity: tiny-oracle. HF f32 tiny golden (tiny-random Qwen3_5ForCausalLM, text path):
  argmax exact, logit cosine 1.000000, greedy continuation exact. The fixture keeps the
  released model's shape character rather than its size — head_dim independent of
  hidden/heads, 3:1 layer_types so both mixers run, GVA value/key head ratio, and
  mrope_section present.

  And the real 27.8B runs. TestQwen38Real_gate loads Qwen/Qwen3.8-27B (18 bf16 shards,
  55.6 GB) at int4 on a 62 GB linux/amd64 box, asserts the geometry and BOTH mixers' tensor
  sets against the released index, and generates 96 greedy tokens: distinct-trigram 0.770,
  three correct Paris landmarks with correct detail (Champ de Mars, the 1889 World's Fair,
  the Mona Lisa). That is coherence, not an oracle — no bf16 reference forward was run
  against it — so the family stays experimental rather than claiming a validated tier.

  Admission: CPU-only, the same posture as every DeltaNet hybrid (no backend implements
  the mixer). GGUF and vision are follow-ons.

goinfer v0.13.0 — expert streaming as flags, faster prefill, Go 1.26.6

goinfer v0.13.0 — MoE expert streaming gets proper flags, prefill is faster, and three reachable stdlib CVEs are closed.

Security

• Go 1.26.5 → 1.26.6 across all four modules, closing three standard-library
  vulnerabilities that are reachable from this project's own code — not dormant
  transitive imports:

  		reached via
  GO-2026-6090	crypto/tls — limit post-handshake messages	serve → http.Server.ListenAndServe → tls.Conn.HandshakeContext; also prequant.readHead → io.ReadFull
  GO-2026-6089	net/http — apply ReadHeaderTimeout on the unencrypted HTTP/2 check	serve → http.Server.ListenAndServe
  GO-2026-5972	encoding/asn1 — enforce maximum recursion depth	serve → signal.Notify → asn1.Unmarshal

  govulncheck ./... reports no vulnerabilities after the bump. If you build
  goinfer serve yourself, use Go 1.26.6 or later; the go directive now requires it.

  The CUDA gate was re-run in full on 1.26.6 before tagging — a toolchain change is a
  compiler change, and the resident parity gates exist to catch a forward that moved.

Performance

• Prefill is ~4.5% faster on one measured shape, from aikit's f32 blocked-matmul rework
  (arriving with the v1.17.1 bump below). Benchmark-level +4.49% (median; bootstrap 95% CI
  +4.24% to +5.26%).

  Measured, and the method is part of the claim: BenchmarkPrefillLong, Qwen2.5-Coder-0.5B
  (dense), f32, 512-token prompt, batch prefill, on one box (Ryzen 7 3700X, linux/amd64).
  Both arms interleaved in a single session, warm-up discard and a 0.6% significance floor fixed in
  advance from the instrument's own characterization, 12 retained samples per arm. The arms do not
  overlap and per-visit medians are consistently ordered across three rounds. Full record, including
  every raw sample and the pre-registration: docs/measurements/aikit-v1.17.1-prefill-ab.md.

  Scope — what this does not say. One model, one prompt length, one quantization, one box,
  prefill only. It says nothing about decode, about other prompt lengths, about MoE or Gemma4
  architectures (which route down a sequential per-token path and never reach this shape), or about
  any other machine. A derived figure of ~8.6% within the reworked kernel itself follows from
  dividing by that path's profiled share of runtime — derived, not measured, and quoted second
  for that reason.

  Recorded because it was measured to the standard this project demands of a regression. A record
  that discloses losses and withholds equally-measured wins is biased, not cautious.

Changed

• aikit v1.16.0 → v1.17.0, aikit/gpu v0.27.0 → v0.28.0 across all five modules. A
  dependency update. The quantized GEMV is untouched — gpu/testdata/gemv_quant.ptx is
  byte-identical across gpu/v0.27.0..gpu/v0.28.0, and the gemv_quant.cu diff is three comment
  lines — as is the vision tower (vit.ptx byte-identical, ViTBlock/LNBLOCK still 256). What
  reaches goinfer is in the root module: a new AVX2 int8 kernel behind w8a8Span on the W8A8 decode
  path, and a reworked inner loop in the blocked f32 matmul. Both are argued bit-identical upstream;
  what demonstrates it here is the forward goldens — 33 passed, 0 failed, 9 skipped, of which 14
  drive a quantized path and 19 are f32, all against recorded values.

  The only performance figure carried for this bump is goinfer's own prefill measurement above.
  Upstream reports its own numbers; those are not reproduced here. Decode was also measured and is
  flat — no benchmark-level change against v1.16.0 — recorded in
  docs/measurements/aikit-v1.17.1-decode-ab.md rather than claimed here, because "no measurable
  change" is not a release-note item.

v0.12.0 — the 26B expert cache sizes itself correctly

MINOR (0.11.0 → 0.12.0): a correctness release for the CUDA MoE expert cache, three
bit-identical performance items, and behaviour changes that are disclosed above. No public API was
removed. The core-numerics surface is unchanged since 6edd1ca; every change to a path covered by
testdata/parity_manifest.json carried a goldens run whose axis composition is printed with the
result.

⚠ Gemma 4 output changes on CUDA and Metal. Gemma 4 previously fell back to the CPU path
unless you set GOINFER_GEMMA4_RESIDENT; it now runs GPU-resident by default. The resident path
is W4A8 — it quantizes activations to int8, which the CPU path does not — so logits differ and
a token can flip at a near-tie. Both paths are parity-gated — argmax-exact with a calibrated
cosine, which is the contract; GPU and CPU are not byte-identical to each other — and
argmax agreed at every position on the new real-width gate, but this is a real output change for
anyone running Gemma 4 on a GPU. It is opt-out: --backend cpu keeps the previous numerics.

This is the third consecutive release to change output for some users — v0.10.3 moved the
sampler tie-break, v0.11.0 shifted the seed→token mapping for the temperature and filtered
sampling paths, and this one moves Gemma 4 from CPU to GPU numerics. Each was individually
justified and each was disclosed, but three in a row is worth stating plainly in one place
rather than leaving a user to assemble it from three release notes. If you depend on
reproducible output across upgrades, pin a version and read this section before moving.

Changed

• Gemma 4 is resident by default on CUDA and Metal. GOINFER_GEMMA4_RESIDENT is now inert and
  can be dropped from any script that sets it. The flag was a bring-up gate that outlived its
  purpose in an instructive way: because every real Gemma-4 checkpoint reaching the resident path
  was only ever compared against itself (expert-cache on/off, CUDA-graphs on/off), the flag had
  become the only thing standing between users and a forward whose numerics no gate asserted at
  real width. It comes off because that gate now exists, not because it looked like residue.
  WebGPU still declines (no Gemma kernels) and E-models (E2B/E4B, PLE) decline everywhere.

Added

• A real-width parity gate for the Gemma-4 resident forward (TestGemma4MoEScaled_residentParity)
  and the composition gate the expert cache never had (TestGemma4MoE_cacheExpertsBitExact_scaled,
  ..._cacheReuse_scaled), on a new fixture that keeps the real per-expert row geometry
  (hidden 2816, moe_intermediate 704) and transplants per-group weight scales from the real 26B.
  The scaled cache gate had never executed since it was written: it named a fixture that did not
  exist. Generate with scripts/pin_gemma4_moe_scaled.py.

Performance

All three are bit-identical — same operands, same order, no tolerance involved.

• Gemma's final-logit softcap runs in parallel: 1.43 ms → 640 µs per sampled token at a 262,144
  vocabulary. Elementwise, so each output depends only on the input at the same index and the split
  cannot change a bit. Sampling path only — greedy reduces the argmax on-device and never paid it.
  The threshold is measured rather than chosen: below ~32k elements the split is a loss (8,192
  elements parallelise at 0.95×), so small vocabularies keep the serial path.
• MoE experts share one gate/up buffer pair per token instead of one per expert — at top-k 8,
  16 allocations per token become 2. The experts run sequentially, so the extra pairs were never
  simultaneously live. Applied to both MoE forwards.
• W4A8 reuses the per-stream Workspace instead of allocating a fresh one per projection per
  token. It was excluded by a dispatch that tested "is this W8A8" rather than "does this weight have
  a form that takes a workspace"; it now asks the second question.

Fixed

• The MoE expert cache sizes itself correctly on an 8 GB card, and the 26B decodes. The cap was
  a division over a raw byte sum; the CUDA driver charges each of four buffers per MoE layer its own
  whole 2 MiB quantum, so the requirement is a step function of the slot count. At 34 slots all
  four tip at once — putting the requirement 203,816,960 B over free — and the granted cap
  allocated successfully and then could not launch. The forward produced zero tokens.
  capSlots is now a search over the granularity form rather than a division (a division plus a
  correction term is wrong at exactly the boundaries the failure lives on), and it is the single
  implementation allocSlots calls. On this card the auto-cap moves 34 → 31 and the 26B decodes
  coherently.
• The deferred first-launch reservation is paid before the cache is sized. The on-GPU router
  kernel declares per-thread scratch, and the driver backs it for the device's occupancy the first
  time that kernel runs — 138,412,032 B retained, 289,013,760 B demanded at the launch itself,
  none of it visible to the free-VRAM reading the cap was computed from. Forcing that launch before
  the reading makes the cap correct by construction instead of covered by a margin. Measured
  after: nothing at all is consumed between the sizing decision and the end of the token.
• A launch that runs out of memory now names the kernel and both slot counts. Previously a bare
  cuLaunchKernel: CUDA_ERROR_OUT_OF_MEMORY with nothing tying it to the setting that caused it. It
  now names the kernel, the requested count and the effective count after capping — they
  differ once the cap fires, and naming only the effective one sends someone who set 48 to lower it
  to 40, which caps to the same value and fails identically.
• A resident decline now always says why. The reason was gated behind GOINFER_RESIDENT_DEBUG
  on all four backends, so a model silently moving its entire forward to CPU looked identical to
  one running resident. Unconditional now — one line at load.

Docs

• The 26B-A4B section is retracted and rewritten. It told users the slot count was "a manual
  workaround for a safety net that is not holding". That was accurate and is no longer: the cap
  holds. The section now records what the old behaviour was (34 slots allocates, then cannot launch),
  names both costs the cap was missing with their measured figures, and gives a version test —
  capping to 33 has the fix, 34 does not. The example sets the slot count high and lets the cap
  choose, because it can now be trusted to.

Verification

• int4 now has forward goldens — 23 fixtures across 16 architectures, comparing int4 output
  against recorded int4 output. int4 is the documented default quantization and nothing gated it:
  every golden that ran was f32, so a change that was correct in f32 and wrong in int4 passed. This
  is a note about what the gates cover, not a feature; nothing in the runtime changed because of it.
  The goldens run that gates a core change went from 19 to 33 tests, 14 of them on a quantized path,
  and now prints that composition alongside the count.

Known unfixed

• ~150 MiB of reported-free VRAM is not allocatable, on this driver and card. cuMemGetInfo
  reports it as free and cuMemAlloc refuses it at any request size down to 1 MiB —
  151,191,552 B, measured, cause unattributed. It is not fragmentation (a request 2.71× smaller
  than free was refused) and not exhaustion.

  This is why the 384 MiB slot margin cannot simply be lowered to recover the two slots the sizing
  fix costs: 151,191,552 B of that margin is this floor, not slack. A 128 MiB margin was measured
  working on this card and is below the floor — it worked because the cap it produced happened to
  leave enough leftover, which is luck rather than safety. The margin holds the correct value for the
  wrong-looking reason, and lowering it needs the floor understood first.