Changelog

Changelog

The project was called NVMAI until 2026-09-14, when it was renamed to TinyTitan. Entries below keep the name each release shipped under, and their links still resolve (GitHub redirects the old repository name).

Compact user-facing release history. Follow the linked GitHub releases for complete source changes and assets.

5.16 — the engine is a library

Released 2026-10-03. The prebuilt binaries, the library archive and their checksums are on the release page.

• TinyTitanLib is a supported Swift package you embed in your own program: an Engine and a Session over the same kernels, format reader and sampler the engine uses — streaming, cancellation, unload() and tools, with no subprocess and no HTTP. examples/embedded is a consumer you can read.
• Every release now ships the library as binaries too: libTinyTitanLib.a, libTinyTitanLib.dylib, the module set and its module maps, the Metal shader sources, and two demo apps — one per link form — for a consumer that cannot or will not build from source.
• Tools work in both directions: offer them with respond(tools:), read the model's calls as events, and close the loop by replaying the turn that asked plus the tool's result.
• Building needs Xcode 27 with Swift 6.4 — exactly. Nothing else is tested, and the installer now says so rather than accepting a newer Swift silently.
• CLI fixes: dense Qwen 3.5 prefills in 4,096-token chunks again (which also puts it back on the ANE), a fifth --stop string works again, and --quiet silences the library's own lines as well.
• No engine output changed: the seven installed golden baselines are byte-identical, and the CLI matches its pre-5.16 output byte for byte.

5.15 — the model format is renamed .ssdai

Released 2026-10-02. The prebuilt binaries and their checksum are on the release page.

• The install format is .ssdai (it was .gturbo): the manifest magic, the Swift vocabulary, the CLI, the tools and the docs all carry the new name.
• Your existing installs keep working, untouched. Reads accept the legacy GTURBO magic for one release, and no manifest is rewritten in place — the receipt binds the manifest's digest and the directory path, so an install moves to the new magic only when it is next rebuilt.
• --input-ssdai is the flag; --input-gturbo is still accepted as a deprecated alias. The directory suffix is a convention, so a folder still named something.gturbo works exactly as before.
• No engine behaviour changed — the seven installed golden baselines are byte-identical, and each of them loads a manifest that still carries the legacy magic.

5.14 — the browser window moves to DeepSeek Harness 0.2.0-rc.2

Released 2026-10-01. The prebuilt binaries and their checksum are on the release page.

• The browser chat window runs on DeepSeek Harness 0.2.0-rc.2. tools/dsh_local.sh installs that release and tools/server_launcher.sh --web opens it with the TinyTitan route already in place. The route now goes through the harness's own settings service and the agent preset is registered with its preset registry — 0.2.0 removed the settings.yaml and preset files the bundle used to write. The preset keeps its id, so sessions already on it keep working; it becomes the default only while nothing else is selected.
• Installing a model while the window is open reaches the model picker without restarting the harness — the watcher had been refreshing through the old file path, which 0.2.0 cannot satisfy.
• compactionHeadroomTokens lets you place the compaction trigger: leave it unset for the harness's policy, 0 to let the 0.8 ratio govern, or a token count for a guard. The stock 65,536-token headroom holds the trigger near 62% of a 262,144-token window, and on a narrow declared window it stops compaction altogether.
• tools/dsh_local.sh status now says which harness version is installed, not just which one is pinned, and the bundle's README and wiki page describe what it actually writes after the 0.2.0 move.
• No engine behaviour changed; the seven installed golden baselines are byte-identical. The internal-speed record is optional from this release on (a machine under load moves the numbers further than the 10% threshold), so no timing record accompanies 5.14.

5.13 — a wrong model path names itself, and a wiki written for users

Released 2026-09-29. The prebuilt binaries and their checksum are on the release page.

• A --model path that does not exist now says so. Any typo used to fail with installed tokenizer is missing chat_template.jinja; reinstall the model, which sent you to reinstall a model that was simply not at that path. It now says model directory not found: <absolute path> — at the CLI, the server and the token-counting endpoint alike.
• The wiki is a user guide. Every page now explains how to use TinyTitan: a new Installation and Configuration page (every model key, the launcher's flags, all the environment variables in one place), a rewritten Runtime Controls (every CLI, server and API parameter with its default), Local Server and API (endpoints, examples, structured output, function tools, client wiring), a larger Cookbook, and Benchmarks reduced to what to expect on an M3. The engineering record and the closed audit findings left the wiki.
• The README is a verified quickstart — what TinyTitan is and who it is for, prerequisites, install, a complete worked example and troubleshooting, with a Last verified line naming the commands that were actually run.
• No engine behaviour changed. 50 source files were split under the 500-line rule as pure code motion, each moved block verified byte-for-byte with the full suite green after every split; sampling, kernels, defaults and the API are untouched.

5.12 — the pre-production audit: no crash paths left, and gates that cannot pass silently

Released 2026-09-25. The prebuilt binaries and their checksum are on the release page.

• A bad state reports itself instead of killing the process. Every force unwrap is gone — 171 sites in the engine and 139 in tests and benchmarks — so the engine returns a typed error where it used to trap. Chunked prefill against a producer that cannot run it now returns the existing chunkedUnsupported error rather than crashing.
• The project verifies itself with eleven pinned gates — SwiftLint and swift-format, strict C99, Ruff, eslint/prettier per plugin package, shellcheck, and the size/architecture/scanner checks — run by the same command locally and in CI. Each fails when its tool is missing or is a different version, so none can pass by skipping.
• Swift code scanning works again. CodeQL had failed before compiling a single file since the toolchain moved; it now scans 249 of 464 Swift files, including the memory, validation and Fleet modules, with no alerts.
• The DeepSeek-V4.1-Flash port reference (docs/deepseek-v41-flash-port.md, docs/deepseek-v41-flash-reference.md) documents what that checkpoint is, its byte and quantization layout, and the two conversion paths available here.
• No model, kernel or API behaviour was retargeted. This is a hardening release: 4,479 SwiftLint findings and 442 files of formatting drift went to zero, and the audit that found them is closed, with its evidence and deliberate decisions live in the closing commits of that work.

5.11 — a browser chat window that installs without a registry, and an installer that refuses what cannot finish

Released 2026-09-25. The prebuilt binaries and their checksum are on the release page.

• The chat window is dsh-tinytitan, a local DeepSeek Harness bundle. The Mac app is gone; this points the harness at the TinyTitan server, keeps the route current as models are installed, and stops the harness's own compaction calls from thinking on a local reasoning model. It is configuration only — the harness's own adapter serves the models, nothing is patched, and it composes with a harness you already run. Install: run the installer and accept the chat window (a model must be installed), then start it with ~/.local/bin/tinytitan-web. Harness and bundle both come from the release's source archive, so no npm account is needed on either side. Benefits: one command sets up engine, models, window and route; the harness is private and pinned under ~/.tinytitan/dsh, so yours is never read, written or stopped, caches included. Limits: exactly one harness release (0.1.6-alpha.2) and it refuses any other; loopback only; session titles are not covered by the compaction row; macOS Apple Silicon only; binaries not notarized. More: TinyTitan Plugin.
• The model installer refuses a download that cannot finish, before fetching a byte. It prints what each volume needs, the two ways out (free space, or TINYTITAN_WORK_DIR on another volume) and TINYTITAN_SKIP_DISK_CHECK=1.
• Staging stays inside the install root and is reclaimed as widths complete. It used to be a relative .build/… — for a factory-new install, ~/.build in the user's home; install_models.sh clean reclaims what is left.
• An EOF at the model menu installs nothing, where Ctrl-D used to fall through to the recommended 36.9 GB download.
• Memory keeps one address per fact. Two sessions in one project could distil at once, so the later read a store the earlier had not written and invented a parallel key; distillations are now chained per project.
• --ram help now says what it accepts: any whole GB from 4 up, not only the interactive menu's 4/8/16/32. No behaviour change.

5.10 — the RAM flag means the whole process, and Qwen3.8's sampling rows are real

Released 2026-09-24. The prebuilt binaries and their checksum are on the release page.

• --ram is a target for the whole process, not just the expert cache. The flag used to bound the cache alone, so --ram 8 produced a process using 11.63 GiB. It now budgets the process — the cache gets the target minus the weight file and a measured 512 MiB reserve — steps down to the largest slot rung that fits, refuses anything below a 4 GB floor, and prints ram target=… cache=… slots=… resident_floor=… estimate=… at every start. This changes an existing flag: --ram 8 now gives 32 slots where it gave 64, so it is slower than it was on 5.9; --ram 12 reproduces the old 64-slot cache. Peak RSS is now within 0.2 GiB of the printed estimate.
• Qwen3.8's instruct row is real. The engine refused every non-zero presence penalty; it now applies one in the same pre-softmax window as the repetition penalty, ships two explicit Qwen3.8 rows (thinking 1.0/0.95/20; instruct 0.7/0.80/20 with presence 1.5), picks the row from the request's thinking mode, and exposes --presence-penalty on the CLI. min_p is accepted but must be zero — the filter does not exist, and refusing beats ignoring.
• The C kernels compile at -O2. The CPU int8 GEMV runs 1.21× faster (2.35 → 1.94 ms, identical checksum); the internal-speed re-record reads decode +2.5%, prefill +6.5%, first token −6.1%.
• The expert-cache ceiling is a third of physical memory, not a half, which is what stopped a 24 GiB machine from swapping (855 → 1,610 MB before; flat swap and 7.29 tok/s after, against 5.58 tok/s at the old cap).
• Twelve decode switches that measured a wash or a loss are gone — including TINYTITAN_PREFETCH_PER_EXPERT, …_AHEAD, …_TOP_M, …_MIN_MARGIN, …_IO_TIER, …_EXPERT_CACHE_POLICY, …_EXPERT_CACHE_LAYOUT, …_EARLY_HITS, …_KEEP_WIRED, …_PARALLEL_IO and …_PREFILL_LAYER_MAJOR. Setting one is now inert; the winning default is what runs.
• Converting Qwen3.8 resumes and works through mirrors. An interrupted conversion keeps the output shards it finished and fetches only what is missing; HF_ENDPOINT (or --endpoint) points the whole fetch at a mirror on the Hub's URL layout; a truncated shard is never adopted; the finished 102 GB n-gram table is reused in place or copied across filesystems; downloads retry with a real timeout and restart a file when a mirror refuses ranges; and a directory left between the shard rename and the index write is refused rather than duplicated. Thirty-six fault-injection cases back it and run in CI.
• Prefetch depth 1 is the Qwen3.8 profile default (+15.7% decode at a short prompt, +14.6% at ~500 tokens, responses byte-identical).

5.9 — the GDN pair loads at the slot's width, and memory's value is measured

Released 2026-09-20. The prebuilt binaries and their checksum are on the release page.

• An install whose manifest names the GDN pair explicitly serves again. A manifest that carries linear_attn.in_proj_a/in_proj_b at the attention slot's own width was refused on load as if it were corrupt, which broke a qwen38flash 4-bit install completely (issue #16). The validator now honours the slot's width and bf16, and names the width it accepts when it still refuses another.
• The ten master prompts are runnable end to end. Long-session worlds as data, with foundation and carryable key sets derived from the truth rather than authored beside it, a scorer over the stored answers, and a driver that runs all ten. The result that matters is the baseline: a client's own 200-word summary wins or ties on three of the first four scenarios, so memory is argued against that, not against memory-off. Where the fact set is dense, arbitrary and revised, memory carries 20 of 20 against the summary's 14.
• Which model judges the side-engine's tasks is a measurement. benchmark/side_engine_judges.py runs the same cases through the resident 4B on the CPU and the model already on the engine: the 35B is better at the reply check the 4B cannot do (100% against 62%) and worse at duplication (75% against 100%), and the pair beats either alone.
• A T7 retrieval hint is queued before the search returns. The background caller's registration was fire-and-forget, so a caller that observed the hint immediately after a search could arrive before the question was queued — the race the thread-sanitizer gate caught. Nothing on the request path waits on a judgement either way.
• The plugin is catalogued. dsh-tinytitan is listed in awesome-dsh-plugin; the npm publish remains the operator's step.

5.8 — a memory side-engine, and a LAN manager for a fleet

Released 2026-09-19. The prebuilt binaries and their checksum are on the release page.

• Agent memory has a resident side-engine. A 4B runs on the CPU and answers one closed question at a time — keep this fact, do these two say the same thing, do they disagree, which kind of change is this — while the main model keeps the GPU. Durability, duplication, contradiction and supersession are wired into consolidation under a budget of six questions; nil means no decision, so an absent engine leaves the old path alone. The 4B is the default, the 9B optional, the 2B unused.
• A stored rule can hold a write back. rules/eyes now stops characters/marcus/eyes from changing, found by key with no model call; only model-derived facts are checked, so the person always overrules a rule.
• Retrieval ranks by rarity. Weighting each term by inverse document frequency takes the authored paraphrase set from recall@1 1 of 4 to 3 of 4, and T7 — the one scorer that reaches the semantic miss — now works in the background during idle windows instead of on the request path.
• The 102 GB n-gram table is shared between builds, not copied per quantization, with a gate that refuses a table whose constants do not match.
• Per-tensor bit widths resolve in the resident index, the enabler for a ~10 MB precision promotion. Its quality case was measured twice and is absent (18/20 against 18/20; 0.0097 ± 0.0067 nats, inside the instrument's floor), so the converter policy is deliberately not built.
• A LAN manager for a DeepSeek Harness fleet. dsh-lan-manager adds a LAN-scoped API for workspaces, sessions and prompts, and ttlanmanager is a terminal dashboard over it; reaching it from another machine is blocked until the harness allows a non-loopback bind.
• Both DSH plugins pin exactly 0.1.6-alpha.2 and refuse to run on any other release rather than write into a harness home they cannot identify.

5.7 — one command installs a built engine, and the app is gone

Released 2026-09-17. The prebuilt binaries and their checksum are on the release page.

• The install builds nothing. tools/install_tinytitan.sh downloads the published arm64 executables, verifies their sha256, unpacks them under ~/.tinytitan and asks one question — which model. No Xcode, Homebrew, Python or Node is needed, and --version TAG pins a release while --from-source keeps the clone-and-build path. See Getting Started.
• The model is a menu. All 16 builds are listed with their installed size and what each is for; Enter takes the default, --model NAME skips the menu, and through a pipe the installer takes the default and says so rather than hanging.
• Every script runs on the shell a factory Mac actually has. /bin/bash there is 3.2.57, and these scripts had only ever run under Homebrew's 5.x. Three classes were closed: a heredoc apostrophe that stopped 3.2 parsing the launcher at all, ${v^^}/mapfile that died mid-menu, and "${a[@]}" on an empty array (unbound variable under set -u), which broke the default --web path. tools/lint.sh shell now fails on all three.
• An optional browser chat window, isolated from any DeepSeek Harness you run: its own home, npm prefix, pnpm store and port, under ~/.tinytitan. The launcher's --web opens it already pointed at your model, with a chat preset that cuts a nine-word question from 4222 to 124 prompt tokens and a warm expert cache, so the first answer lands in 8.3 s instead of 161 s on the 35B.
• The Mac app is removed. The engine and its loopback server are the product; the supported clients are the ones you already have, plus the browser window as a client of the loopback server. This also leaves ServerVersion.current as the tree's single version literal, which release.sh enforces.
• POST /v1/responses/compact returns one opaque state item standing in for a conversation, preserving the system prompt and the core reasoning. The note is metered with the server's own tokenizer and compressed rather than truncated when it is over budget. Verified on the 4B and 9B.
• The concurrent width is any power of two up to 256 (the default stays 1), with the per-load memory budget doing the clamping and saying so in the log.
• Repository cleanup: the forum article series is gone with the forum, and adding a model is eight wiring points now that the app descriptor is not one.

5.6 — the three reported bugs, and an ANE answer of "no"

Released 2026-09-17. The prebuilt binaries and their checksum are on the release page.

• The app can change models. Model → Change Model now lists every selectable build and records the choice; a banner says to reopen, because the model directory, its settings file and the decode-service process are bound at launch. Previously the app read a model preference it had no way to write.
• The model installer finds its own binary. tools/install_models.sh used a target-triple build path that exists only when SwiftPM's triple matches the toolchain, and can point at a stale tree when it does. It uses the stable product path now, and a lint gate fails any build path that hardcodes a triple.
• A sidecar the Neural Engine refused can no longer be written or run. The exporter checks the compile markers, loads every function it records, and asks the compute plan which device each operation is assigned to — the refused Qwen 3.8 h12288 measures 0 of 173 operations on the ANE. The runtime declines a sidecar that lacks aneCompileVerified, so an old sidecar costs a GPU prefill instead of a ~38× slower CPU one.
• Qwen 3.8's sparse attention is correct on the ANE, and still stays on the GPU. The runtime folds the indexer's key selection into the mask the sidecar already takes — 0.47% against an independent reference, where a causal-only mask is off by 7.6%. Measured, the ANE is 0.72× the GPU at 4-bit and 0.87× at 8-bit, so no sidecar is installed for it; the gather-graph variant that would have removed the extra arithmetic was sized and rejected.
• Building from source needs Swift 6.4 (Xcode 27). The manifest floor now matches the toolchain the project is built and tested on.
• The dense Qwen 3.5 installs reach the ANE. They were on the 128-token default prefill chunk, which never matches the sidecar's fixed 4,096-token program; at 4,096 the 2B prefill measured 1.30× faster at ~2,500 tokens.
• The server has per-session KV and GDN state and admits up to four sessions, but serves them one at a time: width > 1 produced degenerate output on the real 2B and 4B, so the batched width is held at 1 until multi-session output is verified correct. The Responses API now echoes the sampling the server used instead of null.
• The wiki is a user guide. New: a Cookbook of copy-paste recipes with the output to expect; the engineering write-ups it used to index live under docs/ in the repository.
• Repository cleanup: the processed deep-audit register was removed, the project tracker no longer carries a closure log, and three oversized source files were split as pure code motion with the build and 1,566 tests verified.
• Internal speeds are a release gate with a committed baseline, so a kernel or bandwidth regression fails a release instead of being noticed later.

5.5 — the project is TinyTitan, and JSON is enforced by a grammar

Released 2026-09-14. The prebuilt binaries and their checksum are on the release page.

• The project is TinyTitan everywhere — package, targets, executables, environment variables, repository and app. The binaries are TinyTitanServer, TinyTitanMac, TinyTitanCLI, TinyTitanRepack, TinyTitanDecodeService and TinyTitanBench; the environment variables are TINYTITAN_*; the app installs as TinyTitan.app; and the archive is tinytitan-5.5-macos-arm64.tar.gz. The old repository URL redirects, and releases before this one keep the name they shipped under.
• JSON output is enforced, not requested. response_format on Chat Completions, text.format on the Responses API and output_config.format on Messages compile into a byte-level grammar that masks the sampler, so the model can only emit a document the schema allows. The supported subset is small and explicit, and everything outside it is refused by name.
• Thinking is the request's own on every surface. On /v1/messages, thinking.disabled really turns it off, adaptive still means "you decide", and an enabled budget maps onto the levels the served model renders. Chat Completions gained the same per-request control, and reasoning tokens are reported in usage.
• A developer message no longer fails with HTTP 500. It renders as the system turn it stands for — what a harness that switches to that role once a model reasons needs.
• The launcher lists only installed models, and warns in red above 40% of this Mac's physical memory before using a larger --ram anyway.
• One client list for the launcher and the coder harness — Codex, Claude Code, Qwen Code, OpenCode and the Zed editor — with --round clients checking every client's wiring without loading a model.
• DeepSeek Harness support. tools/dsh_route.sh generates the harness's llm-pi-ai route from the installed models, and plugins/dsh-tinytitan/ keeps it current at boot and mounts a compaction backend that does not think.
• The six dense Qwen 3.5 installs have golden baselines, so a release now re-checks 2B/4B/9B at both widths instead of declaring them exceptions.
• The README is one benchmark table with a reproducible GPU-versus-CPU column for the dense models and no per-release callout — this page is the announcement.

5.4 — KAT-Coder-V2.5-Dev, and verification that never fetches a model

Released 2026-09-14. The prebuilt binaries and their checksum are on the release page. This release first ships the 5.3 work to users (see the note below), together with the release-verification policy and a native app icon.

• KAT-Coder-V2.5-Dev 35B-A3B is supported at 4-bit and 8-bit — Kwaipilot's agentic-coding fine-tune of Qwen 3.6, with its own temperature (1.0) rather than the series' 0.6. 17.86 tok/s at 4-bit, 6.91 at 8-bit, with the three verified continuations and a golden baseline per width.
• Routed experts are filed by index, not by arrival order. KAT is the first converted checkpoint whose experts ship one tensor per expert; fusing them in arrival order silently paired each routing decision with a different expert's weights. The install loaded, matched the checkpoint byte for byte, passed every structural check, and answered nonsense. A tools/lint.sh gate now fails if the order regresses.
• Both widths install from one download: tools/install_models.sh <model> both converts 4-bit and 8-bit in a single pass over one ~70 GB checkpoint, and a later second width reuses the snapshot instead of re-fetching it.
• Three size caps that refused legitimate files are fixed — the snapshot index (4 MiB against KAT's 9.7 MB, copied into both install paths), the resident index (a 1 MB staging budget applied to a ~28 MB output), and the runtime's manifest cap (4 MiB against the 6.25 MB file --verify-install had just accepted).
• The downloader survives a truncating link: length-verified 64 MiB ranges, a small connection pool, HTTP/1.1, stall detection, and no resume that could append to a truncated prefix.
• The tools resolve their own Python by capability rather than a pinned version.
• Release verification uses only the models already installed. The golden gate reports every target it could not check instead of skipping it silently, an installed model that no gate covers is an error, and the gate fails if it changes models/ at all — so a release is never made to pass by downloading, converting or re-installing a model.
• A native NVMAI app icon, replacing the upstream fork's bird: the wordmark's own palette, and reproducible with tools/make_app_icon.py.
• The binary archive carries NOTICE alongside LICENSE and THIRD_PARTY_NOTICES.md.

5.3 — tagged, never published

5.3 was prepared and tagged (v5.3) but never published, because the golden gate could not complete on the machine it was cut from: a storage provider had left most installs online-only and reading them failed. Nothing was missing from the release itself, so its content first shipped to users in 5.4. The tag is kept as the history it records.

5.2 — one server for every model, dense Qwen 3.5 on both engines

Release v5.2

• One server serves every installed model. --models-dir serves the whole catalogue on one port with one model resident at a time; /v1/models lists them and a request naming another switches to it. One launcher replaces the eight per-model start scripts and the separate client launcher: it starts the API alone or with Codex, Claude Code, Qwen Code, OpenCode or Zed, and asks what to launch, the answer style, the model, the thinking level and — only where a model has an expert cache — the RAM limit.
• The dense Qwen 3.5 2B / 4B / 9B run on the GPU as well as the CPU, and the engine is a per-request choice for them: the bare id is the GPU spelling and <id>@cpu / <id>@gpu name an engine explicitly. The refusal was lifted only after the GPU's logits matched the CPU engine's on the real install, layer by layer — which is the gate that caught k_proj/v_proj being read at the attention slot's 4-bit width while the install stores them at 8, a plausible wrong answer rather than an error.
• Optional agent memory: NVMAI_MEMORY=1 gives a model durable facts scoped to the repository a client is working in, inside the server process, with no database to install. The extraction marks whether the person asserted a fact or the model inferred it, and the guard (on by default where memory is on) stops an inferred fact from silently superseding an asserted one: unguarded the store answered 61% of a recorded session's questions, guarded 98%.
• Thinking is not the answer. Reasoning reaches clients as reasoning_content, apart from content, on every surface, and each model offers only the levels its template renders. A thought the model opens while thinking is off is now split out too — Qwen-AgentWorld 8-bit does exactly that, and its scaffold used to arrive as the answer — and the server logs it.
• The APIs coding agents speak: the OpenAI Responses API in full and the Anthropic Messages API, both exercised against the real Codex and Claude Code CLIs rather than only against test doubles.
• A deep audit of the whole tree: 89 code findings and 8 documentation defects, 0 open. The worst were silent rather than loud — an out-of-bounds router write on Qwen3.8-Flash-Next, a prompt-cache restore that left the sparse indexer holding the previous conversation's keys, and a fused kernel that could never be built, so every measurement of that path had measured the unfused one.
• Watchdogs, off by default: NVMAI_WATCHDOGS=1 watches for a repetition loop, a stall, a stub answer and a ping-pong turn, and names which of them may stop a generation rather than only report it.
• Structure and language baseline: the 2,599-line HTTP server is eight files, the test tree mirrors the source tree, the compiler warning count is zero, and the three upcoming Swift features the tree was already clean under are now enforced (the three with a real migration cost are recorded as deliberate, with what they would cost).
• This release's own measurements — every model, both engines, three repeats, every reply quoted verbatim, including the install that thinks with the switch off — are in benchmark/benchmark-results/capital-of-paris-20260911T1935/ in the repository. The 512-token story table under Benchmarks was measured at 5.1 and has not been re-run.

5.1 — the 35B models decode 10-13% faster at 8-bit

Release v5.1

• The expert prefetch ring was clogged and is now repaired. A reclaim rule introduced in 5.0.2 freed a speculative slot only once its layer index had passed, so a prediction made for the last layer of a token was never freed and the ring ran with almost no free slots. Every prefetch measurement since compared variants of a mechanism that was not running.
• Prefetch is on again for the 35B models, one read deep, measured per install: Qwen 3.6 +1.8% at 4-bit and +11.3% at 8-bit, Ornith 1.5 +1.8% / +12.6%, Qwen-AgentWorld +1.4% / +11.4%. Qwen3.8-Flash-Next keeps it off, where the repaired ring loses at every setting: its experts are larger and arrive after the layer that would have used them.
• The expert cache stays wired through prefill. It used to be released at prefill, swapped out under memory pressure, and faulted back in by the first decode token, costing 1.6-4.7 s per request. Short answers gain 5-7%; long ones about 1%; prefill is unaffected.
• A start script per model and quantization, eight in all, each on its own port so two can run at once. The interactive launchers and the start scripts now share one catalogue of models, paths and ports, and both read the served model id from the running server rather than assuming it.
• The Mac app recognizes the models you have installed. Its catalogue listed only MLX repacks, so after 5.0.1 moved installs to bf16-sourced builds it treated all of them as foreign checkpoints. It now fingerprints each of the eight, and where it cannot download a build itself it prints the installer command instead of a Download button.
• New decode diagnostics, and three levers measured and left off by default: early expert hits, a probe-weight prefetch gate, and a decayed-frequency cache policy.
• Story benchmark, base M3 24 GB: AgentWorld 21.74 / 12.28 tok/s (4 / 8-bit), Ornith 21.65 / 11.93, Qwen 3.6 21.41 / 12.37, Qwen 3.8 5.46 / 2.10.

5.0.2 — per-model tuning profiles, bigger expert caches for the 35B family

Release v5.0.2

• Every installed model and width has its own tuning row (expert-cache budget, prefetch depth and disk I/O tier, prefill chunk, sampling defaults, kernel switches), resolved family default, then row, then environment. Editing one install never moves another; NVMAI_* variables still override.
• The 35B-A3B models get bigger expert caches. A per-role split showed a fifth of the 4-bit token and over a third of the 8-bit token waiting on expert reads at the old 128 / 64 slots per layer. Qwen 3.6, Ornith 1.5 and Qwen-AgentWorld now ship 160 slots at 4-bit (10 GiB) and 96 at 8-bit (12 GiB), measured on each install with swap sampled: +9% at 4-bit and +15% at 8-bit, output byte-identical. 192 slots at 4-bit measured +4% more but starts paging on a 24 GB machine; --ram-budget 12G selects it.
• Qwen3.8-Flash-Next: 4-bit prefetch runs two deep on the utility disk I/O tier (+3%); 8-bit takes 40 slots (+8%). Two-layer-ahead prefetch and the fused hyper-connection gates measured washes and stay opt-in.
• Ornith 1.5 is rebuilt from its bf16 release through the shared converter; no third-party quantization remains in the installer.
• Story benchmark, base M3 24 GB: AgentWorld 21.28 / 11.16 tok/s (4 / 8-bit), Ornith 20.99 / 10.89, Qwen 3.6 20.95 / 11.23, Qwen 3.8 5.40 / 2.06.

5.0.1 — 35B decode regression fix, Qwen-AgentWorld

Release v5.0.1

• Fixed a 5.0 decode regression on the 35B models: per-kernel environment reads (about 800 per token) cost most of a 45 ms token. Qwen 3.6 4-bit went from 6.67 back to 19.21 tok/s; output unchanged on every golden.
• Added Qwen-AgentWorld 35B-A3B (4-bit and 8-bit), installed straight from Qwen's bf16 release: one download quantized a shard at a time into both widths, router, shared-expert gate, DeltaNet gating projections and every norm kept at bf16.
• Every install now comes from its own bf16 release through one converter; the embedding and head stay 8-bit for quality (--head-bits 4 exists and is worth about 20% on 35B 4-bit builds).
• Top-8 routing uses the one-simdgroup selector (byte-identical, a few percent on the 35B family).

5.0 — Qwen3.8-Flash-Next, and per-family decode tuning

Release v5.0

• Added Qwen3.8-Flash-Next 125B-A6B in 4-bit: 48 layers of gated DeltaNet with sparse-indexed attention, 512 experts at top-10, hyper-connection residual streams, a QSA key indexer, and hashed n-gram embeddings at layer 1. Text-only; the checkpoint's vision tower is not repacked. An install is about 161 GB on SSD, of which the n-gram table alone is 95 GB, and it runs on a 24 GiB machine because only a bounded slice of its experts is resident.
• The expert-cache budget and expert prefetch are now tuned per family and quantization rather than being one number for the catalogue. Both are governed by how much of the token is expert I/O, and that varies by nearly 3x across the shipped models. Qwen3.8-Flash-Next takes a 12 GiB budget (96 slots) and depth-1 prefetch, worth +21.3% together (5.735 to 6.957 tok/s); Qwen 3.6 and Ornith take depth-1 prefetch at 8-bit (+5.5% / +5.9%) and no prefetch at 4-bit, where it measured -4.2% / -3.9%. The tuned budget is clamped to half of physical memory, so a 16 GiB Mac is not handed a budget measured on a 24 GiB one. --ram-budget and NVMAI_PREDICTIVE_PREFETCH override both, in either direction.
• Speculative prefetch now filters the predicted experts by residency before capping the ring rather than after, so the budget is aimed at experts that would otherwise miss instead of being spent on ones already cached. Ring depth is a bandwidth decision and is sharply peaked at one read in flight: depth 1 measured +12.2% where depth 4 measured -9.8%.
• ANE prefill is now the default (NVMAI_PREFILL_ANE=off opts out). Models without an exported sidecar fall back to the GPU automatically.
• Breaking: API model ids now always end in the routed-expert width, for example ornith-1.5-35b-a3b_8-Bit. The width comes from the manifest rather than the name, so two quantizations of the same weights are distinguishable in /v1/models. The bare id is no longer accepted.
• NVMAICLI now derives its expert-cache slots the way the server does instead of defaulting to a flat 64 for every model, so the two front ends agree.

4.6 — Sampler rewrite and ANE prefill

Release v4.6

• Production sampling (Top-K 1...64, which includes the default Top-K 20) now runs through the tiled three-stage GPU reduction instead of a single-threadgroup kernel that re-scanned the full vocabulary once per kept candidate. Sampled output at a fixed seed is unchanged token-for-token; the per-token sampling cost fell from 15.5 ms to 1.4 ms. NVMAI_SAMPLER_PATH=generic keeps the old kernel as a measured control.
• Added the experimental opt-in ANE prefill path (NVMAI_PREFILL_ANE=on): full-attention prefill blocks run on the Neural Engine from a Core ML sidecar exported once per model with tools/export_ane_prefill.py. A 6,103-token prompt measured 2.31x faster prefill (132.90 s to 57.52 s) with decode unchanged. Off by default because its fp16 attention is not byte-identical to the GPU path; the greedy golden baseline runs with it off. Short prompts, prompt-cache resumes, and MTP chunks stay on the GPU automatically.
• The MTP verify pass now plans both rows' experts as one union, overlaps one parallel miss fetch with the shared expert, and reads the lm_head once per pass. Faster than the old width-4096 tile scheduler at width 2 with byte-identical output; NVMAI_MTP_VERIFY=tile retains the old schedule. MTP itself remains off: re-qualification still failed promotion at both quantizations even at 84-86% draft acceptance.
• Sampling is now visible to NVMAI_KERNEL_STATS as its own role, and MTP runs report per-pass phase attribution under NVMAI_RUNNER_STATS.
• Retained the v4.2 control-plane and v4.3 predictive-prefetch experiments as documented, non-default controls; production decode scheduling is unchanged from v4.1.
• On the release M3/24 GB benchmark, median Ornith decode improved from 16.45 to 22.53 tok/s at 4-bit and from 8.75 to 9.40 tok/s at 8-bit.

4.1 — Expert hit/fixup streaming

Release v4.1

• Decode now executes phase 1 for already-cached routed experts while missing experts are read from SSD, then runs a bounded fixup for those misses. Shared expert work remains overlapped with routing and I/O.
• Cache-slot leases prevent in-flight expert buffers from being evicted or overwritten. The default pread path still bypasses the macOS page cache and preserves the configured RAM bound.
• Added per-run hit/miss, reload, read-latency, hidden-I/O, and hit/fixup-layer telemetry plus a deterministic barrier-versus-hit/fixup benchmark.
• Kept the Metal expert-I/O backend experimental; production continues to use bounded pread because Metal I/O is not yet a complete faster replacement.
• Fixed command-buffer completion tracking found during release benchmarking. Golden output remained identical after the scheduling change.
• On the release M3/24 GB benchmark, median Ornith decode improved from 15.31 to 16.45 tok/s at 4-bit and from 8.31 to 8.75 tok/s at 8-bit.

4.0 — Ornith and extended context

Release v4.0

• Added an eight-cell Ornith coding/tooling matrix for 4-bit and 8-bit weights, Concise on/off, and Thinking on/off. Four isolated Swift, Python, TensorFlow, and PyTorch repairs receive up to 25 turns and must pass both public execution and chained hidden validation.
• Exposed Ornith/Qwen thinking consistently as the models' official binary off|on switch in the server, CLI, Mac app, launchers, and benchmark profile. Low/Medium/High modes are not offered because the model templates define no effort levels or thinking-token budgets.
• Made Ornith 1.5 35B-A3B 8-bit with Concise off and Thinking off the default installer, Mac app, launcher, benchmark, release gate, and real-inference golden baseline. Qwen 3.6 and Ornith 4-bit remain explicit options.
• Added pinned, text-only Ornith 1.5 35B-A3B installers for 4-bit and 8-bit. Ornith reuses the existing bounded Qwen3.5-MoE runtime and SSD expert streaming; its tokenizer and chat template remain model-specific. Vision is deferred.
• Added reproducible preparation and local import of Ornith's official native one-layer MTP draft. It shares the matching target embedding and head while keeping draft experts SSD-streamed and bounded. Cross-model Qwen/Ornith pairings are rejected. MTP remains experimental and off by default because the first M3 benchmarks found no repeatable speedup.
• Standardized Mac app, CLI, Chat Completions, Responses API, decode-service, and benchmark defaults across Qwen and Ornith quantizations: temperature 0.6, Top-P 0.95, Top-K 20, and presence penalty 0.0. Nonzero presence penalties remain unsupported; greedy correctness and MTP measurements still override temperature to 0 explicitly.
• Added 16-, 8-, and 4-bit KV-cache storage with 8-bit as the default. The cache remains bounded and grows on demand independently of model weight precision.
• Added optional YaRN context scaling for 512K and 1M contexts. Native RoPE is still the default; YaRN defaults to 1M when enabled and currently excludes MTP.

3.9 — Streaming and memory

Release v3.9

• KV state now grows on demand, bounded expert I/O is the default, and the server accepts a RAM budget that derives the expert-cache size.
• Six-bit support was withdrawn; current supported installations are 4-bit and 8-bit.

3.8 — Runtime measurement

Release v3.8

• Reduced transition idle time and improved decode throughput while preserving deterministic output.
• Added clearer GPU occupancy, routing, and runtime timing instrumentation.

3.7 — Correctness and release safety

Release v3.7

• Fixed sampler, moved-install, unload, and concurrency defects found in a production audit.
• Added lint, ThreadSanitizer, and real-inference golden-output gates.

3.6 — Model residency

Release v3.6

• Added lazy loading, idle unload, and manual model unload.
• Improved receipt recovery and prompt-cache consistency for fast-alias turns.

3.5 — Robustness

Release v3.5

• Hardened streaming tool calls, cancellation, UTF-8 detokenization, installer durability, and server queue behavior.
• Aligned server sampling defaults with the Mac app and CLI.

3.4 — Launchers and reasoning

Release v3.4

• Added the unified server and coding-CLI launchers.
• Added optional reasoning mode and removed the artificial default response cap when clients omit one.

3.3 — Fast alias

Release v3.3

• Added qwen3.6-35b-a3b-fast for chat-only coding-CLI requests with much less prompt prefill.

3.2 — Runtime controls

Release v3.2

• Corrected expert-cache and prefill defaults and improved API compatibility.

3.1 — Responses API

Release v3.1

• Added native POST /v1/responses support and direct Codex connectivity.

3.0 — Concise mode

Release v3.0

• Added per-model concise prompts and standardized model-directory names.

2.0 — Performance controls

Release v2.0

• Added parallel expert fills, larger expert caches, read advice, larger prefill chunks, and the kernel benchmark suite.

0.1 — Initial version

• Released Qwen 3.6 inference with the installer, CLI, Mac app, bounded expert streaming, and local OpenAI-compatible server.