TinyTitan 5.10

TinyTitan 5.10 — the RAM flag means the whole process, and Qwen3.8's sampling rows are real

--ram used to name the expert cache alone, so --ram 8 produced a process using 11.6 GiB. It now
targets the whole process, with a floor and a printed estimate — a deliberate change to an existing
flag, and this release's headline. Beside it: Qwen3.8's two published sampling rows are implemented,
the C kernels compile at -O2, the expert-cache ceiling is a third of physical memory, twelve
decode switches that measured a wash or a loss are gone, and converting Qwen3.8 resumes instead of
restarting a 360 GB fetch.

--ram is a target for the whole process, with a 4 GB floor

• The flag bounded the expert cache, so the number and the memory used were different things: --ram 8 built a 7.92 GiB cache and, with 3.8 GiB of weights, runtime and KV on top, used 11.63 GiB.
• The cache now gets target - (model_weights.bin + residentRuntimeReserveBytes), where the 512 MiB
  reserve is measured (the server loads at 3.75 GiB against a 3.22 GiB weight file, and the smallest
  cache takes it to 4.74 GiB — exactly the cache delta), and the slot count is the largest supported
  rung whose cache fits, stepping down so the estimate never exceeds the number given.
• Every start prints it:
• TinyTitan ram target=8.00G cache=3.96G slots=32 resident_floor=3.72G estimate=7.68G
• Below the floor (about 4 GiB on the 125B install) the target is refused with the real floor named;
  4G is the minimum accepted.
• This changes an existing flag: --ram 8 now gives 32 slots where it gave 64 (cache 3.96 GiB
  against 7.92), so a 4-bit Qwen3.8 run at that setting is slower than it was on 5.9, and --ram 12
  (64 slots, 7.92 GiB cache, 11.71 GiB peak) is the flag that reproduces the old behaviour.
• Measured on the 24 GiB M3 with peak RSS sampled every 0.5 s (qwen38 4-bit, 256 greedy tokens),
  peak now tracks the estimate to within 0.2 GiB at every rung: 4→8 slots/4.73 GiB, 6→16/5.84,
  8→32/7.83, 10→48/9.83, 12→64/11.71.
• On an idle machine and a 198-token prompt, two interleaved rounds: --ram 8G gives 32 slots,
  8,057 MiB peak in both rounds and 3.34–3.58 tok/s; --ram 10G gives 48 slots, 10,098 MiB
  both rounds and 3.97 tok/s; neither grows swap and both answers are byte-identical. With the
  machine busy (load 6.5) the rates fell ~15% and the 10G arm paged — load moves the rate and the
  peak, not the plan.

Qwen3.8's two sampling rows are implemented, and the row follows the request

• The engine refused every non-zero presence penalty: the sampler's validation threw and the OpenAI
  layer rejected presence_penalty outright.
• Qwen3.8's card asks for 1.5 outside thinking mode, so this implements it.
• Applied in the same host-side, pre-softmax window as the repetition penalty, in the softcap's
  space, once per distinct id in the history.
• Two explicit rows replace the single one: thinking 1.0 / top-p 0.95 / top-k 20, and
  instruct 0.7 / 0.80 / 20 with presence 1.5. A request's own temperature, top-p or top-k
  still beats the row.
• The row is chosen from the request's thinking mode, so a non-thinking request no longer runs at
  thinking's temperature with no presence penalty.
• --presence-penalty (−2…2, OpenAI's range) joins the CLI, and --thinking selects the row, so
  the scripted path can express the instruct row at all.
• min_p is accepted but must be zero: the filter does not exist, and refusing beats sampling as if
  it did. Every shipped row uses 0.0.

The expert-cache ceiling is a third of physical memory, not a half

• min(wanted, physicalMemory / 2) handed a 24 GiB machine a 12 GiB budget and 64 slots — 4.22 GiB
  of cache against a 70.8 MB slot — which pages: swap 855 → 1,610 MB at 5.58 tok/s, against flat
  swap at 7.29 tok/s for the 40 slots a third selects.
• A third also reproduces the budget the decode constants were tuned on and scales down where a
  constant could not (an 8 GB mini previously got 64 slots).
• The cost on the large machine is small: −1.3% decode (16.55 against 16.76 tok/s, three
  interleaved pairs) with first token improving (1.27–1.32 s against 1.44–1.56 s).
• A floor at the tuned budget, so only machines below the tune are cut, is the obvious next
  experiment and is not done here.

The C kernels compile at -O2

• SwiftPM's swiftbuild compiles C at -Os where the older native planner used -O2, which is not
  neutral here: the CPU int8 affine GEMV (8192×8192, 8 threads) runs 2.35 ms a pass at -Os
  against 1.94 ms at -O2, minimum of six interleaved rounds each with an identical checksum —
  1.21×.
• Package.swift sets .unsafeFlags(["-O2"]) on TinyTitanKernelsC, so the portable build carries
  it; the price is that .unsafeFlags makes the package unusable as a dependency, which is fine for
  an application nothing depends on.
• The internal-speed re-record on the -O2 default reads decode +2.5%, prefill +6.5%, first
  token −6.1%.

Twelve decode switches that measured a wash or a loss are gone

• Each was measured on this build, recorded, and deleted rather than left in its losing position;
  the shipped path is unchanged, which the goldens confirm (qwen38-4 and qwen35-4b-4 byte-
  identical).
• Removed: TINYTITAN_PREFETCH_PER_EXPERT (−2.0%/−3.3%), …_PREFETCH_AHEAD=2 (−2.8%),
  …_PREFETCH_TOP_M (−6.6%/−9.8%), …_PREFETCH_MIN_MARGIN (−4.2%), …_PREFETCH_IO_TIER
  (−0.1%/−0.6%), …_EXPERT_CACHE_POLICY (washes), …_CACHE_DECAY_HALFLIFE, …_EXPERT_CACHE_LAYOUT
  (−0.75%), …_EARLY_HITS (+2.2%, 1 of 2 runs), …_KEEP_WIRED (−0.37%), …_PARALLEL_IO (+0.4%, 2
  of 3 runs) and …_PREFILL_LAYER_MAJOR (−2.7×).
• Setting one is now inert: the winning default is what runs.
• The options kept are product API or hold an unmeasured balance, and the wiki's runtime-controls
  page lists them.

Converting Qwen3.8 resumes, and mirrors work

• Building the Qwen3.8 snapshot means fetching 131 shards, 360 GB, and any interruption used to mean
  fetching all of it again.
• The conversion now:
• adopts the output shards a previous run finished and skips the checkpoint shards whose every
  tensor is already present. A shard is adopted only when its payload matches the header's declared
  size — a kill mid-write leaves a file whose header parses and whose payload is short, and trusting
  the header alone is how a truncated snapshot gets indexed as complete. Flushes are written beside
  the destination and renamed, so a kill leaves a *.partial the next run deletes.
• reuses a finished n-gram table in place when its size and the constants addressing it match,
  builds a new one under a temporary name and renames it only when whole, and copies rather than
  hardlinking across filesystems; the installer passes --share-ngram-table only when staging and
  the install share a device.
• retries a shard six times with backoff and a 20-minute --max-time, and treats curl exit 33
  (a mirror answering a range request with 200) as "drop the partial and start that file again"
  instead of retrying a request that can never progress.
• fetches through a mirror via HF_ENDPOINT or --endpoint on the Hub's URL layout, for the
  weights, the small JSON files and the tokenizer alike.
• refuses two states rather than duplicating work: a directory with finished
  model-*-of-*.safetensors but no index, and a resume whose recorded width differs from the
  --bits in hand.
• Thirty-six end-to-end cases (benchmark/test_qwen38_resume_e2e.py) run the real converter and
  real curl against a local mirror with injected drops, truncations, 404s, stalls and refused
  ranges, apply a real SIGKILL mid-conversion and a cross-filesystem copy, and require every
  recovery to end byte-identical to a clean run.
• They run in CI.

Also in this release

• Prefetch depth 1 is the Qwen3.8 profile default, re-measured on this engine: decode +15.7%
  at a 7-token prompt and +14.6% at ~500 tokens, expert misses −17% / −11%, responses byte-
  identical. The 8-bit row inherits it by inference — the ring is family-level and width-
  independent, and that install is not present to A/B — and the row comment says so.
• The native-build experiment is gone: tools/build-native.sh was dropped after its CPU flag
  measured ~1% (noise) on top of the -O2 default, and the finding stays in AGENTS.md for a non-
  portable build.
• The RAM-budget curve is measured 1–16 GB, the page-cache trade is +4–5% decode for nothing on
  prefill, and three lines are closed with numbers: layer-major prefill (−2.7×), lossless
  compression of expert reads, and moving the decompression to another engine stage.

Performance

• Measured on this commit for this release against the 5.9 record (benchmark/internal- speeds/v5.10.json), on an otherwise idle 24 GiB M3:
• GPU QKV GEMV 67.4 GB/s (+6.1%), routed MoE 41.4 GB/s (−5.0%), GDN in-projection 76.8
  GB/s (+13.1%);
• CPU int8 affine GEMV 54.7 GB/s (+15.2%);
• generation prefill 25.9 tok/s (+7.4%), decode 26.1 tok/s (+4.2%), effective decode
  70.7 GB/s (+4.3%), first token 0.27 s (−6.9%);
• ANE prefill 50.1 tok/s (+3.5%);
• the greedy response is byte-identical to 5.9 (quality.response_sha256 unchanged), and so are
  coverage (0.417) and trigram repetition (0.0958) — no arithmetic moved.

Verification

• Measured on this commit by the release dry run:
• six lint gates clean, 2,030 functions scanned, the shell gate over 20 scripts on bash 3.2.57;
• 1,491 tests in 223 suites, all passing;
• 7 golden baselines byte-identical — qwen36-4, qwen36-8, qwen38-4, qwen35-4b-4, qwen35-4b-8,
  qwen35-9b-4, qwen35-9b-8;
• a clean scratch release build with the compiler-warning scan clean, and the archive staged and
  packaged from that tree;
• the engine's speeds re...

TinyTitan 5.9

TinyTitan 5.9 — the GDN pair loads at the slot's width, and memory's value is measured

One runtime fix carries this release: an install whose manifest names the GDN
linear_attn.in_proj_a/in_proj_b pair explicitly at the attention slot's own width is honoured
again instead of refused on load, which is what stopped a qwen38flash 4-bit install from serving
at all (issue #16). Beside it is a one-line ordering fix the thread-sanitizer gate caught on this
release's own commit, and the rest of what landed since 5.8 is measurement: the ten master prompts
are runnable end to end, and which model judges the side-engine's tasks — the resident 4B on the CPU
or the model already on the engine — is now a number instead of a belief.

A GDN a/b override at the attention slot's width is honoured

• Model.validateRoleUniformity refused every quantized override on the GDN
  in_proj_a/in_proj_b pair, comparing only against 16, because the kernel that reads the pair
  takes a bf16-or-slot flag rather than a fixed width.
• That is correct for an override at some third width and wrong for one that names the attention
  slot's own width: the runtime already reads the pair at that width, so refusing it is refusing the
  width the kernel is using.
• A qwen38flash 4-bit install that spells the pair out at 4 bits therefore failed on load with
  in_proj_a.weight size N does not match expected M, which reads like corruption rather than a
  limit (issue #16, reported against c20f688).
• The check now takes attentionBits and honours an override that is either the slot's width or
  bf16, naming the accepted width in the refusal it still makes for any other
  (sources/TinyTitan/Runtime/Inference/Model+Loading.swift).
• Verified on this checkout by adding that override to the shipped 125B manifest and its receipt:
  the reported error fired verbatim, and with the fix the same install answered.
• RoleUniformityTests pins the regression, and the qwen38 4-bit golden passes unchanged.

A T7 hint is queued before the search returns

• The T7 background caller's registration was fire-and-forget: memory_search scheduled the
  question in an unstructured task and returned, so the question could still be unqueued when the
  search's answer was.
• Nothing on the request path waits on a judgement either way, but the thread-sanitizer gate widened
  that window until MemoryRetrievalTests failed on the release commit — a caller that awaits the
  hint right after a search could arrive before the question was registered.
• Registration is now awaited and only enqueues; the sweep still runs on its own task in the idle
  window (sources/TinyTitanMemory/MemoryService.swift).

The ten master prompts are runnable, and memory has a baseline that is not "off"

• benchmark/master_scenarios.py holds the ten long-session worlds as data — a session-1 brief that
  fixes the facts, per-session instructions, the sessions that change one, and a quiz scored against
  what is true by then — with foundation (never changes) and carryable (changes at least once)
  derived from the truth rather than authored beside it (benchmark/test_memory_scenarios.py).
• benchmark/memory_master.py scores the stored answers, and benchmark/memval_master.sh runs all
  ten, one invocation per scenario.
• Four of the ten are complete and recorded in docs/master-benchmark-results.md (photograph,
  pong, ledger, pigeon).
• The result that matters is the baseline: a client's own 200-word summary — what a compaction does
  — wins or ties on three of the four, so memory has to be argued against that, not against
  memory-off.
• Memory's win appears exactly where the design predicts, a dense, arbitrary, revised fact set: on
  pigeon it carries the carryable keys 20/20 against the summary's 14/20, and the summary
  is the arm that goes stale.

Which judge: the 4B on the CPU, or the model already loaded

• The side-engine's decisions had a default judge — a dense 4B on the CPU — chosen for what it does
  not cost.
• benchmark/side_engine_judges.py runs the same prepared case file through either judge
  (cpu:<install> or server:<url>:<model>) and scores both with the task scorer, so the choice is
  measured.
• Bigger is not uniformly better: the served 35B is worse on duplication (75% against the 4B's
  100%, reading characters/marcus/eyes = grey and notes/marcus = marcus's eyes are grey as
  different facts) and better on the reply check the 4B cannot do at all (100% against 62%).
• Split by task the pair beats either alone.
• Three more worlds (ops, lab, contract) were added for diversity, producing 53 cases; the
  full matrix and what it does not change yet are in docs/side-engine-tasks.md.

Also in this release

• The plugin is catalogued. dsh-tinytitan is listed in awesome-dsh-plugin — PR #5396 merged
  2026-09-19 — so only the npm publish remains, and that is an operator action. docs/dsh-plugin- publication.md says what is left.
• The harness asks are answered and recorded, with the correction the replies forced (docs/dsh- upstream-asks.md): the auxiliary-reasoning failure is plugin-side (dsh #7109), the LAN manager's
  remote reach is blocked by the webserver schema's two host literals rather than the startup guard
  (#7111), and a bind would also have to fold the bound address into resolveLanTrust.
• The handover brief now starts from 5.8 (docs/handover-tinytitan.md).

Performance

• The README's benchmark table was not re-measured for this release; its rows are quoted as they
  stand.
• Measured on this commit for this release against the 5.8 record (benchmark/internal- speeds/v5.9.json):
• routed MoE 43.6 GB/s (−1.4%), CPU int8 GEMV 47.5 GB/s (−3.5%), prefill 24.1 tok/s
  (−6.9%), decode 25.0 tok/s (−6.4%), first token 0.29 s (+7.4%), effective decode 67.8
  GB/s (−6.4%), ANE prefill 48.4 tok/s (−1.9%) — every one inside the 10% gate;
• the greedy response is byte-identical to 5.8 (quality.response_sha256 unchanged), so no
  arithmetic moved;
• the two synthetic kernel metrics read low — QKV GEMV 63.5 GB/s (−19.2%) and GDN in-projection
  67.9 GB/s (−12.3%) — and ### Verification records why: the machine was under system-
  maintenance load, and those metrics have ranged 55.4–78.6 and 66.8–77.4 GB/s across the v5.5–v5.8
  records here.

Verification

• Measured on this commit by the release dry run:
• six lint gates clean, 2,035 functions scanned, the shell gate over 20 scripts on bash 3.2.57;
• 1,484 tests in 222 suites, all passing; and 7 MemoryRetrievalTests under
  --sanitize=thread, 15 runs in a row, after the ordering fix above;
• 11 golden baselines byte-identical;
• a clean scratch release build with the compiler-warning scan clean, and the archive staged and
  packaged from that tree;
• the engine's speeds recorded against the 5.8 baseline and committed (benchmark/internal- speeds/v5.9.json): every generation metric and the quality proxy inside the gate, with the greedy
  response hash unchanged.
• Two synthetic kernel metrics are past the 10% speed gate and are not a code regression. QKV
  GEMV measured 63.5 GB/s against 78.6 (−19.2%) and GDN in-projection 67.9 against 77.4 GB/s
  (−12.3%).
• The measurement ran while macOS's Duet Activity Scheduler held a core at ~95% and Chrome was
  active (dasd sampled at 93–97% throughout, load average ≈6); the same synthetic metrics have
  ranged 55.4–78.6 GB/s (QKV) and 66.8–77.4 GB/s (GDN) across the v5.5–v5.8 records on this
  machine, with 5.8 the series' high-water mark — each earlier release shows the same first-run-low,
  re-run-high pattern the 5.9 re-run repeated.
• Nothing in this release touches those kernels (the runtime changes are a load-time validation and
  a memory-path ordering fix), every generation metric came back inside the gate on the re-run, and
  the greedy response is byte-identical to 5.8.
• The record is committed as measured rather than re-rolled to flatter it.
• Five golden targets are not checked, because their install is not under models/ and nothing
  may be fetched to change that: ornith-8, ornith-4, qwen38-8, katcoder-4, katcoder-8.

Checksum

tinytitan-5.9-macos-arm64.tar.gz sha256: 2c9a6657a5516dcb754837542b20f95122a02370be10dc983c1e581a4030f8d7
tinytitan-5.9-macos-arm64.tar.gz size: 15437857 bytes

TinyTitan 5.8

TinyTitan 5.8 — a resident side-engine for memory, and a LAN manager for a fleet

Agent memory grew a second, small model: a 4B on the CPU that answers one closed question at a time
— is this fact worth keeping, do these two say the same thing, do they disagree, which kind of
change is this, could this fact answer this question — while the main model keeps the GPU and the
person keeps their turn. Around it, memory learned to hold a write a stored rule forbids, to rank a
search by how rare a term is rather than how often it appears, and to let the retrieval question
work in the background instead of on the request path. The repository also gained dsh-lan-manager
and ttlanmanager, a LAN-scoped control plane for a fleet of DeepSeek Harness instances. Everything
here was already on main.

Agent memory has a resident side-engine

• A small Qwen 3.5 runs on the CPU as a resident helper and decides one thing at a time, because a
  small model composes badly but decides well (docs/side-engine-tasks.md).
• The 4B 4-bit install is the default and the verification instrument; the 9B is optional and buys
  the reply check while being worse at durability; the 2B is not used.
• Only the tasks with a caller are on the port, and the caller is consolidation.
• A model-derived fact is asked first whether it is worth keeping (T2), and one the engine
  rejects is not stored at all.
• A changed value the store already holds is asked which kind of change it is (T4) once a stored
  rule fixes it.
• New keys are checked for duplication (T5) and contradiction (T3) against the session's own
  scope and the shared workspace.
• nil is "no decision", so an absent, shut-down or confused engine leaves the deterministic path
  exactly as it was.
• The questions are budgeted like the model calls they are: measured at 15.2 s on the 4B, so one
  consolidation puts at most six questions in total and three to any one fact — about a minute and a
  half in the pause consolidation already runs in.
• Verified against the real model: the release-only test drives durability, duplication and
  supersession on the 4B (133.1 s for six judgements) and the 9B (255.4 s), and the task matrix is
  in docs/side-engine-tasks.md.

A stored rule can hold a write back

• T4 was measured ready but unreachable: it decides 100% once the stored rule is supplied, and
  nothing supplied one.
• MemoryRuleLookup now finds it by key — the last segment of the changed key, so
  characters/marcus/eyes looks for rules/eyes — which is free, because it is a key match and not
  a model call.
• A .conflict keeps the old value and logs only the key, never the rule or either value; .update
  changes nothing.
• Only a model-derived fact is asked about, so the person always overrules a rule, and a change with
  no rule is not asked about at all.

Retrieval ranks by rarity, and T7 works in the background

• The token ranking scaled every term the same, so a question naming a common word won: "does it
  ever rain in this town?" matched the town's key for 3 and the rain rule's value for 2.
• MemoryRanking now weights each term by its inverse document frequency over the candidates, which
  takes the authored paraphrase set from recall@1 1 of 4 to 3 of 4 while the ten questions
  phrased in the store's own words stay 10 of 10 (benchmark/side_engine_recall.py --baseline).
• The one miss left shares no term in any form — "How often does the boat cross the water?" against
  rules/ferry = runs only on Sundays — so no weighting reaches it.
• T7 does, and it now has a caller that can afford it: MemoryRetrievalHinter registers the
  question a search just asked and walks the scope's facts only in the idle window
  (ServerCoordinator.generating read as isIdle), never on the request path.
• Each YES becomes a ranking hint keyed by the fact plus an FNV-1a fingerprint of the value it
  judged, so the next search for that question puts the hinted facts first, including one the token
  ranking never returned.
• It covers at most 64 facts a question and keeps 16 questions, least recently asked evicted; a
  value that changed cannot be promoted on an answer about the old one.
• Without an engine, or while a client is generating, the path is byte-for-byte the token ranking it
  was.

The n-gram table is shared between builds instead of copied

• Qwen3.8-Flash-Next's ngram_table.bin is 102 GB, and every quantization of the model stored its
  own copy.
• TinyTitanRepack --share-ngram-table hardlinks it from the source snapshot instead, and
  prepare_qwen38.py --reuse-ngram-table links an existing table into a new build.
• The table is a hash table with no self-description, so a wrong one would read silently as garbage
  ids: the reuse gate refuses a table whose PLE constants differ, is pinned by five cases in
  benchmark/test_prepare_qwen38.py, and the link-and-size path has a Swift test.

Per-tensor bit widths resolve in the resident index

• HyperConnection, PLEBlock and QSAIndexer read weights whose width came from the attention
  slot, and the loader never checked those tensors' widths — so a per-tensor override on them was
  unhonoured and unguarded: packed at one width, read at another, silently wrong.
• The three families now resolve their width through the manifest's overrides with the slot as
  fallback and validate the resolved width against the kernel, with the GDN a/b pair refused a
  quantized override by name.
• This is the enabler for the ~10 MB promotion tools/precision_plan_qwen35.py proposed.
• Its quality case is measured twice and is not there: the 4B's own 16 MB k_proj/v_proj
  promotion scores 18/20 against a control's 18/20 on a twenty-prompt suite, and on a paired
  held-out perplexity A/B it is 0.0097 ± 0.0067 nats ahead of the control (t −1.46), inside the
  instrument's ~0.013-nat floor.
• The converter policy is therefore deliberately not built (benchmark/quant_perplexity_ab.py).

The DSH LAN Manager: a control plane for a DSH fleet

• plugins/dsh-lan-manager gives a DeepSeek Harness host a LAN-scoped management API — list active
  workspaces and their sessions, prompt one or all of them, read a session's messages back, archive
  or delete, start a session through the harness's own controller, and an aggregate inventory of
  every member of the group.
• Members find each other over tailnet peers, Bonjour, configured peers and an optional subnet
  sweep, and every gossiped address is validated against the same allowlist the request fence uses
  before anything is dialled.
• sources/TinyTitanFleet builds ttlanmanager, a live terminal dashboard over that API with a
  pure renderer and key map (so layout and keys are tested with no terminal).
• It is an operator tool and deliberately not part of the installed engine.
• Verified by 44 tests in 11 suites; reaching it from another machine is blocked upstream, because
  the harness refuses any bind but loopback (TT-020).

Both DSH plugins pin exactly one harness release

• dsh-tinytitan and dsh-lan-manager support 0.1.6-alpha.2 and refuse any other release —
  older, newer, a build from main, or one whose version cannot be read — with one line on stderr
  and no writes into the harness home.
• Neither throws, so DSH still boots and removing ours leaves nothing to undo.
• The pin lives in the peer dependencies, the launcher and each plugin's constant, and CI now runs
  both plugin suites, which nothing did before.

Also in this release

• Steady-state decode past the ANE handover costs nothing — +0.7% on AgentWorld 35B-A3B 4-bit,
  measured after the old figures turned out to be a ~60-token window (TT-007).
• The Qwen3.8 expert-cache budget stays at 96 slots (TT-011), and TINYTITAN_KEEP_WIRED is a
  tri-state so =0 pages the expert cache out (TT-008).
• The MTP verify pass is attributed on the installed pair (TT-006), the ANE re-warm is separated
  from drift and the pin removes it (TT-005), the E5RT arenas are returned (TT-004), and the top-2
  logits of both sampling paths are traced (TT-002).
• The QSA host prefill selection is 2.75% of a long prefill, measured before building a GPU
  path, and the GPU path is not built (TT-010).
• The ThreadSanitizer gate carries one top-frame suppression (TT-001), and the project has a
  written standard for its single open-task table (docs/task-table-standard.md).

Performance

• The README's benchmark table was not re-measured for this release; its rows are quoted as they
  stand.
• Measured on this commit for this release:
• the engine's own speeds against the 5.7 record (benchmark/internal-speeds/v5.8.json): QKV GEMV
  78.6 GB/s (+4.4%), routed MoE 44.2 GB/s (+4.0%), GDN in-projection 77.4 GB/s (0.0%),
  CPU int8 GEMV 49.2 GB/s (+1.7%), prefill 25.9 tok/s (0.0%), decode 26.7 tok/s (+5.2%),
  first token 0.27 s, ANE prefill 49.3 tok/s (−2.6%). No metric is past the 10% gate and the
  response hash did not change;
• the side-engine's wired judgements, 15.2 s each on the 4B;
• the held-out perplexity A/B, 1,023 paired token positions per install, about three minutes per
  install on the CPU.

Verification

• Measured on this commit by the release dry run:
• six lint gates clean, 2,035 functions scanned, the shell gate over 19 scripts on bash 3.2.57;
• 1,482 tests in 222 suites, all passing;
• 11 golden baselines byte-identical: the 125B at 4-bit, AgentWorld 4- and 8-bit, qwen36 4- and
  8-bit, and the dense 2B/4B/9B at both widths;
• a clean scratch release build with the compiler-warning scan clean, and the archive staged and
  packaged from that tree;
• the engine's speeds recorded against the 5.7 baseline and committed (benchmark/internal- speeds/v5.8.json), every metric inside the gate.
• Five golden targets are not checked, because their install is not under `m...

TinyTitan 5.7

TinyTitan 5.7 — one command installs a built engine, and the app is gone

The install no longer builds anything: tools/install_tinytitan.sh downloads the published arm64
executables, verifies the checksum, unpacks them under ~/.tinytitan and asks one question — which
model — so a Mac with no Xcode, Homebrew, Python or Node can go from nothing to a served model. The
Mac app is removed, leaving the engine and its loopback server as the whole product, with an
optional browser window as a client of it. And every script now runs on /bin/bash 3.2.57,
which is the shell a factory Mac actually has — the launcher did not even parse there before this
release. Everything here was already on main.

The installer downloads the engine instead of compiling it

• It cloned the source and ran swift build, so a new Mac needed Xcode and ten minutes of compiling
  before anything worked — while the release already publishes the same four executables, built for
  arm64, in a 25 MB tarball.
• It now: checks the Mac; downloads the newest release's tarball, verifies its published sha256 and
  unpacks it into ~/.tinytitan/bin; downloads the matching tag's source for the tools, the DSH
  plugin and the docs into ~/.tinytitan/src; asks which model; writes ~/.local/bin/tinytitan and
  ~/.local/bin/tinytitan-web; offers to start, and opens the page.
• --version TAG pins a release instead of taking the newest and --from-source keeps the clone-
  and-build path for contributors.
• The launcher, the model installer and the route writer all take TINYTITAN_BIN_DIR and
  TINYTITAN_MODELS_DIR, so an installed ~/.tinytitan/bin and a checkout's .build/release are
  the same code path; checkout defaults are unchanged.
• Verified against the real published artifact — v5.6, because a tag cannot name the release it is
  in — in an isolated HOME: download, checksum, unpack, tools, wrappers, and the installed
  TinyTitanServer --catalog against real installs.
• Then the launcher from this checkout, pointed at that bin, started the installed binary on a
  real model and answered a completion with 42 and finish: stop — a release install serving with
  no .build anywhere.
• Not verified: the model download itself, a 20–37 GB fetch that must never be run to satisfy a
  check.

The model is a menu, not a yes/no about one default

• The installer asked a yes/no about exactly one model, so anything else meant knowing the target
  name and finding install_models.sh --help first.
• It now lists all 16 builds with their installed size and what each is for; Enter takes the
  verified default, so the shortest path is still one keypress.
• --model NAME skips the menu, and through a pipe the installer takes the default and says so
  instead of hanging — --choose refuses a pipe with a usable message.
• The list lives in TINYTITAN_MODEL_CHOICES next to the client catalogue so the two installers
  cannot disagree about what exists or how big it is.

Every script runs on the shell a factory Mac has

• #!/usr/bin/env bash finds Homebrew's 5.x on a development machine and /bin/bash 3.2.57 on
  a new one, and these scripts had only ever run under 5.x.
• Three classes of defect were found and closed, each verified by running it:
• Parse. 3.2 cannot parse a single-quoted heredoc holding an apostrophe inside $( ). The
  launcher's expert-cache warm-up did exactly that, so /bin/bash -n tools/server_launcher.sh
  failed with unexpected EOF while looking for matching ' — on a new Mac not one line would have
  run. The warm-up is built with printf now, which also drops a python3 requirement from a path
  that must work without one.
• Run time. ${v^^} and mapfile are bash 4; 3.2 parses them and then dies mid-menu, in the
  launcher's engine column, in dsh_route.sh --from-server and in repack_dense.sh. All three use
  tr and a while read loop.
• Empty arrays. Under set -u, "${a[@]}" on an empty array is a[@]: unbound variable on 3.2
  and fine on 5.x. The shipped --web path had two — the default browser window died before dsh
  was exec'd. All 62 whole-array expansions in the tree are now ${a[@]+"${a[@]}"}, which means the
  same thing for a non-empty array on both shells.
• tools/lint.sh shell fails on any of the three shapes now, and scans every shell script in the
  tree (including docs/).
• It was validated by injection: a bare array expansion and a ${v^^} each produce a FAIL and exit
  1.
• Verified by running the paths under /bin/bash 3.2.57, not by reading them: every script parses
  under both shells; the installer completes a real release install (download, checksum, unpack,
  wrappers) under 3.2 in an isolated HOME; the launcher dry run, the model menu, the status table,
  the DSH status and the engine column all run under it; and tools/lint.sh all is clean under
  3.2.57 and 5.3.20 alike.

An optional browser chat window, isolated from any dsh you run

• A user who wants a window now gets one already pointed at their model: TinyTitan's own DeepSeek
  Harness, opened in the default browser by the launcher's --web.
• Nothing is built or forked — it is upstream's MIT harness plus the plugins/dsh-tinytitan bundle
  this repository already ships, installed under ~/.tinytitan with its own DSH_HOME, npm prefix,
  pnpm store and port (7788, stepping up to the first free one).
• The user's ~/.dsh, a dsh on PATH and a stock UI on 3080 are never read, written or stopped;
  Node is reused when the Mac has one and fetched privately only when it does not.
• The harness version is pinned to 0.1.5-rc.2, because DSH is a developer preview that says
  outright it will break compatibility between releases.
• Driving the real page in a headless browser found what a curl cannot: the plugin rewrote the
  route's port at every boot; the route forced thinking on against a server started with it off (the
  page sat on "Deep diving..." while the model spent 32768 tokens reasoning and answered nothing);
  and a fresh home has no workspace, which disables the composer.
• The private home now seeds one, marks the harness's developer-preview notice as seen, and points
  the harness default at our route instead of DeepSeek's hosted one (MISSING_CREDENTIAL: llm- deepseek).
• Three tunings make the window usable on the intended 35B MoE, measured on Qwen-AgentWorld 35B-A3B
  4-bit:
• the chat preset drops the three injected-context rows a prompt box does not want: 4222 → 124
  prompt tokens for a nine-word question;
• --web warms the expert cache before opening the window — 89 s at startup, so the person's
  first question does not pay the cold sweep (161 s cold against 72 s warm);
• together, the first answer through the real page lands in 8.3 s of engine time and the smoke
  test passes in 16 s wall clock, against 161.5 s and 2 m 53 s before.

The Mac app is removed

• The app, its out-of-process decode service, its library and test targets and its icon generator
  are gone.
• A second front end is a second surface to build, keep in step with every engine feature, and
  support: the supported way to use a model is the loopback OpenAI-compatible server with a client
  you already have — Zed, Codex, Claude Code, DeepSeek Harness, curl — and --web is a client we
  merely install and configure.
• Removing it also removed the tree's only other version literal, CFBundleVersion;
  ServerVersion.current is the single one, and release.sh refuses a tag that disagrees with it.

Server: POST /v1/responses/compact

• The Open Responses compaction endpoint, implemented as the value endpoint it is — a conversation
  in, a compacted input window out, nothing stored and no session started.
• The note is metered with the server's own tokenizer and one over budget is compressed by a
  second pass rather than truncated, because truncation drops the end of a session, which a
  continuation needs most.
• Three guards keep a bad pass out of the caller's history: instruction lines the model copied back
  are stripped, a repetition loop is recognised as a failed pass, and a pass that says nothing
  usable falls back to the newest text trimmed to budget; mode reports which path produced the
  note.
• The summariser runs at temperature 0 with thinking off, since a model that reasons inside its
  own output cap returns an empty note.
• Verified on the 4B and 9B installs at 4-bit: an eight-turn session compacted in 16.5 s and 39.7 s,
  both notes keeping all four load-bearing facts, and the model answering from the replayed window.

Server: the concurrent width is any power of two

• --max-concurrent-sequences was capped at 4, a policy limit rather than an engineering one; it
  takes any power of two up to 256 now, and the launcher offers 1 / 2 / 4 / 8 / 16 or a custom
  entry.
• The default stays 1, the warning above 1 stays, and two lines were added above 16 saying the clamp
  is likely to bind.
• KVCacheManager.maximumSlots moved with it (8 → 256), so a large request is served or clamped
  rather than tripping a precondition; what a machine can really hold is still decided per load by
  BatchedMemoryBudget.
• Verified against the 2B at a 32k context on a 24 GB Mac: 32 builds 32 slots and serves 16
  concurrent requests with no 429, and 256 is accepted and clamped — "per-slot 223 MiB, budget
  12288 MiB; serving 54 at once".

Also in this release

• The build floor is Swift 6.4 (Xcode 27) for source builds; the published binaries need none of
  it.
• AGENTS.md now says ad-hoc model runs use the 4B or 9B, not the 2B — the 2B fails
  instruction-following in ways that read as defects in the code under test.
• The forum article series is removed with the forum; the wiki is the user documentation, and
  every live reference was repointed.
• Repository cleanup. `docs/adding-a-...

TinyTitan 5.6

TinyTitan 5.6 — the three reported bugs, and an ANE answer of "no"

Closes the three bug reports open against 5.5 — the app could not change models, the model installer
looked for its binary in the wrong directory, and the ANE exporter would write a sidecar the Neural
Engine had refused and then run the whole prefill on the CPU — and records what a year of Neural
Engine work actually bought. Also raises the build floor to Swift 6.4, gives the server per-session
KV and GDN state, and turns the wiki into a user guide with a Cookbook. Everything here was already
on main.

Change the model from the app (#9)

• The app read a persisted model preference at launch and had no way to write it, so the model
  could not be changed from the app at all.
• Model → Change Model now lists every selectable build, records the choice and shows a banner
  saying the app has to be reopened — the model directory, its settings file and the decode-service
  process are bound at launch, and the runtime has no supported way to rebind them under a running
  generation.
• A directory name is accepted as a selector too, and the checkout is found on a case-sensitive
  volume.
• Backed by swift test --no-parallel --filter AppModelSelectionTests (8 tests): every recognizable
  build is selectable, the choice persists and round-trips, and a change is refused while a
  generation or an install is in flight.

The installer finds its own binary (#8)

• tools/install_models.sh built its path from the target triple (.build/arm64-apple- macosx/release/TinyTitanRepack).
• That directory exists only when SwiftPM's triple happens to match the toolchain — macOS 26 and 27
  differ — and when it does exist it can point at a stale tree.
• It now uses the stable product path, and tools/lint.sh arch-path fails if any build path
  hardcodes a triple again.

A sidecar the Neural Engine refused can no longer be written, or run (#7)

• Exporting with a large --max-history produced .mlpackage files Core ML could not compile for
  the ANE.
• The exporter exited 0 and wrote the sidecar, and the runtime then ran the whole prefill on the CPU
  — measured at roughly 38× the GPU cost, with no error and no fallback line.
• Three layers now stand in the way, each catching something the one before it cannot:
• Compile markers — Core ML reports an ANE refusal on the native stderr and still returns, so
  the exporter reads it back and refuses the export.
• Load — every function the metadata records must load under its h<history> name. A variant
  that loads but can never predict() is caught here.
• Assignment — the exporter asks MLComputePlan which device each operation is assigned to and
  fails on zero. Measured on the real 3.8 h12288: 0 of 173 operations on the Neural Engine,
  against 64–74 for a healthy variant. This is the layer that closes the silent case, because a CPU
  fallback prints nothing.
• The runtime refuses a sidecar that does not record aneCompileVerified, so a sidecar written by
  an older exporter is declined and the GPU path is used rather than a 38× prefill.
• tools/verify_ane_sidecar.py keeps the graph check that a wrong geometry cannot pass.
• 22 exporter tests, including the marker scan, which had none.

Sparse-indexed attention is now correct on the ANE — and still stays on the GPU

• Qwen 3.8's full-attention layers choose keys with a QSA indexer, so a sidecar fed the causal mask
  attended to keys the model drops past 2,051 visible keys.
• The runtime now folds the indexer's compacted selection into the additive mask the graph already
  takes — no graph change — and refuses to attend densely: a chunk past the dense-exact window with
  no selection throws.
• Correctness was measured, because the obvious check was not sensitive enough:
• the graph tracks an independent NumPy reference under a QSA-shaped mask to 0.47 %, against
  7.6 % for the causal-only mask, on both widths;
• a 32-token greedy continuation is textually identical to the GPU — and so was the causal-only
  control, which is why the logits (13.6–17.8 % apart) are what separate them;
• lowering the QSA budget from 2,048 to 8 moves L3_after by 10.2 % and the logits by 69 %, so the
  mask demonstrably reaches the graph.
• Measured, the ANE loses on this model: 0.72× at 4-bit and 0.87× at 8-bit. The GPU path already
  gathers ~2,051 keys while the ANE scores the whole context and masks the rest, and per-variant
  setup dominates — one variant loads in 6.7 s (h0) / 13.6 s (h4096) / 37.5 s (h8192) against
  the ~0.5 s the runtime's own note records for the 35B.
• So no sidecar is installed for 3.8 and tools/ane_sidecars.sh skips it.
• The gather-graph variant that would remove the extra arithmetic was sized and rejected: a gathered
  key must be materialised once per query that selects it — 64× the dense score matrix, 103 GB at
  the real chunk — and measured it is 5.70× slower where it runs and fails one chunk size up.
• The full write-up is on the wiki: ANE Prefill
  Research.

The build floor is Swift 6.4 (Xcode 27)

• The manifest declared 6.3 while the tree already carried a Swift 6.4 workaround (the generation
  decode state is boxed because 6.4 rejects sending the captured mutable struct).
• The floor is now the toolchain this project builds and tests on.
• Building from source needs Xcode 27 or a matching Swift 6.4 toolchain.

The server has per-session state, and serves one session at a time

• KVCacheManager and GDNStateManager gained slots, so each admitted session's prefill and decode
  land in its own KV and GDN state, and the forward step is serialized because the runner's scratch
  is not per-slot.
• Two wiring defects that made four admitted requests serialize at the engine were fixed, and so
  were three in the prefill path that made slots above zero decode the prompt to garbage.
• The batched width is deliberately held at 1. Width > 1 produced degenerate output on the real
  2B and 4B installs, in both int8 and fp16 KV, while width 1 and the CLI are coherent — so the
  server admits several sessions but serves them one at a time until multi-session output is
  verified correct on the real models.
• The Responses API also echoes the sampling the server actually used, instead of null where the
  schema requires a number.

The dense Qwen 3.5 family reaches the ANE

• The dense installs were on the 128-token default prefill chunk, and the sidecar is a fixed
  4,096-token program that only engages when the configured chunk matches it — so the family saw no
  ANE prefill even though the switch has been on by default since 4.6.
• They now run the 4,096 chunk: 1.30× faster prefill at ~2,500 tokens on the 2B.

Also in this release

• One instruction file. AGENTS.md is the only agent instruction file (Claude Code reaches it
  through the committed CLAUDE.md), and RELEASE.md carries the release and build standard for
  this repository. Both are edited here; nothing is deployed from another repository.
• The issue workflow is written down in AGENTS.md: verify the report against the code, fix
  only what is true and unfixed, test the guard, verify again with the reporter's own reproduction,
  audit, reply with evidence, close.
• A wiki user guide. The wiki gained a
  Cookbook — a recipe per common task with
  the command and the output to expect — and a Technical
  Articles section for engineering
  write-ups that were previously nowhere.
• Repository cleanup. The processed deep-audit register was removed (nothing in it was open, and
  its references were followed and updated), the project tracker no longer carries a closure log,
  and three files were split as pure code motion: ServerInference.swift (1,897 → 1,664 lines),
  RealForwardRunner+Decode.swift (1,829 → 1,572) and RealForwardRunner+Prefill.swift (1,875 →
  1,524).
• Internal speeds are now a release gate with a committed baseline, so a kernel or bandwidth
  regression fails a release rather than being noticed later.

Performance

• The README's benchmark table was not re-measured for this release, and the previously
  published rows are quoted as they stand.
• What was measured on this range:
• Qwen 3.8 125B-A6B prefill, ~4,333-token prompt, chunk 4,096: the ANE is 0.72× the GPU at 4-bit
  and 0.87× at 8-bit, so this model stays on the GPU (benchmark/ane- prefill/v5.6-ane-38-{4,8}bit.json).
• The dense 2B at ~2,500 tokens: 23.33 s → 17.88 s, 1.30×, with the 1,024-token sidecar
  (benchmark/ane-prefill/ane-chunk1024-2b-4bit.json).
• The gather-graph probe: the ANE accepts the gather but is 5.70× slower where it runs and fails
  outright one chunk size up (benchmark/ane-prefill/ane-gather-probe-v5.6-gather-*.json).

Verification

• Measured on this commit, by the release dry run:
• five lint gates clean, 2,108 functions scanned;
• 1,566 tests in 237 suites, all passing;
• 9 golden baselines byte-identical: the 125B at both widths, AgentWorld 4-bit, and the dense
  2B/4B/9B at both widths;
• a clean scratch release build with the compiler-warning scan clean, and the archive staged and
  packaged from that tree.
• Seven golden targets are not checked, because their install is not under models/ and nothing
  may be fetched to change that: ornith-8, ornith-4, qwen36-4, qwen36-8, agentworld-8,
  katcoder-4, katcoder-8.
• The engine's own speeds are recorded against the 5.5 baseline and committed with this release
  (benchmark/internal-speeds/v5.6*.json).
  -...

TinyTitan 5.5

TinyTitan 5.5 — the name, structured output, and a harness route

First release under the project's own name. Carries the two features that landed
after 5.4 — JSON the sampler is not allowed to leave, and a thinking switch that
belongs to the request instead of to load time — plus the launcher, client and
DeepSeek Harness work around them. Everything here was already on main.

The project is TinyTitan

• One mechanical rename — NVMAI_→TINYTITAN_, NVMAI→TinyTitan, nvmai→tinytitan — across the package, all 28 SwiftPM targets and their 581 paths, the executables, env vars, launcher, benchmark scripts, DSH bundle, docs, wiki and repository. The old repository URL redirects.
• Binaries are now TinyTitanServer, TinyTitanMac, TinyTitanCLI, TinyTitanRepack, TinyTitanDecodeService, TinyTitanBench. A stale .build/ may still hold the old NVMAI* executables; nothing updates them.
• Env vars are TINYTITAN_* (TINYTITAN_PORT, TINYTITAN_MODELS_DIR, TINYTITAN_CLIENTS, …).
• Installs as TinyTitan.app; the archive is tinytitan-5.5-macos-arm64.tar.gz. Earlier versions keep the name they shipped under.
• Fixed by hand after the pass: the wordmarks now read Tiny + Titan with a widened canvas, and the app icon is the brand image clipped to the macOS rounded square (tools/make_app_icon.py). One defect the pass introduced — a resource-bundle glob that matched nothing, because bundles are named after the package plus target — was found and fixed before release.

Structured output is enforced by a grammar

• response_format (Chat Completions), text.format (Responses) and output_config.format (Messages) are no longer refused: a named JSON format compiles into a byte-level grammar that masks the sampler on both engines, so every token comes from the set that keeps the document inside the schema. The schema removes what is forbidden; among what remains the model's distribution still decides.
• Supported subset is explicit: type, properties, required, additionalProperties, items, enum, const. Everything outside it ($ref, anyOf, pattern, numeric bounds, tuple items, …) is refused by name at request time, and unsatisfiable shapes are refused with the reason. Special tokens carry no bytes and are never allowed.
• Thinking is off for a constrained request; MTP is skipped, because a draft ahead of the sampler never writes the logits a mask would edit.
• One whitespace-only token is allowed between structural tokens and a second in a row is not. A response truncated by max_tokens is a truncated document — content correctness is still the model's.
• Check: verified on a real install on both engines and through all three surfaces — docs/structured-output.md.

Thinking belongs to the request, on all three surfaces

• /v1/messages: thinking.disabled is a real off, adaptive still means "you decide", and thinking.enabled maps Anthropic's budget_tokens onto the ladder the OpenAI path already uses (under 4k low, under 16k medium, else xhigh). Anthropic's own budget rules stay refusals.
• Chat Completions gained the same per-request control through chat_template_kwargs.enable_thinking and reasoning_effort, and reports reasoning tokens in usage.
• A server loaded with thinking on can be told to think less — or not at all — for one turn, without a restart.

developer is the system turn, not an HTTP 500

• A leading developer message — the role that replaced system — was handed to templates that define only system/user/assistant/tool, so raise_exception('Unexpected message role.') surfaced as HTTP 500. Harnesses that switch to developer once a model reasons hit it.
• The role now renders as system in both prompt paths, and a leading developer message takes the effort instruction into itself.

The launcher offers only what is installed, and warns above 40% of RAM

• The built-in fallback filters itself against the install directories (the same check applies to a stale TINYTITAN_CATALOG_JSON); families and widths left out are named in one line, an empty models/ is a hard error carrying the install runbook, and an uninstalled width answers with the widths that are.
• The RAM recommendation moved from half to 40% of physical memory (floored to whole GB), because the expert cache is wired and cannot be paged out.
• An explicit --ram above that line is warned about in bold red — swapping, instability, slower tokens — and used anyway, because it is the operator's call. The default path keeps the install's measured profile, still clamped to half of physical memory.

One client catalogue for the launcher and the coder harness

• TINYTITAN_CLIENTS in tools/tinytitan_models.sh is now the single list; the launcher menu and benchmark/coder_cli_benchmark.py both build from it, so they cannot disagree.
• Four coder clients — Codex, Claude Code, Qwen Code, OpenCode — and Zed is an editor: --clients zed is refused and points at the new --round clients, which checks every client's wiring without loading a model.

DeepSeek Harness: a generated route, and a thin bundle

• tools/dsh_route.sh generates the llm-pi-ai route block from the server's own catalog — served ids, each template's thinking ladder, and the three switches that are easy to get wrong by hand (thinkingFormat: chat-template, the keyless-route auth header, a stream idle timeout that outlives a cold prefill). --write replaces just that section of ~/.dsh/settings.yaml after a timestamped backup, line-based so comments survive.
• plugins/dsh-tinytitan/ refreshes the route at boot and generates a compaction preset whose backend forces thinking off for compaction and session titles only. It registers no adapter and copies no protocol implementation, so a harness upgrade cannot leave a stale copy behind. Twenty node --test tests.

Also in this release

• Six dense Qwen 3.5 installs have golden baselines of their own. 2B/4B/9B at either width had no target and were declared exceptions; they have targets now and the gate checks them.
• A route refresh no longer orphans its own header. The generated block's three comment lines sit above llm-pi-ai:, so a section replacement left the previous header behind and every boot added three more. The writer removes its own header, a rewrite is byte-identical, and a regression test pins it.
• CI scans the Swift runtime. CodeQL covered only actions, c-cpp and python, and open alerts were zero — which is exactly why the gap was invisible. Swift is scanned by an advanced-setup workflow, weekly and on demand, building outside the checkout on arm64 because these sources use Float16.
• The coder harness can finish a round against a local model. A cold prefill pays minutes before the first token and Codex abandoned an idle stream after five, retrying into another cold prefill; the harness sets the stream idle timeout and disables retries for Codex.
• Every benchmark starts its server through tools/server_launcher.sh, so a stored baseline and a live measurement cannot diverge through a different launch. --round features refuses the dense installs by name, because they have no routed experts.
• The README is one benchmark table with a reproducible GPU-versus-CPU column for dense Qwen 3.5, a names-only supported-model list, and no per-release callout.
• The plugin package is publishable metadata-wise: a repository field pointing at plugins/dsh-tinytitan, and peer ranges widened to ^0.1.5-rc.2 || ^0.1.6-rc.1.

Performance

No performance number was re-measured for this release, and the README table is
unchanged from 5.4's — this release renames, constrains the sampler, fixes
request handling and adds configuration. The grammar masks the logits buffer the
repetition penalty already made, and a request naming no format generates
byte-identically, which the golden baselines re-check rather than a benchmark.

Verification

Cut from tag v5.5 on the base M3 with 24 GB: macOS 26.6.2, Swift 6.3.3, Apple M3, 24 GB.

• tools/lint.sh — all four gates clean: force-cast, func-length (0 baselined, 0 new, 2059 functions scanned), unchecked-Sendable, and the converter expert-order probe.
• swift test --no-parallel — 1523 tests in 234 suites passed (119.6 s).
• Clean scratch release build — warning-free, 107.9 s, staging the six executables and the .bundle resources the runtime loads its kernels from.
• Golden baselines, byte-identical — all ten targets installed here: katcoder-4, katcoder-8, qwen38-4, qwen38-8, qwen35-2b-4, qwen35-2b-8, qwen35-4b-4, qwen35-4b-8, qwen35-9b-4 and qwen35-9b-8.
• Not checked — no install under models/, and none may be fetched to fix that: ornith-4, ornith-8, qwen36-4, qwen36-8, agentworld-4 and agentworld-8. They are absent because the operator deleted those installs to save disk; nothing was downloaded, converted, repacked or re-installed to satisfy this gate.
• models/ holds eleven installs. The eleventh, qwen3.8-flash-next_125B_A6B_MTP_4Bit, is the MTP draft head — a sidecar the covered qwen38 targets already exercise — declared in NON_GOLDEN_INSTALLS with that reason rather than silently unchecked.

These results are from the dry run of this commit; --publish repeats every gate
from scratch and rebuilds the archive, which is why the digest and size below are
filled in only at publish time.

Not re-measured for this release: every performance number in the README,
including the dense GPU-versus-CPU rows, quoted from the wiki's
One Prompt, Every Model
page.

Checksum

tinytitan-5.5-macos-arm64.tar.gz sha256: 1e6f10bbdcad18daed92565fe8ccaa03935227402b1a8b1e63c1139013946775
tinytitan-5.5-macos-arm64.tar.gz size: 26094346 bytes

NVMAI 5.4

NVMAI 5.4 — KAT-Coder-V2.5-Dev, and verification that never fetches a model

This release first ships the 5.3 work to users — 5.3 was tagged but never
published — together with a hardened release gate and a native app icon.

KAT-Coder-V2.5-Dev 35B-A3B

Kwaipilot's agentic-coding fine-tune of Qwen 3.6 35B-A3B joins the family at
4-bit and 8-bit (tools/install_models.sh katcoder, katcoder-8bit, or
katcoder both). Same geometry as Qwen 3.6 and its own sampling — temperature
1.0, from the checkpoint's generation_config.json, not the series' 0.6.

Verified on the real install at both widths: the three continuations this
project uses behave (The capital of France is → Paris, Once upon a →
time, The quick brown fox jumps over the lazy → dog), each receipt
verifies, and each width has a golden baseline that re-checks byte-identical.
The baselines' 96-token answer is coherent technical prose about mutexes.

Measured on this base M3 with 24 GB through benchmark/nvmai_maxthroughput.py,
512-token greedy generations: 17.86 tok/s at 4-bit and 6.91 at 8-bit. The
8-bit build streams 36.9 GB of routed experts from SSD, which makes it the most
expert-locality-sensitive install of the 35B family; its count prompt is the
worst case at both widths (11.34–17.86 tok/s at 4-bit, 1.00–6.91 at 8-bit).

The expert axis was ordered by arrival, not by index

KAT's checkpoint is the first this project has converted whose routed experts
ship one tensor per expert rather than fused. The converter stacked them by
appending, which is only correct when they arrive in ascending index order. They
do not: the checkpoint index is lexicographically sorted, so layer 0's experts
arrive 0, 1, 10, 100, … 109, 11, 110, … — 48 non-consecutive steps — and 12 of
the 40 layers also split their experts across two shards. The fused axis was
ordered by arrival, so the runtime routed to expert k and read expert j's
weights.

Nothing in the pipeline could see it. Every expert's bytes matched the
checkpoint exactly; the shapes, the manifest and the packed_experts layout
were right; validateRoleUniformity passed; the receipt verified; and
gturbo_diff_snapshot reported all 613 resident tensors byte-identical. The
model answered fluently and partly correctly — "the capital of France"
appeared, then collapsed into repetition — at both widths, because the logits
came from the wrong experts.

It was found by converting Qwen 3.6 through the same converter as a control:
same geometry, but its source ships experts already fused, and it answered
correctly. That isolated the fault to the per-expert path, and comparing expert
255 rather than 0 or 1 exposed the ordering — two earlier readings had called
the fusion correct because they spot-checked the first two experts, which happen
to arrive first.

The accumulator now preallocates the expert axis and files each expert at its own
index, rejecting a duplicate instead of overwriting. tools/lint.sh converter
feeds experts in shuffled order and asserts each lands at its index; reverting to
append reproduces [3,0,7,1,5,2,6,4] and fails it. It lives in the lint gate
rather than the Swift suite because no Swift test can observe a Python converter
bug.

Three size caps that refused legitimate files

Found in the same run, each a literal chosen when an unbounded read was made
bounded, and each below what a real checkpoint produces:

• The snapshot index was capped at 4 MiB and KAT's is 9.7 MB, because the
  bound scales with tensors × key length and the converter's renames roughly
  double key length. The same literal was copied into the remote loader, so
  neither install path could build the model.
• The resident index was capped by the per-worker staging budget (1 MB)
  rather than the format's own ceiling, refusing an index of about 28 MB.
• The runtime's manifest cap was 4 MiB while --verify-install accepted the
  same 6.25 MB file against its own 64 MiB cap — so the install verified and
  then refused to load. A cross-module test now asserts the two ceilings agree.

The runtime now streams KAT's experts from SSD

Before the fusion fix, KAT's routed experts were classified as resident
weights: the install declared expertsPerLayer: 0, carried no packed expert
files, and would have held all 256 experts per layer in RAM instead of
streaming them. The install built by this release declares 256 experts,
expertStride 1769472, 41 packed expert files, and 1.8 GB of resident
weights with 17 GB streamed.

Release verification uses only the models already installed

models/ is deliberately kept below the full supported set to save disk, and the
golden gate now says so out loud instead of skipping silently:

• a target with no install is printed as not checked and collected;
• --publish refuses unless the release notes name every target that was not
  checked, absent ones included;
• an installed model that no check_golden line covers is a hard error, so a
  model cannot join the fleet unchecked; an intentional exception is declared in
  NON_GOLDEN_INSTALLS with its reason — the MTP draft head, and the dense
  Qwen 3.5 2B/4B/9B, which have no stored baseline at all;
• the phase fingerprints every verified-install.json before and after and fails
  if models/ changed at all.

No release step downloads, converts, repacks or re-installs a model to make a
check pass. docs/release-process.md §5 states the policy and
tools/release.sh enforces it.

Also in this release

• tools/install_models.sh <model> both installs 4-bit and 8-bit from one
  download for every Qwen3.5-MoE checkpoint (Ornith 1.5, Qwen 3.6,
  Qwen-AgentWorld, KAT). One width alone already converted both and kept the
  other snapshot; the install path now reuses it in either spelling, so a
  second width never re-fetches the checkpoint.
• A downloader that survives a truncating link. Shards are fetched as
  length-checked 64 MiB ranges, three at a time, with --http1.1 (this host
  resets HTTP/2 streams continuously), a stall floor that aborts a dead
  connection, and no resume that could append to a truncated prefix. A
  truncation costs one chunk instead of 5 GB.
• The tool scripts resolve their own Python by capability — 3.10+ with
  numpy, ml_dtypes and safetensors — instead of a pinned python3.13, so
  they work wherever the analysis stack lives; NVMAI_PYTHON overrides.
• A native NVMAI app icon, replacing the upstream fork's bird. It uses the
  wordmark's own palette and is reproducible with tools/make_app_icon.py
  (issue #5).
• The archive carries NOTICE. The binary distribution now ships LICENSE,
  NOTICE and THIRD_PARTY_NOTICES.md: Apache-2.0 requires the first two to
  travel with the binaries, and the third carries the upstream attributions.
• The app recognizes both KAT widths, with descriptors carrying each
  snapshot's own fingerprint, and the install table in
  AppModelInstallTests covers ten builds.
• benchmark/nvmai_maxthroughput.py --engine cpu|gpu selects the engine and
  folds it into the result label, so a CPU row cannot be read as a GPU one.
• The README benchmark table carries KAT's measured rows and a
  GPU-versus-CPU table for the dense Qwen 3.5 models.

Performance

KAT's rows are the only numbers this project measured itself, on this base M3
with 24 GB, through benchmark/nvmai_maxthroughput.py; they are in the README
table above and were not re-measured for 5.4. The dense Qwen 3.5
GPU-versus-CPU table added to the README is not a fresh measurement either: it
quotes the decode rates already recorded on the wiki's
One Prompt, Every Model
page, which was measured on this machine. Attempts to re-run the dense CPU
numbers were abandoned as unreliable — the 9B 8-bit thrashes on the CPU engine
(0.4–1.2 tok/s, one 512-token generation taking 1225 s) because that engine holds
the model resident instead of streaming experts — and the recorded numbers are
short-generation rates, not 512-token peaks like the rows above them.

Verification

Cut on 2026-09-14 from tag v5.4 on the base M3 with 24 GB this project measures
on — macOS 26.6.2, Swift 6.3.3, Apple M3, 24 GB.

• tools/lint.sh — all four gates clean: force-cast, func-length (0
  baselined, 0 new, 2012 functions scanned), unchecked-Sendable, and the
  converter expert-order probe.
• swift test --no-parallel — 1470 tests in 228 suites passed (118.3 s).
• Clean scratch release build — warning-free, all six executables and the
  .bundle resources staged (the Metal shader library among them).
• Golden baselines, byte-identical: qwen38-4, qwen38-8,
  katcoder-4 and katcoder-8. These are the four targets installed on the
  machine; the archive is 24770200 bytes.

The gate verifies every golden target that has an install under models/ and
reports the rest. Six targets have no install here and are not checked:
ornith-8, ornith-4, qwen36-4, qwen36-8, agentworld-4 and
agentworld-8. They are absent because the operator deleted those installs to
save disk. They were not downloaded to satisfy this gate, and they must not
be: release.sh names every unchecked target, refuses to publish unless these
notes repeat the list, and fails if the golden phase changed the install set
under models/ — a different install, a removed one, or a stray file left
behind. (It is not a payload hash: the receipt the runtime verifies attests the
payload.)

The six dense Qwen 3.5 installs are not golden-checked at all. There has
never been a stored baseline for 2B/4B/9B at either width — benchmark/golden/
holds ten files, all for the MoE families. That is a pre-existing gap, not one
this release introduced; it was surfaced by this release's new coverage guard,
which had...

NVMAI 5.2

NVMAI 5.2 — one server for every model, the dense family on both engines, and memory that outlives a conversation

This is the largest NVMAI release so far. Three changes alter what you can do
with it, and a fourth alters what it will do quietly:

• One server serves every installed model. The launcher starts it with the
  client of your choice, and any other installed model stays reachable by name
  on the same port — one resident at a time, switched on demand.
• The dense Qwen 3.5 models run on the GPU as well as the CPU, and the
  engine is a per-request choice for them.
• Optional agent memory, inside the server process: durable facts scoped to
  the repository you are working in, with no database to install.
• Thinking is not the answer. A model's reasoning reaches clients as
  reasoning_content, apart from content, including the case where a model
  thinks although the request turned thinking off.

Everything below is in the tagged commit; the checks that back it are named as
they come up.

One server, every model, one launcher

NVMAIServer --models-dir models serves the whole catalogue: /v1/models
lists every installed model and quantization, and a request naming another one
unloads the resident model and loads that one. The launcher
(tools/server_launcher.sh) asks what to launch (the API alone, or the API
plus Codex, Claude Code, Qwen Code, OpenCode or Zed), whether to keep the
coding-CLI boilerplate (full or fast), which model and quantization, the
answer style, the thinking level, and — only for a model that has one — the
RAM limit for the expert cache. Its model list comes from the server's own
catalogue, so it cannot disagree with what the server serves.

Retired with it: the eight per-model start scripts and the separate
client-wiring launcher. Both are replaced by this one entry point, and the
launcher reads the served model id from the running server rather than
assuming it.

The dense Qwen 3.5 family runs on both engines

The 2B, 4B and 9B dense models (4- and 8-bit) were CPU-only in 5.1 because the
GPU runtime refused their family (qwen3_5_dense) by name: the decode and
prefill pipelines encoded the router, the prefetch probes, the residency
classification and the streamed routed FFN unconditionally, which is the
silent-fluent-nonsense class of failure this project has shipped once before.
That stage is now conditional, and a dense model skips it entirely: its
gate/up/down FFN runs through the block that already implements a shared
expert, the tensor schema is the family's own, and the layer conventions come
from the manifest's arch block rather than from a guess.

The engine is now selectable per request for these installs: the bare
catalogue id is the GPU spelling, and <id>@cpu / <id>@gpu name an engine
explicitly, so a client can choose without restarting anything. Residency is
still one model at a time, so switching engines reloads the install.

The port's acceptance bar was numerical, not "it loads": logits equivalent to
the CPU engine's on the real install, layer by layer, before the refusal was
lifted. That gate paid for itself — it found that the GPU path read
k_proj/v_proj through the attention slot's 4-bit kernel while the install
stores them at 8 bits, which is a plausible-looking wrong attention output
rather than an error. Widths now resolve per tensor stem, and the runtime
builds one affine dispatcher per width a role needs. Three further
assumptions that had never run without a routed mixture (a manifest that
demanded packed expert files for every layer, a layout validator that required
at least one expert, and range guards that trapped on top-0) were fixed with
it.

Memory that outlives a conversation (optional)

NVMAI_MEMORY=1 gives a model durable facts it writes in one session and reads
in another, scoped to the repository the client is standing in. It runs inside
the server process — no database, no port, no connection to lose — on a
continuity engine that journals one file per workspace. Six memory tools are
answered by the engine itself, and the extraction that writes facts marks
whether the person asserted something or the model inferred it.

That distinction is enforced: the guard (NVMAI_MEMORY_GUARD=1, on by default
wherever memory is on) stops a model-derived fact from silently superseding one
the person asserted. Measured by replaying a recorded long session, an
unguarded store answered 61% of the questions the recording supports and a
guarded one 98%; on a control session the guard scored 97%, one point off the
unguarded control, so it is not buying safety with silence. It is off by
default overall and the serving path does not depend on it; see
docs/agent-memory.md.

The Qwen 3.5 2B was measured as a resident helper that would propose and check
facts. It is not wired in: one decision per call with a closed answer set
scored 92%, but asked to decompose a long fact in one shot it echoed its input
in 38 of 47 answers, and one CPU thread while a 35B generates costs that 35B 3%.
The measurement is recorded so the design stays honest about what a 2B can be
trusted with.

Thinking is not the answer

A model's reasoning now reaches clients as reasoning_content, apart from
content, on every surface: a client that knows the field shows the thought
apart from the answer, and one that does not sees the answer alone. The
reasoning levels a model offers are the levels its template actually renders —
the binary switch for Ornith, Qwen 3.6, Qwen-AgentWorld and the dense Qwen 3.5
models; off, low, medium, xhigh for Qwen3.8-Flash-Next — and a level a
model does not define is refused rather than mapped to a neighbour.

Two defects were found here and fixed. A model's reasoning_effort change was
an HTTP 500 on every installed CPU model, because both re-render paths handed
the tokenizer loader a model directory where a .gturbo install keeps a
tokenizer/ sidecar. And a thought the model opens while thinking is off was
streamed as the answer: Qwen AgentWorld 35B-A3B 8-bit, asked a false-premise
question with the switch off, reopens a <think> block and spends the whole
token budget inside it, so a client capping max_tokens received a thinking
transcript where it expected an answer. The thought is now split into
reasoning_content on every engine, and the server logs
thinking off, but the model wrote N characters of reasoning on the request's
line. The measurements, with every reply verbatim: the wiki's
One Prompt, Every Model
page.

The protocols coding agents actually speak

One server speaks OpenAI Chat Completions, the OpenAI Responses API in full
(stored responses, previous_response_id, the complete event grammar) and the
Anthropic Messages API (/v1/messages, count_tokens, streaming). All
three were exercised against the real Codex and Claude Code CLIs, not only
against test doubles, and the launcher wires a chosen client to the model the
server advertises. Reasoning levels a client requests are mapped to what the
served model renders, and the mapping is logged rather than left to be
inferred.

The deep audit: 89 code findings, 8 documentation defects, none open

Seven read-only passes over every module, then verification of each finding
against the source before any fix — a subagent's word is not evidence. The
register is docs/audit-2026-09-11-findings.md: 89 code findings and 8
documentation defects resolved, 0 open, 3 disproved (and recorded as
disproved rather than deleted).

The worst were the ones that were silent rather than loud: an out-of-bounds
router write on Qwen3.8-Flash-Next that did not fault only because driver
allocations are page-granular; a prompt-cache restore that left the sparse
indexer holding the previous conversation's pooled keys, which is silently
wrong output on a pinned model; a fused hyper-connection read kernel that could
never be built, so every measurement of that path had measured the unfused one;
and unbounded request headers on a loopback server, with an oversized-body 413
that arrived only after the whole body had been read.

Structure, naming, and the Swift 6.3 baseline

HTTPServer.swift (2,599 lines: an actor, a 2,111-line handler and five
support types) is eight files now, the largest 604 lines; Model.swift split
into the model and its loading path; three benchmark families that shared one
1,153-line main.swift are separate files; and two files named main.swift
while holding @main are named after their type. The test tree mirrors the
source tree. The compiler warning count is zero, and a defer that deleted a
demo's scratch directory before the demo wrote to it is one of the warnings
that had been learned past. The conventions are written down in
docs/repository-layout.md.

The tree is Swift 6.3.3, tools format 6.3, Swift 6 language mode, and now also
enforces the three upcoming features it was already clean under:
InferIsolatedConformances, ImmutableWeakCaptures and
MemberImportVisibility (which needed direct imports in twelve files). The
three features with a real migration cost — ExistentialAny,
InternalImportsByDefault, NonisolatedNonsendingByDefault — are recorded as
deliberate, with their measured cost, in docs/swift-language-standard.md.

Also in this release

• Watchdogs, opt-in: NVMAI_WATCHDOGS=1 watches four failure modes — a
  repetition loop, a stall, a stub answer and a ping-pong turn — and
  NVMAI_WATCHDOG_ACT=<kinds> names the ones that may stop a generation
  instead of only reporting it. Off by default.
• A whole-model CPU verifier: tools/verify_cpu_models.sh runs the dense
  installs end to end through the CPU engine (all continuations correct), and
  the bench commands that did this had been pinned to a conversion
  intermediate that no longer exists.
• **The dense inst...

NVMAI 5.1

NVMAI 5.1 — the 35B models decode 10–13% faster at 8-bit

NVMAI 5.1 repairs the expert prefetch ring, which had been effectively off
since 5.0.2, and turns it back on for the 35B models. Qwen 3.6, Ornith 1.5
and Qwen-AgentWorld gain 10–13% at 8-bit and 2–4% at 4-bit. Output is
byte-identical to 5.0.2 on all eight goldens.

The prefetch ring was clogged

Per-token counters added for this release showed the ring issuing about
five speculative reads per token where it should issue tens, with 1.78 of
its 2 slots permanently held by completed reads whose layer had already
passed. The reclaim rule frees a slot only when its layer index is at or
below the current one, so a prediction made for layer 46 or 47 at the end
of one token carried that index into the next, where the layer counter
restarts at zero. A prediction for the last layer was never reclaimed at
all. The rule arrived in 5.0.2 with the two-layer-ahead experiment.

Every prefetch measurement taken since then compared variants of a
mechanism that was not running, which is why the 35B rows had asked for a
speculative read and received nothing.

The ring now reclaims every finished read at the token boundary.

What that is worth, per install

Measured on the repaired ring, arms interleaved within each round so the
machine's own drift cancels, 512-token generations, five rounds on
Qwen 3.6 and three on the others. Every 95% interval excludes zero:

Model	4-bit	8-bit
Qwen 3.6 35B-A3B	+1.8%	+11.3%
Ornith 1.5 35B-A3B	+1.8%	+12.6%
Qwen-AgentWorld 35B-A3B	+1.4%	+11.4%

One read in flight remains the right depth: two returns +2.7% where one
returns +9.8%. On Qwen3.8-Flash-Next the repaired ring loses at every
setting, because its experts are larger and land after the layer that
would have used them, so that model ships with prefetch off. The two
families now differ on purpose.

The expert cache stays wired through prefill

Prefill released the 10–12 GiB slot cache, memory pressure swapped it out,
and the first decode token faulted all of it back in: 1.6–4.7 seconds per
request, which a short answer pays in full. The cache now stays wired for
every model. Worth +7.1% and +5.1% on 48-token generations (Qwen 3.6
4-bit and 8-bit) and about +1% on long ones; prefill is not slower.

Two smaller fixes on the same path: the pin check no longer rebuilds a
dictionary from the process environment once per layer per token, and it
returns early once every layer is wired instead of walking all of them.

A start script per model and quantization

Eight scripts that start the server for one install with no questions and
each on its own port, so two configurations can run at once:

tools/start-ornith-4bit.sh        tools/start-ornith-8bit.sh
tools/start-qwen3.6-4bit.sh       tools/start-qwen3.6-8bit.sh
tools/start-agentworld-4bit.sh    tools/start-agentworld-8bit.sh
tools/start-qwen3.8-4bit.sh       tools/start-qwen3.8-8bit.sh

Ornith keeps ports 8081 and 8083. The model list, install paths and ports
live in one catalogue that both the start scripts and the two interactive
launchers read, so they cannot disagree about where a model is or which
port it serves. tools/server_launcher.sh still asks, and
tools/cli_launcher.sh also wires up Codex, Qwen Code or OpenCode.

The Mac app recognizes the models you have installed

The app's checkpoint catalogue listed only MLX repacks. After 5.0.1 moved
every install to a build quantized from the model's own bf16 release, the
app matched none of them: it reported each as a foreign checkpoint and
offered to download an MLX build over it. It now carries a fingerprint for
each of the eight installs, and separates what it can download itself from
what needs the command-line converter — for those it prints the
tools/install_models.sh target instead of a Download button. Older MLX
installs stay recognized and downloadable under -mlx selectors.

Also in this release

• Every launch path takes its tuning from the model's profile row: the Mac
  app sizes the expert cache from it instead of a flat 64 slots, and its
  sampling follows the model until you change a value.
• Both launchers read the served model id from the running server. Since
  5.0 those ids carry the quantization suffix, and the bare name the
  launchers wrote into Codex and Qwen Code configs was rejected.
• New decode diagnostics under NVMAI_RUNNER_STATS: per-segment token
  timings, prefetch reads issued and adopted per token, and a prefetch-ring
  slot-state line.
• Opt-in and measured, off by default: early expert hits
  (NVMAI_EARLY_HITS=1, a wash within ±0.16 tok/s over five paired runs),
  a probe-weight prefetch gate, and a decayed-frequency cache policy (+4%
  on a 3000-word prompt, −4% on a short one).
• Three test races fixed that only a parallel swift test exposed.

Performance

512-token story-generation benchmark at the shipped defaults, base M3,
24 GB, measured on this release:

Model	Quantization	5.0.2	5.1
Qwen-AgentWorld 35B-A3B	4-bit	21.28 tok/s	21.74 tok/s
Ornith 1.5 35B-A3B	4-bit	20.99 tok/s	21.65 tok/s
Qwen 3.6 35B-A3B	4-bit	20.95 tok/s	21.41 tok/s
Qwen 3.6 35B-A3B	8-bit	11.23 tok/s	12.37 tok/s
Qwen-AgentWorld 35B-A3B	8-bit	11.16 tok/s	12.28 tok/s
Ornith 1.5 35B-A3B	8-bit	10.89 tok/s	11.93 tok/s
Qwen3.8-Flash-Next 125B-A6B	4-bit	5.40 tok/s	5.46 tok/s
Qwen3.8-Flash-Next 125B-A6B	8-bit	2.06 tok/s	2.10 tok/s

A decode profile of Qwen3.8-Flash-Next, including every lever measured and
closed, is in docs/qwen38-decode-profile-2026-09-05.md.

Checksum

nvmai-5.1-macos-arm64.tar.gz sha256: 38b9124bb870f88c2fb37e7448ce68aa08f70fb824fa3c8c26f4192b54bad647

NVMAI 5.0.2

NVMAI 5.0.2 — per-model tuning profiles, bigger expert caches for the 35B family

NVMAI 5.0.2 gives every installed model and width its own tuning row and
uses it to lift the three 35B-A3B models by 9% at 4-bit and 15% at 8-bit,
with no change to any weight, head or KV width. Output is byte-identical
to 5.0.1 on all eight goldens.

Per-model profiles

ModelProfile is a table with one row per (model, routed-expert width):
expert-cache budget, prefetch depth and disk I/O tier, prefill chunk,
sampling defaults and the kernel switches. Resolution is family default,
then the row, then the environment, so a row can be edited without moving
any other install and an experiment can still override a shipped value
with NVMAI_* variables. The resolved profile is logged once at load under
NVMAI_RUNNER_STATS.

Where the 35B token went

A per-role split of a Qwen 3.6 token at the 5.0.1 defaults: 49.8 ms at
4-bit, of which 13.2 ms was expert I/O with only a quarter hidden behind
the GPU, at an 87.7% cache hit rate with 128 slots per layer; 121.9 ms at
8-bit, of which 51.7 ms was expert I/O, 12% hidden, at 79.3% with 64
slots. The 8-bit vocabulary head, which a head-width test had pointed at,
runs at 89.9 GB/s, the practical ceiling on this GPU.

The 128-slot cap was only a list entry. The allowed slot counts gain 160,
192 and 256, and each install was measured on its own with the story
prompt, arms interleaved, swap sampled around every run:

Install	4-bit, 128 → 160 slots	8-bit, 64 → 96 slots
Qwen 3.6 35B-A3B	19.50 / 20.45 → 20.55 / 21.04	9.85 / 9.72 → 11.13 / 11.19
Ornith 1.5 35B-A3B	19.91 / 20.41 → 20.84 / 21.02	8.69 / 9.12 → 10.83 / 10.86
Qwen-AgentWorld 35B-A3B	20.52 / 20.50 → 21.11 / 20.92	9.31 / 9.25 → 11.15 / 11.21

All six rows now ship 10 GiB at 4-bit (160 slots) and 12 GiB at 8-bit (96
slots); swap stayed flat on every arm. 192 slots at 4-bit measured another
4% on Qwen 3.6 (21.61 / 21.60) but pushed 1.5 GB to swap on first contact
on a 24 GB machine, so it stays one --ram-budget 12G away for machines
with more memory.

Qwen3.8-Flash-Next

• 4-bit: predictive prefetch runs two deep on the utility disk I/O tier,
  +3% over one deep on the default tier (5.46 / 5.40 vs 5.21 / 5.33). The
  throttle tier costs 22%; two-layer-ahead prefetch (NVMAI_PREFETCH_AHEAD=2,
  opt-in) measured a wash.
• 8-bit: 40 slots (9.5 GiB) instead of 32, +8% (2.18 / 2.27 vs 2.05 / 2.06)
  with no paging; 48 slots grew swap by about 1 GB per run.
• The prefetch levers that helped Qwen 3.8 measured a wash on every 35B
  install at both widths, which is why the rows differ.

Also in this release

• Ornith 1.5 rebuilt from ornith-ai's bf16 release through the shared
  converter, like Qwen 3.6 and AgentWorld; no third-party quantization is
  left in install_models.sh. Goldens ornith-4 / ornith-8 pin it.
• The prefetch disk I/O tier is a profile field (NVMAI_PREFETCH_IO_TIER
  still overrides it).
• NVMAIBench head_affine8 | head_affine4 | head_int4 benchmarks the
  vocabulary head GEMV at the 35B shape.

Performance

512-token story-generation benchmark at the shipped defaults, base M3,
24 GB, one run each on the release binary:

Model	Quantization	5.0.1	5.0.2
Qwen-AgentWorld 35B-A3B	4-bit	18.57 tok/s	21.28 tok/s
Ornith 1.5 35B-A3B	4-bit	19.24 tok/s	20.99 tok/s
Qwen 3.6 35B-A3B	4-bit	19.21 tok/s	20.95 tok/s
Qwen3.8-Flash-Next 125B-A6B	4-bit	5.25 tok/s	5.40 tok/s
Qwen 3.6 35B-A3B	8-bit	9.73 tok/s	11.23 tok/s
Qwen-AgentWorld 35B-A3B	8-bit	9.46 tok/s	11.16 tok/s
Ornith 1.5 35B-A3B	8-bit	9.72 tok/s	10.89 tok/s
Qwen3.8-Flash-Next 125B-A6B	8-bit	2.03 tok/s	2.06 tok/s

Checksum

nvmai-5.0.2-macos-arm64.tar.gz sha256: 618152f197a880b57ec4053b9522f4cd80308b7637a1209ae7715d52541b786a