Engineering Notes
The reference half of the engineering record: traps that have already cost time, measurements that closed a question, observations nobody has explained, and decisions that should not be re-litigated. No tasks live here — open work, one ID per item, is in the Project Tracker.
This page is engineering-facing. The rest of the wiki describes TinyTitan as it ships today; nothing here is a promise or a supported feature. Every claim carries the measurement it came from, and anything unmeasured says so.
Current handover: docs/handover-tinytitan.md in the checkout — the only one,
after the three earlier handovers were deleted on 2026-09-14 so nothing competes
with it. It starts with the prompt for the next session, and carries the rules a
gate or a harness has to follow — installed-models-only verification, when a
baseline can be captured, and the launcher mandate for the benchmark scripts.
The DeepSeek Harness bundles in plugins/ are clients and sidecars around the
loopback server, not the engine: they follow the harness's pinned version instead
of the engine's release, and they are not part of what a TinyTitan install ships.
What each one does is on its own page — TinyTitan Plugin and
LAN Manager — and their open work is in the
Project Tracker. What follows is what was measured and decided.
2026-09-18 — the plugin's work is committed on main (from its first commit
0ddce77); the notes below record what was measured as it landed. The plugin mounts
a LAN-scoped management API
on a running harness's web server; ttlanmanager (sources/TinyTitanFleet/) is
the CLI that drives a group of them. The plugins find each other; the CLI is
the only thing that prompts or mutates, and it talks to the member that owns the
thing it is acting on — a plugin never relays a prompt. User-facing page:
LAN Manager.
Working, and measured:
- Discovery finds tailnet peers on any OS and any continent (the first cut
filtered on
OS === "macOS", which hid every Linux box in every datacenter), plus Bonjour browse, configured seeds and an opt-in/24sweep, on a jittered 60 s timer. One cycle over 513 candidates: 2.5 s to find, 8.1 s to probe, and a worst event-loop delay of 26 ms — discovery does not block the harness it runs inside. - A member's answer is cached including a failure. Twelve stale
.localnames cost 30 s per cycle uncached and 0 s on the second cycle cached, which is the difference between a Bonjour-heavy LAN working and not. - Endpoints:
/peers,/peers/:id,/inventory,/prompt,/prompt-all,/sessions/:id/messages,POST /workspaces,/sessions/:id/archive,/workspaces/:id/delete,/health. - 107 plugin tests and 44 manager tests. The source-address fence is the best-covered part: every allowed range, the address just outside each one, the malformed literals that used to be repaired into an allowed range, and the spoofed forwarded header.
Starting a session is wired (TT-028) — by delegating to the harness's own
session controller (sessionController.create({ workspaceId | cwd, agentPreset? })),
which composes the agent's world from the preset, resolves the default model,
creates the working directory, mints the id and attaches the session to the
workspace. Reimplementing that here would be a second copy of harness logic,
drifting at every release. A profile composing no controller still answers 501
rather than claiming a session it did not create.
Bonjour is browse-only by decision (TT-029), not by omission: registering
_dsh-lan._tcp would advertise a loopback-only API, so a peer would be discovered
and then fail to probe. The plugin says so at every mount.
The route was driven end to end against a real model, and both write paths were broken (TT-014, 2026-09-18)
What ran. A private harness (tools/dsh_local.sh, pinned 0.1.6-alpha.2, port
7788) with the plugin installed, and TinyTitanServer serving
qwen3.5-4b_4-Bit on 8080 — an install already under models/, nothing fetched.
Driven with curl against the running harness:
-
POST /dsh-lan/sessionson the seeded workspace →session-9bb889e8…, presettinytitan. -
POST /dsh-lan/prompt→delivered: true,wakeup: true, message87e73078…. -
GET /dsh-lan/sessions/:id/messages→ the assistant's "42".
The model call is in the server's own log, so the answer was not a harness cache:
chatcmpl-5d9c051f… completed in 297.432 s, prompt=7439 cached=0 completion=3 finish=stop. The 7439 prompt tokens are the harness's agentic system
prompt, not the question; the harness also issued its session-title call in the same
second, and the server's one-generation-at-a-time queue serialized them. The 297 s
is a first-call figure, left standing as an observation rather than explained here.
Two defects, both invisible to the unit tests, both fixed (TT-032).
-
POST /sessionsasked the harness for a service namedsessions. That name is@deepseek-ai/dsh-session's raw store, whosecreate(id, options)mints a bare session; the controller is registered assessionController(@deepseek-ai/dsh-api-session-controller). The call died withsession header id "[object Object]" does not match session id "[object Object]"and created nothing. The plugin's fake had encodedsessionsas the controller, so 104 green tests pinned the wrong name. -
POST /promptbuilt a user message withcreateUserMessage({ content })and nosource. Upstream's factory mints an id and a role but deliberately does not invent a source, and the agent loop readsmessage.source.kind. Delivery reporteddelivered: true, wrote the message into the inbox, started a turn and killed it 5 ms later withCannot read properties of undefined (reading 'kind')— no model call, no error at the route. The session's ownturn/endevent is the only witness.
The fallback message shape already carried source: { kind: "user" }, so the plugin
worked only where the dynamic dsh-llm import failed — the opposite of the intent,
and the same "a claim broader than the code" shape as the two defects that reached a
release in this project. The receipt also reported wakeup: options.wakeup !== false
while Agent.followup(message) takes no such option and always wakes; it now says
true as a fact and the dead option is gone.
The regression tests pin the service name (the fake records every service read
and asserts sessions was never asked for) and the source on both factory
paths (a stub @deepseek-ai/dsh-llm module drives the primary path the suite had
never exercised). 105 plugin tests pass.
Also driven live, all sound: POST /sessions/:id/archive (archived: true, and
the session leaves GET /sessions), POST /workspaces for a throwaway folder,
POST /workspaces with "startSession": true (returns the session and preset), and
POST /workspaces/:id/delete (with archiveSessions, which reported the archived
session id).
GET /sessions/:id/messages used to answer 404 whenever the session had no live
agent, so an idle or archived session — the exact thing a fleet audit asks about —
could not be read at all. It reads storage now.
The cold path is deliberately the harness's own: sessionQuery.readSession()
loads and replay-validates the stored log, and the plugin hands that
{ header, events, inheritedEventCount } back to the sessions service's
prepare() with eventState: "detached" — the same call sessionQuery makes
internally to build an observation — then calls the same deriveMessages() the live
path uses. Surface markers and compaction rules therefore stay the harness's, no
frame parser is written here, and the plugin still imports no harness internals.
Verified live against the private harness with no TinyTitan server running: after
a restart, so no agent was live, the TT-014 session read back its four messages,
including the assistant's 42; an unknown id is a clean 404 no session …; and a
freshly created session still takes the live path. A profile that composes no cold
reader answers 503 and says so, rather than reporting an empty conversation.
107 plugin tests pass.
2026-09-18. Two throwaway harnesses on this machine — separate DSH_HOMEs, ports
3199 and 3198, each seeded with the other — found each other in both directions
(source=seed, 9–11 ms). The first was booted before the second, so its boot cycle
could see nothing; the fact that it found the peer anyway proves the periodic
cycle, not just the boot-time one.
Two things the run taught. The interval has to be changed in the row config rather
than the environment, because the shipped patch writes
discoveryIntervalSeconds: 60 and a row value beats the documented
DSH_LAN_DISCOVERY_SECONDS fallback — that was TT-031, fixed the same day: the
two pinned fields are commented in the patch now, both fallbacks apply, and
test/patch.test.js fails if a field with an environment fallback is ever written
back in. And a peer is a member only after a GET /inventory that returns 200 with
the shared key: a machine in the group with the wrong key is reachable, logs as
401, and is deliberately not a peer.
Bonjour stays browse-only (TT-029, 2026-09-18). The plugin browses
_dsh-lan._tcp and nothing registers it, so the LAN source finds only a
third-party advertiser. Registering the service was rejected rather than
deferred: Bonjour advertises this host's LAN address, and the API answers on
loopback only (TT-020), so a registered service would be a false beacon — a peer
would appear in the group and then fail to probe, which costs more than an empty
browse. The plugin says so in the log at every mount, and registration belongs with
a reachable bind.
Verified against the harness we now pin. tools/dsh_local.sh moved from
0.1.5-rc.2 to 0.1.6-alpha.2 on 2026-09-18 — an alpha, and npm agrees:
latest and next are still 0.1.5-rc.2. Two consequences were checked rather
than assumed: the notice version the pin "moves with" is unchanged
(2026-08-13.1), and the alpha falls below ^0.1.6-rc.1 in semver, so
dsh-tinytitan's dsh-compaction-basic peer range was widened to
^0.1.5-rc.2 || ^0.1.6-alpha.1.
The supported range is now exactly the pin (2026-09-18). Those ranges were
widened only to include the release; they also admitted everything else, and
"supports this alpha" is not the same claim as "supports whatever npm resolves". So
dsh-compaction-basic and dsh-agent-presets are exact 0.1.6-alpha.2, and both
plugins refuse to run on any other harness — older, newer, a build from main, or
a version that cannot be read at all. dsh-lan-manager carries the same gate, and
its constant is asserted against the launcher's pin so the two cannot drift apart.
plugins/*/test/support.test.js fails if either peer range, either constant, or the
pin disagrees — and CI now runs those suites, which nothing ran before.
Neither gate throws. A refusal is one log line naming both versions followed by a
return: no route, no preset, no watcher, and no writes into the harness home. DSH
still boots, every other plugin still loads, and removing ours leaves nothing to
undo. An unreadable version is refused rather than assumed, because a plugin that
writes into ~/.dsh has no business proceeding on a harness it cannot identify.
And the refusal had to move to stderr (2026-09-18). Booting a throwaway harness
in a temporary DSH_HOME — with both plugins installed, on an unsupported version —
found what reading had not: the harness collects a plugin's log records into its
startup diagnostic and prints them only when the boot itself fails, and its
startup exporter is registered at levels: { default: 2 }, so ctx.logger.info
reaches nothing at all. The first gate therefore refused silently: DSH was fine
and nothing was written, but the operator had no way to learn why the plugin had
stopped. Both refusals now go to stderr as well as the host logger. The throwaway
run also showed the LAN manager never activates in a headless profile at all — it
injects webServer, so it sits pending and its gate is only reachable behind a web
profile, which is where it is tested.
Where the gate stands (2026-09-18). Both installed copies are refreshed to it.
The global harness was reloaded and took 0.1.6-alpha.2 without a refusal — its
settings.yaml was rewritten at boot by the route step while the agent preset was
left untouched, which is the gate passing and the preset generation staying
idempotent. The private harness's copy is refreshed but idle until its next start.
The four fleet nodes run plain DSH with no TinyTitan plugin, so there the gate has
nothing to refuse.
Deliberate, so it is not re-litigated:
- The group key ships with a public default, so by default it groups rather
than protects. That was an explicit operator decision; change
DSH_LAN_KEYon a network that is not solely yours. - A plugin never sends a prompt or mutates another instance. Only the CLI does.
- Gossiped addresses are candidates, validated like any other against the LAN/Tailscale fence before anything is dialled, and the list is capped.
-
discoverSubnetis off by default: it is the only source that touches hosts which never opted in. -
ttlanmanageris not a release product. It is an operator tool and stays out ofrelease.sh'sPRODUCTS. - The manager's scanner was deliberately not built on
worker_threads: it never blocks the loop, and libuv's threadpool is per process, so a worker would share the same four threads and buy nothing.
The preset and route bundle that the launcher's --web installs. Its open item,
publication, is TT-019 in the Project Tracker.
2026-09-18 — status corrected. The research is settled and the code half is
done; what changed is the submission itself. PR #5094 was closed on 2026-09-15
with no comment, and the catalogue carries no entry for this plugin
(data/plugins/Pummelchen__TinyTitan--plugins-dsh-tinytitan.yml → 404). It is not
on npm either (dsh-tinytitan → 404), which was always optional for a listing.
The submission itself is TT-019 in the Project Tracker: either
resubmit the entry — the file is ready in docs/dsh-plugin-publication.md — or take
the other route the research already found (npm, or a git host). What follows is the
research, so it is not repeated.
What is settled, so it is not re-researched:
-
Where a plugin is published. Upstream has no registry and no catalogue;
dsh plugin --profile <name> …forwards to pnpm in the profile directory. Distribution is therefore npm, apnpm packtarball, or a git host — the last needing apreparescript and the user'sallowBuildspermission. Discovery is the community listawesome-dsh-plugin/awesome-dsh-plugin, whose whole submission is one generated file,data/plugins/<owner>__<repo>.yml: the handover's "one-file catalogue PR" is that repository, not upstream. - The licence is MIT on purpose, recorded with its reason in the plugin README: the package is an independent work that talks to the server over its HTTP API and copies nothing from the fork's lineage, so it is deliberately not aligned with the repository's Apache-2.0.
-
The code half is closed (
2d9daa9): the route refresh no longer needs a checkout —generate.jsruns the discoveredTinyTitanServer --catalog --models-dir <dir>— andtest/generate.test.jspins the generated block byte-for-byte totools/dsh_route.sh --print. -
Readiness passed against the list's checklist before the first attempt:
dsh.bundle, theplugins/subpackage, therepositoryfield, 34 tests, repo age, thedsh-plugintopic, and the npm name unclaimed. One trap the dry run found, worth keeping:build-site.mjsparses the generated READMEs rather thandata/plugins/*.yml, so a yml-only submission builds a site missing its own entry until the regeneration step runs — a local row count one belowreadEntries()is that, not a dropped entry.
The wiki Roadmap page was retired on 2026-09-18. Planned work is recorded one
item at a time in the Project Tracker; there is deliberately no
separate roadmap to re-create.
Recorded so it is not re-litigated: the transport design (pinned host to VRAM
over PCIe, cudaMalloc pools, Unified Memory interplay) and the residency cost
model. Unified memory has no VRAM/host split to manage, our bottleneck is NVMe
to RAM under F_NOCACHE, and their planner assumes every KV page is needed
every token. Routed experts are sparsely selected — ten of 512 — and reused, so
our problem is prediction, not bounded full scans.
The icon request in issue #5 is done and shipped in 5.4. The same issue listed three Fieldfare features the requester wanted here, and all three were features of a GUI front end — which TinyTitan does not have: it is an engine and a loopback server, and the client owns everything with a window. They are recorded as closed rather than open, so nothing here sends a future session after a front end:
- Image upload — a prompt that carries an image. Every model here is text-only, the Qwen 3.5 9B among them (its vision tower is deliberately not repacked), so nothing in the current engine can consume an image with or without a UI.
- Conversation history — durable, resumable conversations. This was a renderer feature. A client gets the equivalent from the server's stored Responses and its prompt-state reuse; the engine persists no conversation of its own.
- LaTeX rendering — typeset math in responses. This was a renderer feature; typesetting is the client's job now.
Not scheduled, not sized, and not a commitment.
The one item still blocked is upstream: the LAN manager's remote access, TT-020. What follows is why.
Three hardware-blocked items were closed on 2026-09-19, not resolved: TT-021 (validate on M1/M2/M4/M5/M6) and TT-022 (ANE behaviour across chip generations) needed machines that do not exist here, and TT-023 (long-context parity beyond the exactness window) needed a machine that can hold the ~360 GB bf16 reference, which this one has no disk for. The two caveats below therefore stand: they are why the tasks existed, and closing the tasks did not answer them.
The chip-range claim is the one to watch. The README and wiki state M1 through M6; all published measurements come from a single M3. Nothing is known to be wrong on other chips — nothing has been tested on them. Treat the range as a design intent, not a measured result, until someone runs the benchmark suite on other hardware.
Long-context behaviour for Qwen 3.8 is verified only at a lowered budget, where the sparse path engages after 67 tokens. The arithmetic is identical at 2,051, but nothing has been diffed against a reference at that length because the reference cannot hold this model.
A related harness limit, not hardware: steady-state decode past the ANE handover transient had never been measured, because every decode figure in the ANE work is a ~60-token window (the pinned benchmark prompt hits end-of-turn early). Measured 2026-09-18 — see Steady-state decode after ANE prefill below: on AgentWorld 35B-A3B 4-bit it costs +0.7%, i.e. nothing.
Open questions with real measurements behind them and no accepted cause.
2026-09-11, "Capital of Paris" matrix, three repeats each. Qwen 3.5 2B 4-bit, asked the ambiguous prompt, answers degenerately on the GPU path -- "The capital of France, and the capital of the country France, is Paris. Paris is the capital and capital city of France. It is the capital of the French Republic and the capital of the French Republic." -- and cleanly on the CPU path with the same weights: "The capital of France is Paris. Located on the left bank of the Seine River..." (51 tokens). The GPU text is identical in all three repeats, and the plain control question is clean on both engines ("The capital of France is Paris.", 8-9 tokens), so this is not sampling noise and not the prompt alone.
The port's equivalence gate is what makes this a puzzle rather than a bug
report: it drives both engines from the same prompt and the layer dumps agree
-- L0-L2 matched the Python reference to bf16 rounding, and after the width fix
L3's v matched exactly. Greedy decoding needs only one near-tie to diverge,
and at 4-bit this one diverges into a repetition on the GPU path. Measured
2026-09-18 — see TT-002: the 2B's two answers are one near-tie below: the
divergence is the fifth generated token, , against is after "The capital of
France", and the cross-engine logit spread there is 0.37–1.44, well above a
0.41 margin. The gate compares dumps, not the argmax of the final logits, which
is why it could not see this.
Measurements: One Prompt, Every Model, prompt 1, rows 1-2.
2026-09-11, "Capital of Paris" matrix. Qwen AgentWorld 35B-A3B 8-bit, asked
that prompt with the server's --reasoning off, spends its whole 128-token cap
inside a <think> block and stops there (finish_reason: length); the re-run
reproduced it exactly (128 tokens, 9.02 tok/s). The 4-bit install under the same
server answers in 8 tokens (stop).
The template is not the difference, and neither is the request path: both
installs carry a byte-identical chat_template.jinja, which with
enable_thinking: false renders a closed <think>\n\n</think> block, and the
only differing sidecar is config.json's quantization spec (4-bit honours two
8-bit overrides -- embedding and head -- and 8-bit is uniform). So the 8-bit
weights reopen a block the prompt had closed. It is not a runaway either: at a
512-token cap it stops by itself at 482 tokens with the answer after the closing
</think>.
What the model does to a client is fixed (C90): a thought the model
starts while the switch is off is reasoning, not the answer, so it rides
reasoning_content and the server logs thinking off, but the model wrote N characters of reasoning on the request's line. Before that fix the scaffold
arrived as content and a client capping tokens got a thinking transcript where
it expected an answer.
What stays unexplained is why two widths of one checkpoint disagree this
sharply, and what the catalog should advertise: /v1/models offers off for
this install as for every other Qwen 3.6-family one, and for the 8-bit install
off still means "think anyway" -- it now says so in the log and in
reasoning_content rather than silently. Three repeats make the shape of it
sharper: the thought appears on the ambiguous prompt only (3/3, 477 characters
each time), never on the plain control question, and never on the 4-bit install
of the same family. No cause is accepted; the report is
One Prompt, Every Model and the raw rows are in
benchmark/benchmark-results/capital-of-paris-20260911T1935/.
Three hypotheses are tested and dead:
- handover wiring latency — 136 ms, cannot account for it;
- expert-cache eviction during prefill — wiring the cache across prefill measured identically, 3,139.9 ms of expert-I/O await against 3,128.7 ms;
- extra expert I/O — this was the documented claim and it is false. Decode reads the same bytes either way: 9.18 GiB at 70.5% hit with ANE off, 9.16 GiB at 70.6% with it on.
What differs is how long those identical reads take to complete: 859 ms of
await against 3,129 ms — same reads, 3.6x the wait. The fourth candidate was
that Core ML's E5RT arenas are not actually returned at releaseModels() and keep
costing memory bandwidth or residency through decode. Tested 2026-09-18 (TT-004)
and dead: the arena measures ~152 MiB and is returned synchronously, with nothing
surviving into decode — see TT-004: the E5RT arenas are returned below. The 3.6x
wait now has no accepted cause.
After the allocation-time pin (1b3fdfe) the 4-bit ANE decode gap went from
−53.6% to −14.7%. Separated 2026-09-18 — see The ANE window penalty is a
re-warm, and the pin removes it below: the pinned configuration performs no unpin
at all, so there is no re-warm to find, steady state is ~0, and the remainder was
drift.
A width-2 verify measured 1.965x a single token where the union model predicts under 1.3x. Attributed 2026-09-18 — see Where the MTP verify pass goes below: the union comes in at the model's price, and the rest is the prefill path's non-expert kernels (1.70x where the model assumes 1.0x) plus ~200 ms/pass of host and commit overhead.
Each of these produced a confident wrong conclusion at least once.
-
finddoes not follow a symlink given as its own starting point, and a release shipped without its resources because of it.release.shstaged the products withcp "$BIN/$product"and the bundles withfind "$BIN" -maxdepth 1 -name '*.bundle'. Both are correct as written — except that this toolchain's$SCRATCH/releaseis a symlink toout/Products/Release, whichcpfollows andfinddoes not. The bundle copy matched nothing, no gate noticed (nothing asserted the archive's contents), and the 5.6 release was published with six executables and noTinyTitan_TinyTitan.bundle: every downloaded binary died withFatal error: unable to find bundle named TinyTitan_TinyTitanon the first model load. The script now resolves the products directory physically, refuses to stage without that bundle, and greps the finished archive for it. The rule this cost a release to learn: after publishing, download the artifact and run it from a clean directory. Every gate in the runbook passed while the published binaries were unusable; the asset was repaired from the same tagged build, without moving the published tag. -
Renaming the checkout folder invalidates more than the model receipts.
.build's debug half is compiled against absolute paths, and afterDownloads/NVMAI→Downloads/TinyTitanit held 7,312 files naming the old path, soswift testdied withprecompiled file …_Builtin_stdbool….pcm was compiled with module cache path '/Users/andreborchert/Downloads/NVMAI/…'before a single test ran.release.shsurfaces that asswift test did not report a passing run, which reads like a failing test and is not one — check the test log for a compiler error. The release build had been rebuilt after the rename and was fine; only the debug tree was stale. The fix is to remove.build/arm64-apple-macosx/debugand let it rebuild: it is not a cache purge to make a gate pass, it is invalid build output. Every install receipt is invalidated by the same rename, for the same reason (absolute paths) — see Renaming an install invalidates its receipt below. -
Benchmark noise is ±15% run-to-run. Only interleaved within-run comparisons are load-bearing. Absolute numbers are not comparable across runs and must be re-taken on a verified-idle machine before publication.
-
Sequential config sweeps fake wins through page-cache warming. Interleave arms, discard a warmup per arm, and always re-measure an unchanged config as a control.
-
GPU contention looks exactly like a regression. A background game halved decode once. Check
ioregbefore believing any regression. -
A full
swift testcan starve a task for tens of seconds. Wall-clock waits are wrong by construction: a test that sleeps for a fixed interval and then asserts is a flake, so wait on a signal or inject the clock instead. -
Top-k A/B is chaotic. Compare sparse-selection paths at the first layer on shared input, never on downstream logits.
-
Dispatch-count arithmetic predicts cost badly here. Folding two gates removed 192 encoders per token and recovered 0.59 ms, not the 3.8 ms that the usual ~20 µs per encoder implies — the real figure is nearer 3 µs.
-
Family constants silently carry over. Four of five Qwen 3.8 porting bugs were Qwen 3.6 constants reused unchanged. Audit every baked function constant when adding a family.
-
The parity harness reads the same weights as the runtime. Quantization is common-mode by design, which is a feature — but so is any weight-mapping or shared-architecture error, and those are then invisible. Ground-truth against
transformers' own modules and against the source checkpoint, not only against the runtime. -
Two false alarms from the Qwen 3.8 debug, both measurement error, both worth not re-chasing: a "layer 0 GDN cosine 0.803" that compared a first-token reference against a position-1 dump, and a "PLE conv 2.6x magnitude" that compared a pre-silu value against a post-silu buffer.
-
A model install in a synced folder can be online-only, and then the golden gate fails with a runtime-looking error. Found cutting 5.3. Dropbox had left seven installs with zero blocks allocated (
fileproviderctl evaluate <path>→isDownloaded = 0), and every packed-expert read returnedparallel expert read failed: Operation timed out, whichrelease.shreports asgolden baseline mismatch (<target>). It is neither a mismatch nor a runtime fault:caton one of the files fails the same way, in 0.6–12 s. Two things make it look like a bug in the model — the message names a read failure inside the streaming pool, and the first target in the list still passes because that install happened to be local. Diagnose withfind models -type f -size +1M -exec stat -f "%b %z %N" {} \; | awk '$1*512 < $2*0.9'before touching the runtime. Materializing is the fix (cat <file> > /dev/null, which is what the provider hydrates on, at ~2–15 MB/s); it needs the full install's worth of free disk, and a volume that is too full makes the provider refuse the fetch outright, so the failure mode appears with no disk-space message at all. Seven installs cost 24 GB to hydrate; the 125B 8-bit install would have needed 134 GB against 123 GB free, soqwen38-8was skipped by name for the 5.3 release rather than deleted from the gate's list — seeTINYTITAN_RELEASE_SKIP_GOLDENSindocs/release-process.md.
-
A model that loads and generates plausible text is not evidence it was built correctly. KAT's first install passed every structural check and produced nonsense, because the converter filed routed experts by arrival order instead of by index: the checkpoint's index sorts lexicographically, so layer 0's experts arrive
0, 1, 10, 100, …, and 12 of 40 layers also split them across two shards. Every expert's bytes matched the checkpoint, so nothing downstream could see it. Two things stand in the way now —tools/lint.sh converterasserts the fused axis is index-ordered, and the converter's--planprints whether the converted experts land underswitch_mlp— but the general lesson is the one to keep: had the manifest cap been permissive, the wrong install would have loaded, written plausible prose, and silently ignored SSD streaming. -
A package update replaces the files under a running harness, and can take it down. Confirmed the hard way on 2026-09-18:
npm install -g @deepseek-ai/dsh@0.1.6-alpha.2changed 488 packages whilenode /opt/homebrew/bin/dsh webwas serving a live session, and that process (pid 15900) ended inside that window. It left no crash report, so it was a signal or a clean error exit rather than a segfault — the exact trigger is not known, and it should not be guessed at. What is not in doubt is the shape: a Node process whosenode_modulesis replaced beneath it has nowhere to load its next not-yet-imported module from. The session came back only because the harness was restarted, and DSH resumes a session from its log — so restarting the harness you are talking through costs the session, not the conversation. The rule: replace the files, then restart, and do not update the harness that is hosting the work you are doing. A source tree is no different —pnpm install && pnpm buildunder a livedsh webrewritesapps/cli/libandnode_modulesexactly the same way. -
npm install -gcan exit 0, report "changed N packages", and still be an incomplete install. npm 11 blocks lifecycle scripts unless they are allow-listed, and DSH needs five of them. The one that matters is silent:node-pty'sspawn-helper— the 0.1.6 sidebar terminal — keeps its prebuilt binary but loses the executable bit, and@deepseek-ai/dsh-subprocess-local's postinstall is what restores it. The version reports correctly and the harness boots either way, so nothing looks wrong until someone opens a terminal. Finish it withnpm install -g --allow-scripts=@deepseek-ai/dsh-subprocess-local,koffi,node-pty,@google/genai,protobufjs @deepseek-ai/dsh@<version>and check thatnode-pty/prebuilds/darwin-arm64/spawn-helperis-rwxr-xr-x. -
A hand-set
baseURLcan 404 every model call, because the harness defaults to the Messages protocol. Found on 2026-09-18: all four fleet nodes (node1–node4) were failing every turn withDeepSeek Messages request failed (404), which the UI shows asHTTP_404with the agent preset loading fine and the system prompt and context injections succeeding — so it reads as a model problem, not a configuration one. Each node carriedllm-deepseek: { baseURL: https://api.deepseek.com/v1 }.dsh-llm-deepseekdefaults toprotocol: messages(lib/index.js:3004), whose official root ishttps://api.deepseek.com/anthropic(MESSAGES_BASE_URL, line 2952);https://api.deepseek.com/v1is the chat-completions root, andmessagesApiRoot()leaves a path already ending in/v1alone — so the requests went to/v1/messages, a path DeepSeek does not serve. Settled by request rather than by reading:/v1/messages→ 404,/anthropic/v1/messages→ 200 withdeepseek-flash. All four were corrected to the anthropic root (each beside asettings.yaml.bak-*) and verified end to end —dsh headless "Reply with exactly: OK"→OKon every one. MacBook-AB was never affected, and that is the part worth keeping: its DeepSeek config is generated —tools/dsh_route.shwrites anllm-pi-airoute and nollm-deepseekblock at all, so it inherits the correct default. The nodes were provisioned by hand in one sitting (all foursettings.yamlmtimes fall within 11 seconds of each other, 2026-09-15 16:27), which is exactly why one wrong value reached all four. AbaseURLcopied from generic DeepSeek/OpenAI docs is wrong here unlessprotocol: chat-completionsis set alongside it; generated config does not make that mistake, and config typed once per machine does. One member is still unaudited:node5(100.99.92.66) is up with sshd listening but refuses the key for every username tried, so what it runs is not known.
Two facts about verified-install.json that cost time on 2026-09-01 and will
confuse a provenance audit later.
The receipt binds to an absolute path. Renaming or moving an installed
model makes it refuse to load: "the receipt was issued for <old> but the
model is now at <new>". The runtime is right to refuse -- it will not run an
unverified payload -- and the error names both paths and the exact fix. Repair
is TinyTitanRepack --verify-install --input-gturbo <dir>, which re-hashes in
place with no re-download: a few minutes for a 162 GiB install, seconds for a
1.4 GiB draft head. Plan the re-issue as part of any rename, not as a
discovery at load.
A re-issued receipt drops sourceRepoID. After re-issue the field reads
None and toolVersion becomes TinyTitanRepack verify-install. This is not
damage from the re-issue path being wrong -- Ornith installs nobody has touched
show the same, having been verified at some point too -- but it means a
re-issued receipt records what the payload hashes to and no longer where it
came from. Provenance survives in sourceRevision, which keeps the snapshot
hash, and in manifest.json's own sourceSnapshotHash.
Practical consequence: do not read sourceRepoID as evidence of origin.
For the Qwen3.8 pair it is None on both the target and the draft head, and
both were nevertheless built from Qwen's own bf16 release -- the snapshot
hashes are what prove it.
Shipped. 8-bit was refused at load from 3e86f78 until the kernels were
given a path for it; it is now the model's golden build.
HyperConnection, PLEBlock and QSAIndexer each constructed a
DequantInt4GEMV with no affine variant and no bit-width parameter, so an
8-bit install handed them 8-bit weights that they read as nibbles. It did not
throw: the install built, passed the snapshot verifier, loaded, answered
" Paris" and then degenerated. Each kernel now dispatches on width.
The build pipeline was verified by that exercise, which was the point of
running 8-bit before the 4-bit rebuild. It also found five defects, four latent
in code predating the day: an argument-parser loop that never terminated, a
tokenizer requirement that refused draft heads, an entryKind double-optional
that made a "file exists" test true for absent files, and a converter that had
never fetched tokenizer files -- every snapshot it ever produced would have
been rejected at repack, hidden because the 4-bit one was hand-repaired.
The lesson is not "be more careful". These paths had one caller and one hand-held run; every assumption in them went unchallenged until a second consumer arrived. The same shape recurred four more times during the promotion work below.
8-bit exists for a different user than 4-bit does: someone with a correctness-
sensitive, time-insensitive job -- reviewing a design, fixing a subtle bug --
who will run the model overnight or on a dedicated machine and needs to trust
the output. For that user, speed is the cheap currency and quality is the dear
one. So eight tensor suffixes across seven families are carried at bf16
even in the 8-bit build, chosen by per-tensor error from
tools/precision_probe.py -- the tensors where quantisation error changes a
decision rather than nudging a value:
| family | params | what it decides |
|---|---|---|
mlp.gate |
62.91M | which experts run |
ple.key_proj |
26.21M | n-gram retrieval keys |
indexer.index_q_proj / index_k_proj
|
19.66M | which KV entries QSA attends to |
linear_attn.in_proj_a / in_proj_b
|
8.85M | GDN gating |
*_hyper_connection.block_inject_weight |
3.93M | the write gate on every layer |
mlp.shared_expert_gate |
0.12M | 1 row; highest measured error |
| total | 121.7M |
Every one of these is a resident weight. model_weights.bin grows by about
114 MB and the streamed expert files are byte-for-byte unchanged, so this does
not touch the decode I/O floor in Decode throughput has a hard ceiling at all -- the cost is RAM and a little
compute, not bytes.
Promotion is decided in prepare_qwen38.py:PROMOTE_TO_BF16_AT_8BIT, checked
before the slot rules, because the runtime reads a promoted tensor's width
from its own dtype rather than from the slot it nominally belongs to.
Four separate consumers had to be taught the same lesson, each found only after
the previous fix: encodePrimaryGEMV (8-bit builds skip the fused GDN path
entirely), encodeAffineProjection's batched MPP/QMM, the prefillGateProjection
closure, and the PLE batched projection closure. Each one read scale and bias
pointers out of a struct that a promoted tensor does not populate.
Three of the four surfaced as NaN at L0_attn_out and were quick. The fourth
was not: it produced fluent output on most prompts and truncated on some.
"What is a mutex?" and a haiku both answered correctly while "Explain what a
mutex is and when you would use one." ended the turn after one token -- a
corrupted PLE injection does not always overwhelm a strong prior. Probe count
did not find it; prompt diversity did. Sequential prefill was correct
throughout, which localised it to the batched path both times.
- install audit: 497 8-bit entries, 582 bf16, 0 problems
- full-stack parity against the numpy reference, all 48 layers, stack-out cosine 0.99999
- 10-prompt sweep on the batched path, all coherent (arithmetic, code, translation, factual recall)
-
qwen38-8golden captured and re-checked identical;qwen38-4golden still reproduces, so the shared-kernel changes did not regress 4-bit
Not measured: the decode throughput cost of the promotion, and the end-to-end quality gain. The first is predicted small (compute-only, no extra bytes) but the machine has been contended for days and this project has a standing rule against timings taken under contention. The second was argued from per-tensor error, not benchmarked -- there is no coding-quality harness here yet, and this page's own rule is to label that rather than imply a number.
docs/qwen38-decode-20tps-concept.md. Decode reads 480 expert records of
2.77 MB per token -- 1.33 GB if every read missed. The cache measured
saturating at 78% around 128 slots, and effective read bandwidth measured
3.6 GB/s, so 292 MB / 3.6 GB/s = 81 ms of I/O alone: a 12.3 tok/s
ceiling for perfect engineering at the shipped geometry.
Overlap, prefetch depth, dispatch fusion and kernel work hide latency. None reduces bytes, and the floor is bytes over bandwidth. Engineering is worth 6.82 -> ~12 tok/s (+75%); 20 tok/s needs an 86.5% hit rate at that bandwidth, or 7.2 GB/s at that hit rate, and only three things cross it: more resident memory, faster storage, or reading fewer experts -- the last of which changes the model's output.
Provisional. The decode split for this model has never been measured; everything but the cache curve and the 3.6 GB/s is derived. The concept states what would falsify it, and this project has twice designed against estimates wrong by more than an order of magnitude.
TINYTITAN_KEEP_WIRED used to be read as == "1" in three places —
ModelProfile.resolve, RealForwardRunner.keepExpertCacheWired and
Model.keepExpertCacheWired — so the only thing it could express was "on". Every
table row that streams experts already sets keepWired, so on this 24 GB Mac the
12 GiB cache could not be paged out at all: 14.72 GB RSS, 11% system memory free,
CoreAudio glitching while a model was loaded.
It is a tri-state with one reader now. ExpertCacheWiring.override(environment:)
returns true for 1, false for 0 and nil for anything else; resolve folds
it over the row in ModelProfile.resolve, and the resolved profile carries the
answer. The two duplicate statics are deleted rather than kept in step — the
runner reads profile.keepExpertCacheWired, and Model.openLayerLocked reads the
flag setKeepExpertCacheWired already set from that profile — so the drift class is
gone, not just this instance. releasePrefillCacheWiring is what makes =0 pay:
nothing pins at allocation, prefill's release is a real munlock, and the first
decode token pins exactly as before — the trade the variable was supposed to offer.
Verified. =0 on qwen3.6-35b-a3b 4-bit — a row that wires it — logs
keep_wired=false under TINYTITAN_RUNNER_STATS, =1 logs true, and an
unrecognised value leaves the row alone. 1,407 package tests pass (two new ones
pin the tri-state vocabulary and both directions), tools/lint.sh is clean, and the
golden check is byte-identical on qwen38-4 (the broad gate: hyper-connections,
PLE and QSA), qwen36-4 and agentworld-4. ornith-8 is reported not
checked — its install is absent and nothing is fetched to change that. The default
path is provably unchanged: with the variable unset the old env == "1" || profile
reduced to profile, which is exactly what the call sites read now.
Found 2026-09-02 while checking whether the 4-bit MTP head pairs with the promoted 8-bit target. The pairing failed with
tensor ...layers.0.mlp.gate.weight size 2621440 does not match expected 1310720
which is exactly the promoted bf16 router -- 512 x 2560 at 2 bytes against the 8-bit slot's 1. It looked like a fifth unpatched consumer of a promoted tensor.
It was a stale binary. TinyTitanServer was 15 hours older than the last
promotion fix: the whole four-bug loop in the 8-bit Qwen3.8 work had rebuilt TinyTitanCLI and nothing
else. On a rebuild the server loads the promoted model, and the 4-bit sidecar
attaches to it and speculates correctly (mtp=on:384MiB,
emitted_per_pass=2.000). Mixed-width pairing is supported by design --
sharingTargetWeights checks family, hidden size, vocab and lineage, never
width.
The reason it went unnoticed is structural, not careless. golden-baseline.sh
ran TinyTitanCLI exclusively, and its contention guard refuses to run while a
server is up, so no gate in the repository had ever executed a line of the
server path. It could drift arbitrarily far behind and every check would still
be green.
tools/golden-baseline.sh --server now closes it, and deliberately stores
nothing. A recorded server baseline would need re-capturing after every
intentional numerics change and could go stale exactly as the binary did.
Instead it renders one message list through both front ends at the same
settings and requires the text to match, so:
| mode | catches |
|---|---|
| the stored file | the runtime changed, against its own past output |
--server |
the two front ends diverged, on one build |
Both legs go through --messages-file: the server has no raw completions route
and always applies the chat template, so comparing it against raw --prompt
output would have compared two different prompts. Verified on qwen38-8 --
agrees, 439 bytes. The comparator was checked separately to actually fail on
differing text, because a gate that can only print "ok" is worse than none.
The generalisation, third instance now: verification that drives one caller proves one caller. It was true of the family kernels, of the four promoted-tensor call sites, and now of an entire front end.
The thread-sanitizer CI job had been red on some commits and green on others. It
was never diagnosed from a CI log here, so the first job was to make it reproduce:
six instrumented TinyTitanServerTests processes at once, which report the same
race in several of them. A single process is almost always clean, which is why the
gate read as flaky rather than broken.
The report is always this shape:
Write of size 8 … by thread T3:
#0 closure #1 in SSEOutbox.next() HTTPServerSupport.swift
Previous read of size 8 … by thread T72:
#0 UnsafeContinuation.resume(returning:) <compiler-generated>
#1 closure #1 in closure #1 in EventLoopFuture.get() AsyncAwaitSupport.swift:90
… SelectableEventLoop.run …
Location is heap block of size 1016 allocated by thread T2:
#2 closure #1 in closure #3 in ServerHTTPHandler.handleResponses(…) Responses.swift:108
It is not a race in SSEOutbox. The racy block is the drainer task's own heap
allocation — the Task { … } closure context created at Responses.swift:108, the
allocation the async frame is carved from — not the outbox, which is a separate small
object. Every SSEOutbox field (frames, pendingDrain, closed, overflowed,
abandoned, closeAfterDrain, drainCancelled) is only touched under its NSLock,
and all four resume sites were read looking for a double-resume path: there is none.
What the two stacks show is the NIO event loop resuming the drainer
(EventLoopFuture.get(), from writeSSEChunk) and that task, woken on a GCD worker,
writing its own frame: the happens-before edge of a swift_task_switch that TSan does
not see when the resumer is not a Swift task. Contention is what makes it land, which
is why CI — and six local processes — expose it while one run does not.
Decision: one suppression, race_top:UnsafeContinuation.resume, in
tools/tsan-suppressions.txt, wired into the CI job through TSAN_OPTIONS.
race_top matches only the top frame of one of the two stacks, so the rule fires on
Swift's own continuation-resume helper and nothing else: a real race in TinyTitan code
still has a TinyTitan top frame and still fails the job. Measured: with the file, a
six-process storm reports nothing and TSan prints Matched 1 suppressions on the runs
that would have fired; with a pattern that matches nothing, the same storm reports the
race three times out of six.
tools/tsan-storm.sh is the reproduction, kept so the decision can be re-checked —
--no-suppressions runs the raw storm — and the entry should be deleted if a
toolchain or swift-nio update makes it clean. This is a toolchain false positive worth
reporting upstream; it is not in this repository's gift to fix without giving up the
async→NIO bridge the whole streaming path is built on.
Every ANE decode figure in this project was a ~60-token window, because the
pinned qualification prompt asks for a 40-word summary and hits end-of-turn at
about that point; --max-new never bound. Such a window contains the one-time
handover cost, so those figures were upper bounds and the steady-state cost had
never been measured at all.
The prompt fix. The qualification body plus "Continue … write at least 700
more words … without stopping early" generates to the cap instead of stopping:
stop=maxTokens, 448 tokens asked for and 448 delivered. That one prompt change
is what makes the measurement possible.
The measurement. benchmark/ane_steady_state_decode.py runs the same arm at
two generation lengths and differences them,
(new_big − new_small) / (decode_big − decode_small), so every one-time cost both
runs pay — first token, wiring walk, expert-cache re-warm — cancels exactly. Arms
alternate gpu ane ane gpu. On AgentWorld 35B-A3B 4-bit, an 11,948-token
prompt (three ANE chunks), greedy, two runs per arm and length:
| prefill median | steady state [64, 448] | window ≤64 tok | window ≤448 tok | |
|---|---|---|---|---|
| GPU | 622.5 s | 7.21 tok/s | 6.66 tok/s | 7.17 tok/s |
| ANE | 187.7 s | 7.26 tok/s | 6.79 tok/s | 7.19 tok/s |
ANE prefill costs +0.7% of steady-state decode — nothing, inside the run-to-run spread. The 64-token windows show no penalty either (+1.9%). The ~60-token upper-bound worry does not survive a generation long enough to leave the window: there is no steady-state decode cost to find on this install, at this prompt length, with the shipped wiring.
One new trap: the first ANE prefill of a process pays a compile warm-up. It measured 390.44 s against 187.83 s for the identical second run — 2.1x — and the two later ANE runs sat at 185–188 s. Any single ANE prefill figure taken from a cold process overstates it, and a benchmark that does not discard a warm-up per arm will report the compile cost as the sidecar's speed. The qualification harness already discards one warm-up per arm, which is why its numbers never showed this.
Caveats, so this is not over-read. Two runs per cell, one block (medians of
2). The GPU arm's decode follows a ~10-minute GPU prefill, the ANE arm's a
~3-minute ANE one, so a thermal advantage for the ANE arm cannot be excluded —
the earlier Ornith figures (8.70 against 8.68 tok/s) showed no such asymmetry.
Ornith, the install every earlier ANE number was taken on, is not installed; this
uses AgentWorld 4-bit, the same qwen36 family and geometry. And AgentWorld's row
wires the expert cache through prefill, so the expert re-warm that TT-005 is
about is not present in the shipped configuration measured here — whether a
pageable cache reintroduces a transient is TT-005's question, and
TINYTITAN_KEEP_WIRED=0 (TT-008) is the switch that can now ask it.
The proposal was to raise this family's routed-expert budget to 128 slots
(15.84 GiB), on the strength of a simulated LRU trace that put the hit rate at
65% for 64 slots and 78% at 128, "where it saturates". The runtime's own counters
say something different, so benchmark/expert_cache_slots.py swept the count
directly (--expert-cache-slots), 64/96/128 then 128/96/64, on a 7,550-token
prompt, 256 new tokens, greedy, two runs per point:
| slots | cache | hit | tok/s | MiB/token | max RSS |
|---|---|---|---|---|---|
| 64 | 7.92 GiB | 67.7% | 3.44 | 408.0 | 12.0 GiB |
| 96 (shipped) | 11.88 GiB | 76.5% | 3.66 | 296.7 | 15.4 GiB |
| 128 | 15.84 GiB | 82.6% | 1.21 | 220.0 | 15.8 GiB |
The hit-rate case for 128 is real, and the throughput case against it is
decisive. From 96 to 128 the hit rate gains 6.1 points and per-token expert
reads fall 26% — the curve is still climbing, so the simulated "saturates at 128"
is wrong — but decode falls from 3.66 to 1.21 tok/s, reproducibly (1.22 and 1.21).
At 15.84 GiB the slot cache plus the mapped weights no longer fit a 24 GiB machine
and the runtime pages: the win from fewer misses is spent several times over on
swap. 64 slots is cheaper still (12.0 GiB RSS) and also slower (3.44), so the
shipped 96 is the right point for this machine class — which is what
affordableExpertCacheBudget's half-of-RAM cap already encodes.
No code change. Raising the row's wanted budget to 15.84 GiB would be inert on
a 24 GiB Mac (the cap clamps it back to 12 GiB) and would change behaviour only on
machines this repository cannot measure. The escape hatch for those is
--ram-budget 16G or --expert-cache-slots 128, and this curve is what says when
to reach for it.
A width-2 verify measured 1.965x a scalar token where the union model predicted
well under 1.3x, and 0.4–0.7x of that had never been attributed. The B1 work in
docs/v4.4-decode-width-plan.md measured the phases on Ornith 8-bit; that install
is gone, so this re-does the attribution on the pair that is installed —
Qwen3.8-Flash-Next 4-bit with its own draft head
(benchmark/tinytitan_mtp_phases.py, which now takes --target/--sidecar
instead of the uninstalled Ornith map).
256 greedy tokens, 86.9% acceptance, 1.869 emitted per pass, 137 passes, two runs per arm agreeing to 0.05%:
| component | ms/pass | × a 214.45 ms scalar token |
|---|---|---|
| verify backbone (wall) | 417.35 | 1.946 |
| — its GPU kernels | 217.2 | 1.013 |
| ·· routed pair (the union) | 51.9 | 0.242 |
| ·· non-expert prefill kernels | 165.3 | 0.771 |
| — host + commit + wait | 200.2 | 0.933 |
| verify head | 22.17 | 0.103 |
| proposal (draft layer) | 23.59 | 0.110 |
| commit (accepted draft) | 15.13 | 0.071 |
| pass total | 480.03 | 2.238 |
The union is not the problem. verify_routed_pair at 51.9 ms/pass is 1.45x the
scalar routed GPU time (35.9 ms), at or under the 1.585x the union model predicts.
What the model assumes is ~1.0x is the non-expert path, and it measures 165.3 ms
against the scalar's 97.3 ms — 1.70x — because the two rows run the 32-token
prefill kernels, not the decode ones. The largest single piece is not a kernel at
all: the backbone's wall exceeds its kernel time by 200 ms/pass, 0.93x a whole
scalar token, which is per-tile MTLArgumentBuffer construction, a command
buffer per tile, the cache plan, and the sequential fetch awaits that have nothing
to overlap at width 2. TINYTITAN_MTP_VERIFY=pair — the default, and what these
numbers describe — recovers only 3–5% of it (v4.4 B2).
End to end this is why acceptance is not the lever: 86.9% acceptance emits 1.869 tokens per pass and the pass costs 2.238, so MTP decodes 16.5% slower than scalar (3.892 against 4.663 tok/s), with byte-identical output. The floor analysis in v4.4 says a decode-style two-row attention and perfect overlap still lands near 1.6x — above what 2.0x-max acceptance can buy once the scalar baseline is fast. MTP stays off by default.
Caveats. A browser renderer was on the CPU (GPU reported 3% busy); the two MTP-on runs agreed to 0.05% and their kernel roles to within 0.1%, so the ratios are reproducible, but the 200 ms host figure could be inflated by that contention. It is also what v4.4 B1 found independently on a different model, so the conclusion does not rest on it. Kernel role sums overlap by design, so the GPU split is a role measurement, not an additive budget.
The residual −14.7% at 4-bit, after the allocation-time pin, had three candidate explanations: a first-token re-warm still inside the ~60-token window, a remaining Core ML arena footprint, or drift. They are separated now, and the separation is the wire trace rather than a rate comparison.
TINYTITAN_WIRE_TRACE=1 records every setExpertCachePinned call that takes more
than 2 ms. AgentWorld 35B-A3B 4-bit, 4,043-token prompt, two runs per arm and
length, fresh process each:
arm (TINYTITAN_KEEP_WIRED) |
init pin | unpin at prefill | re-pin at first decode token |
|---|---|---|---|
1 — shipped row, pinned at allocation |
8.8–15.2 ms | none | none (early return) |
0 — pageable through prefill |
under 2 ms | walked, under 2 ms | 888 / 1,588 / 2,084 / 22,077 ms |
Correction, later the same day: the prompt above was too short to engage the
sidecar. The ANE serves chunks of exactly 4,096 tokens, and 4,043 tokens is one
partial chunk, so both "ANE" arms above prefilled on the GPU — the runtime logs
ane-prefill fallback: chunk at 0 (+4043) outside sidecar coverage, which was not
checked at the time. The wiring result stands, because it is cache state rather than
engine state, but it was not measured after an ANE prefill. Re-run on an eligible
11,948-token prompt (two full chunks, no fallback line):
| arm | first decode pin | decode, 8 tokens |
|---|---|---|
KEEP_WIRED=1 (shipped) |
13.4 ms, walked 1 (already pinned) | 2.72 s → 2.95 tok/s |
KEEP_WIRED=0 (pageable) |
10,776.5 ms, walked 40 | 12.04 s → 0.66 tok/s |
After a real ANE prefill the pageable cache re-pins all 40 layers in 10.78 s — 810x the pinned arm's 13 ms, and about half of a 60-token window at this model's ~3 tok/s. That is the window penalty the pre-pin and phase-scoped figures recorded, now measured with the sidecar engaged. The GPU-prefill numbers above (0.9–22 s, memory-state dependent) remain what the same mechanism costs when the eviction pressure comes from elsewhere.
1. First-token re-warm. It cannot exist in the pinned configuration: the cache is never unpinned, so nothing is evicted and nothing faults back. When the cache is pageable it is real, and it is the size of the old window penalties — 0.9–2.1 s against a 64-token window of 7–9 s is 13–24%, and the one run that met memory pressure paid 22.1 s, longer than the whole window. The pre-pin and phase-scoped window figures (−27.9%, −53.6%) are this mechanism; the spread across those GPU-prefill runs is the machine's memory state, and with the sidecar engaged the pageable re-warm measured 10.78 s (the correction table above).
2. Remaining arena footprint. Not measurable at steady state. The wired arm's
ANE-vs-GPU decode here is −1.1% (window ≤64: −1.0%), reproducing TT-007's
+0.7% on a 12K prompt. TT-004 has since tested the arena directly: it is ~152 MiB
and releaseModels() returns it synchronously, with nothing surviving into decode —
so this candidate is dead, not merely unmeasurable.
3. Drift. What is left, and already retracted once: the −14.7% comparison came from a session whose GPU arm drifted 8.369 → 7.389 tok/s, and v4.6 says only within-run comparisons are load-bearing there. This session demonstrates how large that can be — prefill for one identical configuration ranged 96–162 s, and the pageable arm's decode ranged 2.2–11.7 tok/s — while the wired arm's four runs agreed to 1%. With the pin, the interleaved A/B shows no residual to explain.
Verdict. The residual −14.7% was drift, not a mechanism. The window-contained
re-warm belongs to the pageable wiring, which no longer ships, and the pin is what
removed it; TINYTITAN_KEEP_WIRED=0 (TT-008) is how to get that configuration back
deliberately.
Caveats. This session ran with a browser renderer holding CPU and ~19% GPU, so the rate comparisons are directional — one identical prefill ranged 96–162 s. The wire-trace times are direct measurements of the pin calls and do not depend on that; the pageable arm's decode rates are not usable and are not quoted.
What was recorded. On 2026-09-11, with --reasoning off, Qwen AgentWorld
35B-A3B 8-bit answered "Capital of Paris" by opening a <think> block of its own
and spending the whole budget inside it — 477 characters, three repeats,
byte-identical, and at a 512-token cap it closes and answers at 482. Its own
control prompt was clean, and the 4-bit install answered in 8 tokens, so the tracker
carried it as "the 8-bit does, the 4-bit does not". Raw rows:
benchmark/benchmark-results/capital-of-paris-20260911T1935/.
The install was restored and the comparison re-run with more than one prompt.
The 8-bit install (converted and repacked 2026-09-18, byte-identical to its stored
golden) turns the single observation into this, at --reasoning off, greedy,
128-token cap, both widths:
| prompt | 8-bit reasoning chars | 4-bit reasoning chars |
|---|---|---|
Capital of Paris |
477 (length, empty answer) |
0 (answered, 31 chars) |
What is the capital of Paris? |
470 (length) |
471 (length) |
Who is the president of Paris? |
0 (answered) | 0 (answered) |
How many capitals does Paris have? |
487 (length) |
504 (length) |
What is the capital of France? |
0 | 0 |
Name the capital city of Japan. |
0 | 0 |
Both widths reopen a closed block, on different prompts. The 4-bit install thinks
on two of the four ambiguous prompts and not on the one the original row used; the
8-bit thinks on three. So this is not a property of the width — it is a
false-premise prompt pulling a deliberation out of a family whose weights treat the
closed <think> as a weak cue — and the original "8-bit does, 4-bit does not" was one
prompt over-generalised. The controls are clean on both, and --reasoning on changes
the thought's content (1,025 characters against 1,860) rather than deciding whether
there is one: the level is a request, not a gate.
What is still true from the earlier analysis. The template renders
enable_thinking: false as a closed block
(models/qwen-agentworld_35B_A3B_4Bit/tokenizer/chat_template.jinja:154), and the two
installs' templates were checked byte-identical; C90 (c7fca53) routes an
unrequested thought to reasoning_content and logs its size, which it did on all five
of these runs (thinking off, but the model wrote N characters of reasoning).
The catalogue decision: advertise nothing differently. /v1/models offering off
describes what the server asks for, and the server does ask for it. Special-casing the
8-bit width would encode a single-prompt artefact, and removing off from the family
would be worse than useless because the level does not gate the behaviour in either
width. What a client needs to know is that off is not a promise about the weights,
that an ambiguous prompt can produce a thought, and that a small max_tokens can
therefore return an empty content; C90's split and log line are the honest surface
for that, and the wiki page for the matrix now says so.
Qwen 3.5 2B 4-bit answers "Capital of Paris" degenerately on the GPU path ("The capital of France, and the capital of the country France, is Paris…") and cleanly on the CPU path ("The capital of France is Paris…") from the same weights, deterministically, three repeats each. The port's equivalence gate compares layer dumps, so the argmax of the final logits was never compared and the divergence had no name.
TINYTITAN_LOGIT_TRACE=1 — added for this, on both sampling paths: sampleOnce for
the GPU and CPUSampler.pick for the CPU — prints each step's top-2 and margin. One
server, the same chat request, temperature 0, 64-token cap:
| generated step | GPU top-1 | GPU top-2 | GPU margin | CPU top-1 | CPU top-2 | CPU margin |
|---|---|---|---|---|---|---|
| 3 |
France 23.33 |
Paris 22.52 |
0.81 |
France 23.71 |
Paris 23.14 |
0.57 |
| 4 | , 24.67 |
is 24.27 |
0.41 | is 25.71 |
, 24.30 |
1.40 |
The engines agree for the first four generated tokens and diverge at the fifth,
on the same two candidates with the ranking swapped: the GPU continues with a comma,
the CPU writes "…France is Paris". That is not a tie-break on an exact tie — the two
engines' logits for the same token differ by 1.44 for is and 0.37 for ,, so
the CPU's ordering is a genuinely different score, not a wobble.
How close a tie has to be. Over the shared top-2 candidates of the first 17 steps, the cross-engine |logit difference| is min 0.007, median 0.90, max 4.02. A step can therefore flip whenever its top-2 margin is below that spread: step 4 flipped at a GPU margin of 0.41, while steps 11 and 34 held margins of 0.45 and 0.16 and did not flip, because the error's sign happened to favour the same token. Margins under about 1 are the danger zone on this install; nothing above the observed 4.0 spread can flip at all.
Consequence. Neither engine is wrong — each computes the 4-bit model correctly
and they disagree about a continuation those weights make nearly equiprobable. The
GPU's text is a second, lower-probability continuation (a comma after "France" is the
start of the degenerate repetition) and nothing in the equivalence gate could see it.
A third opinion would need the numpy reference, which needs the
.build/qwen35-2b-affine-4bit snapshot; it is not present, and re-converting is a
separate job. This is a 4-bit quality finding, not a defect in either engine.
The ANE decode gap had one untested candidate: that Core ML's E5RT arenas are not
actually returned at releaseModels() and keep costing residency and bandwidth
through decode. TINYTITAN_ANE_MEMORY_TRACE=1 now reports phys_footprint at the
points that decide it — when a layer's MLModel becomes resident, at
releaseModels() before and after the drop, and at the prefill→decode boundary.
AgentWorld 35B-A3B 4-bit, 11,948-token prompt (two full 4,096-token chunks, so the
sidecar really serves it — no fallback line), --max-new 8:
| point | footprint |
|---|---|
releaseModels(), last model resident |
14,893.8 MiB |
immediately after residentModel = nil
|
14,741.6 MiB |
| prefill→decode boundary, ANE on, cache pinned | 14,741.6 MiB |
| prefill→decode boundary, ANE on, cache pageable | 14,184.6 MiB |
| prefill→decode boundary, ANE off (GPU prefill) | 14,423.5 MiB |
The arena is ~152 MiB and it is returned synchronously. The footprint falls by
exactly that (152.2 MiB pinned, 158.9 MiB pageable) at the moment the last MLModel
reference is dropped, and 498 MiB more when our own masks and shadow rows are freed.
Nothing arena-sized survives into decode: the ANE arm's boundary footprint sits
−239 to +318 MiB from the GPU arm's, which is run-to-run spread, not the ~1 GB a
retained arena would need. (The ~1 GB figure in ANEPrefillAttention's comment is
about the h4096 score tensors on the larger variant; on this install the last resident
model's contribution at release measured 152 MiB.)
So the fourth hypothesis is dead: the 3.6x longer expert-read awaits with ANE prefill (859 ms against 3,129 ms on identical bytes — the reads themselves are the same) are not caused by an E5RT arena staying resident. TT-004 closes with that negative result; the awaits remain unexplained, and TT-005 shows what else was measured around them.
Three cheap-looking ways to raise 4-bit quality, each measured and each rejected. The numbers are here so the ideas are not re-derived from first principles a third time. A fourth, per-tensor bit widths inside a 4-bit install (TT-025), is now closed the same way: its runtime half ships, and the two end-to-end checks under Does precision buy quality? below both find no measurable quality case for the converter policy.
Quantizing real checkpoint tensors with TinyTitan's own quantiser, affine group-64:
| 4-bit | 8-bit | |
|---|---|---|
| weight error (mean|err|/mean|w|) | 10.8-12.0% | 0.64-0.72% |
| cosine vs bf16 | 0.9936-0.9949 | 0.99990-0.99998 |
The shipped "4-bit" build is already mixed precision -- router and
embedding (which carries lm_head) are 8-bit, attention, routedExpert
and sharedExpert are 4-bit. The parts whose error does not average away are
already protected.
Both QSA key selections are host computations behind a barrier. The decode one
already has a GPU implementation (encodeSelectKeys, opt-in through
TINYTITAN_QSA_GPU_SELECT) and it measured a wash, which is why the profile row
keeps qsa_select off. The prefill one (selectKeysPrefill) has no GPU path, so
before building one this measured what it could be worth.
TINYTITAN_QSA_SELECT_TRACE=1 times it per layer per chunk on
qwen3.8-flash-next 4-bit, GPU prefill:
| prompt | prefill | QSA host selection | share |
|---|---|---|---|
| 2,410 tok | 95.64 s | 185.7 ms — 12 calls, mean 15.5 | 0.19% |
| 11,948 tok | 864.90 s | 23,748.7 ms — 36 calls, mean 660, max 1,461 | 2.75% |
The cost is superlinear in the visible window: on the long prompt the first chunk's twelve layers cost 1,775 ms, the second 9,222 ms and the third 12,752 ms, because a row past the ~2,051-key exactness window pays a block sort plus an O(visible) compaction, and later chunks have more such rows and more blocks to rank. (Both runs shared the machine with a 70 GB model conversion, so the host times are upper bounds.)
Rejected on the ceiling, not on the implementation. A GPU prefill path would need
a multi-row version of the existing single-row mask kernel and GPU compaction — the
ascending index list and counts the prefill attention loops over, for which the decode
path has no equivalent. It could recover at most the 2.75% above, on one family, on
the longest prompts, minus whatever the GPU spends doing the same work; the decode
half of the identical idea shipped and was a wash. The mask semantics are also
exact: any difference in a tie-break changes which keys are attended, so the
implementation carries a numerics risk a ≤2.7% TTFT ceiling does not pay for.
TINYTITAN_QSA_SELECT_TRACE=1 stays, so the number can be re-taken if the window, the
compress ratio or the prompt-length mix changes.
Min/max per group wastes levels when one outlier stretches the range, so search
a clip factor that minimises MSE instead. Zero runtime cost, same format, same
kernels -- a change to quantize_affine alone.
Measured across five real tensors: 10.92% -> 10.49%, a 3.9% reduction. Real weights inside a 64-group are well-conditioned enough that min/max is already near-optimal. Not worth the change.
The indexer is the only ranking still at 4 bits. A synthetic test on random Gaussian input suggested a large effect -- 90.0% top-8 agreement at 4-bit against 99.3% at 8-bit -- which is what made this look worth building.
On real activations it is much smaller. Driving the numpy reference through layer 3 for 110 real tokens at budget 32 (keeping 32% of keys, a reasonable proxy for a ~6k-token context at the production 2048 budget):
| keep-set agreement with bf16 | mean | worst position |
|---|---|---|
| 8-bit indexer | 99.77% | 91.43% |
| 4-bit indexer (shipped) | 97.41% | 85.71% |
2.4 points, and the keys that differ are by construction the marginal ones at the selection boundary -- the lowest-scoring blocks, where swapping one for another is close to a no-op. Real activations separate the block scores far better than random input does, which is why the synthetic number was misleading.
The cost side is not small either. QSAIndexer unconditionally constructs
DequantInt4GEMV with no bits parameter, where every other call site uses the
weightBits == 4 ? int4 : affine pattern -- so this is a kernel-selection
change, a new manifest slot to carry the width, a converter change, and a
rebuilt install. And a second install does not fit: 128 GB free against a
165 GB install, so it would need the unimplemented hardlink sharing.
The lesson worth keeping: measure ranking stability on real activations, never on synthetic input. The two answers differed by 7 points and pointed to opposite decisions.
The one option that needs no code change: the runtime already supports the slot
at 8 bits (Ornith 8-bit runs attention=8), so it is a converter policy
change. It is also the largest lever, because dense is resident rather than
streamed:
| active params per token | |
|---|---|
| routed experts (10 of 512, 48 layers) | 2.36 B (39%) |
| dense / attention slot | 3.68 B (61%) |
Taking 61% of the active computation from 11% error to 0.7% costs no SSD bandwidth, which is what bounds decode. It costs +2.10 GB resident.
The measured RAM is what constrains it. Qwen3.8 4-bit on the 24 GB M3, 3093-token context: RSS 3.66 GB after load, peak RSS 15.12 GB, phys_footprint peak 18 GB. That is 3.2 GB of resident weights plus a 12 GiB expert-cache budget, so the cache is essentially all of it. Adding 2.10 GB pushes peak to ~20 GB of 24 GB, and the only way to claw it back is shrinking the expert cache -- which is what buys the throughput.
Worth revisiting on a machine with more memory, or against a measured cache-size/quality trade. Not free on this one.
The premise was that decode leaves seven of eight cores idle while the GPU is busy 39.5 to 48.5 percent of the token, so a share of the routed experts could be computed on the CPU in the gaps. A streaming-read probe supported it: 45–60 GB/s during inference without slowing the GPU. The probe was wrong about the workload — a sequential read is prefetch-friendly and low-power, a dequant kernel is neither.
Running the actual CPU expert kernel (8 threads, int4 dequant plus GEMV) as a load generator during decode:
| CPU load | tok/s | GPU busy/token |
|---|---|---|
| off | 19.886 | 27.335 ms |
| 8 threads of real dequant work | 15.387 | 39.608 ms |
GPU-busy rose 44.9% and throughput fell 22.6%: the dequant kernel competes with the GPU for the memory controller and the package power budget. The arithmetic closes it rather than deferring it — at best the CPU absorbs ~32% of the work, but the GPU then does the remaining 68% at 0.69x, which is slower than the GPU doing all of it alone. There is no split ratio that wins.
CPUExpertFFN and the TinyTitanKernelsC target stay in the tree because they
are correct, tested and cheap to keep; nothing may call them from the decode
path. docs/cpu-coexecution-plan.md carries the full argument, including what
is left for a 2x: the idle is the only slack (a 1.71x ceiling), and during
compute the bus already runs at 83% of what a pure CPU streaming read achieves.
The conversion-time question was which precision-sensitive tensors to keep at 8
bits rather than 4. tools/precision_plan_qwen35.py found k_proj/v_proj on
the full-attention layers, where 8 bits removes 94% of their error for 16 MB
resident — the smallest sensible promotion, against +2.10 GB for the whole
attention slot. It was measured end to end twice and is closed: 18/20 against
18/20 on the twenty-prompt suite, and −0.009727 ± 0.006665 nats (t −1.46) on the
paired held-out perplexity A/B, with the whole-slot 8-bit build at −0.001249
nats (t −0.09). Both point estimates are small and in precision's favour, and
neither clears the instrument's ~0.013-nat floor. The converter policy and the
GDN a/b kernel's int8 branch are therefore not built. The runtime half —
per-tensor widths in the resident index — ships and is tested (3af08ee).
Do not revive without a machine that can hold the 125B bf16 source; the
effect is bounded below what this one can resolve.
These numbers are weight-space fidelity and selection stability: they bound how much the model could differ, not how much the answers do. The two TT-025 measurements below are the end-to-end answer, and they close the question at this machine's resolution.
TT-025's premise is that a few megabytes of precision-sensitive tensors are
worth promoting inside a 4-bit install. The weight-space case is measured
(tools/precision_plan_qwen35.py: k_proj/v_proj at 8 bits removes 94% of
their error for 16 MB), and the note in prepare_qwen35.py says plainly that
the 4B keeps the rule unmeasured end to end. So this measures it, with
benchmark/quant_quality_ab.py: twenty prompts whose answers a script checks —
arithmetic including multi-step, a counting task, a syllogism, a format
constraint, a reversal, a calendar step, a fact — greedy, identical prompts,
identical system message, one generation each.
| installs compared | what differs | first | second |
|---|---|---|---|
| Qwen 3.5 4B 4-bit vs 8-bit | ~2x the bytes, whole model | 18/20 | 17/20 |
Qwen 3.5 4B 4-bit vs a --no-promote 4-bit build |
the 16 MB k_proj/v_proj promotion, nothing else |
18/20 | 18/20 |
| Qwen 3.5 9B 4-bit vs 8-bit | ~2x the bytes, whole model | 18/20 | 18/20 |
A wash, everywhere. The two 4B configurations miss the same two cases and produce the same wrong answers — "if today is Wednesday, what day is it in 10 days?" → Tuesday, and "the sum of the first five prime numbers" → 17, which is the first four. Both are reasoning failures that precision does not touch. The 9B gets both right at either width, which is a capacity effect, not a precision one, and its two misses differ by configuration (4-bit fumbles "stressed" → "destrets", 8-bit fumbles Wednesday+10 → Monday) — noise rather than an advantage.
What this establishes. At this instrument's resolution, and for this family, there is no end-to-end quality case for a per-tensor promotion: the whole-slot 8-bit build doubles the bytes for nothing the check can see, so a ~10 MB subset of the same change will not show either. TT-025's mechanism ships and is tested; its policy has no measured justification, and the 125B rebuild that would test it there is not affordable on this machine (the bf16 source is 360 GB, and the install's 95.4 GiB table is the only part that could have been reused).
What it does not establish, and what the next section answers. Twenty short prompts are a coarse instrument and a small perplexity effect could sit below its floor. That is why the held-out perplexity A/B below was run — it is the sharper instrument, and it agrees: the promotion is 0.0097 ± 0.0067 nats ahead of its control, which is inside the ~0.013-nat floor rather than above it.
The controls are what make it readable: the promotion pair differs only in
those tensors — the shipped manifest carries k_proj/v_proj at 8 bits on the
eight full-attention layers, the control at 4 everywhere else — and the 4-bit
and 8-bit installs are the shipped ones, not rebuilds. The control takes five
minutes:
python3.13 tools/prepare_qwen35.py --size 4b --bits 4 --no-promote \
--output .build/qwen35-4b-uniform --work .build/qwen35-4b-shards
.build/release/TinyTitanRepack --input-snapshot .build/qwen35-4b-uniform \
--model-id qwen3.5-4b-uniform --output .build/qwen35-4b-uniform.gturbo
Both come from Qwen/Qwen3.5-4B at its pinned revision — the official repo.
The twenty-prompt check above is a floor: both 4B configurations scored 18/20
and missed the same two reasoning cases, so it cannot see a small perplexity
effect. This is the sharper instrument. TinyTitanBench cpu35ppl scores a
fixed text through the CPU forward pass — step returns the whole
vocabulary's logits for every position, so scoring needs no sampling and no
generation — and records one negative log-likelihood per token:
.build/release/TinyTitanBench cpu35ppl <install> <text> [tokens] [nll-out]
python3.13 benchmark/quant_perplexity_ab.py models/qwen3.5_4B_4Bit \
.build/qwen35-4b-uniform.gturbo models/qwen3.5_4B_8Bit --tokens 1024
Every install scores the same token positions (token hash
5d5dc2a7b319ad43 for all three), so the comparison is paired: mean dNLL, its
standard error, and t over the 1,023 positions. Text: the four repository
documents benchmark/quant_perplexity_ab.py names by default.
| install | mean nll | perplexity | dNLL vs the promoted 4-bit | t | ppl ratio |
|---|---|---|---|---|---|
4B 4-bit, k_proj/v_proj promoted |
2.666170 | 14.384774 | — | — | — |
4B 4-bit, --no-promote control |
2.675897 | 14.525374 | −0.009727 ± 0.006665 | −1.46 | 0.990320 |
| 4B 8-bit, whole slot | 2.667420 | 14.402758 | −0.001249 ± 0.014438 | −0.09 | 0.998751 |
A sharper instrument, the same answer. dNLL is baseline minus install, so a negative number means the promoted install has the lower NLL. Both point estimates are small and in precision's favour — the 16 MB promotion is 0.0097 nats (about 1% perplexity) ahead of its uniform control, and the whole-slot 8-bit build is 0.0012 nats ahead — but neither is distinguishable from zero at this length (|t| 1.46 and 0.09). The instrument resolves roughly 0.013 nats, so the promotion is bounded at about 1% perplexity rather than shown to buy it. That is what "the policy has no measured justification" means here: the effect is either absent or below a floor this machine can reach for a few minutes of CPU, and the 125B that would resolve it needs a 360 GB bf16 source.
What it does not establish. The paired t treats token positions as independent, which they are not, so treat |t| as indicative rather than exact. The text is English prose from this repository; a code-heavy or multilingual held-out set could move the estimate. And no downstream task — the book scenario's continuity quiz, a coding pass@k — was run, so this bounds weight-space quality, not task quality.
TT-025 closes here. The mechanism ships and is tested; the quality case is absent at both resolutions this machine can reach, and the 125B rebuild that would sharpen it further needs a 360 GB bf16 source there is no disk for. The converter policy and the GDN a/b kernel's int8 branch are not built, and this entry is the reason not to re-propose them.
T7 ranked better than the token match from the first measurement — recall@1 4
of 4 against 1 of 4 on the authored paraphrase set — and had no caller, because
every question-shaped caller is a request the person is waiting on and a
judgement is 15.2 s on the 4B. The inverse-document-frequency weight then took
the token ranking to recall@1 3 of 4 (878cea6), leaving exactly one miss no
weighting reaches: "How often does the boat cross the water?" against
rules/ferry = runs only on Sundays, which share no term in any form.
The caller is MemoryRetrievalHinter, and it is deliberately off the request
path. memory_search answers from the token ranking immediately and registers
its question; one background task walks the scope's facts while the server is
idle — ServerCoordinator.generating read as isIdle — and records each YES
as a ranking hint keyed by the fact plus a fingerprint of the value it judged.
A later search for the same question puts the hinted facts first, including a
fact the token ranking never returned, which is the semantic case. Nothing
awaits a judgement, so the 15.2 s stays off the turn.
The bounds are what make it safe to leave running. The sweep covers at most 64
facts a question (MemoryRetrievalHinter.coverageLimit, and never more than a
search itself returns), keeps hints for at most 16 questions (least recently
asked evicted), and a verdict carries the value's FNV-1a fingerprint, so a fact
whose value changed is not promoted on an answer about the old one. A hint is
filtered by the query's own prefix, tags and importance, so it cannot return a
fact the caller excluded. Without an engine, or while a client is generating,
the path is byte-for-byte the token ranking it was.
MemoryRetrievalTests pins the seven behaviours the design rests on:
never-waiting, promotion of a dropped fact, reordering within the returned set,
staleness by fingerprint, the query filters, the coverage bound, and a log line
that carries counts but never the question or a value. The real model's ranking
is unchanged and already measured above; what was missing was a caller, and
this is one.
The 2B is not used. The 4B 4-bit is the side-engine: the install
ServerSideEngineFactory loads by default, and the smallest that decides
contradiction, duplication, retrieval and — with the third durability draft —
durability on the shipped prompts. The 9B is optional and chosen on quality
benchmark results, not size for its own sake: it adds the reply check (T6) and
is worse at durability (65% against the 4B's 95%), so a deployment that wants
T6 takes it and one that wants durability does not. docs/side-engine-tasks.md
carries the matrix these follow from.
The report. A qwen38flash 4-bit install would not serve: unsupported architecture: qwen38flash declares model.language_model.layers.0.linear_attn.in_proj_a at 4 bits; the GDN a/b kernel reads that pair at the attention slot's width or as bf16, so a quantized override there is not honoured. The install verified
clean; only serving failed.
What the check got wrong. The rule in the message is right; the code
compared against 16 alone. Model.validateRoleUniformity refused every
quantized in_proj_a/b override, including one that names the attention
slot's own width — the width the kernel already reads. A manifest carrying such
an override was therefore refused before any weight was read. The shipped
install carries no override at all, which is why this checkout never hit it; the
reporter's manifest did, and any manifest built with explicit per-tensor widths
(naming even the widths that match the slot) would.
Reproduced and fixed on this checkout. Adding just that override to our
qwen3.8-flash-next_125B_A6B_4Bit manifest — with the receipt's manifestSha256
updated so the load is not refused for the edit — produced the reported error
verbatim. With the fix the same install answered Paris for "The capital of
France is" (8 tokens, 2.8 tok/s), and both files were restored byte-identically
afterwards. The check now takes the attention slot width and accepts that width
or bf16; RoleUniformityTests pins the slot-width case (the regression), both
mismatch directions, and bf16. The full package suite and all six lint gates
pass, and the qwen38-4 golden is identical to its baseline.
What to carry forward. The validator was written from the comment — "bf16 or the slot's width" — and implemented as "bf16 only": the parameter the rule needs was never passed in. A rule that names a slot has to be given that slot. And a manifest may name a width that matches the slot; equality is a description, not an override that needs honouring.
Nothing is parked. TT-025, the one idea that stood here, is closed by measurement: both the twenty-prompt floor and the paired perplexity check put its quality case below this machine's floor, so the converter policy and the GDN a/b int8 branch are not built. TT-033 closed when its caller landed above.
Three ideas were closed here earlier instead of revived. TT-026 (sharing
ngram_table.bin) and TT-027 (a streaming path for KAT-Coder and AgentWorld)
were already implemented when their rows were checked —
TinyTitanRepack --share-ngram-table with prepare_qwen38.py --reuse-ngram-table on one side, prepare_agentworld.py's three-way shard
fetcher on the other — and TT-024 (CPU co-execution) is closed by
measurement, in the entry above.
- MTP expert union — closed on both models: on Qwen3.8 a width-2 verify costs about 2.8x against a 2.0x ceiling, and 71% acceptance does not close that.
Start
Use TinyTitan
DeepSeek Harness
Reference
Engineering
Project