Benchmarks

Benchmarks

TinyTitan performance depends on the Mac, available memory, SSD behavior, prompt length, generated tokens, quantization, and cache state. Treat these as measured examples, not hardware limits.

One prompt, every served model

A different question from throughput: what each install answers, and what the engine choice costs, measured on one deliberately ambiguous prompt with thinking off, on both engines where an install has two. Every reply is quoted verbatim, and the page records which installs think even with the switch off. See One Prompt, Every Model.

5.0 continuous-generation benchmark

On 2026-08-30, a base 8-core M3 MacBook Pro with 24 GB ran every shipped model at its shipped defaults -- nothing pinned for the benchmark that a user would not get. Same workload as v4.1 and v4.6: 512 tokens of continuous plain English prose, temperature 0.6, Top-P 0.95, Top-K 20, native 262K context, multi-prefix prompt cache on, 8-bit KV, MTP off. One discarded warmup, then measured fresh processes.

Model	Quantization	Median decode	Runs	Spread
Qwen3.8-Flash-Next 125B-A6B	4-bit	6.82 tok/s	3	2.2%
Ornith 1.5 35B-A3B	4-bit	22.65 tok/s	7	0.6%
Ornith 1.5 35B-A3B	8-bit	11.89 tok/s	3	1.3%
Qwen 3.6 35B-A3B	4-bit	23.28 tok/s	7	1.8%
Qwen 3.6 35B-A3B	8-bit	12.72 tok/s	3	0.9%

Three configurations moved with the per-family expert-cache sizing and depth-1 speculative expert prefetch that shipped in 5.0 — Qwen3.8-Flash-Next 4-bit, Ornith 8-bit and Qwen 3.6 8-bit. The two 4-bit 35B configurations were not changed by that work and reproduce their published numbers to within 0.5%, which is what makes the other three readable as gains rather than drift.

The percentages this paragraph carried were withdrawn on 2026-09-18, because their baselines could not be re-derived: two of the three figures appear nowhere else in the repository, the v4.6 table below gives Ornith 8-bit as 9.40 rather than the quoted 10.17, and Qwen3.8-Flash-Next did not exist at v4.6 at all. Re-measure before quoting a gain.

Why these exceed the A/B that motivated them

The same changes measured +21.3%, +5.9% and +5.5% in an interleaved off/on/on/off A/B. The interleaved figures are the smaller ones because interleaving keeps every configuration page-cache-warm, and warming flatters the baseline far more than the tuned configuration -- the tuned one barely reaches the disk. Measured cold, the way a server actually starts, the gains are two to three times what the interleaved test credited.

Interleaving is still the right tool for deciding between configurations, because it removes the ordering advantage. It is the wrong tool for publishing an absolute, for the same reason.

A note on run counts

The 4-bit 35B rows use seven runs rather than three. At three runs, Qwen 3.6 4-bit returned 26.35 / 23.32 / 26.32 tok/s, and a median taken there would have published a 13% gain for a configuration that had not changed. Seven clean runs give 23.278 with 1.8% spread and no run above 23.35; the two fast readings were transient. busy_per_token was flat across all of them, so the GPU work never varied -- only stall time did. Where a delta approaches the within-configuration spread, take more runs before believing it.

v4.6 continuous-generation benchmark

On 2026-08-22, a base 8-core M3 MacBook Pro with 24 GB ran Ornith 1.5 35B-A3B using the v4.6 runtime (tiled Top-K sampling; decode scheduling unchanged from v4.1). The fixed prompt requested continuous plain English about an ordinary day in a small town. Each quantization used one discarded warmup and three measured fresh processes, generating 512 tokens per run:

Quantization	Decode runs	Median decode	Median wall time	Change from v4.1
4-bit	22.415 / 22.530 / 22.543 tok/s	22.53 tok/s	25.59 s	+37.0%
8-bit	9.373 / 9.399 / 9.506 tok/s	9.40 tok/s	59.48 s	+7.4%

The settings match the v4.1 protocol: temperature 0.6, Top-P 0.95, Top-K 20, presence penalty 0.0, native 262K context, multi-prefix prompt cache on, 8-bit KV, an 8 GiB expert-cache budget, MTP off, Concise off, Thinking off, and the base model alias. The gain over v4.1 is the sampler path; sampled output at a fixed seed is unchanged token-for-token, and TINYTITAN_SAMPLER_PATH=generic reproduces the old speed for comparison.

v4.6 experimental ANE prefill

With the opt-in sidecar (TINYTITAN_PREFILL_ANE=on, exported once per model with tools/export_ane_prefill.py), an interleaved gpu/ane/ane/gpu A/B on a 6,103-token prompt, greedy, prompt cache off, fresh server per run:

Path	Prefill median	Runs	Decode after prefill
GPU	132.90 s	132.85 / 132.95	8.70 tok/s
ANE	57.52 s	57.58 / 57.46	8.68 tok/s

Prefill improved 2.31x with decode unchanged. The ANE path's fp16 attention is not byte-identical to the GPU path; each arm's greedy output was internally deterministic across its runs. It was opt-in when measured and has since become the default (TINYTITAN_PREFILL_ANE=off opts out), falling back to the GPU when a model has no exported sidecar.

Steady state, past the handover transient (2026-09-18). The decode column above is a ~60-token window — the pinned prompt hits end-of-turn early — so it contains the one-time handover re-warm. A long-generation re-measurement on AgentWorld 35B-A3B 4-bit (11,948-token prompt, greedy, three ANE chunks; benchmark/ane_steady_state_decode.py) differences two generation lengths, 64 and 448 tokens, so any one-time cost cancels:

Path	Prefill median	Steady-state decode	Window ≤64 tok	Window ≤448 tok
GPU	622.5 s	7.21 tok/s	6.66 tok/s	7.17 tok/s
ANE	187.7 s	7.26 tok/s	6.79 tok/s	7.19 tok/s

ANE prefill costs +0.7% of steady-state decode — inside the run-to-run spread, i.e. nothing — and even the 64-token windows show no penalty. Two runs per arm and length. The first ANE prefill of a process paid a 2.1x compile warm-up (390.4 s against 185–188 s afterwards), so a single ANE prefill figure taken from a cold process overstates it.

v4.1 continuous-generation benchmark

On 2026-08-21, the same base 8-core M3 MacBook Pro with 24 GB ran Ornith 1.5 35B-A3B using the v4.1 hit/fixup runtime with the same prompt and protocol. This is the baseline the v4.6 table compares against:

Quantization	Decode runs	Median decode	Median wall time	Change from v4.0
4-bit	16.372 / 16.454 / 16.471 tok/s	16.45 tok/s	34.24 s	+7.5%
8-bit	8.738 / 8.746 / 8.799 tok/s	8.75 tok/s	63.99 s	+5.2%

The settings were identical to the v4.6 run above. All six measured responses were byte-identical to their matching v4.0 responses.

Ornith native MTP qualification

On 2026-08-20, the same base M3/24 GB machine qualified Ornith's official one-layer MTP draft against commit 982ceb7. Both modes used greedy decoding, prompt cache off, native RoPE, 8-bit KV, a 4-bit MTP sidecar, and the same bounded SSD expert path. Higher is better:

Target	Scenario	Target only	Native MTP	Acceptance	Result
4-bit	Short Fibonacci code	8.425 tok/s	6.616 tok/s	67.1%	-21.5%
4-bit	Medium AsyncLRU code	10.326 tok/s	7.480 tok/s	65.2%	-27.6%
4-bit	16,314-token code review	5.679 tok/s	no response by 900 s	n/a	failed
8-bit	Predictable 53-token function	6.416 tok/s	6.288 tok/s	92.6%	-2.0%

The 8-bit row is the strongest acceptance case — a three-repetition median with byte-identical, complete MTP and target-only answers. The 4-bit short and medium rows were single qualification probes, and the long MTP request exceeded the fixed 900-second timeout and was recorded as failed rather than replaced. No tested scenario produced a repeatable benefit on this M3, so MTP remains opt-in and experimental. It may become worthwhile on different hardware or longer, highly predictable output, but that is not established by these measurements.

Ornith four-program coding/tooling matrix

On 2026-08-21, the M3/24 GB machine ran eight fresh-server cells across Ornith 4-bit/8-bit, Concise off/on, and Thinking off/on. Each cell repaired and ran independent Swift, Python 3.14, two-layer TensorFlow LSTM, and PyTorch attention programs. Every program received a fresh conversation with up to 25 assistant turns; separate hidden inputs decided the chained final result.

Weights	Concise	Thinking	Hidden result	Wall time
4-bit	Off	Off	Pass	14m 17s
4-bit	Off	On	Fail	17m 46s
4-bit	On	Off	Fail	11m 02s
4-bit	On	On	Fail	13m 13s
8-bit	Off	Off	Pass	13m 37s
8-bit	Off	On	Pass	29m 25s
8-bit	On	Off	Pass	18m 27s
8-bit	On	On	Pass	23m 22s

Five of eight cells passed all four hidden checks. All three failed cells were 4-bit attention failures. The fastest correct configuration was 8-bit, Concise off, Thinking off, which is therefore the default. This is one stochastic run per cell, not a quality proof or performance median; the public and hidden checks make it a stronger default-selection signal than output rate alone.

Fast alias latency

The same M3 asked “What is the capital of France?” through three coding CLIs. The -fast alias uses the same weights but removes the clients' agent scaffolding before prefill.

Client	Base 4-bit	Fast 4-bit	Fast 8-bit
Codex	about 250 s	24.6 s	32.8 s
Qwen Code	over 300 s, stalled	4.6 s	12.8 s
OpenCode	about 218 s	5.9 s	10.2 s

The fast alias is for direct questions. It strips tool definitions and cannot perform an agent tool loop; use the base model for file and command work.

Compare results correctly

• Compare only the same model, quantization, prompt, generated-token count, settings, cache state, and stop reason.
• Report decode separately from prefill and total wall time.
• Reject repeated, incomplete, or corrupted output.
• Record the commit, hardware, RAM, macOS, Swift version, exact command, and complete timing footer.

Use the Benchmarking Guide to collect a new result.