v0.3.0 — 10.50 tok/s, faster than the C reference on every stage

What's Changed

• measure the memory system; correct three claims; record two dead ends by @Bellman281 in #20
• build: 120 MHz MSPI as an opt-in overlay — 9.75 tok/s, one flag away by @Bellman281 in #21
• perf(head): 10.50 tok/s — i16 activations and two rows per pass by @Bellman281 in #22
• docs: 91.8 ms with --fast-mspi; language bar says Rust by @Bellman281 in #23

Full Changelog: v0.2.0...v0.3.0

v0.2.0 — faster than C, verified

A 28.9M-parameter language model on an $8 microcontroller, in Rust — now faster than the C implementation it was ported from.

v0.1.0 marked the port running correctly on hardware. This one marks it running faster than the thing it was ported from: 9.41 tok/s on an ESP32-S3-DevKitC, 14.8% faster per token than the C reference, with the board itself proving the output is identical.

Twelve measured steps since v0.1.0, 3.21 → 9.41 tok/s.

Verified, not asserted

Every performance claim below is reported alongside "byte-identical output". As of this release that phrase is checked by the hardware:

token digest:   0x327578cb136fd6aa (418 tokens) OK -- matches models.toml gen_digest

The firmware folds every token id it emits into an FNV-1a 64 digest and prints it. model/models.toml pins the expected value; llm-host/tests/token_stream_digest.rs recomputes it on the host through the same output-head arithmetic the board uses; benchmark_settings_match.rs stops the firmware's copy of the constant from drifting from the registry. A run that diverges by one token says so, in the same output as the timings.

That matters more than it sounds: a firmware that is fast because it computes something else is not a result.

Measured on device

ESP32-S3-DevKitC N16R8, 400 tokens / 18-token prompt. Both engines gated to the same settings by benchmark_settings_match.rs, because attn is the only stage that scales with sequence position and mismatched settings have produced a wrong conclusion in this repo before.

ms/token	input	attn	ffn	ple	head	stages	wall	tok/s
C reference	4.4	42.9	6.9	8.5	57.1	119.8	122.0	8.20
Rust, first run	10.6	66.7	15.8	20.4	193.8	307.3	311.5	3.21
+ dual-core head, unroll	8.0	58.9	11.8	15.1	110.9	204.7	208.8	4.79
+ wider caches	7.4	53.7	11.2	14.3	87.6	174.2	177.3	5.64
Rust, before	7.8	54.4	11.9	15.2	94.6	183.9	187.2	5.34
Rust, nibble LUT	6.6	50.8	10.1	12.7	98.0	178.2	181.6	5.51
Rust, + KV layout	6.6	47.0	10.1	12.7	94.8	171.2	174.6	5.73
+ int4 head	6.6	47.0	10.1	12.7	94.8	171.2	174.6	5.73
+ 4 accumulators	6.6	47.0	10.1	12.7	81.3	157.7	161.0	6.21
+ bias hoist	6.6	47.0	10.1	12.7	61.5	137.9	140.9	7.09
+ dual-core matvec	3.9	40.2	6.2	7.5	61.6	119.4	122.7	8.15
+ argmax/stdout	3.8	39.8	6.0	7.3	61.5	118.4	121.1	8.25
+ argmax in head	3.9	40.0	6.1	7.5	61.5	119.0	119.0	8.40
+ attn instrumented	3.9	39.0	6.2	7.6	61.9	118.6	118.5	8.43
+ attn heads split	3.9	26.4	6.2	7.6	62.7	106.8	106.8	9.36
Rust, now	3.9	26.3	6.2	7.6	62.3	106.3	106.3	9.41
ratio vs C reference	0.89x	0.61x	0.90x	0.89x	1.09x	0.89x	0.87x
absolute gap	-0.5	-16.6	-0.7	-0.9	+5.2	-13.5	-15.7
change this brought	+0.0	-0.1	+0.0	+0.0	-0.4	-0.5	-0.5

attn broken down — the sub-stages the five-stage profile hides. qkv/proj are the fp32 matvec that also drives ffn and ple; core is attention proper, the only part that grows with sequence position. The C reference has no equivalent instrumentation, so it is absent rather than zero.

attn ms/token	qkv	rope	core	proj	sum	both cores
+ attn instrumented	7.8	0.14	28.3	2.7	38.9	qkv, proj
+ attn heads split	7.9	0.15	15.6	2.7	26.3	qkv, proj, core
Rust, now	7.8	0.15	15.6	2.7	26.2	qkv, proj, core

Rust is 14.8% faster per token than the C reference it was ported from — 106.3 ms against 122.0, 9.41 tok/s against 8.20, on the same board with byte-identical output. It beats C on 4 of the 5 stages.

Generated from benchmarks/device.toml by scripts/plot_stages.py — the table, the chart and the sentence above all come from one file, so no figure here was typed by hand.

How it got there

Twelve measured steps, 3.21 → 9.41 tok/s, every one bit-exact. No tolerance was spent: the parity suite holds these paths to exact agreement, not to a threshold.

The changes that moved it most, in order:

1. Bias hoist in the int4 dot (−19.8 ms). Codes are stored biased, and Σ(cⱼ−8)aⱼ = Σcⱼaⱼ − 8Σaⱼ, where Σaⱼ is the same for all 25,353 rows. One operation per element deleted, across 2.43M elements per token — plus the lookup table that had been standing in for it.
2. Every matvec on both cores (−18.6 ms). The output head had been the only stage using both LX7s; 55% of a token ran on one with the worker task blocked. llm_forward_with_matvec_override had shipped since Phase 3 with no user.
3. Attention's heads on both cores (−12.7 ms). Attention is not a matvec, so the hook above could not reach it. core was 24% of a token on one core.
4. Four accumulators in the head dot (−9.7 ms). A single accumulator makes every add wait on the previous one.
5. Nibble→f32 lookup table and pair-unrolled unpack (−9.1 ms across four stages).
6. Argmax where the logits are made (−2.7 ms). The greedy pick had been reading 25,353 floats back out of PSRAM with a data-dependent branch.

Full reasoning for each, including the ones that did nothing: BENCHMARKING.md.

Three things this project learned the hard way

The head was not bandwidth-bound, and this repo said it was. Acting on that produced an int4-staged head that halved what the stage streams and moved it 0.0 ms. Instrumenting it took one run to answer: 13.4 MB/s used against 72 MB/s available, with both cores independently hitting the same time. It was compute-bound throughout. The null result is what forced the instrument, and the instrument was worth more than any guess about the stage.

The host is an oracle for arithmetic and useless for memory shape. It predicted the nibble table's device result well. It predicted the KV-layout change would be slightly worse; the device gave −7.5% on the stage.

This board has no run-to-run noise, which is less comfortable than it sounds. A given binary reproduces to 0.01 s wall and 0.1 ms per stage — so a change in a stage you did not touch is real. The output head has been measured between 61.5 and 98.0 ms across binaries that do not change a line it executes: instruction and .rodata placement, worth several percent. The first reading of that was thermal drift. One repeat run killed it.

Correctness

cargo test --release                              35 passed, 0 failed
cargo test --release --features int8-activations  32 passed, 0 failed

• golden.rs — logits against a PyTorch reference (max diff 5.5e-7).
• cli_parity.rs — byte-for-byte against the compiled C binary's stdout.
• ppl_parity.rs — exact fp32 cross-entropy over real validation tokens.
• matvec_split.rs / attn_split.rs — every row and head split point, bit-for-bit, so a device mismatch would have to be the FreeRTOS plumbing rather than the arithmetic.
• token_stream_digest.rs — the on-device token stream.
• benchmark_settings_match.rs — both firmwares against the registry.

Two real bugs were caught this way during the port: an out-of-bounds scratch buffer for one model config, and a rounding-mode mismatch against C's lrintf.

Known gaps

• The output head is the one stage still behind C, at 1.09x and 59% of a token. It runs ~12.4 cycles per multiply-accumulate against C's ~11.3 — the remaining gap is scheduling, which means the ESP32-S3's PIE vector unit rather than another algebraic identity.
• 1.35 ms of the attention split did not come back. A perfect head split predicts 14.25 ms including sync; the board gives 15.6. Either both cores now contend for the KV cache, or it is the layout tax — the same binary's boot probe reads 68 MB/s against the previous one's 72. The experiment that separates them is running the bandwidth probe on both cores at once. Recorded in device.toml rather than rounded away.
• The digest ties the board to this engine's host path, not directly to the C firmware. No capture of C's own 400-token run exists in the repo. Closing that means capturing one, or teaching the C firmware the same digest.
• stdin prompts do not work on hardware. The firmware uses a baked-in prompt, as the C reference does. See read_prompt_ids.
• Phases 4–6 (display, bandwidth bench port, dropping FreeRTOS for esp-hal) are not started.

Credits

Ported from slvDev/esp32-ai's C implementation, which remains vendored in reference-c/ as the correctness reference. Also built on llama2.c, TinyStories (arXiv:2305.07759), and Google's Gemma 3n Per-Layer Embeddings.