TiinyBench 0.1.8

Four more kinds of model can be measured, the report is rebuilt, and every run
now records what makes it comparable to somebody else's box.

Every run carries its provenance

A run taken without this is a run that cannot be compared tomorrow. Each file
now has a provenance block with a schema number on it.

Per run: the host that drove the benchmark (os, release, version, arch,
python, logical CPUs, physical RAM, machine model), the transport (plane,
gateway port, vhost if used, and the round trip measured before any work
starts), the device (name, model, serial, TiinyOS, service version, NPU
unit budget), and the tool (bench version, suite version, envelope schema,
git commit). A benchmark driven from a Mac over a cable and one driven from a
Windows box over Wi-Fi are not the same measurement, and until now nothing in
the file said which.

Per model rather than per run, because a sweep loads and unloads as it goes
and the conditions the fifth model met are not the ones the first met: the
resident set with each model's NPU units at the moment that model's tests
begin, units used and free, whether another process held the device lock, the
sampling parameters actually sent, which tests ran and which were skipped, and
the served context window. The lock is read and never taken, because a
benchmark that fought for it would change the thing it measures.

Temperature and power are recorded as null with a reason rather than omitted.
A missing key reads as an oversight and a zero reads as a measurement, and
neither is true of something nobody could measure. The machine model on macOS
is the same: reading it means starting a program, which this app may not do.

public_view is the one function that makes a record shareable, so an upload
path and a report cannot disagree about what is safe. It hashes the serial so
runs group by box without naming one, drops the device address, the vhost and
the owner's chosen name for the box, drops any string that looks like a path
under a home directory, and adds an account attribution. Every measured number
survives untouched: it makes a record publishable, not smaller.

All additive. Files written before any of this still load, render, export and
publish.

Four new test types

Twenty-nine models are installed across nine classes and the suite had tests
for five of them.

Class	Test	The figure
ASR	asr	seconds of audio transcribed per second of wall clock
Image-to-Text	ocr	seconds per page, and whether the digits came back
Music Generation	music	seconds of audio produced per second of wall clock
Text Reranking	rerank	query-document pairs scored per second

Their fixtures are generated in code, not committed: a WAV whose header states
its duration exactly, because the real-time factor divides by it, and a PNG of
seven-segment digits, because a font table is three hundred lines nobody can
check by eye. Both are byte for byte identical every run.

Three copies of one list, now one each

The set of measurable model classes was written down three times and all three
had drifted. CHAT_TYPES was a second hand-written copy, which is the actual
reason four classes had no test. The report's leaderboard named the three
result blocks it knew how to read, so a class added later scored blank. And
the run page kept the narrowest copy of all, greying out every model that was
not a chat model, which was already wrong for the image, speech and embedding
tests that already existed.

All three read the registry now. The run page also plans per model, so a
Text-to-Image model picked beside a chat model runs its own test rather than
refusing "thinking", and the time estimate counts tests that will really run.

The report, rebuilt

Exports: HTML hands over the existing self-contained file rather than
generating a second one that could disagree with it; Markdown renders server
side off the same loader; PDF is the browser's own print through a stylesheet,
because a PDF library is a dependency this app will not take.

The print path was tested rather than assumed, by emulating print media and
then producing the PDF. That testing caught three real defects that would
otherwise have shipped: a stranded near-empty page, a grey block where an
empty grid cell printed, and a bordered section stretched down a mostly empty
sheet.

Per model: a dual axis prefill chart with time to first token solid on the
left and decode rate dashed on the right and every point labelled on both, the
same points as a table, concurrency as grouped bars with wall time under each
level, reasoning as two horizontal bars carrying wall time and token count, a
device telemetry strip, and a caption under every chart computed from that
model's own numbers. A box whose aggregate is flat gets a caption that says it
is queueing, and the words come out of the ratio rather than out of a template.

A cross-model section with a stable colour per model taken from a hash of the
id, so adding or dropping a model never repaints the others: sustained decode,
throughput per NPU unit, every prefill curve on one axis, and aggregate
throughput against stream count. Newest run per model, never best, because a
better old run would flatter the box.

Long model names are unreadable under vertical bars, so those two charts are
horizontal. The name gutter is measured from the widest label in the set: it
was capped, which fitted every name but the longest, and that one ran under
the start of its own bar.

The version string

v0.1.5 and v0.1.6 were both tagged and both shipped VERSION = "0.1.4", so
the app printed 0.1.4 in its banner, in --version and in the selfcheck, and
stamped bench_version: 0.1.4 into every result file it wrote. Anyone reading
a result file written before 19 September should treat that field as
unreliable; the measurements themselves are unaffected. A test now holds
bench.py and tiiny-app.json to the same number.

Two things learned about the device

/openapi.json stopped being per-service in TiinyOS 1.0.0: the AI Gateway
moved behind port 80 and that path is served by the front proxy, so every
vhost spelling returns the same device management spec. Confirm a route
without running it by asking with GET, where a POST-only route that exists
answers 405 and one that does not answers 404.

A route whose class of model is not loaded answers 503 with an envelope
nothing else on the device uses. Code that reads it as a transport failure
reports a missing model as a broken box.

173 tests, offline, no device needed.