How We Measured It

How We Measured It

Almost every hard number in this wiki came from an instrument built for the purpose. This page is the method, because the method is more reusable than the numbers.

Build the instrument before iterating

Guessing at a fix and re-running is slower than it feels, and it produces conclusions that cannot be checked later. Where a fact was needed, something was built to report it:

Instrument	Answers
visualizer --probe	Is capture reaching real audio, and if not, which kind of nothing is it seeing?
visualizer --selftest	Is the analysis chain correct, on signal whose answer is known beforehand?
visualizer --sweep	Does every mode hold its frame target?
bench	What is the panel's actual upload ceiling?
calibrate	Where does a touch at a known point actually land?
preview	What does a screen look like, without owning the device?
--selftest (editor)	Does every view still build, and survive being revisited?

A check needs a control in both directions

A test that only ever passes proves nothing. Every meaningful check here was run against a case that should fail.

The negative control caught the biggest bug in the project. Before any audio code existed, two captures were taken: one from a RUNNING monitor with music playing, one from a SUSPENDED monitor. The result came back backwards — the running one gave pure zeros, the suspended one gave −52.9 dBFS of noise. That inversion is what exposed the pw-record targeting failure. Had only the "should work" arm been run, zeros would have passed for a quiet passage.

The positive control caught a broken exit code. The DSP self-test passes on correct code; to check it could fail, the bin mapping was deliberately broken by an octave. Every tone case then failed by 3.1–3.6 bands — an octave is 3.58 bands at that spacing, so the instrument failed by precisely the right amount. Running that control also revealed that the command printed FAILED and exited 0, which made it useless to any script. (Top-level statements with a trailing return 0 compile to Task<int> Main, and that final return overwrites Environment.ExitCode.)

Prediction is not measurement

Plans here mark predicted numbers as predicted, and replace them with measured ones when they arrive. An un-run cause reads as a finding a week later, and a predicted latency budget presented as a measured one is simply false.

Where a number cannot be measured, say which part is measured and which is reported by something else — the latency figures in Music Visualizers name exactly what they exclude rather than summing to a tidy total.

An instrument that fires on correct work is worse than none

A dependency check in the packaging test reported seven perfectly resolvable libraries as missing, while the same package launched and ran four different ways. The cause was ldconfig -p | grep -q under set -o pipefail: grep -q exits on the first match, ldconfig then writes into a closed pipe, takes SIGPIPE and returns 141, and pipefail surfaces that as failure.

It was intermittent, which is worse than broken. Snapshotting the cache to a file once and matching against that fixed it — and the matcher was then checked against a library that genuinely does not exist, to be sure it could still fail.

Delete or fix such a check. Never silence it. A check people learn to ignore is a check that will be ignored on the day it is right.

Verify the product, not the step

Every signal in a build pipeline reports on the step. "Build green", "package created", "install added N files" are all perfectly consistent with an application that cannot start.

On a sibling project a packaging step that copied files by an allow-list of names dropped all 218 managed assemblies. makepkg exited 0, the package manager installed 54 files without a complaint, and the application died instantly.

So packages here are verified by asserting an inventory against the tree they were built from, and then launching the binaries out of the extracted package with the development toolchain's environment scrubbed. Copy payloads wholesale; never by an allow-list of names.

Believe the user's eyes

Colour and legibility on this panel are not things a program can check. The 18-bit truncation and the yellow-shifted green mean a rendering that is correct in memory can still be wrong on the glass.

One mode was reported as "hard to tell what it was supposed to be doing" at the same time the per-mode timing sweep flagged it as the only mode missing its frame target. Two independent signals, one cause — but the aggregate number alone would never have named it, and the visual report alone would not have found the SetPixel cost behind it. Both were needed.