amy-bench: on-target A/B performance benchmark (ESP32-S3)

Measures what an AMY DSP change actually costs on hardware. The harness compiles an AMY src/ tree in place as an ESP-IDF component, runs deterministic synth scenes headless, and reports per-block wall time, CPU cycles, and output CRCs over serial as JSONL. abrun.py drives the whole A/B: two git refs in, one report out.

This repo is the harness, not a copy of AMY. It measures an AMY checkout you point it at, which is what lets a run compare any two refs that checkout can reach - your branch against upstream/main, two upstream tags, or a dirty working tree against its own merge-base. The only thing AMY itself needs is two documented #ifndef config guards, and abrun.py applies those to its own throwaway scratch tree, so upstream refs work untouched. See AMY-EDITS.md.

flowchart LR
    subgraph amy["AMY checkout (--amy-repo)"]
        src["src/*.c *.h<br/>any two refs"]
    end
    subgraph harness["amy-bench (this repo)"]
        comp["esp32s3/components/amy<br/>compiles a src/ tree in place"]
        mainc["esp32s3/main/<br/>scenes, metrics, JSONL emitter"]
        tools["tools/abrun.py<br/>capture.py, abcompare.py"]
    end
    src -->|"git archive<br/>one scratch tree per side"| comp
    comp --> fw["bench firmware, one per side"]
    mainc --> fw
    fw -->|"A into ota_0<br/>B into ota_1"| s3["ESP32-S3 board"]
    s3 -->|"JSONL over serial<br/>A B A B ..."| logs["repeated captures"]
    logs --> tools
    tools --> report["compare.json<br/>per-scene deltas vs measured noise, CRC diff"]
    report --> md["tools/abreport.py<br/>summary.md: the table you read"]

Loading

Prerequisites

• An AMY checkout. git clone https://github.com/shorepine/amy. Point the bench at it with --amy-repo, or $AMY_REPO, or by cloning it as a sibling ../amy next to this repo, which is the default.
• ESP-IDF 6.0, target esp32s3. Source its export.sh before anything else.
• An ESP32-S3 with 16 MB flash and octal PSRAM (matching the production S3-Amysynth board), on a serial port.
• pyserial, in the interpreter that runs abrun.py. It ships in the IDF python env, and sourcing export.sh puts that env on PATH, so a plain python3 after sourcing already has it. abrun.py checks for it up front rather than after spending minutes on builds.
• On WSL the board must be attached to the VM first (usbipd attach from an admin PowerShell), or no /dev/ttyACM* will exist.

Two app slots

The partition table carries ota_0 and ota_1 rather than one factory app, so abrun.py keeps both firmwares on the board at once and alternates with a boot-slot switch and a reset (about a second) instead of a ~20s reflash. Cheap repeats are the point: the noise estimate needs samples. Interleaving A B A B rather than A A B B keeps board drift from masquerading as the change under test.

Side A is flashed with the bootloader and partition table into ota_0; side B is written straight to ota_1's offset with esptool. (Not otatool.py write_ota_partition: in ESP-IDF 6.0 that entry point is broken - it dispatches input to a function whose parameter is named input_file - so it dies with a TypeError.)

Quick start

One command builds both sides, flashes them into the two app slots, alternates between them, and reports:

cd esp32s3
source $IDF_PATH/export.sh
python ../tools/abrun.py --port /dev/ttyACM0 --amy-repo ../../amy \
                         --base upstream/main --head exp/faster-filter

With a sibling ../amy clone, --amy-repo can be dropped:

python ../tools/abrun.py --port /dev/ttyACM0 --head exp/faster-filter

Logs, compare.json and summary.md land in tools/abrun-out/ (--outdir to change). summary.md is the table to read; compare.json is the evidence behind it (every sample, every CRC), kept so an archived run can be re-reported later without the board.

abrun.py takes the harness from this repo for both sides and swaps only src/, so the two firmwares are measured with the same ruler no matter how far apart the two AMY refs are.

Every side is a snapshot, taken into a scratch dir: committed refs with git archive, the working tree with a copy. The AMY checkout is only ever read, so a run cannot disturb it, and it cannot move underneath a run either - editing src/ while a build is in flight can no longer change what is being measured. Because --head defaults to the working tree, uncommitted work is measurable without committing, stashing, or branching first.

The guards in AMY-EDITS.md go into those snapshots, so AMY never needs patching. Upstream carries neither guard and merge bases are old, so most sides arrive without them; abrun.py wraps the two #defines in the throwaway tree instead. They change no instructions unless a define is injected, and the same definitions go to both sides, so they cannot bias the comparison. Without them a side would silently build at 44100/fixed-point and compare, looking perfectly healthy, against a 48000/float one.

What you can compare

A side is any ref the AMY checkout can resolve, so the shape of a run is just the pair you name. --head is what you are testing, --base is what it has to beat.

A branch against where it left main. The default: omit --base and it resolves to the merge base of --head and main, which isolates the branch's own effect from anything that landed on main since it forked.

python ../tools/abrun.py --port /dev/ttyACM0 --head exp/faster-filter --repeat 5

Your working tree, uncommitted. The default --head. Edit src/ in the AMY checkout, run, read the number - no commit, no stash, no branch. The tree is snapshotted into the scratch dir like any other side, so it is only read, and it needs no preparation: untracked files come along, and the AMY-EDITS guards are applied to the snapshot rather than to your checkout.

python ../tools/abrun.py --port /dev/ttyACM0 --repeat 5          # worktree vs merge-base
python ../tools/abrun.py --port /dev/ttyACM0 --base my-branch    # worktree vs its own branch

That last one is the tight loop: it answers "did the edit I just made help?" while the edit is still in the buffer.

Two branches, neither of them yours. Nothing privileges main or your own work. Any two refs the checkout can reach will do, which is what makes the harness usable on someone else's PR or across upstream releases.

python ../tools/abrun.py --port /dev/ttyACM0 --base upstream/main --head pr-1234
python ../tools/abrun.py --port /dev/ttyACM0 --base 1.2.52 --head 1.2.58

The same ref twice. A null run: the delta is known to be zero, so whatever it reports is the instrument's own noise. See Measured noise floor.

python ../tools/abrun.py --port /dev/ttyACM0 --base HEAD --head HEAD --repeat 5

Whatever the pair, two things hold. The harness comes from this repo for both sides and only src/ is swapped, so the two firmwares are measured with the same ruler however far apart the refs are. And attribution is only as clean as the branch: a branch carrying one change tells you what that change cost, a branch carrying five tells you what five changes cost together. If you want the numbers to name a cause, keep the branch to one.

Then read the verdict (below); if a scene moved, ask whether its CRC moved too. To see where the time went, rerun the same pair with --profile for a per-tag breakdown - keeping those captures separate, since the instrumentation adds a timestamp read per profiled call and inflates the absolute numbers.

Reading the result

A delta is meaningless until you know how much the measurement wanders on its own, so abcompare.py reports both, per scene:

scene         A med_cyc  B med_cyc     d_cyc    noise verdict
dx76             850121     765283    -9.98%   ±0.01% IMPROVEMENT
juno6           1159320    1159402    +0.01%   ±0.01% within noise
saw_lpf6         442452     442450    -0.00%   ±0.01% within noise

noise is the boot-to-boot spread of that side's own repeated measurements of the same firmware - what the number does when nothing changed.

summary.md

That dump is the evidence; the answer is the table abreport.py folds it into, which abrun.py writes to summary.md and echoes when the run finishes:

Scene	A cyc	B cyc	delta	noise	headroom	verdict
juno6	1,159,437	1,128,461	-2.67%	±0.00%	9.4% -> 11.9%	IMPROVEMENT
dx76	765,568	672,689	-12.13%	±0.00%	40.2% -> 47.5%	IMPROVEMENT
idle	34,967	29,072	-16.86%	±0.00%	97.3% -> 97.7%	IMPROVEMENT

Each delta sits next to the noise that qualifies it and the headroom transition that gives it consequence, and rows are ordered tightest scene first - the one closest to overrunning the block budget is the one whose delta you need to read, whatever the others did. A closing count says how many scenes improved, how many regressed, and how many are left under the headroom floor (--headroom-floor, default 20%), because a win that leaves a scene at 11% headroom is still a scene one regression away from glitching.

It recomputes nothing - every number and every verdict is abcompare's - so it can be re-run over an archived compare.json long after the board is gone:

python ../tools/abreport.py abrun-out/compare.json -o RESULTS.md

Given several reports it emits the attribution matrix instead: scenes down the side, one delta column per run. That is how a stack of A/B runs sharing a baseline separates one change's effect from another's - if run 1's delta plus run 2's composes to run 3's, the two changes are independent and additive.

python ../tools/abreport.py abrun-out/*/compare.json

A worked example: five runs over the ESP32-S3 PIE port, two of them measuring the same change against different baselines, showed the filters.c half of that port winning on nothing and costing saw_lpf6 ~0.9% - so the shipping arm keeps only the block clears and copies. The matrix is what made that visible.

verdict	meaning
within noise	the delta does not clear the scene's own spread. Not a result, whatever its sign.
small	clears the noise but is below --threshold. Real, but minor.
REGRESSION / IMPROVEMENT	clears the noise and the threshold.
1 capture	judgement withheld: one capture per side cannot estimate noise (see below).

A verdict needs at least two captures per side (--repeat 2; default 3). With one, the only spread available is between passes of a single boot, which can see neither reboot nor relink effects and therefore understates the real noise - so abcompare withholds judgement rather than trust it.

The unit of measurement is the median across one boot's passes, not the individual passes. Passes are not interchangeable samples: some scenes cost systematically more on a particular pass (saw_lpf6's pass 1 runs ~4.6% hot, reproducibly, on every boot and on both sides of an A/B). That is a fixed property of the scene which cancels in a comparison; pooling it into the noise estimate would inflate it ~500x and hide every real regression behind it.

The output column diffs the rendered audio's CRC, pass by pass. A change there means the DSP change altered the audio: expected for a real algorithm change, a bug for a supposedly-pure optimization. Do not accept "faster" without deciding which of the two it is.

Baseline

What the scenes cost on a stock ESP32-S3 at AMY 1.2.58-1-g06a4dd3, as a reference to read your own numbers against. Measured with a null run (--base HEAD --head HEAD, 5 captures per side), so the delta column is a self-check on the instrument rather than a result: both sides are built from identical sources, and every scene comes back inside its own noise with identical CRCs.

240 MHz · fixed-point · 48000 Hz · block 256 · free pacing · 3 passes · 5 captures/side · block budget 5333 us

Scene	cycles	us	headroom	null-run delta (noise)
juno6	1,159,320	4830	9.4%	+0.01% (±0.01%)
dx76	765,564	3189	40.2%	-0.00% (±0.01%)
saw_lpf6	442,452	1843	65.4%	-0.00% (±0.01%)
fx_sine8	386,559	1610	69.8%	-0.00% (±0.01%)
sine8	162,847	678	87.3%	-0.01% (±0.01%)
pcm4	141,812	590	88.9%	+0.00% (±0.00%)
idle	34,967	145	97.3%	+0.00% (±0.00%)

Ordered tightest first. juno6 at 9.4% headroom is the scene that matters: it is the one already close to overrunning the block budget, so it is the one whose delta you read first, whatever the others did.

A note on fx_sine8. Its three passes render different audio from each other within a single boot, because AMY's reverb and chorus buffers are never reset between scenes (see Known AMY gap). It is still perfectly usable as a benchmark, and the reason is worth being precise about: each pass is bit-identical across boots, and abcompare.py compares the two sides pass by pass. So the check that has to hold - a given pass renders the same CRC on every boot - does hold. The one that does not is "all passes agree with each other", and nothing depends on it.

It is the noisiest scene here regardless, so it is the one where a marginal delta deserves the least trust. Two captures are enough to see it move: at --repeat 2 it reads -0.03% against a ±0.01% spread and gets flagged, which on a null run can only be an artifact of an under-sampled noise estimate. At --repeat 5, above, it settles to -0.00%. If a sub-0.1% delta on this scene is load-bearing for a decision, spend the extra captures before believing it.

Measured noise floor

From this harness on an ESP32-S3 (240 MHz, octal PSRAM), free-running, fixed-point, 5 captures per side:

source of error	measured	how
boot-to-boot, same binary	±0.01%	--base HEAD --head HEAD - identical images, one per OTA slot
relink / code layout	≤0.07%	identical src/, binary relinked (a 64-byte shift)

The default --threshold 0.5 is derived from the second number, not the first, and the distinction is the whole point: repeated boots of one binary can never reveal layout noise, because layout is fixed per binary. It lands in the delta, never in the noise column. So the boot floor tells you the instrument is sound; only the relink floor tells you what a delta has to clear.

Re-derive both if you change the board, the scene set, or the build profile:

# null test: identical binaries, one per slot
python ../tools/abrun.py --port /dev/ttyACM0 --base HEAD --head HEAD --repeat 5

Every scene must come back within noise with identical CRCs. This also confirms that which OTA slot a firmware runs from does not affect its speed - the assumption the two-slot design rests on. If it fails, nothing else the tool says counts.

abrun.py options

flag	default	notes
--port	-	required unless --build-only
--head REF	working tree	ref under test
--base REF	merge-base of head and main	baseline
--repeat N	3	captures per side, interleaved A B A B. 2 is the minimum that yields a verdict.
--passes N	3	scene-list repetitions inside one run
--threshold PCT	0.5	regression threshold; see the noise floor above
--float	off (fixed-point)	production runs float on the S3's FPU
--paced	off (free-running)	GPTimer-paced at the real block period: headroom and overrun counting
--profile	off	per-tag AMY_DEBUG breakdown; inflates absolute numbers
--lto	off	also applies the gcc.cmake patch; see LTO below
--build-only	-	build both sides and stop, no board needed
--outdir DIR	tools/abrun-out	where logs and compare.json go
--timeout SEC	180	per-capture; a wedged board fails rather than hangs
--quiet	-	do not mirror the serial stream while capturing

Build options are set identically on both sides by these flags. Note this means idf.py menuconfig does not affect an abrun build: it generates a fresh sdkconfig per side from sdkconfig.defaults plus an overlay, precisely so the two sides cannot drift apart. Use menuconfig only for hand-built single-side runs. Either way abcompare re-checks the two run headers and shouts RUN CONFIG MISMATCH if the firmwares disagree about sample rate, block size, pacing, arithmetic mode or profiling.

Working by hand (single side)

idf.py build
idf.py -p /dev/ttyACM0 flash
python ../tools/capture.py --port /dev/ttyACM0 --out runA.log
python ../tools/abcompare.py -A runA.log runA2.log -B runB.log runB2.log

capture.py replaces idf.py monitor | tee - it resets the board, records the run, and stops on its own at the run_end footer, so it can be scripted. It exits non-zero on timeout, so "the board is wedged" is distinguishable from "the run was slow".

abcompare.py also takes --metric us (default cyc; the cycle counter is less jittery than wall time) and --json PATH for machine-readable output.

LTO profile

Cross-TU inlining changes codegen materially (loop forms, inlining depth), so wins and losses should be confirmed under LTO before they are trusted for an LTO-enabled production build. abrun.py --lto does this for both sides, including re-applying the gcc.cmake patch that a fresh component fetch overwrites (the published cmake_utilities component's IPO check fails against the ESP-IDF cross toolchain). By hand:

idf.py -D SDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.defaults.lto" reconfigure
cp ../tools/build-patches/espressif__cmake_utilities-gcc.cmake \
   managed_components/espressif__cmake_utilities/gcc.cmake
idf.py build

Never compare an LTO capture against a non-LTO one.

Known AMY gap: no FX state reset

AMY has no way to reset effects state, and this is an upstream bug, not just a bench inconvenience. Reverb's ten delay lines and its four IIR filter states (reverb_params_t, src/amy.h) are zeroed exactly once, at allocation - the bzero in new_reverb() and the clearing loop in new_delay_line(), both in src/delay.c - and never again. Chorus is the same. Nothing in AMY's RESET_* vocabulary (RESET_AMY, RESET_TIMEBASE, RESET_EVENTS, RESET_SYNTHS) touches them.

Turning an effect off (h0, k0) only stops it being processed. The tail does not drain - it freezes in the buffers, and is still sitting there when the effect is switched back on. Any host that reuses an AMY instance across songs inherits the previous one's reverb tail.

The visible symptom here is fx_sine8, the only scene with reverb and chorus: each pass starts the reverb from the previous pass's leftovers, so each pass renders different audio (60c02a8e, a50befbc, d9cac803). Every other scene is bit-identical across passes, which is what isolates FX state as the cause - sine8 is the same eight oscillators with the effects switched off, and it is perfectly stable.

This is not measurement error. Each pass is bit-identical across boots (10/10 captures), so abcompare.py compares the two sides pass by pass and keeps a fully working output oracle for the scene. Its timing is unaffected and as solid as any other scene (±0.01% boot-to-boot), so fx_sine8 stays in the scene list and needs no special handling from you.

The upstream fix is a reset that zeroes the FX delay lines and filter states, slotting into the existing RESET_* flags for a scene teardown to call. Until then, do not read fx_sine8's three per-pass CRCs as a fault.

Troubleshooting

symptom	cause
no such port	board not attached. On WSL, usbipd attach first.
this interpreter has no pyserial	running a python from outside the IDF env. Source export.sh.
could not apply the build-config guard(s)	amy.h no longer matches the shape AMY-EDITS.md describes, so the guards cannot be applied to that side's snapshot by pattern. Apply them by hand. abrun refuses to measure a tree whose sample rate and arithmetic mode it cannot verify.
RUN CONFIG MISMATCH	the two firmwares disagree about sample rate / pacing / float / profiling. Never compare them.
capture TIMEOUT ... no run_end	the board never finished a run: wedged, crashed, or a scene got much slower. The partial log is kept.
NON-DETERMINISTIC - the same pass renders differently on different boots	real nondeterminism (unseeded PRNG, wall-clock dependence). The scene's output oracle is void until fixed.
a scene reports 1 capture	--repeat 1. No noise estimate is possible; raise it.

Flashing over an existing firmware

Flashing the bench replaces the partition table and the app slots on the board. Data partitions of other firmware living above the bench's own partitions are not touched at the flash level; reflashing that firmware (with its own partition table) restores everything.

Output format

See tools/schema.md. Scenes are defined in esp32s3/main/scenes.c as plain AMY wire-format messages - add new workloads there. A new scene must be deterministic in the sense that matters: a given pass renders the same CRC on every boot. It need not render the same audio on every pass (see the FX gap above).