Skip to content

HW CI: average 3 capture runs per build to cut bench noise - #897

Merged
bwhitman merged 1 commit into
mainfrom
claude/hwci-usec-regression-6notes-a16f33
Jul 15, 2026
Merged

HW CI: average 3 capture runs per build to cut bench noise#897
bwhitman merged 1 commit into
mainfrom
claude/hwci-usec-regression-6notes-a16f33

Conversation

@bwhitman

Copy link
Copy Markdown
Collaborator

Back-to-back HW CI runs of effectively no-op PRs have been showing full-chord render µs deltas of ~±33 (e.g. #896), which is bench noise, not the PR's cost. This makes each bench measurement the mean of 3 runs: flash once, capture 20 s, reset the board, capture again — 3× — for both the PR build and its merge-base baseline.

Changes

  • measure.py --runs N: flashes once, then resets and re-captures N times (LoadTestChord restarts its chord sequence on reset, and capture() already pulses RTS at t=0, so no re-flash is needed). Run 1 keeps load.csv/serial.log; runs 2+ add loadK.csv/serialK.log. meta.json gains a per-run runs list, and the top-level render_us_final/render_us_max/n_samples become the mean/max/total over the clean runs. A run that gets doubly hit by external bench interference keeps its logs but is dropped from the averages (and still FAILs the verdict, as before). Default is --runs 1, so make speedtest and the sweep are unchanged.

  • hwci_report.py: averages each chord size's settled load across the clean runs, prefixes per-run problems with run N:, and adds a per-run full-chord line to the PR comment so run-to-run spread is visible at a glance:

    Full chord settled render µs: 1610 (was 1555, Δ +3.5%) (peak 1640, 117 samples, mean of 3 runs)

    per-run full-chord µs — this PR: 1600 / 1600 / 1630 · main @ abc1234: 1550 / 1560 / 1555

    Old single-run meta.json files (no runs key) still analyze, so a dir from an old sweep or a mixed-version bench run doesn't break the report.

  • amy-hwci.yml: passes --runs 3 for both builds, uploads the per-run logs/traces (serial*.log / load*.csv globs), and updates the comment text. Adds ~80 s per bench job — well inside the 20 min timeout.

Testing

Simulated the whole pipeline with upload/capture stubbed to synthetic UART lines: 3 clean runs (means + spread line verified against per-run stats), an interfered run 2 (excluded from means, run 2: errors, FAIL verdict), single-run mode (byte-identical report shape to today), and a legacy no-runs baseline dir. All 15 checks pass; workflow YAML validated.

Note: the bench checks out harness scripts from main, so this PR's own HW CI comment will still be a single-run one — the 3× averaging takes effect for PRs benched after this merges.

🤖 Generated with Claude Code

Back-to-back HW CI runs of no-op PRs were showing full-chord render_us
deltas of ~±33 us (e.g. PR #896), which is pure bench noise. Reduce it by
capturing each flashed build 3 times instead of once:

- measure.py --runs N: flash once, then reset the board and re-capture N
  times (the sketch restarts its chord sequence on reset). Run 1 keeps
  load.csv/serial.log; runs 2+ add loadK.csv/serialK.log. meta.json gains
  a per-run "runs" list; the top-level render_us_final/max/n_samples
  become the mean/max/total over the clean runs. A doubly-interfered run
  keeps its logs but is dropped from the averages.
- hwci_report.py: averages each chord size's settled load across the
  clean runs, prefixes per-run problems with "run N:", and shows the
  per-run full-chord values so run-to-run spread is visible on the PR.
  Old single-run meta.json (no "runs" key) still analyzes.
- amy-hwci.yml: pass --runs 3 for both the PR and merge-base builds,
  upload the per-run logs/traces, update the comment text.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🎛️ AMY HW CI (AMYboard bench)

Flashed this PR's AMY (LoadTestChord: 6-voice Juno patch=1, one held note every 2 s) onto the physical AMYboard and measured the smoothed render load as the chord grows — back-to-back with the same sketch built at the PR's merge base, so Δ is this PR's own cost.

PASS — the bench ran the test to completion.

notes held main @ 73da71e this PR Δ
1 1603 1605 +2
2 1800 1804 +4
3 2479 2489 +10
4 2720 2720 +0
5 3465 3466 +1
6 3674 3677 +3

Full chord settled render μs: 3678 (was 3672, Δ +0.2%) (peak 3686, 39 samples)

⬇️ Artifacts: serial log · load trace · report

Self-hosted bench (amyboardci). FAIL means only that the test could not run — the load values are informational, with no threshold and no audio compare. See tools/arduino_loadsweep/.

@bwhitman
bwhitman merged commit 4bf060b into main Jul 15, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant