HW CI: average 3 capture runs per build to cut bench noise - #897
Merged
Conversation
Back-to-back HW CI runs of no-op PRs were showing full-chord render_us deltas of ~±33 us (e.g. PR #896), which is pure bench noise. Reduce it by capturing each flashed build 3 times instead of once: - measure.py --runs N: flash once, then reset the board and re-capture N times (the sketch restarts its chord sequence on reset). Run 1 keeps load.csv/serial.log; runs 2+ add loadK.csv/serialK.log. meta.json gains a per-run "runs" list; the top-level render_us_final/max/n_samples become the mean/max/total over the clean runs. A doubly-interfered run keeps its logs but is dropped from the averages. - hwci_report.py: averages each chord size's settled load across the clean runs, prefixes per-run problems with "run N:", and shows the per-run full-chord values so run-to-run spread is visible on the PR. Old single-run meta.json (no "runs" key) still analyzes. - amy-hwci.yml: pass --runs 3 for both the PR and merge-base builds, upload the per-run logs/traces, update the comment text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
🎛️ AMY HW CI (AMYboard bench)Flashed this PR's AMY (LoadTestChord: 6-voice Juno ✅ PASS — the bench ran the test to completion.
Full chord settled render μs: 3678 (was 3672, Δ +0.2%) (peak 3686, 39 samples) ⬇️ Artifacts: serial log · load trace · report Self-hosted bench (amyboardci). FAIL means only that the test could not run — the load values are informational, with no threshold and no audio compare. See |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Back-to-back HW CI runs of effectively no-op PRs have been showing full-chord render µs deltas of ~±33 (e.g. #896), which is bench noise, not the PR's cost. This makes each bench measurement the mean of 3 runs: flash once, capture 20 s, reset the board, capture again — 3× — for both the PR build and its merge-base baseline.
Changes
measure.py --runs N: flashes once, then resets and re-captures N times (LoadTestChord restarts its chord sequence on reset, andcapture()already pulses RTS at t=0, so no re-flash is needed). Run 1 keepsload.csv/serial.log; runs 2+ addloadK.csv/serialK.log.meta.jsongains a per-runrunslist, and the top-levelrender_us_final/render_us_max/n_samplesbecome the mean/max/total over the clean runs. A run that gets doubly hit by external bench interference keeps its logs but is dropped from the averages (and still FAILs the verdict, as before). Default is--runs 1, somake speedtestand the sweep are unchanged.hwci_report.py: averages each chord size's settled load across the clean runs, prefixes per-run problems withrun N:, and adds a per-run full-chord line to the PR comment so run-to-run spread is visible at a glance:Old single-run
meta.jsonfiles (norunskey) still analyze, so a dir from an old sweep or a mixed-version bench run doesn't break the report.amy-hwci.yml: passes--runs 3for both builds, uploads the per-run logs/traces (serial*.log/load*.csvglobs), and updates the comment text. Adds ~80 s per bench job — well inside the 20 min timeout.Testing
Simulated the whole pipeline with
upload/capturestubbed to synthetic UART lines: 3 clean runs (means + spread line verified against per-run stats), an interfered run 2 (excluded from means,run 2:errors, FAIL verdict), single-run mode (byte-identical report shape to today), and a legacy no-runsbaseline dir. All 15 checks pass; workflow YAML validated.Note: the bench checks out harness scripts from main, so this PR's own HW CI comment will still be a single-run one — the 3× averaging takes effect for PRs benched after this merges.
🤖 Generated with Claude Code