Skip to content

ci(visual): stop churn PRs from non-deterministic tray screenshots#39

Merged
afonsojramos merged 3 commits into
mainfrom
istanbul
Jun 3, 2026
Merged

ci(visual): stop churn PRs from non-deterministic tray screenshots#39
afonsojramos merged 3 commits into
mainfrom
istanbul

Conversation

@afonsojramos
Copy link
Copy Markdown
Member

Summary

The Visual Tray workflow opens a PR (e.g. #38) on nearly every push to main, even when nothing visually changed. The screenshots are not byte-stable: the PNGs carry no metadata, but the decoded pixels differ run to run. Two distinct sources:

  • macOS (14, 15): sub-pixel anti-aliasing on the window's rounded corners. ~400 pixels differ but 0 survive a 2% fuzz — invisible, yet enough to make every capture a new file.
  • Windows: genuine render variance — the popover window settles a few pixels off between runs (windows-2025), and the tray marker only paints once Explorer has finished repainting the promoted icon (windows-2022).

Then the update-platforms job did rm -f *.png; cp *.png unconditionally and create-pull-request opened a PR on any byte change — so AA jitter alone produced a churn PR.

Changes

  1. Gate screenshot updates on a meaningful pixel diff. Compare each new capture against the committed one with compare -metric AE -fuzz 2% and only overwrite when more than 100 pixels differ. Cosmetic AA jitter (0 px) is ignored; real tray/window changes still land. Dimension changes and new platforms fall through to an update. (Adds an ImageMagick install step to the job.)
  2. Wait longer for the Windows window to settle before capture (VISUAL_POST_READY_DELAY_MS=6000), so the frameless fixture window is at its pinned (400,200) position rather than caught mid-move.
  3. Extend the Explorer repaint wait (6s -> 10s) so the promoted magenta/green tray marker renders reliably instead of falling back to a generic glyph.

Verification

Ran the exact gated-sync shell against the real PR #38 artifacts with the actual compare binary:

file diff @2% fuzz outcome
macos-14 / macos-15 0 px keep (no churn)
windows-2022 (marker) 256 px update
windows-2025 (shift) 2438 px update
identical capture 0 px keep
dimension change large / non-numeric update

shellcheck clean on the embedded script; workflow YAML validated.

Notes

  • Behavior change: the sync no longer deletes committed screenshots — strictly safer, since the old rm would have wiped a good screenshot whenever a platform job produced no PNG (transient failure). Trade-off: a platform removed from the matrix leaves an orphaned (unreferenced) screenshot to clean up manually.
  • Change 1 fully fixes the macOS churn (provably 0 px). Changes 2-3 are determinism mitigations for the Windows diffs; I had a single before/after to work from, so their effect should be confirmed across a few real main runs. If windows-2025 still churns, the deeper fix is to re-read getBounds() immediately before capture in tests/visual/run.ts.

@afonsojramos afonsojramos merged commit 04cd05f into main Jun 3, 2026
6 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant