ci: keep visual-regression baselines refreshing and surface Cypress failures - #2669
Open
balzss wants to merge 3 commits into
Open
ci: keep visual-regression baselines refreshing and surface Cypress failures#2669balzss wants to merge 3 commits into
balzss wants to merge 3 commits into
Conversation
…ailures The baseline job ran Cypress without continue-on-error, so failing a11y/console assertions aborted the job before the "Push baselines" step — silently freezing the visual-baselines branch on every merge since the dark/light theme suite landed. Decouple baseline capture from the assertion gate: Cypress runs with continue-on-error, the screenshots always flatten and push (if: always()), and a trailing step fails the job for visibility if Cypress failed. Baselines now refresh on every merge regardless of the a11y/console outcome. Also add a "Cypress assertions" pass/fail line to the sticky PR comment so failures are visible even when the visual diff is clean, and remove the leftover TEMP diff-demo fixture step and regression-test/cypress/diff-demo/. Refs: INSTUI-5137 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
Contributor
Visual regression reportCypress assertions (a11y + console errors): ❌ Failing — see the run logs. Visual diff:
Diff images (32)alert.png — baseline no longer producedavatar.png — baseline no longer producedbadge.png — baseline no longer producedbillboard.png — baseline no longer producedbreadcrumb.png — baseline no longer producedbutton-and-derivatives.png — baseline no longer producedbyline.png — baseline no longer producedcalendar.png — baseline no longer producedcheckbox.png — baseline no longer producedcheckboxgroup.png — baseline no longer producedcolorpicker.png — baseline no longer producedcontextview.png — baseline no longer producedcustom-and-lucide-icons.png — baseline no longer produceddateinput-dateinput2.png — baseline no longer produceddatetimeinput.png — baseline no longer produceddrilldown.png — baseline no longer producedfiledrop.png — baseline no longer producedform-errors.png — baseline no longer producedheading.png — baseline no longer producedimg.png — baseline no longer producedlink.png — baseline no longer producedmenu.png — baseline no longer producedmetric-pill-tag-timeselect-text.png — baseline no longer producedoptions.png — baseline no longer producedpagination.png — baseline no longer producedprogressbar.png — baseline no longer producedselect-simpleselect.png — baseline no longer producedtable.png — baseline no longer producedtabs.png — baseline no longer producedtooltip.png — baseline no longer producedtreebrowser.png — baseline no longer producedview.png — baseline no longer producedBaselines come from the |
The regression suite already ran axe per page and theme, but the violations were only printed to the Cypress log and discarded. Persist them and render them in the report so designers can triage accessibility issues alongside the pixels. - regression-test: a new recordA11y Cypress task writes cypress/a11y.json keyed by "<slug>-<theme>" (mirroring meta.json); the spec records each violation's rule id, impact, help text, helpUrl, and offending node selector/summary. - ui-scripts visual-diff: a new --a11y flag reads that file and adds a per-row "a11y" badge, an "A11y" filter chip, a header count, and a collapsible list of violations (rule linked to its axe helpUrl, impact, selector, contrast summary) per screenshot. a11yPages/a11yViolations are added to summary.json. - visual-regression.yml passes --a11y cypress/a11y.json to the diff step. Additive: without --a11y the report is unchanged. Unit-tested esc() and a11yFor(), and verified end to end against a real suite run (15 flagged pages). Refs: INSTUI-5137 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… HTML view Builds on the a11y report data: make the violations visible without hunting. - Expand each row's violation list by default (Tier 1 was collapsed). - In the lightbox HTML view, outline the offending nodes inside the live iframe using the recorded axe selectors, and add a legend that lists each violation and scrolls/flashes its element on click. - Embed the a11y data as a JSON block for the client; same-origin iframe access means no pixel math. Runtime-generated selectors (Emotion hashes, generated ids) resolve because the iframe app and the axe run share one build; if they ever diverge the outline no-ops but the legend and text details still show. Refs: INSTUI-5137 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repairs the self-hosted visual-regression pipeline and makes the report far more useful for triage. Two related concerns, in separate commits:
1. Baseline pipeline fix (
ci)visual-baselines.yml: decouple baseline capture from the a11y/console gate — Cypress runs withcontinue-on-error, screenshots always flatten + push (if: always()), and a trailing step fails the job for visibility if Cypress failed. Fixes the silent baseline freeze: thevisual-baselinesbranch had stopped updating on every merge since the dark/light theme suite landed (Cypress aborted the job before the push step).visual-regression.yml: the sticky PR comment now leads with a "Cypress assertions (a11y + console errors)" ✅/❌ line, so failures are visible even when the visual diff is clean.TEMP inject diff-demo fixturestep andregression-test/cypress/diff-demo/.2. A11y in the visual-diff report (
feat)recordA11yCypress task now persists them tocypress/a11y.json, keyed by<slug>-<theme>(mirrorsmeta.json).ui-scripts visual-diffgains a--a11yflag that renders a per-page⚠ N a11ybadge, an⚠ A11yfilter chip, a header count, and a collapsible violation list (rule linked to its axehelpUrl, impact, offending selector, contrast summary).a11yPages/a11yViolationsadded tosummary.json. Additive — without--a11ythe report is unchanged.--a11y cypress/a11y.jsoninto the diff step.Test Plan
visual-diffsuite (34 tests, incl. newesc/a11yFor).a11y.json(15 flagged page/theme entries) and the report renders badges/filter/list with the real axe contrast data.visual-baselineseven while the suite is red, and ends red with the::error::annotation.color-contrastviolations remain (tracked in INSTUI-5137) — the suite stays red-but-visible until those are fixed by design/component work.Fixes INSTUI-5137
🤖 Generated with Claude Code