Report every failed image comparison in a visual-regression case, not the first (BL-16799) - #8291
Merged
Merged
Conversation
… the first Each case in src/BloomVisualRegressionTests/index.spec.ts threw on its first mismatch, so the comparisons after it never captured their images. A stale baseline therefore surfaced one image per run, and each run takes about three minutes: BL-16638 needed three accept-and-rerun rounds to get through one layout change. comparePreviewImage now records a mismatch in a per-case list instead of throwing, and the case body throws once at the end, naming every image that did not match. Anything thrown while comparing, such as Pixelmatch's "Image sizes do not match", is recorded the same way, so it is still a failure and it no longer hides the comparisons behind it. The list resets at the start of each case. Retires the AUTOMATION-DEBT.md entry "Visual-regression cases stop at the first failed comparison". Verified by type check only. This suite's baselines match the CI runner rather than a developer machine, so it cannot go green here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hatton
force-pushed
the
BL-16799-vr-collect-failures
branch
from
September 3, 2026 19:40
e80c2eb to
6d05507
Compare
Master took two changes this branch had to be written against again. In src/BloomVisualRegressionTests/index.spec.ts, master's "Render the visual-regression player pages from Bloom's own fonts" deleted the likelyCause and andikaIsInstalled mechanism: the fonts no longer differ, so there is no longer an installed Andika to blame a mismatch on. This branch had extended that mechanism. So the change this branch makes, which is to collect every failed image comparison and fail once at the end of the case, is here written against the comparePreviewImage master leaves, and carries no likely cause. In src/BloomE2E/AUTOMATION-DEBT.md, master reworked the table of this stack to give each pull request its number. This branch deletes its own row and the entry it pays off, so the merge applies both deletions to master's text. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BaWw4EHWrbFF2fNJvMJVC8
Member
Author
|
[Claude Opus 5 from Hatton's machine during devin-review] Consulted Devin on 2026-09-03 22:45 UTC up to commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each case in src/BloomVisualRegressionTests/index.spec.ts threw on its first
mismatch, so the comparisons after it never captured their images. A stale
baseline therefore surfaced one image per run, and each run takes about three
minutes: BL-16638 needed three accept-and-rerun rounds to get through one
layout change.
comparePreviewImage now records a mismatch in a per-case list instead of
throwing, and the case body throws once at the end, naming every image that did
not match. Anything thrown while comparing, such as Pixelmatch's "Image sizes
do not match", is recorded the same way, so it is still a failure and it no
longer hides the comparisons behind it. The list resets at the start of each
case.
Retires the AUTOMATION-DEBT.md entry "Visual-regression cases stop at the first
failed comparison".
Verified by type check only. This suite's baselines match the CI runner rather
than a developer machine, so it cannot go green here.
This is one of eleven stacked pull requests (BL-16799)
Each one pays down one entry of
src/BloomE2E/AUTOMATION-DEBT.md, and each branches off the one before it. Base:BL-16799-automation-scripts. Review only this pull request's own commit; the ones below it are reviewed in their own pull requests. The first six change test and tooling code only; the last five also change product code.BL-16799-automation-scripts— Make the bloom-automation scripts safe to ask for helpBL-16799-vr-collect-failures— Report every failed image comparison in a visual-regression case, not the firstBL-16799-component-tests-in-ci— Run the component-tester Playwright suites nightlyBL-16799-vite-port— Let an e2e run test the working tree's front endBL-16799-type-in-one-call— Type into a text box in one call, not one key press per characterBL-16799-page-screenshot— Capture a whole book page from a testBL-16799-toolbox-registration— Register the toolbox tools from one list both callers shareBL-16799-shell-document— Stop a test attaching to a shell document Bloom does not driveBL-16799-tab-test-ids— Click a workspace tab by a test id, not by its localized labelBL-16799-page-change— Refuse a page change the Edit tab cannot do, and wait before askingBL-16799-collection-languages— Set a collection's languages through an e2e hook, not by writing XMLReplaces #8276, which did all of this in one pull request.
Verification of the whole stack, at its tip: the C# suite passes (3338 passed, 13 skipped), the front-end vitest suite passes (781 passed, 5 skipped), and the
src/BloomE2Esuite passes against a Vite dev server on the working tree (36 passed, 0 skipped, 8.2 minutes). Each pull request also has its own type check and lint.🤖 Generated with Claude Code
Devin review
This change is