fix(ci): Surface screenshot test failures and add retries#119
Merged
Conversation
Three related fixes so the screenshot CI job actually fails PRs when ScreenshotUITests fails: - Extend waitForExistence timeouts in ScreenshotUITests to 10s across the board. The 1s/2s/3s timeouts were fine locally but too tight under CI runner load, particularly around long-press + context-menu interactions (lines 63/66 and 150/153), which have been flaky. - Set number_of_retries: 3 on both capture_screenshots calls (generate_screenshots and generate_screenshots_ci). Absorbs genuine simulator flakes without hiding real bugs. - Set stop_after_first_error: true on both capture_screenshots calls. Without this, snapshot's simulator_launcher swallows exhausted-retry failures (see snapshot/lib/snapshot/simulator_launchers/ simulator_launcher.rb:131) and returns as if everything succeeded — which is why every screenshot run on main since #116 reported ✅ while actually producing incomplete screenshot sets.
📲 Install BuildsiOS
|
philprime
added a commit
that referenced
this pull request
Apr 22, 2026
The matrix screenshot jobs use scan (run_tests) rather than snapshot (capture_screenshots), so the retries added in #119 don't apply here. iPad matrix jobs are consistently flaking at the long-press / context-menu interaction (ScreenshotUITests:66) and failing the whole release because release-upload needs the matrix. Add number_of_retries: 3 to the run_tests call so scan retries the test before giving up, matching the snapshot lane's behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The screenshot CI job has been green on main since #116 despite the underlying
ScreenshotUITests.testScreenshotsfailing on most runs — the failures silently disappeared, and the empty/partial artifacts never raised an alert.Root cause in
fastlane/snapshot/lib/snapshot/simulator_launchers/simulator_launcher.rb:131:Snapshot only raises on exhausted retries when
stop_after_first_erroris true. Without it, the error_proc falls off the end and snapshot returns as if nothing happened. Our lanes had it at the default (false).Three changes:
stop_after_first_error: trueon bothcapture_screenshotscalls. This is the key fix — makes failures actually fail the lane, which fails the job, which blocks the PR.number_of_retries: 3on both calls. Absorbs real simulator flake (long-press + context-menu timing) without hiding real bugs.waitForExistencetimeouts to 10s inScreenshotUITests.testScreenshots. The previous 1–5s timeouts are fine on a dev Mac but too tight on a contended hosted runner. Lines 63/66 and 150/153 (press(forDuration:) → editButton.waitForExistence(timeout: 3)) have been the main flake source.Flag name
stop_after_first_erroris misleading — despite the name, it does not stop before retries. It only affects what happens after retries are exhausted: crash (true) vs. return silently (false). Combined withnumber_of_retries: 3, the real behaviour is "retry 3 times, then fail loudly", which is what we want.Depends on #118 (EDR simulator fallback) being in main — without it, retries would still fail on every attempt due to the QR accessibility issue.