test(e2e): De-flake remix-hydrogen client tests#22051
Merged
Merged
Conversation
Two changes to reduce flakiness in the remix-hydrogen E2E client tests (CI runs these with retries: 0, so any transient failure is reported as flaky): - Drop `replayIntegration` (and `debug`) from the client SDK init. Replay was recorded intermittently (`replaysSessionSampleRate: 0.1`, and always on the error tests via `replaysOnErrorSampleRate: 1.0`), spinning up the replay worker and sending extra envelopes through the same tunnel — intermittent, timing- sensitive load that no test here asserts on. This matches the leaner hydrogen-react-router-7 sibling app. - Gate the "client-side exception" test on hydration by awaiting the pageload transaction before clicking the button, mirroring the existing mitigation in the navigation test. A click that lands before the onClick handler is attached never captures the exception. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mydea
requested review from
a team,
chargome,
nicohrubec and
s1gr1d
and removed request for
a team
chargome
approved these changes
Jul 9, 2026
chargome
approved these changes
Jul 9, 2026
Member
|
I have no idea why this is reviewed twice here lol |
mydea
added a commit
that referenced
this pull request
Jul 9, 2026
…22061) Companion to #22051 (remix-hydrogen): the sibling hydrogen-react-router-7 app has the same click-before-hydration flakiness in its client tests. ### Root cause Three client tests click an element immediately after `page.goto()`. Playwright's `click()` waits for actionability but **not** React hydration, so the click can land before the element's `onClick` handler is attached: - the exception / ErrorBoundary buttons then capture no error, and - the `<Link>` behaves like a plain anchor, producing a full-page `pageload` instead of the expected client-side `navigation` transaction. ### Fix Await the pageload transaction — which only completes once the client SDK and router have hydrated — before clicking, in the exception, ErrorBoundary and navigation tests. This is the exact mitigation the remix-hydrogen navigation test already uses. This app already omits Replay/`debug`, so no SDK-config change is needed. The two remaining tests in the group (`pageload`, `meta tags`) are load-only with no click; they were swept into the same flaky-run batch and have no per-test hydration race to fix here. These E2E tests require the packed SDK tarballs + Shopify Hydrogen toolchain to run, so they're validated in CI rather than locally. Fixes #21737 Fixes #21736 Fixes #21710 Fixes #21735 Fixes #21734 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The remix-hydrogen E2E client tests are a recurring source of flaky-CI reports — all five below were filed from a single run. CI runs this app with
retries: 0, so any transient client-side hiccup surfaces as a flaky failure.Root cause
Every one of the flaky tests exercises the client (
page.goto+ wait for an envelope or click). The remix-hydrogen client SDK differs from the leaner, more reliablehydrogen-react-router-7sibling app in two ways that add timing variance:replaysSessionSampleRate: 0.1records a replay in ~10% of runs, andreplaysOnErrorSampleRate: 1.0records one on both error tests. When it records, it spins up the replay worker and pushes extra envelopes through the samelocalhost:3031tunnel the tests wait on — intermittent, timing-sensitive load that no test here asserts on. Intermittent-by-sample-rate behavior is a classic flakiness signature.#exception-buttonimmediately aftergoto('/'). Playwright'sclick()waits for actionability but not React hydration, so the click can land before theonClickhandler is attached — the exception is then never captured and the test times out. The navigation test already worked around exactly this by awaiting the pageload transaction first.Fix
replayIntegration(anddebug) fromentry.client.tsx, matching thehydrogen-react-router-7app.The hydration gate directly fixes the exception test; trimming Replay reduces the intermittent client-side load behind the broader batch of client-test flakes. These E2E tests can't be run against real CI infra locally, so the changes are conservative and reduce known flake surface rather than depending on unverifiable timing.
Fixes #21742
Fixes #21739
Fixes #21740
Fixes #21741
Fixes #21738
🤖 Generated with Claude Code