👷 Add pinned Firefox 119 + WebKit 17.4 e2e + Chromium 120 test runner without BrowserStack#4612
Merged
Conversation
…rStack - Add `playwright.pinned.config.ts` that boots a Playwright 1.40.1 `run-server` (which bundles FF 119 / WK 17.4) plus a translation proxy, and connects the current 1.58 client over wsEndpoint. - Add `scripts/pinned-proxy.mjs` to bridge the JSON-RPC schema gap between Playwright 1.58 and 1.40 (UA spoofing for the version check, patches to __create__ initializers and a few command params). - Add `test:e2e:pinned:init` / `test:e2e:pinned` scripts to install the pinned browsers locally and run the pinned matrix. - Ignore the local `test/e2e/.pinned-browsers/` install directory.
In WebKit, `beforeunload` doesn't always fire when navigating to a different domain. Specifically, navigating from `localhost` to `foo.localhost` or between `localhost` and `127.0.0.1` silently skips the event, so the SDK never flushes. Tests using `withHostName()` were affected. Fix by navigating to `/flush` on the same origin when the current page is served by the base server, and falling back to the base server origin otherwise.
Firefox does not allow setting cookies from iframes without src, so the SDK won't start and these tests are not meaningful there.
- rename test/e2e/scripts/pinned-proxy.mjs to pinnedProxy.ts - add ws and @types/ws to root devDependencies (was an unlisted import) - add types for JSON-RPC messages, RawData handling, and header forwarding - update playwright.pinned.config.ts to invoke `node pinnedProxy.ts`
Without this directive, GitLab relies on the project-level "Auto-cancel redundant pipelines" setting, which is currently not effective for this project. Explicitly enabling `auto_cancel.on_new_commit: interruptible` ensures interruptible jobs (e.g. unit-bs, e2e-bs queued behind the browserstack resource group) are cancelled when a new commit lands on the same branch.
The previous check allowed a margin of 2 for Firefox, but Firefox async stack traces can exceed that margin. Instead of continuously adjusting the margin, we drop the upper bound assertion entirely.
In WebKit, the dd_cookie_test_* probe cookie (written by areCookiesAuthorized) lingers in the browser's in-memory store after deletion, appearing before the real _dd_s_v2 session cookie. Tests reading cookies[0] or asserting the full cookie array shape would fail because they hit the probe cookie instead.
Co-authored-by: Benoit Zugmeyer <benoit.zugmeyer@datadoghq.com>
Firefox renders the inner setTimeout callback as `window.RUM_INIT/<` instead of `<anonymous>` with the npm setup, because the test's rumInit is wrapped in a `window.RUM_INIT` arrow.
- install Chromium via the pinned Playwright 1.40.1 in Dockerfile and test:e2e:init - add chromium-pinned project (Chrome 120) to playwright config
Two RUM tests (rum/actions: "associate a long tasks to its action", microfrontend: "LOAf should have service and version from source code context") rely on PerformanceLongAnimationFrameTiming, which shipped in Chrome 123 and is not available in chromium-pinned (Chrome 120). Replace the previous "browserName !== 'chromium'" gate with a runtime feature check via PerformanceObserver.supportedEntryTypes — also covers non-Chromium browsers without an explicit version assumption.
…vert after)" This reverts commit f942810.
The Playwright 1.40 server used by the pinned Chromium runtime never emits service-worker-sourced console events on any channel, so the NO_SESSION_WARNING check would always fail there. Detect the pinned runtime via project metadata (presence of `version`) and skip the assertion, mirroring the existing Firefox carve-out. - add `isPinnedRuntime` helper based on project metadata - make `os`/`osVersion` optional in BrowserConfiguration (pinned projects don't set them) and apply `satisfies BrowserConfiguration` to project metadata for type safety
- A bare `yarn test:e2e` now runs only the four non-pinned projects, so a fresh checkout that hasn't run `yarn test:e2e:init` doesn't fail trying to launch pinned browsers it doesn't have. - Pinned projects (and their `run-server` + proxy web servers) only surface when `--project=...` is passed; CI already does this. - Tighten the SW-console comment in the no-session warning assertion.
Earlier on this branch yarn.lock drifted with many unintended downgrades (esbuild 0.27.7→0.27.3, @babel/helpers 7.29.2→7.28.6, @emnapi/* 1.10.0→1.9.2, @1natsu/wait-element 4.2.0→4.1.2, and others) — none of which match a package.json change in this PR. Restore yarn.lock from origin/v7 and rerun yarn install so the lockfile reflects only the real additions on this branch (ws, @types/ws). Net diff vs v7 is now +18/-16.
…config Filtering projects via process.argv broke worker processes: Error: Project "chromium-pinned" not found in the worker process. Make sure project name does not change. Worker processes don't inherit the test runner's argv, so getSelectedProjects() returned [] inside the worker, getProjects() stripped the pinned entries, and the worker couldn't find the project name assigned to it. ~700 retries later the job died. Project list must be deterministic across processes. Restore the original shape: always return all 7 projects; only the webServer boot decision (which runs only in the main process) gates on argv. The 'fresh-checkout boots pinned servers' UX concern from the review is best handled in docs, not config.
- Use node:util parseArgs for --project / -p extraction - Drop stale reference to playwright.base.config.ts in test/e2e/AGENTS.md
- Drop internal notes for the CI eval loop; not intended for the repo
Bundles Sizes Evolution
🚀 CPU Performance
🧠 Memory Performance
|
mormubis
approved these changes
May 12, 2026
BenoitZugmeyer
approved these changes
May 12, 2026
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ef76412dfb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+40
to
+44
| RUN npx -y playwright@${PLAYWRIGHT_VERSION} install --with-deps chromium | ||
|
|
||
| # Pinned Playwright browsers: Chromium 120 + Firefox 119 + WebKit 17.4 (used by the e2e-pinned job) | ||
| ARG PINNED_PLAYWRIGHT_VERSION=1.40.1 | ||
| RUN npx -y playwright@${PINNED_PLAYWRIGHT_VERSION} install --with-deps chromium firefox webkit |
There was a problem hiding this comment.
The root AGENTS.md for /workspace/browser-sdk says this project uses Yarn workspaces and to “Never use npm or npx”, but rebuilding the CI image now invokes npx for both Playwright installs. This bypasses the repo’s Yarn-only dependency/registry setup during the ci-image job, so please switch these image-build installs to the Yarn equivalent.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Replicate the BrowserStack Firefox 119 + WebKit 17.4 coverage locally so we can keep running pinned-browser e2e tests without BrowserStack, and lay the groundwork for a CI eval loop that iterates on GitLab jobs autonomously.
Changes
e2e-pinnedGitLab job andtest:e2e:pinnedscript driven by a newplaywright.pinned.config.ts.run-serverplus a small translation proxy (test/e2e/scripts/pinned-proxy.mjs) so the current 1.58 client can talk to the older server.CURRENT_CI_IMAGEto 106.flushEventsso the flush works for pages served from custom domains (WebKit fix).iframe-without-srcsession cookie tests on Firefox.APPROACH.md.Test instructions
yarn test:e2e:pinned:initthenyarn test:e2e:pinnedlocally to run Firefox 119 + WebKit 17.4.e2e-pinnedjob runs green in GitLab on this branch.Checklist