Skip to content

✅ improve E2E test reliability on BrowserStack#4494

Merged
BenoitZugmeyer merged 3 commits intomainfrom
benoit/e2e-bs-reliability
Apr 16, 2026
Merged

✅ improve E2E test reliability on BrowserStack#4494
BenoitZugmeyer merged 3 commits intomainfrom
benoit/e2e-bs-reliability

Conversation

@BenoitZugmeyer
Copy link
Copy Markdown
Member

@BenoitZugmeyer BenoitZugmeyer commented Apr 16, 2026

Motivation

E2E tests running on BrowserStack have been flaky due to high network latency between the remote browser and the local test server. Several issues compounded this:

  • The idle wait duration was fixed at 500ms regardless of actual latency, causing waitForServersIdle() to return before all requests had arrived
  • Requests sent by the browser could still be in-flight when waitForServersIdle() was called, making it return immediately despite pending network activity
  • The browser was issuing spurious /favicon.ico requests that added noise to server idle tracking

Changes

  • Measure browser-to-server latency once per worker before tests run, and use Math.max(500, latency * 1.5) as the idle wait duration
  • waitForServersIdle() now waits idleWaitDuration ms upfront to let in-flight requests arrive before checking idle state
  • Add a dedicated /flush route (replaces /ok?duration=200) that delays 200ms and returns HTML — used when navigating away to flush SDK events, including session replay Web Worker segments
  • Suppress spurious /favicon.ico requests by embedding an empty favicon in all test pages
  • Skip redundant flushEvents() in teardown when the page is already on /flush
  • Stop the CI test run after the first failure (post-retries) to save time

Test instructions

Run the E2E tests on BrowserStack and verify that previously flaky tests pass consistently.

Checklist

  • Tested locally
  • Tested on staging
  • Added unit tests for this change.
  • Added e2e/integration tests for this change.
  • Updated documentation and/or relevant AGENTS.md file

In BrowserStack, the browser runs remotely and the round-trip latency to
the test server can be significant. A fixed idle wait duration may not be
sufficient to catch all in-flight requests. We now measure the latency
once per worker before tests run and use `Math.max(500, latency * 1.5)`
as the idle wait duration.
…avicon requests

- waitForServersIdle() now waits idleWaitDuration ms before checking idle
  state, so in-flight browser requests have time to reach the server before
  idle is declared
- replace /ok?duration=200 with a dedicated /flush route that delays 200ms
  and returns HTML with an empty favicon, preventing spurious favicon requests
  during event flushing
- skip teardown flushEvents() when already on /flush to avoid redundant work
- add empty favicon link to all test pages to suppress /favicon.ico requests
@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da Bot commented Apr 16, 2026

Bundles Sizes Evolution

📦 Bundle Name Base Size Local Size 𝚫 𝚫% Status
Rum 178.84 KiB 178.84 KiB 0 B 0.00%
Rum Profiler 6.16 KiB 6.16 KiB 0 B 0.00%
Rum Recorder 27.03 KiB 27.03 KiB 0 B 0.00%
Logs 56.36 KiB 56.36 KiB 0 B 0.00%
Rum Slim 134.70 KiB 134.70 KiB 0 B 0.00%
Worker 23.63 KiB 23.63 KiB 0 B 0.00%
🚀 CPU Performance
Action Name Base CPU Time (ms) Local CPU Time (ms) 𝚫%
RUM - add global context 0.0041 0.0066 +60.98%
RUM - add action 0.0135 0.0186 +37.78%
RUM - add error 0.0131 0.0171 +30.53%
RUM - add timing 0.0032 0.0033 +3.12%
RUM - start view 0.0136 0.0158 +16.18%
RUM - start/stop session replay recording 0.0008 0.0009 +12.50%
Logs - log message 0.0167 0.0178 +6.59%
🧠 Memory Performance
Action Name Base Memory Consumption Local Memory Consumption 𝚫
RUM - add global context 31.72 KiB 32.66 KiB +966 B
RUM - add action 102.50 KiB 106.93 KiB +4.43 KiB
RUM - add timing 32.67 KiB 32.62 KiB -42 B
RUM - add error 106.81 KiB 107.49 KiB +692 B
RUM - start/stop session replay recording 32.50 KiB 32.27 KiB -235 B
RUM - start view 482.63 KiB 489.43 KiB +6.81 KiB
Logs - log message 101.45 KiB 98.45 KiB -3.00 KiB

🔗 RealWorld

@datadog-prod-us1-3
Copy link
Copy Markdown

datadog-prod-us1-3 Bot commented Apr 16, 2026

Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 77.01% (-0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6ad6464 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@BenoitZugmeyer BenoitZugmeyer changed the title 👷 improve e2e:bs reliability ✅ improve E2E test reliability on BrowserStack Apr 16, 2026
@BenoitZugmeyer BenoitZugmeyer marked this pull request as ready for review April 16, 2026 15:07
@BenoitZugmeyer BenoitZugmeyer requested a review from a team as a code owner April 16, 2026 15:07
@BenoitZugmeyer BenoitZugmeyer merged commit 16af0e2 into main Apr 16, 2026
24 checks passed
@BenoitZugmeyer BenoitZugmeyer deleted the benoit/e2e-bs-reliability branch April 16, 2026 15:09
@github-actions github-actions Bot locked and limited conversation to collaborators Apr 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants