fix(playwright): preserve late failure screenshots - #9896
Conversation
Overall package sizeSelf size: 8.28 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 445.14 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 6d2232a | Docs | View more details | Give us feedback! |
BenchmarksBenchmark execution time: 2026-08-21 10:49:05 Comparing candidate commit 6d2232a in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2285 metrics, 7 unstable metrics.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9896 +/- ##
========================================
Coverage 98.55% 98.55%
========================================
Files 983 983
Lines 145311 145551 +240
Branches 12565 12864 +299
========================================
+ Hits 143206 143448 +242
+ Misses 2105 2103 -2 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…288-playwright-screenshot-handoff
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2fa88dc6cd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![datadog-prod-us1-4[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
The new fixture calls a saved Array.push method without its attachment array receiver. The deferred screenshot case throws before it tests the new handoff.
🤖 Datadog Autotest · Commit 2fa88dc · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
@codex review |
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9f0b8d440
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
A restored disabled test reserves a screenshot FIFO slot but sends no worker trace. This shifts later screenshot data and can remove or misassign a failure screenshot.
🤖 Datadog Autotest · Commit f9f0b8d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
More details
The PR keeps stable per-test screenshot arrays,
aligns them with worker traces,
and excludes expected skips and skipped EFD retries.
The new commit fixes the detached Array.push call,
so no open finding remains.
🤖 Datadog Autotest · Commit f9f0b8d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
@codex review |
|
/datadog autotest review |
There was a problem hiding this comment.
More details
This PR keeps late Playwright failure screenshots until the worker trace uses them.
The new commit excludes restored disabled tests from the screenshot queue.
This change fixes the earlier queue shift that could lose or misassign a later screenshot.
It also adds direct coverage for a failure after a disabled serial-retry sibling.
No open finding remains in the current PR diff.
🤖 Datadog Autotest · Commit 69d8d55 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
More details
The PR preserves late Playwright failure screenshots.
It aligns screenshot arrays with worker traces.
It excludes expected skips, skipped EFD retries, and disabled serial retries.
The earlier detached push defect is fixed.
No open finding remains.
🤖 Datadog Autotest · Commit 69d8d55 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
BridgeAR
left a comment
BridgeAR
left a comment
There was a problem hiding this comment.
While I do not know the domain parts, the code is LGTM
Just the skip part would be nice to land
|
@codex review |
|
/datadog autotest review |
There was a problem hiding this comment.
More details
The PR keeps late Playwright failure screenshots until the worker trace consumes them.
It keeps the screenshot queue aligned when EFD or restored disabled tests do not emit worker traces.
The prior disabled-test queue issue remains fixed and has direct serial-retry coverage.
The latest commit registers two Playwright 1.60 tests as explicit skips on unsupported versions.
No open finding remains.
🤖 Datadog Autotest · Commit 6d2232a · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
More details
The PR preserves late Playwright failure screenshots.
It aligns screenshot arrays with worker traces.
It excludes expected skips, skipped EFD retries, and disabled serial retries.
The earlier detached push defect remains fixed.
The latest test registration change adds no new issue, and no open finding remains.
🤖 Datadog Autotest · Commit 6d2232a · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
3 participants
What does this PR do?
Preserves Playwright automatic failure screenshot attachments when the marked
attachevent reaches the dispatcher aftertestEndbut before the worker trace is consumed.The handoff now keeps a stable per-test attachment array and reserves an aligned FIFO slot for every worker-emitted test trace. Expected skips remain excluded because their spans are created in the main process and do not emit worker traces.
Motivation
There is a report of an intermittent failed Playwright tests where Playwright wrote
test-failed-1.png, but Test Optimization received neithertest.failure_screenshot.uploadednortest.failure_screenshot.upload_error.Previously, the dispatcher deleted screenshot state at
testEndand only queued attachments that had already arrived. A marked automatic attachment delivered just after that boundary was silently lost before upload. This is independent of RUM correlation.This PR hardens that vulnerable boundary. Whether deferred attachment delivery is the customer's exact trigger is still under investigation.
Additional Notes
The regression uses only Playwright test fixtures: a test-scoped fixture defers the automatic screenshot attachment, and a worker-scoped fixture releases it from worker teardown. It does not inspect or reorder Playwright IPC or Datadog's private attachment marker.
Against the pre-fix production path, the focused regression fails with the upload tag absent. After the change:
PLAYWRIGHT_VERSION=latest ./node_modules/.bin/mocha --timeout 180000 integration-tests/playwright/playwright-reporting.spec.js --grep 'uploads a failure screenshot deferred by test code'— 1 passingPLAYWRIGHT_VERSION=latest ./node_modules/.bin/mocha --timeout 180000 integration-tests/playwright/playwright-reporting.spec.js --grep "uploads only automatic failure screenshots with screenshot: 'only-on-failure'"— 1 passing./node_modules/.bin/eslint packages/datadog-instrumentations/src/playwright.js integration-tests/ci-visibility/playwright-tests-screenshot/failure-screenshot-test.js integration-tests/playwright/playwright-reporting.spec.js— passing