CONSOLE-5306: Surface Playwright report link in Prow Spyglass

[rhamilto]

Analysis / Root cause:
The Playwright HTML report URL printed in CI logs used a wildcard (*) for the ci-operator step name, which doesn't resolve in a browser. Additionally, the Spyglass JUnit lens renders <nil> for failed tests that have no error details (e.g. tests that never executed due to a prior timeout). Prow Spyglass also showed duplicate Flaky Tests and Failed Tests sections because JUnit XML files in the bulk-copied playwright-test-results/ subdirectory were being picked up alongside the intended top-level junit-playwright.xml.

Solution description:

• Derive the ci-operator step name from JOB_NAME and PULL_BASE_REF to build a direct, resolvable URL to the Playwright HTML report.
• Write a custom-link-playwright-report.html artifact that the Spyglass html lens renders as a clickable link on the Prow job page.
• Ensure <failure> elements always have text content to prevent the Spyglass JUnit lens from rendering <nil>.
• Rename the standard JUnit copy from junit-playwright-standard.xml to playwright-standard-junit.xml so it doesn't match Prow's junit*.xml glob.
• Remove *.xml files from the bulk playwright-test-results/ copy to prevent Prow from recursively discovering duplicate JUnit files.
• Style the Spyglass report link with font-size: 14px and color: #ff9999.

Screenshots / screen recording:

Screen.Recording.2026-07-09.at.9.04.39.PM.mov

Test setup:
Changes only affect CI reporter output. Verify on the next Prow Playwright job run.

Test cases:

• Verify the Playwright HTML Report link appears as a Spyglass panel on the Prow job page
• Verify the link resolves to the correct playwright-report/index.html
• Verify failed tests with no error details show "No error details available" instead of <nil>
• Verify only a single Flaky Tests / Failed Tests section appears in Spyglass (no duplicates from playwright-test-results/ subdirectory)

Browser conformance:

• Chrome
• Firefox
• Safari (or Epiphany on Linux)

Additional info:
N/A

Reviewers and assignees:

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@rhamilto: This pull request references CONSOLE-5306 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Analysis / Root cause:
The Playwright HTML report URL printed in CI logs used a wildcard (*) for the ci-operator step name, which doesn't resolve in a browser. Additionally, the Spyglass JUnit lens renders <nil> for failed tests that have no error details (e.g. tests that never executed due to a prior timeout).

Solution description:

• Derive the ci-operator step name from JOB_NAME and PULL_BASE_REF to build a direct, resolvable URL to the Playwright HTML report.
• Write a custom-link-playwright-report.html artifact that the Spyglass html lens renders as a clickable link on the Prow job page.
• Ensure <failure> elements always have text content to prevent the Spyglass JUnit lens from rendering <nil>.

Screenshots / screen recording:

Test setup:
Changes only affect CI reporter output. Verify on the next Prow Playwright job run.

Test cases:

• Verify the Playwright HTML Report link appears as a Spyglass panel on the Prow job page
• Verify the link resolves to the correct playwright-report/index.html
• Verify failed tests with no error details show "No error details available" instead of <nil>

Browser conformance:

• Chrome
• Firefox
• Safari (or Epiphany on Linux)

Additional info:
N/A

Reviewers and assignees:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

The Playwright JUnit reporter now derives step-specific report links from CI variables, writes a custom HTML link next to JUnit output, normalizes empty failure text, and the integration test script copies the standard JUnit report under a new artifact name.

Changes

Playwright reporting updates

Layer / File(s)	Summary
Derived report URL and link file frontend/e2e/reporters/prow-junit-reporter.ts	Derives an optional step name from CI variables, builds a step-specific Playwright report URL when available, writes custom-link-playwright-report.html beside the JUnit output, and prints the computed report URL in the summary.
Failure text normalization frontend/e2e/reporters/prow-junit-reporter.ts	Strips ANSI codes from failed-test error text and falls back to No error details available when the extracted text is empty.
JUnit artifact copy name frontend/integration-tests/test-playwright-e2e.sh	Removes XML files from the copied Playwright test-results directory and copies the standalone JUnit file to playwright-standard-junit.xml.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• openshift/console#16501: Both PRs change the Playwright JUnit reporter and the integration-test artifact naming used for Prow-style reporting.

Suggested reviewers: spadgett

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (14 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	No Ginkgo test titles were added or modified in the touched files; the PR only changes reporter/output scripts, so there are no dynamic test names to flag.
Test Structure And Quality	✅ Passed	PR changes only a TypeScript reporter and a shell artifact script; no Ginkgo test code or cluster-wait patterns are present to review.
Microshift Test Compatibility	✅ Passed	No new Ginkgo specs were added; the diff only changes reporter/artifact copying, with no new MicroShift-unsupported APIs or assumptions introduced.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; the PR only changes reporter/artifact handling, so there are no SNO-specific multi-node assumptions to review.
Topology-Aware Scheduling Compatibility	✅ Passed	Only Playwright reporter and artifact-copy scripts changed; no deployment manifests, controllers, node selectors, affinity, PDBs, or replica logic were introduced.
Ote Binary Stdout Contract	✅ Passed	PASS: The PR only changes a Playwright reporter and shell artifact-copy script; it adds no Go main/init/TestMain/top-level stdout writes or other process-level JSON stdout regressions.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added in the changed files; the diff only updates reporter/artifact handling, so this IPv6/disconnected check is not applicable.
No-Weak-Crypto	✅ Passed	Reporter/script changes are URL and artifact handling only; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons found.
Container-Privileges	✅ Passed	The PR only changes a TS reporter and shell script; no container/K8s manifests or privilege flags were added.
No-Sensitive-Data-In-Logs	✅ Passed	No credentials/PII were added to logs; new output only prints artifact paths and a Playwright report URL.
Title check	✅ Passed	The title is concise, Jira-prefixed, and accurately summarizes surfacing the Playwright report link in Prow Spyglass.
Description check	✅ Passed	The description covers the required template sections and gives enough detail on root cause, solution, tests, and browser conformance.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[rhamilto]

/test e2e-playwright

[rhamilto]

/test e2e-playwright

[coderabbitai]

🧹 Nitpick comments (1)

frontend/integration-tests/test-playwright-e2e.sh (1)

117-118: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Align the artifact naming convention. playwright-standard-junit.xml is the only filename here that uses the *-junit suffix, while the sibling artifact still uses the junit-* prefix. Keeping both on one convention would make the artifact set easier to scan.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@frontend/integration-tests/test-playwright-e2e.sh` around lines 117 - 118,
The artifact naming in the Playwright test script is inconsistent because the
standard JUnit file uses a different naming pattern than the sibling artifact.
Update the copy target in the test-results handling block of the shell script so
it follows the same convention as the other JUnit artifact, and keep the echoed
message in sync with the new filename.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@frontend/integration-tests/test-playwright-e2e.sh`:
- Around line 117-118: The artifact naming in the Playwright test script is
inconsistent because the standard JUnit file uses a different naming pattern
than the sibling artifact. Update the copy target in the test-results handling
block of the shell script so it follows the same convention as the other JUnit
artifact, and keep the echoed message in sync with the new filename.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 27b5d73b-0728-41dc-8b14-c53485cf6290

📥 Commits

Reviewing files that changed from the base of the PR and between f14e240 and 18e47da.

📒 Files selected for processing (2)

• frontend/e2e/reporters/prow-junit-reporter.ts
• frontend/integration-tests/test-playwright-e2e.sh

🚧 Files skipped from review as they are similar to previous changes (1)

• frontend/e2e/reporters/prow-junit-reporter.ts

[rhamilto]

/test e2e-playwright

[rhamilto]

/test e2e-playwright

[rhamilto]

/test e2e-gcp-console
/assign @jhadvig
/approve
/label px-approved
/label docs-approved
/verified by CI

[openshift-ci-robot]

@rhamilto: This PR has been marked as verified by CI.

Details

In response to this:

/test e2e-gcp-console
/assign @jhadvig
/approve
/label px-approved
/label docs-approved
/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhamilto

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• frontend/OWNERS [rhamilto]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@rhamilto: This pull request references CONSOLE-5306 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Analysis / Root cause:
The Playwright HTML report URL printed in CI logs used a wildcard (*) for the ci-operator step name, which doesn't resolve in a browser. Additionally, the Spyglass JUnit lens renders <nil> for failed tests that have no error details (e.g. tests that never executed due to a prior timeout).

Solution description:

• Derive the ci-operator step name from JOB_NAME and PULL_BASE_REF to build a direct, resolvable URL to the Playwright HTML report.
• Write a custom-link-playwright-report.html artifact that the Spyglass html lens renders as a clickable link on the Prow job page.
• Ensure <failure> elements always have text content to prevent the Spyglass JUnit lens from rendering <nil>.

Screenshots / screen recording:

Screen.Recording.2026-07-09.at.9.04.39.PM.mov

Test setup:
Changes only affect CI reporter output. Verify on the next Prow Playwright job run.

Test cases:

• Verify the Playwright HTML Report link appears as a Spyglass panel on the Prow job page
• Verify the link resolves to the correct playwright-report/index.html
• Verify failed tests with no error details show "No error details available" instead of <nil>

Browser conformance:

• Chrome
• Firefox
• Safari (or Epiphany on Linux)

Additional info:
N/A

Reviewers and assignees:

Summary by CodeRabbit

• New Features
• Added a companion link file next to the generated Playwright HTML report artifacts.
• When CI job details are available, the Playwright HTML report link is now step-specific for easier navigation.
• Bug Fixes
• Improved failure text handling: if error details are empty after cleanup, a clearer “no details available” message is shown.
• Corrected the naming/copying of the standard Playwright JUnit XML artifact to the expected destination.
• Prevented duplicate XML files from being picked up during artifact collection.
• Other Changes
• Simplified the console summary for the Playwright HTML Report to show only the final report URL.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@rhamilto: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.