fix(ci): count JUnit errors in Slack failure report

[gustavolira]

Summary

The Slack CI alert was reporting 0 tests failed with a :circleci-fail: failure emoji, which is contradictory and misleading.

Root cause: The JUnit XML spec distinguishes failures (assertion failures) from errors (exceptions/timeouts). Playwright reports TimeoutError and similar issues as errors, not failures. The old code in testing.sh only parsed the failures attribute:

failed_tests=$(grep -oP 'failures="\K[0-9]+' "${e2e_tests_dir}/${JUNIT_RESULTS}" | head -n 1)

So any test that failed due to a timeout or exception was silently uncounted, producing 0 tests failed on Slack even though tests actually failed.

Evidence from build 2061711989599637504:

• showcase JUnit: failures="0" errors="4" → Slack showed "0 tests failed" (should be 4)
• showcase-runtime JUnit: failures="0" errors="8" → Slack showed "0 tests failed" (should be 8)

Fix: Sum both failures and errors from the root <testsuites> element. Also adds a safety net: if Playwright exits non-zero but JUnit reports 0 of both, report "some" instead of the misleading "0".

Test plan

• Verified fix logic against actual JUnit XML from the failing build
• Next nightly run should show correct failure counts on Slack

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[rhdh-qodo-merge]

Code Review by Qodo

Sorry, something went wrong

We weren't able to complete the code review on our side. Please try again

[rhdh-qodo-merge]

Review Summary by Qodo

Fix Slack CI alert to count JUnit errors in failure report

🐞 Bug fix

Walkthroughs

Description

• Fix Slack CI alert reporting incorrect test failure counts
• Sum both JUnit failures and errors attributes
• Handle Playwright crashes/timeouts with safety fallback
• Playwright reports timeouts as errors, not failures

Diagram

flowchart LR
  A["Playwright Test Exit Code"] --> B{"Exit Code = 0?"}
  B -->|Yes| C["Report 0 failures"]
  B -->|No| D["Parse JUnit XML"]
  D --> E["Extract failures attribute"]
  D --> F["Extract errors attribute"]
  E --> G["Sum failures + errors"]
  F --> G
  G --> H{"Sum = 0?"}
  H -->|Yes| I["Report 'some'"]
  H -->|No| J["Report actual count"]
  I --> K["Slack Notification"]
  J --> K

Loading

File Changes

1. .ci/pipelines/lib/testing.sh 🐞 Bug fix +16/-2

Count both JUnit failures and errors in test report

• Extract both failures and errors from JUnit XML root element
• Sum both values to get accurate failed test count
• Add fallback to "some" when Playwright exits non-zero but JUnit reports 0 failures and 0 errors
• Add detailed comments explaining JUnit XML distinction between failures and errors

.ci/pipelines/lib/testing.sh

---

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.25%. Comparing base (c64c1dd) to head (e043cc6).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4912      +/-   ##
==========================================
- Coverage   55.82%   55.25%   -0.58%     
==========================================
  Files         121      109      -12     
  Lines        2350     2132     -218     
  Branches      562      513      -49     
==========================================
- Hits         1312     1178     -134     
+ Misses       1032      954      -78     
+ Partials        6        0       -6     

Flag	Coverage Δ
rhdh	55.25% <ø> (-0.58%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c64c1dd...e043cc6. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-4912
• quay.io/rhdh-community/rhdh:pr-4912-e043cc6c

[sanketpathak]

Thanks @gustavolira
/lgtm