[#664] Gated advisory accessibility violations out of JUnit failures.

[AlexSkrypnyk]

Closes #664

Summary

AccessibilityTrait::accessibilityRenderJunit() wrote a <failure> for every raw violation regardless of the scenario's threshold, while the pass/fail gate filtered the same violations through accessibilityFilterViolations(). As a result, a @accessibility:warning scenario (threshold never) passed in Behat but its JUnit report still declared failures, turning JUnit-consuming CI reporters red. The renderer now consults accessibilityEffectiveThreshold() and gates <failure> output through accessibilityFilterViolations(), so advisory findings are recorded as passing <testcase> elements with the finding in <system-out> instead of a <failure>.

Changes

• src/AccessibilityTrait.php: accessibilityRenderJunit() resolves the scenario's effective threshold and filters each violation through accessibilityFilterViolations() before deciding whether it becomes a <failure>. Violations below the threshold are still emitted as <testcase> elements carrying the finding in <system-out>, keeping them visible without failing the report. tests and failures are now accumulated per emitted <testcase> (one per affected node) rather than derived up front from count($violations) and count($passes), which also fixes a pre-existing mismatch for violations that affect multiple nodes.
• tests/behat/bootstrap/BehatCliContext.php: Added a file matching :pattern should contain: and should not contain: steps backed by a shared assertFileMatchingContains() helper. Glob matches are sorted before the first one is inspected, so the assertion target is deterministic when several files match the pattern.
• tests/behat/features/accessibility.feature: Added a @trait:AccessibilityTrait scenario that runs a nested @accessibility:warning Behat scenario and asserts the generated JUnit report has failures="0", contains no <failure> element, and still contains a <system-out> entry for the advisory finding.
• tests/phpunit/src/AccessibilityTraitTest.php: Added coverage for accessibilityRenderJunit() across thresholds (never/any/critical/serious/moderate), parsed with SimpleXML rather than substring matching, plus a dedicated test for per-node failure counting and one confirming advisory findings surface in <system-out> without a <failure>.

Before / After

BEFORE - @accessibility:warning scenario (threshold: never)
┌──────────────────────────────────────────────────────────────────────┐
│ Behat pass/fail gate      -> PASS (violations filtered by threshold) │
│ accessibilityRenderJunit() -> ignores the threshold entirely         │
│                                                                        │
│ <testsuites tests="5" failures="3">                                  │
│   <testsuite failures="3">                                           │
│     <testcase name="img.logo">                                       │
│       <failure type="critical" message="[critical] ...">...</failure>│
│     </testcase>                                                      │
│     ... 2 more <failure> cases                                       │
│   </testsuite>                                                       │
│ </testsuites>                                                        │
│                                                                        │
│ JUnit-consuming CI reporter -> FAILED (3 failures for a passing run)  │
└──────────────────────────────────────────────────────────────────────┘

AFTER - @accessibility:warning scenario (threshold: never)
┌──────────────────────────────────────────────────────────────────────┐
│ Behat pass/fail gate      -> PASS (violations filtered by threshold) │
│ accessibilityRenderJunit() -> filters through the same threshold via │
│                                accessibilityFilterViolations()        │
│                                                                        │
│ <testsuites tests="5" failures="0">                                  │
│   <testsuite failures="0">                                           │
│     <testcase name="img.logo">                                       │
│       <system-out>[advisory] Rule: image-alt ...</system-out>        │
│     </testcase>                                                      │
│     ... 2 more advisory <testcase> entries                           │
│   </testsuite>                                                       │
│ </testsuites>                                                        │
│                                                                        │
│ JUnit-consuming CI reporter -> PASSED (findings stay visible in      │
│ system-out; report stays green)                                      │
└──────────────────────────────────────────────────────────────────────┘

I’m checking the repo’s contribution rules first so I can flag any step-definition issues accurately in the summary, especially since you asked for Critical marking on violations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 3c70d310-91a2-4dec-a6a1-309dd619b9fb

📥 Commits

Reviewing files that changed from the base of the PR and between 148f5af and 4f1aad9.

📒 Files selected for processing (4)

• src/AccessibilityTrait.php
• tests/behat/bootstrap/BehatCliContext.php
• tests/behat/features/accessibility.feature
• tests/phpunit/src/AccessibilityTraitTest.php

---

Walkthrough

accessibilityRenderJunit() now filters violations per the scenario's effective threshold, emitting <failure> only for threshold-meeting violations while sub-threshold violations become passing testcases with <system-out> details. New Behat step definitions and PHPUnit/Behat tests validate this behavior.

Changes

Threshold-aware JUnit rendering

Layer / File(s)	Summary
Threshold-based failure filtering src/AccessibilityTrait.php	accessibilityRenderJunit() fetches the effective threshold, filters each violation via accessibilityFilterViolations(), and emits failing testcases with <failure> or advisory testcases with <system-out>, updating per-violation test/failure counts.
PHPUnit coverage for threshold rendering tests/phpunit/src/AccessibilityTraitTest.php	New parameterized tests verify failure counts by threshold, advisory-mode rendering, and stable testcase counts; adds createJunitResults() fixture and updates testRenderJunit() to accept an optional threshold.
Behat file-content assertions and warning-mode scenario tests/behat/bootstrap/BehatCliContext.php, tests/behat/features/accessibility.feature	Adds #[Then] steps to assert substring presence/absence in glob-matched files, and a new feature scenario verifying warning-mode runs produce a JUnit report with failures="0" and <system-out> content.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• drevops/behat-steps#643: Introduced the axe-based JUnit reporting/gating behavior that this PR's threshold-aware failure logic builds on.
• drevops/behat-steps#660: Also modifies accessibilityRenderJunit(), changing URL formatting in JUnit details alongside this PR's failure/testcase counting changes.
• drevops/behat-steps#661: Also modifies accessibilityRenderJunit() testcase/failure serialization concurrently with this PR.

Suggested labels: Needs review

Poem

A rabbit checked the JUnit log so neat,
"No more false failures, that's a treat!"
Advisory hops now sit calm and mild,
While real failures stay stern and riled,
🐇 green checks bloom, the report's complete!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: preventing advisory accessibility violations from becoming JUnit failures.
Linked Issues check	✅ Passed	The renderer now gates JUnit failures by the effective threshold and preserves advisory violations as informational output, matching #664.
Out of Scope Changes check	✅ Passed	The added Behat and PHPUnit test/support changes are directly tied to validating the JUnit threshold fix and do not appear unrelated.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/664-a11y-advisory-junit

---

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.85%. Comparing base (148f5af) to head (4d4405f).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #666   +/-   ##
=======================================
  Coverage   96.85%   96.85%           
=======================================
  Files          46       46           
  Lines        3817     3819    +2     
=======================================
+ Hits         3697     3699    +2     
  Misses        120      120           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.