fix(gates): the a11y family read only Vue, so a PHP-template app got a green over nothing (#225) - #261
Merged
Merged
Conversation
#242, #240) Gates 19 (e2e-coverage), 25 (contract-coverage), 62 (store-plane) and 63 (settings-surface) diff-scoped themselves INSIDE their own helpers, below the runner's base resolution, and did it UNCONDITIONALLY — the base ref was defaulted to origin/development even when the caller had asked for no scoping at all. Two consequences, and the second is why it stayed hidden: 1. A full-tree run — the mode a fleet audit uses — was silently narrowed to a diff against origin/development, which on a mainline checkout is empty. 2. The verdict for "I inspected nothing" was PASS, not a skip. So --require-full-coverage, the one assertion built to catch gates that did not run, had nothing to catch. gate-63 was the clearest case (#240): its log printed "gate skipped" on the line above a verdict that said PASS. Both cannot be true, and PASS is the one every consumer counted. Measured on openconnector 2026-08-08: gate-19 5 findings as the runner invoked it -> 412 over the full tree gate-25 PASS as the runner invoked it -> 32 over the full tree - the base ref is now the caller's decision and is never defaulted: set means diff-scoped, unset means full-tree audit - an empty scope returns a distinct status and the runner maps it to `_skip … structural`, which counts against coverage and fails --require-full-coverage - absent subject matter returns its own status and maps to `_skip … na` - gate-25 scopes on CONTROLLER files, not "any file": a docs-only diff opened no controller, and PASS there claimed a wire contract had been read when none was - gate-25 now returns a STATUS rather than the finding count, matching the convention gate-19 settled on in #209; the count is read from stdout Mutation-checked, three mutants, all killed: 1. pristine pre-fix runner — every arm red 2. the #242 defect reintroduced on gate-19 — ARM 1 red: the planted true positive stops being caught 3. ANTI-WIDENING CONTROL, checkers forced to inspect nothing — ARM 1 and ARM 3 red, so the suite cannot be satisfied by skipping everything
Follow-up to the empty-scope work, found by test_check_e2e_coverage.py. Adding the NOT APPLICABLE verdict introduced a regression of exactly the kind this issue is about: `openspec/specs` missing and the app dir being unreadable produce the SAME empty set, and the new code reported both as NOT APPLICABLE. That would retire the gate on the strength of a typo in a path. A missing directory is now an ERROR — a failure to look, not an absence of specs. Three tests updated, and two of them encoded the defect as an expectation: - test_pass_when_no_spec_files_in_diff asserted PASS for a repo with no specs. Renamed and now asserts NOT APPLICABLE: "I inspected nothing" and "I inspected everything and it was fine" cannot share a verdict. - test_diff_scope_only_changed_spec_flagged asserted PASS when the diff touched no spec. It now asserts EMPTY SCOPE, and STILL asserts the ADR-020 invariant it was written for — an untouched legacy spec is never flagged. - test_run_gate_raising_is_reported_as_ERROR monkeypatched changed_spec_files, which an unscoped run no longer calls. It now sets a base ref, so it exercises the path it claims to. Without this it would have passed for a reason unrelated to what it checks. All 105 gate-19 helper tests green; all 33 discovered helper suites green.
…a green over nothing (#225) Gates 31, 32, 34, 35, 36, 37, 39, 40, 42, 43, 44 and 45 enumerated `find src -name '*.vue'` and nothing else. An app that renders its UI from PHP templates had every one of those gates iterate an empty list and report PASS. The `[ -d src ]` guard did not save it. nldesign HAS a src/ — containing only manifest.json. The directory existed, the glob matched nothing, the loop ran zero times, and twelve gates printed PASS. Measured on nldesign, one textbook true positive planted per gate into templates/settings/ — an <img> with no alt, a positive tabindex, a focusable element inside aria-hidden="true", an icon-only <button>, an unlabelled <input>, a <table> with no <th>, "click here" link text, and the rest: before 0 of 12 gates caught their planted true positive after 12 of 12 Removing the plants then surfaced 8 GENUINE pre-existing findings in nldesign's real admin template (gates 40, 43 and 44) that no run had ever reported. WCAG does not care which templating language produced the DOM. - `_a11y_markup_files` enumerates src/, templates/ and appinfo/templates/ for .vue, .php, .html and .htm; node_modules / vendor / dist / build / coverage / phpmetrics are excluded, and the repo root is deliberately NOT scanned — nldesign ships a generated phpmetrics/*.html tree and auditing build output would manufacture findings nobody can act on - gates 34 and 36 grep templates/ as well as src/ - gate 35 accepts a literal `src="…"` carrying a semantic noun, not only a Vue binding, so `src="<?php p($avatarUrl) ?>" alt=""` is caught - gates 3, 12 and 13 stay .vue-only on purpose: a fetch*() stub, <NcSelect> and <NcModal> are Vue component patterns. Gate 40 already covers the language-agnostic label rule that gate 12 covers for NcSelect. Mutation-checked, both killed: A. pristine .vue-only runner — all 12 gates report "verdict: PASS", exactly the #225 symptom B. widening that REPLACES rather than ADDS (src/ dropped) — the .vue arm goes red The suite includes an ANTI-WIDENING CONTROL: a clean PHP template must raise no finding. Without it, "widen the glob" could be satisfied by flagging everything.
rubenvdlinde
added a commit
that referenced
this pull request
Aug 8, 2026
…the errexit invariant Resolved in favour of BOTH sides, not either: * KEPT from main (#225 / #261): gates 37 and 43 enumerate `_a11y_markup_files`, not `find src -name '*.vue'`, and guard on `_a11y_has_markup_dir`, not `[ -d src ]`. Taking my side wholesale would have silently REVERTED the glob fix for these two gates — the gates would have gone back to seeing nothing in a PHP-template app while reporting PASS. Both new helpers read markup rather than Vue specifically, so an `aria-hidden` focusable or an unscoped `<th>` is now caught in a .php template too. * KEPT from this branch: the tabindex="-1" and per-header-scope fixes, the tested helpers, and the SKIPPED-on-crash wiring. * DROPPED from this branch (#243): the `case $- in *e*) … set -e` restore dance. main established that errexit is OFF for the whole script and nothing may turn it on — a trailing `set -e` is an unconditional ENABLE, not a restore, which is the defect that left 45 gates running under an errexit they did not expect. Restore sites now say `set +e` only, and scripts/lib/test_gate_errexit_discipline.sh enforces it. My earlier workaround is obsolete; the underlying bug is fixed upstream. Verified after the merge: 38 helper suites pass, errexit discipline passes (64 verdicts for 64 declared gates, no abort banner under a crashing checker).
rubenvdlinde
added a commit
that referenced
this pull request
Aug 8, 2026
… errexit invariant Same resolution as the gate-37/43 merge: * KEPT from main (#225/#261): gate-39 enumerates _a11y_markup_files and guards on _a11y_has_markup_dir. Taking my side wholesale would have REVERTED the glob fix for this gate. check_button_name.py reads markup, not Vue specifically, so an unnamed <button> in a .php template is now caught too. * KEPT from this branch: bound-attribute acceptance, the quote-aware parser that could finally see 19 previously-invisible buttons, and the tested helper. * DROPPED (#243): the `set -e` restore dance. errexit is OFF for the whole script and nothing may turn it on; restore sites say `set +e`.
rubenvdlinde
added a commit
that referenced
this pull request
Aug 8, 2026
…adopt the errexit invariant TWO SCOPE DEFINITIONS BECOME ONE (#225 / #261). This arm had its own `find templates appinfo/templates -name '*.php'`, which is `_a11y_markup_files` minus its exclusions — so a generated `phpmetrics/` or a `vendor/` template would have been audited HERE and nowhere else in the a11y family. gate-38 now filters the shared enumeration to `*.php`. There is deliberately no third definition. ERREXIT (#243). Dropped the `case $- in *e*) … set -e` restore dance. main established that errexit is OFF for the whole script and nothing may turn it on — a trailing `set -e` is an unconditional ENABLE, not a restore. My workaround was written before that landed and is now obsolete; the underlying bug is fixed upstream, and test_gate_errexit_discipline.sh enforces the rule. Verified after the merge: skip-link control pairs 18/18, errexit discipline ALL PASS.
This was referenced Aug 8, 2026
rubenvdlinde
added a commit
that referenced
this pull request
Aug 8, 2026
…o full of markup, and three reported PASS over a crashed checker (#272) * fix(gates 35,40,42,44): four a11y gates excused themselves from a repo full of markup, and three reported PASS over a crashed checker Measured at package sha cdfbd7a against opencatalogi (93 .vue) and nldesign (zero .vue, one PHP template), one textbook true positive planted per gate in BOTH — the asymmetry that made #225/#261 possible. All 11 gates in the 34-44 band fired and named the plant in both arms, and all returned to their exact prior verdict on removal. Two defects survive that. 1. FOUR GATES GO `na` ON A TEMPLATES-ONLY REPO Gates 35, 40, 42 and 44 still guarded on `[ -d src ]` while 34/36/37/39/43 had moved to `_a11y_has_markup_dir`, and the central applicability table listed the whole family under `[ -d src ]`. On a repo with a `templates/` full of markup and no `src/`, same run, same files: gate-34/36/37/38/39/41/43 ran; four of them FAILED on the plants gate-35/40/42/44 NOT APPLICABLE — "this repo ships no frontend, so there is no .vue/.js/.ts source for this gate to inspect" `na` is the one verdict that removes a gate from coverage accounting, and the reason was contradicted by the same run's own output three lines above it. No fleet app is templates-only today; nldesign is one `rm` away, since its `src/` holds a single `manifest.json` — the exact shape that made twelve gates pass over nothing in #225. The guards now call `_a11y_has_markup_dir`, and the applicability declaration calls THE SAME FUNCTION rather than restating it, so the two cannot drift again. No third scope definition was added. 2. A CRASHED CHECKER REPORTED PASS (#147 / #249) — gates 40, 42, 44 With a `python3` on PATH that exits 1 on every call, run against opencatalogi: gate-40 PASS gate-42 PASS gate-44 PASS <- the three inline ones gate-34/37/38/39/41/43 SKIPPED (wiring) <- the six behind a helper gate-40 printed PASS over the 13 real findings it had reported one run earlier. gate-40 discarded its status with `2>/dev/null || true`; 42 and 44 ran per-file inline heredocs and never had one. 42 and 44 move to scripts/lib/check_link_text.py and scripts/lib/check_autocomplete.py — one interpreter for the whole file set, findings on stdout, exit code as a status — and 40 gains the same return-code guard. FOUND WHILE WRITING THE TESTS * gate-44 judged an input on the FIRST of name/id/v-model and stopped, so `<input id="e" type="text" name="email">` — the plainest textbook case this gate has — passed. Fleet effect, measured across 15 repos: openregister 0 -> 1 (an OpenAI Organization ID field), pipelinq 4 -> 5 (a "Colleague email" field). Both genuine, nothing lost. * gates 35, 36 and 44 read attribute values out of DOUBLE QUOTES ONLY. `tabindex='5'`, `alt=''` and `name='telephone'` render identically and reported PASS in both arms. Zero occurrences in the fleet today, which is why they could sit there indefinitely. * `[^>]*` in gates 42 and 44: a `>` inside an attribute value is not the end of a tag — the parse that hid 19 buttons from gate-39 (#259, #198, #236). * gates 42 and 44 scanned RAW text, so a commented-out `<a>click here</a>` or `<input name="email">` counted. That is gate-64's defect (#184), the one gate-38 (#247) and gate-41 (#266) each shipped a fix for. MEASURED AFTER, NOT ONLY BEFORE * 15 repos, gates 34-44, before vs after: every verdict and every finding count identical except the two new gate-44 true positives above. The rewrites of 42 and 44 removed nothing. * opencatalogi and nldesign return to their exact pre-plant baselines. * ARM 4 of test_gate_a11y_markup_scope.sh was mutation-checked: reverting gate-42's guard to `[ -d src ]` turns it red with the finding it was written for. TESTS * scripts/lib/test_check_link_text.py, test_check_autocomplete.py — 32 assertions; every relaxation ships with the true positive it must not swallow, comment/script exclusions ship with their positive control, and each ends with the whole PRE-FIX checker replayed as the mutant, asserting it answers DIFFERENTLY on every fixture. * test_gate_a11y_helper_wiring.sh gains gates 39, 40, 42, 44 (39 was wired correctly but never listed, so nothing held it to that) — 70 assertions. * test_gate_a11y_markup_scope.sh gains ARM 4, the templates-only repo. * Full discovered suite: 49 passed, 0 failed, 2 pre-existing quarantines. tests/test-hydra-gates-bin.sh: 59 passed, 0 failed. * fix(test): SC2194 — the case word was the constant, not the subject `case " 38 45 " in *" ${_g} "*)` matches a constant against a pattern built from the variable, which is the comparison written backwards. It happened to work, and ShellCheck is right that it reads as a mistake. Verified with shellcheck 0.10.0 at full severity: clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #225. Stacked on #258 — review that first; this branch contains its commits.
The defect
Gates 31, 32, 34, 35, 36, 37, 39, 40, 42, 43, 44, 45 enumerated
find src -name '*.vue'and nothing else. An app that renders its UI from PHP templates had every one of those gates iterate an empty list and report PASS.The
[ -d src ]guard did not save it. nldesign HAS asrc/— containing onlymanifest.json. The directory existed, the glob matched nothing, the loop ran zero times, and twelve gates printed PASS.The acceptance test: planted true positives in nldesign
One textbook true positive per gate, planted into
templates/settings/— an<img>with no alt, a positivetabindex, a focusable element insidearia-hidden="true", an icon-only<button>, an unlabelled<input>, a<table>with no<th>, "click here" link text, and the rest:Removing the plants then surfaced 8 genuine pre-existing findings in nldesign's real admin template — gate-40 (3 unlabelled inputs), gate-43 (2 tables with no
<th scope>), gate-44 (3 inputs with no autocomplete). None had ever been reported.Worth noting: two of my first plants were wrong, and the gates were right to ignore them — gate-35 requires an explicit
alt="", and gate-37 inspects the tag carryingaria-hidden, not its descendants. I corrected the plants rather than the gates.The fix
_a11y_markup_filesenumeratessrc/,templates/andappinfo/templates/for.vue,.php,.html,.htmnode_modules/vendor/dist/build/coverage/phpmetricsexcluded, and the repo root deliberately NOT scanned — nldesign ships a generatedphpmetrics/*.htmltree, and auditing build output would manufacture findings nobody can act ontemplates/as well assrc/src="…"carrying a semantic noun, not only a Vue binding, sosrc="<?php p($avatarUrl) ?>" alt=""is caught.vue-only on purpose — afetch*()stub,<NcSelect>and<NcModal>are Vue component patterns. Gate 40 already covers the language-agnostic label rule for PHP templates.WCAG does not care which templating language produced the DOM.
Mutation check — both killed
.vue-only runner — all 12 gates reportverdict: PASS, exactly the The a11y gate family scans src/**/*.vue only — a PHP-template app (nldesign) gets a green over an EMPTY SCOPE, and the PASS is indistinguishable from a measured zero #225 symptomsrc/dropped from the enumerator) — the.vuearm goes redThe suite carries an anti-widening control: a clean PHP template must raise no finding, and a
.vueviolation must still be caught. Without those, "widen the glob" could be satisfied by flagging everything, or by swapping one blind spot for another.Coordination
Deliberately does not touch gate-38 — that is #247's, which also adds
php_template_scope.py. Once #247 lands,_a11y_markup_filesand that helper should converge on one scope definition; I have kept mine to a shell function in the a11y section to keep the conflict surface to zero.🤖 Generated with Claude Code