fix(gates 35,40,42,44): four a11y gates excused themselves from a repo full of markup, and three reported PASS over a crashed checker - #272
Merged
rubenvdlinde merged 2 commits intoAug 8, 2026
Conversation
…o full of markup, and three reported PASS over a crashed checker Measured at package sha cdfbd7a against opencatalogi (93 .vue) and nldesign (zero .vue, one PHP template), one textbook true positive planted per gate in BOTH — the asymmetry that made #225/#261 possible. All 11 gates in the 34-44 band fired and named the plant in both arms, and all returned to their exact prior verdict on removal. Two defects survive that. 1. FOUR GATES GO `na` ON A TEMPLATES-ONLY REPO Gates 35, 40, 42 and 44 still guarded on `[ -d src ]` while 34/36/37/39/43 had moved to `_a11y_has_markup_dir`, and the central applicability table listed the whole family under `[ -d src ]`. On a repo with a `templates/` full of markup and no `src/`, same run, same files: gate-34/36/37/38/39/41/43 ran; four of them FAILED on the plants gate-35/40/42/44 NOT APPLICABLE — "this repo ships no frontend, so there is no .vue/.js/.ts source for this gate to inspect" `na` is the one verdict that removes a gate from coverage accounting, and the reason was contradicted by the same run's own output three lines above it. No fleet app is templates-only today; nldesign is one `rm` away, since its `src/` holds a single `manifest.json` — the exact shape that made twelve gates pass over nothing in #225. The guards now call `_a11y_has_markup_dir`, and the applicability declaration calls THE SAME FUNCTION rather than restating it, so the two cannot drift again. No third scope definition was added. 2. A CRASHED CHECKER REPORTED PASS (#147 / #249) — gates 40, 42, 44 With a `python3` on PATH that exits 1 on every call, run against opencatalogi: gate-40 PASS gate-42 PASS gate-44 PASS <- the three inline ones gate-34/37/38/39/41/43 SKIPPED (wiring) <- the six behind a helper gate-40 printed PASS over the 13 real findings it had reported one run earlier. gate-40 discarded its status with `2>/dev/null || true`; 42 and 44 ran per-file inline heredocs and never had one. 42 and 44 move to scripts/lib/check_link_text.py and scripts/lib/check_autocomplete.py — one interpreter for the whole file set, findings on stdout, exit code as a status — and 40 gains the same return-code guard. FOUND WHILE WRITING THE TESTS * gate-44 judged an input on the FIRST of name/id/v-model and stopped, so `<input id="e" type="text" name="email">` — the plainest textbook case this gate has — passed. Fleet effect, measured across 15 repos: openregister 0 -> 1 (an OpenAI Organization ID field), pipelinq 4 -> 5 (a "Colleague email" field). Both genuine, nothing lost. * gates 35, 36 and 44 read attribute values out of DOUBLE QUOTES ONLY. `tabindex='5'`, `alt=''` and `name='telephone'` render identically and reported PASS in both arms. Zero occurrences in the fleet today, which is why they could sit there indefinitely. * `[^>]*` in gates 42 and 44: a `>` inside an attribute value is not the end of a tag — the parse that hid 19 buttons from gate-39 (#259, #198, #236). * gates 42 and 44 scanned RAW text, so a commented-out `<a>click here</a>` or `<input name="email">` counted. That is gate-64's defect (#184), the one gate-38 (#247) and gate-41 (#266) each shipped a fix for. MEASURED AFTER, NOT ONLY BEFORE * 15 repos, gates 34-44, before vs after: every verdict and every finding count identical except the two new gate-44 true positives above. The rewrites of 42 and 44 removed nothing. * opencatalogi and nldesign return to their exact pre-plant baselines. * ARM 4 of test_gate_a11y_markup_scope.sh was mutation-checked: reverting gate-42's guard to `[ -d src ]` turns it red with the finding it was written for. TESTS * scripts/lib/test_check_link_text.py, test_check_autocomplete.py — 32 assertions; every relaxation ships with the true positive it must not swallow, comment/script exclusions ship with their positive control, and each ends with the whole PRE-FIX checker replayed as the mutant, asserting it answers DIFFERENTLY on every fixture. * test_gate_a11y_helper_wiring.sh gains gates 39, 40, 42, 44 (39 was wired correctly but never listed, so nothing held it to that) — 70 assertions. * test_gate_a11y_markup_scope.sh gains ARM 4, the templates-only repo. * Full discovered suite: 49 passed, 0 failed, 2 pre-existing quarantines. tests/test-hydra-gates-bin.sh: 59 passed, 0 failed.
`case " 38 45 " in *" ${_g} "*)` matches a constant against a pattern
built from the variable, which is the comparison written backwards. It
happened to work, and ShellCheck is right that it reads as a mistake.
Verified with shellcheck 0.10.0 at full severity: clean.
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The acceptance test for gates 34–44 (the accessibility band), run at package sha
cdfbd7a: plant one textbook true positive per gate in both a.vueapp (opencatalogi, 93.vue) and a PHP-template app (nldesign, zero.vue, one template) — the asymmetry that made #225/#261 possible — require the gate to FAIL and name the plant in both, then remove it and require the prior verdict back.All 11 gates fired and named the plant in all 22 arms, and all 22 returned to baseline exactly. The
_a11y_markup_filesfix holds. Two defects survive that test, and neither is visible from it.1. Four gates declare themselves NOT APPLICABLE over a repo full of markup
Gates 35, 40, 42, 44 still guarded on
[ -d src ]while 34/36/37/39/43 had moved to_a11y_has_markup_dir, and the central applicability table listed the whole family under[ -d src ]. On a repo with atemplates/full of markup and nosrc/— same run, same files:nais the one verdict that removes a gate from coverage accounting, and the reason is contradicted by the same run's own output three lines above it. The a11y family has read.phpand.htmlsince #261; the declaration never caught up.No fleet app is templates-only today. nldesign is one
rmaway — itssrc/holds a singlemanifest.json, the exact shape that made twelve gates pass over nothing in #225.The four guards now call
_a11y_has_markup_dir, and the applicability declaration calls the same function instead of restating it, so the two cannot drift again. No third scope definition was added (#247/#261).2. A crashed checker reported PASS — gates 40, 42, 44 (#147 / #249)
With a
python3on PATH that exits 1 on every invocation, against opencatalogi:gate-40 discarded its status with
2>/dev/null || true; 42 and 44 ran per-file inline heredocs and never had one. 42 and 44 move intocheck_link_text.py/check_autocomplete.py— one interpreter for the whole file set, findings on stdout, exit code as a status — and 40 gains the same guard.Found while writing the tests
name/id/v-modeland stopped, so<input id="e" type="text" name="email">— the plainest textbook case this gate has — passed. Fleet effect, measured over 15 repos: openregister 0 → 1 (an OpenAI Organization ID field), pipelinq 4 → 5 (a "Colleague email" field). Both genuine.tabindex='5',alt='',name='telephone'render identically and reported PASS in both arms. Zero fleet occurrences today — which is why they could sit there indefinitely.[^>]*in gates 42 and 44 — a>inside an attribute value is not the end of a tag; the parse that hid 19 buttons from gate-39 (fix(gate-39): a BOUND name is still a name, and[^>]*could not see 19 buttons #259, fix(gate-9): the admin rule matched no real isAdmin() call, and prose bought the self-auth exemption #198, gates 12/32/38: two text-extraction false positives and one that scores a comment #236).<a>click here</a>counted. gate-64's defect (fix(hydra-gates): gate-64 read spelling, not code — 2 false REDs and 1 false GREEN #184), the one fix(gate-38): a mount point is not a page root, and the remedy it demanded regressed a11y (#214, #216, #227) #247 and gate-41 (html-lang) has gate-38's defect verbatim: a PHP COMMENT mentioning <html> makes a mount point a page root #266 each shipped a fix for.Measured after, not only before
Tightening a gate creates its own false positives, so the after-measure is the load-bearing one:
[ -d src ]turns it red with the finding it was written for. A check that cannot fail is not a check.What the controls confirmed still holds
Verified independently, not assumed from the PR that claimed them: gate-34 both arms (#269 — the comment does not fire,
window['confirm']()does) · gate-37tabindex="-1"is not focusable (#251) · gate-38/41 a PHP comment mentioning<html>does not make a mount point a page root, and a commented-out<html lang>does not satisfy a real unlangged one (#247/#266) · gate-39 a bound:title/:aria-labelledbyis a name, and a>inside an attribute does not hide the button (#259) · gate-40 a slottedNcCheckboxRadioSwitchis not flagged, a self-closing one is (#251) · gate-43 anaria-hiddenspacer<th>is not flagged (the 8 false openconnector findings).One finding NOT fixed here, reported instead
gate-40 flags
<input type="file" aria-hidden="true" tabindex="-1">— the exact element gate-37's #222 fix declared canonical and correct. Two gates in the same band contradict each other, andaria-labelon anaria-hiddenelement is inert, so the finding cannot be closed honestly. Zero fleet occurrences today (nldesign's three aredisplay:none, notaria-hidden), and excludingdisplay:nonewould be blinding — a style can be toggled by JS. Filing rather than guessing.Tests
test_check_link_text.py,test_check_autocomplete.py— 32 assertions. Every relaxation ships with the true positive it must not swallow; each comment/script exclusion ships with its positive control; each file ends with the whole pre-fix checker replayed as the mutant, asserting it answers differently on every fixture.test_gate_a11y_helper_wiring.shgains gates 39, 40, 42, 44 — 39 was wired correctly but never listed, so nothing held it to that. 70 assertions.test_gate_a11y_markup_scope.shgains ARM 4, the templates-only repo.test-hydra-gates-bin.sh59 passed / 0 failed.🤖 Generated with Claude Code