hydra-gates: an EMPTY ADR-020 diff scope is classified 'structural', so require-full-coverage fails every PR that touches no spec and no manifest (exit 98)

[rubenvdlinde]

Summary

Since c1f7deb (merge of #258, main at 2026-08-08T14:21:55Z), gates 19 / 25 / 62 / 63 classify an empty ADR-020 diff scope as structural. structural counts against --require-full-coverage, which is ON by default. The result: any PR whose diff happens not to touch a spec file and not to touch a manifest/menu-layout fails with exit 98, with zero gate findings and nothing wrong with the code.

This is not the same as #169 (gates 4/24/33, subject matter genuinely absent from the repo). Here the subject matter is present, clean, and passes when the gate is allowed to look at it.

Reproduction

Repo: ConductionNL/doriath, PR #191 — a 3-file change (one controller + two new unit test files). Every gate that ran passed.

##[error]hydra-gates passed every gate that ran, but a gate whose SUBJECT MATTER EXISTS
did not report, and hydra-gates-require-full-coverage is set.
GATES THAT DID NOT RUN: 19 62 63

Reproduced locally against hydra-gates @ 846baed:

$ bash .../run-hydra-gates.sh --require-full-coverage --scope-to-diff origin/development .
[gate-19] e2e-coverage:     SKIPPED (structural) — the diff touched NO spec file …
[gate-62] store-plane:      SKIPPED (structural) — the diff touched no manifest or menu-layout …
[gate-63] settings-surface: SKIPPED (structural) — the diff touched no manifest or menu-layout …
[hydra-gates] 57 GATE(S) GREEN — but 3 of 60 APPLICABLE gates DID NOT RUN
EXIT=98

The subject matter exists, is inspectable, and is clean

Run gate-63's own checker on the same tree with and without --base:

$ python3 scripts/lib/check_store_and_settings_surface.py . --gate settings
rc=0
checked 2 manifest(s): 0 failure(s), 0 warning(s).

$ python3 scripts/lib/check_store_and_settings_surface.py . --gate settings --base origin/development
rc=3
No changed manifest / menu-layout — EMPTY SCOPE (ADR-020 diff scoping against 'origin/development').
2 manifest(s) exist here and NONE were inspected; this gate's placement rules are UNVERIFIED by this run.

The gate did not fail to run. It ran, found the diff filter empty, and returned rc=3. Nothing is missing, broken or unmeasurable — ADR-020 scoping simply excluded it, which is the entire point of ADR-020.

Why structural is the wrong category here

The runner's own definitions (run-hydra-gates.sh, _skip):

structural  The subject matter EXISTS and nothing produced the gate's input.
            An app with src/ that ships no axe report has runtime
            accessibility defects it has not looked for. A real gap.

…and the header it prints for na:

[hydra-gates] NOT APPLICABLE — subject matter absent from this repo or this diff.

"Absent from this diff" is exactly the rc=3 case, and the runner already advertises that as na. structural was written for the axe case — an input the repo should have produced and didn't. A PR that legitimately doesn't touch a manifest has not failed to produce anything.

The substance of #258 is right — gates 62/63 previously printed PASS over an empty scope, which is a false green (confirmed: at 68254af the same tree reports [gate-62] PASS / [gate-63] PASS with nothing inspected). The regression is only in the category chosen for rc=3, and its interaction with --require-full-coverage.

Control: development is red too — the green it is compared against is a stale measurement

doriath's development tip 4bfeab7c8 was gated success at 14:21:47Z. The classification change landed on main at 14:21:55Z — 8 seconds later. hydra-gates-ref defaults to main, so the two runs are different programs, not different code.

Same commit, zero changes, today's package, faithful push scope:

$ cd <worktree @ 4bfeab7c8>
$ HYDRA_GATE_PUSH_BEFORE=b11ee90be bash .../run-hydra-gates.sh --require-full-coverage --scope-to-diff .
[hydra-gates] SCOPE-FILE-COUNT: 2        # playwright.config.ts, tests/e2e/playwright.config.ts
[gate-19] e2e-coverage:      SKIPPED (structural)
[gate-25] contract-coverage: SKIPPED (structural)
[gate-62] store-plane:       SKIPPED (structural)
[gate-63] settings-surface:  SKIPPED (structural)
EXIT=98

So the branch every PR is compared against does not itself pass. Any strict-subset merge rule reading "green on development, red on the PR" is reading a 30-second-old build of a different program.

Measured blast radius

Fleet PR runs created after 14:21:55Z whose Hydra Gates job failed, classified by annotation:

repo	run	cause
doriath	31266516836	coverage-98
doriath	31266161406	coverage-98
larpingapp	31265829882	coverage-98
softwarecatalog	31266322853	coverage-98
opencatalogi	31266489179	real finding
pipelinq	31262984939	real finding
larpingapp	31266504079	real finding
softwarecatalog	31264780566	real finding
openconnector	31262941378	real finding

4 runs across 3 repos blocked on nothing. It is diff-dependent, not repo-dependent: a PR is blocked purely by whether it happened to touch a spec file and a manifest.

There is no honest workaround in the consuming repo

• gate-19 wants a changed spec scenario.
• gates 62/63 want a changed manifest or menu-layout.

A bugfix PR has no legitimate reason to edit src/manifest.json. Editing one to make a gate run is manufacturing the gate's input — the exact false-green this package exists to prevent. The only lever left in the repo is hydra-gates-require-full-coverage: false, which switches the control off fleet-wide. Both are worse than the bug.

Suggested fix (for the maintainer to weigh)

Map "empty ADR-020 diff scope" to na rather than structural, for every diff-scoped gate (19, 25, 62, 63 at minimum), while keeping #258's real fix — never print PASS over a scope that was never opened. A third category (e.g. scoped-out: reported, excluded from the coverage verdict, never rendered as PASS) would preserve both properties and keep the distinction visible in the output.

Whatever the shape, the invariant worth pinning with a test: a gate whose checker returns rc=0 when run unscoped must never fail a run merely because the diff did not reach it.

[rubenvdlinde]

Fresh reproduction today on doriath, still live.

doriath#191 is a controller-only fix (lib/Controller/SettingsController.php plus two new unit test files). It touches no spec, no manifest, no menu-layout. quality / Hydra Gates fails with:

##[error]hydra-gates passed every gate that ran, but a gate whose SUBJECT MATTER EXISTS
did not report, and hydra-gates-require-full-coverage is set.
GATES THAT DID NOT RUN: 19 62 63

The three skips are all structural, and each is correct on its own terms:

[gate-19] e2e-coverage:     SKIPPED (structural) - the diff touched NO spec file
[gate-62] store-plane:      SKIPPED (structural) - the diff touched no manifest or menu-layout
[gate-63] settings-surface: SKIPPED (structural) - the diff touched no manifest or menu-layout

Every gate that ran passed. Nothing is wrong with the PR.

Why this one is worth noting: gate-63 is settings-surface, and the PR changes the settings controller. The gate whose name matches the change most closely is the one that declined to inspect it - it keys off the manifest / menu-layout (ADR-079 placement), not off SettingsController.php. Coverage enforcement then reads that decline as a gap.

The baseline it is compared against is not a real green. doriath's development run passed, but its own log says why:

NOT APPLICABLE - scope was empty - 0 lib/Controller PHP file(s) in this diff,
so NOTHING was inspected

development was green because that diff contained no controllers; the PR is red because it does. Under the fleet's "failing job names must be a strict subset of development's" merge rule that is a new failing name, so the PR is unmergeable - for a reason with nothing to do with its contents.

Net effect, consistent with this issue's title: any narrow, correctly-scoped PR is unmergeable in a repo where require-full-coverage is enforced and the pin distinguishes SKIPPED from NOT APPLICABLE. A structural skip under ADR-020 diff scoping means "this gate had no input in this diff", which is not the same as "this gate has an unverified gap". The two are being conflated.

Cross-ref: ConductionNL/doriath#191 - held red rather than merged, and rather than waiving the gate.

[rubenvdlinde]

Fixed by #270, merged to main at 17:19:09Z (32fa23ea).

What landed: an empty ADR-020 diff scope is now na, not structural, in all four gates (19/25/62/63). Audited every remaining structural call site (24, 28×2, 30, 33) — each is a genuine gap inside a non-empty scope and keeps its category. #258's fix is intact: an unopened scope prints NOT APPLICABLE, which is not PASS.

Verified against live main — all four coverage-98 cases clear, with the emitted gate count identical on both sides (nothing widened away, only the denominator moved):

	old package	now
doriath #191	98	0 (57 green both sides)
doriath development 4bfeab7c8	98	0 (53 green)
larpingapp b0c60f734	98	0 (53 green)
softwarecatalog 31d87d1	98	0 (57 green)

doriath#191's real CI now reports quality / Hydra Gates: SUCCESS.

On the suggestion that gate-63 should instead treat a changed SettingsController.php as its subject — it should not, and the source settles it. Every rule in check_settings() reads a manifest or src/menu-layout.json; ADR-079 governs where a settings surface is declared, not what a settings endpoint does. lib/Controller/ is not even lib/Settings/, the only PHP path the gate consults (as a boolean, in a WARN). On #191 the controller is judged — by gates 5, 6, 7, 8, 9, 14, 16, 25, 47, 48, all PASS. There is no single global scope gates "agree" on; each declares its own inputs. And this widening was already tried and reverted: check_store() records that walking lib/ while the manifest half honoured ADR-020 "blocked EVERY manifest-touching PR in that repo, permanently".

Tests grew from three arms to five and pin both directions — an empty scope must not fail the run, and na must not swallow a subject that is in the diff. Mutation-checked (each mutation asserting old in source first, so a mutation that never applied cannot pose as a survivor). The mutant "gate-63 declares na unconditionally" is killed by ARM 5 only — ARM 3 misses it, because a clean manifest still returns rc=0 and passes.

The larger finding in this issue is not dropped. #270 emits the gate package's SHA into the run output, which makes the drift visible; the policy question of making the strict-subset comparison sound is now tracked in #271.