gates 19/25/26 are still diff-scoped on every pull_request — HYDRA_GATE_BASE_REF leaks past the SCOPE_TO_DIFF guard, so #378 does not reach the event that gates merges

[rubenvdlinde]

Summary

Since #378 made whole-tree the default, gates 19, 25 and 26 are still silently diff-scoped on every pull_request run — i.e. on exactly the event that gates a merge. They report their whole-tree backlog only on push, after the merge, where it can no longer block anything.

The cause is not the scope flag. It is that those three helpers receive the base ref through the environment, and HYDRA_GATE_BASE_REF is present in the environment on pull_request and absent on push.

Isolated control — same tree, same base commit, only the delivery channel differs

Local run of bin/hydra-gates against docudesk, base f1931874 in both arms, SCOPE-MODE: full in both:

gate	HYDRA_GATE_BASE_REF=f1931874 bin/hydra-gates --app-dir D	env -u HYDRA_GATE_BASE_REF bin/hydra-gates --app-dir D --base f1931874
19 e2e-coverage	NOT APPLICABLE	FAIL — 396 scenario(s)
25 contract-coverage	NOT APPLICABLE	PASS (it actually ran)
26 visual-coverage	NOT APPLICABLE	FAIL — 6 new page component(s)
	COVERAGE: 56 of 65	COVERAGE: 58 of 65

Both arms print the same resolved base — only the source label changes (($HYDRA_GATE_BASE_REF) vs (--base)). Nothing else about the invocation differs.

The same split reproduces in CI as a push-vs-PR difference

repo	event	base line in the log	gate-19	COVERAGE
hermiq	push (2 changed files)	Delta base: 51e128e… (github.event.before (push))	FAIL — 491	60 of 65
hermiq	pull_request (4 changed files)	Delta base: origin/development ($HYDRA_GATE_BASE_REF)	NOT APPLICABLE	58 of 65
docudesk	push (0 changed files)	Delta base: 18e17b1f… (github.event.before (push))	FAIL — 396	62 of 65
docudesk	pull_request (1 changed file)	Delta base: origin/development ($HYDRA_GATE_BASE_REF)	NOT APPLICABLE	58 of 65

On push the wrapper resolves the base from github.event.before into a shell variable and forwards it as an argument; the env var is not set, so the helper sees no base and sweeps the tree. On pull_request the caller exports HYDRA_GATE_BASE_REF, and the helper picks it up regardless of SCOPE_TO_DIFF.

Why the existing guard does not catch it

run-hydra-gates.sh gate-19 already knows about this class — its comment cites #242:

SCOPE ONLY WHEN THE CALLER ASKED FOR IT (#242). BASE_REF was passed unconditionally, so an UNSCOPED run — the mode a fleet audit uses — was silently narrowed to the diff against origin/development, came back empty, and the helper printed PASS over a repo it never opened. Measured on openconnector: 5 findings scoped, 412 over the full tree.

and the code correctly gates the explicit pass:

if [ "${SCOPE_TO_DIFF}" = "1" ]; then
    HYDRA_GATE_BASE_REF="${BASE_REF}" python3 …/check_e2e_coverage.py . >> "${_e2e_log}" 2>&1
else
    python3 …/check_e2e_coverage.py . >> "${_e2e_log}" 2>&1
fi

The else branch does not unset the variable. An inherited HYDRA_GATE_BASE_REF walks straight past the if into the child process, and #242 is re-opened by the environment rather than by the argument.

⚠️ This also silently sabotages local reproduction: my first attempt at reproducing docudesk's gate-19 used HYDRA_GATE_BASE_REF=… and got NOT APPLICABLE on a tree I already knew carried 396. That reads exactly like a clean repo.

Impact

#378's stated purpose is that a tightened gate should force the inherited debt to be fixed. For the three coverage gates that does not happen where it would bite:

• docudesk: 402 findings (396 + 6) invisible on every PR.
• hermiq: 502 findings (491 + 5 + 6) invisible on every PR.

And the two events produce numbers that look like progress. docudesk development push reads RESULT: 5 GATE(S) FAILED; a one-file PR against it reads RESULT: 2 GATE(S) FAILED. Nothing was fixed in between. Anyone quoting a PR gate count as a repo state — or comparing one to a push count — records a three-gate improvement that does not exist.

Suggested fix

Unset the variable in the else branch for each of the three helpers, so the environment cannot re-introduce the scoping the flag declined:

else
    env -u HYDRA_GATE_BASE_REF python3 …/check_e2e_coverage.py . >> "${_e2e_log}" 2>&1
fi

⚠️ Expect the number to jump on the next PR in every repo, which is the point of #378 — but it should land deliberately rather than as a surprise on someone's unrelated one-line change.

Worth adding to the gate-acceptance fixtures as a two-arm case: same base, delivered two ways, must produce the same verdict. That property is what failed here, and it is cheap to pin.

Measured on ConductionNL/.github@923f57d locally and against CI runs using 5e73e640 and e3da4103; the split is present in all three.

[rubenvdlinde]

Fixed in #418.

The split reproduced, and then twice more. Clean clone of docudesk, one tree, one base, SCOPE-MODE: full in both arms, only the delivery channel different — gates 19/25/26 went not-applicable through the environment and 396 / passed / 6 through --base, coverage 56 of 65 against 59 of 65. Diffing the complete 65-gate verdict sets, exactly three lines differ. The same split then reproduced through the real wrapper on openregister (845 / 66 / 27) and procest (1142 / 118 / 78).

One correction to the issue's scope: it is seven gates, not three. The same unguarded else is at gates 19, 25, 26, 51, 52, 54 and 55, and all seven helpers read the variable from os.environ. I could not demonstrate a behavioural difference for the last four — their whole-tree count is zero in every app I measured, so scoped and swept print identically there — but they were exposed by the same mechanism and are in the fix.

The fix is not the else-branch env -u suggested here, though that would work. The runner now consumes the variable where it reads it and unsets it, so every checker is spawned into an environment that does not contain it and can only receive a base the runner explicitly supplies. The reason for preferring it: the else-branch repair has to be remembered by each gate, and it already failed six times — gate-19's author wrote the #242 comment explaining the hazard, and gates 25, 26, 51, 52, 54 and 55 were each written afterwards with the same unguarded shape. The invariant is that the scope decision has exactly one source. The variable remains a supported input for direct runner callers; only the ambient inheritance is gone.

Acceptance arm, as suggested here: scripts/lib/test_gate_base_ref_delivery_channel.sh pins it generically — for one tree and one base at full scope, the per-gate verdict set must be identical whichever channel delivered the base — so a gate added tomorrow is covered without editing the file. It refuses to grade until a positive control proves the fixture's scenario is findable, and separately requires gate-19 to fail in both arms, so two arms agreeing about nothing can never satisfy it. Revert-proven.

Blast radius, measured on all 18 core apps: gate-19 5,505 · gate-25 241 · gate-26 146. Per-app table is in the PR. These are not new findings — they are what the push run has been printing all along, now arriving where they can block. gate-19 is a coverage backlog, not a defect, and #345 scores its only annotation-shaped remedy as positive coverage, so mass-excluding would turn every cell green while covering nothing.