gate-19: a test QUARANTINED behind a tracked blocker has no honest end state — its only remedy is to declare the scenario permanently out of e2e scope (59 findings measured)

[rubenvdlinde]

Split out of #239's secondary observation, which #249 deliberately did not address because it is a policy change rather than a false-positive fix.

The gap

gate-19 is correct that a test.fixme / test.skip('title', …) does not run, so the scenario is not proven. But the only remedy it prints is:

replace the tag with a reason-bearing @e2e exclude

and @e2e exclude means "this scenario is out of scope for e2e, forever". A scenario that should be e2e-tested and is blocked on a tracked fixture or product gap has no way to say so. The honest states are three, and the vocabulary has two.

Measured, root-commit-scoped, against .github@main (7b66766)

These are the findings that survive #249 — i.e. every one is a correct report of a test that genuinely does not run:

repo	findings	what they actually are
openbuild	36	test.skip('REQ-OBR-002 — builder route mounts a nested CnAppRoot …', …) — 8+ declarations in openbuild-runtime.spec.ts alone
larpingapp	23	test.fixme(…), every one carrying // FIXME(detail-actions-blocker): detail Actions menu never renders — DETAIL_ACTIONS_BLOCKER.
procest	26	test.skip(true, 'Requires a seeded case fixture in the dev container') at top of body
openconnector	6	test.describe.skip('Catalog page — browse, filter, badges (REQ-001)', …)
scholiq	2

93 findings, and the 59 in larpingapp + openbuild are the clearest case: the author already wrote down the blocker (DETAIL_ACTIONS_BLOCKER, REQ-OBR-*) in a comment the gate cannot read. Closing these honestly today means asserting something false.

What the vocabulary is missing

Three distinct facts, two words:

fact	today	should be
not suitable for e2e (backend contract, covered by PHPUnit/Newman)	@e2e exclude <reason>	unchanged
should be e2e-tested, blocked on a tracked issue	no representation — pressure to use @e2e exclude	e.g. @e2e blocked <issue> <reason>
should be e2e-tested, nobody has written it	missing @e2e	unchanged

Why it matters more than a wording nit

The remedy the gate prints is the one people follow. Every scenario pushed from blocked into @e2e exclude is permanently removed from the e2e programme's denominator — silently, and by a gate that was working correctly. That is the same shape as #239's "complying deletes a true coverage claim", but reached through an accurate finding instead of a false one, which makes it harder to notice.

Suggested shape

• accept @e2e blocked <issue-ref> <reason> in the spec scenario block, with both parts required (a bare @e2e blocked is non-compliant, mirroring the existing bare-exclude rule);
• it does not satisfy coverage — the scenario stays in the denominator and is reported;
• but it is reported in its own category, so FAIL — n scenario(s) separates nobody wrote this from someone wrote it and it is blocked on ;
• and consider a rot check: a @e2e blocked whose issue is CLOSED is itself a finding, so the quarantine can only shrink — the same discipline tests/run-helper-suites.sh already applies to its own QUARANTINE list.

Related: #239 (where this was first observed), #249 (the fix that left it out on purpose).

[rubenvdlinde]

gate-46's answer to the shared-convention question, with numbers

I was asked to consider whether gate-46 should grow a machine-readable "resolved elsewhere / out of scope" state so it and gate-19 share one convention. Having now measured the whole fleet, my answer is no — gate-46 does not have this problem, and giving it an escape hatch would create one.

The distinguishing property in #252 is a finding that is correct but has no legitimate way to be closed. A scenario quarantined behind a tracked blocker is genuinely stuck: the work is real, it cannot be done yet, and the only available closure is a lie. That is worth a vocabulary.

A dangling @spec anchor is not stuck. It has three honest terminal actions, all available today:

1. repoint the tag at the capability's canonical spec;
2. write the requirement the tag names, if it should exist;
3. delete the tag — @spec exclude <reason> already exists and gate-46 skips it before the resolution check, so "no spec applies to this code" is already expressible and machine-readable.

Measured on origin/development for all 20 repos that carry openspec/, using the gate as merged in #246:

• 602 findings from 249 distinct targets (down from 784 pre-fix).
• 0 of 602 point outside their own repo's openspec/ tree. There is no cross-repo class where a local fix is impossible.
• 0 of 602 are findings on tags that already carry an exclusion — the hatch is honoured, not being defeated.

So every one of the 602 can be closed truthfully by someone with commit rights to that repo. Adding a fourth state — "this anchor is fine really" — would not describe a real situation; it would only offer a way to silence a genuine dangling reference, which is the exact defect gate-46 exists for. Given how many of these were manufactured by tooling rather than authored (see #228), a cheap suppression keyword would be reached for at scale, and the gate would quietly become decorative.

The part of #228 that WAS the same shape, and how it was closed

There was a real "no honest end state" problem here, and it is worth naming because the fix generalises. The blocking phpcs SpecTagSniff instructed authors to point @spec at openspec/changes/{name}/tasks.md#task-N. Following that instruction guaranteed a dangling anchor the moment the change was archived — work that regenerates itself. But repointing at the canonical openspec/specs/ was ALSO not durable, because archiving in these repos does not reliably promote the delta spec there: procest carried 18 tags written exactly as the corrected advice says, whose spec only ever existed inside the archived change directory.

That is the shape #252 describes: both available actions led back to a finding.

#246 closed it by resolving a spec by capability across all three homes it occupies — in-flight change dir, archived change dir, canonical openspec/specs/ — rather than by change name. A tag written once now keeps resolving through the whole lifecycle. The honest end state was created by making one of the actions actually work, not by adding a word that means "stop asking".

That is the transferable lesson for the 93 gate-19 survivors, and especially for the 59 in larpingapp/openbuild that already name their blocker in a comment the gate cannot read: before minting a new suppression vocabulary, check whether one of the existing terminal actions is failing for a mechanical reason that can be fixed. A blocker named in a comment is evidence the authors already tried to express the state and had nowhere to put it — that is an argument for a readable field, which is a different thing from an escape hatch, because it still has to name something real and can be audited.

What the 602 actually are, so nobody grinds the wrong thing

Split: 447 "anchor not found" (the file resolves, the requirement/task named in the fragment does not exist) and 155 "target file not found". 557 of 602 still point at openspec/changes, which is why the sniff rewording matters more than any individual repo's cleanup.

I verified the survivors exhaustively in five repos (portaliq, doriath, openconnector, larpingapp, procest) and sampled the three largest others. They are dominated by tags naming requirements and task numbers that exist nowhere — doriath's 77, for instance, name Listing and download, Masked presentation and Lease management API, none of which appear in any of the three homes. Those are true positives and no resolution rule should rescue them.