gate-19: 'tag present but the test does not run' fires on tests that DO run — 34 of 190 findings on nldesign (18% inflation), disproved by the same CI run

[rubenvdlinde]

Summary

Measuring nldesign's true gate-19 debt (root-commit-scoped, per #242) gives 190 findings. 34 of them are false positives in one specific branch: @e2e tag present but the test does not run: referenced only by a test that never runs (<file>).

Every file named in those 34 has passing tests in the same CI run. The real debt is 156.

This is the counting hazard in its concrete form: a finding count is not a defect count, and here the inflation is 18%.

Measured

$ run-hydra-gates.sh --scope-to-diff --base <root commit>
[hydra-gates] SCOPE-FILE-COUNT: 3611
[gate-19] e2e-coverage: FAIL — 190 scenario(s) missing @e2e

Split by reason (191 log lines, 191 distinct — no duplicate-emission inflation, so the 34 are the whole story):

reason	count	verdict
— missing @e2e	156	genuine
— @e2e tag present but the test does not run	34	false positive

Files named in the 34, against passing tests in nldesign CI job 93093896316:

file	flagged as "never runs"	tests that PASSED in that run
admin-settings.spec.ts	14	17
token-editor-ui.spec.ts	6	8
app-theming.spec.ts	5	4
token-set-apply-dialog.spec.ts	3	5
token-set-contrast-audit.spec.ts	2	2
custom-token-set-upload.spec.ts	2	4
token-set-apply-toggle-behavior.spec.ts	1	3
token-import-export.spec.ts	1	2

Not one of them is a file whose tests never run.

Five flagged anchors matched to passing tests, verbatim

The scenario names and the CI ✓ lines are the same tests:

admin-settings::settings-panel-appears-in-admin-area                    -> ✓ 10 admin-settings.spec.ts:24:6  › Settings panel appears in admin area (3.2s)
admin-settings::settings-panel-position-relative-to-nextcloud-theming   -> ✓ 11 admin-settings.spec.ts:35:6  › Settings panel position relative to Nextcloud theming (2.6s)
admin-settings::dropdown-populated-with-token-sets                      -> ✓ 12 admin-settings.spec.ts:78:6  › Dropdown populated with token sets (2.6s)
admin-settings::dropdown-label-is-associated-with-select                -> ✓ 13 admin-settings.spec.ts:95:6  › Dropdown label is associated with select (2.6s)
admin-settings::preview-box-renders-with-token-set-colors               -> ✓ 15 admin-settings.spec.ts:133:6 › Preview box renders with token set colors (2.8s)

admin-settings.spec.ts contains test.describe('admin-settings', () => { — a plain describe, not .skip — 17 test( blocks, and zero test.skip / test.fixme / describe.skip anywhere in the file.

Likely mechanism

nldesign writes the tag inside the test( call, between the opening paren and the title:

test(
    // @e2e openspec/specs/admin-settings/spec.md#settings-panel-appears-in-admin-area
    'Settings panel appears in admin area',
    async ({ page }) => {
        await page.goto(THEMING_URL)
        ...
    },
)

The helper's forward search from the tag looks for the test( that owns it — but here test( is on the line above the tag. The search runs forward, so it either finds the next test's declaration or runs off the end of the file, and the "no live declaration found" path reports "referenced only by a test that never runs".

That fits the distribution: 14 of 17 tagged tests in admin-settings.spec.ts are flagged, and the file's tag layout is uniform.

The helper already carries a comment about this class of bug — "never runs", about a file whose tests run fine — so the ancestor/describe.skip direction has been fixed. The declaration-above-the-tag layout looks like the remaining one.

Suggested fix

When resolving a tag to its owning test, search backwards as well as forwards for the nearest test( / it( declaration, and prefer the nearer of the two. Both layouts are in fleet use:

// @e2e ...          |   test(
test('title', ...)   |       // @e2e ...
                     |       'title',

Whatever the fix, the discriminator worth adding is: "I could not resolve this tag to a declaration" is not "the test never runs." They are different facts and only the second is a finding — the same distinction gate-5 already draws between "the attribute is absent" and "I could not open the class".

Why this was only visible at the root commit

A normal run scopes to the diff, so these never appear. A Scope: full repo run does not help either: gate-19 is unconditionally diff-scoped inside its helper, so an empty diff makes it call _pass (#242). It takes --scope-to-diff --base <root> to see any of it.

Related: #242 (gates 19/25/63 invisible to --require-full-coverage), #243 (set -e leak aborts the run mid-way).

[rubenvdlinde]

What it actually was — diagnosed, not assumed

Fixed in #249. The issue asked for a diagnosis before a fix, so here it is.

Your "likely mechanism" was right about the first half, and that half is the whole thing. The tag sits between the open paren and the title, and the resolver only ever searched forward, so each tag bound to the next test( in the file — a different test entirely.

But that alone would not have produced findings, because the next test also runs. The second half was #234. Every nldesign declaration ends:

		},
	)

and the body finder stepped back from ) over whitespace requiring a }, hit the ,, and returned an empty body. So the tag bound to the wrong test, and the wrong test then read as empty.

Two defects, one symptom. That is why the count alone could not have told them apart, and why the fixture asserts the binding (doc.owner(pos) returns the declaration containing the tag, not the one after it) as well as the liveness.

The guessed remedy would not have worked

The suggestion was "search backwards as well as forwards and prefer the nearer". On the nldesign layout the nearer declaration going backwards is the right one, so it would have fixed these 34 — but it would have mis-bound the conventional layout wherever a preceding test's closing paren is nearer than the following test's opening one, which is common in the spec-coverage files. The fix in #249 is structural instead: build the tree, and the owner is the declaration whose header contains the tag, else the next declaration inside the same body — the search can no longer escape the enclosing node at all.

On the discriminator you asked for

"I could not resolve this tag to a declaration" is not "the test never runs."

Agreed, and it is now structurally impossible to confuse them: a tag that resolves to no declaration at all is a file-level annotation and stays live. It is only reported when a declaration was found and that declaration is switched off, empty, or under an unconditional skip.

Measured, root-commit-scoped

nldesign 190 → 156. Exactly the 34, and not one other finding changed on nldesign. Fleet-wide across 24 checkouts: 8790 → 8698, all 92 removals false positives, zero findings added anywhere.

The 34 came from 8 distinct spec files — worth recording next to your 18%-inflation figure, since a finding count is not a defect count in either direction.