refactor(smoke): discover test files instead of registering them

[chubes4]

Closes #2402 item 4.

The smoke manifest enumerated 128 commands by hand. A test ran only if someone remembered to register it — which is how #2402 found 153 test: scripts that no gate invoked, and why package.json absorbs 15% of all commits acting as the de facto test registry.

This replaces enumeration with discovery.

What changes

scripts/smoke-discovery.ts finds tests/*.test.{ts,mjs} and scripts/*-smoke.{ts,php} by convention. Adding a test file is now enough to make it run. Anything that must not run is listed in DISCOVERY_EXCLUSIONS with a stated reason, so exclusion is an explicit, reviewable act rather than the silent default.

The manifest keeps only work that is not a single test file:

kept	why
build	tsc -b for runtime-core/playground/cli, plus the CLI bin and provenance steps
test:cloudflare-runtime	carries tsc -p packages/runtime-cloudflare --noEmit

scripts/smoke-manifest.ts drops from 198 lines and 128 entries to 50 lines and 2.

Coverage is verified, not assumed

The risk in this change is silently losing coverage. Measured against the old manifest:

files covered by the OLD manifest: 147
of those, NOT in the discovered set: 2
   scripts/ensure-cli-bin-executable.mjs
   scripts/write-cli-build-provenance.ts

Both are build helpers that npm run build still runs. 145 of 147 are discovered directly; the remaining 2 are covered by the retained build command. Nothing is lost, and discovery adds 212 files that were never gated.

Concurrency, because 357 files sequentially is 30 minutes

Discovered files are independent processes, so the discovered phase runs at concurrency 8 and captures output per file, reporting every failure rather than stopping at the first. The old runner failed fast, which is why #2402 needed an out-of-tree harness to produce a full picture.

Five files contend on the shared Playground WordPress archive cache and fail when run alongside each other. They are correct in isolation, so rather than excluding them they run in a serial phase afterwards, listed in DISCOVERY_SERIAL. Verified: all five pass serially in 90s total.

Measured

npm run check
# [smoke] check: 2 declared, 352 discovered (concurrency 8), 5 serial
# [smoke] check passed: 359 command(s)
# real 7m59.321s

	before	after
manifest lines	198	50
manifest entries	128	2
files executed	147	357
wall time	~5 min	~8 min

143% more files for about three more minutes.

Exclusions

14 total, each with a reason in the source. Three categories:

• Environment (3): need Docker, exceed the per-file budget, or need the 427 MB plugin zip. All already run in the agent-task-contracts workflow or the Homeboy gate.
• The runner itself (1): scripts/run-smoke.ts matches *-smoke.ts and would recurse.
• Failing and unmaintained (10): added June 2026, never wired to a gate, untouched since. Discovery is what surfaced them.

Three of those ten fail for one shared root cause worth recording: packages/runtime-playground/src/query-recorder.ts:80 emits PHP inside a JS template literal containing \0 and \x0B, which JavaScript interprets as actual control characters. The generated PHP therefore carries a raw NUL byte, and Node refuses to pass it as an argv argument. The string value PHP ends up with is the same either way, so this is a portability defect rather than a behavior bug, but it blocks any harness that shells out with php -r.

I have deliberately not fixed those ten here. This PR should be evaluated on whether discovery is correct, not bundled with ten unrelated repairs. They are #2402 item 5.

Update: four CI rounds to converge

Local runs passed on the first try. CI did not, and the gap was instructive — my machine has enough cores to hide the problems. Each round found a distinct class:

round	failures	cause	fix
1	19	prepare-declaration-rebuild deletes runtime-core/dist and runs npm install mid-run, breaking every concurrent import of @automattic/wp-codebox-core	excluded as destructive to shared state
2	2	browser + Playground tests starved at concurrency 8 on a 2-core runner	moved to the serial phase
3	1	cancellation-timing test, same starvation class	moved to the serial phase
4	1	native-mariadb-runtime-service lost the ordering test:runtime-services gave it	chains restored, their files excluded from discovery

Round 4 corrected a design mistake. My original analysis flagged four commands as must-keep; I overrode two of them because their member files were individually discoverable. Being discoverable is not the same as being safe standalone or concurrently. test:generic-primitives and test:runtime-services are back as declared commands, and the 22 files they own are excluded from discovery so nothing runs twice.

Round 1 was worth the trip on its own: a test that deletes shared build output while other tests import from it is a real defect, and it was invisible while the suite ran sequentially.

Final CI: smoke aggregate green in 11m16s.

Final shape

	before	after
manifest lines	198	55
manifest entries	128	4
files executed	147	356
CI wall time	n/a (not run)	11m16s

Coverage verified against origin/main: 145 of the 147 files the old manifest executed are still covered, and the other two are build helpers build still runs.

Exclusions total 15, each with a reason, plus 22 chain-owned files executed by their declared chain.

Follow-up

The 218 test: scripts in package.json are now largely redundant with discovery, but removing them is a separate diff with its own review surface — docs and operator habits reference them. Item 5 covers that alongside the ten excluded files.

---

AI assistance disclosure: authored by Claude (Sonnet 4.5) running in OpenCode, directed by @chubes4. The model probed all 226 previously-ungated files individually to establish their status, measured sequential vs concurrent execution to size the runtime, identified the five archive-cache contenders empirically and confirmed they pass serially, and proved the 145-of-147 coverage overlap before collapsing the manifest. All figures are measured. Reviewed by a human before opening.