Skip to content

ci: shard the test phase across 3 runners - #118

Merged
wormeyman merged 1 commit into
mainfrom
ci/shard-verify
Aug 3, 2026
Merged

ci: shard the test phase across 3 runners#118
wormeyman merged 1 commit into
mainfrom
ci/shard-verify

Conversation

@wormeyman

Copy link
Copy Markdown
Owner

verify took 9m03s as a single job on #116, and the test phase is ~95% of it. This splits the phases into a static job and a 3-way sharded tests matrix, cutting the gate to roughly 3-4 minutes. Nothing about what is checked changes.

Why sharding, and not a vitest setting

Measured locally: the suite is 497s of CPU in 68s of wall, ~7.5x on 12 cores (8 performance + 4 efficiency). Per-box parallelism is already exhausted - CLAUDE.md records maxWorkers 4/8/11 at 74.7/61.7/61.8s against a 61.2s default, because the extra cores are E-cores. More machines is the only lever left.

Two alternatives measured and rejected:

Candidate Result
isolate: false 66 of 171 files fail. Those same files pass individually with --no-isolate, so it's cross-file module-state pollution, not misconfiguration. Also only 7.6% (68.24s → 63.07s).
environment: "node" by default Worth ~3s. Only ~30 of 164 spec files touch document/window, so the rest pay happy-dom for nothing - but not worth the churn alone.

Why three shards - measured, not guessed

vitest splits by file, so a single file is an unsplittable floor, and test/previewAgreement.spec.ts is 67s of the 68s suite. Slowest-shard wall is what the gate waits on:

shard walls gate runner-seconds
N=3 55.7 / 25.0 / 18.0 55.7s 159s
N=4 53.5 / 37.1 / 23.4 / 16.9 53.5s 211s

N=4 is 2.3s faster - inside run-to-run noise - for 33% more runner minutes, because both are pinned by the same file. So 3 is the cost-optimal point, not merely adequate. Raise it only after splitting previewAgreement into its three independent tests, and re-measure.

The shards are unbalanced (55.7 vs 18.0) because the split is by file count, not duration. That costs nothing here: even perfect balance can't beat the 67s file floor.

Two things done carefully

Anti-drift. The old job ran pnpm run verify verbatim so CI couldn't drift from local. That rule is kept by a different mechanism: this workflow names only package.json scripts, never the underlying commands, and verify, verify:static and verify:shard all compose the same verify:lint. One definition of each phase, still.

The required check. Ruleset EJ requires a check named verify, and a required check that never appears blocks every PR forever. The aggregating job therefore keeps the id verify, and asserts needs.*.result explicitly - because a needs: job whose dependency failed is skipped, and a skipped required check does not block a merge. No ruleset change is needed, which is the point.

No blob reports

--reporter=blob works, but vp test --merge-reports does not merge - it re-runs, reporting 114 files from a 57-file shard's blob. So no artifact plumbing. The directory stays gitignored anyway, since running that reporter by hand drops a ~500 KB JSON in the repo root that vp check's format step trips over.

Verification

pnpm run verify green on this branch after the script refactor: 1,423 app + 12 worker + 3 container tests, 0 warnings. Arg passthrough checked directly - pnpm run verify:shard -- --shard=3/3 runs 57 files, not 171. This PR's own verify run is the real test of the workflow.

🤖 Generated with Claude Code

https://claude.ai/code/session_018AB7J1qK6kSBnJJmgDqMst

`verify` took 9m03s as a single job on PR #116, and the test phase is ~95%
of it. Splitting the phases into a `static` job and a 3-way sharded `tests`
matrix cuts the gate to roughly 3-4 minutes. Nothing about what is checked
changes.

WHY SHARDING AND NOT A SETTING. Measured locally: the suite is 497s of CPU
in 68s of wall, ~7.5x on 12 cores (8 performance + 4 efficiency). Raising
parallelism on one box is already exhausted - CLAUDE.md records maxWorkers
4/8/11 at 74.7/61.7/61.8s against a 61.2s default. More machines is the only
remaining lever. Two other candidates were measured and rejected:

  - `isolate: false`: 66 of 171 files FAIL. The same files pass individually
    with --no-isolate, so it is cross-file module-state pollution, not a
    misconfiguration. It also only bought 7.6% (68.24s -> 63.07s).
  - `environment: node` by default (only ~30 of 164 spec files touch
    document/window): worth ~3s. Real but not worth the churn on its own.

WHY THREE, MEASURED NOT GUESSED. vitest splits by FILE, so one file is an
unsplittable floor, and `test/previewAgreement.spec.ts` is 67s of the 68s
suite. Slowest-shard wall, which is what the gate waits on:

  N=3   55.7 / 25.0 / 18.0        -> 55.7s   159 runner-seconds
  N=4   53.5 / 37.1 / 23.4 / 16.9 -> 53.5s   211 runner-seconds

N=4 is 2.3s faster - inside run-to-run noise - for 33% more runner minutes,
because both are pinned by the same file. So 3 is the cost-optimal point,
not merely an adequate one. Raise it only after splitting previewAgreement
into its three independent tests, and re-measure rather than assuming.

The shards are unbalanced (55.7 vs 18.0) because the split is by file count,
not duration. That costs nothing here: even a perfect balance could not beat
the 67s file floor.

ANTI-DRIFT. The old single job ran `pnpm run verify` verbatim so CI could not
drift from local. That rule is kept by a different mechanism: this workflow
names only package.json SCRIPTS, never the underlying commands, and
`verify`, `verify:static` and `verify:shard` all compose the same
`verify:lint`. There is still one definition of each phase.

REQUIRED CHECK. Ruleset `EJ` requires a check named `verify`, so the
aggregating job keeps that id - a required check that never appears blocks
every PR forever. It asserts `needs.*.result` explicitly, because a `needs:`
job whose dependency failed is SKIPPED, and a skipped required check does not
block a merge. No ruleset change is needed.

NO BLOB REPORTS. `--reporter=blob` works, but `vp test --merge-reports` does
not merge - it re-runs, reporting 114 files from a 57-file shard's blob. The
directory stays gitignored anyway: running that reporter by hand drops a
~500 KB JSON in the repo root that `vp check`'s format step trips over.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AB7J1qK6kSBnJJmgDqMst
@wormeyman
wormeyman merged commit d7feb3c into main Aug 3, 2026
6 checks passed
@wormeyman
wormeyman deleted the ci/shard-verify branch August 3, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant