ci: shard the test phase across 3 runners - #118
Merged
Merged
Conversation
`verify` took 9m03s as a single job on PR #116, and the test phase is ~95% of it. Splitting the phases into a `static` job and a 3-way sharded `tests` matrix cuts the gate to roughly 3-4 minutes. Nothing about what is checked changes. WHY SHARDING AND NOT A SETTING. Measured locally: the suite is 497s of CPU in 68s of wall, ~7.5x on 12 cores (8 performance + 4 efficiency). Raising parallelism on one box is already exhausted - CLAUDE.md records maxWorkers 4/8/11 at 74.7/61.7/61.8s against a 61.2s default. More machines is the only remaining lever. Two other candidates were measured and rejected: - `isolate: false`: 66 of 171 files FAIL. The same files pass individually with --no-isolate, so it is cross-file module-state pollution, not a misconfiguration. It also only bought 7.6% (68.24s -> 63.07s). - `environment: node` by default (only ~30 of 164 spec files touch document/window): worth ~3s. Real but not worth the churn on its own. WHY THREE, MEASURED NOT GUESSED. vitest splits by FILE, so one file is an unsplittable floor, and `test/previewAgreement.spec.ts` is 67s of the 68s suite. Slowest-shard wall, which is what the gate waits on: N=3 55.7 / 25.0 / 18.0 -> 55.7s 159 runner-seconds N=4 53.5 / 37.1 / 23.4 / 16.9 -> 53.5s 211 runner-seconds N=4 is 2.3s faster - inside run-to-run noise - for 33% more runner minutes, because both are pinned by the same file. So 3 is the cost-optimal point, not merely an adequate one. Raise it only after splitting previewAgreement into its three independent tests, and re-measure rather than assuming. The shards are unbalanced (55.7 vs 18.0) because the split is by file count, not duration. That costs nothing here: even a perfect balance could not beat the 67s file floor. ANTI-DRIFT. The old single job ran `pnpm run verify` verbatim so CI could not drift from local. That rule is kept by a different mechanism: this workflow names only package.json SCRIPTS, never the underlying commands, and `verify`, `verify:static` and `verify:shard` all compose the same `verify:lint`. There is still one definition of each phase. REQUIRED CHECK. Ruleset `EJ` requires a check named `verify`, so the aggregating job keeps that id - a required check that never appears blocks every PR forever. It asserts `needs.*.result` explicitly, because a `needs:` job whose dependency failed is SKIPPED, and a skipped required check does not block a merge. No ruleset change is needed. NO BLOB REPORTS. `--reporter=blob` works, but `vp test --merge-reports` does not merge - it re-runs, reporting 114 files from a 57-file shard's blob. The directory stays gitignored anyway: running that reporter by hand drops a ~500 KB JSON in the repo root that `vp check`'s format step trips over. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AB7J1qK6kSBnJJmgDqMst
wormeyman
force-pushed
the
ci/shard-verify
branch
from
August 3, 2026 22:33
67b745b to
8c31784
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
verifytook 9m03s as a single job on #116, and the test phase is ~95% of it. This splits the phases into astaticjob and a 3-way shardedtestsmatrix, cutting the gate to roughly 3-4 minutes. Nothing about what is checked changes.Why sharding, and not a vitest setting
Measured locally: the suite is 497s of CPU in 68s of wall, ~7.5x on 12 cores (8 performance + 4 efficiency). Per-box parallelism is already exhausted - CLAUDE.md records
maxWorkers4/8/11 at 74.7/61.7/61.8s against a 61.2s default, because the extra cores are E-cores. More machines is the only lever left.Two alternatives measured and rejected:
isolate: false--no-isolate, so it's cross-file module-state pollution, not misconfiguration. Also only 7.6% (68.24s → 63.07s).environment: "node"by defaultdocument/window, so the rest pay happy-dom for nothing - but not worth the churn alone.Why three shards - measured, not guessed
vitest splits by file, so a single file is an unsplittable floor, and
test/previewAgreement.spec.tsis 67s of the 68s suite. Slowest-shard wall is what the gate waits on:N=4 is 2.3s faster - inside run-to-run noise - for 33% more runner minutes, because both are pinned by the same file. So 3 is the cost-optimal point, not merely adequate. Raise it only after splitting
previewAgreementinto its three independent tests, and re-measure.The shards are unbalanced (55.7 vs 18.0) because the split is by file count, not duration. That costs nothing here: even perfect balance can't beat the 67s file floor.
Two things done carefully
Anti-drift. The old job ran
pnpm run verifyverbatim so CI couldn't drift from local. That rule is kept by a different mechanism: this workflow names only package.json scripts, never the underlying commands, andverify,verify:staticandverify:shardall compose the sameverify:lint. One definition of each phase, still.The required check. Ruleset
EJrequires a check namedverify, and a required check that never appears blocks every PR forever. The aggregating job therefore keeps the idverify, and assertsneeds.*.resultexplicitly - because aneeds:job whose dependency failed is skipped, and a skipped required check does not block a merge. No ruleset change is needed, which is the point.No blob reports
--reporter=blobworks, butvp test --merge-reportsdoes not merge - it re-runs, reporting 114 files from a 57-file shard's blob. So no artifact plumbing. The directory stays gitignored anyway, since running that reporter by hand drops a ~500 KB JSON in the repo root thatvp check's format step trips over.Verification
pnpm run verifygreen on this branch after the script refactor: 1,423 app + 12 worker + 3 container tests, 0 warnings. Arg passthrough checked directly -pnpm run verify:shard -- --shard=3/3runs 57 files, not 171. This PR's ownverifyrun is the real test of the workflow.🤖 Generated with Claude Code
https://claude.ai/code/session_018AB7J1qK6kSBnJJmgDqMst