Skip to content

perf(test): split vitest into ui (jsdom) and node projects#3814

Merged
cixzhang merged 2 commits into
facebook:mainfrom
Han5991:perf/vitest-split-projects
Jul 11, 2026
Merged

perf(test): split vitest into ui (jsdom) and node projects#3814
cixzhang merged 2 commits into
facebook:mainfrom
Han5991:perf/vitest-split-projects

Conversation

@Han5991

@Han5991 Han5991 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

All ~316 test files ran under one root vitest config: every file got a fresh jsdom environment plus the React/StyleX babel transform — including the ~121 files (packages/cli, build, scripts, internal tooling) that never touch the DOM. Vitest's own accounting put that overhead at environment 180s, setup 27s (worker-summed) per full run.

This PR splits the suite into two workspace projects so node-only code stops paying for the DOM:

project scope environment plugins
ui packages/core + packages/lab (195 files) jsdom (unchanged, extends root config) StyleX babel + jest-dom setup
node everything else the root include matched (121 files) node none

Key decisions

  • Partitioning is exhaustive by construction: ui lists core/lab explicitly; node uses the same root globs with core/lab excluded. A new package can't silently fall out of CI — it lands in node by default and fails loudly there if it needs the DOM.
  • Include lists moved out of vitest.config.ts: workspace extends merges arrays, so an include list left in the root config leaks into extending projects and double-runs files (bit me in the first attempt: 121 files ran twice).
  • node project stays on the forks pool: several CLI tests call process.chdir(), which worker threads don't support.

Alternatives measured and rejected

  • pool: 'threads' globally: 63 failuresprocess.chdir() is not supported in workers across 11 CLI test files.
  • isolate: false: 2,138 failures — widespread cross-file DOM/module state pollution. Not viable for this suite.

Measurements (local, M-series 10-core)

  • File/test parity verified via JSON reporter diff: identical 316 files / 5,893 tests, zero missing, zero duplicated.
  • node project alone: 50.4s with environment 16ms (was jsdom per file) and transform 3.5s (was 33s with StyleX babel).
  • Full-run wall time roughly unchanged locally — a 10-core machine absorbs the overhead in parallel. The win is total compute, which matters most on 4-vCPU CI runners: this PR's own test job is the real measurement (baseline: 4m50s–5m30s).
  • Targeted runs are dramatically faster for CLI development: vitest --project node.
  • --coverage smoke-tested under the workspace config.
  • One flaky observed during load testing (TabList.test.tsx roving-tabindex focus) — passes in isolation on both old and new configs; unrelated to this change.

Measurements (actual results, by perceived time)

Per-scenario wall clock — what a human actually feels — measured by applying/reverting this diff on the same tree (local: M-series 10-core) and from this PR's own test job runs (CI: 4-vCPU ubuntu). Baseline CI range comes from recent green runs of other PRs against main.

scenario before after felt difference
CLI-dev targeted run (node scope, 121 files / 1,791 tests) 61.6s 50.3s (vitest --project node) −18%
single CLI test file (edit → rerun loop) 1.69s 1.09s −35% per iteration
local full suite (10-core) 97.5s 117.8s unchanged (run-to-run noise)
CI test job wall 3m56s–5m29s 4m09s / 5m07s within baseline variance so far

The saving itself lives in total compute, not wall time (worker-summed, from vitest's own summary line):

metric baseline this PR delta
environment (jsdom boot) 198.7–204.3s 100.8–124.1s ~−45%
setup (jest-dom) 21.6–22.4s 9.7–13.4s ~−45%
vitest wall (CI) 283.9–293.2s 218.8–269.6s −5% to −25%

So the accurate claim for this PR is not "tests got faster" — it is: the CLI development feedback loop got 18–35% faster, and ~90–110s of worker CPU per CI run stops being spent booting jsdom/setup for files that never touch the DOM. A 10-core machine was absorbing that waste in parallel all along (which is why local full-run wall doesn't move); on saturated 4-vCPU runners the freed compute should convert to wall time, but two runs can't prove that — the post-merge test job distribution on main is the deciding measurement.

  • Per-file overhead removed for node files: environment 283ms → 0ms, setup 192ms → 0ms, transform 173ms → 8ms (single-file run, both configs).
  • File/test parity re-confirmed: identical 316 files / 5,893 tests under both configs.

First CI run failed — pre-existing core-build race, fixed in this PR

The first test job run failed 2 of 4 tests in build-theme.variants.test.mjs with expected 1 to be +0. That assertion is expect(result.code).toBe(0): the astryx theme build invocation under test exited 1 because packages/core/dist was being rewritten out from under it.

Root cause: #3562 already serializes the concurrent pnpm -F @astryxdesign/core build calls (which collide on core's leading rimraf dist) behind the ensure-core-built.mjs filesystem lock — but the variants suite was missed in that migration and still ran its own unguarded check-then-build in beforeAll. The race is latent on main; this PR made it likely to fire, because the 121 node files no longer pay a per-file jsdom boot and now start fast and dense on a fresh CI checkout, so two suites hit the "dist missing → build it" path concurrently.

Fixed in 02b6634 by routing the suite through the shared ensureCoreBuilt() helper, same as the other build-theme suites. The job is green on the current run (316/316 files).

Follow-ups (non-blocking)

  • vitest.workspace.ts/defineWorkspace is deprecated in Vitest 3.2 and removed in Vitest 4 in favor of test.projects in the root config. We're on ^2.1.0 where the workspace file is the supported mechanism; migrate when upgrading.
  • The node project intentionally does not inherit the root config's poolOptions.forks.execArgv: --max-old-space-size=4096 — that heap bump exists for jsdom-heavy suites (e.g. Chat composer) and node-only files don't need it.
  • After merge, track the test job duration distribution on main to confirm how much of the compute saving converts to wall time on 4-vCPU runners.

@vercel

vercel Bot commented Jul 11, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
astryx Ready Ready Preview, Comment Jul 11, 2026 12:31pm

Request Review

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jul 11, 2026
@Han5991 Han5991 marked this pull request as draft July 11, 2026 11:48
@Han5991 Han5991 marked this pull request as ready for review July 11, 2026 12:01
Han5991 added 2 commits July 11, 2026 21:27
Every one of the ~316 test files ran under a single root config: a fresh
jsdom environment per file plus the React/StyleX babel transform — even
for the ~121 files (packages/cli, build, scripts, internal tooling)
that never touch the DOM. Per vitest's own accounting that overhead was
substantial: environment 180s, setup 27s (worker-summed).

Split into two workspace projects:

- ui: packages/core + packages/lab. Extends the root config unchanged
  (jsdom, StyleX babel, jest-dom setup, forks heap bump).
- node: everything else the root include matched. Plain node
  environment, no plugins, no setup file. Stays on the forks pool —
  several CLI tests call process.chdir(), which worker threads don't
  support.

The include lists moved from vitest.config.ts into the workspace file:
workspace `extends` MERGES arrays, so an include list left in the root
config leaks into every extending project and double-runs files (~121
files ran twice in the first attempt).

Partitioning is deliberately exhaustive: ui lists core/lab explicitly,
node takes the same root globs minus core/lab — a new package cannot
silently fall out of CI; it lands in node by default and fails loudly
there if it needs the DOM.

Measured locally (M-series, 10 cores): identical file/test sets
(316 files, 5893 tests), node project alone drops from ~jsdom-dominated
to 50s with environment at 16ms and transform 33s -> 3.5s. Full-run
wall time is roughly unchanged locally (CPU-saturated); the win is
total compute, which should matter more on 4-vCPU CI runners — the PR's
own CI run measures exactly that. Targeted runs (--project node) are
dramatically faster for CLI development.

Also considered and rejected: pool=threads globally (63 failures:
process.chdir unsupported in workers) and isolate:false (2138 failures:
cross-file DOM/module state pollution).
…hared lock

build-theme.variants.test.mjs was the one build-theme suite still running
its own unguarded 'if (!exists) pnpm -F core build' in beforeAll instead
of the ensureCoreBuilt() lock helper that exists precisely because
concurrent core builds collide on packages/core/dist (the build starts
by wiping dist).

The ui/node project split changed file scheduling so the build-theme
suites start near-simultaneously on fresh-checkout CI, which made the
latent race fire: variants' unlocked build ran concurrently with the
lock-holder's build, dist was wiped mid-suite, and the CLI under test
exited 1 on missing core .d.ts files ('expected 1 to be +0' in CI run
29151203513, while the first two tests of the same suite passed).

Verified by deleting packages/core/dist and running all four build-theme
suites concurrently in the node project — all pass with the build
serialized behind the lock.
@Han5991 Han5991 force-pushed the perf/vitest-split-projects branch from 02b6634 to 6453d64 Compare July 11, 2026 12:27
@github-actions github-actions Bot added the community Authored by a community contributor (not on the eng/design team) label Jul 11, 2026

@cixzhang cixzhang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this optimization

@cixzhang cixzhang merged commit 82b47b6 into facebook:main Jul 11, 2026
17 checks passed
@Han5991 Han5991 deleted the perf/vitest-split-projects branch July 11, 2026 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot. community Authored by a community contributor (not on the eng/design team)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants