Contributor experience: startup cost when iterating on tests

[Pr1p]

Contributor experience: startup cost when iterating on tests

I’d like to share a local development observation: for commands targeting a small number of tests, startup and loading time can sometimes far exceed the time spent running the tests themselves.

These are a few Windows runs using the repository’s Vitest configuration. They are examples, not a controlled benchmark, and the commands have different dependency scopes.

Test command	Test result	Total time
agent-launch-failure-code.test.ts	3 passed	0.28 s
plugin-host-methods.test.ts	20 passed	about 3–5 s
orca-runtime.test.ts, filtered to 4 cases	4 passed, 1,354 skipped	about 54 s

In one runtime test run, Vitest reported that about 86% of the time went to transform, about 13% to import, and about 1% to the tests themselves. This suggests that for some tests, preparing the test environment and loading dependencies can dominate even when only a few cases are selected.

I understand that the recent test-suite reductions and owner-based restructuring are already addressing maintenance costs. This post is meant to add the external-contributor perspective: is there a recommended fast iteration workflow, or a lightweight runner for logic-only tests?

For example, a Vitest config that skips jsdom and Electron setup could help with tests for parsers, state machines, and reducers that do not need a UI environment. If ongoing work already covers this, pointers would be welcome.

Wall-clock time (each █ is about 5 seconds)

Pure logic test    0.28 s   ▏
Plugin host        ~3–5 s   █
Filtered runtime   ~54 s    ███████████

[OrcaWin]

Thanks for measuring this. You're right, and the numbers pointed us at the actual cause.

Skipping jsdom won't help. The unit config already runs in environment: 'node', and the DOM setup only loads when a DOM environment is present. A separate "logic-only" config would run the same pipeline.

orca-runtime.test.ts is the outlier. It's one entry file that imports ~146 spec fragments (~58k lines). A -t filter still transforms every one of them, which is why you saw 4 passed / 1,354 skipped in 54s. That's a structural problem on our side, not something you were doing wrong.

Workaround for now: run through pnpm test <path>, not vitest / npx vitest directly. The wrapper turns on a persistent transform cache (node_modules/.cache/orca-vitest-modules), and raw vitest skips it. On Linux, with the same filtered runtime run:

Command	Time
npx vitest run …orca-runtime.test.ts -t …	~31s
pnpm test …orca-runtime.test.ts -t …, first run	~26s
pnpm test …orca-runtime.test.ts -t …, next runs	~6s

The cache is keyed on the lockfile, tsconfig and runner scripts, so it survives normal edits. Most small test files are already in your 0.3–5s range.

What we're changing: we're going to split the runtime suite so each part loads on its own and the aggregator goes away. Then targeting one area won't pay for the other 145. We'll also document the cache in CONTRIBUTING's "Targeting Local Tests" section.

[nwparker]

[nwparker]

If still an issue feel free to re-oepn