Repository navigation
Test runs design
How the feature is built. What it does for users is in Test runs; what it must do is in the requirements, and the IDs below (T2, T6…) point there.
flowchart TD
start[Start / save / ▶ Run] --> sm[SessionManager._runFiles]
sm --> queue["buildBatchQueue<br/>per framework · open files first"]
queue --> workers["workers: min(budget, framework maxParallel)"]
workers --> br["batchRunner.runFiles<br/>Jest · Vitest · pytest"]
br --> exec["Executor: spawn, poll the result file"]
exec -->|per file| store[ResultStore + trace + coverage]
br -->|missing files| retry{"all missing?"}
retry -->|"retry hint"| once[run the batch once more]
retry -->|yes| plain[plain run, one file at a time]
retry -->|some| split[split the batch in half, retry]
| File | Role |
|---|---|
src/session/SessionManager.ts → _runFiles, _runTestCases
|
Batches, workers, retries, fallbacks, progress, Stop token |
src/session/batchQueue.ts |
buildBatchQueue (per framework, open files first), promoteInQueue (T2) |
packages/runner/src/AngularRunner.ts, framework/adapters/AngularAdapter.ts, framework/angularWorkspace.ts
|
Angular: ng test <project> with --include / --filter / --list-tests, our reporters, a runner config in the session folder (includeTaskLocation, update, coverage folder from LTR_NG_COVERAGE_DIR, maxWorkers within the budget: the builder has no flag for it), CI=true (Angular's build cache off). Reads angular.json for the projects tested with the unit-test builder (T14) |
src/session/AngularBatchRunner.ts, jsImporters.ts
|
Splits a batch by Angular project over VitestBatchRunner; saves rerun the importing test files (T15) |
src/session/JestBatchRunner.ts, VitestBatchRunner.ts, PytestBatchRunner.ts
|
One framework process per batch with our reporter / plugin, a temp config, streamed results, hang detection |
src/session/IBatchRunner.ts |
The contract: runFiles returns { missing, retry? }
|
src/framework/FrameworkServices.ts → batchPolicy
|
Per framework: maxParallel (pytest: 1 unless the project uses pytest-xdist -n), maxFilesPerBatch, workersPerProcess
|
packages/runner/src/execution/ |
Spawning, killing the process tree, the result file |
src/cache/Storage.ts |
Every cached path, Clear Cache, pruning after 30 days (T12) |
-
Budget (T4, T16, T22, G4).
totalWorkers()from the speed level (setSpeedLevel, fromliveTestRunner.speedinextension.ts): fullfloor(3·cores/4), balancedfloor(cores/2), lightmin(2, floor(cores/4)), at least 1.parallelBatches()=max(1, floor(total / 2))batches at once (light: 1; balanced equals the oldfloor(cores / 4)) (runnerutils/resources.ts). Read when a run starts, so a change applies from the next run. Adaptive (T24, default): unlessliveTestRunner.speedStrictis on, the level is where the budget starts. Before each test process the scheduler asksworkersNow(ours): the machine's busy cores (fromos.cpus()times, at least a second apart) less what our own test processes really used (every process below the extension host, frompsCPU times, sampled at most once a second; a window in which one of them ended keeps the other programs' share from the window before (oursInWindow): its last CPU time is gone with it, and a batch's end looked like 2–3 busy cores just when the next batch asked; on Windows each running worker counts as a busy core) are the other programs' share; the budget is every core they leave free but one, between Light's budget andadaptiveCeiling()(Full's, and no more than the physical cores but one:sysctl hw.physicalcpu,/proc/cpuinfo, WindowsWin32_Processor.NumberOfCoresasked in the background at activation (primePhysicalCores); never below the level's own).maxBatchSlots()gives the run as many slots as that ceiling, and each process is gated by its workers (BatchPolicy.workersPerProcess: Jest and pytest 1, Vitest / AngularworkersPerBatch()), so Jest no longer uses half its share. The check reserves the process and its workers in the same step (_batchesRunning,_workersRunning): checked and counted after an await, every slot passed at once. The open files' batches start 1.5 s before the rest of a full run (OPEN_FILES_HEAD_START_MS: started together with five other Jest processes, the open file's result came 2 s later). Jest batches run with--maxWorkers=1, so a batch is one process. Vitest batches get--maxWorkers=workersPerBatch(): half the cores split across the parallel batches (at least 1), or the project's own lower limit (maxWorkers, a pool'smaxThreads/maxForks,fileParallelism: false), which the config probe reads from the resolved config. Below Vitest 4--minWorkers=1goes with it (a configminWorkersabove the cap would crash the pool; Vitest 4 rejects the flag). Without the cap every batch started Vitest's default of cores − 1 workers. A framework'sbatchPolicy.maxParallelcaps its own processes (pytest: 1). Memory (T19): before a batch or scoped rerun starts beside others,SessionManager._memoryForAnotherBatchasksmemoryLow(available, total, another)(under min(2 GB, total / 5) free onceanotheris taken off: the largest running test process with its workers,largestTestProcessMemory()from the samepspass as the load (macOS / Linux); Sentry's Jest processes reached 1.4 GB each and 6 of them 8.4 GB on a 16 GB machine, fromprocess.availableMemory, elseos.freememexcept on macOS, where it counts only wholly free pages); when low the batch goes back to the front of its queue and waits for a batch to end or 2 s._batchesRunningcounts every run in the window. Priority (T20):ExecutorcallslowerPriority(child.pid)right after spawning (os.setPriority, below normal); workers inherit it. -
Batches. Before them, at Start: the framework's own list of the files it runs (
jest --listTests,pytest --collect-only, Vitest's list), remembered per project until a test or config file changes. What Vitest's config resolves to (projects, coverage include / exclude, worker limit: thevitestProjectsprobe, ~1 s) is remembered too and used at once; a probe checks it in the background (T21). When the list is not remembered, the test files on screen start while it is made (T18; not in a mixed project, whose lists also route files to frameworks, and not with a budget of one process, since the list is a process too). A file the list then leaves out loses its results, coverage contribution and trace entries and is marked excluded. Files are grouped per framework (never mixed). Batch size (batchSizeinbatchQueue.ts) is 5 to 50 files (a framework may raise the top:BatchPolicy.maxFilesPerBatch: Angular 1000, since eachng testbuilds before a test runs (2,500 files: 98 s in 167-file batches, 71.5 s in one per process; plainng test --coverage56 s); pytest 300), chosen per framework so its batches fill whole rounds of its parallel batches (500 files, 3 at a time: 12 batches of 42, not 10 of 50 with a last round of one while the rest of the budget waits). A process's startup is shared across files while results still stream, and a crash or hang retries few files. Open test files form batches of their own at the front;promote()moves a file opened later to the front (T2). - Streaming (T3). Our Jest reporter, Vitest reporter and pytest plugin append one JSON line when a file starts and one when it ends. The batch runner polls that file every 200 ms. A file is marked running only on its start record.
-
Hangs (T5). Per-test heartbeats from the reporter or plugin. A test past
hangTimeoutSecondsstops the batch; its file is failed with the reason; the batch's unfinished files run again. A worker that never reports a test start is stopped after three times the limit without progress. -
No result (T6, T7).
runFilesreturns the files that got no result. If the runner gives aretryreason (a file vanished mid-scan), the batch runs once more. If every file is missing, they fall back to a plain framework run, one at a time within the budget. If only some are missing, the batch is split in half and retried, down to a plain run of a single file. -
Stop (T8). Stop increments the session token and kills every batch runner's process tree. Every retry, fallback and queued batch checks the token first, so nothing starts afterwards. Running files go back to their previous state. The runners stay stopped until the next explicit run without a session (▶ Run, rerun, an AI assistant's run:
_beginRunresets them) or the next Start (T8a); that run, and ▶ Run on a test file the framework's list does not have yet, first ask the framework for its file list again (_askAboutNewTestFile). -
Compile once (T17). Jest names its transform cache and file map after the config file's path. The generated config is therefore one file per content hash in the project's transform-cache folder (
writeStableConfig), and nothing in its text changes per batch or session: the reporter file and the resolved-config JSON come through the environment (LTR_REPORTER_FILE,LTR_JEST_RESOLVED_CONFIG). With a new path per batch, every batch re-instrumented every file and re-crawled the project (2,500 test files: 51 caches, 564 MB, full run 192 s; now one cache, 61 MB, about 115 s). Nor may anything the config evaluates to change per session: Jest copies options such ascoverageDirectoryinto each project config, and that config is part of every file's transform cache key.coverageDirectoryis therefore a fixedjest-coveragefolder in the transform-cache folder, not the session folder (which recompiled the whole project on every Start: Sentry, 100 test files in one process 285 s → 188 s). -
Same results as the terminal (T1, T10). The project's own framework binary and config run; Jest gets a temp config that extends the project's (
jest --showConfig), Vitest a wrapper config, pytest our plugin by-p. The test script's flags are read and passed on.
| Decision | Reason | Alternatives considered |
|---|---|---|
| One budget from the speed level, adapting to free CPU and memory (G4) | Uses an idle machine fully and gives way when the user's other programs need it | A fixed share of the cores (a quarter until 3.4.0): an idle machine stays half used; all cores: the editor stutters |
| Batches, not one process per file | Framework startup is the main cost; batching shares it | One process per file: simple, slow |
| Jest batches stay at most 50 files | One long-lived Jest process slows as it ages on a large real project: Sentry with 167-file batches took 2,756–2,880 s against about 1,476 s with 50 (the 2,500-file synthetic project gained 13% from bigger batches, the real one lost) | 200-file batches |
| One process per batch, not one wide process with every worker (Vitest, Angular) | Vitest computes per-test-file coverage serially on its main thread (~5.6 ms per test file): one wide process took 124 s against 70 s for batches; doing that work during the run slowed Vite's serving (154 s) | One process for the whole run |
| Results streamed from a file we poll | Works the same for all three frameworks and survives a crashed process | Parse stdout: interleaved and truncated on Windows |
| Open files start before the runnable list on the first Start (T18) |
pytest --collect-only takes 29–36 s on 2,500 files and nothing ran meanwhile; the open file is almost always runnable, and the rare one that is not is taken back. Both runs take their processes from one BatchSlots (batchQueue.ts): with their own limit each, a Vitest Start ran four batches at once on a budget of three |
Wait for the list (first result after half a minute); run every discovered file (runs files the config excludes) |
| The Vitest config answer is used from the last session, then checked | Start waited for the probe (0.75–1.2 s on 500 files, 2.8–4.8 s on heavier configs) before the first batch; the stamp (config, package.json, installed Vitest) cannot see files a config imports, so a background probe corrects it for the rest of the session | Wait for the probe every Start; trust the stamp alone (a changed imported file would stay wrong) |
| Speed levels, default balanced (no battery mode) | One setting covers "my laptop heats up" and "use the whole machine" without breaking zero config: the default is today's budget. Battery mode would need the power state from the OS (a new read): left out for now | A worker-count setting (users would have to know their cores); automatic levels from power state |
The level is where the budget starts, adapting by default; speedStrict keeps it |
A level as a hard cap leaves an idle machine half used; adapting to other programs' load keeps the user's work first and uses what is free. Measured on 6 cores / 12 threads, 2,500 files: Jest 110–116 s strict → 96–98 s adaptive, Vitest about the same; with 8 busy programs the run kept 2–4 processes instead of 6 | The level as a cap (the owner's call: an idle machine would never go high); logical cores as the ceiling (9 Jest processes on 6 cores: no faster, three times the memory) |
| Test processes below normal priority; memory gates extra batches, not a lower budget | Priority costs nothing on an idle machine and gives the user the CPU when they need it; the memory gate slows only a machine short of memory (Sentry, 16 GB: 3 Jest processes reached 6.1 GB and the machine swapped 9 GB) | A smaller budget for everyone (slower on every machine); a speed setting (adds config; a later option) |
| Generated Jest config named by its content hash, per-run paths in the environment (T17) | Jest's cache id is the config path: one stable path = one cache shared by every batch and session; a changed text = a new file, so stale code is never served | Set Jest's id in the config (an internal option, not accepted by every Jest version); one fixed file name (two windows with different extension versions would overwrite each other's config) |
| Fall back to plain runs | A config we can't instrument still gives results (G9) | Fail the batch |
spawn, never exec
|
exec buffers output and truncates large runs |
— |
pytest batches reuse a folder's collection (_collect_folders_once in the plugin) |
pytest 8+ collects a file argument's folder again for every file argument: a batch of 2,500 files in folders of 100 made 250,000 nodes (52 s against 15 s); reused only when every argument is a different file, so a file named twice runs as in plain pytest | Folders as arguments plus a filter: different rules for files outside python_files
|
pytest one process at a time, unless the project runs pytest-xdist itself (-n / --numprocesses in its pytest config) |
Session fixtures, test servers on fixed ports and shared files assume it, as in the terminal; a project that already runs in parallel has shown it is safe (2,500 files: 173 s one at a time, plain pytest --cov 61 s) |
Parallel pytest for every project: faster, but flaky for those projects |
| Angular through the builder, not plain Vitest (T14) | Angular's tests need its compilation and TestBed setup; vitest run fails with describe is not defined. The builder takes our reporters and a runner config of our own |
Generate a Vite config from angular.json: re-implements the builder |
| Angular batches up to 1000 files, Vitest workers capped in the runner config | Every ng test builds its files first (~7 s before a test runs), and left alone Vitest starts a worker per core in each batch. 2,500 test files: 52 batches of 49 with uncapped workers took 528 s; 16 batches with 2 workers each 124 s; one batch per process 71.5 s (plain ng test --coverage: 56 s) |
Vitest's default batch size; one ng test for the whole run (no results until the build is done, a crash reruns everything) |
| No partial trace and no lighter discovery parser for large projects | Measured on 2,500-file projects: the trace costs almost nothing (Jest ~0%, Vitest ~3% of the run) and discovery takes 1–5 s; the time was in per-batch caches, coverage work and scheduling | Trace only open / recently changed files; a regex discovery pass before Babel |
One ng test per Angular project |
ng test with no name runs each project in turn, all writing the same result file |
One command for the workspace |
| Coverage only with the project's provider | The builder prepares its build for the provider it finds in the project; a provider outside it measured nothing | Ship a provider (later research) |
scripts/perf-project.js --framework jest|vitest|pytest|angular generates a synthetic project (2,500 test files, 600 sources, 32,800 tests by default); scripts/perf-measure.js <project> runs the e2e scenario diag-perf (discovery, first result, full run, parse times, memory, the window's CPU by process type, one save rerun) and times the plain framework run with and without coverage on the same project (details in the developer guide). Compare the extension with the plain run with coverage measured at the same moment: timings in VS Code vary ±15% between runs.
Same-moment numbers at 3.4.1 (6-core laptop): Jest ≈ plain (Sentry: 1,476 s against jest --coverage 1,460 s), Vitest 500 files 77 s against 56 s, Angular 2,500 files 70 s against 48 s, pytest 2,500 files one process at a time 173 s against 61 s. The remaining gaps are what the features cost: coverage per test file and the covering tests, batches that stream results and keep a crash small.
- e2e:
basic-full-run,basic-open-first,basic-hang,basic-stop,basic-save-during-run,basic-scoped-run,basic-clear-restart,vitest-batch,vitest-hang,vitest-stop,pytest-hang,pytest-stop,mixed-stop,esm-fn-config,ts-config,cra-app, and the real-repo matrix. - Unit:
batchQueue,vanishedFile,storage, runnerexecutor,jestRunnerArgs,vitestRunnerArgs. - Gaps: the crashed-batch fallback (T6) is not triggered by any scenario.
Source: docs/development/design/test-runs.md. This Wiki is generated from the repository; to change a page, open a pull request that edits its file in docs/.
Features
- Test runs
- Test Explorer and editor
- On-save reruns
- Results panel
- Coverage
- Debugging
- Monorepos and mixed frameworks
- AI assistants
Requirements
- Project-wide requirements
- Test runs: requirements
- Test Explorer and editor: requirements
- On-save reruns: requirements
- Results panel: requirements
- Coverage: requirements
- Debugging: requirements
- Monorepos and mixed frameworks: requirements
- AI assistants: requirements
Development
- Developer Guide
- Architecture
- Framework feature matrix
- AI context
- Feature designs
Generated from docs/. Edits made here are overwritten: change a page with a pull request.