Repository navigation
Test runs requirements
github-actions[bot] edited this page Nov 19, 2026
·
4 revisions
What the feature must do, and how we check it does. For the plain-language description see Test runs; for how it is built see the design. The project-wide requirements apply too, above all G2 (never touch the project), G3 (same results as the user's own runs), G4 (the resource budget) and G5 (nothing before Start, nothing after Stop).
| # | The extension must… | Checked by |
|---|---|---|
| T1 | Give the same result for every test as the framework's own run of the project |
vitest-full-run, pytest-full-run, cra-app, the real-repo matrix |
| T2 | Run the test files open in the editor first, and move a test file opened during the run to the front |
basic-open-first; unit: batchQueue
|
| T3 | Show a file as running only once the framework starts it, and its result as soon as it ends |
vitest-batch, basic-full-run
|
| T4 | Run batches in parallel within the budget; pytest one process at a time | unit: batchQueue, resources
|
| T5 | Stop a test that runs longer than hangTimeoutSeconds (timed by the extension's clock, not the test's: fake timers do not count), fail its file with the reason, and finish the other files |
basic-hang, vitest-hang, pytest-hang
|
| T6 | Retry the files of a crashed batch in smaller batches, then on their own with a plain run; log a warning | |
| T7 | Retry a batch once when a file vanished while the framework scanned the project (tests that create and delete files) | unit: vanishedFile
|
| T8 | On Stop, end every framework process and its children; start no retry, fallback or queued batch; leave nothing spinning |
basic-stop, vitest-stop, pytest-stop, mixed-stop
|
| T8a | After Stop, explicit runs (▶ Run, rerun, an AI assistant's run) still run: Stop ends what was in flight, not later clicks; a test file the framework's list (asked at Start) lacks is asked about again before it is run |
new-tests-and-stop, new-tests-and-stop-vitest, new-tests-and-stop-pytest
|
| T9 | Announce "finished" once, after the last run in flight; progress never above 100% |
basic-full-run, basic-save-during-run
|
| T10 | Run the project's own framework with its config, setup files and test-script flags (--config, --resetMocks, Node options) |
basic-full-run, esm-fn-config, ts-config, jest-svelte-esm
|
| T11 | Pass names to the framework intact: spaces, ^, |, &
|
basic-scoped-run |
| T12 | Keep caches outside the project; Clear Cache deletes a project's data; delete data unused for 30 days |
basic-clear-restart, commands; unit: storage
|
| T14 | Angular 21+ (Vitest through Angular's unit-test builder): detected from angular.json (not run as plain Vitest), each Angular project run with ng test <name>; results with line numbers, per-test output, coverage with the project's provider, a single test by name, Update snapshot; nothing written into the project (Angular's build cache off, coverage in the session folder); Vitest's workers within the budget and batches of up to 200 files (each ng test builds first); Karma projects get a clear message |
angular-full-run, angular-debug, angular-snapshot-update; unit: angularRunner, frameworkRegistry, batchQueue
|
| T15 | Angular saves: rerun the test files that import the saved file (relative imports), a template's component tests, else every test of its project |
angular-full-run; unit: jsImporters
|
| T13 | Without the framework installed, or with an environment that can't run it, run nothing, mark nothing failed, and say how to fix it | pytest-no-pytest |
| T16 | Vitest batches together use at most half the cores: each gets --maxWorkers from the shared budget, a lower limit in the project's config is kept, and --minWorkers=1 is added below Vitest 4 only |
unit: resources, vitestRunnerArgs; e2e: vitest-full-run, vitest-batch (real Vitest accepts the flags) |
| T17 | Compile a source file once, not once per batch: every batch and later session of a project reuses the framework's compiled files (Jest: one generated config per config text, so one transform cache and file map); a changed config or extension never reuses stale output | unit: stableJestConfig; e2e: basic-full-run (several batches, one cache; a second Start writes no cache entry) |
| T18 | When the framework's list of runnable files is not remembered from the last session, start the test files on screen while the list is made (single-framework projects, budget of two or more processes); the early batches and the rest of the run share one budget (BatchSlots); a file the list leaves out loses its results, coverage and trace and shows as not run |
pytest-open-first-while-listing, basic-open-first; unit batchQueue
|
| T19 | Start a run's first test process always, and another (any run in the window) only while free memory, less what the new process will take (the largest running test process with its workers, where it can be measured), stays at least 2 GB, or a fifth of the machine's memory when that is less; otherwise wait until one ends or memory frees up, logging it once. Free memory that cannot be told (macOS before VS Code 1.90) holds nothing back | unit: resources |
| T20 | Run every test process (and the workers it starts) below normal priority: Unix nice 10, Windows below normal |
basic-hang, vitest-hang, pytest-hang; unit: resources
|
| T21 | Start the first Vitest batch without waiting for the config probe when the config, package.json and installed Vitest are unchanged since the last session: use that session's answer, check it with a probe in the background, and use a different answer for the rest of the session and the next (coverage globs compared as a set) | unit: vitestConfigMemory, storage; repo-vitest (vueuse, second run) |
| T22 | Size the budget by liveTestRunner.speed: full 3/4 of the cores, balanced (default, unknown values too) half, light one test process with at most 2 workers; with speedStrict every level stays within its budget (otherwise see T24); a change applies from the next run and is logged once |
basic-speed; unit: resources
|
| T23 | Never run more processes of a framework at once than its limit allows (pytest: one, as the user's own pytest: projects share ports, files and session fixtures; as many as the budget allows when the project runs pytest in parallel itself, pytest-xdist workers in its addopts), whatever else waits (memory, other runs) |
unit: pytest (addopts that run in parallel); perf-measure.js: one pytest process at a time on 2,500 files, three with -n 2 in addopts) |
| T24 | Unless liveTestRunner.speedStrict is on, adapt the budget before each test process: the cores other programs leave free (the machine's busy cores less what our test processes used) but one, between Light's budget and Full's (no more than the physical cores but one, Windows included; never below the level's own); the level until measured; Jest and pytest processes count one worker each; the open files' batches start before the rest of a full run |
unit: resources (incl. oursInWindow: a test process that ended in the window); e2e: basic-speed (strict and adaptive lines); measured with perf-measure.js (8 busy programs: 2–4 Jest processes instead of 6) |
| T25 | A pytest batch collects each folder once (not once per test file in it), with the same tests as plain pytest; a file named twice runs as plain pytest runs it | plugin tests test_a_batch_of_files_collects_their_folder_once, test_a_file_given_twice_runs_as_plain_pytest_runs_it; measured with perf-measure.js
|
| # | Requirement |
|---|---|
| Q1 | The framework always runs in a child process, never inside VS Code's extension host. |
| Q2 | Processes are spawned (streamed), never exec'd (buffered and truncated). |
| Q3 | Every run writes one INFO summary line: what started it, how many files and tests, how long. |
- Running tests on another machine or in a container.
Source: docs/requirements/test-runs.md. This Wiki is generated from the repository; to change a page, open a pull request that edits its file in docs/.
Features
- Test runs
- Test Explorer and editor
- On-save reruns
- Results panel
- Coverage
- Debugging
- Monorepos and mixed frameworks
- AI assistants
Requirements
- Project-wide requirements
- Test runs: requirements
- Test Explorer and editor: requirements
- On-save reruns: requirements
- Results panel: requirements
- Coverage: requirements
- Debugging: requirements
- Monorepos and mixed frameworks: requirements
- AI assistants: requirements
Development
- Developer Guide
- Architecture
- Framework feature matrix
- AI context
- Feature designs
Generated from docs/. Edits made here are overwritten: change a page with a pull request.