Skip to content

Project wide requirements

github-actions[bot] edited this page Nov 19, 2026 · 4 revisions

Project-wide requirements

These apply to every feature. A change that breaks one of them is a bug, even if every feature still works.

# Requirement Why How it is checked
G1 Zero configuration. Install the extension, open a project, click Start. No setup steps, no config file of ours. The tool must be faster to adopt than running tests by hand Every e2e fixture is a plain project with no extension config
G2 Never touch the user's project. Never install packages into it or its environment, and never edit its code, config or files. Anything we write is a temporary file in our own session folder. The one exception is Update snapshot: only after the user says yes in a dialog that names the file, and the test framework itself rewrites that test's snapshot. The user's repository and their own terminal runs must stay exactly as they were pytest-full-run, pytest-coverage and vitest-coverage check that nothing is written into the project; the real-repo matrix runs on copies of real projects; snapshot-update / vitest-snapshot-update (nothing written when the dialog is declined; only that test's snapshot otherwise)
G3 The user's own test runs keep working side by side with ours: same results, no shared caches or lock files. People still run jest, vitest or pytest in a terminal and in CI The real-repo matrix compares our results with the framework's own run, test for test
G4 Stay within the machine's budget. Every kind of parallelism (batches, frameworks, projects, workers) shares one budget sized from the free cores and memory. When the budget is 1, things run one at a time. Whatever else the user does must stay responsive Unit tests on the batch queue, project limits and resources (memory check, priority); e2e basic-hang, vitest-hang, pytest-hang (a test reads its own priority). The budget is half the cores by default (liveTestRunner.speed: Full 3/4, Light one process with at most 2 workers, T22): parallel batches get one per two workers, Vitest workers split the rest (T16); another batch starts only while memory is not low (T19); test processes run below normal priority (T20). ⚠️ No e2e of the memory wait (it needs a machine short of memory)
G5 Nothing runs in the background before Start, and nothing after Stop. Explicit clicks (▶ Run, ▷ Debug, rerun buttons) always run, because the user asked for exactly that. Resource use is predictable and under the user's control vitest-session, basic-stop, pytest-stop and similar e2e scenarios
G6 One window never disturbs another. Each VS Code window works in its own session folder. People open several projects at once Session folders are named by process id; stale ones are pruned on start
G7 Show only what was measured. A test that was never run on its own shows a placeholder, not output borrowed from its file. Wrong information is worse than missing information Results-store unit tests; output scenarios
G8 Windows, macOS and Linux behave the same: paths with spaces, symlinks, short names, drive-letter case. Users are on all three The e2e suite runs on all three in CI
G9 Fail soft and say why. When something cannot run the best way (no trace, no coverage package, an unknown config), tests still run in a simpler way, and the log or the UI says what is missing. A degraded result is better than none, and the user can fix the cause Fallback scenarios (pytest-no-coverage, trace-off runs)
G10 Supported versions follow a written rule. A Node major is supported until 2 years after its end of life; the minimum VS Code is the first whose bundled Node is still supported; everything we ship runs on the oldest supported Node. Users on older editors and Node keep working as long as their platform is maintained, and we stop carrying versions with no security fixes scripts/support-floor.js --check in the PR checks; the monthly support-floor.yml workflow opens an issue when the minimum can move; the PR job E2E (minimum VS Code) runs the main scenarios on that version
G11 No known vulnerabilities in what we ship, and none high or critical in our tooling. Users and their security teams install code we vouch for; our CI and machines run the rest scripts/security-audit.js and dependency review in security.yml (every PR, weekly); CodeQL for our own code
G12 No network calls, no telemetry. The extension talks to nothing outside the machine: no analytics, no error reporting, no accounts, no remote resources in its views (the built-in guide included). Links open in the browser only when clicked. AI features hand data only to tools the user runs, and only on their action: the clipboard, VS Code's language-model tools, a local MCP server over stdio. The extension never calls an AI model or service itself and never opens a network port. What it keeps is listed in PRIVACY.md, which changes with it. Users, and their companies' security teams, must be able to trust it with private code packages/vscode-extension/test/noNetwork.test.ts (bundle modules, the MCP relay bundle, sources, shipped code, dependencies incl. AI-service SDKs, webview CSPs incl. the AI setup page, the shipped guide); guidePanel.test.ts (the guide's build step); aiCatalog.test.ts (assistants get no Start / Stop / settings / command tools)

The IDs (G1, G2…) are for discussing these rules in issues and reviews. Code and tests don't refer to them; each comment explains its own reason.


Source: docs/requirements/project-rules.md. This Wiki is generated from the repository; to change a page, open a pull request that edits its file in docs/.

Clone this wiki locally