diff --git a/.github/instructions/learning.instructions.md b/.github/instructions/learning.instructions.md new file mode 100644 index 000000000000..28b085f486ce --- /dev/null +++ b/.github/instructions/learning.instructions.md @@ -0,0 +1,34 @@ +--- +applyTo: '**' +description: This document describes how to deal with learnings that you make. (meta instruction) +--- + +This document describes how to deal with learnings that you make. +It is a meta-instruction file. + +Structure of learnings: + +- Each instruction file has a "Learnings" section. +- Each learning has a counter that indicates how often that learning was useful (initially 1). +- Each learning has a 1 sentence description of the learning that is clear and concise. + +Example: + +```markdown +## Learnings + +- Prefer `const` over `let` whenever possible (1) +- Avoid `any` type (3) +``` + +When the user tells you "learn!", you should: + +- extract a learning from the recent conversation + _ identify the problem that you created + _ identify why it was a problem + _ identify how you were told to fix it/how the user fixed it + _ generate only one learning (1 sentence) that helps to summarize the insight gained +- then, add the reflected learning to the "Learnings" section of the most appropriate instruction file + +Important: Whenever a learning was really useful, increase the counter!! +When a learning was not useful and just caused more problems, decrease the counter. diff --git a/.github/instructions/testing_feature_area.instructions.md b/.github/instructions/testing_feature_area.instructions.md new file mode 100644 index 000000000000..9036ab4e3acd --- /dev/null +++ b/.github/instructions/testing_feature_area.instructions.md @@ -0,0 +1,181 @@ +--- +applyTo: 'src/client/testing/**' +--- + +# Testing feature area — Discovery, Run, Debug, and Results + +This document maps the testing support in the extension: discovery, execution (run), debugging, result reporting and how those pieces connect to the codebase. It's written for contributors and agents who need to navigate, modify, or extend test support (both `unittest` and `pytest`). + +## Overview + +- Purpose: expose Python tests in the VS Code Test Explorer (TestController), support discovery, run, debug, and surface rich results and outputs. +- Scope: provider-agnostic orchestration + provider-specific adapters, TestController mapping, IPC with Python-side scripts, debug launch integration, and configuration management. + +## High-level architecture + +- Controller / UI bridge: orchestrates TestController requests and routes them to workspace adapters. +- Workspace adapter: provider-agnostic coordinator that translates TestController requests to provider adapters and maps payloads back into TestItems/TestRuns. +- Provider adapters: implement discovery/run/debug for `unittest` and `pytest` by launching Python scripts and wiring named-pipe IPC. +- Result resolver: translates Python-side JSON/IPCPayloads into TestController updates (start/pass/fail/output/attachments). +- Debug launcher: prepares debug sessions and coordinates the debugger attach flow with the Python runner. + +## Key components (files and responsibilities) + +- Entrypoints + - `src/client/testing/testController/controller.ts` — `PythonTestController` (main orchestrator). + - `src/client/testing/serviceRegistry.ts` — DI/wiring for testing services. +- Workspace orchestration + - `src/client/testing/testController/workspaceTestAdapter.ts` — `WorkspaceTestAdapter` (provider-agnostic entry used by controller). +- Provider adapters + - Unittest + - `src/client/testing/testController/unittest/testDiscoveryAdapter.ts` + - `src/client/testing/testController/unittest/testExecutionAdapter.ts` + - Pytest + - `src/client/testing/testController/pytest/pytestDiscoveryAdapter.ts` + - `src/client/testing/testController/pytest/pytestExecutionAdapter.ts` +- Result resolution and helpers + - `src/client/testing/testController/common/resultResolver.ts` — `PythonResultResolver` (maps payload -> TestController updates). + - `src/client/testing/testController/common/testItemUtilities.ts` — helpers for TestItem lifecycle. + - `src/client/testing/testController/common/types.ts` — `ITestDiscoveryAdapter`, `ITestExecutionAdapter`, `ITestResultResolver`, `ITestDebugLauncher`. + - `src/client/testing/testController/common/debugLauncher.ts` — debug session creation helper. + - `src/client/testing/testController/common/utils.ts` — named-pipe helpers and command builders (`startDiscoveryNamedPipe`, etc.). +- Configuration + - `src/client/testing/common/testConfigurationManager.ts` — per-workspace test settings. + - `src/client/testing/configurationFactory.ts` — configuration service factory. +- Utilities & glue + - `src/client/testing/utils.ts` — assorted helpers used by adapters. + - Python-side scripts: `python_files/unittestadapter/*`, `python_files/pytestadapter/*` — discovery/run code executed by adapters. + +## Python subprocess runners (what runs inside Python) + +The adapters in the extension don't implement test discovery/run logic themselves — they spawn a Python subprocess that runs small helper scripts located under `python_files/` and stream structured events back to the extension over the named-pipe IPC. This is a central part of the feature area; changes here usually require coordinated edits in both the TypeScript adapters and the Python scripts. + +- Unittest helpers (folder: `python_files/unittestadapter`) + + - `discovery.py` — performs `unittest` discovery and emits discovery payloads (test suites, cases, locations) on the IPC channel. + - `execution.py` / `django_test_runner.py` — run tests for `unittest` and, where applicable, Django test runners; emit run events (start, stdout/stderr, pass, fail, skip, teardown) and attachment info. + - `pvsc_utils.py`, `django_handler.py` — utility helpers used by the runners for environment handling and Django-specific wiring. + - The adapter TypeScript files (`testDiscoveryAdapter.ts`, `testExecutionAdapter.ts`) construct the command line, start a named-pipe listener, and spawn these Python scripts using the extension's ExecutionFactory (activated interpreter) so the scripts execute inside the user's selected environment. + +- Pytest helpers (folder: `python_files/vscode_pytest`) + + - `_common.py` — shared helpers for pytest runner scripts. + - `run_pytest_script.py` — the primary pytest runner used for discovery and execution; emits the same structured IPC payloads the extension expects (discovery events and run events). + - The `pytest` execution adapter (`pytestExecutionAdapter.ts`) and discovery adapter build the CLI to run `run_pytest_script.py`, start the pipe, and translate incoming payloads via `PythonResultResolver`. + +- IPC contract and expectations + + - Adapters rely on a stable JSON payload contract emitted by the Python scripts: identifiers for tests, event types (discovered, collected, started, passed, failed, skipped), timings, error traces, and optional attachments (logs, captured stdout/stderr, file links). + - The extension maps these payloads to `TestItem`/`TestRun` updates via `PythonResultResolver` (`src/client/testing/testController/common/resultResolver.ts`). If you change payload shape, update the resolver and tests concurrently. + +- How the subprocess is started + - Execution adapters use the extension's `ExecutionFactory` (preferred) to get an activated interpreter and then spawn a child process that runs the helper script. The adapter will set up environment variables and command-line args (including the pipe name / run-id) so the Python runner knows where to send events and how to behave (discovery vs run vs debug). + - For debug sessions a debug-specific entry argument/port is passed and `common/debugLauncher.ts` coordinates starting a VS Code debug session that will attach to the Python process. + +## Core functionality (what to change where) + +- Discovery + - Entry: `WorkspaceTestAdapter.discoverTests` → provider discovery adapter. Adapter starts a named-pipe listener, spawns the discovery script in an activated interpreter, forwards discovery events to `PythonResultResolver` which creates/updates TestItems. + - Files: `workspaceTestAdapter.ts`, `*DiscoveryAdapter.ts`, `resultResolver.ts`, `testItemUtilities.ts`. +- Run / Execution + - Entry: `WorkspaceTestAdapter.executeTests` → provider execution adapter. Adapter spawns runner in an activated env, runner streams run events to the pipe, `PythonResultResolver` updates a `TestRun` with start/pass/fail and attachments. + - Files: `workspaceTestAdapter.ts`, `*ExecutionAdapter.ts`, `resultResolver.ts`. +- Debugging + - Flow: debug request flows like a run but goes through `debugLauncher.ts` to create a VS Code debug session with prepared ports/pipes. The Python runner coordinates attach/continue with the debugger. + - Files: `*ExecutionAdapter.ts`, `common/debugLauncher.ts`, `common/types.ts`. +- Result reporting + - `resultResolver.ts` is the canonical place to change how JSON payloads map to TestController constructs (messages, durations, error traces, attachments). + +## Typical workflows (short) + +- Full discovery + + 1. `PythonTestController` triggers discovery -> `WorkspaceTestAdapter.discoverTests`. + 2. Provider discovery adapter starts pipe and launches Python discovery script. + 3. Discovery events -> `PythonResultResolver` -> TestController tree updated. + +- Run tests + + 1. Controller collects TestItems -> creates `TestRun`. + 2. `WorkspaceTestAdapter.executeTests` delegates to execution adapter which launches the runner. + 3. Runner events arrive via pipe -> `PythonResultResolver` updates `TestRun`. + 4. On process exit the run is finalized. + +- Debug a test + 1. Debug request flows to execution adapter. + 2. Adapter prepares ports and calls `debugLauncher` to start a VS Code debug session with the run ID. + 3. Runner coordinates with the debugger; `PythonResultResolver` still receives and applies run events. + +## Tests and examples to inspect + +- Unit/integration tests for adapters and orchestration under `src/test/` (examples): + - `src/test/testing/common/testingAdapter.test.ts` + - `src/test/testing/testController/workspaceTestAdapter.unit.test.ts` + - `src/test/testing/testController/unittest/testExecutionAdapter.unit.test.ts` + - Adapter tests demonstrate expected telemetry, debug-launch payloads and result resolution. + +## History & evolution (brief) + +- Migration to TestController API: the code organizes around VS Code TestController, mapping legacy adapter behaviour into TestItems/TestRuns. +- Named-pipe IPC: discovery/run use named-pipe IPC to stream events from Python runner scripts (`python_files/*`) which enables richer, incremental updates and debug coordination. +- Environment activation: adapters prefer the extension ExecutionFactory (activated interpreter) to run discovery and test scripts. + +## Pointers for contributors (practical) + +- To extend discovery output: update the Python discovery script in `python_files/*` and `resultResolver.ts` to parse new payload fields. +- To change run behaviour (args/env/timouts): update the provider execution adapter (`*ExecutionAdapter.ts`) and add/update tests under `src/test/`. +- To change debug flow: edit `common/debugLauncher.ts` and adapters' debug paths; update tests that assert launch argument shapes. + +## Django support (how it works) + +- The extension supports Django projects by delegating discovery and execution to Django-aware Python helpers under `python_files/unittestadapter`. + - `python_files/unittestadapter/django_handler.py` contains helpers that invoke `manage.py` for discovery or execute Django test runners inside the project context. + - `python_files/unittestadapter/django_test_runner.py` provides `CustomDiscoveryTestRunner` and `CustomExecutionTestRunner` which integrate with the extension by using the same IPC contract (they use `UnittestTestResult` and `send_post_request` to emit discovery/run payloads). +- How adapters pass Django configuration: + - Execution adapters set environment variables (e.g. `MANAGE_PY_PATH`) and modify `PYTHONPATH` so Django code and the custom test runner are importable inside the spawned subprocess. + - For discovery the adapter may run the discovery helper which calls `manage.py test` with a custom test runner that emits discovery payloads instead of executing tests. +- Practical notes for contributors: + - Changes to Django discovery/execution often require edits in both `django_test_runner.py`/`django_handler.py` and the TypeScript adapters (`testDiscoveryAdapter.ts` / `testExecutionAdapter.ts`). + - The Django test runner expects `TEST_RUN_PIPE` environment variable to be present to send IPC events (see `django_test_runner.py`). + +## Settings referenced by this feature area + +- The extension exposes several `python.testing.*` settings used by adapters and configuration code (declared in `package.json`): + - `python.testing.pytestEnabled`, `python.testing.unittestEnabled` — enable/disable frameworks. + - `python.testing.pytestPath`, `python.testing.pytestArgs`, `python.testing.unittestArgs` — command path and CLI arguments used when spawning helper scripts. + - `python.testing.cwd` — optional working directory used when running discovery/runs. + - `python.testing.autoTestDiscoverOnSaveEnabled`, `python.testing.autoTestDiscoverOnSavePattern` — control automatic discovery on save. + - `python.testing.debugPort` — default port used for debug runs. + - `python.testing.promptToConfigure` — whether to prompt users to configure tests when potential test folders are found. +- Where to look in the code: + - Settings are consumed by `src/client/testing/common/testConfigurationManager.ts`, `src/client/testing/configurationFactory.ts`, and adapters under `src/client/testing/testController/*` which read settings to build CLI args and env for subprocesses. + - The setting definitions and descriptions are in `package.json` and localized strings in `package.nls.json`. + +## Coverage support (how it works) + +- Coverage is supported by running the Python helper scripts with coverage enabled and then collecting a coverage payload from the runner. + - Pytest-side coverage logic lives in `python_files/vscode_pytest/__init__.py` (checks `COVERAGE_ENABLED`, imports `coverage`, computes per-file metrics and emits a `CoveragePayloadDict`). + - Unittest adapters enable coverage by setting environment variable(s) (e.g. `COVERAGE_ENABLED`) when launching the subprocess; adapters and `resultResolver.ts` handle the coverage profile kind (`TestRunProfileKind.Coverage`). +- Flow summary: + 1. User starts a Coverage run via Test Explorer (profile kind `Coverage`). + 2. Controller/adapters set `COVERAGE_ENABLED` (or equivalent) in the subprocess env and invoke the runner script. + 3. The Python runner collects coverage (using `coverage` or `pytest-cov`), builds a file-level coverage map, and sends a coverage payload back over the IPC. + 4. `PythonResultResolver` (`src/client/testing/testController/common/resultResolver.ts`) receives the coverage payload and stores `detailedCoverageMap` used by the TestController profile to show file-level coverage details. +- Tests that exercise coverage flows are under `src/test/testing/*` and `python_files/tests/*` (see `testingAdapter.test.ts` and adapter unit tests that assert `COVERAGE_ENABLED` is set appropriately). + +## Interaction with the VS Code API + +- TestController API + - The feature area is built on VS Code's TestController/TestItem/TestRun APIs (`vscode.tests.createTestController` / `tests.createTestController` in the code). The controller creates a `TestController` in `src/client/testing/testController/controller.ts` and synchronizes `TestItem` trees with discovery payloads. + - `PythonResultResolver` maps incoming JSON events to VS Code API calls: `testRun.appendOutput`, `testRun.passed/failed/skipped`, `testRun.end`, and `TestItem` updates (labels, locations, children). +- Debug API + - Debug runs use the Debug API to start an attach/launch session. The debug launcher implementation is in `src/client/testing/testController/common/debugLauncher.ts` which constructs a debug configuration and calls the VS Code debug API to start a session (e.g. `vscode.debug.startDebugging`). + - Debug adapter/resolver code in the extension's debugger modules may also be used when attaching to Django or test subprocesses. +- Commands and configuration + - The Test Controller wires commands that appear in the Test Explorer and editor context menus (see `package.json` contributes `commands`) and listens to configuration changes filtered by `python.testing` in `src/client/testing/main.ts`. +- Execution factory & activated environments + - Adapters use the extension `ExecutionFactory` to spawn subprocesses in an activated interpreter (so the user's venv/conda is used). This involves the extension's internal environment execution APIs and sometimes `envExt` helpers when the external environment extension is present. + +## Learnings + +- Never await `showErrorMessage()` calls in test execution adapters as it blocks the test UI thread and freezes the Test Explorer (1)