Skip to content

Add meta-instruction files for learnings and testing feature area #25473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 20, 2025
Merged

Add meta-instruction files for learnings and testing feature area #25473

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/instructions/learning.instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
applyTo: '**'
description: This document describes how to deal with learnings that you make. (meta instruction)
---

This document describes how to deal with learnings that you make.
It is a meta-instruction file.

Structure of learnings:

- Each instruction file has a "Learnings" section.
- Each learning has a counter that indicates how often that learning was useful (initially 1).
- Each learning has a 1 sentence description of the learning that is clear and concise.

Example:

```markdown
## Learnings

- Prefer `const` over `let` whenever possible (1)
- Avoid `any` type (3)
```

When the user tells you "learn!", you should:

- extract a learning from the recent conversation
_ identify the problem that you created
_ identify why it was a problem
_ identify how you were told to fix it/how the user fixed it
_ generate only one learning (1 sentence) that helps to summarize the insight gained
- then, add the reflected learning to the "Learnings" section of the most appropriate instruction file

Important: Whenever a learning was really useful, increase the counter!!
When a learning was not useful and just caused more problems, decrease the counter.
181 changes: 181 additions & 0 deletions .github/instructions/testing_feature_area.instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
---
applyTo: 'src/client/testing/**'
---

# Testing feature area — Discovery, Run, Debug, and Results

This document maps the testing support in the extension: discovery, execution (run), debugging, result reporting and how those pieces connect to the codebase. It's written for contributors and agents who need to navigate, modify, or extend test support (both `unittest` and `pytest`).

## Overview

- Purpose: expose Python tests in the VS Code Test Explorer (TestController), support discovery, run, debug, and surface rich results and outputs.
- Scope: provider-agnostic orchestration + provider-specific adapters, TestController mapping, IPC with Python-side scripts, debug launch integration, and configuration management.

## High-level architecture

- Controller / UI bridge: orchestrates TestController requests and routes them to workspace adapters.
- Workspace adapter: provider-agnostic coordinator that translates TestController requests to provider adapters and maps payloads back into TestItems/TestRuns.
- Provider adapters: implement discovery/run/debug for `unittest` and `pytest` by launching Python scripts and wiring named-pipe IPC.
- Result resolver: translates Python-side JSON/IPCPayloads into TestController updates (start/pass/fail/output/attachments).
- Debug launcher: prepares debug sessions and coordinates the debugger attach flow with the Python runner.

## Key components (files and responsibilities)

- Entrypoints
- `src/client/testing/testController/controller.ts` — `PythonTestController` (main orchestrator).
- `src/client/testing/serviceRegistry.ts` — DI/wiring for testing services.
- Workspace orchestration
- `src/client/testing/testController/workspaceTestAdapter.ts` — `WorkspaceTestAdapter` (provider-agnostic entry used by controller).
- Provider adapters
- Unittest
- `src/client/testing/testController/unittest/testDiscoveryAdapter.ts`
- `src/client/testing/testController/unittest/testExecutionAdapter.ts`
- Pytest
- `src/client/testing/testController/pytest/pytestDiscoveryAdapter.ts`
- `src/client/testing/testController/pytest/pytestExecutionAdapter.ts`
- Result resolution and helpers
- `src/client/testing/testController/common/resultResolver.ts` — `PythonResultResolver` (maps payload -> TestController updates).
- `src/client/testing/testController/common/testItemUtilities.ts` — helpers for TestItem lifecycle.
- `src/client/testing/testController/common/types.ts` — `ITestDiscoveryAdapter`, `ITestExecutionAdapter`, `ITestResultResolver`, `ITestDebugLauncher`.
- `src/client/testing/testController/common/debugLauncher.ts` — debug session creation helper.
- `src/client/testing/testController/common/utils.ts` — named-pipe helpers and command builders (`startDiscoveryNamedPipe`, etc.).
- Configuration
- `src/client/testing/common/testConfigurationManager.ts` — per-workspace test settings.
- `src/client/testing/configurationFactory.ts` — configuration service factory.
- Utilities & glue
- `src/client/testing/utils.ts` — assorted helpers used by adapters.
- Python-side scripts: `python_files/unittestadapter/*`, `python_files/pytestadapter/*` — discovery/run code executed by adapters.

## Python subprocess runners (what runs inside Python)

The adapters in the extension don't implement test discovery/run logic themselves — they spawn a Python subprocess that runs small helper scripts located under `python_files/` and stream structured events back to the extension over the named-pipe IPC. This is a central part of the feature area; changes here usually require coordinated edits in both the TypeScript adapters and the Python scripts.

- Unittest helpers (folder: `python_files/unittestadapter`)

- `discovery.py` — performs `unittest` discovery and emits discovery payloads (test suites, cases, locations) on the IPC channel.
- `execution.py` / `django_test_runner.py` — run tests for `unittest` and, where applicable, Django test runners; emit run events (start, stdout/stderr, pass, fail, skip, teardown) and attachment info.
- `pvsc_utils.py`, `django_handler.py` — utility helpers used by the runners for environment handling and Django-specific wiring.
- The adapter TypeScript files (`testDiscoveryAdapter.ts`, `testExecutionAdapter.ts`) construct the command line, start a named-pipe listener, and spawn these Python scripts using the extension's ExecutionFactory (activated interpreter) so the scripts execute inside the user's selected environment.

- Pytest helpers (folder: `python_files/vscode_pytest`)

- `_common.py` — shared helpers for pytest runner scripts.
- `run_pytest_script.py` — the primary pytest runner used for discovery and execution; emits the same structured IPC payloads the extension expects (discovery events and run events).
- The `pytest` execution adapter (`pytestExecutionAdapter.ts`) and discovery adapter build the CLI to run `run_pytest_script.py`, start the pipe, and translate incoming payloads via `PythonResultResolver`.

- IPC contract and expectations

- Adapters rely on a stable JSON payload contract emitted by the Python scripts: identifiers for tests, event types (discovered, collected, started, passed, failed, skipped), timings, error traces, and optional attachments (logs, captured stdout/stderr, file links).
- The extension maps these payloads to `TestItem`/`TestRun` updates via `PythonResultResolver` (`src/client/testing/testController/common/resultResolver.ts`). If you change payload shape, update the resolver and tests concurrently.

- How the subprocess is started
- Execution adapters use the extension's `ExecutionFactory` (preferred) to get an activated interpreter and then spawn a child process that runs the helper script. The adapter will set up environment variables and command-line args (including the pipe name / run-id) so the Python runner knows where to send events and how to behave (discovery vs run vs debug).
- For debug sessions a debug-specific entry argument/port is passed and `common/debugLauncher.ts` coordinates starting a VS Code debug session that will attach to the Python process.

## Core functionality (what to change where)

- Discovery
- Entry: `WorkspaceTestAdapter.discoverTests` → provider discovery adapter. Adapter starts a named-pipe listener, spawns the discovery script in an activated interpreter, forwards discovery events to `PythonResultResolver` which creates/updates TestItems.
- Files: `workspaceTestAdapter.ts`, `*DiscoveryAdapter.ts`, `resultResolver.ts`, `testItemUtilities.ts`.
- Run / Execution
- Entry: `WorkspaceTestAdapter.executeTests` → provider execution adapter. Adapter spawns runner in an activated env, runner streams run events to the pipe, `PythonResultResolver` updates a `TestRun` with start/pass/fail and attachments.
- Files: `workspaceTestAdapter.ts`, `*ExecutionAdapter.ts`, `resultResolver.ts`.
- Debugging
- Flow: debug request flows like a run but goes through `debugLauncher.ts` to create a VS Code debug session with prepared ports/pipes. The Python runner coordinates attach/continue with the debugger.
- Files: `*ExecutionAdapter.ts`, `common/debugLauncher.ts`, `common/types.ts`.
- Result reporting
- `resultResolver.ts` is the canonical place to change how JSON payloads map to TestController constructs (messages, durations, error traces, attachments).

## Typical workflows (short)

- Full discovery

1. `PythonTestController` triggers discovery -> `WorkspaceTestAdapter.discoverTests`.
2. Provider discovery adapter starts pipe and launches Python discovery script.
3. Discovery events -> `PythonResultResolver` -> TestController tree updated.

- Run tests

1. Controller collects TestItems -> creates `TestRun`.
2. `WorkspaceTestAdapter.executeTests` delegates to execution adapter which launches the runner.
3. Runner events arrive via pipe -> `PythonResultResolver` updates `TestRun`.
4. On process exit the run is finalized.

- Debug a test
1. Debug request flows to execution adapter.
2. Adapter prepares ports and calls `debugLauncher` to start a VS Code debug session with the run ID.
3. Runner coordinates with the debugger; `PythonResultResolver` still receives and applies run events.

## Tests and examples to inspect

- Unit/integration tests for adapters and orchestration under `src/test/` (examples):
- `src/test/testing/common/testingAdapter.test.ts`
- `src/test/testing/testController/workspaceTestAdapter.unit.test.ts`
- `src/test/testing/testController/unittest/testExecutionAdapter.unit.test.ts`
- Adapter tests demonstrate expected telemetry, debug-launch payloads and result resolution.

## History & evolution (brief)

- Migration to TestController API: the code organizes around VS Code TestController, mapping legacy adapter behaviour into TestItems/TestRuns.
- Named-pipe IPC: discovery/run use named-pipe IPC to stream events from Python runner scripts (`python_files/*`) which enables richer, incremental updates and debug coordination.
- Environment activation: adapters prefer the extension ExecutionFactory (activated interpreter) to run discovery and test scripts.

## Pointers for contributors (practical)

- To extend discovery output: update the Python discovery script in `python_files/*` and `resultResolver.ts` to parse new payload fields.
- To change run behaviour (args/env/timouts): update the provider execution adapter (`*ExecutionAdapter.ts`) and add/update tests under `src/test/`.
- To change debug flow: edit `common/debugLauncher.ts` and adapters' debug paths; update tests that assert launch argument shapes.

## Django support (how it works)

- The extension supports Django projects by delegating discovery and execution to Django-aware Python helpers under `python_files/unittestadapter`.
- `python_files/unittestadapter/django_handler.py` contains helpers that invoke `manage.py` for discovery or execute Django test runners inside the project context.
- `python_files/unittestadapter/django_test_runner.py` provides `CustomDiscoveryTestRunner` and `CustomExecutionTestRunner` which integrate with the extension by using the same IPC contract (they use `UnittestTestResult` and `send_post_request` to emit discovery/run payloads).
- How adapters pass Django configuration:
- Execution adapters set environment variables (e.g. `MANAGE_PY_PATH`) and modify `PYTHONPATH` so Django code and the custom test runner are importable inside the spawned subprocess.
- For discovery the adapter may run the discovery helper which calls `manage.py test` with a custom test runner that emits discovery payloads instead of executing tests.
- Practical notes for contributors:
- Changes to Django discovery/execution often require edits in both `django_test_runner.py`/`django_handler.py` and the TypeScript adapters (`testDiscoveryAdapter.ts` / `testExecutionAdapter.ts`).
- The Django test runner expects `TEST_RUN_PIPE` environment variable to be present to send IPC events (see `django_test_runner.py`).

## Settings referenced by this feature area

- The extension exposes several `python.testing.*` settings used by adapters and configuration code (declared in `package.json`):
- `python.testing.pytestEnabled`, `python.testing.unittestEnabled` — enable/disable frameworks.
- `python.testing.pytestPath`, `python.testing.pytestArgs`, `python.testing.unittestArgs` — command path and CLI arguments used when spawning helper scripts.
- `python.testing.cwd` — optional working directory used when running discovery/runs.
- `python.testing.autoTestDiscoverOnSaveEnabled`, `python.testing.autoTestDiscoverOnSavePattern` — control automatic discovery on save.
- `python.testing.debugPort` — default port used for debug runs.
- `python.testing.promptToConfigure` — whether to prompt users to configure tests when potential test folders are found.
- Where to look in the code:
- Settings are consumed by `src/client/testing/common/testConfigurationManager.ts`, `src/client/testing/configurationFactory.ts`, and adapters under `src/client/testing/testController/*` which read settings to build CLI args and env for subprocesses.
- The setting definitions and descriptions are in `package.json` and localized strings in `package.nls.json`.

## Coverage support (how it works)

- Coverage is supported by running the Python helper scripts with coverage enabled and then collecting a coverage payload from the runner.
- Pytest-side coverage logic lives in `python_files/vscode_pytest/__init__.py` (checks `COVERAGE_ENABLED`, imports `coverage`, computes per-file metrics and emits a `CoveragePayloadDict`).
- Unittest adapters enable coverage by setting environment variable(s) (e.g. `COVERAGE_ENABLED`) when launching the subprocess; adapters and `resultResolver.ts` handle the coverage profile kind (`TestRunProfileKind.Coverage`).
- Flow summary:
1. User starts a Coverage run via Test Explorer (profile kind `Coverage`).
2. Controller/adapters set `COVERAGE_ENABLED` (or equivalent) in the subprocess env and invoke the runner script.
3. The Python runner collects coverage (using `coverage` or `pytest-cov`), builds a file-level coverage map, and sends a coverage payload back over the IPC.
4. `PythonResultResolver` (`src/client/testing/testController/common/resultResolver.ts`) receives the coverage payload and stores `detailedCoverageMap` used by the TestController profile to show file-level coverage details.
- Tests that exercise coverage flows are under `src/test/testing/*` and `python_files/tests/*` (see `testingAdapter.test.ts` and adapter unit tests that assert `COVERAGE_ENABLED` is set appropriately).

## Interaction with the VS Code API

- TestController API
- The feature area is built on VS Code's TestController/TestItem/TestRun APIs (`vscode.tests.createTestController` / `tests.createTestController` in the code). The controller creates a `TestController` in `src/client/testing/testController/controller.ts` and synchronizes `TestItem` trees with discovery payloads.
- `PythonResultResolver` maps incoming JSON events to VS Code API calls: `testRun.appendOutput`, `testRun.passed/failed/skipped`, `testRun.end`, and `TestItem` updates (labels, locations, children).
- Debug API
- Debug runs use the Debug API to start an attach/launch session. The debug launcher implementation is in `src/client/testing/testController/common/debugLauncher.ts` which constructs a debug configuration and calls the VS Code debug API to start a session (e.g. `vscode.debug.startDebugging`).
- Debug adapter/resolver code in the extension's debugger modules may also be used when attaching to Django or test subprocesses.
- Commands and configuration
- The Test Controller wires commands that appear in the Test Explorer and editor context menus (see `package.json` contributes `commands`) and listens to configuration changes filtered by `python.testing` in `src/client/testing/main.ts`.
- Execution factory & activated environments
- Adapters use the extension `ExecutionFactory` to spawn subprocesses in an activated interpreter (so the user's venv/conda is used). This involves the extension's internal environment execution APIs and sometimes `envExt` helpers when the external environment extension is present.

## Learnings

- Never await `showErrorMessage()` calls in test execution adapters as it blocks the test UI thread and freezes the Test Explorer (1)
Loading