Repository navigation
Affected Test Selection 2026 10 30
On a pull request most of a suite tests code the change didn't touch. Running only the tests a change can affect makes CI faster, but a selection that misses a test hides a failure until main breaks. Taltal asked for it (2026-10-30) with three conditions: map changes to tests from past coverage or configurable path-to-test globs, fall back to running everything when unsure, and show why each test was picked. Off by default.
qasight/services/affected_tests.py picks the tests; two opt-in entry points use it:
-
qasight affectedprints the tests and the reason for each (text,--json, or--listfor other runners' command lines). -
pytest --qasight-affected(orTESTO_AFFECTED=1) deselects the rest inside pytest, so pytest users need no script. It runs before--qasight-shard, so shards split only the affected tests.
Changed files are the working tree against the merge base with the baseline branch, plus untracked files (so a local run sees uncommitted work). git diff --no-renames lists both names of a moved file. The base is --base, else affected.base, else the pull request's target branch from the CI variables qasight/services/ci_provenance.py already reads, else origin/main.
Mapping, for each changed file, first match wins (affected: in qasight.yaml, parsed by qasight/config/affected.py):
| Step | Source | Result |
|---|---|---|
| 1 |
run_all_when (built in: conftest.py, pyproject.toml, lockfiles, qasight.yaml, ...) |
run everything |
| 2 |
ignore (built in: __pycache__/, *.pyc, .pytest_cache/, ...) |
no test |
| 3 |
test_files name patterns (test_*.py, *.spec.ts, *_test.go, ...) |
that file; nothing if it was deleted |
| 4 |
rules: changed globs to tests globs or pytest node ids |
those tests |
| 5 | a coverage.py JSON report with test contexts (pytest --cov --cov-context=test, coverage json --show-contexts) |
the tests that ran a line of the file |
When unsure, run everything. A file no step maps, a file the coverage report lists but no test ran (only imported), a rule whose test globs match no file, a failed git diff, an unknown base and an empty diff each add a reason to run_all_reasons, and any reason means the whole suite. In pytest, a picked Python test that wasn't collected (renamed since the coverage report, or outside the paths pytest was given) also runs everything. Only ignore can make a change run nothing; pytest then exits 0 instead of its "no tests ran" 5.
Why each test. Every picked test carries "<changed file>: <why>" lines; qasight affected prints them, --json returns them, and pytest prints the first 20 in its summary.
- Nothing changes unless a user passes
--qasight-affectedor callsqasight affected.qasight rundoesn't select by itself: a stage's command line belongs to the user, and--listgives them the tests to put in it. - Run history isn't a mapping source: a run records outcomes, not which files a test touched. Coverage contexts are the "past" signal, and they go stale; the run-all fallbacks (a test that isn't collected, a file the report doesn't list) catch the common cases, and a periodic full run (nightly, or on main) catches the rest.
- Coverage is pytest-only for now (contexts are pytest node ids). Other runners use
rulesand changed test files. - Reference: Command Reference § qasight affected.
Generated from docs/Specs & ADRs/Affected Test Selection - 2026-10-30.md. Edit it there; changes made in the wiki are overwritten.
Architecture
CLI Commands
Processes & Guides
- CI-CD Pipeline Setup
- Closed Network Install
- E2E Harness Operations Guide
- GitLab Pages Demo
- Publishing Docker Images
- Publishing to Artifactory
- Publishing to PyPI
- Setup Snippets
- Shared Resource Pools
- Stability and Deprecation Policy
- Team Server
Prompts & Snippets
Roadmap & Strategy
- Feature Gaps and Quick Wins - 2026-10-31
- New User Journey Audit - 2026-11-01
- Paid Features to Offer Free - 2026-10-31
- Product Roadmap
Specs & ADRs
- Adoption Blockers - 2026-11-12
- Affected Test Selection - 2026-10-30
- Alerts Where Developers Look - 2026-10-22
- Architecture Consolidation - 2026-10-06
- Architecture Review - 2026-10-31
- Automation and Unification - 2026-11-12
- BehaveX Reports - 2026-10-21
- Better Flaky Signal - 2026-11-01
- CLI-Only Bug Hunt - 2026-11-15
- Closed Network Install - 2026-11-15
- Command Adapter and JUnit Import - 2026-09-25
- Config JSON Schema - 2026-10-22
- Crash Detection - 2026-10-31
- Delta Comparison Policy
- Docs Site - 2026-10-19
- Docs Vault and Wiki Sync - 2026-10-07
- Easy Setup - 2026-11-01
- Engine Fixes - 2026-11-01
- Engineering Health Review - 2026-10-31
- Exporting Results - 2026-11-01
- External Audit Tools - 2026-11-11
- Flaky Test Issues - 2026-11-01
- Flaky Test Quarantine - 2026-10-22
- Free Copies of Paid Features - 2026-11-01
- GitHub Action PR Comment - 2026-10-19
- Human-Run Resource Provider - 2026-10-30
- Importer and Integration Bug Hunt - 2026-10-31
- Ingest and Team Server - 2026-10-22
- Interview Readiness Fixes - 2026-10-07
- Library Packaging
- Lifecycle Fixes - 2026-11-02
- MCP Server - 2026-11-01
- Menu by Question - 2026-11-12
- New User Journey Automation - 2026-11-01
- Open Source Features - 2026-10-19
- OpenTelemetry Export - 2026-10-30
- Phase 3 Delta Comparison Engine Plan
- Phase 3 Unified Dashboard Plan
- Phase 4 BYOK and Failure Analysis
- Phase 5 UI Redesign - Cycles-First Navigation
- Playwright Adapter - 2026-10-19
- Playwright JSON Import - 2026-10-22
- Plug and Play - 2026-11-12
- Plugin API - 2026-10-19
- Product Roadmap Work - 2026-10-25
- Product to Ten - 2026-10-25
- Project Health - 2026-10-26
- QASight Logo - 2026-11-05
- Qualflare Follow-ups - 2026-10-26
- Quarantine From the UI - 2026-11-01
- Specs & ADRs
- Release Gate - 2026-11-12
- Release Scorecard - 2026-11-12
- Rename to QASight - 2026-11-04
- Repository Pattern - Database-Agnostic Refactor
- Resource Lifecycle Hooks - 2026-10-26
- Resource Pools and Leases - 2026-10-26
- Run Deletion and Stop - 2026-11-02
- Runner Coverage - 2026-10-26
- Scale Runs and Stages - 2026-10-24
- Scorecard on MegaLinter - 2026-11-12
- Security and Install Review - 2026-10-31
- Standards Compliance - 2026-11-12
- Switchable Team Features - 2026-10-24
- Test Delivery Metrics - 2026-11-02
- Test Labels, Environments and Releases - 2026-11-02
- Test Source in AI Summaries - 2026-11-01
- Throwaway Stage Containers - 2026-10-24
- Tidy Cards and Demo Examples - 2026-11-05
- UI Design Review - 2026-10-31
- UI Polish - 2026-11-08
- UI Restyle - 2026-10-19
- UI Review Fixes - 2026-11-01
Testing Workflows
Vault