Select tests from a per test coverage recording - #1660
Open
sferik wants to merge 1 commit into
Open
Conversation
Mutant selects a subject's tests by matching the subject's expression against each test's expression, which RSpec derives from the leading token of an example description. A spec headed `describe 'clamping behaviour'` covers `Calculator#clamp` without ever saying so, and the selector cannot find it. Its mutations then report alive even though the suite kills them. Add a `context_map` selection strategy that measures instead of guessing. It reads the per test data simplecov 1.2.0 and newer add to `coverage/coverage.json` under `track_tests`, and selects the tests that executed the subject's own source lines, wherever those tests live. Mutant takes on no dependency for this. It reads `coverage.json` rather than the internal resultset, so it consumes simplecov's published schema: a version to gate on, and a `meta.root` naming the project root the recording was taken against, which is what paths resolve against. The default strategy is `auto`. Mutant uses the recording when there is a usable one and the expressions otherwise, so a project that records per test coverage gets the better selection without asking for it and a project that does not is unaffected. Every run prints the strategy it settled on. `--selection context_map` turns every reason the recording cannot be used into a failed run, `--selection expression` never reads it, and `--selection-path` reads a report outside `coverage`. The same settings live under a `selection` key in `mutant.yml`. Subjects whose lines no test executes, such as a constant body evaluated while the suite loads, fall back to expression based selection. Reporting their mutations alive without running a test would be worse than the default, not better. The selected tests are ordered likeliest killer first. A run stops at the first failing test, so the order decides whether one test runs or all of them. A test the expressions would also have picked is the subject's own unit test and goes first as a block, then comes the test spending the largest share of its own footprint inside the subject, and reach breaks the remaining ties. Minitest and Test::Unit run the given order and stop at the first failure, so the ranking arrives intact. RSpec picks its own order for groups and examples, so mutant installs an ordering strategy in the ordering registry. Its public `register_ordering(:global)` cannot serve here, because it returns early once `--order` was forced from the command line or `.rspec`, which most suites do. On a suite where thirty tests execute a subject incidentally and one asserts on it, the ranking cut the examples run from 129 to 39 for the same nine kills. `Mutant::Test` gains a `location`, which is how a test recorded by file and line joins back to the test the integration can run.
sferik
force-pushed
the
context-map-test-selection
branch
from
September 4, 2026 18:50
e5bce80 to
e071f78
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The latest version of SimpleCov (1.2.0) adds the ability to
track_tests, which creates a bitmap incoverage/coverage.jsonfrom lines of code to the specific tests/specs that exercise them. If this bitmap exists, it can more efficiently select which tests to run (and in which order) to kill a mutation on a particular line of code.The selected tests come back ordered likeliest killer first: the ones the expressions would also have picked, then whichever spends the largest share of its own footprint inside this subject, then reach, and finally the framework’s test id to break a tie. Minitest and Test::Unit already run the order they are given and stop at the first failure. RSpec picks its own order, so Mutant installs an ordering strategy.
The default is
auto, meaning use the recording when there is a usable one, otherwise use expressions, so nothing changes for a project that doesn’t use SimpleCov.--selection context_mapinstead turns every reason the recording cannot be used into a hard failure,--selection expressionnever looks at it, and--selection-pathpoints at a report outsidecoverage. Users can specify the same configuration options under aselectionkey inmutant.yml. Every run prints which one it settled on, because auto-detection you cannot see is a trap.I’ve added full documentation in
docs/test-selection.mdas well as aselectionsection todocs/configuration.md.To benchmark this change, I ran mutant for SimpleCov (206 subjects and 7000 mutations) against this branch. Median wall clock dropped from 230s to 185s (20% faster), owing largely to running fewer tests per subject (from 161 down to 79).