The mutation gate returns a different score for the same tree

[VDBBjorn]

The Changed code mutation gate returns a different score for the same tree. On PR #164 I ran it four times:

run	score	survived
CI	66.67%	3
local	55.56%	4
local, after adding 6 tests	77.78%	2
local, identical tree, immediately after	88.89%	1

The last two runs are the same commit on the same machine, back to back. The threshold is 80%.

The mutants reported as survived are demonstrably killed

Two mutants in src/Alberto.Messaging/MessagingBuilderExtensions.cs were reported Survived in run 3. I hand-applied each mutation and ran the fast suite:

• L79, configureMappings(registry); → ; — 2 tests fail.
• L86, the AddSingleton<IHostedService>(… OutboxRetentionService …) statement → ; — WithOutbox_RegistersRetentionServiceEvenWithoutTransport fails at OutboxHandlerTests.cs:212.

So the suite kills both. Stryker said otherwise, and then said otherwise again differently on the next run.

Suspected cause

Both flaky mutants report coveredBy = 2069 tests, i.e. the entire suite — Stryker's fallback when it cannot attribute coverage to a block. Both live inside the deferred builder.Register(context => { … }) lambda, which does not execute at the point WithOutbox is called; it runs later, when a test drives DeferredRegistrations.

stryker-config.json sets "test-runner": "mtp". Every run prints:

The Microsoft Test Platform testrunner is currently in preview. Results should be verified
since this feature is still being tested.

tests/Alberto.Tests has no xunit.runner.json, so xUnit v3 parallelism is at its default. A per-test coverage collector that races parallel execution would produce exactly this: unattributable coverage collapsing to "all tests", and an unstable per-mutant verdict on the deferred-lambda mutants.

Also visible every run, and possibly related: 5 mutants got status CompileError in this same file, and 3 mutants got status Ignored. Reason: Removed by block already covered filter — and which 3 are filtered moves between runs, so the scored set itself is not stable.

Why this matters more than the percentage

Small packages amplify it. Alberto.Messaging scores 9 mutants, so one flaky verdict is 11 points and straddles the 80% line. A PR can fail the gate for reasons that have nothing to do with its diff, and — worse in the other direction — a genuinely uncovered mutant can be reported killed and pass. A gate whose verdict is not reproducible cannot distinguish "this diff is well tested" from "we got a lucky roll", which is the same false-all-clear shape as #155.

Suggested directions

Not proposing a fix here, just what seems worth trying:

1. Pin test parallelism during mutation runs (an xunit.runner.json with parallelizeTestCollections: false, or maxParallelThreads: 1) and re-run the same tree several times to see whether the variance disappears. Costs wall-clock; the diff gate is already the throughput constraint, so measure before adopting.
2. Try "test-runner": "vstest" instead of the preview mtp runner and compare stability.
3. Investigate the 5 CompileError mutants in MessagingBuilderExtensions.cs — Stryker failing to compile its own mutants of that file suggests its instrumentation of it is shaky, which may be the same root cause.
4. Whatever the fix, add a repeatability check: run the gate twice on an unchanged tree in the nightly sweep and fail loudly if the two scores disagree. Right now nothing would have caught this.

Do not "fix" this by lowering the threshold or excluding the file. The tests are real and the mutants are genuinely killed; the reporting is what is broken.

[VDBBjorn]

Related, and possibly the same root cause: the All core packages job in mutation.yml already documents that the preview MTP runner does not attribute coverage per test, so every covered mutant runs all 1890 tests. That is tracked as #133.

If coverage attribution is absent rather than merely slow, then the coveredBy: 2069 I observed on both flaky mutants is not a fallback for two unusual mutants — it is what every covered mutant reports, and the per-mutant verdict is decided by a full parallel suite run each time. That would make the instability a property of the runner rather than of MessagingBuilderExtensions.cs, and it means the fix for #133 and the fix for this are likely the same piece of upstream work.

Worth confirming before anyone spends time on the parallelism theory in the issue body: if coverage is never attributed, pinning maxParallelThreads may reduce the variance without removing the cause.

[VDBBjorn]

A hand-verified false survivor

The score table above shows the run-to-run drift. This is a stronger datapoint: a mutant the gate reports as survived that six existing tests demonstrably kill.

src/Alberto/Subscriptions/CachingCheckpointStore.cs:282 — the resync skip for entries with a pending write:

// Skip entries that have pending writes — our value is ahead of the DB intentionally
if (_dirty.ContainsKey(processorId))
    continue;

Stryker's Negate expression mutant on that line came back Survived in a run of

build/mutation-test.sh Alberto.csproj \
  --mutate '**/Subscriptions/CachingCheckpointStore.cs' \
  --mutate '**/Subscriptions/ICheckpointInventory.cs' \
  --mutate '**/ServiceCollectionExtensions.cs'

Applying that exact mutation by hand and running the same filter Stryker runs (Category!=Integration, Release):

Failed CachingCheckpointStoreTests.ResyncFromStore_ExternalDelete_ShouldUpdateCache
Failed CachingCheckpointStoreTests.ResyncFromStore_ExternalReset_ShouldUpdateCache
Failed CachingCheckpointStoreTests.ResyncFromStore_DirtyEntry_ShouldNotOverwrite
Failed CachingCheckpointStoreTests.ResyncTimer_ShouldDetectExternalReset
Failed CachingCheckpointStoreFlushAndResyncTests.ResyncFromStore_AfterDetectingReset_UpdatesPersistedSoSubsequentFlushSucceeds
Failed CachingCheckpointStoreFlushAndResyncTests.OnResyncTimer_WhenInnerGetThrows_LogsError

Failed: 6, Passed: 1638, Skipped: 3, Total: 1647

Six failures, one of them (ResyncFromStore_DirtyEntry_ShouldNotOverwrite) a direct specification of the exact rule the mutant breaks. This is not a thin or incidental kill.

So the gate is not merely noisy about borderline mutants — it can report a well-covered line as unspecified. That has a cost beyond the score: acting on the report leads you to write a duplicate of a test that already exists. It nearly did here.

It also sharpens the diagnosis. Combined with #133 (the MTP runner does not attribute coverage per test, so every covered mutant runs the full suite), the plausible failure is in how the run's result is attributed back to the mutant, not in mutant selection or in test selection. A mutant whose test session is misattributed — or whose session is one of the 138 the same run counts as compile errors — would be scored Survived while the tests that kill it were never consulted.

Worth reproducing on a second machine before concluding, since the runs here were all on one host.

Unchanged from the original report: the fix is not to lower the threshold, exclude the file, or add // Stryker disable.

[VDBBjorn]

The gate is not noisy — it is measuring something other than the test suite

Three consecutive local runs of the same command on the same branch, with only tests added between them, every added test hand-verified to kill a mutant by breaking the production line and watching it fail:

Run	tests in suite	scored	killed	score
1	1639	118	96	82.20 %
2	1642 (+3 verified kills)	117	94	81.20 %
3	1644 (+2 verified kills)	114	86	76.32 %

Adding tests that demonstrably kill mutants moves the score down, and shrinks the denominator too. That is not variance around a true value; the number is not a function of the test suite.

The datapoint that settles it

Run 3 reports ServiceCollectionExtensions.cs:112 as Survived:

RegisterModule(services, declared, builder.DeferredRegistrations);

That is the entire non-sharded module registration. Deleting it means AddAlberto registers nothing and every DI resolution in the suite fails. By hand, with the same filter and configuration Stryker uses:

Failed: 62, Passed: 1582, Skipped: 3, Total: 1647

Sixty-two failures, scored as a surviving mutant.

Two more in the same run that are directly and deliberately tested:

• ServiceCollectionExtensions.cs:42 — ArgumentException.ThrowIfNullOrWhiteSpace(moduleKey), which AddAlbertoArgumentTests has a whole region for
• CachingCheckpointStore.cs:182 — ArgumentNullException.ThrowIfNull(handler), killed in run 1 by a test written for exactly it, survived in run 3 with that test untouched

What this changes

The earlier framing in this issue was "the score drifts a couple of points between runs." That understates it. A run can mark an arbitrary subset of well-covered mutants as survived, and the subset grows as the suite gets longer — runs 1→3 went 46 → 53 → 54 minutes as the score fell. The most likely mechanism is that test sessions which fail to report (crash, are killed, or are lost) are attributed as Survived rather than as errors; note RuntimeError: 7 and CompileError: 138 alongside in run 3.

That makes the gate actively harmful, not just unreliable:

1. It cannot fail honestly. A red gate is now uninformative, so the standing response becomes "re-run it", which is how a real regression gets waved through.
2. It sends you to write duplicate tests. Acting on run 3's report means writing a second test for a null guard that already has one. That nearly happened here — see the previous comment.
3. A green run is not evidence either. If attribution is unreliable in one direction it is not trustworthy in the other.

Suggested order of attack

1. Confirm on a second machine and on the CI runner that a hand-broken line is scored Survived. One host is not enough to rule out something local.
2. Check whether the count of misattributed mutants tracks suite wall-clock — run the same branch with an artificially padded suite.
3. Get the per-mutant test-session logs out of the mtp runner for a known-false survivor. This overlaps A quarter of mutants time out because tests hang instead of failing fast #133: without per-test coverage attribution every mutant runs all ~1890 tests, so any per-session failure mode is amplified across the whole set.
4. Until then, treat the Changed code result as advisory and verify any surviving mutant by hand before writing a test for it.

Still not a reason to lower the threshold, add an exclude, or write // Stryker disable. The threshold is fine; the measurement is broken.