feat(execution): add execution-based mutation verification engine - #104
Merged
Conversation
Static analysis, however precise, can only ever predict whether a mutant would be noticed. This adds the core mechanism to actually find out: a new NetEvolve.FrameShift.Execution library that applies a candidate Mutation to a real Compilation, emits it to a real assembly image, loads that image into an isolated, collectible AssemblyLoadContext, and invokes a test method by reflection to observe a genuine pass or fail. - MutantAssemblyBuilder emits the mutated compilation to bytes, reusing the same mutation-application logic MutantCompiler already uses to verify a mutant re-binds, taken one step further to a runnable program. - IsolatedAssemblyRunner loads that image into its own AssemblyLoadContext and invokes one parameterless method, catching a thrown exception (or a faulted returned Task) as the kill signal every mainstream assertion library already produces on a failed assertion. - MutationExecutionEngine orchestrates both per mutant and aggregates a batch into a MutationScore (Killed / Survived / BuildFailed). This library targets net6.0+ only: the collectible AssemblyLoadContext it needs is CoreCLR-only, unlike the analyzer itself, which stays on netstandard2.0 to ship as a single Roslyn component. Scope: this is the core execution mechanism, proven end to end against a dogfood fixture in NetEvolve.FrameShift.Tests.Execution, one mutation genuinely killed and one genuinely surviving. It runs exactly one named test method per mutant, resolved by the caller; discovering and running a whole test suite through a real test host per framework, and wiring that into a CLI or MSBuild gate, is intentionally left as follow-up work.
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
NetEvolve.FrameShift.Execution and its test project have no consumer and no compatibility promise yet, so they track the newest supported runtime instead of the widest one. Broadening the matrix back to $(_NetTargetFrameworks) is a one-line change once there is a reason to.
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (94.81%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #104 +/- ##
==========================================
- Coverage 95.36% 95.32% -0.04%
==========================================
Files 93 107 +14
Lines 6233 6677 +444
Branches 1340 1389 +49
==========================================
+ Hits 5944 6365 +421
- Misses 128 140 +12
- Partials 161 172 +11 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The in-process reflection runner proved the core mechanism works but still picks one test method by hand. This adds the orchestration a real build-time or CI gate would actually use: run a whole, already-built test project's own test host as a real subprocess against a copy of its build output with only the production assembly swapped for the mutant. - MutantSwapWorkspace copies a test project's build output directory to a temporary location and overwrites the production assembly file with the mutant bytes, so the already-compiled test assembly keeps resolving its reference by file name straight into the mutant, no recompilation involved. - ProcessTestHostRunner runs the swapped test assembly with `dotnet exec` and reads nothing but the process exit code: 0 for every supported test framework (TUnit, xUnit, NUnit, MSTest, under VSTest or under Microsoft.Testing.Platform) means every test passed, anything else means at least one did not. That is what keeps this runner test-framework agnostic without parsing a single result format. - MutationExecutionEngine.ExecuteViaTestHostAsync/RunViaTestHostAsync wire both together and add a Timeout verdict, excluded from the mutation score exactly like a build failure is, for a host that had to be killed without ever reporting an exit code. Proven end to end in MutationExecutionEngineTestHostTests: a real `dotnet exec` subprocess against a real swapped-in mutant assembly, one mutation of the exercised method genuinely exits non-zero, one mutation of an unrelated method genuinely exits zero. Still not wired into a CLI or an MSBuild target, and still resolved by the caller which build output directory and which assembly names to use; discovering that automatically from a real project is the next step.
The library API existed but nothing actually invoked it: this adds
Program.cs, a real entry point that can be started with `dotnet exec` or
`dotnet run` instead of only from a caller that links against the
library.
frameshift-execute --test-output <dir> --production-dll <file.dll>
--test-dll <file.dll> --source <file.cs> [--source <file.cs> ...]
[--timeout-seconds <n>]
It recompiles the given production source files fresh (referencing every
other assembly already sitting in the test output directory, plus the
current process's own shared framework assemblies, since a
framework-dependent build output never contains those itself), generates
every candidate mutation with the same MutantGenerator the analyzer uses,
and runs each one through MutationExecutionEngine.ExecuteViaTestHostAsync,
printing progress per mutant and the final mutation score.
Running it against a real, `dotnet build`-produced project surfaced a real
defect no earlier fixture caught: Roslyn defaults an unversioned
compilation to assembly version 0.0.0.0, while the .NET SDK's own build
defaults to 1.0.0.0. A recompiled mutant at the wrong version does not
fail to compile - it fails to load, because the test host's own
*.deps.json pins the production assembly's expected version and rejects a
same-named file that does not match, which surfaced as an unhandled
FileNotFoundException inside the test host and would have silently
counted every mutant as a false "Killed". Fixed by reading the original
assembly's real version once and stamping the recompiled mutant with a
matching [assembly: AssemblyVersion] attribute.
MutationExecutionCliRealProjectTests is the regression test for exactly
that: it builds a real project pair with a real `dotnet build`, not a
hand-emitted fixture, and asserts the CLI reports the correct score
against it.
…verage gaps Rename tests/NetEvolve.FrameShift.Tests.Execution to NetEvolve.FrameShift.Execution.Tests.Unit per the sub-product naming convention, and split the subprocess- and build-spawning tests into a new NetEvolve.FrameShift.Execution.Tests.Integration project. Add coverage for every previously untested branch of the execution CLI: argument parsing, the console entry point, the file-swap workspace's failure paths, the async test-invocation branch, and the test-host subprocess runner's timeout and output-capture paths, including a Verify-based snapshot of the CLI usage text.
3 tasks
…leanup The CI Windows runner reports a killed process's still-open file handle as UnauthorizedAccessException rather than IOException, which the existing retry loop did not catch, making the timeout test fail intermittently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FrameShift's static analysis (FSH0001/FSH0006/FSH0007) can only ever predict whether a mutant would be noticed by a test — it never actually runs one. This PR adds the real mechanism, at three depths, plus a CLI that actually invokes it.
1. In-process, one method at a time
MutantAssemblyBuilderapplies a candidateMutationto aCompilationand emits it to a real assembly image, reusing the same apply-mutation logicMutantCompileralready uses to verify a mutant re-binds — taken one step further, to a runnable program.IsolatedAssemblyRunnerloads that image into its own collectibleAssemblyLoadContextand invokes one parameterless method by reflection, treating a thrown exception (or a faulted returnedTask) as the kill signal — the same signal every mainstream assertion library (xUnit, NUnit, MSTest, TUnit) already produces on a failed assertion.MutationExecutionEngine.Execute/.Runorchestrate this and aggregate a batch into aMutationScore(Killed / Survived / BuildFailed).2. Real subprocess, a whole test host at a time
MutantSwapWorkspacecopies a test project's build output directory to a temporary location and overwrites the production assembly file with the mutant bytes, so the already-compiled test assembly keeps resolving its reference by file name straight into the mutant — no recompilation of the test assembly involved.ProcessTestHostRunnerruns the swapped test assembly withdotnet execand reads nothing but the process exit code:0for every supported test framework (TUnit, xUnit, NUnit, MSTest, under VSTest or under Microsoft.Testing.Platform) means every test passed, anything else means at least one did not — keeping this runner test-framework agnostic without parsing a single result format.MutationExecutionEngine.ExecuteViaTestHostAsync/.RunViaTestHostAsyncwire both together and add aTimeoutverdict, excluded from the mutation score exactly like a build failure is.3. A CLI that actually invokes it
Recompiles the given production source files (referencing every assembly already in the test output directory, plus the current process's own shared framework assemblies — a framework-dependent build output never ships those itself), generates every candidate mutation with the same
MutantGeneratorthe analyzer uses, and runs each one throughExecuteViaTestHostAsync, printing per-mutant progress and the final mutation score.Running the CLI against a real
dotnet build-produced project surfaced a real defect no hand-emitted fixture had caught: Roslyn defaults an unversioned compilation to assembly version0.0.0.0, while the .NET SDK's own build defaults to1.0.0.0. A recompiled mutant at the wrong version doesn't fail to compile — it fails to load, because the test host's own*.deps.jsonpins the production assembly's expected version and rejects a same-named file that doesn't match. That surfaced as an unhandledFileNotFoundExceptioninside the test host and would have silently counted every single mutant as a false "Killed". Fixed by reading the original assembly's real version once and stamping the recompiled mutant with a matching[assembly: AssemblyVersion].MutationExecutionCliRealProjectTestsis the regression test — it builds a real project pair with a realdotnet build, not a hand-emitted fixture, and asserts the CLI reports the correct score against it.Scope
NetEvolve.FrameShift.Executiontargets net10.0 only, not the repo's usual wide matrix: the collectibleAssemblyLoadContextit needs is CoreCLR-only (unlike the analyzer itself, which stays on netstandard2.0 to ship as a single Roslyn component), and this library has no consumer yet to keep a wider compatibility promise for.Still not packaged as an installable
dotnet tool, not wired into an MSBuild target, and the production compilation is rebuilt from explicit source files rather than a real MSBuild project evaluation (so source generators, conditional compilation symbols, and non-trivial reference graphs aren't reproduced faithfully yet). Those remain the natural next increments.Test plan
dotnet build FrameShift.slnx— 0 warnings, 0 errorsdotnet testonNetEvolve.FrameShift.Tests.Execution(net10.0) — all 7 tests pass, including a genuine subprocess run and a full real-dotnet buildregression testdotnet build-produced project pair and confirmed the exact expected score (4 killed, 1 survived)NetEvolve.FrameShift.Tests.UnitandNetEvolve.FrameShift.Tests.Integrationsuites unaffected (3629 + 260 on net10.0, 0 failures)