feat(analyzers): add FSH0007 behavioral verification of mutation coverage - #103
Conversation
…rage Plain reachability lets a test that only captures a method reference and asserts IsNotNull() clear FSH0001 without ever exercising the method's behaviour. FSH0007 closes that gap: a mutation point is only treated as covered when a test both invokes the member and backs that call with a recognised, non-trivial assertion (AssertionRecognition), not just a bare null check. The test-surface manifest gains a 'B' line marking behaviorally verified references, additive and backward compatible with manifests that predate it. TestSurfaceCollector classifies each reference as invoked-vs-bare and tracks whether a test's reachable code calls a real assertion. ReachabilityClosure.ComputeBehavioral seeds a second, narrower closure from those references, computed lazily so a compilation where it is never needed never pays for it. GeneratorDriverBridge recognises the CSharpGeneratorDriver.Create(...) .RunGenerators(...) test-harness pattern and synthesizes the otherwise invisible reachability edge to IIncrementalGenerator.Initialize / ISourceGenerator.Initialize+Execute, so a source-generator test no longer needs an artificial delegate-reference just to satisfy the analyzer. It is the first implementation of the new IInvocationBridge extension point under TestSurface.Bridges, so a second recognised pattern can be added without collector changes. Static-analysis based, not execution based: FSH0007 narrows where the existing heuristics are most obviously wrong, it does not run mutants against the real test suite.
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #103 +/- ##
==========================================
+ Coverage 95.32% 95.36% +0.03%
==========================================
Files 91 93 +2
Lines 5928 6233 +305
Branches 1295 1340 +45
==========================================
+ Hits 5651 5944 +293
- Misses 122 128 +6
- Partials 155 161 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1 participant
Summary
FrameShift's coverage check (FSH0001) only asks "does any test reach this code?" A test that captures a method reference and asserts
IsNotNull()on it clears that check without ever calling the method or verifying its behavior — one test shaped like that cleared 700+ real warnings in a source-generator project without asserting anything.This PR adds FSH0007, a second-layer check: a mutation point only counts as covered when a test both invokes the member and backs that call with a recognized, non-trivial assertion — not just a bare null check.
Bline marking behaviorally verified references, additive and backward compatible with manifests written before this change.TestSurfaceCollector: classifies each reference as invoked-vs-bare-method-group, and detects whether a test's reachable code calls a real assertion (AreEqual,IsTrue,Contains,Throws, TUnit'sIsEqualTo, etc.) vs. only a trivial null check (AssertionRecognition).ReachabilityClosure.ComputeBehavioral: a second, narrower transitive closure seeded from behaviorally verified references, computed lazily so a compilation that never needs the answer never pays for it.GeneratorDriverBridge: recognizes theCSharpGeneratorDriver.Create(...).RunGenerators(...)test-harness pattern and synthesizes the otherwise-invisible reachability edge toIIncrementalGenerator.Initialize/ISourceGenerator.Initialize+Execute— no more artificial delegate-reference tests needed just to satisfy the analyzer. It's the first implementation of a newIInvocationBridgeextension point underTestSurface.Bridges, so a second recognized pattern can be added without touching the collector.Scope note: this is still static analysis, not execution-based mutation testing. It narrows where the existing heuristics are most obviously wrong (reached-but-never-asserted code); it does not inject a mutant and run the real test suite. That remains a separate, larger effort tracked for follow-up.
Test plan
dotnet build FrameShift.slnx— 0 warnings, 0 errorsdotnet test(both test projects, all 8 target frameworks) — 30712 test cases, 0 failuresIsNotNulltest) and confirms it no longer counts as coveredComputeBehavioral,AssertionRecognition,GeneratorDriverBridge, and FSH0007/FSH0006 precedence rules