feat(justdummies): add JD007-JD010 for draws that escape the seed scope - #369
Merged
Merged
Conversation
The four rules cover the ways a value is drawn outside the scope [Reproducible] pins, each verified against xunit.v3 3.2.2 by probe before the rule was written rather than inferred from the adapter's code. JD007 — xUnit constructs the test-class instance, and awaits IAsyncLifetime.InitializeAsync, before running the before/after hooks the adapter pins the seed from. The probe confirmed it: under [Reproducible(Seed = 555)] a constructor draw and a field-initializer draw both differed from the first draw of a fresh UseSeed(555) scope, while the body's draw matched it exactly. The test therefore advertises full reproducibility and delivers it only for the body. JD008 — a theory's data provider is evaluated at discovery, before any case runs. The probe's [MemberData] value likewise fell outside the pinned scope. One draw serves every case, replayable from no seed. JD009 — a static initializer draws once, lazily, under whichever test first touched the type; the value is then shared, making the tests order-dependent. RandomSource resolves the source at Generate() time, so a shared *generator* is safe — the compliant shape the message names. JD010 — [Reproducible] on a method with no IFactAttribute is never read. It matters because a working attribute is silent by design, so the inert form is indistinguishable from the working one until a failure that should have named a seed does not. Three shared helpers carry the rules: XunitFacts for the lifecycle and provider shapes, plus GeneratorFacts.IsGenerateCall and RootsAtAmbientAny. The latter is deliberately conservative — it reports only a chain proven to start at the ambient Any, so a draw from an isolated AnyContext, which no ambient scope governs, can never be reported. Dogfooded across JustDummies.UnitTests, JustDummies.PropertyTests, JustDummies.Xunit.UnitTests, FirstClassErrors.Testing.UnitTests and FirstClassErrors.UnitTests: no hits. A planted positive control confirmed all four fire at the expected locations, so the clean sweep is the suites' doing, not a rule that never runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKqyhG9Y4AfdxEYekP2Evq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second wave of the JustDummies analyzer study: four rules covering the ways an arbitrary value is drawn outside the scope
[Reproducible]pins. Each rests on a claim about xUnit's lifecycle, and each claim was verified against xunit.v3 3.2.2 by probe before the rule was written — not inferred from the adapter's code.Type of change
Changes
DrawOutsideThePinnedScope(Warning) — xUnit constructs the test-class instance, and awaitsIAsyncLifetime.InitializeAsync, before running the before/after hooks the adapter pins the seed from. The failure mode is worse than no reproducibility: the test advertises it, the reader pins the reported seed, the arrangement still differs, and the seed looks broken.ArbitraryValueInTheoryData(Warning) — a theory's data provider is evaluated at discovery, before any case runs. One draw serves every case, replayable from no seed, in a theory that reads as if it enumerated arbitrary cases.DrawInStaticInitializer(Warning) — a type initializer draws once, lazily, under whichever test first touched the type; the value is then shared by every test in the class, making them order-dependent.ReproducibleOnNonTestMethod(Warning) —[Reproducible]on a method with noIFactAttributeis never read. It earns a rule because a working attribute is silent by design, so the inert form is indistinguishable from the working one until a failure that should have named a seed does not.XunitFacts— lifecycle and data-provider shapes, all resolved by metadata name, so a compilation referencing neither xUnit nor the adapter never reaches these rules and JustDummies stays standalone.GeneratorFacts.IsGenerateCallandRootsAtAmbientAny. The latter is deliberately conservative: it reports only a chain proven to start at the ambientAny, so a draw from an isolatedAnyContext— which no ambient scope governs — can never be reported. That under-reports a generator held in a local or field, and the help pages say so.The lifecycle claims, and how they were checked
A probe under
[Reproducible(Seed = 555)]compared each draw against the first draw of a freshAny.UseSeed(555)scope:bodyDraw == firstDrawOfSeed555[MemberData]providerThe probe was removed before commit; what it established is now pinned by the rules' own tests and stated in the help pages.
Dogfooding
Run over
JustDummies.UnitTests,JustDummies.PropertyTests,JustDummies.Xunit.UnitTests,FirstClassErrors.Testing.UnitTestsandFirstClassErrors.UnitTests: no hits. Unlike the previous wave, nothing here changed the design.Because a clean sweep is also what a rule that never runs produces, a planted positive control was built and checked: all four fired at the expected locations, then the control was removed. The clean sweep is the suites' doing.
Testing
dotnet build FirstClassErrors.sln— 0 warnings, 0 errorsdotnet test FirstClassErrors.sln— 1 983 tests, 0 failures across all 13 test projectsFirstClassErrors.Analyzers.UnitTests, 132) — the rules added here live inJustDummies.Analyzers.UnitTests(66)Mutation testing was not run locally.
Documentation
doc/updated —JD007–JD010rule pages and the rule indexJD007.fr.md–JD010.fr.mdand the French entries ofdoc/handwritten/for-users/analyzers/README.fr.mdArchitecture decisions
Proposed: ADR-____These four apply decisions already recorded — ADR-0044 (first-party analyzers for a mistake the type system cannot express) and ADR-0059 (enforcement follows what the mechanism can know). No new lasting decision.
Open question
JD007 exists because the adapter cannot pin earlier.
BeforeAfterTestAttribute.Beforeruns after construction, which is xUnit's design, not the adapter's choice — so the analyzer is the available answer rather than the only conceivable one. If a v3 extensibility point could open the scope before the test class is constructed, JD007 would become unnecessary and the adapter would simply be more correct. I did not investigate that: it changesJustDummies.Xunit's behaviour rather than adding a diagnostic, and it is your call whether it is worth exploring.Related issues
None. Second wave of the analyzer-opportunity study; the first shipped in #368.
🤖 Generated with Claude Code
https://claude.ai/code/session_01GKqyhG9Y4AfdxEYekP2Evq
Generated by Claude Code