Skip to content

Add incremental edit tests — verify editing one file only regenerates affected contracts#214

Merged
Skymly merged 2 commits into
mainfrom
test/incremental-edit-tests
Jul 4, 2026
Merged

Add incremental edit tests — verify editing one file only regenerates affected contracts#214
Skymly merged 2 commits into
mainfrom
test/incremental-edit-tests

Conversation

@Skymly

@Skymly Skymly commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add RunIncrementalEdit<TGenerator> helper to SourceGeneratorTestContext that runs a generator on an initial compilation, replaces one syntax tree with a modified version, and re-runs. This enables testing the core incremental value of source generators: editing one file should only regenerate affected contracts.
  • Add GetTrackedStepReasons and GetAllTrackedStepNames helpers for inspecting tracked step reasons from a GeneratorDriverRunResult.
  • Add 3 new incremental edit tests to IncrementalCacheTests:
    • RegisterStrategy_EditOneFile_UnmodifiedContractStaysCached — two strategy contracts in separate files, add a strategy to one file, assert StrategyCombine is Modified/New and transform reflects the new strategy.
    • GenerateSingleton_EditOneFile_TransformReflectsChange — two singleton targets in separate files, add a singleton to one file, assert transform has at least one Modified/New output.
    • StateTransition_EditOneFile_CollectIsModified — two state machines in separate files, add a transition to one file, assert StateMachineCombine is Modified/New and transform reflects the change.

What these tests prove

The existing AssertCacheHit tests only verify "same compilation, second run → all stages Cached". The new tests verify the real incremental scenario: when a file is edited, the pipeline detects the change (Combine is Modified) and the transform stage produces new/modified outputs for the changed content. This is the primary performance benefit of IIncrementalGenerator in large projects.

Design notes

  • StrategyCollect and StateMachineCollect tracking names are defined in TrackingNames.cs but not used in the generators (no WithTrackingName on Collect()). The tests use StrategyCombine/StateMachineCombine instead, which cover the Collect+Combine aggregation.
  • GenerateSingletonGenerator has no Collect stage (per-item output), so the test checks the transform tracking name directly.

Related Issue

Closes #213

Solution module

  • SourceGenerators (tests)

Type of change

  • Test improvement

Test plan

  • dotnet build DesignPatterns.slnx -c Release — 0 errors, 0 warnings
  • dotnet test DesignPatterns.slnx -c Release --no-build — all 644 tests pass (641 existing + 3 new)
  • New tests verify Combine stage is Modified/New after edit
  • New tests verify transform stage has Modified/New output after edit

Breaking changes

  • None

…tateTransition

Add RunIncrementalEdit helper to SourceGeneratorTestContext that runs a
generator on an initial compilation, replaces one syntax tree with a
modified version, and re-runs. This enables testing the core incremental
value: editing one file should only regenerate affected contracts.

Three new test methods in IncrementalCacheTests:
- RegisterStrategy_EditOneFile_UnmodifiedContractStaysCached: two
  strategy contracts in separate files, add a strategy to one file,
  assert Combine is Modified and transform reflects the new strategy.
- GenerateSingleton_EditOneFile_TransformReflectsChange: two singleton
  targets in separate files, add a singleton to one file, assert
  transform has at least one Modified/New output.
- StateTransition_EditOneFile_CollectIsModified: two state machines in
  separate files, add a transition to one file, assert Combine is
  Modified and transform reflects the change.

Also add GetTrackedStepReasons and GetAllTrackedStepNames helpers for
inspecting tracked step reasons from a GeneratorDriverRunResult.

Closes #213

@Skymly Skymly left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: PR #214 — Incremental edit tests

Overall this is a solid addition — the RunIncrementalEdit helper and 3 edit tests fill an important gap in the incremental cache test coverage. A few issues to address before merge:

Issues

1. Stale <param> in XML doc comment (line ~134)
RunIncrementalEdit has a <param name="unchangedSources"> tag in its XML doc, but the method signature is (initialSources, modifiedPath, modifiedSource) — there is no unchangedSources parameter. This is a leftover from an earlier design. Remove the stale <param> line.

2. GetAllTrackedStepNames is dead code
GetAllTrackedStepNames is defined in SourceGeneratorTestContext.cs but never called anywhere in the test suite. Either add a test that uses it (e.g. a diagnostic helper that dumps all tracked names on failure), or remove it to keep the helper surface minimal.

3. Tests don't assert that unmodified-file transforms are Cached
The 3 edit tests assert that the Combine stage is Modified/New and that at least one transform output is Modified/New. But the core value of incremental editing is that unchanged inputs stay Cached. Consider adding an assertion that at least one transform output for the unmodified file is Cached (or that the count of Cached outputs is >= 1). Without this, the tests would still pass even if the pipeline incorrectly re-ran all transforms.

For RegisterStrategy_EditOneFile_UnmodifiedContractStaysCached — the test name promises "UnmodifiedContractStaysCached" but doesn't actually verify caching of the unmodified contract. This is misleading.

Nits

  • The using Microsoft.CodeAnalysis; import was added at the top of IncrementalCacheTests.cs — verify it's actually needed (for IncrementalStepRunReason). It's fine since that's where the type lives.
  • The box-drawing comment separator is fine but unusual in this codebase — check that .editorconfig doesn't flag non-ASCII comment characters.

What's good

  • RunIncrementalEdit correctly uses compilation.ReplaceSyntaxTree (not WithSyntaxTrees which doesn't exist on CSharpCompilation).
  • disabledOutputs: IncrementalGeneratorOutputKind.None + trackIncrementalGeneratorSteps: true is the right configuration for cache testing.
  • The 3 tests cover the 3 distinct pipeline shapes: Collect+Combine (Strategy), per-item output (Singleton), and Collect+Combine with single stream (StateTransition).
  • Test method names are descriptive and follow the Generator_Scenario_ExpectedBehavior convention.

- Remove stale <param name='unchangedSources'> from RunIncrementalEdit
  XML doc comment (the parameter does not exist in the method signature).
- Remove dead code GetAllTrackedStepNames (defined but never called).
- Add Cached/Unchanged assertions to GenerateSingleton and
  StateTransition edit tests: the unmodified file's transform should
  be Cached or Unchanged, verifying the core incremental guarantee.
- Rename RegisterStrategy_EditOneFile_UnmodifiedContractStaysCached to
  RegisterStrategy_EditOneFile_CombineReflectsChange: the unmodified
  file's transform is Modified (not Cached) because Transform returns
  List<KeyedRegistration> (reference equality). This will be fixed by
  PR #216 (LocationInfo + EquatableArray), but until then the test
  correctly verifies only the Combine stage behavior. Added XML doc
  explaining the limitation.
@Skymly

Skymly commented Jul 4, 2026

Copy link
Copy Markdown
Owner Author

Review issues fixed in latest commit

  1. Stale <param> removed — Removed the <param name="unchangedSources"> doc comment that referenced a non-existent parameter.
  2. Dead code removed — Removed GetAllTrackedStepNames which was defined but never called.
  3. Caching assertions added — Added Cached/Unchanged assertions to GenerateSingleton_EditOneFile_TransformReflectsChange and StateTransition_EditOneFile_CollectIsModified — the unmodified file's transform should be Cached or Unchanged, verifying the core incremental guarantee.
  4. Test renamedRegisterStrategy_EditOneFile_UnmodifiedContractStaysCachedRegisterStrategy_EditOneFile_CombineReflectsChange. The unmodified file's transform is Modified (not Cached) because Transform returns List<KeyedRegistration> (reference equality). This will be fixed by PR Replace Location with value-equatable LocationInfo — fix incremental cache invalidation on Clone #216 (LocationInfo + EquatableArray), but until then the test correctly verifies only the Combine stage behavior. XML doc explains the limitation.

All 133 tests pass.

@Skymly Skymly merged commit 3849263 into main Jul 4, 2026
3 checks passed
@Skymly Skymly deleted the test/incremental-edit-tests branch July 4, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add incremental edit tests — verify editing one file only regenerates affected contracts

1 participant