Skip to content

fix: use normal endpoint for incremental updates, preserve domains#62

Merged
jonathanpopham merged 2 commits intosupermodeltools:mainfrom
jonathanpopham:fix/incremental-update-churn
Apr 7, 2026
Merged

fix: use normal endpoint for incremental updates, preserve domains#62
jonathanpopham merged 2 commits intosupermodeltools:mainfrom
jonathanpopham:fix/incremental-update-churn

Conversation

@jonathanpopham
Copy link
Copy Markdown
Contributor

@jonathanpopham jonathanpopham commented Apr 7, 2026

Problem

Incremental updates are broken in two ways:

  1. 99% graph churn. AnalyzeIncremental sends a changedFiles multipart field the API doesn't understand. The API returns a full graph with new UUIDs for every node. Editing 1 file churns 1797 out of 1801 nodes.

  2. Domain corruption. mergeGraph unconditionally replaces domains with the incremental response. The API reclassifies domains from 1 file and returns garbage (5 real domains → 3 wrong ones like LocalCache, GraphAPI, CLIConfig).

Full investigation with repro data: #50

Fix

  • Swap AnalyzeIncrementalAnalyzeSidecars in the daemon's incrementalUpdate. The small zip goes through the normal endpoint. Graph-fusion proved this returns a correct subgraph with 0% churn.
  • Never replace domains on incremental merge. Domains are preserved from the last full generate. Domain classification requires full repo context — partial results are always wrong.
  • Delete AnalyzeIncremental and postIncrementalZip — dead code after the fix.

Results

Metric Before After
Node churn (1 file edit) 99% (1797/1801) 0% (8/1821)
Domains preserved No — replaced with garbage Yes — identical
Same entity, new UUID 1795 8 (only the changed file's nodes)

Tests

9 new tests covering merge logic:

  • TestMergeGraph_BasicMerge — new nodes added alongside existing
  • TestMergeGraph_FileReplacement — re-processed file replaces old nodes
  • TestMergeGraph_UUIDRemapping — relationships from unchanged files remap to new UUIDs
  • TestMergeGraph_ExternalDependencyResolution — suffix matching resolves LocalDependency placeholders
  • TestMergeGraph_EmptyIncremental — doesn't panic
  • TestMergeGraph_IncrementalNoMatchingExisting — new file added cleanly
  • TestMergeGraph_NilExisting — incremental into empty daemon
  • TestMergeGraph_DomainsPreservedOnIncremental — domains survive incremental merge
  • TestMergeGraph_DomainsPreservedEvenWhenIncrementalHasMore — domains preserved even when incremental has more

Test plan

  • go test ./... -race — all tests pass
  • go vet ./... — clean
  • Live test: supermodel watch → trigger incremental via supermodel hook → diff graph before/after → 0% churn, domains identical
  • Verified sidecar imported-by relationships correct after incremental

Known issue (pre-existing, not introduced by this PR)

The suffix matcher in mergeGraph can resolve LocalDependency nodes to the wrong file when multiple files share a common suffix (e.g. config/config.go matches archdocs/pssg/config/config.go). This affects both the old and new code paths. Filed separately.

Summary by CodeRabbit

  • Changes

    • Unified incremental and full analysis into a single analysis path (incremental changed-files upload removed)
    • Domains are preserved from the last full analysis during incremental updates and are no longer refreshed by incremental runs
  • Tests

    • Added comprehensive tests and test helpers covering incremental merge behavior, relationship remapping, file-level replacement, and domain preservation

The daemon's incrementalUpdate was calling AnalyzeIncremental which
sent a changedFiles multipart field the API doesn't understand. This
caused the API to return a full graph with new UUIDs for every node,
resulting in 99% graph churn on every single-file update.

Additionally, mergeGraph unconditionally replaced domains with the
incremental response. Since the API reclassifies domains from scratch
based on whatever's in the zip, sending 1 file produced garbage
domains that overwrote the correct domain model.

Fix:
- Use AnalyzeSidecars (normal endpoint) instead of AnalyzeIncremental
- Never replace domains on incremental merge — preserve from last
  full generate
- Delete dead AnalyzeIncremental and postIncrementalZip code

Verified: incremental update for 1 file now shows 0% node churn
(was 99%) and domains are preserved identically.

Closes supermodeltools#50
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Apr 7, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 96e18310-2476-4903-a154-19056b655bc3

📥 Commits

Reviewing files that changed from the base of the PR and between 74b3f12 and c7b8554.

📒 Files selected for processing (1)
  • internal/files/daemon_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • internal/files/daemon_test.go

Walkthrough

Removed the client's incremental-upload polling path (AnalyzeIncremental/postIncrementalZip) and switched daemon incremental analysis to call AnalyzeSidecars. Domains from incremental responses are no longer applied during merges. Added test helpers and a comprehensive merge test suite for Daemon.

Changes

Cohort / File(s) Summary
Client API
internal/api/client.go
Deleted exported Client.AnalyzeIncremental and its postIncrementalZip helper — incremental multipart changedFiles upload + polling path removed; only ZIP-only AnalyzeSidecars polling remains.
Daemon logic
internal/files/daemon.go
Replaced AnalyzeIncremental(ctx, zipPath, changedFiles, idemKey) calls with AnalyzeSidecars(ctx, zipPath, "incremental-"+idemKey[:8]). Modified mergeGraph to preserve existing d.ir.Domains (incremental responses no longer overwrite repo domains).
Tests / Test helpers
internal/files/daemon_export_test.go, internal/files/daemon_test.go
Added NewTestDaemon, MergeGraph, GetIR test helpers and a suite validating node/relationship merges, file-level replacement, relationship remapping, external dependency handling, nil/empty IR resilience, and domain preservation rules.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • greynewell

Poem

Incremental paths trimmed down to one,
Sidecars hum where two once ran,
Domains stand steady — not undone,
Tests stitch graphs like a careful hand,
Merge and keep the map as planned. 🎉

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 26.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main changes: switching to the normal endpoint for incremental updates and preserving domains during merges.
Description check ✅ Passed The description fully covers the problem, fix, results, tests, and test plan sections matching the template structure with comprehensive detail.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
internal/files/daemon_test.go (1)

286-289: Minor: Variable shadowing - d used for both daemon and loop variable.

On line 287, the loop variable d shadows the outer d (your test daemon). Doesn't break anything right now since you don't use the daemon after this point, but could cause confusion if someone adds code later.

Quick fix would be to rename the loop variable to something like dom:

✨ Suggested tweak
 	names := make(map[string]bool)
-	for _, d := range result.Domains {
-		names[d.Name] = true
+	for _, dom := range result.Domains {
+		names[dom.Name] = true
 	}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@internal/files/daemon_test.go` around lines 286 - 289, The loop uses the
variable name `d` which shadows the outer test daemon variable `d`; rename the
loop variable (for example to `dom`) in the loop that iterates over
`result.Domains` where you populate the `names` map so the outer `d` is not
shadowed and future code remains clear (update occurrences of the loop variable
within that loop accordingly).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@internal/files/daemon_test.go`:
- Around line 286-289: The loop uses the variable name `d` which shadows the
outer test daemon variable `d`; rename the loop variable (for example to `dom`)
in the loop that iterates over `result.Domains` where you populate the `names`
map so the outer `d` is not shadowed and future code remains clear (update
occurrences of the loop variable within that loop accordingly).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ad592569-cb00-4ddd-9ac9-3e3ed596f1ce

📥 Commits

Reviewing files that changed from the base of the PR and between f830ccf and 74b3f12.

📒 Files selected for processing (4)
  • internal/api/client.go
  • internal/files/daemon.go
  • internal/files/daemon_export_test.go
  • internal/files/daemon_test.go
💤 Files with no reviewable changes (1)
  • internal/api/client.go

Rename `d` → `dom` in domain iteration to avoid shadowing the test
daemon variable.
@jonathanpopham
Copy link
Copy Markdown
Contributor Author

Integration test data

Before fix (on main)

Triggered incremental update for 1 file (internal/cache/cache.go) via supermodel hook while supermodel watch was running.

=== NODE DIFF ===
Before: 1801 nodes
After:  1818 nodes
Kept (same ID): 4 (0%)
Removed: 1797
Added: 1814
Churn: 99%

Same entity, new UUID: 1795
Genuinely new entities: 19

=== DOMAINS ===
Before: ['CommandCLI', 'ApiClient', 'SidecarEngine', 'ArchDocsGenerator', 'SharedKernel']
After:  ['LocalCache', 'GraphAPI', 'CLIConfig']
Match:  False

Only 4 nodes survived (2 Domain + 2 Subdomain). The API returned the entire graph with new UUIDs. Domains were replaced with garbage from 1-file classification.

Sidecar corruption: cache.go imports internal/api and internal/config, but after the incremental update the sidecar said imports internal/archdocs/pssg/config/config.go — suffix matcher resolved to the wrong file when the whole graph was churned.

After fix (this PR)

Same test — 1 file incremental update via hook.

=== NODE DIFF ===
Before: 1821 nodes
After:  1837 nodes
Kept (same ID): 1813 (99%)
Removed: 8
Added: 24
Churn: 0%

=== DOMAINS ===
Before: ['Command', 'SupermodelAPI', 'SidecarEngine', 'ArchDocs', 'SharedKernel']
After:  ['Command', 'SupermodelAPI', 'SidecarEngine', 'ArchDocs', 'SharedKernel']
Match:  True

8 nodes removed (the cache.go File + Function nodes being replaced), 24 added (re-analyzed subgraph for that file). Domains identical.

Graph-fusion control (same API, same repo, same file)

For comparison, graph-fusion's incremental update (which uses the normal endpoint without changedFiles) on the same repo:

GRAPH-FUSION:
Before: 1801 nodes
After:  1817 nodes
Kept (same ID): 1793 (99%)
Removed: 8
Added: 24
Churn: 0%

This PR's results match graph-fusion's proven approach.

Unit tests

9 tests, all pass with -race:

=== RUN   TestMergeGraph_BasicMerge                          PASS
=== RUN   TestMergeGraph_FileReplacement                     PASS
=== RUN   TestMergeGraph_UUIDRemapping                       PASS
=== RUN   TestMergeGraph_ExternalDependencyResolution         PASS
=== RUN   TestMergeGraph_EmptyIncremental                    PASS
=== RUN   TestMergeGraph_IncrementalNoMatchingExisting        PASS
=== RUN   TestMergeGraph_NilExisting                         PASS
=== RUN   TestMergeGraph_DomainsPreservedOnIncremental        PASS
=== RUN   TestMergeGraph_DomainsPreservedEvenWhenIncrementalHasMore  PASS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant