feat: route auto-sync per substrate, so k8s drift stops triggering the AWS apply - #124
Merged
Conversation
…e AWS apply Implements #117, to the design in docs/design/per-substrate-autosync.md (#105). Auto-sync had two assumptions that only hold on a single-substrate estate. Detection was estate-wide: `driftDigest` reduced the whole overlay to one string, answering "did anything move" and never "what moved", so a drifted k8s Service and a drifted security group were the same event. Selection took the first match: `pickAutoSyncOp` was `ops.find((o) => o.kind === kind)` over a name-sorted list. Together, on a mixed estate whose halves are applied by different Ops, drift in the k8s half triggered the AWS apply — silently, because the Op ran and succeeded without touching what had actually moved. A wrong action, not a missing feature. It has bitten nobody only because auto-sync is off by default. - poll.ts: `driftDigestsByLexicon` + `changedLexicons`. Every IR node already carries `lexicon`; the reduction was throwing it away. Nodes only, so #103's cross-substrate anchor edges cannot make one substrate look like it moved because it is now joined to another that did. No new read. - ops.ts: `OpInfo.substrate`, scraped from the declared `target` the same way `name:`/`signalName:`/`env:` already are. chant's ApplyTarget is cloudformation|kubectl|arm -> aws|k8s|azure. An unrecognised or absent target is unscoped, never a guess. - autosync.ts: `pickAutoSyncOps` returns a pick per moved substrate plus the declines. Several Ops matching one substrate, or none, declines and says which — the discipline soleManagedCluster (#103) and addValueMatchEdges follow. The now-line carries it, because a self-heal loop that declines silently reads as a broken one. - history.ts + server.ts: the rollback interlock. While a rollback branch is open, `pull-request` mode must not reconcile — it would re-adopt exactly what the rollback removed. `apply` mode is untouched: healing toward rolled-back source is the intended completion of a rollback. Two decisions the design left open, both written up in its new Implemented section: An Op declaring no target is NOT automatically ambiguous. chant's ReconcileOp accepts no `target` at all, so every reconcile Op is unscoped, and reading an absent target as ambiguity would have disabled pull-request mode on every project. Declared substrate wins; failing that a single unscoped Op covers the estate; two or more competing still declines. The interlock suspends the whole estate rather than a subset, because behold's rollback takes no directory scope — narrowing would invent a distinction the command does not make. Gate: tsc clean, 430 tests (up from 399), build clean. Verified end to end against real .op.ts fixtures: a drifted k8s Service routes to k8s-apply, where the old find-by-kind returned cloud-apply. Closes #117
lex00
force-pushed
the
feat/117-per-substrate-autosync
branch
from
August 3, 2026 17:05
4de8f52 to
18a3eb2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #117. Implements the design in
docs/design/per-substrate-autosync.md(#105).The bug
Auto-sync had two assumptions that only hold on a single-substrate estate.
Detection was estate-wide.
driftDigestreduced the whole overlay to one string — it answered did anything move and never what moved, so a drifted k8s Service and a drifted security group were indistinguishable.Selection took the first match.
pickAutoSyncOpwasops.find((o) => o.kind === kind)over a list sorted by name.Together, on a mixed estate whose halves are applied by different Ops (a
cloudformationApplyOp for the cloud half, akubectlone for the k8s half), drift in the k8s half triggered the AWS apply — silently, because the Op ran and succeeded without touching what had actually moved. That is a wrong action, not a missing feature. It has bitten nobody only because auto-sync is off by default.What changed
poll.ts—driftDigestsByLexicon+changedLexicons. Every IR node already carrieslexicon; the reduction was throwing it away. Nodes only, so feat: cross-substrate live overlay -- preserve the IR's cross-provider edges in --live #103's cross-substrate anchor edges cannot make one substrate appear to move because it is now joined to another that did. No new read, no chant change.ops.ts—OpInfo.substrate, scraped from the declaredtargetexactly the wayname:/signalName:/env:already are. chant'sApplyTargetiscloudformation | kubectl | arm->aws | k8s | azure. An absent or unrecognised target is unscoped, never a guess.autosync.ts—pickAutoSyncOpsreturns a pick per moved substrate plus the declines. Several Ops matching one substrate, or none, declines and says which — the same disciplinesoleManagedCluster(feat: cross-substrate live overlay -- preserve the IR's cross-provider edges in --live #103) andaddValueMatchEdgesfollow. The declines reach the now-line, because a self-heal loop that declines silently reads as a broken one.history.ts+server.ts— the rollback interlock. While a rollback branch is open,pull-requestmode must not reconcile: it would re-adopt exactly what the rollback removed.applymode is untouched, since healing toward newly-rolled-back source is the intended completion of a rollback.Two decisions the design left open
Both are written up in the design doc's new Implemented section, since each narrows a claim it makes.
An Op declaring no target is not automatically ambiguous. Question 2 reads an undeclared target as the ambiguity case. Taken literally that disables
pull-requestmode everywhere — chant'sReconcileOpaccepts notargetat all (it reads the estate and opens a PR; there is no transport to name), so every reconcile Op is unscoped. Implemented rule: a declared substrate wins; failing that a single unscoped Op covers everything that moved; two or more competing still declines.The interlock suspends the whole estate, not a subset. Question 3 anticipates narrowing by the rolled-back
sourceDir, but behold's rollback (/api/rollback->chant lifecycle rollback <env> --to <ref>) takes no directory scope, so the rollback really is whole-project. Narrowing would invent a distinction the command does not make.suspendedByRollbackalready takes the moved set, so it narrows there if rollback ever gains a scope.One behaviour worth flagging
A substrate whose Op is refused by the one-write-at-a-time guard is not retried — the poll advances its baseline whether or not a trigger started, so it waits until it drifts again. That was equally true of the single-Op loop, which dropped the same event just as silently; the change is that the refusal is now visible.
Gate
tscclean, 430 tests (up from 399), build clean.Verified end to end against real
.op.tsfixtures rather than only in units — discovery through routing on a mixed estate:The old find-by-kind returned
cloud-applyfor that same event.