GitSync adopts a stale prebuilt assembly over the source it just synced — silently running last week's code, and it destroyed client data

[rbuergi]

What happened

A git_hub_sync update pulled new source for Crm/Migration, then adopted a prebuilt assembly instead of compiling what it had just pulled. The node reported success. The next run executed the old code and destroyed live client data.

Sync activity Crm/_Activity/0c389edf, 2026-08-30T17:06:04Z:

Fetching the branch HEAD from GitHub and importing the deltas…
Recompiling 2 NodeType(s): Crm/Migration, Crm/Opportunity
Adopted 2 prebuilt assembly(ies) — the rest compile.
Imported Imported (6 node(s)).

Status Succeeded. Nothing in that log tells an operator that the assembly now serving Crm/Migration was not built from the source in the line above it.

Why it was wrong, provably

The synced commit (904854f) contained three fixes merged ~40 minutes earlier. A migration run at 17:06:37 exhibited all three pre-fix behaviours:

fixed in the synced source	what the adopted assembly did
read the document body shape-tolerantly	stripped the body from 4 promoted documents
a bullet whose bold text is a link is not a person	promoted 2 documents to Crm/Contact
never promote/suggest a node already typed Crm/*	suggested the client's own Crm/Opportunity

I then forced a compile via the compile verb. Afterwards compiledSources matched currentSourceVersions and the MVID changed (ed21acce… → 07d5ca73…), confirming the adopted assembly had been built from different source than the node was holding.

The damage

Four client documents lost their entire text — one a 7,000-word commercial offer. Recovered from version history.

One did not recover. PartnerRe/EslLifeCycleProcess has exactly one version in history — the one the stale code wrote. restore_from_point_in_time returns "No version found at or before…", and content indexing is off on that deployment, so there is no chunk copy either. Its markdown body is permanently gone.

Why this is worse than a normal staleness bug

The operator did everything right: merged a green PR, ran check, ran update, saw Succeeded, saw the NodeType at compilationStatus: Ok with compiledSources == currentSourceVersions. Every signal available said the fix was live. The adopted assembly is what made all of those true while being false in the only way that mattered.

What I think the fix is

1. Never adopt a prebuilt for a NodeType whose source just changed in this import. The import knows which sources it wrote; that set should force a local compile.
2. If adoption stays, it must be verifiable. Record on the NodeType which commit/source-hash the adopted assembly was built from, and refuse adoption when it does not match currentSourceVersions. Today compiledSources is populated in a way that makes an adopted assembly indistinguishable from a compiled one.
3. Say it in the log. Adopted 2 prebuilt assembly(ies) should name them and say what they were built from — an operator reading that line cannot currently tell it is a warning.

Point 2 is the important one: this class of bug is invisible precisely because the verification I was taught to do (compiledSources vs currentSourceVersions) reported healthy.

Secondary: the hub did not recover

After the forced compile, Crm/Migration's hub stopped responding — the control-plane watcher no longer picks up an actionable request, and get @{instance}/area/Run times out. recycle on the NodeType did not clear it. Possibly a separate defect around compile-while-hub-live; noting it here because it is the state the forced recompile left behind, and it is currently blocking the migration.

Evidence: Admin/Migration/restructure-all-2 on memex.systemorph.com sits at requestedAction: Run, state: Requested, execute: true and is never picked up.

[rbuergi]

Same defect from the CI end — and it already has a home in #2702

I had an agent chasing what turns out to be this, filed as a read-your-own-write hypothesis. This report corrects it to adoption, which is the more dangerous reading, and I have redirected accordingly.

The CI face

NodeTypeCompileParkTest.ParkedNodeType_CompileToolRetry_AfterFix_SettlesOkAndUnparks (core, shard 1 of run 33322993649, on a documentation-only PR whose diff cannot reach any of this):

Expected value to be "Ok" because a deliberate compile retry of a parked type with fixed
source must run a REAL compile and succeed, but found "Error"

The test parks a NodeType with broken source, writes valid source and awaits that Update, then asks the MCP compile tool for a deliberate retry. It fails 4 seconds after TEST START — a wrong answer, not a slow one — and its output carries the BROKEN source's diagnostics (CS1040 line 58, from BrokenConfiguration). The retry did not compile the fixed source; it adopted bytes built from the old one.

Three siblings fail with it, all the same "fix source → recompile" shape: ParkedNodeType_RecycleRetry_AfterFix_SettlesOkAndUnparks, ParkedNodeType_SourceFix_AutoRecompilesAndUnparks_WithoutDeliberateRetry, CodeEditRecompileTest.FailedCompile_DirectCompileToolRetry_RunsFreshCompileAndCreatesRelease.

In the wild, on a third surface

MeshWeaver.Reinsurance main, run 33322605771 (RED at 16:29Z, GREEN at 16:35Z on the same commit):

bake-seed[0] Prebuilt assembly ADOPTED for Ifrs17/AocConfiguration at version 0
             (framework s770d75c0738de799a1c67802f05bf03f) — no compile needed
…
GATE FAILED — render: Ifrs17/AocConfiguration — 1 new failure(s)

So the same adoption path produces: a false CI red on satellites, a false CI red on core, and permanent data loss on a client portal. That is not three flakes.

Consolidation, not a fourth issue

• Prebuilt adoption stamps the LIVE source snapshot onto bytes compiled from an OLDER one — a stale bundle reports compiledSources == currentSourceVersions and runs old code as current #2702 — "Prebuilt adoption stamps the LIVE source snapshot onto bytes compiled from an OLDER one" — is this defect, already stated. It should be closed into this one or vice versa; this issue keeps the evidence because it has the data loss.
• Adopted prebuilt build's source stamp is refused by MergeGuard — IsDirty never converges (the gate now names it compile-status-unrecorded, #2649) #2463 — "Adopted prebuilt build's source stamp is refused by MergeGuard — IsDirty never converges" — is live on the very node in this report. memex portal logs, Crm/Migration, 17:11Z: [MergeGuard] Crm/Migration: refused stale/reordered cross-hub write to 'compilationStatus' and … to 'lastModified', then MeshNode Conflict … 2 field(s) changed on the owner since the writer's base and nothing was applied.

The requirement that matters most

The reporter's sharpest finding is not the stale bytes — it is that every staleness detector said the fix was live: compilationStatus: Ok and compiledSources == currentSourceVersions, which is the exact check the Crm operator docs teach people to run after a deploy. The MVID only moved (ed21acce… → 07d5ca73…) once a compile was forced.

So a fix that merely makes the compile correct is insufficient. An operator must be able to tell an adopted assembly from a compiled one. The proposal in the body — record the commit/source-hash the adopted bytes were built from, and refuse adoption when it disagrees with currentSourceVersions — is the part that closes the data-loss path, and the MVID shift is the regression test's assertion.

Separately: the wedge reported at the end of this issue is NOT compile-while-hub-live

memex portal logs at 17:11:21.408Z:

AccessControlPipeline: Access denied: user 'rbuergi' lacks Update permission on 'Crm/Migration'
   (triggered by message=PatchDataRequest sender=cache/_ciqCj6MNUWJjheeu08jRg)
[UpdateRemote] OWNER_DENIED … [UpdateQueue] FAILED path=Crm/Migration seq=632
Recycle: failed to stamp release request for Crm/Migration — disposing the hub anyway

plus, on the other replica, [UpdateRemote] NO-OP … contentType=JsonElement — lambda returned unchanged; check the lambda's content-type pattern match. The grant is missing, so the control-plane request is never written; recycle tore the hub down anyway; and where a write does reach, the node reads as an untyped JsonElement so the lambda's pattern match silently no-ops (the .As<T>() trap-door, same shape as #2729). Worth not splitting out as its own compile-while-hub-live defect.

[rbuergi]

Two corrections and one hazard, from the reporter's follow-up probe

1. The hub is UP. The NODETYPE is not resolving.

A read (rather than another write) settles it — get @Admin/Migration/restructure-all-2/area/Run returns a rendered response, not a timeout:

Area not found. No renderer is registered for area Run on hub Admin/Migration/restructure-all-2.
Available named areas: $Collection, $Content, $Data, … Edit, Export, … Versions

That list is only the generic node areas — no Run, no Tests, nothing from Crm/Migration's own AddMigrationLayoutAreas() / AddMigrationControlPlane(). The instance activates and serves; it just does not know it is a Crm/Migration. The type's configuration is never applied, so no watcher exists to pick up an actionable request.

This is the same fact as the portal-log lines from the other end — [UpdateRemote] NO-OP … contentType=JsonElement — lambda returned unchanged and EnrichWithNodeType slow path for 'Crm/Migration' faulted (TimeoutException). The type never registers, so every instance falls back to generic behaviour and every typed Update lambda misses its pattern match.

Stating it as "the hub is down" would send the next reader after grain lifetime, and the hub is demonstrably alive. Correcting my earlier comment on this issue accordingly.

2. Adoption + an armed control-plane node is a loaded gun

The reporter's queued run was sitting at requestedAction: Run, execute: true. If the type re-registers against the stale adopted assembly, that fires the pre-fix code unattended — stripping document bodies again, with nobody watching. They have set requestedAction: None and written the reason into the node's description so it cannot fire and whoever finds it knows why.

This is the fix requirement arriving from a second direction: the window between "the type re-registers" and "the operator notices which assembly it got" is unattended execution of whichever bytes won, and the operator has no signal that distinguishes them. Any fix must make the adopted-vs-compiled distinction visible before a control plane can act on it — not only auditable afterwards.

3. Why forcing a sync is not the recovery

A git_hub_sync update runs as system-security and would re-register the type — but it can re-adopt the same stale prebuilt, and with a control-plane node in the space that is exactly the combination above. So the obvious recovery is the dangerous one. The space is being held deliberately.

Also worth its own line

The permanently-lost document (PartnerRe/EslLifeCycleProcess) had one version in history and content indexing off, so there was no second copy anywhere. That is a deployment setting, not a code path — a portal holding client documents with neither version depth nor an index has nothing to restore from. It is the difference between what was recovered here and what was not.

[rbuergi]

Not the same defect as the NodeTypeCompileParkTest CI red — measured, so the two do not get conflated

Nothing here disputes this issue: the sync/adoption defect and its data loss are real and the
proposed fix (never adopt a prebuilt for a NodeType whose source changed in this import; make an
adopted assembly distinguishable from a compiled one) stands.

I was pointed at this issue as the explanation for the four "fix the source, then recompile" tests
that failed in run 33322993649 shard 1 (PR #2807). It is not — those are #890, and the log
settles it, so recording it here to stop the two being merged:

• No adoption anywhere near them. The shard log carries no adoption line for any ParkTest/*
  node. A real compile ran.
• The retry's error is not a stale diagnostic. ParkTest/CompileRetry's output does contain the
  broken source's CS1040 line 58 — from the first compile at 16:43:45.406, before the test
  wrote the valid Configuration. The retry ran at 16:43:46.528 and the compile tool returned
  NullReferenceException from Cci.MetadataWriter plus canary=BELOW-ROSLYN. Adjacency in the
  captured output made the two look like one event; the timestamps separate them.
• The process, not the code. After 16:42:42.375, 7 of 7 compiles in that process that reached
  Roslyn's metadata writer failed identically over 6 m 15 s and none succeeded, while compiles that
  needed only diagnostics kept returning correct CS####. Full measurement:
  CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 (comment)

Where the two do touch, and it is worth stating because it argues for point 2 of this issue:
both are "the recorded state says the compile succeeded/failed on grounds it never actually
established". #890's half is now fixed in the failure direction — an emit the canary proved the
process could not do is recorded as Unavailable (not evaluated) instead of Error, so it no longer
writes a verdict nothing formed and no longer stops the automatic re-drive. The success direction —
an adopted assembly reporting compilationStatus: Ok with compiledSources == currentSourceVersions
while being built from different bytes — is exactly the same class of unearned claim, and is this
issue's to fix.

[rbuergi]

Diff shape for the adoption fix — posted BEFORE writing it, because it is cross-repo and one premise needs correcting first

1. Confirmed at code level, and the two halves are in the same file

PrebuiltAssemblySeeder.IsAlreadyAdopted says, verbatim:

🚨 Note in particular what this does NOT compare: … Whether a build is stale with respect to its
SOURCES is CompiledSources/CurrentSourceVersions's question, answered by the compile
watcher, and it is deliberately not re-asked here.

…and 100 lines later Seed sets RequestedSourceStampAt, whose contract (NodeTypeDefinition.cs)
is that the owner then writes CompiledSources = CurrentSourceVersions.

So the seeder defers staleness to the compile watcher, and the same adoption makes the compile
watcher's staleness test true by construction. Nobody compares bytes to source text at any point.
That is why both detectors read clean: they are not broken, they are answering a question the
adoption already answered for them.

The gate is DeclineReason(frameworkMvid) (framework API-surface hash — deliberately stable
across internal merges) ∧ the per-type dependency record ∧ the IsAlreadyAdopted skip. None is a
function of the source text, and I confirmed the bundle carries nothing source-derived either:
PluginBundleClient passes only manifest.FrameworkMvid and a.Dependencies into Seed.

2. So the fix is cross-repo — there is nothing to compare against today

• Producer (the bake, node-repo-publish-bake.yml in every node repo): record a per-entry
  fingerprint of the source set the bytes were built from (the source hash, or the commit).
• Consumer: carry it through PluginBundleClient → Seed → onto the node beside
  RequestedSourceStampAt.
• The check must run on the OWNER, not in Seed — for the exact reason RequestedSourceStampAt
  exists (Prebuilt adoption stamps CompiledSources from a field its own activation is still publishing — the adopted build is born dirty and the next release request compiles it anyway #1834): the seeder writes CROSS-HUB and its lambda sees the mirror's snapshot, which
  predates the first-activation write of CurrentSourceVersions that its own subscribe triggers.
  Only the owner holds both authoritatively.

The good news: there is exactly one place to change, and it is pure —
NodeTypeCompilationHelpers.ApplyAdoptedSourceStamp, deliberately shared by all three writers that
can fulfil the request (the sources watcher's publication, the release-request dispatch, and
InstallAdoptedSourceStampWatcher) "so two of them can never disagree about what adopted and
current means". Turning it from assert to check fixes all three at once, and it is
unit-testable with no hub and no timing:

fingerprint vs the owner's live CurrentSourceVersions	today	proposed
matches	stamp CompiledSources = CurrentSourceVersions	unchanged
disagrees	stamp anyway ⇒ IsDirty=false, the lie	do not stamp; flip CompilationStatus = Pending so the live source compiles locally
absent (legacy bundle)	stamp anyway	adopt, but do not stamp a match it cannot prove — record the absence so compiledSources == currentSourceVersions stops being something an operator can be taught to trust

The legacy row is the one that needs a scope call, and it is why I am posting rather than pushing:
refusing unproven bundles breaks every bundle published to date and the node-repo CI gates that
depend on prebuilt fetches, so the rollout has to be producer-first, with the consumer tolerant
until the fleet has re-baked. That is a maintainer decision, not mine.

3. Two items from the escalation I could not verify, flagged rather than assumed

• "Force-release is a no-op under a sealed bundle." If that holds it is a separate defect, in
  the release-request watcher's "satisfied by the existing current build" branch — which is
  precisely the branch ApplyAdoptedSourceStamp's contract exists to satisfy (IsDirty false by
  construction). Worth its own line here, because "there is no operator recovery verb" is a
  different severity from "the adoption is wrong".
• The armed control plane. Agreed that the adopted-vs-compiled distinction has to be visible
  before a control plane can act on it. Note that the proposed Pending flip gives that for free
  in the disagreement case — the type is not Ok, so nothing can arm against it.

4. 🚨 Correction: NodeTypeCompileParkTest is NOT the regression for this

I was asked to treat it as the unit-level regression that "already fails without the fix". It is
not, and using it as the proof would produce a guard that checks nothing.

Those tests never adopt anything — there is no adoption line for any ParkTest/* node anywhere in
the shard log — and their failure in run 33322993649 is #890: the retry ran a real compile and
Roslyn threw NullReferenceException from Cci.MetadataWriter with canary=BELOW-ROSLYN. The
broken source's CS1040 line 58 in the same test's output is the first compile at
16:43:45.406, 1.1 s before the retry; adjacency in the captured output is what makes them look
like one event. Measurement:
#890 (comment)

Those tests will pass and fail identically with or without the adoption fix. This fix needs its
own regression — the natural one is pure and cheap: ApplyAdoptedSourceStamp given a fingerprint
that disagrees with the live snapshot must not stamp, with the MVID shift as the integration-level
assertion that the bytes actually changed.

Related work already landed

#2816 fixes the failure direction of the same "unearned claim" class — an emit the process could
not perform was recorded as Error and stamped with a verdict-inputs token, which then stopped
every automatic retry. Same shape as this issue's point 2 (a record asserting something nothing
established), opposite direction. It does not touch adoption.

[rbuergi]

🚨 Retraction: the park-test failures I posted here are NOT this defect

Earlier in this thread I offered NodeTypeCompileParkTest.ParkedNodeType_CompileToolRetry_AfterFix_SettlesOkAndUnparks and its three siblings as "the CI face" of stale-prebuilt adoption, and the Reinsurance AocConfiguration red alongside them. That attribution was wrong. A dedicated investigation has measured it, and the correction matters because it would have sent the adoption fix looking for a regression test that cannot detect it.

What those five failures actually were: #890 — one Roslyn Emit poisoning the whole process. From the job log of run 33322993649, after 16:42:42.375:

	count
compiles reaching Roslyn's metadata writer	7 — all failed identically, over 6m15s
…that succeeded	0
compiles needing only diagnostics	all correct (CS1040, CS0246)

Parse and bind stayed healthy; emit was dead, permanently. The exit=124 shard timeout is a consequence, not a second problem.

Both hypotheses I put forward are dead, with proof:

• Read-your-own-write — the retry did run a real compile. The CS1040 line 58 I read as "the retry compiled the old source" came from the first compile at 16:43:45.406, before the test wrote the valid Configuration. The retry ran at 16:43:46.528 and returned NullReferenceException with canary=BELOW-ROSLYN. Adjacency in the captured output made two events look like one — and I built an argument on that adjacency.
• ReleaseRequestWatcherHighWaterTest was a timing casualty of the shard cap — no. Its compile threw 174 ms after TEST START. The 50-second wait was waiting for an Ok the process could no longer produce. fix: four silent defects from the issue sweep (#2729, #2731, #2793, #2797) #2800's re-split does not touch it.

So: I was wrong twice here, and the peer heuristic I corrected earlier ("all five are cap casualties") was also wrong, in the other direction. All five are one event, and that event is #890.

What this does and does not change for THIS issue

Unchanged: the adoption defect is real, confirmed in code, and caused the data loss. The mechanism stands — adoption asserts rather than checks, the gate keys on identity + node version, neither of which is a function of the source text, and the owner then stamps CompiledSources = CurrentSourceVersions.

Changed, and important for whoever writes the fix: NodeTypeCompileParkTest never adopts anything and would pass or fail identically with or without an adoption fix. Using it as the regression proof would ship a guard that checks nothing — the exact failure mode this repo keeps paying for. The adoption fix needs a test that actually exercises adoption.

Better news from the same investigation: ApplyAdoptedSourceStamp is one pure function shared by all three writers, so changing it from assert to check fixes all three call sites at once and is unit-testable directly — a far better home for the regression than any park test.

[rbuergi]

Refined diff shape (three-way), plus two findings that change it — still not pushed

Working from the scope call: don't refuse legacy bundles, mark them; refuse only a stamp that
demonstrably disagrees; the field must be readable on the node, not in a log. Agreed on all three.
Two things surfaced while shaping it that need a decision before code.

🚨 Finding 1 — the table and constraint 1 contradict each other on the legacy row

The table says legacy ⇒ adopt, record Unverified. Constraint 1 says stop stamping
CompiledSources = CurrentSourceVersions on an unverified adoption. Applied to the legacy row
those are not the same instruction, and the difference is a fleet outage:

IsDirty is exactly CompiledSources != CurrentSourceVersions. Not stamping ⇒ every legacy-bundle
type is IsDirty on arrival ⇒ the release watcher's "satisfied by the existing current build"
branch (which requires !IsDirty) stops absorbing ⇒ every install recompiles every type — the
43 activations / 13.5 s of boot the prebuilt lane exists to remove. And on a Modules:RequirePrebuilt
mesh it is worse than slow: a local compile is refused by design (PrebuiltRequiredException), so
not stamping would park every legacy-bundle type. That is the outage refusal was rejected for,
arriving through a different door.

So constraint 1 is right for the mismatch row and unsafe for the legacy row. My proposal:

bundle stamp	matches live sources	CompiledSources	CompilationStatus	BuildProvenance
yes	yes	stamp (as today)	Ok	AdoptedVerified
yes	no	do not stamp	Pending → real local compile	AdoptedRefused
no (legacy)	unknown	stamp (lane keeps working)	Ok	AdoptedUnverified

The legacy row keeps the staleness answer it has today but stops the provenance claim being
invisible — which is the load-bearing requirement as you stated it (visibility, not refusal). If you
want the stricter reading instead, it has to be producer-first with a fleet re-bake before the
consumer tightens, and that is a bigger programme than this fix.

On the middle row: "refuse" means fall back to a local compile, which is the existing decline path
(Seed emitting false already means "compile normally"). On a require-prebuilt mesh that fallback
is forbidden, so a stale bundle parks the type. That is the correct trade — a parked type beats
silently running last week's code over live client data — but it is loud and should be stated in the
PR, not discovered.

🚨 Finding 2 — "force-release is a no-op under a sealed bundle" is not what the code does, and #2813 itself contradicts it

I checked both absorb branches in InstallReleaseRequestWatcher. Both explicitly guard on force:

if (!def.RequestedReleaseForce && def.CompilationStatus is CompilationStatus.Ok
    && !def.IsDirty && HasUsableBuild(curr, def, GuardsOf(hub)))   // "satisfied by existing build"

internal static bool IsSatisfiedByInFlightCompile(NodeTypeDefinition def, string requestedToken)
    => !def.RequestedReleaseForce && …                              // "satisfied by in-flight"

A forced request therefore falls through to the COMMIT path and flips CompilationStatus = Pending,
which dispatches a real compile. And this issue's own report says the same thing happened in
practice: "I then forced a compile via the compile verb. Afterwards compiledSources matched
currentSourceVersions and the MVID changed (ed21acce… → 07d5ca73…)".

So there is a recovery verb, and it worked on the affected node. If a peer session observed force
being inert somewhere else, the code says the cause is not the release watcher. The candidate that
does explain it is Modules:RequirePrebuilt: on such a mesh the forced compile is refused and the
type parks, so force looks inert. That is a genuinely missing recovery verb, but only on
require-prebuilt meshes, and it is a different defect from this one. I did not want to write "no
recovery verb exists" into a PR on a claim the code contradicts — if the observation is real, please
point me at the deployment and node and I will pin it as its own issue rather than fold it in here.

Mechanics, once the rows are settled

• Producer (node-repo-publish-bake.yml in every node repo): a per-entry source fingerprint.
• Carrier: PluginBundleClient → PrebuiltAssemblySeeder.Seed(…, sourceFingerprint) → written
  onto the node beside RequestedSourceStampAt. Today Seed receives only manifest.FrameworkMvid
  and a.Dependencies — nothing source-derived — so this is the cross-repo half.
• The check runs on the OWNER, never in Seed — the Prebuilt adoption stamps CompiledSources from a field its own activation is still publishing — the adopted build is born dirty and the next release request compiles it anyway #1834 reason: the seeder writes CROSS-HUB and
  its lambda sees the MIRROR's snapshot, which predates the first-activation write of
  CurrentSourceVersions that its own subscribe triggers. Only the owner holds both authoritatively.
• One function to change: NodeTypeCompilationHelpers.ApplyAdoptedSourceStamp — pure, and
  deliberately shared by all three writers that can fulfil the request (sources-watcher publication,
  release-request dispatch, InstallAdoptedSourceStampWatcher) "so two of them can never disagree
  about what adopted and current means". Assert → check fixes all three at once.
• Both new fields are operational: adoptedSourceFingerprint and buildProvenance must be added
  to NodeTypeOperationalContent's list (stripped on export, preserved from the live node on import,
  ignored by change detection) — otherwise an authored value would forge a provenance claim, which is
  the same class of bug as the one being fixed.

Regression — on the pure function, all three rows mutation-proved

ApplyAdoptedSourceStamp given (a) a matching fingerprint stamps and records AdoptedVerified;
(b) a disagreeing fingerprint does not stamp and flips Pending — the data-loss row, and the
one whose mutation must fail loudest; (c) an absent fingerprint stamps but records
AdoptedUnverified, never Verified. Plus the reporter-side assertion the incident actually needed:
a node reading Ok + compiledSources == currentSourceVersions must not be indistinguishable
from a compiled one — buildProvenance is what makes that stateable.

🚨 To be explicit, since it caused a wrong attribution earlier: NodeTypeCompileParkTest is not
this fix's regression and must not be cited as one. It never adopts; its failures in run
33322993649 are #890.

Ready to implement on a word from the maintainer on Finding 1's row and on whether Finding 2's
observation is real.

[rbuergi]

🚨 Retraction #2: "force-release is a no-op under a sealed bundle" — the code and this issue both contradict it

I relayed that claim from another session and amplified it into "operators have no recovery verb at all". Both halves are wrong, and this issue's own report is the refutation.

• InstallReleaseRequestWatcher's both absorb branches explicitly guard !def.RequestedReleaseForce, so a forced request falls through to the COMMIT path and dispatches a real compile.
• The body of this issue says exactly that happened: "I then forced a compile… the MVID changed (ed21acce… → 07d5ca73…)". The recovery verb exists and worked on the affected node.

I passed a peer's observation on as established fact because it fitted the shape I already had. That is the second retraction I owe this thread and the third unverified claim I have relayed today; recording it here rather than quietly deleting it, because a data-loss issue is the worst place for a confident wrong statement to sit.

If that behaviour was genuinely observed, the one configuration that would produce it is Modules:RequirePrebuilt, where the forced compile is refused by design and the type parks — which looks identical to "force did nothing" from outside. That is a real gap, but a different defect on require-prebuilt meshes only, and it belongs in its own issue with the deployment and node named. I have asked for those.

Correction to my own proposed fix, from the same review

My earlier scope note said an unverified adoption must stop stamping CompiledSources = CurrentSourceVersions. That would cause the outage it was meant to prevent. IsDirty is CompiledSources != CurrentSourceVersions, so withholding the stamp makes every legacy-bundle type dirty on arrival, the !IsDirty absorb branch stops firing, and every install recompiles everything — and on a Modules:RequirePrebuilt mesh, where a local compile is refused by design, it parks every legacy-bundle type.

The corrected shape, which keeps the load-bearing requirement (visibility, not refusal) without that blast radius:

bundle carries a source stamp	matches currentSourceVersions	outcome
yes	yes	adopt, BuildProvenance = Verified
yes	no	🚨 REFUSE — the data-loss case, and the only hard fail
no (legacy)	—	adopt, keep the stamp, BuildProvenance = AdoptedUnverified

An operator can then tell an adopted assembly from a compiled one, and a verified adoption from an unverified one, before a control plane acts on it — which is what turns "adoption plus an armed control-plane node is a loaded gun" back into something disarmable on sight. Withholding the stamp was never necessary for that; a provenance field is.

Regression home: ApplyAdoptedSourceStamp — one pure function shared by all three writers, unit-testable without a mesh. All three rows mutation-proved, especially the middle one.

[rbuergi]

🚨 Retracting my Finding 2 — the peer session was right, I was wrong, and the mechanism is worse than either of us had

I wrote that "force-release is a no-op under a sealed bundle" was contradicted by the code, because
both absorb branches in InstallReleaseRequestWatcher guard on !def.RequestedReleaseForce. That
part is true and it is also irrelevant, because I stopped looking one layer too early.

RequestedReleaseForce is honoured by the release watcher — and then silently discarded by the
compile watcher. On a Pending flip, before any Roslyn pass:

attempt.Disposable = prebuiltConsumer.SeedForTypes([hubPath])
    .Subscribe(landed =>
    {
        if (!landed) { DispatchOrPark(); return; }
        logger?.LogInformation(
            "Compile watcher: {HubPath} adopted a prebuilt assembly on demand — "
            + "settling without a Roslyn pass.", hubPath);
    }, …);

RequestedReleaseForce does not appear anywhere in that region. So a forced request re-adopts the
same sealed bytes and returns: same assembly, same MVID, Ok, request recorded as handled.
DispatchOrPark() — the only route to Roslyn — is reached only when adoption misses.

That is exactly the reported symptom, and the reporter's wording was exact: "under a sealed
bundle" is the resolving case. This issue's own "I forced a compile and the MVID changed" is the
missing case. Both observations are correct; the discriminator is whether SeedForTypes resolves
an artifact at that moment. I had two true reports and treated the one that did not fit as an error —
which is the same failure I flagged in the attribution above, committed by me.

Filed as #2818 with the mechanism, the proposed fix (an explicit force must suppress on-demand
re-adoption), and the require-prebuilt consequence.

This matters for the shape below: the three-way's middle row refuses a stale adoption and drives a
local compile. Until #2818 lands, an operator who sees AdoptedUnverified on a node still has no
verb that rebuilds it while its bundle resolves. So #2818 is not a nice-to-have beside this fix — it
is the recovery path this fix's diagnosis points at, and the two should land together or in that
order.

My earlier comment asking whether the observation was real is withdrawn; no one needs to hunt for a
deployment and node.

[rbuergi]

The diff, as agreed — posted before pushing so the Crm session sees it first

Branch fix/2813-adoption-provenance, cut from main at 2160d6525. Not pushed.

 src/MeshWeaver.Mesh.Contract/Services/BuildProvenance.cs        |  63 ++++++  (new)
 src/MeshWeaver.Mesh.Contract/NodeTypeOperationalContent.cs      |   6 +
 src/MeshWeaver.Graph/Configuration/NodeTypeDefinition.cs        |  52 +++++
 src/MeshWeaver.Graph/Configuration/NodeTypeCompileState.cs      |  27 +++
 src/MeshWeaver.Graph/Configuration/NodeTypeCompilationHelpers.cs| 114 ++++++--
 src/MeshWeaver.Graph/Configuration/PrebuiltAssemblySeeder.cs    |  52 +++++
 test/MeshWeaver.Graph.Test/AdoptedBuildProvenanceTest.cs        | 191 ++++++  (new)
 + 3 one-line cref fixes (the overload made them ambiguous — CS0419)

The three-way, exactly as scoped

if (def.AdoptedSourceFingerprint is not { Length: > 0 } adopted)
    return stamped with { BuildProvenance = BuildProvenance.AdoptedUnverified };   // legacy: KEEPS the stamp

if (def.CurrentSourceFingerprint is not { Length: > 0 } live)
    return stamped with { BuildProvenance = BuildProvenance.AdoptedUnverified };   // not compared yet

if (string.Equals(adopted, live, StringComparison.Ordinal))
    return stamped with { BuildProvenance = BuildProvenance.AdoptedVerified };

return def with                                                                    // 🚨 the data-loss row
{
    RequestedSourceStampAt = null,
    CompilationStatus = CompilationStatus.Pending,   // compile the LIVE source
    BuildProvenance = BuildProvenance.AdoptionRefused,
};                                                    // note: no CompiledSources — the stamp IS the lie

One pure function, three writers, all fixed at once. BuildProvenance.Compiled is the zero value, so
every record written before this field existed reads as the honest default.

Three design points that were not obvious, and where they landed

1. The fingerprint must be CONTENT, not versions. CurrentSourceVersions is
{path → LastModified.UtcTicks} — mesh-local mtimes the producer cannot know and does not have (the
bake writes zeros). A fingerprint over ticks would never match and every adoption would be refused.
So both sides use PartitionSourceFingerprint — content-based, order-independent, already in
Mesh.Contract, already the static-repo importer's change detector.

2. The comparison cannot happen in Seed, for the same #1834 reason the stamp request exists:
the seeder writes CROSS-HUB and its lambda sees the MIRROR's snapshot, which predates the
first-activation write its own subscribe triggers. So the owner publishes its own
CurrentSourceFingerprint in the same update as CurrentSourceVersions, and
ApplyAdoptedSourceStamp compares two strings already on the node — staying pure, no new I/O at any
of the three writers.

3. The live fingerprint is computed only when it can be USED — when the node carries an adopted
fingerprint or a pending stamp request:

if (pendingStamp || def.AdoptedSourceFingerprint is { Length: > 0 })
    updated = updated with { CurrentSourceFingerprint = PartitionSourceFingerprint.Compute(
        published.Nodes, versioned: false, hub.JsonSerializerOptions) };

Otherwise every source emission for every NodeType would pay a SHA-256 over the whole serialised
source set on a path that today reads only timestamps, for an answer nothing consumes. A
locally-compiled type pays nothing.

Two things the repo's own guards caught, worth recording

• NodeTypeCompileState had to carry all three fields. StateMembers_AreExactlyTheMaskedOperationalMembers
  failed with "'adoptedSourceFingerprint' is masked by the sync seams but missing from
  NodeTypeCompileState — the satellite would silently drop it". Exactly the paired-contract guard
  this repo builds, doing its job on a field I had added to one half.
• Seed is public surface, so this is an OVERLOAD, not a signature change — adding an eighth
  optional parameter is a binary break the compatibility gate is right to refuse. The new parameter
  carries no default, so existing seven-argument calls still bind unambiguously. Fallout: five (then
  six) bare <see cref="…Seed"/> became CS0419-ambiguous; all now carry explicit signatures, and a
  repo-wide grep confirms none is left.

Tests — all three rows mutation-proved

AdoptedBuildProvenanceTest, 6 cases. Reverting the three-way to the old unconditional stamp fails
5 of 6, and the messages name each row:

Expected value to be AdoptionRefused because the bundle states which sources it was built
  from and they are not the live ones …
Expected value to be AdoptedVerified,   but found Compiled.
Expected value to be AdoptedUnverified, but found Compiled.
Expected value to be AdoptedUnverified because no fingerprint means UNKNOWN provenance …

The one that still passes is the enum-default case, which the three-way does not decide — correct.

Two of the six are not row assertions and are the ones I would keep if the design changes:

• the legacy row asserts IsDirty == false — i.e. it pins that the stamp is KEPT. If someone
  later "tightens" this to withhold it, that test fails with the reason written out: every install
  recompiles everything, and on a RequirePrebuilt mesh every legacy-bundle type parks.
• AVerifiedAndAnUnverifiedAdoption_AgreeOnEveryOldSignal_AndDifferOnlyInProvenance — the
  reporter-side assertion the incident actually needed. Two nodes identical on CompilationStatus,
  IsDirty and CompiledSources must still be distinguishable. If provenance ever stops carrying
  that, this is what says so.

What this does NOT do — named, not buried

1. 🚨 Every adoption reads AdoptedUnverified until the bake stamps fingerprints. The refusal row
   is armed and correct but cannot fire yet. Nothing writes sourceFingerprint into a bundle today,
   so every adoption genuinely is unverifiable and a field saying so is TRUE — a uniform value here
   is the honest state, not a bug, and nobody should later read it as one. The producer half is
   node-repo-publish-bake.yml plus wherever the manifest's FrameworkMvid is written; the day a
   bundle carries a fingerprint the refusal row starts discriminating with no further change.
   Consumer-first is safe precisely because legacy = adopt+mark.

2. No recovery verb while a bundle resolves — that is Force-release is silently discarded: the compile watcher re-adopts the prebuilt bundle instead of compiling, so there is no verb that forces a build of the live source #2818, found while shaping this:
   RequestedReleaseForce is honoured by the release watcher and then silently discarded by the
   compile watcher, which re-adopts the same sealed bundle and settles "without a Roslyn pass".
   An operator who sees AdoptedUnverified can see the problem and still cannot force a rebuild —
   which is exactly where the Crm session is standing.

   I am NOT folding it into this change set, deliberately. The fix is ~10 lines (skip the
   on-demand adoption branch when the pending definition carries the force flag, go straight to
   DispatchOrPark()). Its regression is not: nothing in test/ registers an
   IPrebuiltAssemblyConsumer today, so proving "a forced trigger reaches Roslyn while an unforced
   one still settles by adoption" means standing one up on a Monolith mesh with real bundle bytes.
   Shipping a behaviour change to the compile watcher's hot path without that test is the exact move
   this repo has spent the evening catching. So: provenance lands first, the recovery verb is Force-release is silently discarded: the compile watcher re-adopts the prebuilt bundle instead of compiling, so there is no verb that forces a build of the live source #2818
   and it is still open, and I would rather it got a real test than a ride-along.

3. A dependency worth stating: the fingerprint's meaning rests on NodeSources.GetSources not
   projecting away content — which that helper already documents as load-bearing ("a projection
   omitting content would make the adapter emit NULL::jsonb … and the type would compile against
   empty sources with no error anywhere"). If that ever regresses, both sides would hash empty
   content and match. It would break compilation far more loudly first, but it is a real coupling.

Verification

Mesh.Contract / Graph / Hosting / PluginCatalog / Graph.Test / PluginCatalog.Test
                                        -c Release -warnaserror   0 Error(s)
dotnet test MeshWeaver.Graph.Test                                 1582 passed, 0 failed
mutation (three-way reverted)                                     5 of 6 failed
dotnet test MeshWeaver.PluginCatalog.Test                          (in flight as I post; its
                                                                   -warnaserror build is clean and
                                                                   my only change there is one cref)

Say the word and I will push it; I would rather #2818 went first, and I would like the Crm session to
say whether AdoptedUnverified on their nodes is enough to unblock the hold or whether they need the
producer half too.

🚨 One residual I did NOT decide unilaterally — please rule on it

On the refusal row I return def with { … }, not stamped — so CompiledSources is untouched
and the staleness question stays open. But Seed has already run by then, and it stamps the adopted
build's assembly coordinates (LatestAssemblyCollection / LatestAssemblyPath / LastCompiledVersion
/ LatestAssemblyMvid). Those I leave alone.

So between the refusal and the fresh compile completing, the rejected bytes are still what serves.
That is strictly better than today — today they serve forever with no signal, and now the node says
AdoptionRefused and a compile is running — but it is not nothing.

Clearing the coordinates on refusal would stop the stale assembly serving immediately. I did not do
it, because the consequence is availability, not correctness:

	serve stale until the compile lands (what I implemented)	clear coordinates on refusal
ordinary mesh	a short window of known-stale code, marked	the type serves NOTHING for the length of one compile
Modules:RequirePrebuilt mesh	same short window	the compile is refused by design ⇒ the type is left with no assembly at all, indefinitely

"Stop serving code we have proven is stale" is a defensible reading and may well be the right one for
a data-integrity fix — but on a require-prebuilt mesh it converts a marked-stale type into a dead one,
and that is a product call about which failure a client should get. Given #2813 is a data-loss issue I
lean toward clearing, but I am not making that trade on my own.

[rbuergi]

Resolving the open refusal-row question — and the answer moves the interlock to EXECUTE time

From the session that owns this incident, and it reframes the question I posed:

The damage needed two ingredients — stale bytes, and something armed to run them. Stale bytes sitting unloaded harm nobody. Refusing to execute matters more than refusing to load. A type that renders read-only pages from unverified bytes is a degraded system; a type that WRITES from them is the incident.

That is right, and it demotes the clear-or-keep question I had been treating as central. The first-order requirement is that BuildProvenance be readable by a control plane, so a watcher can refuse to act on an unverified type. A Crm/Migration-shaped node that fires on type registration would otherwise still fire — against exactly the assembly the loader refused to verify. That interlock, not any load-time decision, is what would have saved the four documents.

The load-time row still resolves as a conditional, now as the second line of defence:

the mesh can compile	outcome
yes	clear the adopted coordinates — the next activation builds, exposure is seconds
no (RequirePrebuilt)	keep, log Critical, and mark the type visibly unverified — clearing there trades a silent wrong answer for a certain outage, and an outage on a client's production mesh is not obviously the kinder failure

Making it conditional keeps the dangerous branch explicit rather than resting on an assumption about how a flag is set. 🚨 And it must not rest on "RequirePrebuilt is set nowhere" — I measured it absent from pod env, deployment inline env and ConfigMap on memex and memex-cloud, which is two instances and says nothing about pearl, atioz, local installs, or any external instance the registry serves.

A neighbouring signal that is NOISE — do not build on it

The same session has retracted (Crm#33) a staleness check they had shipped into their own AGENTS.md: "the Release is what instances load, so a mismatch means the portal is not running your change." Falsified by their next sync — Crm/Offer came back with all three fields mismatched:

NodeType   Crm_Offer/v63-s13fb7c2-…   fw s13fb7c25…
Release    Crm_Offer/v25-s770d75c-…   fw s770d75c0…

…and served perfectly: title, DRAFT pill, all 7,111 chars. latestReleasePath lags routinely — all fifteen of their types have stale release pointers and all fifteen are healthy.

So release-vs-NodeType disagreement is not what distinguished the broken type from the healthy ones, and a check of that shape would fire constantly on healthy meshes. The source-fingerprint-on-the-bundle comparison this fix uses is the right signal precisely because it does not have that property. Recording it here so the next reader does not reach for the cheaper-looking neighbour.

Sequencing

The incident owner is not waiting on this fix — their migration is complete, done by script, with every write verified by read-back — and they have said AdoptedUnverified everywhere is the honest state and preferable to a field that pretends. So the provenance change lands on its own merits, with #2818 (no operator verb forces a build of the live source) and the execute-time interlock both named as open follow-ups rather than folded in untested.

[rbuergi]

PR is up — #2821, CI running

Branch fix/2813-adoption-provenance, base main. It came up DIRTY on open (#2816 had landed on the same two files); both sides were pure additions at the same anchor, so the resolution keeps both — verified by EmitPipeline.cs and EmitDeadProcessIsNotAVerdictTest.cs showing zero diff against main, and by Graph.Test going 1583 → 1589 after the merge, which is #2816's six tests arriving intact.

Post-merge, locally: 5 projects clean at -c Release -warnaserror; Graph.Test 1589 passed, PluginCatalog.Test 698 passed, Documentation.Test 121 passed.

Holding it for the relay — the reviewing session asked to read the legacy path first, which is ALegacyBundle_StillAdoptsAndStamps_ButIsNeverRecordedAsVerified plus the if (def.AdoptedSourceFingerprint is not { Length: > 0 }) branch in ApplyAdoptedSourceStamp. That test asserts IsDirty == false on purpose: it is the tripwire for anyone later "tightening" the legacy row, and it fails with the parking consequence written out in the message.

The two things I most want challenged there:

1. The legacy row keeps the stamp, so a legacy bundle still reads IsDirty == false — i.e. this change does not make unverified adoptions re-compile. It only makes them say they were never checked. If the hold on the Crm space needs them to actually rebuild, this change alone does not do that and Force-release is silently discarded: the compile watcher re-adopts the prebuilt bundle instead of compiling, so there is no verb that forces a build of the live source #2818 is the blocker, not this.
2. The refusal row is armed but idle until a bake records a fingerprint, so nothing in the fleet changes behaviour today except the new buildProvenance field reading AdoptedUnverified everywhere an assembly was adopted.

[rbuergi]

The recovery verb is up: #2824 (fixes #2818) — a FORCED release now skips on-demand prebuilt adoption in the compile watcher and compiles the live source, and the force is spent by that compile. The provenance half (verify/refuse an adoption against the source) stays with #2821. Found alongside in #2824: a type that already had a build could be stranded at Pending by an adoption reported but never written back — the landed-predicate now requires the Ok stamp a real Seed writes.

[rbuergi]

Reopening — #2821 merging did not fix the defect in this issue's title

#2821 (a6c5e87aa) closed this automatically via its closing keyword. That is wrong, and the PR itself says why in its own body: it changes what an adopted build says, not what it does.

This issue's headline is "GitSync adopts a stale prebuilt assembly over the source it just synced — silently running last week's code." Check it against what actually shipped:

	before #2821	after #2821
can a sync adopt a bundle built from older source?	yes	yes
does it then run that older code?	yes	yes
does the node claim compiledSources == currentSourceVersions?	yes	yes (legacy row keeps the stamp — deliberately, or every legacy type parks on a RequirePrebuilt mesh)
can an operator tell the bytes were never checked?	no	yes — BuildProvenance = AdoptedUnverified

Only the last row changed. And because nothing writes a sourceFingerprint into a bundle yet, every adoption in the fleet reads AdoptedUnverified — the refusal row is armed and idle. That is the honest state and a real improvement, but it is visibility, not prevention.

What still has to land before this issue is genuinely closed:

• Force-release is silently discarded: the compile watcher re-adopts the prebuilt bundle instead of compiling, so there is no verb that forces a build of the live source #2818 — the compile watcher discards RequestedReleaseForce and re-adopts without a Roslyn pass, so there is still no operator verb that forces a build of the live source on any mesh where the bundle resolves. That is the recovery path this incident needed and did not have.
• A control plane must refuse to EXECUTE against a type whose build provenance is not verified — refusing to load is the second line of defence #2820 — the execute-time interlock. The damage here needed two ingredients: stale bytes and something armed to run them. Until a control plane refuses to act on an AdoptedUnverified type, a Crm/Migration-shaped node still fires against exactly the assembly nobody verified.
• the producer half — a bake that records the source fingerprint. Without it the middle row of the three-way can never fire.

Reopening rather than leaving a closed issue that reads as "the data loss is handled". The document that was permanently lost is still the measure of this one, and nothing shipped today would have prevented it — it would only have made it visible to an operator who thought to look.