🚨 Every rollout since 3.0.0-ci.8059 stalls at the bake gate: #3628's MeshNodeContentDegradedException fires during the NodeType bake for instance nodes typed by their own NodeType's source (4 Doc types 'regressed')

[rbuergi]

What happened

Rolling the fleet to 3.0.0-ci.8059 (core a57a6b5 × plugins a7abdb2c) on 2026-09-08 00:22–00:50Z: on memex.systemorph.com the new pod (memex-portal-deployment-7d5d458cc4-cbztk) never passes its startup probe — /health answers 503 from nodetype_bake:

Health check nodetype_bake with status Unhealthy … NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 4 NodeType(s) regressed on this image: Doc/Architecture/BusinessRules/Cession, Doc/DataMesh/PythonPandasNode/PandasExplorer, Doc/DataMesh/SocialMedia/Post, Doc/DataMesh/SocialMedia/Profile

The same pod's log names the cause for each of the four — one instance node per type, read during the bake:

MeshWeaver.Mesh.MeshNodeContentDegradedException: MeshNodeStreamCache.GetStream: content for 'Doc/Architecture/BusinessRules/Cession/MotorXL' (nodeType 'Doc/Architecture/BusinessRules/Cession') stayed an untyped JsonElement — the $type discriminator resolved to no registered CLR type on the reading hub and the mesh-wide content-type registry did not recover it …
  … 'Doc/DataMesh/PythonPandasNode/PandasExplorer/LiveFrame' ($type PandasExplorer)
  … 'Doc/DataMesh/SocialMedia/Post/Post-001' ($type SocialMediaPost)
  … 'Doc/DataMesh/SocialMedia/Profile/Roland-LinkedIn' ($type SocialMediaProfile)

The gate did its job: the old 8055 pods keep serving, the deploy run (Memex 34173391343) reported the stalled rollout red at its 25-minute budget. Nothing broken is serving. But the release is un-rollable.

Root cause

8055 (core 7bfe2d9) rolled clean on the same portal 2 h earlier (Job 36 complete, 3/3 ready). The only core change between the sets is #3628 (7bfe2d907..a57a6b52b, "the untyped-content gate can fire"), which introduced MeshNodeContentDegradedException and throws it from MeshNodeStreamCache.ConvertContentJsonElementToTyped when content stays a JsonElement. The four regressed types are exactly the doc types whose instance nodes carry a $type declared in the type's OWN Source/ (CessionData, PandasExplorer, SocialMediaPost, SocialMediaProfile): during the bake, the instance is read on a hub where that type is not yet registered (the type is what the bake is compiling), the content is legitimately untyped for that moment, and what used to be a silent degradation is now a throw inside the bake — which the sweep scores as a regression. LateContentTypeRegistrationTest (added by #3628) was meant to cover the late-registration case; the bake path is a case it does not model.

Both portals show it: memex-cloud's 8059 pod logs the same exception for Posts/ClaudeAbstractsErrors (SocialMedia/Post), plus CS0246: 'CessionData' could not be found on Doc/Architecture/BusinessRules/Cession — the same type, failing to compile its own source in the same window.

What needs to happen

The throw must not fire for content whose type is registered by the NodeType being baked — the bake must read those instances after their type compiled, or the conversion must degrade (as before) rather than throw while that type's compilation is in flight. Until then every set ≥ a57a6b5 stalls every rollout at the bake gate; the fleet stays on 8055 (whose migration image lacks Plugins#1488's budget — the databases are already migrated to v55 by the 8059 Job, so 8055 pods serve fine).

Evidence: Memex runs 34173111536 (memex-cloud) and 34173391343 (memex); pods memex-portal-deployment-7d5d458cc4-cbztk (memex) and memex-portal-deployment-8b96fdbd4-h7cxj (memex-cloud).

[rbuergi]

🚨 The stated mechanism is falsified — there is no throw. Measured on the stalled pod, which is still live.

This issue says the degradation "is now a throw inside the bake". MeshNodeContentDegradedException is never thrown anywhere in src/. On origin/main there are exactly three construction sites, all in MeshNodeStreamCache.cs (lines 2295, 2735, 2758) and all three are passed as the exception ARGUMENT to logger.LogWarning(...). grep -rn "throw new MeshNodeContentDegradedException" src/ returns nothing. A logged exception renders in the pod log with the same Namespace.Type: message prefix as a thrown one, which is what makes the two indistinguishable by eye.

What the stalled pod actually reports

memex-cloud/memex-portal-deployment-8b96fdbd4-h7cxj (8059) is still running and still not ready, so this is measured, not reconstructed:

DynamicTypePreWarmer: warm-up complete in 00:03:31.9
  compiled=329  alreadyBaked=14  compileErrors=4  timedOut=0  skipped=0  contentBroken=7  faulted=0

compileErrors=4 — the four gating types are PreWarmStatus.CompileError, i.e. genuine compile failures. contentBroken=7 is the already-carved-out non-gating class. The four errors are types failing to resolve names declared in their own Source/:

CS0246 The type or namespace name 'SocialMediaProfile' could not be found
CS0246 The type or namespace name 'Platform' could not be found
CS0103 The name 'Platform' does not exist in the current context
CS1061 'LayoutDefinition' does not contain a definition for 'AddSocialMediaProfileLayoutAreas'
CS0246 The type or namespace name 'OperationRequestContent' could not be found      (rbuergi/OperationRequest)
CS1061 'MessageHubConfiguration' does not contain a definition for 'AddOperationRequestControlPlane'

And the degradation exception that IS present in the log names a different node entirely — content for 'Hosting' (nodeType 'Store/Plugin'), $type: PluginContent — logged at Warning, not gating anything.

So the causal claim needs re-deriving, and I am not asserting a replacement

What is established: the gate fires on CompileError, and those four types cannot see their own declared types. What is NOT established is that #3628 caused it. A plausible chain worth testing FIRST, because it would explain both symptoms at once:

a NodeType's definition content degrades to an untyped JsonElement → ContentAs<NodeTypeDefinition>() answers null → its Sources read as empty → the compile runs against an incomplete source set → CS0246 on the type's own types.

That is a content→compile chain, not a throw, and it would make the degradation the upstream of the compile errors rather than the gating event itself. It is a hypothesis. It should be confirmed or killed before any code changes.

🚨 Do not "fix the throw" — there is nothing to fix there, and a change built on that reading would be inert at best. Likewise do not revert #3628 on this evidence: it closed a real defect (#3625, the gate could not fire at all for five days, verified), and reverting on a falsified mechanism trades a rollout stall for a silent blind spot.

Control arm not yet available

The serving 8055 pod started 2026-09-07T22:20:14Z and its warm-up summary has already rolled out of the retained log, so I could not read compileErrors= for the set that rolled clean. That comparison is the decisive measurement — if 8055 also had these four CompileErrors, then the gate's WasHealthyBeforeBake logic (a type already broken must not gate) is where the behaviour changed, and #3628 is not implicated at all. Get it from the 8055 pod before it restarts, or from the deploy run's archived logs.

Related: NodeTypeBakeGate already distinguishes a CONTENT verdict from an IMAGE verdict (NoSources, UpstreamContentBroken, TimedOut, UpstreamUnevaluated all deliberately do not gate, on the stated principle that "no image can change what a mesh query matches"). If these four are content drift rather than an image regression, that existing carve-out — not a new one — is probably where they belong.

[rbuergi]

Attribution CONFIRMED, mechanism still wrong — and the mesh's own record contradicts the pod

The attribution in this issue is right. git log 7bfe2d907..a57a6b52b is exactly 8 commits, all of them #3628 — nothing else is between 8055 and 8059. So #3628 is the only candidate, as stated. (My previous comment falsified the mechanism, not the attribution; both matter and they are separate claims.)

🚨 #3628 changed more than logging, and the extra change is the one worth looking at. Commit 344b743ad — "the gate's first real catch — every instance's module inventory was unreadable" — is a live src/ fix, not test scaffolding:

• registered DeploymentReport in AddPluginCatalogTypes (it was registered nowhere, and that config is applied to both the mesh hub and the default node hub);
• changed DeploymentReportService.InventoryContentType from the hand-typed literal "ModuleInventoryContent" — a name no CLR type in the fleet carries — to nameof(DeploymentReport).

That is a type-registration change on the hubs that also serve NodeType compilation, shipped in the same set that broke the bake. It is a far better suspect than a throw that does not exist, and it is where I would look first.

The mesh's own compilation record disagrees with the pod

Swept the live mesh (nodeType:NodeType content.compilationStatus:Error, one non-truncated call, count: 2):

type	recorded	lastModified
LinkedIn/Skill	Error	2026-08-21
rbuergi/OperationRequest	Error	2026-09-06

Two facts follow:

1. rbuergi/OperationRequest was ALREADY broken before 8055 — its Error state predates both sets by a day. Its CS0246 OperationRequestContent on the 8059 pod is therefore not a regression, and per this gate's own rule (a type already broken must not gate) it must not be among the four holding readiness. Worth confirming it is being excluded correctly.
2. The other three named types — Doc/Architecture/BusinessRules/Cession, Doc/DataMesh/PythonPandasNode/PandasExplorer, Doc/DataMesh/SocialMedia/{Post,Profile} — are NOT recorded as Error in the mesh at all. So the 8059 pod fails to compile types the mesh considers healthy, which is the genuine regression signal and is exactly what the gate is for.

What is still missing, and it is the decisive measurement

compiled=329 alreadyBaked=14 compileErrors=4 timedOut=0 skipped=0 contentBroken=7 faulted=0 is the 8059 number. I could not obtain the same line for 8055 — that pod started 2026-09-07T22:20:14Z and its warm-up summary has already rolled out of the retained log, and I did not restart it because it is the only thing serving.

Get compileErrors= for 8055 from the deploy run's archived logs (Memex run 34173111536 / 34173391343) before changing code. If 8055 also reported four, the change is in what the gate does with them, not in the compile.

[rbuergi]

The search space is now ONE line of src/, and the control arm is the mesh itself

#3628's entire src/ footprint is two files and 24 added lines. Everything else in those 8 commits is tests, a gate script, and docs:

src/MeshWeaver.PluginCatalog/DeploymentReportService.cs          13 ++++++++++++-
src/MeshWeaver.PluginCatalog/PluginCatalogConfigurationExtensions.cs  11 ++++++++++-

and the functional change in the second is a single statement appended to AddPluginCatalogTypes:

.WithType(typeof(RegistryReconcileLedger), nameof(RegistryReconcileLedger))
+ .WithType(typeof(DeploymentReport), nameof(DeploymentReport));

plus InventoryContentType moving from the literal "ModuleInventoryContent" to nameof(DeploymentReport).

So whatever broke the bake is downstream of registering one more type on a surface applied to BOTH the mesh hub and the default node hub. That is the whole candidate set. There is no throw, and there is nothing else in src/ to suspect.

The control arm exists after all — it is the mesh's own record

I could not read compileErrors= for 8055 (that pod's warm-up has rolled out of its retained log, and I will not restart the only thing serving; the deploy run's log observes the rollout and carries no pod output). But the mesh's per-NodeType compilationStatus is the 8055 verdict — the serving pod wrote it, and per AGENTS.md the readiness gate reads the same field, so the two agree by construction rather than by coincidence.

type	mesh record (written by 8055)	8059 pod
Doc/Architecture/BusinessRules/Cession	not Error	CompileError
Doc/DataMesh/PythonPandasNode/PandasExplorer	not Error	CompileError
Doc/DataMesh/SocialMedia/Post · /Profile	not Error	CompileError
rbuergi/OperationRequest	Error since 2026-09-06	CompileError — pre-existing, must not gate
LinkedIn/Skill	Error	—

Three types that compile on 8055 do not compile on 8059, and the only difference between the images is the one registration above. That is a real image regression, and #3628 owns it — the attribution in this issue is correct even though its mechanism is not.

Suggested next step, for whoever takes it

Reproduce locally against a mesh holding one of the three (SocialMediaProfile is the cheapest — its failure is CS0246 on a type declared in its own Source/), with and without that single .WithType(...) line. That is a one-line revert and a two-arm falsification, and it will either name the mechanism immediately or clear the line and move the search to DeploymentReportService.

🚨 Do not ship the one-line revert as the fix without understanding it: the registration cures a real defect (Ops/Modules/{deployment} unreadable on every instance), and trading a rollout stall for that is a swap, not a repair.

[rbuergi]

Closing the obvious objection to the comparison above: it is the SAME portal, verified.

A mesh-vs-pod comparison is worthless if the two are different deployments, and the MCP servers here are named after deployments rather than after which portal they hit. Checked rather than assumed:

server 'memex'       -> https://memex.meshweaver.cloud
server 'systemorph'  -> https://memex.systemorph.com
server 'memexlocal'  -> https://memex.localhost

The compilationStatus sweep ran on memex, i.e. memex.meshweaver.cloud — the same portal whose stalled pod (memex-cloud/memex-portal-deployment-8b96fdbd4-h7cxj, 8059) I read the compileErrors=4 line from, and whose serving pod (8055) wrote those records. Same mesh, same deployment, one image apart.

So the finding stands: three types the serving image compiles, the new image does not, with a two-file, 24-line src/ delta between them.

Also worth stating plainly, because it cuts against my own suspect: neither src/ change is an obvious mechanism. They are (1) one .WithType(typeof(DeploymentReport), nameof(DeploymentReport)) appended to AddPluginCatalogTypes, and (2) a const string moving from "ModuleInventoryContent" to nameof(DeploymentReport). Neither adds an assembly or touches the compile reference set, and a NodeType's references are the whole TPA. So the mechanism is likely INDIRECT — something reads that registry or that constant during the bake — and the two-arm local repro is what will name it. I have not been able to run that repro (the doc gate needs a staged tree plus the tester image), so I am handing over a narrowed search, not a diagnosis.

[rbuergi]

Third independent sighting — and it names the SAME THREE TYPES, from a different repo, on a different instrument

MeshWeaver.Education#290 pins that repo to 3.0.0-ci.8059 (lane refs on a57a6b52b) and its
disposable-mesh install gate fails. The portal container's own boot line, from run
34179258418, shard 3/4:

DynamicTypePreWarmer: 3 dynamic NodeType(s) were NOT covered by the prebuilt bundles and will each
pay a Roslyn compile on FIRST ACCESS:
  Doc/Architecture/BusinessRules/Cession        (NeverBuilt: no assembly recorded),
  Doc/DataMesh/PythonPandasNode/PandasExplorer  (NeverBuilt: no assembly recorded),
  Doc/DataMesh/SocialMedia/Post                 (NeverBuilt: no assembly recorded)

Three for three with the types this issue names, reached from an entirely different direction —
a disposable mesh in another repository's CI, not an AKS rollout.

What this adds beyond corroboration

NeverBuilt: no assembly recorded is a bake fact, not a runtime one. The same boot reports
ShippedPrebuiltBundles: 17 prebuilt assembly(ies) from 40 shipped bundle(s) … 0 adopted now, 17 already current. So on the 8059 set the bake produced no assembly at all for these three, and
every consumer then has to compile them on first access — which is what the install test trips over.

That closes a gap in my earlier comments. I had the symptom on the portal (compileErrors=4, the
readiness gate refusing) and the mesh's contrary record (these types not marked Error). This shows
the missing middle: the bake output for those three is simply absent on 8059. The failure is
therefore upstream of readiness — it is in what the bake emits, not only in how the gate scores it.

Blast radius is wider than the two portals

This is no longer "rollouts stall". Any repo that pins the 8059 set inherits it:

• both AKS portals — readiness refused, fleet held on 8055;
• MeshWeaver.Education — disposable-mesh install gate red on a pin-move PR whose diff is only
  pin values;
• and by construction any satellite that moves its pin to this set next.

🚨 Practical consequence, worth acting on before the root cause is found: do not move further pins
onto 8059. Education#290 is a pin move onto a set with a known unresolved defect; it will stay red
for a reason that has nothing to do with its diff. Same caution for the Crm and Reinsurance lane
moves that are staged.

[rbuergi]

🚨 Correcting myself: the bake did NOT fail. It compiled all four types successfully.

In an earlier comment I wrote that "on 8059 the bake emits no assembly at all for those three". That
is wrong, and I am retracting it. I inferred it from the Education portal's
NeverBuilt: no assembly recorded, which is a statement about what that instance found, not about
what the bake produced. Reading the bake's own log settles it the other way.

The 8059 CD bake (run 34167788582, job 101885370459) compiled every one of them:

Doc/Architecture/BusinessRules/Cession [compile:Passed render:Passed tests:Skipped]
Cession: compile=Ok render=ok tests=skipped
compile: 4/4 NodeType(s) compiled, 1 bundle(s), framework=sb43f9287dbd6922a7937bd24be103937
         — 0 known-debt failure(s) allowed, 0 new, 0 stale allow entr(ies)
seed: adopted 4 of 4 baked assembly(ies) for the 4 installed NodeType(s)
bake published: identity=sb43f9287… source-sha=a57a6b52b… bundles=1 targets-published=2 release=3.0.0-ci.8059

Zero compile errors in the whole job. The assemblies exist and were published.

So the failure is ADOPTION, not compilation — a different defect class entirely

Three readings that now have to be held together:

where	what it says
8059 CD bake	4/4 compiled, published under identity sb43f9287…
8059 portal pod	compileErrors=4 — it is recompiling them, and failing
Education disposable mesh	"3 dynamic NodeType(s) were NOT covered by the prebuilt bundles", NeverBuilt

A consumer only recompiles a NodeType when it has not adopted a prebuilt assembly for it. So the
assemblies were built and published, and the consumers are not taking them. That is the
bake-identity / adoption family (Doc/Architecture/BakeIdentityMismatch — "why a green CD can
publish a bake no portal adopts"), not a compile regression.

It also explains the shape that puzzled me: the mesh records these types as healthy (they are —
they compiled at bake time) while the pod reports them broken (it never received the bytes and its
own recompile fails).

A second oddity, worth someone's attention

The run I identified as 8055's CD (34166682137, head 7bfe2d907) did not bake. Its bake job
found the work already done and skipped:

holds a COMPLETE publication of THIS content … (sentinel present, source a57a6b52b…) — already published; skipping
bake published: identity=sb43f9287… source-sha=a57a6b52b… targets-published=0 release=3.0.0-ci.8058

Note source-sha=a57a6b52b and release=3.0.0-ci.8058 on a run whose head is 7bfe2d907. Both
sets share ONE content publication, made from a57a6b52b under one identity. Whether that
sharing is correct by design (same content identity ⇒ one publication) or is itself part of the
problem, I have not established — but it means "8055's content" and "8059's content" are the same
bytes, which is worth knowing before anyone reasons about the difference between the two sets.

What this changes for the fix

The one-line-revert experiment I proposed earlier is now the wrong first step: there is no
compile failure at bake time to revert. The question to answer first is why a consumer on
a57a6b52b declines a bundle published under sb43f9287… — compare the identity the 8059 portal
image computes at runtime against the identity the bundle carries. If they differ, that is the bug
and #3628 changing src/ is a plausible cause of the identity moving.

My apologies for the earlier misdirection; the corrected reading is above and the evidence is in the
two job logs named.

[rbuergi]

A clue that sharpens this: both sets' bakes carry the SAME framework identity

Measured from the two CD bake jobs:

8059 bake (run 34167788582)  identity=sb43f9287dbd6922a7937bd24be103937
8055 run  (run 34166682137)  identity=sb43f9287dbd6922a7937bd24be103937

Identical. And the 8055 run did not bake at all — it found the work already done
(already published; skipping, targets-published=0), so both sets share ONE content publication
under that single identity, produced from source-sha=a57a6b52b.

That is worth stating because it removes the most obvious explanation. If the bundle's identity were
different between the sets, "the 8059 portal declines a bundle sealed under another identity" would
be the whole story. It is not: there is only one bundle, one identity, and 8055 adopts it while
8059 does not.

So the difference has to be on the consumer side — what the 8059 portal image computes as its own
framework identity at runtime, or what its adoption path does with a bundle whose identity it
otherwise accepts. Both images were built from the same content publication; only the image differs,
and the image differs by exactly #3628's two files.

Concrete next measurement, which needs a running pod and so is beyond what I can do from here:
on the stalled 8059 pod, read the identity it computes and compare it to sb43f9287…. The boot log
already prints an adoption summary of the form
ShippedPrebuiltBundles: N prebuilt assembly(ies) from M shipped bundle(s) … adopted / skipped, and
the Education instance's version of that line reported 0 adopted now, 17 already current while
three types were NeverBuilt. Whether the 8059 portal's line shows a wholesale decline (identity
mismatch) or a partial one (three types genuinely absent from the bundle) discriminates the two
remaining hypotheses in one reading.

Recording this rather than acting on it: I have now been wrong twice on this issue (the "throw", then
"the bake emits nothing"), and this is a clue rather than a conclusion.

[rbuergi]

🚨 Correction — 8059 DOES roll. memex is now serving it, two ready pods.

Measured 2026-09-08T04:25Z on memex:

memex-portal-deployment-7d5d458cc4-6fq52   ready=true   memex-portal-ai:3.0.0-ci.8059
memex-portal-deployment-7d5d458cc4-cbztk   ready=true   memex-portal-ai:3.0.0-ci.8059

…-cbztk is the same pod this issue opens with — the one that "never passes its startup probe".
It passed. Both portals serve HTTP 200.

So my repeated characterisation of 8059 as "the broken set" was wrong, and I want that on the
record more clearly than the individual corrections above. I measured a rollout that had not yet
succeeded and generalised it into a property of the set. That is the second time on this issue I
turned a snapshot into a conclusion — the first being "the bake emits no assembly", retracted when
the bake log showed 4/4 NodeType(s) compiled. A reading of a system mid-transition describes the
transition, not the destination, and I did not hold that line.

What is now known, and what is still open

Not in dispute: the stall was real at the time. The gate refused readiness, the deploy run
reported it at its 25-minute budget, MeshWeaver.Education#290 reproduced it independently, and the
fleet genuinely sat on 8055 for hours. Whatever cleared it — a later boot, a peer fix such as #3676
(the module-set reader opens only the deciding records, merged tonight and explicitly about a
stuck rollout), or simply a retry after the bake settled — is not established by this comment,
and someone should establish it before closing.

🚨 Do not read this as "#3663 is fixed." A rollout that stalls for hours and then succeeds is a
different defect from one that never succeeds, and it is the more dangerous shape: it will present
as a slow deploy rather than a failure, and the next occurrence will look like flakiness. The
question to answer is why the first boots refused and the later ones did not, on an image that
did not change.

Corrected standing summary of this issue:

claim	status
the degradation "throws"	false — all three sites LogWarning, no throw in src/
the bake emits no assembly	false, retracted — compile: 4/4 NodeType(s) compiled
both sets share one identity sb43f9287…	holds — and the 8055 run did not bake at all
8059 is undeployable	false — it is serving on memex now
the stall happened, and its cause is unexplained	holds — this is what remains

I have also withdrawn the advice I gave elsewhere on the strength of the wrong reading: the "do not
move further pins onto 8059" caution on MeshWeaver.Plugins#1464 and MeshWeaver.Education#290 was
based on it, and MeshWeaver.Plugins#1502 pinning to 8059 was the right call rather than the risky
one.

[rbuergi]

ROOT CAUSE FOUND, with the denominator. The stall was a SHORT SOURCE-DISCOVERY READ, and the three hours were the startup probe's budget.

Measured 2026-09-08 08:05–08:15Z, read-only, against Loki and the live cluster. Nothing rolled, restarted or patched.

The one measurement that settles it

BatchBake logs the size of every source-discovery pass. Every boot on memex (memex.systemorph.com — the namespace this issue opens with) in the window 09-07 22:40Z → 09-08 07:37Z:

boot (UTC)	Code nodes resolved	compileErrors=	readiness
09-07 22:40:02	1236	2	granted
09-07 22:43:42	1236	2	granted
09-08 00:31:22	1145	6	REFUSED
09-08 03:32:37	1237	2	granted
09-08 03:35:30	1237	2	granted
09-08 07:34:44	1241	2	granted
09-08 07:36:57	1241	2	granted

{namespace=~"memex|memex-cloud"} |= "source discovery resolved"
{namespace=~"memex|memex-cloud"} |= "warm-up complete"

Denominator: 29 boots across both portals in 15 hours; exactly ONE refused. Three of the six boots that granted readiness ran the same image as the one that refused. 2 is that portal's standing baseline of already-broken types (which MarkOutcome correctly refuses to let gate). The four extra are precisely the four types in the issue title — and they are 4 of 4 NodeTypes in the Doc partition, not 4 of 40.

memex-cloud never shows this at all: 6837 → 6841 Code nodes, compileErrors= steady at 4–5 both before and after 8059. Its stall was the other defect — the 10-second pending_module_activation health check over 687 module-set records, which is #3676. Two different stalls were conflated into this issue from the start.

The mechanism

The Doc types use the default source queries. On the short pass their sets resolved to zero, Roslyn was handed the generated provider file alone, and it emitted a perfectly honest CS0246 'CessionData' could not be found about a type declared in that very NodeType's own Source/. The gate then did exactly its job on a genuine CompileError.

The invariant that exists to stop this — NodeTypeBatchBake.Assemble → SourceDiscoveryFailedException, written for #1216, whose doc comment even quotes the identical CS0246 'ScopeLibrary' shape — did not fire, because it read:

if (matched.Count == 0 && pending.DeclaresSources && !pending.SourcesKnownDeleted)

DeclaresSources is false for a type that uses the defaults, i.e. for very nearly the whole population — the same mistake DynamicTypePreWarmer.ClassifyCompileFailure had already had to be corrected for in #1391 ("an empty Sources does not mean configuration-only — it means uses the DEFAULT queries"), sitting here in the opposite direction. Verified on the live node: Doc/DataMesh/SocialMedia/Profile has no sources and a populated currentSourceVersions with its 2 files. The mesh's own record said "this type has 2 source files" while the pass matched none — an available, unused witness to a contradiction.

Why three hours, and why that is the dangerous shape

RetractRegression is level-triggered on the type recompiling, and nothing recompiled — the content never changed, so the park registry's source-change retry never fired. What ended it was the kubelet: startupProbe is failureThreshold: 1080 × periodSeconds: 10 = exactly 3 h, and the container died at 03:29:52Z having started at 00:30:23Z (2 h 59 m 29 s). The restart's discovery pass came back complete and the pod went Ready immediately.

#3676 did NOT clear this — falsified, not asserted

The recovery is at 03:33Z; #3676 merged at 04:20:48Z, 47 minutes later, and the pod was on 3.0.0-ci.8059, which by construction cannot contain a commit merged after it was built. Had #3676 been the cause, the recovery would have had to postdate that merge and arrive on a later image. It did neither. (#3676 is a real fix — for memex-cloud's separate startup-probe stall.)

One more seam, recorded but not fixed here

The same boot logged ShippedPrebuiltBundles: bundle Doc.zip: adopted 4/4 prebuilt assembly(ies) at 00:31:08, and ten seconds later the sweep enumerated 204 of 209 … need building — 5 already on the share and recompiled them anyway. Adoption seeding and the sweep's store probe disagree on a cold boot; that is what put these four on the compile path at all. Separate issue's worth of work — it costs time, not correctness.

Fix

PR incoming: the invariant now consults the positive witness (CurrentSourceVersions.Count > 0 ⇒ a zero match contradicts the record ⇒ unestablished ⇒ abandon the batch for the activation-driven sweep). All three shapes of the snapshot stay distinguished — explicitly empty still classifies as NoSources and still must not gate (#1204), absent still falls back to the old rule. Guard test with a control arm: against the pre-fix predicate 1 of its 4 cases fails and the other 3 pin unchanged semantics on both. Doc: Doc/Architecture/SourceSetEstablishment, which carries the table above and the two Loki queries to re-measure with.

No bound was widened, no retry added, and the gate's Regressed rule is untouched: the gate was right, the input was wrong, and the detector that should have rejected the input was switched off for most of the population.

[rbuergi]

The denominator, with its positive control

Same instrument, same 15-hour window (09-07 18:00Z → 09-08 09:00Z), counted over the whole window on Loki's instant endpoint so the range-vector over-count trap does not apply:

count_over_time({namespace="memex"}       |= "NodeType bake regressed" [15h])   →  1065, ONE stream
count_over_time({namespace="memex-cloud"} |= "NodeType bake regressed" [15h])   →     0

All 1065 come from memex-portal-deployment-7d5d458cc4-cbztk boot 0 and nowhere else — one line per readiness probe at the 10-second cadence for 2 h 58 m, which is the startup-probe budget. The non-zero arm is what makes the zero arm mean something: across two portals in fifteen hours, exactly one container boot ever put the bake gate into Regressed, and memex-cloud's simultaneous stall did not involve this gate at all. Two different stalls, one issue.

Fix + doc: #3698.

[rbuergi]

Closing — fixed on main

#3698 merged through the queue at 08:59Z (3ee8dda6e). Verified on origin/main by both ancestry and symbol presence, not by the merge event alone: git merge-base --is-ancestor 3ee8dda6e origin/main → 0, and origin/main:src/MeshWeaver.Hosting/NodeTypeBatchBake.cs carries DiscoveryUnestablished. The merge queue does not fire Fixes # links, which is why this is being closed by hand.

Corrected standing summary — replacing the table in the 04:25Z comment:

claim	status
the degradation "throws"	false — all three sites LogWarning
the bake emits no assembly	false, retracted — compile: 4/4 NodeType(s) compiled
both sets share one identity sb43f9287…	holds, and it was never the difference
8059 is undeployable	false — both portals have served it
the stall happened and its cause is unexplained	RESOLVED — one short source-discovery pass (1145 Code nodes vs 1236/1237/1241 on the same portal), an invariant that could not see it because the four types use the DEFAULT source queries, and a three-hour window that is the startup probe's budget, not anything that understands the defect
#3676 cleared it	false — recovery at 03:33Z, 47 min before #3676 merged, on an image that cannot contain it
memex-cloud had the same defect	false — count_over_time({namespace="memex-cloud"} |= "NodeType bake regressed" [15h]) = 0 against 1065 on memex from one stream. Two stalls, one issue; memex-cloud's was #3676's 10-second health check

Fleet state now. Both portals sweep clean of these types — memex.systemorph.com has 2 NodeTypes at Error (Crm/Migration, Edu/Course), memex.meshweaver.cloud has 2 (LinkedIn/Skill, rbuergi/OperationRequest), all four pre-existing and none of them a Doc/** type. Those pairs ARE the compileErrors=2 / compileErrors=4 baselines seen in every boot log.

Two things deliberately left open, neither of which this fix needs:

1. Why the pass was short. Not measured. Leading candidate is RunQuery's one-second quiet window — QueryResultChange<T> carries no terminal marker, so a chunk gap wider than it truncates the fold silently, and this pass ran 14 s after a 35-second bundle-seeding burst. Named in Doc/Architecture/SourceSetEstablishment with the measurement that would settle it (per-query change-event count and largest inter-chunk gap, short pass vs complete pass on the same portal). 🚨 Widening the window is explicitly the wrong answer — the invariant makes rarity irrelevant.
2. Adoption vs the sweep's store probe disagree on a cold boot. Doc.zip: adopted 4/4 prebuilt assembly(ies) at 00:31:08, then 204 of 209 … need building — 5 already on the share at 00:31:18. That is what put these four on the compile path at all. It costs time, not correctness. Worth its own issue if anyone wants the cold-boot minutes back.

[rbuergi]

The two open halves this issue named are now filed, so the chain is followable from either end:

• Unexplained: one boot's batched source discovery came back 91 Code nodes short — RunQuery completes on a 1-second quiet window and QueryResultChange<T> has no terminal marker #3704 — why the discovery pass was short. The actual unexplained fact. Carries the 1145-vs-1236/1237/1241 table, the candidate (RunQuery's 1-second quiet window; QueryResultChange<T> has no terminal marker), the discriminating measurement (per-query change-event count + largest inter-chunk gap, short pass vs complete pass on the same portal), and an explicit ruling-out of widening the window with the three reasons.
• Cold boot: 78 prebuilt assemblies adopted, the sweep's store probe counts 5 ten seconds later — 197 compiles instead of ~20, and the upstream half of #3663 #3703 — the upstream half. Doc.zip: adopted 4/4 at 00:31:08, 5 already on the share at 00:31:18, and the control that makes it a disagreement rather than a cold-store fact (a boot with 77 already current sees 189 baked; a boot with 78 adopted now sees 5). This is what put those four types on the compile path at all, so it is what made the short pass reachable.

#3698 fixed what a short pass may CONCLUDE. Neither of the above is fixed.