Store cover shows the persisted 'Compiling…' placeholder forever: re-activated instance hubs replay the stored area doc instead of re-deriving from compilationStatus

[rbuergi]

Live on memex prod, 2026-08-26 ~17:30Z: 10 of 12 store covers render the Roslyn compile-progress screen to anonymous visitors, more than an hour after the compile succeeded. Found by plugins-ff; diagnosed live via MCP.

Evidence

• Store/Plugin (NodeType): compilationStatus: Ok, compile 16:26:31→16:26:43Z (12 s), lastCompiledVersion 12379, release 20260826162643-U8NsdxXf. The type is healthy.
• get @Chess/area/Cover returns the compile-progress control tree — ### ⏳ Compiling… — Plugin, "Running Roslyn against 9 source binding(s)", and a LayoutAreaControl bound to activity Store/Plugin/_Activity/compile-2026082616091603… — i.e. the 16:09 compile, not even the last one. The same store's data.progress = 100: the datum updated, the controls never re-rendered.
• Recycling does not heal it. Recycled @Store/Plugin (type), then @Chess (instance), then @Chess again after the type settled — the freshly initialized hub re-serves the identical placeholder tree every time.
• Stuck (10): AdvancedBusinessRules, ThinkInStreams, DataModeling, AgenticPrimer, RiskTransfer, Chess, Voice, Teams, Publish, Edu. Fine (2): AgenticEngineering, AgenticBusiness.
• Portal pods were restarted 17:10Z (module convergence) — the stuck view survives a full pod restart too.

Read

The placeholder was written into the persisted layout-area store when the 16:09 compile started. Completion (16:26) re-renders only instances whose hubs are ACTIVE at that moment — plausibly the two healthy covers had live viewers. Every inactive instance missed the transition, and on (re)activation the hub replays the persisted area document without re-checking the type's current compilationStatus — so recycle/restart faithfully re-serve the stale placeholder. Classic missed-transition + replay-on-init pair; one stale side effect, no error, no log.

Fix direction (no band-aids)

On instance-hub init (and on cover render), derive the compile-placeholder-vs-content decision from the type's current compilationStatus, never from the stored area snapshot — or invalidate/rewrite instances' persisted cover areas as part of compile completion so the store never outlives the state it depicts. A watchdog that re-renders periodically is the forbidden shape.

Interim

The covers are the public storefront; until the fix, any touch that rewrites the cover area doc (a content-version bump per stuck node) would clear it — that is a data mutation on 10 prod nodes and is left to the maintainer's call.

Also noted by plugins-ff, smaller and separate: rbuergi/ThinkInStreams installed pre-apps-wave has no _App tile; AppTileRefresh heals only self-installing packages.

🤖 Generated with Claude Code

[rbuergi]

Fixed in #2435 (merged, b3095765f).

What was actually wrong

Not a rendering bug, and not a persisted area document — the MCP get {path}/area/{Name} route renders live through RenderArea, so the tree in the report was a live render off a stale type-node snapshot. The tell is in the report itself: the embedded LayoutAreaControl pointed at the 16:09 compile, not the 16:26 one that finished.

The bind path and the heal path consulted different authorities.

IMeshNodeStreamCache hands every consumer one process-wide ReplaySubject(1) per path. When the owner's sync stream goes silent — as opposed to faulting — that subject replays its last value forever and nothing can evict it: fault-eviction needs a fault, and idle-eviction needs zero subscribers, which the overlay's own long-lived watchers guarantee can never happen. With WaitForCompileSettled's no-progress budget disarmed while in-flight, a latched Compiling snapshot makes the overlay branch deterministic, not occasional — every activation reaches it after the 5 s grace.

The self-heal watcher was right the whole time. It re-reads STORAGE (AuthoritativeTypeRead), saw a usable build, and correctly recycled the instance. But only the bind path decides what the restarted instance serves, and it read the latched mirror — so re-enrichment overlaid it again seconds later. Heal → recycle → re-poison, which OverlayHealBudget then widened to a ~10-minute steady state. That is why recycling could not clear it: the remedy and the defect were the same loop.

Answering the question in the brief — did a write silently fail?

Yes, but not the compile's. Two writes on this very page were Content is T trap-doors that UpdateRemote turns into silent no-ops on an un-materialized JsonElement (the normal mirror shape):

• the Recompile button — the one remedy the stuck page offers a user did nothing at all, with no error;
• TriggerRecompileAndRetry's Ok→Pending flip and its version gate — so the automatic recompile for a type believed stranded could not fire either.

Both now read with ContentAs<NodeTypeDefinition>. The compile's own terminal Ok write is not implicated: the type node read compilationStatus: Ok, so it landed.

The fix

BuildSlowPath now makes both halves consult the same authority at the moment it decides: on the in-flight branch only, one storage read before committing an instance to the overlay. Not a poller, not a watchdog — when stream and storage agree the overlay is applied exactly as before; when they disagree the instance binds the build that actually exists. No query core, no storage row, or no answer within NodeTypeProbeTimeout all keep the previous behaviour, and the last is logged, because "could not check" and "checked, still compiling" must not look alike.

OverlayAuthorityPreferenceTest pins the decision without a mesh, including that preferring storage never strips HubConfiguration ([JsonIgnore, NotMapped], so its absence on a storage row is the read's blind spot, not information — dropping it would trade a stuck placeholder for an empty page).

Two things NOT fixed, named so they are not mistaken for covered

1. The silence itself. JsonSynchronizationStream's exhausted re-arm budget, and a cross-process write that announces no version locally. This change lets an instance recover from that silence instead of being wedged by it; it does not end it.
2. A durable row that itself still reads Compiling. The PG upsert drops any write whose version is not higher (WHERE target.version <= EXCLUDED.version) — the Skill idempotence flake: BuiltInSkillProvider vs the Skill package's durable install race on Skill/presentation + Skill/slide #2361 refused-write family. Discriminating needs the prod row, which I could not read. If that is what happened on those covers, this change is inert for them rather than wrong; the in-code mechanism it fixes is fully proven.

Verification, stated plainly

Not verified by watching a stuck cover recover — the prod state had already cleared on the mesh I can reach (@Chess/area/Cover now answers "Area not found", which is the honest reply: Cover is not a registered area on that hub even when healthy, so the issue's probe used an area name that never existed). Verified instead by the unit-pinned decision plus Graph 1630/1630, the Monolith compile/overlay set 35/35 (including FrameworkStaleAssembly), and full CI green.

Separately filed from this work: #2441 — the impersonation ratchet guard is blind to AccessContextScope.AsSystem, which is why two banned Observable.Using identity scopes shipped in this very file while the guard reported it as clean.