6,925 Started sync/ hubs hold ~2.7 GB of a 3.5 GB live heap — population MEASURED, cause NOT established (split from #3321)

[rbuergi]

Split out of #3321, which is CLOSED. Its step 3 released the dead hubs; this is the remainder,
and it is the larger half by bytes. Filing it so the finding does not live only in a closed issue's
comments.

What was measured, and where it came from

Five --type Heap dumps of memex-cloud/…-gs6j6 (26 h uptime, 6 859 MiB working set,
~210 MB/h growth, 267 gen2 collections freeing nothing), 2026-09-04 22:32–23:14Z, read-only
throughout, dump files deleted from the pod afterwards. The full walk is in
Doc/Architecture/PortalHeapIsHubs (#3325).

A ClrMD RunLevel histogram over all 9 398 MessageHub instances:

TOTAL MessageHub: 9398
    6925  sync  RunLevel=1Started      ← THIS ISSUE
    1495  sync  RunLevel=6Dead         ⎫ #3321 step 3
     974  <node-or-other>  RunLevel=1Started
       4  sync  RunLevel=0Starting

At the ~390 KB a hub retains (its own Autofac ILifetimeScope + its own TypeRegistry; ~173 KB of
that is framework metadata duplicated per hub), the Started population is ≈2.7 GB of a 3.51 GB
live heap. The Dead population #3321 addressed was ≈580 MB.

For the record, the split #3321's step 3 actually turned on:

	count
Dead sync/ hubs held by a stream	1 496
…under a stream that had itself been disposed	11
…under a stream nobody ever disposed	1 485
SynchronizationStream total / disposed	8 461 / 11

What step 3 fixed, and why it needed two hooks rather than one

The framing everywhere — including this issue's parent — was "SynchronizationStream.Dispose()
clears Hub". Against the dump that is 11 of 1 496.

A sync/ hub is a HOSTED hub. Its parent disposes it during the parent's own teardown — a Blazor
circuit ending, a DisposeRequest, a recycle — while the stream that created it is owned by a
workspace somewhere else and is never told. The stream therefore keeps isDisposed == false,
keeps a strong reference to the corpse, and keeps being handed out as usable. Dispose() never runs
for those 1 485, so clearing the field there reclaims nothing.

So the release happens at both ends, through one idempotent ReleaseHub():

end	where	covers
the stream is disposed	Dispose(), after Hub.Dispose()	the 11
the hub dies underneath it	syncHub.RegisterForDisposal(_ => ReleaseHub()), from the constructor	the 1 485

Completed in #3427 (core, f41f8bda) and Systemorph/MeshWeaver.Plugins#1405 (cdfaef28, merged
first — it guards 11 Blazor-view dereferences that would otherwise have NRE'd).

🚨 Expect the curve to FLATTEN, not to vanish. ≈580 MB of ≈3.5 GB was released; the ≈2.7 GB this
issue is about was not touched. Nobody should read a flattening as a fix, and nobody should read a
still-rising curve as step 3 having failed.

🚨 What is NOT established — treat everything below as hypothesis

This issue is a measured population plus an unproven cause. The population is solid; the
mechanism is not, and #3321's own history is the argument for not guessing: two separate sessions
proposed fixes for the Dead half from a type histogram alone, and both were wrong until a
referrer walk named the actual holder.

Not established:

• Whether these 6 925 are garbage at all. 8 450 undisposed streams is consistent with
  subscribers that really are attached — a portal with many open Blazor circuits, each holding live
  layout-area streams, is supposed to look like this. It is equally consistent with streams
  abandoned without Dispose(). A type histogram cannot separate those two, and they call for
  opposite fixes: one is a leak, the other is the working set being genuinely larger than the
  ceiling.
• What holds them. Unknown. For the Dead half the answer turned out to be exactly one
  referrer (SynchronizationStream<T>.Hub, 1 496 of 1 496). Nothing establishes that the Started
  half has a single holder, or the same one.
• Whether the ~390 KB-per-hub cost is the real story. ≈1.58 GB of the 3.51 GB live heap — 45 % —
  is identical framework metadata (Autofac MiddlewareDeclaration / ExternalComponentRegistration
  / ResolvePipelineBuilder, plus TypeDefinition and its Lazy/Func fan-out) held 9 386 times.
  If most of the 6 925 are legitimately live, then there is no leak here at all and the issue is
  per-hub COST, which is a different fix (sharing the registration metadata across sibling hubs)
  and probably a different issue.
• Whether step 3 changed this number. Not re-measured. The hub-death hook releases a reference
  from an undisposed stream, which could plausibly move some hubs out of the Started bucket
  earlier — or change nothing here at all.

The next measurement, and what would falsify the theory

One dump, against a replica running a build that carries f41f8bda, compared on the counters
#3321 already established so the comparison is like-for-like:

1. dumpheap -stat's Total … objects line (🚨 not dotnet-gcdump, which silently truncated
   at exactly 10 000 000 objects and reported a 568 MB heap — one sixth of the truth — with a type
   table skewed to whatever it reached first).
2. The ClrMD RunLevel histogram over MessageHub, reproducing the table above.
3. SynchronizationStream total / disposed.
4. The discriminator this issue actually turns on: the same one-pass referrer walk that named
   the Dead half's holder, retargeted at the streams behind the Started hubs. 🚨 SOS dumpobj
   and gcroot segfault on this process — anything that follows a field must be ClrMD, built
   and run inside the debug container.

What each outcome means:

result	reading
Started count fell materially and SynchronizationStream disposed rose	step 3's hub-death hook reaches further than claimed; re-scope this issue to what is left
Started count unchanged, referrer walk finds ONE dominant holder	a leak with a named retainer — the #3321 shape again, and actionable
Started count unchanged, referrers are diffuse and match live subscribers (circuits, open layout areas)	the current "leak" framing is FALSIFIED. This is per-hub COST, not retention; close this and open one about sharing DI/type-registry metadata across sibling hubs
Started count ROSE	step 3 moved hubs between buckets rather than releasing them — a regression to investigate before anything else

The third row is the one worth stating plainly, because it is the outcome that would make this
issue wrong: if most of the 6 925 are legitimately live, the portal's working set is simply larger
than its ceiling and the ~390 KB-per-hub DI cost is the whole story.

Related

• Portal managed heap leaks ~215 MB/h through 267 gen2 collections, with ZERO sync/bake events — the cause #2194 is not tracking #3321 (parent, closed) — the Dead half, released
• fix(#3321): step 3 — a disposed stream releases its hub #3427 (core) · Systemorph/MeshWeaver.Plugins#1405 (dependent, merged first)
• Doc/Architecture/PortalHeapIsHubs (docs: name the portal heap retention root — 8,419 of 9,393 hubs are sync/ stream hubs (#3321) #3325) — the five measurements, the retaining root with
  file:line, the three falsified candidates, and the counter recipe that discriminates
  retention-vs-fragmentation without a dump (gen2 heap size vs gen2 fragmentation in one
  read; dotnet.timer.count as a cheap proxy for live hubs — 11 387 at 6.9 GB, 37 796 at 14.0 GB)
• Doc/Architecture/StreamLivenessAndTheHubReference — step 3's design, including the 11-vs-1 485
  split and why it needed two hooks

[rbuergi]

The retainer chain is still NOT established — and the measurement that would establish it is blocked twice over

Session 2026-09-06. No sibling: this issue had zero comments and no branch/PR referenced it.

I went for the one discriminator this issue turns on — the referrer walk retargeted at the
Started population's streams — and did not get it. Two things stopped it, both now measured,
and the second one cost a container restart. Reporting both plainly rather than substituting a
plausible cause.

1. No deployment carries f41f8bda, so the "did the number move" row cannot be read

• fix(#3321): step 3 — a disposed stream releases its hub #3427 merged as f41f8bda at 15:49Z. The earliest CD run publishing an image from it is
  ci.7906 (CD 7905 was f41f8bda itself and was cancelled; 7906 = 71c194ca9, success).
• Both memex and memex-cloud run meshweaver.azurecr.io/memex-portal-ai:3.0.0-rc9.ci.7693 —
  over two hundred CD runs older — because prod was rolled back by hand that evening (Adopt-time identity guard: a process must refuse a bake keyed to a framework identity that is not its own, and compilationStatus must not read Ok for a type whose hubs cannot activate #3472/A process refused by its readiness gate still JOINS the mesh — readiness gates traffic, not membership, and a refused pod stamped module generations for 2h (caused #3472) #3478).

So any dump taken tonight measures the pre-step-3 population. It cannot answer the first row of
this issue's outcome table (Started count fell / unchanged / rose), because there is no
post-f41f8bda replica to compare against.

2. 🚨 The dump itself is not read-only in the way this issue assumes — it restarted the replica

memex-cloud/memex-portal-deployment-58c859d4cc-gqqg7 (8 h, 6 609 MiB, ci.7693):

21:37:38Z  dotnet-dump collect -p 1 --type Heap    (process suspended)
21:39:17Z  kubelet: Container memex-portal failed liveness probe, will be restarted
21:39:26Z  Complete                                (resumed — 106 s of freeze)
21:39:35Z  ephemeral container SIGKILLed with the restart
21:42:18Z  pod 1/1 Running, RESTARTS 1

The probes on the live deployment: readiness /ready 10 s × 3 → out of the Service at ≈30 s;
liveness /alive 15 s × 6 → container restart at ≈90 s. A 106 s freeze crosses it.

The freeze is not write-bound, so a faster disk does not help: dd oflag=direct on the same
node measured 169 MB/s (3.1 GB / 18.6 s), which puts a 5.5 GB dump file at ≈33 s of the 106.
The remaining ≈73 s is createdump walking ≈50 M objects with every thread suspended — i.e. the
freeze scales with exactly the thing that makes this heap interesting.

That gives the governing constraint for this issue:

The population only exists on a replica fat enough that dumping it restarts the replica. A
freshly restarted portal here was back to 2 977 MiB within five minutes, so ~3 GiB is the
floor, not the accumulation — a small replica dumps safely and has nothing to show.

The other four replicas stayed 1/1 Running / RESTARTS 0 throughout, so the deployment served
from four of five for ≈2 min. I stopped there rather than dumping a second replica: the brief said
to stop rather than disturb a serving pod, and I now had proof it does. No dump file was left
behind (the restart wiped the container's writable layer; verified /tmp clean afterwards). No
cluster object was mutated, no deploy/roll/scale was performed, memex was never touched.

This also corrects Doc/Architecture/PortalHeapIsHubs, which says of the 2026-09-04 dumps
"the dump file was deleted from the pod afterwards; no cluster object was mutated". True as far as
it goes, but it reads as "no impact", and at that specimen's 6 859 MiB the same freeze would have
crossed the same 90 s budget.

What is established, and what is not

Retainer chain for the 6 925 Started sync/ hubs	NOT established. Unchanged from the issue as filed.
Post-f41f8bda population	Not measurable — no replica carries it.
Cost of the measurement	Established: 106 s freeze at 6.6 GiB ⇒ one container restart.
Growth floor	Established: a fresh replica reaches ~3 GiB in ~5 min.

I deliberately did not write a fix. Nothing here names a retainer, and #3321's history is the
argument against guessing one. (For the record, the two candidates I read out of src/ — the
per-hub stale-callback scanner and SynchronizationStream.sharedReduceCache — are hypotheses:
the scanner already holds a WeakReference by design (MessageHub.InstallStaleCallbackScanner),
and sharedReduceCache is unbounded but cleared on dispose. Neither is evidence.)

What the next attempt needs

1. A replica carrying ci.7906 or later. Until a deployment is rolled forward past the
   ci.7693 rollback, the comparison this issue is built on cannot be made at all.
2. A deliberate decision to spend one replica restart on the dump — with five replicas that is
   ≈2 min at four-of-five, which may well be acceptable, but it is a call to make in advance and
   not at the end of an outage night.
3. The walk itself is ready and cheap. The ClrMD field recipe — DAC path, Address.Host (so a
   Started sub-hub under a Dead parent separates from one under a live parent), and the
   ReplaySubject → _implementation → _observers → _data[] subscriber count that answers "is
   anyone actually listening to this stream" — is now written down in the doc, verified to compile
   against Microsoft.Diagnostics.Runtime 4.0.732401.

Doc changes (the hazard, the constraint, the walk recipe, and the correction) are in the PR below.
This issue stays open and its title stays accurate: population measured, cause not established.

[rbuergi]

Narrowed, not fixed — and one of the issue's two open questions is now closed from src/

Session 2026-09-07. No dump was taken and no replica was touched: everything below is read out of
src/ with file citations, and the arithmetic in this issue is what confirms it. The durable form is
committed as Doc/Architecture/SyncHubPopulation in #3636; this comment is the pointer.

✅ CLOSED — "what holds them": a sync/ hub is one field of a SynchronizationStream

SynchronizationStream<T>'s constructor mints exactly one, unconditionally, for every stream
(src/MeshWeaver.Data/Serialization/SynchronizationStream.cs, Host.GetHostedHub(SynchronizationAddress.Create(ClientId), …, HostedHubCreation.Always) → Hub = syncHub). And one cross-hub
subscription builds TWO streams: the client's (JsonSynchronizationStream.CreateExternalClient,
Host = the subscriber) and the owner's twin (Workspace.SubscribeToClient →
CreateSynchronizationStream → ReduceStream(… WithClientId(request.StreamId)), Host = the owner)
— same ClientId, different parents.

This issue's own numbers agree, and that is the check that makes it more than a story:

6 925 Started sync/  +  1 495 Dead sync/  =  8 420
SynchronizationStream total                =  8 461      (0.5 % apart)

So there is no separate sync-hub population to explain. The single dominant holder is the same
shape as the Dead half's — a stream's own Hub field, plus HostedHubsCollection on its parent.
The question reduces to "why are there 8 461 live SynchronizationStreams", and the ~390 KB per hub
is then a multiplier on that count, not an independent story.

✅ ANSWERED — "which is broken": the retirement path EXISTS and cannot fire on the paths that generate the garbage

Three things retire a sync/ hub, and only one of them reaches a Started hub under a live
parent:

path	reaches
SynchronizationStream.Dispose() disposes its own Hub	the 11
parent teardown → RegisterForDisposal(_ => ReleaseHub()) (#3427)	the 1 485 measured Dead
MeshNodeStreamCache idle sweep → TryReleaseUnwatched → DetachUpstreams → TearDownEntry → stream.Dispose()	the Started population — and nothing else does

MeshNodeStreamHandle.DetachUpstreams() is reachable from nowhere else, and it is also the only
thing that collects change-feed-parked predecessors.

Its predicate is a conjunction — !evicted && subscribers == 0 && now - lastActiveAt >= 10 min
(MeshNodeStreamCache.Entry.TryMarkIdleEvicted; window/period in MeshNodeStreamCacheOptions) —
and that option's own summary says what refreshes the window: "Every GetStream subscription,
unsubscription and Update on the path refreshes the window."

🚨 The self-defeating loop. Workspace.EvictForPath runs on every change-feed event and removes
every identity's cached stream for that owner, parking it; ReclaimIfUnheld then disposes it only
if no declared lease remains — and the mesh-node cache's hydration is a declared holder for the
entry's whole life (MeshNodeStreamHandle.Subscribe says so in as many words). Nothing re-points
that hydration, because the sweep "only ever CLOSES idle entries; it never re-subscribes anything".
So the first write to a cached path leaves the entry reading a stream that is out of the cache while
the next reader builds a fresh one — two live streams, four sync/ hubs, where the design intends
one — and the same Update that orphaned it also Touch()es the entry and pushes the 10-minute
window out. A path written at least once every ten minutes can never be released, so its orphan can
never be collected.

Answer to the question this issue poses: present-but-unable-to-fire, not missing. Named code
path above.

🚨 A correction for anyone reading Workspace.cs first

Workspace.cs warns that "counting Rx subscribers CANNOT answer 'is anyone still reading this' …
an evicted stream measures 2–3 subscribers and never reaches zero — implemented, measured and
reverted". That is a different, abandoned mechanism at the workspace layer.
MeshNodeStreamCache.Entry.subscribers is a purpose-built refcount moved only by
TryAddSubscriber/RemoveSubscriber from SharedView, and the internal hydration subscription
bypasses it — it genuinely can reach zero. Reasoning about this issue from the Workspace
comment alone mis-attributes the blocker to the wrong conjunct.

🚨 Still NOT established

• Whether the 8 461 are garbage. Thousands of open circuits each holding live layout-area streams
  is what a healthy portal of this size looks like. The above explains the population's shape, not
  that it is waste.
• The split between the per-path floor above and legitimately-live streams — which is exactly
  what decides between this issue's outcome rows 2 and 3.
• Whether fix(#3321): step 3 — a disposed stream releases its hub #3427 moved the number. Still not re-measured.

The measurement I would take next — a RATIO, off the histogram the plan already collects

The plan's discriminator is a ClrMD referrer walk, which needs the dump that (measured, 2026-09-06)
restarts the replica. The identity above makes a far cheaper reading decisive, and it comes off
dumpheap -stat, which that plan already takes:

count MeshNodeStreamCache+Entry alongside SynchronizationStream.

reading	means	next
streams ≈ 2 × entries	the orphan floor dominates	a fix with a named code path: release the entry's lease on eviction (or re-point the hydration) so ReclaimIfUnheld collects it — halves the population, touches no live view
streams ≈ 1 × entries	no orphaning	the leak framing is falsified (this issue's own third row): it is per-hub DI cost — close this, open the metadata-sharing issue
streams ≫ 2 × entries	something outside the mesh-node cache mints them	look at the Identity dimension of _remoteStreamCache's key first

One extra histogram line, no referrer walk, no second dump. It does not replace the walk for row 3 —
it tells you whether you need one.

This issue stays OPEN and its title stays accurate: population measured, cause narrowed to a named
mechanism, not established. I deliberately did not write a fix: #3321's history is that two
sessions proposed one from a histogram and both were wrong, and the ratio above is one cheap reading
away from telling us whether the fix named in row 1 is the right one.

_{No code changed. Doc page committed; no replica was dumped, restarted or otherwise touched.}

[rbuergi]

✅ "What holds them" is CLOSED for a named mechanism — established by a controlled experiment, not a dump

Session 2026-09-08. No replica was dumped, restarted or touched. Everything below is read out of src/ with file citations plus an in-process two-arm test. The durable form is committed as Doc/Architecture/EvictedStreamRetention in #3733; this comment is the pointer.

The retainer

Workspace.ReclaimIfUnheld opens by collapsing two different situations into one branch:

// src/MeshWeaver.Data/Workspace.cs
if (!_remoteStreamLeases.TryGetValue(stream, out var leases) || leases != 0)
    return;

leases != 0 is "a holder is still declared". TryGetValue == false is "no holder was ever declared". Only the first is a reason to keep the stream.

So a stream nobody leased, parked by EvictForPath on a change-feed event, is never disposed. It sits in Workspace._evictedRemoteStreams — an instance dictionary on the singleton workspace — holding its client sync/ hub and its owner-side twin, until Workspace.Dispose(), i.e. process exit.

🚨 DiscardFaultedRemoteStream, in the same file, handles the identical case correctly — it checks ContainsKey explicitly and disposes on the spot. Two sites, one situation, opposite outcomes.

Nothing else collects them

DetachRemoteStreams is the only other drain, reachable from exactly one call site (MeshNodeStreamHandle.DetachUpstreams()), always with new MeshNodeReference(), matching on Equals(parked.Reference, reference).

parked stream's Reference	reaped by the mesh-node cache's idle release?
MeshNodeReference	yes, if the cache holds an entry and the sweep fires
LayoutAreaReference / CollectionReference / anything else	no — nothing matches it

And LayoutAreaReference is the volume case. LayoutExtensions.GetControlStream — every rendered layout area — is GetRemoteStream(address, new LayoutAreaReference(area){Id=id}): cached, no lease. EvictForPath matches on the owner alone and ignores the reference, so a change event on that owner parks the layout area's stream too. When the Blazor circuit ends, its subscription drops and the stream is disposed by nobody.

Which of the issue's three categories

RETENTION. Not a leak — nothing is created-and-forgotten; the parking is deliberate and documented ("an undeclared reader … may still be attached"). Not correct-and-costly — the cache's own invariant is "one stream per key, never two competing live subscriptions". The fix belongs in the holder, and it is the remedy the source already names: "Opting a call site in is one line."

The deterministic repro (EvictedUnleasedStreamRetentionTest, #3733)

One (owner, reference, identity) key, five change-feed events, run twice, differing in exactly one thing — whether a lease is taken. Live sync/ hubs counted off HostedHubsCollection:

unleased: clientSyncHubs=+5      ← grows once per change event
leased:   clientSyncHubs=+0 / +1 ← the denominator: one mirror per key

🚨 The leased arm is a positive control and it earned its keep. The first version measured reclamation by counting UnsubscribeRequest at the owner and the control FAILED — closed=0 in both arms. The instrument was wrong, not the mechanism. Had it been omitted, the unleased arm's closed=0 would have been reported here as proof of a retention it does not actually measure.

Ruled OUT — nobody needs to re-run these

• No static field anywhere in src/ holds a hub or a stream. Zero of type IMessageHub, MessageHub, ISynchronizationStream, IWorkspace or IMeshService. Every registry is already an instance field on a mesh-scoped singleton.
• No timer roots a hub. The per-hub stale-callback scanner, the sync-stream heartbeat, the MeshDataSource sampler, the kernel idle disconnect and the NodeType debounce all hold a WeakReference and self-dispose one tick after the target is collected.

A second, independent root on the same population

JsonSynchronizationStream registers the stream's owner-protocol subscription twice — the second on the outer, long-lived hub ("Belt-and-suspenders"). MessageHub.RegisterForDisposal adds to a plain Rx CompositeDisposable, and Add never prunes. So the outer hub accumulates one strong reference per remote stream ever opened, and the closure captures the stream. The line above it already ties the subscription to the stream's own lifetime.

🚨 The measurement that closes this issue — and it is NOT the referrer walk

The plan's ClrMD referrer walk needs the dump that (measured 2026-09-06) freezes the replica 106 s and restarts it. The retention above is a field on one object, so:

On the singleton Workspace, read _evictedRemoteStreams.Count and _remoteStreamCache.Count.

Two field reads on a single instance — no referrer walk, no heap traversal, no histogram.

reading	means
_evictedRemoteStreams ≫ _remoteStreamCache	this mechanism dominates — fix at the call sites
_evictedRemoteStreams ≈ 0	retention is elsewhere; the outer-hub composite becomes prime suspect
both small vs 6 925	streams are legitimately live — this issue's third outcome row, per-hub DI cost

Both counts are also derivable without any dump from existing Debug logging, which already emits one line per open, one per eviction and one per reclaim: opened − reclaimed is the retained count.

🚨 Still NOT established

That this accounts for all 6 925. The mechanism and its unboundedness are established; the magnitude is not. That is exactly what the two counts above settle.

Why no fix here

GetControlStream is public, the change alters every layout area's stream lifetime, resolving inside Observable.Create moves resolution from call time to subscribe time, the most-affected consumers are in-mesh layout areas no dotnet build or CI job ever type-checks, and AcquireRemoteStreamUnchecked is internal to MeshWeaver.Data with no InternalsVisibleTo for MeshWeaver.Layout. That is a scope call, not a detail. The shape is written out in §9 of the doc.

This issue stays OPEN. Its title can now read cause established for a named mechanism, magnitude not measured.

_{No src/ behaviour changed. The unleased test arm PINS the defect: when the retention is fixed it goes red, and that red is the fix landing.}

[rbuergi]

🚨 The two sites are NOT one situation — and "fix" the TryGetValue == false branch and healthy streams get disposed under undeclared readers

The comment above closes "what holds them" on this reading:

leases != 0 is "a holder is still declared". TryGetValue == false is "no holder was ever
declared". Only the first is a reason to keep the stream. … DiscardFaultedRemoteStream, in the
same file, handles the identical case correctly. Two sites, one situation, opposite outcomes.

The population measurement is solid and I am not disputing it. The inference is contradicted by
the design, in writing, at the call site — and acting on it would be a silent correctness
regression bought with memory.

What the code says, at the site that discards

Workspace.cs, immediately above the DiscardFaultedRemoteStream call (read on origin/main
today):

🚨 … Unlike the change-feed eviction, which parks a HEALTHY stream because a reader may still be
attached, a faulted store can never notify anyone again (Rx grammar), so there is nobody left
to keep it alive for. Faulted ONLY: a stream that is unusable because its hub is winding
down is already inside that cascade …

and at the site that parks, in EvictForPath:

Do NOT unconditionally dispose the evicted stream — an undeclared reader (e.g. a
MeshDataSource reduce callback that handed the stream on) may still be attached and needs to
keep receiving updates until it drops on its own. … A stream whose holders DECLARED
themselves … and have all left is reclaimed immediately.

So the discriminator is not "was a lease ever declared". It is whether the stream can still serve
anyone:

	faulted (DiscardFaultedRemoteStream)	change-feed eviction (ReclaimIfUnheld)
stream state	terminal — Rx grammar forbids another notification	healthy, merely superseded
an undeclared reader still attached	gets nothing either way	is still being served
disposing when never leased	loses nothing	cuts a live reader off, silently

ReclaimIfUnheld's own header states the choice as a choice: "THE LIFETIME OF AN EVICTED STREAM —
declared holders, NOT Rx subscriber counts." Reclaiming on TryGetValue == false inverts it into
"no declared holder ⇒ nobody at all", which is precisely the assumption the comment refuses.

So the retention is real and it is a CONSEQUENCE, not the defect

The parked population is not the lease bookkeeping failing. It is eviction-per-write minting
streams the bookkeeping is deliberately unable to collect — because it cannot see undeclared
readers, and by design will not guess.

And the file already names the fix, one paragraph above EvictForPath:

The real fix is to make the writer's base version-aware rather than to buy its freshness with
a full re-subscribe — i.e. wait for the mirror to reach the version the change feed announced.
That is a change to the write path, not to this method.

with the measurement that makes the scale obvious, in the same block: "six progress writes to one
activity node with one live reader mint 7 client-side and 11 owner-side sync/ hubs." Six writes,
eighteen hubs. At ~390 KB retained per hub, that is where 2.7 GB comes from — and no amount of
collection-side cleverness makes eviction-per-write stop minting them.

What I would do instead

1. Do not change the ReclaimIfUnheld predicate. If it is ever changed, it needs a positive
   demonstration that no undeclared reader can be attached — not the absence of a lease, which is
   the very thing that says nothing about them.
2. Attack the minting: version-aware writes, so a change-feed event does not evict a healthy
   stream at all. That is the named fix, it is a write-path change, and it makes the collection
   question moot rather than harder.
3. If an intermediate is wanted, the honest one is to make undeclared readers declarable — i.e.
   close the gap the design apologises for — so ReclaimIfUnheld can answer the question it is
   actually asking. That is a real change of contract and belongs in its own issue.

Doc/Architecture/EvictedStreamRetention (#3733) should carry this distinction; as summarised in
the comment above it reads as though the branch is an oversight, and the next reader will delete it.

[rbuergi]

Reopened — the 2026-09-10 sweep was not a triage decision

This issue was closed not_planned at 2026-09-10T14:31:27Z by a bulk sweep that PATCHed every open issue in all 13 Systemorph repositories — 311 issues in twelve minutes, no comment on any of them — after a session was asked to "close all issues in all repos". Nothing on this thread was read, and no judgement was made about this issue.

All 82 issues the sweep closed in the MeshWeaver fleet were reviewed afterwards. This one is unresolved: Cause NOT established — the last comment disputes the mechanism; ~2.7 GB of a 3.5 GB live heap is still unexplained.

Sixteen of the 82 were left closed — their fix had actually landed, or the author had withdrawn them; those should have read completed, not not planned.

[rbuergi]

Reproduced, and the seam the earlier attempts were missing

PR #3897 carries a deterministic repro that FAILS without the fix and passes with it.

The seam is the CREATE, not the read. All three earlier attempts tried to make the monolith rig
degrade the content on the way out. It never will: a create hands the store a LIVE CLR instance
and the same process reads that instance straight back, so existing.Content is always typed and
the pipeline's typing step is never reached at all. (Measured here too, before changing anything:
on a NodeType-changing update the validator saw RetypeBeforeContent — the old type, intact — on
the unfixed code. That is the benign branch the previous comment describes, and it is real.)

So the test creates the node with the degraded shape already in place — a bare JsonElement
with no $type:

{"title":"Original","sequence":5}

That is not synthetic. It is literally what ObjectPolymorphicConverter.Write persists for a
content type no hub registered (its final else: "Type not registered, serialize without type
information"), and with no discriminator left, every later read on every hub reproduces it
forever. It is the #3056 world the typing step exists for.

With the snapshot in that state and RetypeBeforeContent(Title, Sequence) →
RetypeAfterContent(Title, Reason), the conversion succeeds (UnmappedMemberHandling = Skip)
and the validator is handed RetypeAfterContent("Original", null) — the silent case the previous
comment predicted would be worse than the reported JsonException, now pinned:

Did not expect value to be of type RetypeAfterContent because an in-place NodeType change is
precisely where the pipeline's own premise — 'same node, same NodeType' — is false …

On the proposed fix. The discriminator route is not needed and would be dead code:
MeshNodeStreamExtensions.EnsureTypedContent runs on EVERY emission of the stream this snapshot
comes from and already applies the exact recovery — IMeshContentTypeRegistry.TryRecoverForNodeType
keyed on the node's OWN NodeType — plus a late re-type wait for a type registered later. A snapshot
still untyped when the pipeline sees it is untyped because nothing in the process can type it. So
the change is the gate (RetypesTheNode, requiring BOTH sides to name a type so an absent
NodeType reads as unchanged and ordinary partial updates keep the #3056 cure) plus the Warning
the earlier comment asked for. No acceptance behaviour changes — the retype still lands; a
validator that wants to refuse one refuses it on the NodeType, which both sides always carry. That
was the half awaiting a maintainer decision, and it turns out not to be needed.

[rbuergi]

Follow-up: this population is NOT #3593 — measured, not assumed

A lead worth ruling out: #3593 reports a mass event where every affected hub was also a sync/ hub also at RunLevel=Started — but reading Disposal=Pending, buffer=1, exec=0: Dispose() HAD been called and the action block never dequeued the buffered ShutdownRequest, so "each pending hub pins its stream, subscriptions and object graph until restart". Same family, opposite mechanism.

The census asked directly rather than reasoning about it — every sync/ hub on the replica classified by the same fields MessageHub.AppendDiagnostics prints (disposalStarted, DisposalSignalled, MessageService.GetQueueSnapshot()). memex-cloud pod …-f9lb5, uptime 123.5 min:

CENSUS6  sync=4649  withBufferedMessage=0  executingATurn=0  snapshotUnreadable=0
   4648  Started  / Disposal=<not started>
      1  Starting / Disposal=<not started>

Zero Pending. Zero buffered. Zero executing. And snapshotUnreadable=0 states the denominator — every one of the 4,649 was actually read, not skipped.

So these hubs were never asked to dispose: nothing is wedged, no pump is stuck, and nothing is being retained against a teardown in progress. They are alive because the stream that owns each of them is alive and nothing ever released it — which is exactly where the attribution landed (unbounded creation on the read path, one hub per GetDataRequest).

#3593 keeps its own defect and its own fix; nothing here should be filed under it. Committed as §4b of Doc/Architecture/ReadPathStreamMinting in #3952.

[rbuergi]

Post-fix measurement on the rolled image: the population is still growing. Do not close this.

#3952 (the read-path stream-minting fix) merged 2026-09-10T18:57:25Z, and memex.meshweaver.cloud has since rolled to 3.0.0+6231c4da (commit dated 03:32:11Z), so the fix is live — verified by symbol with the fetch size checked, since an empty body greps to zero and reads as absence.

Re-ran rbuergi/HubCensus0910c against that pod. Two readings, because one reading is not a rate:

CENSUS3  pod=memex-portal-deployment-6d7497cb58-jc296
  upMin=54.4  wsMiB=7548  gcMiB=4573  gen2=60
  hubs=9801  sync=8595  parents=1198  explained=5487  residual=3108

  upMin=55.0  wsMiB=7597  gcMiB=5135  gen2=60
  hubs=9818  sync=8610  parents=1199  explained=5499  residual=3111

Δ over 36 s: sync +15, residual +3 ⇒ ≈25 sync hubs/min, still climbing. Working set 7.6 GiB, GC heap 4.6 → 5.1 GiB across the two samples.

What this does and does not establish

Does: on an image carrying #3952, the sync/ population is still growing on a pod only 55 minutes old, and the absolute figures are larger than the ones this issue was filed about (2.7 GiB of a 3.5 GiB heap). 1,199 parents carry excess; 3,111 of 8,610 hubs (36%) are unexplained by any known holder — cache, parked leases, subscriptions, local streams or data-source streams.

Does not:

• 🚨 This is CENSUS3 (the v3 script). The headline pre-fix figures in this thread came from a later v6, which classified by disposal state and is a different instrument. 8,610 vs 4,649 is not a like-for-like comparison and I am not claiming a regression from it.
• A 36-second sampling window is a thin baseline for a rate. It shows growth is non-zero; it does not pin the slope.
• Different pod, different load, different time of day than the earlier reading. A pod 55 minutes past a roll is also still settling.

So: growth continues, the residual fraction is essentially unchanged (38% → 36%), and #3952 did what its author said it did — "stops the growth" was the claim for one creation path, and the report explicitly stated the fix "does not shrink what exists" and left 38% unattributed.

What would settle it

Re-run v6 (the disposal-state census) on this pod so the numbers are comparable, and sample over ≥10 minutes rather than 36 seconds. The parents-with-excess list is the lead: the top entries are _Activity/compile-state and _Activity/content-manifest paths with 6–11 sync hubs each and only 3–5 explained — a per-activity pattern, repeated across 1,199 parents, which is where the 3,111 lives.

Nobody should read #3952's merge as closing this.