Input wanted before I measure: is the retained allocator slack fragmentation, or one allocation pattern in aft?

[iceteaSA]

Follow-on from #201, where the shipped malloc_trim(0) relief turned out not to reclaim anything on a long-lived Linux daemon. The free space is pinned beneath live allocations, so trim structurally can't return it.

The open question is which of two things that is:

mechanism	remedy	whose code
ordinary heap fragmentation	allocator with decay-based purging	aft's #[global_allocator]
one pathological allocation pattern	fix the site	aft, specific

mallinfo2 gives aggregates only. The per-arena and per-bin free-space distribution separates the two, and that's malloc_info(3) — which the crate doesn't bind today.

I've designed an experiment and I'd rather ask before building it, because you may already know the answer or want it shaped differently. Specifically:

1. Do you have prior context on the allocation pattern? Long-lived per-root artifacts (search index, semantic index, callgraph store, symbol cache) are each individually capped but have no aggregate ceiling across roots — the daemon here held 7+. Many large, long-lived, differently-sized buffers arriving and departing on independent schedules is a textbook way to strand free space. If that's a known shape to you, the experiment may be unnecessary.

2. Has a non-default allocator been considered and rejected? jemalloc/mimalloc with decay purging is the obvious remedy for fragmentation-pinned space, and it's a #[global_allocator] change with real consequences — binary size, the panic = "unwind" interaction with ONNX Runtime, platform matrix. If there's a standing reason not to, I'd like to know before I spend a run producing evidence for it.

3. Would you shape the measurement differently? My rig is subc_storm_test.rs — it already drives binds, tool traffic, watcher churn, and semantic collection across N roots via AFT_STORM_SCALE, and it spawns its own ephemeral daemon so nothing production-facing is touched. One known weakness: it runs the module in-process, so malloc_info measures the whole test binary. I'm handling that with start/end deltas plus a zero-workload control arm to measure the scaffolding's own contribution, with a dedicated bin target as fallback if that control shows the scaffolding isn't constant.

Design in brief — four arms (scaffolding-only, stock, MALLOC_ARENA_MAX=2 as a negative control on the rig, decay-purging allocator), malloc_info + mallinfo2 + smaps census at start and end, predictions registered before running so the outcome can't be interpreted after the fact. Full design doc available if useful; happy to paste it here or open it as a PR against a docs path if you'd prefer it in-repo.

What I'm not doing: proposing an allocator change. That would need this evidence first, its own gate cycle, and your agreement that it's a direction worth taking at all.

Context on severity, so this is priced correctly: after #202 landed (inspect threads 88 → 8 on the same daemon) the live instance sits at ~2.9 GB swap after 12 h, against 15.3 GB at 18 h before. It's churn-bound rather than time-linear — regions and VmData go flat across idle hours. So this isn't urgent, and I'd rather spend the effort in the direction you'd actually use.

[ualtinok]

Answers to all three, in order — and the short version is: build it, with one arm-sharpening addition.

1. Prior context on the allocation pattern: yes, and it narrows your experiment rather than obviating it. The long-lived per-root artifacts you named are the known shape: each root independently owns a search-index snapshot (postings blob + delta), semantic embeddings, callgraph store buffers, and symbol caches, and they arrive/depart on independent schedules (bind, watcher churn, idle eviction, reconfigure) with sizes spanning KBs to hundreds of MB. Interleaved lifetimes of differently-sized long-lived buffers is exactly the textbook stranding pattern — so "one pathological site" is unlikely to be the answer; "a structural pattern spread across per-root artifact lifecycles" is the prior. Your per-arena/per-bin distribution measurement still decides it, because the prior doesn't tell us whether the stranding concentrates (fixable at a site) or smears (allocator-level remedy).

2. Non-default allocator: not rejected on principle — never seriously proposed, because the standing costs are real and known. Three we'd weigh against any evidence you produce: (a) ONNX Runtime interaction — we load onnxruntime dylibs in-process and the panic = "unwind" + foreign-allocator combination across that FFI boundary is exactly where allocator swaps produce the subtle class of bug; (b) platform matrix — the swap lands on macOS/Windows too (or gets cfg-gated, which forks allocator behavior across the platforms we debug on); (c) binary size and supply chain on a shipped-binary product. None of these is a veto; all of them mean the evidence bar is "distribution data showing decay-purging fixes what an arena cap cannot", not "fragmentation exists". Your reordering (arena-cap outcome checked before allocator-swap) is exactly the right sequencing from our side.

3. Rig shape: sound, one addition. Storm rig + zero-workload scaffolding control + start/end deltas is right. Add to arms A and D a malloc_info(3) arena-count and per-arena free-distribution capture (not just aggregates) — A-vs-D is your cleanest causal read and the per-arena shape is what separates "many low-occupancy arenas" (arena cap wins) from "few arenas, deep internal fragmentation" (cap does nothing, decay-purging or site-fix). Also pin thread counts in the report (your 128-arenas-vs-150-threads ceiling observation is the kind of number that should ride along).

On the pool-size override (from #201): yes, and we'd take it as your PR — env var, not config field. Exact shape so it lands first-pass:

• AFT_INSPECT_POOL_THREADS read once at pool init in dispatch.rs::default_pool_size(), parsed as usize, clamped to max(1, n), capped at 512 as an absurdity guard; absent/unparseable → current derivation unchanged.
• One comment at the site naming its purpose (reproducing thread-regime-dependent behavior; not a tuning surface) and explicitly NOT documented in docs/config — it's a dev knob, same class as AFT_STORM_SCALE and AFT_SEMANTIC_QUIET_WINDOW_MS.
• A test asserting override wins over derivation and the clamp floor (the existing pool tests give you the harness; note the pool is process-global LazyLock, so the test needs the env set before first pool touch — same pattern the storm-scale tests use).
• Config-field version declined deliberately: pool size is derived behavior with a documented contract, and our standing rule is no low-value config knobs.

Looking forward to the distribution data.

[iceteaSA]

Ran it. The per-arena data points at a bounded arena count — but it contradicts a production measurement I reported to you two days ago, and that contradiction needs to come first because you'd reach it in one step anyway.

The contradiction

On #201 I told you MALLOC_ARENA_MAX=2 was measured ineffective in production: 0.95 GB/h swap accumulation against a 0.85 GB/h uncapped baseline, wrong direction. We removed the cap on that basis.

The rig says the same cap is decisive: 65% less retained free space, and the low-occupancy arena population goes to zero.

I cannot adjudicate that from the data I have. Three candidate reconciliations, none confirmed:

1. Different observables. Production measured VmSwap rate under rising box pressure with 3.5× uptime skew and a baseline captured by a different observer. The rig measures allocator-slack distribution directly. A real reduction in retained free need not show up in a noisy swap-rate series.
2. Synthetic churn vs real artifact lifecycles. The rig drives the storm harness — real AppContexts, real binds, real watcher churn — but 6 roots × 2 sessions for 90 s is not a 41-hour daemon serving real work.
3. The production reference was too confounded to mean anything. It ran the pre-8bd35ff4 binary at 211 threads; the rig runs the post-fix regime.

Explanation 3 was my working assumption until the A-vs-D result, which weakens it — see below.

Numbers

Storm workload, 6 roots × 2 sessions, 90 s, release, each arm in its own systemd-run --scope -p MemoryMax=14G -p MemorySwapMax=0. Four runs per arm.

arm	config	peak threads	arenas	retained free	system	free frac	arenas ≥0.9 free
0	scaffolding, no workload	3	3	0.36 MB	0.39 MB	—	3
A	stock	60	16	28.6 MB	35.3 MB	0.810	9.8 of 16, holding 83% of all free space
B	MALLOC_ARENA_MAX=2	62	2	10.1 MB	20.8 MB	0.486	0 of 2
D	AFT_INSPECT_POOL_THREADS=88	143	16	27.8 MB	33.4 MB	0.79	8.5 of 16

Arm A's end-state distribution, one representative run:

nr   system kB   rest kB   free_fraction   bins
 0      5944       5806        0.977         32
 1      4308        973        0.226         65   ← the only working arena
 2      5916       5773        0.976         42
 3       148        139        0.941          2
 …
15     11772      11242        0.955        115

Fifteen of sixteen arenas sit at ≥0.86 free. One is doing the work.

The finding that complicates your mechanism

Arena count saturates at 16 regardless of thread pressure. Arm D ran 2.4× the peak threads of arm A (143 vs 60) and produced an identical arena count and no material difference in retained free (27.8 vs 28.6 MB).

That cuts against the arena-per-contending-thread story as the whole explanation. If the count saturates well below thread count, then 8bd35ff4's 88→8 inspect-thread cut did not help primarily by reducing arenas. It also weakens reconciliation 3 above: production-B had 2 arenas too, and still accumulated swap faster than baseline.

16 equals this host's core count. I have not established that as causal and am not going to guess at it — flagging it as unexplained rather than smoothing it.

The ask, and the part I'd rather you decide

mallopt(M_ARENA_MAX, N) at daemon startup, if you still want that conversation.

The value is genuinely open and I don't think 2 is right. 2 was the maximum-signal probe number, not a production recommendation. Arm A's data says 16-with-10-idle, which argues for well below 16 — but 4, 8, or cores/2 are all defensible and I have no data separating them.

On contention: 2 arenas under 60+ concurrent threads is exactly the configuration that trades lock contention for footprint, and latency was a non-goal for the experiment but cannot be a non-goal for the remedy. One weak data point in the cap's favour: arm B passes the storm harness's own bind-latency assertion (max bind ack ≤ 2 s across all binds) at 62 peak threads. That is a bound, not a measurement — it says the cap didn't blow the existing budget, not that p50 was unaffected.

If you want a latency arm before deciding, say so and I'll add one rather than ship a footprint win that costs tail latency.

What this does not license

Anything about decay-purging allocators — arm C wasn't run. Its registered trigger required the arena cap being ruled out first, and the cap works, so building a second binary would have been doing the expensive work to answer a question this run closed. Also nothing about which of the three changes was operative in production, and no extrapolation of these absolute numbers (tens of MB) to the 29.5 GB the production daemon reached.

One registered stop clause misfired and I'm retiring it in the open rather than quietly: the design said "rig does not reproduce if arm B comes out materially better than production measured it." It did. But that clause depended on arm B being a clean control, and your arena-per-thread correction had already downgraded B to a rough sanity check — the same correction invalidated the clause, and I failed to update it at the time. The governing half doesn't fire: arm A grew retained free 0.24 → 28.6 MB and arenas 2 → 16, so the rig reproduces.

Full writeup with method and threats to validity available if useful.

[iceteaSA]

Chased the unexplained ceiling from my last comment rather than leaving it flagged. It's explained, and it changes how the A-vs-D null should be read.

16 is glibc's default arena limit on this host — reproducible, and independent of how much contention you apply. Standalone C probe, N concurrent allocating threads then malloc_info, glibc 2.44, 16 cores:

concurrent allocators:   1   2   3   4   15   16   20   64   96  128  256
arenas (default):        2   3   4   5   16   16   16   16   16   16   16

Low counts give exactly main + N, which validates the counting. From 16 up it pins and never moves — not at 256 concurrent allocators.

It's a real limit, not a workload artifact. Raising it explicitly moves the ceiling in lockstep:

MALLOC_ARENA_MAX=  4  ->  4 arenas
                  16  -> 16
                  32  -> 32
                  64  -> 64
                 128  -> 65    (= 64 threads + main; threads bind before the limit does)

Two things worth having on the record:

It is not 8 × ncores. The documented default would be 128 here. Observed default is 16, and an explicit 128 reaches 65 — so the mechanism can exceed 16, the default just doesn't. I don't know whether that's a distro build-time choice or something in the 2.44 defaults, and I'm not going to guess at it.

CPU affinity doesn't move it. taskset -c 0-1 through 0-15 all give 16 at 64 concurrent allocators, so it isn't computed from the affinity mask at runtime.

Still open: 16 equals this host's core count, and one machine can't separate "16 = ncores" from "16 = a constant". A different-core-count host settles it in one run. It doesn't matter for the remedy — the cap gets set explicitly either way — so I've recorded it rather than chased it further.

What this does to the A-vs-D result I reported: both arms were already pinned at the ceiling, so thread count couldn't have moved arena count in either. The null is what the mechanism predicts once you know the ceiling exists — not evidence against arena multiplication. I framed it as complicating your mechanism story; that was too strong, and this is the correction.

The narrower true statement: the 88→8 inspect-thread cut could only have reduced arena count if it dropped concurrent allocators below 16. Production data doesn't tell us whether it did.

Doesn't change the ask. mallopt(M_ARENA_MAX, N) still follows from the free-space distribution — 15 of 16 arenas at ≥0.86 free with one arena doing the work — and the value question is still open, with 4/8/cores÷2 all defensible and this run separating none of them.