perf(daemon): lean idle memory — kill the GC churn, bound the envelope, scope the global passes

[zzet]

Make the daemon lean at idle: kill the GC churn, bound the envelope, scope the global passes

Problem

On a 16 GiB machine the daemon's Activity Monitor footprint sat at 13.7–16.2 GB (pinned at its peak, 96% of it swapped/compressed) with 27–77% CPU while nominally idle. Symbolicated CPU samples showed ~95% of that burn inside the garbage collector (mark/sweep), with SQLite page-cache fetches as the allocation driver: every MCP query allocated enough to keep the GC cycling over a multi-gigabyte heap, and the heap's high-water mark was never returned to the OS.

Live pprof of a running daemon identified the causes this PR fixes:

1. An allocation firehose in config. loadRepoGitignore, called through ConfigManager.EffectiveExclude on every indexer walk and per-file reconcile, re-opened and re-parsed the repo .gitignore on each call — a fresh 64 KiB scanner buffer every time. Measured: 47% of all process allocations (170 GB alloc_space in 17 minutes of life).
2. Whole-graph global passes for one-repo changes. The end-of-batch passes (framework dispatch synthesis, clones, capability edges, implements/overrides, test edges, external calls) re-scanned the entire 26-repo graph whenever ≥1 file changed — at warmup and again on every hourly janitor tick. Measured on a live warmup: 503–1601 s of passes, fn-value-callback alone 264–276 s, and clone detect/rebuild materializing every node in the graph (with meta) three times to process a single changed repo. store_sqlite.scanNode accounted for 107 GB of cumulative allocations.
3. No memory envelope. The daemon set no soft memory limit (GOGC defaults, GOMEMLIMIT unset) — the only tuning lived inside the cold-index window — and nothing ever called a scavenge at burst boundaries, so one balloon pinned the footprint at peak indefinitely.
4. Count-capped caches with unbounded bytes. The sqlite symbol-bundle cache admitted 50,000 entries regardless of size; hub-node bundles run to multiple MB each.
5. Phantom mtime rows. The watcher's delete/rename patch evicted a file's nodes but left its persisted mtime row behind, so every warm restart re-discovered vanished paths as phantom deletions.

Fixes (one commit per cause)

• perf(config): two-layer cache for EffectiveExclude — the .gitignore parse keyed by repo root and invalidated by stat (mtime+size), the layered merge keyed by repo prefix and versioned by config pointer identity. Hot path: 68,672 → 432 B/op, 30 → 3 allocs/op (the residual is one os.Stat). Returned slice is shared, immutable, and clipped so appenders reallocate.
• feat(daemon): a standing soft memory limit at boot — GOMEMLIMIT (respected verbatim if set) > GORTEX_DAEMON_MEMLIMIT > daemon.memory_limit config > default hostRAM/4 clamped to [1 GiB, 8 GiB] — plus debug.FreeOSMemory() at the four burst boundaries (warmup complete, janitor ticks that did work, cold-index window close, MCP whole-graph analysis), so a burst's high-water is handed back instead of pinning the footprint. The cold-index tuning window captures and restores the standing limit. Kill-switch: GORTEX_DAEMON_MEMRELEASE=0.
• perf(indexer,resolver): the global passes take the already-armed changed-repo scope: implements/overrides run their scoped variants (a pair survives when either endpoint is in scope, so cross-repo pairs are kept), capability/test-edge/external-call sweeps drive off per-repo edge readers, clone detect/rebuild read GetRepoNodes(prefix) instead of AllNodes(), and the two worst synthesizers (fn-value-callback, value-ref) scan candidates per-repo while resolving against the whole graph. value-ref also builds its declarator maps only for files that actually carry candidates (a win even unscoped), and fn-value memoizes its name lookups per pass. A nil scope (fresh index) is byte-identical to the old behavior.
• fix(store): the bundle cache is bounded by bytes (64 MiB default, GORTEX_BUNDLE_CACHE_MAX_MB, ≤0 disables) with a deliberately over-counting size estimator; the 50k count cap remains as a secondary guard.
• fix(indexer): the watcher's delete/rename patch now prunes the file's mtime row (in-memory first, then the store, mirroring the durability order of the add side), so warm restarts stop re-processing vanished files.

Verification

• go build ./... and go test -race green across every touched package (config, platform, cmd/gortex incl. the wire-contract golden, indexer, resolver, mcp, graph conformance suites) on the assembled branch.
• New tests: cache-hit-never-rereads (counting reader) + invalidation + clip safety; memory-limit policy resolution table; release kill-switch; byte-budget eviction boundary + concurrent race hammer; scoped-vs-unscoped parity fixtures for cross-repo implements/overrides, fn-value binding into an unchanged repo, and clone reads never materializing a sibling repo; watcher delete-path mtime pruning (5 tests).
• Live measurements on the reporting machine (16 GiB, 26 tracked repos): see the PR discussion for the before/after warmup + idle profile from the deployed build.

Operator notes

• New knobs: daemon.memory_limit (config), GORTEX_DAEMON_MEMLIMIT, GORTEX_DAEMON_MEMRELEASE=0, GORTEX_BUNDLE_CACHE_MAX_MB. Defaults need no configuration.
• An explicit GOMEMLIMIT in the daemon's environment always wins; the daemon then sets nothing.
• The hourly janitor still runs every pass — it just reads the changed repos instead of the whole graph. A fresh full index is intentionally unscoped.

[zzet]

Live before/after on the reporting machine — 16 GiB Apple Silicon, 26 tracked repos, sqlite store, measured with /usr/bin/footprint, sample, and the daemon's own pprof endpoint.

Baseline (pre-branch binary, same store, same workload):

metric	value
physical footprint at idle	13.7–16.2 GB, pinned at the process peak (96% swapped/compressed)
idle CPU	27–77% — samples ~95% GC mark/sweep
warmup, ≥1 repo changed	44 min total: parse 460–496 s, global passes 503–1601 s
worst passes	fn-value-callback 264–276 s · clones 621 s · capability edges 278 s
top allocator	config.loadRepoGitignore — 47% of all allocations (170 GB in 17 min)

This branch (deployed at f6da4fb6, daemon restarted twice to measure both boot shapes):

metric	value
standing memory limit	applied at boot: 4 GiB (hostRAM/4, source=default)
boot, nothing changed	graph queryable in 9.3 s, global passes skipped
boot, 1 repo changed	~6 min total: parse 11.8 s, queryable 109 s, global passes 175.8 s
scoped passes	fn-value-callback 175 ms · value-ref 90 ms · clones 0.49 s · capability 5.8 s · implements+overrides 3.1 s
remaining pass cost	the deliberately-unscoped tail: macro-expansion 46.3 s + ~40 small synthesizers at 1–6 s each
post-warmup release	released heap to OS reason=warmup_complete fired; footprint settled within a minute
footprint at idle	1.35 GB (stable across samples)
idle CPU	~0%

Peak transient footprint during the changed-repo warmup stayed under 3 GB (resolve + enrichment overlap), versus 6.7–16.2 GB before.

Notes:

• The negative freed_bytes value visible in the first deployed build's release log (concurrent allocation re-acquiring pages between the two MemStats reads) is clamped by 471adbba, which landed after that binary was cut.
• The gitignore cache benchmark in the PR body (68,672 → 432 B/op) is the microbenchmark counterpart of the alloc_space drop; the live confirmation is the idle CPU falling to ~0%.