perf(code-index): bound pre-seat text projection memory below admission budget

[ScriptedAlchemy]

Defect

The preserved operator profile on exact pushed PR #707 commit 250c5e01b reached 26.31 GiB RSS before any project graph engine opened, sealed-store publication occurred, or a graph generation seated. This is a pre-seat text/code-index projection working-set defect, distinct from #799's retained Grafeo-engine peak and #830's post-publication release boundary.

Observed live progression for generation 29:

• 1.82 → 8.58 → 13.15 → 18.97 → 22.36 → 23.74 → 26.31 GiB RSS
• graph file remained 2,361,815,040 bytes with no WAL
• shutdown cancelled inside the published text loop; the graph gate is evaluated only afterward
• profile was preserved and the daemon was stopped through its normal lifecycle before OOM

A prior Hotpath allocation capture on the same ~4–5k-file shape attributes the largest overlapping owners to:

• code_index.generation.decode.bundle: 7.4 GB
• publish.segment_encode: 10.8 GB
• decode.segment: 1.7 GB
• restore.file_admit: 1.4 GB
• incremental materialization: 1.1 GB
• load_active deep clone: 533 MB

The decoded active generation, restored file artifacts/evidence, incremental candidate, whole-generation encoder payload/segment/index marker buffers, and one text-source batch can coexist. The current 1.5 GiB ledger bounds only batch/cache estimates; it does not account for the decoded generation, encoder, SQLite/allocator overhead, or arena retention. 0a3642dd4 only changes scheduling throughput and 63777d1d0 only improves diagnostics; neither bounds these owners.

Required outcome

• Decompose and eliminate overlapping whole-generation owners across decode, active-generation loading, incremental materialization, partitioned sealing, and text publication.
• Preserve exact generation/source identity, content-addressed validation, cancellation, rollback, and the existing admission ceiling.
• Make the actual end-to-end working set observable and admission-accounted; do not raise limits or mask the failure with stale RSS telemetry.
• Keep perf(code-index): make incremental capture and seal delta-proportional #800's delta-proportional persistence work distinct, but reuse it where it genuinely deletes whole-generation work.

Acceptance evidence

• A scale regression that includes active-generation decode, one-file incremental materialization, segment sealing, and text publication—not an isolated batch estimator.
• Hotpath/allocator evidence showing the former whole-generation buffers no longer overlap.
• Re-run the preserved-profile generation-29 journey: peak and settled RSS remain below the existing 18.84 GiB high watermark and the generation reaches its truthful seated state without daemon restart.
• Existing tamper, restart, cancellation, rollback, and exact-generation tests remain green.

Related: #799, #800, #830, #847.

[ScriptedAlchemy]

Calibration: the 7.4/10.8 GB Hotpath allocation figures are historical phase/multiplicity evidence from before e95f373 introduced streaming/reused buffers; they are not exact attribution for the 250c5e0 live run. Exact 250 evidence proves a composite pre-seat text/index working set and no graph open. The current codec still simultaneously owns full evidence payload/rewritten segment plus whole-generation marker maps, and the current 2,600-file codec benchmark still reaches about 2.23–2.25 GB VmHWM with 12.1 GB encode and 16 GB decode allocations over five iterations. Acceptance therefore requires a fresh exact-tip phase capture rather than treating the historical byte totals as current.

[ScriptedAlchemy]

First root-cause codec slice is implemented and pushed on codex/issue-852-evidence-shards at f7a348727 (rebased onto PR-tip 0aab74a74).

What changed:

• generation evidence is now one deterministic JSON stream emitted directly into ordered 256 KiB content-addressed shards;
• the manifest authenticates every shard plus the aggregate byte-exact stream digest/length;
• decode verifies and feeds one reusable shard at a time directly to serde, instead of joining/re-writing the full evidence;
• whole-generation payload + rewritten segment buffers and repository-wide symbol/chunk marker maps are gone;
• file encoder buffers are released before evidence begins; unchanged parent file segments keep their existing content addresses.

Fresh exact-branch regression evidence:

• RED cc-3791: the old encoder retained the complete 8,391,004-byte rewritten segment;
• GREEN synthetic fixture: 8.39 MB is emitted as 33 shards; the live-owned gauge is exactly one 262,144-byte shard plus descriptor slots/digest payloads (267,047 bytes), and the test requires that to remain under 1/8 of total evidence bytes;
• multi-shard concatenation is byte-exact, aggregate digest verifies, typed decode round-trips, and a tampered shard is refused.

Post-rebase checks:

• cc-3970: codec unit module, 6/6 passed;
• cc-3971: stable bytes/typed round-trip + one-file parent reuse, 2/2 passed (non-vacuous);
• cc-3972: cargo clippy -p tracedecay-code-index --lib --tests -- -D warnings, passed.

The small golden evidence payload grows from 19,548 to 22,960 bytes because original identities are retained instead of building the removed global marker dictionaries. File segment bytes are unchanged. This is the codec-owner memory slice only; #852 should remain open for the composite end-to-end working-set accounting and preserved-profile generation-29 rerun.

[ScriptedAlchemy]

Superseding the earlier f7a3487 evidence: the per-page durability design was replaced in 9741fa3 (codex/issue-852-evidence-shards) with one content-addressed evidence pack containing authenticated 256 KiB pages. Format 7 changes in place. Decode/verification range-read one page at a time; the manifest retains ordered page digests plus aggregate digest/size.

Production evidence is now bounded: a multi-page seal performs one evidence sync/rename/directory-sync transaction, while an identical retry performs zero. Failure after all streamed pages through the real atomic publisher leaves no temporary pack, generation manifest, or active pointer. Locked restart orphan cleanup, missing-byte refusal, aggregate/page tamper checks, parent file-segment reuse, and strict retention of the active pack plus collection of the retired pack are covered.

Fresh checks after rebasing on 8b6d521:

• cc-4201: focused codec tests, 7/7.
• cc-4202: production multi-page publish/restart/truncate journey, 1/1.
• cc-4203: strict unchanged-parent reuse and retention boundary, 1/1.
• cc-4197: real atomic failure-after-pages rollback, 1/1.
• cc-4156 / cc-4158 / cc-4159: affected-package clippy -D warnings, check/all-targets, and bench target compile green.

Fresh 2,600-file benchmark cc-4177 (9,284,370 corpus bytes, 10 replicas, 5 measured iterations): VmHWM 1,883,029,504 bytes; encode p50/p95 4.143/4.407 s; decode+open p50/p95 9.652/9.934 s. Hotpath tracked 10.1 GB total allocations, including 3.2 GB in segment encode and 5.3 GB in decode bundle across five iterations. The retained-byte regression separately proves the encoder owns one 256 KiB page plus compact descriptors, below one eighth of an 8+ MiB evidence stream.

The independent focused rereview found no remaining blocker after the end-to-end failure test was added. Keeping #852 open until this lands in the integration branch and the full exact-profile/live journey confirms the composite peak.

[ScriptedAlchemy]

Follow-up fix pushed in 41d59bad6 on codex/issue-852-evidence-shards.

The packed-evidence publication now closes the post-pack-rename/pre-manifest failure window in two layers:

• publication keeps a newly committed pack rollback-owned until the manifest rename and directory sync complete, so ordinary manifest failures remove the unattached final pack immediately;
• crash debris is reclaimed by the canonical generation-retention transaction even when no generation is collectable. Publication-store construction no longer attempts final-segment cleanup because it lacks graph-replay liveness authority.

The retention mark/sweep is bounded-memory and lock-safe: it streams segment descriptors, authenticates complete generation/replay manifest bytes against canonical or hidden staged-unlink filename digests, validates descriptor cardinality/occurrence keys and evidence page ordering/bounds/aggregate size, and acquires the graph-replay pool lock before unlinking. Active, retained, parent-reused, canonical replay, and hidden staged-replay references remain marked.

Fresh evidence:

• RED cc-4399 / GREEN cc-4422: unlocked hidden staged-replay window is marked and corrupt same-name replay bytes fail closed.
• GREEN cc-4423: real multi-page post-pack/pre-manifest failure rolls back; simulated crash debris survives constructor reopen and is removed by canonical maintenance without a manifest/pointer.
• GREEN cc-4425: retention library 71/71, including no-generation-delete sweep and replay interleavings.
• GREEN cc-4428: partitioned codec 8/8, including structural under-report refusal and bounded page retention.
• GREEN cc-4429: production multi-page one-pack durability + restart.
• GREEN cc-4430: exact parent segment reuse plus orphan sweep preserving all live segments.
• GREEN cc-4426 affected-package all-target check; cc-4427 strict all-target clippy.
• Focused rereview: no Critical or Warning findings.

Keeping #852 open for integration and the separately required live-profile acceptance evidence.

[ScriptedAlchemy]

Follow-up lifetime-race fix pushed in cd6abd954 (codex/issue-852-evidence-shards).

The replay/canonical readers now acquire the matching generation-store or replay-pool lock before opening the manifest, open and identity-validate the single evidence pack on the first range request while that lock still proves liveness, then release the lock and serve every bounded evidence page from the pinned file handle. Source verification uses the same lifetime handoff. Canonical scheduler decode follows the same rule; cold publication hydration happens before the writer lock and the exact expected pointer is rechecked under that lock.

Fresh evidence:

• RED cc-4471: staged manifest finalized + no-op retention sweep unlinked the pack, next pathname range open failed with ENOENT.
• GREEN barrier cc-4489: 1/1; retention unlinks manifest and pack pathname mid-decode, all pages decode and verify through the pinned handle.
• replay/source-verification module cc-4496: 9/9.
• multi-page restart cc-4494: 1/1.
• parent-segment reuse cc-4495: 1/1.
• replay-pool lock/retention cc-4498: 7/7.
• partitioned codec cc-4499: 8/8.
• failure-before-pack-commit cc-4500: 1/1.
• affected package check cc-4501: 0 errors/warnings.
• affected package clippy cc-4502: clean with -D warnings.
• independent focused rereview: CLEAN.

Keeping #852 open pending integration and live profile evidence.

[ScriptedAlchemy]

Typed replay interruption follow-up pushed in 1ff582429 on codex/issue-852-evidence-shards (separate from cd6abd954).

Revision-7 partitioned decode and source verification now retain callback Cancelled / DeadlineExceeded as typed GraphDbError values and return them before the generic codec-contract-to-corruption mapping. Other callback and codec failures remain fail-closed corruption. The lifetime lock and pinned evidence-pack handoff are unchanged.

The earlier manifest-reopen allegation was withdrawn after confirming the sole production caller already brackets verification with stable-source prove/revalidate authority; this commit contains no manifest-reopen behavior or test.

Fresh TDD evidence against pool-only 1,600-symbol, multi-page seals, armed after the first evidence Range so interruption occurs on the next Range callback:

• RED decode Cancelled cc-4531; GREEN cc-4540.
• RED decode DeadlineExceeded cc-4535; GREEN cc-4541.
• RED verify Cancelled cc-4536; GREEN cc-4542.
• RED verify DeadlineExceeded cc-4532; GREEN cc-4543.
• full manifest module cc-4559: 13/13.
• affected package check cc-4560: zero errors/warnings.
• affected package clippy cc-4561: clean with -D warnings.
• fresh independent rereview: CLEAN.

#852 remains open pending central integration and live profile evidence.

[ScriptedAlchemy]

Central integration update: the complete packed-evidence/lifetime/interruption stack is now on PR #707 at 6030aa639 (component commits ec4c07649, e65e9503c, 2c932589f, 77216e896). The rewritten central branch is clean and pushed. Focused manifest/codec/retention tests and strict checks are green. Keeping #852 open for fresh PR dogfood and the guarded real-profile restart/RSS acceptance; no budget or timeout was raised.

[ScriptedAlchemy]

Live heap attribution for the pre-seat working set, measured on the current PR-tip binary (6030aa6, dev and perf profiles) indexing the tracedecay checkout (4,936 files, 68 MB source) under a 4-CPU systemd scope, sampled with gdb -batch -ex 'call (int)malloc_info(0, (void*)stderr)' and summed per heap:

phase	RSS	glibc arenas (system)	live (in use)	free-but-retained
page pass (files 300/4,936)	7.2 GB	8.5 GB	3.5 GB	4.9 GB
seal / seat (phase ready, serving pending)	13.6 GB	13.4 GB	7.8 GB	5.6 GB

The shipped binary uses glibc malloc (allocator features are opt-in), and 56 per-thread arenas never return interior free chunks. One malloc_trim(0) on the live daemon dropped RSS from 6,905 MB to 3,041 MB immediately, mid page pass. That freed-but-retained share is what tripped daemon_resident_memory_over_budget three times during the seal in the same run (12.7, 14.5, 14.4 GB observed) with nothing reclaimed.

Landed on the #707 branch as 875698e: the allocator trim is registered as the last pressure reclaimer of the process cell (runs after the state reclaimers, reports released bytes as process_allocator_trimmed), and a 30-second RSS sampler publishes measurements on its own cadence — the retention tick that used to publish them is hours apart, so a cold index reached the watermark with no sample.

Still live data, not fragmentation, and untouched by this change: the whole generation's FileGenerationArtifactsV1 (every chunk's subtokens: Vec<String>, exact_terms, sanitized text, plus ExactExtractionAuthorityV1 digest strings) stays resident in memory_files for the entire page pass, and admission re-serializes each chunk canonically and re-hashes it (validate_chunk) before JSON-serializing it again for the page. That is the 3.5 GB live figure at page time and the base of the 7.8 GB at seal.