Bound Hermes import and FTS merge WAL growth

[j2h4u]

What changed

• batch partial Hermes imports at 64 logical units or 8 MiB of serialized input, including deferred edges and file touches
• keep event rows and both FTS projections transactionally consistent and immediately searchable
• persistently suppress FTS5 automatic/crisis merges during Hermes bulk ingest
• compact FTS segments afterward in bounded 1,024-page transactions with strict WAL checkpoints
• serialize bulk imports and optimization with a crash-releasing cross-process guard
• automatically recover interrupted bulk state on the next writable store open
• preserve source-atomic behavior when partial import is disabled

Why

The original transaction-batching draft did not fully solve the reported failure. On the contributor's 206–260 MB Hermes corpus, FTS5 could still perform a crisis/incremental merge inside a single INSERT and grow the WAL by multiple GiB, even with one capture per transaction. A checkpoint cannot bound work performed inside that statement.

This replacement prevents those automatic merges during ingest and performs the merge work explicitly in bounded, recoverable transactions.

Validation

• cargo test -p ctx-history-store -p ctx-history-capture -p ctx --no-fail-fast
• cargo clippy -p ctx-history-store -p ctx-history-capture -p ctx --tests -- -D warnings
• cargo fmt --all -- --check
• scripts/check-loc.sh
• crash recovery, pinned-reader, overlapping-import, optimizer-race, byte-budget, search-consistency, and idempotent-replay regressions
• monitored CLI import of 1,100 Hermes messages with approximately 16 KiB content each: 18.6 MB source, 5.39 MB peak WAL, zero final WAL, 1,100 events imported
• independent final review found no remaining P0/P1 issues

The contributor's private 206–260 MB corpus was not available for direct reproduction.

[j2h4u]

One process question for future contributions: is it generally okay to open a focused PR like this directly, without filing a preliminary issue first? I do not want to overuse maintainers’ trust or create review work in the wrong form. Happy to follow whichever issue-first vs. PR-first workflow you prefer.

[j2h4u]

Benchmark update: the initial batching approach is not sufficient, so please do not review the current draft yet. On a 206–260 MB Hermes database, WAL still reached multiple GiB even with one capture per transaction. The root cause is FTS5 merge work performed inside the INSERT itself (including crisis/incremental merges), which checkpoints cannot bound mid-statement. I am revising the PR toward deferring search projection during bulk ingest and rebuilding/merging it in bounded transactions, then will replace the draft implementation and update the measurements.

[j2h4u]

Real-corpus follow-up after merge (corrected after tracing the setup call path):

• Host binary: current upstream after Bound Hermes import and FTS merge WAL growth #124 (e2f1a3ec)
• Hermes source: 206.0 MiB, 250 sessions, 11,922 events, 83 edges
• Clean ctx setup --no-daemon --wait --progress plain already sets partial: true; my earlier statement that setup selected allow_partial_failures=false was incorrect.
• Despite using the partial policy, the clean multi-source setup run still grew the WAL to 7.0 GiB during the Hermes/bulk-FTS phase in 28m26s before I stopped it for disk safety. Hermes remained pending; the database passed integrity_check after rollback/checkpoint.
• Running Hermes explicitly against the preserved populated database succeeded:
  ctx import --provider hermes --path ~/.hermes/state.db --partial --no-daemon
  • elapsed: 38.35s
  • max RSS: 193,972 KiB
  • imported: 250 sessions / 11,922 events / 83 edges
  • failures: 0
  • final WAL: zero/absent
  • no lingering event_search_bulk_mode_v1 recovery markers
  • integrity_check: ok
• Idempotent replay completed in 15.43s, imported 0 new rows, skipped 12,255, and retained exactly 11,922 Hermes events.

So the bounded partial implementation works very well for the isolated Hermes import, while the clean full-setup path still exposes a real-corpus WAL spike somewhere in the surrounding bulk/FTS workflow. This is separate from the product-semantics question in my following comment about whether source-atomic/non-partial import should remain supported.

[j2h4u]

@luca-ctx one product-semantics question before I prepare any follow-up.

Should source-atomic/non-partial provider import remain a supported mode at all?

For a disposable, idempotently rebuildable local index, it seems useful to separate two concerns that --partial currently couples:

1. Transaction strategy: always commit bounded batches so a large provider source cannot create an unbounded WAL and completed work survives interruption.
2. Error policy: decide whether malformed rows make the command fail/report a partial result, while retaining already committed valid rows.

The non-partial Hermes path preserves true all-or-nothing source atomicity, but on the real 206 MiB corpus it produced a 7 GiB WAL before rollback. Conversely, the batched partial path imported all 250 sessions / 11,922 events / 83 edges in 38.35s with zero failures, no final WAL, and an idempotent replay.

Would you prefer to:

• remove source-atomic import and make bounded commits universal, with strictness affecting only reporting/exit status; or
• retain source atomicity as an explicit product guarantee and continue hardening that path separately?

This is your product decision, so I would rather follow your intended semantics than infer it from the current CLI shape.