perf(history): resume JSONL from bounded seams

[Harry19081]

Problem

Append-only JSONL importers persisted a whole-prefix hash. Every append therefore sought to the saved offset but still reread and rehashed all previously committed transcript bytes before accepting the watermark, making warm ingestion O(total history) instead of O(delta).

The shared line reader also allowed one malformed or hostile JSONL record to grow its buffer without a hard bound.

Solution

Replace the persisted whole-prefix digest with a versioned, fixed-size resume seam while keeping the existing database column and parser state:

• store platform file identity plus the FNV-1a hash and length of at most the final 4 KiB before the last complete-line offset;
• resume only when parser version, file identity, monotonic size and mtime, saved offset, and the 4 KiB boundary fingerprint all match;
• treat legacy whole-prefix hashes, same-size rewrites, shrink, rotation, parser changes, identity changes, and seam mismatches as a cold parse of that one file;
• leave final unterminated lines outside the committed offset and state so an active writer can finish them safely;
• enforce a 1 MiB raw JSONL line cap before extending the read buffer;
• keep persistence schema-compatible by encoding the new fingerprint as JSON in the existing prefix_hash column.

Unchanged metadata remains O(metadata) at discovery; an eligible append reads O(4 KiB + appended bytes). No timer, watcher, worker, or new retained cache is added.

Potential risks

• The first read of a legacy watermark intentionally cold-parses that file once and self-heals the stored fingerprint. No table migration is required.
• A 1 MiB physical JSONL line is now rejected. This prevents memory spikes but may reject an unusually large legitimate single-line tool payload; the existing last-good cache and watermark are not advanced on error.
• The constant-size seam detects rotation and mutations near the committed boundary, but cannot prove that bytes far earlier in a growing same-identity file were not rewritten. Same-size mtime changes, shrink, identity change, and boundary mutation do force cold parse. Full arbitrary-prefix verification would restore O(total history) work.
• Filesystems that cannot provide a stable file identity disable resume and cold-parse safely.
• The fingerprint is an integrity check, not a cryptographic adversarial guarantee.
• Rollback is a revert of this commit. The older reader will treat the JSON fingerprint as a mismatched legacy hash and perform one cold parse before restoring its older format.

Audit

Performance guard verdict: pass. Idle cost is zero; unchanged discovery performs no transcript read; append validation reads at most 4 KiB before the delta; per-line memory is capped at 1 MiB; and reset scope is one changed file.

Architecture review covered watermark ownership, file and parser identity, resume/reset state transitions, persistence compatibility, incomplete-line lifecycle, error and last-good semantics, platform fallbacks, and focused test seams. Provider formats and frontend behavior are unchanged.

No screenshot was captured because this is a backend ingestion primitive with no UI change.

Verification

• Clean detached-worktree validation at commit c94db52: CARGO_BUILD_JOBS=1 cargo test -p orgtrack_core sources::imported_history::watermark::tests — 9 passed.
• Clean detached-worktree validation: CARGO_BUILD_JOBS=1 cargo clippy -p orgtrack_core --lib --no-deps — completed with only two pre-existing warnings in unrelated orgtrack-core files.
• Tests cover suffix-only resume, unterminated tails, same-size rewrite, seam mutation, shrink, parser change, file rotation, oversize rejection, database roundtrip, malformed-row fallback, and self-heal.
• rustfmt and git diff --check — passed.
• Diff scan for secrets, private keys, bearer tokens, and personal paths — passed.
• The isolated worktree lacked the generated Husky shim, so equivalent formatting, tests, scoped clippy, and staged-diff checks were run manually before committing.

Not run: production-home benchmark or filesystem-specific Windows execution.