fix(parser): fold SQLite -wal siblings into source fingerprints - #915
Conversation
Hermes, Cursor, OpenCode and copilot OTel sources all live in SQLite databases that their agents keep open in WAL mode for the life of the process. Committed writes park in <db>-wal until a checkpoint, so the main file's stat can sit hours or days behind the newest committed data. fingerprintFile only statted the main file, which broke two ways: - The date-range mtime pre-filter in parseProviderSources read the stale mtime as "nothing in range" and skipped the source entirely. Every Hermes session committed after the last checkpoint vanished from reports: the today-parse skipped the db (mtime < local midnight) while the backfill only keeps days through yesterday. Exactly the "17 sessions in the DB, 14 reported, the 3 from today missing" report in issue getagentseal#913. - reconcileFile saw an unchanged fingerprint between checkpoints and kept serving stale cached turns for sessions that had since grown. Fold the -wal sibling into the fingerprint: newest mtime wins and sizes add, so both WAL growth and a checkpoint (db grows, wal truncates) move the fingerprint. -shm is deliberately ignored (it mutates on reads). Bare SQLite paths get the fold only when the extension says database, so JSONL transcript fingerprints (offset-based append detection) are untouched. Refs getagentseal#913
iamtoruk
left a comment
There was a problem hiding this comment.
Reviewed in depth on the merged tree. Real data-completeness fix: SQLite state DBs kept open in WAL mode park committed writes in -wal until a checkpoint, so the main file's mtime goes stale, the date-range mtime pre-filter skips the source, and today's sessions committed after the last checkpoint never parse (the 17-in-db / 14-reported split in #913). Folding the -wal sibling in (max mtime, sizes added) moves the fingerprint on both WAL growth and checkpoint; -shm correctly ignored. Gated to bare SQLite paths (.db/.sqlite/.sqlite3/.vscdb) and the #/: virtual-suffix forms, and for a non-SQLite base it degrades to exactly the old fingerprint, so JSONL append-detection is untouched. Mutation-checked: neutering the fold to main-only fails exactly the four wal-fold tests (the #/:/bare-SQLite cases and the Hermes checkpoint-stale e2e), while the no-wal and non-SQLite cases still pass. tsc clean; the two touched suites 78/78; full suite green (2438 parallel + 26 serial cache-locks, 0 failed). Good to merge.
Symptoms (reported in #913)
A Hermes user with 17 token-bearing sessions in
~/.hermes/state.dbsees only 14 in reports; the 3 sessions started today are missing entirely.Root cause
Hermes (and Cursor, OpenCode, copilot OTel) keep their SQLite state DBs open in WAL mode for the life of the agent process. Committed writes park in
<db>-waluntil a checkpoint runs, so the main file's mtime can sit hours or days behind the newest committed data. Real-world stat from a live Hermes install (journal_mode=wal):fingerprintFileonly statted the main file, which broke two ways:parseProviderSources(fp.mtimeMs < dateRange.start-> skip) reads the stale mtime as "nothing in range" and skips every source backed by that db. The today-parse (range start = local midnight) skips the whole db whenever the last checkpoint predates midnight, while the backfill union only keeps days through yesterday. Net effect: sessions started today after the last checkpoint appear nowhere. That is exactly the 17-in-DB / 14-reported split in Support for Hermes Agent? #913.reconcileFilesees an unchanged fingerprint and serves cached turns, so sessions that grew since the last checkpoint report old totals until a checkpoint happens to run.Fix
Fold the
-walsibling into the fingerprint: newest mtime wins, sizes add. Both WAL growth and a checkpoint (db grows, wal truncates) move the fingerprint. Applied to the two virtual-suffix fallbacks (#,:) and to bare paths only when the extension is a SQLite one (.db/.sqlite/.sqlite3/.vscdb), so JSONL transcript fingerprints and their offset-based append detection are untouched.-shmis deliberately ignored: it mutates on reads and would churn the fingerprint without any data change.Testing
fingerprintFile(wal fold for#/:/ bare-db forms, no-wal passthrough, non-SQLite paths never folded)-walsibling ->parseAllSessionsmust still report itFixes the missing-session half of #913. The cost-accuracy half is a separate discussion (recorded-cost precedence, #890 provenance) and is being handled on the issue.