Skip to content

fix(parser): fold SQLite -wal siblings into source fingerprints - #915

Merged
iamtoruk merged 1 commit into
getagentseal:mainfrom
ozymandiashh:fix/913-sqlite-wal-fingerprint
Aug 4, 2026
Merged

fix(parser): fold SQLite -wal siblings into source fingerprints#915
iamtoruk merged 1 commit into
getagentseal:mainfrom
ozymandiashh:fix/913-sqlite-wal-fingerprint

Conversation

@ozymandiashh

Copy link
Copy Markdown
Collaborator

Symptoms (reported in #913)

A Hermes user with 17 token-bearing sessions in ~/.hermes/state.db sees only 14 in reports; the 3 sessions started today are missing entirely.

Root cause

Hermes (and Cursor, OpenCode, copilot OTel) keep their SQLite state DBs open in WAL mode for the life of the agent process. Committed writes park in <db>-wal until a checkpoint runs, so the main file's mtime can sit hours or days behind the newest committed data. Real-world stat from a live Hermes install (journal_mode=wal):

-rw------- 626180096 2026-08-04 03:21:24 state.db      <- 8h stale
-rw-------   7300672 2026-08-04 03:23:47 state.db-wal  <- newest commits

fingerprintFile only statted the main file, which broke two ways:

  1. Sessions vanish from reports. The date-range mtime pre-filter in parseProviderSources (fp.mtimeMs < dateRange.start -> skip) reads the stale mtime as "nothing in range" and skips every source backed by that db. The today-parse (range start = local midnight) skips the whole db whenever the last checkpoint predates midnight, while the backfill union only keeps days through yesterday. Net effect: sessions started today after the last checkpoint appear nowhere. That is exactly the 17-in-DB / 14-reported split in Support for Hermes Agent? #913.
  2. Stale cached data between checkpoints. reconcileFile sees an unchanged fingerprint and serves cached turns, so sessions that grew since the last checkpoint report old totals until a checkpoint happens to run.

Fix

Fold the -wal sibling into the fingerprint: newest mtime wins, sizes add. Both WAL growth and a checkpoint (db grows, wal truncates) move the fingerprint. Applied to the two virtual-suffix fallbacks (#, :) and to bare paths only when the extension is a SQLite one (.db/.sqlite/.sqlite3/.vscdb), so JSONL transcript fingerprints and their offset-based append detection are untouched. -shm is deliberately ignored: it mutates on reads and would churn the fingerprint without any data change.

Testing

  • 5 new unit tests for fingerprintFile (wal fold for # / : / bare-db forms, no-wal passthrough, non-SQLite paths never folded)
  • 1 end-to-end regression: hermes session inside the range, main db utimes'd to before the range, fresh -wal sibling -> parseAllSessions must still report it
  • Mutation check: with the src change stashed, exactly the 4 wal-fold tests fail; restored, 78/78 in the two touched files
  • Full suite: 2702 passed / 0 failed (the two known-flaky files on this machine, cache-refresh-lock and menubar-json timeouts, passed on rerun; the 26 jsdom unhandled errors are a pre-existing local node_modules condition, present on clean main)

Fixes the missing-session half of #913. The cost-accuracy half is a separate discussion (recorded-cost precedence, #890 provenance) and is being handled on the issue.

Hermes, Cursor, OpenCode and copilot OTel sources all live in SQLite
databases that their agents keep open in WAL mode for the life of the
process. Committed writes park in <db>-wal until a checkpoint, so the
main file's stat can sit hours or days behind the newest committed data.

fingerprintFile only statted the main file, which broke two ways:

- The date-range mtime pre-filter in parseProviderSources read the stale
  mtime as "nothing in range" and skipped the source entirely. Every
  Hermes session committed after the last checkpoint vanished from
  reports: the today-parse skipped the db (mtime < local midnight) while
  the backfill only keeps days through yesterday. Exactly the "17
  sessions in the DB, 14 reported, the 3 from today missing" report in
  issue getagentseal#913.

- reconcileFile saw an unchanged fingerprint between checkpoints and kept
  serving stale cached turns for sessions that had since grown.

Fold the -wal sibling into the fingerprint: newest mtime wins and sizes
add, so both WAL growth and a checkpoint (db grows, wal truncates) move
the fingerprint. -shm is deliberately ignored (it mutates on reads).
Bare SQLite paths get the fold only when the extension says database, so
JSONL transcript fingerprints (offset-based append detection) are
untouched.

Refs getagentseal#913

@iamtoruk iamtoruk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed in depth on the merged tree. Real data-completeness fix: SQLite state DBs kept open in WAL mode park committed writes in -wal until a checkpoint, so the main file's mtime goes stale, the date-range mtime pre-filter skips the source, and today's sessions committed after the last checkpoint never parse (the 17-in-db / 14-reported split in #913). Folding the -wal sibling in (max mtime, sizes added) moves the fingerprint on both WAL growth and checkpoint; -shm correctly ignored. Gated to bare SQLite paths (.db/.sqlite/.sqlite3/.vscdb) and the #/: virtual-suffix forms, and for a non-SQLite base it degrades to exactly the old fingerprint, so JSONL append-detection is untouched. Mutation-checked: neutering the fold to main-only fails exactly the four wal-fold tests (the #/:/bare-SQLite cases and the Hermes checkpoint-stale e2e), while the no-wal and non-SQLite cases still pass. tsc clean; the two touched suites 78/78; full suite green (2438 parallel + 26 serial cache-locks, 0 failed). Good to merge.

@iamtoruk
iamtoruk merged commit 45d98a3 into getagentseal:main Aug 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants