feat: add rebuild-fts command for FTS5 shadow-table corruption (#287)#288
Merged
feat: add rebuild-fts command for FTS5 shadow-table corruption (#287)#288
Conversation
msgvault verify surfaces "malformed inverted index for FTS5 table main.messages_fts" on long-lived archives. SQLite's own rebuild pragma reads from the same corrupt shadow tables and cannot clear the state, and delete-all is rejected on contentful FTS5 tables. rebuild-fts drops messages_fts outright (discarding the shadow tables) and repopulates it from messages / message_bodies / message_recipients / participants via the existing batched backfill path. RebuildFTS intentionally bypasses the cached fts5Available flag — the probe that sets that flag is the same one that fails on corrupt shadow tables, so the repair command cannot consult it. SQLITE_BUSY / LOCKED errors surface as an actionable "stop serve / MCP and retry" message. verify now runs the integrity check before OAuth setup so corruption hints reach users with expired tokens, and splits the recovery hint between derived-index corruption (rebuild-fts) and core-table corruption (.recover). docs/recovery.md covers both paths.
roborev: Combined Review (
|
Owner
Author
|
The compile-break finding is a false positive. Verified:
The PostgreSQL dialect referenced in the new method's docstring (`PostgreSQL: TODO (REINDEX / recompute tsvector column)`) is forward-looking documentation for the in-progress PG refactor (see #276 "PostgreSQL dialect refactor - PR1 of 4 - Dialect"). No PG dialect type exists yet. When it lands, whoever writes it will implement `FTSRebuildSchema` and `IsBusyError` as part of satisfying the interface — which is exactly how interface evolution is supposed to work. Not making changes based on this finding. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #287.
What this adds
msgvault rebuild-fts— drops and recreatesmessages_fts, then repopulates it frommessages/message_bodies/message_recipients/participantsvia the existing batched backfill. Recovery path formalformed inverted index for FTS5 table main.messages_ftsinverifyoutput — the case where SQLite's ownrebuildpragma anddelete-alldon't help on a contentful FTS5 table. WrapsSQLITE_BUSY/SQLITE_LOCKEDas "stop msgvault serve / MCP and retry."verifyordering — integrity check now runs before OAuth setup, so corrupt databases surface the repair hint even with an expired token.verifyhints — split between FTS-only corruption (points atrebuild-fts) and core-table corruption (points at.recover).docs/recovery.md— covers both paths and the contentful-FTS5 caveat.What this does not cover
Rowid out of orderinmessages/message_bodies).rebuild-ftsis strictly for the derived index; core corruption still needs.recover._synchronous=FULLdefault. Worth considering as archival-durability hardening in a separate PR; not framed as the fix for FTS5 + B-tree corruption on contentful index: recovery requires manual DROP+CREATE+backfill #287.Usage
Stop
msgvault serveand MCP clients first — needs an exclusive write lock. Peak extra disk ≈ size of the FTS5 shadow tables (a few percent of the DB).