Summary
When an unhandled exception occurs in DbWriter (e.g., in InsertSymbols, InsertReferences, or UpsertFile), the TransactionScope.Dispose() rollback cleans up the transaction state but leaves the readiness metadata stamp at its pre-transaction value. If the crash happens after the metadata has been updated but before the row inserts complete, the DB enters an inconsistent state: the readiness bit claims the index is fresh, but symbols/references from that batch are missing. Subsequent queries return incomplete results without surfacing the inconsistency.
Where
src/CodeIndex/Database/DbWriter.cs:98-118 (TransactionScope.Dispose: best-effort rollback on exception, but does not touch metadata)src/CodeIndex/Cli/IndexCommandRunner.cs:731-837 (file-processing loop: metadata is stamped before batch-insert, crash between stamp and insert corrupts readiness)src/CodeIndex/Database/DbContext.cs:422-600 (InitializeSchema: creates metadata table but no crash-recovery fields like last_successful_batch_id)
Suggested approach
- Add a
batch_in_progress: boolean | null column to codeindex_meta, set to true before starting InsertSymbols/InsertReferences and false after commit.
- In DbContext constructor, check for orphaned
batch_in_progress=true on startup and emit a recovery warning to stderr: "Last batch did not complete; run cdidx index --rebuild to re-index from a known clean state."
- Set the
readiness / fold_ready stamps inside the transaction, not before, so they commit atomically with the symbol/reference data.
- After a graceful commit, verify
batch_in_progress is false before clearing recovery warnings.
- Document: "Batch operations are atomic within a transaction. If the indexer crashes, the next run detects incomplete batches and refuses to trust stale readiness markers until
--rebuild completes."
Summary
When an unhandled exception occurs in DbWriter (e.g., in InsertSymbols, InsertReferences, or UpsertFile), the TransactionScope.Dispose() rollback cleans up the transaction state but leaves the
readinessmetadata stamp at its pre-transaction value. If the crash happens after the metadata has been updated but before the row inserts complete, the DB enters an inconsistent state: the readiness bit claims the index is fresh, but symbols/references from that batch are missing. Subsequent queries return incomplete results without surfacing the inconsistency.Where
src/CodeIndex/Database/DbWriter.cs:98-118(TransactionScope.Dispose: best-effort rollback on exception, but does not touch metadata)src/CodeIndex/Cli/IndexCommandRunner.cs:731-837(file-processing loop: metadata is stamped before batch-insert, crash between stamp and insert corrupts readiness)src/CodeIndex/Database/DbContext.cs:422-600(InitializeSchema: creates metadata table but no crash-recovery fields likelast_successful_batch_id)Suggested approach
batch_in_progress: boolean | nullcolumn to codeindex_meta, set totruebefore starting InsertSymbols/InsertReferences andfalseafter commit.batch_in_progress=trueon startup and emit a recovery warning to stderr: "Last batch did not complete; runcdidx index --rebuildto re-index from a known clean state."readiness/fold_readystamps inside the transaction, not before, so they commit atomically with the symbol/reference data.batch_in_progressisfalsebefore clearing recovery warnings.--rebuildcompletes."