Fix index batch recovery marker

DEVELOPER_GUIDE.md

@@ -143,6 +143,8 @@ Current stable codes and triggers:
 
 Under WAL, `NORMAL` avoids per-commit fsync pressure during 500-row indexing batches while preserving database consistency after crashes. `DbWriter` runs `PRAGMA wal_checkpoint(PASSIVE)` after each outer transaction commit, and SQLite may also checkpoint automatically after the configured 1000-page threshold. Both checkpoint paths are opportunistic: active readers are not blocked, and an uncheckpointed WAL is expected state rather than corruption. If the process is killed after SQLite has committed a transaction but before checkpointing, the next normal opener rolls the WAL forward; no manual recovery step is required. If the process dies before a transaction commits, that transaction is rolled back by SQLite.
 
+Index write batches also stamp `codeindex_meta.batch_in_progress=true` before starting a mutation transaction and clear it inside the transaction that commits the matching rows and readiness metadata. If the indexer crashes after the marker is written but before the commit clears it, the next writable DB open demotes readiness bits and warns: `Last batch did not complete; run cdidx index --rebuild to re-index from a known clean state.` Gracefully handled per-file errors clear the marker after rollback; orphaned markers are reserved for interrupted or crashed batches whose trust metadata should not be treated as clean.
+
 Read-only fallback uses an immutable SQLite URI when the normal writable open cannot create or lock journal/WAL side files, so query commands can still read a DB from read-only or sandboxed storage. That fallback intentionally skips writable pragmas, migrations, and WAL recovery writes; if a WAL is present and must be observed, copy the `.db`, `.db-wal`, and `.db-shm` files together to a writable location or use a SQLite backup from an environment that can open the full WAL set. `status --json` exposes the resolved connection values under `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `page_count`, `freelist_count`, `page_size`) for automation and support diagnostics. `cdidx vacuum` runs `PRAGMA incremental_vacuum` against writable incremental-auto-vacuum DBs, and performs a one-time `PRAGMA auto_vacuum=INCREMENTAL` plus full `VACUUM` conversion for legacy no-autovacuum DBs.
 
 Writable opens also reject databases whose `PRAGMA user_version` contains readiness bits outside the current binary's `CurrentSchemaVersion` mask. Read-only status/query paths may still surface `index_newer_than_reader=true` as a degraded audit signal, but write-capable paths must fail with `E003_SCHEMA_TOO_NEW` so an older cdidx cannot silently rewrite a DB stamped by a newer one.
@@ -2617,7 +2619,7 @@ flowchart TD
    - 次に OS の通常の `dlopen` 検索パス（`/lib`、`/usr/lib` など）
    自己完結型 publish は `libe_sqlite3.so` を publish 出力に同梱し、リリース tarball に含め、修正後の `install.sh` がバイナリの隣に置くため、最初のプローブで成功する。これが欠けていると、`SqliteConnection` のインスタンス生成時点で `DllNotFoundException: Unable to load shared library 'e_sqlite3'` が送出され、**ユーザーコードが実行される前にプロセスが終了する**。
 5. **スキーマ初期化。** `DbContext.ctor` が `PRAGMA journal_mode=WAL`、`PRAGMA busy_timeout=5000`、`CREATE TABLE IF NOT EXISTS` / `CREATE VIRTUAL TABLE IF NOT EXISTS fts_chunks USING fts5 (…)` / トリガー DDL を実行する。成功は、ネイティブライブラリがロードできるだけでなく、FTS5 がビルドに含まれた動作する SQLite であることも証明する（SQLitePCLRaw の同梱ビルドは常に FTS5 有効）。
-6. **スキャンと書き込み。** `FileIndexer` がプロジェクトツリーを走査し、ファイルを読み、言語を検出し、チャンク分割し、シンボルと参照を抽出し、`DbWriter` がトランザクションあたり500件ずつ UPSERT する。進捗は `ConsoleUi.SetProgressTheme()` でレンダリング。
+6. **スキャンと書き込み。** `FileIndexer` がプロジェクトツリーを走査し、ファイルを読み、言語を検出し、チャンク分割し、シンボルと参照を抽出し、`DbWriter` がトランザクションあたり500件ずつ UPSERT する。write batch は transaction の外で `codeindex_meta.batch_in_progress=true` を先に stamp し、同じ transaction 内で rows と readiness metadata を commit するときに marker を clear する。crash により marker が残った DB を次回 writable open すると readiness bit を落として rebuild を促す警告を出すため、stale な trust metadata を clean と誤読しない。進捗は `ConsoleUi.SetProgressTheme()` でレンダリング。
 7. **FTS optimize。** 書き込みのコミット後、`INSERT INTO fts_chunks(fts_chunks) VALUES('optimize')` を実行。
 8. **サマリー表示。** `Files / Chunks / Symbols / Refs / Elapsed`。
 

changelog.d/unreleased/2109.fixed.md

@@ -0,0 +1,21 @@
+---
+category: fixed
+issues:
+  - 2109
+affected:
+  - src/CodeIndex/Database/DbContext.cs
+  - src/CodeIndex/Database/DbWriter.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.cs
+  - tests/CodeIndex.Tests/DatabaseTests.cs
+  - tests/CodeIndex.Tests/IndexCommandRunnerTests.cs
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Index write batches now leave a crash-recovery marker (#2109)** — `cdidx` stamps an in-progress batch marker before mutating index rows and clears it atomically with the committing transaction, so the next writable DB open demotes readiness and warns users to rebuild when a prior batch crashed before completing.
+
+## 日本語
+
+- **index write batch が crash recovery marker を残すようになりました (#2109)** — `cdidx` は index 行を変更する前に in-progress batch marker を stamp し、commit する transaction と同時に clear するため、前回 batch が完了前に crash した DB を次回 writable open したとき readiness を落として rebuild を促す警告を出します。

changelog.d/unreleased/2609.fixed.md

@@ -0,0 +1,16 @@
+---
+category: fixed
+issues:
+  - 2609
+affected:
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs
+  - tests/CodeIndex.Tests/SymbolExtractorTests.cs
+---
+
+## English
+
+- **Rust associated type defaults inside traits are now indexed as properties (#2609)** — The Rust symbol extractor now scans `protocol` trait containers when adding associated type default property symbols, matching the existing Rust trait kind and restoring `type Name = Value;` entries inside traits.
+
+## 日本語
+
+- **Rust trait 内の associated type default が property として index されるようになりました (#2609)** — Rust symbol extractor は associated type default の property symbol を追加するとき、既存の Rust trait kind である `protocol` container を走査するようになり、trait 内の `type Name = Value;` entry を取りこぼさなくなりました。

src/CodeIndex/Cli/IndexCommandRunner.cs

@@ -1766,6 +1766,7 @@ void ThrowIfUpdateCancelled()
                 StartUpdateSpinnerIfNeeded();
                 currentUpdatePath = relPath;
                 var absPath = Path.Combine(projectRoot, relPath.Replace('/', Path.DirectorySeparatorChar));
+                var fileBatchMarked = false;
                 try
                 {
                     if (!File.Exists(LongPath.EnsureWindowsPrefix(absPath)))
@@ -2041,6 +2042,8 @@ void ThrowIfUpdateCancelled()
                 }
 
                 DemoteReadinessOnce();
+                writer.MarkBatchInProgress();
+                fileBatchMarked = true;
                 using var txn = writer.BeginTransaction();
                 writer.PurgeStaleFilesSharingChecksum(projectRoot, record.Path, record.Checksum);
                 if (projectRootWritten)
@@ -2068,6 +2071,7 @@ void ThrowIfUpdateCancelled()
                 // Validate content for encoding issues / エンコーディング問題を検証
                 var issues = FileIndexer.ValidateContent(record.Path, rawBytes, content);
                 writer.InsertIssues(fileId, issues);
+                writer.ClearBatchInProgress();
                 txn.Commit();
 
                 updated++;
@@ -2108,6 +2112,8 @@ void ThrowIfUpdateCancelled()
                 }
 
                 DemoteReadinessOnce();
+                if (fileBatchMarked)
+                    writer.ClearBatchInProgress();
                 GlobalToolLog.Error($"index_update_file_failed path={CollapseLineBreaks(relPath)}\n{GlobalToolLog.FormatExceptionChain(ex)}");
 
                 errors++;
@@ -2167,6 +2173,8 @@ void ThrowIfUpdateCancelled()
                 priorFoldFingerprint == currentFoldFingerprint);
         if (readinessDemoted && errors == 0)
         {
+            writer.MarkBatchInProgress();
+            using var readinessTxn = writer.BeginTransaction();
             // Restore each readiness bit independently based on what the DB carried BEFORE
             // ClearReadyFlags wiped them. A pre-#86 DB (user_version=3, i.e. Graph+Issues but
             // no Fold) must keep Graph+Issues after a successful partial update, even though
@@ -2251,6 +2259,8 @@ void ThrowIfUpdateCancelled()
             }
             writer.WriteCdidxWriterVersion(ConsoleUi.LoadVersion());
             writer.SetMeta(SymbolKindFilterMetaKey, options.SymbolKindFilter.Signature);
+            writer.ClearBatchInProgress();
+            readinessTxn.Commit();
         }
         if (errors == 0)
         {
@@ -3085,6 +3095,7 @@ void ThrowIfFullScanCancelled(int filesProcessed, int? filesTotal)
         // full-scan の書き込み全体を outer transaction に入れ、中断時に readiness clear /
         // purge / per-file write をまとめて rollback する。
         ThrowIfFullScanCancelled(0, files.Count);
+        writer.MarkBatchInProgress();
         using var fullScanTxn = writer.BeginTransaction();
         writer.ClearReadyFlags();
         writer.ClearHotspotFamilyReady();
@@ -3722,6 +3733,7 @@ void StopJsonHeartbeat()
             // ここで stamp する。full scan / partial update を問わず最新の HEAD を保存する。
             StampIndexedHeadMetadata(writer, projectRoot);
         }
+        writer.ClearBatchInProgress();
         fullScanTxn.Commit();
         stopwatch.Stop();
         // Detect cwd drift between option-parsing and finalize. See RunUpdateMode for the

src/CodeIndex/Database/DbContext.cs

@@ -189,6 +189,7 @@ public DbContext(string dbPath)
                     ApplyConnectionPerformancePragmas();
                     RegisterConnectionFunctionsWithRetry(_connection);
                     _isReadOnly = true;
+                    WarnIfBatchInProgress();
                     return;
                 }
                 catch
@@ -231,6 +232,7 @@ public DbContext(string dbPath)
             ExecuteSynchronousPragmaWithFallback(Execute);
             Execute($"PRAGMA wal_autocheckpoint={DefaultWalAutocheckpointPages}");
             Execute("PRAGMA optimize=0x10002");
+            WarnIfBatchInProgress();
         }
         catch (SqliteException ex) when (IsReadOnlyOpenError(ex))
         {
@@ -250,6 +252,7 @@ public DbContext(string dbPath)
                 ApplyConnectionPerformancePragmas();
                 RegisterConnectionFunctionsWithRetry(_connection);
                 _isReadOnly = true;
+                WarnIfBatchInProgress();
             }
             catch
             {
@@ -271,6 +274,17 @@ public DbContext(string dbPath)
         _suppressWriteWorkTracking = false;
     }
 
+    private void WarnIfBatchInProgress()
+    {
+        var raw = GetMetaString(BatchInProgressMetaKey);
+        if (string.Equals(raw, "true", StringComparison.OrdinalIgnoreCase))
+        {
+            Console.Error.WriteLine("Warning: Last batch did not complete; run `cdidx index --rebuild` to re-index from a known clean state.");
+            if (!_isReadOnly)
+                Execute("PRAGMA user_version = 0");
+        }
+    }
+
     private void ApplyConnectionPerformancePragmas()
     {
         Execute($"PRAGMA cache_size=-{ReadPositiveIntEnvironment(CacheSizeEnvironmentVariable, DefaultCacheSizeKb)}");
@@ -1047,6 +1061,7 @@ private static void RegisterConnectionFunctionsWithRetry(
     // ファイル数。index 済み件数ではなく scan coverage の信号であり、現行 index が stamp
     // するまでは reader 側で省略する。
     public const string UnknownExtensionFileCountMetaKey = "unknown_extension_file_count";
+    public const string BatchInProgressMetaKey = "batch_in_progress";
     // Issue #1546: case-sensitivity of the workspace filesystem the most recent successful
     // index ran on, persisted as the string "true" / "false". Resolved via the probe in
     // `PathCasing` (which honors `core.ignorecase` when the project is a git workspace and

src/CodeIndex/Database/DbWriter.cs

@@ -1574,6 +1574,10 @@ public int RecordFtsIncrementalWrite()
         return value;
     }
 
+    public void MarkBatchInProgress() => SetMeta(DbContext.BatchInProgressMetaKey, "true");
+
+    public void ClearBatchInProgress() => SetMeta(DbContext.BatchInProgressMetaKey, "false");
+
     public bool OptimizeFtsIfIncrementalWriteThresholdReached(int threshold = DefaultFtsOptimizeIncrementalWriteThreshold)
     {
         if (threshold <= 0)

src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs

@@ -10070,7 +10070,7 @@ private static void ExtractRustAssociatedTypeDefaultSymbols(long fileId, string[
                             StartColumn = nameGroup.Index,
                             EndLine = lineNumber,
                             Signature = lines[lineIndex].Trim(),
-                            ContainerKind = "interface",
+                            ContainerKind = trait.Kind,
                             ContainerName = trait.Name,
                             ContainerQualifiedName = trait.ContainerQualifiedName,
                             Visibility = match.Groups["visibility"].Success ? match.Groups["visibility"].Value : null,

src/CodeIndex/Mcp/McpToolHandlers.cs

@@ -2419,6 +2419,7 @@ void WriteProjectRootOnce()
 
         foreach (var filePath in files)
         {
+            var fileBatchMarked = false;
             try
             {
                 var (record, content, rawBytes, _) = indexer.BuildRecordWithRawBytes(filePath);
@@ -2442,6 +2443,8 @@ void WriteProjectRootOnce()
                     continue;
                 }
 
+                writer.MarkBatchInProgress();
+                fileBatchMarked = true;
                 using var txn = writer.BeginTransaction();
                 var fileId = writer.UpsertFile(record);
                 var chunks = ChunkSplitter.Split(fileId, content);
@@ -2465,6 +2468,7 @@ void WriteProjectRootOnce()
                 var issues = FileIndexer.ValidateContent(record.Path, rawBytes, content);
                 writer.InsertIssues(fileId, issues);
                 WriteProjectRootOnce();
+                writer.ClearBatchInProgress();
                 txn.Commit();
             }
             catch (FileIndexer.BinaryFileSkippedException)
@@ -2487,6 +2491,8 @@ void WriteProjectRootOnce()
             }
             catch
             {
+                if (fileBatchMarked)
+                    writer.ClearBatchInProgress();
                 errors++;
             }
             processed++;
@@ -2506,6 +2512,8 @@ void WriteProjectRootOnce()
         _ = priorMetadataTargetCsharp;
         if (errors == 0)
         {
+            writer.MarkBatchInProgress();
+            using var readinessTxn = writer.BeginTransaction();
             writer.MarkGraphReady();
             writer.MarkIssuesReady();
             writer.MarkSqlGraphContractReady();
@@ -2616,6 +2624,8 @@ void WriteProjectRootOnce()
             {
                 // Best-effort; never fail an otherwise-successful index run.
             }
+            writer.ClearBatchInProgress();
+            readinessTxn.Commit();
         }
         var (totalFiles, totalChunks, totalSymbols, totalReferences) = writer.GetCounts();
 

tests/CodeIndex.Tests/DatabaseTests.cs

@@ -71,6 +71,70 @@ public void OptimizeFtsIfIncrementalWriteThresholdReached_RunsOnlyAtThreshold()
         Assert.Equal(0, _writer.GetFtsIncrementalWritesSinceOptimize());
     }
 
+    [Fact]
+    public void DbContext_OpenWithBatchInProgress_Warns()
+    {
+        _writer.MarkBatchInProgress();
+
+        var stderr = ConsoleCapture.CaptureError(() =>
+        {
+            using var reopened = new DbContext(_dbPath);
+        });
+
+        Assert.Contains("Last batch did not complete", stderr);
+        Assert.Contains("cdidx index --rebuild", stderr);
+    }
+
+    [Fact]
+    public void DbContext_OpenWithBatchInProgress_DemotesReadiness()
+    {
+        _writer.MarkGraphReady();
+        _writer.MarkIssuesReady();
+        _writer.MarkBatchInProgress();
+
+        using (var reopened = new DbContext(_dbPath))
+        {
+            Assert.Equal(0, reopened.GetUserVersion());
+        }
+    }
+
+    [Fact]
+    public void BatchInProgress_ClearInsideCommittedTransaction_PersistsCleanState()
+    {
+        _writer.MarkBatchInProgress();
+
+        using (var txn = _writer.BeginTransaction())
+        {
+            _writer.ClearBatchInProgress();
+            txn.Commit();
+        }
+
+        var stderr = ConsoleCapture.CaptureError(() =>
+        {
+            using var reopened = new DbContext(_dbPath);
+        });
+
+        Assert.DoesNotContain("Last batch did not complete", stderr);
+    }
+
+    [Fact]
+    public void BatchInProgress_ClearInsideRolledBackTransaction_LeavesRecoveryWarning()
+    {
+        _writer.MarkBatchInProgress();
+
+        using (var txn = _writer.BeginTransaction())
+        {
+            _writer.ClearBatchInProgress();
+        }
+
+        var stderr = ConsoleCapture.CaptureError(() =>
+        {
+            using var reopened = new DbContext(_dbPath);
+        });
+
+        Assert.Contains("Last batch did not complete", stderr);
+    }
+
     [Fact]
     public void Constructor_NewDatabaseEnablesIncrementalAutoVacuum()
     {

tests/CodeIndex.Tests/IndexCommandRunnerTests.cs

@@ -3435,8 +3435,12 @@ public void Run_FullScan_CancelledAfterReadinessDemotion_RollsBackExistingIndex(
 
             Assert.True(hookInvoked);
             Assert.Equal(CommandExitCodes.Interrupted, interruptedExitCode);
-            using (var db = new DbContext(dbPath))
-                Assert.Equal(initialReadiness, db.GetUserVersion());
+            var recoveryWarning = ConsoleCapture.CaptureError(() =>
+            {
+                using var db = new DbContext(dbPath);
+                Assert.Equal(0, db.GetUserVersion());
+            });
+            Assert.Contains("Last batch did not complete", recoveryWarning);
             Assert.DoesNotContain("later.cs", ReadIndexedPaths(dbPath));
         }
         finally

tests/CodeIndex.Tests/SymbolExtractorTests.cs

@@ -14409,13 +14409,13 @@ fn build(&self) {
         Assert.Contains(symbols, s =>
             s.Kind == "property"
             && s.Name == "Output"
-            && s.ContainerKind == "interface"
+            && s.ContainerKind == "protocol"
             && s.ContainerName == "Builder"
             && s.ReturnType == "()");
         Assert.Contains(symbols, s =>
             s.Kind == "property"
             && s.Name == "Error"
-            && s.ContainerKind == "interface"
+            && s.ContainerKind == "protocol"
             && s.ContainerName == "Builder"
             && s.ReturnType == "String");
         Assert.DoesNotContain(symbols, s => s.Kind == "property" && s.Name == "Pending");