Classify validate replacement characters

DEVELOPER_GUIDE.md

@@ -229,6 +229,12 @@ Directory scan / shared path filter (built-in skip lists + `.gitignore` / `.cdid
   → Populate FTS5 index
 ```
 
+`FileIssue` rows may include nullable `origin` and `severity` metadata.
+For `replacement_char`, `origin: source_literal` means the file contains a
+valid encoded U+FFFD literal, while `origin: decode_replacement` means the
+decoder inserted U+FFFD for invalid bytes. `severity: info` is used for source
+literals, and `severity: warning` is used for likely encoding damage.
+
 Scoped `--files` / `--commits` refreshes reuse the same path filter as full scans. Before scanning a nested project root, `FileIndexer` loads ignore files from the resolved ignore-rule root through each existing ancestor directory down to the project root's parent, then loads the project directory's own rules during the normal walk. Within each directory, `FileIndexer` loads `.gitignore` before `.cdidxignore`, appends both rule sets in that order, and honors later `!` patterns as re-includes. If an ancestor ignore directory cannot be read, scanning fails closed with a scan error instead of silently skipping those rules; `ScanFilesResult.AncestorIgnoreDirectories` records the resolved ancestor list for troubleshooting. If a commit-scoped refresh includes `.gitignore` or `.cdidxignore` changes, `IndexCommandRunner` falls back to a full scan so newly ignored files are purged safely. Malformed ignore lines are reported as scan errors and skipped instead of aborting the whole run. Directory symlinks default to `--follow-symlinks none`; `internal` follows only targets that resolve under the workspace root, and `all` preserves the broad historical behavior. Dangling symlinks are counted and warned separately. On Windows, files and directories with Hidden or System attributes are rejected before language detection; clear those attributes before indexing project-owned sources because ignore rules cannot re-include them.
 
 Incremental refreshes that mutate `fts_chunks` increment `codeindex_meta.fts_incremental_writes_since_optimize`. When the counter reaches `DbWriter.DefaultFtsOptimizeIncrementalWriteThreshold`, the update path runs `INSERT INTO fts_chunks(fts_chunks) VALUES('optimize')`, resets the counter, and stamps `fts_last_optimized_at`. Users can run the same maintenance directly with `cdidx optimize --db <path>` or `cdidx index <projectPath> --optimize`; this may briefly hold the writer lock on large indexes.
@@ -2124,6 +2130,12 @@ CI で `NU1004 The packages lock file is inconsistent with the project dependenc
   → FTS5インデックス反映
 ```
 
+`FileIssue` rows には nullable な `origin` / `severity` metadata が入ることがある。
+`replacement_char` では `origin: source_literal` が正規にエンコードされた U+FFFD
+literal、`origin: decode_replacement` が不正 byte に対して decoder が挿入した U+FFFD
+を意味する。source literal は `severity: info`、エンコーディング破損の可能性は
+`severity: warning` として返す。
+
 `--files` / `--commits` の部分更新も、フルスキャンと同じパスフィルタを再利用する。各ディレクトリでは `FileIndexer` が `.gitignore` を `.cdidxignore` より先に読み、この順序でルールを追加し、後続の `!` パターンを再包含として扱う。commit 単位更新に `.gitignore` または `.cdidxignore` の変更が含まれる場合、`IndexCommandRunner` は newly ignored file を安全に purge するため自動でフルスキャンへフォールバックする。malformed な ignore 行は走査エラーとして報告し、その行だけをスキップして index 全体は継続する。Windows では Hidden または System 属性が付いたファイルとディレクトリを言語検出前に拒否する。プロジェクト所有のソースを索引したい場合、ignore ルールでは再包含できないため先にそれらの属性を外す。
 
 ### ignore ファイルの解析

README.md

@@ -158,6 +158,7 @@ downgrading `cdidx`.
 | Area | What cdidx provides |
 |---|---|
 | Search surfaces | CLI-first output for humans and machines; full-text, symbol, reference, caller/callee, dependency, map, inspect, and excerpt commands. `search`, `definition`, `references`, `callers`, `callees`, `find`, and `validate` support `--format count|compact|csv|tsv|lsp|qf|sarif` for token-budgeted agents, scripts, editors, and CI reports. `cdidx lsp --db .cdidx/codeindex.db` starts a read-only stdio Language Server Protocol shim for LSP-native editors. |
+| Validation diagnostics | `validate --json` and MCP `validate` annotate `replacement_char` rows with `origin` (`source_literal` or `decode_replacement`) and `severity` so agents can separate intentional U+FFFD literals from likely encoding damage. |
 | Definition and impact diagnostics | `definition --json` includes C# `disambiguator` hints for overloads, partial types, and extension receivers when indexed metadata can distinguish them. `impact --json` and MCP `impact_analysis` include `impact_failure_chain` and `suggestion_type` for zero-result routing; `impact --strict` exits non-zero when resolution or graph preconditions are unmet. |
 | Ranking and filters | Public/exported symbol matches rank ahead of protected, internal, and private matches. Use `--no-visibility-rank` for legacy order, and `--visibility` / `--exclude-visibility` with `symbols`, `definition`, `unused`, and `hotspots`. Query defaults can be adjusted with `CDIDX_DEFAULT_LIMIT`, `CDIDX_DEFAULT_SNIPPET_LINES`, and `CDIDX_DEFAULT_MAX_LINE_WIDTH`; explicit CLI flags still win. |
 | Project scoping | `.sln` / `.csproj`-aware <code>--project &lt;name&#124;path&gt;</code> filters for indexing and queries, plus `--solution <path>` when a workspace has multiple solution files. |
@@ -437,6 +438,7 @@ upgrade / downgrade 後はインストール済み補完 script を再生成し
 | 分野 | 内容 |
 |---|---|
 | 検索面 | CLI-first の人間向け / 機械処理向け出力。全文検索、シンボル、参照、caller/callee、依存関係、map、inspect、excerpt コマンドを提供します。`cdidx lsp --db .cdidx/codeindex.db` は LSP-native editor 向けの read-only stdio Language Server Protocol shim を起動します。 |
+| validation 診断 | `validate --json` と MCP `validate` は `replacement_char` 行に `origin` (`source_literal` / `decode_replacement`) と `severity` を付け、意図的な U+FFFD literal とエンコーディング破損の可能性を agent が分離できるようにします。 |
 | definition / impact 診断 | `definition --json` は C# overload、partial type、extension receiver を区別できる場合に `disambiguator` を返します。`impact --json` と MCP `impact_analysis` は 0 件時の経路判断用に `impact_failure_chain` と `suggestion_type` を返し、`impact --strict` は解決または graph の前提条件が満たされない場合に非 0 で終了します。 |
 | 順位と filter | public/exported なシンボル一致を protected、internal、private より優先します。従来順は `--no-visibility-rank`、可視性の include / exclude は `symbols`、`definition`、`unused`、`hotspots` の `--visibility` / `--exclude-visibility` で指定できます。query 既定値は `CDIDX_DEFAULT_LIMIT`、`CDIDX_DEFAULT_SNIPPET_LINES`、`CDIDX_DEFAULT_MAX_LINE_WIDTH` で調整でき、明示 CLI flag が常に優先されます。 |
 | project scope | `.sln` / `.csproj` を使った <code>--project &lt;name&#124;path&gt;</code> filter で index と query を .NET project 配下へ絞り込めます。workspace に solution が複数ある場合は `--solution <path>` を指定します。 |

USER_GUIDE.md

@@ -280,7 +280,7 @@ sections below show examples and option details for the most common workflows.
 | Analysis | `impact` | Traverse transitive callers from a resolved symbol | `impact_analysis` |
 | Analysis | `unused` | Find symbols defined but not referenced, with confidence buckets | `unused_symbols` |
 | Analysis | `hotspots` | Rank high-impact symbols or statements by reference volume | `symbol_hotspots` |
-| Analysis | `validate` | Report encoding and line-ending issues in indexed files | `validate` |
+| Analysis | `validate` | Report encoding and line-ending issues in indexed files; U+FFFD rows include origin/severity metadata | `validate` |
 | Status | `status` | Show DB statistics, freshness, and readiness metadata | `status` |
 | Status | `languages` | List language extensions and symbol/graph capabilities | `languages` |
 | Diagnostics | `db --integrity-check` | Run SQLite `PRAGMA integrity_check` against the DB | -- |
@@ -400,13 +400,16 @@ conflicting instructions.
 
 ```bash
 cdidx validate
-cdidx validate --kind replacement-character --path src/
+cdidx validate --kind replacement_char --path src/
 cdidx validate --json --path legacy/
 ```
 
 `validate` reports indexed files that are likely to produce misleading snippets
 or symbol names: U+FFFD replacement characters, UTF-16 BOMs, null bytes, mixed or
 CR-only line endings, likely non-UTF-8 content, and Git LFS pointer placeholders.
+For `replacement_char`, JSON and MCP responses include `origin` (`source_literal`
+or `decode_replacement`) and `severity` so agents can distinguish intentional
+U+FFFD literals from likely encoding damage.
 LFS pointers are recorded as `lfs_pointer_skipped` and their placeholder body is
 not indexed; run `git lfs pull` and then `cdidx index .` to index the real file
 content.
@@ -1945,7 +1948,7 @@ The MCP `tools/list` response includes an `examples` array for every registered
 | `unused_symbols` | Find symbols defined but never referenced, with confidence buckets for dead-code triage |
 | `symbol_hotspots` | Find high-impact hotspots. `groupBy` supports `symbol`, `file`, and `statement`; SQL scopes default to statement grouping while non-SQL scopes default to symbol grouping. |
 | `batch_query` | Execute multiple queries in a single call (MCP only, max 10). The response includes a top-level `metadata` object with `submitted`, `executed`, `errors`, `total_elapsed_ms`, `success_count`, and `failure_count`; every entry in `results` carries `request_index`, `ok`, `elapsed_ms`, and compact `args_summary` fields so callers can correlate partial failures and slow inner queries without relying on positional guesses. |
-| `validate` | Report encoding issues (U+FFFD, BOM, null bytes, mixed/CR-only line endings, UTF-16 BOM detection, likely non-UTF8 encodings) |
+| `validate` | Report encoding issues (U+FFFD with origin/severity, BOM, null bytes, mixed/CR-only line endings, UTF-16 BOM detection, likely non-UTF8 encodings) |
 | `languages` | List all supported languages, file extensions, and capabilities |
 | `ping` | Lightweight connection check |
 | `index` | Index or re-index a project directory |
@@ -2412,7 +2415,7 @@ cdidx index . --quiet
 | Analysis | `impact` | 解決した symbol から transitive callers を探索 | `impact_analysis` |
 | Analysis | `unused` | 参照されていない可能性がある symbols を confidence bucket 付きで表示 | `unused_symbols` |
 | Analysis | `hotspots` | reference volume で high-impact symbols/statements を ranking | `symbol_hotspots` |
-| Analysis | `validate` | indexed files の encoding / line-ending 問題を報告 | `validate` |
+| Analysis | `validate` | indexed files の encoding / line-ending 問題を報告。U+FFFD 行には origin/severity metadata が付く | `validate` |
 | Status | `status` | DB stats、freshness、readiness metadata を表示 | `status` |
 | Status | `languages` | language extensions と symbol/graph capabilities を一覧 | `languages` |
 | Diagnostics | `db --integrity-check` | DB に対して SQLite `PRAGMA integrity_check` を実行 | -- |
@@ -2522,13 +2525,16 @@ render できます。
 
 ```bash
 cdidx validate
-cdidx validate --kind replacement-character --path src/
+cdidx validate --kind replacement_char --path src/
 cdidx validate --json --path legacy/
 ```
 
 `validate` は、snippet や symbol name を誤らせやすい indexed file を報告します。
 対象は U+FFFD replacement character、UTF-16 BOM、null byte、mixed / CR-only line
-ending、likely non-UTF-8 content、Git LFS pointer placeholder などです。LFS pointer
+ending、likely non-UTF-8 content、Git LFS pointer placeholder などです。
+`replacement_char` の JSON / MCP response には `origin` (`source_literal` /
+`decode_replacement`) と `severity` が入り、意図的な U+FFFD literal と
+エンコーディング破損の可能性を agent が区別できます。LFS pointer
 は `lfs_pointer_skipped` として記録され、placeholder 本文は index されません。
 実体を index するには `git lfs pull` の後に `cdidx index .` を再実行してください。
 
@@ -4077,7 +4083,7 @@ OpenAI Codex CLI (`codex.json` または `~/.codex/config.json`):
 | `unused_symbols` | 定義されているが参照されていないシンボルを bucket 付きで検索（デッドコード検出向け） |
 | `symbol_hotspots` | 影響の大きい hotspot を検索。`groupBy` は `symbol` / `file` / `statement` を指定でき、SQL scope は statement grouping、非 SQL scope は symbol grouping が既定。 |
 | `batch_query` | 複数クエリを1回で実行（MCP専用、最大10件）。レスポンスにはトップレベル `metadata`（`submitted` / `executed` / `errors` / `total_elapsed_ms` / `success_count` / `failure_count`）と各 `results` エントリの `request_index` / `ok` / `elapsed_ms` / `args_summary` が含まれ、位置だけに依存せず部分失敗や遅い内部クエリを把握できます。 |
-| `validate` | エンコーディング問題（U+FFFD、BOM、null バイト、改行混在 / CR-only 行末、UTF-16 BOM 検出、UTF-8 以外と推定されるエンコーディング）を報告 |
+| `validate` | エンコーディング問題（origin/severity 付き U+FFFD、BOM、null バイト、改行混在 / CR-only 行末、UTF-16 BOM 検出、UTF-8 以外と推定されるエンコーディング）を報告 |
 | `languages` | 対応言語一覧を拡張子・機能付きで表示 |
 | `ping` | 軽量な接続確認 |
 | `index` | プロジェクトのインデックス作成・更新 |

changelog.d/unreleased/2814.changed.md

@@ -0,0 +1,27 @@
+---
+category: changed
+issues:
+  - 2814
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - src/CodeIndex/Models/FileIssue.cs
+  - src/CodeIndex/Database/DbContext.cs
+  - src/CodeIndex/Database/DbReader.cs
+  - src/CodeIndex/Database/DbWriter.cs
+  - src/CodeIndex/Cli/ConsoleUi.cs
+  - src/CodeIndex/Mcp/McpToolDefinitions.cs
+  - README.md
+  - USER_GUIDE.md
+  - DEVELOPER_GUIDE.md
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+  - tests/CodeIndex.Tests/QueryCommandRunnerTests.cs
+  - tests/CodeIndex.Tests/DatabaseTests.cs
+---
+
+## English
+
+- **`validate` now classifies U+FFFD replacement-character rows (#2814)** — `replacement_char` issues now include `origin` and `severity` metadata so JSON and MCP consumers can distinguish intentional `source_literal` U+FFFD characters from `decode_replacement` encoding damage.
+
+## 日本語
+
+- **`validate` が U+FFFD replacement character 行を分類するようになりました (#2814)** — `replacement_char` issue に `origin` と `severity` metadata を追加し、JSON / MCP consumer が意図的な `source_literal` の U+FFFD とエンコーディング破損を示す `decode_replacement` を区別できるようにしました。

src/CodeIndex/Cli/ConsoleUi.cs

@@ -769,7 +769,7 @@ public static void PrintUsageBrief(bool showBanner = true)
         Console.WriteLine("  inspect <query>            Bundle definition, graph, and nearby symbol context");
         Console.WriteLine("  status                     Show database statistics, freshness, config, and logs");
         Console.WriteLine("  doctor                     Print a redacted environment summary for bug reports");
-        Console.WriteLine("  validate                   Report encoding issues (U+FFFD, BOM, null bytes, mixed line endings, UTF-16 BOM, likely non-UTF8)");
+        Console.WriteLine("  validate                   Report encoding issues (U+FFFD origin/severity, BOM, null bytes, mixed line endings, UTF-16 BOM, likely non-UTF8)");
         Console.WriteLine("  impact <query>             Show transitive callers; type queries may return heuristic file-level dependency hints");
         Console.WriteLine("  deps                       Show file-level dependency edges from the reference graph");
         Console.WriteLine("  unused                     Find symbols defined but never referenced (dead code)");
@@ -880,7 +880,7 @@ private static void PrintCommandSummary()
         Console.WriteLine("  db prune --dry-run|--apply Count or delete orphaned DB rows");
         Console.WriteLine("  diff <db1> <db2>           Compare two index databases; exit 0 identical, 1 drift, 2 schema mismatch, 3 unreadable");
         Console.WriteLine("  report --output <path>     Build a redacted crash-repro tarball (.tgz) for bug reports");
-        Console.WriteLine("  validate                   Report encoding issues (U+FFFD, BOM, null bytes, mixed line endings, UTF-16 BOM, likely non-UTF8)");
+        Console.WriteLine("  validate                   Report encoding issues (U+FFFD origin/severity, BOM, null bytes, mixed line endings, UTF-16 BOM, likely non-UTF8)");
         Console.WriteLine("  impact <query>             Show transitive callers; type queries may return heuristic file-level dependency hints");
         Console.WriteLine("  deps                       Show file-level dependency edges from the reference graph");
         Console.WriteLine("  unused                     Find symbols defined but never referenced (dead code)");

src/CodeIndex/Database/DbContext.cs

@@ -1599,7 +1599,9 @@ CREATE TABLE IF NOT EXISTS file_issues (
                 file_id         INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
                 kind            TEXT NOT NULL,
                 line            INTEGER NOT NULL DEFAULT 0,
-                message         TEXT NOT NULL
+                message         TEXT NOT NULL,
+                origin          TEXT,
+                severity        TEXT
             )");
 
                 // Key-value metadata: fold algorithm version, future per-subsystem schema markers
@@ -1631,6 +1633,8 @@ value  TEXT
                 EnsureColumn("symbols", "family_key", "TEXT");
                 EnsureColumn("symbols", "visibility", "TEXT");
                 EnsureColumn("symbols", "return_type", "TEXT");
+                EnsureColumn("file_issues", "origin", "TEXT");
+                EnsureColumn("file_issues", "severity", "TEXT");
                 EnsureColumn("symbols", "is_metadata_target", "INTEGER");
                 var rebuildsSymbolReferences = !ColumnIsNotNull("symbol_references", "file_id");
                 EnsureColumn(
@@ -1804,10 +1808,12 @@ CREATE TABLE file_issues (
                     file_id         INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
                     kind            TEXT NOT NULL,
                     line            INTEGER NOT NULL DEFAULT 0,
-                    message         TEXT NOT NULL
+                    message         TEXT NOT NULL,
+                    origin          TEXT,
+                    severity        TEXT
                 )
                 """,
-                "id, file_id, kind, line, message");
+                "id, file_id, kind, line, message, origin, severity");
         }
         finally
         {