Summary
FileIndexer.cs:1793-1802 flags any file containing a NUL byte as a possible-binary candidate. UTF-16 text legitimately contains NUL bytes between every ASCII character (the high byte is 0x00). PowerShell scripts saved by older Windows tools and many MS Office text exports land in this trap: the file is flagged with a FileIssue and may be skipped, even though it is plain UTF-16 text. The fix is encoding-aware: NUL density is meaningful only after subtracting expected high-byte zeros from a UTF-16 detection.
Where
src/CodeIndex/Indexer/Scanning/FileIndexer.cs:1793-1802 (NUL byte detection)
Suggested approach
(1) Before flagging on NUL byte presence, run BOM detection (UTF-16 LE/BE BOM ⇒ skip the NUL check entirely; the bytes are encoding artifacts). (2) Without a BOM, fall back to a heuristic: if the NUL bytes are at a regular stride (every 2nd byte for UTF-16), classify as UTF-16 rather than binary. (3) Otherwise, require a NUL density above a threshold (e.g. >5% of the leading 4 KB) before flagging as binary. (4) Persist the detected encoding as file metadata so downstream extractors can decode correctly. (5) Add fixtures: pure UTF-8 text, UTF-16 LE BOM script, UTF-16 BE without BOM, true binary. (6) Cross-link with #442 (shebang UTF-16 detection — same encoding-detection family) and the just-filed #1789 binary detection issue.
Summary
FileIndexer.cs:1793-1802flags any file containing a NUL byte as a possible-binary candidate. UTF-16 text legitimately contains NUL bytes between every ASCII character (the high byte is 0x00). PowerShell scripts saved by older Windows tools and many MS Office text exports land in this trap: the file is flagged with aFileIssueand may be skipped, even though it is plain UTF-16 text. The fix is encoding-aware: NUL density is meaningful only after subtracting expected high-byte zeros from a UTF-16 detection.Where
src/CodeIndex/Indexer/Scanning/FileIndexer.cs:1793-1802(NUL byte detection)Suggested approach
(1) Before flagging on NUL byte presence, run BOM detection (UTF-16 LE/BE BOM ⇒ skip the NUL check entirely; the bytes are encoding artifacts). (2) Without a BOM, fall back to a heuristic: if the NUL bytes are at a regular stride (every 2nd byte for UTF-16), classify as UTF-16 rather than binary. (3) Otherwise, require a NUL density above a threshold (e.g. >5% of the leading 4 KB) before flagging as binary. (4) Persist the detected encoding as file metadata so downstream extractors can decode correctly. (5) Add fixtures: pure UTF-8 text, UTF-16 LE BOM script, UTF-16 BE without BOM, true binary. (6) Cross-link with #442 (shebang UTF-16 detection — same encoding-detection family) and the just-filed #1789 binary detection issue.