Summary
TryDetectLanguageFromShebang (FileIndexer.cs:1866) reads the first 256 bytes and attempts UTF-8 decoding to recover the #!/... line. UTF-16 LE/BE files (rare but legal — Windows PowerShell can save scripts as UTF-16 by default) decode as garbage, so the shebang is missed and the file falls into the "no extension, no shebang" bucket and is silently skipped. Sibling concern to #1587 (shebang probe lacks pre-binary check / cap) but at a distinct layer: this one is encoding detection.
Where
src/CodeIndex/Indexer/Scanning/FileIndexer.cs:1866 (shebang decode)
Suggested approach
(1) Before decoding, probe the leading bytes for a BOM (EF BB BF UTF-8, FF FE UTF-16 LE, FE FF UTF-16 BE, 00 00 FE FF / FF FE 00 00 UTF-32). (2) When a UTF-16 BOM is detected, decode using the matching encoding before searching for #!. (3) Strip the BOM bytes from the buffer used for content hashing so the hash is stable across BOM-stripped re-saves. (4) Add fixtures with shebang lines saved as UTF-8, UTF-8-BOM, UTF-16-LE, UTF-16-BE; assert each is detected. (5) Cross-link with #1587 (shebang pre-binary check) — same probe site, complementary fix. (6) Cross-link with #1538 (extensionless shell scripts) — overall shebang-detection family.
Summary
TryDetectLanguageFromShebang(FileIndexer.cs:1866) reads the first 256 bytes and attempts UTF-8 decoding to recover the#!/...line. UTF-16 LE/BE files (rare but legal — Windows PowerShell can save scripts as UTF-16 by default) decode as garbage, so the shebang is missed and the file falls into the "no extension, no shebang" bucket and is silently skipped. Sibling concern to #1587 (shebang probe lacks pre-binary check / cap) but at a distinct layer: this one is encoding detection.Where
src/CodeIndex/Indexer/Scanning/FileIndexer.cs:1866(shebang decode)Suggested approach
(1) Before decoding, probe the leading bytes for a BOM (
EF BB BFUTF-8,FF FEUTF-16 LE,FE FFUTF-16 BE,00 00 FE FF/FF FE 00 00UTF-32). (2) When a UTF-16 BOM is detected, decode using the matching encoding before searching for#!. (3) Strip the BOM bytes from the buffer used for content hashing so the hash is stable across BOM-stripped re-saves. (4) Add fixtures with shebang lines saved as UTF-8, UTF-8-BOM, UTF-16-LE, UTF-16-BE; assert each is detected. (5) Cross-link with #1587 (shebang pre-binary check) — same probe site, complementary fix. (6) Cross-link with #1538 (extensionless shell scripts) — overall shebang-detection family.