feat(agent-core-v2): read UTF-16 text files by transcoding to UTF-8 - #2647
Conversation
🦋 Changeset detectedLatest commit: 9845c71 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f304c25ab
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (fileType.kind === 'unknown') { | ||
| return { | ||
| isError: true, | ||
| output: notReadableFileOutput(args.path), | ||
| }; | ||
| const detection = detectTextEncoding(header); |
There was a problem hiding this comment.
For a UTF-16 file that has a BOM but no NUL byte in the 512-byte header (for example an empty UTF-16 file or CJK-only content), detectFileType(...) returns text, so this branch is skipped and the readLines(...errors: 'strict') path tries to decode FF FE as UTF-8. Those valid BOM-marked UTF-16 files are still rejected instead of transcoded; run detectTextEncoding/BOM handling before falling through to the UTF-8 text path.
Useful? React with 👍 / 👎.
| if (isBinary && req.encoding !== 'base64') { | ||
| const detection = detectTextEncoding(sample); |
There was a problem hiding this comment.
For fs.read, a BOM-marked UTF-16 file can have no NUL bytes in the sample (e.g. repeated 你好), so detectBinary(sample) is false and this guard never runs. The response then slices raw UTF-16 bytes and labels them UTF-8, causing the web UI file viewer to show replacement/garbage for a valid UTF-16 file; trust detectTextEncoding's BOM result before gating on isBinary.
Useful? React with 👍 / 👎.
- detect UTF-16 LE/BE from a BOM or a zero-byte parity heuristic (tolerant of CJK content), derived from VS Code's encoding detection - Read tool and workspace fs.read transcode UTF-16 text to UTF-8 instead of refusing it as binary; larger than 10 MiB still refused - refuse other non-UTF encodings (e.g. GBK) with a clearer message
9f304c2 to
9845c71
Compare
1 participant
Related Issue
No linked issue — the problem is explained below.
Problem
The Read tool and the web UI file viewer refuse UTF-16 text files as binary. UTF-16 is what Windows Notepad uses when saving
.txt, so opening such a file fails with "not readable as UTF-8 text" even though it is plain text. Non-UTF encodings such as GBK are also refused with a generic message that gives no hint about the actual cause.What changed
Adds encoding detection derived from VS Code (
detectEncodingFromBuffer, MIT) and transcodes UTF-16 text to UTF-8 for display:FF FE/FE FF/EF BB BF), then recognizes BOM-less UTF-16 LE/BE via zero-byte placement — zeros at least twice and at exactly one index parity. This deliberately deviates from VS Code's stricter every-pair rule so mixed Latin/CJK content (whose UTF-16 units carry no zero byte) is still detected; real binaries (zeros at both parities, or an isolated zero byte) stay refused.iconv). Other non-UTF encodings (e.g. GBK) are refused with a clearer message that names the supported encodings and the conversion workaround.fs.readendpoint behind the web UI file viewer transcodes UTF-16 the same way (windowing applies to the decoded UTF-8 bytes); explicitbase64requests keep raw bytes. No protocol schema changes.Checklist
gen-changesetsskill, or this PR needs no changeset.gen-docsskill, or this PR needs no doc update.