fix: correctly output non-ASCII text on Windows - #604
Open
DanielC000 wants to merge 1 commit into
Open
Conversation
Piped output on Windows used the console's active codepage, which defaults to something other than UTF-8, garbling non-ASCII text. This sets it to UTF-8 once at startup and restores it on exit.
|
coverage: 98.824% (-0.03%) from 98.858% — DanielC000:fix/windows-utf8-codepage into open-cli-tools:main |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #302.
I looked into this and your
chcp 65001 && <command>idea from 2023 doesn't actually work if you inline it as a single command like that —chcpruns and reports success, but the CJK text is already garbled by the time it prints. cmd.exe seems to convert the whole/ccommand-line argument using the codepage that was active beforechcpgets a chance to run. Confirmed on a real Windows 11 machine with raw byte output, not just what shows in a terminal.What does work: setting the codepage on concurrently's own process once at startup — basically what the original reporter's own
chcp 65001 && concurrently ...workaround was doing, since the codepage is a property of the console session and a freshly-piped child inherits whatever its parent console is set to. This does that in the CLI entry point only (not the library API, so nothing changes for anyone using concurrently programmatically), restores the original codepage on exit, and is a no-op if the codepage is already UTF-8 (increasingly common with Windows Terminal etc). Cost is onechcpprobe at startup on Windows, and nothing at all on other platforms or under--raw.Also tried cmd's own
/Uflag, since it doesn't needchcpat all. Doesn't help though — it only affects cmd.exe's own builtins, has zero effect on output from an external process like Node, and would have actively broken the currently-correct output from Node-based commands (most real usage), since it flips the encoding to UTF-16LE.Before:
After:
One thing I'm not sure about: since codepage is a per-console-session property, not per-process, this could in principle race with another codepage-sensitive tool running in the same terminal at the same time. It's scoped to the CLI only, and the restore hangs off the same completion path concurrently already uses for Ctrl+C, so I think the risk is small — but happy to put it behind a flag instead if you'd rather it not be automatic. WDYT?