PowerShell: Handle output encoding

[Taitava]

I think on Unix like operating systems the Bash / Dash / Zsh shells work quite nicely with the UTF-8 encoding. At least I haven't yet encountered issues for them.

But on Windows..... It's a sad, sad situation. I tried to research for a solution for encoding problems in CMD in issue #5, but failed. I decided not to try anymore to find a solution for CMD. I thought that the newer shells, PowerShell 5 / PowerShell Core would fix the encoding issues, but no. At least not automatically.

An example of bad output on PowerShell 5 / PowerShell Core

1. Use the following example shell command:

   echo "Ä"
   

2. Set shell to be either PowerShell 5 or PowerShell Core .
3. Set the output channel to be Notification balloon.
4. Execute the shell command, and you'll see that the result is a questionmark:
   

---

This can be fixed by preceding the command with:

$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding; (Source)

So the command becomes:

$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding; echo "Ä"

And the output is now correct:

This is only a workaround, and I want to find a more easy to use solution.

Different ways to try to solve this problem

• Determine the output encoding automatically, and then convert to utf-8 when receiving output? For this, Node.js's child_process module's exec() function has an option named encoding, that Node.js uses to convert output from the specified encoding to utf-8. The issue with this is that the set of supported encodings seems to be quite limited: "ascii" | "utf8" | "utf-8" | "utf16le" | "ucs2" | "ucs-2" | "base64" | "latin1" | "binary" | "hex".
• Let a user pick an encoding for each shell in the settings. The user would choose from the same encoding options defined above, so it might be that some users can't select an encoding that would be correct for them. I also don't like the fact that fixing encoding problems would always be manual work for users. An automatic solution would be better especially for non-tech-savvy users.
• Let a user define a shell specific script file that will be ran before each shell command. PowerShell's version of the script file could contain the above-mentioned $OutputEncoding = ... code. Again, manual work for users, and very confusing for newcomers.
• Does PowerShell have some environment variable that I could use to set a specific output encoding?
• Something else?

What about input encoding?

I want to emphasize that this discussion is only about output encoding. PowerShell's input encoding has problems, too (e.g. discussion #155 - edit: actually, it might be that that discussion does not have an input related issue, but output instead, as the problem occurs in error message output). This whole encoding topic is so huge and confusing to me that I try to solve it in smaller pieces. If I do find a solution to this output encoding problem, maybe it will lead to a solution for input, too. But that discussion can be started in a different discussion topic.

[Taitava]

The Templater plugin has the same problem, and their documentation contains some interesting links: https://silentvoid13.github.io/Templater/faq.html