Terminal: [console]::InputEncoding and [console]::OutputEncoding are set to the wrong UTF-8 encoding (*with* BOM)

[mklement0]

Note: As of v6.1.0-rc.1, the console on Windows is fundamentally not configured to use UTF-8 yet - see #7233

While [console]::InputEncoding and [console]::OutputEncoding on macOS and Linux are set to UTF-8, the specific encoding variant used is the one with a BOM, which is the wrong one (though I'm unclear on what the practical implications are, given that streams, not files are typically involved).

This contrasts with automatic variable $OutputEncoding which correctly uses the BOM-less UTF-8 encoding.

Steps to reproduce (macOS and Linux)

[console]::InputEncoding.GetPreamble().Count, 
[console]::OutputEncoding.GetPreamble().Count, 
$OutputEncoding.GetPreamble().Count

Expected behavior

0
0
0

Actual behavior

3
3
0

That is, the 3-byte BOM is unexpectedly present in [console]::InputEncoding and [console]::OutputEncoding

Environment data

PowerShell Core v6.1.0-rc.1 on macOS 10.13.6
PowerShell Core v6.1.0-rc.1 on Ubuntu 16.04.4 LTS

[PetSerAl]

Well it is not PowerShell issue, but .NET one.

[mklement0]

I haven't looked into it, @PetSerAl - do you know where these encodings are assigned?

[PetSerAl]

@mklement0 I will reference Windows implementation. I am not sure if it behave the same on Linux or MacOS since it use platform dependent function on Windows.
Here it calls native GetConsoleOutputCP. For UTF-8 it would return 65001. There are not way to distinguish BOM/BOM-less flavor of UTF-8. Underlying console simply does not support that.
On the next line it just calls Encoding.GetEncoding which return UTF-8 with BOM if you pass 65001 as argument:

[Text.Encoding]::GetEncoding(65001).GetPreamble().Length # 3

[PetSerAl]

Though, at quick glance on .NET Core implementation, it looks like it supposed to return BOM-less UTF-8 in this case, so maybe I was wrong and it have some PowerShell related issue.

[mklement0]

Thanks, @PetSerAl. Indeed, given that [Text.Encoding]::Default.GetPreamble().Count is 0 - i.e., the BOM-less UTF-8 variant, it would stand to reason that the console use the same encoding.

[iSazonov]

It is here
https://github.com/dotnet/corefx/blob/e1a33f4a1cc582f42bd5ae0f887bf1aa9878d493/src/System.Console/src/System/ConsolePal.Unix.cs#L612
https://github.com/dotnet/corefx/blob/a10890f4ffe0fadf090c922578ba0e606ebdd16c/src/Common/src/System/Text/EncodingHelper.Unix.cs#L34

So on Unixes CoreFX looks in env variables "LC_ALL", "LC_MESSAGES", "LANG".
If the vars is absent the default is Utf8 without BOM. ( return enc ?? new UTF8Encoding(encoderShouldEmitUTF8Identifier: false);)

[mklement0]

Thanks for digging deeper, @iSazonov.

In the absence of said environment variables the behavior is correct, but it is broken if these environment variables are present and specify UTF-8 - and these environment variables are virtually never absent, because they are part of the current locale (culture) setting, and these days they indeed typically specify UTF-8.

I've filed a CoreFx bug - see https://github.com/dotnet/corefx/issues/32004

[BrucePay]

@mklement0 So this should be resolved as external correct?

[mklement0]

@BrucePay: Indeed, thanks - should've made that clearer.