System.Console unexpectedly uses a UTF-8 encoding *with BOM* on Windows

[mklement0]

Note: #27258 (since fixed) addressed the same issue, but for Unix-like platforms only, because at the time I didn't realize that it should be fixed on Windows also.

On Windows too the preamble (BOM) should be removed from the UTF-8 encoding that the Console class uses when code page 65001 (chcp 65001) is in effect, given that cmd.exe - rightfully - has always operated without BOM in that case:

After all, any programs relying on the active code page should be able to blindly assume that stdin input / stdout output is encoded accordingly and shouldn't have to deal with a BOM (neither to redundantly imply the same code page nor to potentially signal a different encoding).

The presence of a BOM, in fact, breaks PowerShell's background-jobs feature (via Start-Job), as detailed here:

• https://windowsserver.uservoice.com/forums/301869-powershell/suggestions/14915283-job-cmdlets-fail-with-utf-8-codepage

Follow-on bugs:

• PowerShell task - remove BOM from [Console]::InputEncoding to prevent issues with Start-Job microsoft/azure-pipelines-tasks#9766
• Powershell scripts that use Start-Job fail on Ansible 2.0+ ansible/ansible#15770

It's fair to assume that other applications/languages/frameworks may be affected as well, which can range from outright failure, as in PowerShell's case, to mistakenly considering the BOM part of the data.

---

Demonstration of cmd's current behavior:

cmd's stdout output has always created BOM-less output with chcp 65001 in effect; e.g. the following creates t.txt as a BOM-less UTF-8 file.

C:\> chcp 65001
C:\> (echo hü)> t.txt
C:\> powershell -noprofile -c Format-Hex t.txt

           Path: C:\Users\jdoe\t.txt

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   68 C3 BC 0D 0A                                   hü..

The above shows that the file contains no BOM (and that char. ü was correctly encoded as UTF-8 as 2-byte sequence 0xC3 0xBC).

[carlossanlop]

@mklement0 would you like to work on this issue?

[mklement0]

Thanks for asking, @carlossanlop, but I'm hoping someone else can take this on.

[eiriktsarpalis]

As far as Console.Out is concerned, it would seem like the current implementation is stripping the preamble for Windows as well. I tested the following console app:

    class Program
    {
        static void Main()
        {
            Console.Write(Console.OutputEncoding.CodePage);
        }
    }

Using the latest .NET 5.0 preview sdk:

chcp 65001
dotnet run > foo.txt
powershell -noprofile -c Format-Hex foo.txt

Produces no BOM artifacts:

           Path: C:\Users\eitsarpa\source\repos\ConsoleApp14\ConsoleApp14\foo.txt

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   36 35 30 30 31                                   65001

It should be noted however that System.Console.OutputEncoding does not strip the preamble, so applying it to the raw stdout stream should not have the same effect.

Perhaps I'm misunderstanding what is broken here?

[mklement0]

My bad, @eiriktsarpalis - I missed that the referenced PR did fix the problem on Windows too, as you demonstrate.

Sorry for the noise - I'm closing this.