Set-Content/Add-Content/Get-Content use an 8-bit character encoding by default, but the help topics state ASCII; problematic Core default file encoding

[mklement0]

This issue has two distinct aspects:

• discussion of an existing documentation bug
• discussion of the problematic fixed default file encoding currently (alpha16) chosen for Core.

Steps to reproduce

'ö' | Set-Content -NoNewline -Encoding ASCII tmp.txt 
'ö' | Add-Content -Encoding ASCII -NoNewline tmp.txt 
Get-Content -Encoding ASCII tmp.txt
(Get-Content -Encoding Byte -TotalCount 2 tmp.txt) | % { '0x{0:x}' -f $_ }
'--'
'ö' | Set-Content -NoNewline tmp.txt   # use default encoding
'ö' | Add-Content -NoNewline tmp.txt   # use default encoding
Get-Content tmp.txt                    # use default encoding
(Get-Content -Encoding Byte -TotalCount 2 tmp.txt) | % { '0x{0:x}' -f $_ }

Expected behavior

??
0x3f
0x3f
--
??
0x3f
0x3f

Actual behavior

??
0x3f
0x3f
--
öö
0xf6
0xf6

That is, ASCII encoding turns a non-ASCII character into literal ? (0x3f)

The fact that Set-Content without an -Encoding argument resulted in ö on reading implies that ASCII encoding wasn't used, and the specific byte value of 0xf6 further implies that that a single-byte, extended-ASCII encoding was used:

• For Windows PowerShell, it is the respective system's legacy codepage ("ANSI"), such as Windows-1252 on US-English systems, or Windows-1251 on Russian systems. In other words: the specific encoding is, to put it in Unix terms, locale-dependent.

• For PowerShell Core, as of alpha 16, it is ISO-8859-1, as @iSazonov helpfully points out (see his comment below for the source-code links).

  • Using a fixed encoding that is limited to 256 code points is problematic, however.
  • See @iSazonov's comment below and the discussion of the RFC about default file encodings.

In contrast, Get-Help Set-Content, Get-Help Add-Content, and Get-Help Get-Content state for parameter -Encoding:

Specifies the file encoding. The default is ASCII.

The help-topic sources (branch live) for the relevant cmdlets can be found here.

Additionally:

• While these cmdlets accept an encoding identifier Default, as used in other cmdlets, the help only mentions String.

• Given that the two appear to result in the same encoding - what is their relationship?

• The description for encoding String in the online help is inadequate:

Uses the encoding type for a string.

Environment data

PowerShell Core v6.0.0-alpha (v6.0.0-alpha.16) on Microsoft Windows 10 Pro (64-bit; v10.0.14393)

[iSazonov]

The default is changed for PowerShell Core to ISO-8859-1
For OEM too ISO-8859-1

File provider
GetContentReader
GetContentWriter

[iSazonov]

This is real regression (breaking change) and we should fix it.

[iSazonov]

The Default is changed for PowerShell Core to ISO-8859-1
For OEM ISO-8859-1 too.

Lets see PowerShell FullCRL.
Default use function GetACP (Retrieves the current Windows ANSI code page identifier for the operating system.) See .Net Framework Reference

OEM is in only PowerShell and use function GetOEMCP (Returns the current original equipment manufacturer (OEM) code page identifier for the operating system.) See PowerShell code

I don't know why the both encodings was added in Powershell. Maybe PowerShell PG comment this.

Default is still not released in CoreCLR - so it is external issue. Maybe PowerShell PG find out this internally with .Net team. And should we use waiting-netstandart20 label?
OEM is not in CoreCLR at all - so it seems is internal issue. Is it make sense to fix it?

[iSazonov]

@mklement0 See .Net Framework Reference
We can not use Windows-1252 as default.

[iSazonov]

Here Default is not PowerShell default, it is OS default. Every system can has own default. Original .Net code use GetACP() to get OS default code page. Modern Unix use en_US.UTF-8 as default.

[SteveL-MSFT]

cc @JamesWTruher @BrucePay

[iSazonov]

@mklement0 Current Windows Powershell behavior is based on .Net Framework and it is dynamic:

System	FileSystemCmdletProviderEncoding.Default (GetACP())	FileSystemCmdletProviderEncoding.OEM (GetOEMCP())
Windows English	1252	437
Windows Russian	1251	866

Current PowerShell Core behavior is hard-coded to ISO-8859-1. If we change it on Windows-1252 we still not get Windows PowerShell behavior:

System	FileSystemCmdletProviderEncoding.Default (GetACP())	FileSystemCmdletProviderEncoding.OEM (GetOEMCP())
Windows English	1252	437
Windows Russian	1252	437

Preferred solution is to fix this in CoreCLR.

As you can see then we can properly read and write files on Windows Russian with 1251 (default system) code page by:
Get-Content -Encoding Default and Set-Content -Encoding Default

As far as Unix locale names mapping Unix uses standard names Table of locales (This is not a complete sample). Therefore, the fix in CoreCLR will not be too difficult.

We should wait for MSFT expert conclusion and then it will be clear that we need to fix in the code and documentation.

[iSazonov]

@mklement0 We have a RFC draft for this. Welcome to discuss PowerShell/PowerShell-RFC#71

[mklement0]

@iSazonov: Good idea. I've cleaned up my comments here, and I've revised the original post to point to the relevant other issues / comments / RFC.