PowerShell use Chinese character “” just like "" ?

[MMMMMoris]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

Both these two lines of code work in PowerShell.

Write-Host “这是一段中文: This a paragrah of Chinese”
Write-Host "这是一段中文: This a paragrah of Chinese"

Did PowerShell just mean to do so or it was just a mistake of our developers ?
I was thinking if it is related to UTF-8 character set.

As you can see from the picture above, my Windows was not set to use UTF-8 character set.
Here is the output of $OutputEncoding of my PowerShell

Preamble          :
BodyName          : utf-8
EncodingName      : Unicode (UTF-8)
HeaderName        : utf-8
WebName           : utf-8
WindowsCodePage   : 1200
IsBrowserDisplay  : True
IsBrowserSave     : True
IsMailNewsDisplay : True
IsMailNewsSave    : True
IsSingleByte      : False
EncoderFallback   : System.Text.EncoderReplacementFallback
DecoderFallback   : System.Text.DecoderReplacementFallback
IsReadOnly        : True
CodePage          : 65001

Expected behavior

`Write-Host “这是一段中文: This a a paragrah of Chinese”`
Code like this perhaps ought to be wrong.

Actual behavior

`Write-Host “这是一段中文: This a a paragrah of Chinese”`
Code like this works fine in PowerShell.

Error details

No response

Environment data

PSVersion                      7.4.1
PSEdition                      Core
GitCommitId                    7.4.1
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

[mklement0]

PowerShell, by - perhaps surprising - design, allows interchangeable use of ASCII-range and non-ASCII-range ("Unicode") punctuation for syntactic purposes.

E.g., " (QUOTATION MARK, U+0022) and “ (LEFT DOUBLE QUOTATION MARK, U+201C) can be used interchangeably, and can even be mixed in a given string literal; e.g.:

# OK, despite use of two different quotation marks.
"Hi!“

For more information and a complete list of interchangeable characters, see this Stack Overflow answer.

[mklement0]

P.S.: $OutputEncoding only comes into play when sending data from PowerShell commands to external (native) programs, and its default value is UTF-8 statically, i.e. it doesn't vary based on the current culture / system locale.

[Console]::OutputEncoding by default reflects the encoding that corresponds to the active console window's (output) code page, which in turn defaults to the OEM code page associated with host computer's system locale. In other words: [Console]::OutputEncoding by default reflects the code page reported by chcp.com

However, [Console]::OutputEncoding too is only relevant with respect to external (native) programs: it determines how PowerShell decodes output from the latter.

[237dmitry]

my Windows was not set to use UTF-8 character set

When I was using Windows I chose utf8 as the system locale and it was the best option. All applications, editors, file managers, cmd.exe, pwsh.exe and powershell.exe simply began to understand each other without additional actions.

[mklement0]

For clarification, I should add:
With respect to the issue at hand:

• None of the following matter: $OutputEncoding , [Console]::OutputEncoding, the system locale (aka language for non-Unicode programs) or current culture (Get-Culture) - they only matter when PowerShell communicates with external (native) programs.

• Irrespective of the settings above, PowerShell code such as Write-Host “这是一段中文: This a paragrah of Chinese” is (in memory) always represented as UTF-16LE, and therefore supports the full range of Unicode characters, including CJK characters and non-ASCII-range punctuation such as “

• As noted, the interchangeable use of " and “ (with analogous interchangeability of other punctuation characters with syntactic function) is by design.

[mklement0]

As an aside, @237dmitry:

• Using the system locale's UTF-8 option sets both the OEM and the ANSI code pages to 65001 (UTF-8).

• Unless prevented by backward-compatibility concerns (see below), this option is indeed preferable, as it makes most problems with respect to PowerShell's interaction with external (native) programs disappear, as you state.

• That said, existing code can break for the following reasons:

  • Using the system locale's UTF-8 option fundamentally changes the interpretation of preexisting BOM-less files: Any such files that were encoded based on a legacy ANSI Ior - less likely - OEM encoding), such as Windows-1252 (or - less likely - CP437) on US-English systems, will then be interpreted as UTF-8-encoded.

  • Any such file that contains characters outside the ASCII range will then be misinterpreted, because such characters require multi-byte sequences in UTF-8.

  • Notably, the following applications, which rely on the ANSI code page (including for BOM-less input files), are affected:

    • Windows PowerShell
    • Python

[MMMMMoris]

Acknowledged, thanks for your answers.

[237dmitry]

Using the system locale's UTF-8 option sets both the OEM and the ANSI code pages to 65001 (UTF-8)

Yes, it was very convenient. Mainly for exchanging files between Windows and Linux, even despite the factor of different line endings. There were no troubles at all.

Notably, the following applications, which rely on the ANSI code page (including for BOM-less input files), are affected

I had no problem with Windows PowerShell, I did not use Python on Windows

As for the issue, it is very bad that pwsh uses other characters as quotes. Moreover, not all of them, for example `u{ab} and `u{bb} work as expected, but `u{201c} and `u{201d} no:

$ «text»   # «text»: The term '«text»' is not recognized as...
$ “text”   # no error

[mklement0]

I had no problem with Windows PowerShell, I did not use Python on Windows

I was clarifying the backward compatibility concerns in general, also for future readers.

In neither environment will you have problems with ASCII files, but any existing files that were encoded based on a legacy ANSI code page containing non-ASCII characters will by default be misinterpreted by existing scripts in ANSI-based environments, which could include source-code files. Additionally, ANSI-based environments such as Windows PowerShell (Set-Content) and Python will then start creating BOM-less UTF-8 files by default, potentially breaking consumers of such files that make the fixed assumption that the legacy ANSI encoding is used.

bad that pwsh uses other characters as quotes.

I agree that this permissiveness was never a good idea (and is incomplete, as you point out), but we're stuck with it now.

I presume that the original design intent was to also accommodate source code created with word processors rather than plain-text editors, as the former have features such as auto-correcting " (QUOTATION MARK, U+0022) to “ (LEFT DOUBLE QUOTATION MARK, U+201C), and - (HYPHEN-MINUS, U+002D) to – (EN DASH, U+2013).

[237dmitry]

I spoke about personal experience; I simply have no local ASCII files left after ten or so years of using UTF8. If I come across such things, for example cue-files, I will recode them to UTF8:

(gc file.cue -enc ansi) | sc file.cue

[mklement0]

@237dmitry, as a general tip: After being told that your personal circumstances aren't really relevant to the discussion, I suggest not following that up with further discussion of them.

[237dmitry]

as a general tip

Thanks, I know what to do without your advices. I did not ask you to answer.

[mklement0]

Whether you asked for the tip or not is similarly irrelevant, as are assertions of your autonomy devoid of any other content.

I realize that I am adding to the existing distraction with the following, but hopefully it will inform future behavior:

• The issue at hand has nothing to do with system locales or code pages, unlike what @MMMMMoris originally thought.

• While it's worth briefly mentioning why the latter do not apply (as I did), their detailed discussion, including apparent guidance, amounts to a distraction.

• Yet you provided guidance, whose specific content I consider problematic, because it suggests a cure-all to character-encoding woes that comes with serious caveats, which weren't mentioned.

  • To counteract that, for the benefit of future readers, I spelled out those caveats - any given user can then decide whether they apply to them or not.

• After you continued with what I perceive to be a distraction, talking about your specific situation, I suggested no longer doing so.

Therefore, allow me to offer more advice:

• Before posting, please consider whether the contribution is relevant to the core issue; if distractions/tangents are warranted, I suggest clearly labeling them as such.

• Please refrain from cantankerous comments such as your previous one. (By contrast, my suggestion - to not continue discussing your personal situation as part of what was itself a distraction - contained actionable advice, which you're free to agree or disagree with, and free to take or not.)

[237dmitry]

As an aside

my suggestion - to not continue discussing your personal situation

What was this “as an aside” for? I just answered you.