Sort order difference between PowerShell 5 and PowerShell 7

[jason-nyc]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

I experienced a slightly different sort order when sorting a list of files (strings) by name. Upon further review, I found that, for instance, the underline (95) and the period (46) are sorted differently.

In PowerShell 5, the underline sorts before the period.
In PowerShell 7, the underline sorts after the period.

This is on the same machine, the same get-culture.

I solved this by converting to an ArrayList and performing an Ordinal sort.
$list = New-Object System.Collections.ArrayList;$list.AddRange($mystrings ) $list.Sort([System.StringComparer]::Ordinal)

If I need to stick with Sort-Object, I convert the string to Hex strings of Unicode bytes and sort by that.
$mystrings | Sort-Object {[system.Text.Encoding]::UTF8.GetBytes($_)|ForEach-Object ToString X2}
Note that Sort-Object has a -culture option, but I couldn't find an InvariantCulture to use.

Expected behavior

# PowerShell 5
#
PS> "_" -gt "."
True
PS> $mystrings = "A","B","C","a","b","c","_","."
$mystrings | Sort-object
.
_
A
a
B
b
C
c
PS> $mystrings | Sort-object | %{[int][char]$_}
46
95
65
97
66
98
67
99

Actual behavior

# PowerShell 7
#
PS> "_" -gt "."
False
PS> $mystrings = "A","B","C","a","b","c","_","."
PS> $mystrings | Sort-object
_
.
A
a
B
b
C
c
PS> $mystrings | Sort-object | %{[int][char]$_}
95
46
65
97
66
98
67
99

Error details

No response

Environment data

$PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.3.9
PSEdition                      Core
GitCommitId                    7.3.9
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

$PSVersionTable

Name                           Value                                                                                                                                
----                           -----                                                                                                                                
PSVersion                      5.1.22621.2506                                                                                                                       
PSEdition                      Desktop                                                                                                                              
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                              
BuildVersion                   10.0.22621.2506                                                                                                                      
CLRVersion                     4.0.30319.42000                                                                                                                      
WSManStackVersion              3.0                                                                                                                                  
PSRemotingProtocolVersion      2.3                                                                                                                                  
SerializationVersion           1.1.0.1

Visuals

No response

[mklement0]

In short: The difference in behavior with respect to sorting and string comparisons is owed to the difference in internationalization libraries used in the legacy .NET Framework (underlying Windows PowerShell, whose latest and last version is 5.1.x) and .NET 5+ (underlying PowerShell (Core)). See .NET globalization and ICU and the following issues:

• Tab completion regression in ordering (does not match Linux shells, Windows explorer, or cmd) #14757
• Incorrect sort order when using tab completion and listing items #19736

[237dmitry]

but I couldn't find an InvariantCulture to use

Result is the same. -Culture 127

[microsoft-github-policy-service]

📣 Hey @jason-nyc, how did we do? We would love to hear your feedback with the link below! 🗣️

🔗 https://forms.office.com/r/P926k48jRJ

Microsoft Forms

[jason-nyc]

mkelement0 is right - sort order difference is an expected compromise as they expand PowerShell to more OS implementations etc. I was hopeful for Sort-object -Culture Inv but as 237dimtry points out, it doesn't help for this issue.

It's unfortunate that you have to build your own sort key if you expect the same sort across versions, even on the same machine.

[mklement0]

@jason-nyc, on Windows you can use the NLS opt-in (setting env. var. DOTNET_SYSTEM_GLOBALIZATION_USENLS to 1 beforehand) to get the old behavior back, but note that it invariably applies to the whole session (and if you persistently set this env. var., it applies to ALL sessions).

---

As an aside re the invariant culture (which indeed doesn't help):

Sort-Object -Culture 127 seems to be the only way to directly request [cultureinfo]::InvariantCulture via its LCID - its name is actually the empty string, so another way to obtain it is with [cultureinfo]::new('').

Thus, you would expect -Culture '' to work, but it doesn't - see:

• Sort-Object, unlike operators, uses the current culture and doesn't support requesting the invariant culture via -Culture #18655

By contrast, -Culture Inv, -Culture InvariantCulture or -Culture Bogus only accidentally works from what I can tell:

• It is the equivalent of, say, [cultureinfo]::new('Bogus'), which creates a custom culture ad hoc (with catch-all LCID 4096), which happens to use the invariant culture's settings by default.
• It is unfortunate that names that don't match a predefined culture are quietly accepted and result in this behavior - a simple typo can result in subtle bugs.
• A more sensible approach would be to accept only the names of predefined cultures, while allowing for the possibility of passing a custom culture as a [cultureinfo] object - which currently isn't supported, because the -Culture parameter is [string]-typed, as discussed in the linked issue.