Incorrect offset returned from String.IndexOf method when 0x1B character is involved

[dmzaharov]

Description

String.IndexOf method returns incorrect offset, when search string begins with 0x1B character. Same code running in .NET Framework 4.7.2 works correctly. Workaround for .NET Core consists of specifying StringComparison.Ordinal option to String.IndexOf call.

Reproduction Steps

        Encoding latin1 = Encoding.GetEncoding("ISO-8859-1");
        var buffer = System.IO.File.ReadAllText(@"C:\Temp\out_before.pcl", latin1);
        var p = buffer.IndexOf(Convert.ToChar(27) + "*p-10000x");  // returns incorrect offset 26, see attached PCL file

out_before.zip

Expected behavior

var p = buffer.IndexOf(Convert.ToChar(27) + "*p-10000x"); // returns 25 in .NET Framework, correct

Actual behavior

Incorrect string offset is returned

Regression?

.NET Framework 4.7.2 --- CORRECT (25)
.NET Core 3.1 --- CORRECT (25)
.NET Core 5 --- INCORRECT (26)
.NET Core 7 --- INCORRECT (26)
.NET Core 8 --- INCORRECT (26)
.NET Core 9 --- INCORRECT (26)

Known Workarounds

var p = buffer.IndexOf(Convert.ToChar(27) + "*p-10000x", StringComparison.Ordinal); // this fixes it in .NET 5,7,8,9

Configuration

No response

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

[tarekgh]

.NET Framework 4.7.2 and .NET Core 3.1 rely on NLS for globalization, whereas .NET 5.0 and later use the ICU library. You can find more details in the documentation here: https://learn.microsoft.com/en-us/dotnet/core/extensions/globalization-icu.

In your code, the IndexOf operation is performed without specifying a StringComparison option. In this case, the comparison is linguistic, meaning it depends on the user’s default culture and the underlying globalization library (NLS or ICU). As a result, different behavior is expected when using ICU, since ICU updates string comparison rules in accordance with the latest Unicode Standard.

Unicode defines certain characters that are ignored during linguistic comparisons. The full list of ignored characters is available here: https://www.unicode.org/charts/collation/chart_Ignored.html. Notably, character 0x1B (char 27) is one of these ignored characters. You can think of an ignored character as if it does not exist in the string during comparison, which explains why the result is position 26.

As you noted, using IndexOf with StringComparison.Ordinal forces a non-linguistic comparison based on ordinal character values, ensuring consistent and predictable results.