strings: EqualFold is not equivalent to comparing ToLower

[bjkail]

#52022 added this statement to the doc for strings.EqualFold:

EqualFold(s, t) is equivalent to Tolower(s) == Tolower(t).

This program seems to be a counterexample.

(Additionally, it seems like the doc meant to say ToLower rather than Tolower?)

[hopehook]

Thanks. I'll follow up on this issue.

[hopehook]

It seems like the doc meant to say ToLower rather than Tolower?

ToLower

[gopherbot]

Change https://go.dev/cl/398736 mentions this issue: strings, bytes: improve the description of simple case-folding in EqualFold

[typesanitizer]

Interesting! It seems like a bug in go staticcheck, which recommends replacing ToLower comparison with EqualFold, even though it is not semantics-preserving.

Also, I did write a fuzz test locally and it passed without finding a counter-example. Maybe the default fuzzing limit is too low.

Given that this transformation has been incorrectly described in multiple places:

• DigitalOcean blog describes it as an "optimization" even though it is not semantics-preserving.
• A popular StackOverflow answer has a comment recommending using < after ToLower, which would be inconsistent with EqualFold.

Would it make sense to add a note of caution to the ToLower docs? (And maybe to the ToUpper docs too?) Something like:

Avoid comparing two strings after case transformation; use EqualFold instead. For example, "ς" (Greek Small Letter Final Sigma) and "σ" (Greek Small Letter Sigma) are equivalent under case-folding but not under comparison via lowercase.

Doesn't need to be pure text, maybe it can be a runnable example.

[ainar-g]

Interestingly enough, this particular example seems to be okay with strings.ToUpper. See https://go.dev/play/p/JFkZxxPOhWe.

[typesanitizer]

Example where comparison via ToUpper is different from comparison via EqualFold. https://go.dev/play/p/9g3vAh5wCKN

[ianlancetaylor]

Thanks for the examples.

@hopehook Sorry for the incorrect suggestion.

[hopehook]

@ianlancetaylor Thanks Ian for the help. I really underestimated the complexity here.

[hopehook]

@typesanitizer @ainar-g Thanks for the examples.

I think you are more professional than me. If there is a good solution that can be implemented, I suggest that you can open another issue, solve it together, and help it become better.