x/text/collate: Key and KeyFromString silently ignore the collate.Force option

[danderson]

Go version

go version go1.22.5 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/dave/.cache/go-build'
GOENV='/home/dave/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/dave/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/dave/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/nix/store/3v17dij8rvg7q99009swxg52995r7s22-go-1.22.5/share/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/nix/store/3v17dij8rvg7q99009swxg52995r7s22-go-1.22.5/share/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.5'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2184709228=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Attempted to sort strings using x/text/collate's collate.Force option, which implements deterministic comparison per Unicode TR10 (https://www.unicode.org/reports/tr10/#Forcing_Deterministic_Comparisons). Specifically, it forces strings to only compare as equal if they are bit-identical. Strings that are equivalent but not bit-equal are forced into a total order by using bytewise comparison of the raw byte sequence as a tie-breaker.

What did you see happen?

Key and KeyFromString ignore collate.Force, and return 0 (a == b) for strings that are equivalent but not bit-identical. Demo: https://go.dev/play/p/s_fwVG8pBD5 compares "Québécois" in normalization forms C and D (equivalent but different character sequences), and gets:

Byte compare : 1
CompareString: 1
KeyFromString: 0

What did you expect to see?

All public APIs of Collator that are specified to compare things should be obeying the provided collation settings, or at least failing loud rather than silently producing a different collation order.

[gabyhelp]

Related Issues and Documentation

• x/text/collate: Compare and CompareString broken in "ka-shifted" collation mode #68166
• x/text/collate: ignores width even if the flag is not specified #23241
• x/text/collate: collation does not work for Korean #19087
• x/text/collate: ignores case and diacritics by default #67296
• x/text/collate: Collator is not concurrency safe #67286 (closed)
• x/text/collate: CompareString is not threadsafe #57314 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[danderson]

I looked through all the above bugs prior to filing this one. None are relevant to this issue.

[cagedmantis]

cc @mpvl