Repository navigation
strings: Split is inconsistent with bytes.Split #53511
Description
Consider the following:
in := "\xff-\xff"
sep := ""
fmt.Printf("%q\n", bytes.Split([]byte(in), []byte(sep))) // ["\xff" "-" "\xff"]
fmt.Printf("%q\n", strings.Split(string(in), string(sep))) // ["�" "-" "\xff"]The results of these two are inconsistent where the strings implementation replaced
\xff with utf8.RuneError, even though the documentation of strings.Split
mentions no such behavior. It only says that it slices up the input.
Furthermore, it is odd, that it only mangles the first element of the result, but not the last.
Activity
gopherbot commentedon Jun 23, 2022
Change https://go.dev/cl/413715 mentions this issue:
strings: avoid utf.RuneError mangling in Splitcagedmantis commentedon Jun 24, 2022
@griesemer
nightlyone commentedon Jul 13, 2022
https://go.dev/play/p/7PPxD1AyxTq shows it together with 2 other situations where utf8 rune splitting is being done that are handled differently again.
To me it looks like the strings and bytes package are both being inconsistent to each of the other documented rune splitting methods and probably both need to be adjusted.
Or maybe I am missing something here and those functions are supposed to support rune splitting in a different way in order to allow for edge cases I am not aware of.
dsnet commentedon Jul 14, 2022
@nightlyone. My CL makes it such that your example prints:
This brings the
bytesandstringspackage in conformance to each other.Regarding the last two lines in your example, there is no documented guarantee that
strings.Split(s, "")needs to be identical to[]rune(s). In fact,strings.Splitis technically bound by the documentation to always produce a list of subslices. Converting 0xff to the rune error violates that. On the other hand,[]rune(s)is documented by the language specification that it does do the rune error mangling.strings: avoid utf8.RuneError mangling in Split
strings: avoid utf8.RuneError mangling in Split