strings: Map produces invalid utf-8 when passed PAD (U+0080)

[petercgrant]

What version of Go are you using (go version)?

go version go1.10.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOOS="darwin"
GOARCH="amd64"

What did you do?

Calling strings.ToLower with a PAD (U+0080) character causes the output to be invalid utf-8. It appears this bug is a faulty boundary condition in strings.Map: the comparisons r <= utf8.RuneSelf should perhaps be rewritten as r < utf8.RuneSelf.
https://play.golang.org/p/YVkdm_KyRPT

What did you expect to see?

0x41c280
true
0x61c280
true

What did you see instead?

0x41c280
true
0x6180
false

[gopherbot]

Change https://golang.org/cl/111286 mentions this issue: strings: fix encoding of \u0080 in map

[robpike]

There may well be a bug here (and I believe you've diagnosed it) but this is the first time I've heard U+0080 called PAD. Where did you get that name?

[as]

The word PAD is used on some websites that enumerate the Latin-1 Supplement block.

http://www.unicode-symbol.com/u/0080.html
https://www.compart.com/en/unicode/U+0080
https://codepoints.net/U+0080

But a document from unicode.org does not mention "PAD" or "PADDING CHARACTER" anywhere. It also does not render the glyph in a way that suggests "PAD" is an identifier for it.

http://www.unicode.org/charts/PDF/U0080.pdf

ISO 7816-4 described 0x80 in a padding scheme, also commonly known in some libraries as OneAndZeroes [SIC] padding, where it is the first byte followed by a number of 0x00 bytes.

[petercgrant]

Best I can tell the name PAD originated in RFC 1345. In addition to the links in the previous reply, the name also appears on the Wikipedia page for C0 and C1 control codes and on Wiktionary. Both sources include a morsel of history. The name PAD ironically appeared in a proposal for C1 control pictures, which was apparently accepted (because we have them now?) even though the control picture in the standard has the name XXX instead of PAD. I called it by the only unambiguous, if unofficial, name I found.

If you think the name (or lack thereof) is a problem, let's appreciate that we aren't tasked with deciphering what the character means. The answer to that starts with It depends... and ends with a headache.

[anupcshan]

@gopherbot Please consider this for backport to 1.10, it's a regression.

[gopherbot]

Backport issue(s) opened: #25479 (for 1.10).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[andybons]

@gopherbot please backport to 1.9 as well