fmt: %q quotes out-of-range codepoints to strings containing invalid runes

[bcmills]

What version of Go are you using (go version)?

$ go version
devel go1.19-d1820f748f Mon Mar 7 14:07:18 2022 +0000

Does this issue reproduce with the latest release?

Yes.

What did you do?

Called fmt.Sprintf("%q", x) with various int32 values that are not valid as UTF-8 runes (https://go.dev/play/p/wbvbvuKIWCC?v=gotip).

	for _, r := range []rune{
		-1,
		surrogateMin,
		unicode.ReplacementChar,
		unicode.MaxRune + 1,
	} {
		s := fmt.Sprintf("%q", r)
		fmt.Printf("%s (%x)\n", s, s)
	}

What did you expect to see?

The documentation for the fmt package says that %q formats the value as “a single-quoted character literal safely escaped with Go syntax.”

Per https://go.dev/ref/spec#Rune_literals, “The escapes \u and \U represent Unicode code points so within them some values are illegal, in particular those above 0x10FFFF and surrogate halves.

So I expected fmt.Sprintf to emit either an invalid \u or \U literal (since these values are too large for \x or \0 escapes), or an explicit format-error string of some sort (along the lines of https://pkg.go.dev/fmt#hdr-Format_errors).

At the very least, I expected to see some explicit description of the invalid-range behavior in the fmt package docs!

What did you see instead?

Invalid codepoints uniformly (and lossily) quoted to the Unicode replacement character:

devel go1.19-d1820f748f Mon Mar 7 14:07:18 2022 +0000
'�' (27efbfbd27)
'�' (27efbfbd27)
'�' (27efbfbd27)
'�' (27efbfbd27)

(CC @robpike)

[gopherbot]

Change https://go.dev/cl/390424 mentions this issue: internal/fuzz: fix encoding for out-of-range ints and runes

[rsc]

I think this is happening in strconv.QuoteRune not fmt. It's also what the Unicode replacement character is for. That rune is the explicit error indicator.

[bcmills]

I think this is happening in strconv.QuoteRune not fmt.

Seems plausible, but fmt could make the behavior clearer (because it already has a format for recovered errors, whereas strconv does not).

It's also what the Unicode replacement character is for. That rune is the explicit error indicator.

Maybe? Again, at the very least I would expect to see an explicit description in the documentation: the word “replacement” doesn't even appear in the fmt docs today, and “replaced” only appears in the context of reflect.Value and the * flag.

[robpike]

I was about to write what @rsc said. I don't think fmt should behave differently from strconv in this regard. But perhaps documentation could be improved.

This is not what I would call a "recovered error", or even a formatting error (such as use of an invalid verb or missing argument). This is just a question of how to represent invalid UTF-8, and there is lots of consistent precedent for that.

[gopherbot]

Change https://go.dev/cl/390436 mentions this issue: fmt: document use of Unicode replacement character in %q

[gopherbot]

Change https://go.dev/cl/390816 mentions this issue: [release-branch.go1.18] internal/fuzz: fix encoding for out-of-range ints and runes