go/doc: reconsider comment rewrites of `''` to `”`

[kortschak]

In ae3d890 as part of addressing #51082, a change was made to go/doc that rewrite all '' to ”. This makes semantic changes to comments where ' is used as a prime and '' is used as double prime, common in mathematical code.

This was raised in #51082 (comment) but essentially disregarded. A work around was suggested,

as a workaround, you could replace those with U+2032 PRIME and U+2033 DOUBLE PRIME. That is, f' → f′ and f'' → f″. (Those may look the same depending on your font.)

However on investigation with relevant fonts (the font here and the font used by pkg.go.dev) at 100%, U+2033 DOUBLE PRIME is barely distinguishable from U+2032 PRIME and worse, also barely distinguishable from other commonly used marks in the same position such as '*' (Comparison: f′ f″ f* ).

The change has made it harder to read these comments, harder to write them in a way that doesn't get mutated and easier for incorrectly formatted comments to be committed (f'' getting mutated to f” which is essentially indistinguishable from f″ at normal font sizes).

[mvdan]

f'' getting mutated to f” which is essentially indistinguishable from f″ at normal font sizes

I had to put my glasses on to see the difference at my regular zoom level, so I think you're right :) And this is on a 25" monitor, let alone a tiny phone screen.

[seankhliao]

cc @rsc

[zikaeroh]

Another example is when docs refer to the empty string, specifically JS where strings aren't necessarily double quoted, leading to undesirable doc replacements (and apparently, inserting quotes at the start/end of paragraphs that are not quotes?) as in: evanw/esbuild@296870e

[ALTree]

An example from #56380 (unable to write '' in a comment, closed as a dup of this one):

Since Go 1.19 (commit ae3d890) '' is unconditionally transformed to “ or ”. This means my comment:

// lexMultilineRawString consumes a raw string. Nothing can be escaped in such
// a string. It assumes that the beginning ''' has already been consumed and
// ignored.

Now becomes:

// lexMultilineRawString consumes a raw string. Nothing can be escaped in such
// a string. It assumes that the beginning ”' has already been consumed and
// ignored.

There are some other cases, especially when parsing things, where one might reasonably want to type '' inside a regular (non-codeblock) comment.

Looking at go/doc/comment/parse.go, there is no way to escape this behaviour, and I don't really see an obvious non-ugly way to write that comment.

[evanw]

I recently encountered a particularly annoying case of this behavior. I wanted to write this:

quoteChar should be either '"' or '\''

But no matter what I do, it keeps being turned into this:

quoteChar should be either '"' or '\”

This replacement makes no sense, and is not helpful at all. Really hoping this behavior can be reverted.

[seebs]

So, the more I look at this, the more I think that:

1. It's just plain wrong to alter quotes like this in documentation comments, even just when rendering.
2. It's extra special extremely wrong to do it to documentation comments and modify the source code.

I just don't think this actually makes things better. If I want Fancy Quotes, I can type them! We have full UTF-8 support. I do not think the assumption should be that, if someone writing documentation for code uses characters in it that have special significance in thousands of contexts across basically every widely-used programming language, the most likely interpretation is that they're wrong and we should fix their code for them.

I think the assumption should be that if someone wrote an apostrophe, they meant apostrophe, and if they wrote two apostrophes, they meant two apostrophes, and that if they wanted a fancy close-quote, they would have typed one.

There may have been a brief window during which everything could display fancy quotes, but nothing could type them or store them in files, and it thus made sense to try to "correct" them. It's not the case now. If I wanted to type those fancy quotes, I could! I could embed them in things, I could use an editor which inserted them where appropriate, I have lots of options.

What I don't have options for is writing documentation for my code and not having it corrupted by gofmt.

[kortschak]

Discussion of this problem has been going on since March 2022 (originally in other issues). It is marked as NeedsDecision and yet no decision maker has even commented here. My full expectation of how this will play out is that when they do turn up, the decision will be that it has been like this for too long to change.

[bcmills]

It is marked as NeedsDecision and yet no decision maker has even commented here.

The go fmt behavior was changed as part of proposal #51082 (see https://github.com/golang/proposal/blob/master/design/51082-godocfmt.md#reformatting-doc-comments).

Revisions to that behavior should also go through the proposal process, so I am marking this issue as a proposal now.