go/doc: improve headings, lists, and links

[rsc]

This proposal improves support for headings, lists, and links in Go doc comments,
while remaining backwards compatible with existing comments.
It includes a new package, go/doc/comment, exposing a parsed syntax tree for
doc comments, and it includes changes to go/printer and therefore gofmt
to format doc comments in a standard way.

For example, existing lists reformat from the display on the left to the one on the right:

URL links can be rewritten to change the display on the left to the one on the right:

And package doc comments (and others) can be rewritten to link to specific symbols:

The full design doc is at https://github.com/golang/proposal/blob/master/design/51082-godocfmt.md.

[gopherbot]

Change https://go.dev/cl/384263 mentions this issue: go/doc/comment: new package

[gopherbot]

Change https://go.dev/cl/384257 mentions this issue: all: fix various doc comment formatting nits

[gopherbot]

Change https://go.dev/cl/384267 mentions this issue: cmd/go: gofmt alldocs.go

[gopherbot]

Change https://go.dev/cl/384258 mentions this issue: all: fix TODO comment hanging indents

[gopherbot]

Change https://go.dev/cl/384265 mentions this issue: go/doc: use go/doc/comment

[gopherbot]

Change https://go.dev/cl/384266 mentions this issue: cmd/doc: use new go/doc APIs

[gopherbot]

Change https://go.dev/cl/384268 mentions this issue: all: gofmt main repo

[gopherbot]

Change https://go.dev/cl/384264 mentions this issue: go/printer: format doc comments

[gopherbot]

Change https://go.dev/cl/384259 mentions this issue: all: remove trailing blank doc comment lines

[gopherbot]

Change https://go.dev/cl/384260 mentions this issue: all: separate doc comment from //go: directives

[gopherbot]

Change https://go.dev/cl/384274 mentions this issue: internal/pkgdoc: update go/doc API usage for links

[rsc]

The top comment has been filled in now.
See #51082 (comment).

[jimmyfrasche]

This solves 90% of my problems working with docstrings and makes the remaining 10% trivial. I am ecstatic!

I'd be perfectly happy to have this accepted as-is but I do have a couple minor thoughts:

I'd s/Text/Inline/g and then s/Plain/Text/g to align with html. That could just be familiarity but it seems clearer to me.

Allow subheadings with ##, ###, and so on. I've seen a few packages since the original discussion that could use at least one extra level of indentation for grouping subsections (I did not think to record the packages in question, sorry). This could be added later, but if so it may be prudent to accept ## x as a heading now so it doesn't change from a paragraph to a heading later. Though at that point you're most of the way to an implementation.

[rsc]

For now I think we should keep things simple and not increase scope past the previous discussion. There would really be no problem with having ## render as a paragraph during a later transition. Single # is rendering as a paragraph during this one.

[kortschak]

In numbered point 5 in the goals and non-goals, the absence of in-line image support is noted for a couple of languages and support by one, but no mention of Python (a much more relevant language than for example Perl). Here it does exist in some projects https://numpy.org/doc/stable/reference/generated/numpy.fft.fft.html#numpy.fft.fft and in general, Python doc strings may be valid reStructuredText which allows embedding of images.

[rsc]

Python supports images in their docs, noted.
All the problems listed for images still exist though, and images remain out of scope for this change.

[ainar-g]

I am extremely excited about this! A couple of questions / suggestions.

1. Currently, the pkg.go.dev renderer automatically converts text like RFC NNNN into a link to the corresponding RFC. From what I can tell, that's a feature of the website's renderer as opposed to one of godoc itself. Will that behaviour of the renderer remain intact?

2. The point about ASCII double quotes doesn't mention if they are reformatted in the preformatted text. I think, it is worth explicitly mentioning that they won't be replaced with the Unicode ones in that position and adding test cases for that if there isn't one.

   (This may seem obvious, but anyone who ever tried editing a Shell script with the macOS TextEdit program will know that it sometimes isn't, heh.)

3. In the numbered list section, it's currently not clear, what happens when N is above 9. I assume that it will become “␠NN.␠”?

4. In the heading section, perhaps there should be an example of a line that begins with a space and then a #. Markdown allows a certain amount of spaces before the heading character, and if the new format doesn't allow that then it's a differrence worth mentioning explicitly, I think.

Again, really looking forward to this in any case.

[kortschak]

Overall this looks good, but I want to restate my concern about the accessibility issues I raised with the approach to marking links.

[rsc]

Thanks for the questions, @ainar-g.

1. I dont know for sure, but I think the idea is to drop the special case RFC NNNN support on pkg.go.dev and let people write links. The current support does not handle references to subsections well, for example. It was probably a mistake for pkg.go.dev to diverge this way. There is a bit more prior discussion at #48305 (comment).

2. Doubled ASCII single quotes (`` and '') are reformatted. This is called out explicitly:

   The ASCII double-single-quote forms that have always been defined to render as “ and ” are replaced with those.

3. If N is 10, then “␠N.␠” would be “␠10.␠”. This seems clearer than writing “␠NN.␠” - wouldn't that be “␠1010.␠”?

4. Thanks for the suggestion. Added that as an example.

[rsc]

@kortschak Noted re link syntax. I don't have anything more to add beyond #48305 (comment).

[kortschak]

It's worth noting that I did follow that up OOB with a detailed response, but did not receive a reply.

[thepudds]

Overall, this is exciting to see.

A few quick questions about linking to headers and the relationship to the base URL.

1. For linking to headers in the same package, would something like the following work?

See [Some Heading] for more information.

[Some Heading]: #some-heading

2. Would that rely on knowing the default header anchor ID algorithm (or doing a copy/paste or similar)? From the API, it looks to be proposed as:

// The default anchor ID is constructed by dropping all characters
// except A-Z, a-z, 0-9, space, and tab, converting to lower case,
// splitting into space-or-tab-separated fields, and then rejoining
// the fields using hyphens. For example, if the heading text is
// “Everybody Says Don't”, the default ID is “everybody-says-dont”.

3. Would you be able to link to headers in other packages without needing to know the base URL, such as something like:

[Another Heading]: /some/import/path#another-heading

3. Anchor IDs removing non-ASCII letters certainly simplifies things, but if linking to something like this (one of the examples from #7349) is contemplated for the future, simply changing the anchor ID algorithm might then break other links that are mixed ASCII/non-ASCII headers?

# これは、ヘッダです

In any event, I think the ability to link to headers would be extremely useful, especially for complex packages.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[shuLhan]

@rsc

I think there is a bug with list formatting when comment start and end with /**/ in the current tip branch, or maybe a missing exception in the proposal since the spec still allow comment start/end with /**/.

Test code

See https://go.dev/play/p/WfIdp2WQphD?v=gotip

Expected result

/*
testCommentSlashStart method with header and list

# Header

Here is the list,

  - a
  - b
*/

Got result

/*
testCommentSlashStart method with header and list

# Header

Here is the list,

* a
* b
*/
func testCommentSlashStart() {}

[ianlancetaylor]

@shuLhan Would you mind opening a new issue for that specific problem? Thanks.

[ianlancetaylor]

I believe that all changes described here have been submitted, so closing this issue. Please comment if you disagree.

[ianlancetaylor]

Still one pending CL: the doc comment for the comment package: https://go.dev/cl/397287.

[bcmills]

The changes for this proposal appear to have introduced a crash in go/doc (#52783).

[mknyszek]

This is only waiting documentation now, right?

[gopherbot]

Change https://go.dev/cl/407137 mentions this issue: all: gofmt main repo