go/doc: improve headings, lists, and links

[rsc]

This proposal improves support for headings, lists, and links in Go doc comments,
while remaining backwards compatible with existing comments.
It includes a new package, go/doc/comment, exposing a parsed syntax tree for
doc comments, and it includes changes to go/printer and therefore gofmt
to format doc comments in a standard way.

For example, existing lists reformat from the display on the left to the one on the right:

URL links can be rewritten to change the display on the left to the one on the right:

And package doc comments (and others) can be rewritten to link to specific symbols:

The full design doc is at https://github.com/golang/proposal/blob/master/design/51082-godocfmt.md.

[gopherbot]

Change https://go.dev/cl/384263 mentions this issue: go/doc/comment: new package

[gopherbot]

Change https://go.dev/cl/384257 mentions this issue: all: fix various doc comment formatting nits

[gopherbot]

Change https://go.dev/cl/384267 mentions this issue: cmd/go: gofmt alldocs.go

[gopherbot]

Change https://go.dev/cl/384258 mentions this issue: all: fix TODO comment hanging indents

[gopherbot]

Change https://go.dev/cl/384265 mentions this issue: go/doc: use go/doc/comment

[gopherbot]

Change https://go.dev/cl/384266 mentions this issue: cmd/doc: use new go/doc APIs

[gopherbot]

Change https://go.dev/cl/384268 mentions this issue: all: gofmt main repo

[gopherbot]

Change https://go.dev/cl/384264 mentions this issue: go/printer: format doc comments

[gopherbot]

Change https://go.dev/cl/384259 mentions this issue: all: remove trailing blank doc comment lines

[gopherbot]

Change https://go.dev/cl/384260 mentions this issue: all: separate doc comment from //go: directives

[gopherbot]

Change https://go.dev/cl/384274 mentions this issue: internal/pkgdoc: update go/doc API usage for links

[rsc]

The top comment has been filled in now.
See #51082 (comment).

[jimmyfrasche]

This solves 90% of my problems working with docstrings and makes the remaining 10% trivial. I am ecstatic!

I'd be perfectly happy to have this accepted as-is but I do have a couple minor thoughts:

I'd s/Text/Inline/g and then s/Plain/Text/g to align with html. That could just be familiarity but it seems clearer to me.

Allow subheadings with ##, ###, and so on. I've seen a few packages since the original discussion that could use at least one extra level of indentation for grouping subsections (I did not think to record the packages in question, sorry). This could be added later, but if so it may be prudent to accept ## x as a heading now so it doesn't change from a paragraph to a heading later. Though at that point you're most of the way to an implementation.

[rsc]

For now I think we should keep things simple and not increase scope past the previous discussion. There would really be no problem with having ## render as a paragraph during a later transition. Single # is rendering as a paragraph during this one.

[kortschak]

In numbered point 5 in the goals and non-goals, the absence of in-line image support is noted for a couple of languages and support by one, but no mention of Python (a much more relevant language than for example Perl). Here it does exist in some projects https://numpy.org/doc/stable/reference/generated/numpy.fft.fft.html#numpy.fft.fft and in general, Python doc strings may be valid reStructuredText which allows embedding of images.

[rsc]

Python supports images in their docs, noted.
All the problems listed for images still exist though, and images remain out of scope for this change.

[ainar-g]

I am extremely excited about this! A couple of questions / suggestions.

1. Currently, the pkg.go.dev renderer automatically converts text like RFC NNNN into a link to the corresponding RFC. From what I can tell, that's a feature of the website's renderer as opposed to one of godoc itself. Will that behaviour of the renderer remain intact?

2. The point about ASCII double quotes doesn't mention if they are reformatted in the preformatted text. I think, it is worth explicitly mentioning that they won't be replaced with the Unicode ones in that position and adding test cases for that if there isn't one.

   (This may seem obvious, but anyone who ever tried editing a Shell script with the macOS TextEdit program will know that it sometimes isn't, heh.)

3. In the numbered list section, it's currently not clear, what happens when N is above 9. I assume that it will become “␠NN.␠”?

4. In the heading section, perhaps there should be an example of a line that begins with a space and then a #. Markdown allows a certain amount of spaces before the heading character, and if the new format doesn't allow that then it's a differrence worth mentioning explicitly, I think.

Again, really looking forward to this in any case.

[kortschak]

Overall this looks good, but I want to restate my concern about the accessibility issues I raised with the approach to marking links.

[rsc]

Thanks for the questions, @ainar-g.

1. I dont know for sure, but I think the idea is to drop the special case RFC NNNN support on pkg.go.dev and let people write links. The current support does not handle references to subsections well, for example. It was probably a mistake for pkg.go.dev to diverge this way. There is a bit more prior discussion at #48305 (comment).

2. Doubled ASCII single quotes (`` and '') are reformatted. This is called out explicitly:

   The ASCII double-single-quote forms that have always been defined to render as “ and ” are replaced with those.

3. If N is 10, then “␠N.␠” would be “␠10.␠”. This seems clearer than writing “␠NN.␠” - wouldn't that be “␠1010.␠”?

4. Thanks for the suggestion. Added that as an example.

[rsc]

@kortschak Noted re link syntax. I don't have anything more to add beyond #48305 (comment).

[kortschak]

It's worth noting that I did follow that up OOB with a detailed response, but did not receive a reply.

[thepudds]

Overall, this is exciting to see.

A few quick questions about linking to headers and the relationship to the base URL.

1. For linking to headers in the same package, would something like the following work?

See [Some Heading] for more information.

[Some Heading]: #some-heading

2. Would that rely on knowing the default header anchor ID algorithm (or doing a copy/paste or similar)? From the API, it looks to be proposed as:

// The default anchor ID is constructed by dropping all characters
// except A-Z, a-z, 0-9, space, and tab, converting to lower case,
// splitting into space-or-tab-separated fields, and then rejoining
// the fields using hyphens. For example, if the heading text is
// “Everybody Says Don't”, the default ID is “everybody-says-dont”.

3. Would you be able to link to headers in other packages without needing to know the base URL, such as something like:

[Another Heading]: /some/import/path#another-heading

3. Anchor IDs removing non-ASCII letters certainly simplifies things, but if linking to something like this (one of the examples from #7349) is contemplated for the future, simply changing the anchor ID algorithm might then break other links that are mixed ASCII/non-ASCII headers?

# これは、ヘッダです

In any event, I think the ability to link to headers would be extremely useful, especially for complex packages.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Yes, this is waiting on docs, which was in turn blocked on my analysis of the rewrites on Google's code base, so I know what to call out in the docs. Just finished the analysis this morning. I'm aiming to have the docs sent out this week.

[rsc]

In addition to docs, this is waiting on integration with pkgsite and x/website/cmd/golangorg.
@jba has done the former but won't submit it until beta1 is available.
I can do x/website.

Only the docs and some minor gofmt fixes I've discovered block the beta.
Once they are submitted we can add okay-after-beta1.

[gopherbot]

Change https://go.dev/cl/410358 mentions this issue: go/doc/comment: add doc comment

[gopherbot]

Change https://go.dev/cl/410359 mentions this issue: go/doc/comment: do not turn ``` into “`

[gopherbot]

Change https://go.dev/cl/410394 mentions this issue: _content/doc: document doc comments at go.dev/doc/comment

[rsc]

All that remains is updating x/website and pkgsite; marked okay-after-beta1.

[heschi]

Ping: beta1 is out.

[mattn]

I leave this comment with a my real experience. I created the following pull-request with a Go tip. It's my fault for using Go tip, so it's not someone else's problem to blame. Just my fault. It should be noted that this change will not be undone if you run the new gofmt and then run it again with the old gofmt.

charmbracelet/bubbletea#343

I am not against this pull-request because in my personal opinion this change is reasonable. This is an issue that everyone should be aware of.

[ianlancetaylor]

@jba what is the status of supporting the new doc comment code on pkgsite? That is the only work remaining for this 1.19 issue. Thanks.

[gopherbot]

Change https://go.dev/cl/414694 mentions this issue: _content/doc/comment: add note about conflicting package imports

[jba]

The pkgsite code has been updated to use the new doc comment syntax. It has not been deployed yet.

[gopherbot]

Change https://go.dev/cl/415174 mentions this issue: go/doc/comment: support links in lists in comments

[rsc]

I think I still need to update x/website as well.

[gopherbot]

Change https://go.dev/cl/415554 mentions this issue: go/doc/comment: support doc links to builtins

[thomaspeugeot]

Naive question : is there a way to reference Note in a link ?

consider the following example

// NOTE(boiled egg cooking time experiment): following
// cooking time have been tested
//
// 6' : too hard
// 5' : fine
// 4' : too soft

// BoiledEggCookingTime is the desired cooking duration
//
// see [NOTE(boiled egg cooking time experiment)]
const BoiledEggCookingTime = time.Duration(5 * time.Minute)

would // see [NOTE(boiled egg cooking time experiment)] works ?
If "boiling egg cooking time experiment" is not typed correctly, would gopls spots the broken link ?