doc comment revisions: headings, lists, and links

[rsc]

I am looking into the possibility of revising Go's doc comment syntax, specifically adjusting headings and adding lists and links. This discussion is meant to gather feedback before writing an official proposal.

The current Go doc comment format has served us well since their introduction in 2009. There has only been one significant change, which was the addition of headings in 2011. But there are also a few long-open issues and proposals about doc comments, including:

• #7349 points out that the headings rule does not work well with non-Roman scripts.
• #31739 points out that lines ending with double quotes cannot be headings.
• #34377 points out that lines ending with parens cannot be headings.
• #7873 asks for list support.
• #45533 proposes linking of symbols, written as [io.EOF], to make it easier to write good top-level doc comments and cross-reference with other packages.

It makes sense, as we approach a decade of experience, to take what we've learned and make one coherent revision, setting the syntax for the next 10 or so years.

Goals and non-goals

The primary design criteria for Go doc comments was to make them readable as ordinary comments when viewing the source code directly, in contrast to systems like C#'s Xmldoc, Java's Javadoc, and Perl's Perlpod. The goal was to prioritize readability, avoiding syntactic ceremony and complexity. This remains as a primary goal.

Another concern, new since 2009, is backwards compatibility. Whatever changes we make, existing doc comments must generally continue to render well. Less important, but still something to keep in mind, is forward compatibility: keeping new doc comments rendering well in older Go versions, for a smoother transition.

Another goal for the revamp is that it include writing a separate, standalone web page explaining how to write Go doc comments. Today that information is squirreled away in the doc.ToHTML comment and is not easily found or widely known.

Within those constraints, the focus I have set for this revamp is to address the issues listed above. Specifically:

1. Make the header syntax more predictable. The headings rule is clearly difficult to remember and has too many false negatives. But further adjustments of the current rule run the risk of false positives.

2. Add support for lists. There are many times in documentation when a bullet or numbered list is called for. Those appear in many doc comments today, as indented <pre> blocks.

3. Add support for links to URLs. Today the only way to link to something is by writing the URL directly, but those can sometimes be quite unreadable and interrupt the text.

4. Add support for links to Go API documentation, in the current package and in other packages. This would have multiple benefits, but one is the ability in large packages to write top-level doc comments that give a good overview and link directly to the functions and types being described.

I believe it also makes sense to add another goal:

5. Add formatting of doc comments to gofmt, to promote consistent appearance and create more room for future changes.

It is not a goal to support every possible kind of documentation or markup. For example:

• Plain text has served us very well so far, and while some might prefer that comments allow font changes, the syntactic ceremony and complexity involved seems not worth the benefit, no matter how it is done.

• People have asked for support for embedding images in documentation (see #39513), but that adds significant complexity as well: image size hints, different resolutions, image sets, images suitable for both light and dark mode presentation, and so on. It is also difficult (but not impossible) to render them on the command line. Although images clearly have important uses, all this complexity is in direct conflict with the primary goal. For these reasons, images are out of scope. I also note that C#'s Xmldoc and Perl's Perlpod seem not to have image support, although Java's Javadoc does.

Markdown is not the answer, but we can borrow good ideas

An obvious suggestion is to switch to Markdown; this is especially obvious given the discussion being hosted on GitHub where all comments are written in Markdown. I am fairly convinced Markdown is not the answer, for a few reasons.

First, there is no single definition of Markdown, as explained on the CommonMark site. CommonMark is roughly what is used on GitHub, Reddit, and Stack Overflow (although even among those there can be significant variation). Even so, let's define Markdown as CommonMark and continue.

Second, Markdown is not backwards compatible with existing doc comments. Go doc comments require only a single space of indentation to start a <pre> block, while Markdown requires more. Also, it is common for Go doc comments to use Go expressions like `raw strings` or formulas like a*x^2+b*x+c. Markdown would instead interpret those as syntactic markup and render as “raw strings or formulas like ax^2+bx+c”. Existing comments would need to be revised to make them Markdown-safe.

Third, many features in Markdown are not terribly readable. The basics of Markdown can be simple and punctuation-free, but once you get into more advanced uses, there is a surfeit of notation which directly works against the goal of being able to read (and write) program comments in source files without special tooling. Markdown doc comments would end up full of backquotes and underscores and stars, along with backslashes to escape punctuation that would otherwise be interpreted specially. (Here is my favorite recent example of a particularly subtle issue.)

Fourth, Markdown is surprisingly complex. Markdown, befitting its Perl roots, provides more than one way to do just about anything: _i_, *i*, and <em>i</em>; Setext and ATX headings; indented code blocks and fenced code blocks; three different ways to write a link; and so on. There are subtle rules about exactly how many spaces of indentation are required or allowed in different circumstances. All of this harms not just readability but also comprehensibility, learnability, and consistency. The ability to embed arbitrary HTML adds even more complexity. Developers should be spending their time on the code, not on arcane details of documentation formatting.

Of course, Markdown is widely used and therefore familiar to many users. Even though it would be a serious mistake to adopt Markdown in its entirety, it does make sense to look to Markdown for conventions that users would already be familiar with, that we can tailor to Go's needs. If you are a fan of Markdown, you can view this revision as making Go adopt a (very limited) subset of Markdown. If not, you can view it as Go adopting a couple extra conventions that can be defined separately from any Markdown implementation or spec.

Headings

The current rule is:

A span that consists of a single line, is followed by another paragraph span, begins with a capital letter, and contains no punctuation other than parentheses and commas is formatted as a heading.

I can never remember the details of this exact rule, despite having chosen it. Every time I write a heading, I worry about whether it's going to be recognized as such. Others clearly have the same problem (#7349, #31739, #34377). The rule avoided the need for visible syntax, but in retrospect visible syntax would have been simpler. As Markdown shows us, that syntax can be very lightweight: a single “#” would suffice. Therefore I suggest the following:

New Rule: If a span of non-blank lines is a single line beginning with # followed by a space or tab and then additional text, then that line is a heading.

# This is a heading

Here are some examples of variations that do not satisfy the rule and are therefore not headings:

#This is not a heading, because there is no space.

# This is not a heading,
# because it is multiple lines.

The next span is not a heading, because there is no additional text:

#

In the middle of a span of non-blank lines,
# this is not a heading either.

Transition: The old heading rule will remain valid, which is acceptable since it mainly has false negatives, not false positives.
Gofmt will rewrite old-style headings into new-style headings, so that the fact of being a heading is made clearer to readers.

Lists

There is no support for lists today. As noted before, documentation needing lists uses indented <pre> blocks instead.

For example, here are the docs for cookiejar.PublicSuffixList:

// PublicSuffixList provides the public suffix of a domain. For example:
//      - the public suffix of "example.com" is "com",
//      - the public suffix of "foo1.foo2.foo3.co.uk" is "co.uk", and
//      - the public suffix of "bar.pvt.k12.ma.us" is "pvt.k12.ma.us".
//
// Implementations of PublicSuffixList must be safe for concurrent use by
// multiple goroutines.

And here are the docs for url.URL.String:

// In the second form, the following rules apply:
//      - if u.Scheme is empty, scheme: is omitted.
//      - if u.User is nil, userinfo@ is omitted.
//      - if u.Host is empty, host/ is omitted.
//      - if u.Scheme and u.Host are empty and u.User is nil,
//         the entire scheme://userinfo@host/ is omitted.
//      - if u.Host is non-empty and u.Path begins with a /,
//         the form host/path does not add its own /.
//      - if u.RawQuery is empty, ?query is omitted.
//      - if u.Fragment is empty, #fragment is omitted.

Ideally, we'd like to adopt a rule that makes these into bullet lists without any edits at all. (Markdown's space-counting rules would make these <pre> blocks, not lists.)

Today, a span of lines all indented by one or more spaces or tabs is always a <pre> block.
I suggest the following:

New Rule: In a span of lines all blank or indented by one or more spaces or tabs (which would otherwise be a <pre> block),
if the first indented line begins with a bullet list marker or a numbered list marker,
then that span of indented lines is a bullet list or numbered list.
A bullet list marker is a dash, star, or plus followed by a space or tab and then text.
In a bullet list, each line beginning with a bullet list marker starts a new list item.
A numbered list marker is a decimal number followed by a period or right parenthesis, then a space or tab, and then text.
In a numbered list, each line beginning with a number list marker starts a new list item.
Item numbers are left as is, never renumbered (unlike Markdown).

Using this rule, the two doc comments above are both recognized and formatted as bullet lists, not as <pre> blocks.

Note that the rule means that a list item followed by a blank line followed by additional indented text continues the list item (regardless of comparative indentation level):

// Text.
//
//  - A bullet.
//
//     Another paragraph of that first bullet.
//
// - A second bullet.

Note also that there are no code blocks inside list items—any indented paragraph following a list item continues the list item, and the list ends at the next unindented line—nor are there nested lists. This avoids all of the space-counting subtlety of Markdown.

To re-emphasize, a critical property of this definition of lists is that it makes existing doc comments written with pseudo-lists turn into doc comments with real lists.

Transition: Gofmt will rewrite recognized bullet and numbered lists to use a standard format. For example, the two doc comments above would reformat to:

// PublicSuffixList provides the public suffix of a domain. For example:
//
//   - the public suffix of "example.com" is "com",
//   - the public suffix of "foo1.foo2.foo3.co.uk" is "co.uk", and
//   - the public suffix of "bar.pvt.k12.ma.us" is "pvt.k12.ma.us".
//
// Implementations of PublicSuffixList must be safe for concurrent use by
// multiple goroutines.

and:

// In the second form, the following rules apply:
//
//   - if u.Scheme is empty, scheme: is omitted.
//   - if u.User is nil, userinfo@ is omitted.
//   - if u.Host is empty, host/ is omitted.
//   - if u.Scheme and u.Host are empty and u.User is nil,
//     the entire scheme://userinfo@host/ is omitted.
//   - if u.Host is non-empty and u.Path begins with a /,
//     the form host/path does not add its own /.
//   - if u.RawQuery is empty, ?query is omitted.
//   - if u.Fragment is empty, #fragment is omitted.

The specific formatting rules are discussed in the Formatting section below.

Markdown recognizes three different bullets: -, *, and +. In the main Go repo, the dash is dominant: in comments of the form `//[ \t]+[-+*] ` (grepping, so some of these may not be in doc comments), 84% use -, 14% use *, and 2% use +. In a now slightly dated corpus of external Go code, the star is dominant: 37.6% -, 61.8% *, 0.7% +.

Markdown also recognizes two different numeric list item suffixes: “1.” and “1)”. In the main Go repo, 66% of comments use “1.” (versus 34% for “1)”). In the external corpus, “1.” is again the dominant choice, 81% to 19%.

We have two conflicting goals: handle existing comments well, and avoid needless variation. To satisfy both, all three bullets and both forms of numbers will be recognized, but gofmt (see below) will rewrite them to a single canonical form: dash for bullets, and “N.” for numbers. (Why dashes and not asterisks? Proper typesetting of bullet lists sometimes does use dashes, but never uses asterisks, so using dashes keeps the comments looking as typographically clean as possible.)

Links to URLs

Documentation is more useful with clear links to other web pages. For example, the encoding/json package doc today says:

// Package json implements encoding and decoding of JSON as defined in
// RFC 7159. The mapping between JSON and Go values is described
// in the documentation for the Marshal and Unmarshal functions.
//
// See "JSON and Go" for an introduction to this package:
// https://golang.org/doc/articles/json_and_go.html

There is no link to the actual RFC 7159, leaving the reader to Google it. And the link to the “JSON and Go” article must be copied and pasted. Loosely following the Markdown shortcut reference link format, I suggest the following:

New Rule: A span of unindented non-blank lines defines link targets when each line is of the form “[Text]: URL”. In other text, “[Text]” represents a link to URL using the given text—in HTML, <a href="URL">Text</a>.

For example:

// Package json implements encoding and decoding of JSON as defined in
// [RFC 7159]. The mapping between JSON and Go values is described
// in the documentation for the Marshal and Unmarshal functions.
//
// For an introduction to this package, see the article
// “[JSON and Go].”
//
// [RFC 7159]: https://tools.ietf.org/html/rfc7159
// [JSON and Go]: https://golang.org/doc/articles/json_and_go.html

Note that the link definitions can only be given in their own “paragraph” (span of non-blank unindented lines), which can contain more than one such definition, one per line. If there is no corresponding URL declaration, then (except for doc links, described in the next section) the text is not a hyperlink, and the square brackets are preserved.

This format only minimally interrupts the flow of the actual text, since the URLs are moved to a separate section. As already noted, it also roughly matches the Markdown shortcut reference link format, without the optional title text.

Transition: Gofmt will move link definitions to the end of the overall doc comment. Go vet will flag unused link targets. Older versions of Go will show the text verbatim, which is fairly readable.

Links to Go API documentation

Documentation is also more useful with clear links to other documentation, whether it's one function linking to another, preferred version or a top-level doc comment summarizing the overall API of the package, with links to the key types and functions. Today there is no way to do this. Names can be mentioned, of course, but users must find the docs on their own.

Following discussion on #45533, I suggest to treat doc links like the links in the previous section, without target definitions. Specifically:

New Rule: Doc links are links of the form “[Name1]” or “[Name1.Name2]” to refer to exported identifiers in the current package, or “[pkg]”, “[pkg.Name1]”, or “[pkg.Name1.Name2]” to refer to identifiers in other packages.
In the second form, “pkg” can be either a full import path or the assumed package name of an existing import.
The assumed package name is either the identifier in a renamed import or else the name assumed by goimports. (Goimports inserts renamings when that assumption is not correct, so this rule should work for essentially all Go code.)
A “pkg” is only assumed to be a full import path if it starts with a domain name (a path element with a dot) or is one of the packages from the standard library (“[os]”, “[encoding/json]”, and so on).
To avoid problems with maps, generics, and array types, doc links must be both preceded and followed by punctuation, spaces, tabs, or the start or end of a line.

For example, if the current package imports encoding/json, then “[json.Decoder]” can be written in place of “[encoding/json.Decoder]” to link to the docs for encoding/json's Decoder.

The implications and potential false positives of this implied URL link are presented by Joe Tsai here. In particular, the false positive rate appears to be low enough not to worry about.

To illustrate the need for the punctuation restriction, consider:

// Constant folding computes the exact constant value ([constant.Value])
// for every expression ([ast.Expr]) that is a compile-time constant.

versus

// The Types field, a map[ast.Expr]TypeAndValue,
// holds type-checking results for all AST expressions.

and

// A SHA1 hash is a [Size]byte.

Transition: Older versions of Go will show the text verbatim, which is still fairly readable.

Formatter

Along with the changes, I suggest we add to go/doc a function

func Format(text string) string

that reformats a doc comment in the conventional presentation, and then to have go/printer and gofmt invoke this formatter.

The formatter would canonicalize the input so that it formatted exactly as before but with the following properties:

• All paragraphs are separated by single blank lines.
• Legacy headings are converted to “#” headings.
• All code blocks are indented by a single tab.
• All code blocks have a blank line before and after.
• All list markers are written as “␠␠-␠“ (space space dash space) or “␠N.␠“ (space number dot space).
• All list item continuation text, including additional paragraphs, is indented by four spaces.
• All lists have a blank line before and after.
• If there is a blank line anywhere in a list, there are blank lines between all list elements.
• All bullet list items use - as the bullet (* and + are converted).
• All numbered list items use the “N.” form (“N)” is converted).
• The ASCII double-single-quote forms that have always been
  defined to render as “ and ” are replaced with those.
• Link URLs are moved to the end of the doc comment.

The formatter would not reflow paragraphs, so as not to prohibit use of the semantic linefeeds convention.

This canonical formatting has the benefit for Markdown aficionados of being compatible with the Markdown equivalents. The output would still not be exactly Markdown, since various punctuation would not be (and does not need to be) escaped, but the block structure Go doc comments and Markdown have in common would be rendered as valid Markdown.

Additional API

The current doc.ToHTML is given only the comment text and therefore cannot implement import-based links to other identifiers. To address this, we would need to add ToHTML and ToText methods to the Package type, and define that the top-level functions are as though calling the methods on a zero value of the struct. The ToHTML method will need to take a new Config struct that, at the least, allows specifying the URL prefix of the documentation server (for example, / or https://pkg.go.dev/ or https://golang.org/pkg/). It would also need to specify the HTML tag for headings, which will depend on the surrounding page where the docs will be presented.

There is an accepted proposal to add doc.ToMarkdown for easy conversion of Go doc comments to Markdown, and we would implement and update that as part of this work. It too would be added to the Package type.

[scott-cotton]

Great to see this overview, thanks!

Generally:
I think that a sort of unspoken looming problem for the doc format is related to the tooling. 10 years ago there was godoc -http= and godoc.org. Now it appears work is shifting to pkg.go.dev, and I have heard that there is work on a command we can run to host docs locally. So now I, and at least some others, use godoc locally and pkg.go.dev otherwise. The problem w.r.t. the format here is that 10 years ago there was more or less one tool. Now there is a doc.ToMarkdown proposal, and momentum is shifting towards pkg.go.dev, without any interest in replicating things like godoc -analysis=xyz.

In such an ecosystem, there is a natural drive to diversify formats, which I do not think is good for Go. However, perhaps it would be fruitful to think of the doc format for the next 10 years as serving a wider set of tools than godoc.org. To me, the principal of "to make [docs] readable as ordinary comments when viewing the source code directly" is not bad because different tools can present the information without needing to delve into rapidly evolving formats or formats with several variants.

Specifically:
I think that the lists as they are are difficult to read in the source code, happy to see proposals to use a list format. I also spend time working around the lack of links, especially to local symbols. It would be great if a new doc format supported links to local symbols, and if, like in the source code, we could refer to those symbols in the scope of where the source code is (taking into account imports).

Looking forward to seeing how this evolves, and personally hoping something a little bit nicer than ascii for math can be included (though unicode math support in editors, eg … ⊧ ∃x ax²+bx+c seems to help a little).

[leetrout]

I'll toss in another vote for AST. I spent a weekend pulling comments out and trying to move them to unist so that we could offer a public rendering of our private package and docs.

I think it would be very useful to have this officially supported so folks could trim down and render their docs however best suits them.

[rsc]

Personally, I think we should only provide an AST, and not get into the business of formatting HTML or Markdown.

Even if we do add an AST, it would be a mistake to remove ToText, ToHTML, and ToMarkdown. Those are incredibly useful. Not everyone wants to reimplement them.

[rsc]

And for what it's worth. I'm mostly convinced about the AST in some form. I will think more about how to make it most convenient.

[dsnet]

Even if we do add an AST, it would be a mistake to remove ToText, ToHTML, and ToMarkdown. Those are incredibly useful.

As a data point, I should note that pkg.go.dev avoids using doc.ToHTML, but has it's own HTML formatter since it needs more customization than what ToHTML provides. I think a ToHTML function is fine to provide, so long as we accept that it's intended for basic usages. I'm primarily concerned about the feature bloat the ToHTML API will grow in order to accommodate all the reasonable ways that a higher-level HTML renderer might want to tweak its output.

[rsc]

FWIW I looked at the pkg.go.dev formatter, and the situation is at the least unclear. It has a comment saying it could be removed once Go 1.14 was out, but clearly other changes have been made since then, and the comment appears stale. But looking at the commits it appears that many of the changes are bug fixes that we probably should put back into the standard go/doc. I have yet to find anything that requires pkg.go.dev to be permanently diverged from go/doc's ToHTML (but it's hard to reconstruct everything). Instead it seems like they have a copy just to be able to make fixes quicker. And the fixes I see are not about HTML customization but about corner cases - real bugs.

[dmitshur]

Issue #38056 may be worth considering, if it hasn't already been, as it has potential intersection with what's described in the "Links to URLs" section.

To add some historical background, it appears godoc.org had automatic RFC linking since its initial commit in 2014, with additional enhancements (golang/gddo#276, golang/gddo#282) in 2015.

[rsc]

I'm a little uneasy with special-casing RFCs.
Why are they above the bar?
What other things might be special cased, and why are they below the bar?

[dsnet]

I feel like the explicit linking syntax with [] is sufficient.

I had code where I wrote "RFC 8785, Appendix B." and pkg.go.dev failed to auto-link it. Automatically linking RFC works in some common cases, but it's annoying when it doesn't. We could improve RFC auto-linking, but the advantage with explicit linking is that the author has full control over making it work as we intended.

[ghost]

A couple of the documentation websites automatically link to RFCs (godocs, pkg.go). Perhaps gofmt should apply the linking logic used by these sites to create the explicit links shown in the issue.

[rsc]

I've definitely considered that, but I am not entirely convinced we should special-case RFCs. It feels like overfitting to one very specific use case for Go. Surely there are other kinds of standards that would merit this too? Where do we draw the line? Or: why are RFCs above the line and not other things?

[rsc]

Let's call this thread a duplicate of #48305 (comment).

[ghost]

I am suggesting that gofmt (perhaps behind a fix switch) create explicit links for what pkg.go.dev does automatically. The line is what go.pkg.dev currently implements. The justification for this line is that people are accustomed to RFC linking on godocs.org and its successors. The end goal is to remove the special case for RFCs from the rest of the ecosystem.

Speaking of a fix switch, should some of the proposed gofmt transformations be behind a switch? Perhaps somebody will want to write text that happens to match the legacy header rule.

[rsc]

Perhaps somebody will want to write text that happens to match the legacy header rule.

In that case I hope it will be useful to see more clearly that it's actually going to be rendered as a header.
This is analogous to gofmt rewriting 1+2 * 3+4 into 1 + 2*3 + 4.
The rewrite also makes it possible, in some future version, to disable the old header rule, but I'm not suggesting that now.

(It's also very easy to avoid matching the header rule: end the text with a period, or split one line into two.)

[arp242]

If an AST is exposed (as discuses earlier) then people can write tools to automatically create/generate links to RFCs, GitHub issues, Debian bug tracker, PornHub videos, or whatever you want.

For example, personally I'd think that writing a tool to automatically create a link from #666 to [#666] and [#666]: https://github.com/issues/666 would be pretty useful, but I don't think it needs to be in the standard Go formatter, and by linking this explicitly rather than doing it in the HTML conversation step also means you have more control (e.g. for linking to specific sections, comments, using a different text, etc.)

[jimmyfrasche]

The ToHTML Config should let you specify the level of heading tag to use, which is important for screen reader UX.

[rsc]

Good point, thanks, added.

[jimmyfrasche]

Similarly it would be nice to be able to override the IDs for the headings to make them play nice with the surrounding html.

[ghost]

For example, if the current package imports encoding/json, then “[json.Decoder]” can be written in place of “[encoding/json.Decoder]” to link to the docs for encoding/json's Decoder.

Because imports are scoped to a file, should this say "if the current file imports" instead of "the current package imports"?

[dsnet]

Since I'm advocating for "current file"-only semantics, I think there are several reasonable workarounds.

• Option 1: Use package-path qualified names. For example: [crypto/rand.Reader]
• Option 2: Import the given package and make use of it:

  // This package uses [rand.Reader].
  package mypkg
  
  import "crypto/rand"
  
  var _ = rand.Reader

• Option 3: Depending on exactly how the rules operate, you could imagine a blank import also being sufficient:

  // This package uses [rand.Reader].
  package mypkg
  
  import _ "crypto/rand"

[dsnet]

Well, OK, but what about doc.go when there is an unambiguous reference to json?

My concern is that it might be an unambiguous reference today, but then someone adds an import in another file that introduces ambiguities for references in doc.go. I'm not sure if it's a good user experience if changing the imports in a specific file requires thinking about how it may affect references in another file.

[AndrewHarrisSPU]

I'd rather accept that linking isn't very magical and manage the mapping between inline markup and well-qualified links myself. Whether or not the tooling can interpret doc symbols unambiguously, I feel like it's fair to ask that the author interpret them unambiguously. It's something I find holistically useful when the author is me.

This code uses different packages for [-rand.Rand] and [-rand.Reader]. See [-blog] for details.

[rand.Rand]: doc:math/rand.Rand
[rand.Reader] doc:crypto/rand.Reader
[blog] https://www.blog.com/

If there is an opportunity to examine an AST generated from a comment, the symbol could be changed to a •, making that an easy way to spot where the AST is hanging, and unobtrusive in plain text.

This code uses multiple rand imports.

  • [•rand.Rand] is from math/rand
  • [•rand.Reader] is from crypto/rand

[rand.Rand]: doc:math/rand.Rand
[rand.Reader] doc:crypto/rand.Reader
[blog] https://www.blog.com/

[ghost]

Attributing the package doc to a single file is not always possible because the package doc is the the concatenation of the non-empty package comments. One workaround is to discourage multiple package comments with a go vet warning.

[earthboundkid]

I favor a hybrid, package/file approach. Suppose a.go imports math/rand, and doc.go imports nothing but says "This package uses [rand.Rand]." If crypto/rand isn't imported in another file, the docs should link [rand.Rand] to math/rand, but if b.go imports crypto/rand, then the package name is ambiguous and it should fail to link. Link imports per package if unambiguous, per file if ambiguous.

[diamondburned]

How exactly would GoDoc handle links that have the same inner text, such as the
snippet below?

// Here is a paragraph. See link 1 [here]; link 2 [here].
//
// [here]: ???
// [here]: ???

The first paragraph contains 2 links that have the same inner text, so there
doesn't seem to be a way to declare them properly.

I think having an additional syntax like [here][here-1] (or [here][1])
similarly to Markdown's hyperlinks would solve this issue.

[dsnet]

What's the use case for this? The text within the link needs to be a unique reference. It seems more sensible to do:

// Here is a paragraph. See [link 1]; [link 2].
//
// [link 1]: ???
// [link 2]: ???

[diamondburned]

There's not really a particular "use case" for this; it's just something that
might often come up when writing things up. For example, it's often natural for
me to say "refer to the documentation [here] or [here]", but that might not
always be translatable over.

With the current design, the issue goes beyond just the word "here" as well. I
think hyperlinking on any single word will be quite difficult, since the common
words would collide more often within a single link paragraph.

I suppose enumerating the text might be easier, but it doesn't always feel
natural, especially when I might not have control over what the comment is,
especially during code generation. If I stumble upon a link with the same inner
text, then I will likely have to enumerate that, making it ugly for the reader.

[fzipp]

@diamondburned Using "here" (or even worse "click here") as a link text is bad practice anyway, see:

https://www.washington.edu/accessibility/checklist/links/
https://www.w3.org/QA/Tips/noClickHere

I don't think we have to support it.

[diamondburned]

So what about code generation? Do I have to enumerate the texts then?

[rsc]

@diamondburned I don't understand your question.

[kylelemons]

For assumed package names, I also (lightly) propose handling a few other common corner cases that I've noticed. These usually come up as the names for GitHub repositories (to avoid conflicts and make the language clear), and so if there are any Go source files at the root of the repository, the assumed package names would be invalid identifiers.

• "pkg-go" is assumed to be named "pkg" (after ignoring any "/vN" suffix)
• "pkg.go" is assumed to be named "pkg" (after ignoring any "/vN" suffix)
• (maybe) "go-foo" is assumed to be named "pkg" -- this hasn't come up for us, but my memory says I've seen this more than once.

(we have had to write some of this down in our style guide since this has come up a few times, though this is at least partly our own fault, as we have Go source files at the root of the baseplate.go repository, following the prior convention from baseplate.py)

[dsnet]

The evidence for this seems strong.

I found:

• 8530 cases where stripping a go- prefix results in the correct package name
• 2029 cases where stripping a -go suffix results in the correct package name
• 204 cases where stripping a go. prefix results in the correct package name
• 114 cases where stripping a .go suffix results in the correct package name

For comparisons, I found:

• 6047 cases where stripping a /vN suffix results in the correct package name
• 1278 cases where stripping a .vN suffix results in the correct pacakge name

[rsc]

Thanks very much for pointing this out.

I thought I was describing what goimports does, but I was not.
Now I've changed to be "what goimports does".
When that doesn't produce the correct identifier, goimports (and gopls) inserts a renaming anyway, so that should make the rule work for essentially all Go code.

[kylelemons]

👍 Tying it to how goimports assumes names definitely makes sense to me.

For the curious, it looks like goimports handles all but the 204 go.-prefixed names, which seems like a pretty good batting average:

https://play.golang.org/p/Xbcmy1EaLKZ

ImportPathToAssumedName("foo")       = "foo"
ImportPathToAssumedName("foo/v1")    = "foo"
ImportPathToAssumedName("foo.v1")    = "foo"
ImportPathToAssumedName("foo.go")    = "foo"
ImportPathToAssumedName("foo.go/v1") = "foo"
ImportPathToAssumedName("foo-go")    = "foo"
ImportPathToAssumedName("foo-go/v1") = "foo"
ImportPathToAssumedName("go-foo")    = "foo"
ImportPathToAssumedName("go-foo/v1") = "foo"
ImportPathToAssumedName("go.foo")    = "go"
ImportPathToAssumedName("go.foo/v1") = "go"

And thanks for the canonical reference! I definitely need to add that to the style guide...

[kortschak]

Is there a reason for choosing the link digraph sets with their low false positive rate over the essentially zero false positive rate with the lighter weight on the page `...' pairing?

[kortschak]

I've raised accessibility here three times. No response has addressed that. Note that this is not a theoretical concern.

[dsnet]

@kortschak I'm not sure anyone disagrees that [...] delimiters have more visual noise than `...' (this quantitatively easy to prove based on the number of pixels needed to print it). As with most design decisions, there are different factors at play such as the familiarity of [...] (due to Markdown) over `...'. Pointing at the advantage of different approach is (in my opinion) a valid response, even if it doesn't directly address your concern of accessibility.

[kylelemons]

I think it's important to note that concerns over accessibility can cut many ways. My screen reader, for example, is much happier with [text] than it is with `text'. For the former, it simply ignores the brackets and reads straight through, but for the latter it narrates the backtick but not the apostrophe. This is likely to vary between software, but given that the quotes are two different kinds of punctuation and they are not opposites, I can imagine that it is going to be pretty common to hear this kind of asymmetry. (I would also be happy for it to narrate both delimiters, which it does in some cases like [...] when there is no word to read, but reading one and not the other is annoying, at least to me)

[kortschak]

Yeah. I'm not saying that I'm right about the solution or that the choice is simple (I come from a family of typographers), but I do feel that the concerns have been ignored.

[rsc]

@kortschak, re:

It avoids needless variation.

I'm trying to understand how requiring a novel pair of digraphs to delimit a link is not variation and how addressing readability on the page is needless. This latter point has not been addressed anywhere here.

The full comment was:

Although I believe it is a mistake to adopt Markdown wholesale, it seems clear that lots of people do know Markdown. It seems like leaning into that a little and using the syntaxes that those people already know is a win. It avoids needless variation.

In that context "It" = reusing Markdown notations. The needless variation is doing something gratuitously different from Markdown.

You have raised readability a few times but I can't fathom how `text' is somehow more readable for a link than [text]. If I saw the former I would think it meant quotation marks (which it does in LaTeX).

I think we have to agree to disagree about which is more readable.

[halturin]

Interfaces, please, let's make them readable in the doc.

[dsnet]

Are you talking about Go interfaces? If so, it seems that this is a concern for the rendering of how interfaces look, rather than the syntax of documentation. I think it's out of scope for this discussion.

[azr]

👋🏼 , Super nice to see this topic. In Packer, we often go generate docs from comments on structs (code, generated, live docs, generator) and then aggregate/import docs into pages. It's rather awesome because changes often happen only in one targeted place for us, and it makes documenting a bit easier and important. Moreover the code becomes the single source of truth.

For links, we use the markdown synthax: [word](url) and some pages can be quite big, containing a lot of links containing the same words/description; and having the description/link grouped together seemed simpler in terms of treatment. It can be slightly less easy on the eye when reading comments though.

[Gregory-Ledray]

Perhaps it makes more sense to have gofmt put the link definition immediately after its use, to make that easier

If that is what gofmt did - perhaps putting the links at the end of the current paragraph - then my argument really boils down to personal preference. In that case, I would very much appreciate it if gofmt automatically converted [text](https://link.io) to

[text]
[text]: https://link.io

Although I would understand if the language did not support that and I had to write a macro to do it instead.

[dsnet]

I prefer the inline syntax because when reading code and I encounter a link, I often want to click on it.

Once a standard GoDoc syntax is ratified, I suspect the tooling ecosystem will adapt. One can imagine that in a future version of VSCode with the Go plugin, you can ctrl+click on [RFC 7159] and it will know fetch the link reference declared later on.

[Gregory-Ledray]

True. But Go should not depend on an IDE to provide a good user experience, and IDEs are not guaranteed to implement such features by default. Case in point: despite Visual Studio being capable of rendering Xmldoc when I'm reading C# code, it does not always do so by default to make editing easier. Consequently I have to read ugly XML when I'm reading through C# code.

[ChrisHines]

Your point about finding the link is a good one. Perhaps it makes more sense to have gofmt put the link definition immediately after its use, to make that easier.

This is the approach I have been using when writing markdown manually because I think it has the best balance of readability of the prose and ease of finding the referenced links.

[arp242]

I prefer the inline syntax because when reading code and I encounter a link, I often want to click on it. With the reference link syntax, if the doc.go file is long or there is an example in the comment, you may have to scroll down quite a ways and hunt for the link among a forest of other links. I know I don't have to say this, but I want to reinforce it anyway: a link is only useful if you can click on it.

Personally I strongly prefer putting the links externally in anything of substance I write (as outlines in the proposal) as it makes the text "flow" a lot better; I usually try to make documentation read well, and I find having a (possible very long) link smack-down in the middle of it makes it a lot harder. I also find it harder to read: 9 times out of 10 I don't really care about the link, so it makes the text harder to read.

It's true that this makes actually clicking the link a bit harder, but that's a problem solvable with fairly simple tooling as mentioned. Conversely, making the text "flow better" is a problem that's harder to solve with tooling.

In the end, both methods have their own set of advantages and disadvantages – which is why Markdown supports both – but the more pragmatical solution for Go seems to me to be to use [text]: https://link as the downsides with that are much easier solvable.

Anecdotally, I've had a lot of trouble in the past remembering whether Markdown's link syntax is text, (text)[url], url, or (url)[text]

The way I remember this is that it's the same as you would write it without any formatting: put the link in parenthesis after the text:

For more information, see the documentation (https://example.com) which has detailed examples.

I still get it wrong on occasion though, especially if I'm in a hurry, and I've heard from many people over the years that they have trouble remembering this.

[peterbourgon]

Markdown is not the answer, but we can borrow good ideas

Markdown is the de facto standard solution for the problems you're trying to solve here. It certainly has flaws, but in practice they're solvable without major effort. And the benefits of using a more-or-less universally understood syntax hugely outweigh those costs.

Go shouldn't spend an innovation token on documentation syntax. It's a solved problem, and the marginal improvements offered by alternatives don't outweigh their concomitant toil.

—

I am fairly convinced Markdown is not the answer, for a few reasons.

To respond to each point...

First, there is no single definition of Markdown . . . Even so, let's define Markdown as CommonMark and continue.

That's just how these things go 🤷 and while it definitely introduces complexity to the parsers and renderers, it's not intractable.

(GitHub-flavored Markdown is what you should be targeting, for the record. It's by far the most widely used.)

Second, Markdown is not backwards compatible with existing doc comments . . .

It seems like all of these issues can be solved individually without too much fuss. You would end up forking GitHub-flavored Markdown and creating Go-flavored Markdown, or something like that. That's an ideal outcome.

Third, many features in Markdown are not terribly readable.

This is subjective, of course, so I'm not sure how much weight you can give to it in the decision-making. Furthermore, most of the enumerated feature requirements are core Markdown stuff, which means basically all Go programmers already know how to read and write them. (Thanks, GitHub!) That usability surely trumps aesthetics in the calculus.

Fourth, Markdown is surprisingly complex . . .

It's true, and in some sense a shame, that Markdown lets you express 1 thing in N ways. IMO, a Go-flavored Markdown could reasonably elide a lot of those "features" — I wouldn't miss HTML elements, or alternate demarcations for headings, for example. Maybe some people would say that it isn't actually Markdown without those features. Fair enough. But that's fine, there's no standards body involved here.

Just a point of feedback. Take it as you will.

[peterbourgon]

In-line links take up a ton of space in a paragraph

I agree that the readability of inline links is decreased in proportion to the length of the link. But the readability of footnote links is decreased in proportion to the distance between the link and the end of the paragraph. Neither is strictly more or less readable than the other; both have their place.

1 lists don’t render correctly as plain text.

That's true, and a good point. If the proposed syntax has "equivalent readability between source form and rendered form" as a design goal, then I agree it makes sense to elide this feature.

—

I also wanted to give a big +1 to not supporting images. Thanks for that.

[peterbourgon]

Markdown is the de facto standard [on GitHub] . . . Go aims to serve a much broader audience than just people who live on GitHub.

And GitHub is the de facto standard VCS. I'm not sure there are many Go programmers in the world who don't use GitHub on a regular basis.

edit: I have no idea why you'd thumbs-down this, @kylelemons. It's a statement of fact followed by a (clearly labeled) statement of opinion. If you want to object, please do so in a productive way.

[azr]

👋🏼 I've been thinking about this !

I also kinda like the renumbering of Markdown, because when I add a new item, I don't have to renumber/recommit the rest. Making it easier to find history in git and making PRs smaller. But the thing here, is that if I use a "simple/dumb" text editor; this magic numbering will be lost. And for editors that do renumber things, this makes it hard to edit docs. Maybe they suddenly need to have 'preview/raw' modes ?

Same thing for links: I also kinda like supporting more ways to display URLs. That's just a personal preference. But it's true that the readability of [description](url) can be a bit 'meh' on the eye. I have used it for a while in go docs, and they sometimes make lines of comments extra super long, breaking the style of godocs. They also very often break the flow of reading documentation. For example, when you're reading/understanding something, and suddenly sort of have to switch your eyes in 'skip mode'. It can be hard for me and I often lose the context and have to re-read again.

This would be solvable by a magic editor too, but I think that optimizing for a 'simple/dumb' text editor is just simpler and makes more sense.

[zephyrtronium]

@peterbourgon
On the Go issue tracker, we use thumbs-down reactions to mean "I disagree." It is not an attack, just a way to indicate one's sentiment regarding an idea without flooding threads with "+1" or "-1" comments.