proposal: x/tools/cmd/godoc: add support for sections #18342

Closed
dsnet opened this Issue Dec 16, 2016 · 62 comments

Comments

Projects
None yet
@dsnet
Member

dsnet commented Dec 16, 2016

Problem Statement

As packages grow organically over time, their API surface often expands to the point where it is hard to understand how to use the package when all functions and types are shown as a vast alphabetically ordered list. Aside from reading examples (and other 3rd party tutorials), it becomes nearly impossible figuring out how to use a package by the godoc alone.

For example, it is not clear from the godoc of the proto package that the core set of functions are Marshal, Unmarshal, Equal, Clone, and Merge apart from reading the example. The vast majority of other functions are for more advanced (and legacy) functionality that should be categorized separately from the core functionality.

I propose that godoc provides support for user-defined sections.

Proposed Solution

(Approach presented here is Solution A)

It is observed that Go code is often written in such a way that declarations that would be grouped together under a section are already located in proximity of each other within the source code. Assuming that most sections follow this pattern, I propose that a special Section: marker in a top-level comment be used to signify the start of a section. All const, var, func, and type declarations below that marker will be considered part of that section. A section will not cover methods since those are already implicitly grouped under the receiver type.

The coverage of the section group extends either until:

  • The end of the source file.
  • The next occurrence of a section marker.
  • The next occurrence of a special End section. marker, which must be a top-level comment with a blank line above and below it. As a matter of style, it will be discouraged practice to use the end marker unless necessary.

The section marker syntax allows for a required title and an optional description:

// Section: My wonderful title
//
// Let me tell you more details about my wonderful section
// in multiple lines of exciting text.

The title is a string that follows the Section: magic string and must be one line. The title may be any arbitrary string, but white space will be collapsed together. The description is optional and comprises of zero or more paragraphs (similar to package documentation) and must be preceded by a blank line.

For example, the proto package contains a set of helper functions that are only used when dealing with proto2 messages. In the source code, they are already co-located near each other with a comment that looks very similar to a section header. This portion could be modified to be more presentable in the public documentation:

// Section: Basic type helpers
//
// The proto2 format uses pointers for optional basic types (ints, uints, floats, and strings)
// It uses the the nil value to signal that they were are not present in the marshaled message.
// The proto package provides some helpers to aid in allocating values for these types.
//
// These helpers are not necessary for proto3 messages.

All of the helper functions (Float32, Float64, Int, Int32, Int64, String, Uint32, Uint64) will be grouped together under this section until the next top-level end marker:

// End section.

Sections without source locality

The observation that related declarations are often close together in the source does not always hold true. There are certain rare use cases of sections where the related declarations are not necessarily close by (or even in the same source file).

To solve this use case, we permit multiple section headers that have the same title. All declarations under sections with the same title will be considered to be under the same section. Only one of those section headers may have a description. This feature, combined with the end marker, can be used to indicate which declarations in separate files belong in the same section.

For example, the proto package has several types and functions that are only used by generated code and should never be called by user code. These types and functions live in different files. A section can be used to clearly identify these types and functions that are of no interest to the user by add a section in lib.go:

// Section: Internal implementation
//
// These types and functions are only used by generated .pb.go files and
// should never be called directly by user code.
// The compatibility agreement does not cover any of these declarations.

Elsewhere in message_set.go:

+ // Section: Internal implementation

  // RegisterMessageSetType is called from the generated code.
  func RegisterMessageSetType(m Message, fieldNum int32, name string) { ... }

+ // End section.

Elsewhere in properties.go:

+ // Section: Internal implementation

  // RegisterFile is called from generated code and maps from the
  // full file name of a .proto file to its compressed FileDescriptorProto.
  func RegisterFile(filename string, fileDescriptor []byte) { ... }

+ // End section.

How is godoc rendered?

All const, var, func, and type declarations that do not have a section will be shown at the top of the index as they are currently shown. Methods will always be grouped under their receiver type. The "core functionality" of a package should be a small set of declarations that do not have a parent section so that they appear first in the index.

For each section, the title will be printed and then the section's declarations will be shown. The list of sections are ordered alphabetically and the list of declarations are also listed alphabetically in the same way that they are shown today.

See the attached images for an example of what the proto package could look like.

Questions:

Q: Will there be support for arbitrary ordering of sections?

No. Since sections are ordered alphabetically, it is trivial for the ordering to be defined by prepending the section title with some prefix (e.g., the section number). It is unlikely that the number of sections will exceed 10, but if it does, a leading 0 can be prepended to the prior 9 sections. People are welcome to design their own ordering system (or carefully choose titles such that they appear in the desired order).

Q: Will there be support for sub-sections?

No. The number of indentations needed to visually show nested sections would make godoc too unreadable. Again, since the sections are ordered alphabetically, a prefix (e.g., 3.1.) can be used to create the concept of a "sub-section".

Q: What about other approaches to describe a section?

I looked other approaches of describing what was in a section and evaluated possible designs. They are described below.

Alternative Solution B (File-based locality)

(Idea by @josharian)

A variation on the source locality idea is to restrict it such that the section header can only be applied at most once at the top of the source file. The same section title may be present on multiple files. This approach avoids the need for a End section. marker since sections effectively span the entire file.

Alternative Solution C (Forward References)

An alternative to defining sections by source locality is to use forward references. In this approach, each section header contains a list of all the consts, vars, funcs, and types that belong in that section.

  // Section: Basic type helpers
  //
  // The proto2 format uses pointers for optional basic types (ints, uints, floats, and strings)
  // It uses the the nil value to signal that they were are not present in the marshaled message.
  // The proto package provides some helpers to aid in allocating values for these types.
  //
  // These helpers are not necessary for proto3 messages.
+ //
+ // References:
+ //  Float32
+ //  Float64
+ //  Int
+ //  Int32
+ //  Int64
+ //  String
+ //  Uint32
+ //  Uint64

Alternative Solution D (Reverse References)

An alternative to defining sections by source locality is to use reverse references. In this approach, the comment for each const, var, func, and type declaration contains a reference to the parent section it belongs in.

  // Int64 is a helper routine that allocates a new int64 value to store v and returns a pointer to it.
+ //
+ // Section: Basic type helpers
  func Int64(v int64) *int64 { ... }

Summary

A summary of the possible solutions:

Source Locality (A) File locality (B) Forward references (C) Reverse references (D)
Provides sections to godoc
Similar to how people naturally document Go code (1) (1)
New syntax does not negatively affect older doc-like tools (2)
Using sections requires no movement of actual code (3)
Renaming a section is trivial ⚠️ (4) ⚠️ (4)
Renaming or deleting a declaration is trivial (5)
Easy to associate declarations with a section (6) ⚠️ (6)
Avoids End section. marker (7)
  1. (A, B) In looking at examples of what could use sections, it was common to find a comment block that was very similar to a section header. Unfortunately, the beautiful prose written is not visible on the godoc.

  2. (D) The Section: Foo title string will be visible for doc-like tools with no understanding of sections.

  3. (B) Since sections are defined on a per-file basis, it may be necessary to move code around, which muddle the code history.

  4. (A, B) In the common case, there is only one section title and renaming is trivial. In rarer cases the same title may be used in multiple places and will need to be updated together.

  5. (C) It is easy for the forward references to get out of sync when declarations are renamed or deleted. This is trivially checked by the lint tool.

  6. (C) Any approach using references needs to provide that reference information somewhere and specifying these references can be cumbersome.

    For forward references, in the event that two section headers references the same declaration, which section should that declaration be in? Also, if there are many identifiers (e.g., lots of constants), do all identifiers have to be listed in the section header? The problem gets worse when different section headers refer to different identifiers within the same declaration block. Should the block be split apart to be in different sections?

    For backward references, a reference is still needed, but it avoids the problem with block declarations since the parent section is applied on the entire block.

  7. (A) The end section marker is not always needed since the end of a file or start of a section ends the current section's scope. Forgetting the end section marker can accidentally add more declarations to a section than intended. Getting this wrong is obvious when viewing the godoc.

I propose that godoc add support for user-defined sections by source locality (Solution A) as opposed to other approaches (using forward references or reverse references). I believe that Go documentation is often clean and palatable when the package is small, but quickly becomes overwhelming as the number of exported declarations in the package grows. User defined documentation sections solves this problem and encourages clean documentations at scale.

Related proposals:

It may be the case that when a package accrues many types over time that they should to be broken off into their own seperate package. However, support for movement of types does not change the fact that these types still exist in the original package for compatibility reasons. If support for #18130 happens, that is more reason that we should support sections so that documentation can clearly identify that a group of types have moved and provide details about the move.

As a possible alternative to #17056, a section could be used to contain all deprecated items under it. However, sections would not be able to group deprecated methods or fields. The existence of both features in godoc do not hamper the other, but could be used powerfully augment each other. You can imagine allowing godoc to hide an entire section with the Deprecated: magic string in the section description.

@dsnet dsnet added the Proposal label Dec 16, 2016

@dsnet dsnet added this to the Proposal milestone Dec 16, 2016

@dsnet

This comment has been minimized.

Show comment
Hide comment
@minux

This comment has been minimized.

Show comment
Hide comment
@minux

minux Dec 16, 2016

Member
Member

minux commented Dec 16, 2016

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Dec 16, 2016

Member

In retrospect, some of these packages should be split apart and doing so will definitely help the situation. The reality is, many packages are currently bloated and are unlikely to shed API surface anytime soon due to compatibility reasons.

  1. Even if we start gradually moving types using aliases (or whatever the solution ends up being), there is still a reference in the package. Sections help categorize and document the move.
  2. Even after some cruft is remove from a package, there are usually still a number of distinct sections that still belong together in a package, and it makes sense to document them as such.
  3. Better introductions help, but I believe are still insufficient. Some internal packages at Google have ridiculously long introductions, and they don't really help make the package docs be any more readable. IMHO, most of that introduction should be split into the relevant sections that they describe.
  4. Examples help, but I believe are still insufficient.
Member

dsnet commented Dec 16, 2016

In retrospect, some of these packages should be split apart and doing so will definitely help the situation. The reality is, many packages are currently bloated and are unlikely to shed API surface anytime soon due to compatibility reasons.

  1. Even if we start gradually moving types using aliases (or whatever the solution ends up being), there is still a reference in the package. Sections help categorize and document the move.
  2. Even after some cruft is remove from a package, there are usually still a number of distinct sections that still belong together in a package, and it makes sense to document them as such.
  3. Better introductions help, but I believe are still insufficient. Some internal packages at Google have ridiculously long introductions, and they don't really help make the package docs be any more readable. IMHO, most of that introduction should be split into the relevant sections that they describe.
  4. Examples help, but I believe are still insufficient.
@neild

This comment has been minimized.

Show comment
Hide comment
@neild

neild Dec 16, 2016

Contributor

+1 for some form of sections in godoc.

Splitting large packages can make sense, but it's very difficult to do after the fact and a maze of highly interdependent small packages is not necessarily any clearer than a single large one. Even small packages can benefit from sections, too; for example, database/sql is not particularly large, but grouping the Null* types into a single section might provide a small boost in readability.

Contributor

neild commented Dec 16, 2016

+1 for some form of sections in godoc.

Splitting large packages can make sense, but it's very difficult to do after the fact and a maze of highly interdependent small packages is not necessarily any clearer than a single large one. Even small packages can benefit from sections, too; for example, database/sql is not particularly large, but grouping the Null* types into a single section might provide a small boost in readability.

@cznic

This comment has been minimized.

Show comment
Hide comment
@cznic

cznic Dec 16, 2016

Contributor

Please keep godoc as simple as it is. It's a great value, though some would realize that only after it's gone.

Projects needing more structured documentation can easily link to any of the uncountable formats available. For example guru or go/types do that.

Contributor

cznic commented Dec 16, 2016

Please keep godoc as simple as it is. It's a great value, though some would realize that only after it's gone.

Projects needing more structured documentation can easily link to any of the uncountable formats available. For example guru or go/types do that.

@bcmills

This comment has been minimized.

Show comment
Hide comment
@bcmills

bcmills Dec 16, 2016

Member

It appears that the godoc on golang.org already supports sections to some extent. For example, there are now heading links to the Examples section of the testing package.

Another option might be to expand godoc to accept package comments from multiple source files. The package comment(s) in source files without any other declarations would become the top-level "Overview", and the package comment in each other source file would be the section header for the declarations in that file. Declarations in files without their own package comment would be grouped into the main section with the overview.

Member

bcmills commented Dec 16, 2016

It appears that the godoc on golang.org already supports sections to some extent. For example, there are now heading links to the Examples section of the testing package.

Another option might be to expand godoc to accept package comments from multiple source files. The package comment(s) in source files without any other declarations would become the top-level "Overview", and the package comment in each other source file would be the section header for the declarations in that file. Declarations in files without their own package comment would be grouped into the main section with the overview.

@bcmills

This comment has been minimized.

Show comment
Hide comment
@bcmills

bcmills Dec 16, 2016

Member

For the example of the proto package in particular: I maintain that it would be cleaner to split that package into two (or more) distinct packages: one with only the user-facing API, and another with the pseudo-internal hooks for generated proto packages.

(I could envision even-finer-grained splits, too: one for the proto2 pointer-to-primitive helper functions, perhaps one for the extension API, etc.)

Member

bcmills commented Dec 16, 2016

For the example of the proto package in particular: I maintain that it would be cleaner to split that package into two (or more) distinct packages: one with only the user-facing API, and another with the pseudo-internal hooks for generated proto packages.

(I could envision even-finer-grained splits, too: one for the proto2 pointer-to-primitive helper functions, perhaps one for the extension API, etc.)

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Dec 16, 2016

Member

It appears that the godoc on golang.org already supports sections to some extent.

I contend that the most useful feature of sections that I want to see is the ability to group a set of const, var, func, and type declarations together. Right now, all we have is the ability to pretty print some header title.

I maintain that it would be cleaner to split that package into two (or more) distinct packages.

Absolutely agree the pseudo-internal hooks should have been a different package. I believe the exported stuff to help with custom implementations of Message should also be split out. But those pointer-to-primitive functions are so pervasively used (over 200k usages!) that it's pretty annoying needing to import both a proto package and proto2 package. The point of sections is to group those functions together without creating a tiny package just for them.


To organize the pointer-to-primitive helpers, you could do something similar to binary.LittleEndian, but I find that method a hack and doesn't always work. I also find it dirty altering the implementation to fit the godoc, rather than the other way around.

Even then, binary.ByteOrder could still benefit from sections. The source code has ByteOrder, LittleEndian, BigEndian all located together in the source, but show up on opposite sides of the godoc.

Member

dsnet commented Dec 16, 2016

It appears that the godoc on golang.org already supports sections to some extent.

I contend that the most useful feature of sections that I want to see is the ability to group a set of const, var, func, and type declarations together. Right now, all we have is the ability to pretty print some header title.

I maintain that it would be cleaner to split that package into two (or more) distinct packages.

Absolutely agree the pseudo-internal hooks should have been a different package. I believe the exported stuff to help with custom implementations of Message should also be split out. But those pointer-to-primitive functions are so pervasively used (over 200k usages!) that it's pretty annoying needing to import both a proto package and proto2 package. The point of sections is to group those functions together without creating a tiny package just for them.


To organize the pointer-to-primitive helpers, you could do something similar to binary.LittleEndian, but I find that method a hack and doesn't always work. I also find it dirty altering the implementation to fit the godoc, rather than the other way around.

Even then, binary.ByteOrder could still benefit from sections. The source code has ByteOrder, LittleEndian, BigEndian all located together in the source, but show up on opposite sides of the godoc.

@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Dec 16, 2016

Contributor

I contend that the most useful feature of sections that I want to see is the ability to group a set of const, var, func, and type declarations together.

Thinking out loud...

What if that grouping was determined, perhaps optionally, by source file? Code is frequently already organized this way, with some exceptions, like autogenerated files. And if feels less fragile than relying on any ordering within a file. Perhaps if you still want section headers and more control, there could be zero or one section header per file, which applied to all code in that file, with identical section headers being coalesced.

Contributor

josharian commented Dec 16, 2016

I contend that the most useful feature of sections that I want to see is the ability to group a set of const, var, func, and type declarations together.

Thinking out loud...

What if that grouping was determined, perhaps optionally, by source file? Code is frequently already organized this way, with some exceptions, like autogenerated files. And if feels less fragile than relying on any ordering within a file. Perhaps if you still want section headers and more control, there could be zero or one section header per file, which applied to all code in that file, with identical section headers being coalesced.

@bcmills

This comment has been minimized.

Show comment
Hide comment
@bcmills

bcmills Dec 16, 2016

Member

But those pointer-to-primitive functions are so pervasively used (over 200k usages!) that it's probably pretty annoying needing to import both a proto package and proto2 package. The point of sections is to group those functions together without creating a tiny package just for them.

  1. The pointer-to-primitive functions aren't needed for proto3 messages, so only a subset of users of the proto package will actually use them.
  2. goimports makes having lots of tiny packages fairly benign, as long as there is some reasonable heuristic to figure out which item is in which package. (A counterexample: io vs. ioutil.)

The net and net/http packages in the standard library might be other good examples to consider. If you had support for sections, how would you want to split those out? (And does that split match the division of files today?)

Member

bcmills commented Dec 16, 2016

But those pointer-to-primitive functions are so pervasively used (over 200k usages!) that it's probably pretty annoying needing to import both a proto package and proto2 package. The point of sections is to group those functions together without creating a tiny package just for them.

  1. The pointer-to-primitive functions aren't needed for proto3 messages, so only a subset of users of the proto package will actually use them.
  2. goimports makes having lots of tiny packages fairly benign, as long as there is some reasonable heuristic to figure out which item is in which package. (A counterexample: io vs. ioutil.)

The net and net/http packages in the standard library might be other good examples to consider. If you had support for sections, how would you want to split those out? (And does that split match the division of files today?)

@neild

This comment has been minimized.

Show comment
Hide comment
@neild

neild Dec 16, 2016

Contributor

I disagree that lots of tiny packages are benign, even with goimports. The user still needs to remember which package contains which function; easy when you have one or two packages, less so when you have a dozen.

Obviously, stuffing unrelated functionality into a single package isn't right either. It's a balance. (For the proto package, I'd say the pointer-to-primitive functions should stay--they're small, cheap, and anyone who uses proto2 will want them--but obviously all the internal hooks should get pulled out. Something to work on once type aliases or the moral equivalent become available.)

Contributor

neild commented Dec 16, 2016

I disagree that lots of tiny packages are benign, even with goimports. The user still needs to remember which package contains which function; easy when you have one or two packages, less so when you have a dozen.

Obviously, stuffing unrelated functionality into a single package isn't right either. It's a balance. (For the proto package, I'd say the pointer-to-primitive functions should stay--they're small, cheap, and anyone who uses proto2 will want them--but obviously all the internal hooks should get pulled out. Something to work on once type aliases or the moral equivalent become available.)

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Dec 16, 2016

Member

What if that grouping was determined, perhaps optionally, by source file?

In the example I posted, when I was organizing the proto package, 5 of the sections were pretty much by file. The last section, which was about "pseudo-internal stuff" was spanning several files. Arguably, those functions and types could be all moved to a single internal.go file and be made one section. Thus, grouping by file-alone can lead to some unfortunate of code-churn.

That said, if we end up going with grouping by file alone, I'd be fine with that approach.

Member

dsnet commented Dec 16, 2016

What if that grouping was determined, perhaps optionally, by source file?

In the example I posted, when I was organizing the proto package, 5 of the sections were pretty much by file. The last section, which was about "pseudo-internal stuff" was spanning several files. Arguably, those functions and types could be all moved to a single internal.go file and be made one section. Thus, grouping by file-alone can lead to some unfortunate of code-churn.

That said, if we end up going with grouping by file alone, I'd be fine with that approach.

@neild

This comment has been minimized.

Show comment
Hide comment
@neild

neild Dec 16, 2016

Contributor

As a specific example from the net package, it contains eight error types:

DNSConfigError
DNSError
Error
InvalidAddrError
OpError
ParseError
UnknownNetworkError

Those are currently spread throughout the package index; collecting them into a single section would improve clarity.

These type names could be rewritten to use a common prefix (e.g., ErrParse) so that they sort together, but using less-natural names to work around godoc doesn't seem like a good situation.

Contributor

neild commented Dec 16, 2016

As a specific example from the net package, it contains eight error types:

DNSConfigError
DNSError
Error
InvalidAddrError
OpError
ParseError
UnknownNetworkError

Those are currently spread throughout the package index; collecting them into a single section would improve clarity.

These type names could be rewritten to use a common prefix (e.g., ErrParse) so that they sort together, but using less-natural names to work around godoc doesn't seem like a good situation.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Dec 16, 2016

Member

These type names could be rewritten to use a common prefix (e.g., ErrParse)

Not to mention that's not the convention. It's ErrFoo (or errFoo) for variables, and FooError (or fooError) for types.

Member

bradfitz commented Dec 16, 2016

These type names could be rewritten to use a common prefix (e.g., ErrParse)

Not to mention that's not the convention. It's ErrFoo (or errFoo) for variables, and FooError (or fooError) for types.

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Dec 16, 2016

Member

As another data point. The need for sections is not just for bloated packages, but even small packages can benefit from sections. I mentioned earlier the example of the binary package and the problem with ByteOrder and LittleEndian being on opposite sides of the page.

Here's what the GoDoc would look like with sections: binary doc

The sections as seen in godoc also matches exactly how they are laid out in source code.

Member

dsnet commented Dec 16, 2016

As another data point. The need for sections is not just for bloated packages, but even small packages can benefit from sections. I mentioned earlier the example of the binary package and the problem with ByteOrder and LittleEndian being on opposite sides of the page.

Here's what the GoDoc would look like with sections: binary doc

The sections as seen in godoc also matches exactly how they are laid out in source code.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 16, 2016

Contributor

I agree that godoc could do better. It would be great if we could do better automatically. It's easy to group errors, for example. Are there automatic groupings that we can explore? Alternately, what do other languages do?

One thing I do not want to see is CLs moving code around and breaking 'git blame' just so that things appear in a different order in godoc.

Contributor

rsc commented Dec 16, 2016

I agree that godoc could do better. It would be great if we could do better automatically. It's easy to group errors, for example. Are there automatic groupings that we can explore? Alternately, what do other languages do?

One thing I do not want to see is CLs moving code around and breaking 'git blame' just so that things appear in a different order in godoc.

@bcmills

This comment has been minimized.

Show comment
Hide comment
@bcmills

bcmills Dec 16, 2016

Member

Are there automatic groupings that we can explore?

Grouping functions with similar signatures would help with the binary package. Grouping types with similar method sets would help with the net package.

Unfortunately, I don't see any obvious heuristics that would help with the net/http package: the logical groupings there, I think, are "client", "server", and "request data", but determining which declaration goes with which cluster seems to require a fairly sophisticated traversal spanning struct fields, method receivers, and method sets.

Member

bcmills commented Dec 16, 2016

Are there automatic groupings that we can explore?

Grouping functions with similar signatures would help with the binary package. Grouping types with similar method sets would help with the net package.

Unfortunately, I don't see any obvious heuristics that would help with the net/http package: the logical groupings there, I think, are "client", "server", and "request data", but determining which declaration goes with which cluster seems to require a fairly sophisticated traversal spanning struct fields, method receivers, and method sets.

@neild

This comment has been minimized.

Show comment
Hide comment
@neild

neild Dec 16, 2016

Contributor

Python has pydoc, which I do not believe has sections. The Python standard library documentation is a separate set of reStructuredText files with internal sections; it does not use pydoc.

I don't believe Javadoc has sections. My experience in reading Javadocs does not lead me to consider it a system in need of emulation.

Contributor

neild commented Dec 16, 2016

Python has pydoc, which I do not believe has sections. The Python standard library documentation is a separate set of reStructuredText files with internal sections; it does not use pydoc.

I don't believe Javadoc has sections. My experience in reading Javadocs does not lead me to consider it a system in need of emulation.

@cznic

This comment has been minimized.

Show comment
Hide comment
@cznic

cznic Dec 16, 2016

Contributor

Someone spends more time in godoc, someone else might spend more time reading source code. Nonetheless, the better/richer/fancier the godoc output gets in terms of anything based on some magic/metadata/markup/... in comments, the less overaly source code comments will be usable without the godoc "interpreter".

It's a zero sum game.

Contributor

cznic commented Dec 16, 2016

Someone spends more time in godoc, someone else might spend more time reading source code. Nonetheless, the better/richer/fancier the godoc output gets in terms of anything based on some magic/metadata/markup/... in comments, the less overaly source code comments will be usable without the godoc "interpreter".

It's a zero sum game.

@bcmills

This comment has been minimized.

Show comment
Hide comment
@bcmills

bcmills Dec 16, 2016

Member

Alternately, what do other languages do?

The Haskell Haddock markup language has heading annotations.

Rust uses Markdown with some distinguished sections and some special parsing for examples.

I believe that ocamldoc keeps comments and declarations in source-file order, but only allows one source file per package.

C and C++ have dedicated third-party websites because their documentation story is so poor. 😦

Member

bcmills commented Dec 16, 2016

Alternately, what do other languages do?

The Haskell Haddock markup language has heading annotations.

Rust uses Markdown with some distinguished sections and some special parsing for examples.

I believe that ocamldoc keeps comments and declarations in source-file order, but only allows one source file per package.

C and C++ have dedicated third-party websites because their documentation story is so poor. 😦

@neild

This comment has been minimized.

Show comment
Hide comment
@neild

neild Dec 16, 2016

Contributor

I don't see any form of zero-sum game between godoc and source code comments. The proposed section comments read naturally in either location.

Contributor

neild commented Dec 16, 2016

I don't see any form of zero-sum game between godoc and source code comments. The proposed section comments read naturally in either location.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jan 9, 2017

Contributor

As a step toward solving the underlying problem, what if the mentions of exported names in the package doc comment automatically turned into links? The net package doc has, for example:

Name Resolution

The method for resolving domain names, whether indirectly with functions like Dial or directly with functions like LookupHost and LookupAddr, varies by operating system.

Imagine if instead each of those words - Dial, LookupHost, LookupAddr - were hyperlinked to the real thing. Then the package doc would work better and could contain the sections people are talking about, without any new syntax or non-prose.

It seems like maybe that should be the first step at least? It's unclear who should work on this. @neild, @dsnet, @bcmills, any takers?

-rsc for @golang/proposal-review

Contributor

rsc commented Jan 9, 2017

As a step toward solving the underlying problem, what if the mentions of exported names in the package doc comment automatically turned into links? The net package doc has, for example:

Name Resolution

The method for resolving domain names, whether indirectly with functions like Dial or directly with functions like LookupHost and LookupAddr, varies by operating system.

Imagine if instead each of those words - Dial, LookupHost, LookupAddr - were hyperlinked to the real thing. Then the package doc would work better and could contain the sections people are talking about, without any new syntax or non-prose.

It seems like maybe that should be the first step at least? It's unclear who should work on this. @neild, @dsnet, @bcmills, any takers?

-rsc for @golang/proposal-review

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Jan 9, 2017

Member

what if the mentions of exported names in the package doc comment automatically turned into links

Not just package doc comments, but all comments.

Member

bradfitz commented Jan 9, 2017

what if the mentions of exported names in the package doc comment automatically turned into links

Not just package doc comments, but all comments.

@neild

This comment has been minimized.

Show comment
Hide comment
@neild

neild Jan 9, 2017

Contributor

Automatic linking will result in some words inadvertently becoming links. I don't know if we care about that. I've certainly wished for internal links on a number of occasions.

Adding sections to the existing index seems simpler to me than creating an entire second table of contents in the package doc.

Contributor

neild commented Jan 9, 2017

Automatic linking will result in some words inadvertently becoming links. I don't know if we care about that. I've certainly wished for internal links on a number of occasions.

Adding sections to the existing index seems simpler to me than creating an entire second table of contents in the package doc.

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Jan 9, 2017

Member

I think that's an interesting and I would like to see that feature as well. ISTM that what you're describing is a form of "bottom-up" documentation. That is, you look at some specific component and the feature helps you answer the question: "how does this relate to all the other components in this package?"

However, sections are designed to be a form of "top-down" documentation. That is, they should help answer the questions: "I'm new to this package, how do I even grasp the high-level organization? what's important?"

Member

dsnet commented Jan 9, 2017

I think that's an interesting and I would like to see that feature as well. ISTM that what you're describing is a form of "bottom-up" documentation. That is, you look at some specific component and the feature helps you answer the question: "how does this relate to all the other components in this package?"

However, sections are designed to be a form of "top-down" documentation. That is, they should help answer the questions: "I'm new to this package, how do I even grasp the high-level organization? what's important?"

@griesemer

This comment has been minimized.

Show comment
Hide comment
@griesemer

griesemer Jan 9, 2017

Contributor

The "top-down" documentation should be given via a good package comment. If we can make the appropriate words in that comment link to the relevant pieces elsewhere we can probably go a long way. We do have all the exported identifiers, so it's a matter of being smart about connecting them with the prose. I think it can be done but it may take some experimentation.

Contributor

griesemer commented Jan 9, 2017

The "top-down" documentation should be given via a good package comment. If we can make the appropriate words in that comment link to the relevant pieces elsewhere we can probably go a long way. We do have all the exported identifiers, so it's a matter of being smart about connecting them with the prose. I think it can be done but it may take some experimentation.

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Jan 9, 2017

Member

@griesemer. A good package comment is not sufficient for complex packages. For example, even with a nice package comment, it only marginally helps the proto package to be readable.

Furthermore, if you spend time writing a good package comment about what is conceptually a "section", then shouldn't that block of text be co-located next to the related identifiers on godoc?

Member

dsnet commented Jan 9, 2017

@griesemer. A good package comment is not sufficient for complex packages. For example, even with a nice package comment, it only marginally helps the proto package to be readable.

Furthermore, if you spend time writing a good package comment about what is conceptually a "section", then shouldn't that block of text be co-located next to the related identifiers on godoc?

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Jan 9, 2017

Member

As a data point: https://godoc.org/github.com/google/gopacket. There was actually time invested into the package documentation for gopacket, but I still it very hard figuring out how the exported indentifiers should be used.

Member

dsnet commented Jan 9, 2017

As a data point: https://godoc.org/github.com/google/gopacket. There was actually time invested into the package documentation for gopacket, but I still it very hard figuring out how the exported indentifiers should be used.

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Jan 9, 2017

Member

@griesemer another easy win would be to automatically generate a table of contents for the package comment that links to its headers. That would make large package comments easier to navigate.

Member

jimmyfrasche commented Jan 9, 2017

@griesemer another easy win would be to automatically generate a table of contents for the package comment that links to its headers. That would make large package comments easier to navigate.

@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Jan 13, 2017

Contributor

what if the mentions of exported names in the package doc comment automatically turned into links

Not just package doc comments, but all comments.

One small piece of evidence that this might be useful: #18644.

Contributor

josharian commented Jan 13, 2017

what if the mentions of exported names in the package doc comment automatically turned into links

Not just package doc comments, but all comments.

One small piece of evidence that this might be useful: #18644.

@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Jan 13, 2017

Contributor

That's indeed pretty small.

Contributor

robpike commented Jan 13, 2017

That's indeed pretty small.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jan 23, 2017

Contributor

It still seems like automatic linking is worth doing regardless, and it may (or may not) help enough that we don't need to do anything else. It seems like we should do that before we consider more invasive changes.

Contributor

rsc commented Jan 23, 2017

It still seems like automatic linking is worth doing regardless, and it may (or may not) help enough that we don't need to do anything else. It seems like we should do that before we consider more invasive changes.

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Jan 24, 2017

Member

I can tackle this (i.e., automatic linking) in a few weeks unless someone else wants to do it. I'll assign it to myself when I start working on it.

Member

dsnet commented Jan 24, 2017

I can tackle this (i.e., automatic linking) in a few weeks unless someone else wants to do it. I'll assign it to myself when I start working on it.

@rsc rsc changed the title from proposal: godoc: add support for sections to proposal: cmd/godoc: add support for sections Jan 30, 2017

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Feb 2, 2017

Member

FWIW, @ianlancetaylor proposed the automatic linking in Jun 2013 in #5637 (still open).

Member

bradfitz commented Feb 2, 2017

FWIW, @ianlancetaylor proposed the automatic linking in Jun 2013 in #5637 (still open).

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Mar 6, 2017

Contributor

On hold for @dsnet's work.

Contributor

rsc commented Mar 6, 2017

On hold for @dsnet's work.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Sep 27, 2017

Change https://golang.org/cl/66311 mentions this issue: godoc: add support for hotlinks

Change https://golang.org/cl/66311 mentions this issue: godoc: add support for hotlinks

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Sep 27, 2017

Change https://golang.org/cl/66312 mentions this issue: godoc: add support for sections

Change https://golang.org/cl/66312 mentions this issue: godoc: add support for sections

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Sep 27, 2017

Member

I hacked up an implementation of "hotlinking" exported identifiers based on @rsc's comment and also an implementation of "sections" based on my initial proposal.

I have mailed out the prototypes as:

  • CL/66311: godoc: add support for hotlinks
  • CL/66312: godoc: add support for sections

I have also setup some test godoc servers running those CLs:

GoDoc hotlinks

  • Examples: net, net/http, cmp
  • My current approach does not handle struct fields and interface methods.
  • The algorithm is susceptible to false-positives (i.e., we hyperlink what looks like an exported identifier to the wrong thing).
  • The algorithm is susceptible to false-negative (i.e., we fail to hyperlink what is obviously a reference to some exported identifier).
  • For both cases, we can keep tuning the algorithm, but it is impossible to be accurate all of the time. False-positives are especially bad since it causes godoc to be lying about what something actually is.
  • We only perform hyperlinking for exported identifiers from within the same package. Hyperlinking external packages (e.g., io.ReadCloser) gets tricky.
  • The proto package (which was my original motivation for this proposal) did not benefit much from hotlinks as each exported identifier is largely independent from each other. As a general observation, hotlinking performs best when the components within a package are tightly coupled with each other (e.g., net/http).

GoDoc sections

  • Examples: encoding/binary, encoding/json, cmp, iolimit, proto
  • Even small packages like binary can benefit from sections since it groups ByteOrder with the specific implementations LittleEndian and BigEndian. It also groups all of the varint-related functions together since the names don't sort nicely.
  • Very large packages like proto actually becomes understandable by grouping related functionality into sections.
  • There is an open question of whether methods can be moved as a result of being in a different section from the parent type. My prototype does allow methods to be separated from their parent type (e.g., methods of iolimit.Limiter).
Member

dsnet commented Sep 27, 2017

I hacked up an implementation of "hotlinking" exported identifiers based on @rsc's comment and also an implementation of "sections" based on my initial proposal.

I have mailed out the prototypes as:

  • CL/66311: godoc: add support for hotlinks
  • CL/66312: godoc: add support for sections

I have also setup some test godoc servers running those CLs:

GoDoc hotlinks

  • Examples: net, net/http, cmp
  • My current approach does not handle struct fields and interface methods.
  • The algorithm is susceptible to false-positives (i.e., we hyperlink what looks like an exported identifier to the wrong thing).
  • The algorithm is susceptible to false-negative (i.e., we fail to hyperlink what is obviously a reference to some exported identifier).
  • For both cases, we can keep tuning the algorithm, but it is impossible to be accurate all of the time. False-positives are especially bad since it causes godoc to be lying about what something actually is.
  • We only perform hyperlinking for exported identifiers from within the same package. Hyperlinking external packages (e.g., io.ReadCloser) gets tricky.
  • The proto package (which was my original motivation for this proposal) did not benefit much from hotlinks as each exported identifier is largely independent from each other. As a general observation, hotlinking performs best when the components within a package are tightly coupled with each other (e.g., net/http).

GoDoc sections

  • Examples: encoding/binary, encoding/json, cmp, iolimit, proto
  • Even small packages like binary can benefit from sections since it groups ByteOrder with the specific implementations LittleEndian and BigEndian. It also groups all of the varint-related functions together since the names don't sort nicely.
  • Very large packages like proto actually becomes understandable by grouping related functionality into sections.
  • There is an open question of whether methods can be moved as a result of being in a different section from the parent type. My prototype does allow methods to be separated from their parent type (e.g., methods of iolimit.Limiter).

@dsnet dsnet removed the Proposal-Hold label Sep 27, 2017

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Sep 27, 2017

Member

@dsnet maybe the precision of hotlinks could be improved by having more strict rules

  • Only attempt to match at the start of a line or when preceded by white space (takes care of Set-Cookie)
  • Only match Name with a top-level variable, const, func, or type.
  • Only match fields/methods with an explicit Type.Name

The first rule would allow the false-positive with Equal to be suppressed by writing:

// [...] and no "Equal" method is defined [...]

Alternately, it could be written to say something along the lines of: "and does not satisfy the Equaler interface." (Which would require defining said interface purely for documentation purposes or writing the definition in a pre block in the comment itself)

That could mean having to rewrite documentation to make sure things link correctly and do not link incorrectly but no matter how fancy the heuristics are that's going to be a problem. At least simple heuristics are easy for people to reason about.

Member

jimmyfrasche commented Sep 27, 2017

@dsnet maybe the precision of hotlinks could be improved by having more strict rules

  • Only attempt to match at the start of a line or when preceded by white space (takes care of Set-Cookie)
  • Only match Name with a top-level variable, const, func, or type.
  • Only match fields/methods with an explicit Type.Name

The first rule would allow the false-positive with Equal to be suppressed by writing:

// [...] and no "Equal" method is defined [...]

Alternately, it could be written to say something along the lines of: "and does not satisfy the Equaler interface." (Which would require defining said interface purely for documentation purposes or writing the definition in a pre block in the comment itself)

That could mean having to rewrite documentation to make sure things link correctly and do not link incorrectly but no matter how fancy the heuristics are that's going to be a problem. At least simple heuristics are easy for people to reason about.

@neild

This comment has been minimized.

Show comment
Hide comment
@neild

neild Sep 27, 2017

Contributor

Subjective impressions: The hotlinks are really nice when they work. The sections are great, even in the case of small packages like encoding/binary. The proto package is hugely improved with sections; that's in large part because the proto package is a complete mess, but I think improving the documentation of packages which are complete messes is a worthy goal.

I think the history of wiki linking (from WikiWords to [[brackets]]) argues that the most practical approach to avoiding false positives and negatives is to explicitly tag links. It's easier to consistently tag things which should link that it is to remember the link-detection heuristics and tag things which should not link. (e.g., and no "Equal" method is defined.) This isn't necessarily an argument for explicit tagging, just an observation about the limitations of a heuristic approach.

Contributor

neild commented Sep 27, 2017

Subjective impressions: The hotlinks are really nice when they work. The sections are great, even in the case of small packages like encoding/binary. The proto package is hugely improved with sections; that's in large part because the proto package is a complete mess, but I think improving the documentation of packages which are complete messes is a worthy goal.

I think the history of wiki linking (from WikiWords to [[brackets]]) argues that the most practical approach to avoiding false positives and negatives is to explicitly tag links. It's easier to consistently tag things which should link that it is to remember the link-detection heuristics and tag things which should not link. (e.g., and no "Equal" method is defined.) This isn't necessarily an argument for explicit tagging, just an observation about the limitations of a heuristic approach.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Oct 2, 2017

Contributor

Thanks Joe. It looks like there are some minor adjustments to make, but the general "automatic" approach does look right. Let's mark this proposal as accepted and leave the fine details for code review. @griesemer will take a look at the CLs.

Contributor

rsc commented Oct 2, 2017

Thanks Joe. It looks like there are some minor adjustments to make, but the general "automatic" approach does look right. Let's mark this proposal as accepted and leave the fine details for code review. @griesemer will take a look at the CLs.

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Oct 2, 2017

Member

To be clear, what has been accepted? hotlinking, sections, or both?

Edit: Nevermind, you're talking about the "automatic" approach. Copy.

Member

dsnet commented Oct 2, 2017

To be clear, what has been accepted? hotlinking, sections, or both?

Edit: Nevermind, you're talking about the "automatic" approach. Copy.

@griesemer

This comment has been minimized.

Show comment
Hide comment
@griesemer

griesemer Oct 2, 2017

Contributor

@dsnet Correct. We didn't look into the sections idea in the meeting - but I see that it looks pretty nice and is not intrusive. Let's discuss this in a 2nd round.

Contributor

griesemer commented Oct 2, 2017

@dsnet Correct. We didn't look into the sections idea in the meeting - but I see that it looks pretty nice and is not intrusive. Let's discuss this in a 2nd round.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Oct 6, 2017

Change https://golang.org/cl/69030 mentions this issue: godoc: add table of contents to Overview section

Change https://golang.org/cl/69030 mentions this issue: godoc: add table of contents to Overview section

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Oct 6, 2017

Member

I made a CL to add a table of contents to the overview section based on the headers in the package document per my comment on #18342 (comment) I realize that didn't get an official go-ahead and that it would likely need to be updated to work with the autolinking CL (I would be happy to do so), but I wanted to see if there was any benefit to having such a ToC. I tried godoc out on some packages with larger comments and it was quite pleasant, imo.

Member

jimmyfrasche commented Oct 6, 2017

I made a CL to add a table of contents to the overview section based on the headers in the package document per my comment on #18342 (comment) I realize that didn't get an official go-ahead and that it would likely need to be updated to work with the autolinking CL (I would be happy to do so), but I wanted to see if there was any benefit to having such a ToC. I tried godoc out on some packages with larger comments and it was quite pleasant, imo.

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Oct 6, 2017

Member

@jimmyfrasche, thanks for prototype, do you have a test server running that you can point people to? Which packages best demonstrate the benefit of this feature?

Member

dsnet commented Oct 6, 2017

@jimmyfrasche, thanks for prototype, do you have a test server running that you can point people to? Which packages best demonstrate the benefit of this feature?

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Oct 6, 2017

Member

I do not, sorry. I'm just running the modified godoc locally.

The standard library does not have many packages with long package comments and those that do do not tend to have headers. The go/build package was the best example I could find—and I've often had to ctrl+F or page down, page down, page down to jump to the section I wanted before so it was personally satisfying to see the ToC there.

Outside the standard library, the nicest obvious improvement in a very commonly used package was github.com/gorilla/websocket which has 5 useful subsections whose quick links benefit the return reader.

The github.com/google/gopacket package you posted earlier in this thread has many headers in the package document so it looks like it would be useful there as well.

Member

jimmyfrasche commented Oct 6, 2017

I do not, sorry. I'm just running the modified godoc locally.

The standard library does not have many packages with long package comments and those that do do not tend to have headers. The go/build package was the best example I could find—and I've often had to ctrl+F or page down, page down, page down to jump to the section I wanted before so it was personally satisfying to see the ToC there.

Outside the standard library, the nicest obvious improvement in a very commonly used package was github.com/gorilla/websocket which has 5 useful subsections whose quick links benefit the return reader.

The github.com/google/gopacket package you posted earlier in this thread has many headers in the package document so it looks like it would be useful there as well.

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Oct 6, 2017

Member

I suppose I could at least post a screenshot, though. Sorry for not thinking of that earlier.
image

Member

jimmyfrasche commented Oct 6, 2017

I suppose I could at least post a screenshot, though. Sorry for not thinking of that earlier.
image

@neild

This comment has been minimized.

Show comment
Hide comment
@neild

neild Oct 6, 2017

Contributor

There's an existing table of contents; why not include the overview sections table in there?

Overview
    Go Path
    Build Constraints
    Binary-Only Packages
Index
Contributor

neild commented Oct 6, 2017

There's an existing table of contents; why not include the overview sections table in there?

Overview
    Go Path
    Build Constraints
    Binary-Only Packages
Index
@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Oct 6, 2017

Member

@neild That was my first thought as well, but this way I only had to render the doc to html once and I didn't have to worry about what happens if someone clicks the link but the overview is collapsed. I don't mind changing it but I figured why go to that trouble for a first iteration.

Member

jimmyfrasche commented Oct 6, 2017

@neild That was my first thought as well, but this way I only had to render the doc to html once and I didn't have to worry about what happens if someone clicks the link but the overview is collapsed. I don't mind changing it but I figured why go to that trouble for a first iteration.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Oct 24, 2017

Change https://golang.org/cl/72890 mentions this issue: godoc/internal/render: add render package for text formatting

Change https://golang.org/cl/72890 mentions this issue: godoc/internal/render: add render package for text formatting

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Oct 24, 2017

Member

CL 72890 is my first cut at implementation hotlinks. It's a vastly cleaned up version of my former CL.

The render_docs.go file uses the render package to statically generate HTML for the entire standard library. You can see the results here: https://static-hotlinks.digitalstatic.net/

Member

dsnet commented Oct 24, 2017

CL 72890 is my first cut at implementation hotlinks. It's a vastly cleaned up version of my former CL.

The render_docs.go file uses the render package to statically generate HTML for the entire standard library. You can see the results here: https://static-hotlinks.digitalstatic.net/

@ugorji

This comment has been minimized.

Show comment
Hide comment
@ugorji

ugorji Dec 25, 2017

Contributor

Any chance we can get this in for go 1.10? It would be really nice to get this in sooner than later. I reckon that this change can go in, as it doesn't affect running programs, but it makes a useful tool much better. Waiting for go 1.11 means waiting until August/Sept 2018 - a very long time away.

Contributor

ugorji commented Dec 25, 2017

Any chance we can get this in for go 1.10? It would be really nice to get this in sooner than later. I reckon that this change can go in, as it doesn't affect running programs, but it makes a useful tool much better. Waiting for go 1.11 means waiting until August/Sept 2018 - a very long time away.

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Dec 25, 2017

Member

The 1.10 release is in beta and only bug fixes are going in. Aside from that, the code still needs some polishing and to go through code review, both of which is going to take time and most certainly will not complete before 1.10 release.

Secondly, suppose the code for this feature gets submitted not long after 1.10 is release, nothing is stopping you from go getting the godoc tool yourself and using it ahead of the 1.11 release. Thus, saying that you have to wait for August 2018 is not a fair assessment either.

Member

dsnet commented Dec 25, 2017

The 1.10 release is in beta and only bug fixes are going in. Aside from that, the code still needs some polishing and to go through code review, both of which is going to take time and most certainly will not complete before 1.10 release.

Secondly, suppose the code for this feature gets submitted not long after 1.10 is release, nothing is stopping you from go getting the godoc tool yourself and using it ahead of the 1.11 release. Thus, saying that you have to wait for August 2018 is not a fair assessment either.

@neild

This comment has been minimized.

Show comment
Hide comment
@neild

neild Apr 13, 2018

Contributor

Experience report: We're considering defining some exported types as interfaces rather than as concrete types, entirely because interfaces allow related methods to be grouped together in godoc and concrete types don't.

e.g.,

type ContainerType interface {
  // NumCats is the number of cats in the container.
  NumCats() int
  // Cat returns the i'th cat in the container.
  Cat(i int) Cat

  // NumPigs is the number of pigs in the container.
  NumPigs() int
  // Pig returns the i'th pig in the container.
  Pig(i int) Pig
}

This is a different than the sections proposal (which would allow grouping types, but not methods within a type), but seems related.

Contributor

neild commented Apr 13, 2018

Experience report: We're considering defining some exported types as interfaces rather than as concrete types, entirely because interfaces allow related methods to be grouped together in godoc and concrete types don't.

e.g.,

type ContainerType interface {
  // NumCats is the number of cats in the container.
  NumCats() int
  // Cat returns the i'th cat in the container.
  Cat(i int) Cat

  // NumPigs is the number of pigs in the container.
  NumPigs() int
  // Pig returns the i'th pig in the container.
  Pig(i int) Pig
}

This is a different than the sections proposal (which would allow grouping types, but not methods within a type), but seems related.

@agnivade agnivade changed the title from proposal: cmd/godoc: add support for sections to proposal: x/tools/cmd/godoc: add support for sections Apr 14, 2018

@neild

This comment has been minimized.

Show comment
Hide comment
@neild

neild May 17, 2018

Contributor

Removed the "accepted" label, because what was accepted was something other than sections (and @griesemer's last comment about sections is that there should be a second review round about sections).

Contributor

neild commented May 17, 2018

Removed the "accepted" label, because what was accepted was something other than sections (and @griesemer's last comment about sections is that there should be a second review round about sections).

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc May 21, 2018

Contributor

Per #18342 (comment) and following, I thought the plan was:

  1. Add automatic linking of function names etc.
  2. Encourage people to write good overview doc comments using those names.
  3. Step back and see what still needs to be done.

How far along are we on that plan? I'm confused about what is being re-suggested in this re-proposal. Also we just approved #25449 which will help with sections in the overview comment, which fits well with (2). I'm still skeptical of per-symbol section annotations.

Contributor

rsc commented May 21, 2018

Per #18342 (comment) and following, I thought the plan was:

  1. Add automatic linking of function names etc.
  2. Encourage people to write good overview doc comments using those names.
  3. Step back and see what still needs to be done.

How far along are we on that plan? I'm confused about what is being re-suggested in this re-proposal. Also we just approved #25449 which will help with sections in the overview comment, which fits well with (2). I'm still skeptical of per-symbol section annotations.

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet May 21, 2018

Member

How far along are we on that plan?

It's not far off from integration with x/tools/godoc (but won't make it for the Go1.11 cycle). When I looked at what it would take to integrate with other godoc tools (e.g., godoc.org), I came to the sad realization that every tool does front-end rendering in their own way, which makes a change like this lots of work.

Arguably, the place that benefits most is in the godoc.org tool, not the x/tools one.

I wonder whether we should strive to unify the godoc tools first. I'm not saying they should be the same tool, but they are a lot of opportunities for shared code that would make adding godoc features easier.

Member

dsnet commented May 21, 2018

How far along are we on that plan?

It's not far off from integration with x/tools/godoc (but won't make it for the Go1.11 cycle). When I looked at what it would take to integrate with other godoc tools (e.g., godoc.org), I came to the sad realization that every tool does front-end rendering in their own way, which makes a change like this lots of work.

Arguably, the place that benefits most is in the godoc.org tool, not the x/tools one.

I wonder whether we should strive to unify the godoc tools first. I'm not saying they should be the same tool, but they are a lot of opportunities for shared code that would make adding godoc features easier.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jun 4, 2018

Contributor

It sounds like the plan is to do both #25449 and #25444. At that point more proposals can be made about what is left. But until then it seems like we should close this proposal as having spawned two concrete next steps. I agree about sharing code between godoc, gddo, etc but that probably doesn't need a proposal (it doesn't affect how programmers write doc comments, for example).

Contributor

rsc commented Jun 4, 2018

It sounds like the plan is to do both #25449 and #25444. At that point more proposals can be made about what is left. But until then it seems like we should close this proposal as having spawned two concrete next steps. I agree about sharing code between godoc, gddo, etc but that probably doesn't need a proposal (it doesn't affect how programmers write doc comments, for example).

@rsc rsc closed this Jun 4, 2018

@spekary

This comment has been minimized.

Show comment
Hide comment
@spekary

spekary Jun 13, 2018

Coming late to this conversation, but does closing this proposal imply that sections will NOT be done? The spawned proposals are good, but they do not easily satisfy the origial premise of being able to guide readers of documentation through an API by pointing them first to the general purpose or most likely used functions, and then to more specialized fuctions. Yes, one could do that through package comments and hotlinking, but sections are easier to maintain.

Another reason for sections is to guide various consumers of an API in a large project. Go does not have protection semantecs like other languages, so large groups have to communicate and use conventions to work well together. It would be nice to reinforce these conventions through the documentation.

Also, I am concerned about this comment from the original proposal:

A section will not cover methods since those are already implicitly grouped under the receiver type.

The purpose of sections applies to methods as well. The grouping provided by receivers is for programmatic purposes, not for documentation purposes. The need to group parts of an API for the consumers of a receiver is the same.

Currently, in order to do this, one could use a naming convention that adds a prefix to the names of functions and methods in order to group them, but I hate having to change an API for something that could be handled with a simple documentation mechanism.

I am not sure if tagging was ever considered as an alternate solution to sections to solve this. It looks like the Deprected: keyword is getting some traction, but really in a limited scope. Being able to put a tag in a comment, like Deprecated: or the BUG(): mechanism, and using those to define sections could alternately be used, and also satisfy the Deprecated: problem (but a sort order would need to be added too.)

Just some thoughts.

spekary commented Jun 13, 2018

Coming late to this conversation, but does closing this proposal imply that sections will NOT be done? The spawned proposals are good, but they do not easily satisfy the origial premise of being able to guide readers of documentation through an API by pointing them first to the general purpose or most likely used functions, and then to more specialized fuctions. Yes, one could do that through package comments and hotlinking, but sections are easier to maintain.

Another reason for sections is to guide various consumers of an API in a large project. Go does not have protection semantecs like other languages, so large groups have to communicate and use conventions to work well together. It would be nice to reinforce these conventions through the documentation.

Also, I am concerned about this comment from the original proposal:

A section will not cover methods since those are already implicitly grouped under the receiver type.

The purpose of sections applies to methods as well. The grouping provided by receivers is for programmatic purposes, not for documentation purposes. The need to group parts of an API for the consumers of a receiver is the same.

Currently, in order to do this, one could use a naming convention that adds a prefix to the names of functions and methods in order to group them, but I hate having to change an API for something that could be handled with a simple documentation mechanism.

I am not sure if tagging was ever considered as an alternate solution to sections to solve this. It looks like the Deprected: keyword is getting some traction, but really in a limited scope. Being able to put a tag in a comment, like Deprecated: or the BUG(): mechanism, and using those to define sections could alternately be used, and also satisfy the Deprecated: problem (but a sort order would need to be added too.)

Just some thoughts.

@akavel

This comment has been minimized.

Show comment
Hide comment
@akavel

akavel Jun 14, 2018

Contributor

@spekary Did you review the above discussion? I believe various possible approaches, as well as their pros and cons, are discussed quite exhaustively, starting already from the Summary section in the initial post. There doesn't seem to be a clear winner, all approaches appear have some cons. On first reading, I'm afraid I can't see what your post adds to the discussion, that was not already discussed.

As far as I understand, the current idea is to start by implementing the 2 linked proposals, which have the advantage of also fixing/improving some other issues. Afterwards, with some more experience w.r.t. how they work, the situation could potentially be reevaluated. That said, as far as I understand, those proposals will enable de facto realizing the "Alternative Solution C (Forward References)" approach, by thoughtfully composing the "package level" docs for a package in appropriate Sections (a.k.a. Headings). Personally, I suppose this will then become the advised approach for solving this issue.

To be a bit more explicit, "Headings" are already supported in package docs, and #25449 aims to show them in ToC. Per #25444, any function/type/... names used in package docs sections would be automatically hotlinked to their definitions. This sounds like it should enable your stated goal, i.e. for the documentation writer to:

[...] easily [be] able to guide readers of documentation through an API by pointing them first to the general purpose or most likely used functions, and then to more specialized functions.

Contributor

akavel commented Jun 14, 2018

@spekary Did you review the above discussion? I believe various possible approaches, as well as their pros and cons, are discussed quite exhaustively, starting already from the Summary section in the initial post. There doesn't seem to be a clear winner, all approaches appear have some cons. On first reading, I'm afraid I can't see what your post adds to the discussion, that was not already discussed.

As far as I understand, the current idea is to start by implementing the 2 linked proposals, which have the advantage of also fixing/improving some other issues. Afterwards, with some more experience w.r.t. how they work, the situation could potentially be reevaluated. That said, as far as I understand, those proposals will enable de facto realizing the "Alternative Solution C (Forward References)" approach, by thoughtfully composing the "package level" docs for a package in appropriate Sections (a.k.a. Headings). Personally, I suppose this will then become the advised approach for solving this issue.

To be a bit more explicit, "Headings" are already supported in package docs, and #25449 aims to show them in ToC. Per #25444, any function/type/... names used in package docs sections would be automatically hotlinked to their definitions. This sounds like it should enable your stated goal, i.e. for the documentation writer to:

[...] easily [be] able to guide readers of documentation through an API by pointing them first to the general purpose or most likely used functions, and then to more specialized functions.

@spekary

This comment has been minimized.

Show comment
Hide comment
@spekary

spekary Jun 14, 2018

I did read it. I saw tagging mentioned but not seriously considered, and I think that should be evaluated.

Another goal and benefit of godoc is the minimal amount of effort developers need to put in to get reasonably good documentation. By using a locality approach or tagging approach, the developer need do nothing else but make sure a function or method is defined in the right place in the file or given a correct tag. If he/she gets that wrong, running godoc will make it clear whether the new functions appear in the wrong place. Developers tend to work serially, writing code first with minimal doc, and then adding more extensive doc later.

Relying on the two linked proposals means developers must maintain documentation in two places. If a developer in a fit of inspiration writes a bunch of new functions, and then later goes back to try to document them, running godoc will not help identify which functions are missing from the package doc, since they will not appear at all. The developer will have to visually compare the package doc with the generated TOC. It will be easy for a developer to miss some, especially in a large project. As the project grows, the header's description will diverge from what is actually in the TOC.

Well, that is just a prediction based on my observed human behavior. As you say, we can see how it goes and revisit later. I am hoping that closing this proposal doesn't end it.

spekary commented Jun 14, 2018

I did read it. I saw tagging mentioned but not seriously considered, and I think that should be evaluated.

Another goal and benefit of godoc is the minimal amount of effort developers need to put in to get reasonably good documentation. By using a locality approach or tagging approach, the developer need do nothing else but make sure a function or method is defined in the right place in the file or given a correct tag. If he/she gets that wrong, running godoc will make it clear whether the new functions appear in the wrong place. Developers tend to work serially, writing code first with minimal doc, and then adding more extensive doc later.

Relying on the two linked proposals means developers must maintain documentation in two places. If a developer in a fit of inspiration writes a bunch of new functions, and then later goes back to try to document them, running godoc will not help identify which functions are missing from the package doc, since they will not appear at all. The developer will have to visually compare the package doc with the generated TOC. It will be easy for a developer to miss some, especially in a large project. As the project grows, the header's description will diverge from what is actually in the TOC.

Well, that is just a prediction based on my observed human behavior. As you say, we can see how it goes and revisit later. I am hoping that closing this proposal doesn't end it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment