Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: restructure module documentation #33637

Open
jayconrod opened this issue Aug 13, 2019 · 68 comments
Open

doc: restructure module documentation #33637

jayconrod opened this issue Aug 13, 2019 · 68 comments

Comments

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Aug 13, 2019

Summary

Go module documentation is spread out over several sources. The canonical source is go help, but it's difficult to read in a terminal, and the HTML equivalent (golang.org/cmd/go) is hard to discover and lacks structure and examples. Many developers seem to have difficulty finding solutions to common problems and discovering reference material.

This issue proposes restructuring module documentation into three main areas.

  • A specification should be added as one or more documents linked from the References section of golang.org/doc. The specification should be written in the main Go repository, and updates should be linked to the Go release cycle.
  • A quick reference should remain in the go command with a brief description of each command and each flag. go help modules and other pages should be shortened, pointing to the specification where appropriate. The quick reference should be mirrored at golang.org/cmd/doc, and updates should be linked to the Go release cycle.
  • A series of guide articles should be published to help developers learn how to start using modules and how to solve specific problems with modules. Guide articles should be linked from the Learning Go section of golang.org/doc. Guide articles may be published outside the Go release cycle and should be kept up to date with current features and best practices.

Specification

A complete reference for everything related to modules should be added as one or more structured documents linked from golang.org/doc under the References section. This is analogous to the Language specification or The Go memory model.

The specification should be written as HTML documents within the main Go repository. Updates on the website will be tied to the Go release cycle. This is desirable since things like the go.mod format and the proxy protocol are tied to the current release. The latest version of the documentation will always be visible from tip.golang.org.

The specification should include the following sections. Note that this is simply a rough list of headings to establish scope, not an outline, which will come later.

  • Introduction
    • Motivation, problems solved (based on https://research.swtch.com/vgo-intro) - very brief description of what modules are and why they exist.
    • Preliminary module support (from go help modules) - how to enable or disable module aware mode.
    • Defining a module (from go help modules) - overview of using modules, with an example go.mod file and mention of go mod init.
    • The main module and the build list (from go help modules) - defining terminology.
    • Pseudo-versions (from go help modules) - definition and format.
    • Module queries (from go help modules) - list of @ suffixes accepted by go get, go mod download, etc.
    • Upgrading and downgrading with go get - more detailed description of go get flags and use of module queries. go help get will link here.
    • Module compatibility and semantic versioning - explanation of semantic versioning, need for major version suffixes, +incompatible version suffix.
  • Mechanics of downloading, verifying, and storing modules
    • Module code layout (from go help modules) - how modules are extracted from vcs repo. Currently a pointer to https://research.swtch.com/vgo-module
    • Module download and verification (from go help modules) - description of how modules are downloaded and verified against go.sum.
    • Module replace semantics
    • Modules and vendoring (from go help modules) - go mod vendor and -mod=vendor. We will need to define vendoring without relying on prior knowledge or GOPATH documentation.
    • Minimal version selection (based on https://research.swtch.com/vgo-mvs) - how versions are chosen
    • Module authentication - go.sum and checksum database.
    • Private modules (from go help module-private) - hosting and accessing private modules. Currently, this is just about configuring the go command to avoid leaking information about private modules to the proxy and sumdb. We should include GOAUTH (when implemented; see #26232) and other mechanisms (.netrc, git config) for connecting to and authenticating with private servers.
  • Formats and protocols
    • go.mod format (from go help go.mod)
    • go.sum format
    • GOPROXY protocol (from go help goproxy) - Should describe fallback behavior.
    • GOSUMDB protocol.

Quick reference

A quick reference for the Go command should be preserved in go help and golang.org/cmd/go/, which is generated from the same text. The quick reference should briefly describe each command and each flag. Quick reference pages may link to sections in the specification or occasionally guide articles, but they should not link to the wiki or external sources that may change or disappear.

As with the specification, updates to the quick reference on the website will be tied to the Go release cycle.

The following changes should be made to go help:

  • environment - should provide a brief description of GOPROXY family variables, referring to 'go help goproxy' for more details.
  • get (module-get) - Should be slightly more concise. Flags should be indented and descriptions should be shortened (like go help build) to make them easier to find quickly. Reference documentation should be linked for details.
  • mod download - No change needed.
  • mod edit - Flags should be indented (like go help build). Content is already concise.
  • mod graph - No change needed.
  • mod init - Should define the module path and explain what it should look like. Consider linking to an introductory guide article.
  • mod tidy - No change needed.
  • mod vendor - Should link to "Modules and vendoring" in reference documentation.
  • mod verify - Should link to "Module authentication".
  • mod why - Flags should be indented. Content is already concise.
  • goproxy - Should provide a brief description of all proxy environment variables. Should link to reference documentation for proxy protocol, sumdb protocol, Module authentication, etc.
  • modules - Should provide a concise introduction (~200-500 words) and link to reference documentation.
  • modules-auth - Should link to "Module authentication" in reference documentation.
  • modules-private - Should link to "Private modules" in reference documentation.

Guide articles

Guide articles should teach developers how to accomplish specific tasks. Each article should focus on a specific topic or small group of related topics. Articles may be more verbose than reference documentation and should include more realistic examples.
Unlike reference documentation, guide articles may be published outside of the Go release cycle. Unlike blog articles, guide articles should be updated after they are published to include new features and best practices. Blog articles may be converted to guide articles after they are published.

Guide articles should be linked from the Learning Go section of golang.org/doc. See the Hosting section below for possible hosting locations.

The following is a non-exhaustive list of articles to be written.

  • Using Go Modules - An introduction to modules. Should be copied mostly as-is with dates and versions removed or updated.
  • How to Write Go Code (#28215) - Currently is an introduction to GOPATH, aimed at new Go developers. Should be rewritten as an introduction to modules. This has some overlap with Using Go Modules, but the audience is new Go developers, rather than experienced Go developers who are new to modules.
  • Publishing Go modules (blog post in progress) - a guide for releasing a new version of a module. Semantic versions, semantic import versioning, tag format for modules not at repository root. How to push new major version.
  • Migrating to Go modules - how to migrate from dep and other systems. Discusses migration with go mod init and limitations. Considerations for v2+ modules. Compatibility with GOPATH (minimal module compatibility).
  • v2 and beyond (blog post planned) - Migration and publishing considerations for v2+ modules.
  • Tools for understanding modules - how to answer common questions using go list [-m] [-json], go mod why [-m], go mod graph.
  • Debugging Go Modules - a guide for how to debug module failures. See https://github.com/golang/go/wiki/Modules#faqs--possible-problems for some existing topics.
  • Multi module repositories - a guide for managing multi-module repositories. (see https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories)
  • Maintaining module requirements - a guide for managing dependencies with modules, loosely based on go help modules. Explains how to accomplish common tasks with go get and go mod tidy in more detail than the above documents.
  • How to upgrade to a new major version of a dependency - Should cover any details left out of Migrating to Go modules. Should keep up with tools that assist with this.
  • Local modifications to dependencies - walks through creating a short-lived fork of a dependency, fixing an issue, submitting a PR, upgrading and unforking once the PR is merged. May be combined with Maintaining module requirements, depending on level of detail.
  • Managing private modules - authentication with private repos, private proxies, -insecure flag.

Hosting

The golang.org website is served using golang.org/x/website/cmd/golangorg. Currently, this binary has some static content built into the content/static package. This includes some HTML, JS, CSS, and templates. Most content is extracted from the Go repository itself. For example, golang.org/doc is doc/docs.html.

The blog is served on a separate site, blog.golang.org using the binary golang.org/x/blog. Content is written in the content directory in golang.org/x/tools/present format.

At the moment, there is no place for non-blog content to be published outside the Go release cycle. We are considering a few locations:

  • The doc/articles directory of the main Go repository. x/website would need to serve pages in this directory at master instead of from the latest release.
  • A directory within golang.org/x/website itself. Currently, this repository has not content (just templates, CSS, and JS). This would reduce the need to serve pages from multiple versions of the Go repository, but we'd at least need to serve doc/docs.html from tip in order to update links outside the release cycle.
  • A separate documentation site, to be determined.

Whereever we end up hosting articles, authors should be able to write in Markdown or a similar convenient format, rather than raw HTML.

@jayconrod jayconrod added this to the Go1.14 milestone Aug 13, 2019
@gopherbot gopherbot added the Proposal label Aug 13, 2019
@jayconrod
Copy link
Contributor Author

@jayconrod jayconrod commented Aug 13, 2019

@mvdan
Copy link
Member

@mvdan mvdan commented Aug 22, 2019

@myitcv noticed that go help modules does mention GOPROXY's special off value, but go help goproxy doesn't. I think the latter help page is where one would look first.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 3, 2019

Is there any reason for this issue to go through the proposal process? It seems like something to be decided by the cmd/go maintainers.

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 3, 2019

@ianlancetaylor, mostly I think we'd like feedback on where to put new documentation that isn't tied to the release cycle: in doc/articles in the main go repo, in x/website, or some new documentation repository (such as x/doc)?

I think our preference would be a new repository, so that the main go repo can remain synchronized to the release cycle and x/website can remain focused on mechanism rather than content.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 3, 2019

OK.

The latter point is related to #29206.

@jayconrod jayconrod changed the title proposal: doc: restructure module documentation doc: restructure module documentation Sep 12, 2019
@jayconrod jayconrod removed the Proposal label Sep 12, 2019
@jayconrod
Copy link
Contributor Author

@jayconrod jayconrod commented Sep 12, 2019

Removed Proposal label. We'll want to talk about modifying x/website and maybe creating a new repo (x/doc?) soon. Perhaps that doesn't need to go through the proposal committee though.

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 26, 2019

Change https://golang.org/cl/197640 mentions this issue: x/website: link module blog posts from golang.org/doc

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 27, 2019

Change https://golang.org/cl/197638 mentions this issue: cmd/golangorg, content: serve /doc from content/static/doc before GOROOT/doc

gopherbot pushed a commit to golang/website that referenced this issue Sep 30, 2019
…OOT/doc

In the "/doc" directory, files are now served from the
content/static/doc, then from GOROOT/doc (where GOROOT is either a zip
file or the GOROOT directory).

This allows documentation to be moved to golang.org outside of the
normal Go release cycle. For example, this CL moves (but does not
modify) doc/docs.html here. Documentation that is tied to the Go
release cycle (for example, doc/spec.html) will not be moved.

Updates golang/go#33637
Updates golang/go#29206

Change-Id: I5f1c577e0ffddc3418653bdd9b0cc36cf79e3fe1
Reviewed-on: https://go-review.googlesource.com/c/website/+/197638
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Sep 30, 2019
Updates golang/go#33637

Change-Id: I37c81d2358c9b2c7ae11939b3884acea2612ff7d
Reviewed-on: https://go-review.googlesource.com/c/website/+/197640
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 4, 2019

Change https://golang.org/cl/199058 mentions this issue: content/static/doc: add code.html

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 4, 2019

Change https://golang.org/cl/199057 mentions this issue: doc: remove code.html

gopherbot pushed a commit to golang/website that referenced this issue Oct 4, 2019
See deletion CL here: https://golang.org/cl/199057

Updates golang/go#33637
Updates golang/go#29206

Change-Id: I992a6f3222b6327927578111ad71bdf3653adf78
Reviewed-on: https://go-review.googlesource.com/c/website/+/199058
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit that referenced this issue Oct 4, 2019
This is being moved to x/website in https://golang.org/cl/199058.

Updates #33637
Updates #29206

Change-Id: I5c58b784fcdd212d7a003cd0f4085059f33a47c7
Reviewed-on: https://go-review.googlesource.com/c/go/+/199057
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 4, 2019

Change https://golang.org/cl/199117 mentions this issue: doc: remove docs.html

gopherbot pushed a commit that referenced this issue Oct 4, 2019
This page has moved to x/website in https://golang.org/cl/197638.

Updates #33637
Updates #29206

Change-Id: I4f5f7822a2bf540a3911470548d38ccc7a66b74c
Reviewed-on: https://go-review.googlesource.com/c/go/+/199117
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Jean de Klerk <deklerk@google.com>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@rsc rsc removed this from the Go1.14 milestone Oct 9, 2019
gopherbot pushed a commit to golang/website that referenced this issue Jun 29, 2020
For golang/go#33637

Change-Id: I60d0b0c2c83c35dbb9ebbc200b781b2196320f87
Reviewed-on: https://go-review.googlesource.com/c/website/+/238277
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 1, 2020

Change https://golang.org/cl/240686 mentions this issue: content/static/doc: add module documentation on private modules

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 1, 2020

Change https://golang.org/cl/240688 mentions this issue: content/static/doc: add module documentation on go.sum files

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 13, 2020

Change https://golang.org/cl/242318 mentions this issue: content/static/doc: add module documentation on module cache

gopherbot pushed a commit to golang/website that referenced this issue Jul 13, 2020
For golang/go#33637

Change-Id: I6b46d319e8f5dcf42db66c54d08ba7136375b43a
Reviewed-on: https://go-review.googlesource.com/c/website/+/234489
Reviewed-by: Michael Matloob <matloob@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Jul 13, 2020
For golang/go#33637

Change-Id: I5afa5373c00e0b0252ade069e84c67478379896f
Reviewed-on: https://go-review.googlesource.com/c/website/+/240686
Reviewed-by: Michael Matloob <matloob@golang.org>
@jayconrod
Copy link
Contributor Author

@jayconrod jayconrod commented Jul 20, 2020

From a comment in CL 243077, we need an article on how and when to use replace directives. It should explain what the implications are for other modules that depend on a module with replace, how to test with and without replace, and how to avoid replace in released versions.

@carnott-snap
Copy link

@carnott-snap carnott-snap commented Jul 20, 2020

We are pretty aggressive internally about not allowing replace in non-main modules, and have found it is easier to just disallow it. Valid use cases are either hacking/testing deps, something that should not be released, or upstream bug fixes, where we prefer temporary, internal forks. Would it be reasonable for gorelease to warn or error on any?

@jayconrod
Copy link
Contributor Author

@jayconrod jayconrod commented Jul 20, 2020

I's not necessarily harmful to keep replace directives around, as long as you've tested that your module works without them. But I suspect that's rarely the case, so I'm inclined to agree: gorelease should report a warning.

If we decide to go forward with #40276, there's a much stronger argument: go get -b will report an error when building an executable from a module with a replace. You really wouldn't want to release a version with a replace left over if anyone might go get -b something inside it.

gopherbot pushed a commit to golang/website that referenced this issue Jul 20, 2020
For golang/go#33637

Change-Id: I22f556cd0e1dcd76682d3f79143f1f28558f8766
Reviewed-on: https://go-review.googlesource.com/c/website/+/240688
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 22, 2020

Change https://golang.org/cl/244417 mentions this issue: content/static/doc: add module documentation on GOPATH compatibility

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 24, 2020

Change https://golang.org/cl/244766 mentions this issue: content/static/doc: set proper header ids in module documentation

gopherbot pushed a commit to golang/website that referenced this issue Aug 3, 2020
For golang/go#33637

Change-Id: I58300dfe59b5e4e3cb118fd07f5a00419acc07b8
Reviewed-on: https://go-review.googlesource.com/c/website/+/242318
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Aug 3, 2020
For golang/go#33637

Change-Id: I839a4075df8494389a69416e49e0fdc0ac812818
Reviewed-on: https://go-review.googlesource.com/c/website/+/244417
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Aug 3, 2020
Previously, we used a separate <a> tag before the header. Headers had
temporary ids, and the table of contents linked to those. With this
change, the header elements themselves will have ids.

For golang/go#33637

Change-Id: Ia79048f0db7cc4e8eac3bb5641278acd354fca01
Reviewed-on: https://go-review.googlesource.com/c/website/+/244766
Reviewed-by: Michael Matloob <matloob@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 10, 2020

Change https://golang.org/cl/247762 mentions this issue: content/static/doc: add module documentation on privacy

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 10, 2020

Change https://golang.org/cl/247761 mentions this issue: content/static/doc: edit module documentation on GOPROXY

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 10, 2020

Change https://golang.org/cl/247758 mentions this issue: content/static/doc: add introduction, resolve several TODOs

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 10, 2020

Change https://golang.org/cl/247759 mentions this issue: content/static/doc: add module documentation on build commands

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 10, 2020

Change https://golang.org/cl/247763 mentions this issue: content/static/doc: add module documentation on authentication

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 10, 2020

Change https://golang.org/cl/247760 mentions this issue: content/static/doc: add diagrams for 'go get' module examples

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 14, 2020

Change https://golang.org/cl/248617 mentions this issue: content/static/doc: fix anchors with dots in module documentation

gopherbot pushed a commit to golang/website that referenced this issue Sep 9, 2020
* Added Introduction. This should say generally what modules are. It
  should also say this is a reference document, and point to tutorials
  for folks looking for those.
* Edited some text in introductory sessions.
* Replaced emphasis on new terms with <dfn> tags. Dropped emphasis
  from terms that are being mentioned, not defined.
* Added glossary term for "vendor directory".
* Removed some TODOs that are no longer relevant.

For golang/go#33637

Change-Id: Iae6dcd28b392a4cd94351bc645a059ead3a7dfa8
Reviewed-on: https://go-review.googlesource.com/c/website/+/247758
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Sep 9, 2020
For golang/go#33637

Change-Id: I629768ab8d2708a0f8dad8fa09840b519a9624cd
Reviewed-on: https://go-review.googlesource.com/c/website/+/247759
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Sep 9, 2020
For golang/go#33637

Change-Id: Ie502018aa23324732942e8fd82b45006096c3e1c
Reviewed-on: https://go-review.googlesource.com/c/website/+/247760
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Sep 9, 2020
Clarify error fallback behavior and explain the difference between
comma and pipe separators.

For golang/go#33637

Change-Id: I3a8cf82284ea7c8e7096b8d8595f7cc7a21d7dba
Reviewed-on: https://go-review.googlesource.com/c/website/+/247761
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Sep 9, 2020
Previously, Privacy was intended to be a top-level section. Now, it's
grouped under Private modules, since it provides different advice
depending on how private modules are being retrieved.

For golang/go#33637

Change-Id: I5c70f7b7a0b01b3d5e2719f30d8bc66ffb25c046
Reviewed-on: https://go-review.googlesource.com/c/website/+/247762
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Sep 9, 2020
Also moved some content out of the go.sum section into
authentication. The go.sum section is mostly focused on the format of
go.sum files themselves, rather than the general topic of
authentication.

For golang/go#33637

Change-Id: Ie427924078c3fb7e1201867b3ef2836578e3b877
Reviewed-on: https://go-review.googlesource.com/c/website/+/247763
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Sep 9, 2020
Our markdown renderer was cutting off everything after the dot, so
"go.mod-files" was rendered as "go".

For golang/go#33637
Fixes golang/go#41080

Change-Id: Iefded0f697c4c9c8833303e83de7317c3930091a
Reviewed-on: https://go-review.googlesource.com/c/website/+/248617
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.