Skip to content

Commit

Permalink
_content/ref/mod: document Go 1.17 lazy loading and module graph pruning
Browse files Browse the repository at this point in the history 
For golang/go#36460

Change-Id: I36b3657103f069412b225356d011a19c1a10109e
Reviewed-on: https://go-review.googlesource.com/c/website/+/333629
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
  • Loading branch information
Bryan C. Mills committed Jul 16, 2021
1 parent 11adafe commit fa093af
Showing 1 changed file with 153 additions and 22 deletions.
175 changes: 153 additions & 22 deletions _content/ref/mod.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -564,6 +564,13 @@ specified by the `go` directive. This has the following effects:
[`go mod vendor`](#go-mod-vendor) since modules were introduced. In lower
versions, `all` also includes tests of packages imported by packages in
the main module, tests of those packages, and so on.
* At `go 1.17` or higher, the `go.mod` file includes an explicit [`require`
directive](#go-mod-file-require) for each module that provides any package
transitively imported by a package or test in the main module. (At `go 1.16`
and lower, an [indirect dependency](#glos-direct-dependency) is included only
if [minimal version selection](#minimal-version-selection) would otherwise
select a different version.) This extra information enables [lazy
loading](#lazy-loading) and [module graph pruning](#graph-pruning).

A `go.mod` file may contain at most one `go` directive. Most commands will add a
`go` directive with the current Go version if one is not present.
Expand All @@ -590,14 +597,22 @@ the [build list](#glos-build-list).

The `go` command automatically adds `// indirect` comments for some
requirements. An `// indirect` comment indicates that no package from the
required module is directly imported by any package in the main module.
The `go` command adds an indirect requirement when the selected version of a
module is higher than what is already implied (transitively) by the main
module's other dependencies. That may occur because of an explicit upgrade
(`go get -u`), removal of some other dependency that previously imposed the
requirement (`go mod tidy`), or a dependency that imports a package without
a corresponding requirement in its own `go.mod` file (such as a dependency
that lacks a `go.mod` file altogether).
required module is directly imported by any package in the [main
module](#glos-main-module).

If the [`go` directive](#go-mod-file-go) specifies `go 1.16` or lower, the `go`
command adds an indirect requirement when the selected version of a module is
higher than what is already implied (transitively) by the main module's other
dependencies. That may occur because of an explicit upgrade (`go get -u ./...`),
removal of some other dependency that previously imposed the requirement (`go
mod tidy`), or a dependency that imports a package without a corresponding
requirement in its own `go.mod` file (such as a dependency that lacks a `go.mod`
file altogether).

At `go 1.17` and above, the `go` command adds an indirect requirement for each
module that provides a package transitively imported by any package or test in
the main module. These extra requirements enable [lazy loading](#lazy-loading)
and [module graph pruning](#graph-pruning).

```
RequireDirective = "require" ( RequireSpec | "(" newline { RequireSpec } ")" newline ) .
Expand Down Expand Up @@ -964,6 +979,89 @@ is changed to 1.1.
suffix after an argument. This works similarly to a downgrade. All versions
of the named module are removed from the module graph.

## Module graph redundancy {#graph-redundancy}

In a module with a [`go` directive](#go-mod-file-go) at `go 1.17` or higher, the
`go.mod` file is expected to include an explicit [`require`
directive](#go-mod-file-require) to the `go.mod` file for the [selected
version](#glos-selected-version) of every module that provides any package
imported (even [indirectly](#glos-indirect-dependency)) by a package or test in
the [main module](#glos-main-module).

When the `go` command loads packages it makes a best effort to check and
maintain this property: [`go mod tidy`](#go-mod-tidy) adds indirect requirements
for all packages imported by the main module, and [`go get`](#go-get) adds
indirect requirements for the packages imported by those named on the command
line.

A module that _does not_ provide any imported package cannot affect the compiled
artifacts produced by `go build` and `go test`, so this added redundancy allows
packages in the main module to be built and tested using only the requirements
listed in the its own `go.mod` file. The `go` command takes advantage of that
property to substantially reduce the size of the overall module graph.

(At `go 1.16` and below, the `go.mod` file includes only [direct
dependencies](#glos-direct-dependency), so a much larger graph must be loaded to
ensure that all indirect dependencies are included.)

See [the design document](/design/36460-lazy-module-loading) for more detail.

### Lazy loading {#lazy-loading}

If the main module is at `go 1.17` or higher, the `go` command avoids loading
the complete module graph until (and unless) it is needed. Instead, it loads
only the main module's `go.mod` file, then attempts to load the packages to be
built using only those requirements. If a package to be imported (for example, a
dependency of a test for a package outside the main module) is not found among
those requirements, then the rest of the module graph is loaded on demand.

If all packages can be found without loading the full module graph, the `go`
command then loads the `go.mod` files for _only_ the modules containing those
packages, and their requirements are checked against the requirements of the
main module to ensure that they are locally consistent. (Inconsistencies can
arise due to version-control merges, hand-edits, and changes in modules that
have been [replaced](#go-mod-file-replace) using local filesystem paths.)

### Module graph pruning {#graph-pruning}

If the main module is at `go 1.17` or higher, the [module
graph](#glos-module-graph) used for [minimal version
selection](#minimal-version-selection) includes only the _immediate_ (not
transitive) requirements for each module dependency that specifies `go 1.17` or
higher in its own `go.mod` file, unless that version of the module is also
(transitively) required by some _other_ dependency at `go 1.16` or below.

Since a `go 1.17` `go.mod` file includes every dependency needed to build any
package or test in that module, the pruned module graph includes every
dependency needed to `go build` or `go test` the packages in every dependency
explicitly [required](#go-mod-file-require) by the main module.

Modules whose requirements have been pruned out still appear in the module graph
and are still reported by `go list -m all`: their [selected
versions](#glos-selected-version) are known and well-defined, and packages can
be loaded from those modules (for example, as transitive dependencies of tests
loaded from other modules). However, since the `go` command cannot easily
identify which dependencies of these modules are satisfied, the arguments to `go
build` and `go test` cannot include packages from modules whose requirements
have been pruned out. [`go get`](#go-get) promotes the module containing each
named package to an explicit dependency, allowing `go build` or `go test` to be
invoked on that package.

Because Go 1.16 and earlier did not support module graph pruning, the full
transitive closure of dependencies — including transitive `go 1.17` dependencies
— is still included for each module that specifies `go 1.16` or lower. (The
`go.mod` file for a `go 1.16` or lower module does not necessarily include the
requirements needed to build the packages and tests in that module.)

The [`go.sum` file](#go-sum-files) recorded by [`go mod tidy`](#go-mod-tidy) for
a module by default includes checksums needed by the Go version _one below_ the
version specified in its [`go` directive](#go-mod-file-go). So a `go 1.17`
module includes checksums needed for the full module graph loaded by Go 1.16,
but a `go 1.18` module will include only the checksums needed for the pruned
module graph loaded by Go 1.17. The `-compat` flag can be used to override the
default version (for example, to prune the `go.sum` file more aggressively in a
`go 1.17` module).

## Compatibility with non-module repositories {#non-module-compat}

To ensure a smooth transition from `GOPATH` to modules, the `go` command can
Expand Down Expand Up @@ -1704,7 +1802,7 @@ to parse, edit, and format `go.mod` files.
Usage:

```
go mod graph
go mod graph [-go=version]
```

The `go mod graph` command prints the [module requirement
Expand All @@ -1729,6 +1827,10 @@ space-separated fields: a module version and one of its dependencies. Each
module version is identified as a string of the form `path@version`. The main
module has no `@version` suffix, since it has no version.

The `-go` flag causes `go mod graph` to report the module graph as
loaded by the given Go version, instead of the version indicated by
the [`go` directive](#go-mod-file-go) in the `go.mod` file.

See [Minimal version selection (MVS)](#minimal-version-selection) for more
information on how versions are chosen. See also [`go list -m`](#go-list-m) for
printing selected versions and [`go mod why`](#go-mod-why) for understanding
Expand Down Expand Up @@ -1786,7 +1888,7 @@ requirements and to drop unused requirements.
Usage:

```
go mod tidy [-e] [-v]
go mod tidy [-e] [-v] [-go=version] [-compat=version]
```

`go mod tidy` ensures that the `go.mod` file matches the source code in the
Expand Down Expand Up @@ -1814,20 +1916,30 @@ with names that start with `.` or `_` unless those packages are explicitly
imported by other packages.

Once `go mod tidy` has loaded this set of packages, it ensures that each module
that provides one or more packages either has a `require` directive in the main
module's `go.mod` file or is required by another required module. `go mod tidy`
will add a requirement on the latest version on each missing module (see
[Version queries](#version-queries) for the definition of the `latest`
version). `go mod tidy` will remove `require` directives for modules that don't
provide any packages in the set described above.
that provides one or more packages has a `require` directive in the main
module's `go.mod` file or, if the main module is at `go 1.16` or below (see
[Module graph redundancy](#graph-redundancy)), is required by another required
module. `go mod tidy` will add a requirement on the latest version on each
missing module (see [Version queries](#version-queries) for the definition of
the `latest` version). `go mod tidy` will remove `require` directives for
modules that don't provide any packages in the set described above.

`go mod tidy` may also add or remove `// indirect` comments on `require`
directives. An `// indirect` comment denotes a module that does not provide
packages imported by packages in the main module. These requirements will be
present if the module that imports packages in the indirect dependency has
no `go.mod` file. They may also be present if the indirect dependency is
required at a higher version than is implied by the module graph; this usually
happens after running a command like `go get -u ./...`.
directives. An `// indirect` comment denotes a module that does not provide a
package imported by a package in the main module. (See the [`require`
directive](#go-mod-file-require) for more detail on when `// indirect`
dependencies and comments are added.)

If the `-go` flag is set, `go mod tidy` will update the [`go`
directive](#go-mod-file-go) to the indicated version, enabling or disabling
[lazy loading](#lazy-loading) and [module graph pruning](#graph-pruning) (and
adding or removing indirect requirements as needed) according to that version.

By default, `go mod tidy` will check that the [selected
versions](#glos-selected-version) of modules do not change when the module graph
is loaded by the Go version immediately preceding the version indicated in the
`go` directive. The versioned checked for compatibility can also be specified
explicitly via the `-compat` flag.

### `go mod vendor` {#go-mod-vendor}

Expand Down Expand Up @@ -3887,6 +3999,12 @@ A deprecated module is marked with a [deprecation
comment](#go-mod-file-module-deprecation) in the latest version of its
[`go.mod` file](#glos-go-mod-file).

<a id="glos-direct-dependency"></a>
**direct dependency:** A package whose path appears in an [`import`
declaration](/ref/spec#import_declarations) in a `.go` source file for a package
or test in the [main module](#glos-main-module), or the module containing such a
package. (Compare [indirect dependency](#glos-indirect-dependency).)

<a id="glos-direct-mode"></a>
**direct mode:** A setting of [environment variables](#environment-variables)
that causes the `go` command to download a module directly from a [version
Expand All @@ -3904,6 +4022,14 @@ files](#go-mod-file).
**import path:** A string used to import a package in a Go source file.
Synonymous with [package path](#glos-package-path).

<a id="glos-indirect-dependency"></a>
**indirect dependency:** A package transitively imported by a package or test in
the [main module](#glos-main-module), but whose path does not appear in any
[`import` declaration](/ref/spec#import_declarations) in the main module;
or a module that appears in the [module graph](#glos-module-graph) but does not
provide any package directly imported by the main module.
(Compare [direct dependency](#glos-direct-dependency).)

<a id="glos-main-module"></a>
**main module:** The module in which the `go` command is invoked. The main
module is defined by a [`go.mod` file](#glos-go-mod-file) in the current
Expand Down Expand Up @@ -4030,6 +4156,11 @@ after it was published. See [`retract` directive](#go-mod-file-retract).
[version](#glos-version) to a specific revision. See [Mapping versions to
commits](#vcs-version).

<a id="glos-selected-version"></a>
**selected version:** The version of a given module chosen by [minimal version
selection](#minimal-version-selection). The selected version is the highest
version for the module's path found in the [module graph](#glos-module-graph).

<a id="glos-vendor-directory"></a>
**vendor directory:** A directory named `vendor` that contains packages from
other modules needed to build packages in the main module. Maintained with
Expand Down

0 comments on commit fa093af

Please sign in to comment.