cmd/go: flags to control changes to go.mod, go.sum

[jayconrod]

Proposed changes

I propose we add three new flags to go subcommands that deal with modules.

• -modfile=go.mod - instead of reading and writing go.mod from the current directory or a parent directory, the go command would read and write the specified file. The original go.mod would still determine the module root directory, so this file could be in any directory (perhaps /tmp). It would be an error to use this flag when no actual go.mod file is present.
• -sumfile=go.sum - instead of reading and writing go.sum from the directory containing go.mod, the go command would read and write the specified file. It would be an error to use this flag when no original go.mod file is present.
• -g - "global mode" - the go command would behave as if no go.mod file were present.

These flags would be allowed when GO111MODULE is set to on or auto (the current default) and rejected when GO111MODULE is set to off. -modfile and -sumfile both require an actual go.mod to be present, so modules must be enabled when they're used. -g does not require an actual go.mod to be present, and in auto mode, it implies that modules are enabled.

Background

The go command updates go.mod and go.sum after any command that needs to find a module for a package not provided by any module currently in the build list. This ensures reproducibility: if you run the same command twice, it should build (or list or test) the same packages at the same versions, even if new versions have been published since the first invocation.

For example, if you run go build ., and a .go file in the current directory imports example.com/m/pkg which is not provided by any known module, the go command will add a requirement on the latest version of the module example.com/m to go.mod. Future runs of go build . will produce the same result.

While these updates are usually helpful, there are many situations where they're not desirable.

Selected issues

gopls (various issues)

gopls loads information about source files in a workspace using golang.org/x/tools/go/packages, which invokes go list. gopls may also run go list directly. In either case, gopls may trigger changes to go.mod and go.sum. This may be caused by user actions that seem unrelated to building anything, for example, opening a file. go.mod appears to change mysteriously on its own, and users don't realize gopls is triggering it.

It's not usually important that the information gopls loads is reproducible; files it operates on are frequently changing. However, it is important that when it resolves an unknown import path to a module, it doesn't need to do so repeatedly since this can add a lot of latency, especially on slow connections.

gopls could set -modfile and -sumfile to temporary copies of the original go.mod and go.sum. The original go.mod and go.sum would not be modified (until the user explicitly runs a command like go build). Resolved module requirements would stay in the temporary files so they would not need to be resolved again.

#25922 - clarify best practice for tool dependencies

Developers need a way to express module requirements that aren't implied by package imports. This is especially useful for tools invoked by go generate. Authors can add tool requirements to go.mod manually or with go get, but these requirements are erased by go mod tidy.

The current recommendation is to create a tools.go file, tag it with // +build tools, then import main packages of needed tools. tools.go will never be built because of the tag, but go mod tidy will read the imports and preserve the requirements. This feels like a hacky workaround rather than a best practice. It also pushes requirements which may not otherwise be needed on downstream modules.

A better solution would be to keep a separate go.tools.mod file with tool requirements, then point to that with -modfile=go.tools.mod when running commands that require tools.

#26640 - allow go.mod.local to contain replace/exclude lines

This is a feature request to keep some go.mod statements out of source control. It's frequently useful for module authors to check out dependencies and point to them with replace statements for local development and debugging. These statements shouldn't necessarily be exposed to users or other developers on the same project though.

Setting -modfile=go.local.mod and -sumfile=go.local.sum would solve this problem, at least partially. The two files could be copied from the regular go.mod and go.sum files and added to .gitignore. Note however, that these local files are used instead of the regular files, not in addition to, so some synchronization might be required.

#30515 - offer a consistent "global install" command

Developers want to be able to install tools from any directory, regardless of the requirements of the current module. go get tool@version may update the current go.mod, so users need to change to a temporary directory without a go.mod file to run commands like this. Tool authors need to be careful when writing installation instructions because of this.

The -g flag would address this issue. It would tell the go command to run as if it were outside any module. Tool authors could write go get -g tool@latest in their installation instructions: this would install the latest version of the tool, regardless of the current directory.

Note that "missing go.mod" is being reconsidered (#32027), so the actual semantics of -g may change: this issue is just about ignoring the current module.

#33710 - module mode removes concept of global docs

In module mode, go doc example.com/pkg prints documentation for the packages named on the command line at the same version they would be built with. Like go build, go doc may add or update requirements in go.mod. This may be undesirable, especially if you're using the documentation to decide whether you want to depend on a package that is not currently imported.

The -g flag would partially solve this. The current module would be ignored, and "global" documentation would be shown.

Note that go doc does not currently work in "missing go.mod" for packages outside std. #33710 would need to be fixed, but -g would provide a useful way to access that mode.

Other related issues

There are a large number of open issues about unexpected and unwanted go.mod changes. The flags suggested here won't solve all these problems, but they provide useful context.

• cmd/go: do not resolve missing imports in 'go mod why' #26977 - go mod why adds a go.mod line
• cmd/go: go list has too many (more than zero) side effects #29452 - go list has too many (more than zero) side effects
• cmd/go: 'go list' should not resolve or record modules that are not relevant to the requested output fields #29869 - 'go list' should not resolve or record modules that are not relevant to the requested output fields
• cmd/go: 'mod verify' should not modify the go.mod file #31372 - 'mod verify' does not respect -mod=readonly
• x/tools/gopls: support go.mod files #31999 - x/tools/gopls: support go.mod files
• cmd/go: rethink "missing go.mod" mode #32027 - rethink "missing go.mod" mode
• cmd/go: go install: don’t fail when go.mod can’t be updated on a read-only system #33326 - go install: don’t fail when go.mod can’t be updated on a read-only system
• cmd/go: 'go mod download' should not modify the go.mod file #34450 - go mod download support -readonly

[jayconrod]

cc @bcmills @rsc @stamblerre @ianthehat

[bcmills]

-g - "global mode" - the go command would behave as if no go.mod file were present.

Per #32027 (comment), we'll probably want to make the go command less willing to resolve dependencies when no go.mod file is present.

If we do that, presumably the -g flag will allow the go command to resolve the transitive dependencies of the modules containing the packages listed on the command line. Should -g also allow the go command to resolve missing dependencies found in the import statements of those packages?

[bcmills]

The original go.mod would still determine the module root directory,

but

These flags would be accepted when GO111MODULE is set to on and rejected when GO111MODULE is set to off.

If GO111MODULE is on but there is no go.mod file above the current working directory, what would we use as the module root?

[jayconrod]

Per #32027 (comment), we'll probably want to make the go command less willing to resolve dependencies when no go.mod file is present.

If we do that, presumably the -g flag will allow the go command to resolve the transitive dependencies of the modules containing the packages listed on the command line. Should -g also allow the go command to resolve missing dependencies found in the import statements of those packages?

I think -g should just make the go command do whatever it would do without a go.mod file. So if go run x.go resolves transitively imports and succeeds, go run -g x.go would do the same within a module. But if we change go run x.go to fail, then go run -g x.go would also fail.

[jayconrod]

If GO111MODULE is on but there is no go.mod file above the current working directory, what would we use as the module root?

I contradicted myself a bit here. Updated the text. -modfile and -sumfile require an actual go.mod to be present in order to set the module root directory.

[heschi]

Regarding the file flags: I think having to specify both -modfile and -sumfile is cumbersome. I can't think of any compelling reason to want to share go.sum with any other module, since go mod tidy will throw away the unnecessary lines anyway. So I would suggest that at a minimum, -sumfile be derived from -modfile if it's not set. A more extreme option would be to specify a single prefix, say -modprefix, and add .mod and .sum to it as needed. That may be too strange, though.

This is also a good precedent to set in case there's ever a third file, since nobody will know to override its location before it exists.

[myitcv]

@jayconrod regarding the "gopls (various issues)" motivation. I'm not entirely clear on what the issue here is and hence why -g/-modfile/other is a solution.

Is the problem that people are not used to cmd/go having side effects in module mode?

Because for me, I would expect that anything I do inside my editor, e.g. adding an import for a package whose module is not in my go.mod, could have side effects on the main module.

Could you give a bit more background?

[mvdan]

#30515 - offer a consistent "global install" command

While not part of the original issue, the consensus for a while seemed to be that a "global" install should also obey replace directives. At least, this is what @ianthehat said in #30515 (comment). Has the team's position generally changed to not obeying replace directives at all?

I'm also a bit confused by the multiple flags, like @heschik. How about a -modroot=path flag? It would roughly be equivalent to cd path && go <args>.

[myitcv]

Possibly worth reiterating that replace directives come in two forms:

• directory-based
• non-directory-based

I think the consensus that @mvdan refers to above was on the latter being applied.

See @mvdan's comment following this one.