Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: default to & improve -mod=readonly #40728

Closed
rsc opened this issue Aug 12, 2020 · 29 comments
Closed

cmd/go: default to & improve -mod=readonly #40728

rsc opened this issue Aug 12, 2020 · 29 comments

Comments

@rsc
Copy link
Contributor

rsc commented Aug 12, 2020

For Go 1.16, I propose to default to & improve -mod=readonly, making -mod=mod completely equivalent to -mod=readonly.
This would mean that go commands other than go mod ... and go get no longer update versions in go.mod.
If go.mod needs to be fixed, these commands would report that error and stop.
And the errors in go list should be reported only for the packages that can't be resolved - the overall go list operation should succeed.

Discussed previously on golang-tools@: https://groups.google.com/g/golang-tools/c/BRCgqwWLwoY.

This is what I sent there:

As you probably know, we're trying to make Go 1.16 the release when there are no blockers remaining for anyone to adopt modules, including things like "go get -b" which Jay has been working on.

I'm wondering, as part of this polishing of modules, if we should change the default behavior and stop doing so much updating of go.mod.

Obviously I can justify the current behavior at length, and have in the past, but I think the past thinking falls apart a bit when users make mistakes.

For example, I was working in a module recently (let's call it m) and had a package (let's call it p), and I decided to rename the package (to q). I caught most of the m/p import paths and updated them to m/q. But not all, and the next time I ran "go build", the go command went off trying to find some other module to provide the no-longer-existent m/p. When m/p is a real thing that exists, automatically updating go.mod to use it makes some sense. When m/p is a mistake that needs updating, doing network work just slows down the real fix.

Another example is adding an import you didn't actually want. You might copy a file into a module m from somewhere else, and when you "go build" it adds something to go.mod silently, perhaps even changing the versions of existing require statements. In this case - since you don't want the import - it would be better for the tool to stop and tell you about it instead of doing additional damage that you will have to undo.

I suspect there are many more of these. In general, while the automatic additions to go.mod seemed good on paper, my impression is that they haven't delivered the vision we wanted: they do the wrong thing too often, they slow down the compile-edit-debug cycle for typos, and they are confusing.

We already have a mode that reports errors instead of making changes, namely -mod=readonly. That flag does not apply to "go get" or "go mod" - those commands' job is to update go.mod and they would still do that - but it stops other commands like "go list", "go build", "go test" from changing go.mod. If we made this mode the default and also cleaned up some of its rough edges (go list can probably do a better job reporting partial results, for example), that might go a long way to fixing the problems I listed above.

If we do change the default, it would probably make sense to change -mod=mod (the name for the current automatic updating) to something like -mod=autoupdate, leaving -mod=mod as an alias for that of course.

The command "go get" (with no arguments) has always done a build of the package in the current directory, including downloading any necessary dependencies. With a default -mod=readonly, "go get" would become the way to do a one-time fix-up of go.mod. If "go build" or "go list" fails because of a missing requirement or other problem with go.mod, "go get" would be the way to fix it. If "go test" fails due to an import specific to the test, we'd need to provide a way to deal with that, probably adding back "go get -t", which was the pre-modules way to download the dependencies for a test.

Along with this I think you'd want to change goimports to have a way to say what new requirements it thinks go along with the import paths. Then tooling invoking goimports could present the new imports and the new go.mod requirements together. Users opting in to that kind of automation would still have the option of both. It just wouldn't happen automatically in commands like "go build". (This listing of new go.mod requirements only matters when goimports has exhausted your module's current requirements and is grubbing around in your module cache. When it finds a match, especially a v0, you probably do want to get the version it found and not whatever the go command decides is latest from the internet. So letting goimports specify the new go.mod requirements would correct a sort of race in the current behavior anyway.)

I haven't heard many people suggesting we should default to -mod=readonly before, so maybe others haven't seen the current behavior as too much of a problem. But maybe people just figure it's not worth bringing up because it can't change at this point. I don't believe that's true - if the behavior is wrong, then Go 1.16 would be the time to get it right (and soon).

@rsc rsc added this to the Proposal milestone Aug 12, 2020
@DisposaBoy
Copy link

DisposaBoy commented Aug 12, 2020

I haven't heard many people suggesting we should default to -mod=readonly before, so maybe others haven't seen the current behavior as too much of a problem.

FWIW, I personally started complaining about this whole magic behaviour as early as #29452, but I didn't feel like anyone was taking it seriously so from that point on I essentially stopped participating in the discussion (IIRC I was even invited to comment on a proposal moving things forward which I naturally ignored).

Not only do I think the default behaviour is C@#0%, I'm 99% sure it's a security issue (which I believe I brought up somewhere else). If you make a simple typo, it goes out and pulls the package off the internet with no input from the user. I know this happens in RL because we previously ran an experiment by registering a few common typos of golang.org and observed people pulling package from it. Those requests returned 404 and was before the checksum database, so I don't know the real security impact but I feel like the fact that it was/is possible to some extent is bad enough.

TL;DR

I'm still salty, but IMO this is the best thing to happen to Go modules since their introduction and I support it fully 💯

@rsc
Copy link
Contributor Author

rsc commented Aug 16, 2020

Another compelling example:

% go run main.g
go: finding module for package main.g
^C

@rasky
Copy link
Member

rasky commented Aug 26, 2020

Actually, running go run MODULE is a very good way to have people running CLI tools written in Go. Would that be broken by this proposal for people running in this command in a folder where go.mod doesn't even exist?

@jayconrod
Copy link
Contributor

jayconrod commented Aug 26, 2020

@rasky In module mode, when invoked outside a module, go run can't build packages outside the standard library that require a module lookup. You'll get an error like this:

$ GO111MODULE=on go run use.go
use.go:3:8: cannot find module providing package golang.org/x/mod/semver: working directory is not part of a module

That was #32027. Before that, pretty much everything required several network fetches, so builds were slow and non-reproducible. We felt it wasn't a good user experience, especially for new users.

@myitcv
Copy link
Member

myitcv commented Aug 27, 2020

@rsc just thinking of side effects of this change, would this change make a go mod download a necessary prestep in CI?

Consider a module which we know to have a complete go.{mod,sum}, that is being tested with an empty module cache (a regular situation on CI). Running go test will necessitate a download of modules, in order to determine the go.{mod,sum} are indeed complete. Are we saying this download will no longer automatically happen? And therefore that a go mod download will be a necessary prestep, that itself will potentially modify the go.{mod,sum} files?

@bcmills
Copy link
Member

bcmills commented Aug 27, 2020

@myitcv, -mod=readonly prevents resolving new modules, but it does not prevent the go command from fetching modules that are already both selected by the requirements in the go.mod file and listed in the go.sum file.

So as long as the module is already tidy (or at least complete) when it gets sent to CI, no explicit go mod download should be needed.

@rasky
Copy link
Member

rasky commented Aug 27, 2020

That was #32027. Before that, pretty much everything required several network fetches, so builds were slow and non-reproducible. We felt it wasn't a good user experience, especially for new users.

Thanks for the pointer, I had missed that discussion. I'm not really happy about the outcome but such is life.

@myitcv
Copy link
Member

myitcv commented Aug 27, 2020

@bcmills - thanks, I was forgetting we are actually defaulting to the semantics of -mod=readonly here, despite that being in the title. Minor brain fade. Apologies for the noise.

@gopherbot
Copy link

gopherbot commented Aug 31, 2020

Change https://golang.org/cl/251881 mentions this issue: cmd/go: default to -mod=readonly in most commands

@gopherbot
Copy link

gopherbot commented Aug 31, 2020

Change https://golang.org/cl/251879 mentions this issue: cmd/go: let 'go list -mod=readonly' succeed if go.mod is inconsistent

@gopherbot
Copy link

gopherbot commented Sep 9, 2020

Change https://golang.org/cl/253837 mentions this issue: cmd/go: update tests to work with -mod=readonly on by default

@gopherbot
Copy link

gopherbot commented Sep 9, 2020

Change https://golang.org/cl/253745 mentions this issue: cmd/go: refactor modload.Import for better -mod=readonly errors

@gopherbot
Copy link

gopherbot commented Sep 9, 2020

Change https://golang.org/cl/253744 mentions this issue: cmd/go: refactor -mod flag parsing

gopherbot pushed a commit that referenced this issue Sep 11, 2020
For #40728

Change-Id: Ic2b025ff75c6e73c0cb58c1737e44e2a41c71571
Reviewed-on: https://go-review.googlesource.com/c/go/+/253837
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
gopherbot pushed a commit that referenced this issue Sep 11, 2020
Keep track of whether the -mod flag was set explicitly. When
-mod=readonly is the default, we'll want to adjust our error messages
if it's set explicitly.

Also, register the -mod, -modcacherw, and -modfile flags in functions
in internal/base instead of internal/work. 'go mod' commands that
don't load packages shouldn't depend on internal/work.

For #40728

Change-Id: I272aea9e19908ba37e151baac4ea8630e90f241f
Reviewed-on: https://go-review.googlesource.com/c/go/+/253744
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
gopherbot pushed a commit that referenced this issue Sep 11, 2020
When -mod=readonly is set, Import will now allow imports from
replacements without explicit requirements. With -mod=mod, this would
add a new requirement but does not trigger a module lookup, so it's
determinisitic.

Before reporting an error for an unknown import with -mod=readonly,
check whether the import is valid. If there's a typo in the import,
that's more relevant.

For #40728

Change-Id: I05e138ff76ba3d0eb2e3010c15589fa363deb8d3
Reviewed-on: https://go-review.googlesource.com/c/go/+/253745
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
gopherbot pushed a commit that referenced this issue Sep 15, 2020
For #40728

Change-Id: I6618f1b5a632e8b353a483a83bb0cdf4ef6df72c
Reviewed-on: https://go-review.googlesource.com/c/go/+/251881
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

gopherbot commented Oct 28, 2020

Change https://golang.org/cl/263267 mentions this issue: cmd/go/internal/modget: resolve paths at the requested versions

gopherbot pushed a commit that referenced this issue Nov 5, 2020
Previously, we resolved each argument to 'go get' to a package path or
module path based on what was in the build list at existing versions,
even if the argument specified a different version explicitly. That
resulted in bugs like #37438, in which we variously resolved the wrong
version or guessed the wrong argument type for what is unambiguously a
package argument at the requested version.

We were also using a two-step upgrade/downgrade algorithm, which could
not only upgrade more that is strictly necessary, but could also
unintentionally upgrade *above* the requested versions during the
downgrade step.

This change instead uses an iterative approach, with an explicit
disambiguation step for the (rare) cases where an argument could match
the same package path in multiple modules. We use a hook in the
package loader to halt package loading as soon as an incorrect version
is found — preventing over-resolving — and verify that the result
after applying downgrades successfully obtained the requested versions
of all modules.

Making 'go get' be correct and usable is especially important now that
we are defaulting to read-only mode (#40728), for which we are
recommending 'go get' more heavily.

While I'm in here refactoring, I'm also reworking the API boundary
between the modget and modload packages. Previously, the modget
package edited the build list directly, and the modload package
accepted the edited build list without validation. For lazy loading
(#36460), the modload package will need to maintain additional
metadata about the requirement graph, so it needs tighter control over
the changes to the build list.

As of this change, modget no longer invokes MVS directly, but instead
goes through the modload package. The resulting API gives clearer
reasons in case of updates, which we can use to emit more useful
errors.

Fixes #37438
Updates #36460
Updates #40728

Change-Id: I596f0020f3795870dec258147e6fc26a3292c93a
Reviewed-on: https://go-review.googlesource.com/c/go/+/263267
Trust: Bryan C. Mills <bcmills@google.com>
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
@mvdan
Copy link
Member

mvdan commented Nov 8, 2020

@rsc @bcmills this is missing from the release notes still, right? It seems like a big change that should be there before beta1 is tagged.

@gopherbot
Copy link

gopherbot commented Nov 10, 2020

Change https://golang.org/cl/268859 mentions this issue: cmd/go: release note for -mod=readonly by default

@gopherbot
Copy link

gopherbot commented Jan 20, 2021

Change https://golang.org/cl/285113 mentions this issue: content/static/doc: document -mod=readonly enabled by default

gopherbot pushed a commit to golang/website that referenced this issue Feb 16, 2021
For golang/go#40728

Change-Id: I9617d2ebd920d1a0de11c3d7ae9d99505f282b84
Reviewed-on: https://go-review.googlesource.com/c/website/+/285113
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@jdog8541
Copy link

jdog8541 commented May 6, 2021

Man you guys are kicking ass!!! Keep up the good work!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests