Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: rethink "missing go.mod" mode #32027

Closed
rsc opened this issue May 14, 2019 · 38 comments

Comments

@rsc
Copy link
Contributor

commented May 14, 2019

A few weeks ago, shortly after GO111MODULE=on became the default, I got to watch Rob using it. One problem he had was that when he worked in preexisting GOPATH directories, or just directories newly created for the purpose of testing one bug or another, there was no go.mod, so every go command reresolved "latest" for any imported dependencies. This made things much slower than they are intended to be (that is, much slower than they are when there is a go.mod).

I think we probably need to rethink what we do without any go.mod. There needs to be some place for the go command to write down its decisions from one command to the next. When people say "modules are slow" I wonder if this case is one of the things they mean.

/cc @bcmills @jayconrod @ianthehat

@rsc rsc added this to the Go1.14 milestone May 14, 2019
@rsc rsc added the release-blocker label May 14, 2019
@bcmills

This comment has been minimized.

Copy link
Member

commented May 14, 2019

Some ideas (separated for emoji-response tracking):

  • When we need to implicitly resolve a package as part of a go build or go test (rather than, say, a go get), we could pull in the latest version found in the module cache instead of trying to get the absolute latest upstream version.
@bcmills

This comment has been minimized.

Copy link
Member

commented May 14, 2019

  • We could revive the “default go.mod” idea (from #24250 (comment) and others).

The downside of that approach is that it would be really easy for users to unintentionally get stuck on some really old version of a dependency and never realize that they need to upgrade it. (That's one of the more annoying failure modes of GOPATH today.)

@bcmills

This comment has been minimized.

Copy link
Member

commented May 14, 2019

  • We could be less willing to run commands outside of a module in general; for example, we could refuse to run go get unless the user passed an explicit “global install” flag (#30515), and we could refuse to resolve any dependencies from outside the standard library during a go build.

That would at least prompt the user to run go mod init earlier in their workflow, which would mitigate some of the sources of slowness.

@beoran

This comment has been minimized.

Copy link

commented May 15, 2019

An important use case for no go.mod file are tools that do not use any imported dependencies. For certain software, such as, say, a Go module proxy, or a simple command line tool, the programmers might want to avoid a chicken-and-egg bootstrap problem and therefore depend only on the Go standard library. Whatever the solution, I'd like this use case to be taken into consideration.

@bcmills

This comment has been minimized.

Copy link
Member

commented May 15, 2019

@beoran, maintaining a go.mod file for a package with few (or no) external dependencies should be a negligible amount of overhead (since it never needs to change), and the go.mod file has some other important effects (such as the go directive, which declares the expected version of the Go language specification).

I think the bug-testing and legacy-GOPATH use cases are more compelling.

@rsc

This comment has been minimized.

Copy link
Contributor Author

commented Sep 19, 2019

Looking at cmd/go/testdata/script/mod_outside.txt, the only "slow success" cases (where there's no go.mod so every execution has to re-resolve unknown imports) are things like 'go run x.go' (or go build, go vet etc) where x.go contains an import of a package from outside the standard library. If we make that case fail with a message about not having a go.mod, then I think we can consider this issue fixed as far as blocking setting GO111MODULE=on by default.

@gopherbot

This comment has been minimized.

Copy link

commented Oct 3, 2019

Change https://golang.org/cl/198778 mentions this issue: cmd/go: forbid resolving import to modules when outside of a module

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

I wrote this comment on the CL:

With this CL applied, I can no longer go run my tools outside a module:

$ go run gioui.org/cmd/gogio
package gioui.org/cmd/gogio: cannot look up module for package "gioui.org/cmd/gogio": working directory is not part of a module

I can't even run (demo) programs:

$ go run gioui.org/apps/hello
package gioui.org/apps/hello: cannot look up module for package "gioui.org/apps/hello": working directory is not part of a module

This behaviour change seems to be a regression, in particular in light of https://golang.org/issue/33518.

"go get" is exempt from the new behaviour, why not "go run"?

@bcmills replied

go get is exempt because people generally only run it once: you run go get, and that resolves dependencies and installs a binary that you then invoke directly.

go run is not exempt because it is more frequently run many times in succession. That imposes a recurring cost (and non-reproducible behavior) to re-resolve the dependencies, which would become a one-time cost (and reproducible behavior) if the resolution could be recorded in a go.mod file.

Let me defend go run outside a module:

First, this CL is taking away a useful, albeit slow, feature without a replacement. I don't consider the usual workaround "use go get and run the installed command" a good enough replacement for go run. See #33518 for my arguments.

Second, this issue seems to be about running local Go packages and standalone programs. Quoting Russ:

Looking at cmd/go/testdata/script/mod_outside.txt, the only "slow success" cases (where there's no go.mod so every execution has to re-resolve unknown imports) are things like 'go run x.go' (or go build, go vet etc) where x.go contains an import of a package from outside the standard library.

I can live with not allowing go run of local files and packages outside a module. What I'm interesting in is running already published modules, such as

$ go run gioui.org/apps/hello
$ go run gioui.org/cmd/gogio -target android gioui.org/apps/hello

Ironically, both gioui.org/cmd/gogio and gioui.org/apps/hello are themselves modules, even though the go run command is run outside one.

Note that even though I personally don't care much about go running local files and packages outside a module doesn't mean it's a good idea to disallow it. Writing a short Go program and then go running it is a very useful and lightweight way to work. Forcing me to do go mod init example.com just to please go run seems like unnecessary bureaucracy (writing this argument out, I'm curious what Rob would say about this extra requirement in his work). play.golang.org doesn't force me to deal with modules, why should go run?

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default.

In short, I think submitting CL 198778 as is would be a mistake and I would be quite disappointed to see a useful feature go away for efficiency reasons.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default.

I ran the following experiment:

$ go version
go version devel +f923c66893 Sat Sep 14 13:08:03 2019 +0200 linux/amd64
$ export GOPATH=$(mktemp -d)
$ time go run gioui.org/cmd/gogio
go: finding gioui.org latest
go: finding gioui.org/cmd latest
go: downloading gioui.org v0.0.0-20191003172637-31e12607d739
go: downloading gioui.org/cmd v0.0.0-20191003172637-31e12607d739
go: extracting gioui.org v0.0.0-20191003172637-31e12607d739
go: extracting gioui.org/cmd v0.0.0-20191003172637-31e12607d739
go: downloading gioui.org v0.0.0-20190930145547-3784ece6dd65
go: extracting gioui.org v0.0.0-20190930145547-3784ece6dd65
go: downloading golang.org/x/image v0.0.0-20190802002840-cff245a6509b
go: downloading golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: extracting golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: extracting golang.org/x/image v0.0.0-20190802002840-cff245a6509b
go: finding golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: finding golang.org/x/image v0.0.0-20190802002840-cff245a6509b
gogio: specify a package
exit status 1

real	0m3,664s
user	0m2,028s
sys	0m0,662s
$ time go run gioui.org/cmd/gogio
go: finding gioui.org latest
go: finding gioui.org/cmd latest
gogio: specify a package
exit status 1

real	0m0,511s
user	0m0,413s
sys	0m0,158s

The first run completed in just under 5 seconds, the second in less than a second.

For comparison, I ran gogio inside the gioui.org/cmd module:

$ time go run ./gogio/
gogio: specify a package
exit status 1

real	0m0,369s
user	0m0,334s
sys	0m0,122s

That is, a difference of ~200ms between inside and outside a module.

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 4, 2019

Ironically, both gioui.org/cmd/gogio and gioui.org/apps/hello are themselves modules, even though the go run command is run outside one.

That's an interesting input for #30515 / #34506. In particular, it suggests that we may want to allow go run (and possibly go install) to accept the same “global install” flag as go get in order to explicitly opt in to the slower path.

Even then, I think that only get, if that, should resolve dependencies that are not already specified by the module(s) containing the package argument(s).

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default. ... I would be quite disappointed to see a useful feature go away for efficiency reasons.

To be clear, we're not planning to remove this for efficiency reasons: we're planning to remove this because we don't feel that it's a good user experience. When GOPATH is eventually deprecated and the default value of GO111MODULE is on, a lot of users, especially those new to Go, will be stuck in this mode without realizing it. In this mode, Go essentially acts as a scripting language that goes out to the network and downloads the latest version of each imported package (and if imported packages haven't migrated to modules, their transitive dependencies as well). While this produces a correct result, it's very slow, and it's not reproducible (a key goal of modules). We don't want people to come away from this thinking "Is Go always this slow? This is horrible."

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default.

GOPROXY certainly speeds things up, but asking for the latest version of a module can still be quite slow. If it's a module that is requested infrequently, the "latest" version may not be cached, and the proxy may need to fetch the repository again.

Additionally, if any of the imported packages don't have go.mod files, we have to resolve the "latest" versions of their dependencies (and so on).

Even in the fast case, you're going out to the network, which will be slower than local builds with a warm or mostly warm cache.

First, this CL is taking away a useful, albeit slow, feature without a replacement. I don't consider the usual workaround "use go get and run the installed command" a good enough replacement for go run. See #33518 for my arguments.

I read through #33518, but I'm not directly familiar with Gio, and I'm not really sure what the workflow is. Why is go get followed by a direct call to the installed binary not usable? In that issue, it seemed like the main concerns were that linked binaries aren't cached (#33468) and that go install might not write to a directory in PATH. Those seem like issues that can be addressed separately.

Also from reading the intro text at https://git.sr.ht/~eliasnaur/gio, it sounds like this tool builds and packages programs. Won't most users always be running it in their own modules? Also, shouldn't a fixed, predictable version of gio be used instead of the latest?

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default. ... I would be quite disappointed to see a useful feature go away for efficiency reasons.

To be clear, we're not planning to remove this for efficiency reasons: we're planning to remove this because we don't feel that it's a good user experience. When GOPATH is eventually deprecated and the default value of GO111MODULE is on, a lot of users, especially those new to Go, will be stuck in this mode without realizing it. In this mode, Go essentially acts as a scripting language that goes out to the network and downloads the latest version of each imported package (and if imported packages haven't migrated to modules, their transitive dependencies as well). While this produces a correct result, it's very slow, and it's not reproducible (a key goal of modules). We don't want people to come away from this thinking "Is Go always this slow? This is horrible."

Let me point out that it's not always slow. With a warm cache, my experiment above measures the difference in the hundreds of milliseconds. Furthermore, the output "go: finding blah go: downloading blah go: extracting blah" is a clear hint to why a cold go run is slow.

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default.

GOPROXY certainly speeds things up, but asking for the latest version of a module can still be quite slow. If it's a module that is requested infrequently, the "latest" version may not be cached, and the proxy may need to fetch the repository again.

Ok, but for the second and subsequent runs, go run only has to do ~1 query to the proxy which is likely to have the result cached, right?

Additionally, if any of the imported packages don't have go.mod files, we have to resolve the "latest" versions of their dependencies (and so on).

As more and more people switch to modules, this problem diminishes. In particular when GO111MODULE=on becomes the default, which to many indicates that Go modules are "ready for general use".

Even in the fast case, you're going out to the network, which will be slower than local builds with a warm or mostly warm cache.

I agree.

First, this CL is taking away a useful, albeit slow, feature without a replacement. I don't consider the usual workaround "use go get and run the installed command" a good enough replacement for go run. See #33518 for my arguments.

I read through #33518, but I'm not directly familiar with Gio, and I'm not really sure what the workflow is. Why is go get followed by a direct call to the installed binary not usable? In that issue, it seemed like the main concerns were that linked binaries aren't cached (#33468) and that go install might not write to a directory in PATH. Those seem like issues that can be addressed separately.

Also from reading the intro text at https://git.sr.ht/~eliasnaur/gio, it sounds like this tool builds and packages programs. Won't most users always be running it in their own modules?

Most of the time, sure. But the critical time is when a newcomer evaluates Gio. I'd like for their experience to be as painfree as possible and being able to run a Gio program with a simple go run without the PATH hassle is invaluable.

When a Gio user finally decides to use Gio, publishes their Gio program to some Git repository, they in turn can tell their users to evaluate it with just

$ go run <github.com/user/shinyprogram>

I imagine most projects would like the simplest possible "get started" steps. go run is that.

Also, shouldn't a fixed, predictable version of gio be used instead of the latest?

Sure, and that's why I suggest go running gogio in my instructions. But that doesn't stop users go installing gogio which is not reproducible. In other words, gogio must support the "use latest version" use case (when Gio reaches 1.0), and if so, it will also support the very convenient

$ go run gioui.org/cmd/gogio -target android <github.com/yourapp/blah>

outside a module.

Responding to both @bcmills'

Ironically, both gioui.org/cmd/gogio and gioui.org/apps/hello are themselves modules, even though the go run command is run outside one.

That's an interesting input for #30515 / #34506. In particular, it suggests that we may want to allow go run (and possibly go install) to accept the same “global install” flag as go get in order to explicitly opt in to the slower path.

and @jayconrod's

I read through #33518, but I'm not directly familiar with Gio, and I'm not really sure what the workflow is. Why is go get followed by a direct call to the installed binary not usable? In that issue, it seemed like the main concerns were that linked binaries aren't cached (#33468) and that go install might not write to a directory in PATH. Those seem like issues that can be addressed separately.

Great, let's implement the replacement before killing off go run outside modules. I don't know if that means fixing #30515, fixing the go install PATH issue, or something else entirely.

P.S. In case you think I'm making too big a deal out of go run versus go install and PATH: Because of Windows' powerful syscall interface for using native DLLs without Cgo, go run builds and runs Gio programs with only Go installed! (There is a runtime dependency on the non-system ANGLE DLLs, but that can be avoided with a future Direct3D backend.)
Additionally, PATH is most alien and difficult to set for Windows users.
The point is, even setting up PATH is a non-trivial increase in "getting started" complexity.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

It didn't occur to me before, but CL 198778 also breaks the gogio tool outside a module:

$ gogio -target js gioui.org/apps/hello
gogio: go list -f {{.ImportPath}} gioui.org/apps/hello failed: can't load package: package gioui.org/apps/hello: cannot look up module for package "gioui.org/apps/hello": working directory is not part of a module

The gogio tool uses go list which is not covered by @bcmills' #30515 suggestion :(

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 4, 2019

Yeah, go list almost certainly should not be allowed to resolve packages when outside of a module.

go list -f {{.ImportPath}} in particular is fairly benign, but outside of a module it only succeeds when it is the identity function — so it's also not a particularly interesting example.

But it would be pretty bad if, say, you ran go list -f '{{with .Module}}{{.Path}}{{end}} example.com/apps/hello' and got back example.com/apps, and then the next time you ran a go command the package was in a different module because the owner had published a commit splitting example.com/apps/hello out into a separate module.

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 4, 2019

Ok, but for the second and subsequent runs, go run only has to do ~1 query to the proxy which is likely to have the result cached, right?

That's assuming that the proxy has the module at all. For untagged, non-public modules, that query may involve fetching the head of the default branch of the repo and checking its go.mod file to see whether the path matches.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

If you're ok with go get outside of a module, can we allow go run, go list and so on outside of a module, but using only the module cache, never the network? Since the module cache is consistent, you won't run into problems where go list will say something that is incorrect when you run the next go command. Your builds are reproducible, at least relative to your module cache. Finally, the commands are fast because they don't access the network.

Quick start instructions for Gio programs will change to

$ go get gioui.org/apps/hello
$ go run gioui.org/apps/hello

and to

$ go get gioui.org/cmd/gogio gioui.org/apps/hello
$ go run gioui.org/cmd/gogio -target android gioui.org/apps/hello

for building an Android APK from a Gio program.

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

Let me point out that it's not always slow. With a warm cache, my experiment above measures the difference in the hundreds of milliseconds. Furthermore, the output "go: finding blah go: downloading blah go: extracting blah" is a clear hint to why a cold go run is slow.

This experiment only measures the fast case: the proxy already has the latest version of a project that has a go.mod file, and that version is cached locally. We're most concerned about the slow path where the proxy hasn't cached @latest: it might actually have that version, but it has to partially fetch the repo in order to verify there's nothing newer. If the project does not have a go.mod file, we might have to do that many times.

But the critical time is when a newcomer evaluates Gio. I'd like for their experience to be as painfree as possible and being able to run a Gio program with a simple go run without the PATH hassle is invaluable.

I definitely sympathize with this, but I think it's important for the "first time setup" instructions to be close to the "everyday best practice". Running a build tool at the latest version makes me nervous: I'd be scared to put that in a build script or CI. As a user, I want to be able to list versions of my dependencies (both libraries and tools), and from that, I want reproducible output. So I'm not convinced that running go run outside of a module as part of a build workflow for a larger project is something we should encourage.

Running these commands within a module seems fine of course, since the version will be recorded and the result will be reproducible. We know there are a number of issues around managing tool dependencies (#25922), and we're exploring some ways to fix that (#34506).

Great, let's implement the replacement before killing off go run outside modules. I don't know if that means fixing #30515, fixing the go install PATH issue, or something else entirely.

I'm happy to hold CL 198778 back for discussion for a little while, but we'd like to get it in for this release, and it should go in before the freeze.

#30515 seems viable if it's accompanied by a better cache eviction policy, but it probably won't go into 1.14.

I don't know what can be done to fix PATH though. It's admittedly confusing for new developers, but we do expect new developers to set that up when they install Go. I'd be surprised if we have a solution for that in 1.14, other than improved documentation.

@ianthehat

This comment has been minimized.

Copy link

commented Oct 4, 2019

once #23439 is fixed you can instead express the quickstart instructions as:

$ go get gioui.org/apps/hello && $(go env GOBIN)/hello

or right now you could do

$ GOBIN=$PWD go get gioui.org/apps/hello && ./hello

I guess we could also warn you if go get is going to write a binary to a place that is not on your PATH, but I think that would probably cause more confusion than it saved.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

Let me point out that it's not always slow. With a warm cache, my experiment above measures the difference in the hundreds of milliseconds. Furthermore, the output "go: finding blah go: downloading blah go: extracting blah" is a clear hint to why a cold go run is slow.

This experiment only measures the fast case: the proxy already has the latest version of a project that has a go.mod file, and that version is cached locally. We're most concerned about the slow path where the proxy hasn't cached @latest: it might actually have that version, but it has to partially fetch the repo in order to verify there's nothing newer. If the project does not have a go.mod file, we might have to do that many times.

go commands outside modules could be restricted to only work with projects with go.mod files. But if you insist that even the first invocation of go run, go install etc. must be fast, I think my module-cache-only proposal above is a solution.

But the critical time is when a newcomer evaluates Gio. I'd like for their experience to be as painfree as possible and being able to run a Gio program with a simple go run without the PATH hassle is invaluable.

I definitely sympathize with this, but I think it's important for the "first time setup" instructions to be close to the "everyday best practice". Running a build tool at the latest version makes me nervous: I'd be scared to put that in a build script or CI. As a user, I want to be able to list versions of my dependencies (both libraries and tools), and from that, I want reproducible output. So I'm not convinced that running go run outside of a module as part of a build workflow for a larger project is something we should encourage.

The difference between not encourage and flat out disabling functionality is significant. And again, please see my proposal above that won't use an arbitrary latest version, but rather the version in your modcache.

Great, let's implement the replacement before killing off go run outside modules. I don't know if that means fixing #30515, fixing the go install PATH issue, or something else entirely.

I'm happy to hold CL 198778 back for discussion for a little while, but we'd like to get it in for this release, and it should go in before the freeze.

#30515 seems viable if it's accompanied by a better cache eviction policy, but it probably won't go into 1.14.

And #30515 is about go install, not (yet) go run. Furthermore, -g is proposed to

As part of #34506, I'm proposing we add a -g flag (global mode) which would cause the go command to behave as if it were outside a module

Which seems to contradict CL 198778 that disables anything else than go get outside a module.

Let's say #34506 brings back the functionality of go run and friends, as long as you add -g. Then we'll have go run working fine (but slow) in Go 1.13, disabled in Go 1.14, and then working again in Go 1.1x (with a -g).

I don't know what can be done to fix PATH though. It's admittedly confusing for new developers, but we do expect new developers to set that up when they install Go. I'd be surprised if we have a solution for that in 1.14, other than improved documentation.

If you expect a certain PATH, why isn't the Go installer doing it? And why expect developers to set up PATH when you added go env -w because "Setting environment variables for go command configuration
is too difficult and system-specific." (#30411)?

That is very frustrating. You're removing functionality in Go 1.14 while leaving the (potential) replacements for Go 1.15 or perhaps later.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

once #23439 is fixed you can instead express the quickstart instructions as:

$ go get gioui.org/apps/hello && $(go env GOBIN)/hello

This doesn't work in Windows' cmd.exe.

or right now you could do

$ GOBIN=$PWD go get gioui.org/apps/hello && ./hello

This also doesn't work in cmd.exe.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

If you're ok with go get outside of a module, can we allow go run, go list and so on outside of a module, but using only the module cache, never the network? Since the module cache is consistent, you won't run into problems where go list will say something that is incorrect when you run the next go command. Your builds are reproducible, at least relative to your module cache. Finally, the commands are fast because they don't access the network.

Quick start instructions for Gio programs will change to

$ go get gioui.org/apps/hello
$ go run gioui.org/apps/hello

and to

$ go get gioui.org/cmd/gogio gioui.org/apps/hello
$ go run gioui.org/cmd/gogio -target android gioui.org/apps/hello

for building an Android APK from a Gio program.

@bcmills responded to disabling go doc with:

Maybe..?
But see also https://golang.org/issue/33710. It might be reasonable for 'go doc' to resolve the module once, and to otherwise display the documentation for the latest version that happens to be in the module cache.

My proposal is a generalization of that suggestion.

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2019

I'm about to go offline, so I won't respond to everything right now. However, I do want to address your proposal, which I haven't replied to yet:

If you're ok with go get outside of a module, can we allow go run, go list and so on outside of a module, but using only the module cache, never the network? Since the module cache is consistent, you won't run into problems where go list will say something that is incorrect when you run the next go command. Your builds are reproducible, at least relative to your module cache.

I think this what Bryan was getting at in #32027 (comment).

This sounds okay to me, and it was the option I voted for (not that I think emoji straw polls are a good way to make technical decisions). I'd even go further and say that if a module is not in the cache at all, we should fetch the latest version of it; it will be there next time, so the build will be (mostly?) reproducible.

The downside of this is that if a user has old stuff in their cache, they'll get old results. There's little visibility into the cache, and there's no way to update all the modules in the cache (like go get -u all in GOPATH mode). This is probably fine for go doc, go list, and probably most small examples.

Does this solve your problem though? It means that if someone tried out gogio a year ago and is trying it out again with go run outside of a module, they won't see new features.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 5, 2019

If you're ok with go get outside of a module, can we allow go run, go list and so on outside of a module, but using only the module cache, never the network? Since the module cache is consistent, you won't run into problems where go list will say something that is incorrect when you run the next go command. Your builds are reproducible, at least relative to your module cache.

I think this what Bryan was getting at in #32027 (comment).

Yes, I see that now. I'm sorry I missed it.

This sounds okay to me, and it was the option I voted for (not that I think emoji straw polls are a good way to make technical decisions). I'd even go further and say that if a module is not in the cache at all, we should fetch the latest version of it; it will be there next time, so the build will be (mostly?) reproducible.

Even better since it eliminates an error mode (missing module). It won't matter much for me, because I will add an extra go get ...@latest step to my instructions.

The downside of this is that if a user has old stuff in their cache, they'll get old results. There's little visibility into the cache, and there's no way to update all the modules in the cache (like go get -u all in GOPATH mode). This is probably fine for go doc, go list, and probably most small examples.

As long as go get gioui.org/apps/hello@latest updates the cache for that particular module, I don't see a problem here.

Does this solve your problem though?

Absolutely!

It means that if someone tried out gogio a year ago and is trying it out again with go run outside of a module, they won't see new features.

This is no different than someone running go install gioui.org/cmd/gogio to their PATH a year ago and then using gogio from then on, never updating it.

Sure, fetching the latest version is convenient while Gio is v0. As soon as Gio approaches or reaches v1, using whatever is in the cache is the better choice. It's fast, reproducible, and consistent.

In other words, I never cared for go run and friends fetching the latest version (it's a dubious choice anyway) outside modules. I just care that they continue to work.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 5, 2019

I'm about to go offline, so I won't respond to everything right now. However, I do want to address your proposal, which I haven't replied to yet:

If you're ok with go get outside of a module, can we allow go run, go list and so on outside of a module, but using only the module cache, never the network? Since the module cache is consistent, you won't run into problems where go list will say something that is incorrect when you run the next go command. Your builds are reproducible, at least relative to your module cache.

I think this what Bryan was getting at in #32027 (comment).

This sounds okay to me, and it was the option I voted for (not that I think emoji straw polls are a good way to make technical decisions). I'd even go further and say that if a module is not in the cache at all, we should fetch the latest version of it; it will be there next time, so the build will be (mostly?) reproducible.

Thinking more about it, I think this is the correct behaviour, because users who don't care about or want modules likely won't even be affected when GO111MODULE=on becomes the default. Their dependencies will live in the module cache and not in GOPATH, but otherwise all their go subcommands work as usual.

Whereas with CL 198778, the users will be met with "working directory is not part of a module". What does that even mean to users unaware or ignorant about modules?

@mvdan

This comment has been minimized.

Copy link
Member

commented Oct 5, 2019

It means that if someone tried out gogio a year ago and is trying it out again with go run outside of a module, they won't see new features.

This is no different than someone running go install gioui.org/cmd/gogio to their PATH a year ago and then using gogio from then on, never updating it.

I was undecided on this topic, but Elias's point here convinced me in favor of using what's in the cache. It's always possible to query the version being used for a module, or updating it via go get.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 5, 2019

Furthermore, I believe the use-the-module-cache-outside-module behaviour fixes #33518 as a side effect. Nice.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 6, 2019

I don't have much experience with CI, but wouldn't module-cache go commands enable CI using only the module proxy and a Go installation:

$ go get module@<version>
$ go test module/...

I think that's quite neat to avoid the dependencies on particular source control tools and just use modules directly.

Or perhaps that is already possible in another way?

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 7, 2019

@mvdan

It's always possible to query the version being used for a module, or updating it via go get.

If we were to use the latest version in the module cache, then the module cache would become effectively equivalent to GOPATH today — with the corresponding mysteries as to why a given path that worked (or did not work) yesterday is not working (or working) today, and corresponding user confusion about bugs they've observed that have already been fixed upstream.

Yes, it is always possible to query the version being used: but that version is only stable if the user has a go.mod file in which to record it.

Besides, empirically users do not often think to run go list -m. To some extent that is an education problem, but to some extent it is also a reality that we must design for.

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 7, 2019

@eliasnaur

@bcmills responded to disabling go doc with:

[…]

My proposal is a generalization of that suggestion.

Giving it some further thought:

  1. The module cache should really be a cache. If a command works with an empty cache, the same command should work in the same way with a pre-populated cache. That implies that go doc should, by default, not resolve packages outside the dependencies of the main module.

  2. For the “global documentation” use-case, it probably makes sense for go doc to accept the same “global” flag as for #30515. That should cause it to accept a slower (and potentially unstable) version-resolution path: invoking go doc on a package from a module you aren't even using seems like an exceptional case.

@thepudds

This comment has been minimized.

Copy link

commented Oct 7, 2019

@bcmills

Besides, empirically users do not often think to run go list -m. To some extent that is an education problem, but to some extent it is also a reality that we must design for.

Quick side comment on your side comment (sorry):

I suspect usage of go list -m would increase a fair amount if it was spelled go mod list. (That ship has likely sailed, but if Go 1.13 could drop go get -m, then maybe Go 1.14 could revisit go list -m given modules are still officially opt-in, though that would admittedly be a bigger change than the 1.13 change).

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 7, 2019

It's always possible to query the version being used for a module, or updating it via go get.

If we were to use the latest version in the module cache, then the module cache would become effectively equivalent to GOPATH today — with the corresponding mysteries as to why a given path that worked (or did not work) yesterday is not working (or working) today, and corresponding user confusion about bugs they've observed that have already been fixed upstream.

Can you elaborate on the mysteries you expect to see?

It's true that like GOPATH, go run gioui.org/cmd/gogio would run whatever version of gogio is in the module cache, perhaps fetching the newest if there is no version in the cache.

Unlike GOPATH, however, the go.mod of module gioui.org/cmd/gogio is then used for reproducible dependencies¹. So unless the user happened to go get a broken version of gogio, it will work, and as the author (in this case me) intended.

¹ To avoid the GOPATH mess regardless of GO111MODULE, go run, go list, etc. could refuse to work with packages that don't belong to a module.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 7, 2019

It's always possible to query the version being used for a module, or updating it via go get.

If we were to use the latest version in the module cache, then the module cache would become effectively equivalent to GOPATH today — with the corresponding mysteries as to why a given path that worked (or did not work) yesterday is not working (or working) today, and corresponding user confusion about bugs they've observed that have already been fixed upstream.

Can you elaborate on the mysteries you expect to see?

It's true that like GOPATH, go run gioui.org/cmd/gogio would run whatever version of gogio is in the module cache, perhaps fetching the newest if there is no version in the cache.

Unlike GOPATH, however, the go.mod of module gioui.org/cmd/gogio is then used for reproducible dependencies*. So unless the user happened to go get a broken version of gogio, it will work, and as the author (in this case me) intended.

And to re-iterate my argument to @jayconrod: go install (or go get) of a main package has the same problem of unreproducible versions: when I invoke gogio, the system runs whatever version that happened to be "current" at the time of building it. That applies even if the current module lists gioui.org/cmd/gogio with a particular version as a dependency.

Furthermore, querying the version of the gogio binary is possible but isn't obvious (I don't know it).

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 7, 2019

go install (or go get) of a main package has the same problem of unreproducible versions:

Go is a compiled language. Instructions for how to use a tool should generally include an explicit go get or go install step, and that is not going to change as part of the switch to module mode.

If you have given the user an explicit go get command to run, then they will know exactly when the installed binary has been updated.

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 7, 2019

It's true that like GOPATH, go run gioui.org/cmd/gogio would run whatever version of gogio is in the module cache, perhaps fetching the newest if there is no version in the cache.

Yes, that is exactly the problem. The module cache should be a cache: it should not affect the semantic meaning of commands. (A command that runs the latest version should always run the latest version, regardless of whether it is cached.)

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 8, 2019

The Go team is sometimes criticized for not paying enough attention to the broader Go community.

In this discussion, they did an exemplary job of patiently, thoroughly and timely addressing every one of my (sometimes long) arguments and concerns. So even though I failed to convince the team, I have no shred of doubt they seriously considered all input.

Thank you, in particular @bcmills and @jayconrod.

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Oct 8, 2019

@eliasnaur Thanks for the kind words! Your feedback has been very helpful, and we always welcome discussion on issues like this. I know you've run into a number of problems with the go tool while developing Gio, and I hope we can fix many of them, if not in 1.14 then hopefully soon after. Ultimately, we're all trying to figure out what's best for our users, and there's not always a clear answer to that question.

Going forward, I'd like to move ahead with CL 198778 (forbidding imports of non-std packages outside of a module). I find @bcmills' argument that the cache should not affect the semantic meaning of commands to be very convincing. While the approach in #32027 (comment) would solve most of the problems we're concerned about, relying on the cache like this means there's a large amount of hidden state that affects outcomes of many commands, and it will be difficult for developers to understand why things go wrong. If it turns out that CL 198778 causes a lot of problems for common workflows during the 1.14 beta, we may reconsider this, but I think we should try this out.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 8, 2019

By all means feel free to go ahead with CL 198778.

I also found the cache argument compelling, and I found a workaround that works on Windows:

$ cd somewhere
$ go mod init not-used.com
$ go run gioui.org/example/hello

The first line feels like busywork, in particular the unused module name, but Bryan's argument does make sense: create somewhere for the go command to record versions into and thereby eliminate any confusion about versions.

gopherbot pushed a commit that referenced this issue Oct 9, 2019
When in module mode outside of any module, 'go build' and most other
commands will now report an error instead of resolving a package path
to a module.

Previously, most commands would attempt to find the latest version of
a module providing the package. This could be very slow if many
packages needed to be resolved this way. Since there is no go.mod file
where module requirements can be saved, it's a repeatedly slow and
confusing experience.

After this change, 'go build' and other commands may still be used
outside of a module on packages in std and source files (.go
arguments) that only import packages in std. Listing any other package
on the command line or importing a package outside std will cause an
error.

'go get' is exempted from the new behavior, since it's expected that
'go get' resolves paths to modules at new versions.

Updates #32027

Change-Id: Ia9d3a3b4ad738ca5423472e17818d62b96a2c959
Reviewed-on: https://go-review.googlesource.com/c/go/+/198778
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot

This comment has been minimized.

Copy link

commented Oct 10, 2019

Change https://golang.org/cl/200297 mentions this issue: cmd/go: forbid module pattern 'all' when outside a module

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.