Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: 'go run' should run executables in module mode outside a module #42088

Open
eliasnaur opened this issue Oct 20, 2020 · 15 comments
Open

Comments

@eliasnaur
Copy link
Contributor

@eliasnaur eliasnaur commented Oct 20, 2020

#40276 implements go install path@version for installing a Go binary outside a module. I propose the same support added to go run, with equivalent behavior. That is,

$ go run gioui.org/cmd/gogio@d5bdf0756a5a

should build and run the gioui.org/cmd/gogio program at version d5bdf0756a5a

Why isn't go install enough for my uses? Consider a README describing how to build and use an auxiliary Go program:

To build the XYZ Android app you need to use the gogio tool:

$ export PATH=$PATH:$GOPATH/bin
$ go install gioui.org/cmd/gogio@d5bdf0756a5a
$ gogio -target android example.com/cmd/xyz

The README has several issues:

  1. go install'ing the binary is reproducible, but running it isn't. For example, the user may have an old gogio in their PATH already, and fail to run the instructed go install. They may remember, but later install a different version of gogio.
  2. go install polutes the user's GOPATH/bin, and PATH if it includes GOPATH/bin.
  3. If GOPATH is not set, the README has to contain hardcoded paths (~/go/bin).

In contrast, with go run path@version support, the README is reduced to just:

To build the XYZ Android app you need to use the gogio tool:

$ go run gioui.org/cmd/gogio@d5bdf0756a5a -target android example.com/cmd/xyz

@myitcv
Copy link
Member

@myitcv myitcv commented Oct 20, 2020

cc @bcmills @jayconrod @ianthehat given previous discussion on this

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 20, 2020

Just to consider as an alternative:

example.com$ GOBIN=$(pwd) go install gioui.org/cmd/gogio@d5bdf0756a5a

example.com$ ./gogio
gogio: specify a package

That has its own problems (namely, $(pwd) is not portable), but it makes clear that for subsequent invocations the user should re-invoke the compiled binary rather than re-resolving and recompiling from upstream.

If we allowed go build to accept the @version syntax, then it could be made portable:

$ go build -o ./gogio.exe gioui.org/cmd/gogio@d5bdf0756a5a
$ ./gogio.exe
@ianthehat
Copy link

@ianthehat ianthehat commented Oct 20, 2020

The go install thing is not portable, because of the varying executable filename, and I think adding this support to go build would be a mistake, I think would rather see us add a -o to go install if that is where we are going.
One of the use cases I find more interesting to talk about is writing reproducible generate lines

  //go:generate go run golang.org/x/tools/cmd/stringer@v1.2.3 -type=Pill
@eliasnaur
Copy link
Contributor Author

@eliasnaur eliasnaur commented Oct 20, 2020

The go install thing is not portable, because of the varying executable filename, and I think adding this support to go build would be a mistake, I think would rather see us add a -o to go install if that is where we are going.
One of the use cases I find more interesting to talk about is writing reproducible generate lines

  //go:generate go run golang.org/x/tools/cmd/stringer@v1.2.3 -type=Pill

What if you have several such invocations of the same tool, each with a duplicate @version modifier?

I omitted the go:generate use-case from this proposal because I think it's better to have such dependencies recorded in the go.mod file by using the idiom of _-importing the tool in a tools.go file.

@ianthehat
Copy link

@ianthehat ianthehat commented Oct 20, 2020

Why would duplicate @version modifiers be a problem?

The _ import is bad because it causes the tool to modify the version selection of your main application, and also to pull things into your dependency graph that your binary does not actually depend on. It also causes the tools to affect each other, rather than being run with the versions the author has tested with. In general it is an acceptable hack while we don't have a better answer, but not a long term acceptable solution in my opinion.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Oct 20, 2020

cc @matloob as well.

Personally I'm in favor of go run supporting this with the same semantics and restrictions as #40276. This came up a few times in the discussion of #40276. Let's set aside go build.

The only technical barrier is that we'd need to cache linked binaries with a different eviction policy than compiled packages. Currently, we don't cache linked binaries at all.

Other than that, it's just a question of CLI design and impact to the ecosystem.

@eliasnaur
Copy link
Contributor Author

@eliasnaur eliasnaur commented Oct 20, 2020

Why would duplicate @version modifiers be a problem?

I was referring to having to update all versions if you want a newer version of the tool. Maybe that's not too bad.

The _ import is bad because it causes the tool to modify the version selection of your main application, and also to pull things into your dependency graph that your binary does not actually depend on. It also causes the tools to affect each other, rather than being run with the versions the author has tested with. In general it is an acceptable hack while we don't have a better answer, but not a long term acceptable solution in my opinion.

Good points. In an ideal world, go:generate dependencies should be recorded in go.mod, but the downsides you point out seem to outweigh the advantages.

@ianthehat
Copy link

@ianthehat ianthehat commented Oct 20, 2020

I think if there was a reasonably common pattern of //go:generate go run package@version args... we could easily write tooling to maintain those lines separately if it turns out to be needed.

@myitcv
Copy link
Member

@myitcv myitcv commented Oct 20, 2020

In general it is an acceptable hack while we don't have a better answer, but not a long term acceptable solution in my opinion.

The argument about dependencies being varied to versions not tested by the author might equally apply to any third party library you are using, so that doesn't sway the argument for me. The fact that, under such a scheme, we would have multiple sites at which to maintain tool versions is a real problem however. Because use of these tools is by no means limited to go:generate directives, e.g. scripts.

@ianthehat
Copy link

@ianthehat ianthehat commented Oct 20, 2020

There is a significant difference between a library you want to include in your code that shares dependancies with other libraries in your graph, and a binary you want to run exactly as the author intended it to be run. The module story has been very focused on the former (for good reason) and the existing approaches have not left people happy with the results for the latter, which is one of the reasons we have talked about these kinds of changes.

I am mostly uninterested in scripts or makefiles because I think they already have all the tools they need, the path and install hacks are good enough for those cases, it might not be beautiful and need some extra lines, but I don't find that a big deal.

@peebs
Copy link

@peebs peebs commented Oct 22, 2020

The argument about dependencies being varied to versions not tested by the author might equally apply to any third party library you are using, so that doesn't sway the argument for me. The fact that, under such a scheme, we would have multiple sites at which to maintain tool versions is a real problem however. Because use of these tools is by no means limited to go:generate directives, e.g. scripts.

I also see a significant difference between depending on libraries and installing a released and versioned binary.

Depending on a library means to me taking ownership of how the library fits into your dep graph and sufficiently testing your code to be confident in the potentially unique dep graph.

I never want to modify a released binary's deps based on independent local code i'm developing. If I need to modify the deps of a released binary i'm either in the process of forking or contributing upsteam to the project directly. I want control over the deps of code I am currently authoring only. If I am building a main package outside of my current project then I want a universally reproducible artifact as much as possible.

@eliasnaur
Copy link
Contributor Author

@eliasnaur eliasnaur commented Oct 26, 2020

This may be a duplicate of #33518

@powerman
Copy link

@powerman powerman commented Oct 27, 2020

The upside of using _ imports for tools is extra dependabot notifications when it's time to update your linter or code generation tools.

I'm afraid //go:generate go run package@version … will open the door for extra inconsistencies and too much flexibility - I don't really like the idea of using multiple versions of same tool in the single project/module and/or get extra headache to keep these versions in sync. So, running tool version defined in go.mod by default is probably better way to go.

Also, the whole story about gobin, go install and now go run for tools have one big downside: not all tools are written in Go, and it's better to have more general (non-Go-specific) way to express tools dependencies and run required tool version. But this is probably offtopic here.

@eliasnaur eliasnaur changed the title cmd/go: 'go run' should run executables in module mode outside a module proposal: cmd/go: 'go run' should run executables in module mode outside a module Oct 29, 2020
@gopherbot gopherbot added Proposal and removed NeedsDecision labels Oct 29, 2020
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Nov 11, 2020
@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Nov 18, 2020

We discussed this for a bit in the golang-tools session this week.

Personally, I'm in favor of this proposal, and it sounded like most others on the call were as well. The main point was that go run pkg@version should build the same binary that go install pkg@version would build. The only differences would be that go run executes the binary instead of writing it to GOBIN. There would be no differences in the semantics used to resolve versions or restrictions on replace directives.

Before we move forward though, I think two things need to be resolved:

  1. We need to have firm agreement on what will be done with replace for go install pkg@version and go run pkg@version. This was discussed at length in #40276. In 1.16, go install will report an error if any replace directive is present in the go.mod file of the module providing the named packages. This eliminates ambiguity, and it leaves the door open for other behaviors. Our experience in 1.16 will inform what we do later on: keep the new behavior, ignore replace directives, or apply some replace directives but not others.
  2. We'll need to change the build cache eviction algorithm. Binaries linked with go run should be cached for a little while so repeated go run commands don't need to re-link.
@rsc rsc moved this from Incoming to Active in Proposals Nov 18, 2020
@mvdan
Copy link
Member

@mvdan mvdan commented Nov 18, 2020

Our experience in 1.16 will inform what we do later on: keep the new behavior, ignore replace directives, or apply some replace directives but not others.

I agree with pretty much everything you said, but I should also say that go run pkg@version with the current "no replaces" semantics would still be very useful with a lot of modules. So even if we can't figure out how to advance what to do with replace directives, I still think it's worth to teach go run this new behavior in 1.17.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.