proposal: cmd/go: provide a portable and simple way to run SDK-like Go tools

[eliasnaur]

The Gio library contains a module (gioui.org/ui) for creating GUI programs in Go. For desktop platforms, running a Gio program is as easy as any other Go program, provided you have a few common system libraries and a C compiler installed:

$ go run gioui.org/apps/hello

Running Gio programs on Android, iOS and in the browser requires packaging several support files and meta data, so the Gio project includes the gogio tool in the `gioui.org/cmd module to automate the tedious work.

Note that the gogio tool is not general purpose: it is intrinsically bound to the gioui.org/ui module. Similar examples are the gobind and gomobile tools from the Gomobile project.

This issue is about providing an easy and portable way to run such support tools.

In #33468 I described a possible solution in terms of go run because go run is very close to what I want. This is the simple and portable one-liner for creating an Android app for a Gio program:

$ go run gioui.org/cmd/gogio -target android gioui.org/apps/gophers

Unfortunately, according to #33468 (comment), #25416 and #25416 (comment), it is almost accidental that go run can run the above one-liner.

go install

@bcmills brings up the usual way to use Go tools: go install and setting up $PATH to run them:

You could give separate instructions for cmd.exe and for Unix-like shells. Or assume that they have their PATH configured appropriately (perhaps by reference to some other document) and tell everyone:

go install gioui.org/cmd/gogio
gio -target android gioui.org/apps/gophers

However, setting up PATH is not portable and not nearly as simple as go run, in particular for Windows users.

Case in point: go env can now set environment variables for the Go tool because (#30411)

Setting environment variables for go command configuration
is too difficult and system-specific.

go build

#25416 (comment) suggests

$ go build -o /some/dir/gogio gioui.org/cmd/gogio
$ /some/dir/gogio ...

which doesn't depend on the environment, but still more awkward than just go run.

(Some) Solutions

• Let go run be the way to conveniently run Go commands without fiddling with the environment. cmd/go: speed up 'go run' by caching binaries #33468 is about speeding up go run so that it is (nearly) as fast as running a go install'ed binary. go run is also the only way to get reproducible builds; see proposal: cmd/go: provide a portable and simple way to run SDK-like Go tools #33518 (comment).
• Set up the user's environment when installing Go so that go install'ed programs are guaranteed to be in PATH.
• As a variant to the above, provide a portable and simple way to run a go install'ed binary that doesn't depend on the environment.

[eliasnaur]

In #33468 (comment), @bcmills brings up an interesting point:

When working within a module in module mode, go run should produce a reproducible result, not always upgrade to the latest version.

I agree that within a module the result should be reproducible which is the reason go run records the version of the command in a current module's go.mod.

However, that means that within a module, go running gio is superior to running a previously go install'ed version, just because the tool version is reproducible with go run, but not with the binary that happens to be in GOBIN.

So go install is not always the correct choice, even if we assume PATH is correctly set up.

[mvdan]

As a variant to the above, provide a portable and simple way to run a go install'ed binary that doesn't depend on the environment.

It is possible to get the path to the installed binary in a portable way, with something like:

go list -f {{.Target}} some/main/pkg

Note that this will still resolve the version and do extra work outside a module, so it's not instantaneous.

[eliasnaur]

It is possible to get the path to the installed binary in a portable way, with something like:

go list -f {{.Target}} some/main/pkg

Note that this will still resolve the version and do extra work outside a module, so it's not instantaneous.

Thank you. My current thinking is that the speed of go run outside a module is not that important; most actual uses will be from inside a module.

[tv42]

Unfortunately, according to #33468 (comment), #25416 and #25416 (comment), it is almost accidental that go run can run the above one-liner.

It seems like your use cases for running this tool would always be within a module, though -- it's only relevant when developing programs using your giou.org/ui module. And for that, go run (or go build blah && ./blah, for caching the executable) seem just right.

[bcmills]

CC @myitcv @rogpeppe @thepudds; see also #31173.

[rogpeppe]

ISTM that #31173 would solve this reasonably, particularly if combined with an idea from @ianthehat of being able to pass a -modfile argument to the go command to ask it to use a different go.mod file .

[myitcv]

A random list of points to add to the conversation, followed by a question on how we should move forward:

• cmd/go: offer a consistent "global install" command #30515 (comment) represents the most recent discussion about this from a golang-tools perspective
• in the golang-tools discussions, it was generally agreed/observed that the "within a module" use case is one part of the problem/solution space: being able to install/run "globally" (i.e. "outside of a module") is also a problem that needs solving. Indeed @ianthehat sort to define this more precisely:

The concrete use cases people have suggested so far are:

• go code generators (often from go generate lines) which must be versioned at possibly per invocation granularity
• code analysis tools (the kinds of things we are trying to merge into gopls, an also gopls itself of course) which may need to be versioned per project
• System wide installs of binaries, for the kinds of instructions people put on their wiki

• per previous threads with @bcmills, @rsc and others, it's not clear to me that these changes have to be made to cmd/go. @rsc made the point (somewhere I can't now recall) that trying to load too much into cmd/go effectively has costs associated with it, costs that aren't necessarily outweighed by the cost of putting this functionality in a different command. I have definitely come to appreciate this point more, although the attraction of having everything in cmd/go remains
• if the changes are not made as part of cmd/go, then the command within which the functionality sits must ultimately be distributed with Go (a la godoc in the old days): the overhead of installing a command to install/run other commands is too high. This obviously doesn't preclude developing something in x/tools as a starting point
• gobin is an experiment that covers the cases of "within a module" and "outside a module". It has been largely successful in acting as a hybrid run/install command that sits atop cmd/go. gobin respects replace directives (and could respect exclude directives, this is simply not implemented)
• gobin seeks to optimise the cost of running main packages. It is faster than go run, but currently nowhere near as fast as running something from PATH. However it does obviate the need for setting PATH which is a big win. I'm sure there is more scope for optimisation here, particularly if we can start to build on assumptions about the pristineness of the module cache (arguably if the user messes with the module cache it's their own problem)
• one conclusion we (read @myitcv, @rogpeppe and @mvdan) have (almost) reached with gobin to date is that in the "within a module" scenario, polluting the main module with tool dependencies is not good. proposal: switch to using nested modules for -m mode resolution myitcv/gobin#81 proposes the use of nested modules for this reason
• whatever and wherever the solution lies:
  • it must be able to support further replace and exclude directives, and build flags/constraints
  • it must be possible to cross compile
  • version information for all modules (including the main package's module) must be available in the resulting binary

Next steps

This issue has been rumbling in the background for some time now and it seems, to my mind, worthy of tackling given the plethora of problems/issues/common stumbling blocks it would help to avoid (arguments along similar lines to why go env -w is preferable to people setting environment variables). Continuing some sort of gobin-like experiment in x/tools feels like a good next step, regardless of whether that work is adopted into cmd/go or not.

@rsc, @bcmills, @ianthehat - what do you see as the next steps? Any thoughts on how best to proceed from here?