Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: offer a consistent "global install" command #30515

Closed
mvdan opened this issue Mar 1, 2019 · 115 comments
Closed

cmd/go: offer a consistent "global install" command #30515

mvdan opened this issue Mar 1, 2019 · 115 comments

Comments

@mvdan
Copy link
Member

@mvdan mvdan commented Mar 1, 2019

As of 1.12, one can run the following from outside a Go module:

GO111MODULE=on go get foo.com/cmd/bar

The same mechanism can be used to download a specific version instead of @latest, such as @v1.2.3.

We can even emulate very similar behavior in Go 1.11, with a one-liner like:

cd $(mktemp -d); go mod init tmp; go get foo.com/cmd/bar

1.13 will likely make GO111MODULE=on a default, so it's likely that project READMEs will be able to justs tell users to run go get foo.com/cmd/bar on Go 1.13 and later.

However, this has a problem - if the user runs the command from within a Go module, the command will add clutter to the module's go.mod and go.sum files. The binary will also be installed as usual, so the user might not even notice the unintended consequences until much later.

This is a fairly common point of confusion among Go developers, particularly those new to modules. Now that most Go projects are modules, the chances of one running cmd/go commands within a module are quite high.

What we need is a "global install" command, which will ignore the current module. For example, imagine a go get -global, to ensure backwards compatibility. Or perhaps even repurposing go install to always mean global installs, since go get can be used for non-global ones.

I think we should make a decision and implement it before the 1.13 release, so that READMEs can finally drop the problematic go get -u foo.com/cmd/bar line. We can almost do it now, minus this confusing edge case with $PWD.

/cc @bcmills @rsc @myitcv @rogpeppe @thepudds

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 1, 2019

@myitcv
Copy link
Member

@myitcv myitcv commented Mar 1, 2019

cc @ianthehat given the recent golang-tools related conversations too.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 1, 2019

See previously #27643, #28400, #28156.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Mar 1, 2019

I think we should repurpose go install for this when GO111MODULE=on. Since the build cache is required now, it doesn't seem like there's much use for go install anymore, especially when the target is a main package not in the build list.

@mvdan
Copy link
Member Author

@mvdan mvdan commented Mar 1, 2019

I agree that repurposing go install for this would be ideal in the long run, as right now go get and go install seem to overlap a bit too much. But we'd need to document and announce the new behavior of go install well, as this would be a breaking change for some users.

@ianthehat
Copy link

@ianthehat ianthehat commented Mar 1, 2019

I think it is fairly easy to make a case for
go install module/binary@version
doing exactly what the user would expect (not modifying your local go.mod, and reliably building that specific version no matter what the local directory/state is)
I don't think it would even be a breaking change if that is all we did (at the moment specifying a version would not be valid)

The harder things are at the edges, things like are a way of doing that work and then running the resulting binary (had to do will with go run, and I am not sure we should even try), or wether the behaviour of go install when not given a version should be changed to avoid any confusion.

@thepudds
Copy link

@thepudds thepudds commented Mar 1, 2019

From what I have observed as well as for my personal usage, this would be very useful regardless of how it is spelled (go install, go get -global, ...).

If it is implemented, I think it would be important to use the remote go.mod if one exists, including respecting any replace and exclude directives in that remote go.mod. In practice, authors will need to use replace or exclude in some cases, even though hopefully it will not be the common case. One of the main goals of modules is 100% reproducible builds, and that seems especially important when publishing a binary.

@thepudds
Copy link

@thepudds thepudds commented Mar 1, 2019

At this point, personally I would vote for go install.

Regarding this:

I think it is fairly easy to make a case for
go install module/binary@version
...
The harder things are at the edges, things like ... whether the behaviour of go install when not given a version should be changed to avoid any confusion.

If go install module/binary@version takes on the behavior suggested by this proposal, then I think go install module/binary would need to be equivalent to go install module/binary@latest (including for consistency with go get foo vs. go get foo@latest behavior).

A related question -- should go get some/cmd and go get some/cmd@v1.2.3 stop installing binaries? That would be a more radical change and would reduce overlapping behavior. One could argue that might be desirable if starting from scratch, but given the number of install instructions out there that say go get some/cmd or go get -u some/cmd, I suspect it would be preferable to let go get keep installing binaries.

@bufdev
Copy link

@bufdev bufdev commented Mar 1, 2019

Just passing by. +1 to having this. My current workaround as of 1.12:

$(STATICCHECK):
$(eval STATICCHECK_TMP := $(shell mktemp -d))
@cd $(STATICCHECK_TMP); GOBIN=path/to/bin go get honnef.co/go/tools/cmd/staticcheck@$(STATICCHECK_VERSION)
@rm -rf $(STATICCHECK_TMP)

@peebs
Copy link

@peebs peebs commented Mar 1, 2019

I also support go install being for global binary installs to gopath and go get only for use inside a module path.

Modules brings some breaking tooling changes and it seems like making some more here is the time to do it and worth the cost since this makes the tooling much easier to understand.

@rogpeppe
Copy link
Contributor

@rogpeppe rogpeppe commented Mar 2, 2019

I support this suggestion.

One question: would this re-purposed go install command respect replace and exclude directives in the installed module? I've argued before that it should, and I still think that's the case.

To take one example, it wouldn't be possible to build a correctly working executable for github.com/juju/juju/cmd/juju without respecting replace directives.

@ianthehat
Copy link

@ianthehat ianthehat commented Mar 2, 2019

I think the results should be identical to as if you had checked out the module at the supplied version to a temporary directory, changed to that directory and done go install ., so yes, fully obeying the go.mod with replace and excludes applied, not treating it as a dependancy.

@mvdan
Copy link
Member Author

@mvdan mvdan commented Mar 2, 2019

Something that we might want to be careful about is go install commands which are given local packages, and not remote ones. For example, I presume that go install . or go install ./cmd/foo should still use the current module.

@ianthehat
Copy link

@ianthehat ianthehat commented Mar 2, 2019

Yes, this is one of the reasons why I am dubious about making any changes to non versioned forms, and thus don't agree that an implied @latest is the right choice.
On the other hand, I am slightly worried that the radical change of behavior adding a version causes might be confusing, but I think it's probably ok.

@thepudds
Copy link

@thepudds thepudds commented Mar 2, 2019

@mvdan I suspect I am missing something obvious… But could the rule be to use the go.mod file (including respecting replace and exclude) based on wherever the "package main" is coming from? In other words, use the remote go.mod file if installing a remote command, and use the local go.mod file of the module you are in currently if doing something like go install .?

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Mar 4, 2019

Trying to distill the discussion so far into a proposal:

We would change the behavior of go install [modules]. [modules] is one or more module patterns, as currently interpreted by go get. go get itself would not change. Each module pattern has an optional version query (@latest if no query is specified). Each module pattern will be handled separately.

We change behavior if:

  • go install is run in module mode AND
  • The module pattern being installed is NOT any of the following:
    • Empty (equivalent to .)
    • A local path without a version (e.g., ./cmd/foo, ./...).
    • A metapackage (all, cmd, std).
    • Anything matching packages in the standard library.

If the conditions above are true, go install will:

  • Download the module at the specified version
    • Patterns like @latest or @v1 will be interpreted the same way that go get interprets them.
  • Build packages and executables matching the pattern, treating the downloaded module as the main module.
    • replace and exclude directives will be observed.
    • vendor directories will be used if they would be used normally, e.g., -mod=vendor is set. After #30240 is implemented, vendoring would be used by default (as if the main module were the installed module).
  • Copy packages and executables to their target locations, i.e., the path indicated by go list -f '{{.Target}}'
    • This will normally be $(go env GOPATH)/bin or $(go env GOPATH)/pkg.
  • go install will not modify the go.mod or go.sum files in the module where go install is invoked. Version constraints in the module where go install is invoked will be ignored.

Examples:

  • These commands will not change behavior:
    • Any variant of go get.
    • go install
    • go install cmd
    • go install ./cmd/foo
    • go install ./...
    • go install main.go
  • These commands will change when run in module mode:
    • go install golang.org/x/tools/packages
    • go install golang.org/x/tools/cmd/goimports
    • go install golang.org/x/tools/cmd/goimports@v1.2.3
    • go install golang.org/x/tools/cmd/...
    • go install golang.org/x/tools/cmd/...@v1.2.3
    • go install ./cmd/foo@latest
  • An error message would be printed if go install is run in GOPATH mode with an argument that includes an @ character, same as go get.
@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Mar 4, 2019

I think a key question is whether go install with a non-local path and no version should behave differently than a non-local path with an explicit version. For example, should go install golang.org/x/tools/cmd/goimports work differently than go install golang.org/x/tools/cmd/goimports@latest?

I'd argue the form with no version should be consistent with other module-related commands. For a new user, it would be strange to accidentally run the former command instead of the latter, then see it take your build constraints into account and possibly modify your go.mod file. This would be a change from current behavior. If we require explicit versions, we'd only be adding new behavior.

@mvdan
Copy link
Member Author

@mvdan mvdan commented Mar 4, 2019

I think go install remote.org/pkg should behave just as if @latest had been given. That's more consistent, and like you say, less confusing to users.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 5, 2019

@rogpeppe @ianthehat The trouble with applying replace directives is that it isn't feasible to apply them consistently. We could apply replacements that specify other modules, but not filesystem paths: the replacement module must have its own go.mod file, and as such will not be found within the same module source tree in the module cache.

That means, for example, that we wouldn't be able to fetch the module from a proxy.

The same consideration holds for vendor directories: if we implement #30240, then the go.mod files in the vendor directory will cause essentially all of the source files in that tree to be pruned out of the module cache.

We could, at least in theory, apply the subset of replacements that specify module paths rather than filesystem paths. However, that would produce a third variation on the build configuration: one that doesn't necessarily match the clean-checkout configuration (because it doesn't apply filesystem replacements), but doesn't necessarily match the clean-external-build configuration either (because it does apply non-filesystem replacements).

As another alternative, we could apply module replacements and emit an error of the module has any filesystem replacements. That would produce the same result as a build within the module, but at the cost of rejecting some otherwise-well-formed modules.

I don't think that the benefit of enabling temporary fixes and workarounds offsets the complexity of adding either of those modes of operation. Users have enough trouble understanding module-mode behavior as it is. It seems much simpler to require that modules build successfully using unmodified dependencies, even if that results in a bit of extra work for the maintainers of large modules to upstream their modifications or maintain a complete fork.

@ianthehat
Copy link

@ianthehat ianthehat commented Mar 5, 2019

As I said before, the rule should be the tool is built exactly as if you had fetched it from a proxy, extracted it to a temporary directory, changed to the directory, and typed go install. I think possibly we should also specify the readonly flag, but that's a separate discussion.
This will apply replace directories in a fully consistent way, with no special rules of any kind required.
More importantly, it will attempt to build exactly the same binary that you would build if you checked out the repository, which is a consistency that is far easier to explain.
It may well fail to work if someone has checked in a bad go.mod, but I am fine with that, people should not be checking in go.mod files with replace directories that are not self contained anyway, and if they do then go install module/binary@version will stop working for them.
I have also said that I strongly disagree with #30240, and I agree this is one more way it will cause problems if we do implement it, but it merely causes a case that does not currently work to still not work, it's not really an argument for not doing this.

@rsc
Copy link
Contributor

@rsc rsc commented Mar 5, 2019

"Repurposing" go install seems like a non-starter to me. It has a defined meaning, and that meaning is not "pretend we're not in the current module". We can't break that. Similarly, go get is by design the only command that accepts @version. If go install adds it, everything has to add it (go build, go test, go vet, etc), and that way lies incoherence.

@rsc
Copy link
Contributor

@rsc rsc commented Mar 5, 2019

Half-applying replace directives also seems like a non-starter to me. It would be a new half-way mode that has similar coherence problems.

@mvdan
Copy link
Member Author

@mvdan mvdan commented Mar 5, 2019

"Repurposing" go install seems like a non-starter to me. It has a defined meaning

That's fair enough, but aren't go install and go get practically the same? Now that install also fetches source code, that is.

This is where the idea of slightly changing the meaning of one of them comes from, to keep both commands useful. We could always add a flag like go get -global, but I find that a bit clunky, since installing a Go program on the system is a fairly common need.

@rsc
Copy link
Contributor

@rsc rsc commented Mar 5, 2019

@bcmills and I talked a bit about various "auto-detect" kind of ways to shove this into the go command and didn't come up with anything palatable. I could see adding a short flag to go get to support this, like maybe go get -b for binary or bare, but the first question to answer is what "this" means.

  • Does it mean "run as if in a directory outside any module?"
  • Does it mean "run using the upgrades implied by the current module's go.mod but just don't write any changes back down to go.mod?"
  • Does it mean something else?
@rsc
Copy link
Contributor

@rsc rsc commented Mar 5, 2019

@mvdan Whether they are practically the same doesn't really matter. What matters is that they have established semantics that can't just be redefined.

@mvdan
Copy link
Member Author

@mvdan mvdan commented Mar 5, 2019

Does it mean "run as if in a directory outside any module?"

This is what I'd imagine.

What matters is that they have established semantics that can't just be redefined.

Wouldn't the same have applied to go install or go build suddenly fetching code in module mode? That was hidden behind GO111MODULE=on, but still, it seems to me like it changed the established semantics somewhat. Similar to what's been suggested here, in my mind.

@peebs
Copy link

@peebs peebs commented Jun 26, 2020

Notice that your comments could apply equally to non-main packages. As things stand today, running go test in a module context obeys replace directives in that main module. That is to say, it's possible to release a version of that module where go test passed with replace directives applied, but fails when those replace directives are not applied (and hence an importer of a package from that module might experience problems).

This seems correct to me as is. If I import a package that uses replace statements as a dependency, then failed tests might help alert me to this problem and I may reconsider the import. I certainly wouldn't ever want replace statements from another module to apply in one i'm using. This is a core principal of replace directives. I just categorize installing a global binary as being equivalent to building inside that binaries module.

Note that if you want to build exactly the same binary that someone else tested and built, you also need to set the same build tags and compiler and linker flags that they used, and if the package has any cgo dependencies you need to use the same C toolchain and library versions. If authors of binary tools want that level of control, I would argue that they really ought to package and release binaries, not just tag versions of the source code.

This is a good point and I agree with it. I will say that many prefer pure go over cgo for this reason and more. Also why open the door for less reproducibility just because its not perfect now?

Looking forward to proposal and summary. Thanks all.

@thepudds
Copy link

@thepudds thepudds commented Jun 27, 2020

A few quick additional details that probably should be covered in an actual proposal (with one answer or another):

  1. It would be nice if go get -g foo (or whatever incantation) has the exact same behavior independent of the setting of GO111MODULE, even if GO111MODULE=off. (Part of the point of this issue is to make it easy for authors of commands to put simple install instructions in their README that yield consistent results, without having to think through or write up many permutations).

  2. It seems reasonable to support go get -g mod1/... and go get -g mod1/cmdA mod1/cmdB, but it might be best to not cross the beams in terms of version selection for any shared dependencies with go get -g mod1 mod2. From a principle of least surprise, the simplest might be to error out in that case? (That could be relaxed in the future if there is a compelling use case, but the most common case for -g is probably for following the instructions from a single README; if someone wants to install from multiple modules using -g, they have a simple solution of running multiple go get commands).

  3. Presumably other standard go get flags like -u and -u=patch would be allowed with -g? (-t is more of a corner case, but seems reasonable to allow, including it could make a difference with -g -t -u. -g -d doesn't seem massively useful, but could just let it do what it would do).

In any event, I'm not sure those are the best answers, and probably better to view those more as details to iron out in the proposed behavior...

stblassitude added a commit to stblassitude/gitea that referenced this issue Jun 28, 2020
Prevent `go get` from touching `go.mod` and `go.sum` when executing
global binaries during the build process. Once
golang/go#30515 is fixed, we should is
whatever solution is provided there.

Fixes: go-gitea#12010

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
lafriks pushed a commit to go-gitea/gitea that referenced this issue Jun 28, 2020
Prevent `go get` from touching `go.mod` and `go.sum` when executing
global binaries during the build process. Once
golang/go#30515 is fixed, we should is
whatever solution is provided there.

Fixes: #12010

Co-authored-by: techknowlogick <techknowlogick@gitea.io>

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 16, 2020

Change https://golang.org/cl/243077 mentions this issue: design: add 30515-go-get-b.md

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jul 17, 2020

Hey everyone, I've opened a new proposal issue, #40276, with a design doc at CL 243077.

It'd be best if we could continue the discussion on that CL: Gerrit's threading and versioning will be more readable than a long GitHub issue.

To concentrate discussion, I'd like to close and lock this issue. Before I do that, if you feel the proposal doesn't address an important use case and would like to continue this discussion separate from the proposal, please comment here.

Also note, this will be on the agenda for the next golang-tools discussion on 2020-07-22 at 15:30 UTC, so feel free to join in then as well.

@mvdan
Copy link
Member Author

@mvdan mvdan commented Jul 22, 2020

I helped Jay with the new proposal above, so I agree that we should close this issue in favor of that newer one. This isue was more of a problem statement than a proposal for a specific solution that can be implemented.

@mvdan mvdan closed this Jul 22, 2020
@mvdan mvdan removed this from the Go1.16 milestone Jul 22, 2020
ydelafollye pushed a commit to ydelafollye/gitea that referenced this issue Jul 31, 2020
Prevent `go get` from touching `go.mod` and `go.sum` when executing
global binaries during the build process. Once
golang/go#30515 is fixed, we should is
whatever solution is provided there.

Fixes: go-gitea#12010

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
successgo added a commit to successopen/gitea that referenced this issue Aug 20, 2020
Prevent `go get` from touching `go.mod` and `go.sum` when executing
global binaries during the build process. Once
golang/go#30515 is fixed, we should is
whatever solution is provided there.

Fixes: go-gitea#12010

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
jpalmerpivotal added a commit to cloudfoundry/cli that referenced this issue Oct 8, 2020
Modules prevent the installation of executables using the standard
`go get` syntax. Using `GOMODULE111=off` should fix this until
[#40276](golang/go#40276) is implemented.

See also:
[#27643](golang/go#27643)
[#30515](golang/go#30515)

[#171660042](https://www.pivotaltracker.com/story/show/171660042)

Co-authored-by: Sebastian Vidrio <svidrio@vmware.com>
lisamburns added a commit to cloudfoundry/cli that referenced this issue Oct 21, 2020
Modules prevent the installation of executables using the standard
`go get` syntax. Using `GOMODULE111=off` should fix this until
[#40276](golang/go#40276) is implemented.

See also:
[#27643](golang/go#27643)
[#30515](golang/go#30515)

[#171660042](https://www.pivotaltracker.com/story/show/171660042)

Co-authored-by: Sebastian Vidrio <svidrio@vmware.com>
svengreb added a commit to svengreb/wand that referenced this issue Nov 22, 2020
>>> Go Executable Installation

When installing a Go executable from within a Go module [1] directory
using the `go install` command [2], it is installed into the Go
executable search path that is defined through the `GOBIN` environment
variable [3] and can also be shown and modified using the `go env`
command [4]. Even though the executable gets installed globally, the
`go.mod` file [5] will be updated to include the installed packages
since this is the default behavior of the `go get` command [6]
running in "module mode" [7].

Next to this problem, the installed executable will also overwrite any
executable of the same module/package that was installed already,
but maybe from a different version. Therefore only one version of a
executable can be installed at a time which makes it impossible to work
on different projects that use the same tool but with different
versions.

>>>> History & Future

The local installation of executables built from Go modules/packages has
always been a somewhat controversial point which unfortunately,
partly for historical reasons, does not offer an optimal and
user-friendly solution up to now. The `go` command [8] is a fantastic
toolchain that provides many great features one would expect to be
provided out-of-the-box from a modern and well designed programming
language without the requirement to use a third-party solution:
from compiling code, running unit/integration/benchmark tests, quality
and error analysis, debugging utilities and many more.
Unfortunately the way the `go install` command [2] of Go versions less
or equal to 1.15 handles the installation of an Go module/package
executable is still not optimal.

The general problem of tool dependencies is a long-time known issue/weak
point of the current Go toolchain and is a highly rated change request
from the Go community with discussions like golang/go#30515 [9],
golang/go#25922 [10] and golang/go#27653 [11] to improve this essential
feature, but they've been around for quite a long time without a
solution that works without introducing breaking changes and most users
and the Go team agree on.
Luckily, this topic was finally picked up for the next upcoming Go
release version 1.16 [12] and golang/go#40276 [13] introduces a way to
install executables in module mode outside a module. The release note
preview also already includes details about this change [14] and how
installation of executables from Go modules will be handled in the
future.

>>>> The Workaround

Beside the great news and anticipation about an official solution for
the problem the usage of a workaround is almost inevitable until Go 1.16
is finally released.

The official Go wiki [15] provides a section on "How can I track tool
dependencies for a module?" [16] that describes a workaround that tracks
tool dependencies. It allows to use the Go module logic by using a file
like `tools.go` with a dedicated `tools` build tag that prevents the
included module dependencies to be picked up included for normal
executable builds. This approach works fine for non-main packages,
but CLI tools that are only implemented in the `main` package can not be
imported in such a file.

In order to tackle this problem, a user from the community created
"gobin" [17], an experimental, module-aware command to install/run main
packages.
It allows to install or run main-package commands without "polluting"
the `go.mod` file by default. It downloads modules in version-aware mode
into a binary cache path within the systems cache directory [18].
It prevents problems due to already globally installed executables by
placing each version in its own directory. The decision to use a cache
directory instead of sub-directories within the `GOBIN` path keeps the
system clean.

"gobin" is still in an early development state, but has already received
a lot of positive feedback and is used in many projects.
There are also members of the core Go team that have contributed to the
project and the chance is high that the changes for Go 1.16 were
influenced or partially ported from it.
It is currently the best workaround to...

1. prevent the Go toolchain to pick up the `GOMOD` environment
   variable [4] (see [`go env GOMOD`][4]) that is initialized
   automatically with the path to the `go.mod` file in the current
   working directory.
2. install module/package executables globally without "polluting" the
   `go.mod` file.
3. install module/package executables globally without overriding
   already installed executables of different versions.

See gobin's FAQ page [19] in the repository wiki for more details about
the project.

>>>> The Go Module Caster

To allow to manage the tool dependency problem, wand now uses "gobin"
through a new caster [20]
`go.mod` file and allows to...

1. install `gobin` itself into `GOBIN` (`go env GOBIN` [4]).
2. cast any spell incantation [21] of kind `KindGoModule` [22] by
   installing the executable globally into the dedicated `gobin` cache.

[1]: https://golang.org/ref/mod
[10]: golang/go#25922
[11]: golang/go#27653
[12]: https://github.com/golang/go/milestone/145
[13]: golang/go#40276
[14]: https://tip.golang.org/doc/go1.16#modules
[15]: https://github.com/golang/go/wiki
[16]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
[17]: https://github.com/myitcv/gobin
[18]: https://golang.org/pkg/os/#UserCacheDir
[19]: https://github.com/myitcv/gobin/wiki/FAQ
[2]: https://golang.org/cmd/go#hdr-Compile_and_install_packages_and_dependencies
[20]: https://pkg.go.dev/github.com/svengreb/wand/pkg/cast/gobin#Caster
[21]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#Incantation
[22]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#KindGoModule
[3]: https://golang.org/cmd/go/#hdr-Environment_variables
[4]: https://golang.org/cmd/go/#hdr-Print_Go_environment_information
[5]: https://golang.org/ref/mod#go-mod-file
[6]: https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them
[7]: https://golang.org/ref/mod#mod-commands
[8]: https://golang.org/cmd/go
[9]: golang/go#30515

GH-22
svengreb added a commit to svengreb/wand that referenced this issue Nov 22, 2020
>>> Go Executable Installation

When installing a Go executable from within a Go module [1] directory
using the `go install` command [2], it is installed into the Go
executable search path that is defined through the `GOBIN` environment
variable [3] and can also be shown and modified using the `go env`
command [4]. Even though the executable gets installed globally, the
`go.mod` file [5] will be updated to include the installed packages
since this is the default behavior of the `go get` command [6]
running in "module mode" [7].

Next to this problem, the installed executable will also overwrite any
executable of the same module/package that was installed already,
but maybe from a different version. Therefore only one version of a
executable can be installed at a time which makes it impossible to work
on different projects that use the same tool but with different
versions.

>>>> History & Future

The local installation of executables built from Go modules/packages has
always been a somewhat controversial point which unfortunately,
partly for historical reasons, does not offer an optimal and
user-friendly solution up to now. The `go` command [8] is a fantastic
toolchain that provides many great features one would expect to be
provided out-of-the-box from a modern and well designed programming
language without the requirement to use a third-party solution:
from compiling code, running unit/integration/benchmark tests, quality
and error analysis, debugging utilities and many more.
Unfortunately the way the `go install` command [2] of Go versions less
or equal to 1.15 handles the installation of an Go module/package
executable is still not optimal.

The general problem of tool dependencies is a long-time known issue/weak
point of the current Go toolchain and is a highly rated change request
from the Go community with discussions like golang/go#30515 [9],
golang/go#25922 [10] and golang/go#27653 [11] to improve this essential
feature, but they've been around for quite a long time without a
solution that works without introducing breaking changes and most users
and the Go team agree on.
Luckily, this topic was finally picked up for the next upcoming Go
release version 1.16 [12] and golang/go#40276 [13] introduces a way to
install executables in module mode outside a module. The release note
preview also already includes details about this change [14] and how
installation of executables from Go modules will be handled in the
future.

>>>> The Workaround

Beside the great news and anticipation about an official solution for
the problem the usage of a workaround is almost inevitable until Go 1.16
is finally released.

The official Go wiki [15] provides a section on "How can I track tool
dependencies for a module?" [16] that describes a workaround that tracks
tool dependencies. It allows to use the Go module logic by using a file
like `tools.go` with a dedicated `tools` build tag that prevents the
included module dependencies to be picked up included for normal
executable builds. This approach works fine for non-main packages,
but CLI tools that are only implemented in the `main` package can not be
imported in such a file.

In order to tackle this problem, a user from the community created
"gobin" [17], an experimental, module-aware command to install/run main
packages.
It allows to install or run main-package commands without "polluting"
the `go.mod` file by default. It downloads modules in version-aware mode
into a binary cache path within the systems cache directory [18].
It prevents problems due to already globally installed executables by
placing each version in its own directory. The decision to use a cache
directory instead of sub-directories within the `GOBIN` path keeps the
system clean.

"gobin" is still in an early development state, but has already received
a lot of positive feedback and is used in many projects.
There are also members of the core Go team that have contributed to the
project and the chance is high that the changes for Go 1.16 were
influenced or partially ported from it.
It is currently the best workaround to...

1. prevent the Go toolchain to pick up the `GOMOD` environment
   variable [4] (see [`go env GOMOD`][4]) that is initialized
   automatically with the path to the `go.mod` file in the current
   working directory.
2. install module/package executables globally without "polluting" the
   `go.mod` file.
3. install module/package executables globally without overriding
   already installed executables of different versions.

See gobin's FAQ page [19] in the repository wiki for more details about
the project.

>>>> The Go Module Caster

To allow to manage the tool dependency problem, wand now uses "gobin"
through a new caster [20]
`go.mod` file and allows to...

1. install `gobin` itself into `GOBIN` (`go env GOBIN` [4]).
2. cast any spell incantation [21] of kind `KindGoModule` [22] by
   installing the executable globally into the dedicated `gobin` cache.

[1]: https://golang.org/ref/mod
[10]: golang/go#25922
[11]: golang/go#27653
[12]: https://github.com/golang/go/milestone/145
[13]: golang/go#40276
[14]: https://tip.golang.org/doc/go1.16#modules
[15]: https://github.com/golang/go/wiki
[16]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
[17]: https://github.com/myitcv/gobin
[18]: https://golang.org/pkg/os/#UserCacheDir
[19]: https://github.com/myitcv/gobin/wiki/FAQ
[2]: https://golang.org/cmd/go#hdr-Compile_and_install_packages_and_dependencies
[20]: https://pkg.go.dev/github.com/svengreb/wand/pkg/cast/gobin#Caster
[21]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#Incantation
[22]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#KindGoModule
[3]: https://golang.org/cmd/go/#hdr-Environment_variables
[4]: https://golang.org/cmd/go/#hdr-Print_Go_environment_information
[5]: https://golang.org/ref/mod#go-mod-file
[6]: https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them
[7]: https://golang.org/ref/mod#mod-commands
[8]: https://golang.org/cmd/go
[9]: golang/go#30515

Closes GH-22
gopherbot pushed a commit to golang/proposal that referenced this issue May 17, 2021
Draft design for 'go install' functionality to install executables in
module mode outside a module at a specific version.

For golang/go#30515
For golang/go#40276

Change-Id: Iaec50eeae92148be51e6acb1d3a9488bb2587bc8
Reviewed-on: https://go-review.googlesource.com/c/proposal/+/243077
Trust: Jay Conrod <jayconrod@google.com>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@golang golang locked and limited conversation to collaborators Jul 22, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet