cmd/go: allow installing programs to GOBIN when outside a module in module mode

[smyrman]

UPDATE: This is a feature request to be able to install a program/main package to a user's home directory (basically $GOBIN) outside of a module context.

Background

While dep, glide and other tools provide a way of fetching project dependencies, part of what they are missing which I think belongs in any official go get with versioning, is a way to install programs (tools, CLI, or a binary compiled from any main package) into a user's PATH (e.g. GOBIN, which may be relevant, even if GOPATH get's deprecated).

---

Please answer these questions before submitting your issue. Thanks!

What did you do?

$ vgo get github.com/golang/dep/cmd/dep@v0.4.1

Example command only; but this would be the closest to how things currently work with go get.

What did you expect to see?

dep binary (in this case) installed to either $GOBIN, $GOPATH[0]/bin or $HOME/go/bin, in that order.

What did you see instead?

$ vgo get github.com/golang/dep/cmd/dep@v0.4.1
cannot determine module root; please create a go.mod file there

System details

go version go1.10 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/smyrman/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/smyrman/.local"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/sj/d5g1dfkd3g75ccz83k505v3r0000gn/T/go-build848907380=/tmp/go-build -gno-record-gcc-switches -fno-common"
VGOMODROOT=""
GOROOT/bin/go version: go version go1.10 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.10
uname -v: Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.13.3
BuildVersion:	17D102
lldb --version: lldb-900.0.64
  Swift-4.0

[kardianos]

@smyrman I think this is a genuine concern, but it isn't just about pulling down a specific version of a program; it is using vgo outside the context of a module. You'll get the same error if you omit the versuib @v0.0.0 specifier as well. vgo thinks you are trying to pull in a dependency; you are trying to install a program into GOBIN.

Maybe:
x/vgo: no longer supports installing an executable from outside of a module
?

[kardianos]

@smyrman I think this is a genuine concern, but it isn't just about pulling down a specific version of a program; it is using vgo outside the context of a module. You'll get the same error if you omit the versuib @v0.0.0 specifier as well. vgo thinks you are trying to pull in a dependency; you are trying to install a program into GOBIN.

Maybe:
x/vgo: no longer supports installing an executable from outside of a module
?

[smyrman]

You'll get the same error if you omit the versuib @v0.0.0 specifier as well. vgo thinks you are trying to pull in a dependency; you are trying to install a program into GOBIN.

@kardianos, thanks for the response. I am actually not concerned about the error message being unclear. This is meant as a feature request to (somehow) still be able to install binaries to a user's PATH, no matter what your current working directory is, similar to what we can with go get today. Even better, the possibility to install a program at as specific version. I believe a language package manger should not be just about dependencies.

I am not to particular about the syntax, but maybe there could be a flag like -home, -user or even -gopath, that allowed installing stuff to either $GOBIN or $GOPATH/bin. This would be similar to how e.g. pip works, for Python, to mention one, where if you use e.g. venev or virtualenv for a per-project experience, you can still add an option to pip to install tools to a user's home:

• https://pip.pypa.io/en/stable/user_guide/#user-installs
• https://docs.python.org/3/install/index.html#alternate-installation-the-user-scheme

While OS package managers are good at doing more or less the same thing (not to a user's home of course, but system wide), there is often a delay for programs to be packaged, and language power users may prefer to use a language package manger for installing his/hers software, while also being able to pull down unreleased software, as we do today with go get. Users operating on a shared system may have a similar need to install things in their home directories.

Further, even though vgo deprecates GOPATH/GOBIN for project development, it's not necessarily right to deprecate GOPATH for other potentially valid use-cases, but that's probably another discussion, where the only thing I am interested in through this issue, is the GOBIN part.

[smyrman]

You'll get the same error if you omit the versuib @v0.0.0 specifier as well. vgo thinks you are trying to pull in a dependency; you are trying to install a program into GOBIN.

@kardianos, thanks for the response. I am actually not concerned about the error message being unclear. This is meant as a feature request to (somehow) still be able to install binaries to a user's PATH, no matter what your current working directory is, similar to what we can with go get today. Even better, the possibility to install a program at as specific version. I believe a language package manger should not be just about dependencies.

I am not to particular about the syntax, but maybe there could be a flag like -home, -user or even -gopath, that allowed installing stuff to either $GOBIN or $GOPATH/bin. This would be similar to how e.g. pip works, for Python, to mention one, where if you use e.g. venev or virtualenv for a per-project experience, you can still add an option to pip to install tools to a user's home:

• https://pip.pypa.io/en/stable/user_guide/#user-installs
• https://docs.python.org/3/install/index.html#alternate-installation-the-user-scheme

While OS package managers are good at doing more or less the same thing (not to a user's home of course, but system wide), there is often a delay for programs to be packaged, and language power users may prefer to use a language package manger for installing his/hers software, while also being able to pull down unreleased software, as we do today with go get. Users operating on a shared system may have a similar need to install things in their home directories.

Further, even though vgo deprecates GOPATH/GOBIN for project development, it's not necessarily right to deprecate GOPATH for other potentially valid use-cases, but that's probably another discussion, where the only thing I am interested in through this issue, is the GOBIN part.

[smyrman]

PS! I am also not concerned about this issue being solved right away, but it would be really cool to solve it before vgo goes main-line, if that's what going to happen.

[smyrman]

PS! I am also not concerned about this issue being solved right away, but it would be really cool to solve it before vgo goes main-line, if that's what going to happen.

[smyrman]

Updated the description.

[smyrman]

Updated the description.

[rsc]

$ vgo get github.com/golang/dep/cmd/dep@v0.4.1
cannot determine module root; please create a go.mod file there

This clearly must work eventually. The thing I'm not sure about is exactly what this does as far as the version is concerned: does it create a temporary module root and go.mod, do the install, and then throw it away? Probably. But I'm not completely sure, and for now I didn't want to confuse people by making vgo do things outside go.mod trees. Certainly the eventual go command integration has to support this.

Note that installs already happen to $GOBIN or else $GOPATH/bin (so default $HOME/go/bin), not the modue tree.

Will mark NeedsDecision because we still need to work out how to do this, but yes absolutely the command must work.

[rsc]

$ vgo get github.com/golang/dep/cmd/dep@v0.4.1
cannot determine module root; please create a go.mod file there

This clearly must work eventually. The thing I'm not sure about is exactly what this does as far as the version is concerned: does it create a temporary module root and go.mod, do the install, and then throw it away? Probably. But I'm not completely sure, and for now I didn't want to confuse people by making vgo do things outside go.mod trees. Certainly the eventual go command integration has to support this.

Note that installs already happen to $GOBIN or else $GOPATH/bin (so default $HOME/go/bin), not the modue tree.

Will mark NeedsDecision because we still need to work out how to do this, but yes absolutely the command must work.

[smyrman]

does it create a temporary module root and go.mod, do the install, and then throw it away? Probably.

This is a reasonable approach, and have some obvious benefits, such as the result of go get github.com/golang/dep/cmd/dep@v0.4.1 not being affected by other tools installed on the system. To make a decision easier, I will supply an alternative approach and highlight some problems with it:

I suppose an alternative would be to have a "default" module that is used when a flag is provided to the go get command.

E.g. one could picture a default module in $GOPATH[0]/src/u (first entry of GOPATH) that is automatically created and used when the -user flag is given to the go get command line. This would let go get -user github.com/golang/dep/cmd/dep@v0.4.1 be semantically equivalent to:

#!/bin/sh
set -e
_module_path=$(echo $GOPATH | awk -F':' '{ print $1}')/src/u
mkdir -p ${_module_path}
cd  ${_module_path}
test -f go.mod || echo 'module "u"' > go.mod
go get  github.com/golang/dep/cmd/dep@v0.4.1

Some benefits

• One easy flag to access a user-specific module for user-installed cross-project binaries.
• The module path is preserved on disk, so power users could sync the folder via tools like Git or Dropbox to keep their tools with them at specific versions across machines.

Major drawback

• Tool installation (e.g. installation of dep at a specific version as used in the example above), may be affected by (minimal) version selection of overlapping dependencies with other tools, leading to potentially weird bugs.

Conclusion

For me, I belive the drawback outweighs any benefit, and a temporary module root / mod.go root is the semantically correct solution; perhaps implementation wise the temporary module root and go.mod file only exists in memory.

[smyrman]

does it create a temporary module root and go.mod, do the install, and then throw it away? Probably.

This is a reasonable approach, and have some obvious benefits, such as the result of go get github.com/golang/dep/cmd/dep@v0.4.1 not being affected by other tools installed on the system. To make a decision easier, I will supply an alternative approach and highlight some problems with it:

I suppose an alternative would be to have a "default" module that is used when a flag is provided to the go get command.

E.g. one could picture a default module in $GOPATH[0]/src/u (first entry of GOPATH) that is automatically created and used when the -user flag is given to the go get command line. This would let go get -user github.com/golang/dep/cmd/dep@v0.4.1 be semantically equivalent to:

#!/bin/sh
set -e
_module_path=$(echo $GOPATH | awk -F':' '{ print $1}')/src/u
mkdir -p ${_module_path}
cd  ${_module_path}
test -f go.mod || echo 'module "u"' > go.mod
go get  github.com/golang/dep/cmd/dep@v0.4.1

Some benefits

• One easy flag to access a user-specific module for user-installed cross-project binaries.
• The module path is preserved on disk, so power users could sync the folder via tools like Git or Dropbox to keep their tools with them at specific versions across machines.

Major drawback

• Tool installation (e.g. installation of dep at a specific version as used in the example above), may be affected by (minimal) version selection of overlapping dependencies with other tools, leading to potentially weird bugs.

Conclusion

For me, I belive the drawback outweighs any benefit, and a temporary module root / mod.go root is the semantically correct solution; perhaps implementation wise the temporary module root and go.mod file only exists in memory.

[rsc]

This isn't blocking Go 1.11. The main reason being that if GO111MODULE=auto (the default) then we can't reasonably do anything with modules when outside a go.mod tree. That's the opt-in. And inside a go.mod tree this already works.

[rsc]

This isn't blocking Go 1.11. The main reason being that if GO111MODULE=auto (the default) then we can't reasonably do anything with modules when outside a go.mod tree. That's the opt-in. And inside a go.mod tree this already works.

[carlmjohnson]

I agree that this doesn't block Go 1.11, but I do think it should be a blocker for 1.12. There are a lot of repos that have READMEs saying "just run go get github.com/me/myrepo to install". That should continue to work once GO111MODULE changes to "on".

[carlmjohnson]

I agree that this doesn't block Go 1.11, but I do think it should be a blocker for 1.12. There are a lot of repos that have READMEs saying "just run go get github.com/me/myrepo to install". That should continue to work once GO111MODULE changes to "on".

[bcmills]

Agreed. I've marked it as a release-blocker for 1.12.

[bcmills]

Agreed. I've marked it as a release-blocker for 1.12.

[thepudds]

@bcmills Just expanding slightly on the downside you listed, I think that means that go get tool-foo when run outside of a go.mod tree would not work in module mode for a tool that itself has not opted in to modules?

If so, that might be problematic as 1.12 behavior, or whenever module-mode becomes the default behavior of the go tooling. Ideally go get tool-foo would would continue to work for tools that have not updated their readmes, including for tools that have not yet opted in to modules (given it will take some time for the ecosystem to fully adopt modules).

Perhaps an alternative would be to do as you outlined for a module-based tool, but liberalize it such that a non-module based tool would also work via go get tool-foo in module-aware mode when run outside of a module (e.g., perhaps by following 1.10 behavior, or perhaps by effectively creating a temporary go.mod as outlined in #24250 (comment), perhaps by having a 'catch all' per-user go.mod file that picks up requirements when in module-mode outside of a go.mod tree, or ___).

[thepudds]

@bcmills Just expanding slightly on the downside you listed, I think that means that go get tool-foo when run outside of a go.mod tree would not work in module mode for a tool that itself has not opted in to modules?

If so, that might be problematic as 1.12 behavior, or whenever module-mode becomes the default behavior of the go tooling. Ideally go get tool-foo would would continue to work for tools that have not updated their readmes, including for tools that have not yet opted in to modules (given it will take some time for the ecosystem to fully adopt modules).

Perhaps an alternative would be to do as you outlined for a module-based tool, but liberalize it such that a non-module based tool would also work via go get tool-foo in module-aware mode when run outside of a module (e.g., perhaps by following 1.10 behavior, or perhaps by effectively creating a temporary go.mod as outlined in #24250 (comment), perhaps by having a 'catch all' per-user go.mod file that picks up requirements when in module-mode outside of a go.mod tree, or ___).

[jrick]

@bcmills We are using replace directives in main modules pointing to submodules in the same repository, so that changes to submodules can immediately affect the main module and be tested without needing an intermediate step to publish a new module version. When you say

we should emit an explicit error if the go.mod file has any replace directives that point to (relative or absolute) file paths outside of that module

do you mean the repo? What does a replace directive pointing "outside the module" mean?

[jrick]

@bcmills We are using replace directives in main modules pointing to submodules in the same repository, so that changes to submodules can immediately affect the main module and be tested without needing an intermediate step to publish a new module version. When you say

we should emit an explicit error if the go.mod file has any replace directives that point to (relative or absolute) file paths outside of that module

do you mean the repo? What does a replace directive pointing "outside the module" mean?

[bcmills]

do you mean the repo? What does a replace directive pointing "outside the module" mean?

I think I mean the module itself, since that's what module proxies will have available. “Outside the module” means either above the directory containing the module's go.mod file, or at or below the directory of a submodule's go.mod file.

[bcmills]

do you mean the repo? What does a replace directive pointing "outside the module" mean?

I think I mean the module itself, since that's what module proxies will have available. “Outside the module” means either above the directory containing the module's go.mod file, or at or below the directory of a submodule's go.mod file.

[myitcv]

@jrick

We are using replace directives in main modules pointing to submodules in the same repository, so that changes to submodules can immediately affect the main module and be tested without needing an intermediate step to publish a new module version.

But presumably this is just an intermediate, transition step? Because then the "released" version of a command/library should not have any directory-target replace directives, thereby indicating it was tested against specific versions of all its dependencies.

By the way, I can empathise with the pain of the multiple steps involved. I added an Experience Report on just this situation (actually with the added complication of a cyclic module dependency):

https://gist.github.com/myitcv/79c3f12372e13b0cbbdf0411c8c46fd5

[myitcv]

@jrick

We are using replace directives in main modules pointing to submodules in the same repository, so that changes to submodules can immediately affect the main module and be tested without needing an intermediate step to publish a new module version.

But presumably this is just an intermediate, transition step? Because then the "released" version of a command/library should not have any directory-target replace directives, thereby indicating it was tested against specific versions of all its dependencies.

By the way, I can empathise with the pain of the multiple steps involved. I added an Experience Report on just this situation (actually with the added complication of a cyclic module dependency):

https://gist.github.com/myitcv/79c3f12372e13b0cbbdf0411c8c46fd5

[bcmills]

I think that means that go get tool-foo when run outside of a go.mod tree would not work in module mode for a tool that itself has not opted in to modules?

Yes.

Ideally go get tool-foo would would continue to work for tools that have not updated their readmes, including for tools that have not yet opted in to modules (given it will take some time for the ecosystem to fully adopt modules).

That's true. Perhaps a fallback per-user go.mod would work for that, but I don't think we should use a fallback if we have the binary's actual go.mod file available.

[bcmills]

I think that means that go get tool-foo when run outside of a go.mod tree would not work in module mode for a tool that itself has not opted in to modules?

Yes.

Ideally go get tool-foo would would continue to work for tools that have not updated their readmes, including for tools that have not yet opted in to modules (given it will take some time for the ecosystem to fully adopt modules).

That's true. Perhaps a fallback per-user go.mod would work for that, but I don't think we should use a fallback if we have the binary's actual go.mod file available.

[jrick]

@myitcv so far, we have been tagging releases with the replace directives, and fetching the repository or tarball has been a necessary step to build a release (I realize this doesn't play well with module proxies who don't operate on repos). It could be possible to remove those after we branch off to make a release and leave the replaces on our master branch.

[jrick]

@myitcv so far, we have been tagging releases with the replace directives, and fetching the repository or tarball has been a necessary step to build a release (I realize this doesn't play well with module proxies who don't operate on repos). It could be possible to remove those after we branch off to make a release and leave the replaces on our master branch.

[thepudds]

For the case where you in module mode but outside a module and do go get example.com/tool-foo, and tool-foo lacks a go.mod file, the go tooling could automatically create something like:

$GOPATH/pkg/mod/....something.../example.com/tool-foo/go.mod

That would track dependencies and provide reproducible builds via that go.mod without cross contamination from other globally installed tools.

This is separate case from whatever is done for a tool that itself has a go.mod.

I've told a couple people that today with Go 1.11 they can manually create something like ~/tools/gorename/go.mod or ~/tools/goimports/go.mod and then do a go get or go install for the proper tool from within the corresponding directory, but approach outlined above would be automating that process and putting the go.mod into some canonical location (and given no human in the loop, avoid collisions by including the full import path in the directory structure).

[thepudds]

For the case where you in module mode but outside a module and do go get example.com/tool-foo, and tool-foo lacks a go.mod file, the go tooling could automatically create something like:

$GOPATH/pkg/mod/....something.../example.com/tool-foo/go.mod

That would track dependencies and provide reproducible builds via that go.mod without cross contamination from other globally installed tools.

This is separate case from whatever is done for a tool that itself has a go.mod.

I've told a couple people that today with Go 1.11 they can manually create something like ~/tools/gorename/go.mod or ~/tools/goimports/go.mod and then do a go get or go install for the proper tool from within the corresponding directory, but approach outlined above would be automating that process and putting the go.mod into some canonical location (and given no human in the loop, avoid collisions by including the full import path in the directory structure).

[siebenmann]

If so, that might be problematic as 1.12 behavior, or whenever module-mode becomes the default behavior of the go tooling. Ideally go get tool-foo would would continue to work for tools that have not updated their readmes, including for tools that have not yet opted in to modules (given it will take some time for the ecosystem to fully adopt modules).

I think that this is essential behavior for any future 'module only' world, whether it is in 1.12 or later. The reality of life is that there are many people who will not immediately add go.mod files to their programs for various reasons and some number of programs that will never add it because they are no longer being actively developed. A world in which go get <program> stops working unless the program has been updated to have a go.mod file is a world where a great many Go programs are no longer conveniently installable, probably for a significant amount of time. People are unlikely to be happy with this.

In my view, retaining GOPATH for at least this usage is the simplest answer, but people may not like it. Otherwise, the best solution I can think of is to create a temporary go.mod file for the package on the fly.

[siebenmann]

If so, that might be problematic as 1.12 behavior, or whenever module-mode becomes the default behavior of the go tooling. Ideally go get tool-foo would would continue to work for tools that have not updated their readmes, including for tools that have not yet opted in to modules (given it will take some time for the ecosystem to fully adopt modules).

I think that this is essential behavior for any future 'module only' world, whether it is in 1.12 or later. The reality of life is that there are many people who will not immediately add go.mod files to their programs for various reasons and some number of programs that will never add it because they are no longer being actively developed. A world in which go get <program> stops working unless the program has been updated to have a go.mod file is a world where a great many Go programs are no longer conveniently installable, probably for a significant amount of time. People are unlikely to be happy with this.

In my view, retaining GOPATH for at least this usage is the simplest answer, but people may not like it. Otherwise, the best solution I can think of is to create a temporary go.mod file for the package on the fly.

[bcmills]

That would track dependencies and provide reproducible builds via that go.mod without cross contamination from other globally installed tools.

Maybe, although go get -u in non-module mode today is a nice forcing function to pick up at least some updates, and if there isn't any go.mod for the requested package, then maybe using other information gleaned from the community about what works isn't really a bad thing.

I could imagine, say, a $HOME/.go.mod file used as a fallback, where any go get or go install outside a module updates $HOME/.go.mod with indirect dependencies. My fear then is that it might be too convenient: we probably do want to retain some incentive for folks to add go.mod files to individual projects, and if we make it too easy to work without them we lose some of the broader benefits of reproducibility.

[bcmills]

That would track dependencies and provide reproducible builds via that go.mod without cross contamination from other globally installed tools.

Maybe, although go get -u in non-module mode today is a nice forcing function to pick up at least some updates, and if there isn't any go.mod for the requested package, then maybe using other information gleaned from the community about what works isn't really a bad thing.

I could imagine, say, a $HOME/.go.mod file used as a fallback, where any go get or go install outside a module updates $HOME/.go.mod with indirect dependencies. My fear then is that it might be too convenient: we probably do want to retain some incentive for folks to add go.mod files to individual projects, and if we make it too easy to work without them we lose some of the broader benefits of reproducibility.

[mwf]

I've been giving this some more thought, and by now I think the right behavior — if we're in module mode and not inside any other module — is to use the go.mod file from the module containing the requested package.

I've also been thinking about it these days and came to the same conclusion. We have all we need in the go.mod of the module we're building.

I've got only one thing to think about.
If we are building an exteranal binary with go get inside some current module - the current module's dependencies could affect the build (implicitly set other versions than are set in the go.mod of the binary we're installing).

I don't think it's a good behaviour if we are driving for high-fidelity builds.

I would vote for a more predictable behaviour. Something like "if we run go get on a main-package having its own go.mod - we shouldn't use the dependencies from current go.mod (if having one)".

If we apply this rule, all binaries installed with go get will have a predictable build list, affected only by their own module requirements.

[mwf]

I've been giving this some more thought, and by now I think the right behavior — if we're in module mode and not inside any other module — is to use the go.mod file from the module containing the requested package.

I've also been thinking about it these days and came to the same conclusion. We have all we need in the go.mod of the module we're building.

I've got only one thing to think about.
If we are building an exteranal binary with go get inside some current module - the current module's dependencies could affect the build (implicitly set other versions than are set in the go.mod of the binary we're installing).

I don't think it's a good behaviour if we are driving for high-fidelity builds.

I would vote for a more predictable behaviour. Something like "if we run go get on a main-package having its own go.mod - we shouldn't use the dependencies from current go.mod (if having one)".

If we apply this rule, all binaries installed with go get will have a predictable build list, affected only by their own module requirements.

[bcmills]

@mwf

If we are building an [external] binary with go get inside some current module - the current module's dependencies could affect the build […].

I don't think it's a good behaviour if we are driving for high-fidelity builds.

It may reduce fidelity somewhat, but seems like a desirable and perhaps even necessary behavior (see #25922). For example, suppose that one of the tools you're using calls out to a parser for some language (perhaps Go itself): if you run into a bug in the parser, you may want to rebuild the tool with an updated parser without waiting to push that update upstream.

[bcmills]

@mwf

If we are building an [external] binary with go get inside some current module - the current module's dependencies could affect the build […].

I don't think it's a good behaviour if we are driving for high-fidelity builds.

It may reduce fidelity somewhat, but seems like a desirable and perhaps even necessary behavior (see #25922). For example, suppose that one of the tools you're using calls out to a parser for some language (perhaps Go itself): if you run into a bug in the parser, you may want to rebuild the tool with an updated parser without waiting to push that update upstream.

[mwf]

You're absolutely right, it seems we just need a support for both cases.

What do you think about adding some build flag to switch the behaviour?
Maybe something like -mod=standalone, telling go build to use the original go.mod and nothing more.

[mwf]

You're absolutely right, it seems we just need a support for both cases.

What do you think about adding some build flag to switch the behaviour?
Maybe something like -mod=standalone, telling go build to use the original go.mod and nothing more.

[punya]

Would it make sense to instead support something analogous to npx from the npm ecosystem? I.e. go mod run <modulespec> <package>? Like other module commands, it would transparently cache the relevant module, and grab dependencies as needed. It would only work for commands contained within modules.

If this seems valuable, I can write up a fuller proposal separate from this issue.

[punya]

Would it make sense to instead support something analogous to npx from the npm ecosystem? I.e. go mod run <modulespec> <package>? Like other module commands, it would transparently cache the relevant module, and grab dependencies as needed. It would only work for commands contained within modules.

If this seems valuable, I can write up a fuller proposal separate from this issue.

[myitcv]

@punya the go tool already has go run pkg which does exactly that in a module context. In a global context it probably make sense to ensure that we retain this behaviour (although I don't think the multiple version support is necessary - major versions, yes, via the import path /vX, but not minor/patch versions).

However, the main issue with using go run is #25416. Although there is a clear way to sidestep that: #25416 (comment)

[myitcv]

@punya the go tool already has go run pkg which does exactly that in a module context. In a global context it probably make sense to ensure that we retain this behaviour (although I don't think the multiple version support is necessary - major versions, yes, via the import path /vX, but not minor/patch versions).

However, the main issue with using go run is #25416. Although there is a clear way to sidestep that: #25416 (comment)

[punya-asapp]

Thanks for the pointer. Russ's comment clearly goes against the substance of what I proposed so I'll hold off.

[punya-asapp]

Thanks for the pointer. Russ's comment clearly goes against the substance of what I proposed so I'll hold off.

[myitcv]

Just as a reminder when we come to look at this issue that we will need to bear in mind #26794

[myitcv]

Just as a reminder when we come to look at this issue that we will need to bear in mind #26794

[thepudds]

Would it make sense to re-title this issue? Some people read this current issue title:

"cmd/go: allow installing programs (at specific versions) to GOBIN"

...as meaning that this issue is just about making go get some/command@v1.2.3 work (in other words, that this issue is about allowing versions for go get with tools). However, the discussion in this issue is broader than that (including in terms of the other GitHub issues that are being closed as duplicates of this issue here).

[thepudds]

Would it make sense to re-title this issue? Some people read this current issue title:

"cmd/go: allow installing programs (at specific versions) to GOBIN"

...as meaning that this issue is just about making go get some/command@v1.2.3 work (in other words, that this issue is about allowing versions for go get with tools). However, the discussion in this issue is broader than that (including in terms of the other GitHub issues that are being closed as duplicates of this issue here).

[coolaj86]

I'm a go noobie (coming in at an excellent time with the introduction of go modules) and I wanted to throw in my 2¢ of my use case.

First

I would expect either success (installed bin) or an error message when from my home directory I do this:

go install github.com/UnnoTed/fileb0x

or this:

go get github.com/UnnoTed/fileb0x

Instead (go1.11 darwin/amd64) I get no error and, even worse, echo $? yields 0.

Second

I don't have a $GOBIN. I would expect that to be... dirname $(command -v go) if not otherwise specified?

Third

The purpose of fileb0x is to be used by go generate, which is why I was trying to install it following the readme.

For anyone else in a similar use case finding this via search, I did find that adding go run worked for me. However, it added everything to my go.mod. I'm not sure if that's desirable or undesirable. Personally I appreciate it.

//go:generate go run github.com/UnnoTed/fileb0x b0x.yaml

[coolaj86]

I'm a go noobie (coming in at an excellent time with the introduction of go modules) and I wanted to throw in my 2¢ of my use case.

First

I would expect either success (installed bin) or an error message when from my home directory I do this:

go install github.com/UnnoTed/fileb0x

or this:

go get github.com/UnnoTed/fileb0x

Instead (go1.11 darwin/amd64) I get no error and, even worse, echo $? yields 0.

Second

I don't have a $GOBIN. I would expect that to be... dirname $(command -v go) if not otherwise specified?

Third

The purpose of fileb0x is to be used by go generate, which is why I was trying to install it following the readme.

For anyone else in a similar use case finding this via search, I did find that adding go run worked for me. However, it added everything to my go.mod. I'm not sure if that's desirable or undesirable. Personally I appreciate it.

//go:generate go run github.com/UnnoTed/fileb0x b0x.yaml

[davecheney]

GOBIN is optional, go install will install to $GOPATH/bin of GOBIN is not provided and in the case that GOPATH is not provided the Go tool will assume it to be $HOME/go, thus go install installs to $HOME/go/bin unless instructed otherwise.

…

On 22 Sep 2018, at 14:59, AJ ONeal ***@***.***> wrote: I'm a go noobie (coming in at an excellent time with the introduction of go modules) and I wanted to throw in my 2¢ of my use case. First I would expect either success (installed bin) or an error message when from my home directory I do this: go install github.com/UnnoTed/fileb0x or this: go get github.com/UnnoTed/fileb0x Instead (go1.11 darwin/amd64) I get no error and, even worse, echo $? yields 0. Second I don't have a $GOBIN. I would expect that to be... dirname $(command -v go) if not otherwise specified? Third The purpose of fileb0x is to be used by go generate, which is why I was trying to install it following the readme. For anyone else in a similar use case finding this via search, I did find that adding go run worked for me. However, it added everything to my go.mod. I'm not sure if that's desirable or undesirable. Personally I appreciate it. //go:generate go run github.com/UnnoTed/fileb0x b0x.yaml — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[davecheney]

GOBIN is optional, go install will install to $GOPATH/bin of GOBIN is not provided and in the case that GOPATH is not provided the Go tool will assume it to be $HOME/go, thus go install installs to $HOME/go/bin unless instructed otherwise.

…

On 22 Sep 2018, at 14:59, AJ ONeal ***@***.***> wrote: I'm a go noobie (coming in at an excellent time with the introduction of go modules) and I wanted to throw in my 2¢ of my use case. First I would expect either success (installed bin) or an error message when from my home directory I do this: go install github.com/UnnoTed/fileb0x or this: go get github.com/UnnoTed/fileb0x Instead (go1.11 darwin/amd64) I get no error and, even worse, echo $? yields 0. Second I don't have a $GOBIN. I would expect that to be... dirname $(command -v go) if not otherwise specified? Third The purpose of fileb0x is to be used by go generate, which is why I was trying to install it following the readme. For anyone else in a similar use case finding this via search, I did find that adding go run worked for me. However, it added everything to my go.mod. I'm not sure if that's desirable or undesirable. Personally I appreciate it. //go:generate go run github.com/UnnoTed/fileb0x b0x.yaml — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[coolaj86]

Thanks @davecheney. Since I'm using go1.11 I don't have a GOPATH either. I do see that the bins were put in $HOME/go/bin/. I'll set GOBIN=$(dirname $(command -v go)).

Interestingly, I actually don't have command -v go in my user PATH variables. It appears that go installed itself sneakily into /etc/paths.d/go when I wasn't looking (probably the mac .pkg installer)

[coolaj86]

Thanks @davecheney. Since I'm using go1.11 I don't have a GOPATH either. I do see that the bins were put in $HOME/go/bin/. I'll set GOBIN=$(dirname $(command -v go)).

Interestingly, I actually don't have command -v go in my user PATH variables. It appears that go installed itself sneakily into /etc/paths.d/go when I wasn't looking (probably the mac .pkg installer)

[coolaj86]

P.S. I'd like to recommend that go install have an additional output of Installed xyz to /some/path/bin/thing. That would be useful also in cases where I happed to be in an environment that would install in places that I didn't expect due to my own forgetfulness (perhaps something like gvm or a folder with a go.mod that causes ambiguity), as well as the case above.

[coolaj86]

P.S. I'd like to recommend that go install have an additional output of Installed xyz to /some/path/bin/thing. That would be useful also in cases where I happed to be in an environment that would install in places that I didn't expect due to my own forgetfulness (perhaps something like gvm or a folder with a go.mod that causes ambiguity), as well as the case above.

[myitcv]

@coolaj86 you can use go list to find the target of go install:

# setup a fresh GOPATH for testing's sake
export GOPATH=$(mktemp -d)

# create a module for demo purposes
cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello

# make this a main package
cat <<EOD > main.go
package main
EOD

# look at the "default" Target
go list -f "{{.Target}}" .

# now compare with GOBIN set
export GOBIN=$PWD/.bin
go list -f "{{.Target}}" .

For more details see go help list.

For anyone else in a similar use case finding this via search, I did find that adding go run worked for me. However, it added everything to my go.mod. I'm not sure if that's desirable or undesirable. Personally I appreciate it.

This falls under the umbrella of #25922 and #27653.

Since I'm using go1.11 I don't have a GOPATH either. I do see that the bins were put in $HOME/go/bin/. I'll set GOBIN=$(dirname $(command -v go)).

Perhaps GOROOT would be more helpful? For more details see go help environment. go env is your friend too.

[myitcv]

@coolaj86 you can use go list to find the target of go install:

# setup a fresh GOPATH for testing's sake
export GOPATH=$(mktemp -d)

# create a module for demo purposes
cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello

# make this a main package
cat <<EOD > main.go
package main
EOD

# look at the "default" Target
go list -f "{{.Target}}" .

# now compare with GOBIN set
export GOBIN=$PWD/.bin
go list -f "{{.Target}}" .

For more details see go help list.

For anyone else in a similar use case finding this via search, I did find that adding go run worked for me. However, it added everything to my go.mod. I'm not sure if that's desirable or undesirable. Personally I appreciate it.

This falls under the umbrella of #25922 and #27653.

Since I'm using go1.11 I don't have a GOPATH either. I do see that the bins were put in $HOME/go/bin/. I'll set GOBIN=$(dirname $(command -v go)).

Perhaps GOROOT would be more helpful? For more details see go help environment. go env is your friend too.

[coolaj86]

I think that GOPATH GOBIN is "The Right Place" ™.

I will (except in rare circumstances) have write access to my GOBIN and GOPATH. I may or may not have write access to my GOBIN GOROOT (i.e. if I install via apt install go-1.11, heaven forbid).

However, I still am a firm believer that go install should print the install path.

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
...
Readability counts.
...
Although practicality beats purity.
...
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
...
If the implementation is hard to explain, it's a bad idea.

Printing the path is beautiful, explicit, obvious, simple, readable, practical, does not require guessing, not difficult to explain over the phone and remember later.

For example, I had to search my email to find the go list command you mentioned, and I had to try it three times to get it right.

• 1st time I used the '.', which was obviously wrong, but I just copy-pasted
• 2nd time I had the " in the wrong place, including the package
• 3rd time the quote was correct, as was my guess that . should be replaced with the qualified name.

I'm assuming there are temporary variables that could be set in the environment (i.e. when using gvm or other tools) that could lead to the package installing into an alternate place that would become non-reproducible if I were to run the go list command at a later time or in another terminal.

[coolaj86]

I think that GOPATH GOBIN is "The Right Place" ™.

I will (except in rare circumstances) have write access to my GOBIN and GOPATH. I may or may not have write access to my GOBIN GOROOT (i.e. if I install via apt install go-1.11, heaven forbid).

However, I still am a firm believer that go install should print the install path.

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
...
Readability counts.
...
Although practicality beats purity.
...
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
...
If the implementation is hard to explain, it's a bad idea.

Printing the path is beautiful, explicit, obvious, simple, readable, practical, does not require guessing, not difficult to explain over the phone and remember later.

For example, I had to search my email to find the go list command you mentioned, and I had to try it three times to get it right.

• 1st time I used the '.', which was obviously wrong, but I just copy-pasted
• 2nd time I had the " in the wrong place, including the package
• 3rd time the quote was correct, as was my guess that . should be replaced with the qualified name.

I'm assuming there are temporary variables that could be set in the environment (i.e. when using gvm or other tools) that could lead to the package installing into an alternate place that would become non-reproducible if I were to run the go list command at a later time or in another terminal.

[bcmills]

I will (except in rare circumstances) have write access to my GOPATH. I may or may not have write access to my GOBIN (i.e. if I install via apt install go-1.11, heaven forbid).

I think you're confusing GOBIN with GOROOT?

You should always have write access to GOBIN, because that's where go get puts binaries normally.

[bcmills]

I will (except in rare circumstances) have write access to my GOPATH. I may or may not have write access to my GOBIN (i.e. if I install via apt install go-1.11, heaven forbid).

I think you're confusing GOBIN with GOROOT?

You should always have write access to GOBIN, because that's where go get puts binaries normally.

[coolaj86]

Probably

s/GOBIN/GOROOT/g

s/GOPATH/GOBIN/g

[coolaj86]

Probably

s/GOBIN/GOROOT/g

s/GOPATH/GOBIN/g