New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: allow "go get" when outside a module in module mode #24250

Closed
smyrman opened this Issue Mar 5, 2018 · 43 comments

Comments

Projects
None yet
@smyrman

smyrman commented Mar 5, 2018

UPDATE: This is a feature request to be able to install a program/main package to a user's home directory (basically $GOBIN) outside of a module context.

Background

While dep, glide and other tools provide a way of fetching project dependencies, part of what they are missing which I think belongs in any official go get with versioning, is a way to install programs (tools, CLI, or a binary compiled from any main package) into a user's PATH (e.g. GOBIN, which may be relevant, even if GOPATH get's deprecated).


Please answer these questions before submitting your issue. Thanks!

What did you do?

$ vgo get github.com/golang/dep/cmd/dep@v0.4.1

Example command only; but this would be the closest to how things currently work with go get.

What did you expect to see?

dep binary (in this case) installed to either $GOBIN, $GOPATH[0]/bin or $HOME/go/bin, in that order.

What did you see instead?

$ vgo get github.com/golang/dep/cmd/dep@v0.4.1
cannot determine module root; please create a go.mod file there

System details

go version go1.10 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/smyrman/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/smyrman/.local"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/sj/d5g1dfkd3g75ccz83k505v3r0000gn/T/go-build848907380=/tmp/go-build -gno-record-gcc-switches -fno-common"
VGOMODROOT=""
GOROOT/bin/go version: go version go1.10 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.10
uname -v: Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.13.3
BuildVersion:	17D102
lldb --version: lldb-900.0.64
  Swift-4.0

@gopherbot gopherbot added this to the vgo milestone Mar 5, 2018

@smyrman smyrman changed the title from x/vgo: feature: Allow installing _programs_ at specific versions to x/vgo: feature: Allow installing programs at specific versions Mar 5, 2018

@smyrman smyrman changed the title from x/vgo: feature: Allow installing programs at specific versions to x/vgo: feature: allow installing programs at specific versions Mar 5, 2018

@kardianos

This comment has been minimized.

Contributor

kardianos commented Mar 5, 2018

@smyrman I think this is a genuine concern, but it isn't just about pulling down a specific version of a program; it is using vgo outside the context of a module. You'll get the same error if you omit the versuib @v0.0.0 specifier as well. vgo thinks you are trying to pull in a dependency; you are trying to install a program into GOBIN.

Maybe:
x/vgo: no longer supports installing an executable from outside of a module
?

@smyrman smyrman changed the title from x/vgo: feature: allow installing programs at specific versions to x/vgo: feature: allow installing programs (at specific versions) Mar 5, 2018

@smyrman

This comment has been minimized.

smyrman commented Mar 5, 2018

You'll get the same error if you omit the versuib @v0.0.0 specifier as well. vgo thinks you are trying to pull in a dependency; you are trying to install a program into GOBIN.

@kardianos, thanks for the response. I am actually not concerned about the error message being unclear. This is meant as a feature request to (somehow) still be able to install binaries to a user's PATH, no matter what your current working directory is, similar to what we can with go get today. Even better, the possibility to install a program at as specific version. I believe a language package manger should not be just about dependencies.

I am not to particular about the syntax, but maybe there could be a flag like -home, -user or even -gopath, that allowed installing stuff to either $GOBIN or $GOPATH/bin. This would be similar to how e.g. pip works, for Python, to mention one, where if you use e.g. venev or virtualenv for a per-project experience, you can still add an option to pip to install tools to a user's home:

While OS package managers are good at doing more or less the same thing (not to a user's home of course, but system wide), there is often a delay for programs to be packaged, and language power users may prefer to use a language package manger for installing his/hers software, while also being able to pull down unreleased software, as we do today with go get. Users operating on a shared system may have a similar need to install things in their home directories.

Further, even though vgo deprecates GOPATH/GOBIN for project development, it's not necessarily right to deprecate GOPATH for other potentially valid use-cases, but that's probably another discussion, where the only thing I am interested in through this issue, is the GOBIN part.

@smyrman smyrman changed the title from x/vgo: feature: allow installing programs (at specific versions) to x/vgo: feature: allow installing programs (at specific versions) to GOBIN Mar 5, 2018

@smyrman

This comment has been minimized.

smyrman commented Mar 5, 2018

PS! I am also not concerned about this issue being solved right away, but it would be really cool to solve it before vgo goes main-line, if that's what going to happen.

@smyrman

This comment has been minimized.

smyrman commented Mar 5, 2018

Updated the description.

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 30, 2018

$ vgo get github.com/golang/dep/cmd/dep@v0.4.1
cannot determine module root; please create a go.mod file there

This clearly must work eventually. The thing I'm not sure about is exactly what this does as far as the version is concerned: does it create a temporary module root and go.mod, do the install, and then throw it away? Probably. But I'm not completely sure, and for now I didn't want to confuse people by making vgo do things outside go.mod trees. Certainly the eventual go command integration has to support this.

Note that installs already happen to $GOBIN or else $GOPATH/bin (so default $HOME/go/bin), not the modue tree.

Will mark NeedsDecision because we still need to work out how to do this, but yes absolutely the command must work.

@rsc rsc added the NeedsDecision label Mar 30, 2018

@smyrman

This comment has been minimized.

smyrman commented Jun 7, 2018

does it create a temporary module root and go.mod, do the install, and then throw it away? Probably.

This is a reasonable approach, and have some obvious benefits, such as the result of go get github.com/golang/dep/cmd/dep@v0.4.1 not being affected by other tools installed on the system. To make a decision easier, I will supply an alternative approach and highlight some problems with it:

I suppose an alternative would be to have a "default" module that is used when a flag is provided to the go get command.

E.g. one could picture a default module in $GOPATH[0]/src/u (first entry of GOPATH) that is automatically created and used when the -user flag is given to the go get command line. This would let go get -user github.com/golang/dep/cmd/dep@v0.4.1 be semantically equivalent to:

#!/bin/sh
set -e
_module_path=$(echo $GOPATH | awk -F':' '{ print $1}')/src/u
mkdir -p ${_module_path}
cd  ${_module_path}
test -f go.mod || echo 'module "u"' > go.mod
go get  github.com/golang/dep/cmd/dep@v0.4.1

Some benefits

  • One easy flag to access a user-specific module for user-installed cross-project binaries.
  • The module path is preserved on disk, so power users could sync the folder via tools like Git or Dropbox to keep their tools with them at specific versions across machines.

Major drawback

  • Tool installation (e.g. installation of dep at a specific version as used in the example above), may be affected by (minimal) version selection of overlapping dependencies with other tools, leading to potentially weird bugs.

Conclusion

For me, I belive the drawback outweighs any benefit, and a temporary module root / mod.go root is the semantically correct solution; perhaps implementation wise the temporary module root and go.mod file only exists in memory.

@rsc rsc modified the milestones: vgo, Go1.11 Jul 12, 2018

@rsc rsc added the modules label Jul 12, 2018

@rsc rsc changed the title from x/vgo: feature: allow installing programs (at specific versions) to GOBIN to cmd/go: feature: allow installing programs (at specific versions) to GOBIN Jul 12, 2018

@rsc rsc changed the title from cmd/go: feature: allow installing programs (at specific versions) to GOBIN to cmd/go: allow installing programs (at specific versions) to GOBIN Jul 17, 2018

@rsc

This comment has been minimized.

Contributor

rsc commented Aug 9, 2018

This isn't blocking Go 1.11. The main reason being that if GO111MODULE=auto (the default) then we can't reasonably do anything with modules when outside a go.mod tree. That's the opt-in. And inside a go.mod tree this already works.

@rsc rsc modified the milestones: Go1.11, Go1.12 Aug 9, 2018

@carlmjohnson

This comment has been minimized.

Contributor

carlmjohnson commented Aug 10, 2018

I agree that this doesn't block Go 1.11, but I do think it should be a blocker for 1.12. There are a lot of repos that have READMEs saying "just run go get github.com/me/myrepo to install". That should continue to work once GO111MODULE changes to "on".

@bcmills

This comment has been minimized.

Member

bcmills commented Aug 10, 2018

Agreed. I've marked it as a release-blocker for 1.12.

@coolaj86

This comment has been minimized.

coolaj86 commented Oct 8, 2018

Probably

s/GOBIN/GOROOT/g

s/GOPATH/GOBIN/g

@rsc rsc changed the title from cmd/go: allow installing programs to GOBIN when outside a module in module mode to cmd/go: allow installing programs when outside a module in module mode Oct 25, 2018

@rsc rsc changed the title from cmd/go: allow installing programs when outside a module in module mode to cmd/go: allow "go get" when outside a module in module mode Oct 25, 2018

@myitcv

This comment has been minimized.

Member

myitcv commented Oct 29, 2018

As something of an experiment in this space, would appreciate feedback etc on https://github.com/myitcv/gobin

@cespare

This comment has been minimized.

Contributor

cespare commented Oct 29, 2018

@bcmills is something going to happen here for the Go 1.12 freeze?

Not being able to use go get in the "fetch/update a tool globally" sense has been the biggest frustration for us with the module changes in Go 1.11. We need a command we can run from anywhere that updates tools, now that go get has been broken for that purpose.

I'm not sure I totally follow the entire lengthy discussion here, but it sounds like even with this change we would still need to do cd /somewhere/outside/of/a/module && go get blah.

@smyrman

This comment has been minimized.

smyrman commented Oct 29, 2018

I'm not sure I totally follow the entire lengthy discussion here, but it sounds like even with this change we would still need to do cd /somewhere/outside/of/a/module && go get blah.

Well, I really hope the final solution will include a flag as suggested here:

#24250 (comment)

Or that it uses a different command entirely... Or that go modules (as it's still an experiement), is in fact changed to use go mod add if you want to download something not a tool, but for the current module...

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 5, 2018

@cespare

is something going to happen here for the Go 1.12 freeze?

I'm trying to get this in today, now that the stack for #26794 has been tested and mailed. Whether it makes the freeze will depend largely on how invasive the fix is.

@gopherbot

This comment has been minimized.

gopherbot commented Nov 8, 2018

Change https://golang.org/cl/148517 mentions this issue: cmd/go: enable module mode without a main module when GO111MODULE=on

dsnet referenced this issue in golang/protobuf Nov 28, 2018

protoc-gen-go: generate XXX_OneofWrappers instead of XXX_OneofFuncs (#…
…760)

The marshaler, unmarshaler, and sizer functions are unused ever since
the underlying implementation was switched to be table-driven.
Change the function to only return the wrapper structs.

This change:
* enables generated protos to drop dependencies on certain proto types
* reduces the size of generated protos
* simplifies the implementation of oneofs in protoc-gen-go

Updates #708

@bcmills bcmills added the NeedsFix label Nov 28, 2018

@gopherbot gopherbot closed this in cdbd4d4 Nov 29, 2018

@dsnet dsnet referenced this issue Nov 30, 2018

Closed

regen #117

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment