Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: no way to `go get` source code for editing in module mode #31529

Open
mholt opened this issue Apr 17, 2019 · 7 comments

Comments

Projects
None yet
3 participants
@mholt
Copy link

commented Apr 17, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/matt/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/matt/Dev"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/sf/7w4p15_15jb59d7kwy0wglh40000gn/T/go-build796937585=/tmp/go-build -gno-record-gcc-switches -fno-common"

What are you trying to do?

(Added this question, hope that's OK.)

go get a Go project's source code and install its command (the repo contains both a command and library packages) while respecting module dependencies.

In the past we used vendor to pin dependencies, but we'd rather use Go modules. go get has always worked great, even with an empty GOPATH. If possible, we'd like for the project to stay go-gettable.

What did you do?

  1. Make an empty GOPATH. (or, as I did, export GOPATH= and ensure no ~/go folder exists)
  2. $ GO111MODULE=on go get github.com/mholt/testrepo/caddy

What did you expect to see?

The $GOPATH/src (~/go/src) folder populated with the project's source code, and the binary installed to the bin folder, while honoring the versions specified in go.mod.

What did you see instead?

Only module folders were created.

How does one get the source code, with dependencies, while honoring go.mod?

The one-step install flow of go get was really nice. Will this no longer be the case with Go 1.13 when GO111MODULE=on becomes the default behavior?

More info

We're in an awkward position in our transition to modules. Many people are unable to build Caddy from source without some wrangling.

We've added go.mod and go.sum, removed our vendor folder, and tagged a beta release (in the actual repo, not my test repo here). Unfortunately, modules seem to ignore the beta release tag so all module operations fail unless we explicitly specify the beta version in our commands. So it is not obvious how to test that we've transitioned correctly.

Additionally, without modules (GO111MODULE=off), running go get on Caddy currently fails with:

package github.com/lucas-clemente/quic-go/h2quic: cannot find package "github.com/lucas-clemente/quic-go/h2quic" in any of:
	/usr/local/go/src/github.com/lucas-clemente/quic-go/h2quic (from $GOROOT)
	/Users/matt/go/src/github.com/lucas-clemente/quic-go/h2quic (from $GOPATH

because one of our upstream dependencies have had a breaking change. Our go.mod pins the working version of this dependency, but go get does not honor this (yet?).

But setting GO111MODULE=on also doesn't work because the beta version isn't recognized. The last non-prerelease tag doesn't use modules, so the build also fails.

(As an aside, @v1.0.0-beta1 needs to be specified explicitly, which (temporarily?) breaks our build instructions and isn't obvious... it confused a lot of people including me.)

Even still, if we start over on the test repo but set GO111MODULE=on this time, it does not pull down the source code so we can start developing on it.

If go get is how the GOPATH is populated for development, it should put the source code in it so we can develop with it. If go get becomes a command for a read-only workflow, what is the alternative for "developers" as opposed to just "builders"?

@mholt

This comment has been minimized.

@mholt mholt closed this Apr 19, 2019

@mholt mholt reopened this Apr 19, 2019

@bcmills

This comment has been minimized.

Copy link
Member

commented Apr 19, 2019

go get a Go project's source code and install its command (the repo contains both a command and library packages) while respecting module dependencies.

Note that GO111MODULE=on go get does this today. It just unpacks the source code into the module cache instead of GOPATH/src.

There is currently some question as to what to do about replace directives (see #31173), but if you don't need those then you should be set as far as installing binaries.

@bcmills

This comment has been minimized.

Copy link
Member

commented Apr 19, 2019

But setting GO111MODULE=on also doesn't work because the beta version isn't recognized. The last non-prerelease tag doesn't use modules, so the build also fails.

Yep, that behavior is expected and intended. We want the choice to use an unreleased version (when released alternatives exist) to be explicit.

You can always go get […]@master or go get […]@v1.0.0-beta1. (Or finish up that release and tag it! 🙂)

@bcmills

This comment has been minimized.

Copy link
Member

commented Apr 19, 2019

what is the alternative for "developers" as opposed to just "builders"?

Generally we're trying to move the go command away from autonomously invoking VCS tools in potentially-nested directories, since that has historically been a source of security vulnerabilities.

The alternative for developers is to run git clone (or your alternative VCS of choice) explicitly. If that alternative doesn't work for you, please let us know more detail so that we can understand the problem and the use-cases.

(See also #18387.)

@bcmills bcmills changed the title cmd/go: There appears to be no way to `go get` source code while honoring go.mod cmd/go: no way to `go get` source code for editing in module mode Apr 19, 2019

@bcmills bcmills added this to the Unplanned milestone Apr 19, 2019

@bcmills

This comment has been minimized.

Copy link
Member

commented Apr 19, 2019

(CC @jayconrod, but I don't think we'll be able to make significant changes here until 1.14 at the earliest.)

@mholt

This comment has been minimized.

Copy link
Author

commented Apr 19, 2019

Thanks @bcmills for taking a quick look at this.

(Sorry for the close+open earlier; my trackpad sometimes does phantom clicks.)

I had always viewed go get as a tool to "go get" source code and populate my GOPATH so that I can work on it.

But now with GO111MODULE=on, and with Go 1.13, I will need to view it almost solely as a dependency manager. (This is a significant change that I think is not obvious.)

I agree it seems like a good idea to not get pre-releases by default... just makes the transition awkward since our beta release is the first to try using modules.

Running git clone is reasonable I suppose, just something new to my workflow which I never had to do much with Go code.

@thepudds gave this 5-star explanation in Slack:

The way it works is if you are inside of a module foo and do go get some/cmd, then some/cmd is treated as a dependency of your current module foo, complete with resolving any shared version constraints with the versions listed in go.mod for foo, updating the go.mod for foo to include a require for some/cmd, and storing the code for some/cmd in the module cache just like other dependencies of foo.

The way it works if you are outside of a module with GO111MODULE=on is briefly documented here:

https://golang.org/doc/go1.12#modules

When GO111MODULE is set to on, the go command now supports module-aware operations outside of a module directory, provided that those operations do not need to resolve import paths relative to the current directory or explicitly edit the go.mod file. Commands such as go get, go list, and go mod download behave as if in a module with initially-empty requirements. In this mode, go env GOMOD reports the system's null device (/dev/null or NUL).

In other words, in that case, if you do cd /tmp; go get some/cmd, then some/cmd is still treated "as if" some/cmd is a dependency of a temporary and otherwise empty module, complete with storing the code for some/cmd in the modules cache along with dependencies of other modules.

So now I think I have something I can tell the community, many of whom are currently having trouble building Caddy from source:

  1. git clone https://github.com/mholt/caddy.git
  2. cd caddy
  3. git checkout <version> ⬅️ optional, but necessary if wanting a specific version
  4. go build or go install ⬅️ will get all dependencies as specified by go.mod

Thank you for the help, coming to understand this.

I'll leave the issue open since being able to consolidate those steps into one command (like it has always been before) would be convenient, especially if one (like me! 😄) wants to continue to use a traditional GOPATH. In other words, being able to go get source code as before, but with respect to go.mod, would be great.

One last question: in the future, GOPATH is still where module dependencies will be stored (read-only); will it still be recommended and good practice to store working copies of repos in GOPATH? I ask because I really like its structure, but if it messes with go tooling + modules, I should try to avoid that...

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

One last question: in the future, GOPATH is still where module dependencies will be stored (read-only); will it still be recommended and good practice to store working copies of repos in GOPATH? I ask because I really like its structure, but if it messes with go tooling + modules, I should try to avoid that...

$GOPATH/src is still a good place to keep working copies of repositories. I'd recommend it to keep code organized. However, since the Go command won't look for dependencies there, avoid writing tools or scripts that require this convention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.