Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: global go get does not reference main package's module's go.sum #37225

Open
myitcv opened this issue Feb 14, 2020 · 8 comments
Open

cmd/go: global go get does not reference main package's module's go.sum #37225

myitcv opened this issue Feb 14, 2020 · 8 comments

Comments

@myitcv
Copy link
Member

@myitcv myitcv commented Feb 14, 2020

What version of Go are you using (go version)?

$ go version
go version devel +b7689f5aa3 Fri Jan 31 06:02:00 2020 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOENV="/home/myitcv/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/tmp/tmp.lzY0uMO3kp"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myitcv/gos"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build624481406=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The current advice for globally installing a tool is some variant of:

(cd $(mktemp -d); GO111MODULE=on go get example.com/tool)

The following command sequence succeeds:

export GOPATH=$(mktemp -d)
cd $(mktemp -d)
GO111MODULE=on go get github.com/myitcvscratch/badmain@v0.0.0-20200214123625-d46c956f729b

However the following command sequence does not:

cd $(mktemp -d)
git clone -q https://github.com/myitcvscratch/badmain
cd badmain/
git checkout d46c956
go install

It fails with:

verifying golang.org/x/tools@v0.0.0-20200213224642-88e652f7a869: checksum mismatch
        downloaded: h1:DPqS0AlgYBVHhG5jnEVScBXXIS+xjgn7O8s1E3sDqxc=
        go.sum:     h1:DPqS0AlgYBVHhG5jnEVScBXXIS+xjgn7O8s1E3sD=

(an error I have deliberately introduced)

What did you expect to see?

Given we are installing a tool outside of a module context, have no go.{mod,sum} and hence are relying on a temporary module, part of me is not surprised by this behaviour.

However I think it's problematic. It feels like the default advice for installing tools globally should use some go.sum reference when installing said tool, rather than blindly accepting whatever it sees. In this case it could use the go.sum that is part of the github.com/myitcvscratch/badmain module.

Arguably go get's of this sort should fail if either the go.mod or go.sum is incomplete, but that's perhaps another issue.

What did you see instead?

As above.

@myitcv

This comment has been minimized.

Copy link
Member Author

@myitcv myitcv commented Feb 14, 2020

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

This is closely related to #28802, which was withdrawn.

The default go.sum reference, as of Go 1.13, is the global checksum database. (The only modules that are not verified today during a global install are those that intentionally bypass the database due to GOPRIVATE or GONOSUMDB.)

@bcmills bcmills added the modules label Feb 14, 2020
@bcmills bcmills added this to the Unplanned milestone Feb 14, 2020
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

@myitcv

This comment has been minimized.

Copy link
Member Author

@myitcv myitcv commented Feb 14, 2020

Thanks @bcmills

The default go.sum reference, as of Go 1.13, is the global checksum database

Indeed. My description above was inaccurate ("blindly accepting whatever it sees") - thanks for the correction.

My asking this question is in the context of whether a go get of this sort should fail if the main package's module's go.mod is incomplete. Because if it is then we're not guaranteed to have reproducible builds.

Hence it seemed logical (to me at least!) to use the main package's module's go.sum for authentication, and similarly fail if it was incomplete/didn't verify.

The extent to which either of these points are a problem, though, is not totally clear to me. Hence I raised this issue more for discussion.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

My asking this question is in the context of whether a go get of this sort should fail if the main package's module's go.mod is incomplete.

Ah, that is a different (and also interesting) question!

go get explicitly updates module dependencies, so in general (within or outside of a module) it does resolve missing imports. But if we're in an explicit “global install” mode (#30515) — and especially if that “global install” mode has a well-defined notion of “main module” — then it might make sense to use the main module's go.mod and go.sum file as well.

(But I think that observation probably belongs on #30515, since absent an explicit “global install” flag we would like go get to have the same behavior both within and outside of a main module.)

@myitcv

This comment has been minimized.

Copy link
Member Author

@myitcv myitcv commented Feb 14, 2020

Per Slack, I don't even know what name to give this mode:

(cd $(mktemp -d); GO111MODULE=on go get example.com/tool)

So for the purposes of what follows, I will refer to it as "mode Y" of go get 😄

go get explicitly updates module dependencies, so in general (within or outside of a module) it does resolve missing imports

I guess my point is that in "mode Y" the user is blind to the fact that a temporary module is being created, used and then thrown away. So they are also blind to the fact that the step is potentially not reproducible.

But if we're in an explicit “global install” mode (#30515)

Indeed, there's definitely overlap with that discussion. Per my comment above, I suppose I'm not really clear what "mode Y" is useful for, or whether we should be advising people to rely on/use it.

... then it might make sense to use the main module's go.mod and go.sum file as well

We could even use the main module's non-directory replace directives as well 🏃

But I think that observation probably belongs on #30515,

Agreed. This issue was about the go.sum point. As I said above, happy for you to close this as "working as intended" if we don't think there's an issue here.

On that very point, is there some documentation somewhere that describes what happens in "mode Y"? Per my Slack message, I couldn't find any.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 14, 2020

We've been using the term “module mode outside of a main module” for “mode Y”, but that is admittedly pretty verbose.

The behavior outside of a module is not very thoroughly documented at this point because it has been in flux (#24250 in 1.12, #32027 in 1.14, perhaps #30515 in 1.15 if we can straighten out the semantics, maybe #36513 at some point). It probably still needs some fine-tuning, so we certainly appreciate the input!

@FiloSottile

This comment has been minimized.

Copy link
Member

@FiloSottile FiloSottile commented Feb 14, 2020

For what it's worth, there is a degree of per-machine reproducibility because the tree head of the checksum database which is stored in $GOPATH/pkg/sumdb/sum.golang.org/latest can only ratchet forward on a certain machine, and it can't include disagreeing values for the same module version without detection by auditors.

“Module mode outside of a main module” can't get any better than that I suspect, because there is no way to verify the go.sum of the go geted module itself, so relying on it would be fairly moot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.