cmd/go: get panics with "can't find reason for requirement on"

[ash2k]

What version of Go are you using (go version)?

$ go version
go version go1.17 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/mikhail/Library/Caches/go-build"
GOENV="/Users/mikhail/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/mikhail/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/mikhail/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/opt/go/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/opt/go/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.6"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/mikhail/src/gitlab-agent/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/f5/v0r6vhks50d61jjmrhfgnlbh0000gn/T/go-build1390498625=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I'm trying new pruned modules functionality to see if it helps reduce the number of tracked dependencies in my project.

• I've changed one of the libraries to go 1.17 and ran go1.17 mod tidy -compat=1.17. I had to change the go directive in go.mod to 1.17 manually, re-run the command, and only then it worked - is this expected? Pushed this into a branch (commit).
• Then in my project I did the same as above, and then ran:

go1.17 get gitlab.com/gitlab-org/labkit@fc0c07abad6a32b5b77e16289fcc72d38f8411ab
go: downloading gitlab.com/gitlab-org/labkit v1.9.1-0.20210826103322-fc0c07abad6a
panic: internal error: can't find reason for requirement on github.com/google/gofuzz@v1.1.0

goroutine 1 [running]:
cmd/go/internal/modget.(*resolver).updateBuildList.func1({{0xc000ac4a68, 0xc00065d920}, {0xc00002afd0, 0xc000010570}})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1794 +0x10a
cmd/go/internal/modget.(*resolver).updateBuildList(0xc0005cc100, {0x166fac0, 0xc00002a0c0}, {0x0, 0x0, 0x0})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1799 +0x5bb
cmd/go/internal/modget.(*resolver).resolveQueries(0xc0005cc100, {0x166fac0, 0xc00002a0c0}, {0xc000010050, 0x1, 0x1ac4658})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1278 +0x1e5
cmd/go/internal/modget.runGet({0x166fac0, 0xc00002a0c0}, 0xc000026318, {0xc0000201d0, 0x1, 0x1})
        /usr/local/go/src/cmd/go/internal/modget/get.go:300 +0x374
main.invoke(0x1983c80, {0xc0000201c0, 0x2, 0x2})
        /usr/local/go/src/cmd/go/main.go:216 +0x2f6
main.main()
        /usr/local/go/src/cmd/go/main.go:173 +0x78e

• Then I thought "It shouldn't crash, but maybe I've messed something up in the library repo?". I went there, ran go1.17 mod tidy (without -compat=1.17) and indeed there was a new change (commit). I pushed it too.
• Ran go get in my project:

go1.17 get gitlab.com/gitlab-org/labkit@c3248dffceabec66f9f26a08feb847a438d33707
go: downloading gitlab.com/gitlab-org/labkit v1.9.1-0.20210826104146-c3248dffceab
panic: internal error: can't find reason for requirement on github.com/google/gofuzz@v1.1.0

goroutine 1 [running]:
cmd/go/internal/modget.(*resolver).updateBuildList.func1({{0xc0004c74d0, 0xc00073ed20}, {0xc00002afd6, 0xc000604090}})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1794 +0x10a
cmd/go/internal/modget.(*resolver).updateBuildList(0xc00059a100, {0x166fac0, 0xc00002a0c0}, {0x0, 0x0, 0x0})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1799 +0x5bb
cmd/go/internal/modget.(*resolver).resolveQueries(0xc00059a100, {0x166fac0, 0xc00002a0c0}, {0xc000010050, 0x1, 0x1ac4658})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1278 +0x1e5
cmd/go/internal/modget.runGet({0x166fac0, 0xc00002a0c0}, 0xc000026318, {0xc0000201d0, 0x1, 0x1})
        /usr/local/go/src/cmd/go/internal/modget/get.go:300 +0x374
main.invoke(0x1983c80, {0xc0000201c0, 0x2, 0x2})
        /usr/local/go/src/cmd/go/main.go:216 +0x2f6
main.main()
        /usr/local/go/src/cmd/go/main.go:173 +0x78e

The second time it actually was not crashing immediately but after a few seconds (looks like it downloaded the library this time) it still did.

Here is the commit in my project, on which I'm trying to run the above commands and it crashes.

What did you expect to see?

No crashes. If something is wrong with my modules or their dependencies, a comprehensible error.

What did you see instead?

See above.

[seankhliao]

cc @bcmills @matloob @jayconrod

[thomaspeugeot]

FYI, I met an identical issue.

thomaspeugeot@MacBook-Pro-de-Thomas gongsvg % go version
go version go1.17 darwin/amd64

to reproduce the issue:

git clone https://github.com/fullstack-lang/gongsvg
cd gongsvg
go get -d github.com/fullstack-lang/gongdoc@HEAD

panic

> Executing task: go get -d github.com/fullstack-lang/gongdoc@HEAD <

panic: internal error: can't find reason for requirement on gonum.org/v1/plot@v0.8.1

goroutine 1 [running]:
cmd/go/internal/modget.(*resolver).updateBuildList.func1({{0xc000814408, 0xc0003b6858}, {0xc0000b2da6, 0xc00028c480}})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1794 +0x10a
cmd/go/internal/modget.(*resolver).updateBuildList(0xc0002de300, {0x166fac0, 0xc0000b2000}, {0x0, 0x0, 0x0})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1799 +0x5bb
cmd/go/internal/modget.(*resolver).resolveQueries(0xc0002de300, {0x166fac0, 0xc0000b2000}, {0xc0000c0048, 0x1, 0x1ab7040})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1278 +0x1e5
cmd/go/internal/modget.runGet({0x166fac0, 0xc0000b2000}, 0xc0000da420, {0xc000098070, 0x1, 0x1})
        /usr/local/go/src/cmd/go/internal/modget/get.go:300 +0x374
main.invoke(0x1983c80, {0xc000098050, 0x3, 0x3})
        /usr/local/go/src/cmd/go/main.go:216 +0x2f6
main.main()
        /usr/local/go/src/cmd/go/main.go:173 +0x78e
The terminal process "zsh '-c', 'go get -d github.com/fullstack-lang/gongdoc@HEAD'" terminated with exit code: 2.

[bcmills]

@ash2k, thanks for the detailed steps to reproduce! I can reproduce the panic with a go command built from head in your repo at the named commit:

~/tmp/gitlab-agent$ go get gitlab.com/gitlab-org/labkit@fc0c07abad6a32b5b77e16289fcc72d38f8411ab
go: downloading gitlab.com/gitlab-org/labkit v1.9.1-0.20210826103322-fc0c07abad6a
panic: internal error: can't find reason for requirement on github.com/google/gofuzz@v1.1.0

goroutine 1 [running]:
cmd/go/internal/modget.(*resolver).updateBuildList.func1({{0xc00067f6e0, 0xc0002ee948}, {0xc0000b8d86, 0xc000684a18}})
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/modget/get.go:1810 +0x114
cmd/go/internal/modget.(*resolver).updateBuildList(0xc00052e100, {0xa82780, 0xc0000b8000}, {0x0, 0x0, 0x0})
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/modget/get.go:1815 +0x59b
cmd/go/internal/modget.(*resolver).resolveQueries(0xc00052e100, {0xa82780, 0xc0000b8000}, {0xc0000c6048, 0x1, 0x1010000004e4ea6})
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/modget/get.go:1293 +0x1e5
cmd/go/internal/modget.runGet({0xa82780, 0xc0000b8000}, 0xc0000e01e0, {0xc0000b2170, 0x1, 0x1})
        /usr/local/google/home/bcmills/go/src/cmd/go/internal/modget/get.go:300 +0x374
main.invoke(0xd9be40, {0xc0000b2160, 0x2, 0x2})
        /usr/local/google/home/bcmills/go/src/cmd/go/main.go:216 +0x2f6
main.main()
        /usr/local/google/home/bcmills/go/src/cmd/go/main.go:173 +0x78e

I'll investigate, and I should have a fix for either 1.17.1 or (possibly) 1.17.2.

[bcmills]

Somehow go get is ending up thinking that it needs github.com/google/gofuzz@v1.1.0. Adding v1.2.0 explicitly causes the command to succeed:

~/tmp/gitlab-agent$ go1.17 get gitlab.com/gitlab-org/labkit@fc0c07abad6a32b5b77e16289fcc72d38f8411ab github.com/google/gofuzz@v1.2.0
go get: upgraded github.com/Microsoft/go-winio v0.4.19 => v0.5.0
go get: upgraded github.com/google/gofuzz v1.1.0 => v1.2.0
go get: upgraded github.com/lightstep/lightstep-tracer-go v0.24.0 => v0.25.0
go get: upgraded github.com/philhofer/fwd v1.0.0 => v1.1.1
go get: upgraded github.com/uber/jaeger-client-go v2.27.0+incompatible => v2.29.1+incompatible
go get: upgraded gitlab.com/gitlab-org/labkit v1.9.0 => v1.9.1-0.20210826103322-fc0c07abad6a
go get: upgraded gopkg.in/DataDog/dd-trace-go.v1 v1.30.0 => v1.32.0

There really doesn't seem to be any reason to reject github.com/google/gofuzz v1.2.0, so the panic here really does indicate a bogus constraint. Still investigating how that bogus constraint gets locked in.

[bcmills]

Ah, I think I've got it!

gitlab.com/gitlab-org/labkit@fc0c07abad6a32b5b77e16289fcc72d38f8411ab specifies go1.17, so this interacts with module graph pruning.

We have an existing indirect dependency on gopkg.in/DataDog/dd-trace-go.v1:

~/tmp/gitlab-agent$ go mod why -m gopkg.in/DataDog/dd-trace-go.v1
# gopkg.in/DataDog/dd-trace-go.v1
gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v14/internal/tool/tracing
gitlab.com/gitlab-org/labkit/tracing/impl
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/opentracer

When we upgrade to fc0c07ab, we pull in an upgrade of dd-trace-go.v1 to v1.32.0, which is still on go 1.12. So upgrading labkit indirectly pulls in the new dependencies of dd-trace-go.v1, even though those dependencies are pruned out of the labkit dependency itself.

[ash2k]

@bcmills

so this interacts with module graph pruning.

Yes, I was trying to see if module graph pruning helps to reduce the number of dependencies. Thanks for investigating this!

[gopherbot]

Change https://golang.org/cl/347290 mentions this issue: cmd/go/internal/modload: scan dependencies of root paths when raising version limits in editRequirements

[bcmills]

@gopherbot, please backport to Go 1.17. This is a cmd/go panic due to an internal invariant violation, which may be confusing for end users to diagnose.