Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 'go get […]/v2@v2.X.Y' fails when the go.mod file for […]/v2 is at the repo root #34383

Closed
abraithwaite opened this issue Sep 18, 2019 · 14 comments
Assignees
Milestone

Comments

@abraithwaite
Copy link
Contributor

@abraithwaite abraithwaite commented Sep 18, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/abraithwaite/.cache/go-build"
GOENV="/home/abraithwaite/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/abraithwaite/Projects/golang"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build983177457=/tmp/go-build -gno-record-gcc-switches"

What did you do?

We were upgrading a library (github.com/segmentio/events) to match proper major version module semantics (we already had some v2 release tags from before modules), and we kept running into the following when trying to go get the module.

GO111MODULE=on GOPROXY=direct go get -u github.com/segmentio/events/v2@v2.3.2
go: finding github.com/segmentio/events v2.3.2
go get github.com/segmentio/events/v2@v2.3.2: github.com/segmentio/events@v2.3.2: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

What did you expect to see?

We expected to see the module fetched without an error.

What did you see instead?

GO111MODULE=on GOPROXY=direct go get -u github.com/segmentio/events/v2@v2.3.2
go: finding github.com/segmentio/events v2.3.2
go get github.com/segmentio/events/v2@v2.3.2: github.com/segmentio/events@v2.3.2: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

We're aware that the other v2 tags are invalid: that is they don't have a correct go.mod file.

When we remove the v2 branch from the repository, go get works again without issues. We're skipping the module proxy and doing this with a clean modules cache. We expect that the error is a bit more clear at the very least. At the best, it ignores the v2 branch entirely and just uses the tag specified.

We're leaving the v2 branch in tact for now so that others can test and debug this issue.

Also interestingly, doing go get without specifying the exact tag works fine:

16:58:46 $ GO111MODULE=on GOPROXY=direct go get -u github.com/segmentio/events/v2
go: finding golang.org/x/sys latest
16:59:14 $ cat go.mod
module github.com/segmentio/blah

go 1.13

require github.com/segmentio/events/v2 v2.3.2 // indirect
@thepudds thepudds changed the title go mod v2.* tags conflict with v2 branch causing go get to fail cmd/go: go mod v2.* tags conflict with v2 branch causing go get to fail Sep 19, 2019
@abraithwaite

This comment has been minimized.

Copy link
Contributor Author

@abraithwaite abraithwaite commented Sep 23, 2019

Seeing a similar issue with our stats package:

https://gist.github.com/abraithwaite/15a0ffd0021b4539793bc0ceb1dc3bc1

@abraithwaite

This comment has been minimized.

Copy link
Contributor Author

@abraithwaite abraithwaite commented Sep 23, 2019

This repo can consistently break the command:
go get github.com/abraithwaite/broken/v2@v2.1.0

When the repo you run this from is lacking the broken module in it's go.mod, this results in:

16:37:38 $ GO111MODULE=on GOPROXY=direct go get 'github.com/abraithwaite/broken/v2@v2.1.0'
go: finding github.com/abraithwaite/broken v2.1.0
go: finding github.com/abraithwaite/broken/v2 v2.1.0
go: downloading github.com/abraithwaite/broken/v2 v2.1.0
go: extracting github.com/abraithwaite/broken/v2 v2.1.0
go get github.com/abraithwaite/broken/v2@v2.1.0: github.com/abraithwaite/broken@v2.1.0: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

It also fails to update the go.mod file with any version of the broken module whatsoever:

16:38:00 $ cat go.mod
module example.com/test

go 1.13
@abraithwaite

This comment has been minimized.

Copy link
Contributor Author

@abraithwaite abraithwaite commented Sep 23, 2019

For my own sanity, I checked to see if this issue could be reproduced for modules which have a "correct" git history. that is: the go.mod package directive has been updated to include the vX suffix throughout this history, for tags that reference a major version. It does still trigger the bug.

This repo has the correct v2 suffix in the go.mod file for the v2.0.0 and v2.1.0 tags and it still triggers this issue.

@bcmills bcmills self-assigned this Sep 24, 2019
@bcmills bcmills added this to the Go1.14 milestone Sep 24, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 24, 2019

@gopherbot, please backport to 1.13. This is a regression, and the workaround (editing the go.mod file instead of using go get) does not work for downgrading.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Sep 24, 2019

Backport issue(s) opened: #34497 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 24, 2019

This turns out to be closely related to #34432, in that we should notice the mismatched module path and treat that as a “not found” error rather than a hard error.

@abraithwaite

This comment has been minimized.

Copy link
Contributor Author

@abraithwaite abraithwaite commented Sep 24, 2019

This turns out to be closely related to #34432, in that we should notice the mismatched module path and treat that as a “not found” error rather than a hard error.

Ah, so it's looking for the subdirectory erroneously? None of these repos that I've created/used have a vX subdirectory in their tree. Only git tags.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 24, 2019

It's checking whether github.com/segmentio/events@v2.3.2 contains a /v2 subpackage, and erroring out because v2.3.2 is not a valid version for github.com/segmentio/events.

Instead, it should notice that the module path doesn't match and ignore the repo.

(As a smaller, shorter-term fix, we could apply the same solution as for #34094: if we've already found the /v2 module, simply ignore the error for the module with the shorter path.)

@abraithwaite

This comment has been minimized.

Copy link
Contributor Author

@abraithwaite abraithwaite commented Sep 24, 2019

(As a smaller, shorter-term fix, we could apply the same solution as for #34094: if we've already found the /v2 module, simply ignore the error for the module with the shorter path.)

I think that makes sense anyway. As a consumer, I'm surprised that the root module is even searched, but maybe that has something to do with the fact that there isn't a subdirectory for the module?

Also, by "ignore the repo" you mean that it will skip github.com/segmentio/events and only scan github.com/segmentio/events/v2?

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Sep 24, 2019

Change https://golang.org/cl/197059 mentions this issue: cmd/go: suppress errors in package-to-module queries if the package is already found

@bcmills bcmills changed the title cmd/go: go mod v2.* tags conflict with v2 branch causing go get to fail cmd/go: 'go get […]/v2@v2.X.Y' fails when the go.mod file for […]/v2 is at the repo root Sep 24, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 24, 2019

Also, by "ignore the repo" you mean that it will skip github.com/segmentio/events and only scan github.com/segmentio/events/v2?

Yes. It should interpret the mismatched path as meaning “the module github.com/segmentio/events does not exist at the requested version”.

gopherbot pushed a commit that referenced this issue Sep 24, 2019
…s already found

In CL 173017, I changed the package-to-module query logic to query all
possible module paths in parallel in order to reduce latency. (For
long package paths, most such paths will not exist and will fail with
little overhead.)

The module resolution algorithm treats various kinds of non-existence
as “soft errors”, to be reported only if package resolution fails, but
treats any remaining errors as hard errors that should fail the query.

Unfortunately, that interacted badly with the +incompatible version
validation added in CL 181881, causing a regression in the 'direct'
fetch path for modules using the “major branch” layout¹ with a post-v1
version on the repository's default branch. Because we did not
interpret a mismatched module path as “no such module”, a go.mod file
specifying the path 'example.com/foo/v2' would cause the search for
module 'example.com/foo' to error out. (That regression was not caught
ahead of time due to a lack of test coverage for 'go get' on a package
within a /vN module.)

The promotion of hard errors during parallel search also made the 'go'
command less tolerant of servers that advertise 'go-import' tags for
nonexistent repositories. CL 194561 mitigated that problem for HTTP
servers that return code 404 or 410 for a nonexistent repository, but
unfortunately a few servers in common use (notably GitLab and
pre-1.9.3 releases of Gitea) do not.

This change mitigates both of those failure modes by ignoring
“miscellaneous” errors from shorter module paths if the requested
package pattern was successfully matched against a module with a
longer path.

¹https://research.swtch.com/vgo-module#from_repository_to_modules

Updates #34383
Updates #34094

Change-Id: If37dc422e973eba13f3a3aeb68bc7b96e2d7f73d
Reviewed-on: https://go-review.googlesource.com/c/go/+/197059
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Sep 24, 2019

Change https://golang.org/cl/197063 mentions this issue: [release-branch.go1.13] cmd/go: suppress errors in package-to-module queries if the package is already found

gopherbot pushed a commit that referenced this issue Sep 25, 2019
…queries if the package is already found

In CL 173017, I changed the package-to-module query logic to query all
possible module paths in parallel in order to reduce latency. (For
long package paths, most such paths will not exist and will fail with
little overhead.)

The module resolution algorithm treats various kinds of non-existence
as “soft errors”, to be reported only if package resolution fails, but
treats any remaining errors as hard errors that should fail the query.

Unfortunately, that interacted badly with the +incompatible version
validation added in CL 181881, causing a regression in the 'direct'
fetch path for modules using the “major branch” layout¹ with a post-v1
version on the repository's default branch. Because we did not
interpret a mismatched module path as “no such module”, a go.mod file
specifying the path 'example.com/foo/v2' would cause the search for
module 'example.com/foo' to error out. (That regression was not caught
ahead of time due to a lack of test coverage for 'go get' on a package
within a /vN module.)

The promotion of hard errors during parallel search also made the 'go'
command less tolerant of servers that advertise 'go-import' tags for
nonexistent repositories. CL 194561 mitigated that problem for HTTP
servers that return code 404 or 410 for a nonexistent repository, but
unfortunately a few servers in common use (notably GitLab and
pre-1.9.3 releases of Gitea) do not.

This change mitigates both of those failure modes by ignoring
“miscellaneous” errors from shorter module paths if the requested
package pattern was successfully matched against a module with a
longer path.

¹https://research.swtch.com/vgo-module#from_repository_to_modules

Updates #34383
Updates #34094
Fixes #34497
Fixes #34215

Change-Id: If37dc422e973eba13f3a3aeb68bc7b96e2d7f73d
Reviewed-on: https://go-review.googlesource.com/c/go/+/197059
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
(cherry picked from commit a3426f2)
Reviewed-on: https://go-review.googlesource.com/c/go/+/197063
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 25, 2019

This should be working using gotip, and with go1.13.2 whenever that is released. (Note that go1.13.1 is a security-only release and thus will not contain the fix for this non-security issue.)

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 8, 2019

Change https://golang.org/cl/199839 mentions this issue: cmd/go: suppress more errors in package-to-module loading

gopherbot pushed a commit that referenced this issue Oct 9, 2019
In CL 197059, I suppressed errors if the target package was already found.
However, that does not cover the case of passing a '/v2' module path to
'go get' when the module does not contain a package at its root.

This CL is a minimal fix for that case, intended to be backportable to 1.13.

(Longer term, I intend to rework the version-validation check to treat
all mismatched paths as ErrNotExist.)

Fixes #34746
Updates #34383

Change-Id: Ia963c2ea00fae424812b8f46a4d6c2c668252147
Reviewed-on: https://go-review.googlesource.com/c/go/+/199839
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.