Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: does not distinguish pseudo-versions from pre-release versions when both are included in module proxy @v/list endpoint #32715

Closed
dmitshur opened this issue Jun 20, 2019 · 5 comments
Assignees
Milestone

Comments

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Jun 20, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.6 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

It also happens on tip, I tried with go version devel +71a6a444 (1~ day old).

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/dmitshur/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/dmitshur/go"
GOPROXY="https://proxy.golang.org"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/3m/rg2zm24d1jg40wb48wr0hdjw00jwcj/T/go-build318521526=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Consider a repository with 3 commits and a pre-release tag named v0.5.0-alpha pointing to the middle commit. I've made such a test repository at https://github.com/dmitshur-test/modtest5. So, the module with the module path github.com/dmitshur-test/modtest5 has (at least) these three versions:

  1. v0.0.0-20190619020302-197a620e0c9a - pseudo-version for the 1st commit
  2. v0.5.0-alpha - pre-release version for the 2nd commit
  3. v0.5.0-alpha.0.20190619023908-3da23a9deb9e - pseduo-version for the 3rd commit
(Source.)
$ go list -m github.com/dmitshur-test/modtest5@197a620e0c9a7a44f03a73314220f8faccaf76bb
github.com/dmitshur-test/modtest5 v0.0.0-20190619020302-197a620e0c9a
$ go list -m github.com/dmitshur-test/modtest5@ac1e91846d66c96f83d3013692715ead23d978db
github.com/dmitshur-test/modtest5 v0.5.0-alpha
$ go list -m github.com/dmitshur-test/modtest5@3da23a9deb9e9f230e64fbae87befd59ffb46457
github.com/dmitshur-test/modtest5 v0.5.0-alpha.0.20190619023908-3da23a9deb9e

Based on the rules described in the 5th paragraph of https://golang.org/cmd/go/#hdr-Module_aware_go_get, since a pre-release tag exists, pseudo-versions are not considered when determining the module version that the module query latest resolves to. So, latest should be v0.5.0-alpha (pointing to 2nd commit) rather than a pseudo-version pointing to the 3rd commit.

This works when fetching directly:

$ go list -m -json github.com/dmitshur-test/modtest5@latest
{
	"Path": "github.com/dmitshur-test/modtest5",
	"Version": "v0.5.0-alpha",
	"Time": "2019-06-19T02:04:46Z"
}

The last paragraph in the module proxy protocol section of cmd/go documentation says:

Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy.

Now, if one downloads the 3 versions to a local module cache, then uses its resultant contents on disk as a module proxy implementation, the module query latest ends up incorrectly resolving to the pseduo-version v0.5.0-alpha.0.20190619023908-3da23a9deb9e instead of the pre-release version v0.5.0-alpha:

$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOPATH=$(mktemp -d)

$ go mod download github.com/dmitshur-test/modtest5@v0.0.0-20190619020302-197a620e0c9a
go: finding github.com/dmitshur-test/modtest5 v0.0.0-20190619020302-197a620e0c9a
$ go mod download github.com/dmitshur-test/modtest5@v0.5.0-alpha
go: finding github.com/dmitshur-test/modtest5 v0.5.0-alpha
$ go mod download github.com/dmitshur-test/modtest5@v0.5.0-alpha.0.20190619023908-3da23a9deb9e
go: finding github.com/dmitshur-test/modtest5 v0.5.0-alpha.0.20190619023908-3da23a9deb9e

$ cat $GOPATH/pkg/mod/cache/download/github.com/dmitshur-test/modtest5/@v/list
v0.0.0-20190619020302-197a620e0c9a
v0.5.0-alpha
v0.5.0-alpha.0.20190619023908-3da23a9deb9e

$ export GOPROXY="file://${GOPATH}/pkg/mod/cache/download"
$ export GOPATH=$(mktemp -d)

$ go list -m -json github.com/dmitshur-test/modtest5@latest
{
	"Path": "github.com/dmitshur-test/modtest5",
	"Version": "v0.5.0-alpha.0.20190619023908-3da23a9deb9e",
	"Time": "2019-06-19T02:39:08Z"
}

What did you expect to see?

I expected pre-release version to be consistently used regardless if fetching from direct or a module proxy:

$ GOPROXY=direct go list -m github.com/dmitshur-test/modtest5@latest
github.com/dmitshur-test/modtest5 v0.5.0-alpha
$ GOPROXY=<some proxy whose list includes pseudo-versions> go list -m github.com/dmitshur-test/modtest5@latest
github.com/dmitshur-test/modtest5 v0.5.0-alpha

What did you see instead?

A pseudo-version was treated as a pre-release version and used as latest when fetching via the module proxy protocol that includes pseudo-versions in the @v/list endpoint:

$ GOPROXY=direct go list -m github.com/dmitshur-test/modtest5@latest
github.com/dmitshur-test/modtest5 v0.5.0-alpha
$ GOPROXY=<some proxy whose list includes pseudo-versions> go list -m github.com/dmitshur-test/modtest5@latest
github.com/dmitshur-test/modtest5 v0.5.0-alpha.0.20190619023908-3da23a9deb9e

Based on my understanding, this is a minor problem that is unlikely to affect a lot of users. I'm reporting it anyway. This issue is similar/related to #32695.

I believe a solution could be... Either it should be clarified in the module proxy protocol specification that the @v/list endpoint must not include pseudo-versions (otherwise they get treated as if they were pre-release versions), or the go command should correctly distinguish pseudo-versions from pre-release versions when both are included in the @v/list output.

/cc @jayconrod @hyangah @bcmills

@dmitshur dmitshur added this to the Go1.14 milestone Jun 20, 2019
@bcmills bcmills added the NeedsFix label Jun 21, 2019
@bcmills bcmills self-assigned this Jun 21, 2019
@bcmills bcmills modified the milestones: Go1.14, Go1.13 Jun 21, 2019
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jun 21, 2019

Change https://golang.org/cl/183402 mentions this issue: cmd/go/internal/modfetch: filter pseudo-versions from proxy /list endpoints

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jun 21, 2019

Change https://golang.org/cl/183401 mentions this issue: cmd/go/testdata: update addmod.go to work with the mainline go command

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jun 21, 2019

Given that file trees including pseudo-versions already exist in the wild, I think it's simplest to filter them out within (*modfetch.ProxyRepo).Versions. That turns out to be fairly trivial.

gopherbot pushed a commit that referenced this issue Jun 21, 2019
Updates #32715

Change-Id: I4ea41d7bd1a2daeba8cebb283aaa096b6c3d4533
Reviewed-on: https://go-review.googlesource.com/c/go/+/183401
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jun 24, 2019

Change https://golang.org/cl/183619 mentions this issue: cmd/go/internal/modfetch: return structured errors from proxy operations

gopherbot pushed a commit that referenced this issue Jun 25, 2019
CL 181881 added structured error types for direct fetches.
Use those same structured errors to format proxy errors consistently.

Also ensure that an empty @v/list is treated as equivalent to the module
not existing at all.

Updates #27173
Updates #32715

Change-Id: I203fd8259bc4f28b3389745f1a1fde936b0fa24d
Reviewed-on: https://go-review.googlesource.com/c/go/+/183619
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@gopherbot gopherbot closed this in 1969005 Jun 25, 2019
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jun 25, 2019

Change https://golang.org/cl/183845 mentions this issue: cmd/go/internal/modfetch: halt proxy fallback if the proxy returns a non-404/410 response for @latest

gopherbot pushed a commit that referenced this issue Jun 26, 2019
…non-404/410 response for @latest

The @latest proxy endpoint is optional. If a proxy returns a 404 for
it, and returns an @v/list with no matching versions, then we should
allow module lookup to try other module paths. However, if the proxy
returns some other error (say, a 403 or 505), then the result of the
lookup is ambiguous, and we should report the actual error rather than
"no matching versions for query".

(This fix was prompted by discussion with Dmitri on CL 183619.)

Updates #32715
Updates #26334

Change-Id: I6d510a5ac24d48d9bc5037c3c747ac50695c663f
Reviewed-on: https://go-review.googlesource.com/c/go/+/183845
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.