Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: cannot go get when all versions are retracted #42648

Open
hyangah opened this issue Nov 17, 2020 · 11 comments
Open

cmd/go: cannot go get when all versions are retracted #42648

hyangah opened this issue Nov 17, 2020 · 11 comments

Comments

@hyangah
Copy link
Contributor

@hyangah hyangah commented Nov 17, 2020

What version of Go are you using (go version)?

$ go version
go version devel +f9108d168e Mon Nov 16 17:46:12 2020 -0500 darwin/amd64

Does this issue reproduce with the latest release?

Reproducible only with tip that understands 'retract' rules.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/hakim/Library/Caches/go-build"
GOENV="/Users/hakim/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/hakim/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/hakim/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/hakim/go_tip/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/hakim/go_tip/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="devel +f9108d168e Mon Nov 16 17:46:12 2020 -0500"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bw/6r6k9d113sv1_vvzk_1kfxbm001py5/T/go-build001158049=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Module name: github.com/hyangah/retracttest/a
Retracted all released versions [v1.0.0,v1.0.1] - as noted in go.mod@v1.0.1

What did you expect to see?

Retract only the released versions of a module. I thought I could let future users pick up the latest pseudo-versions after retraction - #41700.

What did you see instead?

I ended up making the entire module unavailable.

$ go get -x github.com/hyangah/retracttest/a@latest
# get https://proxy.golang.org/github.com/@v/list
# get https://proxy.golang.org/github.com/hyangah/retracttest/a/@v/list
# get https://proxy.golang.org/github.com/hyangah/@v/list
# get https://proxy.golang.org/github.com/hyangah/retracttest/@v/list
# get https://proxy.golang.org/github.com/@v/list: 410 Gone (0.283s)
# get https://proxy.golang.org/github.com/hyangah/retracttest/@v/list: 200 OK (0.288s)
# get https://proxy.golang.org/github.com/hyangah/retracttest/@latest
# get https://proxy.golang.org/github.com/hyangah/@v/list: 410 Gone (0.312s)
# get https://proxy.golang.org/github.com/hyangah/retracttest/a/@v/list: 200 OK (0.320s)
# get https://proxy.golang.org/github.com/hyangah/retracttest/a/@latest
# get https://proxy.golang.org/github.com/hyangah/retracttest/@latest: 200 OK (0.044s)
# get https://proxy.golang.org/github.com/hyangah/retracttest/a/@latest: 200 OK (0.144s)
go get: module github.com/hyangah/retracttest@latest found (v0.0.0-20201117000603-58d79ad1d7ed), but does not contain package github.com/hyangah/retracttest/a

go get with GOPROXY=direct works though, so it could be bad interaction with proxy.golang.org.

And go list result is confusing:

With default GOPROXY:

$ go list -m --versions -json github.com/hyangah/retracttest/a
{
	"Path": "github.com/hyangah/retracttest/a",
	"Version": "v1.0.1",
	"Time": "2020-11-16T22:37:32Z",
	"Dir": "/Users/hakim/go/pkg/mod/github.com/hyangah/retracttest/a@v1.0.1",
	"GoMod": "/Users/hakim/go/pkg/mod/cache/download/github.com/hyangah/retracttest/a/@v/v1.0.1.mod",
	"GoVersion": "1.15"
}

$ go list -m --versions -json github.com/hyangah/retracttest/a@latest
go list -m: module github.com/hyangah/retracttest/a: no matching versions for query "latest"

With GOPROXY=direct:

$ GOPROXY=direct go list -m --versions -json github.com/hyangah/retracttest/a
{
	"Path": "github.com/hyangah/retracttest/a",
	"Version": "v1.0.1",
	"Time": "2020-11-16T22:37:32Z",
	"Dir": "/Users/hakim/go/pkg/mod/github.com/hyangah/retracttest/a@v1.0.1",
	"GoMod": "/Users/hakim/go/pkg/mod/cache/download/github.com/hyangah/retracttest/a/@v/v1.0.1.mod",
	"GoVersion": "1.15"
}

$ GOPROXY=direct tipgo list -m --versions -json github.com/hyangah/retracttest/a@latest
{
	"Path": "github.com/hyangah/retracttest/a",
	"Version": "v0.0.0-20201117000603-58d79ad1d7ed",
	"Time": "2020-11-17T00:06:03Z",
	"Dir": "/Users/hakim/go/pkg/mod/github.com/hyangah/retracttest/a@v0.0.0-20201117000603-58d79ad1d7ed",
	"GoMod": "/Users/hakim/go/pkg/mod/cache/download/github.com/hyangah/retracttest/a/@v/v0.0.0-20201117000603-58d79ad1d7ed.mod",
	"GoVersion": "1.15"
}

cc @jayconrod @bcmills @matloob @heschik @katiehockman

@hyangah
Copy link
Contributor Author

@hyangah hyangah commented Nov 17, 2020

This is because proxy.golang.org isn't yet using go 1.16 that understands version retraction. According to #24031 (comment),

If no pre-release version is available, the go command either requests a version from the proxy's $module/@latest endpoint or derives a pseudo-version from the module repository's default branch (in direct mode).

And according to golang.org/ref/mod

$base/$module/@latest
Returns JSON-formatted metadata about the latest known version of a module in the same format as $base/$module/@v/$version.info. The latest version should be the version of the module that the go command should use if $base/$module/@v/list is empty or no listed version is suitable. This endpoint is optional, and module proxies are not required to implement it.

As go1.16 rolls out, proxy maintainers need to make sure either to implement the end point correctly, or let the go command fallback to direct (if users chose to do so) by responding with 404/410.

It would be nice if the go command's error message gets improved though.

@jayconrod jayconrod self-assigned this Nov 17, 2020
@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Nov 17, 2020

So it looks like the proxy returns the highest release version from the @latest point, rather than a pseudo-version. Since v1.0.1 is retracted, the go command has no suitable version to use.

$ curl -L https://proxy.golang.org/github.com/hyangah/retracttest/a/@latest
{"Version":"v1.0.1","Time":"2020-11-16T22:37:32Z"}

The first error message is a bit obscure, but I think it's reasonable. The go command failed to find a suitable versions of the submodule, but it did find a version of the parent module, and that version didn't contain the package. You might see this if the author split the repo up into submodules, decided it was a bad idea, and retracted all versions of the submodules.

module github.com/hyangah/retracttest@latest found (v0.0.0-20201117000603-58d79ad1d7ed), but does not contain package github.com/hyangah/retracttest/a

The second error message is less helpful. This one should explicitly mention that all versions were retracted.

go list -m: module github.com/hyangah/retracttest/a: no matching versions for query "latest"

I think a couple changes to the proxy are needed though:

  • When fetching the version list, the proxy should use go list -m -versions -retracted $mod. The new -retracted flag will include retracted versions in the list.
  • For the @latest endpoint, the proxy should always return a pseudo-version. That can be used when all versions from the @v/list endpoint are retracted or excluded.
    • One way to get this version with GOPROXY=direct would be to run go list -m -versions -retracted $mod as above, then create a temporary go.mod file with an exclude directive for each version, then run go list -m $mod@latest. #42545 would provide an easier way to get that, but we won't be able to implement that for 1.16.

@hyangah Does this sound reasonable? Should I open a separate issue for the proxy changes?

@toothrot
Copy link
Contributor

@toothrot toothrot commented Jan 7, 2021

@hyangah @jayconrod Any update on this as a release blocking issue for Go 1.16?

@hyangah
Copy link
Contributor Author

@hyangah hyangah commented Jan 7, 2021

This is not a release blocker.
@jayconrod we have an internal issue open to update the proxy's go version when go1.16 is released.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jan 7, 2021

Thanks @hyangah, I just commented on the internal issue. I'd still like the @latest endpoint to return a pseudo-version so the go command can fall back to that if everything in @v/list is retracted or excluded. That would also give us a way to implement #42545.

I'm still planning to update the wording for the error message for 1.16, but it's not a release blocker.

@hyangah
Copy link
Contributor Author

@hyangah hyangah commented Jan 7, 2021

@jayconrod I commented internally - the proxy returns whatever go command returns as @latest in GOPROXY=direct mode. Given that the proxy depends on the go command (go list -m, go mod download) to fetch modules, implementing the feature from the proxy side to help #42545 (a rarely used feature) seems weird.

@odeke-em
Copy link
Member

@odeke-em odeke-em commented Feb 7, 2021

Thanks for the reports and discussion @hyangah, @jayconrod, @toothrot, @bcmills! Given that this issue is a chicken and egg kind for closing out Go1.16 i.e. the proxy needs to be updated to Go1.16 for which it’ll then understand version retraction, yet this issue is marked for Go1.16, shall we mark this instead for Go1.17 or Backlog?

@jayconrod jayconrod modified the milestones: Go1.16, Go1.17 Feb 8, 2021
@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Feb 8, 2021

Let's move this to Go1.17. There's an internal issue listing the changes needed for the proxy (b/173606721). I still need to improve the wording of the error message though.

@hyangah
Copy link
Contributor Author

@hyangah hyangah commented Feb 16, 2021

I just noticed the 1.16rc1 now behaves differently and the retraction doesn't seem to work as expected even in the direct mode.

$ GO111MODULE=on GOPROXY=direct GOSUMDB=off GOPATH=/tmp /Users/hakim/sdk/go1.16rc1/bin/go  list -m -json --versions -x -v github.com/hyangah/retracttest/a
mkdir -p /tmp/pkg/mod/cache/vcs # git3 https://github.com/hyangah/retracttest
# lock /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e.lockmkdir -p /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e # git3 https://github.com/hyangah/retracttest
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git init --bare
0.037s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git init --bare
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git remote add origin -- https://github.com/hyangah/retracttest
0.008s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git remote add origin -- https://github.com/hyangah/retracttest
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git ls-remote -q origin
0.286s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git ls-remote -q origin
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git tag -l
0.006s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git tag -l
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' fc9f13e7bc5a89df7463770ed8fee012fde0ca45 --
0.009s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' fc9f13e7bc5a89df7463770ed8fee012fde0ca45 --
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git fetch -f --depth=1 origin refs/tags/a/v1.0.1:refs/tags/a/v1.0.1
0.343s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git fetch -f --depth=1 origin refs/tags/a/v1.0.1:refs/tags/a/v1.0.1
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' refs/tags/a/v1.0.1 --
0.006s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' refs/tags/a/v1.0.1 --
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git cat-file blob fc9f13e7bc5a89df7463770ed8fee012fde0ca45:a/go.mod
0.006s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git cat-file blob fc9f13e7bc5a89df7463770ed8fee012fde0ca45:a/go.mod
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git cat-file blob fc9f13e7bc5a89df7463770ed8fee012fde0ca45:a/go.mod
0.008s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git cat-file blob fc9f13e7bc5a89df7463770ed8fee012fde0ca45:a/go.mod
{
        "Path": "github.com/hyangah/retracttest/a",
        "Version": "v1.0.1",
        "Time": "2020-11-16T22:37:32Z",
        "GoVersion": "1.15"
}

Edit: Maybe false alarm. Adding @latest picks up the pseudo version.

@hyangah
Copy link
Contributor Author

@hyangah hyangah commented Feb 19, 2021

proxy.golang.org uses 1.16 and now I see

$ curl -L https://proxy.golang.org/github.com/hyangah/retracttest/a/@latest
{"Version":"v0.0.0-20201117044135-c0d2dc84e45f","Time":"2020-11-17T04:41:35Z"}
$ go list -m --versions -json github.com/hyangah/retracttest/a@latest
{
        "Path": "github.com/hyangah/retracttest/a",
        "Version": "v0.0.0-20201117044135-c0d2dc84e45f",
        "Time": "2020-11-17T04:41:35Z",
        "GoVersion": "1.15"
}

@jayconrod is it ok to close this issue? Or do you want to keep this to track the error message improvement?

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Feb 19, 2021

Let's keep this open to track the error message improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants