cmd/go: cannot go get when all versions are retracted

[hyangah]

What version of Go are you using (go version)?

$ go version
go version devel +f9108d168e Mon Nov 16 17:46:12 2020 -0500 darwin/amd64

Does this issue reproduce with the latest release?

Reproducible only with tip that understands 'retract' rules.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/hakim/Library/Caches/go-build"
GOENV="/Users/hakim/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/hakim/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/hakim/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/hakim/go_tip/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/hakim/go_tip/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="devel +f9108d168e Mon Nov 16 17:46:12 2020 -0500"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bw/6r6k9d113sv1_vvzk_1kfxbm001py5/T/go-build001158049=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Module name: github.com/hyangah/retracttest/a
Retracted all released versions [v1.0.0,v1.0.1] - as noted in go.mod@v1.0.1

What did you expect to see?

Retract only the released versions of a module. I thought I could let future users pick up the latest pseudo-versions after retraction - #41700.

What did you see instead?

I ended up making the entire module unavailable.

$ go get -x github.com/hyangah/retracttest/a@latest
# get https://proxy.golang.org/github.com/@v/list
# get https://proxy.golang.org/github.com/hyangah/retracttest/a/@v/list
# get https://proxy.golang.org/github.com/hyangah/@v/list
# get https://proxy.golang.org/github.com/hyangah/retracttest/@v/list
# get https://proxy.golang.org/github.com/@v/list: 410 Gone (0.283s)
# get https://proxy.golang.org/github.com/hyangah/retracttest/@v/list: 200 OK (0.288s)
# get https://proxy.golang.org/github.com/hyangah/retracttest/@latest
# get https://proxy.golang.org/github.com/hyangah/@v/list: 410 Gone (0.312s)
# get https://proxy.golang.org/github.com/hyangah/retracttest/a/@v/list: 200 OK (0.320s)
# get https://proxy.golang.org/github.com/hyangah/retracttest/a/@latest
# get https://proxy.golang.org/github.com/hyangah/retracttest/@latest: 200 OK (0.044s)
# get https://proxy.golang.org/github.com/hyangah/retracttest/a/@latest: 200 OK (0.144s)
go get: module github.com/hyangah/retracttest@latest found (v0.0.0-20201117000603-58d79ad1d7ed), but does not contain package github.com/hyangah/retracttest/a

go get with GOPROXY=direct works though, so it could be bad interaction with proxy.golang.org.

And go list result is confusing:

With default GOPROXY:

$ go list -m --versions -json github.com/hyangah/retracttest/a
{
	"Path": "github.com/hyangah/retracttest/a",
	"Version": "v1.0.1",
	"Time": "2020-11-16T22:37:32Z",
	"Dir": "/Users/hakim/go/pkg/mod/github.com/hyangah/retracttest/a@v1.0.1",
	"GoMod": "/Users/hakim/go/pkg/mod/cache/download/github.com/hyangah/retracttest/a/@v/v1.0.1.mod",
	"GoVersion": "1.15"
}

$ go list -m --versions -json github.com/hyangah/retracttest/a@latest
go list -m: module github.com/hyangah/retracttest/a: no matching versions for query "latest"

With GOPROXY=direct:

$ GOPROXY=direct go list -m --versions -json github.com/hyangah/retracttest/a
{
	"Path": "github.com/hyangah/retracttest/a",
	"Version": "v1.0.1",
	"Time": "2020-11-16T22:37:32Z",
	"Dir": "/Users/hakim/go/pkg/mod/github.com/hyangah/retracttest/a@v1.0.1",
	"GoMod": "/Users/hakim/go/pkg/mod/cache/download/github.com/hyangah/retracttest/a/@v/v1.0.1.mod",
	"GoVersion": "1.15"
}

$ GOPROXY=direct tipgo list -m --versions -json github.com/hyangah/retracttest/a@latest
{
	"Path": "github.com/hyangah/retracttest/a",
	"Version": "v0.0.0-20201117000603-58d79ad1d7ed",
	"Time": "2020-11-17T00:06:03Z",
	"Dir": "/Users/hakim/go/pkg/mod/github.com/hyangah/retracttest/a@v0.0.0-20201117000603-58d79ad1d7ed",
	"GoMod": "/Users/hakim/go/pkg/mod/cache/download/github.com/hyangah/retracttest/a/@v/v0.0.0-20201117000603-58d79ad1d7ed.mod",
	"GoVersion": "1.15"
}

cc @jayconrod @bcmills @matloob @heschik @katiehockman

[hyangah]

This is because proxy.golang.org isn't yet using go 1.16 that understands version retraction. According to #24031 (comment),

If no pre-release version is available, the go command either requests a version from the proxy's $module/@latest endpoint or derives a pseudo-version from the module repository's default branch (in direct mode).

And according to golang.org/ref/mod

$base/$module/@latest
Returns JSON-formatted metadata about the latest known version of a module in the same format as $base/$module/@v/$version.info. The latest version should be the version of the module that the go command should use if $base/$module/@v/list is empty or no listed version is suitable. This endpoint is optional, and module proxies are not required to implement it.

As go1.16 rolls out, proxy maintainers need to make sure either to implement the end point correctly, or let the go command fallback to direct (if users chose to do so) by responding with 404/410.

It would be nice if the go command's error message gets improved though.

[jayconrod]

So it looks like the proxy returns the highest release version from the @latest point, rather than a pseudo-version. Since v1.0.1 is retracted, the go command has no suitable version to use.

$ curl -L https://proxy.golang.org/github.com/hyangah/retracttest/a/@latest
{"Version":"v1.0.1","Time":"2020-11-16T22:37:32Z"}

The first error message is a bit obscure, but I think it's reasonable. The go command failed to find a suitable versions of the submodule, but it did find a version of the parent module, and that version didn't contain the package. You might see this if the author split the repo up into submodules, decided it was a bad idea, and retracted all versions of the submodules.

module github.com/hyangah/retracttest@latest found (v0.0.0-20201117000603-58d79ad1d7ed), but does not contain package github.com/hyangah/retracttest/a

The second error message is less helpful. This one should explicitly mention that all versions were retracted.

go list -m: module github.com/hyangah/retracttest/a: no matching versions for query "latest"

I think a couple changes to the proxy are needed though:

• When fetching the version list, the proxy should use go list -m -versions -retracted $mod. The new -retracted flag will include retracted versions in the list.
• For the @latest endpoint, the proxy should always return a pseudo-version. That can be used when all versions from the @v/list endpoint are retracted or excluded.
  • One way to get this version with GOPROXY=direct would be to run go list -m -versions -retracted $mod as above, then create a temporary go.mod file with an exclude directive for each version, then run go list -m $mod@latest. cmd/go: support @tip as an alias to "the latest development version" #42545 would provide an easier way to get that, but we won't be able to implement that for 1.16.

@hyangah Does this sound reasonable? Should I open a separate issue for the proxy changes?

[toothrot]

@hyangah @jayconrod Any update on this as a release blocking issue for Go 1.16?

[hyangah]

This is not a release blocker.
@jayconrod we have an internal issue open to update the proxy's go version when go1.16 is released.

[jayconrod]

Thanks @hyangah, I just commented on the internal issue. I'd still like the @latest endpoint to return a pseudo-version so the go command can fall back to that if everything in @v/list is retracted or excluded. That would also give us a way to implement #42545.

I'm still planning to update the wording for the error message for 1.16, but it's not a release blocker.

[hyangah]

@jayconrod I commented internally - the proxy returns whatever go command returns as @latest in GOPROXY=direct mode. Given that the proxy depends on the go command (go list -m, go mod download) to fetch modules, implementing the feature from the proxy side to help #42545 (a rarely used feature) seems weird.

[odeke-em]

Thanks for the reports and discussion @hyangah, @jayconrod, @toothrot, @bcmills! Given that this issue is a chicken and egg kind for closing out Go1.16 i.e. the proxy needs to be updated to Go1.16 for which it’ll then understand version retraction, yet this issue is marked for Go1.16, shall we mark this instead for Go1.17 or Backlog?

[jayconrod]

Let's move this to Go1.17. There's an internal issue listing the changes needed for the proxy (b/173606721). I still need to improve the wording of the error message though.

[hyangah]

I just noticed the 1.16rc1 now behaves differently and the retraction doesn't seem to work as expected even in the direct mode.

$ GO111MODULE=on GOPROXY=direct GOSUMDB=off GOPATH=/tmp /Users/hakim/sdk/go1.16rc1/bin/go  list -m -json --versions -x -v github.com/hyangah/retracttest/a
mkdir -p /tmp/pkg/mod/cache/vcs # git3 https://github.com/hyangah/retracttest
# lock /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e.lockmkdir -p /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e # git3 https://github.com/hyangah/retracttest
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git init --bare
0.037s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git init --bare
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git remote add origin -- https://github.com/hyangah/retracttest
0.008s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git remote add origin -- https://github.com/hyangah/retracttest
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git ls-remote -q origin
0.286s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git ls-remote -q origin
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git tag -l
0.006s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git tag -l
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' fc9f13e7bc5a89df7463770ed8fee012fde0ca45 --
0.009s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' fc9f13e7bc5a89df7463770ed8fee012fde0ca45 --
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git fetch -f --depth=1 origin refs/tags/a/v1.0.1:refs/tags/a/v1.0.1
0.343s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git fetch -f --depth=1 origin refs/tags/a/v1.0.1:refs/tags/a/v1.0.1
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' refs/tags/a/v1.0.1 --
0.006s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' refs/tags/a/v1.0.1 --
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git cat-file blob fc9f13e7bc5a89df7463770ed8fee012fde0ca45:a/go.mod
0.006s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git cat-file blob fc9f13e7bc5a89df7463770ed8fee012fde0ca45:a/go.mod
cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git cat-file blob fc9f13e7bc5a89df7463770ed8fee012fde0ca45:a/go.mod
0.008s # cd /tmp/pkg/mod/cache/vcs/fbf1b28828ad1177f688ff5d63aae461dda590b0b1e1c864f1fc7e8c5462053e; git cat-file blob fc9f13e7bc5a89df7463770ed8fee012fde0ca45:a/go.mod
{
        "Path": "github.com/hyangah/retracttest/a",
        "Version": "v1.0.1",
        "Time": "2020-11-16T22:37:32Z",
        "GoVersion": "1.15"
}

---

Edit: Maybe false alarm. Adding @latest picks up the pseudo version.

[hyangah]

proxy.golang.org uses 1.16 and now I see

$ curl -L https://proxy.golang.org/github.com/hyangah/retracttest/a/@latest
{"Version":"v0.0.0-20201117044135-c0d2dc84e45f","Time":"2020-11-17T04:41:35Z"}

$ go list -m --versions -json github.com/hyangah/retracttest/a@latest
{
        "Path": "github.com/hyangah/retracttest/a",
        "Version": "v0.0.0-20201117044135-c0d2dc84e45f",
        "Time": "2020-11-17T04:41:35Z",
        "GoVersion": "1.15"
}

@jayconrod is it ok to close this issue? Or do you want to keep this to track the error message improvement?

[jayconrod]

Let's keep this open to track the error message improvement.

[bcmills]

I don't remember what the error message improvement here was, but the core issue has been addressed.

If someone remembers or notices the bad error message to be fixed, we can file a separate issue for that.