cmd/go/internal/modfetch: re-evaluate the undocumented proxy behavior in proxyRepo.latest()

[hyangah]

Forked from the discussion during the review of https://golang.org/cl/183402

1. undocumented /@v/list format

proxyRepo.latest() assumes /@v/list could return a list of version, commit timestamp pairs. This is not documented in the proxy protocol doc, and I am not sure if there is any proxy that's following this format. Please reject the undocumented format or document it.

One of the benefit of preserving this feature is, when this proxyRepo.latest() is invoked, the commit timestamp info here is more reliable than the timestamp parsed from the pseudo-version like version string and helps avoiding an extra proxyRepo.Stat() call. On the other hand, the code path can be completely eliminated if proxies implement the /@latest endpoint, which is also not documented.

2. whether to include pseudo-versions or not in /@v/list

The proxyRepo.latest() itself is also relying on part of undocumented behavior of /@v/list, which is supposed to include pseudo-versions. Before https://golang.org/cl/183402 (pre-1.13), we assumed it's not a good idea to include the pseudo versions in the /@v/list output and planned to include that in the proxy protocol spec.

https://go-review.googlesource.com/c/mod/+/176540/5/proxy/server.go#40

// List returns a list of tagged versions of the module identified by path.
// The versions should all be canonical semantic versions
// and formatted in a text listing, one per line.
// Pseudo-versions derived from untagged commits should be omitted.
// The go command exposes this list in 'go list -m -versions' output
// and also uses it to resolve wildcards like 'go get m@v1.2'.`

If the spec draft is valid, the proxyRepo.latest() implementation doesn't do anything but an extra network round trip for the proxies that follow the spec. If the go command can handle the list output including pseudo-versions, we may want to revisit the decision.

So, I propose either

1. tighten the spec to require the /@latest endpoint. Then, remove the code that leads to the proxyRepo.latest(), or
2. relax the /@v/list format and allow pseudo versions in its output, and document it.

Option 2) implies more code to maintain on cmd/go side. Option 1) implies breakage of proxies that doesn't implement the /@latest endpoint.

p.s. having the go command's proxyRepo.Versions ensure not to return pseudo versions is a good fix though.

[bcmills]

s/also not implemented/also not documented/?

[dmitshur]

2. whether to include pseudo-versions or not in /@v/list

As an observation, the last paragraph at https://golang.org/cmd/go/#hdr-Module_proxy_protocol says:

Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy.

If one looks inside the local module cache as produced by cmd/go, pseudo-versions are included in /@v/list output. E.g.:

$ cat $(go env GOPATH)/pkg/mod/cache/download/golang.org/x/text/@v/list
v0.3.0
v0.3.1-0.20180807135948-17ff2d5776d2
v0.3.1-0.20181227161524-e6919f6577db
v0.3.2

[gopherbot]

Change https://golang.org/cl/190838 mentions this issue: cmd/go: document <module>/@latest endpoint in 'go help goproxy'

[jayconrod]

1. undocumented /@v/list format

I think we should confirm no proxies are using extra fields. If there are none, we should start rejecting them in 1.14.

Currently, it looks like this can be used to set timestamps. We only really need timestamps for ordering pseudo-versions, and the timestamp should match the timestamp encoded in the pseudo-version, so there's no need to specify it separately.

2. whether to include pseudo-versions or not in /@v/list

CL 190838 updates go help goproxy to include /@latest. It also points out that /@v/list may include pseudo-versions.

Note that go1.13 treats /@v/list differently from go1.12 and earlier.

In go1.13, if /@v/list contains tagged versions and pseudo-versions, the pseudo-versions will only be considered if all tagged versions (including pre-release) are excluded.

In go1.12 and earlier, if /@v/list contains tagged versions and pseudo-versions, the pseudo-versions will be considered together with tagged pre-release versions (non-pre-release versions still take precedence).

So if a proxy stopped serving /@latest and instead included that version in /@v/list (if not already present), it would change behavior of go1.12 and earlier if that version is a pseudo-version newer than pre-release versions in /@v/list. I don't think that change would affect proxy.golang.org since a pseudo-version is only returned from /@latest if /@v/list is empty.