Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go/internal/modfetch: re-evaluate the undocumented proxy behavior in proxyRepo.latest() #32789

Open
hyangah opened this issue Jun 26, 2019 · 4 comments

Comments

@hyangah
Copy link
Contributor

@hyangah hyangah commented Jun 26, 2019

Forked from the discussion during the review of https://golang.org/cl/183402

  1. undocumented /@v/list format

proxyRepo.latest() assumes /@v/list could return a list of version, commit timestamp pairs. This is not documented in the proxy protocol doc, and I am not sure if there is any proxy that's following this format. Please reject the undocumented format or document it.

One of the benefit of preserving this feature is, when this proxyRepo.latest() is invoked, the commit timestamp info here is more reliable than the timestamp parsed from the pseudo-version like version string and helps avoiding an extra proxyRepo.Stat() call. On the other hand, the code path can be completely eliminated if proxies implement the /@latest endpoint, which is also not documented.

  1. whether to include pseudo-versions or not in /@v/list

The proxyRepo.latest() itself is also relying on part of undocumented behavior of /@v/list, which is supposed to include pseudo-versions. Before https://golang.org/cl/183402 (pre-1.13), we assumed it's not a good idea to include the pseudo versions in the /@v/list output and planned to include that in the proxy protocol spec.

https://go-review.googlesource.com/c/mod/+/176540/5/proxy/server.go#40

// List returns a list of tagged versions of the module identified by path.
// The versions should all be canonical semantic versions
// and formatted in a text listing, one per line.
// Pseudo-versions derived from untagged commits should be omitted.
// The go command exposes this list in 'go list -m -versions' output
// and also uses it to resolve wildcards like 'go get m@v1.2'.`

If the spec draft is valid, the proxyRepo.latest() implementation doesn't do anything but an extra network round trip for the proxies that follow the spec. If the go command can handle the list output including pseudo-versions, we may want to revisit the decision.

So, I propose either

  1. tighten the spec to require the /@latest endpoint. Then, remove the code that leads to the proxyRepo.latest(), or
  2. relax the /@v/list format and allow pseudo versions in its output, and document it.

Option 2) implies more code to maintain on cmd/go side. Option 1) implies breakage of proxies that doesn't implement the /@latest endpoint.

p.s. having the go command's proxyRepo.Versions ensure not to return pseudo versions is a good fix though.

@bcmills

This comment was marked as resolved.

Copy link
Member

@bcmills bcmills commented Jun 26, 2019

s/also not implemented/also not documented/?

@bcmills bcmills added this to the Go1.14 milestone Jun 26, 2019
@andybons andybons added the NeedsFix label Jun 28, 2019
@gopherbot gopherbot removed the NeedsFix label Aug 12, 2019
@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Aug 16, 2019

  1. whether to include pseudo-versions or not in /@v/list

As an observation, the last paragraph at https://golang.org/cmd/go/#hdr-Module_proxy_protocol says:

Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy.

If one looks inside the local module cache as produced by cmd/go, pseudo-versions are included in /@v/list output. E.g.:

$ cat $(go env GOPATH)/pkg/mod/cache/download/golang.org/x/text/@v/list
v0.3.0
v0.3.1-0.20180807135948-17ff2d5776d2
v0.3.1-0.20181227161524-e6919f6577db
v0.3.2
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Aug 19, 2019

Change https://golang.org/cl/190838 mentions this issue: cmd/go: document <module>/@latest endpoint in 'go help goproxy'

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Aug 19, 2019

  1. undocumented /@v/list format

I think we should confirm no proxies are using extra fields. If there are none, we should start rejecting them in 1.14.

Currently, it looks like this can be used to set timestamps. We only really need timestamps for ordering pseudo-versions, and the timestamp should match the timestamp encoded in the pseudo-version, so there's no need to specify it separately.

  1. whether to include pseudo-versions or not in /@v/list

CL 190838 updates go help goproxy to include /@latest. It also points out that /@v/list may include pseudo-versions.

Note that go1.13 treats /@v/list differently from go1.12 and earlier.

In go1.13, if /@v/list contains tagged versions and pseudo-versions, the pseudo-versions will only be considered if all tagged versions (including pre-release) are excluded.

In go1.12 and earlier, if /@v/list contains tagged versions and pseudo-versions, the pseudo-versions will be considered together with tagged pre-release versions (non-pre-release versions still take precedence).

So if a proxy stopped serving /@latest and instead included that version in /@v/list (if not already present), it would change behavior of go1.12 and earlier if that version is a pseudo-version newer than pre-release versions in /@v/list. I don't think that change would affect proxy.golang.org since a pseudo-version is only returned from /@latest if /@v/list is empty.

gopherbot pushed a commit that referenced this issue Sep 12, 2019
Updates #32789

Change-Id: Ie5e8e3b7b6a923aa9068c8af3ac8f081bd92c830
Reviewed-on: https://go-review.googlesource.com/c/go/+/190838
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.