Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: add optional /mirrored endpoint to the module proxy spec #35400

Open
katiehockman opened this issue Nov 6, 2019 · 7 comments
Labels
Milestone

Comments

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Nov 6, 2019

Users would benefit from more transparency around whether or not a specific module version is being temporarily cached in a proxy or whether it is being permanently mirrored. There are a number of reasons why a proxy may choose not to mirror something forever: licensing is one notable example.

The proposal would be to add an additional optional endpoint to the proxy spec (ie. go help goproxy), which proxies could implement if they choose to, which would give this information. For example:
https://proxy.golang.org/golang.org/x/text/@v/v0.3.2/mirrored
would return "true" or "false" as plaintext.

This is something we could pair with a utility in x/mod which would accept a go.sum file and indicate which versions aren't being permanently mirrored by any of the proxies listed in GOPROXY. That might help you decide to use a different version of the module, vendor that dependency, or encourage you to file an issue against the module if you see that a suitable license is missing, for example.

/cc @jayconrod @bcmills @heschik @hyangah @rsc

@katiehockman katiehockman added this to the Proposal milestone Nov 6, 2019
@bradfitz

This comment has been minimized.

Copy link
Member

@bradfitz bradfitz commented Nov 6, 2019

The Expires headers could/should(?) say the same thing.

e.g. Expires 1 week vs Expires 100 years.

@katiehockman

This comment has been minimized.

Copy link
Member Author

@katiehockman katiehockman commented Nov 6, 2019

In many cases there is a difference between when the response is considered "stale", ie. Expires, and when the underlying server may be unable to continue serving a zip. The common case would be if a proxy is using a CDN where the cached response provided by the CDN may be stale after a few hours but the proxy server intends to continue serving that zip for much longer.

@ankushchadha

This comment has been minimized.

Copy link

@ankushchadha ankushchadha commented Nov 6, 2019

Having module versions cached temporarily means non-deterministic builds for the users.

IMO availability is one of the fundamental requirements of a public goproxy. And this is true atleast in gocenter.io.

Also, what does it mean for a user who also uses a local goproxy that further points to a public goproxy? Should they selectively clean up the local goproxy's cache always given this new endpoint? Maybe let users decide what they want to consume based on the metadata provided by the goproxy.

@arschles

This comment has been minimized.

Copy link

@arschles arschles commented Nov 6, 2019

+1 to the above. Except for extenuating circumstances like DMCA takedowns etc..., why would a module not be stored (purposefully not using the term "cache" because it implies expiration) forever on proxy.golang.org?

If there were modules that proxies/mirrors might not or did not store, then as @ankushchadha said, builds become nondeterministic. One of the major apparent benefits of proxy.golang.org right now is that it enables deterministic builds.

Edit: the proxy enables deterministic builds

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Nov 6, 2019

@arschles, a module might not be stored if the proxy maintainer is not confident that the module's license permits it to be stored.

Builds in that case do not become “nondeterministic”: they may either succeed or fail, depending on whether the needed modules are available (locally or from any configured remote source), but if they succeed they will produce the same result as any other successful build.

@arschles

This comment has been minimized.

Copy link

@arschles arschles commented Nov 7, 2019

@bcmills understood, I agree that this feature may be useful for on-prem proxies.

I'm talking about this endpoint in the context of public, hosted proxies. It introduces the possibility that a host may cache modules, and if you get a false back, that module@version could expire at any time. As the developer of an application who relies heavily on proxy.golang.org (we use it to build github.com/gomods/athens), I would prefer that everything returns true at all times (exceptions being made for special cases)

@hyangah

This comment has been minimized.

Copy link
Contributor

@hyangah hyangah commented Nov 7, 2019

I agree we need a way to tell users what proxy.golang.org will do about a specific module version. But I am not 100% sure about whether this endpoint belongs to the proxy protocol - at this moment, it seems too specific to proxy.golang.org.

It will sound more convincing if there are proxies other than proxy.golang.org that would utilize this new endpoint in a meaningful way.

The endpoint doesn't make much sense for enterprise and private proxies.

gocenter.io is trying to mirror everything once it decides to serve a module version. Most of other public proxies I've seen didn't make any official commitment about their data retention policy. Can other public proxy owners chime in?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.