Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go mod download -json should add 'Latest bool' #32239

Open
hyangah opened this issue May 24, 2019 · 11 comments

Comments

Projects
None yet
5 participants
@hyangah
Copy link
Contributor

commented May 24, 2019

go get <modulepath>@<version>' triggers GET $GOPROXY//@v/.{info, mod, zip}` queries when it talks with GOPROXY.

When it's the very first time the module proxy sees the version, the proxy may want to update its database and, if necessary, serve the newest version as the latest version of the module. Module owners may utilize this to tell the module proxy or Go Module Mirror as soon as the version becomes available in the origin.

But the proxy need to be careful because the the may be a commit or branch, and there is no guarantee that the newly fetched version is suitable as the latest version. The proxy may perform extra go list queries itself for every new version it observes, but it would be nice if the 'go mod download' command designed for the proxy provides this info in its output.

@rsc @bcmills @jayconrod @katiehockman @heschik

@hyangah hyangah added the modules label May 24, 2019

@julieqiu julieqiu added this to the Go1.13 milestone May 28, 2019

@bcmills bcmills modified the milestones: Go1.13, Go1.14 May 29, 2019

@bcmills

This comment has been minimized.

Copy link
Member

commented May 29, 2019

But the proxy need to be careful because the [version] may be a commit or branch, and there is no guarantee that the newly fetched version is suitable as the latest version.

It seems to me that that information can already be resolved from the go mod download output.

Specifically, we can examine the Version field:

  • If it is a pseudo-version (or other prerelease) derived from the latest version (or something older), then the proxy does not need to update latest.
  • If it is a pseudo-version derived from a release version after latest, then the proxy is probably missing that release (and can find it using go list -versions).
  • If it is a release version, then the proxy should update latest if (and only if) the new version is semantically higher than the existing latest version.

That leaves open the question of what to do if the proxy sees a pseudo-version or prerelease tag derived from something newer than latest, but those are likely rare anyway.

@hyangah

This comment has been minimized.

Copy link
Contributor Author

commented May 29, 2019

Specifically, we can examine the Version field:

If it is a pseudo-version (or other prerelease) derived from the latest version (or something older), then the proxy does not need to update latest.
If it is a pseudo-version derived from a release version after latest, then the proxy is probably missing that release (and can find it using go list -versions).
If it is a release version, then the proxy should update latest if (and only if) the new version is semantically higher than the existing latest version.

This is already too much of complexity to implement correctly outside go commands. Would be nice if it's done through a library (maintained to be consistent with how go command sees) or throug go mod download. And we'd like to avoid the run of go list -versions followed by every go mod download if possible (due to our internal architectural limitation :-().

@bcmills

This comment has been minimized.

Copy link
Member

commented May 29, 2019

Would be nice if it's done through a library (maintained to be consistent with how go command sees)

You mean like golang.org/x/mod/semver? 🙂

You could easily skip the “pseudo-version derived from a release version after latest” part, in which case you get:

if semver.Prerelease(m.Version) == "" && semver.Max(latest, m.Version) == m.Version {
	// Update latest.
	[…]
}

For the “derived from a release version after” case, perhaps we could add a semver.ReleaseBefore(v string) string, which would return the highest non-prerelease version (if any) immediately preceding v.

@hyangah

This comment has been minimized.

Copy link
Contributor Author

commented May 29, 2019

@bcmills, do you mean the cached latest version in the proxy's database by 'latest' in your code snippet? (again, we are looking for a solution that avoids extra run of 'go list -versions' or 'go mod download module@latest' - because we are running each command in an isolated sandbox currently).

I think the pseudo-version can be newer than the actual latest (in the origin) if the pseudo-version was derived from a development branch or non-default branch.

@bcmills

This comment has been minimized.

Copy link
Member

commented May 29, 2019

do you mean the cached latest version in the proxy's database by 'latest' in your code snippet?

Yes.

@bcmills

This comment has been minimized.

Copy link
Member

commented May 29, 2019

I think the pseudo-version can be newer than the actual latest (in the origin) if the pseudo-version was derived from a development branch or non-default branch.

Yes: the pseudo-version is calculated based on the highest tag from a parent commit, which may be the latest tag or an arbitrarily higher pre-release tag. (For example, the commit may be on the default branch after the latest release, or may be after a higher prerelease tag on a development branch, or so on.)

However, note that every pseudo-version is also semantically a prerelease version: hence the semver.Prerelease(m.Version) == "" check in the code snippet.

@hyangah

This comment has been minimized.

Copy link
Contributor Author

commented May 29, 2019

A bit more context behind this issue: we are investigating if it's possible to determine (without extra list query) whether the newly downloaded module version from go mod download is suitable for the 'latest' query when the origin doesn't have any released version tags. Only pseudo-versions are available so m.Version and latest in the code example are both pseudo-versions.

@bcmills

This comment has been minimized.

Copy link
Member

commented May 29, 2019

we are investigating if it's possible to determine (without extra list query) whether the newly downloaded module version from go mod download is suitable for the 'latest' query when the origin doesn't have any released version tags.

Ah, ok!

In that case, I think you need only one more function, IsPseudoVersion(v string) bool, plus the ability to iterate over the versions known to the proxy:

if semver.Max(latest, m.Version) == m.Version {
	allowed := true
	if module.IsPseudoVersion(m.Version) {
		for _, v := range cachedVersions {
			if !module.IsPseudoVersion(v) {
				allowed = false
				break
			}
		}
	} else if semver.Prerelease(m.Version) != "" {
		onlyPseudoVersions := true
		for _, v := range cachedVersions {
			if semver.Prerelease(m.Version) == "" {
				allowed = false
				break
			}
		}
	}
	if allowed {
		// Update latest.
		[…]
	}
}

That's admittedly a fair amount of code, but it's comparable to what we would have to add to cmd/go anyway. Moreover, within cmd/go we would need to do a significant amount of extra work to determine whether the requested version is latest in general, and I don't think we want to do that extra work for every invocation of go mod download -json.

@rsc

This comment has been minimized.

Copy link
Contributor

commented May 31, 2019

I don't believe you can derive latest-ness from the pseudo-version strings themselves. Whether something is latest depends, for Git, on what master is set to. Master can be changed arbitrarily in ways that don't respect semantic version ordering.

We should provide Latest bool in the go mod download output. Let the go command say for sure instead of making clients guess (or do a second invocation).

@rsc rsc changed the title cmd/go: include whether the downloaded version is the latest version or not in the 'go mod download -json' output cmd/go: go mod download should add 'Latest bool' to JSON output May 31, 2019

@rsc rsc changed the title cmd/go: go mod download should add 'Latest bool' to JSON output cmd/go: go mod download -json should add 'Latest bool' May 31, 2019

@rsc

This comment has been minimized.

Copy link
Contributor

commented May 31, 2019

I think this will require a little bit of thought and then very little actual coding. If that's the case, it would be good to get this into Go 1.13 so that proxies can use it before 1.14 comes out.

@rsc rsc modified the milestones: Go1.14, Go1.13 May 31, 2019

@rsc rsc added the release-blocker label May 31, 2019

@bcmills bcmills self-assigned this Jun 11, 2019

@gopherbot

This comment has been minimized.

Copy link

commented Jun 25, 2019

Change https://golang.org/cl/183841 mentions this issue: cmd/go: add a Latest field to the output of 'go mod download -json'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.