proposal: x/website: have endpoint to query for the latest bugfix version

[mvdan]

Background

#34864 covers documenting the endpoint better. Right now, it's documented at https://godoc.org/golang.org/x/website/internal/dl. Listing the versions is very useful, so that endpoint is great.

However, there is one common operation that's not covered by the endpoint: showing the latest Go patch release for a major version like 1.12 or 1.13. This is useful to quickly download and install such a latest version, or to use it for a line like docker run golang:$(<query> 1.13.x) ..., and so on.

CI systems like Travis or GitHub Actions allow you to specify 1.13.x as a Go version in their configs. Internally, they query /dl?mode=json, and figure out what the latest version is. That's fine, because that software is pretty complex anyway, so a bit of JSON parsing is not terrible.

However, this gets harder when one is writing shell script one-liners, or portable scripts in general. There's just no way to do it in POSIX Shell or even Bash, let alone shells for other systems.

/VERSION?m=text almost satisfies this purpose, but it only shows the latest single release (not 1.12.x, 1.11.x, etc), and it sometimes lags behind even days after a newer release has gone out.

Proposed solution

Given that /VERSION is a static file and not an endpoint, I'm going to suggest that we extend the existing /dl endpoint instead. But this choice can be changed.

We don't want the result to be JSON, for the reasons laid out before. So the first step is adding ?mode=text alongside ?mode=json. Similar to /VERSION?m=text, it simply prints one version name per line, and nothing else. This is already useful by itself; one could figure out the absolute latest version via curl .../dl?mode=text | sed 1q, for example.

To support querying for the latest bugfix version for a particular "major" release like 1.12, we'd add another parameter latest: /dl?mode=text&latest=1.12.x. Given a string ${version}.x, it returns the latest bugfix release for that version. The version must be in the format ${number}.{number}, and the suffix .x must be present, to avoid ambiguity with 1.12 (actually 1.12.0).

Queries like /dl?mode=json&latest=1.12.x should also work, doing what you'd imagine. Instead of returning one line with just the version name, you'd get a JSON element with the structured data for the release in question.

/cc @dmitshur @myitcv @andybons

[mvdan]

Another option would be to rely more on the shell, and simply add mode=text. Then, the equivalent of curl .../dl?mode=text&latest=1.12.x would be something like curl /dl?mode=text | sed -n '/1\.12\./{p;q}'. However, I didn't include this suggestion because regexes are easy to mess up, and portability remains very difficult.

[dmitshur]

Thanks for filing this.

/VERSION?m=text almost satisfies this purpose, but it only shows the latest single release (not 1.12.x, 1.11.x, etc), and it sometimes lags behind even days after a newer release has gone out.

I agree the /VERSION endpoint is not a great long term solution. It can reliably answer the question of "what VERSION file does the golang.org website serve", but that cannot be reliably correlated with what versions of Go are released or tagged.

Since you haven't discussed it explicitly in the original issue, I'll copy what I wrote in #36886 (comment):

If you're looking for more up-to-date Go release information, I suggest using either the https://golang.org/dl/?mode=json endpoint, which is updated as soon as release downloads are available, or the maintner API (see usage example here), which is updated as soon as releases are cut (the tag is published).

I want to ask what you think about relying on the maintner API for this? Its ListGoReleases endpoint seems to return exactly the information you're looking for. The questions are:

• whether making an unauthenticated gRPC call is considered "convenient" enough for the needs you stated in the original issue
• whether the expectation of stability of the API based on the domain name is good enough (the golang.org vs maintner.golang.org domains may be interpreted differently)
• whether including a new minor release as soon as it is tagged is what you're looking for, or whether you want to know about it only when that minor release is completed (i.e., release artifacts such as downloads are published)

It's still an option to expose the same information from the golang.org/dl?mode=json API, which this issue is about investigating, but I wanted to ask a few of those questions first.

[mvdan]

whether making an unauthenticated gRPC call is considered "convenient" enough

I want to make use of this in scripts, so it should be plan HTTP+plaintext, just like the existing endpoint with mode=text. So a gRPC client won't do, I'm afraid.

whether the expectation of stability of the API based on the domain name is good enough

I'd prefer this to live under golang.org, simply because that's also where one downloads the releases from. It's also a bit more convenient (i.e. shorter) to use. Not a big problem, but if what we want is to consolidate all of this into the maintainer API, we should just deprecate the existing /dl REST endpoint.

Note that, as said before, the replacement needs to support REST and plaintext endpoints too, otherwise we'd be losing valuable features.

whether including a new minor release as soon as it is tagged is what you're looking for

I imagine a large amount of the use cases (including mine) involve using the release, i.e. downloading it. So yeah, the feature should show the latest fully released version.

[mvdan]

I haven't gotten much feedback, so I'm turning this into a proposal in hopes that it goes through the formal process soon. I did write the original issue almost like a small proposal, anyway.

[rsc]

It's worth noting that https://golang.org/dl/?mode=json has the information.

This also requires a developer to change their CI scripts for upgrading the latest Go release.

I guess the question is: do you want your CI system to break automatically the moment we release a new version of Go (if the CI system is fetching "latest"), or do you want to control when the update happens? In general we've encouraged the latter and discouraged references like "latest".

Given that "latest" is possible already with the JSON (and shell scripts can use jq), and that maybe "latest" shouldn't be made trivial to discourage these kinds of blind updates, maybe things are just right as is.

[Copied from #38150. Since this is the older issue, let's put the discussion here.]