release: provide sha256 checksums in machine readable format

[Scorpiion]

Hi,

Correct me if I'm wrong, but it seems it's not possible to get the checksums of Go downloads without actually parsing a whole HTML page?

My expected behavior to download Go and it's checksum would be:

wget https://storage.googleapis.com/golang/go1.6.linux-amd64.tar.gz
wget https://storage.googleapis.com/golang/go1.6.linux-amd64.tar.gz.sha256sum

But the checksum is not there, the only place I can find it is hardcoded into the HTML of the download page (probably autogenerated during a build I assume).

I would like to do this download from a script and that's why manually copying the checksum is a bit tedious.

Here is an example of how it's possible to download and verify Dart's SDK download:

# Example of how to download and verify the Dart SDK
mkdir /tmp/download
cd /tmp/download
wget https://storage.googleapis.com/dart-archive/channels/stable/release/1.14.2/sdk/dartsdk-linux-x64-release.zip
wget https://storage.googleapis.com/dart-archive/channels/stable/release/1.14.2/sdk/dartsdk-linux-x64-release.zip.sha256sum
sha256sum -c dartsdk-linux-x64-release.zip.sha256sum
cd -

Would it be possible to add so that checksums are served up in the golang storage bucket?

[dsnet]

This is somewhat related to #13220. Presumably some API to download the release binaries would also provide checksums for them.

[ianlancetaylor]

CC @adg @broady

[broady]

Would an API for golang.org/dl be sufficient? Perhaps some kind of JSON output?

[broady]

You can get a CRC32 and MD5 hash through the Google Storage API:
https://www.googleapis.com/storage/v1/b/golang/o/go1.6.linux-amd64.tar.gz

No SHA hash, though.

[adg]

We could just upload a .sha256 file alongside each binary, and then make the SHA256 on the downloads page be a link to that file.

[Scorpiion]

Hi, thanks for all the replies.

I think a quick and easy solution would be:

1. Enable directory listing on the bucket (this effectively allows the user to parse out what versions are available)
2. Upload a .sha256 file alongside each binary as @adg suggested.

To get a better idea here is a sample of how I parse out Dart versions currently from their storage bucket:

$ STABLE_VERSIONS=$(gsutil ls gs://dart-archive/channels/stable/release/1.*/sdk/dartsdk-linux-x64-release.zip | sed 's|gs://dart-archive/channels/stable/release/||g' | sed  's|/sdk/dartsdk-linux-x64-release.zip||g')
$ echo $STABLE_VERSIONS                                                                                                                                                                              

After you have that list it's easy to just add start and end strings for the final url's (if a sha256 upload exists). The start and end of url I think is okay to hardcode as they are unlikely to change and trivial to update.

Even better than the bucket listing would be a REST endpoint that simply returned the equivalent of $STABLE_VERSIONS above, that would indeed be convenient, it does not need to be that complicated, just a list of versions and the url pattern can be described on the download page.

As a reference, the manual SDK download page of Dart looks like this, the last section "Direct download URLs" is a nice adition I think (code for that page).

[gopherbot]

CL https://golang.org/cl/19783 mentions this issue.

[adg]

I made the bucket world-readable (so it can be listed) and the fix above modifies the release tool to upload a .sha256 file alongside each release binary.

[Scorpiion]

Great Andrew, I tried listing the versions and it works great. For people wanting to listing versions, I provide some samples below.

Here is a sample of how to listing all Linux 64 bit releases (with raw gs:// urls):

gsutil ls gs://golang | grep "linux-amd64"

Here is a sample of parsing out stable release version numbers:

gsutil ls gs://golang | grep "linux-amd64" | sed 's|gs://golang/go||g' | sed  's|.linux-amd64.tar.gz||g' | grep -v "beta" | grep -v "rc"

Here is a sample of getting download links for all stable releases (https:// urls):

gsutil ls gs://golang | grep "linux-amd64" | sed 's|gs://golang/|https://storage.googleapis.com/golang/|g' | grep -v "beta" | grep -v "rc"

When the .sha256 files have been uploaded I think it's okay to close this issue, leaving it open until then. Thanks again for the quick replies and fixes.