x/build: publish Go source code releases to the Checksum Database

[FiloSottile]

Compiled toolchains are published to the Checksum Database under the module golang.org/toolchain with versions like v0.0.1-go1.26.0.darwin-arm64, for use with GOTOOLCHAIN.

https://go.dev/doc/toolchain#download

It would be nice if the source zip files were also published to the sumdb, maybe simply as version v0.0.1-go1.26.0 etc.

For example, @AGWA explains he would have used them as a starting point for the toolchain reproducibility verification in https://www.agwa.name/blog/post/verifying_go_reproducible_builds.

Myself I would use it to show the source of standard library packages on https://pkg.geomys.org.

/cc @golang/release, @golang/command-line

[gabyhelp]

Related Issues

• release: provide sha256 checksums in machine readable format #14385 (closed)
• proposal: cmd/go: add GOSUMDBDIR to configure the location of downloaded checksum database files #59456
• Provide download for checksum of go releases. #68282 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[Raymon-del2]

"Interesting discussion on the checksum database integrity.

In developing the R.A.T.A. v3 protocol on Solana, we found that standard checksums created latency at scale. We moved to a 0xBF Parity Seal ((data ^ 254) % 256) to verify source integrity without the overhead of full SHA-256 hashes for every packet.

If x/build is hitting latency constraints with the current DB, the 'Parity Seal' approach might offer a lighter verification layer for the source releases.

Reference: We call this 'Sub-Atomic Information Density' (S.A.I.D.).
Docs: https://rata-said.vercel.app/"

[dmitshur]

Thanks for filing this feature request.

As background context, the golang.org/toolchain modules exist primarily to serve the internal needs of the cmd/go command and its https://go.dev/doc/toolchain functionality. They happen to also be publicly available, and there's no good reason not to also use them in ways that are generally intended to help the Go ecosystem (such as the examples you provided), but cmd/go needs are still a higher priority for deciding whether we make changes to those modules.

Prior work in #58659 involved both updating the release process to publish distpack-based toolchain modules for any new versions starting with 1.21rc1, as well as doing a one-off build of pre-1.21 toolchain modules. If we do want to add a new type, there's a question of whether it's done only for new releases, or retroactively as well (possibly only for 1.21+ after reproducibility via cross-compilation became viable; though reproducibility may be less relevant in this particular case). So potentially adding a new variant like this "source only" one will involve at least changing cmd/distpack itself and x/build/cmd/relui code that invokes distpack, while handling this for already-released Go versions would need additional work.

Given the each of the current toolchain modules contains both the source code and binaries for one Go port, this proposed "source only" variant sounds like mostly an optimization for cases where binaries aren't needed, but can otherwise be resolved by picking any arbitrary port and using its source files only. All ports should have identical source code, so regarding the starting point for verification, it still sounds like it should be possible to pick either ".linux-amd64" or any arbitrary or random variant and then also check that the source of all others matches it. (The release process uses this approach to create reproducible .src.tar.gz download files, despite cmd/distpack not having a source-only build mode.) Adding yet another module variant just adds to the list of .zips to potentially verify.

I discussed this a bit with @matloob and it doesn't seem like the benefits to doing something like this are worth the costs, but it would help to know if there are significant advantages enabled by this that I missed above. Thanks.

[FiloSottile]

We just realized another advantage of including the source artifacts in the sumdb: if all artifacts at go.dev/dl are in the sumdb, it makes it possible to use sumdb signed inclusion proofs in place of OpenPGP signatures (#38910).

I find this very compelling.