x/build: ensure that TryBot builders are using a released toolchain for GOROOT_BOOTSTRAP

[bcmills]

The OpenBSD builder is using an unreleased development build of the go toolchain:

buildlet$ ./go1.4/bin/go version
go version devel +f14067f3c1 Tue Oct 30 17:45:19 2018 +0000 openbsd/amd64

That builder also runs as a TryBot. That means:

1. Failures of the builder stand in the way of active Go development (such as cmd/go: change default of GO111MODULE from 'auto' to 'on' #30228).
2. Such failures are difficult to reproduce, and may be due to bugs fixed after Oct. 30.

Now that Go 1.12 is released, we should make sure that all of the TryBot builders are using a released go toolchain for their GOROOT_BOOTSTRAP.

CC @dmitshur @bradfitz

[bradfitz]

We no longer do releases of Go 1.4. Often times when Go 1.4 no longer bootstraps on a modern version of an OS, we need to use a tip or release-branch.go1.14 commit to get it bootstrapped.

Now that Go 1.12 is released,

What does Go 1.12 have to do with it? The bootstrap builder only needs to be Go 1.4-ish.

[bcmills]

It doesn't really matter what version the builder uses, as long as it is some version for which we can easily reproduce failures. A released Go 1.11 (if it works) or Go 1.12 should be fine.

The reason 1.12 is relevant is that it presumably supports OpenBSD — meaning that there is at least one supported release that could be used instead of the current mid-cycle development build.

[bradfitz]

Such failures are difficult to reproduce,

I've seen your OpenBSD failures keep occuring. It seems that reproducing it isn't the problem here. Presumably gomote will also fail in the same way. I assume you mean locally on Linux? Have you tried using that git rev as your local GOROOT_BOOTSTRAP version on Linux?

But yeah, I agree that the bootstrap tarball shouldn't be from some random revision. Something on release-branch.go1.4 seems preferable (to test our minimum supported version and keep us honest), but Go 1.4 didn't have the cross-compiling bootstrap.bash support for x/build/cmd/genbootstrap. So I guess we can just rebuild from release-branch.go1.12.

I have other things on my plate at the moment so feel free to work on this or we can just keep it open.

[bcmills]

Have you tried using that git rev as your local GOROOT_BOOTSTRAP version on Linux?

Yeah, and that failure mode is reproducible on Linux at that rev. (Which makes it even more frustrating, since any workaround would be specific to that unreleased version of the toolchain.)

[bradfitz]

Great. Sounds like an easy fix then, just re-running genbootstrap & upload a new tarball (with unique filename) & updating the URL in dashboard/builders.go. (Note that the OpenBSD ABI changed some time ago so there was a period where some binaries from certain Go versions wouldn't run on some OS versions, so it's safer to only make new tarballs with unique URLs so we can roll back.)

[gopherbot]

Change https://golang.org/cl/165747 mentions this issue: dashboard: set openbsd builders to use a Go 1.12 bootstrap toolchain

[4a6f656c]

The other option here would be to install the Go package that is available for the given OpenBSD release. That way you would be using a more recent version and one that is easily reproducible (e.g. install OpenBSD and run 'pkg_add go'). This also avoids any ABI issues with a Go 1.4 tarball.

[bcmills]

@4a6f656c, I would really rather we not rely on system-packaged versions of the toolchain.

I don't know about OpenBSD in particular, but packagers on some systems have been known to remove files or otherwise alter the package from the standard go distribution. We should test against the software that we provide, not against a version of that software with (potentially) arbitrary alterations and deletions.

[bcmills]

Remaining work for this issue is to audit the other builders to determine whether they're all using released versions.

[gopherbot]

Change https://golang.org/cl/170863 mentions this issue: cmd/gopherbot: CC triaged issues to owners