build: adopt Go 1.17 as bootstrap toolchain for Go 1.19

[rsc]

I propose that starting with the Go 1.18 dev cycle, we require Go 1.16 as the Go bootstrap version (instead of Go 1.4).

When we switched to writing the Go compiler in itself, I wrote two proposals related to the bootstrap process.

The original proposal, https://golang.org/s/go13compiler (December 2013), was about the overall process of converting the compiler to Go.
In that proposal, I wrote:

With a Go compiler written in Go, there must be a plan for bootstrapping from scratch. The rule we plan to adopt is that the Go 1.3 compiler must compile using Go 1.2, Go 1.4 must compile using Go 1.3, and so on. Then there is a clear path to generating current binaries: build the Go 1.2 toolchain (written in C), use it to build the Go 1.3 toolchain, and so on. There will be a shell script to do this; it will take CPU time but not human time. The bootstrapping only needs to be done once per machine; the Go 1.x binaries can be kept in a known location and reused each time all.bash is run during the development of Go 1.(x+1).

Obviously, this bootstrapping path scales poorly over time. Before too many releases have gone by, it may make sense to write a back end for the compiler that generates C code. The code need not be efficient or readable, just correct. That C version would be checked in, just as today we check in the y.tab.c file generated by yacc. The bootstrap sequence would invoke gcc on that C code to build a bootstrap compiler, and the bootstrap compiler would be used to build the real compiler. Like in the other scheme, the bootstrap compiler binary can be kept in a known location and reused (not rebuilt) each time all.bash is run.

This was all fairly hypothetical, and I certainly no longer believe it makes any sense to write a C back end for the Go compiler. (In fact, for the windows/arm64 port I did, I now have a working Go toolchain but still don't know what C compiler I'm supposed to use!)

The final proposal, https://golang.org/s/go15bootstrap (January 2015), simplified the process from an iterative one to hard-coding Go 1.4 as the bootstrap toolchain:

To build Go 1.x, for x ≥ 5, it will be necessary to have Go 1.4 (or newer) installed already, in $GOROOT_BOOTSTRAP. The default value of $GOROOT_BOOTSTRAP is $HOME/go1.4. In general we'll keep using Go 1.4 as the bootstrap base version for as long as possible. The toolchain proper (compiler, assemblers, linkers) will need to be buildable with Go 1.4, whether by restricting their feature use to what is in Go 1.4 or by using build tags.

This was an important simplification, especially for people packaging Go for other systems. That decision has served us well.

But it has now been over six years since Go 1.4. Much has happened in the world of Go, and many bugs have been fixed. Many of the systems Go runs on today aren't supported by Go 1.4 (including darwin/arm64 for M1 Macs). Those are using newer toolchains to bootstrap, and the other systems could too. At a higher level, Go is far more mature and widely available now than it was in the Go 1.4 era. There are tons of available binary distributions to use for bootstrapping.

I propose that Go 1.17 be the last version of Go requiring Go 1.4 for bootstrapping, and that Go 1.18 require Go 1.16 for bootstrapping.
Go 1.16 would remain the bootstrap version for the next few years at least.

Why not Go 1.15?

• Go 1.16 added //go:build support. We don't want the compiler to have to keep using +build even once the Go world has moved on. (Edit: this is only half true: see #44505 (comment).)

Why not Go 1.17?

• There's no compelling feature being added in Go 1.17.
• Using Go 1.16 as the new bootstrap toolchain gives packagers the chance to test the changes they'd need to make using Go 1.17,
  which would still work with Go 1.4.
• Go 1.17 will be the first release to use the new register-based ABI, which will likely have some lingering bugs that take a while to shake out.
  (See next section.)

Why not Go 1.18?

• Go 1.18 will contain the initial implementation of generics, which is going to require many many new lines of code, and certainly some of them will contain subtle bugs. If we adopt Go 1.18 as the bootstrap toolchain, we will be stuck working around those bugs for many years to come. Better to lock in the last release before all that churn as the version that will need to hold up for the long term.

Why not a quickly rolling version?

As noted above, I think it has served us well to have a fixed version required to build Go, as opposed to an automatically sliding version as originally envisioned. Packagers benefit from not having to update their package-building scripts to provide a different environment to each new Go release.

At the same time, bumping the version forward every five years or so lets us take advantage of newer Go capabilities and ports and to let us retire old compatibility shims (standard library packages like sort are carrying various +build'ed files to keep them building with Go 1.4). Modern C compilers are not written in pre-ANSI C.

What about a slow-rolling version?

That's essentially what this proposal would establish as our practice, although without a specific timeline.

The next obvious entry in the sequence after Go 1.4 and Go 1.16 is Go 1.256, followed by Go 1.65536.
(Or perhaps that is not quite the right pattern to establish.)

Using dates instead, assuming we switch to Go 1.16 in Go 1.18 (Feb 2022), it seems reasonable to me to revisit the bootstrap version four years later, which at our current release cycle would mean using Go 1.24 (Feb 2025) for Go 1.26 (Feb 2026).

But this proposal is not about establishing the sequence, which would depend on many other factors.

It is only about picking Go 1.16 as the bootstrap version starting in Go 1.18.

[rsc]

Actually, I confused myself while writing this.
Go 1.16 does not know how to read //go:build lines,
but it does know to reject files that contain //go:build without // +build.
And all future gofmts will know to sync (≠ insert) // +build lines alongside //go:build
So that's enough for the bootstrap toolchain.

[mdempsky]

Being able to assume a recent compiler for bootstrapping would be really nice for not having to remember all the quirks of Go 1.4, or needing to worry about working around long-fixed issues.

FWIW, OpenJDK appears to require the immediately previous release for bootstrapping (see https://openjdk.java.net/groups/build/doc/building.html, "Build JDK Requirements"), and also has a 6-month release cycle (https://en.wikipedia.org/wiki/OpenJDK#OpenJDK_versions), but building OpenJDK takes way longer than running make.bash does. I don't think we need to be nearly as aggressive at uprev'ing Go bootstrap versions, but I think it at least suggests end users could tolerate another uprev before 1.256. :)

[beoran]

This is a good idea, however, perhaps it would make sense to additionaly declare that the bootstrap version of Go is also going to be the long term support or "LTS" version, which keeps receiving backports of bugfixes for 4 years (or whatever the support period for the bootstrap version is going to be). In some organizations and for some developments, having a stable LTS version that keeps receiving bug fixes is extremely valuable.

[tianon]

FWIW, the level of aggressive bootstrap version bumps OpenJDK does are a big hurdle towards having recent OpenJDK packaged for backported distribution releases (and require a lot of extra coordination / "fancy footwork" from maintainers to coordinate or "hack" around).

As a concrete example for Go, this could make adding new architectures more difficult for a distribution until/unless gccgo gets updated to 1.16 (because gccgo is often used as a bootstrap-builder in order to onboard new architectures without having to do something drastic like uploading an old version of Go with the newer version number and "stepping" upwards until it's supported, which is really disruptive for the other architecture too -- the other alternative is cross-building, which has challenges of its own).

IMO this would be easier to understand/stomach if the proposed version were one that's already supported by at least the most recent gccgo release (which I believe in GCC 10 is 1.14?)

[ianlancetaylor]

Just a note that GCC 11 will support Go 1.16.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[zephyr]

This should be mentioned in the release notes for Go 1.18.

And maybe even in the release notes for Go 1.17, if still possible? (since the removal of support for old macOS versions is usually noted one release note in advance)

[rsc]

We missed implementing this for Go 1.18.
The builders are not ready and the release team has too much other work to do.

We will shoot for bumping the minimum bootstrap toolchain in Go 1.19 instead.
Given that the bump is pushed forward a cycle,
I suggest that we also push the minimum release forward a cycle:
Go 1.19 would require Go 1.17, not Go 1.16.

(The main concern with Go 1.17 was the stability of the register ABI, but that has been rock solid.)

Thoughts? Concerns? Thanks.

[gopherbot]

Change https://golang.org/cl/355690 mentions this issue: cmd/link/internal/ld: don't use linkname before 1.12

[gopherbot]

Change https://golang.org/cl/369914 mentions this issue: build: for default bootstrap, use Go 1.17 if present, falling back to Go 1.4

[fzipp]

Building from source no longer works for me after this change:

src % ./all.bash 
Building Go cmd/dist using /usr/local/go. (go1.17.4 darwin/arm64)
Building Go toolchain1 using /Users/frederik/go1.4.
go tool dist: FAILED: /Users/frederik/go1.4/bin/go install -gcflags=-l -tags=math_big_pure_go compiler_bootstrap bootstrap/cmd/...: fork/exec /Users/frederik/go1.4/bin/go: no such file or directory

I have Go 1.17.4 installed under /usr/local/go, but it still looks for go1.4 in my home directory.

[zikaeroh]

I'm experiencing the same thing on Arch (where I bootstrap with what's in $PATH, the system install).

Skimming the CL, make.bash no longer exports GOROOT_BOOTRSTRAP, so it's not set when dist is run (which then uses a different logic that only looks for go1.17 and go1.4).

Adding export GOROOT_BOOTSTRAP after the new code that checks $HOME/sdk/go1.17 and $HOME/go1.17 fixes the issue.

[gopherbot]

Change https://golang.org/cl/370138 mentions this issue: Revert "build: for default bootstrap, use Go 1.17 if present, falling back to Go 1.4"

[prattmic]

I am also seeing failures in a Google Cloud Build environment which effectively does apt-get update && apt-get install -y golang git && ./make.bash in an Ubuntu container.

[fsouza]

I'm experiencing the same thing on Arch (where I bootstrap with what's in $PATH, the system install).

Skimming the CL, make.bash no longer exports GOROOT_BOOTRSTRAP, so it's not set when dist is run (which then uses a different logic that only looks for go1.17 and go1.4).

Adding export GOROOT_BOOTSTRAP after the new code that checks $HOME/sdk/go1.17 and $HOME/go1.17 fixes the issue.

Yeah I did that on my side too. Perhaps GOROOT_BOOTSTRAP should default to go env GOROOT? (assuming go is on the PATH)

[zikaeroh]

I'm pretty sure it was just an oversight (as the new code "reassigns" the variable to set it up); the behavior you're describing is the intended behavior (see #48155), and what it'd do with the missing export added.

[zikaeroh]

Critically, note that make.bat and make.rc implicitly export GOROOT_BOOTSTRAP to child processes; it's only bash where you have to explicitly do it (and so it was done before).

The odd thing to me is why dist has logic to conjure up a default for GOROOT_BOOTSTRAP at all, given it's just a part of the make process; I'm sure I've overlooked something there, but it means implementing that same fallback logic twice, which is awkward.

[ianlancetaylor]

I sent https://golang.org/cl/370156 to restore the lost export of GOROOT_BOOTSTRAP.

[rsc]

The odd thing to me is why dist has logic to conjure up a default for GOROOT_BOOTSTRAP at all, given it's just a part of the make process; I'm sure I've overlooked something there, but it means implementing that same fallback logic twice, which is awkward.

Dist can be run directly by hand too, in rare circumstances. But yes, the bug was the missing export. Thanks for finding it.

[gopherbot]

Change https://golang.org/cl/370274 mentions this issue: build: for default bootstrap, use Go 1.17 if present, falling back to Go 1.4

[zikaeroh]

I'm not sure whether or not this should be its own issue, but with the new code in make.bash, go in $PATH will be ignored even if it's new enough to satisfy 1.17, which may be surprising for some distro packagers as their builds may now use something that happens to be in their home. Maybe it doesn't matter thanks to the bootstrap process iterating until stability, but I'd probably be confused if I went to run make.bash and it chose that over my preferred install.

Additionally, the checks are for explicitly the directories $HOME/sdk/go1.17 and $HOME/go1.17, but I think the former is unlikely to exist given there are patch releases, where golang.org/dl downloaders put things into places like $HOME/sdk/go1.17.4, as go1.17 explicitly means the first release of 1.17 (or, 1.17.0 in semver).

[rsc]

Retitled as suggested in #44505 (comment).

[gopherbot]

Change https://go.dev/cl/386774 mentions this issue: doc/go1.18: document Go 1.17 bootstrap and //go:build fix

[gopherbot]

Change https://go.dev/cl/399514 mentions this issue: cmd/asm: use constraint.IsGoBuild and remove TODO

[mvdan]

Are we on track for 1.19? I see https://go-review.googlesource.com/c/go/+/386774/ was merged, but the builders still seem to bootstrap with Go 1.4.x from what I can see.