Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/release: evaluate releaselets used for each upcoming major Go release #40561

Open
dmitshur opened this issue Aug 4, 2020 · 26 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. recurring Issues that should never be closed, but moved to the next milestone once fixed in the current one. release-blocker
Milestone

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Aug 4, 2020

At the start of each code freeze, we should evaluate the choice of releaselets (builders that are used for constructing a release artifact) used to make the upcoming major Go release, and confirm they are the most appropriate choice out of what is available.

The start of a code freeze is a safe time to introduce a builder change, as it will be tested during pre-release versions, starting with beta 1 and onwards.

Making this a release blocker for Go 1.16 (and not okay after beta 1—it needs to be done before). Once completed, this issue should be moved to the next major release milestone.

CC @golang/release, @ianlancetaylor.

@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) release-blocker recurring Issues that should never be closed, but moved to the next milestone once fixed in the current one. labels Aug 4, 2020
@dmitshur dmitshur added this to the Go1.16 milestone Aug 4, 2020
@toothrot toothrot added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Aug 4, 2020
@toothrot
Copy link
Contributor

toothrot commented Nov 5, 2020

/cc @cagedmantis

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Nov 19, 2020
@dmitshur
Copy link
Contributor Author

dmitshur commented Nov 21, 2020

The @golang/release team has considered the current builders, and we are proposing the following builder changes for the Go 1.16 release:

go1.16*.linux-amd64.tar.gz

Builder Name Host Type
Current linux-amd64-jessie host-linux-jessie
New linux-amd64 host-linux-stretch

(Or rather a new builder, identical to the linux-amd64 one, just named linux-amd64-stretch instead.)

Rationale: Debian 8 "Jessie" Long Term Support (LTS) has ended on June 30, 2020 (see https://wiki.debian.org/LTS). We want to start using the next stable release, Debian 9 "Stretch", which has LTS support until June 2022.

Note that this has caused #31293 in April 2019 when this change was made unintentionally as part of a minor release, and it was reverted. We hope that it should be okay to try again for Go 1.16 a year later, since this is being done intentionally this time, and for a major release. CC @ianlancetaylor, @jayconrod, @bradfitz.

go1.16*.linux-386.tar.gz

Builder Name Host Type
Current linux-386 host-linux-jessie
New linux-386-stretch host-linux-stretch

Rationale: Same as the one above for linux/amd64.

go1.16*.linux-armv6l.tar.gz

Builder Name Host Type
Current linux-arm host-linux-arm-scaleway
New linux-arm-aws host-linux-arm-aws

Rationale: The replacement builder has proven to be more reliable, and it is easier for us to maintain.

go1.16*.linux-arm64.tar.gz

Builder Name Host Type
Current linux-arm64-packet host-linux-arm64-packet
New linux-arm64-aws host-linux-arm64-aws

Rationale: The replacement builder has proven to be more reliable, and it is easier for us to maintain.

Also see #42304 (comment).


Feedback is welcome: we are happy to alter the plan based on information we haven't already considered. These builder changes are planned for Go 1.16 Beta 1, and we will re-evaluate this as needed based on any new findings and feedback.

@dmitshur dmitshur self-assigned this Dec 3, 2020
@gopherbot
Copy link

gopherbot commented Dec 8, 2020

Change https://golang.org/cl/276034 mentions this issue: cmd/release, dashboard: implement builder plan for Go 1.16

gopherbot pushed a commit to golang/build that referenced this issue Dec 9, 2020
See https://golang.org/issues/40561#issuecomment-731482962
for the plan description and rationale.

Add new builder definitions that are needed for the plan.
Don't rely on generic builder name like "linux-amd64" to
avoid a repeat of golang.org/issue/31293; use a specific
builder name instead.

Remove support and test cases for Go 1.13, it's unsupported
per release policy.

For golang/go#40561.

Change-Id: I070744e15be9f1932d649a27ee7e4599e467553f
Reviewed-on: https://go-review.googlesource.com/c/build/+/276034
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@dmitshur
Copy link
Contributor Author

dmitshur commented Dec 9, 2020

CL 276034 has implemented the aforementioned plan for Go 1.16.

We'll need to revisit this issue next time during the Go 1.17 development cycle. (If we discover problems with the Go 1.16 plan, we'll come back here, or file new issues.) Moving to the next milestone for now.

@dmitshur dmitshur modified the milestones: Go1.16, Go1.17 Dec 9, 2020
@dmitshur dmitshur removed their assignment Dec 9, 2020
@thanm
Copy link
Contributor

thanm commented Dec 15, 2020

While working on issue #39326, it has become clear to me that the C compile we have installed on our windows gomotes (at least the ones I have used) is pretty ancient -- GCC 5. It would be great if we could update the version to something a bit more recent (V10 maybe). Also worth noting that many folks are now using clang instead of gcc on windows -- it would be great if our gomotes had clang available as well.

@dmitshur
Copy link
Contributor Author

dmitshur commented Dec 15, 2020

@thanm Can you please file a separate issue to track that work, and CC @golang/release? It may need more information/investigation. Thanks.

@thanm
Copy link
Contributor

thanm commented Dec 15, 2020

Filed #43195. Thanks.

@dmitshur dmitshur changed the title x/build/cmd/release: evaluate builders used for upcoming major Go release x/build/cmd/release: evaluate releaselets used for each upcoming major Go release Dec 15, 2020
@gopherbot
Copy link

gopherbot commented Dec 15, 2020

Change https://golang.org/cl/278357 mentions this issue: cmd/release: update checkRelocations for Go 1.16

gopherbot pushed a commit to golang/build that referenced this issue Dec 15, 2020
The plan in https://golang.org/issue/40561#issuecomment-731482962
involved updating the linux-amd64 target from Debian Jessie to
Debian Stretch, which in turn comes with a newer version of GCC.

checkRelocations was added in CL 171317 in response to a bad minor
release that was accidentally issued without the intended fix, and
needs to be updated for Go 1.16 and newer releases. It's not clear
if we want to maintain it indefinitely for future Go versions, but
keep it for a bit longer. Add a note to make it clear that it can
be removed as needed in the future.

For golang/go#40561.
Updates golang/go#31293.

Change-Id: I2da419eff6379575eb2648787f0dac6bba07b304
Reviewed-on: https://go-review.googlesource.com/c/build/+/278357
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@gopherbot
Copy link

gopherbot commented Apr 23, 2021

Change https://golang.org/cl/313070 mentions this issue: dashboard: make FreeBSD 11.4 and 12.2 builders the default

gopherbot pushed a commit to golang/build that referenced this issue Apr 26, 2021
With this change:
- The default FreeBSD builders used on master are 11.4 and 12.2.
- The FreeBSD 12.0 builders have been removed.
- The FreeBSD 11.2 builders have been set to run on at most Go 1.16.
  They will continue to be used for Go 1.16.x and 1.15.x minor releases.
- The freebsd-amd64-race builder has been updated to use the FreeBSD
  12.2 host.
- The slowbot aliases for FreeBSD have been updated to use 12.2.

Fixes golang/go#44431
Update golang/go#45727
Update golang/go#40561

Change-Id: Ib558fca65c2de1916b4633711d3e320c390bd2b2
Reviewed-on: https://go-review.googlesource.com/c/build/+/313070
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@cagedmantis
Copy link
Contributor

cagedmantis commented Apr 30, 2021

The @golang/release team has considered the current builders, and we are proposing the following builder change for the Go 1.17 release:

go1.17.*.freebsd-amd64.tar.gz

Builder Name Host Type
Current freebsd-amd64-11_2 host-freebsd-11_2
New freebsd-amd64-11_4 host-freebsd-11_4

Rationale: Using the latest "patch" release of the oldest supported "major" release of FreeBSD.

#45727 (comment)

go1.17.*.freebsd-386.tar.gz

Builder Name Host Type
Current freebsd-386-11_2 host-freebsd-11_2
New freebsd-386-11_4 host-freebsd-11_4

Rationale: Using the latest "patch" release of the oldest supported "major" release of FreeBSD.

#45727 (comment)

@gopherbot
Copy link

gopherbot commented Apr 30, 2021

Change https://golang.org/cl/315889 mentions this issue: cmd/release: start using the FreeBSD 11.4 builder for Go 1.17

@cagedmantis
Copy link
Contributor

cagedmantis commented Nov 9, 2021

The @golang/release team has considered the current builders, and we are proposing the following builder change for the Go 1.18 release:

Port Current Builder Current Host Type Proposed Builder Proposed Host Type
freebsd-386 freebsd-386-11_4 host-freebsd-11_4 freebsd-386-12_2 host-freebsd-12_2
freebsd-amd64 freebsd-amd64-11_4 host-freebsd-11_4 freebsd-amd64-12_2 host-freebsd-12_2
windows-386 windows-386-2008 host-windows-amd64-2008 windows-386-2012 host-windows-amd64-2012
windows-amd64 windows-amd64-2008 host-windows-amd64-2008 windows-amd64-2012 host-windows-amd64-2012

Rationale: Using the latest "patch" release of the oldest supported "major" release of FreeBSD. Using the latest supported version of Windows server.

@gopherbot
Copy link

gopherbot commented Nov 19, 2021

Change https://golang.org/cl/365694 mentions this issue: cmd/release: update the releaselets for go1.18

@cagedmantis
Copy link
Contributor

cagedmantis commented Dec 1, 2021

The @golang/release team has revised the proposed releaselets to be used for the Go 1.18 release:

Port Current Builder Current Host Type Proposed Builder Proposed Host Type
freebsd-386 freebsd-386-11_4 host-freebsd-11_4 freebsd-386-12_2 host-freebsd-12_2
freebsd-amd64 freebsd-amd64-11_4 host-freebsd-11_4 freebsd-amd64-12_2 host-freebsd-12_2

Rationale: Using the latest "patch" release of the oldest supported "major" release of FreeBSD. The Windows change was removed because of the discussions in #47845 and release team meetings.

@gopherbot
Copy link

gopherbot commented Dec 1, 2021

Change https://golang.org/cl/368394 mentions this issue: cmd/release: revert to macOS 11 builder for 1.17, 1.16 minor releases

gopherbot pushed a commit to golang/build that referenced this issue Dec 1, 2021
This change updates the builders used for go1.18 by:
- Using FreeBSD 12.2

It also removes unused releaselets (Go version < 1.16).

Updates golang/go#40561

Change-Id: I3f46d197e9d235cf4577a1687848f8b52981c5e3
Reviewed-on: https://go-review.googlesource.com/c/build/+/365694
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@dmitshur
Copy link
Contributor Author

dmitshur commented Dec 1, 2021

By now we have darwin builders with macOS 12 both on amd64 and arm64 architectures (#49149), though that work is still evolving. We will consider whether to use them or still use macOS 11 for building 1.18 before Beta 1. The current trajectory is to use macOS 12 builder at least for arm64, but that may change. (Feedback on this is welcome.)

gopherbot pushed a commit to golang/build that referenced this issue Dec 1, 2021
CL 365736 updated the builder to macOS 12 out of necessity, since
the macOS 11 builder was not available then. Based on discussion
since, the builder is back though in limited capacity. Continue
to use it for Go 1.17 and 1.16 minor releases, with the plan to
start using macOS 12 builder for Go 1.18 and onwards.

Fixes golang/go#49889.
Updates golang/go#40561.

Change-Id: Ie9b6341b7c1f378e26e2226fc2bb4c3cb7042f9b
Reviewed-on: https://go-review.googlesource.com/c/build/+/368394
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

gopherbot commented Dec 7, 2021

Change https://golang.org/cl/369958 mentions this issue: cmd/release: start using macOS 12 releaselets in Go 1.18 Beta 1

@dmitshur
Copy link
Contributor Author

dmitshur commented Dec 7, 2021

We've discussed macOS today and agreed to use macOS 12 releaselets both for amd64 and arm64 architectures starting with Go 1.18 Beta 1. Sent CL 369958.

gopherbot pushed a commit to golang/build that referenced this issue Dec 8, 2021
The work to add macOS 12 builders for amd64 and arm64 was completed
ahead of Go 1.18 Beta 1, so we have the opportunity to pick between
starting to use them or stay with existing macOS 11 builders longer.

As part of go.dev/issue/40561 we discussed our available options and
opted to select macOS 12 releaselets for Go 1.18 on both architectures,
starting with first beta release. This CL implements that decision.

Tested:
  I ran 'release -target=darwin-amd64 -version=go1.18beta123 -rev=HEAD'
  locally and it completed successfully; all.bash release tests passed.

For golang/go#40561.

Change-Id: Ia80c4862a5137b5264c7a855452be25bddd79840
Reviewed-on: https://go-review.googlesource.com/c/build/+/369958
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@dmitshur dmitshur self-assigned this Dec 8, 2021
@dmitshur dmitshur moved this from In Progress to Done in Go Release Team Dec 8, 2021
@dmitshur
Copy link
Contributor Author

dmitshur commented Dec 8, 2021

All looks ready for Go 1.18 Beta 1, so moving this recurring issue to the next milestone.

@dmitshur dmitshur modified the milestones: Go1.18, Go1.19 Dec 8, 2021
@gopherbot
Copy link

gopherbot commented Jan 12, 2022

Change https://golang.org/cl/377474 mentions this issue: dashboard: clean up builders affected by memory corruption

gopherbot pushed a commit to golang/build that referenced this issue Jan 12, 2022
Remove freebsd 12.2, which is replaced by 12.3 with the XSAVE fix.
Move freebsd 11.* to N2 machines, which are not affected.
Remove openbsd and netbsd e2/n1/n2/n2d-specific configurations now that
we have mostly understood the nature of that problem. Keep one around so
that the runtime team can create gomotes. Move the "official" builder to
the n2 cpu that works.

For golang/go#49967, golang/go#49209, golang/go#40561.
Fixes golang/go#50496.

Change-Id: If6989317f06cbec95d5addb19d9e968aecfa3f8a
Reviewed-on: https://go-review.googlesource.com/c/build/+/377474
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@heschi
Copy link
Contributor

heschi commented Jan 12, 2022

I've updated 1.18 to use FreeBSD 12.3, which has the fix for the memory corruption bug discussed in #49209.

@dmitshur
Copy link
Contributor Author

dmitshur commented Jun 1, 2022

I've looked through our current releaselet selection for 1.18:

darwin/amd64    darwin-amd64-12_0
darwin/arm64    darwin-arm64-12
freebsd/386     freebsd-386-12_3
freebsd/amd64   freebsd-amd64-12_3
linux/386       linux-386-stretch
linux/amd64     linux-amd64-stretch
linux/arm       linux-arm-aws
linux/arm64     linux-arm64-aws
linux/ppc64le   linux-ppc64le-buildlet
linux/s390x     linux-s390x-crosscompile
windows/386     windows-386-2008
windows/amd64   windows-amd64-2008
windows/arm64   windows-arm64-11

In general it is quite up to date and there's not much that needs to (or can) change for 1.19.

For example, we're already using macOS 12 (the latest macOS release today) builders for darwin amd64 and arm64 ports, Windows 11 builder for windows arm64 port. Nothing to do there.

FreeBSD builders were just recently updated from 11.4 to 12.3 in 1.18, so they're in good shape. (FreeBSD 12.x is still supported for some time.)

One opportunity I'm spotting is to update linux 386 and amd64 from Debian Stretch to Debian Buster. We updated to stretch in Go 1.16, and Stretch is documented to exit its LTS support on June 30, 2022. (Go 1.19 final is aiming for August 2022.) Buster is supported July 2022 to June 2024.

The linux arm and arm64 ports use the AWS builders which are already running buster, so they don't need any changes.

So my only change suggestion for 1.19 that we can consider is:

linux/386       linux-386-stretch → linux-386-buster
linux/amd64     linux-amd64-stretch → linux-amd64-buster

@dmitshur dmitshur self-assigned this Jun 2, 2022
@gopherbot
Copy link

gopherbot commented Jun 2, 2022

Change https://go.dev/cl/410094 mentions this issue: internal/releasetargets: update linux-{386,amd64} builders to buster

gopherbot pushed a commit to golang/build that referenced this issue Jun 2, 2022
We've updated the Linux builders used during Go releases from
Debian 8 "Jessie" to Debian 9 "Stretch" in time for Go 1.16,
since Jessie support had already ended on June 30, 2020 and
wasn't suitable for the lifetime of Go 1.16.

We're in a similar position now, where Debian 9 "Stretch" LTS
support is ending on June 30, 2022, before Go 1.19's anticipated
release date. Update the selected builder to Debian 10 "Buster",
whose LTS support keeps going until June 2024 and should cover
the expected lifespan of Go 1.19 better.

This is easy to do thanks to the builder already being added
as of CL 267937.

For golang/go#40561.
Updates golang/go#42410.

Change-Id: I06b43e1cdd66996852ed19a1cc928eff29d09c86
Reviewed-on: https://go-review.googlesource.com/c/build/+/410094
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@dmitshur
Copy link
Contributor Author

dmitshur commented Jun 2, 2022

CL 410094 makes Go 1.19 start using the buster builder as mentioned above.

There's nothing more planned to do here at this time, moving to next milestone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. recurring Issues that should never be closed, but moved to the next milestone once fixed in the current one. release-blocker
Projects
Status: No status
Development

No branches or pull requests

6 participants