Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/internal/releasetargets: evaluate for each upcoming major Go release #40561

Open
dmitshur opened this issue Aug 4, 2020 · 50 comments
Open
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. recurring Issues that should never be closed, but moved to the next milestone once fixed in the current one. release-blocker
Milestone

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Aug 4, 2020

At the start of each code freeze, we should evaluate the choice of releaselets (builders that are used for constructing a release artifact) used to make the upcoming major Go release, and confirm they are the most appropriate choice out of what is available.

The start of a code freeze is a safe time to introduce a builder change, as it will be tested during pre-release versions, starting with beta 1 and onwards.

Making this a release blocker for Go 1.16 (and not okay after beta 1—it needs to be done before). Once completed, this issue should be moved to the next major release milestone.

Edit: Starting with Go 1.21, releaselets are only used for running tests. Release artifact construction happens by reproducibly cross-compiling a distpack distribution on a fixed builder.

CC @golang/release, @ianlancetaylor.

@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) release-blocker recurring Issues that should never be closed, but moved to the next milestone once fixed in the current one. labels Aug 4, 2020
@dmitshur dmitshur added this to the Go1.16 milestone Aug 4, 2020
@toothrot toothrot added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Aug 4, 2020
@toothrot
Copy link
Contributor

toothrot commented Nov 5, 2020

/cc @cagedmantis

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Nov 19, 2020
@dmitshur
Copy link
Contributor Author

dmitshur commented Nov 21, 2020

The @golang/release team has considered the current builders, and we are proposing the following builder changes for the Go 1.16 release:

go1.16*.linux-amd64.tar.gz

Builder Name Host Type
Current linux-amd64-jessie host-linux-jessie
New linux-amd64 host-linux-stretch

(Or rather a new builder, identical to the linux-amd64 one, just named linux-amd64-stretch instead.)

Rationale: Debian 8 "Jessie" Long Term Support (LTS) has ended on June 30, 2020 (see https://wiki.debian.org/LTS). We want to start using the next stable release, Debian 9 "Stretch", which has LTS support until June 2022.

Note that this has caused #31293 in April 2019 when this change was made unintentionally as part of a minor release, and it was reverted. We hope that it should be okay to try again for Go 1.16 a year later, since this is being done intentionally this time, and for a major release. CC @ianlancetaylor, @jayconrod, @bradfitz.

go1.16*.linux-386.tar.gz

Builder Name Host Type
Current linux-386 host-linux-jessie
New linux-386-stretch host-linux-stretch

Rationale: Same as the one above for linux/amd64.

go1.16*.linux-armv6l.tar.gz

Builder Name Host Type
Current linux-arm host-linux-arm-scaleway
New linux-arm-aws host-linux-arm-aws

Rationale: The replacement builder has proven to be more reliable, and it is easier for us to maintain.

go1.16*.linux-arm64.tar.gz

Builder Name Host Type
Current linux-arm64-packet host-linux-arm64-packet
New linux-arm64-aws host-linux-arm64-aws

Rationale: The replacement builder has proven to be more reliable, and it is easier for us to maintain.

Also see #42304 (comment).


Feedback is welcome: we are happy to alter the plan based on information we haven't already considered. These builder changes are planned for Go 1.16 Beta 1, and we will re-evaluate this as needed based on any new findings and feedback.

@dmitshur dmitshur self-assigned this Dec 3, 2020
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/276034 mentions this issue: cmd/release, dashboard: implement builder plan for Go 1.16

gopherbot pushed a commit to golang/build that referenced this issue Dec 9, 2020
See https://golang.org/issues/40561#issuecomment-731482962
for the plan description and rationale.

Add new builder definitions that are needed for the plan.
Don't rely on generic builder name like "linux-amd64" to
avoid a repeat of golang.org/issue/31293; use a specific
builder name instead.

Remove support and test cases for Go 1.13, it's unsupported
per release policy.

For golang/go#40561.

Change-Id: I070744e15be9f1932d649a27ee7e4599e467553f
Reviewed-on: https://go-review.googlesource.com/c/build/+/276034
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@dmitshur
Copy link
Contributor Author

dmitshur commented Dec 9, 2020

CL 276034 has implemented the aforementioned plan for Go 1.16.

We'll need to revisit this issue next time during the Go 1.17 development cycle. (If we discover problems with the Go 1.16 plan, we'll come back here, or file new issues.) Moving to the next milestone for now.

@dmitshur dmitshur modified the milestones: Go1.16, Go1.17 Dec 9, 2020
@dmitshur dmitshur removed their assignment Dec 9, 2020
@thanm
Copy link
Contributor

thanm commented Dec 15, 2020

While working on issue #39326, it has become clear to me that the C compile we have installed on our windows gomotes (at least the ones I have used) is pretty ancient -- GCC 5. It would be great if we could update the version to something a bit more recent (V10 maybe). Also worth noting that many folks are now using clang instead of gcc on windows -- it would be great if our gomotes had clang available as well.

@dmitshur
Copy link
Contributor Author

dmitshur commented Dec 15, 2020

@thanm Can you please file a separate issue to track that work, and CC @golang/release? It may need more information/investigation. Thanks.

@thanm
Copy link
Contributor

thanm commented Dec 15, 2020

Filed #43195. Thanks.

@dmitshur dmitshur changed the title x/build/cmd/release: evaluate builders used for upcoming major Go release x/build/cmd/release: evaluate releaselets used for each upcoming major Go release Dec 15, 2020
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/278357 mentions this issue: cmd/release: update checkRelocations for Go 1.16

gopherbot pushed a commit to golang/build that referenced this issue Dec 15, 2020
The plan in https://golang.org/issue/40561#issuecomment-731482962
involved updating the linux-amd64 target from Debian Jessie to
Debian Stretch, which in turn comes with a newer version of GCC.

checkRelocations was added in CL 171317 in response to a bad minor
release that was accidentally issued without the intended fix, and
needs to be updated for Go 1.16 and newer releases. It's not clear
if we want to maintain it indefinitely for future Go versions, but
keep it for a bit longer. Add a note to make it clear that it can
be removed as needed in the future.

For golang/go#40561.
Updates golang/go#31293.

Change-Id: I2da419eff6379575eb2648787f0dac6bba07b304
Reviewed-on: https://go-review.googlesource.com/c/build/+/278357
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/313070 mentions this issue: dashboard: make FreeBSD 11.4 and 12.2 builders the default

gopherbot pushed a commit to golang/build that referenced this issue Apr 26, 2021
With this change:
- The default FreeBSD builders used on master are 11.4 and 12.2.
- The FreeBSD 12.0 builders have been removed.
- The FreeBSD 11.2 builders have been set to run on at most Go 1.16.
  They will continue to be used for Go 1.16.x and 1.15.x minor releases.
- The freebsd-amd64-race builder has been updated to use the FreeBSD
  12.2 host.
- The slowbot aliases for FreeBSD have been updated to use 12.2.

Fixes golang/go#44431
Update golang/go#45727
Update golang/go#40561

Change-Id: Ib558fca65c2de1916b4633711d3e320c390bd2b2
Reviewed-on: https://go-review.googlesource.com/c/build/+/313070
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@cagedmantis
Copy link
Contributor

The @golang/release team has considered the current builders, and we are proposing the following builder change for the Go 1.17 release:

go1.17.*.freebsd-amd64.tar.gz

Builder Name Host Type
Current freebsd-amd64-11_2 host-freebsd-11_2
New freebsd-amd64-11_4 host-freebsd-11_4

Rationale: Using the latest "patch" release of the oldest supported "major" release of FreeBSD.

#45727 (comment)

go1.17.*.freebsd-386.tar.gz

Builder Name Host Type
Current freebsd-386-11_2 host-freebsd-11_2
New freebsd-386-11_4 host-freebsd-11_4

Rationale: Using the latest "patch" release of the oldest supported "major" release of FreeBSD.

#45727 (comment)

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/315889 mentions this issue: cmd/release: start using the FreeBSD 11.4 builder for Go 1.17

gopherbot pushed a commit to golang/build that referenced this issue Jun 15, 2023
A longtest builder for the darwin/amd64 port was added in the Go 1.21
development cycle. Start using it as for non-advisory release testing.

For golang/go#40561.
For golang/go#35678.
For golang/go#29252.

Change-Id: Ic65e84e5e10bcb786cb28c36c1d1b137e2f6202e
Reviewed-on: https://go-review.googlesource.com/c/build/+/503758
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/504525 mentions this issue: internal/releasetargets: set LongTestBuilder for linux-arm64

gopherbot pushed a commit to golang/build that referenced this issue Jun 22, 2023
The linux/arm64 port is first class and a longtest builder was added
for it during the Go 1.20 development cycle. It's a mandatory trybot
on 1.20+ release branches and is working well.

The builder recently found a real problem in go.dev/issue/60825, which
has been resolved by now. Upgrade this builder from an advisory TryBot
to have more complete release test coverage.

For golang/go#40561.
For golang/go#49649.
For golang/go#29252.

Change-Id: I1765e820deb0d34b7bb9a72aba397d87c1369ced
Reviewed-on: https://go-review.googlesource.com/c/build/+/504525
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/527017 mentions this issue: internal/releasetargets: drop Go 1.19 targets

gopherbot pushed a commit to golang/build that referenced this issue Sep 8, 2023
There won't be new Go 1.19 releases per go.dev/doc/devel/release#policy.

For golang/go#62076.
For golang/go#40561.

Change-Id: I30e09c9f47ec0006caeb67e15e03b8cdbff7c175
Reviewed-on: https://go-review.googlesource.com/c/build/+/527017
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@dmitshur dmitshur self-assigned this Nov 29, 2023
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/547935 mentions this issue: internal/releasetargets: regenerate all ports for Go 1.22

gopherbot pushed a commit to golang/build that referenced this issue Dec 7, 2023
It's the same list as 1.21, with the addition of the openbsd/ppc64 port.

Generated with 'go generate'.

For golang/go#40561.
For golang/go#56001.

Change-Id: I93d2cfd191134f524255393e8b60bf6ce7328940
Reviewed-on: https://go-review.googlesource.com/c/build/+/547935
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@dmitshur
Copy link
Contributor Author

dmitshur commented Dec 7, 2023

CL 547935 takes care of the port list for Go 1.22.

After Go 1.22.0 is out, all release artifacts we produce will be via reproducible cross-compilation and distpack, so the work of "evaluate the choice of releaselets (builders that are used for constructing a release artifact)" that this issue tracks becomes obsolete. We still need to handle the port list, though it'll be possible to replace its generation with a run-time computation if that's easier. And the releasetargets package can be simplified. We do still need to continuously maintain and update OS versions, but that's out of scope for here.

Dealing with all that is for after Go 1.22.0 is out, so moving this to the next milestone to revisit later.

@dmitshur dmitshur modified the milestones: Go1.22, Go1.23 Dec 7, 2023
@dmitshur dmitshur removed their assignment Dec 7, 2023
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/564217 mentions this issue: internal/releasetargets: update MinMacOSVersion for Go 1.23

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/564216 mentions this issue: internal/releasetargets: drop Go 1.20 release targets

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/564255 mentions this issue: internal/releasetargets: delete fields not relevant to distpack releases

gopherbot pushed a commit to golang/build that referenced this issue Feb 16, 2024
Delete the old before starting to update for the new.
"Targets for release 1.21" and newer in releases.txt
doesn't change.

For golang/go#40561.

Change-Id: I40abd20bbee165a641e2a22a6e791cfbdf950058
Reviewed-on: https://go-review.googlesource.com/c/build/+/564216
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Feb 16, 2024
For golang/go#40561.
Fixes golang/go#64207.

Change-Id: I1bbd60d0c2bcae28856c7f7c8293cb2f5a2397f5
Reviewed-on: https://go-review.googlesource.com/c/build/+/564217
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Feb 20, 2024
The Target struct fields Builder, LongTestBuilder, and BuildOnly were
applicable to releases prior to distpacks (i.e., Go 1.20 and older).

By now, all releases are built on a secured Linux machine (and verified
by rebuilding on another secured Windows machine), installers are built
by the same pipeline that's responsible for performing signing, and the
tests are run via the "advisory builder" loop. These fields have become
meaningless and confusing, so drop them.

Print whether a port is primary or not in releases.txt, so that if a map
entry like "linux-amd64": &Target{} that only has implicit effects left
by now is accidentally removed from allReleases, it'll be easy to spot
the effect via the releases.txt diff.

There's a bit more to do before the entire package becomes obsolete.

For golang/go#40561.

Change-Id: Ib66ca6958db695db0093556edc822dcdfbfdde0d
Reviewed-on: https://go-review.googlesource.com/c/build/+/564255
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
@dmitshur dmitshur self-assigned this Apr 19, 2024
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/585216 mentions this issue: internal/releasetargets: add openbsd/ppc64 target for Go 1.22 onwards

gopherbot pushed a commit to golang/build that referenced this issue May 13, 2024
The release target list has two underlying sources: the allReleases map
and the allPorts map. The former is a small curated map, mostly used to
indicate which ports are first-class, and to set a few port properties.
The latter is generated from 'go tool dist list' output for each major
Go release.

Unfortunately, the logic in TargetsForGo1Point ended up not taking into
account that CL 547935 added allports/go1.22.txt with the openbsd/ppc64
port in it because the allReleases map didn't have an entry for go1.22.
As a result, we haven't been making releases for the openbsd/ppc64 port
as intended. :( Fix this by making sortedReleases consult both sources.

For golang/go#40561.
For golang/go#56001.

Change-Id: Icbabad6f7e2920c1b24a69e309d65aea77c9c461
Reviewed-on: https://go-review.googlesource.com/c/build/+/585216
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/588137 mentions this issue: internal/releasetargets: regenerate all ports for Go 1.23

@dmitshur dmitshur added the FixPending Issues that have a fix which has not yet been reviewed or submitted. label May 24, 2024
gopherbot pushed a commit to golang/build that referenced this issue May 24, 2024
The release freeze for Go 1.23 has started, so now is a good
time to regenerate this list. Done with 'go generate'.

The list is the same as for Go 1.22 with the addition of the
new openbsd/riscv64 port.

For golang/go#40561.
For golang/go#55999.

Change-Id: Ib920c3b4337fcd9aa4a765e5622e6c158f829bda
Reviewed-on: https://go-review.googlesource.com/c/build/+/588137
Reviewed-by: Carlos Amedee <carlos@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@dmitshur
Copy link
Contributor Author

Done for the Go 1.23 cycle.

@dmitshur dmitshur removed the FixPending Issues that have a fix which has not yet been reviewed or submitted. label May 24, 2024
@dmitshur dmitshur modified the milestones: Go1.23, Go1.24 May 24, 2024
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/612555 mentions this issue: internal/releasetargets: drop 1.21 targets, add 1.24 targets

gopherbot pushed a commit to golang/build that referenced this issue Sep 11, 2024
The Go 1.21 entries are no longer needed now that go1.23.0 is out.

The list for tip is the same as for Go 1.23, except that the windows/arm
port is not included because it's currently marked broken (CL 601777).

I considered renaming allReleases to something more appropriate since
it tracks only first class ports now, but leaving that to a future CL.

For golang/go#40561.
For golang/go#68552.

Change-Id: Ic61e797a125aef5b9cc2782b87ea558e7c88035c
Reviewed-on: https://go-review.googlesource.com/c/build/+/612555
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Tim King <taking@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. recurring Issues that should never be closed, but moved to the next milestone once fixed in the current one. release-blocker
Projects
Status: Planned
Status: No status
Development

No branches or pull requests

6 participants