Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: add a Windows ARM64 builder #42604

Open
cagedmantis opened this issue Nov 13, 2020 · 16 comments
Open

x/build: add a Windows ARM64 builder #42604

cagedmantis opened this issue Nov 13, 2020 · 16 comments

Comments

@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Nov 13, 2020

A builder should be added in support of the Windows/arm64 port #36439

Note: this is independent of #38607 which adds a Windows arm builder.

@golang/release

@zx2c4
Copy link
Contributor

@zx2c4 zx2c4 commented Nov 20, 2020

I have a ARM64 builder now, with a preinstalled gcc for each architecture supported on the machine:

image

@zx2c4
Copy link
Contributor

@zx2c4 zx2c4 commented Nov 20, 2020

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 20, 2020

Change https://golang.org/cl/272106 mentions this issue: dashboard: add new Windows ARM builder

gopherbot pushed a commit to golang/build that referenced this issue Nov 21, 2020
This partially reverts CL 229559, while adjusting it to use a new
builder. The builder will be used now for windows/arm and later for
windows/arm64.

Updates golang/go#42604.
Updates golang/go#38607.
Updates golang/go#36439.

Change-Id: I1ab661a6585e64de8196955c077715b32dc732b8
Reviewed-on: https://go-review.googlesource.com/c/build/+/272106
Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Trust: Alex Brainman <alex.brainman@gmail.com>
Trust: Jason A. Donenfeld <Jason@zx2c4.com>
@cagedmantis cagedmantis added this to Planned in Go Release Team Mar 23, 2021
@cagedmantis cagedmantis modified the milestones: Backlog, Go1.17 Apr 8, 2021
@cagedmantis cagedmantis self-assigned this Apr 13, 2021
@cagedmantis cagedmantis moved this from Planned to In Progress in Go Release Team Apr 13, 2021
@toothrot toothrot assigned toothrot and unassigned cagedmantis Apr 20, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 27, 2021

Change https://golang.org/cl/314430 mentions this issue: dashboard: add compile-only TryBot coverage for windows/arm64 port

gopherbot pushed a commit to golang/build that referenced this issue Apr 27, 2021
The windows/arm64 port has been completed in golang.org/issue/36439
recently. Add pre-submit compile-only coverage via a misc-compile
TryBot for the new port. It can be upgraded to a normal TryBot that
also executes tests after a windows/arm64 builder becomes available.

Tested locally by running 'buildall.bash windows-arm64' on a recent
Go tip commit (CL 313849), which passed.

For golang/go#36439.
Updates golang/go#42604.

Change-Id: Ie4988a876aa18392a583eb407c6c3621eb30a4f8
Reviewed-on: https://go-review.googlesource.com/c/build/+/314430
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Trust: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 19, 2021

Change https://golang.org/cl/321309 mentions this issue: doc/go1.17: add release note for windows/arm64 port

gopherbot pushed a commit that referenced this issue May 20, 2021
Updates #44513, #42604.

Change-Id: I8200e8087c219a0042ab2a6770a7275c3b17942a
Reviewed-on: https://go-review.googlesource.com/c/go/+/321309
Trust: Cherry Mui <cherryyz@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 25, 2021

Change https://golang.org/cl/321959 mentions this issue: env/windows-arm: Add Ubuntu/Windows ARM64 VM configuration

gopherbot pushed a commit to golang/build that referenced this issue May 26, 2021
This change introduces a Windows ARM64 builder image for Go, running on
Ubuntu on an a1.metal instance on AWS.

The AMI image successfully boots, and automatically starts a Windows
ARM64 VM, exposing port 443. The buildlet still needs to be updated for
the appropriate mingw path. Additionally, the qemu script does not
yet automatically halt the instance on termination, which makes
debugging easier.

- Ubuntu was chosen over Debian, as the official Debian AMI would not
boot on a1.metal instances.
- a1.metal is the most affordable AWS instance to support KVM on ARM64.
- A prepared qemu image exists in the Go AWS account, under
s3://go-builder-data.
- Only 4 processors are currently supported in order to avoid a
CLOCK_WATCHDOG_TIMEOUT. We're working on investigating how to increase
this.
- The EC2 metadata service is proxied to the guest OS via qemu.

For golang/go#42604

Change-Id: I2a505a8218c2a818f0e52d179c02e6d264d2e422
Reviewed-on: https://go-review.googlesource.com/c/build/+/321959
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 26, 2021

Change https://golang.org/cl/322654 mentions this issue: buildlet: increase timeout period for client

@gopherbot
Copy link

@gopherbot gopherbot commented May 26, 2021

Change https://golang.org/cl/322655 mentions this issue: env/windows-arm64/aws: increase VM resources

@gopherbot
Copy link

@gopherbot gopherbot commented May 26, 2021

Change https://golang.org/cl/322653 mentions this issue: cmd/buildlet: support Windows ARM64 qemu guests

@gopherbot
Copy link

@gopherbot gopherbot commented May 26, 2021

Change https://golang.org/cl/322656 mentions this issue: dashboard: add windows-arm64 builder

@gopherbot
Copy link

@gopherbot gopherbot commented May 26, 2021

Change https://golang.org/cl/322657 mentions this issue: env/windows-arm64/aws: halt after VM quits

@toothrot
Copy link
Contributor

@toothrot toothrot commented May 26, 2021

As mentioned in cl/322655, we occasionally hit a CLOCK_WATCHDOG_TIMEOUT or failure to boot the vm with -smp set higher than 4. I'm still investigating a fix to this.

With -smp 4, it takes us about 33 minutes from VM creation to completion to run a build, which is longer than I'd like. Also, I'd like to use all the resources we're paying for on the a1.metal instances.

@zx2c4
Copy link
Contributor

@zx2c4 zx2c4 commented May 26, 2021

Looks like you guys have this sorted now for arm64. My arm builder is running the same 64-bit OS, but simply building for GOARCH=arm GOARM=7 GOOS=windows and testing that. You guys should be able to do the same with better resources than my raspi. So, I'll be decommissioning that shortly, and you can build a new one on AWS as you've done for 64-bit.

gopherbot pushed a commit to golang/build that referenced this issue May 26, 2021
The Windows ARM builders on EC2 run on qemu via KVM. We use a userspace
network device for simplicity, which also conveniently restricts which
ports are exposed. In order to make a request to the metatadata service,
which is routed in a different way than the public internet, we
explicitly forward the port to the guest VM on a special ip/port.

This introduces a hard-coded value for Windows ARM64 buildlets. We
should be able to improve this code to detect it, but this will get the
buildlet unblocked.

The buildlet image also includes llvm-arm64 mingw rather than a gcc
based distribution. This change also adds the correct directory to the
path.

For golang/go#42604

Change-Id: Ife2ebb900a08034d6e0dfa0982a24b312ee6d70a
Reviewed-on: https://go-review.googlesource.com/c/build/+/322653
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue May 26, 2021
The a1.metal instances take about four minutes to become available
before beginning to boot the Windows ARM64 VM. This means we need
somewhere between five and ten minutes before we can first connect to
the buildlet.

It's possible that other instances are faster to boot, but this is
currently the most economical choice.

For golang/go#42604

Change-Id: Ib76944307c9ad28fd1adca0dc54f3fbeaabfffa0
Reviewed-on: https://go-review.googlesource.com/c/build/+/322654
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 26, 2021

Change https://golang.org/cl/322855 mentions this issue: internal/coordinator/pool: add a hard limit for a1.metal instances

gopherbot pushed a commit to golang/build that referenced this issue May 26, 2021
This change allows the ledger to track how many a1.metal instances
have either been created or are in the process of being created. A
hard limit of 1 has been set for a1.metal limits while testing is
being performed.

For golang/go#42604

Change-Id: I5bcb3a65407af07d225caf2884877ce040ee011b
Reviewed-on: https://go-review.googlesource.com/c/build/+/322855
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue May 26, 2021
a1.metal instances have 32Gb of ram, and 16 cores. We should try to use
as much as possible.

We run into CLOCK_WATCHDOG_TIMEOUT when using more than 4 of the 16
cores. The VM can also fail to boot. We have a thread discussing how to
resolve this.

A new AMI will need to be created and configured after any changes to
this script.

For golang/go#42604

Change-Id: If49b89718fca6dc5cee30e609a8ded9f99bf1f05
Reviewed-on: https://go-review.googlesource.com/c/build/+/322655
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue May 26, 2021
Add the windows-arm64 builder to the dashboard and coordinator with a
known issue until we can verify builds are succeeding regularly.

Also, tweak the logic around validating ec2 configurations. The arm64
buildlet sets VMImage but not ContainerImage, unlike other ec2
buildlets. This should work for both.

For golang/go#42604

Change-Id: I7176b8f6bba837830a0565e7713c9e9a0eb76e90
Reviewed-on: https://go-review.googlesource.com/c/build/+/322656
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@toothrot
Copy link
Contributor

@toothrot toothrot commented May 26, 2021

@zx2c4 Sounds good. Thanks so much for your assistance! Your help was very useful in getting this set up, and thanks for providing builders in the interim.

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 15, 2021

Change https://golang.org/cl/328311 mentions this issue: dashboard: add windows-arm64-10 reverse buildlet

gopherbot pushed a commit to golang/build that referenced this issue Jun 17, 2021
This adds a reverse buildlet for Windows ARM64 running under qemu on M1
Mac Minis. The VM currently boots, but more changes are needed to
stage0 and buildlet in order for the builder to run.

For golang/go#42604.

Change-Id: Ie86e596c0f166d2ca8c075c0c971fcd1e34e940f
Reviewed-on: https://go-review.googlesource.com/c/build/+/328311
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Go Release Team
In Progress
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants