Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: add Amazon EC2 ARM instances for builders #36841

Open
cagedmantis opened this issue Jan 28, 2020 · 33 comments
Open

x/build: add Amazon EC2 ARM instances for builders #36841

cagedmantis opened this issue Jan 28, 2020 · 33 comments
Assignees
Milestone

Comments

@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Jan 28, 2020

As @bradfitz noted, we should explore adding ephemeral AWS ARM instances for both 32-bit and 64-bit ARM builds.

@toothrot @dmitshur

@gopherbot gopherbot added this to the Unreleased milestone Jan 28, 2020
@gopherbot gopherbot added the Builders label Jan 28, 2020
@toothrot
Copy link
Contributor

@toothrot toothrot commented Jan 28, 2020

See also #36170, for discussion of some of our requirements on ARM builders. We still have work to do there, but it might be worth taking care of it all at once.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jan 28, 2020

@toothrot, if we did this bug, that bug would be not applicable.

We'd run each buildlet in its own VM (like we do on GCE, except using AWS), not using Docker (like Packet & Scaleway & PPC{,64}, etc).

We now have quota from AWS to run this for free, so it's just a matter of writing an AWSBuildletPool implementation.

@toothrot
Copy link
Contributor

@toothrot toothrot commented Jan 28, 2020

I understand that, but I wanted to make sure some of the other requirements were still being paid attention to, like creating a highcpu builder.

@cagedmantis cagedmantis self-assigned this Feb 26, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 1, 2020

Change https://golang.org/cl/226843 mentions this issue: internal/spanlog: moves spanlog into the internal directory

gopherbot pushed a commit to golang/build that referenced this issue Apr 2, 2020
This is the first in a series of patches which include an
effort to refactor how the cmd/coordinator is structured. In order
to move buildlet pools into their own package, it would facilitate
the move to have spanlog in it's own internal package. This CL will
be followed by one which removes cmd/coordinator/spanlog and
changes the path of the package for all the callers.

Updates golang/go#36841

Change-Id: Ic310a38f807a3099815e0bdb988edc42b2ca5c85
Reviewed-on: https://go-review.googlesource.com/c/build/+/226843
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 7, 2020

Change https://golang.org/cl/227023 mentions this issue: cmd/coordinator/spanlog: remove spanlog package

gopherbot pushed a commit to golang/build that referenced this issue Apr 7, 2020
This removes the spanlog package located inside the
coordinator package. The spanlog package has already been
copied over to the internal package. All imports in the build
repository have been changes to point to the internal spanlog
package.

Updates golang/go#36841

Change-Id: I84499bcfe782c566b12dabcbfa743cd8ce559f18
Reviewed-on: https://go-review.googlesource.com/c/build/+/227023
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 7, 2020

Change https://golang.org/cl/227141 mentions this issue: internal/pool: move the gce buildlet pool into a pool package

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 8, 2020

Change https://golang.org/cl/227578 mentions this issue: internal/pool: move the kube buildlet pool into a pool package

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 9, 2020

Change https://golang.org/cl/227769 mentions this issue: internal/pool: move the reverse buildlet pool into a pool package

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 9, 2020

Change https://golang.org/cl/227768 mentions this issue: internal/pool: move the kubernetes buildlet pool into a pool package

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 10, 2020

Change https://golang.org/cl/227920 mentions this issue: internal/pool: remove package level accessors

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 17, 2020

Change https://golang.org/cl/228799 mentions this issue: all: add linux-arm64-aws builders

gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2020
This CL creates the internal/coordinator/pool package intended to
contain all buildlet pool implementations. In order to keep this
change small and carefully discover where the interactions are
between the gce buildlet pool and the rest of the coordinator
are, this change only moves the gce buildlet over to the new
package.

The next steps will be to move the rest of the buildlet pools
over to this package. After that we will restructure the
implementations themselves in order to increase test coverage
and increase the ease of testing.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: If82ae1b584bd77c697aa84fadf9011c9e79fa409
Reviewed-on: https://go-review.googlesource.com/c/build/+/227141
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2020
This is a set in a series of steps which will move everything buildlet
pool related into a pool package.

Updates golang/go#36841

Change-Id: I8efb1f94c7b929be559004d9f455bca0370c7800
Reviewed-on: https://go-review.googlesource.com/c/build/+/227768
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2020
This is a set in a series of steps which will move everything buildlet
pool related into a pool package.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: Ic7a0ccd7838345036df2e72b13084070541cb63c
Reviewed-on: https://go-review.googlesource.com/c/build/+/227769
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2020
This change moves all package level accessors into a struct as
suggested by Alex:
https://go-review.googlesource.com/c/build/+/227141/6/internal/coordinator/pool/gce.go#227
This will make it easier to move away from global state in
future changes.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: I005884ccd206fe49651d31ef1d3d336fac9c3d5f
Reviewed-on: https://go-review.googlesource.com/c/build/+/227920
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 4, 2020

Change https://golang.org/cl/232077 mentions this issue: buildlet: add an AWS buildlet client

gopherbot pushed a commit to golang/build that referenced this issue May 4, 2020
This adds the ability for linux-arm64 builder images to be
created on AWS. Instead of writing a bash script which would create
an image with all of the dependencies on it, this experiments with
using packer to create the image.

The image is configured to install and daemonize rundockerbuildlet.
Rundockerbuildlet will download and run the latest stage0 image.
Stage0 then downloads the latest buildlet and begins processing jobs.

A follow-up change will enable rundockerbuildlet to initialize the
buildlet without setting it in reverse mode.

Updates golang/go#36841

Change-Id: Iee07c2600e2b91d34f5e6a1e062f763833d79904
Reviewed-on: https://go-review.googlesource.com/c/build/+/228799
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue May 6, 2020
This change adds an AWS buildlet client which allows us to
create EC2 instances on AWS. With this change we have also
moved a portion of the gce creation logic into a helper
function which allows multiple clients to use it. Metadata
for the instances are stored in the user data fields.

The creation of a buildlet pool and modifications to
rundocker buildlet be made in order to enable this change.

Updates golang/go#36841

Change-Id: Ice03e1520513d51a02b9d66542e00012453bf0d9
Reviewed-on: https://go-review.googlesource.com/c/build/+/232077
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 8, 2020

Change https://golang.org/cl/232997 mentions this issue: buildlet: ensure buildlet client creation timeout is correct

gopherbot pushed a commit to golang/build that referenced this issue May 8, 2020
The buildlet client creation will either timeout at the current
default timeout period or the timeout set in the context. The
existing effective timeout is the five seconds set in the
probeBuildlet function.

Fixes golang/go#38956
Updates golang/go#36841

Change-Id: I09e96e2c5abcc45ccd535596104c52998ddb0d7a
Reviewed-on: https://go-review.googlesource.com/c/build/+/232997
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 13, 2020

Change https://golang.org/cl/233800 mentions this issue: cmd/buildlet: retrieve metadata on EC2 instances

@gopherbot
Copy link

@gopherbot gopherbot commented May 13, 2020

Change https://golang.org/cl/233798 mentions this issue: buildlet: modify the request logic and parameters for EC2 instances

@gopherbot
Copy link

@gopherbot gopherbot commented May 13, 2020

Change https://golang.org/cl/233801 mentions this issue: dashboard, buildenv: label EC2 instances

@gopherbot
Copy link

@gopherbot gopherbot commented May 13, 2020

Change https://golang.org/cl/233799 mentions this issue: cmd/buildlet/stage0: add arm aws hosts

@gopherbot
Copy link

@gopherbot gopherbot commented May 15, 2020

Change https://golang.org/cl/234114 mentions this issue: cmd/rundockerbuildlet: enable running arm EC2 instances

gopherbot pushed a commit to golang/build that referenced this issue May 15, 2020
This enables rundockerbuildlet to run non-reverse buildlet
image on EC2. It will only run a single instance of rundockerbuildlet
once. It exposes port 443 for the coordinator to authenticate with the running
buildlet.

This also adds the buildlet name and buildlet container URL to
the EC2 user data struct retrieved by rundockerbuildlet.

Updates golang/go#36841

Change-Id: I31de754e2ac8970c6f18993104de0e0baea5dc31
Reviewed-on: https://go-review.googlesource.com/c/build/+/234114
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jun 3, 2020

Change https://golang.org/cl/236301 mentions this issue: internal/secret: add references to AWS secrets

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 3, 2020

Change https://golang.org/cl/236297 mentions this issue: env/linux-arm64/aws, dashboard: update linux-arm64 builder image

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 3, 2020

Change https://golang.org/cl/236298 mentions this issue: buildlet: add EC2 security groups to configuration

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 3, 2020

Change https://golang.org/cl/236300 mentions this issue: buildlet: use cloud client with AWS buildlet

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 3, 2020

Change https://golang.org/cl/236299 mentions this issue: internal/cloud: add AWS client libary implementation

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 4, 2020

Change https://golang.org/cl/236577 mentions this issue: cmd/debugnewvm: add EC2 buildlets

gopherbot pushed a commit to golang/build that referenced this issue Jun 5, 2020
This updates the instruction for creating a VM image and
a buildlet container image. The instructions will provide
a better description for the steps required to create the images.

The VM image has been updated to no longer require sudo when
the docker command is executed.

The host configuration has been updated with a new VM type and
VM image.

The rundockerbuildlet service configuration has been modified:
* The memory limit has been removed because rundockerbuildlet will
now use all of the available memory for buildlets running on AWS.
* The service will report as running even after rundockerbuildlet has
exited. Rundockerbuildlet will initiate a single buildlet and then exit.
* The service will not be restarted after rundockerbuildlet has exited.

Changes made to rundockerbuildlet which required these changes have been
made in CL 234114.

Updates golang/go#36841

Change-Id: I3ff51e414b11a2b1427cbd7c6e24e9135fc2a5c3
Reviewed-on: https://go-review.googlesource.com/c/build/+/236297
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Jun 5, 2020
This adds security groups to the builder environment
configuration. Security groups are used to set firewall
rules for the virtual machines.

Updates golang/go#36841

Change-Id: I93235b63144fa80add4986e58246ffc9b44c2e73
Reviewed-on: https://go-review.googlesource.com/c/build/+/236298
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jun 5, 2020

Change https://golang.org/cl/236797 mentions this issue: buildenv, cmd/debugnewvm: add a region to the AWS configuration

gopherbot pushed a commit to golang/build that referenced this issue Jun 12, 2020
This creates an AWS client libary which contains a subset of the
AWS client library. This package provides a fake implementation of
the client library facilitating testing throughout the repository.

Updates golang/go#36841

Change-Id: Id91cd778ee794a6e38a3273c250505a97c6a0b02
Reviewed-on: https://go-review.googlesource.com/c/build/+/236299
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Jun 16, 2020
Use the cloud package for AWS functions. Remove the unused
destroyVM function.

Updates golang/go#36841

Change-Id: I00e1a20c904f7c4be6460ac302085b28f518d161
Reviewed-on: https://go-review.googlesource.com/c/build/+/236300
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Jun 16, 2020
Add references to AWS secrets in the secret package.

Updates golang/go#36841

Change-Id: I9b4232591fe67d3b2cf12ba0236f9b9e5aeca41e
Reviewed-on: https://go-review.googlesource.com/c/build/+/236301
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Jun 16, 2020
This adds the ability to debug the creation of buildlets which run
on EC2.

Updates golang/go#36841

Change-Id: Ib6891bc6ea985716f76e5668ade178541073a344
Reviewed-on: https://go-review.googlesource.com/c/build/+/236577
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 16, 2020

Change https://golang.org/cl/243198 mentions this issue: internal/cloud: add quota and instance type functions

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 17, 2020

Change https://golang.org/cl/243337 mentions this issue: internal/pool: create a common buildlet naming function

gopherbot pushed a commit to golang/build that referenced this issue Jul 17, 2020
This change adds a region for AWS services to the build environment
configuration. The region for the debug vm tool is set in the
corresponding enviornment chosen by the flags set by
buildenv.RegisterFlags call. The region can be overriden
by setting the awsRegion flag.

The identifier for the security groups has been changed to the name of
the group instead of the id since that is what the AWS API expects.
The AWS availability zones have been added to the staging environment.

Updates golang/go#36841

Change-Id: Iee64257dd68d3a75027aaed13f3e767af48a406c
Reviewed-on: https://go-review.googlesource.com/c/build/+/236797
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Jul 20, 2020
This change adds a quota function to the AWS client. It enables
the caller to call the AWS API in order to retrieve quota (soon
to be called limits) on resource limits for caller. This will be used
to set the resource limits on how many vCPU's will be allowed to
reserved for use in the buildlet pool.

The InstanceTypesArm function has been added which will call the AWS
API and retrieve all instance types which support the arm64
architecture. Adding this allows us to store the instance types which
could possibly be called by the buildlet pool and know how many
vCPU's would be reseved for each instance that has been requested.

Updates golang/go#36841

Change-Id: Ib280a41c72f9859876fe03ee2a0d8d5eaf12cc9b
Reviewed-on: https://go-review.googlesource.com/c/build/+/243198
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Jul 20, 2020
The logic for naming buildlets appears in each particular buildlet
pool implementation. This change creates a function which contains
the buildlet naming logic. This is being added before the addition of
a new buildlet pool.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: I8c03f9b513efde14414bcc6d823f3cf1a59c8f70
Reviewed-on: https://go-review.googlesource.com/c/build/+/243337
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 4, 2020

Change https://golang.org/cl/246639 mentions this issue: dashboard: remove obsolete disabled builders

gopherbot pushed a commit to golang/build that referenced this issue Aug 4, 2020
FreeBSD 9.3, FreeBSD 10.x, and OpenBSD 6.0 have
reached end-of-life and are no longer supported.
Go 1.12 was the last release¹ that ran on FreeBSD 10.x.
Go 1.10 was the last release² that ran on OpenBSD 6.0.

These builders have long since been disabled and won't be needed in
the future, so remove them. This work helped detect golang/go#40563.

Modify TestHostConfigsAllUsed to report errors instead of warnings
when it finds unused host configs. Make a map of known exceptions
with rationale or a linked issue for each entry.

¹ https://golang.org/doc/go1.12#ports
² https://golang.org/doc/go1.10#ports

Updates golang/go#36841.
Updates golang/go#19938.

Change-Id: Id7fcdc671bfacf7e70174f4168c9c9cc46ba53d7
Reviewed-on: https://go-review.googlesource.com/c/build/+/246639
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 7, 2020

Change https://golang.org/cl/247400 mentions this issue: buildlet: unexport WaitUntilVMExists method

gopherbot pushed a commit to golang/build that referenced this issue Aug 7, 2020
This change modifies the WaitUntilVMExists method and sets it to
not be exported. It will never be directly called by anything outside
of the EC2Client.

Updates golang/go#36841

Change-Id: If6dfea8c4c191036c8fadd9ad8767e85e55cd224
Reviewed-on: https://go-review.googlesource.com/c/build/+/247400
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 11, 2020

Change https://golang.org/cl/247901 mentions this issue: build: stop explicitly selecting random availability zones

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 11, 2020

Change https://golang.org/cl/247908 mentions this issue: cmd/coordinator: enable EC2 buildlet pool

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 11, 2020

Change https://golang.org/cl/247906 mentions this issue: internal/coordinator/pool: add ledger

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 11, 2020

Change https://golang.org/cl/247907 mentions this issue: internal/coordinator/pool: add ec2 pool

gopherbot pushed a commit to golang/build that referenced this issue Aug 11, 2020
This change stops explicitly selecting a random availability zone upon
EC2 vm creation if one is not set. The EC2 api will select an
availability zone with capacity for the request when an availability
zone is not set. The caller maintains the ability to set an
availability zone if needed.

Updates golang/go#36841

Change-Id: I87d4e8e66fcfadfedbe584a2c016696dc4d62062
Reviewed-on: https://go-review.googlesource.com/c/build/+/247901
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.