Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: add Amazon EC2 ARM instances for builders #36841

Closed
cagedmantis opened this issue Jan 28, 2020 · 39 comments
Closed

x/build: add Amazon EC2 ARM instances for builders #36841

cagedmantis opened this issue Jan 28, 2020 · 39 comments

Comments

@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Jan 28, 2020

As @bradfitz noted, we should explore adding ephemeral AWS ARM instances for both 32-bit and 64-bit ARM builds.

@toothrot @dmitshur

@toothrot
Copy link
Contributor

@toothrot toothrot commented Jan 28, 2020

See also #36170, for discussion of some of our requirements on ARM builders. We still have work to do there, but it might be worth taking care of it all at once.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jan 28, 2020

@toothrot, if we did this bug, that bug would be not applicable.

We'd run each buildlet in its own VM (like we do on GCE, except using AWS), not using Docker (like Packet & Scaleway & PPC{,64}, etc).

We now have quota from AWS to run this for free, so it's just a matter of writing an AWSBuildletPool implementation.

@toothrot
Copy link
Contributor

@toothrot toothrot commented Jan 28, 2020

I understand that, but I wanted to make sure some of the other requirements were still being paid attention to, like creating a highcpu builder.

@cagedmantis cagedmantis self-assigned this Feb 26, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 1, 2020

Change https://golang.org/cl/226843 mentions this issue: internal/spanlog: moves spanlog into the internal directory

gopherbot pushed a commit to golang/build that referenced this issue Apr 2, 2020
This is the first in a series of patches which include an
effort to refactor how the cmd/coordinator is structured. In order
to move buildlet pools into their own package, it would facilitate
the move to have spanlog in it's own internal package. This CL will
be followed by one which removes cmd/coordinator/spanlog and
changes the path of the package for all the callers.

Updates golang/go#36841

Change-Id: Ic310a38f807a3099815e0bdb988edc42b2ca5c85
Reviewed-on: https://go-review.googlesource.com/c/build/+/226843
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 7, 2020

Change https://golang.org/cl/227023 mentions this issue: cmd/coordinator/spanlog: remove spanlog package

gopherbot pushed a commit to golang/build that referenced this issue Apr 7, 2020
This removes the spanlog package located inside the
coordinator package. The spanlog package has already been
copied over to the internal package. All imports in the build
repository have been changes to point to the internal spanlog
package.

Updates golang/go#36841

Change-Id: I84499bcfe782c566b12dabcbfa743cd8ce559f18
Reviewed-on: https://go-review.googlesource.com/c/build/+/227023
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 7, 2020

Change https://golang.org/cl/227141 mentions this issue: internal/pool: move the gce buildlet pool into a pool package

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 8, 2020

Change https://golang.org/cl/227578 mentions this issue: internal/pool: move the kube buildlet pool into a pool package

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 9, 2020

Change https://golang.org/cl/227769 mentions this issue: internal/pool: move the reverse buildlet pool into a pool package

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 9, 2020

Change https://golang.org/cl/227768 mentions this issue: internal/pool: move the kubernetes buildlet pool into a pool package

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 10, 2020

Change https://golang.org/cl/227920 mentions this issue: internal/pool: remove package level accessors

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 17, 2020

Change https://golang.org/cl/228799 mentions this issue: all: add linux-arm64-aws builders

gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2020
This CL creates the internal/coordinator/pool package intended to
contain all buildlet pool implementations. In order to keep this
change small and carefully discover where the interactions are
between the gce buildlet pool and the rest of the coordinator
are, this change only moves the gce buildlet over to the new
package.

The next steps will be to move the rest of the buildlet pools
over to this package. After that we will restructure the
implementations themselves in order to increase test coverage
and increase the ease of testing.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: If82ae1b584bd77c697aa84fadf9011c9e79fa409
Reviewed-on: https://go-review.googlesource.com/c/build/+/227141
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2020
This is a set in a series of steps which will move everything buildlet
pool related into a pool package.

Updates golang/go#36841

Change-Id: I8efb1f94c7b929be559004d9f455bca0370c7800
Reviewed-on: https://go-review.googlesource.com/c/build/+/227768
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2020
This is a set in a series of steps which will move everything buildlet
pool related into a pool package.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: Ic7a0ccd7838345036df2e72b13084070541cb63c
Reviewed-on: https://go-review.googlesource.com/c/build/+/227769
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2020
This change moves all package level accessors into a struct as
suggested by Alex:
https://go-review.googlesource.com/c/build/+/227141/6/internal/coordinator/pool/gce.go#227
This will make it easier to move away from global state in
future changes.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: I005884ccd206fe49651d31ef1d3d336fac9c3d5f
Reviewed-on: https://go-review.googlesource.com/c/build/+/227920
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 4, 2020

Change https://golang.org/cl/232077 mentions this issue: buildlet: add an AWS buildlet client

gopherbot pushed a commit to golang/build that referenced this issue May 4, 2020
This adds the ability for linux-arm64 builder images to be
created on AWS. Instead of writing a bash script which would create
an image with all of the dependencies on it, this experiments with
using packer to create the image.

The image is configured to install and daemonize rundockerbuildlet.
Rundockerbuildlet will download and run the latest stage0 image.
Stage0 then downloads the latest buildlet and begins processing jobs.

A follow-up change will enable rundockerbuildlet to initialize the
buildlet without setting it in reverse mode.

Updates golang/go#36841

Change-Id: Iee07c2600e2b91d34f5e6a1e062f763833d79904
Reviewed-on: https://go-review.googlesource.com/c/build/+/228799
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue May 6, 2020
This change adds an AWS buildlet client which allows us to
create EC2 instances on AWS. With this change we have also
moved a portion of the gce creation logic into a helper
function which allows multiple clients to use it. Metadata
for the instances are stored in the user data fields.

The creation of a buildlet pool and modifications to
rundocker buildlet be made in order to enable this change.

Updates golang/go#36841

Change-Id: Ice03e1520513d51a02b9d66542e00012453bf0d9
Reviewed-on: https://go-review.googlesource.com/c/build/+/232077
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 8, 2020

Change https://golang.org/cl/232997 mentions this issue: buildlet: ensure buildlet client creation timeout is correct

gopherbot pushed a commit to golang/build that referenced this issue May 8, 2020
The buildlet client creation will either timeout at the current
default timeout period or the timeout set in the context. The
existing effective timeout is the five seconds set in the
probeBuildlet function.

Fixes golang/go#38956
Updates golang/go#36841

Change-Id: I09e96e2c5abcc45ccd535596104c52998ddb0d7a
Reviewed-on: https://go-review.googlesource.com/c/build/+/232997
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented May 13, 2020

Change https://golang.org/cl/233800 mentions this issue: cmd/buildlet: retrieve metadata on EC2 instances

@gopherbot
Copy link

@gopherbot gopherbot commented May 13, 2020

Change https://golang.org/cl/233798 mentions this issue: buildlet: modify the request logic and parameters for EC2 instances

@gopherbot
Copy link

@gopherbot gopherbot commented May 13, 2020

Change https://golang.org/cl/233801 mentions this issue: dashboard, buildenv: label EC2 instances

@gopherbot
Copy link

@gopherbot gopherbot commented May 13, 2020

Change https://golang.org/cl/233799 mentions this issue: cmd/buildlet/stage0: add arm aws hosts

gopherbot pushed a commit to golang/build that referenced this issue Jul 17, 2020
This change adds a region for AWS services to the build environment
configuration. The region for the debug vm tool is set in the
corresponding enviornment chosen by the flags set by
buildenv.RegisterFlags call. The region can be overriden
by setting the awsRegion flag.

The identifier for the security groups has been changed to the name of
the group instead of the id since that is what the AWS API expects.
The AWS availability zones have been added to the staging environment.

Updates golang/go#36841

Change-Id: Iee64257dd68d3a75027aaed13f3e767af48a406c
Reviewed-on: https://go-review.googlesource.com/c/build/+/236797
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Jul 20, 2020
This change adds a quota function to the AWS client. It enables
the caller to call the AWS API in order to retrieve quota (soon
to be called limits) on resource limits for caller. This will be used
to set the resource limits on how many vCPU's will be allowed to
reserved for use in the buildlet pool.

The InstanceTypesArm function has been added which will call the AWS
API and retrieve all instance types which support the arm64
architecture. Adding this allows us to store the instance types which
could possibly be called by the buildlet pool and know how many
vCPU's would be reseved for each instance that has been requested.

Updates golang/go#36841

Change-Id: Ib280a41c72f9859876fe03ee2a0d8d5eaf12cc9b
Reviewed-on: https://go-review.googlesource.com/c/build/+/243198
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Jul 20, 2020
The logic for naming buildlets appears in each particular buildlet
pool implementation. This change creates a function which contains
the buildlet naming logic. This is being added before the addition of
a new buildlet pool.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: I8c03f9b513efde14414bcc6d823f3cf1a59c8f70
Reviewed-on: https://go-review.googlesource.com/c/build/+/243337
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 4, 2020

Change https://golang.org/cl/246639 mentions this issue: dashboard: remove obsolete disabled builders

gopherbot pushed a commit to golang/build that referenced this issue Aug 4, 2020
FreeBSD 9.3, FreeBSD 10.x, and OpenBSD 6.0 have
reached end-of-life and are no longer supported.
Go 1.12 was the last release¹ that ran on FreeBSD 10.x.
Go 1.10 was the last release² that ran on OpenBSD 6.0.

These builders have long since been disabled and won't be needed in
the future, so remove them. This work helped detect golang/go#40563.

Modify TestHostConfigsAllUsed to report errors instead of warnings
when it finds unused host configs. Make a map of known exceptions
with rationale or a linked issue for each entry.

¹ https://golang.org/doc/go1.12#ports
² https://golang.org/doc/go1.10#ports

Updates golang/go#36841.
Updates golang/go#19938.

Change-Id: Id7fcdc671bfacf7e70174f4168c9c9cc46ba53d7
Reviewed-on: https://go-review.googlesource.com/c/build/+/246639
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 7, 2020

Change https://golang.org/cl/247400 mentions this issue: buildlet: unexport WaitUntilVMExists method

gopherbot pushed a commit to golang/build that referenced this issue Aug 7, 2020
This change modifies the WaitUntilVMExists method and sets it to
not be exported. It will never be directly called by anything outside
of the EC2Client.

Updates golang/go#36841

Change-Id: If6dfea8c4c191036c8fadd9ad8767e85e55cd224
Reviewed-on: https://go-review.googlesource.com/c/build/+/247400
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 11, 2020

Change https://golang.org/cl/247901 mentions this issue: build: stop explicitly selecting random availability zones

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 11, 2020

Change https://golang.org/cl/247908 mentions this issue: cmd/coordinator: enable EC2 buildlet pool

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 11, 2020

Change https://golang.org/cl/247906 mentions this issue: internal/coordinator/pool: add ledger

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 11, 2020

Change https://golang.org/cl/247907 mentions this issue: internal/coordinator/pool: add ec2 pool

gopherbot pushed a commit to golang/build that referenced this issue Aug 11, 2020
This change stops explicitly selecting a random availability zone upon
EC2 vm creation if one is not set. The EC2 api will select an
availability zone with capacity for the request when an availability
zone is not set. The caller maintains the ability to set an
availability zone if needed.

Updates golang/go#36841

Change-Id: I87d4e8e66fcfadfedbe584a2c016696dc4d62062
Reviewed-on: https://go-review.googlesource.com/c/build/+/247901
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Aug 17, 2020
This change adds a ledger to the coordinator pool package. The ledger
is used to store a record of EC2 instance types created and their
attributes. It also keeps a record of resource usage for EC2 instances
in order to ensure that we do not exceed the resource limits for the account.

Updates golang/go#36841

Change-Id: I6a0afdb31c8e3a83634e7c1fc8b2b733b7a50c01
Reviewed-on: https://go-review.googlesource.com/c/build/+/247906
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Aug 18, 2020
The EC2 buildlet pool added by this commit will manage the lifecycle
of buildlets running on EC2. EC2 VMs will only be created for the
ARM64 architecture. As VMs are requested, the pool will ensure that
there are enough resources for the VM to be created and keep track of
the VMs created. Once a VM is destroyed, the resources will be made
available for other VMs. This pool will only keep instances as they
are needed.

Updates golang/go#36841
Updates golang/go#38337

Change-Id: Ic777485c0b0a69ec13726c58b49e9fdc1df4777e
Reviewed-on: https://go-review.googlesource.com/c/build/+/247907
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 18, 2020

Change https://golang.org/cl/249118 mentions this issue: dashboard: add host-linux-arm64-aws builder

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 18, 2020

Change https://golang.org/cl/249121 mentions this issue: internal/coordinator/pool: check if EC2 instance is a buildlet

gopherbot pushed a commit to golang/build that referenced this issue Aug 19, 2020
Before destroying an untracked EC2 instance, verify that it is a
buildlet. Non buildlets will be destroyed manually as they are most
likely instances created while an operator is debugging a problem.

For golang/go#36841

Change-Id: Id0f65192ce4e72f9ecc495dd3e94750724091b51
Reviewed-on: https://go-review.googlesource.com/c/build/+/249121
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Aug 19, 2020
Add the host-linux-arm64-aws builder. The HostConfig IsVM method has
been modified to support the concept that an EC2 VM can be created
which runs the buildlet binary in a container. This configuration
would have both a VM image and a container image set in the host
configuration.

For golang/go#36841

Change-Id: I092b48b1dda24d24bcd0b10dff0b2126f5f43017
Reviewed-on: https://go-review.googlesource.com/c/build/+/249118
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 1, 2020

Change https://golang.org/cl/252317 mentions this issue: cmd/rundockerbuildlet: retrieve host type from metadata on EC2

gopherbot pushed a commit to golang/build that referenced this issue Sep 1, 2020
When rundockerbuildlet is running on an EC2 instance, the
configuration should be pulled in from the metadata. This is part of
the effort to move some configuration for builders over to the
coordinator.

For golang/go#36841

Change-Id: Icbf0e73301faadd06c87f02aa1172bd0e6f87624
Reviewed-on: https://go-review.googlesource.com/c/build/+/252317
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 16, 2020

Change https://golang.org/cl/255358 mentions this issue: internal, internal/coordinator/pool: add a shared polling function

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 16, 2020

Change https://golang.org/cl/255359 mentions this issue: buildenv: remove EC2 zone configuration options

gopherbot pushed a commit to golang/build that referenced this issue Sep 16, 2020
There is no need to set availability zones for EC2 resources at this
time. The resources currently in use are available to all zones within
a single region.

For golang/go#36841

Change-Id: Ifcb2c3404fba2489741c31fe63ad4e7747639ad9
Reviewed-on: https://go-review.googlesource.com/c/build/+/255359
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Oct 1, 2020
This adds a polling function that is tested and can be reused by
callers in the x/build repository. This was suggested in the review of
CL 247907. The polling function replaces the pollers used in the EC2 buildlet pool.

For golang/go#36841

Change-Id: I120ceb83e2740f0bdc5ee2423e0edd3ad727bf4b
Reviewed-on: https://go-review.googlesource.com/c/build/+/255358
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 30, 2020

Change https://golang.org/cl/266697 mentions this issue: dashboard, env/linux-arm64/aws: update EC2 ami search query

gopherbot pushed a commit to golang/build that referenced this issue Nov 2, 2020
This change updates which source ami is used as a base for the
creation of the linux-arm64-aws builder. Packer will now search for
the latest version of the arm64 Debian image to use as the source ami.

For golang/go#36841

Change-Id: I036dbaf7ee65d4e7028cd6a0cdf6710bf898fdd2
Reviewed-on: https://go-review.googlesource.com/c/build/+/266697
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants