Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: coordinator support for performance test execution #49207

Open
prattmic opened this issue Oct 28, 2021 · 10 comments
Open

x/build: coordinator support for performance test execution #49207

prattmic opened this issue Oct 28, 2021 · 10 comments

Comments

@prattmic
Copy link
Member

@prattmic prattmic commented Oct 28, 2021

Background

#48803 tracks the creation of a performance monitoring system for the Go toolchain. This issue covers the first bullet: adding support to the build coordinator for running performance tests.

The performance tests we plan to run fall into one of these categories:

  • Standard “testing” package benchmarks living in the main Go repo or x/benchmarks.
  • “bent” third-party build- and micro-benchmarks.
  • Additional third party application large-scale benchmarks added to x/benchmarks.

Initial Work

Limitations

To start collecting data sooner rather than later, the initial version as described here will simplify the problem by applying the following limitations, which we intend to eventually remove:

  • Builds will be scheduled using the default coordinator priority (LIFO by commit time), rather than a more complex bisection scheme.
  • Benchmarks only run against the Go toolchain version under test, not the baseline.
  • No support for TryBot performance testing.
  • No special snapshotting for benchmark external dependencies.

MVP Design

Builds will initially run on VMs of consistent size and microarchitecture (buildlet named host-linux-amd64-perf). We will characterize the noise level using VMs and may later switch to sole tenant VMs or a dedicated physical machine to reduce noise. The benchmarks have a few external dependencies, notably rsync and perflock. These will be pre-installed in the machine image.

The new build configuration named linux-amd64-perf runs the performance tests as x/benchmarks sub-repo tests. It is configured to run only x/benchmarks tests by setting RunBench = true.

The benchmarks in x/benchmarks are not all run via go test, so runSubrepoTests will have special support for running x/benchmarks benchmarks. The tooling in x/benchmarks is still in flux to improve usability, so rather than encoding minor details about the tools into the coordinator, it will simply execute a to-be-written tool go run golang.org/x/benchmarks/cmd/bench, which is responsible for the details of running all benchmarks.

cmd/bench outputs results to stdout in the Go Benchmark Data Format. The coordinator uploads these results to perfdata.golang.org, adding additional configuration keys like:

  • Go toolchain commit
  • x/benchmarks commit
  • Build time

Old benchmark support

The coordinator has some support for running benchmarks out of x/benchmarks from 2017. This support was disabled in 2018 due to lack of support for some migrations in the coordinator. See CL 354315 for the full enumeration of this code.

This proposal will initially remove nearly all this support, as it is not relevant to the benchmarks we want to run today, and may be confusing to future readers. Some parts will be reused or repurposed, such as:

  • BuildConfig.RunBench to indicate performance test builders.
  • Client code to upload results to perfdata.golang.org.

Future Work

The design above is the minimum necessary to start running tests and collecting data, and is a starting point for future improvements we will want. Here I discuss the future changes we expect to make and the general expected design. We expect the priority and design of these to change as we learn from the running the MVP.

Baseline testing

To minimize noise from environmental changes like OS updates, we would like to run all tests against both the toolchain under test and a “fixed” baseline toolchain, which only changes occasionally (monthly?).

The main change here is to adjust runAllSharded to build both the baseline and test toolchain.

The baseline toolchain version is exposed to the tests as GOROOT_BASELINE.

To take advantage of toolchain snapshotting, we likely want to extend buildStatus.build to support individually fetching the baseline and test toolchain from different snapshots.

We expect this to be the first extension from the MVP design.

Scheduling priority

With many benchmarks and only a single buildlet, we expect that there may not be enough capacity to run every single commit. The coordinator’s currently scheduling algorithm is LIFO by commit time.

During lulls (such as weekends), the system will backfill in LIFO order, which may leave large gaps of untested commits. Instead, we would like to adjust the algorithm to prefer testing commits which will shrink the largest untested gap. i.e., effectively a binary search ordering. This order may even be an improvement to apply to all builds, not just performance tests.

Adding support for this will require plumbing more information about the completed builds into the scheduler.

Benchmark dependency snapshotting / caching

“bent” has large external dependencies fetched over the internet. “bent” fetches many third-party packages (via simple go get). Future benchmarks may fetch pre-built binary assests.

If these operations prove to be very expensive parts of testing, we may want to explore snapshotting these dependencies to save time across builds. The coordinator’s built-in snapshotting mechanism may not provide any speed boost vs fetching over the internet. Instead, read-only and checksummed copies of the dependencies could be saved on the buildlet for use across multiple builds.

The most important aspect here is to ensure that a test of a bad version of the toolchain can’t corrupt the cache in a way that breaks future builds.

cc @mknyszek @aclements @dr2chase @jeremyfaller @golang/release

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 28, 2021

Change https://golang.org/cl/354315 mentions this issue: cmd/coordinator,dashboard,internal/buildgo: remove benchmark support

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 29, 2021

Change https://golang.org/cl/359854 mentions this issue: cmd/bench: add benchmark wrapper

Loading

gopherbot pushed a commit to golang/build that referenced this issue Nov 4, 2021
Running benchmarks has been disabled since 2018. Remove all the code to
keep things more maintainable and understandable.

We will be adding new benchmarking support soon, and may reuse some of
this code, but don't want half-working code adding confusion.

For golang/go#49207.

Change-Id: I11d52b0315bed4d91651c162af11853895012868
Reviewed-on: https://go-review.googlesource.com/c/build/+/354315
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/benchmarks that referenced this issue Nov 4, 2021
The coordinator is getting support for running the benchmarks in this
repository. Since the benchmarks and interface are in flux, encoding all
of the details of running Go tests, bent arguments, etc into the
coordinator will likely cause churn and frustrating migration issues.

Instead, add cmd/bench which serves as the simple entrypoint for the
coordinator. The coordinator runs cmd/bench with the GOROOT to test
(eventually multiple GOROOTs), and this binary takes care of the
remaining details.

Right now, we just do a basic go test golang.org/x/benchmarks/... and
simple invocation of bent. Note that bent does not pass without
https://golang.org/cl/354634.

For golang/go#49207

Change-Id: I5c9cf89540cab605c0a64e17af85311d37985c25
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/359854
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 4, 2021

Change https://golang.org/cl/361418 mentions this issue: cmd/coordinator: upload performance test results to perfdata.golang.org

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 4, 2021

Change https://golang.org/cl/361417 mentions this issue: cmd/coordinator: run performance tests from x/benchmarks

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 4, 2021

Change https://golang.org/cl/354311 mentions this issue: dashboard: add linux-amd64-perf host and builder

Loading

gopherbot pushed a commit to golang/build that referenced this issue Nov 5, 2021
Add initial support for running the performance tests from x/benchmarks.
Since there are a variety of different test suites (some `go test`,
bent, etc), x/benchmarks provides a basic wrapper command,
golang.org/x/benchmarks/cmd/bench which know the minute details. The
coordinator just needs to run that one command.

This build mode is limited to builds of x/benchmarks on builders with
RunBench set to true. Currently there are none, a future CL will add the
initial such linux-amd64 builder.

For golang/go#49207

Change-Id: Ie006ec4a3757a5c2fed0925a3f9eb91edeaa5224
Reviewed-on: https://go-review.googlesource.com/c/build/+/361417
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Nov 5, 2021
The performance test output contains benchfmt-formatted benchmark
results. Upload the output wholesale to perfdata.golang.org for
long-term storage and analysis.

The results for now are a bit rough, as the output may also contain
unrelated output that lines that look like benchfmt. For example. "go:
downloading github.com/BurntSushi/toml v0.3.1" adds a "go" label with
the value "downloading ...". In the future, we will ideally filter these
a bit better (perhaps in x/benchmarks/cmd/bench).

For golang/go#49207

Change-Id: Ifd2512c93902a74f9040db0f9d0c600348fc1849
Reviewed-on: https://go-review.googlesource.com/c/build/+/361418
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Nov 5, 2021
Add a new builder to run the x/benchmarks performance tests on
linux-amd64.

For now, this runs on a GCE C2 instance type, as these instances have
well-defined, consistent CPUs and other server architecture components.

In basic noise testing, even standard VMs of this type appear to be
fairly low noise. As we gain experience with actual monitoring, we may
change this to a sole-tenant VM type or even a dedicated machine if
necessary.

For golang/go#49207

Change-Id: I17eaeeb5349af925249940bebd5b860a2579e6df
Reviewed-on: https://go-review.googlesource.com/c/build/+/354311
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 5, 2021

Change https://golang.org/cl/361656 mentions this issue: internal/coordinator/pool: count C2 and N2 quotas separately

Loading

gopherbot pushed a commit to golang/build that referenced this issue Nov 5, 2021
We currently use E2, C2, and N2 instances on GCE. C2 and N2 instances
have their own quotas, which are accounted separately from the CPUS
quotas.

This could probably be cleaned up to keep track of all CPU quotas and
handle more instance types, but this should work for the time being.

See: https://cloud.google.com/compute/quotas#cpu_quota

For golang/go#49207

Change-Id: Ida1e8de3c857560637095d57e972bca7222284ed
Reviewed-on: https://go-review.googlesource.com/c/build/+/361656
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 5, 2021

Change https://golang.org/cl/361734 mentions this issue: cmd/bent: remove required dependencies

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 5, 2021

Change https://golang.org/cl/361754 mentions this issue: dashboard: SkipSnapshot for linux-amd64-perf

Loading

gopherbot pushed a commit to golang/build that referenced this issue Nov 5, 2021
Since this builder doesn't build the go repo, it will be waiting forever
for a snapshot. Instead, just build Go for each run.

For golang/go#49207

Change-Id: I34a73b507278db402c478b4f5956633996772aae
Reviewed-on: https://go-review.googlesource.com/c/build/+/361754
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 5, 2021

Change https://golang.org/cl/361874 mentions this issue: cmd/coordinator: add basic metadata to perfdata upload

Loading

gopherbot pushed a commit to golang/benchmarks that referenced this issue Nov 5, 2021
Make rsync optional with fallback to cp.

Remove use of /usr/bin/time and replace with measuring time directly
from Go.

For golang/go#49207

Change-Id: Ief5a7a90f9460ddec1d5a51c99d4a534e38a5d9c
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/361734
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: David Chase <drchase@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Nov 8, 2021
These make it possible to tell what was run, as well as a convenience
field stating whether this was a post-submit build or a trybot run.

For golang/go#49207

Change-Id: Iba979bcfd5a3bbdc11e2df0b8de4094cc7212356
Reviewed-on: https://go-review.googlesource.com/c/build/+/361874
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 8, 2021

Change https://golang.org/cl/362375 mentions this issue: cmd/bench: wait for load average to drop before starting

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants