all: Go compiler/runtime performance monitoring system #48803

prattmic · 2021-10-05T20:18:02Z

The runtime/compiler team is investigating the creation of a performance monitoring system, with a primary goal of reducing release toil by catching performance regressions as early as possible.

Requirements are still being discussed, but highlights include:

Automation of performance data generation.
- Benchmark execution solution that tracks commits to Go (e.g., x/build coordinator scheduling benchmark runs).
Storage and visualization of performance data over time.
Key set of "application-level" and "feature-focused" benchmarks.
Active monitoring of changepoints.
Easy debugging/reproduction.

cc @aclements @dr2chase @mknyszek @jeremyfaller @golang/release

mvdan · 2021-10-05T20:27:30Z

cc @odeke-em

odeke-em · 2021-10-05T20:37:25Z

cc @odeke-em

Thank you for the tag @mvdan!

Indeed, at Orijtech Inc we've been working on such a product "Bencher" for the past 2 quarters of 2021 and produced it recently https://twitter.com/odeke_et/status/1428650768135974919?s=21: we have an integration on the GitHub marketplace at https://github.com/marketplace/gobencher and we actually have something on our roadmap to add a Gerrit integration during Q4 2021 or Q1 2022. I had raised and shown the product to @Sajmani, @spf13, @ianlancetaylor and @ianthehat and others :-)

For example, take a look at one of our public users Entgo https://dashboard.github.orijtech.com/benchmark/2bb34b8e166747d2974f76825819acbd

We'd be delighted to work with y'all to bring this first class to the Go tooling ecosystem!

Kindly cc-ing my colleagues @cuonglm @kirbyquerby @willpoint @jhusdero

mengzhuo · 2021-10-06T14:41:43Z

I have some thoughts about this site/app:

Transparent. All results should be available to public like benchsave instead of internal site from Google.
Informative. Regression notice to Gerrit/owner of CL.
Easy to compare with different baselines like major versions of Go.

mknyszek · 2021-10-06T15:13:15Z

@mengzhuo All three of those things are already on our radar. :) Thanks.

gopherbot · 2021-10-06T15:48:11Z

Change https://golang.org/cl/353909 mentions this issue: cmd/coordinator: don't snapshot in dev mode

In dev mode, we have no GCS storage client, so attempting to write a snapshot will panic. For golang/go#48803 Change-Id: Id37264ebd765f914a55acf2fd18274020850331f Reviewed-on: https://go-review.googlesource.com/c/build/+/353909 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

prattmic · 2021-10-06T18:56:27Z

@odeke-em We are still evaluating our options. We've looked at several different off-the-shelf options, including Bencher, as well as extending the existing http://build.golang.org infrastructure with what we need. We are currently leaning towards extending http://build.golang.org, as we already maintain it and it supports much of what we need already (fully open source, precise control over dedicated hardware used for benchmarking, etc).

gopherbot · 2021-10-07T17:44:26Z

Change https://golang.org/cl/354629 mentions this issue: cmd/buildlet: add revdial test

gopherbot · 2021-10-07T17:44:26Z

Change https://golang.org/cl/354630 mentions this issue: all: handle revdial redirects on connect

gopherbot · 2021-10-07T20:55:51Z

Change https://golang.org/cl/354638 mentions this issue: cmd/coordinator: find work in dev mode

gopherbot · 2021-10-07T20:55:52Z

Change https://golang.org/cl/354637 mentions this issue: cmd/coordinator: make listen address configurable

This replaces the broken test in internal/coordinator/pool. For golang/go#48803 Change-Id: I7bd9265bba555562ffa7d59169a9c8792ed97d3c Reviewed-on: https://go-review.googlesource.com/c/build/+/354629 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>

In dev mode, the coordinator uses hostPathHandler, which redirects /reverse and /revdial to /farmer.golang.org/reverse, etc. Establishing a revdial connection chokes on this redirect, as there it expects to complete the protocol switch in a single request. Add rudimentary redirect support via a helper so this works in dev mode. Note that linkRewriter must implement http.Hijacker as revdial.ConnHandler type-asserts the http.ResponseWriter to a Hijacker. For golang/go#48803 Change-Id: I191fa6ff17bbd334203430f3c1f2c5e03db407ff Reviewed-on: https://go-review.googlesource.com/c/build/+/354630 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>

Though it is a good default, it is cumbersome for dev mode to listen only on localhost when developing over SSH. Add a -listen flag to allow overriding this with any address. For golang/go#48803 Change-Id: If01a0b44926a33f2aa01548319508166592936e3 Reviewed-on: https://go-review.googlesource.com/c/build/+/354637 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>

Start the main findWorkLoop in dev mode, to discover work to run from the dashboard. In dev mode, was also replace the linux-amd64 builder config with one using host-linux-amd64-localdev reverse buildlets. This provides a complete lifecycle to test out builds with a local dev coordinator and buildlet. For golang/go#48803 Change-Id: I8ade6c8bccf3bc51437ca9e7d11c232753fe7465 Reviewed-on: https://go-review.googlesource.com/c/build/+/354638 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>

shawndx · 2021-11-17T02:05:43Z

Curious to know will benchmark fluctuation be taken into account, and how? Thanks.

prattmic · 2021-11-17T16:02:51Z

@shawn-xdji By fluctuation, I assume you mean noise in the results. We are taking a few approaches up front:

Running benchmarks on a low-noise machines. For now we plan to use "sole-tenant" VMs on GCE, though we may move to physical machines if necessary.
Rerunning the baseline benchmarks each time we test a new commit to help account for long term changes (e.g., OS upgrade changes performance).
Changepoint detection will need to statistical handle a small amount of noise.
We are only planning to track a curated set of benchmarks. Benchmarks that a fundamentally noisy will either be excluded or fixed.

Finally, all of this is something we'll need to learn on as we go and find out how big the issues are.

shawndx · 2021-11-18T02:42:08Z

Thanks @prattmic, that answers my question, anticipate you will share some best practices later.

gopherbot · 2022-01-07T21:20:54Z

Change https://golang.org/cl/376634 mentions this issue: cmd/coordinator: baseline toolchain for benchmarks

When benchmarking, we want to benchmark both the toolchain under test (i.e., buildStatus.Rev) as well as an older "baseline" toolchain, which will be compared against. For now, the baseline toolchain is the latest stable release. In the future we may want to update more frequently, but this is a simple starting point. This CL determines the baseline toolchain commit for a given test and installs it on the buildlet at BENCH_BASELINE_GOROOT. golang.org/x/benchmarks/cmd/bench is responsible for utilizing the baseline toolchain. CL 376096 is the corresponding change to cmd/bench. Most of the baseline toolchain logic is limited to runBenchmarkTests(). In theory, it logically fits a bit better with the rest of the toolchain logic in build() et al, but keeping it limited to runBenchmarkTests() helps keep the common build() path from getting much more complex for a minor edge-case feature. For golang/go#49207. For golang/go#48803. Change-Id: Id63f8333cf9d1ff952850c3347e999b5e98f7294 Reviewed-on: https://go-review.googlesource.com/c/build/+/376634 Reviewed-by: Alex Rakoczy <alex@golang.org> Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

gopherbot · 2022-03-16T21:42:49Z

Change https://go.dev/cl/392635 mentions this issue: influx: InfluxDB instance setup

This CL creates a Docker image for running an InfluxDB instance for the Go performance monitoring dashboard. The image is based on the Google-maintained GCP InfluxDB 2 image, with an additional small program to perform initial database setup and push access credentials to Google Secret Manager. See README.md for instructions on running the image locally or on GCP. For golang/go#48803 Change-Id: Ica34bd624f6daf89e483f898b110ccaceac83559 Reviewed-on: https://go-review.googlesource.com/c/perf/+/392635 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>

gopherbot · 2022-03-21T20:38:05Z

Change https://go.dev/cl/394354 mentions this issue: influx: InfluxDB instance setup

gopherbot · 2022-03-21T20:38:05Z

Change https://go.dev/cl/394358 mentions this issue: WIP: influx: add kubernetes deployment

gopherbot · 2022-06-21T15:45:30Z

Change https://go.dev/cl/413419 mentions this issue: perf: add loading message

The ?benchmark query param specifies the benchmarks to fetch. "all" fetches all benchmarks. Omitting it returns the default set. For golang/go#48803. Change-Id: I221ac3a702eb0cbb263ef86b2723e5d951662cef Reviewed-on: https://go-review.googlesource.com/c/build/+/412998 Reviewed-by: Michael Knyszek <mknyszek@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

For golang/go#48803. Change-Id: I4ec7b51918fc780729c218d8f25478e539700bc5 Reviewed-on: https://go-review.googlesource.com/c/build/+/413418 Reviewed-by: Michael Knyszek <mknyszek@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

gopherbot · 2022-06-21T17:15:21Z

Change https://go.dev/cl/413421 mentions this issue: perf: link benchmark titles

gopherbot · 2022-06-21T17:30:17Z

Change https://go.dev/cl/413423 mentions this issue: perf: add a few additional homepage benchmarks

Loading can take a few seconds, so add a basic note that something is coming. For golang/go#48803. Change-Id: Ic24483f5729395cfde87ba751c851950abe48281 Reviewed-on: https://go-review.googlesource.com/c/build/+/413419 Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>

For golang/go#48803. Change-Id: I678a2dc5e668179d0fbaf4401aa810ad930c7933 Reviewed-on: https://go-review.googlesource.com/c/build/+/413421 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com> Run-TryBot: Michael Pratt <mpratt@google.com>

For golang/go#48803. Change-Id: I2cee352261b1e432e625d6d5d2d081b78d926c1a Reviewed-on: https://go-review.googlesource.com/c/build/+/413423 Reviewed-by: Michael Knyszek <mknyszek@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

gopherbot · 2022-06-21T19:23:18Z

Change https://go.dev/cl/413426 mentions this issue: perf: add brief dashboard overview

prattmic · 2022-06-21T21:57:22Z

For those that have been following along, the current basic dashboard is available at https://perf.golang.org/dashboard/.

prattmic · 2022-06-21T22:04:45Z

Note that we are still refining the set of quality, stable benchmarks and additional work on reducing nodes on executors.

jake-ciolek · 2022-06-22T12:14:30Z

@prattmic Awesome stuff, would it be possible to expand the date range further back?

prattmic · 2022-06-22T13:36:31Z

@jake-ciolek Yes, I'd like to make the date range configurable (at least to show the entire release cycle), but haven't gotten to that yet.

gopherbot · 2022-06-22T15:41:59Z

Change https://go.dev/cl/413574 mentions this issue: cmd/coordinator: add header link to perf dashboard

gopherbot · 2022-06-22T19:52:23Z

Change https://go.dev/cl/413577 mentions this issue: perf: don't catch 404 throw

gopherbot · 2022-06-22T19:52:24Z

Change https://go.dev/cl/413576 mentions this issue: perf: shrink data.json

gopherbot · 2022-06-23T15:16:52Z

Change https://go.dev/cl/413578 mentions this issue: perf: make duration configurable

For golang/go#48803. Change-Id: I15393e80fa9c2a961e7feacc3acd2b72de312b82 Reviewed-on: https://go-review.googlesource.com/c/build/+/413426 Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>

For golang/go#48803. Change-Id: Ied4ff67757282bdd33f6e85b652762a18c369e8d Reviewed-on: https://go-review.googlesource.com/c/build/+/413574 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

/dashboard/data.json?benchmark=all reduced from 7.8MB to 2.0MB with gzip compression, and from 2.0MB to 1.1MB by removing the indentation. For golang/go#48803. Change-Id: I9031d23b1b751e130521658fd45551771cec04da Reviewed-on: https://go-review.googlesource.com/c/build/+/413576 Run-TryBot: Michael Pratt <mpratt@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

This catches all exceptions, so e.g., a json parsing error is completely swallowed without even appearing in the JS console, which makes debugging difficult. For golang/go#48803. Change-Id: If62f153413cdb2dfbeab2d83ddf6aaedc43cce8c Reviewed-on: https://go-review.googlesource.com/c/build/+/413577 Reviewed-by: Michael Knyszek <mknyszek@google.com>

gopherbot · 2022-06-24T18:48:24Z

Change https://go.dev/cl/414036 mentions this issue: cmd/coordinator, perf: improve header consistency

I forgot there was another copy of the template in status.go. Move at least coordinator's 2 copies of the header into one common place; we can do more later. Also update the perf header to be consistent and link back to the coordinator status page for easier navigation. For golang/go#48803. Change-Id: Id3f789d529e349cde1557ed17dd713549f55b942 Reviewed-on: https://go-review.googlesource.com/c/build/+/414036 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>

This adds a ?day=N parameter that adjusts the query range, and ?end=TIME parameter that sets the view end time. Addition form fields allow adjusting these settings. Though the search box and duration/end boxes are conceptually different adjustments, and have separate submit buttons, put them inside the same <form> so that the browser automatically sets all query parameters as expected. e.g., when on a single benchmark page, changing duration/end should not change the selected benchmark. For golang/go#48803. Change-Id: I8ff366983c568dc0e887289a174082c248934c2d Reviewed-on: https://go-review.googlesource.com/c/build/+/413578 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com> Run-TryBot: Michael Pratt <mpratt@google.com>

prattmic · 2022-06-27T14:37:20Z

@jake-ciolek https://go.dev/cl/413578 is rolled out, so now the graph duration and end date are adjustable.

gopherbot · 2022-06-27T15:23:30Z

Change https://go.dev/cl/414474 mentions this issue: perf: remove /trend

jake-volvo · 2022-06-28T09:26:07Z

@jake-ciolek https://go.dev/cl/413578 is rolled out, so now the graph duration and end date are adjustable.

Thank you. Great work.

I have noticed the data goes back only to 22nd of March (even if I query for more). Would it be possible to extend it?

Also, could we look in to other metrics, like the time to compile Go itself (compilebench?), the size of Go binary broken down by funcs (feed data from something like compilecmp). Perhaps timings of individual SSA passes. Just some ideas for things that could be useful.

prattmic · 2022-06-28T17:46:36Z

I have noticed the data goes back only to 22nd of March (even if I query for more). Would it be possible to extend it?

That is when we started collecting data. It is theoretically possible to test older commits, but I don't think it is worth the effort since we have plenty to investigate already.

Also, could we look in to other metrics, like the time to compile Go itself (compilebench?), the size of Go binary broken down by funcs (feed data from something like compilecmp). Perhaps timings of individual SSA passes. Just some ideas for things that could be useful.

The benchmarks we run are in x/benchmarks/cmd/bench (which most consists of configurations of x/benchmarks/cmd/bent (various unit benchmarks from the community) and x/benchmarks/sweet (large application-scale benchmarks)). We have a minimal set of benchmarks right now, and are certainly open to benchmarking additional things, like the ideas you suggested.

It has been superseded by /dashboard, so let's remove it to keep things simple. For golang/go#48803. Change-Id: I5b7ca91c72ee8f7bcfe762fdcd7400a619611058 Reviewed-on: https://go-review.googlesource.com/c/build/+/414474 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Pratt <mpratt@google.com> Reviewed-by: Austin Clements <austin@google.com>

prattmic added the Performance label Oct 5, 2021

mknyszek added the NeedsFix The path to resolution is known, but the work has not been done. label Oct 5, 2021

mknyszek added this to the Unreleased milestone Oct 5, 2021

aclements mentioned this issue Oct 8, 2021

testing: benchmark unit properties #43744

Open

This was referenced Oct 25, 2021

x/perf/storage: support postgres for db #49121

Open

x/build, x/build/cmd/coordinator: support for performance test execution #49207

Closed

prattmic mentioned this issue Jun 24, 2022

x/tools,x/build: performance monitoring for gopls #53538

Open

12 tasks

all: Go compiler/runtime performance monitoring system #48803

all: Go compiler/runtime performance monitoring system #48803

prattmic commented Oct 5, 2021 •

edited

mvdan commented Oct 5, 2021

odeke-em commented Oct 5, 2021 •

edited

mengzhuo commented Oct 6, 2021

mknyszek commented Oct 6, 2021

gopherbot commented Oct 6, 2021

prattmic commented Oct 6, 2021

gopherbot commented Oct 7, 2021

gopherbot commented Oct 7, 2021

gopherbot commented Oct 7, 2021

gopherbot commented Oct 7, 2021

shawndx commented Nov 17, 2021

prattmic commented Nov 17, 2021

shawndx commented Nov 18, 2021

gopherbot commented Jan 7, 2022

gopherbot commented Mar 16, 2022

gopherbot commented Mar 21, 2022

gopherbot commented Mar 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

prattmic commented Jun 21, 2022

prattmic commented Jun 21, 2022

jake-ciolek commented Jun 22, 2022

prattmic commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 23, 2022

gopherbot commented Jun 24, 2022

prattmic commented Jun 27, 2022

gopherbot commented Jun 27, 2022

jake-volvo commented Jun 28, 2022

prattmic commented Jun 28, 2022

all: Go compiler/runtime performance monitoring system #48803

all: Go compiler/runtime performance monitoring system #48803

Comments

prattmic commented Oct 5, 2021 • edited

mvdan commented Oct 5, 2021

odeke-em commented Oct 5, 2021 • edited

mengzhuo commented Oct 6, 2021

mknyszek commented Oct 6, 2021

gopherbot commented Oct 6, 2021

prattmic commented Oct 6, 2021

gopherbot commented Oct 7, 2021

gopherbot commented Oct 7, 2021

gopherbot commented Oct 7, 2021

gopherbot commented Oct 7, 2021

shawndx commented Nov 17, 2021

prattmic commented Nov 17, 2021

shawndx commented Nov 18, 2021

gopherbot commented Jan 7, 2022

gopherbot commented Mar 16, 2022

gopherbot commented Mar 21, 2022

gopherbot commented Mar 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

prattmic commented Jun 21, 2022

prattmic commented Jun 21, 2022

jake-ciolek commented Jun 22, 2022

prattmic commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 23, 2022

gopherbot commented Jun 24, 2022

prattmic commented Jun 27, 2022

gopherbot commented Jun 27, 2022

jake-volvo commented Jun 28, 2022

prattmic commented Jun 28, 2022

prattmic commented Oct 5, 2021 •

edited

odeke-em commented Oct 5, 2021 •

edited