all: Go compiler/runtime performance monitoring system

[prattmic]

The runtime/compiler team is investigating the creation of a performance monitoring system, with a primary goal of reducing release toil by catching performance regressions as early as possible.

Requirements are still being discussed, but highlights include:

• Automation of performance data generation.
  • Benchmark execution solution that tracks commits to Go (e.g., x/build coordinator scheduling benchmark runs).
• Storage and visualization of performance data over time.
• Key set of "application-level" and "feature-focused" benchmarks.
• Active monitoring of changepoints.
• Easy debugging/reproduction.

cc @aclements @dr2chase @mknyszek @jeremyfaller @golang/release

[mvdan]

cc @odeke-em

[odeke-em]

cc @odeke-em

Thank you for the tag @mvdan!

Indeed, at Orijtech Inc we've been working on such a product "Bencher" for the past 2 quarters of 2021 and produced it recently https://twitter.com/odeke_et/status/1428650768135974919?s=21: we have an integration on the GitHub marketplace at https://github.com/marketplace/gobencher and we actually have something on our roadmap to add a Gerrit integration during Q4 2021 or Q1 2022. I had raised and shown the product to @Sajmani, @spf13, @ianlancetaylor and @ianthehat and others :-)

For example, take a look at one of our public users Entgo https://dashboard.github.orijtech.com/benchmark/2bb34b8e166747d2974f76825819acbd

We'd be delighted to work with y'all to bring this first class to the Go tooling ecosystem!

Kindly cc-ing my colleagues @cuonglm @kirbyquerby @willpoint @jhusdero

[mengzhuo]

I have some thoughts about this site/app:

1. Transparent. All results should be available to public like benchsave instead of internal site from Google.
2. Informative. Regression notice to Gerrit/owner of CL.
3. Easy to compare with different baselines like major versions of Go.

[mknyszek]

@mengzhuo All three of those things are already on our radar. :) Thanks.

[gopherbot]

Change https://golang.org/cl/353909 mentions this issue: cmd/coordinator: don't snapshot in dev mode

[prattmic]

@odeke-em We are still evaluating our options. We've looked at several different off-the-shelf options, including Bencher, as well as extending the existing http://build.golang.org infrastructure with what we need. We are currently leaning towards extending http://build.golang.org, as we already maintain it and it supports much of what we need already (fully open source, precise control over dedicated hardware used for benchmarking, etc).

[gopherbot]

Change https://golang.org/cl/354629 mentions this issue: cmd/buildlet: add revdial test

[gopherbot]

Change https://golang.org/cl/354630 mentions this issue: all: handle revdial redirects on connect

[gopherbot]

Change https://golang.org/cl/354638 mentions this issue: cmd/coordinator: find work in dev mode

[gopherbot]

Change https://golang.org/cl/354637 mentions this issue: cmd/coordinator: make listen address configurable

[shawn-xdji]

Curious to know will benchmark fluctuation be taken into account, and how? Thanks.

[prattmic]

@shawn-xdji By fluctuation, I assume you mean noise in the results. We are taking a few approaches up front:

• Running benchmarks on a low-noise machines. For now we plan to use "sole-tenant" VMs on GCE, though we may move to physical machines if necessary.
• Rerunning the baseline benchmarks each time we test a new commit to help account for long term changes (e.g., OS upgrade changes performance).
• Changepoint detection will need to statistical handle a small amount of noise.
• We are only planning to track a curated set of benchmarks. Benchmarks that a fundamentally noisy will either be excluded or fixed.

Finally, all of this is something we'll need to learn on as we go and find out how big the issues are.

[shawn-xdji]

Thanks @prattmic, that answers my question, anticipate you will share some best practices later.