New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: mention notable benchmark changes in Gerrit comment #20412

Open
bradfitz opened this Issue May 18, 2017 · 4 comments

Comments

Projects
None yet
4 participants
@bradfitz
Member

bradfitz commented May 18, 2017

I think it's going to be too easy to miss significant benchmark changes with the current state of things where the URL to perf.golang.org is always included in the final Gerrit comment.

I think the gerrit comment should also mention any notable changes.

/cc @quentinmit @josharian

@gopherbot gopherbot added this to the Unreleased milestone May 18, 2017

@gopherbot gopherbot added the Builders label May 18, 2017

@josharian

This comment has been minimized.

Contributor

josharian commented May 18, 2017

Yes please! And perhaps also always include any relevant geomeans?

This should get implemented behind a flag that is easy to turn on and off, because past experience with performance dashboards suggests that we're going to have lots of problems with noise.

@bradfitz

This comment has been minimized.

Member

bradfitz commented May 18, 2017

I'd rather start with some noise than have no information. We have lots of noise as-is with flaky tests, sadly.

@josharian

This comment has been minimized.

Contributor

josharian commented May 18, 2017

I'm not so sure noise is better than nothing, particularly if it desensitizes people and causes them to systematically ignore/discount the results. I'll file an issue with an example and we can discuss some details there.

@gopherbot

This comment has been minimized.

gopherbot commented Jun 2, 2017

CL https://golang.org/cl/44613 mentions this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment