Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: provide a way to test that benchmarks work without invalidating cache results #44499

Open
bcmills opened this issue Feb 22, 2021 · 0 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Feb 22, 2021

In #41824 (comment), @rsc wrote:

The fact that CI systems can today opt in to running 1 iteration of benchmarks (using -benchtime=1x) but do not seems to me strong evidence that the Go command should not start doing it either.

As has been pointed out, CI systems and users have the ability to opt in even today, using -benchtime=1x. You could even set GOFLAGS=-benchtime=1x to force it into all your go test commands.

As an experiment I tried doing this for my own day-to-day work, and it was awful. It turns out that the -benchtime=1x causes test results not to be cached, which means that GOFLAGS=-benchtime=1x causes every test to be re-run whenever go test is invoked, even if the source files haven't changed and even if the tests contain no actual benchmarks.

Invalidating the test cache is probably the right behavior when -benchtime is actually being used for benchmarking but is entirely inappropriate when the reason for setting the -benchtime flag is to test that the benchmarks continue to run successfully (which was the point of #41824 to begin with).

This issue tracks the need for “testing that benchmarks still work” without also invalidating cache results.

CC @jayconrod @matloob

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant