x/build: record and collect compiler benchmarks

[mdempsky]

Recently we added a -bench flag to cmd/compile to benchmark compiler phases. It might be interesting if the builders used this flag and then persisted the output file so we could monitor compiler performance over time.

There's a risk that measuring the standard library compile time is confounded with changes to the standard library, but I think only a couple packages (runtime, net, net/http) still seem under active development. So it's probably still a decent signal.

/cc @bradfitz @griesemer

[mdempsky]

/cc @davecheney too, since he's been benchmarking build times against other code bases too.

[bradfitz]

I've been logging more and more to Datastore & BigQuery, and it's on my short-term list to add more timing info there, as part of optimizing build+test speeds for #17104

So, this isn't much more work.

/cc @quentinmit

[quentinmit]

Yep, and I'm working on collecting benchmark results. This sounds great. Where does -bench put its output?

Also, which compile command would you instrument? There are a bunch involved in all.bash.

[mdempsky]

You can use -bench=foo.txt and the compiler will append to foo.txt, so the entire build can be accumulated into a single file. The individual measurements indicate the package being compiled.

I would probably tweak all.bash so that the "go install std cmd" step adds -gcflags="-bench=/tmp/bench.txt" on the builders, and then save /tmp/bench.txt somewhere we can easily collect and analyze.

I think benchmarking the Go1 benchmarks' compile times would be nice too, since they're a fixed target.

[bradfitz]

We don't use all.bash on the builders. And make.bash already respects the env var $GO_GCFLAGS

[mdempsky]

I meant "all.bash" generically to refer to whatever build process was used by the builders. GO_GCFLAGS sounds good to me if that means we can do it non-intrusively.

[davecheney]

SGTM!

On Wed, Sep 21, 2016 at 4:19 AM, Matthew Dempsky notifications@github.com
wrote:

/cc @davecheney https://github.com/davecheney too, since he's been
benchmarking build times against other code bases too.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17167 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA26_YEbWWITcnZNrkOVI6n0kZvA3ks5qsCO2gaJpZM4KB8V8
.

[gopherbot]

CL https://golang.org/cl/51192 mentions this issue.

[dmitshur]

@mdempsky This issue might be a candidate for closing as a duplicate given we now have #48803 and #49207. What do you think? (CC @prattmic.)

[prattmic]

Yes, we have benchmark tracking now. I think the action item here would be to add some compiler phase benchmarks to x/benchmarks. Perhaps alongside our existing overall build benchmarks in https://cs.opensource.google/go/x/benchmarks/+/master:sweet/harnesses/go-build.go?