New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: collect stats, logs #12669

Open
bradfitz opened this Issue Sep 17, 2015 · 12 comments

Comments

Projects
None yet
4 participants
@bradfitz
Member

bradfitz commented Sep 17, 2015

The coordinator should record per-build logs, and also keep statistics about timing information in BigQuery or something.

Then we can graph build times, test times, individual test times, machine bootup times, etc, all over time.

Related: #12668 (shootout tests talke 30 mins on arm)

@bradfitz bradfitz added the Builders label Sep 17, 2015

@bradfitz bradfitz added this to the Unplanned milestone Sep 17, 2015

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Sep 17, 2015

I wanted something similar many times. We have many people changing things now. But no one is looking after performance. Are things getting better over time? Worse? It is impossible to tell.

Alex

@davecheney

This comment has been minimized.

Contributor

davecheney commented Sep 17, 2015

I have been keeping an eye on performance for the changes that are going
into the compiler. Fwiw, we clawed back from 91mins to 88mins (on arm) over
the last few weeks. Of course, we lost 30 mins because something has gone
bonkers with the shootout tests.

I agree, it would be great to have something like Dmitry's performance
dashboard back again.

On Fri, 18 Sep 2015 09:27 Alex Brainman notifications@github.com wrote:

I wanted something similar many times. We have many people changing things
now. But no one is looking after performance. Are things getting better
over time? Worse? It is impossible to tell.

Alex


Reply to this email directly or view it on GitHub
#12669 (comment).

@bradfitz

This comment has been minimized.

Member

bradfitz commented Apr 5, 2016

Btw, this bug is not about the performance dashboard so much as it is about monitoring the health of the build system itself.

@gopherbot

This comment has been minimized.

gopherbot commented Apr 5, 2016

CL https://golang.org/cl/21570 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 6, 2016

cmd/coordinator: log event spans with associated success, not stop/st…
…art text

Now events and phases of the build are represented by a span, with a
stop and start, and each has timing and a success status.

These will visualize nicer (later) and will be able to be logged and
queried in Cloud Datastore + BigQuery.

Also, remove the overloaded and not-to-be-used "rev" parameter from
the GetBuildlet interface. An upcoming change disconnects the
buildlet's lifetime from that of a single build, so the rev isn't even
known at the time of the GetBuildlet call; the buildlet will be handed
out to the most worthy current waiter (and the rev might not have even
existed at the tim the GetBuildlet call started)

Updates golang/go#12669

Change-Id: I02b28fea8bbbc16dc95c1ea0507418be01aa8ca8
Reviewed-on: https://go-review.googlesource.com/21570
Reviewed-by: Andrew Gerrand <adg@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Apr 7, 2016

CL https://golang.org/cl/21640 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 7, 2016

cmd/coordinator: push new logging down into the buildlet creation
Next step is logging it to datastore/bigquery. Then stats and
visualizing with trace viewer later.

Updates golang/go#12669

Change-Id: I039097791993dbe3d4aceb18d2a08db8299f3c05
Reviewed-on: https://go-review.googlesource.com/21640
Reviewed-by: Andrew Gerrand <adg@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Apr 13, 2016

CL https://golang.org/cl/21968 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 13, 2016

cmd/coordinator: give each build and try set a globally unique random ID
These IDs will become the cloud datastore keys for keeping history and logs.

Many more CLs will follow: changing farmer URLs, logging to datastore,
fetching old logs from datastore, etc.

Updates golang/go#13076 (make all logs URLs permanent)
Updates golang/go#12669 (collect logs, stats)

Change-Id: I5b9fd21bf23581c59724b0ed32c8459baa9683f7
Reviewed-on: https://go-review.googlesource.com/21968
Reviewed-by: Andrew Gerrand <adg@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented May 2, 2016

CL https://golang.org/cl/22701 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 2, 2016

cmd/coordinator: start of logging
For now, just log the process's start time and alive/heartbeat time.

Updates golang/go#12669

Change-Id: I4dd28f7ba761b5487f86a4cdb0f721b2aeb4fa57
Reviewed-on: https://go-review.googlesource.com/22701
Reviewed-by: Andrew Gerrand <adg@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented May 2, 2016

CL https://golang.org/cl/22705 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 2, 2016

cmd/coordinator: log builds to datastore
Updates golang/go#12669

Change-Id: I62a245492081519ba02776366d633e31d74a3c76
Reviewed-on: https://go-review.googlesource.com/22705
Reviewed-by: Andrew Gerrand <adg@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Sep 13, 2016

CL https://golang.org/cl/29112 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Sep 14, 2016

cmd/buildstats: add new buildstats command for analyzing builds
Currently this tool just syncs build data from Datastore to Bigquery.

It'll do more later.

Updates golang/go#12669

Change-Id: Ic544cf112ea8699c86943c32f8887d31c81daf79
Reviewed-on: https://go-review.googlesource.com/29112
Reviewed-by: Quentin Smith <quentin@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Oct 6, 2016

CL https://golang.org/cl/30631 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Oct 7, 2016

cmd/coordinator: log per-span info to datastore also
Updates golang/go#12669

Change-Id: Ic7dcc9876557abdb13cd2e6024eb98cdff97c9bb
Reviewed-on: https://go-review.googlesource.com/30631
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Apr 24, 2017

CL https://golang.org/cl/30716 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 24, 2017

cmd/coordinator, cmd/buildstats: start of using logs to schedule tests
We've been logging event spans to datastore for years, but I'd lost
this CL and just found it back. This does two things: syncs the
datastore logs to BigQuery, and starts to use the from-BigQuery timing
info in the coordinator for scheduling sharded tests.

The plan was to have a job occasionally do a BigQuery query and write
out the results to a CSV file on GCS. The code to read that CSV file
is in this CL, but that code path is disabled, so this CL should be a
no-op.

A future change will periodically do the query and write the CSV file,
and then we can start using the new code path and remove the static
map of expected test durations.

Updates golang/go#12669

Change-Id: Ibe5b41d6a3009c2ade8ab728fa1cad646788e621
Reviewed-on: https://go-review.googlesource.com/30716
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Aug 3, 2017

Change https://golang.org/cl/53072 mentions this issue: cmd/buildstats, cmd/coordinator: run buildstats sync in background

gopherbot pushed a commit to golang/build that referenced this issue Aug 3, 2017

cmd/buildstats, cmd/coordinator: run buildstats sync in background
This CL is almost entirely code movement. There is new code in
cmd/coordinator/gce.go.

This library-itizes the buildstats code and makes it used by both
cmd/buildstats (when run by hand) and cmd/coordinator, which now runs
it in a loop in the background.

Running cmd/buildstats by hand is still pretty useless because it
doesn't yet(?) have any query subcommands. It only has the original
-sync command, which is now automatic in the coordinator. But it could
still be useful for debugging or running manually in the staging
environment.

This will permit running BigQuery queries at any time, without Brad or
somebody first manually running cmd/buildstats -sync.

This also means we'll be able to get rid of a bunch of the static (and
stale) timing information in the coordinator code and use empirical
numbers instead for buildlet creation times and per-test durations.
This will help the test sharding schedule things better and result in
faster builds.

Updates golang/go#12669

Change-Id: If9a3b414f8d458a65305c4916ea16d59e5a3971c
Reviewed-on: https://go-review.googlesource.com/53072
Reviewed-by: Sarah Adams <shadams@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment