x/build: collect stats, logs #12669
Comments
|
I wanted something similar many times. We have many people changing things now. But no one is looking after performance. Are things getting better over time? Worse? It is impossible to tell. Alex |
|
I have been keeping an eye on performance for the changes that are going I agree, it would be great to have something like Dmitry's performance On Fri, 18 Sep 2015 09:27 Alex Brainman notifications@github.com wrote:
|
|
Btw, this bug is not about the performance dashboard so much as it is about monitoring the health of the build system itself. |
|
CL https://golang.org/cl/21570 mentions this issue. |
…art text Now events and phases of the build are represented by a span, with a stop and start, and each has timing and a success status. These will visualize nicer (later) and will be able to be logged and queried in Cloud Datastore + BigQuery. Also, remove the overloaded and not-to-be-used "rev" parameter from the GetBuildlet interface. An upcoming change disconnects the buildlet's lifetime from that of a single build, so the rev isn't even known at the time of the GetBuildlet call; the buildlet will be handed out to the most worthy current waiter (and the rev might not have even existed at the tim the GetBuildlet call started) Updates golang/go#12669 Change-Id: I02b28fea8bbbc16dc95c1ea0507418be01aa8ca8 Reviewed-on: https://go-review.googlesource.com/21570 Reviewed-by: Andrew Gerrand <adg@golang.org>
|
CL https://golang.org/cl/21640 mentions this issue. |
Next step is logging it to datastore/bigquery. Then stats and visualizing with trace viewer later. Updates golang/go#12669 Change-Id: I039097791993dbe3d4aceb18d2a08db8299f3c05 Reviewed-on: https://go-review.googlesource.com/21640 Reviewed-by: Andrew Gerrand <adg@golang.org>
|
CL https://golang.org/cl/21968 mentions this issue. |
These IDs will become the cloud datastore keys for keeping history and logs. Many more CLs will follow: changing farmer URLs, logging to datastore, fetching old logs from datastore, etc. Updates golang/go#13076 (make all logs URLs permanent) Updates golang/go#12669 (collect logs, stats) Change-Id: I5b9fd21bf23581c59724b0ed32c8459baa9683f7 Reviewed-on: https://go-review.googlesource.com/21968 Reviewed-by: Andrew Gerrand <adg@golang.org>
|
CL https://golang.org/cl/22701 mentions this issue. |
For now, just log the process's start time and alive/heartbeat time. Updates golang/go#12669 Change-Id: I4dd28f7ba761b5487f86a4cdb0f721b2aeb4fa57 Reviewed-on: https://go-review.googlesource.com/22701 Reviewed-by: Andrew Gerrand <adg@golang.org>
|
CL https://golang.org/cl/22705 mentions this issue. |
Updates golang/go#12669 Change-Id: I62a245492081519ba02776366d633e31d74a3c76 Reviewed-on: https://go-review.googlesource.com/22705 Reviewed-by: Andrew Gerrand <adg@golang.org>
|
CL https://golang.org/cl/29112 mentions this issue. |
Currently this tool just syncs build data from Datastore to Bigquery. It'll do more later. Updates golang/go#12669 Change-Id: Ic544cf112ea8699c86943c32f8887d31c81daf79 Reviewed-on: https://go-review.googlesource.com/29112 Reviewed-by: Quentin Smith <quentin@golang.org>
|
CL https://golang.org/cl/30631 mentions this issue. |
Updates golang/go#12669 Change-Id: Ic7dcc9876557abdb13cd2e6024eb98cdff97c9bb Reviewed-on: https://go-review.googlesource.com/30631 Reviewed-by: Ian Lance Taylor <iant@golang.org>
|
CL https://golang.org/cl/30716 mentions this issue. |
We've been logging event spans to datastore for years, but I'd lost this CL and just found it back. This does two things: syncs the datastore logs to BigQuery, and starts to use the from-BigQuery timing info in the coordinator for scheduling sharded tests. The plan was to have a job occasionally do a BigQuery query and write out the results to a CSV file on GCS. The code to read that CSV file is in this CL, but that code path is disabled, so this CL should be a no-op. A future change will periodically do the query and write the CSV file, and then we can start using the new code path and remove the static map of expected test durations. Updates golang/go#12669 Change-Id: Ibe5b41d6a3009c2ade8ab728fa1cad646788e621 Reviewed-on: https://go-review.googlesource.com/30716 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
|
Change https://golang.org/cl/53072 mentions this issue: |
This CL is almost entirely code movement. There is new code in cmd/coordinator/gce.go. This library-itizes the buildstats code and makes it used by both cmd/buildstats (when run by hand) and cmd/coordinator, which now runs it in a loop in the background. Running cmd/buildstats by hand is still pretty useless because it doesn't yet(?) have any query subcommands. It only has the original -sync command, which is now automatic in the coordinator. But it could still be useful for debugging or running manually in the staging environment. This will permit running BigQuery queries at any time, without Brad or somebody first manually running cmd/buildstats -sync. This also means we'll be able to get rid of a bunch of the static (and stale) timing information in the coordinator code and use empirical numbers instead for buildlet creation times and per-test durations. This will help the test sharding schedule things better and result in faster builds. Updates golang/go#12669 Change-Id: If9a3b414f8d458a65305c4916ea16d59e5a3971c Reviewed-on: https://go-review.googlesource.com/53072 Reviewed-by: Sarah Adams <shadams@google.com>
The coordinator should record per-build logs, and also keep statistics about timing information in BigQuery or something.
Then we can graph build times, test times, individual test times, machine bootup times, etc, all over time.
Related: #12668 (shootout tests talke 30 mins on arm)
The text was updated successfully, but these errors were encountered: