The coordinator should record per-build logs, and also keep statistics about timing information in BigQuery or something.
Then we can graph build times, test times, individual test times, machine bootup times, etc, all over time.
Related: #12668 (shootout tests talke 30 mins on arm)
The text was updated successfully, but these errors were encountered:
I have been keeping an eye on performance for the changes that are going
I agree, it would be great to have something like Dmitry's performance
On Fri, 18 Sep 2015 09:27 Alex Brainman email@example.com wrote:
…art text Now events and phases of the build are represented by a span, with a stop and start, and each has timing and a success status. These will visualize nicer (later) and will be able to be logged and queried in Cloud Datastore + BigQuery. Also, remove the overloaded and not-to-be-used "rev" parameter from the GetBuildlet interface. An upcoming change disconnects the buildlet's lifetime from that of a single build, so the rev isn't even known at the time of the GetBuildlet call; the buildlet will be handed out to the most worthy current waiter (and the rev might not have even existed at the tim the GetBuildlet call started) Updates golang/go#12669 Change-Id: I02b28fea8bbbc16dc95c1ea0507418be01aa8ca8 Reviewed-on: https://go-review.googlesource.com/21570 Reviewed-by: Andrew Gerrand <firstname.lastname@example.org>
These IDs will become the cloud datastore keys for keeping history and logs. Many more CLs will follow: changing farmer URLs, logging to datastore, fetching old logs from datastore, etc. Updates golang/go#13076 (make all logs URLs permanent) Updates golang/go#12669 (collect logs, stats) Change-Id: I5b9fd21bf23581c59724b0ed32c8459baa9683f7 Reviewed-on: https://go-review.googlesource.com/21968 Reviewed-by: Andrew Gerrand <email@example.com>
We've been logging event spans to datastore for years, but I'd lost this CL and just found it back. This does two things: syncs the datastore logs to BigQuery, and starts to use the from-BigQuery timing info in the coordinator for scheduling sharded tests. The plan was to have a job occasionally do a BigQuery query and write out the results to a CSV file on GCS. The code to read that CSV file is in this CL, but that code path is disabled, so this CL should be a no-op. A future change will periodically do the query and write the CSV file, and then we can start using the new code path and remove the static map of expected test durations. Updates golang/go#12669 Change-Id: Ibe5b41d6a3009c2ade8ab728fa1cad646788e621 Reviewed-on: https://go-review.googlesource.com/30716 Reviewed-by: Brad Fitzpatrick <firstname.lastname@example.org>
This CL is almost entirely code movement. There is new code in cmd/coordinator/gce.go. This library-itizes the buildstats code and makes it used by both cmd/buildstats (when run by hand) and cmd/coordinator, which now runs it in a loop in the background. Running cmd/buildstats by hand is still pretty useless because it doesn't yet(?) have any query subcommands. It only has the original -sync command, which is now automatic in the coordinator. But it could still be useful for debugging or running manually in the staging environment. This will permit running BigQuery queries at any time, without Brad or somebody first manually running cmd/buildstats -sync. This also means we'll be able to get rid of a bunch of the static (and stale) timing information in the coordinator code and use empirical numbers instead for buildlet creation times and per-test durations. This will help the test sharding schedule things better and result in faster builds. Updates golang/go#12669 Change-Id: If9a3b414f8d458a65305c4916ea16d59e5a3971c Reviewed-on: https://go-review.googlesource.com/53072 Reviewed-by: Sarah Adams <email@example.com>