New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: meta bug tracking trybot build speed (goal: 5 minutes) #17104

Open
bradfitz opened this Issue Sep 14, 2016 · 11 comments

Comments

Projects
None yet
2 participants
@bradfitz
Member

bradfitz commented Sep 14, 2016

The goal is to make all trybot runs take 5 minutes.

@bradfitz bradfitz added the Builders label Sep 14, 2016

@bradfitz bradfitz added this to the Unreleased milestone Sep 14, 2016

@bradfitz bradfitz self-assigned this Sep 14, 2016

@gopherbot

This comment has been minimized.

gopherbot commented Sep 14, 2016

CL https://golang.org/cl/29153 mentions this issue.

gopherbot pushed a commit that referenced this issue Sep 14, 2016

cmd/dist: skip compiling 100 packages without benchmarks in race mode
The go_test_bench:* tests run:

    go test -short -race -run=^$ -benchtime=.1s -cpu=4 $PKG

... on each discovered package with any tests. (The same set used for
the "go_test:*" tests)

That set was 168 packages:

$ go tool dist test -list | grep go_test: | wc -l
168

But only 76 of those have a "func Benchmark", and running each
"go_test_bench:" test and compiling it in race mode, just to do
nothing took 1-2 seconds each.

So stop doing that and filter out the useless packages earlier. Now:

$ go tool dist test -list -race | grep go_test_bench:  | wc -l
76

Should save 90-180 seconds. (or maybe 45 seconds for trybots, since
they're sharded)

Updates #17104

Change-Id: I08ccb072a0dc0454ea425540ee8e74b59f83b773
Reviewed-on: https://go-review.googlesource.com/29153
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Sep 14, 2016

CL https://golang.org/cl/29156 mentions this issue.

gopherbot pushed a commit that referenced this issue Sep 14, 2016

regexp: don't run slow benchmarks on race builders
Shave 6.5 minutes off the *-race build time.

The *-race builders run:

    go test -short -race -run=^$ -benchtime=.1s -cpu=4 $PKG

... for each package with benchmarks.

The point isn't to measure the speed of the packages, but rather to
see if there are any races. (which is why a benchtime of 0.1 seconds
is used)

But running in race mode makes things slower and our benchmarks aren't
all very fast to begin with.

The regexp benchmarks in race were taking over 6.5 minutes. With this
CL, it's now 8 seconds.

Updates #17104

Change-Id: I054528d09b1568d37aac9f9b515d6ed90a5cf5b0
Reviewed-on: https://go-review.googlesource.com/29156
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Crawshaw <crawshaw@golang.org>

gopherbot pushed a commit that referenced this issue Sep 14, 2016

bytes: cut 10 seconds off the race builder's benchmark test
Don't benchmark so many sizes during the race builder's benchmark run.

This package doesn't even use goroutines.

Cuts off 10 seconds.

Updates #17104

Change-Id: Ibb2c7272c18b9014a775949c656a5b930f197cd4
Reviewed-on: https://go-review.googlesource.com/29158
Reviewed-by: David Crawshaw <crawshaw@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Sep 14, 2016

CL https://golang.org/cl/29163 mentions this issue.

@gopherbot

This comment has been minimized.

gopherbot commented Sep 14, 2016

CL https://golang.org/cl/29159 mentions this issue.

gopherbot pushed a commit that referenced this issue Sep 14, 2016

sort: cut 140 seconds off race build tests
No coverage is gained by running the 1e6 versions of the test over the
1e4 versions. It just adds 140 seconds of race overhead time.

Updates #17104

Change-Id: I41408aedae34a8b1a148eebdda20269cdefffba3
Reviewed-on: https://go-review.googlesource.com/29159
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>

gopherbot pushed a commit that referenced this issue Sep 14, 2016

math/big: cut 2 minutes off race tests
No need to test so many sizes in race mode, especially for a package
which doesn't use goroutines.

Reduces test time from 2.5 minutes to 25 seconds.

Updates #17104

Change-Id: I7065b39273f82edece385c0d67b3f2d83d4934b8
Reviewed-on: https://go-review.googlesource.com/29163
Reviewed-by: David Crawshaw <crawshaw@golang.org>
@bradfitz

This comment has been minimized.

Member

bradfitz commented Sep 14, 2016

Current race build speeds:

  2016-09-14T18:51:56Z finish_start_tests after 3.059513602s; 10.0.2.4:80: [go_test_bench:compress/bzip2]
  2016-09-14T18:51:17Z finish_start_tests after 3.143345345s; 10.0.0.9:80: [go_test_bench:image/jpeg]
  2016-09-14T18:51:18Z finish_start_tests after 3.182452293s; 10.0.2.5:80: [go_test_bench:encoding/hex]
  2016-09-14T18:51:12Z finish_start_tests after 3.490162906s; 10.0.1.6:80: [go_test_bench:expvar]
  2016-09-14T18:51:55Z finish_start_tests after 3.609317114s; 10.0.2.6:80: [go_test_bench:crypto/cipher]
  2016-09-14T18:52:17Z finish_start_tests after 3.628982201s; 10.0.1.6:80: [go_test_bench:compress/lzw]
  2016-09-14T18:51:49Z finish_start_tests after 3.693163398s; 10.0.0.9:80: [go_test_bench:database/sql]
  2016-09-14T18:51:32Z finish_start_tests after 3.807438591s; 10.0.1.6:80: [go_test_bench:math/rand]
  2016-09-14T18:51:37Z finish_start_tests after 3.882166683s; 10.0.2.4:80: [go_test_bench:bufio]
  2016-09-14T18:52:20Z finish_start_tests after 4.038173785s; 10.0.2.5:80: [go_test_bench:context]
  2016-09-14T18:51:08Z finish_start_tests after 4.107135055s; 10.0.0.10:80: [go_test_bench:hash/crc32]
  2016-09-14T18:52:23Z finish_start_tests after 4.205641826s; 10.0.2.6:80: [go_test_bench:unicode/utf8]
  2016-09-14T18:51:58Z finish_start_tests after 4.587359311s; 10.0.0.10:80: [go_test_bench:regexp/syntax]
  2016-09-14T18:52:04Z finish_start_tests after 4.660599666s; 10.0.2.6:80: [go_test_bench:sort]
  2016-09-14T18:51:30Z finish_start_tests after 5.311264213s; 10.0.2.6:80: [go_test_bench:math/cmplx]
  2016-09-14T18:51:23Z finish_start_tests after 5.326788419s; 10.0.0.9:80: [go_test_bench:encoding/gob]
  2016-09-14T18:51:55Z finish_start_tests after 5.777081055s; 10.0.0.9:80: [go_test_bench:reflect]
  2016-09-14T18:51:23Z finish_start_tests after 6.12439885s; 10.0.0.10:80: [go_test_bench:image/png]
  2016-09-14T18:51:14Z finish_start_tests after 6.765132418s; 10.0.2.5:80: [go_test_bench:html/template]
  2016-09-14T18:51:14Z finish_start_tests after 7.476528843s; 10.0.0.9:80: [go_test_bench:fmt]
  2016-09-14T18:52:12Z finish_start_tests after 7.526458261s; 10.0.2.5:80: [go_test_bench:sync]
  2016-09-14T18:51:33Z finish_start_tests after 7.782424696s; 10.0.2.4:80: [go_test_bench:archive/zip]
  2016-09-14T18:52:00Z finish_start_tests after 8.428459563s; 10.0.1.6:80: [go_test_bench:regexp]
  2016-09-14T18:51:17Z finish_start_tests after 8.666510786s; 10.0.0.10:80: [go_test_bench:image/draw]
  2016-09-14T18:52:11Z finish_start_tests after 10.836201759s; 10.0.1.6:80: [go_test_bench:strings]
  2016-09-14T18:52:18Z finish_start_tests after 10.952476479s; 10.0.2.6:80: [go_test_bench:time]
  2016-09-14T18:51:25Z finish_start_tests after 11.373276098s; 10.0.1.6:80: [go_test_bench:image/gif]
  2016-09-14T18:51:25Z finish_start_tests after 11.547950173s; 10.0.2.6:80: [go_test_bench:encoding/json]
  2016-09-14T18:51:57Z finish_start_tests after 11.548834754s; 10.0.2.5:80: [go_test_bench:crypto/tls]
  2016-09-14T18:52:13Z finish_start_tests after 12.819669296s; 10.0.0.10:80: [go_test_bench:strconv]
  2016-09-14T18:51:38Z finish_start_tests after 13.7889302s; 10.0.0.9:80: [go_test_bench:math]
  2016-09-14T18:51:51Z finish_start_tests after 14.845086695s; 10.0.2.6:80: [go_test_bench:net]
  2016-09-14T18:51:52Z finish_start_tests after 15.288519219s; 10.0.1.6:80: [go_test_bench:net/http]
  2016-09-14T18:51:53Z finish_start_tests after 15.809308703s; 10.0.2.4:80: [go_test_bench:bytes]
  2016-09-14T18:51:41Z finish_start_tests after 23.69239388s; 10.0.2.5:80: [go_test_bench:index/suffixarray]
  2016-09-14T18:52:23Z finish_start_tests after 26.906228664s; 10.0.2.4:80: [go_test_bench:compress/flate]
  2016-09-14T18:51:54Z finish_start_tests after 28.82127674s; 10.0.0.10:80: [go_test_bench:math/big]

gopherbot pushed a commit to golang/build that referenced this issue Sep 14, 2016

cmd/coordinator: update the timing hints table a bit
If the coordinator has a better guess as to how long various tests
take, then it can do better critical path scheduling and reduce the
overall time the sharded tests take to complete.

This table still needs to die and be based on recent empirical data,
but at least it's more accurate now, after a long delay in being
updated.

Update is from golang/go#17104 (comment)

Updates golang/go#17104

Change-Id: I115aad23fbdb0cde1b196e71a4131fbe36480cc0
Reviewed-on: https://go-review.googlesource.com/29167
Reviewed-by: David Crawshaw <crawshaw@golang.org>

gopherbot pushed a commit to golang/build that referenced this issue Sep 15, 2016

cmd/buildlet: remount OpenBSD & FreeBSD filesystems on boot
OpenBSD is deployed and tested. FreeBSD has only been tested by hand,
but this CL doesn't fail the buildlet if the remount fails. It only
logs either way.

Updates golang/go#17104

Change-Id: Ia9662b42ae8305ad9eaa4292c94fa3194cc26b11
Reviewed-on: https://go-review.googlesource.com/29238
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
@gopherbot

This comment has been minimized.

gopherbot commented Sep 20, 2016

CL https://golang.org/cl/29430 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Sep 20, 2016

cmd/buildlet: don't log one line per untarred file
Some builders (OpenBSD, FreeBSD, Plan9 at least?) have their buildlet
process's stdout/stderr hooked up to their serial console.

The log line for each untarred Go1.4 + Go src tarball going to the
serial console added just shy of 1 minute (!!) to the build time.  Now
it takes 3 seconds. (Or 12 seconds before change to use an async+noatime
filesystem on the BSDs)

Updates golang/go#17104

Change-Id: I1e6f00bcca955ead26b279a79729e50319384593
Reviewed-on: https://go-review.googlesource.com/29430
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
@gopherbot

This comment has been minimized.

gopherbot commented Sep 21, 2016

CL https://golang.org/cl/29473 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Sep 21, 2016

cmd/coordinator: support skipping redundant tests on try runs
Also, don't start obtaining test sharding buildlets early if they're
reverse buildlets. Reverse buildlets are either immediately available,
or they're not. No point monopolizing them earlier than needed.

Updates golang/go#17104

Change-Id: If5a0bbd0c59b55750adfeeaa8d0f81cdbcc8ad48
Reviewed-on: https://go-review.googlesource.com/29473
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
@gopherbot

This comment has been minimized.

gopherbot commented Sep 22, 2016

CL https://golang.org/cl/29551 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Sep 22, 2016

all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".

Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.

Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):

   linux-amd64
   linux-amd64-race
   linux-386
   linux-386-387

And these are the same host type (a GCE VM):

   windows-amd64-gce
   windows-amd64-race
   windows-386-gce

This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.

Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.

Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.

gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.

This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.

Updates golang/go#17104

Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
@gopherbot

This comment has been minimized.

gopherbot commented Sep 23, 2016

CL https://golang.org/cl/29670 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Sep 23, 2016

dashboard: add a make.bash-only builder on real ARM hardware
This is a new builder in prep for the change to the "linux-arm"
builder where the GOARCH=arm make.bash will be cross-compiled from a
Kubernetes container on fast hardware.

Updates golang/go#17105 (cross-compile ARM builders' make.bash)
Updates golang/go#17104 (5 minute trybots)

Change-Id: Icfd2644d77639f731151abe54839322960418254
Reviewed-on: https://go-review.googlesource.com/29670
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
@gopherbot

This comment has been minimized.

gopherbot commented Sep 23, 2016

CL https://golang.org/cl/29677 mentions this issue.

@gopherbot

This comment has been minimized.

gopherbot commented Sep 24, 2016

CL https://golang.org/cl/29751 mentions this issue.

gopherbot pushed a commit that referenced this issue Sep 24, 2016

runtime: remove TestCollisions from -short
Takes a bit too long to run it all the time.

Fixes #17217
Update #17104

Change-Id: I4802190ea16ee0f436a7f95b093ea0f995f5b11d
Reviewed-on: https://go-review.googlesource.com/29751
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

gopherbot pushed a commit to golang/build that referenced this issue Sep 30, 2016

cmd/coordinator: cross-compile ARM build on Kubernetes
Saves 4.5 minutes or so by using fast x86 machines to build the ARM
build instead of running make.bash on Scaleway ARM machines.

We still run the tests on ARM, and have a separate builder only
running make.bash on ARM (see prior golang.org/cl/29670)

Fixes golang/go#17105
Updates golang/go#17104

Change-Id: I1cb7b0e5b1cc8b644195f262328884ed3aff120a
Reviewed-on: https://go-review.googlesource.com/29677
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment