Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: shard tests over N VMs to speed all.bash #10029

Closed
bradfitz opened this issue Feb 27, 2015 · 4 comments

Comments

Projects
None yet
3 participants
@bradfitz
Copy link
Member

commented Feb 27, 2015

As part of the rewrite of the coordinator and building subrepos (#8646 and #9506), we also want the coordinator to use N VMs to do an all.{bash,bat,rc} instead of 1, and then shard test execution over a bunch of VMs, to speed things up.

First step is to rewrite the bottom half of all.bash into Go and ditch the three variants (bash, bat, rc).

/cc @adg

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Mar 3, 2015

bradfitz added a commit that referenced this issue Mar 3, 2015

build: convert run.bash, run.bat, and run.rc into a Go program
This will enable test sharding over multiple VMs, to speed trybot answers.

Update #10029

Change-Id: Ie277c6459bc38005e4d6af14d22effeaa0a4667e
Reviewed-on: https://go-review.googlesource.com/6531
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@rsc rsc changed the title build: shard tests over N VMs to speed all.bash x/build: shard tests over N VMs to speed all.bash Apr 14, 2015

@rsc rsc modified the milestones: Unreleased, Unplanned Apr 14, 2015

@rsc rsc removed the builder label Apr 14, 2015

bradfitz added a commit to golang/build that referenced this issue May 14, 2015

dashboard, cmd/coordinator: add concept of trybot-only builders
Make all-compiler a trybot-only builder. This will be used for
temporary sharded ARM trybots.

Updates golang/go#10029

Change-Id: I6cc3188887c2f24edef97d0a9354e39553a15896
Reviewed-on: https://go-review.googlesource.com/10051
Reviewed-by: Andrew Gerrand <adg@golang.org>

bradfitz added a commit that referenced this issue May 14, 2015

cmd/dist: support test filtering via repurposed env variable, negation
For upcoming sharded ARM builders.

Updates #10029

Change-Id: I3b1df9560be697c514a8ced0462814d406e23132
Reviewed-on: https://go-review.googlesource.com/10055
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented May 14, 2015

CL https://golang.org/cl/10056 mentions this issue.

bradfitz added a commit to golang/build that referenced this issue May 14, 2015

dashboard, cmd/coordinator: sharded ARM trybot-only builders
This is a quick & hacky version of our eventual sharding plan.  Later
we want to run make.bash once, snapshot it, and then intelligently
schedule all the tests out over N machines (starting with the built
snapshot), giving each the next test to run as they become idle.
And then we want to stitch together all of their output.

But this CL just makes 6 ARM builders, each of which runs make.bash (4
minutes) and then each which runs a statically-configured subset of
the tests. Based on measurements, each should run between 8.2 and 10.6
minutes. We can break up some of the larger test cases later into more
shardable pieces (in particular: "test").

Update golang/go#10029

Change-Id: Ib3e008b61ce6f28d799ae298c8467bf62d89e5d4
Reviewed-on: https://go-review.googlesource.com/10056
Reviewed-by: Andrew Gerrand <adg@golang.org>

bradfitz added a commit that referenced this issue Jun 3, 2015

runtime: fix TestFixedGOROOT to properly restore the GOROOT env var a…
…fter test

Otherwise subsequent tests won't see any modified GOROOT.

With this CL I can move my GOROOT, set GOROOT to the new location, and
the runtime tests pass. Previously the crash_tests would instead look
for the GOROOT baked into the binary, instead of the env var:

--- FAIL: TestGcSys (0.01s)
        crash_test.go:92: building source: exit status 2
                go: cannot find GOROOT directory: /home/bradfitz/go
--- FAIL: TestGCFairness (0.01s)
        crash_test.go:92: building source: exit status 2
                go: cannot find GOROOT directory: /home/bradfitz/go
--- FAIL: TestGdbPython (0.07s)
        runtime-gdb_test.go:64: building source exit status 2
                go: cannot find GOROOT directory: /home/bradfitz/go
--- FAIL: TestLargeStringConcat (0.01s)
        crash_test.go:92: building source: exit status 2
                go: cannot find GOROOT directory: /home/bradfitz/go

Update #10029

Change-Id: If91be0f04d3acdcf39a9e773a4e7905a446bc477
Reviewed-on: https://go-review.googlesource.com/10685
Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Jun 4, 2015

CL https://golang.org/cl/10688 mentions this issue.

bradfitz added a commit that referenced this issue Jun 4, 2015

test: make test/run.go support sharding
Also modifies 'dist test' to use that sharding, and removes some old
temporary stuff from dist test which are no longer required.

'dist test' now also supports running a list of tests given in
arguments, mutually exclusive with the existing -run=REGEXP flag. The
hacky fast paths for avoiding the 1 second "go list" latency are now
removed and only apply to the case where partial tests are run via
args, instead of regex.  The build coordinator will use both styles
for awhile. (the statically-sharded ARM builders on scaleway will
continue to use regexps, but the dynamically-shared builders on GCE
will use the list of tests)

Updates #10029

Change-Id: I557800a54dfa6f3b5100ef4c26fe397ba5189813
Reviewed-on: https://go-review.googlesource.com/10688
Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

@rsc rsc added the Builders label Jun 11, 2015

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Mar 31, 2016

This was done some time ago.

@bradfitz bradfitz closed this Mar 31, 2016

@golang golang locked and limited conversation to collaborators Mar 31, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.