Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: misc-compile TryBots are too slow #32632

Closed
alexbrainman opened this issue Jun 15, 2019 · 8 comments
Closed

x/build: misc-compile TryBots are too slow #32632

alexbrainman opened this issue Jun 15, 2019 · 8 comments

Comments

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jun 15, 2019

I am waiting for try-bots on https://go-review.googlesource.com/c/go/+/177959, and generally try-bots are finished withing about 5 minutes, but misc-compile-mips takes more than 11 minutes from https://farmer.golang.org/try?commit=fa91b668


• android-amd64-emu | pass
-- | --
• freebsd-amd64-12_0 | pass
• js-wasm | pass
• linux-386 | pass
• linux-amd64 | pass
• linux-amd64-race | pass
• misc-compile-darwin | pass
• misc-compile-freebsd | pass
• misc-compile-linuxarm | pass
• misc-compile-mips | running 11m3s
• misc-compile-nacl | pass
• misc-compile-netbsd | pass
• misc-compile-openbsd | pass
• misc-compile-other | pass
• misc-compile-plan9 | pass
• misc-compile-ppc | pass
• misc-compile-solaris | pass
• nacl-amd64p32 | pass
• openbsd-amd64-64 | pass
• windows-386-2008 | pass
• windows-amd64-2016 | pass

Every bot is finished, except misc-compile-mips. Maybe misc-compile-mips should not be part of try-bots, if it is soo slow.

Alex

@agnivade agnivade changed the title misc-compile-mips try-bot is too slow x/build: misc-compile-mips try-bot is too slow Jun 15, 2019
@gopherbot gopherbot added this to the Unreleased milestone Jun 15, 2019
@bcmills bcmills changed the title x/build: misc-compile-mips try-bot is too slow x/build: misc-compile TryBots are too slow Dec 20, 2019
@bcmills
Copy link
Member

@bcmills bcmills commented Dec 20, 2019

On https://farmer.golang.org/try?commit=c29ef957, I'm seeing multiple misc-compile TryBots (in this case, mips, netbsd, openbsd) taking upwards of 10m, when all of the real (sharded test) builders are already long done.

CC @dmitshur @toothrot @cagedmantis

@bcmills
Copy link
Member

@bcmills bcmills commented Feb 20, 2020

On https://farmer.golang.org/try?commit=5b2140c8, I'm again seeing misc-compile TryBots taking upwards of 10m.

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Feb 23, 2021

I too have observed this:

When these trybots take the longest, they become a bottleneck for overall TryBot completion speed (e.g., #17104).

The "misc-compile" trybots do much less work than normal trybots: they check only that compilation is successful, but don't execute tests. They're generally split by OS, matching all architectures available:

https://github.com/golang/build/blob/f539cd3e8d00b786b2c500b824a0d69d196ea3c6/dashboard/builders.go#L1600-L1609

Those architectures are currently tested sequentially. So one possible explanation is if newly added architectures start to get matched, and there become too many for sequential testing to be viable.

If that's the case, we can fix this by adjusting the split to avoid putting too much work into any single trybot.

@dmitshur dmitshur added this to Planned in Go Release Team Feb 23, 2021
@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Mar 3, 2021

I observed a case where misc-compile-openbsd was the slowest run:

Looking at its logs confirmed it was building as many as 5 different architectures, all sequentially:

### Building openbsd-386

### Building openbsd-amd64

### Building openbsd-arm

### Building openbsd-arm64

### Building openbsd-mips64

So the root problem here is pretty clear: we need to parallelize the workload better for OSes that have many architectures.

A fix can look like this:

-addMiscCompile("-openbsd", "^openbsd-")                 // 4: amd64, 386, arm, arm64
+addMiscCompile("-openbsd-amd", "^openbsd-(386|amd64)$") // 2: 386, amd64
+addMiscCompile("-openbsd-arm", "^openbsd-(arm|arm64)$") // 2: arm, arm64
+addMiscCompile("-openbsd-mips", "^openbsd-(mips64)$")   // 1: mips64

(We have good test coverage for missing trybots, so we'll know if something needs updating.)

@FiloSottile
Copy link
Contributor

@FiloSottile FiloSottile commented Apr 19, 2021

I am looking at a TryBot run where misc-compile takes longer than 15 minutes, and longer than linux-amd64-longtest.

image

If there is an easy fix, even if hacky, I would recommend applying it, because this is adding significant latency to everyone's development cycle.

TryBots are supposed to take at most 5 minutes, IIRC, so even running two architectures in a row would probably be too slow. (15min / 4 * 2 = 7.5min)

@dmitshur dmitshur self-assigned this Apr 23, 2021
@dmitshur dmitshur moved this from Planned to In Progress in Go Release Team Apr 23, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 23, 2021

Change https://golang.org/cl/313210 mentions this issue: dashboard: include at most 2 ports per misc-compile TryBot

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Apr 23, 2021

With the work done to migrate x/build to OpenCensus in CL 303669, we're closer to starting to measure and track TryBot completion time, which has been a blocker for making progress on #17104.

However, the problem here is quite clear and a simple fix is available, so I've sent CL 313210 that implements the suggestion in #32632 (comment). Our updated target time for TryBot completion time is 10 minutes (see subject of issue #17104), so going with at most 2 3 ports per misc-compile TryBot should be a good compromise until we measure holistically and find something that works even better.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 26, 2021

Change https://golang.org/cl/313533 mentions this issue: dashboard: consider disabled TryBot as not providing test coverage

gopherbot pushed a commit to golang/build that referenced this issue Apr 26, 2021
Some TryBot builders are disabled or configured with a custom
try policy that doesn't test the master branch of the main Go
repository. For example, openbsd-386-68 is configured with:

	tryBot: explicitTrySet("sys"),

So it's only a TryBot for the x/sys repo, not the main one.

Modify TestTryBotsCompileAllPorts to not consider such TryBots
when determining whether a port has sufficient pre-submit test
coverage.

This will help in the next change which re-arranges target
selection between misc-compile TryBots.

For golang/go#32632.

Change-Id: If393610d11b04b75133a4f10f89cfdfff9dbaece
Reviewed-on: https://go-review.googlesource.com/c/build/+/313533
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot gopherbot closed this in 646d48d Apr 26, 2021
Go Release Team automation moved this from In Progress to Done Apr 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants