Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: misc-compile TryBots are too slow #32632

alexbrainman opened this issue Jun 15, 2019 · 8 comments

x/build: misc-compile TryBots are too slow #32632

alexbrainman opened this issue Jun 15, 2019 · 8 comments
Builders FrozenDueToAge NeedsFix ToolSpeed


Copy link

@alexbrainman alexbrainman commented Jun 15, 2019

I am waiting for try-bots on, and generally try-bots are finished withing about 5 minutes, but misc-compile-mips takes more than 11 minutes from

• android-amd64-emu | pass
-- | --
• freebsd-amd64-12_0 | pass
• js-wasm | pass
• linux-386 | pass
• linux-amd64 | pass
• linux-amd64-race | pass
• misc-compile-darwin | pass
• misc-compile-freebsd | pass
• misc-compile-linuxarm | pass
• misc-compile-mips | running 11m3s
• misc-compile-nacl | pass
• misc-compile-netbsd | pass
• misc-compile-openbsd | pass
• misc-compile-other | pass
• misc-compile-plan9 | pass
• misc-compile-ppc | pass
• misc-compile-solaris | pass
• nacl-amd64p32 | pass
• openbsd-amd64-64 | pass
• windows-386-2008 | pass
• windows-amd64-2016 | pass

Every bot is finished, except misc-compile-mips. Maybe misc-compile-mips should not be part of try-bots, if it is soo slow.


@agnivade agnivade changed the title misc-compile-mips try-bot is too slow x/build: misc-compile-mips try-bot is too slow Jun 15, 2019
@gopherbot gopherbot added this to the Unreleased milestone Jun 15, 2019
@gopherbot gopherbot added the Builders label Jun 15, 2019
@agnivade agnivade added the NeedsInvestigation label Jun 15, 2019
@bcmills bcmills changed the title x/build: misc-compile-mips try-bot is too slow x/build: misc-compile TryBots are too slow Dec 20, 2019
Copy link

@bcmills bcmills commented Dec 20, 2019

On, I'm seeing multiple misc-compile TryBots (in this case, mips, netbsd, openbsd) taking upwards of 10m, when all of the real (sharded test) builders are already long done.

CC @dmitshur @toothrot @cagedmantis

Copy link

@bcmills bcmills commented Feb 20, 2020

On, I'm again seeing misc-compile TryBots taking upwards of 10m.

Copy link

@dmitshur dmitshur commented Feb 23, 2021

I too have observed this:

When these trybots take the longest, they become a bottleneck for overall TryBot completion speed (e.g., #17104).

The "misc-compile" trybots do much less work than normal trybots: they check only that compilation is successful, but don't execute tests. They're generally split by OS, matching all architectures available:

Those architectures are currently tested sequentially. So one possible explanation is if newly added architectures start to get matched, and there become too many for sequential testing to be viable.

If that's the case, we can fix this by adjusting the split to avoid putting too much work into any single trybot.

@dmitshur dmitshur added this to Planned in Go Release Team Feb 23, 2021
Copy link

@dmitshur dmitshur commented Mar 3, 2021

I observed a case where misc-compile-openbsd was the slowest run:

Looking at its logs confirmed it was building as many as 5 different architectures, all sequentially:

### Building openbsd-386

### Building openbsd-amd64

### Building openbsd-arm

### Building openbsd-arm64

### Building openbsd-mips64

So the root problem here is pretty clear: we need to parallelize the workload better for OSes that have many architectures.

A fix can look like this:

-addMiscCompile("-openbsd", "^openbsd-")                 // 4: amd64, 386, arm, arm64
+addMiscCompile("-openbsd-amd", "^openbsd-(386|amd64)$") // 2: 386, amd64
+addMiscCompile("-openbsd-arm", "^openbsd-(arm|arm64)$") // 2: arm, arm64
+addMiscCompile("-openbsd-mips", "^openbsd-(mips64)$")   // 1: mips64

(We have good test coverage for missing trybots, so we'll know if something needs updating.)

@dmitshur dmitshur added NeedsFix and removed NeedsInvestigation labels Mar 3, 2021
Copy link

@FiloSottile FiloSottile commented Apr 19, 2021

I am looking at a TryBot run where misc-compile takes longer than 15 minutes, and longer than linux-amd64-longtest.


If there is an easy fix, even if hacky, I would recommend applying it, because this is adding significant latency to everyone's development cycle.

TryBots are supposed to take at most 5 minutes, IIRC, so even running two architectures in a row would probably be too slow. (15min / 4 * 2 = 7.5min)

@dmitshur dmitshur self-assigned this Apr 23, 2021
@dmitshur dmitshur moved this from Planned to In Progress in Go Release Team Apr 23, 2021
Copy link

@gopherbot gopherbot commented Apr 23, 2021

Change mentions this issue: dashboard: include at most 2 ports per misc-compile TryBot

Copy link

@dmitshur dmitshur commented Apr 23, 2021

With the work done to migrate x/build to OpenCensus in CL 303669, we're closer to starting to measure and track TryBot completion time, which has been a blocker for making progress on #17104.

However, the problem here is quite clear and a simple fix is available, so I've sent CL 313210 that implements the suggestion in #32632 (comment). Our updated target time for TryBot completion time is 10 minutes (see subject of issue #17104), so going with at most 2 3 ports per misc-compile TryBot should be a good compromise until we measure holistically and find something that works even better.

Copy link

@gopherbot gopherbot commented Apr 26, 2021

Change mentions this issue: dashboard: consider disabled TryBot as not providing test coverage

gopherbot pushed a commit to golang/build that referenced this issue Apr 26, 2021
Some TryBot builders are disabled or configured with a custom
try policy that doesn't test the master branch of the main Go
repository. For example, openbsd-386-68 is configured with:

	tryBot: explicitTrySet("sys"),

So it's only a TryBot for the x/sys repo, not the main one.

Modify TestTryBotsCompileAllPorts to not consider such TryBots
when determining whether a port has sufficient pre-submit test

This will help in the next change which re-arranges target
selection between misc-compile TryBots.

For golang/go#32632.

Change-Id: If393610d11b04b75133a4f10f89cfdfff9dbaece
Trust: Dmitri Shuralyov <>
Run-TryBot: Dmitri Shuralyov <>
TryBot-Result: Go Bot <>
Reviewed-by: Heschi Kreinick <>
Go Release Team automation moved this from In Progress to Done Apr 26, 2021
@golang golang locked and limited conversation to collaborators Apr 26, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Builders FrozenDueToAge NeedsFix ToolSpeed

No branches or pull requests

6 participants