x/build: misc-compile TryBots are too slow

[alexbrainman]

I am waiting for try-bots on https://go-review.googlesource.com/c/go/+/177959, and generally try-bots are finished withing about 5 minutes, but misc-compile-mips takes more than 11 minutes from https://farmer.golang.org/try?commit=fa91b668

• android-amd64-emu | pass
-- | --
• freebsd-amd64-12_0 | pass
• js-wasm | pass
• linux-386 | pass
• linux-amd64 | pass
• linux-amd64-race | pass
• misc-compile-darwin | pass
• misc-compile-freebsd | pass
• misc-compile-linuxarm | pass
• misc-compile-mips | running 11m3s
• misc-compile-nacl | pass
• misc-compile-netbsd | pass
• misc-compile-openbsd | pass
• misc-compile-other | pass
• misc-compile-plan9 | pass
• misc-compile-ppc | pass
• misc-compile-solaris | pass
• nacl-amd64p32 | pass
• openbsd-amd64-64 | pass
• windows-386-2008 | pass
• windows-amd64-2016 | pass

Every bot is finished, except misc-compile-mips. Maybe misc-compile-mips should not be part of try-bots, if it is soo slow.

Alex

[bcmills]

On https://farmer.golang.org/try?commit=c29ef957, I'm seeing multiple misc-compile TryBots (in this case, mips, netbsd, openbsd) taking upwards of 10m, when all of the real (sharded test) builders are already long done.

CC @dmitshur @toothrot @cagedmantis

[bcmills]

On https://farmer.golang.org/try?commit=5b2140c8, I'm again seeing misc-compile TryBots taking upwards of 10m.

[dmitshur]

I too have observed this:

When these trybots take the longest, they become a bottleneck for overall TryBot completion speed (e.g., #17104).

The "misc-compile" trybots do much less work than normal trybots: they check only that compilation is successful, but don't execute tests. They're generally split by OS, matching all architectures available:

https://github.com/golang/build/blob/f539cd3e8d00b786b2c500b824a0d69d196ea3c6/dashboard/builders.go#L1600-L1609

Those architectures are currently tested sequentially. So one possible explanation is if newly added architectures start to get matched, and there become too many for sequential testing to be viable.

If that's the case, we can fix this by adjusting the split to avoid putting too much work into any single trybot.

[dmitshur]

I observed a case where misc-compile-openbsd was the slowest run:

Looking at its logs confirmed it was building as many as 5 different architectures, all sequentially:

### Building openbsd-386

### Building openbsd-amd64

### Building openbsd-arm

### Building openbsd-arm64

### Building openbsd-mips64

So the root problem here is pretty clear: we need to parallelize the workload better for OSes that have many architectures.

A fix can look like this:

-addMiscCompile("-openbsd", "^openbsd-")                 // 4: amd64, 386, arm, arm64
+addMiscCompile("-openbsd-amd", "^openbsd-(386|amd64)$") // 2: 386, amd64
+addMiscCompile("-openbsd-arm", "^openbsd-(arm|arm64)$") // 2: arm, arm64
+addMiscCompile("-openbsd-mips", "^openbsd-(mips64)$")   // 1: mips64

(We have good test coverage for missing trybots, so we'll know if something needs updating.)

[FiloSottile]

I am looking at a TryBot run where misc-compile takes longer than 15 minutes, and longer than linux-amd64-longtest.

If there is an easy fix, even if hacky, I would recommend applying it, because this is adding significant latency to everyone's development cycle.

TryBots are supposed to take at most 5 minutes, IIRC, so even running two architectures in a row would probably be too slow. (15min / 4 * 2 = 7.5min)