all: plan9-arm builder failing frequently with a variety of errors #49338
For most of plan9-arm failures, the cause is filesystem flakiness on the builder machines (a cluster of 3 x raspberry pi 4). I've just reconfigured the build script to stop using ramfs (which has a locking bug) for /tmp, and stop using the ad hoc directory cache (which still seems to have some flaws). This should bypass most of the failures but will make the test suite a lot slower to run (by go.17 it had already regressed to about 2 hours even with these filesystem tweaks).
There are some other flakes of the "shouldn't be possible" variety, which are are rare and perplexing and impossible to replicate. They are occurring in general go runtime code, not specific to plan 9. There's a slight whiff of possible memory cache coherence trouble in some of these:
It would be good to have a server-class ARM platform capable of running plan 9 as a more reliable builder. I don't know of one.
The plan9 builders have relatively slow filesystems, and the website repo is relatively filesystem-intensive — many of its tests walk and process large trees of website content. Instead of trying to precisely tune the x/website tests to pass on the plan9 builders, let's skip this repo on them. For golang/go#49338. Change-Id: Ieaf50fb6955a6b39012a23ff72e61c1c31e4ce7c Reviewed-on: https://go-review.googlesource.com/c/build/+/362975 Trust: Bryan C. Mills <email@example.com> Run-TryBot: Bryan C. Mills <firstname.lastname@example.org> TryBot-Result: Go Bot <email@example.com> Reviewed-by: Carlos Amedee <firstname.lastname@example.org> Reviewed-by: Dmitri Shuralyov <email@example.com>
To prevent interleaving of output when 'go build' compiles several packages in parallel, the output mutex in the Builder struct must be locked around any calls to Builder.Print which could generate arbitrary amounts of text (ie more than is guaranteed to be written atomically to a pipe). Fixes #49987 For #49338 Change-Id: I7947df57667deeff3f03f231824298d823f8a943 Reviewed-on: https://go-review.googlesource.com/c/go/+/369018 Reviewed-by: Bryan Mills <firstname.lastname@example.org> Run-TryBot: Bryan Mills <email@example.com> TryBot-Result: Gopher Robot <firstname.lastname@example.org> Trust: Russ Cox <email@example.com>
This test has timed out several times recently on this builder. The test is pretty slow even on Linux, and may be especially filesystem-intensive. For golang/go#49338 Change-Id: Ife2bb399b10f369f815055bb0ad44bb007f606b7 Reviewed-on: https://go-review.googlesource.com/c/text/+/380414 Trust: Bryan Mills <firstname.lastname@example.org> Run-TryBot: Bryan Mills <email@example.com> TryBot-Result: Gopher Robot <firstname.lastname@example.org> Reviewed-by: Ian Lance Taylor <email@example.com>
A sampling from the past week or so. Undiagnosed failures in bold.
2022-01-24T16:42:11-2cc1836-19d819d/plan9-arm (#50775, mitigated)
That's running a
2022-01-21T21:27:57-e7c9de2-9eba5ff/plan9-arm (#46520, not specific to
For golang/go#52653. Updates golang/go#49338. Change-Id: Ib23be56bd79bce8b3168cb40064916a2a91dbb88 Reviewed-on: https://go-review.googlesource.com/c/build/+/408697 Auto-Submit: Bryan Mills <firstname.lastname@example.org> Reviewed-by: Alex Rakoczy <email@example.com> Run-TryBot: Bryan Mills <firstname.lastname@example.org> TryBot-Result: Gopher Robot <email@example.com>