Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: frequent failures on freebsd-arm-paulzhol builder with "signal: killed" since 2021-12-23 #50540

Open
bcmills opened this issue Jan 10, 2022 · 2 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 10, 2022

The freebsd-arm-paulzhol builder has failed a substantial fraction of builds with signal: killed since late December, and also seems to have a lot of missing runs over that interval (perhaps the buildlet itself is also being killed?).

greplogs --dashboard -md -l -e '(?ms)\Afreebsd-arm-paulzhol .* signal: killed\n' --since=2021-01-01

2022-01-08T00:24:25-90860e0/freebsd-arm-paulzhol
2022-01-08T00:24:25-5cfca57-90860e0/freebsd-arm-paulzhol
2022-01-08T00:24:25-1d35b9e-90860e0/freebsd-arm-paulzhol
2022-01-07T22:46:47-7f3eb61/freebsd-arm-paulzhol
2022-01-07T22:46:47-5cfca57-7f3eb61/freebsd-arm-paulzhol
2022-01-07T22:46:47-1d35b9e-7f3eb61/freebsd-arm-paulzhol
2022-01-07T22:40:23-c74be77/freebsd-arm-paulzhol
2022-01-07T22:40:23-5cfca57-c74be77/freebsd-arm-paulzhol
2022-01-07T21:15:14-be26ca9/freebsd-arm-paulzhol
2022-01-07T21:15:14-1d35b9e-be26ca9/freebsd-arm-paulzhol
2022-01-07T19:22:09-8b8fc08-f1596d7/freebsd-arm-paulzhol
2022-01-07T18:40:16-f1596d7/freebsd-arm-paulzhol
2022-01-07T18:40:16-1d35b9e-f1596d7/freebsd-arm-paulzhol
2022-01-07T18:20:24-5b0dc2d-ade5488/freebsd-arm-paulzhol
2022-01-07T06:34:04-5b0dc2d-11b28e7/freebsd-arm-paulzhol
2022-01-07T06:34:04-1d35b9e-11b28e7/freebsd-arm-paulzhol
2022-01-07T06:34:04-11b28e7/freebsd-arm-paulzhol
2022-01-07T02:37:20-40afced/freebsd-arm-paulzhol
2022-01-07T02:37:20-1d35b9e-40afced/freebsd-arm-paulzhol
2022-01-07T02:32:39-5b0dc2d-2bb7f6b/freebsd-arm-paulzhol
2022-01-07T02:32:39-2bb7f6b/freebsd-arm-paulzhol
2022-01-07T02:32:39-1d35b9e-2bb7f6b/freebsd-arm-paulzhol
2022-01-07T02:32:03-ab4556a/freebsd-arm-paulzhol
2022-01-07T02:32:03-5b0dc2d-ab4556a/freebsd-arm-paulzhol
2022-01-07T01:36:17-c1e7c51/freebsd-arm-paulzhol
2022-01-07T01:36:17-1d35b9e-c1e7c51/freebsd-arm-paulzhol
2022-01-07T00:15:59-07525e1/freebsd-arm-paulzhol
2022-01-07T00:02:57-c295137/freebsd-arm-paulzhol
2022-01-06T22:12:11-b9cae6f/freebsd-arm-paulzhol
2022-01-06T22:12:11-5b0dc2d-b9cae6f/freebsd-arm-paulzhol
2022-01-06T19:42:27-5b0dc2d-10f1ed1/freebsd-arm-paulzhol
2022-01-06T19:42:27-1d35b9e-10f1ed1/freebsd-arm-paulzhol
2022-01-06T19:42:27-10f1ed1/freebsd-arm-paulzhol
2021-12-29T04:10:07-1d35b9e-91e7821/freebsd-arm-paulzhol
2021-12-23T17:27:50-ed766b6/freebsd-arm-paulzhol
2021-10-02T16:05:55-a7fe161/freebsd-arm-paulzhol
2021-10-02T00:31:26-64da5e0/freebsd-arm-paulzhol
2021-09-17T08:20:48-6602c86/freebsd-arm-paulzhol
2021-08-02T17:18:57-8a7ee4c/freebsd-arm-paulzhol
2021-02-11T18:02:48-864d4f1/freebsd-arm-paulzhol
2021-01-21T19:15:21-3c2f11b/freebsd-arm-paulzhol

Attn @paulzhol; CC @golang/release.

@paulzhol
Copy link
Member

@paulzhol paulzhol commented Jan 10, 2022

The most recent ones which look like:

Building Go cmd/dist using /usr/home/paulzhol/go1.4. (go1.4-bootstrap-20170531 freebsd/arm)
Building Go toolchain1 using /usr/home/paulzhol/go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
go build cmd/compile/internal/ssa: /tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/tool/freebsd_arm/compile: signal: killed
go tool dist: FAILED: /tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/tool/freebsd_arm/go_bootstrap install -gcflags=all= -ldflags=all= -a -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1

Are due to out of memory during bootstrap. This also sometimes happened mid test like:
https://build.golang.org/log/04777783e4d4c1b95e6e9868f3d61e8bfaa076fe

They were due to the loss of a network connectivity to a block device used as swap, I've restored it. But it stil seems 1.18 is a much heavy memory user.

There used to be a way to control the bots by canceling and triggering a run a new. I think I've been locked out of that for several years. And besides it can never keep up with all the builds when it takes more than 3 houres per run.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 11, 2022

it stil seems 1.18 is a much heavy memory user.

That could be due to #44167 — including non-heap sources of GC work in the pacing decisions can cause less frequent collection when those sources are a significant fraction of memory usage.

There used to be a way to control the bots by canceling and triggering a run a new. I think I've been locked out of that for several years.

The command that does that is golang.org/x/build/cmd/retrybuilds, although I'm not sure what permissions that requires. (Someone from @golang/release could probably fill in more detail.)

And besides it can never keep up with all the builds when it takes more than 3 [hours] per run.

I think that's fine in general, as long as the builder makes regular forward progress? (It seems ok to be missing test runs in the middle of a burst of changes as long as we occasionally get an up-to-date run at the end of the burst.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants