Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: frequent dragonfly-amd64 failures with "No space left on device" since 2021-12-07 #50538

Open
bcmills opened this issue Jan 10, 2022 · 7 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 10, 2022

# golang.org/x/website/cmd/golangorg.test
/tmp/workdir-host-dragonfly-amd64-master/go/pkg/tool/dragonfly_amd64/link: running gcc failed: exit status 1
/usr/libexec/binutils234/elf/ld.gold: fatal error: $WORK/b330/golangorg.test: No space left on device
collect2: error: ld returned 1 exit status

FAIL	golang.org/x/website/cmd/golangorg [build failed]

It's not clear to me whether this is due to the Go buildlet failing to clear temporary data between runs, or due to some other programs running on the host filling up /tmp.

greplogs --dashboard -md -l -e '(?ms)\Adragonfly-amd64.*: No space left on device\n'

2022-01-10T04:23:42-5dfff35-6df0957/dragonfly-amd64
2022-01-09T18:43:51-5dfff35-2639f2f/dragonfly-amd64
2022-01-08T00:24:25-5dfff35-90860e0/dragonfly-amd64
2022-01-07T21:20:23-5dfff35-9de1ac6/dragonfly-amd64
2022-01-07T18:40:16-0b28845-f1596d7/dragonfly-amd64
2022-01-07T15:48:28-0b28845-9de1ac6/dragonfly-amd64
2022-01-07T06:34:04-f4f7323-11b28e7/dragonfly-amd64
2022-01-07T02:37:20-f4f7323-40afced/dragonfly-amd64
2022-01-07T02:32:39-f4f7323-2bb7f6b/dragonfly-amd64
2022-01-07T02:32:03-f4f7323-ab4556a/dragonfly-amd64
2022-01-07T01:36:17-f4f7323-c1e7c51/dragonfly-amd64
2022-01-07T00:15:59-f4f7323-07525e1/dragonfly-amd64
2022-01-07T00:02:57-f4f7323-c295137/dragonfly-amd64
2022-01-06T23:48:15-f4f7323-9de1ac6/dragonfly-amd64
2022-01-06T23:48:15-f4f7323-042548b/dragonfly-amd64
2022-01-06T23:39:43-e63aea0-042548b/dragonfly-amd64
2022-01-06T22:12:11-e63aea0-b9cae6f/dragonfly-amd64
2022-01-06T21:38:59-e63aea0-61014f0/dragonfly-amd64
2022-01-06T20:42:09-e63aea0-9de1ac6/dragonfly-amd64
2022-01-06T20:42:09-e63aea0-10f1ed1/dragonfly-amd64
2022-01-06T19:42:27-b511507-10f1ed1/dragonfly-amd64
2022-01-06T01:14:01-58a5fe9-b5bfaf4/dragonfly-amd64
2022-01-06T01:14:01-58a5fe9-ab70d7c/dragonfly-amd64
2022-01-06T00:26:47-4a88b4c-b5bfaf4/dragonfly-amd64
2022-01-05T21:22:03-4a88b4c-2b39d86/dragonfly-amd64
2022-01-05T19:39:16-4a88b4c-ab70d7c/dragonfly-amd64
2022-01-05T19:39:16-4a88b4c-002283e/dragonfly-amd64
2022-01-05T15:54:37-7a18f71-2c58bb2/dragonfly-amd64
2022-01-05T09:50:29-2aac4c0-2c58bb2/dragonfly-amd64
2022-01-05T01:53:39-2aac4c0-301db3f/dragonfly-amd64
2022-01-04T22:27:20-2aac4c0-f154f8b/dragonfly-amd64
2022-01-04T22:05:15-2aac4c0-e39ab9b/dragonfly-amd64
2022-01-04T20:37:27-2aac4c0-95b240b/dragonfly-amd64
2022-01-04T19:56:33-2b23ec4-95b240b/dragonfly-amd64
2022-01-04T19:56:33-2b23ec4-1242f43/dragonfly-amd64
2022-01-04T19:30:38-995577f-95b240b/dragonfly-amd64
2022-01-04T17:38:42-b24dbb6-95b240b/dragonfly-amd64
2022-01-04T17:38:42-b24dbb6-1242f43/dragonfly-amd64
2022-01-04T16:25:19-bc5b8e0-95b240b/dragonfly-amd64
2022-01-03T23:45:12-f07bd53-95b240b/dragonfly-amd64
2022-01-02T14:27:43-f07bd53-c886143/dragonfly-amd64
2022-01-02T14:27:32-f07bd53-d181885/dragonfly-amd64
2021-12-29T23:18:47-f07bd53-6178d25/dragonfly-amd64
2021-12-29T23:18:47-f07bd53-1242f43/dragonfly-amd64
2021-12-29T20:20:32-56b50be-6178d25/dragonfly-amd64
2021-12-29T19:23:11-56b50be-8a306e2/dragonfly-amd64
2021-12-29T17:53:46-56b50be-91e7821/dragonfly-amd64
2021-12-29T04:10:07-ee35583-91e7821/dragonfly-amd64
2021-12-29T01:44:55-ee35583-a78532a/dragonfly-amd64
2021-12-28T17:18:11-ee35583-b357b05/dragonfly-amd64
2021-12-23T20:03:38-faa0890-b357b05/dragonfly-amd64
2021-12-23T19:58:13-faa0890-0e4b878/dragonfly-amd64
2021-12-23T18:58:36-faa0890-af3b8cf/dragonfly-amd64
2021-12-23T17:27:50-faa0890-ed766b6/dragonfly-amd64
2021-12-22T23:31:40-faa0890-d2ce939/dragonfly-amd64
2021-12-22T20:47:09-faa0890-8cfcee1/dragonfly-amd64
2021-12-22T18:43:55-faa0890-0f3becf/dragonfly-amd64
2021-12-22T00:11:21-91908ba-90fb5a4/dragonfly-amd64
2021-12-21T23:09:59-91908ba-ebac50e/dragonfly-amd64
2021-12-21T22:12:49-91908ba-a1ce550/dragonfly-amd64
2021-12-21T19:17:52-dc59b70-a1ce550/dragonfly-amd64
2021-12-21T17:58:16-dc59b70-2d1d548/dragonfly-amd64
2021-12-21T15:58:14-983ac79-2d1d548/dragonfly-amd64
2021-12-21T14:25:10-49c3839-de690c2/dragonfly-amd64
2021-12-21T14:25:10-49c3839-2d1d548/dragonfly-amd64
2021-12-21T03:55:43-d4a18cb-2d1d548/dragonfly-amd64
2021-12-21T03:54:36-d4a18cb-b5e0654/dragonfly-amd64
2021-12-21T01:51:23-d4a18cb-4dfbb89/dragonfly-amd64
2021-12-21T01:10:17-d4a18cb-e087949/dragonfly-amd64
2021-12-21T00:00:05-d4a18cb-1555062/dragonfly-amd64
2021-12-20T23:46:23-c471c5c-1555062/dragonfly-amd64
2021-12-20T23:32:14-c471c5c-6713b5d/dragonfly-amd64
2021-12-20T15:13:56-4e7a786-7c94355/dragonfly-amd64
2021-12-20T15:13:38-4e7a786-a2004de/dragonfly-amd64
2021-12-19T20:16:45-4e7a786-87b2a54/dragonfly-amd64
2021-12-17T16:05:31-4e7a786-c5fee93/dragonfly-amd64
2021-12-17T03:59:09-4e7a786-5299390/dragonfly-amd64
2021-12-17T03:28:06-4e7a786-3c49583/dragonfly-amd64
2021-12-17T02:00:14-4e7a786-55a25aa/dragonfly-amd64
2021-12-17T01:59:32-4e7a786-63077bf/dragonfly-amd64
2021-12-17T00:14:53-4e7a786-2e6e9df/dragonfly-amd64
2021-12-16T23:52:17-4e7a786-0c24038/dragonfly-amd64
2021-12-16T22:41:31-4e7a786-1387b5e/dragonfly-amd64
2021-12-16T22:39:00-4e7a786-b1a53ec/dragonfly-amd64
2021-12-16T22:15:57-4e7a786-d01cd8f/dragonfly-amd64
2021-12-16T22:15:22-4e7a786-d93677a/dragonfly-amd64
2021-12-16T21:08:09-4e7a786-ae695cd/dragonfly-amd64
2021-12-16T21:02:37-4e7a786-ea26ce7/dragonfly-amd64
2021-12-16T15:44:40-4e7a786-1d3a5b4/dragonfly-amd64
2021-12-16T14:37:07-4e7a786-7f23145/dragonfly-amd64
2021-12-15T14:48:19-0c791db-b1c7703/dragonfly-amd64
2021-12-15T00:33:55-18bc0f9-9d0ca26/dragonfly-amd64
2021-12-14T22:42:47-18bc0f9-1540239/dragonfly-amd64
2021-12-08T17:24:46-0dd6a27-ac7e950/dragonfly-amd64
2021-12-07T03:44:02-0dd6a27-dc65c48/dragonfly-amd64
2021-02-18T17:57:36-a76efea/dragonfly-amd64-5_8

CC @golang/release; attn @tuxillo

@bcmills bcmills added this to the Go1.18 milestone Jan 10, 2022
@bcmills bcmills removed this from the Go1.18 milestone Jan 10, 2022
@bcmills bcmills added this to the Backlog milestone Jan 10, 2022
@tuxillo
Copy link
Contributor

@tuxillo tuxillo commented Jan 10, 2022

This is what I can see in the builder:

root@gobuilder:~ # du -hs /tmp/*
 88K    /tmp/gobuilder.log
1.3G    /tmp/workdir-host-dragonfly-amd64-master
root@gobuilder:~ # df -h /tmp
Filesystem   Size   Used  Avail Capacity  Mounted on
tmpfs       1967M  1363M   603M    69%    /tmp

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 12, 2022

<2GiB seems awfully small for a /tmp partition. Any chance it could be increased?

Barring that, any idea what changed around Dec. 7 to cause this builder to start running over the limit? It looks like the failures overwhelmingly occur when linking golang.org/x/website/cmd/golangorg.test or tests closely before or after that one, but there isn't an obvious culprit CL –- it isn't even clear to me whether this is due to a change in x/website proper, or in the size of the linker inputs or outputs due to some other change in the toolchain.

@bcmills bcmills removed this from the Backlog milestone Jan 12, 2022
@bcmills bcmills added this to the Go1.18 milestone Jan 12, 2022
@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 12, 2022

Marking as release-blocker for Go 1.18 via #11811. This is clearly a mid-cycle regression of some sort, and it's not obvious to me what the root cause is or how it can be mitigated.

@tuxillo
Copy link
Contributor

@tuxillo tuxillo commented Jan 12, 2022

We did not do anything on that VM, in fact it's up for 70 days and running a kernel from Oct, 17th:

root@gobuilder:~ # uname -a
DragonFly gobuilder.muscles.dragonflybsd.org 6.1-DEVELOPMENT DragonFly v6.1.0.477.gc2e50-DEVELOPMENT #0: Sun Oct 17 10:27:56 UTC 2021     root@pkgbox64.dragonflybsd.org:/usr/obj/usr/src/sys/X86_64_GENERIC  x86_64
root@gobuilder:~ # uptime
 8:25AM  up 70 days,  9:39, 1 user, load averages: 5.79, 4.98, 4.47

The /tmp size can be increased indeed. I was also thinking in upgrading to latest bleeding-edge but I'll make no changes until you tell me to.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 12, 2022

Since the dragonfly builder is reverse, @dmitshur suggested that you could go ahead and increase the /tmp partition on this builder and we can investigate the size increases independently (for example, on a GCE builder).

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 12, 2022

I would suggest not upgrading the OS image at this point in the cycle, though. (Early in the 1.19 development window would give more time to chase down any regressions that may occur.)

@tuxillo
Copy link
Contributor

@tuxillo tuxillo commented Jan 13, 2022

@bcmills ok, i'll stop the builder, increase the /tmp partition and restart the VM. I won't be upgrading until you guys tell me to do so. With regards to a GCE builder, I still have to create images for GCE, unless somebody beats me to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants