Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: plan9-arm builder "disk is full" failures starting 2022-01-26 #50857

Closed
bcmills opened this issue Jan 27, 2022 · 4 comments
Closed

x/build: plan9-arm builder "disk is full" failures starting 2022-01-26 #50857

bcmills opened this issue Jan 27, 2022 · 4 comments
Labels
arch-arm Builders NeedsInvestigation OS-Plan9
Milestone

Comments

@bcmills bcmills added OS-Plan9 Builders NeedsInvestigation arch-arm labels Jan 27, 2022
@gopherbot gopherbot added this to the Unreleased milestone Jan 27, 2022
@millerresearch
Copy link
Contributor

@millerresearch millerresearch commented Jan 27, 2022

Yes, the disk (partition) was full. That particular machine was running the buildlet in a loop without re-booting between runs. I think it was the testcache which grew too much.

Given the overhead cost of saving test inputs in the cache (especially on Plan 9, but it's a nonzero cost anywhere), and given that every builder run should be independent and never use cached results, might it be a good idea to disable testcache for test runs on the builders? For the subrepo tests it's just a matter of adding -count 1 to the parameters of the go test command sent to the builder. For the main tree test, I'm not sure how to disable the testcache with go tool dist test, but there ought to be simple way.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 27, 2022

Given that most of the Go project's own builders run in VMs, I would guess that the buildlet binary assumes that the build cache goes away completely in between runs.

Ah — here it is!
https://cs.opensource.google/go/x/build/+/master:cmd/buildlet/buildlet.go;l=211-212;drc=6d1c239cf4abb0e612c783ff67bf13966713df5f

For some reason the logic to clean the cache seems to be skipped on windows and plan9 builders, and I don't see any comments in the code or in CL 144637 explaining why. @dmitshur, @bradfitz: can you offer any insight?

@millerresearch
Copy link
Contributor

@millerresearch millerresearch commented Jan 27, 2022

The usual plan9-arm builders re-boot between runs with a fresh empty disk partition, so normally there wouldn't be a cache to clean; so that's not a problem. My suggestion to disable the testcache for builders is because of the time overhead in saving testcache input information (involving sometimes huge numbers of os.Stat calls) which will never be used.

@bcmills
Copy link
Member Author

@bcmills bcmills commented May 26, 2022

greplogs -l -e '(?ms)\Aplan9-arm .* disk is full' --since=2022-01-28

(0 matching logs)

@bcmills bcmills closed this as completed May 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-arm Builders NeedsInvestigation OS-Plan9
Projects
None yet
Development

No branches or pull requests

3 participants