Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: Dragonfly builder is failing to remove workdir #30853

Open
tdfbsd opened this Issue Mar 14, 2019 · 4 comments

Comments

Projects
None yet
6 participants
@tdfbsd
Copy link

tdfbsd commented Mar 14, 2019

What version of Go are you using (go version)?

go version go1.11.4 dragonfly/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
amd64

When the builder tries to remove the existing /tmp workdir, this happens:

2019/03/14 20:05:58 buildlet starting.
2019/03/14 20:05:58 remove /tmp/workdir-host-dragonfly-amd64-tdfbsd/tmp/TestSimpleCases_import_grouping_not_path_dependent_no_groups_Modules_GoPackages625130825/modcache/pkg/mod/github.com/local@v1.0.0/go.mod: permission denied
^C

% ls -l /tmp/workdir-host-dragonfly-amd64-tdfbsd/tmp/TestSimpleCases_import_grouping_not_path_dependent_no_groups_Modules_GoPackages625130825/modcache/pkg/mod/github.com/local@v1.0.0/
total 16
dr-xr-xr-x 2 tim wheel 88 Mar 14 16:04 bar
-r--r--r-- 1 tim wheel 24 Mar 14 16:04 go.mod

So, the direct cause is that go.mod is being created with read-only permissions. I have no idea when this started happening though, or why.

@bradfitz bradfitz changed the title builder: Dragonfly builder is failing to remove workdir x/build: Dragonfly builder is failing to remove workdir Mar 14, 2019

@gopherbot gopherbot added this to the Unreleased milestone Mar 14, 2019

@gopherbot gopherbot added the Builders label Mar 14, 2019

@gopherbot gopherbot added the Builders label Mar 14, 2019

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Mar 14, 2019

That probably comes from here:

~/src/golang.org/x$ grep -r TestSimpleCases
tools/imports/fix_test.go:func TestSimpleCases(t *testing.T) {

So, two questions:

  1. Why is TestSimpleCases not cleaning up the directories in creates in /tmp? (@ianthehat, who owns golang.org/x/tools/imports at this point? Possibly @bradfitz or @heschik?)
  2. Why is the builder using a “remove” operation that can fail to remove things? (If it is using rm -rf, it should probably chmod -R u+w before attempting that.)
@tdfbsd

This comment has been minimized.

Copy link
Author

tdfbsd commented Mar 14, 2019

When this happens, it goes into a loop of continually trying the remove and prevents the builder from running. This just started sometime in the past week.

@heschik

This comment has been minimized.

Copy link
Contributor

heschik commented Mar 14, 2019

I don't know why https://golang.org/cl/156078 wouldn't take care of this. I just double-checked and on my workstation it doesn't leave directories behind, even in cases of test failure. I guess we can change Cleanup() to return an error and maybe that'll shed light on the problem?

@tdfbsd

This comment has been minimized.

Copy link
Author

tdfbsd commented Mar 15, 2019

I can try a debug patch, if you have one. For now, I'm running the builder as root to work around this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.