[7.2.1] Permission error when deleting local docker sandbox/workdir

[layus]

Description of the bug:

WARNING: Failed to delete contents of sandbox /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox: com.google.devtools.build.lib.vfs.FileAccessException: /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/docker/9817 -> /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/185 (Permission denied)
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.IllegalStateException: Found unexpected files in sandbox base. Please report this in https://github.com/bazelbuild/bazel/issues.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:512)
        at com.google.devtools.build.lib.sandbox.SandboxModule.checkSandboxBaseTopOnlyContainsPersistentDirs(SandboxModule.java:529)
        at com.google.devtools.build.lib.sandbox.SandboxModule.afterCommand(SandboxModule.java:572)
        at com.google.devtools.build.lib.runtime.BlazeRuntime.afterCommand(BlazeRuntime.java:627)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:688)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:244)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:573)
        at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:644)
        at io.grpc.Context$1.run(Context.java:566)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

This happens when I enable local docker execution, using --experimental_enable_docker_sandbox. Usually when I cancel a build using Ctrl-C. The broken state seems to persist, as all following builds fail the same way until I clear the _moved_trash_dir using sudo. It also happens when the build has been running for too long, or the _moved_trash_dir has not been cleared for a while.

Which operating system are you running Bazel on?

Linux 5.14.0-1052 #59-Ubuntu SMP x86_64 GNU/Linux

What is the output of bazel info release?

release 7.1.2- (@non-git)

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

Patched version of nixpkgs bazel. None of the patch seem related to this issue.

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

Nope

Any other information, logs, or outputs that you want to share?

My initial assumption was that it was rules_nixpkgs related as copying files from the nix store copies readonly files that need careful handling to delete properly chmod +rwx && rm but looking at the content of that dir, I see root-owned subtrees that probably accumulate, which may explain why I also get the crash when the build goes on for too long.

The content itself is also weird with all these zeroes. But maybe it's by design ?

$ fd . --owner root  /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir | xargs ls -ld
[...]
drwxr-xr-x 3 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/615/execroot/
drwxr-xr-x 2 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/615/execroot/project_name/
drwxr-xr-x 3 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/618/
drwxr-xr-x 3 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/618/execroot/
drwxr-xr-x 2 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/618/execroot/project_name/
drwxr-xr-x 3 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/619/
drwxr-xr-x 3 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/619/execroot/
drwxr-xr-x 2 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/619/execroot/project_name/
drwxr-xr-x 3 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/629/
drwxr-xr-x 3 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/629/execroot/
drwxr-xr-x 2 root root 4096 Jun  7 14:48 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/1/629/execroot/project_name/
drwxr-xr-x 3 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/51/
drwxr-xr-x 3 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/51/execroot/
drwxr-xr-x 2 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/51/execroot/project_name/
drwxr-xr-x 3 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/52/
drwxr-xr-x 3 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/52/execroot/
drwxr-xr-x 2 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/52/execroot/project_name/
drwxr-xr-x 3 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/53/
drwxr-xr-x 3 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/53/execroot/
drwxr-xr-x 2 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/53/execroot/project_name/
drwxr-xr-x 3 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/55/
drwxr-xr-x 3 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/55/execroot/
drwxr-xr-x 2 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/55/execroot/project_name/
drwxr-xr-x 3 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/56/
drwxr-xr-x 3 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/56/execroot/
drwxr-xr-x 2 root root 4096 Jun  7 20:14 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/0/0/0/0/0/0/1/56/execroot/project_name/
drwxr-xr-x 3 root root 4096 Jun 11 17:54 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/1/381/
drwxr-xr-x 3 root root 4096 Jun 11 17:54 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/1/381/execroot/
drwxr-xr-x 2 root root 4096 Jun 11 17:54 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/1/381/execroot/project_name/
drwxr-xr-x 3 root root 4096 Jun 11 17:54 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/1/382/
drwxr-xr-x 3 root root 4096 Jun 11 17:54 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/1/382/execroot/
drwxr-xr-x 2 root root 4096 Jun 11 17:54 /home/gmaudoux/.cache/bazel/_bazel_gmaudoux/ab04c0d4cbd57d0aa51406a8273f8e84/sandbox/_moved_trash_dir/0/0/0/0/0/0/0/1/382/execroot/project_name/