Bazel clean sometimes fails with Directory not empty

[guibou]

Description of the problem / feature request:

bazel clean on my CI workers sometimes fails with the following error:

ERROR: unlinkat(/var/lib/buildkite-agent-ci/.cache/bazel/output_base/...) (Directory not empty)

It is really difficult to reproduce because the next invocation of bazel clean does succeed most of the time. I do actually run bazel clean || bazel clean || bazel clean in my CI as a workaround.

I'm using a custom --output_base shared between multiples bazel workspace (called output_base here).

There is no other bazel process running on the worker at the same time.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I'm sorry, I have no idea. I'm opening the bug report in order to document my issue. I'll try to add more information if they become available.

What operating system are you running Bazel on?

Linux

What's the output of bazel info release?

release 3.3.1- (@non-git)

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

Using bazel from nixpkgs.

Have you found anything relevant by searching the web?

#5907 is similar, but is fixed since years and was apparently only on Windows.

[aiuto]

Does anyone know if sharing output_base actually supported?

[BalestraPatrick]

We're seeing this issue as well due to having a custom --output_base for our IDE integration. When the Xcode version changes, we need to run bazelisk clean --expunge.