"unlinkat ([path]) (Directory not empty)" errors with experimental_worker_for_repo_fetching

[jfirebaugh]

Description of the bug:

After upgrading to Bazel 7.2.0 and removing --experimental_worker_for_repo_fetching=off from .bazelrc (because #21803 was fixed), we started to observe failures of the following form:

ERROR: /home/circleci/figma/WORKSPACE.bazel:157:27: fetching node_repositories rule //external:nodejs_linux_amd64: java.io.IOException: unlinkat (/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/nodejs_linux_amd64/bin/nodejs) (Directory not empty)
ERROR: no such package '@@nodejs_linux_amd64//': unlinkat (/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/nodejs_linux_amd64/bin/nodejs) (Directory not empty)

These errors appear to be linked with repository fetch restarts, particularly "fetch interrupted due to memory pressure; restarting". In two of two cases where I have seen this error, it happened in a repo where the repo fetch got restarted, judging by either the presence of that log message or the JSON trace profile.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 7.2.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

Bazel Slack discussion: https://bazelbuild.slack.com/archives/CA31HN1T3/p1718057176146129

Any other information, logs, or outputs that you want to share?

No response

[jfirebaugh]

I am presently confirming that --experimental_worker_for_repo_fetching=off mitigates the issue.

[jfirebaugh]

Still on 7.2.0, haven't seen this again with --experimental_worker_for_repo_fetching=off.

[fmeum]

I managed to reproduce this via #22748 (comment):

INFO: repository @@aspect_rules_js~~npm~npm__decode-uri-component__0.2.2' used the following cache hits instead of downloading the corresponding file.
 * Hash '16a51843ef28d79f06c864eb305266b3daa1dc2a932af02a82ab139e42c8f2c2aed34dbca2ba8187134c16415e9f4cc6ca0e9ea40083df6a63564e7c9e0204ad' for https://registry.npmjs.org/decode-uri-component/-/decode-uri-component-0.2.2.tgz
If the definition of 'repository @@aspect_rules_js~~npm~npm__decode-uri-component__0.2.2' was updated, verify that the hashes were also updated.
com.google.devtools.build.lib.rules.repository.RepositoryFunction$RepositoryFunctionException: java.io.IOException: /private/var/tmp/_bazel_fmeum/0465a0857e26a35c072f48ab751a1794/external/aspect_rules_js~~npm~npm__decode-uri-component__0.2.2 (Directory not empty)
	at com.google.devtools.build.lib.rules.repository.RepositoryFunction.setupRepoRoot(RepositoryFunction.java:139)
	at com.google.devtools.build.lib.bazel.repository.starlark.StarlarkRepositoryFunction.lambda$fetch$1(StarlarkRepositoryFunction.java:154)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.ThreadPerTaskExecutor$TaskRunner.run(ThreadPerTaskExecutor.java:314)
	at java.base/java.lang.VirtualThread.run(VirtualThread.java:309)
Caused by: java.io.IOException: /private/var/tmp/_bazel_fmeum/0465a0857e26a35c072f48ab751a1794/external/aspect_rules_js~~npm~npm__decode-uri-component__0.2.2 (Directory not empty)
	at com.google.devtools.build.lib.unix.NativePosixFiles.remove(Native Method)
	at com.google.devtools.build.lib.unix.UnixFileSystem.delete(UnixFileSystem.java:444)
	at com.google.devtools.build.lib.vfs.FileSystem.deleteTree(FileSystem.java:232)
	at com.google.devtools.build.lib.vfs.Path.deleteTree(Path.java:594)
	at com.google.devtools.build.lib.rules.repository.RepositoryFunction.setupRepoRoot(RepositoryFunction.java:136)
	... 6 more
ERROR: <builtin>: fetching npm_import_rule rule //:aspect_rules_js~~npm~npm__decode-uri-component__0.2.2: java.io.IOException: /private/var/tmp/_bazel_fmeum/0465a0857e26a35c072f48ab751a1794/external/aspect_rules_js~~npm~npm__decode-uri-component__0.2.2 (Directory not empty)
ERROR: no such package '@@aspect_rules_js~~npm~npm__decode-uri-component__0.2.2//': /private/var/tmp/_bazel_fmeum/0465a0857e26a35c072f48ab751a1794/external/aspect_rules_js~~npm~npm__decode-uri-component__0.2.2 (Directory not empty)
ERROR: /Users/fmeum/git/examples/frontend/BUILD.bazel:10:22: //:.aspect_rules_js/node_modules/decode-uri-component@0.2.2/pkg depends on @@aspect_rules_js~~npm~npm__decode-uri-component__0.2.2//:pkg in repository @@aspect_rules_js~~npm~npm__decode-uri-component__0.2.2 which failed to fetch. no such package '@@aspect_rules_js~~npm~npm__decode-uri-component__0.2.2//': /private/var/tmp/_bazel_fmeum/0465a0857e26a35c072f48ab751a1794/external/aspect_rules_js~~npm~npm__decode-uri-component__0.2.2 (Directory not empty)
Use --verbose_failures to see the command lines of failed build steps.
ERROR: Analysis of target '//react/src:test_lib' failed; build aborted: Analysis failed
INFO: Elapsed time: 38.951s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
FAILED:
    Fetching repository @@aspect_rules_swc~~swc~swc_darwin-arm64; fetch interrupted due to memory pressure; restarting. 19s
    Fetching repository @@aspect_rules_js~~npm~npm__buffer-from__1.1.2; fetch interrupted due to memory pressure; restarting. 11s
    Fetching https://registry.npmjs.org/@typescript-eslint/type-utils/-/type-utils-5.44.0.tgz; 11s
    Fetching repository @@aspect_rules_js~~npm~npm__esbuild-windows-64__0.15.18; fetch interrupted due to memory pressure; restarting. 10s
    Fetching repository @@aspect_rules_js~~npm~npm__esbuild-freebsd-arm64__0.15.18; fetch interrupted due to memory pressure; restarting. 8s
    Fetching https://registry.npmjs.org/is-weakset/-/is-weakset-2.0.2.tgz; 7s
    Fetching repository @@aspect_rules_js~~npm~npm__at_next_swc-win32-arm64-msvc__13.0.5; fetch interrupted due to memory pressure; restarting. 6s
    Fetching repository @@aspect_rules_js~~npm~npm__randombytes__2.1.0; fetch interrupted due to memory pressure; restarting. 5s ... (20 fetches)

[meteorcloudy]

@bazel-io fork 7.2.1