remote_download_minimal: Error running tests with a directory dependency

[emfree]

Description of the problem / feature request:

Hello there! It looks like latest bazel fails when running tests:

• with remote execution
• with the flags --nobuild_runfile_links or --remote_download_minimal
• when the test specifies a directory in srcs or data. See the minimal example below.

I reckon this is most likely related to the changes in #9244 plus #9315? +cc @buchgr

That said, the changes in #9244 are very very compelling from a performance perspective!

The stacktrace looks like this.

ERROR: /home/efreeman/bazel_regression/BUILD:1:1:  failed due to unexpected I/O exception: /home/efreeman/.cache/bazel/_bazel_efreeman/74164ebd52c68743881f0bc0b22c6057/execroot/__main__/bazel-out/k8-fastbuild/bin/nop.runfiles/__main__/test/data (No such file or directory)
java.io.FileNotFoundException: /home/efreeman/.cache/bazel/_bazel_efreeman/74164ebd52c68743881f0bc0b22c6057/execroot/__main__/bazel-out/k8-fastbuild/bin/nop.runfiles/__main__/test/data (No such file or directory)
	at com.google.devtools.build.lib.unix.NativePosixFiles.readdir(Native Method)
	at com.google.devtools.build.lib.unix.NativePosixFiles.readdir(NativePosixFiles.java:284)
	at com.google.devtools.build.lib.unix.UnixFileSystem.readdir(UnixFileSystem.java:156)
	at com.google.devtools.build.lib.vfs.Path.readdir(Path.java:395)
	at com.google.devtools.build.lib.remote.merkletree.InputTree.explodeDirectory(InputTree.java:339)
	at com.google.devtools.build.lib.remote.merkletree.InputTree.explodeDirectory(InputTree.java:332)
	at com.google.devtools.build.lib.remote.merkletree.InputTree.build(InputTree.java:304)
	at com.google.devtools.build.lib.remote.merkletree.InputTree.build(InputTree.java:255)
	at com.google.devtools.build.lib.remote.merkletree.MerkleTree.build(MerkleTree.java:94)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:166)
	at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:225)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:122)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:87)
	at com.google.devtools.build.lib.actions.SpawnActionContext.beginExecution(SpawnActionContext.java:41)
	at com.google.devtools.build.lib.exec.ProxySpawnActionContext.beginExecution(ProxySpawnActionContext.java:60)
	at com.google.devtools.build.lib.exec.StandaloneTestStrategy.beginTestAttempt(StandaloneTestStrategy.java:304)
	at com.google.devtools.build.lib.exec.StandaloneTestStrategy.access$100(StandaloneTestStrategy.java:73)
	at com.google.devtools.build.lib.exec.StandaloneTestStrategy$StandaloneTestRunnerSpawn.beginExecution(StandaloneTestStrategy.java:447)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction$1.execute(TestRunnerAction.java:765)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction$RunAttemptsContinuation.execute(TestRunnerAction.java:977)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction.beginExecution(TestRunnerAction.java:771)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:790)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:781)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$4.execute(SkyframeActionExecutor.java:854)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:988)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:960)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:116)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:77)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:579)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:768)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:275)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:454)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:399)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I can reproduce the issue using the following minimal setup:

$ cat BUILD
sh_test(
    name = "nop",
    srcs = ["nop.sh"],     # A trivial shell script that just exits 0
    data = ["test/data"],  # A directory
)

(Note that if I replace the data dependency with data = glob(["test/data/**/*"]) no error is thrown.)

$ cat WORKSPACE   # This just has the boilerplate for remote execution
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
  name = "bazel_toolchains",
  urls = [
    "https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/archive/0.29.3.tar.gz",
    "https://github.com/bazelbuild/bazel-toolchains/archive/0.29.3.tar.gz",
  ],
  strip_prefix = "bazel-toolchains-0.29.3",
  sha256 = "1411f2648185b0e7d8c2bb88b25cc8f2c477cc4223133461652ddce2b3154ac4",
)

load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig")
rbe_autoconfig(name = "rbe_default")

$ cat .bazelrc
build --remote_cache=grpcs://remotebuildexecution.googleapis.com
build --remote_executor=grpcs://remotebuildexecution.googleapis.com
build --remote_instance_name=projects/[fill in a project here]
build --google_default_credentials=true
build --strategy=TestRunner=remote
build --platforms=@rbe_default//config:platform
build --host_platform=@rbe_default//config:platform
build --nocache_test_results
build --remote_download_minimal

What operating system are you running Bazel on?

Ubuntu 18.04

What's the output of bazel info release?

development version

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

Latest master, built from source.

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/bazelbuild/bazel
54ec37db189c839f8d2b5b9626f7dd78b3add0d8
54ec37db189c839f8d2b5b9626f7dd78b3add0d8

Have you found anything relevant by searching the web?

I have not.

[sgowroji]

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.