Unrecoverable error while evaluating node 'ActionLookupData3{...}'

[gottagofaster236]

Description of the bug:

I'm not sure I've configured everything correctly, but I'm pretty sure an "unrecoverable error" shouldn't be occurring anyways. The rule in question is an "android_local_test". See the next section for instructions on reproduction.

bazel build @@//:GetProvisioningModeActivityTest
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:GetProvisioningModeActivityTest (123 packages loaded, 2084 targets configured).
[1 / 1] checking cached actions
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData3{actionLookupKey=ConfiguredTargetKey{label=//:GetProvisioningModeActivityTest, config=BuildConfigurationKey[b2c79ba563c5c430c2d80869a5b38c4137f7418fd255f18fa6221441ea694164]}, actionIndex=3}' (requested by nodes 'ArtifactNestedSetKey[2]@1114439354')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException: Cannot invoke "String.length()" because "replacement" is null
        at com.google.devtools.build.lib.util.StringUtilities.replaceAllLiteral(StringUtilities.java:70)
        at com.google.devtools.build.lib.analysis.actions.LocalTemplateExpansionStrategy.getExpandedTemplateUnsafe(LocalTemplateExpansionStrategy.java:80)
        at com.google.devtools.build.lib.analysis.actions.LocalTemplateExpansionStrategy.expandTemplate(LocalTemplateExpansionStrategy.java:48)
        at com.google.devtools.build.lib.analysis.actions.TemplateExpansionAction.execute(TemplateExpansionAction.java:140)
        at com.google.devtools.build.lib.analysis.actions.TemplateExpansionAction.execute(TemplateExpansionAction.java:160)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1144)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1061)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:165)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:94)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:558)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:859)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:333)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:171)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
        ... 7 more

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

• Clone https://github.com/gottagofaster236/android-testdpc
• Run bazel build @@//:GetProvisioningModeActivityTest

Which operating system are you running Bazel on?

macOS

What is the output of bazel info release?

release 7.0.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

https://github.com/gottagofaster236/android-testdpc
84c37ea28d23b5cf69882e674126993bf74eb770

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

#6170
#5296

Any other information, logs, or outputs that you want to share?

No response

[derekperkins]

Similar error trying to upgrade from v6.5.0 to v7.1.0

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'UnshareableActionLookupData{actionLookupKey=ConfiguredTargetKey{label=//pkg/hubspot:go_default_test, config=BuildConfigurationKey[107978ce4da7f89b23c7ed2d5a8e7aa1fe7225cbd1812a31c7e3b2c5b88f8a03]}, actionIndex=9}' (requested by nodes 'TestCompletionKey{configuredTargetKey=ConfiguredTargetKey{label=//pkg/hubspot:go_default_test, config=BuildConfigurationKey[107978ce4da7f89b23c7ed2d5a8e7aa1fe7225cbd1812a31c7e3b2c5b88f8a03]}, topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@90904c3b, exclusiveTesting=false}')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:903)
	at com.google.devtools.build.lib.vfs.Path.getRelative(Path.java:122)
	at com.google.devtools.build.lib.sandbox.SandboxStash.takeStashedSandboxInternal(SandboxStash.java:99)
	at com.google.devtools.build.lib.sandbox.SandboxStash.takeStashedSandbox(SandboxStash.java:73)
	at com.google.devtools.build.lib.sandbox.SymlinkedSandboxedSpawn.filterInputsAndDirsToCreate(SymlinkedSandboxedSpawn.java:79)
	at com.google.devtools.build.lib.sandbox.AbstractContainerizingSandboxedSpawn.createFileSystem(AbstractContainerizingSandboxedSpawn.java:135)
	at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.runSpawn(AbstractSandboxSpawnRunner.java:146)
	at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.exec(AbstractSandboxSpawnRunner.java:113)
	at com.google.devtools.build.lib.sandbox.SandboxModule$SandboxFallbackSpawnRunner.exec(SandboxModule.java:456)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:159)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:119)
	at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:45)
	at com.google.devtools.build.lib.exec.StandaloneTestStrategy.runTestAttempt(StandaloneTestStrategy.java:656)
	at com.google.devtools.build.lib.exec.StandaloneTestStrategy.beginTestAttempt(StandaloneTestStrategy.java:315)
	at com.google.devtools.build.lib.exec.StandaloneTestStrategy$StandaloneTestRunnerSpawn.execute(StandaloneTestStrategy.java:581)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction.executeAllAttempts(TestRunnerAction.java:1163)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:975)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:952)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1144)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1061)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:165)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:94)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:558)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:859)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:333)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:171)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
	... 7 more

[fmeum]

The second stack trace is a different and known bug. @oquenchil I lost our chat history about this and failed to find the issue duplicated by #21632 (comment), do you remember whether we fixed it?

[fmeum]

Found bazelbuild/bazel-skylib#488.

@sgowroji Have you seen any similar failures recently? I believe this is flaky behavior and we probably still lack the actual fix.

[fmeum]

@bazel-io fork 7.1.1

[ahumesky]

I was able to reproduce this issue on 7.0.2 and 6.5.0. I was trying to go back to a version of bazel where this worked, and ran into Java strict deps issues with robolectric annotation and robolectric shadow deps, and java package paths issues in 5.0.0. I fixed those and got it to work in bazel 5.0.0. Then I tried going back to 6.5.0 and 7.0.2 and surprisingly, this also seemed to fix the problem in those versions:

~/android-testdpc-gottagofaster236$ git diff
diff --git a/BUILD b/BUILD
index 2a9818d..e819dfd 100644
--- a/BUILD
+++ b/BUILD
@@ -69,6 +69,8 @@ android_library(
     name = "test_deps",
     exports = [
         "@maven//:org_robolectric_robolectric",
+        "@maven//:org_robolectric_shadows_framework",
+        "@maven//:org_robolectric_annotations",
         "@maven//:com_google_truth_truth",
         "@robolectric//bazel:android-all",
         "@maven//:androidx_test_core",
@@ -158,6 +160,7 @@ android_local_test(
         ":testdpc_lib",
         ":test_deps"
     ],
+    custom_package = "com.afwsamples.testdpc.provision",
 )
 
 java_test(
@@ -323,4 +326,4 @@ java_test(
         ":testdpc_lib",
         ":test_deps"
     ],
-)
\ No newline at end of file
+)
diff --git a/WORKSPACE b/WORKSPACE
index 4538530..29e8a65 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -47,6 +47,8 @@ maven_install(
         "org.bouncycastle:bcpkix-jdk15on:1.70",
         "org.bouncycastle:bcprov-jdk15on:1.70",
         "org.robolectric:robolectric:4.7.3",
+        "org.robolectric:shadows-framework:4.7.3",
+        "org.robolectric:annotations:4.7.3",
         "com.google.truth:truth:1.4.2",
         "androidx.test:core:1.5.0",
         "com.google.testparameterinjector:test-parameter-injector:1.15"

I'm not sure yet why these missing dependencies would result in null values being passed to TemplateExpansionAction. @gottagofaster236 @derekperkins could you see if adding the depndencies and custom_package as above avoid the problem here?

[ahumesky]

And, for what it's worth, the test passes with bazel test

[ahumesky]

After some debugging, the problem appears to be here:

bazel/src/main/java/com/google/devtools/build/lib/rules/android/AndroidLocalTestBase.java

Lines 165 to 167 in ad21130

	substitutions.add(
	Substitution.of(
	"%android_custom_package%", resourceApk.getValidatedResources().getJavaPackage()));

And setting (just) custom_package (in bazel 7.0.2) does avoid the issue (and then reveals the separate strict deps issue)

Normally, and as in bazel 5.0.0, if custom_package is needed, the android rules would print:

ERROR: /usr/local/google/home/ahumesky/android-testdpc-gottagofaster236/BUILD:153:19: in android_local_test rule //:GetProvisioningModeActivityTest: The location of your BUILD file determines the Java package used for Android resource processing. A directory named "java" or "javatests" will be used as your Java source root and the path of your BUILD file relative to the Java source root will be used as the package for Android resource processing. The Java source root could not be determined for "". Move your BUILD file under a java or javatests directory, or set the 'custom_package' attribute.

So something must have changed / broken between bazel 5 and bazel 6 with how custom_package is handled.

[ahumesky]

Looks like the change that caused this is 18e8ce7

I'm surprised it took over 2 years for this to be noticed. Since this is not a regression in bazel 7 and there's a straight forward workaround, we don't need to block 7.1.1 on this

[gottagofaster236]

@gottagofaster236 could you see if adding the depndencies and custom_package as above avoid the problem here?

Yep, the diff you provided fixes it, thanks a lot! I would've figured it out probably if not for the cryptic error message

[ahumesky]

Great, I have a change in review that will avoid the NullPointerException and the cryptic error message