IllegalStateException: Did not write as much data as expected #6136

drigz · 2018-09-12T13:15:13Z

We've seen the following failure cause 2 flakes on our CI server (out of around 30 builds since upgrading from Bazel 0.15.0 to Bazel 0.16.1).

Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false, actionIndex=0}' (requested by nodes 'external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/_ijar/jar/external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/jackson-dataformat-yaml-2.9.0-ijar.jar @com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false')
INFO: Elapsed time: 178.732s, Critical Path: 6.46s
INFO: 186 processes: 186 remote cache hit.
FAILED: Build did NOT complete successfully
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false, actionIndex=0}' (requested by nodes 'external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/_ijar/jar/external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/jackson-dataformat-yaml-2.9.0-ijar.jar @com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
	at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:973)
	at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
	at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
	at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:90)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:63)
	at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:287)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:294)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:978)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:910)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$900(SkyframeActionExecutor.java:120)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:457)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:513)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:227)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:400)
	... 4 more

I haven't seen it occur on my workstation - only on the CI build.

What operating system are you running Bazel on?

Ubuntu 14.04 host, running Bazel inside a docker container based on debian:jessie-backports.

What's the output of `bazel info release`?

0.16.1

Have you found anything relevant by searching the web?

bazelbuild/rules_scala#589 seems to be the same issue.

The text was updated successfully, but these errors were encountered:

benjaminp · 2018-09-28T06:14:53Z

This is a rather disturbing error that looks like either an internal protobuf bug or a HotSpot bug. I'm leaning towards that latter, since it appeared with the embedded Java 9 upgrade. --host_jvm_args=-XX:-InlineUnsafeOps seems to make the problem go away.

jgavris · 2018-09-28T15:18:24Z

Sorry to chime in, but we've been seeing this too. I suspect it could also be that the cache is poisoned and somehow a partial object was written to the cache? And when it is read back, this assertion is thrown.

drigz · 2018-09-30T10:33:26Z

If it helps any others: I applied the workaround by adding this like to .bazelrc:

startup --host_jvm_args=-XX:+UnlockDiagnosticVMOptions --host_jvm_args=-XX:-InlineUnsafeOps

ensonic · 2018-10-19T09:50:24Z

The workaround seems to not apply everywhere:

Starting local Bazel server and connecting to it...
.INFO: Reading 'startup' options from /path/to/.bazelrc: --host_jvm_args=-XX:+UnlockDiagnosticVMOptions --host_jvm_args=-XX:-InlineUnsafeOps
Server crashed during startup. Now printing /home/kbuilder/.cache/bazel/_bazel_kbuilder/7b053e52cb9de0c352841a1b58c652ee/server/jvm.out
Error: VM option 'InlineUnsafeOps' is develop and is available only in debug version of VM.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

ensonic · 2018-10-19T12:49:03Z

Sorry I was too quick. This particular build was still using bazel 0.15.0.

ixdy · 2018-10-25T18:06:01Z

We just ran into this on our CI with bazel 0.18.0.

https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/70245/pull-kubernetes-e2e-gce-device-plugin-gpu/51169/

W1025 17:56:49.640] Target //:push-build failed to build
W1025 17:56:49.642] Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//vendor/github.com/mindprince/gonvml:go_default_library%linux_amd64%cgo_c_lib BuildConfigurationValue.Key[3190fc7906a0bb10bcf73823d08c1469] false, actionIndex=4}' (requested by nodes 'File:[[<execution_root>]bazel-out/k8-fastbuild/bin]vendor/github.com/mindprince/gonvml/libgo_default_library%linux_amd64%cgo_c_lib.lo')
W1025 17:56:49.642] 	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:495)
W1025 17:56:49.643] 	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
W1025 17:56:49.643] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
W1025 17:56:49.643] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
W1025 17:56:49.643] 	at java.base/java.lang.Thread.run(Unknown Source)
W1025 17:56:49.643] Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
W1025 17:56:49.643] 	at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:977)
W1025 17:56:49.644] 	at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
W1025 17:56:49.644] 	at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
W1025 17:56:49.644] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
W1025 17:56:49.644] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
W1025 17:56:49.644] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.644] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645] 	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
W1025 17:56:49.646] 	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:101)
W1025 17:56:49.646] 	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
W1025 17:56:49.646] 	at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
W1025 17:56:49.646] 	at com.google.devtools.build.lib.rules.cpp.CppLinkAction.execute(CppLinkAction.java:315)
W1025 17:56:49.646] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:994)
W1025 17:56:49.646] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:923)
W1025 17:56:49.647] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
W1025 17:56:49.647] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
W1025 17:56:49.647] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
W1025 17:56:49.647] 	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
W1025 17:56:49.647] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:471)
W1025 17:56:49.648] 	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:505)
W1025 17:56:49.648] 	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:215)
W1025 17:56:49.648] 	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:418)
W1025 17:56:49.648] 	... 4 more
W1025 17:56:49.648] 
W1025 17:56:49.652] INFO: Elapsed time: 77.314s, Critical Path: 29.61s
W1025 17:56:49.654] INFO: 362 processes: 355 remote cache hit, 7 linux-sandbox.
W1025 17:56:49.659] FAILED: Build did NOT complete successfully
W1025 17:56:49.663] java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//vendor/github.com/mindprince/gonvml:go_default_library%linux_amd64%cgo_c_lib BuildConfigurationValue.Key[3190fc7906a0bb10bcf73823d08c1469] false, actionIndex=4}' (requested by nodes 'File:[[<execution_root>]bazel-out/k8-fastbuild/bin]vendor/github.com/mindprince/gonvml/libgo_default_library%linux_amd64%cgo_c_lib.lo')
W1025 17:56:49.666] 	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:495)
W1025 17:56:49.666] 	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
W1025 17:56:49.666] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
W1025 17:56:49.667] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
W1025 17:56:49.667] 	at java.base/java.lang.Thread.run(Unknown Source)
W1025 17:56:49.667] Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
W1025 17:56:49.667] 	at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:977)
W1025 17:56:49.668] 	at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
W1025 17:56:49.668] 	at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
W1025 17:56:49.668] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
W1025 17:56:49.668] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
W1025 17:56:49.668] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.669] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.669] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.669] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.669] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.670] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.670] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.670] 	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
W1025 17:56:49.670] 	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:101)
W1025 17:56:49.670] 	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
W1025 17:56:49.671] 	at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
W1025 17:56:49.671] 	at com.google.devtools.build.lib.rules.cpp.CppLinkAction.execute(CppLinkAction.java:315)
W1025 17:56:49.671] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:994)
W1025 17:56:49.671] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:923)
W1025 17:56:49.672] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
W1025 17:56:49.672] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
W1025 17:56:49.672] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
W1025 17:56:49.672] 	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
W1025 17:56:49.672] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:471)
W1025 17:56:49.673] 	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:505)
W1025 17:56:49.673] 	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:215)
W1025 17:56:49.673] 	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:418)
W1025 17:56:49.673] 	... 4 more
W1025 17:56:49.674] java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//vendor/github.com/mindprince/gonvml:go_default_library%linux_amd64%cgo_c_lib BuildConfigurationValue.Key[3190fc7906a0bb10bcf73823d08c1469] false, actionIndex=4}' (requested by nodes 'File:[[<execution_root>]bazel-out/k8-fastbuild/bin]vendor/github.com/mindprince/gonvml/libgo_default_library%linux_amd64%cgo_c_lib.lo')
W1025 17:56:49.674] 	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:495)
W1025 17:56:49.674] 	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
W1025 17:56:49.674] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
W1025 17:56:49.675] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
W1025 17:56:49.675] 	at java.base/java.lang.Thread.run(Unknown Source)
W1025 17:56:49.675] Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
W1025 17:56:49.675] 	at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:977)
W1025 17:56:49.675] 	at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
W1025 17:56:49.676] 	at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
W1025 17:56:49.676] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
W1025 17:56:49.676] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
W1025 17:56:49.676] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.677] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.677] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.677] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.677] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.678] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.678] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.678] 	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
W1025 17:56:49.678] 	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:101)
W1025 17:56:49.679] 	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
W1025 17:56:49.679] 	at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
W1025 17:56:49.679] 	at com.google.devtools.build.lib.rules.cpp.CppLinkAction.execute(CppLinkAction.java:315)
W1025 17:56:49.679] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:994)
W1025 17:56:49.680] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:923)
W1025 17:56:49.680] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
W1025 17:56:49.680] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
W1025 17:56:49.680] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
W1025 17:56:49.680] 	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
W1025 17:56:49.681] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:471)
W1025 17:56:49.681] 	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:505)
W1025 17:56:49.681] 	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:215)
W1025 17:56:49.681] 	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:418)
W1025 17:56:49.682] 	... 4 more
W1025 17:56:49.682] Remote logging disabled for testing, forcing abrupt shutdown.
W1025 17:56:49.683] com.google.devtools.build.lib.util.LoggingUtil#logToRemote: bazel crashed with args: run --startup_time=10 --restart_reason=no_daemon --binary_path=/usr/local/bin/bazel --rc_source=client --default_override=0:common=--isatty=0 --default_override=0:common=--terminal_columns=80 --rc_source=/etc/bazel.bazelrc --rc_source=/go/src/k8s.io/kubernetes/build/root/.bazelrc --default_override=1:build=--experimental_remote_spawn_cache --default_override=1:build=--remote_local_fallback --default_override=1:build=--remote_http_cache=http://bazel-cache.default.svc.cluster.local.:8080/kubernetes/kubernetes,de0d6280eabe96c208a99ad636f792c2 --default_override=2:test:integration=--local_test_jobs --default_override=2:test:integration=4 --default_override=2:test:integration=--test_tag_filters=integration --default_override=2:test:unit=--features=race --default_override=2:test:unit=--test_tag_filters=-e2e,-integration --default_override=2:test:unit=--flaky_test_attempts=3 --default_override=2:test=--test_output=errors --default_override=2:build:unit=--features=race --default_override=2:build=--verbose_failures --default_override=2:build=--workspace_status_command --default_override=2:build=hack/print-workspace-status.sh --default_override=2:build=--sandbox_tmpfs_path=/tmp --default_override=2:build=--sandbox_fake_username --client_cwd=/go/src/k8s.io/kubernetes //:push-build /go/src/k8s.io/kubernetes/_output/gcs-stage/v1.13.0-alpha.2.65+4992a5afe427fe gs://kubernetes-release-pull/ci/pull-kubernetes-e2e-gce-device-plugin-gpu/v1.13.0-alpha.2.65+4992a5afe427fe
W1025 17:56:49.684] java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//vendor/github.com/mindprince/gonvml:go_default_library%linux_amd64%cgo_c_lib BuildConfigurationValue.Key[3190fc7906a0bb10bcf73823d08c1469] false, actionIndex=4}' (requested by nodes 'File:[[<execution_root>]bazel-out/k8-fastbuild/bin]vendor/github.com/mindprince/gonvml/libgo_default_library%linux_amd64%cgo_c_lib.lo')
W1025 17:56:49.684] 	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:495)
W1025 17:56:49.684] 	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
W1025 17:56:49.684] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
W1025 17:56:49.684] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
W1025 17:56:49.685] 	at java.base/java.lang.Thread.run(Unknown Source)
W1025 17:56:49.685] Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
W1025 17:56:49.685] 	at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:977)
W1025 17:56:49.685] 	at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
W1025 17:56:49.685] 	at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
W1025 17:56:49.686] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
W1025 17:56:49.686] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
W1025 17:56:49.686] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.686] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687] 	at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687] 	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
W1025 17:56:49.688] 	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:101)
W1025 17:56:49.688] 	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
W1025 17:56:49.688] 	at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
W1025 17:56:49.688] 	at com.google.devtools.build.lib.rules.cpp.CppLinkAction.execute(CppLinkAction.java:315)
W1025 17:56:49.688] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:994)
W1025 17:56:49.688] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:923)
W1025 17:56:49.688] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
W1025 17:56:49.688] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
W1025 17:56:49.689] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
W1025 17:56:49.689] 	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
W1025 17:56:49.689] 	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:471)
W1025 17:56:49.689] 	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:505)
W1025 17:56:49.689] 	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:215)
W1025 17:56:49.689] 	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:418)
W1025 17:56:49.689] 	... 4 more
W1025 17:56:50.300] 
W1025 17:56:50.300] Server terminated abruptly (error code: 14, error message: '', log file: '/bazel-scratch/.cache/bazel/_bazel_prow/48d5366022b4e3197674c8d6e2bee219/server/jvm.out')

sdqali · 2018-11-16T18:39:50Z

For what it is worth, we are seeing recurrence of this error on our CI builds as well, with version 0.18.10.

burkpojken · 2018-12-19T06:00:50Z

We got the same error in our CI builds using bazel 0.20.0

keith · 2019-02-14T22:03:29Z

We have seen this on macOS CI with 0.22.0

benjaminp · 2019-02-14T22:26:36Z

Presumably this will be resolved when 8db50b6 is released with 0.23.

…

On Thu, Feb 14, 2019, at 14:03, Keith Smiley wrote: We have seen this on macOS CI with 0.22.0 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6136 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANZTn4Evt8zMybc0gcBRdbnUsiW41taks5vNd1FgaJpZM4WlW9y>.

jgavris · 2019-02-14T23:06:12Z

We haven't seen it on 0.22 since applying the recommended workaround here #6136 (comment)

mariusgrigoriu · 2019-03-09T02:01:21Z

Seeing in 0.20.0 in Linux pipeline.

ensonic · 2019-05-23T12:36:56Z

Unfortunately the workaround in
#6136 (comment)
breaks the build on macos x.

jgavris · 2019-05-23T12:46:23Z

@ensonic what version of bazel are you using? We haven't seen it on 0.22 and later on macOS.

mariusgrigoriu · 2020-11-18T19:27:34Z

I wonder if this can be closed. Haven't seen this in ages.

drigz · 2020-11-19T08:28:36Z

Seems reasonable. I'll remove the workaround from our CI build and reopen if it recurs with a recent Bazel version.

drigz mentioned this issue Sep 12, 2018

CI failure on osx reproducibility check bazelbuild/rules_scala#589

Open

iirina added team-Execution untriaged labels Sep 25, 2018

philwo assigned buchgr Oct 17, 2018

philwo added type: bug P2 We'll consider working on this in future. (Assignee optional) category: remote execution / caching and removed untriaged labels Oct 17, 2018

ixdy mentioned this issue Oct 25, 2018

Use debian-base instead of busybox as base image for server images kubernetes/kubernetes#70245

Merged

jin added team-Local-Exec Issues and PRs for the Execution (Local) team and removed team-Execution labels Jan 14, 2019

buchgr added team-Remote-Exec Issues and PRs for the Execution (Remote) team and removed category: remote execution / caching team-Local-Exec Issues and PRs for the Execution (Local) team labels Jan 19, 2019

keith mentioned this issue Feb 15, 2019

Unrecoverable error while evaluating node 'ActionLookupData{...}' #6170

Closed

ishikhman added this to To do in remote: bugs Jul 12, 2019

buchgr removed their assignment Jan 9, 2020

drigz closed this as completed Nov 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IllegalStateException: Did not write as much data as expected #6136

IllegalStateException: Did not write as much data as expected #6136

drigz commented Sep 12, 2018

benjaminp commented Sep 28, 2018

jgavris commented Sep 28, 2018

drigz commented Sep 30, 2018

ensonic commented Oct 19, 2018 •

edited

ensonic commented Oct 19, 2018

ixdy commented Oct 25, 2018

sdqali commented Nov 16, 2018

burkpojken commented Dec 19, 2018

keith commented Feb 14, 2019

benjaminp commented Feb 14, 2019 via email

jgavris commented Feb 14, 2019

mariusgrigoriu commented Mar 9, 2019

ensonic commented May 23, 2019

jgavris commented May 23, 2019

mariusgrigoriu commented Nov 18, 2020

drigz commented Nov 19, 2020

IllegalStateException: Did not write as much data as expected #6136

IllegalStateException: Did not write as much data as expected #6136

Comments

drigz commented Sep 12, 2018

What operating system are you running Bazel on?

What's the output of bazel info release?

Have you found anything relevant by searching the web?

benjaminp commented Sep 28, 2018

jgavris commented Sep 28, 2018

drigz commented Sep 30, 2018

ensonic commented Oct 19, 2018 • edited

ensonic commented Oct 19, 2018

ixdy commented Oct 25, 2018

sdqali commented Nov 16, 2018

burkpojken commented Dec 19, 2018

keith commented Feb 14, 2019

benjaminp commented Feb 14, 2019 via email

jgavris commented Feb 14, 2019

mariusgrigoriu commented Mar 9, 2019

ensonic commented May 23, 2019

jgavris commented May 23, 2019

mariusgrigoriu commented Nov 18, 2020

drigz commented Nov 19, 2020

What's the output of `bazel info release`?

ensonic commented Oct 19, 2018 •

edited