Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bazel build fails with remote exec errors #196

Closed
rahul-nitkkr opened this issue Nov 21, 2018 · 2 comments
Closed

Bazel build fails with remote exec errors #196

rahul-nitkkr opened this issue Nov 21, 2018 · 2 comments

Comments

@rahul-nitkkr
Copy link

rahul-nitkkr commented Nov 21, 2018 •
edited

carrying over form the slack channel
i am running into weird grpc errors on the buildfarm server while usign the smae to remote exec with bazel.
All the actions sent to the server error out while running GetActionResult
Logs from buildfarm server


07:19:29.943 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND DATA: streamId=19 padding=0 endStream=true length=51 bytes=000000002e122c0a2861326466643638346165383464623230653133336433356631656635646439393134373932306237105e
07:19:29.943 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] OUTBOUND RST_STREAM: streamId=19 errorCode=8
07:19:29.944 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] OUTBOUND HEADERS: streamId=19 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-status: 13, grpc-message: Half-closed without a request] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true
07:19:29.944 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND RST_STREAM: streamId=19 errorCode=5
07:19:29.946 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND RST_STREAM: streamId=19 errorCode=8
07:19:30.319 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND HEADERS: streamId=21 headers=GrpcHttp2RequestHeaders[:path: /build.bazel.remote.execution.v2.ActionCache/GetActionResult, :authority: localhost:8980, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.10.0, build.bazel.remote.execution.v2.requestmetadata-bin: ChsKBWJhemVsEhIwLjE5LjAtIChAbm9uLWdpdCkSKGEyZGZkNjg0YWU4NGRiMjBlMTMzZDM1ZjFlZjVkZDk5MTQ3OTIwYjcaJGU0ZDBiOTRlLTY4MTktNDJlMS04YzRlLTIwMGZmNjhiNmY4ZSIkYTJkNGYxYzgtZjNiYy00ZWM3LTk0YzItNjQ0NzNmYmM5NTlk, grpc-accept-encoding: gzip, grpc-timeout: 59999851u, grpc-trace-bin: ] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
07:19:30.319 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND DATA: streamId=21 padding=0 endStream=true length=51 bytes=000000002e122c0a2861326466643638346165383464623230653133336433356631656635646439393134373932306237105e
07:19:30.320 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] OUTBOUND RST_STREAM: streamId=21 errorCode=8
07:19:30.320 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] OUTBOUND HEADERS: streamId=21 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-status: 13, grpc-message: Half-closed without a request] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true
07:19:30.321 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND RST_STREAM: streamId=21 errorCode=5
07:19:30.324 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND RST_STREAM: streamId=21 errorCode=8
07:19:31.109 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND HEADERS: streamId=23 headers=GrpcHttp2RequestHeaders[:path: /build.bazel.remote.execution.v2.ActionCache/GetActionResult, :authority: localhost:8980, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.10.0, build.bazel.remote.execution.v2.requestmetadata-bin: ChsKBWJhemVsEhIwLjE5LjAtIChAbm9uLWdpdCkSKGEyZGZkNjg0YWU4NGRiMjBlMTMzZDM1ZjFlZjVkZDk5MTQ3OTIwYjcaJGU0ZDBiOTRlLTY4MTktNDJlMS04YzRlLTIwMGZmNjhiNmY4ZSIkYTJkNGYxYzgtZjNiYy00ZWM3LTk0YzItNjQ0NzNmYmM5NTlk, grpc-accept-encoding: gzip, grpc-timeout: 59999849u, grpc-trace-bin: ] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
07:19:31.110 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND DATA: streamId=23 padding=0 endStream=true length=51 bytes=000000002e122c0a2861326466643638346165383464623230653133336433356631656635646439393134373932306237105e
07:19:31.111 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] OUTBOUND RST_STREAM: streamId=23 errorCode=8
07:19:31.111 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] OUTBOUND HEADERS: streamId=23 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-status: 13, grpc-message: Half-closed without a request] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true
07:19:31.112 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND RST_STREAM: streamId=23 errorCode=5
07:19:31.113 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND RST_STREAM: streamId=23 errorCode=8
07:19:32.764 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND HEADERS: streamId=25 headers=GrpcHttp2RequestHeaders[:path: /build.bazel.remote.execution.v2.ActionCache/GetActionResult, :authority: localhost:8980, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.10.0, build.bazel.remote.execution.v2.requestmetadata-bin: ChsKBWJhemVsEhIwLjE5LjAtIChAbm9uLWdpdCkSKGEyZGZkNjg0YWU4NGRiMjBlMTMzZDM1ZjFlZjVkZDk5MTQ3OTIwYjcaJGU0ZDBiOTRlLTY4MTktNDJlMS04YzRlLTIwMGZmNjhiNmY4ZSIkYTJkNGYxYzgtZjNiYy00ZWM3LTk0YzItNjQ0NzNmYmM5NTlk, grpc-accept-encoding: gzip, grpc-timeout: 59999842u, grpc-trace-bin: ] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
07:19:32.765 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] INBOUND DATA: streamId=25 padding=0 endStream=true length=51 bytes=000000002e122c0a2861326466643638346165383464623230653133336433356631656635646439393134373932306237105e
07:19:32.766 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] OUTBOUND RST_STREAM: streamId=25 errorCode=8
07:19:32.766 [grpc-default-worker-ELG-1-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc6f64687, L:/127.0.0.1:8980 - R:/127.0.0.1:49953] OUTBOUND HEADERS: streamId=25 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-status: 13, grpc-message: Half-closed without a request] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true

Logs from bazel client with verbose_logging

SUBCOMMAND: # //my-annotations:my_my_annotations_main [action 'Building my-annotations/libmy_my_annotations_main.jar ()']
(cd /local/data/scratch/.cache/bazel/_bazel_p2dpgnp/b6669dedfcd2eab0c289aa50224e42cf/execroot/my_root && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
  external/embedded_jdk/bin/java -XX:+TieredCompilation '-XX:TieredStopAtLevel=1' -Xbootclasspath/p:/sw/external/jdk-1.8.0_60_x64/jre/lib/tools.jar -jar external/bazel_tools/tools/jdk/VanillaJavaBuilder_deploy.jar @bazel-out/k8-fastbuild/bin/my-annotations/libmy_my_annotations_main.jar-0.params)
ERROR: /local/data/scratch/royrah2/bazel10/my-annotations/BUILD:43:1: Couldn't build file my-annotations/libmy_my_annotations_main.jar: Building my-annotations/libmy_my_annotations_main.jar () failed (Exit 34). Note: Remote connection/protocol failed with: execution failed Call failed after 5 retry attempts: io.grpc.StatusRuntimeException: CANCELLED: HTTP/2 error code: CANCEL
Received Rst Stream: com.google.devtools.build.lib.remote.Retrier$RetryException: Call failed after 5 retry attempts: io.grpc.StatusRuntimeException: CANCELLED: HTTP/2 error code: CANCEL
Received Rst Stream
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:290)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:121)
        at com.google.devtools.build.lib.remote.GrpcRemoteCache.getCachedActionResult(GrpcRemoteCache.java:373)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:174)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:106)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
        at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:288)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:295)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:1001)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:930)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:770)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:725)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:478)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:519)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:216)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:422)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.grpc.StatusRuntimeException: CANCELLED: HTTP/2 error code: CANCEL
Received Rst Stream
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:221)
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:202)
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:131)
        at build.bazel.remote.execution.v2.ActionCacheGrpc$ActionCacheBlockingStub.getActionResult(ActionCacheGrpc.java:347)
        at com.google.devtools.build.lib.remote.GrpcRemoteCache.lambda$getCachedActionResult$2(GrpcRemoteCache.java:376)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:268)
        ... 22 more
SUBCOMMAND: # //my-annotations:my_my_annotations_java_main [action 'Building my-annotations/libmy_my_annotations_java_main.jar (8 source files)']
(cd /local/data/scratch/.cache/bazel/_bazel_p2dpgnp/b6669dedfcd2eab0c289aa50224e42cf/execroot/my_root && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
  external/embedded_jdk/bin/java -XX:+TieredCompilation '-XX:TieredStopAtLevel=1' -Xbootclasspath/p:/sw/external/jdk-1.8.0_60_x64/jre/lib/tools.jar -jar external/bazel_tools/tools/jdk/VanillaJavaBuilder_deploy.jar @bazel-out/k8-fastbuild/bin/my-annotations/libmy_my_annotations_java_main.jar-0.params)
ERROR: /local/data/scratch/royrah2/bazel10/my-annotations/BUILD:15:1: Couldn't build file my-annotations/libmy_my_annotations_java_main.jar: Building my-annotations/libmy_my_annotations_java_main.jar (8 source files) failed (Exit 34). Note: Remote connection/protocol failed with: execution failed Call failed after 5 retry attempts: io.grpc.StatusRuntimeException: INTERNAL: Half-closed without a request: com.google.devtools.build.lib.remote.Retrier$RetryException: Call failed after 5 retry attempts: io.grpc.StatusRuntimeException: INTERNAL: Half-closed without a request
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:290)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:121)
        at com.google.devtools.build.lib.remote.GrpcRemoteCache.getCachedActionResult(GrpcRemoteCache.java:373)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:174)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:106)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
        at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:288)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:295)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:1001)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:930)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:770)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:725)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:478)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:519)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:216)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:422)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.grpc.StatusRuntimeException: INTERNAL: Half-closed without a request
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:221)
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:202)
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:131)
        at build.bazel.remote.execution.v2.ActionCacheGrpc$ActionCacheBlockingStub.getActionResult(ActionCacheGrpc.java:347)
        at com.google.devtools.build.lib.remote.GrpcRemoteCache.lambda$getCachedActionResult$2(GrpcRemoteCache.java:376)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:268)
        ... 22 more
Target //my-annotations:my_my_annotations_main failed to build
INFO: Elapsed time: 7.022s, Critical Path: 3.24s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
INFO: 0 processes.
FAILED: Build did NOT complete successfully

this is my bazelrc in case:

startup --host_jvm_args=-Dbazel.DigestFunction=SHA1 --host_jvm_args=-Djsi.branch='~ROYRAH!BAZELFS_CLEAN;SOURCE' --host_jvm_args=-Dworkspace.dir='/local/data/scratch/royrah2/bazel10/'  --host_jvm_args=-Dfs.enable.jsi='true'
build --nojava_header_compilation --jobs=1 --profile=/local/data/scratch/foo.profile --keep_going --java_toolchain=:custom_jdk --spawn_strategy=remote --genrule_strategy=remote --strategy=Javac=remote --strategy=Closure=remote --remote_executor=localhost:8980 --remote_cache=localhost:8980 --verbose_failures  --auth_enabled=false --tls_enabled=false
@werkt
Copy link
Collaborator

werkt commented Nov 23, 2018

You know that you can split your build lines in bazelrc out into multiple lines, right? (just prefix each line with build )

Specifying --remote_cache is unnecessary with --remote_executor, and may have some deleterious effects.

Have you tried this out with an unmodified bazel client? I would have expected some more interesting logging from the server side if it was making it into the actual getActionResult handler.

@werkt
Copy link
Collaborator

werkt commented Dec 26, 2018

Any update on this? Will close soon assuming this is no longer an issue

@werkt werkt closed this as completed Mar 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@werkt @rahul-nitkkr