Skip to content

BlockingStub call hangs indefinitely in ClientCalls$ThreadlessExecutor.waitAndDrain after server completion #12456

@halibobo1205

Description

@halibobo1205

What version of gRPC-Java are you using?

gRPC-Java 1.75

What is your environment?

Linux arm64,JDK 17.0.15

What did you expect to see?

The blocking stub should return once the server sends a grpc-status: 0 completion, or throw a timeout exception if the deadline expires — it should never hang indefinitely.

What did you see instead?

When using a blocking stub with a timeout interceptor, the client occasionally hangs forever in:

io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:817)

even though the server has already completed the RPC successfully (grpc-status: 0).
The thread remains parked indefinitely, and the application never proceeds.

This issue seems to occur randomly (low frequency)

Steps to reproduce the bug

###nterceptor

public class TimeoutInterceptor implements ClientInterceptor {

  private final long timeout;

  /**
   * @param timeout ms
   */
  public TimeoutInterceptor(long timeout) {
    this.timeout = timeout;
  }

  @Override
  public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
      MethodDescriptor<ReqT, RespT> method,
      CallOptions callOptions,
      Channel next) {
    return next.newCall(method,
        callOptions.withDeadlineAfter(timeout, TimeUnit.MILLISECONDS));
  }
}

stack

"Test worker" #1 prio=5 os_prio=0 cpu=68854.65ms elapsed=51528.48s tid=0x0000ffffb40311a0 nid=0x321d1c waiting on condition  [0x0000ffffb99fb000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@17.0.15/Native Method)
    - parking to wait for  <0x00000000d69d5a18> (a io.grpc.stub.ClientCalls$ThreadlessExecutor)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
    at io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:817)
    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:166)
    at org.tron.api.DatabaseGrpc$DatabaseBlockingStub.getBlockByNum(DatabaseGrpc.java:343)
    at org.tron.common.client.DatabaseGrpcClient.getBlock(DatabaseGrpcClient.java:44)
    ...

grpc log

12:45:36.755 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[:authority: 127.0.0.1:57932, :path: /protocol.Database/GetBlockByNum, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.75.0, grpc-accept-encoding: gzip] padding=0 endStream=false
12:45:36.756 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] OUTBOUND DATA: streamId=5 padding=0 endStream=true length=5 bytes=0000000000
12:45:36.756 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] INBOUND HEADERS: streamId=5 headers=GrpcHttp2RequestHeaders[:path: /protocol.Database/GetBlockByNum, :authority: 127.0.0.1:57932, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.75.0, grpc-accept-encoding: gzip] padding=0 endStream=false
12:45:36.757 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] INBOUND DATA: streamId=5 padding=0 endStream=true length=5 bytes=0000000000
12:45:36.758 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip] padding=0 endStream=false
12:45:36.759 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND DATA: streamId=5 padding=0 endStream=false length=663 bytes=00000002920a720a705a6e0801126a0a2d747970652e676f6f676c65617069732e636f6d2f70726f746f636f6c2e5472616e73666572436f6e74726163741239...
12:45:36.759 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] padding=0 endStream=true
12:45:36.760 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND HEADERS: streamId=5 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip] padding=0 endStream=false
12:45:36.760 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND DATA: streamId=5 padding=0 endStream=false length=663 bytes=00000002920a720a705a6e0801126a0a2d747970652e676f6f676c65617069732e636f6d2f70726f746f636f6c2e5472616e73666572436f6e74726163741239...
12:45:36.761 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND HEADERS: streamId=5 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] padding=0 endStream=true


12:46:36.790 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND GO_AWAY: lastStreamId=2147483647 errorCode=0 length=8 bytes=6d61785f69646c65
12:46:36.791 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND PING: ack=false bytes=40715087873
12:46:36.791 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND GO_AWAY: lastStreamId=2147483647 errorCode=0 length=8 bytes=6d61785f69646c65
12:46:36.795 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND PING: ack=false bytes=40715087873
12:46:36.795 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] OUTBOUND PING: ack=true bytes=40715087873
12:46:36.795 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] INBOUND PING: ack=true bytes=40715087873
12:46:36.795 DEBUG [i.g.n.NettyServerHandler] [id: 0xa59d39e1, L:/127.0.0.1:57932 - R:/127.0.0.1:48716] OUTBOUND GO_AWAY: lastStreamId=5 errorCode=0 length=8 bytes=6d61785f69646c65
12:46:36.796 DEBUG [i.g.n.NettyClientHandler] [id: 0xe05d631e, L:/127.0.0.1:48716 - R:/127.0.0.1:57932] INBOUND GO_AWAY: lastStreamId=5 errorCode=0 length=8 bytes=6d61785f69646c65

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions