"StatusRuntimeException: INTERNAL: HTTP/2 error code: INTERNAL_ERROR" ,"StatusRuntimeException:UNKNOWN"

[kaiyangzhang]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.0.0, 1.2.0

What JVM are you using (java -version)?

1.8

What did you do?

I am running a gRPC-java server with multiple grpc services defined.

When running multiple stubs which multiple threads share, so multiple rpc calls simultaneously, I am seeing this exception below, the exception is only occasionally appear.

(1) io.grpc.statusRuntimeException: UNKNOWN
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:227)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:208)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141)
...............................
.................................
(2) io.grpc.statusRuntimeException: INTERNAL: HTTP/2 error code: INTERNAL_ERROR
Received Rst Stream
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:227)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:208)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141)
...............................
.................................

What is benchmark scenario?

1. my benchmark is a distributed program: one client to three servers.

2. client tps: 150k+ /s, each server receive message : 50k+ /s

question:

1. why the exception is only occasionally appear? Concurrency is too big?
2. mutiple thread can call the same stub object at the same time?

[carl-mastrangelo]

RST STREAM only happens on protocol level errors. What version is the gRPC server running?

[ejona86]

Are the clients and servers both using grpc-java?

RST_STREAM with CANCEL happens in some other common cases. RST_STREAM with INTERNAL_ERROR can occur in some error cases. I'd expect a protocol error to have error code PROTOCOL_ERROR.

For the UNKNOWN, I'd expect a "caused by" exception for it. Was there? If not, then that came from the server. I'd expect the server threw an exception and it should be logged.

@kaiyangzhang, did you check the server logs at all? I'd expect that they logged something when this happened.

[carl-mastrangelo]

@ejona86:

I am running a gRPC-java server with multiple grpc services defined.

and

at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:227)

I believe this implies it is both Java client and server.

[kaiyangzhang]

@carl-mastrangelo @ejona86

1. Clients and servers both used grpc-java-1.0.0, and tried grpc-java-1.2.0.
2. For the UNKNOWN, the client did not “caused by”exception, and the server did not throw exception.
3. When the client catch these two exceptions, and then retry once or several times, then no more exceptions are thrown. Because these two exceptions is only occasionally appear, I temporarily use this retry strategy to avoid.
4. When the client concurrently sends data to the server, these two exception will not be thrown. But client concurrent query scene, the client concurrent receive data from servers, these two exception will be thrown.

[kaiyangzhang]

The problem has been solved, and the problem is caused by the server. The queue size of grpc server's thread pool is too small, resulting in tasks submit failed. the server throw exception below:

rejected execution of io.grpc.internal.SerializingExecutor$TaskRunner@5f207f61 on AsThreadPoolExecutor[GrpcServer_ASTERIA_SERVER, queue capacity=200, com.fiberhome.asteriadb.common.concurrent.AsthreadPoolExecutor@5dcfbbfe[Running, pool size=100, active threads =100, queued tasks =200, completed tasks =3473355]]
at com.fiberhome.asteriadb.common.concurrent.AsAbortPolicy.rejectedExecution
at java.util.concurrent.ThreadPoolExecutor.reject
at java.util.concurrent.ThreadPoolExecutor.execute
at com.fiberhome.asteriadb.common.current.AsThreadPoolExecutor.execute
at io.grpc.internal.SerializingExecutor.execute
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener
at io.grpc.internal.AbstractServerStream$TransportState.closeListener
at io.grpc.internal.AbstractServerStream$TransportState.complete
at io.grpc.nettyServerStream$TransportState.complete
.............................
.............................

[longbozhan]

I have the same problems when I send a bytes to server, the bytes's size is about 14Mb, and I have set MAX_SIZE = 50 *1024*1024;
channelBuilder.maxMessageSize(MAX_SIZE) (NettyChannelBuilder)
nettyServerBuilder.maxMessageSize(MAX_SIZE);
@ejona86