Unexpected exception: io.grpc.StatusRuntimeException: UNKNOWN: channel closed
#7376
Comments
|
If I turn off On jdk1.8.0_261: When using jdk11, it first starts with then follows with when |
|
@ejona86 - please let me know if there is anything more I can do. |
There should be a stacktrace along with that, for the ClosedChannelException. gRPC goes out of its way to collect a bit more information in hopes of figuring out what happened.
That is the end of the message. There should have been additional details before it. Was this an UNKNOWN status code as well? Looking at the code, it seems likely it was. When we see "Channel Pipeline" in the failure, we know we didn't get a working connection. In this specific case, there was a failure in the middle of a TLS handshake. If this is the problem you are experiencing for the UNKNOWN, we can definitely look into this.
Those frames just show an actual working connection.
So the connection received a GO_AWAY. Depending on details, if you see an error after this it could be that the replacement connection had trouble. With a hacked server, I've reproduced an UNKNOWN failure caused by diff --git a/netty/src/main/java/io/grpc/netty/ProtocolNegotiators.java b/netty/src/main/java/io/grpc/netty/ProtocolNegotiators.java
index 27b2c74be..432fcf79b 100644
--- a/netty/src/main/java/io/grpc/netty/ProtocolNegotiators.java
+++ b/netty/src/main/java/io/grpc/netty/ProtocolNegotiators.java
@@ -170,12 +170,18 @@ final class ProtocolNegotiators {
}
@Override
- public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
+ public void handlerAdded(final ChannelHandlerContext ctx) throws Exception {
super.handlerAdded(ctx);
SSLEngine sslEngine = sslContext.newEngine(ctx.alloc());
ctx.pipeline().addBefore(ctx.name(), /* name= */ null, this.executor != null
? new SslHandler(sslEngine, false, this.executor)
: new SslHandler(sslEngine, false));
+ ctx.flush();
+ ctx.executor().execute(new Runnable() {
+ @Override public void run() {
+ ctx.close();
+ }
+ });
}
@OverrideThat backtrace is strange because AbstractChannel:818 is part of deregister(). It'll need some more poking. |
|
Additional context: |
Yep.
I only experience the UNKNOWN and channel closed when TLS is involved.
When I stop the server, it sends go away, but I have also tried with |
|
@ejona86 I have tried 1.32.1 and tcnative-boring 2.30.1, no luck. I can trigger that error relatively easy by killing the server randomly. I also tried to see if using Conscrypt helped, jdk 11 / 15 etc. Is there something I could to do in order to isolate the issue? Some race perhaps, as you mention deregister? |
|
@ejona86 I have tried to dig a bit and added some I kill a server that my client talks to and see the following. Note that If I use |
ejona86
added a commit
to ejona86/grpc-java
that referenced
this issue
Oct 1, 2020
Normally the first exception/event experienced is the cause and is followed by a stampede of ClosedChannelExceptions. In this case, SslHandler is manufacturing a ClosedChannelException of its own and propagating it _before_ the trigger event. This might be considered a bug, but changing SslHandler's behavior would be very risky and almost certainly break someone's code. Fixes grpc#7376
|
@ahjohannessen, thanks for that. Taking another look at the stack trace and the other errors gave me an "aha" to realize what was happening. The original stack trace actually included enough information, but it didn't click. |
|
@ejona86 Awesome 👍 :) |
ejona86
added a commit
that referenced
this issue
Oct 1, 2020
Normally the first exception/event experienced is the cause and is followed by a stampede of ClosedChannelExceptions. In this case, SslHandler is manufacturing a ClosedChannelException of its own and propagating it _before_ the trigger event. This might be considered a bug, but changing SslHandler's behavior would be very risky and almost certainly break someone's code. Fixes #7376
|
@ejona86 Seems like that did it: |
ejona86
added a commit
to ejona86/grpc-java
that referenced
this issue
Oct 1, 2020
Normally the first exception/event experienced is the cause and is followed by a stampede of ClosedChannelExceptions. In this case, SslHandler is manufacturing a ClosedChannelException of its own and propagating it _before_ the trigger event. This might be considered a bug, but changing SslHandler's behavior would be very risky and almost certainly break someone's code. Fixes grpc#7376
ejona86
added a commit
that referenced
this issue
Oct 1, 2020
Normally the first exception/event experienced is the cause and is followed by a stampede of ClosedChannelExceptions. In this case, SslHandler is manufacturing a ClosedChannelException of its own and propagating it _before_ the trigger event. This might be considered a bug, but changing SslHandler's behavior would be very risky and almost certainly break someone's code. Fixes #7376
dfawley
pushed a commit
to dfawley/grpc-java
that referenced
this issue
Jan 15, 2021
Normally the first exception/event experienced is the cause and is followed by a stampede of ClosedChannelExceptions. In this case, SslHandler is manufacturing a ClosedChannelException of its own and propagating it _before_ the trigger event. This might be considered a bug, but changing SslHandler's behavior would be very risky and almost certainly break someone's code. Fixes grpc#7376
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What version of gRPC-Java are you using?
What is your environment?
What did you expect to see?
io.grpc.StatusRuntimeException: UNAVAILABLE: channel closedor similar.What did you see instead?
io.grpc.StatusRuntimeException: UNKNOWN: channel closedSteps to reproduce the bug
Not sure how to reproduce it, but if I kill the server I get the following.
This happens before the exception:
The text was updated successfully, but these errors were encountered: