Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TLS sslv3 alert bad record mac error from 4.1.61 to 4.1.63 (inclusive) #11146

Closed
rkapsi opened this issue Apr 8, 2021 · 9 comments · Fixed by #11156
Closed

TLS sslv3 alert bad record mac error from 4.1.61 to 4.1.63 (inclusive) #11146

rkapsi opened this issue Apr 8, 2021 · 9 comments · Fixed by #11156

Comments

@rkapsi
Copy link
Member

rkapsi commented Apr 8, 2021

I'm upgrading from Netty 4.1.58 to 4.1.63 (TCN 2.0.34) and am running into an error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac error in one of my own Unit Tests. I have bisected the starting point of the error to this commit 0b0c234 and I do see a follow up commit 6b48e69 to fix it but it's still happening for me.

The two slightly different but same error stack traces are below.

Expected behavior

Actual behavior

4.1.61 commit 0b0c234

javax.net.ssl.SSLException: javax.net.ssl.SSLException: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:897) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:511) ~[?:1.8.0_282]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1094) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:845) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:813) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:794) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:189) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:540) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.channelReadComplete0(SslHandler.java:1356) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:1345) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1415) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.finishPeerRead0(LocalChannel.java:445) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.access$400(LocalChannel.java:50) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel$5.run(LocalChannel.java:403) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: javax.net.ssl.SSLException: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1347) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1308) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1384) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1427) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:208) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1302) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1339) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:299) ~[netty-all-4.1.61.Final-SNAPSHOT.jar:4.1.61.Final-SNAPSHOT]
	... 8 more

4.1.63 commit 6b48e69

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:299) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.read(SslHandler.java:754) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.read(Http2ConnectionHandler.java:524) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.read(DefaultChannelPipeline.java:1004) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannel.read(AbstractChannel.java:290) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.readIfIsAutoRead(DefaultChannelPipeline.java:1422) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1417) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.finishPeerRead0(LocalChannel.java:445) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.access$400(LocalChannel.java:50) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel$5.run(LocalChannel.java:403) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: javax.net.ssl.SSLException: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1066) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1359) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1308) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1384) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1427) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:208) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1358) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1265) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1302) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	... 36 more

Steps to reproduce

Minimal yet complete reproducer code (or URL to code)

Netty version

JVM version (e.g. java -version)

OS version (e.g. uname -a)
@chrisvest
Copy link
Contributor

@rkapsi Are you able to say more about how you're running into this, perhaps narrow it down to a reproducer?

@rkapsi
Copy link
Member Author

rkapsi commented Apr 9, 2021

@chrisvest It's an Unit Test that uses a LocalChannel and LocalServerChannel to setup HTTP/2 client+server. The server in particular is using a HttpToHttp2ConnectionHandler to handle it like HTTP/1.1 but it doesn't do anything special. The client sends a request upon connect, the server handles it and sends a response.

@Scottmitch
Copy link
Member

Scottmitch commented Apr 9, 2021 •
edited

@rkapsi - reproducer unit test would be very helpful if you could provide one. Are you entering the below conditional block, can you describe the behavior if/when you do (e.g. is this client/server, stack trace, is renegotiation happening, does this happen with jdk provider, what does wrap write and what is the SSLEngineResult, etc..), and if you comment out the wrap call does the handshake complete?

https://github.com/netty/netty/blob/4.1/handler/src/main/java/io/netty/handler/ssl/SslHandler.java#L969-L971

if (inUnwrap && !pendingUnencryptedWrites.isEmpty()) {
  wrap(ctx, true);
}

TCN 2.0.34 -> can you also try with 2.0.38.Final?

@rkapsi
Copy link
Member Author

rkapsi commented Apr 10, 2021

Netty 4.1.63.Final

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:299) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.read(SslHandler.java:754) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.read(Http2ConnectionHandler.java:524) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.read(DefaultChannelPipeline.java:1004) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannel.read(AbstractChannel.java:290) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.readIfIsAutoRead(DefaultChannelPipeline.java:1422) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1417) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.finishPeerRead0(LocalChannel.java:445) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.access$400(LocalChannel.java:50) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel$5.run(LocalChannel.java:403) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: javax.net.ssl.SSLException: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1066) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1359) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1308) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1384) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1427) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:208) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1358) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1265) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1302) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	... 36 more

@rkapsi
Copy link
Member Author

rkapsi commented Apr 10, 2021

Netty 4.1.63.Final with TCN 2.0.38.Final

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:299) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.read(SslHandler.java:754) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.read(Http2ConnectionHandler.java:524) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.read(DefaultChannelPipeline.java:1004) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannel.read(AbstractChannel.java:290) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.readIfIsAutoRead(DefaultChannelPipeline.java:1422) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1417) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.finishPeerRead0(LocalChannel.java:445) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.access$400(LocalChannel.java:50) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel$5.run(LocalChannel.java:403) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: javax.net.ssl.SSLException: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1066) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1359) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1308) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1384) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1427) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:208) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1358) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1265) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1302) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	... 36 more

@rkapsi
Copy link
Member Author

rkapsi commented Apr 10, 2021

Maybe useful: INFO level Http2FrameLogger shows this output before the Exception.

[id: 0x96d6d169, L:local:E:a478f228 - R:local:testhttp2httpbridge-0.3771874423247368] OUTBOUND SETTINGS: ack=false settings={MAX_HEADER_LIST_SIZE=8192}
[id: 0x96d6d169, L:local:E:a478f228 - R:local:testhttp2httpbridge-0.3771874423247368] INBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=8, MAX_HEADER_LIST_SIZE=8192}
[id: 0x96d6d169, L:local:E:a478f228 - R:local:testhttp2httpbridge-0.3771874423247368] OUTBOUND SETTINGS: ack=true
[id: 0x96d6d169, L:local:E:a478f228 - R:local:testhttp2httpbridge-0.3771874423247368] INBOUND SETTINGS: ack=true
[id: 0x96d6d169, L:local:E:a478f228 - R:local:testhttp2httpbridge-0.3771874423247368] OUTBOUND HEADERS: streamId=3 headers=DefaultHttp2Headers[:path: /hello, :method: GET, :scheme: https, :authority: localhost, accept-encoding: gzip, accept-encoding: deflate] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true
[id: 0x96d6d169, L:local:E:a478f228 - R:local:testhttp2httpbridge-0.3771874423247368] INBOUND HEADERS: streamId=3 headers=DefaultHttp2Headers[:status: 200, content-length: 8192] padding=0 endStream=false
[id: 0x96d6d169, L:local:E:a478f228 - R:local:testhttp2httpbridge-0.3771874423247368] INBOUND DATA: streamId=3 padding=0 endStream=false length=16 bytes=00000000000000000000000000000000
An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final.jar:4.1.63.Final]
...

And there is generally a second Exception (followed by refCnt Exceptions) which I have ignored so far because it seems to happen after the Exception I have mentioned above but it may contain some relevant information. In particular flush() calls being triggered from read() calls. I feel this could have some relevance.

An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
javax.net.ssl.SSLException: javax.net.ssl.SSLException: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:897) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:511) ~[?:1.8.0_282]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1059) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:842) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:813) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:794) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:189) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:540) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.channelReadComplete0(SslHandler.java:1319) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:1308) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1415) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.readIfNeeded(SslHandler.java:1327) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.channelReadComplete0(SslHandler.java:1316) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:1308) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1415) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.readIfNeeded(SslHandler.java:1327) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.channelReadComplete0(SslHandler.java:1316) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:1308) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1415) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.readIfNeeded(SslHandler.java:1327) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.channelReadComplete0(SslHandler.java:1316) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:1308) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1415) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.read(SslHandler.java:754) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete0(Http2ConnectionHandler.java:552) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:538) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.channelReadComplete0(SslHandler.java:1319) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:1308) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1415) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1774) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.setHandshakeSuccessIfStillHandshaking(SslHandler.java:1763) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:893) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:813) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:794) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:189) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:967) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannel.flush(AbstractChannel.java:254) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at com.squarespace.echo.http2.streams.HttpToHttp2Handler.flush(HttpToHttp2Handler.java:92) [main/:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at com.squarespace.echo.http2.streams.Http2HttpBridgeTest$1.channelRead(Http2HttpBridgeTest.java:122) [test/:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at com.squarespace.echo.core.pipe.PipeChannel.fireChannelRead(PipeChannel.java:125) [main/:?]
	at com.squarespace.echo.core.pipe.PipeChannel.writeInbound(PipeChannel.java:139) [main/:?]
	at com.squarespace.echo.core.pipe.PipeChannel.writeInbound(PipeChannel.java:118) [main/:?]
	at com.squarespace.echo.http2.streams.Http2ToHttpFrameHandler.onHeaders(Http2ToHttpFrameHandler.java:237) [main/:?]
	at com.squarespace.echo.http2.streams.Http2ToHttpFrameHandler.onHeadersRead(Http2ToHttpFrameHandler.java:205) [main/:?]
	at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onHeadersRead(Http2FrameListenerDecorator.java:48) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2EmptyDataFrameListener.onHeadersRead(Http2EmptyDataFrameListener.java:63) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:420) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:65) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$1.processFragment(DefaultHttp2FrameReader.java:457) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:464) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:254) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:181) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1504) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1265) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1302) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:299) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.finishPeerRead0(LocalChannel.java:445) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.access$400(LocalChannel.java:50) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel$5.run(LocalChannel.java:403) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: javax.net.ssl.SSLException: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1347) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1308) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1384) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1427) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:208) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1358) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1265) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1302) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:299) ~[netty-all-4.1.63.Final.jar:4.1.63.Final]
	... 77 more

@Scottmitch
Copy link
Member

@rkapsi - the control flow with LocalChannel is interesting where reads are done immediately on the stack, perhaps we should wait until the stack unwinds if read is called in a re-entry fashion. However it isn't clear if this is directly responsible for the ssl alert, as this alert is typically triggered from a previous wrap operation on the peer (e.g. being out of order).

wdyt about #11146 (comment) ? would be good to understand if the new call to wrap has any effect.

@rkapsi
Copy link
Member Author

rkapsi commented Apr 13, 2021

@Scottmitch I've commented out lines 969-971 in SslHandler of commit 6b48e69 and re-ran my Unit Test. The first exception goes away. The subsequent exception(s) remain or rather it has become now the primary exception.

javax.net.ssl.SSLException: javax.net.ssl.SSLException: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:897) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:511) ~[?:1.8.0_282]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1059) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:842) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:813) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:794) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:189) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:540) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.channelReadComplete0(SslHandler.java:1319) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:1308) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1415) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.readIfNeeded(SslHandler.java:1327) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.channelReadComplete0(SslHandler.java:1316) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:1308) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1415) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.read(SslHandler.java:754) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete0(Http2ConnectionHandler.java:552) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:538) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.channelReadComplete0(SslHandler.java:1319) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:1308) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:390) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1415) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:397) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:925) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:302) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.doBeginRead(LocalChannel.java:322) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.beginRead(AbstractChannel.java:856) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1774) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.setHandshakeSuccessIfStillHandshaking(SslHandler.java:1763) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:893) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:813) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:794) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:189) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:967) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannel.flush(AbstractChannel.java:254) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at com.squarespace.echo.http2.streams.HttpToHttp2Handler.flush(HttpToHttp2Handler.java:92) [main/:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at com.squarespace.echo.http2.streams.Http2HttpBridgeTest$1.channelRead(Http2HttpBridgeTest.java:127) [test/:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at com.squarespace.echo.core.pipe.PipeChannel.fireChannelRead(PipeChannel.java:125) [main/:?]
	at com.squarespace.echo.core.pipe.PipeChannel.writeInbound(PipeChannel.java:139) [main/:?]
	at com.squarespace.echo.core.pipe.PipeChannel.writeInbound(PipeChannel.java:118) [main/:?]
	at com.squarespace.echo.http2.streams.Http2ToHttpFrameHandler.onHeaders(Http2ToHttpFrameHandler.java:237) [main/:?]
	at com.squarespace.echo.http2.streams.Http2ToHttpFrameHandler.onHeadersRead(Http2ToHttpFrameHandler.java:205) [main/:?]
	at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onHeadersRead(Http2FrameListenerDecorator.java:48) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2EmptyDataFrameListener.onHeadersRead(Http2EmptyDataFrameListener.java:63) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:420) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:65) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$1.processFragment(DefaultHttp2FrameReader.java:457) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:464) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:254) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:181) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1504) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1265) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1302) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:299) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.finishPeerRead0(LocalChannel.java:445) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.access$400(LocalChannel.java:50) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel$5.run(LocalChannel.java:403) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: javax.net.ssl.SSLException: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1347) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1308) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1384) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1427) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:208) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1358) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1265) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1302) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	at io.netty.channel.local.LocalChannel.readInbound(LocalChannel.java:299) ~[netty-all-4.1.63.Final-SNAPSHOT.jar:4.1.63.Final-SNAPSHOT]
	... 77 more

@Scottmitch Scottmitch mentioned this issue Apr 14, 2021
Merged
@Scottmitch
Copy link
Member

@rkapsi - can you try again with #11156?

Scottmitch added a commit to Scottmitch/netty that referenced this issue Apr 15, 2021
@Scottmitch
SslHandler LocalChannel read/unwrap reentry fix
17df836 
Motivation:
SslHandler invokes channel.read() during the handshake process. For some
channel implementations (e.g. LocalChannel) this may result in re-entry
conditions into unwrap. Unwrap currently defers updating the input
buffer indexes until the unwrap method returns to avoid intermediate
updates if not necessary, but this may result in unwrapping the same
contents multiple times which leads to handshake failures [1][2].

[1] ssl3_get_record:decryption failed or bad record mac
[2] ssl3_read_bytes:sslv3 alert bad record mac

Modifications:
- SslHandler#unwrap updates buffer indexes on each iteration so that if
  reentry scenario happens the correct indexes will be visible.

Result:
Fixes netty#11146
Scottmitch added a commit that referenced this issue Apr 16, 2021
@Scottmitch
SslHandler LocalChannel read/unwrap reentry fix (#11156)
0ef2f05 
Motivation:
SslHandler invokes channel.read() during the handshake process. For some
channel implementations (e.g. LocalChannel) this may result in re-entry
conditions into unwrap. Unwrap currently defers updating the input
buffer indexes until the unwrap method returns to avoid intermediate
updates if not necessary, but this may result in unwrapping the same
contents multiple times which leads to handshake failures [1][2].

[1] ssl3_get_record:decryption failed or bad record mac
[2] ssl3_read_bytes:sslv3 alert bad record mac

Modifications:
- SslHandler#unwrap updates buffer indexes on each iteration so that if
  reentry scenario happens the correct indexes will be visible.

Result:
Fixes #11146
Scottmitch added a commit that referenced this issue Apr 16, 2021
@Scottmitch
SslHandler LocalChannel read/unwrap reentry fix (#11156)
59867fa 
Motivation:
SslHandler invokes channel.read() during the handshake process. For some
channel implementations (e.g. LocalChannel) this may result in re-entry
conditions into unwrap. Unwrap currently defers updating the input
buffer indexes until the unwrap method returns to avoid intermediate
updates if not necessary, but this may result in unwrapping the same
contents multiple times which leads to handshake failures [1][2].

[1] ssl3_get_record:decryption failed or bad record mac
[2] ssl3_read_bytes:sslv3 alert bad record mac

Modifications:
- SslHandler#unwrap updates buffer indexes on each iteration so that if
  reentry scenario happens the correct indexes will be visible.

Result:
Fixes #11146
raidyue pushed a commit to raidyue/netty that referenced this issue Jul 8, 2022
@Scottmitch
SslHandler LocalChannel read/unwrap reentry fix (netty#11156)
13c17bd 
Motivation:
SslHandler invokes channel.read() during the handshake process. For some
channel implementations (e.g. LocalChannel) this may result in re-entry
conditions into unwrap. Unwrap currently defers updating the input
buffer indexes until the unwrap method returns to avoid intermediate
updates if not necessary, but this may result in unwrapping the same
contents multiple times which leads to handshake failures [1][2].

[1] ssl3_get_record:decryption failed or bad record mac
[2] ssl3_read_bytes:sslv3 alert bad record mac

Modifications:
- SslHandler#unwrap updates buffer indexes on each iteration so that if
  reentry scenario happens the correct indexes will be visible.

Result:
Fixes netty#11146
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

SslHandler LocalChannel read/unwrap reentry fix Scottmitch/netty
3 participants
@chrisvest @rkapsi @Scottmitch