Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JVM crash during test-run #5692

Closed
normanmaurer opened this issue Aug 12, 2016 · 3 comments
Closed

JVM crash during test-run #5692

normanmaurer opened this issue Aug 12, 2016 · 3 comments

Comments

@normanmaurer
Copy link
Member

normanmaurer commented Aug 12, 2016 •
edited by Scottmitch
Loading 

[18:52:38][io.netty:netty-handler] 18:52:38.742 [localEventLoopGroup-4-2] DEBUG i.n.h.s.u.InsecureTrustManagerFactory - Accepting a server certificate: CN=example.com
[18:52:38][io.netty:netty-handler] 18:52:38.753 [localEventLoopGroup-4-3] DEBUG io.netty.handler.ssl.SslHandler - [id: 0xd2a5b20f, L:local:test - R:local:E:8daf060a] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
[18:52:38][io.netty:netty-handler] 18:52:38.755 [localEventLoopGroup-4-2] DEBUG io.netty.handler.ssl.SslHandler - [id: 0x8daf060a, L:local:E:8daf060a - R:local:test] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
[18:52:38][io.netty:netty-handler] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.71 sec - in io.netty.handler.ssl.OpenSslRenegotiateTest
[18:52:38][io.netty:netty-handler] Running io.netty.handler.ssl.ReferenceCountedOpenSslEngineTest
[18:52:38][io.netty:netty-handler] 18:52:38.908 [threadDeathWatcher-1-1] DEBUG io.netty.buffer.PoolThreadCache - Freed 3 thread-local buffer(s) from thread: Time-limited test
[18:52:39][io.netty:netty-handler] 18:52:39.189 [nioEventLoopGroup-7-1] DEBUG i.n.u.i.JavassistTypeParameterMatcherGenerator - Generated: io.netty.util.internal.__matchers__.io.netty.buffer.ByteBufMatcher
[18:52:39][io.netty:netty-handler] 18:52:39.220 [nioEventLoopGroup-7-1] DEBUG i.n.h.s.u.InsecureTrustManagerFactory - Accepting a server certificate: CN=example.com
[18:52:39][io.netty:netty-handler] 18:52:39.225 [nioEventLoopGroup-6-1] DEBUG io.netty.handler.ssl.SslHandler - [id: 0x76841c34, L:/127.0.0.1:34028 - R:/127.0.0.1:43804] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[18:52:39][io.netty:netty-handler] 18:52:39.227 [nioEventLoopGroup-7-1] DEBUG io.netty.handler.ssl.SslHandler - [id: 0xc4855007, L:/127.0.0.1:43804 - R:/0:0:0:0:0:0:0:0:34028] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[18:52:39][io.netty:netty-handler] 18:52:39.408 [nioEventLoopGroup-7-1] DEBUG io.netty.buffer.PoolThreadCache - Freed 11 thread-local buffer(s) from thread: nioEventLoopGroup-7-1
[18:52:39][io.netty:netty-handler] 18:52:39.429 [nioEventLoopGroup-6-1] DEBUG io.netty.buffer.PoolThreadCache - Freed 9 thread-local buffer(s) from thread: nioEventLoopGroup-6-1
[18:52:39][io.netty:netty-handler] 18:52:39.553 [main] DEBUG i.n.h.s.u.InsecureTrustManagerFactory - Accepting a server certificate: CN=example.com
[18:52:39][io.netty:netty-handler] 18:52:39.641 [nioEventLoopGroup-9-1] DEBUG io.netty.handler.ssl.SslHandler - [id: 0xc05b0b8c, L:/0:0:0:0:0:0:0:1:50202 - R:/0:0:0:0:0:0:0:1:40161] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[18:52:39][io.netty:netty-handler] 18:52:39.645 [nioEventLoopGroup-10-1] DEBUG io.netty.handler.ssl.SslHandler - [id: 0x5baec8a4, L:/0:0:0:0:0:0:0:1:40161 - R:/0:0:0:0:0:0:0:1%1:50202] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[18:52:39][io.netty:netty-handler] 18:52:39.696 [nioEventLoopGroup-9-1] DEBUG io.netty.buffer.PoolThreadCache - Freed 11 thread-local buffer(s) from thread: nioEventLoopGroup-9-1
[18:52:39][io.netty:netty-handler] 18:52:39.721 [nioEventLoopGroup-10-1] DEBUG io.netty.buffer.PoolThreadCache - Freed 12 thread-local buffer(s) from thread: nioEventLoopGroup-10-1
[18:52:39][io.netty:netty-handler] 18:52:39.900 [main] DEBUG i.n.h.s.u.InsecureTrustManagerFactory - Accepting a server certificate: CN=example.com
[18:52:39][io.netty:netty-handler] 18:52:39.970 [main] DEBUG i.n.h.s.u.InsecureTrustManagerFactory - Accepting a server certificate: CN=example.com
[18:52:40][io.netty:netty-handler] 18:52:40.032 [nioEventLoopGroup-13-1] DEBUG i.n.h.s.ReferenceCountedOpenSslContext - verification of certificate failed
[18:52:40][io.netty:netty-handler] sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[18:52:40][io.netty:netty-handler]  at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
[18:52:40][io.netty:netty-handler]  at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
[18:52:40][io.netty:netty-handler]  at sun.security.validator.Validator.validate(Validator.java:260)
[18:52:40][io.netty:netty-handler]  at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
[18:52:40][io.netty:netty-handler]  at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:283)
[18:52:40][io.netty:netty-handler]  at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:138)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:219)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:588)
[18:52:40][io.netty:netty-handler]  at org.apache.tomcat.jni.SSL.readFromSSL(Native Method)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:456)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:823)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:930)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:973)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1093)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:965)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:899)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:610)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:551)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:465)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:437)
[18:52:40][io.netty:netty-handler]  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:52:40][io.netty:netty-handler]  at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:52:40][io.netty:netty-handler]  at java.lang.Thread.run(Thread.java:745)
[18:52:40][io.netty:netty-handler] Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[18:52:40][io.netty:netty-handler]  at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
[18:52:40][io.netty:netty-handler]  at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
[18:52:40][io.netty:netty-handler]  at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
[18:52:40][io.netty:netty-handler]  ... 32 common frames omitted
[18:52:40][io.netty:netty-handler] io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
[18:52:40][io.netty:netty-handler]  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:610)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:551)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:465)
[18:52:40][io.netty:netty-handler]  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:437)
[18:52:40][io.netty:netty-handler]  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:52:40][io.netty:netty-handler]  at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:52:40][io.netty:netty-handler]  at java.lang.Thread.run(Thread.java:745)
[18:52:40][io.netty:netty-handler] Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:592)
[18:52:40][io.netty:netty-handler]  at org.apache.tomcat.jni.SSL.readFromSSL(Native Method)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:456)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:823)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:930)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:973)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1093)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:965)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:899)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
[18:52:40][io.netty:netty-handler]  ... 16 more
[18:52:40][io.netty:netty-handler] Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[18:52:40][io.netty:netty-handler]  at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
[18:52:40][io.netty:netty-handler]  at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
[18:52:40][io.netty:netty-handler]  at sun.security.validator.Validator.validate(Validator.java:260)
[18:52:40][io.netty:netty-handler]  at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
[18:52:40][io.netty:netty-handler]  at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:283)
[18:52:40][io.netty:netty-handler]  at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:138)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:219)
[18:52:40][io.netty:netty-handler]  at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:588)
[18:52:40][io.netty:netty-handler]  ... 25 more
[18:52:40][io.netty:netty-handler] Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[18:52:40][io.netty:netty-handler]  at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
[18:52:40][io.netty:netty-handler]  at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
[18:52:40][io.netty:netty-handler]  at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
[18:52:40][io.netty:netty-handler]  ... 32 more
[18:52:40][io.netty:netty-handler] #
[18:52:40][io.netty:netty-handler] # A fatal error has been detected by the Java Runtime Environment:
[18:52:40][io.netty:netty-handler] #
[18:52:40][io.netty:netty-handler] #  SIGSEGV (0xb) at pc=0x00007f4768e9232b, pid=11364, tid=139944098862848
[18:52:40][io.netty:netty-handler] #
[18:52:40][io.netty:netty-handler] # JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build 1.7.0_80-b15)
[18:52:40][io.netty:netty-handler] # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode linux-amd64 compressed oops)
[18:52:40][io.netty:netty-handler] # Problematic frame:
[18:52:40][io.netty:netty-handler] # V  [libjvm.so+0x65a32b]  JNI_ArgumentPusherVaArg::JNI_ArgumentPusherVaArg(_jmethodID*, __va_list_tag*)+0x1b
[18:52:40][io.netty:netty-handler] #
[18:52:40][io.netty:netty-handler] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
[18:52:40][io.netty:netty-handler] #
[18:52:40][io.netty:netty-handler] # An error report file with more information is saved as:
[18:52:40][io.netty:netty-handler] # /var/lib/teamcity-agent/work/22d507c1ba6aba57/handler/hs_err_pid11364.log
[18:52:40][io.netty:netty-handler] #
[18:52:40][io.netty:netty-handler] # If you would like to submit a bug report, please visit:
[18:52:40][io.netty:netty-handler] #   http://bugreport.java.com/bugreport/crash.jsp
[18:52:40][io.netty:netty-handler] #
[18:52:40][io.netty:netty-handler] Aborted
[18:52:40][io.netty:netty-handler] 
[18:52:40][io.netty:netty-handler] Results :
[18:52:40][io.netty:netty-handler] 
[18:52:40][io.netty:netty-handler] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0
@normanmaurer
Copy link
Member Author

Maybe related to:
https://bugs.openjdk.java.net/browse/JDK-8146230

@Scottmitch
Copy link
Member

Can you reliably reproduce ... and if so is it always when running ReferenceCountedOpenSslEngineTest?

what java version, OS, etc..

@normanmaurer
Copy link
Member Author

@Scottmitch only saw on the ci so far.

@Scottmitch Scottmitch mentioned this issue Aug 17, 2016
Closed
Scottmitch added a commit to Scottmitch/netty that referenced this issue Aug 17, 2016
@Scottmitch
ReferenceCountedOpenSslEngineTest cleanup sequencing bug
6342337 
Motivation:
ReferenceCountedOpenSslEngine depends upon the the SslContext to cleanup JNI resources. If we don't wait until the ReferenceCountedOpenSslEngine is done with cleanup before cleaning up the SslContext we may crash the JVM.

Modifications:
- Wait for the channels to close (and thus the ReferenceCountedOpenSslEngine to be cleaned up) before cleaning up the associated SslContext.

Result:
Cleanup sequencing is correct and no more JVM crash.
Fixes netty#5692
normanmaurer pushed a commit that referenced this issue Aug 18, 2016
@Scottmitch @normanmaurer
ReferenceCountedOpenSslEngineTest cleanup sequencing bug
0a3e699 
Motivation:
ReferenceCountedOpenSslEngine depends upon the the SslContext to cleanup JNI resources. If we don't wait until the ReferenceCountedOpenSslEngine is done with cleanup before cleaning up the SslContext we may crash the JVM.

Modifications:
- Wait for the channels to close (and thus the ReferenceCountedOpenSslEngine to be cleaned up) before cleaning up the associated SslContext.

Result:
Cleanup sequencing is correct and no more JVM crash.
Fixes #5692
liuzhengyang pushed a commit to liuzhengyang/netty that referenced this issue Sep 10, 2017
@Scottmitch
ReferenceCountedOpenSslEngineTest cleanup sequencing bug
bd4e1f9 
Motivation:
ReferenceCountedOpenSslEngine depends upon the the SslContext to cleanup JNI resources. If we don't wait until the ReferenceCountedOpenSslEngine is done with cleanup before cleaning up the SslContext we may crash the JVM.

Modifications:
- Wait for the channels to close (and thus the ReferenceCountedOpenSslEngine to be cleaned up) before cleaning up the associated SslContext.

Result:
Cleanup sequencing is correct and no more JVM crash.
Fixes netty#5692
pulllock pushed a commit to pulllock/netty that referenced this issue Oct 19, 2023
@Scottmitch @normanmaurer
ReferenceCountedOpenSslEngineTest cleanup sequencing bug
426750b 
Motivation:
ReferenceCountedOpenSslEngine depends upon the the SslContext to cleanup JNI resources. If we don't wait until the ReferenceCountedOpenSslEngine is done with cleanup before cleaning up the SslContext we may crash the JVM.

Modifications:
- Wait for the channels to close (and thus the ReferenceCountedOpenSslEngine to be cleaned up) before cleaning up the associated SslContext.

Result:
Cleanup sequencing is correct and no more JVM crash.
Fixes netty#5692
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@normanmaurer @Scottmitch