New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak when creating ReferenceCountedSslServerContext instances #6249

Closed
rkapsi opened this Issue Jan 18, 2017 · 46 comments

Comments

Projects
None yet
4 participants
@rkapsi
Contributor

rkapsi commented Jan 18, 2017

We're currently running a 4.1.7.Final-SNAPSHOT version of Netty that pre-dates the official 4.1.7.Final release (our SNAPSHOT is roughly from January 6, 2017).

We today attempted a release with 4.1.8.Final-SNAPSHOT and I've tried 4.1.7.Final as well. There is suddenly a large number of the following (new) Exceptions and the server runs out of memory quickly until Linux's oom_killer steps in and axes the process. There are no Java OOMEs nor do YourKit memory snapshots show anything out of the ordinary which points towards direct memory being leaked.

Looking the recent commit history there appears to be this dd055c0 change that is potentially related (happened after January 6 but before 4.1.7.Final release).

javax.net.ssl.SSLException: error:140D00CF:SSL routines:SSL_write:protocol is shutdown
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:719) ~[netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:708) ~[netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:685) ~[netty-all-4.1.7.Final.jar:4.1.7.Final]
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:509) ~[?:1.8.0_102]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:746) ~[netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:578) ~[netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:550) ~[netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:531) ~[netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:530) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:355) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) [netty-all-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.7.Final.jar:4.1.7.Final]
...
@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 18, 2017

Contributor

I forgot to mention that I tried to run it with -Dio.netty.noPreferDirect=true" and -Dio.netty.allocator.type=unpooled" which doesn't make a difference.

Contributor

rkapsi commented Jan 18, 2017

I forgot to mention that I tried to run it with -Dio.netty.noPreferDirect=true" and -Dio.netty.allocator.type=unpooled" which doesn't make a difference.

@doom369

This comment has been minimized.

Show comment
Hide comment
@doom369

doom369 Jan 18, 2017

Contributor

@rkapsi you also need to turn openssl off in order to avoid leak (+ above options). I think @normanmaurer will update us regarding those tickets soon.

Contributor

doom369 commented Jan 18, 2017

@rkapsi you also need to turn openssl off in order to avoid leak (+ above options). I think @normanmaurer will update us regarding those tickets soon.

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 18, 2017

Contributor

In our case not really an option. Our product depends on it but I can thankfully continue using the working version of Netty in the meantime.

Contributor

rkapsi commented Jan 18, 2017

In our case not really an option. Our product depends on it but I can thankfully continue using the working version of Netty in the meantime.

@Scottmitch

This comment has been minimized.

Show comment
Hide comment
@Scottmitch

Scottmitch Jan 19, 2017

Member

@rkapsi - Can you try to run with #6238 ? There was a bug in the commit you referenced which removed a call to shutdown in error.

Member

Scottmitch commented Jan 19, 2017

@rkapsi - Can you try to run with #6238 ? There was a bug in the commit you referenced which removed a call to shutdown in error.

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

@Scottmitch will do tomorrow first thing in the morning (NYC time).

Contributor

rkapsi commented Jan 19, 2017

@Scottmitch will do tomorrow first thing in the morning (NYC time).

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi the exception is fixed by #6238 ... Will have another PR this morning to test with.

Member

normanmaurer commented Jan 19, 2017

@rkapsi the exception is fixed by #6238 ... Will have another PR this morning to test with.

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi PTAL at #6252 as well.

Member

normanmaurer commented Jan 19, 2017

@rkapsi PTAL at #6252 as well.

@normanmaurer normanmaurer self-assigned this Jan 19, 2017

@normanmaurer normanmaurer added the defect label Jan 19, 2017

@normanmaurer normanmaurer added this to the 4.0.44.Final milestone Jan 19, 2017

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

@normanmaurer confirm, this fixes the Exception. Stay tuned regards the memory problem. It appears #6252 does not fix it.

This is how one of our graphs looked like: https://i.imgsafe.org/0d77f1a4e6.png

I was wrong regards the SNAPSHOT version we use (the one that works). It's a build from January 9. I have a few more builds I can try.

Contributor

rkapsi commented Jan 19, 2017

@normanmaurer confirm, this fixes the Exception. Stay tuned regards the memory problem. It appears #6252 does not fix it.

This is how one of our graphs looked like: https://i.imgsafe.org/0d77f1a4e6.png

I was wrong regards the SNAPSHOT version we use (the one that works). It's a build from January 9. I have a few more builds I can try.

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi can I have a heap dump?

Member

normanmaurer commented Jan 19, 2017

@rkapsi can I have a heap dump?

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

Unfortunately no. It contains sensitive PKI material. If you have a hunch, class names etc. then let me know. I can poke around and share screenshots and numbers.

I've just double checked #6252 with a second build. The leak persists. The graph from above show's Linux's free memory (e.g. the value that's shown in top). As you can see it drops from ~12GB down to almost nothing. Our JVM heap is configured to be 4GB and the heap dump is only 2.3GB. So clearly something outside the JVM heap.

Contributor

rkapsi commented Jan 19, 2017

Unfortunately no. It contains sensitive PKI material. If you have a hunch, class names etc. then let me know. I can poke around and share screenshots and numbers.

I've just double checked #6252 with a second build. The leak persists. The graph from above show's Linux's free memory (e.g. the value that's shown in top). As you can see it drops from ~12GB down to almost nothing. Our JVM heap is configured to be 4GB and the heap dump is only 2.3GB. So clearly something outside the JVM heap.

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi bummer... maybe ChannelOutboundBuffer.Entry instances. Would be interested to see what ByteBuf instances are in there and what there indices + capacity are.

Member

normanmaurer commented Jan 19, 2017

@rkapsi bummer... maybe ChannelOutboundBuffer.Entry instances. Would be interested to see what ByteBuf instances are in there and what there indices + capacity are.

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi also if you can give me any idea which builds worked and which started to show this would help

Member

normanmaurer commented Jan 19, 2017

@rkapsi also if you can give me any idea which builds worked and which started to show this would help

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

https://i.imgsafe.org/0ed39effb0.png
https://i.imgsafe.org/0ed437aecf.png

OK: netty-all-4.1.7.Final-20170109.120749-87 (the version we use)
OK: netty-all-4.1.7.Final-20170110.120845-88
OK: netty-all-4.1.7.Final-20170111.122656-89 (last build I have)

So it's something after January 11. Going to cut releases from commits as next.

Contributor

rkapsi commented Jan 19, 2017

https://i.imgsafe.org/0ed39effb0.png
https://i.imgsafe.org/0ed437aecf.png

OK: netty-all-4.1.7.Final-20170109.120749-87 (the version we use)
OK: netty-all-4.1.7.Final-20170110.120845-88
OK: netty-all-4.1.7.Final-20170111.122656-89 (last build I have)

So it's something after January 11. Going to cut releases from commits as next.

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi ok thanks... keep me posted

Member

normanmaurer commented Jan 19, 2017

@rkapsi ok thanks... keep me posted

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

I forgot to say, the second screenshot shows only one Entry/ByteBuf expanded but the values are the same for the others (max capacity and writer index are 65k).

Contributor

rkapsi commented Jan 19, 2017

I forgot to say, the second screenshot shows only one Entry/ByteBuf expanded but the values are the same for the others (max capacity and writer index are 65k).

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi also can you tell me if its direct memory or heap memory that cause the problem ? And also what are the objects that take most space in the heap dump ?

Member

normanmaurer commented Jan 19, 2017

@rkapsi also can you tell me if its direct memory or heap memory that cause the problem ? And also what are the objects that take most space in the heap dump ?

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

By Object: https://i.imgsafe.org/0f22ec87c3.png
By Size: https://i.imgsafe.org/0f2335cb22.png

I'd be direct memory that causes the problem. Our JVM heap is configured to be 4GB and the dump file is only 2.3GB and YourKit says 1.8GB is effectively in use.

Contributor

rkapsi commented Jan 19, 2017

By Object: https://i.imgsafe.org/0f22ec87c3.png
By Size: https://i.imgsafe.org/0f2335cb22.png

I'd be direct memory that causes the problem. Our JVM heap is configured to be 4GB and the dump file is only 2.3GB and YourKit says 1.8GB is effectively in use.

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

Alright, found the commit: be8c16c

We do use mutual TLS but I don't quite understand what's happening there.

Memory usage after each deploy: https://i.imgsafe.org/109d30b4c9.png
Normal memory: https://i.imgsafe.org/109dd92c71.png
Bad memory: https://i.imgsafe.org/109d838047.png

Contributor

rkapsi commented Jan 19, 2017

Alright, found the commit: be8c16c

We do use mutual TLS but I don't quite understand what's happening there.

Memory usage after each deploy: https://i.imgsafe.org/109d30b4c9.png
Normal memory: https://i.imgsafe.org/109dd92c71.png
Bad memory: https://i.imgsafe.org/109d838047.png

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi thanks a lot... Not sure how this could lead to this but I will have a look shortly!

Member

normanmaurer commented Jan 19, 2017

@rkapsi thanks a lot... Not sure how this could lead to this but I will have a look shortly!

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi talking with @Scottmitch atm... @Scottmitch may be on something... will keep you posted

Member

normanmaurer commented Jan 19, 2017

@rkapsi talking with @Scottmitch atm... @Scottmitch may be on something... will keep you posted

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi also is it correct to assume that you create / destroy a lot of ReferenceCountedOpenSslServerContext instances ?

Member

normanmaurer commented Jan 19, 2017

@rkapsi also is it correct to assume that you create / destroy a lot of ReferenceCountedOpenSslServerContext instances ?

@Scottmitch

This comment has been minimized.

Show comment
Hide comment
@Scottmitch

Scottmitch Jan 19, 2017

Member

@rkapsi - Thanks for the commit reference ... I think I found a potential leak netty/netty-tcnative#220 .. I'm going to run Netty tests now but can you try this?

Member

Scottmitch commented Jan 19, 2017

@rkapsi - Thanks for the commit reference ... I think I found a potential leak netty/netty-tcnative#220 .. I'm going to run Netty tests now but can you try this?

Scottmitch added a commit to Scottmitch/netty-tcnative that referenced this issue Jan 19, 2017

SSL_CTX_add_client_CA does not take ownership of the X509 cert
Motivation:
OpenSSL's SSL_CTX_add_client_CA does not transfer ownership of the x509 certificate. Instead it just makes a copy of the subject name via X509_get_subject_name. We should always call X509_free after the call to SSL_CTX_add_client_CA.

Modifications:
- Call X509_free after calling SSL_CTX_add_client_CA

Result:
No more memory leak.
Fixes netty/netty#6249.
@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

@normanmaurer yes we do. Possibly by the millions per day.

Contributor

rkapsi commented Jan 19, 2017

@normanmaurer yes we do. Possibly by the millions per day.

@Scottmitch

This comment has been minimized.

Show comment
Hide comment
@Scottmitch

Scottmitch Jan 19, 2017

Member

@rkapsi - Netty unit tests pass with netty/netty-tcnative#220 ... based upon the commit you referenced I think this will help.

Member

Scottmitch commented Jan 19, 2017

@rkapsi - Netty unit tests pass with netty/netty-tcnative#220 ... based upon the commit you referenced I think this will help.

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

@Scottmitch excellent. Do you publish nightly SNAPSHOTs of tcnative anywhere? We depend on the linux-x86_64-fedora artifact and I have no easy way to build it.

Contributor

rkapsi commented Jan 19, 2017

@Scottmitch excellent. Do you publish nightly SNAPSHOTs of tcnative anywhere? We depend on the linux-x86_64-fedora artifact and I have no easy way to build it.

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi we do on oss.sonatype.org but not for PRs. Should I send you the jars via email ?

Member

normanmaurer commented Jan 19, 2017

@rkapsi we do on oss.sonatype.org but not for PRs. Should I send you the jars via email ?

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

@normanmaurer ok, that works - rkapsi at geeemail dot com

Contributor

rkapsi commented Jan 19, 2017

@normanmaurer ok, that works - rkapsi at geeemail dot com

Scottmitch added a commit to netty/netty-tcnative that referenced this issue Jan 19, 2017

SSL_CTX_add_client_CA does not take ownership of the X509 cert
Motivation:
OpenSSL's SSL_CTX_add_client_CA does not transfer ownership of the x509 certificate. Instead it just makes a copy of the subject name via X509_get_subject_name. We should always call X509_free after the call to SSL_CTX_add_client_CA.

Modifications:
- Call X509_free after calling SSL_CTX_add_client_CA

Result:
No more memory leak.
Fixes netty/netty#6249.
@Scottmitch

This comment has been minimized.

Show comment
Hide comment
@Scottmitch

Scottmitch Jan 19, 2017

Member

I will re-open until we verify there are no other issues.

Member

Scottmitch commented Jan 19, 2017

I will re-open until we verify there are no other issues.

@Scottmitch Scottmitch reopened this Jan 19, 2017

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@Scottmitch shouldnt we wait until @rkapsi verified it before closing it ?

Member

normanmaurer commented Jan 19, 2017

@Scottmitch shouldnt we wait until @rkapsi verified it before closing it ?

@Scottmitch

This comment has been minimized.

Show comment
Hide comment
@Scottmitch

Scottmitch Jan 19, 2017

Member

(it closed automatically bcz the tcnative commit referenced this issue)

Member

Scottmitch commented Jan 19, 2017

(it closed automatically bcz the tcnative commit referenced this issue)

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

lol... You beat me by a few secs ;)

Member

normanmaurer commented Jan 19, 2017

lol... You beat me by a few secs ;)

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member

@rkapsi you have mail :)

Member

normanmaurer commented Jan 19, 2017

@rkapsi you have mail :)

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

Testing...

Contributor

rkapsi commented Jan 19, 2017

Testing...

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 19, 2017

Member
Member

normanmaurer commented Jan 19, 2017

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 19, 2017

Contributor

@normanmaurer @Scottmitch victory! thx gentleman.

Contributor

rkapsi commented Jan 19, 2017

@normanmaurer @Scottmitch victory! thx gentleman.

@Scottmitch

This comment has been minimized.

Show comment
Hide comment
@Scottmitch

Scottmitch Jan 20, 2017

Member

great ... thanks for verifying!

Fixed by netty/netty-tcnative#220

Member

Scottmitch commented Jan 20, 2017

great ... thanks for verifying!

Fixed by netty/netty-tcnative#220

@Scottmitch Scottmitch closed this Jan 20, 2017

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 20, 2017

Member
Member

normanmaurer commented Jan 20, 2017

@doom369

This comment has been minimized.

Show comment
Hide comment
@doom369

doom369 Jan 20, 2017

Contributor

@normanmaurer does adding tcnative-snapshot would be enough to buiild? Or I should also rebuild netty with it in order to apply this patch?

Contributor

doom369 commented Jan 20, 2017

@normanmaurer does adding tcnative-snapshot would be enough to buiild? Or I should also rebuild netty with it in order to apply this patch?

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 20, 2017

Member

@doom369 just the snapshot... That said I just released netty-tcnative-1.1.33.Fork26 which includes the fix. Just waiting for maven central to sync before update the version in netty.

Member

normanmaurer commented Jan 20, 2017

@doom369 just the snapshot... That said I just released netty-tcnative-1.1.33.Fork26 which includes the fix. Just waiting for maven central to sync before update the version in netty.

@doom369

This comment has been minimized.

Show comment
Hide comment
@doom369

doom369 Jan 20, 2017

Contributor

@normanmaurer thanks, super. As I just reproduced same behaviour on small prod instance.

Contributor

doom369 commented Jan 20, 2017

@normanmaurer thanks, super. As I just reproduced same behaviour on small prod instance.

normanmaurer added a commit that referenced this issue Jan 20, 2017

Update netty-tcnative
Motivation:

We released a new netty-tcnative version as a memory leak was fixed.

Modifications:

Update netty-tcnative.

Result:

Fixes [#6249].
@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 20, 2017

Member

@doom369 already its on maven central now... See also: #6259

Member

normanmaurer commented Jan 20, 2017

@doom369 already its on maven central now... See also: #6259

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 20, 2017

Contributor

@normanmaurer FYI: I have now moved to 4.1.8-SNAPSHOT and Fork26. Things are good but that shutdown exception is still happening (but at a much lower velocity).

 javax.net.ssl.SSLException: error:140D00CF:SSL routines:SSL_write:protocol is shutdown
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:745) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:734) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:711) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:509) ~[?:1.8.0_102]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:764) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:596) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:568) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:549) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:530) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:355) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
...
$ md5sum netty-all-4.1.8.Final-SNAPSHOT.jar 
9b729fc5155999e5ea46c3ef889844fc  netty-all-4.1.8.Final-SNAPSHOT.jar
Contributor

rkapsi commented Jan 20, 2017

@normanmaurer FYI: I have now moved to 4.1.8-SNAPSHOT and Fork26. Things are good but that shutdown exception is still happening (but at a much lower velocity).

 javax.net.ssl.SSLException: error:140D00CF:SSL routines:SSL_write:protocol is shutdown
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:745) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:734) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:711) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:509) ~[?:1.8.0_102]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:764) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:596) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:568) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:549) ~[netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:530) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:355) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:750) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:777) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:769) [netty-all-4.1.8.Final-SNAPSHOT.jar:4.1.8.Final-SNAPSHOT]
...
$ md5sum netty-all-4.1.8.Final-SNAPSHOT.jar 
9b729fc5155999e5ea46c3ef889844fc  netty-all-4.1.8.Final-SNAPSHOT.jar
@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 20, 2017

Member

@rkapsi can you tell me what commit the snapshot is on ?

Member

normanmaurer commented Jan 20, 2017

@rkapsi can you tell me what commit the snapshot is on ?

@rkapsi

This comment has been minimized.

Show comment
Hide comment
@rkapsi

rkapsi Jan 20, 2017

Contributor

That is the current/most recent nightly.

netty-all-4.1.8.Final-20170120.120745-9.jar	Fri Jan 20 12:07:46 UTC 2017	3599129
Contributor

rkapsi commented Jan 20, 2017

That is the current/most recent nightly.

netty-all-4.1.8.Final-20170120.120745-9.jar	Fri Jan 20 12:07:46 UTC 2017	3599129
@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 20, 2017

Member

@rkapsi hmm... I will have a look ( can you open a new issue with the stack please ?)

Member

normanmaurer commented Jan 20, 2017

@rkapsi hmm... I will have a look ( can you open a new issue with the stack please ?)

@normanmaurer normanmaurer changed the title from error:140D00CF:SSL routines:SSL_write:protocol is shutdown = memory leak? to Memory leak when creating ReferenceCountedSslServerContext instances Jan 20, 2017

normanmaurer added a commit that referenced this issue Jan 20, 2017

Update netty-tcnative
Motivation:

We released a new netty-tcnative version as a memory leak was fixed.

Modifications:

Update netty-tcnative.

Result:

Fixes [#6249].

normanmaurer added a commit that referenced this issue Jan 20, 2017

Update netty-tcnative
Motivation:

We released a new netty-tcnative version as a memory leak was fixed.

Modifications:

Update netty-tcnative.

Result:

Fixes [#6249].
@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 20, 2017

Member

@rkapsi I created it as #6260 ... will investigate

Member

normanmaurer commented Jan 20, 2017

@rkapsi I created it as #6260 ... will investigate

liuzhengyang pushed a commit to liuzhengyang/netty that referenced this issue Sep 10, 2017

Update netty-tcnative
Motivation:

We released a new netty-tcnative version as a memory leak was fixed.

Modifications:

Update netty-tcnative.

Result:

Fixes [#6249].
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment