IO worker threads BLOCKED on SSL Session Cache while CMS full gc #832
Comments
|
@redstrato this really cracks me up... But I think there is nothing we can do about it except have the user set the cache to a "sane" level. :/ |
|
You can configure the session cache timeout to a lower value than the default to make SSL sessions expire sooner for earlier garbage collection. |
|
@trustin yes that was what I tried to say :) |
|
thanks for reply. :-) session cache timeout (sessionTimeout) 's default value is 86400s (24hr), and I reduced to 60s thanks shoud I close this? |
normanmaurer
pushed a commit
that referenced
this issue
Dec 19, 2012
normanmaurer
pushed a commit
that referenced
this issue
Dec 19, 2012
ghost
assigned
|
@redstrato @trustin I added some text to the javadocs of SslHandler to let the user know about this. So let us close it as there is nothing we can do about it... |
Closed
Closed
stepheno
added a commit
to urbanairship/pushy
that referenced
this issue
May 22, 2017
Default caching behavior is not desirable. See netty/netty#832, and the default netty ssl implementation in https://github.com/AsyncHttpClient/async-http-client/
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Jul 17, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size - SSL session timeout - SSL handshake timeout - SSL close notify flush timeout
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Jul 17, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size - SSL session timeout - SSL handshake timeout - SSL close notify flush timeout
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Jul 17, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size - SSL session timeout - SSL handshake timeout - SSL close notify flush timeout
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Jul 17, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size - SSL session timeout - SSL handshake timeout - SSL close notify flush timeout
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Jul 18, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size - SSL session timeout - SSL handshake timeout - SSL close notify flush timeout
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Jul 19, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size - SSL session timeout - SSL handshake timeout - SSL close notify flush timeout
NicoK
pushed a commit
to dataArtisans/flink
that referenced
this issue
Aug 1, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size - SSL session timeout - SSL handshake timeout - SSL close notify flush timeout
NicoK
pushed a commit
to dataArtisans/flink
that referenced
this issue
Aug 1, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size - SSL session timeout - SSL handshake timeout - SSL close notify flush timeout
NicoK
pushed a commit
to dataArtisans/flink
that referenced
this issue
Aug 8, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size - SSL session timeout - SSL handshake timeout - SSL close notify flush timeout
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Aug 13, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.session-cache-size' - SSL session timeout via 'security.ssl.session-timeout' - SSL handshake timeout via 'security.ssl.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.close-notify-flush-timeout'
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Aug 13, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.session-cache-size' - SSL session timeout via 'security.ssl.session-timeout' - SSL handshake timeout via 'security.ssl.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.close-notify-flush-timeout'
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Aug 14, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.session-cache-size' - SSL session timeout via 'security.ssl.session-timeout' - SSL handshake timeout via 'security.ssl.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.close-notify-flush-timeout'
NicoK
added a commit
to apache/flink
that referenced
this issue
Aug 20, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.session-cache-size' - SSL session timeout via 'security.ssl.session-timeout' - SSL handshake timeout via 'security.ssl.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.close-notify-flush-timeout' This closes #6355.
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Oct 13, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.session-cache-size' - SSL session timeout via 'security.ssl.session-timeout' - SSL handshake timeout via 'security.ssl.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.close-notify-flush-timeout'
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Oct 13, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.session-cache-size' - SSL session timeout via 'security.ssl.session-timeout' - SSL handshake timeout via 'security.ssl.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.close-notify-flush-timeout'
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Oct 13, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.internal.session-cache-size' - SSL session timeout via 'security.ssl.internal.session-timeout' - SSL handshake timeout via 'security.ssl.internal.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.internal.close-notify-flush-timeout'
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Oct 13, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.internal.session-cache-size' - SSL session timeout via 'security.ssl.internal.session-timeout' - SSL handshake timeout via 'security.ssl.internal.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.internal.close-notify-flush-timeout'
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Oct 22, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.internal.session-cache-size' - SSL session timeout via 'security.ssl.internal.session-timeout' - SSL handshake timeout via 'security.ssl.internal.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.internal.close-notify-flush-timeout'
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Oct 22, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.internal.session-cache-size' - SSL session timeout via 'security.ssl.internal.session-timeout' - SSL handshake timeout via 'security.ssl.internal.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.internal.close-notify-flush-timeout'
NicoK
pushed a commit
to NicoK/flink
that referenced
this issue
Oct 22, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.internal.session-cache-size' - SSL session timeout via 'security.ssl.internal.session-timeout' - SSL handshake timeout via 'security.ssl.internal.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.internal.close-notify-flush-timeout' This closes apache#6895.
NicoK
pushed a commit
to apache/flink
that referenced
this issue
Oct 22, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.internal.session-cache-size' - SSL session timeout via 'security.ssl.internal.session-timeout' - SSL handshake timeout via 'security.ssl.internal.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.internal.close-notify-flush-timeout' This closes #6895.
NicoK
pushed a commit
to apache/flink
that referenced
this issue
Oct 22, 2018
This is mostly to tackle bugs like netty/netty#832 (JDK issue during garbage collection when the SSL session cache is not limited). We add the following low-level configuration options for the user to fine-tune their system: - SSL session cache size via 'security.ssl.internal.session-cache-size' - SSL session timeout via 'security.ssl.internal.session-timeout' - SSL handshake timeout via 'security.ssl.internal.handshake-timeout' - SSL close notify flush timeout via 'security.ssl.internal.close-notify-flush-timeout' This closes #6838.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It seems a jdk issue. but...
while testing on 60,000 connections, 300 reconnects/s, 4000 rw/s
IO worker threads were BLOCKED for 10s ~ 3 minutes
oracle jdk 1.6.0_33 64bit
netty-3.5.10
we found SSLEngine's ssl session cache is a problem via thread dump.
Removing huge amount of ssl session from cache while full gc seems cause IO Thread owned lock delayed, and other IO threads BLOCKED.
but jdk's ssl session cache couldn't be disabled
so we set SSLSessionContext's ssl session cache size as 1 and session timeout as 60s
SSLContext context = ...;
context.getServerSessionContext().setSessionCacheSize(sessionCahceSize);
context.getServerSessionContext().setSessionTimeout(sessionTimeoutSeconds);
it seems working. but still I'm not sure.
for verifying, I'm running on a long term test
-thread dump-
[BLOCKING thread]
"New I/O worker #69" - Thread t@88
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.removeEntryForKey(HashMap.java:569)
at java.util.HashMap.remove(HashMap.java:538)
at sun.security.util.MemoryCache.emptyQueue(Cache.java:274)
at sun.security.util.MemoryCache.put(Cache.java:336)
- locked <7a41228e> (a sun.security.util.MemoryCache)
at com.sun.net.ssl.internal.ssl.SSLSessionContextImpl.put(SSLSessionContextImpl.java:162)
at com.sun.net.ssl.internal.ssl.ServerHandshaker.clientFinished(ServerHandshaker.java:1245)
at com.sun.net.ssl.internal.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:224)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:976)
- locked <33178df1> (a com.sun.net.ssl.internal.ssl.SSLEngineImpl)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:845)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
- locked <4f67e36b> (a java.lang.Object)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1118)
- locked <35f530e> (a java.lang.Object)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:814)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:422)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- locked <2d83ae87> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
[BLOCKED thread; 47 threads are BLOCKED]
"New I/O worker #70" - Thread t@89
java.lang.Thread.State: BLOCKED
at sun.security.util.MemoryCache.put(Cache.java:336)
- waiting to lock <7a41228e> (a sun.security.util.MemoryCache) owned by "New I/O worker #69" t@88
at com.sun.net.ssl.internal.ssl.SSLSessionContextImpl.put(SSLSessionContextImpl.java:162)
at com.sun.net.ssl.internal.ssl.ServerHandshaker.clientFinished(ServerHandshaker.java:1245)
at com.sun.net.ssl.internal.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:224)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:976)
- locked <675628ce> (a com.sun.net.ssl.internal.ssl.SSLEngineImpl)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:845)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
- locked <49912162> (a java.lang.Object)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1118)
- locked <5726cea9> (a java.lang.Object)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:814)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:422)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- locked <34c9e5f2> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
The text was updated successfully, but these errors were encountered: