[Tomcat BIO] Thread Count Explosion

[ironmike]

Similar to #716, #717, #766, I am seeing the same behaviours. Version of Atmosphere is 1.0.12, configuration is set to IDLE_DESTROY, timeout the default of 300000ms (5mins).

Server creeps up to about 300 threads in WAITING state, and then refuses any more connections.

Each thread from the dump appears as follows:

"ajp-bio-8009-exec-5297" daemon prio=10 tid=0x00007fbe24333800 nid=0x72e9 waiting on condition [0x00007fbe044a4000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000000068c2db968> (a java.util.concurrent.CountDownLatch$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
    at org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
    at org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
    at org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
    at org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:87)
    at org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
    at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1342)
    at org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:298)
    at org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:284)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at com.dms.customer.CustomerTokenFilter$1.run(CustomerTokenFilter.java:51)
    at com.dms.customer.CustomerTransactionRunner$1.call(CustomerTransactionRunner.java:31)
    at com.dms.customer.CustomerTransactionRunner$1.call(CustomerTransactionRunner.java:29)
    at com.dms.customer.CustomerTransactionRunner.execute(CustomerTransactionRunner.java:61)
    at com.dms.customer.CustomerTransactionRunner.execute(CustomerTransactionRunner.java:29)
    at com.dms.customer.CustomerUtils.executeCustomerTransaction(CustomerUtils.java:80)
    at com.dms.customer.CustomerTokenFilter.doFilter(CustomerTokenFilter.java:49)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
    - locked <0x000000068bbe7ce0> (a org.apache.tomcat.util.net.SocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
   Locked ownable synchronizers:
    - 0x00000006873ff130> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Broadcaster lookup is done via:

    Broadcaster broadcaster = BroadcasterFactory.getDefault().lookup(id, true);

and when debugging does return the same Broadcaster, so it doesn't appear that the Broadcasters are multiplying, just the servlets not cleaning up and going away properly.

[jfarcand]

It's expected since you are using Blocking I/O and Tomcat. One thread will be used for every connection. You need to use NIO or APR if you don't want that behavior

[ironmike]

But it clearly states on startup that it is using Tomcat7AsyncSupportWithWebSocket, unless I am missing something?

08:26:25.223 [localhost-startStop-1] WARN o.atmosphere.cpr.AtmosphereFramework

• Missing META-INF/atmosphere.xml but found the Jersey runtime. Starting Jersey
  08:26:25.239 [localhost-startStop-1] INFO o.atmosphere.cpr.AtmosphereFramework
• Installed AtmosphereHandler org.atmosphere.handler.ReflectorServletProcessor mapped to context-path: /*
  08:26:25.240 [localhost-startStop-1] INFO o.atmosphere.cpr.AtmosphereFramework
• Auto detecting WebSocketHandler in /WEB-INF/classes/
  08:26:25.241 [localhost-startStop-1] INFO o.atmosphere.cpr.AtmosphereFramework
• Installed WebSocketProtocol org.atmosphere.websocket.protocol.SimpleHttpProtocol
  08:26:25.251 [localhost-startStop-1] INFO o.atmosphere.cpr.AtmosphereFramework- Atmosphere is using async support: org.atmosphere.container.Tomcat7AsyncSupportWithWebSocket running under container: Apache Tomcat/7.0.29
  08:26:25.256 [localhost-startStop-1] INFO o.a.h.ReflectorServletProcessor - Installing Servlet com.sun.jersey.spi.container.servlet.ServletContainer

[ironmike]

Is this something I have misconfigured in Atmosphere, or at the Tomcat level?

[jfarcand]

Your configuration of Tomcat is wrong. Take a look at their documentation about configuring connector.

[ironmike]

Ok, so by switching to org.apache.coyote.http11.Http11NioProtocol I shouldn't see this threads kicking around when I have IDLE_DESTROY as the BroadcasterLifecyclePolicy? Or at least see them for a while but they should actually be terminated at the 300s mark after idle?

[jfarcand]

If you still see

 at org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
    at org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)

that means your configuration is still wrong. For threads, setup yourself the Executors and set them to clear their threads faster. Please jump on the ML for more discussion as this is not a bug. Thanks