Getting threads blocked

[strokine]

Hello, under normal load everything works just fine, but when server gets a lot of traffic I'm getting response time spikes. While I'm analyzing the thread dump I see a lot of blocked threads, which I don't see at all during no spike time, even under heavy load:

qtp1331498204-743" prio=10 tid=0x00007f68b02ca800 nid=0x324e waiting for monitor entry [0x00007f685f977000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.security.Provider.getService(Provider.java:680)
    - waiting to lock <0x0000000787dd34c8> (a sun.security.provider.Sun)
    at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:436)
    at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375)
    at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485)
    at java.security.Signature.getInstance(Signature.java:223)
    at io.jsonwebtoken.impl.crypto.RsaProvider.newSignatureInstance(RsaProvider.java:73)
    at io.jsonwebtoken.impl.crypto.RsaProvider.createSignatureInstance(RsaProvider.java:38)
    at io.jsonwebtoken.impl.crypto.RsaSignatureValidator.isValid(RsaSignatureValidator.java:40)
    at io.jsonwebtoken.impl.crypto.DefaultJwtSignatureValidator.isValid(DefaultJwtSignatureValidator.java:47)
    at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:283)
    at com.mycomp.JjwtServiceAccessTokenDeserializer.deserialize(JjwtServiceAccessTokenDeserializer.java:25)
    at com.mycomp.ServiceAccessToken.deserialize(ServiceAccessToken.java:248)
    at com.mycomp.SatFilter.doFilter(SatFilter.java:66)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:744)

I'm not sure what could cause the problem, but if you know, please fix it.

Thanks for the library anyway,
Eugene

[lhazlewood]

This has nothing to do with JJWT - JJWT is completely stateless and does not manage threads or locks itself anywhere. You're apparently hitting what looks to be like a design flaw in the sun.security.jca.ProviderList or maybe the java.security.Provider.getService implementation. What JDK are you using?

[strokine]

Right, looks like this is not a problem of your library, but JDK:
http://bugs.java.com/view_bug.do?bug_id=7092821
It looks like Provider.getService method is synchronized.
We are using Java 7.
But maybe there is a work around of this JDK bug. For example someone did this:
http://stackoverflow.com/questions/27944331/why-is-the-java-method-provider-getservicestring-type-string-algorithm-synchr
I guess this is not an option in your case, but just to get inspiration maybe?

[lhazlewood]

Could you please offer a pull request for the fix?

[lhazlewood]

Also, what SDK are you using? Oracle JDK? Open JDK?

[lhazlewood]

@strokine Could you please indicate which SDK you were using? Also, did you try using BouncyCastle to see if that helped?

[strokine]

Oracle JDK. No, we didn't use BouncyCastle.

[lhazlewood]

Closing since there's not much we can do here (it's a JDK implementation issue). There is a workaround identified in the comments of http://bugs.java.com/view_bug.do?bug_id=7092821.