Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Occasional deadlock Lift + Jetty #1455

Closed
serioga opened this Issue · 3 comments

3 participants

@serioga

Lift 2.5-M4
Jetty 7.6.9.v20130131
Discussion https://groups.google.com/d/msg/liftweb/RyDjZ_2It2o/XDgDA3EuT6sJ

I rarely get a lock (in several thread pairs, CPU starts eating, jetty restart required, happened 2 or 3 times after upgrade to 2.5-M4, not clear how to reproduce) like:


"qtp1988368988-4348" - Thread t@4348
   java.lang.Thread.State: BLOCKED
    at net.liftweb.http.LiftSession.fixSessionTime(LiftSession.scala:796)
    - waiting to lock <62789d78> (a net.liftweb.http.LiftSession) owned by "qtp1988368988-3601" t@3601
    at net.liftweb.http.LiftRules.net$liftweb$http$LiftRules$$_getLiftSession(LiftRules.scala:281)
    at net.liftweb.http.LiftRules$$anonfun$11.apply(LiftRules.scala:250)
    at net.liftweb.http.LiftRules$$anonfun$11.apply(LiftRules.scala:250)
    at net.liftweb.http.LiftServlet.getLiftSession(LiftServlet.scala:92)
    at net.liftweb.http.LiftServlet.doService(LiftServlet.scala:293)
    at net.liftweb.http.LiftServlet$$anonfun$doIt$1$1.apply(LiftServlet.scala:145)
    at net.liftweb.http.LiftServlet$$anonfun$doIt$1$1.apply(LiftServlet.scala:144)
    at net.liftweb.util.TimeHelpers$class.calcTime(TimeHelpers.scala:345)
    at net.liftweb.util.Helpers$.calcTime(Helpers.scala:34)
    at net.liftweb.util.TimeHelpers$class.logTime(TimeHelpers.scala:364)
    at net.liftweb.util.Helpers$.logTime(Helpers.scala:34)
    at net.liftweb.http.LiftServlet.doIt$1(LiftServlet.scala:144)
    at net.liftweb.http.LiftServlet.service(LiftServlet.scala:155)
    at net.liftweb.http.provider.HTTPProvider$$anonfun$service$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply$mcV$sp(HTTPProvider.scala:72)
    at net.liftweb.http.provider.HTTPProvider$$anonfun$service$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(HTTPProvider.scala:71)
    at net.liftweb.http.provider.HTTPProvider$$anonfun$service$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(HTTPProvider.scala:71)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.URLRewriter$.doWith(Req.scala:1278)
    at net.liftweb.http.provider.HTTPProvider$$anonfun$service$2$$anonfun$apply$mcV$sp$1.apply$mcV$sp(HTTPProvider.scala:70)
    at net.liftweb.http.provider.HTTPProvider$$anonfun$service$2$$anonfun$apply$mcV$sp$1.apply(HTTPProvider.scala:70)
    at net.liftweb.http.provider.HTTPProvider$$anonfun$service$2$$anonfun$apply$mcV$sp$1.apply(HTTPProvider.scala:70)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.provider.HTTPProvider$$anonfun$service$2.apply$mcV$sp(HTTPProvider.scala:66)
    at net.liftweb.http.provider.HTTPProvider$$anonfun$service$2.apply(HTTPProvider.scala:60)
    at net.liftweb.http.provider.HTTPProvider$$anonfun$service$2.apply(HTTPProvider.scala:60)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.provider.HTTPProvider$class.service(HTTPProvider.scala:60)
    at net.liftweb.http.LiftFilter.service(LiftServlet.scala:922)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun$doFilter$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply$mcV$sp(ServletFilterProvider.scala:82)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun$doFilter$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(ServletFilterProvider.scala:82)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun$doFilter$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(ServletFilterProvider.scala:82)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$class.handleLoan$1(ServletFilterProvider.scala:59)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$class.net$liftweb$http$provider$servlet$ServletFilterProvider$$handleLoanWrappers(ServletFilterProvider.scala:63)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun$doFilter$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ServletFilterProvider.scala:82)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun$doFilter$1$$anonfun$apply$mcV$sp$1.apply(ServletFilterProvider.scala:77)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun$doFilter$1$$anonfun$apply$mcV$sp$1.apply(ServletFilterProvider.scala:77)
    at net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$14$$anonfun$apply$15$$anonfun$apply$16$$anonfun$apply$17.apply(Vars.scala:619)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$14$$anonfun$apply$15$$anonfun$apply$16.apply(Vars.scala:618)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$14$$anonfun$apply$15.apply(Vars.scala:617)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$14.apply(Vars.scala:616)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.CoreRequestVarHandler$class.apply(Vars.scala:615)
    at net.liftweb.http.RequestVarHandler$.apply(Vars.scala:515)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun$doFilter$1.apply$mcV$sp(ServletFilterProvider.scala:75)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun$doFilter$1.apply(ServletFilterProvider.scala:75)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$$anonfun$doFilter$1.apply(ServletFilterProvider.scala:75)
    at net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$14$$anonfun$apply$15$$anonfun$apply$16$$anonfun$apply$17.apply(Vars.scala:619)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$14$$anonfun$apply$15$$anonfun$apply$16.apply(Vars.scala:618)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$14$$anonfun$apply$15.apply(Vars.scala:617)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.CoreRequestVarHandler$$anonfun$apply$14.apply(Vars.scala:616)
    at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
    at net.liftweb.http.CoreRequestVarHandler$class.apply(Vars.scala:615)
    at net.liftweb.http.TransientRequestVarHandler$.apply(Vars.scala:528)
    at net.liftweb.http.provider.servlet.ServletFilterProvider$class.doFilter(ServletFilterProvider.scala:74)
    at net.liftweb.http.LiftFilter.doFilter(LiftServlet.scala:922)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:532)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:367)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - None

"pool-10-thread-15" - Thread t@121
   java.lang.Thread.State: BLOCKED
    at net.liftweb.http.LiftSession.exitComet(LiftSession.scala:683)
    - waiting to lock <7ff773cc> (a net.liftweb.http.LiftSession) owned by "pool-9-thread-16" t@110
    at net.liftweb.http.LiftServlet$ContinuationActor$$anonfun$messageHandler$1.applyOrElse(LiftServlet.scala:683)

    at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
    at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
    at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
    at net.liftweb.actor.LiftActor$class.execTranslate(LiftActor.scala:440)
    at net.liftweb.http.LiftServlet$ContinuationActor.execTranslate(LiftServlet.scala:662)
    at net.liftweb.actor.SpecializedLiftActor$class.liftedTree2$1(LiftActor.scala:288)
    at net.liftweb.actor.SpecializedLiftActor$class.net$liftweb$actor$SpecializedLiftActor$$proc2(LiftActor.scala:287)
    at net.liftweb.actor.SpecializedLiftActor$$anonfun$net$liftweb$actor$SpecializedLiftActor$$processMailbox$1.apply$mcV$sp(LiftActor.scala:210)
    at net.liftweb.actor.SpecializedLiftActor$$anonfun$net$liftweb$actor$SpecializedLiftActor$$processMailbox$1.apply(LiftActor.scala:210)
    at net.liftweb.actor.SpecializedLiftActor$$anonfun$net$liftweb$actor$SpecializedLiftActor$$processMailbox$1.apply(LiftActor.scala:210)
    at net.liftweb.actor.SpecializedLiftActor$class.around(LiftActor.scala:224)
    at net.liftweb.http.LiftServlet$ContinuationActor.around(LiftServlet.scala:662)
    at net.liftweb.actor.SpecializedLiftActor$class.net$liftweb$actor$SpecializedLiftActor$$processMailbox(LiftActor.scala:209)
    at net.liftweb.actor.SpecializedLiftActor$$anonfun$2$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiftActor.scala:173)
    at net.liftweb.actor.LAScheduler$$anonfun$9$$anon$2$$anon$3.run(LiftActor.scala:64)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - locked <5ccd8418> (a java.util.concurrent.ThreadPoolExecutor$Worker)
@dpp dpp closed this in afc9487
@davewhittaker

Should we roll a new 2.5 RC for this?

@dpp
Owner

Let's see if the OP sees it as a fix. If so, I am +1 on a new RC

@serioga

Full thread dump for my case is available in this message https://groups.google.com/d/msg/liftweb/RyDjZ_2It2o/kS6utswHNLUJ

@dpp dpp referenced this issue from a commit
@dpp dpp Closes #1455. Finer grained sync 701ce71
@dpp dpp referenced this issue from a commit
@dpp dpp Closes #1455. Finer grained sync a23b88b
@steven-sortable steven-sortable referenced this issue from a commit in sortable/framework
@steven-sortable steven-sortable Merge tag '2.5.3-release' of https://github.com/lift/framework into d…
…epress

* tag '2.5.3-release' of https://github.com/lift/framework: (409 commits)
  Prepare for 2.5.3
  build adj
  [PATCH] Lock down SecurityHelpers.secureXML further.
  Prepare for 2.5.2
  Fixed minor patch error small changes to publishing script
  Provide a secure XML parser in SecurityHelpers, use it throughout.
  Prepare for 2.5.1
  minor adjustments to unsafePublish.sh
  Replaced many HashMap with ConcurrentHashMap
  Fixed a real deadlock
  Closes #1455. Finer grained sync
  change version to 2.5.1-SNAPSHOT until we get all the changes we want in 2.5.1
  Fix an edge case for HTML5 -> ByteStream
  Prepare for 2.5
  updated publishing script to build from last RC
  Prepare for 2.5-RC6
  Fix for Issue #1442. NPE in compactRender for List with null values
  Deprecate XmlMenu
  None should result in JNothing rather than JNull to optimize the rendered JSON. * Also bumped the reserved code cache size to prevent errors when building locally.
  Made a bunch of methods public so they can be used from user-code:
  ...

Conflicts:
	project/build.properties
	project/plugins/Plugins.scala
aa8121a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.