ScalatraBroadcaster.broadcast() blows up with IllegalStateException when corresponding Jetty HttpSession is invalidated

[wkozaczuk]

Following exception get raised:

java.lang.IllegalStateException: null
        at org.eclipse.jetty.server.session.AbstractSession.checkValid(AbstractSession.java:110) ~[jetty-server-9.1.2.v20140210.jar:9.1.2.v20140210]
        at org.eclipse.jetty.server.session.HashedSession.checkValid(HashedSession.java:79) ~[jetty-server-9.1.2.v20140210.jar:9.1.2.v20140210]
        at org.eclipse.jetty.server.session.AbstractSession.getAttribute(AbstractSession.java:141) ~[jetty-server-9.1.2.v20140210.jar:9.1.2.v20140210]
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_05]
        at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_05]
        at org.scalatra.servlet.AttributesMap$class.get(AttributesMap.scala:26) ~[scalatra_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at org.scalatra.servlet.RichSession.get(RichSession.scala:9) ~[scalatra_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at org.scalatra.servlet.RichSession.get(RichSession.scala:9) ~[scalatra_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at scala.collection.MapLike$class.apply(MapLike.scala:140) ~[scala-library-2.10.4.jar:na]
        at org.scalatra.servlet.RichSession.apply(RichSession.scala:9) ~[scalatra_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at org.scalatra.atmosphere.package$$anon$1.client(package.scala:40) ~[scalatra-atmosphere_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_05]
        at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_05]
        at org.scalatra.atmosphere.ScalatraBroadcaster$$anonfun$1.apply(ScalatraBroadcaster.scala:21) ~[scalatra-atmosphere_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at org.scalatra.atmosphere.ScalatraBroadcaster$$anonfun$1.apply(ScalatraBroadcaster.scala:21) ~[scalatra-atmosphere_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[scala-library-2.10.4.jar:na]
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[scala-library-2.10.4.jar:na]
        at scala.collection.Iterator$class.foreach(Iterator.scala:727) ~[scala-library-2.10.4.jar:na]
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) ~[scala-library-2.10.4.jar:na]
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[scala-library-2.10.4.jar:na]
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[scala-library-2.10.4.jar:na]
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) ~[scala-library-2.10.4.jar:na]
        at scala.collection.AbstractTraversable.map(Traversable.scala:105) ~[scala-library-2.10.4.jar:na]
        at org.scalatra.atmosphere.ScalatraBroadcaster$class.broadcast(ScalatraBroadcaster.scala:21) ~[scalatra-atmosphere_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at org.scalatra.atmosphere.DefaultScalatraBroadcaster.broadcast(DefaultScalatraBroadcaster.scala:8) ~[scalatra-atmosphere_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at org.scalatra.atmosphere.AtmosphereClient$$anonfun$broadcast$1.apply(AtmosphereClient.scala:31) ~[scalatra-atmosphere_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at org.scalatra.atmosphere.AtmosphereClient$$anonfun$broadcast$1.apply(AtmosphereClient.scala:31) ~[scalatra-atmosphere_2.10-2.3.0.RC1.jar:2.3.0.RC1]
        at scala.Option.foreach(Option.scala:236) ~[scala-library-2.10.4.jar:na]
        at org.scalatra.atmosphere.AtmosphereClient$.broadcast(AtmosphereClient.scala:31) ~[scalatra-atmosphere_2.10-2.3.0.RC1.jar:2.3.0.RC1]

I think that the method atmoResourceWithClient (line 39 & 40) in org.scalatra.atmoshpere.package needs to be refined to take into account that underlying HttpSession may be invalid (for example http sessions in Jetty 9 have 30 minutes inactivity timeout). We also noticed that even if websocket client keeps sending data this does not touch http session and after 30 minutes it gets invalid and leads to IllegalStateException if subsequently server side code calls AtmosphereClient.broadcast to push data to the client. The worst part is that once this exception happens it affects all subsequent attempts to call AtmosphereClient.broadcast and the only way to solve it is to restart JVM.

I am not sure what the right solution in Scalatra is (or maybe it is problem with atmosphere) but currently we have following workaround:

  AtmosphereClient.lookup(BroadcastPath).foreach( _.getAtmosphereResources.asScala.toSet foreach(
    (resource:AtmosphereResource) => if( resource.getRequest.getSession(false) == null ) {
      logger.warn(s"Encountered atmosphere resource $resource associated with invalid session")
      resource.asInstanceOf[AtmosphereResourceImpl]._destroy()
    }))

before we call AtmosphereClient.broadcast(BroadcastPath, jsonMessage, clientFilter )

That takes care of removing AtmosphereResource instances associated with invalid http session. We also implemented simple keepAlive mechanism where websocket would periodically send keep alive message and following code on server side would be executed:

  AtmosphereClient.lookup(BroadcastPath).foreach( (broadcaster:ScalatraBroadcaster) => {
    val myResources = broadcaster.getAtmosphereResources.asScala filter { _.uuid() == uuid }
    myResources foreach( (resource:AtmosphereResource) => {
      val session = resource.getRequest.getSession(false)
      session.setMaxInactiveInterval(session.getMaxInactiveInterval + 900)
      logger.info(s"Extended life of session ${session.getId} for another 15 minutes " +
        s"maximum inactivity period in seconds of ${session.getMaxInactiveInterval}")
    })
  })

This is not ideal but works. Another solution would be to change setting on Jetty side to keep http session for infinite amount time. But it seems there is some fundamental issue here and it should be fixed in more elegant way in Scalatra.

[casualjim]

I'm not entirely sure if this is possible but I would think that when a websocket connection is opened the session in jetty should indeed be alive forever for that session id.
Thank you for the workaround, I'll add it to scalatra for 2.3.0