namerd - "session expired"

[JonathanBennett]

Error message gets thrown over and over in the logs and gets raised as a HTTP 502 to the client.

E 0704 16:09:06.818 THREAD22: service failure Failure(session expired, flags=0x100000000) with NoSources at com.twitter.finagle.NoStacktrace(Unknown Source)

From @olix0r :
fwiw we it would ​_seem_​ like a zk exception:
:; git grep -lF session\ expired finagle-serversets/src/main/scala/com/twitter/finagle/serverset2/ZkSession.scala
but we probably should try to prevent that from bubbling up into the request flow if we can

[adleong]

Hey @JonathanBennett, I'm having trouble reproducing this. As I understand it, namerd's zk client sends a heartbeat to keep the zk session alive. Do you have any idea what could cause the session to expire? By default, the client requests a 10 second session timeout.

Is the error message you pasted from linkerd or namerd? If it's from linkerd, is there anything interesting in the namerd logs?

[JonathanBennett]

Hey,
I don't have a reason why zk would have failed to respond to a session. Rather annoyingly due to our log rotation in dev I don't have the logs for this as it looks to have been failing for quite a while so I'll be on the lookout to try and reproduce it; only thing I can think of is maybe ZK was underload and therefore failed to respond but I would imagine that as unlikely.

In such a case how should namerd recover? Should I be restarting namerd or should it be more resilient than this in failing from a zk timeout? From my logs I do have (about 4 days of failures over a weekend) I just see all lookups/sets failing with an error similar to the following:

E 0701 01:47:17.364 THREAD25: binding name /http/1.1/GET/zeppelin.xxxxx.co
java.lang.Exception: session expired
    at com.twitter.finagle.serverset2.buoyant.ZkSession$$anonfun$watchedOperation$1$$anonfun$com$twitter$finagle$serverset2$buoyant$ZkSession$$anonfun$$loop$2$1$$anonfun$2.apply(ZkSession.scala:162)
    at com.twitter.finagle.serverset2.buoyant.ZkSession$$anonfun$watchedOperation$1$$anonfun$com$twitter$finagle$serverset2$buoyant$ZkSession$$anonfun$$loop$2$1$$anonfun$2.apply(ZkSession.scala:142)
    at com.twitter.util.Witness$$anon$17.notify(Event.scala:434)
    at com.twitter.util.Var$$anon$3$$anonfun$register$1.apply(Var.scala:99)
    at com.twitter.util.Var$$anon$3$$anonfun$register$1.apply(Var.scala:99)
    at com.twitter.util.Var$Observer.publish(Var.scala:151)
    at com.twitter.util.UpdatableVar$$anonfun$update$2.apply(Var.scala:474)
    at com.twitter.util.UpdatableVar$$anonfun$update$2.apply(Var.scala:469)
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:778)
    at scala.collection.immutable.RedBlackTree$._foreachKey(RedBlackTree.scala:109)
    at scala.collection.immutable.RedBlackTree$.foreachKey(RedBlackTree.scala:105)
    at scala.collection.immutable.TreeSet.foreach(TreeSet.scala:154)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:777)
    at com.twitter.util.UpdatableVar.update(Var.scala:469)
    at com.twitter.finagle.serverset2.client.EventDeliveryThread$.run(EventDeliveryThread.scala:18)``` 

[adleong]

namerd should definitely recover by retrying on a new session. I have a change that I suspect will fix it, but without being able to reproduce the error it's hard to be sure.