Incorrect SessionExpiredError when ZooKeeper leader process down

[bwtakacy]

I encountered SessionExpiredError when ZooKeeper leader process down with below stack trace, but there was no session expiration at that time in ZooKeeper server logs:

Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/home/postgres/zha/zha.py", line 284, in run
    self.in_elector_loop()
  File "/home/postgres/zha/zha.py", line 303, in in_elector_loop
    lock_result = self.lock.acquire(timeout=self.zha.config.get("elector_interval",3))
  File "/usr/lib/python2.6/site-packages/kazoo-2.2.1-py2.6.egg/kazoo/recipe/lock.py", line 174, in acquire
    self._delete_node(self.node)
  File "/usr/lib/python2.6/site-packages/kazoo-2.2.1-py2.6.egg/kazoo/recipe/lock.py", line 266, in _delete_node
    self.client.delete(self.path + "/" + node)
  File "/usr/lib/python2.6/site-packages/kazoo-2.2.1-py2.6.egg/kazoo/client.py", line 1295, in delete
    return self.delete_async(path, version).get()
  File "/usr/lib/python2.6/site-packages/kazoo-2.2.1-py2.6.egg/kazoo/handlers/utils.py", line 72, in get
    raise self._exception
SessionExpiredError

, where zha.py is a tools using kazoo.

After debugging, I have found that _call() of KazooClient raised this error when KeeperState became CONNECTING and this is what happened in my environment.

        elif self._state in (KeeperState.EXPIRED_SESSION,
                             KeeperState.CONNECTING):
            async_object.set_exception(SessionExpiredError())
            return False

Is there some historical reason about this?
Is it OK to change the condition will catch only the KeeperState.EXPIRED_SESSION ?

[harlowja]

Said change seems ok although I'm not sure on the history of why it checks against both.

[bwtakacy]

Thank you for take a look!
I will create a patch and test it.

[bwtakacy]

I tried to remove checking KeeperState.CONNECTIONG, but tests failed because the below test in test_client.py assumes that.

 468     def test_create_on_broken_connection(self):
 469         client = self.client
 470         client.start()
 471 
 472         client._state = KeeperState.EXPIRED_SESSION
 473         self.assertRaises(SessionExpiredError, client.create,
 474                           '/closedpath', b'bar')
 475 
 476         client._state = KeeperState.AUTH_FAILED
 477         self.assertRaises(AuthFailedError, client.create,
 478                           '/closedpath', b'bar')
 479 
 480         client._state = KeeperState.CONNECTING
 481         self.assertRaises(SessionExpiredError, client.create,
 482                           '/closedpath', b'bar')

Hmm.

By the way, when I also remove the assertion of case of KeeperState.CONNECTING, the all of rest tests are passed.

[eric-spriggy]

I've also experienced the same as @bwtakacy. Any ideas what's the history of checking for CONNECTING and then raising a SessionExpiredError?

It was introduced in this commit: 4eabbd1

[suyogmapara]

@bbangert Can you please comment on this?

[bbangert]

Offhand I can't recall why its returning expired when it isn't expired.

[adityagujral]

Was there any consensus about fixing the issue?