ZK Ledger Manager doesn't handle zookeeper session expires

[sijie]

BUG REPORT

Problem

@jvrao mentioned this at slack channel

Just trying to understand the current code in the following scenario:

Two clients
Client-1 writing a ledger
Client-2 tailing the same ledger (with lac updates etc)
client-2 lost connection with ZK
client-1 lost connection with bookies and replaced entire ensemble.
Now client-2 can't get ensemble change update as its lots its session
also client-2 can't read new LAC as all bookies that it is talking to have old LAC
and apparently we don't handle session expiry on readonly ledger handle watcher
so what happens?
the reader hangs for-ever?

The expected behavior : ledger manager will handle session expires and register watcher on the ledger metadata. However in current apache/master there is no session expire handling logic in ledger manager. This logic exists only at twitter/master.

This issue is to port that change back from twitter/master to apache/master.

[sijie]

/cc @jvrao

[sijie]

This has been addressed by #1130 in master.

It will be cherry-picked to branch-4.6 by #1161

[sijie]

The change has been cherry-picked to 4.6 as well.