bookie exited and Error contacting zookeeper

[xuesongxs]

Describe the bug

22:38:01.722 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - UnavailableException while replicating fragments
org.apache.bookkeeper.replication.ReplicationException$UnavailableException: Error contacting zookeeper
        at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.isLedgerReplicationEnabled(ZkLedgerUnderreplicationManager.java:728) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.waitIfLedgerReplicationDisabled(ZkLedgerUnderreplicationManager.java:619) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicate(ZkLedgerUnderreplicationManager.java:600) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:272) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
        at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:238) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.45.Final.jar:4.1.45.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ledgers/underreplication/disable
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[org.apache.pulsar-pulsar-zookeeper-2.5.1.jar:2.5.1]
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[org.apache.pulsar-pulsar-zookeeper-2.5.1.jar:2.5.1]
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:2021) ~[org.apache.pulsar-pulsar-zookeeper-2.5.1.jar:2.5.1]
        at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2301(ZooKeeperClient.java:70) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]

Additional context
Add any other context about the problem here.

[Ghatage]

@xuesong172500
From the looks of it, the auto recovery process failed to replicate underreplicated ledgers because it was unable to contact Zookeeper.

Can you please give us more information as to why the connections might be failing?
Is the Zookeeper process up?
If it is up and you still got this, is there a Zookeeper log that we can check?

[f-ld]

Dropping a comment here since I got similar issue and fixed it. In my cases logs were like:

11:43:49.562 [main-EventThread] ERROR org.apache.bookkeeper.proto.BookkeeperInternalCallbacks - Error in multi callback : -9
11:43:49.562 [main-EventThread] ERROR org.apache.bookkeeper.meta.AbstractZkLedgerManager - Could not read metadata for ledger: 1952798
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ledgers/00/0195/L2798
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[org.apache.pulsar-pulsar-zookeeper-2.6.4.jar:2.6.4]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[org.apache.pulsar-pulsar-zookeeper-2.6.4.jar:2.6.4]
	at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:448) [org.apache.bookkeeper-bookkeeper-server-4.11.1.jar:4.11.1]
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) [org.apache.bookkeeper-bookkeeper-server-4.11.1.jar:4.11.1]
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:601) [org.apache.pulsar-pulsar-zookeeper-2.6.4.jar:2.6.4]
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [org.apache.pulsar-pulsar-zookeeper-2.6.4.jar:2.6.4]

And the solution was to allocate more memory/cpu to recovery instances.

[tisonkun]

Closed as answered. Thanks for sharing your use case @f-ld!