Enable reorder read sequence for bk client by default

[hangc0276]

Motivation

If one ledger's ensemble is [bk0, bk1] and bk0 is down, the bookie client may send a read request to bk0 first then fail with the following errors, and resend the read request to bk1 in the end.

2023-10-19T18:33:52,042 - ERROR - [BookKeeperClientWorker-OrderedExecutor-3-0:PerChannelBookieClient@563] - Cannot connect to 192.168.31.216:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 192.168.31.216:3181, bookie does not exist or it is not running
2023-10-19T18:33:52,042 - INFO  - [BookKeeperClientWorker-OrderedExecutor-3-0:DefaultBookieAddressResolver@77] - Cannot resolve 192.168.31.216:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2023-10-19T18:33:52,042 - INFO  - [BookKeeperClientWorker-OrderedExecutor-3-0:PendingReadOp$LedgerEntryRequest@223] - Error: Bookie handle is not available while reading L6 E40 from bookie: 192.168.31.216:3181

One of the related issues is in the auto-recovery decommission and there is one PR in the BookKeeper repo: #4113

However, the bookie client already knows the bk0 is down and we should send the read request to bk1 first. So we can reorder the read request based on the known bookie list. If one bookie is lost, it will reorder the lost bookie to the end of the read list.

Modifications

Enable the reorderReadSequence by default for auto-recovery.

[zymap]

Should we change the default value in the clientconfiguration.java? I think it should be applied to all the clients, right? It will avoid reading a lost bookie as much as possible

[hangc0276]

Good point. We can change the default value in the next major release.

[hangc0276]

rerun failure checks

[hangc0276]

rerun failure checks