[Bug] Bookkeeper failed to execute simpletest

[ktpktr0]

Search before asking

• I searched in the issues and found nothing similar.

Version

pulsar: 2.10.5 or 3.2.0
os: centos7.9

Minimal reproduce step

I use the following configuration
#21998

When I tried to use a simple test to verify if the bookkeeper cluster was working, it encountered an error

1、./bin/bookkeeper shell bookiesanity

2024-03-11T10:12:34,332+0800 [main] WARN  org.apache.bookkeeper.client.BookKeeper - The mainWorkerPool did not shutdown cleanly
2024-03-11T10:12:34,458+0800 [main] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x20019f7f5f00015 closed
2024-03-11T10:12:34,459+0800 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Bookie sanity test succeeded
2024-03-11T10:12:34,459+0800 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x20019f7f5f00015

2、bin/bookkeeper shell simpletest --ensemble 3 --writeQuorum 3 --ackQuorum 3 --numEntries 1000

2024-03-11T10:13:46,438+0800 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - 192 entries written
2024-03-11T10:13:48,449+0800 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - 405 entries written
2024-03-11T10:13:50,453+0800 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - 619 entries written
2024-03-11T10:13:52,459+0800 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - 855 entries written
2024-03-11T10:13:53,711+0800 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - 1000 entries written to ledger 369
2024-03-11T10:13:53,776+0800 [main] ERROR org.apache.bookkeeper.client.LedgerHandle - ReadAsync exception on ledgerId:369 firstEntry:0 lastEntry:1000 lastAddConfirmed:998
2024-03-11T10:13:53,835+0800 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 192.168.0.148:3181
2024-03-11T10:13:53,842+0800 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 192.168.0.148:3181
2024-03-11T10:13:53,842+0800 [bookkeeper-io-3-3] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x7adb7bf2, L:/192.168.0.150:53672 ! R:192.168.0.148/192.168.0.148:3181]
2024-03-11T10:13:53,842+0800 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 192.168.0.29:3181
2024-03-11T10:13:53,843+0800 [bookkeeper-io-3-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x345880f5, L:/192.168.0.150:34250 ! R:192.168.0.29/192.168.0.29:3181]
2024-03-11T10:13:53,843+0800 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 192.168.0.29:3181
2024-03-11T10:13:53,843+0800 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 192.168.0.150:3181
2024-03-11T10:13:53,844+0800 [bookkeeper-io-3-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x341a0059, L:/192.168.0.150:41502 ! R:192.168.0.150/192.168.0.150:3181]
2024-03-11T10:13:53,844+0800 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 192.168.0.150:3181
2024-03-11T10:13:53,848+0800 [main] WARN  org.apache.bookkeeper.client.BookKeeper - The mainWorkerPool did not shutdown cleanly
2024-03-11T10:13:53,975+0800 [main] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x10014b110c70009 closed
2024-03-11T10:13:53,978+0800 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10014b110c70009
2024-03-11T10:13:53,975+0800 [main] ERROR org.apache.bookkeeper.tools.cli.helpers.ClientCommand - Failed to process command 'simpletest'
org.apache.bookkeeper.client.BKException$BKReadException: Error while reading ledger
        at org.apache.bookkeeper.client.LedgerHandle.readAsync(LedgerHandle.java:754) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
        at org.apache.bookkeeper.client.api.ReadHandle.read(ReadHandle.java:58) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
        at org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand.run(SimpleTestCommand.java:115) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
        at org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand.run(SimpleTestCommand.java:48) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
        at org.apache.bookkeeper.tools.cli.helpers.ClientCommand.apply(ClientCommand.java:66) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
        at org.apache.bookkeeper.tools.cli.helpers.ClientCommand.apply(ClientCommand.java:60) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
        at org.apache.bookkeeper.bookie.BookieShell$SimpleTestCmd.runCmd(BookieShell.java:914) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
        at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:248) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
        at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:2349) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
        at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:2446) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]

What did you expect to see?

Bookie sanity test succeeded

What did you see instead?

As above

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[Technoboy-]

@horizonzy Could you help take a look ?

[lhotari]

@horizonzy Could you help take a look ?

@Technoboy- I have fixed this in Bookkeeper. simpletest was broken. The fix was part of https://github.com/apache/bookkeeper/pull/4197/files#diff-67a4811fdce2ed6838425537a195b28d454f789f9dcf2a617ff5b09dbf12a8a6L111 . More details in https://lists.apache.org/thread/9ly956ss15jo6y2155jwcdq9vzn6tjfd .

[lhotari]

This will be fixed with Bookkeeper 4.16.5 and 4.17.0, once they are released and used in Pulsar.