Bookkeeper High CPU usage when StreamStorageLifecycleComponent extra component is active

[aschiazza]

Hi, I've a trouble with Bookkeeper v4.9.2 ( I'm using Pulsar Docker image v2.4.1).
When I activate the StreamStorageLifecycleComponent adding this line

extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent

in bookkeeper.conf, the bookkeeper container is using 100% CPU.

In my test environment I have 1 Zookeeper server and 1 Bookkeeper bookie.

I report my bookkeeper.conf configuration section for table service

# enable table service api for pulsar function state management
extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent
ignoreExtraServerComponentsStartupFailures=false

##################################################################
##################################################################
# Settings below are used by stream/table service
##################################################################
##################################################################

### Grpc Server ###

# the grpc server port to listen on. default is 4181
storageserver.grpc.port=4181

### Dlog Settings for table service ###

#### Replication Settings
dlog.bkcEnsembleSize=1
dlog.bkcWriteQuorumSize=1
dlog.bkcAckQuorumSize=1

Thanks for your help

[eolivelli]

Are you able to take a dump of the stacktrace of the JVM? With jstack for instance

[aschiazza]

Thanks for the reply.

I have attached the JVM stacktrace and ps output.

jstack_bookkeeper.txt
ps_bookkeeper.txt

[eolivelli]

It looks like a problem with some leader electiin.
I can't tell more now (I don't have my laptop)
Is the bookie able to talk to zookeeper?

[aschiazza]

I think bookie is able to talk to zookeeper. I test it in that way:
1- I attached a shell in bookkeeper container.
2 - started a zookeeper shell (bin/pulsar zookeeper-shell -server zookeeper:2181)
3 - in zk shell I typed ls /stream/servers/available
4 - result is [172.25.0.6:4181] ( that is the IP of bookie container)

Can I test in some other way?

[aschiazza]

Any suggestions?

[sijie]

@aschiazza do you have the log of the bookies?

[aschiazza]

Hi, thanks for the reply.
I attached the log of the bookie.
bookkeeper_log.txt

[zyllt]

@aschiazza Hi,I started StreamStorageLifecycleComponent according to your configuration, but it is 127.0.0.1:4181 in zk,I tried many methods without success. Is there any other configuration?
my pulsar version is 2.5.0.

[aschiazza]

Hi @zyllt, I've attached my bookkeeper conf file.
bookkeeper.conf.txt
I'm still using version 2.4.1 for my project.

in functions_worker.yml conf file I've added this line stateStorageServiceUrl: bk://bookkeeper:4181

I'm using docker-compose for my environment and the command for bookie container is /pulsar/bin/bookkeeper bookie

I hope it will help you.

[zyllt]

@aschiazza Thanks for your reply. I start bookie bin/bookkeeper bookie use your bookkeeper conf file without docker ,i type ls /stream/servers/available at zk ,it is still 127.0.0.1:4181.
but when i use docker-compose for my environment and the cmd is bin/pulsar standalone,the output of bin/pulsar zookkeeper-shell && ls /stream/servers/available is [172.17.0.2:4181]. that is the IP of bookie container.
I am very confused but I can't find the source code to get ip when registering zk.
@sijie Can you give me some suggestions?

[aschiazza]

If you start pulsar in standalone mode all components (broker, bookie zk) start in a single container and registered to the loopback interface.
You should start each component (broker, bookie, zk) with proper command, for example
/pulsar/bin/pulsar zookeeper for zk
/pulsar/bin/bookkeeper bookiefor bookie
/pulsar/bin/pulsar brokerfor pulsar broker
/pulsar/bin/pulsar proxyfor pulsar proxy
/pulsar/bin/pulsar functions-worker for functions worker

[zyllt]

If you start pulsar in standalone mode all components (broker, bookie zk) start in a single container and registered to the loopback interface.
You should start each component (broker, bookie, zk) with proper command, for example
/pulsar/bin/pulsar zookeeper for zk
/pulsar/bin/bookkeeper bookiefor bookie
/pulsar/bin/pulsar brokerfor pulsar broker
/pulsar/bin/pulsar proxyfor pulsar proxy
/pulsar/bin/pulsar functions-worker for functions worker

@aschiazza thanks for your reply.May be my previous expression was inaccurate. I really mean when I start bookie bin/bookkeeper bookie use your bookkeeper conf file without docker,my environment is product.
because my product environment do not support docker,i start bookie with docker in my local environment.
First I test bin/pulsar standalone command with docker,the output of bin/pulsar zookkeeper-shell && ls /stream/servers/available is [172.17.0.2:4181].
Second i start each component (broker, bookie, zk) use bin/pulsar zookeeper and bin/bookkeeper bookie command with docker, the input is [172.17.0.4:4181].
Last i test each component (broker, bookie, zk) in my local environment without docker,but input is [127.0.0.1:4181].
172.17.0.2 and 172.17.0.4 are the IP of bookiecontainer.