[Bug] CPU Spike and Extended Duration in BanyanDB's etcd Watching Registration Process

[hanahmily]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

BanyanDB (apache/skywalking-banyandb)

What happened

During the etcd watching registration process in BanyanDB, I've noticed that the process takes longer than expected. Additionally, there is a significant CPU spike during this process.

{"level":"info","module":"ETCD","name":"measure","kind":"group","time":"2024-03-03T23:53:17Z","message":"registering watcher"}                               │
│ {"level":"info","module":"ETCD","name":"measure","kind":"measure","time":"2024-03-03T23:53:17Z","message":"registering watcher"}                             │
│ {"level":"info","module":"ETCD","name":"measure","kind":"indexRuleBinding","time":"2024-03-03T23:53:32Z","message":"registering watcher"}                    │
│ {"level":"info","module":"ETCD","name":"measure","kind":"indexRule","time":"2024-03-03T23:53:43Z","message":"registering watcher"}                           │
│ {"level":"info","module":"ETCD","name":"measure","kind":"topNAggregation","time":"2024-03-03T23:53:54Z","message":"registering watcher"}                     │
│ {"level":"info","module":"ETCD","name":"stream","kind":"group","time":"2024-03-03T23:53:54Z","message":"registering watcher"}                                │
│ {"level":"info","module":"ETCD","name":"stream","kind":"stream","time":"2024-03-03T23:53:54Z","message":"registering watcher"}                               │
│ {"level":"info","module":"ETCD","name":"stream","kind":"indexRuleBinding","time":"2024-03-03T23:53:54Z","message":"registering watcher"}                     │
│ {"level":"info","module":"ETCD","name":"stream","kind":"indexRule","time":"2024-03-03T23:53:54Z","message":"registering watcher"}                            │
│ {"level":"info","module":"ETCD","name":"liaison","kind":"group","time":"2024-03-03T23:53:54Z","message":"registering watcher"}                               │
│ {"level":"info","module":"ETCD","name":"liaison","kind":"stream","time":"2024-03-03T23:53:54Z","message":"registering watcher"}                              │
│ {"level":"info","module":"ETCD","name":"liaison","kind":"group","time":"2024-03-03T23:53:54Z","message":"registering watcher"}                               │
│ {"level":"info","module":"ETCD","name":"liaison","kind":"measure","time":"2024-03-03T23:53:54Z","message":"registering watcher"} 

What you expected to happen

The etcd watching registration process should be completed in a reasonable amount of time without causing a CPU spike.

How to reproduce

1. Using OAP to create schema in BanyanDB, don't have to insert any data to DB.
2. Restart BanyanDB.
3. Monitor the CPU usage and etcd watching registration process duration

Anything else

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct