[feature] Add Metrics to understand Load Balancing Decisions

[dave2wave]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

When pushing pulsar clusters to see maximum performance it is important to understand which metric is causing bundle splits and unloading. The metrics shared only show that the split and/or unload occurred.

1. bundle unload metrics
2. bundle split metrics

The logs give values, but it is hard to reason about these or monitor them.

Solution

Add reason counts for load balancing: bundle splits, and unloads.

1. Bandwidth
2. CPU
3. Direct Memory
4. Memory
5. Admin

Alternatives

Make the log files be much clearer about which metric has triggered a bundle split or unload.

Anything else?

This is related to #16691

Are you willing to submit a PR?

• I'm willing to submit a PR!

[github-actions]

The issue had no activity for 30 days, mark with Stale label.