[test][broker] Exposing metrics in BrokerInterceptor

[crossoverJie]

PIP: PIP-264

Motivation

Shows how to expose metrics in BrokerInterceptor.

Modifications

Using OTel in BrokerInterceptor to expose metrics.

Verifying this change

• Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: crossoverJie#26

[crossoverJie]

@asafm PTAL, thanks.

[asafm]

Since many will record topic-level metrics, we need a common infra to allow you get an AttributesBuilder already filled with the topic level attributes.
@dragosvictor is currently working on implementing many OTel metrics, I suggest you get his plan for this, so you won't overlap.

[crossoverJie]

This is just a demo, it will actually be a custom attribute.

[asafm]

Not sure I get the full intent here, as this metric is in a test.

If you want to learn how to add a metric it's best by doing it for a real metric.

[crossoverJie]

This just show how to customize a metric in an BrokerInterceptor.

In fact, this will be a business-customized metric, and the MESSAGE_COUNTER here is just a demo.

The background comes from this #20772

[asafm]

Got it. Thanks.
So if I understand correctly you would like to have the ability to add your own metrics when you write your own BrokerInterceptor.

So this is not a real PR, but a Draft PR - I would convert it to such, so it won't by mistake gets merged.

The same use case you described exists for any "plugin" people can add to the broker: Function, Offloader plugin, Authentication plugin and many more.

As I wrote in the parent PIP of OTel (pip-264), each such plugin should create their own meter, which reflect their own namespace.

[crossoverJie]

So this is not a real PR, but a Draft PR - I would convert it to such, so it won't by mistake gets merged.

Yes, you are right.

I think this PR is also necessary; this PR can verify that using pulsarService.getOpenTelemetry().getMeter() in a BrokerInterceptor to expose metrics is as expected.