kafkaexporter: Improve internal telemetry

[1pkg]

Description

This PR adds the following internal telemetry metrics to improve the exporter telemetry to make it easier to operate.

Click to expand

Name	Description	Type
kafka_exporter_records	The number of exported records	sum
kafka_exporter_bytes	The number of exported bytes, accounting for client compression	sum
kafka_exporter_bytes_uncompressed	The number of exported uncompressed bytes	sum
kafka_exporter_latency	Time it took to export a batch of records	histogram
kafka_broker_connects	The number of connections opened	sum
kafka_broker_disconnects	The number of connections closed	sum
kafka_broker_throttling_duration	The throttling duration imposed by the broker when exporting messages	histogram

Link to tracking issue

Fixes

#40815

Testing

Unit tests are added, also tested manually.

Documentation

Documentation is autogenerated with mdatagen.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: 1pkg / name: Kostiantyn Masliuk (f962f14, 3782746, 5d5f917, ca5a147, 4673f45, 20b75ff, ca83672, fc1f035, 3212a64, 9103ba8, 054bca6)

[axw]

Looks great! Just a few small things.

Maybe we should add exponential histogram support to mdatagen, and use that for measuring the latency. Not a blocker of course, just doesn't seem ideal to use explicit bounds.

[1pkg]

Looks great! Just a few small things.

Maybe we should add exponential histogram support to mdatagen, and use that for measuring the latency. Not a blocker of course, just doesn't seem ideal to use explicit bounds.

I can raise an issue in opentelemetry-collector to add the support in mdatage and contribute the implementation. As this is not a blocker I will do this asynchronously.

[vigneshshanmugam]

Maybe we should add exponential histogram support to mdatagen, and use that for measuring the latency. Not a blocker of course, just doesn't seem ideal to use explicit bounds.

This is a problem we are running in to for other components as well, Worth raising it as a separate issue.

[vigneshshanmugam]

LGTM other than the attribute duplication logic.