Prometheus client metrics support

[libretto]

This code introduces the Metrics class, which allows clients migrating to Karapace from alternative products to collect metrics similar to those they are accustomed to. It supports using Prometheus as an alternative to StatsD for collecting statistics in Karapace.

[libretto]

@jjaakola-aiven @aiven-anton could You review this PR too?

[libretto]

@aiven-anton done. Review please.

[libretto]

BTW. Is the usage of .gauge() and .timing() in the following functions acceptable?

def are_we_master(self, is_master: bool) -> None:
    if not self.is_ready or self.stats_client is None:
        return
    if not isinstance(self.stats_client, StatsClient):
        raise RuntimeError("no StatsClient available")
    self.stats_client.gauge("master-slave-role", int(is_master))


def latency(self, latency_ms: float) -> None:
    if not self.is_ready or self.stats_client is None:
        return
    if not isinstance(self.stats_client, StatsClient):
        raise RuntimeError("no StatsClient available")
    self.stats_client.timing("latency_ms", latency_ms)

[eliax1996]

Can we have some tests?

[libretto]

Can we have some tests?

We can certainly add some tests. However, the current version of Karapace Stats lacks test coverage, so we have no existing framework to guide us. At this point, it seems that only unit tests are feasible. I'm unsure how to implement integration tests for this part task.

[eliax1996]

Can we have some tests?

We can certainly add some tests. However, the current version of Karapace Stats lacks test coverage, so we have no existing framework to guide us. At this point, it seems that only unit tests are feasible. I'm unsure how to implement integration tests for this part task.

Let's kick off with the unit tests. I'll provide an example soon on how to execute an integration test. It's crucial to have a tangible test that ensures our output its compliant to the standard format.
We need to verify that compliant consumers can effectively utilize and extract the metrics.

Without integration tests, even if the feature is technically sound, we lack a mechanism preventing us from the introduction of code that might break the consumers

[eliax1996]

stats_client: StatsClient | None = None, I've run this locally with metrics_extended set to False and its failing because the stats_client its not going to be set

[libretto]

Fixed and unit test for this case is added.

[nosahama]

Thoughts:

• The PR description makes mention of only prometheus, it's confusing to see statsd implementations.
• It'd be nice to see also somewhere in the code, all possible metrics and their related tags if any, that way I as a programming would be working based off an interface than just passing string values to the metrics handler.
• I feel this could be simplified by providing only Prometheus support and using integrations when needed which should also be left to the users, i.e. if StatsD metrics are then needed, one could add the related mapping and deploy for themselves a statsd-exporter.
• If properly tested and benchmarked, it doesn't hurt to run the WSGI server (stats) as it's own thread, but it doesn't feel right in 2024 to write this much application code just to export some metrics with all the tools available.
• Prometheus is pull based, and as we already have a webserver running, we need to add an endpoint to scrape the metrics and probably a directory for the prometheus multiprocess registry.
• Even though we want to provide similar metrics as confluent, we do not need to provide all of them via the application, we can provide a parent metric and then use promql and recording rules to get the rates, avg, etc.

I will continue the work by @eliax1996 and try to add a sample metric and also some integration test.

[libretto]

Thoughts:

* The PR description makes mention of only prometheus, it's confusing to see statsd implementations.

* It'd be nice to see also somewhere in the code, all possible metrics and their related tags if any, that way I as a programming would be working based off an interface than just passing string values to the metrics handler.

* I feel this could be simplified by providing only Prometheus support and using integrations when needed which should also be left to the users, i.e. if StatsD metrics are then needed, one could add the related mapping and deploy for themselves a [statsd-exporter](https://github.com/prometheus/statsd_exporter).

* If properly tested and benchmarked, it doesn't hurt to run the WSGI server (stats) as it's own thread, but it doesn't feel right in 2024 to write this much application code just to export some metrics with all the tools available.

* Prometheus is pull based, and as we already have a webserver running, we need to add an endpoint to scrape the metrics and probably a directory for the prometheus multiprocess registry.

I will continue the work by @eliax1996 and try to add a sample metric and also some integration test.

This PR introduces the Metrics class, aiming to implement metrics similar to those found in comparable products. We intend to support both StatsD and Prometheus. Are you suggesting that we should enable the concurrent use of both?
I just revised the PR description for better clarity.

[nosahama]

Thoughts:

* The PR description makes mention of only prometheus, it's confusing to see statsd implementations.

* It'd be nice to see also somewhere in the code, all possible metrics and their related tags if any, that way I as a programming would be working based off an interface than just passing string values to the metrics handler.

* I feel this could be simplified by providing only Prometheus support and using integrations when needed which should also be left to the users, i.e. if StatsD metrics are then needed, one could add the related mapping and deploy for themselves a [statsd-exporter](https://github.com/prometheus/statsd_exporter).

* If properly tested and benchmarked, it doesn't hurt to run the WSGI server (stats) as it's own thread, but it doesn't feel right in 2024 to write this much application code just to export some metrics with all the tools available.

* Prometheus is pull based, and as we already have a webserver running, we need to add an endpoint to scrape the metrics and probably a directory for the prometheus multiprocess registry.

I will continue the work by @eliax1996 and try to add a sample metric and also some integration test.

This PR introduces the Metrics class, aiming to implement metrics similar to those found in comparable products. We intend to support both StatsD and Prometheus. Are you suggesting that we should enable the concurrent use of both? I just revised the PR description for better clarity.

I do not think it is necessary to have both StatsD and Prometheus within the application itself, especially if there are tools to integrate one on top of the other. Nevertheless, we already provide statsd support within the code, this is something to discuss internally, we'd go ahead and add prometheus metrics and then reason around what to do with the current stats implementation which both calls statsd and sentry at the same time.

[nosahama]

@libretto I will close this PR once #902 is merged.
In #902 we've introduced the building blocks for instrumenting the service using Prometheus. Following this, we can add more metrics as needed, until it is sufficient enough to provide compatibility with Confluent SR metrics.