Consider using Micrometer as metrics facade

[npepinpe]

Is your feature request related to a problem? Please describe.

As Zeebe was built for the cloud, we initially picked Prometheus as our metrics solution. That's a solid choice, but it does not play very well necessarily with every user's deployment, and would force them to have either adapters for their own metrics solutions or to run Prometheus as well.

Describe the solution you'd like

We could use micrometer, a well supported, mature metrics facade, which as they say is "SLF4J but for metrics". This would still allow us to use Prometheus for our own deployments, but at the same time give more freedom for users to use whatever system they wish to use.

Describe alternatives you've considered

Nothing very interesting - there are workarounds, so this isn't a particularly high priority, but it does reduce friction when it comes to deploy Zeebe in existing environments which may or may not use Prometheus but would benefit from monitoring metrics (which in a distributed environment is critical).

Additional context

Micrometer also come with some additional goodies for the JVM which we could make use of - it will require us to rewrite our Grafana dashboard, but I think it could be worth it.

[npepinpe]

Let's tackle that as we continue integrating Spring concepts.

[Prassud]

@npepinpe is there any update on this ?, we are planning to use a different metrics framework, this could help us to use our own metrics registry

[Zelldon]

@npepinpe I reopened in order to discuss it in our triage

[npepinpe]

Had a second look, and it wouldn't be so straightforward to migrate. The approach of both libraries is different - Prometheus has you registering metrics at startup time, and modifying their value over time with some label values, whereas Micrometer expects you to define the label values at construction time. So for each label value combination you end up with one instance of the metric. It's a little strange, and maybe you can use it differently, but the documentation is sparse on that.

On top of that, for things like gauges, you're expected to measure specific object instances. Here's how you would use/register a gauge:

AtomicInteger myGauge = registry.gauge("numberGauge", new AtomicInteger(0));
myGauge.set(27);

Whereas we tend to simply provide the value independently. It seems Micrometer decorates your object to track them and measure the changes. It's an interesting approach, but quite different.

See this thread about dynamic tags: micrometer-metrics/micrometer#535

It looks like you can sort of work around it, and they try to minimize the object creation penalty by returning you the same instance if it already exists (though it still needs to create this lightweight Meter.Id object, though possibly with escape analysis this is "free"?).

[Zelldon]

Since we will not tackle this in the near future I will close this issue. Feel free to reopen it again.