Micrometer compatibility mode for Metrics

[jmartisk]

Users migrating from Spring Boot and Micrometer might want to expose the same JVM metrics as they used to, so that they don't have to rework their Grafana dashboards.
The naming and organization of base/vendor metrics exposed by Quarkus is different than what Micrometer does, so it would be useful to have a configuration option that changes them to align with that scheme. In particular, we should mimic the way how jvm metrics are handled in https://github.com/micrometer-metrics/micrometer/tree/master/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/jvm so we will have to add some new metrics that we don't normally have, and change metadata of some metrics that we do normally have

[jmartisk]

The main obstacle in the road that I currently see is that the MP Metrics implementation works with the base/application/vendor scopes (as is required by the spec) and prepends the scope of each metric before its name in the Prometheus output. Micrometer doesn't do anything like that.
To overcome that, I suppose we would physically place all these new jvm metrics in the vendor registry and add functionality in SmallRye Metrics to suppress prepending the "vendor_" string before them in the output. This is still against the MP Metrics spec, but probably is the least disruptive solution.

[gsmet]

Couldn't you just add some sort of aliases if a configuration property is enabled?

[geoand]

@ebullient I think you were also interested in this?

[jmartisk]

Couldn't you just add some sort of aliases if a configuration property is enabled?

The metrics that Micrometer exposes are actually quite different. For example MP Metrics has a gc.time counter for each garbage collector, while Micrometer has three distinct metrics (count, max, sum) that differentiate major and minor GCs and they even need to subscribe to JMX notifications to gather data of this granularity on-the-fly.
So I can't quite imagine that right now - the mapping would need to be very complex (names, tags, HELP lines, Prometheus types, and even completely different metric data) and I feel it would turn the MetricRegistry into a huge mess. So I think just registering a different set of metrics would be better.

[gsmet]

@jmartisk makes perfect sense!

[ebullient]

Just as commentary, jvm_* (as done by Spring micrometer) is more parallel to what other runtimes do, e.g. the default labels used by a popular nodejs prometheus client: https://www.npmjs.com/package/prom-client

The micrometer implementation is also pulling in/referincing the prometheus java client: https://github.com/micrometer-metrics/micrometer/blob/master/implementations/micrometer-registry-prometheus/build.gradle .. To me, this means they are using "recommended" labels, in addition to providing their own Spring magic naming conventions for custom metrics. Example from the prometheus java client: https://github.com/prometheus/client_java/blob/master/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot/MemoryPoolsExports.java

[jmartisk]

After some more thinking, I guess we'll have to place these JVM metrics in the base scope instead of vendor, because vendor might potentially contain metrics published by various Quarkus extensions, and I suppose we want to keep all other metrics unaffected by this special mode. This will allow us to simply skip prepending the base_ prefix for all metrics in the base scope (if we placed them together with some other metrics, we'd have to introduce a way to mark some of them as "don't prepend the scope for this metric"). Other scopes than base will remain unaffected by this mode.

This holds under the assumption that we want to keep other metrics unaffected - but I guess we do, because

• application-specific metrics can't be migrated automatically anyway (they need to be rewritten using MP Metrics API), so just changing the way they are exported won't actually help anyone with the migration
• metrics exposed by various Quarkus extensions will most likely be different from what you get with Spring Boot, getting all of them to be the same would be far too complicated, so I'd say we keep the scope of this issue limited just to the jvm metrics.

So TLDR, my plan is that this mode will:

• Place the Micrometer-like JVM metrics into the base scope instead of the ones that are normally there
• Suppress adding the base_ prefix before metrics from the base scope. All other scopes/metrics will be unaffected.

Any objections?

[geoand]

I think it's a valid first step

[jmartisk]

I have a prototype ready, it will need changes in SmallRye Metrics to get in first, then we can upgrade it in Quarkus.

It means quite a lot of bending of the MP Metrics specification, but it seems that I managed to bend the jvm_ metric output sufficiently so that it looks the same as in the Micrometer output. Except the TYPE lines which have a few differences, but I suppose that this should not matter as long as the actual metrics have the same names and tags. In Grafana it works for me when testing with this dashboard https://grafana.com/grafana/dashboards/3308

By the way none of these metrics work in native mode.
EDIT: it looks like at least jvm.threads.live, jvm.threads.peak and jvm.threads.daemon work after all :)

[ebullient]

As alternative (experiment), I created a quarkus extension to use micrometer directly: https://github.com/ebullient/quarkus-micrometer-extension