Problem with prometheus metrics and mongodb

[gnom7]

I noticed that common tags are not applied on service that uses MongoDB. In debugging I see that for some reason MongoMetricsConnectionPoolListener logic kicks (and interacts with MeterRegistry) before MetricsAutoConfiguration had a chance to configure and apply MeterRegistryPostProcessor (spring boot v3.2.3).

Might be related to #26665

For now I just disabled (exclude = [MongoMetricsAutoConfiguration::class]) mongo metrics and it fixes behavior.

[mhalbritter]

Thanks for the report. If you'd like us to spend some time investigating, please take the time to provide a complete minimal sample (something that we can unzip or git clone, build, and deploy) that reproduces the problem.

[gnom7]

I have tried to create small repro using spring initializr, but unfortunately basic project doesn't help.

Not sure what exactly causes out of order processing in my application, it looks like this:
MongoAutoConfiguration -> PrometheusMetricsExportAutoConfiguration -> MongoMetricsAutoConfiguration -> MetricsAutoConfiguration

One thought was that premature execution of MongoAutoConfiguration (which in turn causes metrics to be initialized via MongoMetricsConnectionPoolListener when mongo's connection pool gets created) was caused by Mongock which applies MongoDB migrations at the application startup, but adding Mongock with some migrations to the repro project doesn't help either.

BTW #26665 (comment) this workaround also works in my case, so no need to disable mongo metrics in the end.

[mhalbritter]

A breakpoint in org.springframework.boot.autoconfigure.AutoConfigurationSorter#getInPriorityOrder on the return orderedClassNames; may help with debugging. orderedClassNames contains the list of auto-configurations in the order they will be loaded.

I suspect something in the application is using the MeterRegistry quite early and breaks stuff (see this comment).

[gnom7]

You are right, after I have carefully investigated call stack from the MongoAutoConfiguration#mongo I was able to find the place where mongo repository was injected eagerly in some early stages of application startup. Making that injection lazy helped.

Thanks @mhalbritter