Add auto-configuration for Micrometer 2.0 Observation API

[marcingrzejszczak]

Micrometer 2.0.0 changes

Micrometer 2.0.0 comes with the following changes

• Moving part of Micrometer Core to Micrometer API
• Introduction of the Observation API
• Tracing (see Add support for Micrometer tracing #30156)

Observation API

The most notable change is adding the ObservationRegistry interface. MeterRegistry implements it so there wouldn't be any change there. The thing that does change is that MeterRegistry now has a observationConfig configuration that allows to plug in ObservationHandlers.

Required by Boot AutoConfiguration

To create an Observation one has to call factory methods on Observation (start or createNotStarted). The MeterRegistry#ObservationConfig can have a list of ObservationPredicate injected to define whether for given Observation name and metadata an Observation should be created or a NoOp observation should be returned. Example use case is AutoTimer in Spring Boot. Now AutoTimer is an inbuilt feature of the Observation API.

ObservationHandler

An ObservationHandler allows plugging in into the lifecycle of an Observation. That means that we can react to events such as observation was started, stopped, there was an error etc.

Micrometer comes with an interface MeterObservationHandler whose default implementation is TimerObservationHandler. That in turn will create a Timer when an Observation was started and will stop it when the Observation was finished. MeterRegistry comes with a method withTimerObservationHandler that will by default create this handler. Theoretically there might be case that someone adds Micrometer but doesn't want to create a Timer when an Observation is created.

Micrometer Tracing comes with an interface TracingObservationHandler that contains additional logic for tracing related handlers.

Micrometer comes with 2 additional interfaces that group ObservationHandlers. First is ObservationHandler.FirstMatchingCompositeObservationHandler . It will group ObservationHandlers and pick the first one whose supportsContext method returns true.
Second is ObservationHandler.AllMatchingCompositeObservationHandler. It will group ObservationHandlers and pick all whose supportsContext method returns true.

After having grouped the handlers they have to be injected to the ObservationRegistry via the ObservationRegistry#observationConfig()#observationHandler method.

Required by Boot AutoConfiguration

The following rules of handler groupings should take place:

• All MeterObservationHandler implementations should be automatically grouped in a ObservationHandler.FirstMatchingCompositeObservationHandler.
• All TracingObservationHandler implementations should be automatically grouped in ObservationHandler.FirstMatchingCompositeObservationHandler.
• All ObservationHandler implementations should be grouped in ObservationHandler.AllMatchingCompositeObservationHandler (without duplicates)

Example:

• We have a MeterObservationHandler bean called Meter coming from the framework
• We have a TracingObservationHandler bean called Tracing coming from the framework
• We have a ObservationHandler bean called Handler coming from the framework
• We have a FirstMatchingCompositeObservationHandler bean called First coming from the framework
• We have a AllMatchingCompositeObservationHandler bean called All coming from the framework
• We have a FirstMatchingCompositeObservationHandler bean called First from the user coming from the user
• We have a AllMatchingCompositeObservationHandler bean called All from the user coming from the user
• We have a MeterObservationHandler bean called Meter from the user coming from the user
• We have a TracingObservationHandler bean called Tracing from the user coming from the user
• We have a ObservationHandler bean called Handler from the user coming from the user

We should have the following setup of beans by Boot

• FirstMatchingCompositeObservationHandler created by Boot that groups all meter related handlers
  • MeterObservationHandler called Meter
  • MeterObservationHandler caled Meter from the user (order can vary depending on Ordered)
• FirstMatchingCompositeObservationHandler created by Boot that groups all tracing related handlers
  • TracingObservationHandler called Tracing
  • TracingObservationHandler caled Tracing from the user (order can vary depending on Ordered)
• ObservationHandler bean called Handler (order can vary depending on Ordered)
• ObservationHandler bean called Handler from the user (order can vary depending on Ordered)
• FirstMatchingCompositeObservationHandler bean called First (order can vary depending on Ordered)
• FirstMatchingCompositeObservationHandler bean called First from the user (order can vary depending on Ordered)
• AllMatchingCompositeObservationHandler bean called All (order can vary depending on Ordered)
• AllMatchingCompositeObservationHandler bean called All from the user (order can vary depending on Ordered)

ObservationRegistry.observationConfig().observationHandler() will be called to register the beans presented above.

Following conditionals have to be applicable

• When this feature is disabled (no Micrometer on the classpath / a property is explicitly disabled) nothing is created
• FirstMatchingCompositeObservationHandler created by Boot that groups all meter related handlers
  • supportsContext should return false when a property to disable metrics is turned on (Micrometer on the classpath / a property for metrics enabling is explicitly disabled) - we can toggle this on and off at runtime
• FirstMatchingCompositeObservationHandler created by Boot that groups all tracing related handlers
  • should not be created when Micrometer Tracing is not on the classpath
  • supportsContext should return false when a property to disable tracing is turned on (Micrometer & Micrometer Tracing on the classpath / a property for tracing enabling is explicitly disabled) - we can toggle this on and off at runtime

GlobalTagsProvider

A user can provide a GlobalTagsProvider that can be applied to all observations. We would need Boot to autowire a list of GlobalTagsProvider and put them on the ObservationRegistry via the ObservationRegistry.tagsProvider(...) method.

[wilkinsona]

I've only taken a first pass through this so my thoughts probably aren't yet complete. Things I've spotted thus far:

MeterRegistry#ObservationConfig can have a list of BiPredicate<String, Observation.Context> injected. … If a BiPredicate is in your opinion not a good contract for this we can introduce a new, dedicated interface.

If we want Boot's auto-configuration to be able to find the predicates as beans and inject them into the config, I think a dedicated interface would be better. For example, it makes it easier to identify the beans the should be injected without having to worry about generics.

With regards to the composite observation handlers, what should Boot do with the 3 that it creates? I am guessing that they need to be registered with Micrometer but I didn't see that mentioned above. Apologies if I missed it. Also, what should happen with the normal, non-composite handler beans? Do they also need to be registered or should it only be the composites that are registered with Micrometer?

When this feature is disabled (no Micrometer on the classpath / a property is explicitly disabled) nothing is created

Generally speaking, we try to avoid having properties to disable features. I wonder if the classpath could be used as a signal instead but that would depend on the structure of the code in Micrometer.

With regards to the tracing bridges, do you have a feel for the dependency management implications? I imagine that we'll want to manage the version of the Brave and OTel dependencies. Are their release schedules and approach to maintenance amenable to that?

[marcingrzejszczak]

If we want Boot's auto-configuration to be able to find the predicates as beans and inject them into the config, I think a dedicated interface would be better. For example, it makes it easier to identify the beans the should be injected without having to worry about generics.

Makes sense micrometer-metrics/micrometer#3003

With regards to the composite observation handlers, what should Boot do with the 3 that it creates? I am guessing that they need to be registered with Micrometer but I didn't see that mentioned above. Apologies if I missed it. Also, what should happen with the normal, non-composite handler beans? Do they also need to be registered or should it only be the composites that are registered with Micrometer?

I've forgotten to put it there - I've updated the description. Yes, they will need to be registered in ObservationRegistry#observationConfig.

Generally speaking, we try to avoid having properties to disable features. I wonder if the classpath could be used as a signal instead but that would depend on the structure of the code in Micrometer.

That should happen for sure. The thing is that people might state that e.g. tracing is lowering performance and they want to disable just that at runtime.

[wilkinsona]

This is blocked for the same reasons as #29753.

[mhalbritter]

Hey @marcingrzejszczak, got a question about the grouping logic:

• All ObservationHandler implementations should be grouped in ObservationHandler.AllMatchingCompositeObservationHandler

Why is this necessary? Handlers added via observationRegistry.observationConfig().observationHandler(...); are essentially in a AllMatchingCompositeObservationHandler, why wrap them again?

I imagine our code for registration something like this:

	private void registerHandlers(List<ObservationHandler<?>> observationHandlers, ObservationRegistry observationRegistry) {
		// API is forcing us to use raw types - need to create an issue for that
		List<ObservationHandler> meterHandlers = new ArrayList<>();
		List<ObservationHandler> tracingHandlers = new ArrayList<>();

		for (ObservationHandler<?> observationHandler : observationHandlers) {
			if (observationHandler instanceof MeterObservationHandler) {
				meterHandlers.add(observationHandler);
			} else if (observationHandler instanceof TracingObservationHandler) {
				tracingHandlers.add(observationHandler);
			} else {
				observationRegistry.observationConfig().observationHandler(observationHandler);
			}
		}

		if (!meterHandlers.isEmpty()) {
			observationRegistry.observationConfig().observationHandler(new FirstMatchingCompositeObservationHandler(meterHandlers));
		}
		if (!tracingHandlers.isEmpty()) {
			observationRegistry.observationConfig().observationHandler(new FirstMatchingCompositeObservationHandler(tracingHandlers));
		}
	}

Does that work or am I'm missing something?

[mhalbritter]

I edited your ticket and moved the changes for Micrometer Tracing in a separate issue, #30156

[marcingrzejszczak]

Yes, that's OK (I've forgotten to update the ticket with that suggestion).

[mhalbritter]

Thanks for confirmation. I opened an issue regarding the use of raw types (micrometer-metrics/micrometer#3064)

[marcingrzejszczak]

I see one problem actually. Tracing might not be in the classpath. That means that if it's not then accessing the Tracing handler will lead to class not found exception

[mhalbritter]

Yes, we will handle that with some @Conditionals. Thanks for the hint.

[mhalbritter]

Hey @marcingrzejszczak, regarding the option to disable metrics and just leave tracing: Ideally, Micrometer's modules would be structured such that we can figure this out just from what's on the classpath. We don't really like having configuration options if we can avoid it. So if a user wants tracing, they include micrometer-tracing-api. If they don't want metrics, they don't include the micrometer-xxx dependency, whatever that may be.

[mhalbritter]

For further discussions on the implementation, let's please use #30186 - this makes it easier to have them in one place and I can mark them as resolved. Thanks!

[marcingrzejszczak]

If I add micrometer-tracing-api I automatically get micrometer-core so there's no way to have only tracing

[mhalbritter]

We'll delay the merge until the 3.0.0-M2 release is done, as this depends on micrometer snapshots.

[mhalbritter]

3.0.0-M2 is released, no longer blocking the merge.

[mhalbritter]

Build for the PR is currently broken, as Spring Boot uses Micrometer 2.0.0-SNAPSHOT and Spring Batch uses Micrometer 2.0.0-M3, which is incompatible. We need to wait for spring-projects/spring-batch#4081 to be merged, then we can build against Spring Batch 5.0.0-SNAPSHOT.

[mhalbritter]

Spring Batch now builds against micrometer 2.0.0-SNAPSHOT, i've updated the PR, it now builds. No longer blocked.

[mhalbritter]

We need to wait until all the projects in the release train build against micrometer 2.0.0-SNAPSHOT until we can merge this.