Consolidate OpenTelemetry support in Spring

[ThomasVitale]

I was about to open a few issues related to the OpenTelemetry support in Spring/Micrometer and volunteer to implement some of the existing ones, but I thought that it might help gathering here what's the current status so to put each issue within the right context and perspective. Looking at the bigger picture helps me explain some of those issues without pointing the reader to too many different GitHub issues. And I hope it will help everyone who joins the conversation. If this is not the right place to share such information, please let me know and I'll fix it accordingly.

I'll be happy to discuss this topic further and contribute pull requests as needed.

TL;DR; I go through a few issues and suggestions to consolidate OpenTelemetry support in Spring and unifying the configuration for metrics and traces (and enabling doing the same for logs in the future).

Metrics

Spring Boot Actuator provides dependency management and autoconfiguration for Micrometer, which offers a convenient facade over many different monitoring systems. One of them is OpenTelemetry.

Spring Boot developers can enable their applications to export metrics via the OTLP protocol to an OpenTelemetry backend by adding Spring Boot Actuator and the dedicated OpenTelemetry dependency from Micrometer to their projects.

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    runtimeOnly 'io.micrometer:micrometer-registry-otlp'
    ...
}

The exporter provided by the Micrometer Registry OTLP is an HTTP exporter and can be configured via properties thanks to the autoconfiguration for OpenTelemetry.

management:
  otlp:
    metrics:
      export:
        url: http://localhost:4318/v1/metrics
        step: 5s

OpenTelemetry supports additional key/value pairs (called resource attributes) to be included in the telemetry data. Spring Boot Actuator provides autoconfiguration for those and makes it possible to add new resource attributes via properties.

management:
  opentelemetry:
    resource-attributes:
      cluster: local
      "service.name": ${spring.application.name}

I made a demo application to showcase this setup.

Issue M.1 - Wrong autoconfiguration in Spring Boot 3.2

In this pull request delivered to spring Boot 3.2.0, some OpenTelemetry configuration regarding resource attributes has been consolidated to be able to share it between metrics and traces implementations (which is really great and useful, thanks for that!).

That works fine when using OpenTelemetry for tracing, but when it's used only for metrics the autoconfiguration doesn't work. The OpenTelemetryProperties bean which is now autowired in the OtlpMetricsExportAutoConfiguration requires the OpenTelemetry SDK to be in the classpath. However, the Micrometer Registry OTLP library doesn't include the OpenTelemetry SDK, causing the autoconfiguration to fail.

A workaround for now is to add an explicit dependency to OpenTelemetry SDK and rely on the Spring Boot dependency management to resolve the correct version.

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    runtimeOnly 'io.micrometer:micrometer-registry-otlp'
    
    // Added dependency
    implementation 'io.opentelemetry:opentelemetry-sdk-common'
}

Possible solutions:

• include workaround to the Spring Boot Actuator documentation;
• add missing dependency as part of the autoconfiguration;
• wait until further consolidation activities have been completed to unify the OpenTelemetry support in Spring.

Issue M.2 - Service Name resource attribute is set to "unknown_service"

By default, Micrometer Registry OTLP includes a few resource attributes to the exported metrics (see the full list). One of them is the standard service.name OpenTelemetry attribute. The default value for this attribute is unknown_service since the library can't know the name of the application being instrumented.

Relying on the Spring Boot Actuator autoconfiguration for OpenTelemetry resource attributes, developers can overwrite the value of service.name with the actual name of the application as configured in Spring Boot (spring.application.name).

management:
  opentelemetry:
    resource-attributes:
      "service.name": ${spring.application.name}

Since Spring Boot can tell if an application name has been configured, it would be nice if some autoconfiguration existed to do that out-of-the-box. Such autoconfiguration already exists in Spring Boot Actuator (see OpenTelemetryAutoConfiguration). However, that is only actually used for traces.

Possible solutions:

• leave it to the developers to assign a proper value to the service.name resource attribute;
• add dedicated autoconfiguration only for metrics when using OpenTelemetry;
• wait until further consolidation activities have been completed to unify the OpenTelemetry support in Spring.

Issue M.3 - OTLP Metrics Exporter is HTTP-only and not configurable via OpenTelemetry

The Micrometer Registry OTLP module uses an OTLP-compatible HTTP client to export metrics to an OpenTelemetry backend. Internally, the OtlpMeterRegistry uses a private HttpSender object to configure the HTTP client. The benefit of this approach is that the module is lightweight and doesn't need any dependency on the OpenTelemetry SDK. It only uses the io.opentelemetry.proto library which provides the protobuf configuration for OpenTelemetry.

On the other hand, such an approach means that:

• it's not possible to configure the metrics exporter via standard OpenTelemetry configuration;
• it's not possible to use gRPC instead of HTTP/protobuf;
• it's not possible to share OpenTelemetry configuration in Spring Boot between metrics and traces (see, for example, issue M.2).

The new generic OpenTelemetryAutoConfiguration in Spring Boot 3.2 autoconfigures an OpenTelemetry bean and makes it possible to configure an SdkMeterProvider bean. What if Micrometer could use that for setting up the OpenTelemetry metrics exporter?

Possible solutions:

• update the existing Micrometer Registry OTLP module to implement a MeterRegistry that accepts an OpenTelemetry object for configuration (similar to the OpenTelemetryMeterRegistry used by the OpenTelemetry Java Instrumentation library). If backward compatibility is necessary, the new implementation would need to co-exist with the existing one (maybe a feature flag could switch between the two?);
• create a new Micrometer Registry OpenTelemetry module to implement what described in the previous point, but without having issues with backward compatibility.

Building (or updating) such a module would make it possible to re-use the same OpenTelemetryAutoConfiguration introduced in Spring Boot 3.2 for both metrics and traces (and, in the future, for logs as well).

• It would solve the issue M.1 described before since the Micrometer Registry module would have a dependency on the OpenTelemetry SDK.
• It would solve the issue M.2 described before because Micrometer would use the autoconfigured Resource object from OpenTelemetry to add resource attributes to the metrics.
• An issue already exists on the Spring Boot project to autoconfigure an SdkMeterProvider bean and, in general, for using metrics with OpenTelemetry. But we are missing Micrometer support before that is doable.
• It would also make it possible to switch the client implementation between HTTP and gRPC using the standard OpenTelemetry approach (i.e. configuring either an OtlpHttpMetricExporter or OtlpGrpcMetricExporter).
• Finally, it would make it easier for developers using Micrometer in applications together with the OpenTelemetry Java Agent or, in general, the OpenTelemetry Java Instrumentation. Those projects currently maintain a shim between Micrometer and OpenTelemetry. That shim would probably not be needed anymore. So, such a change would also reduce the workload on maintaining the OpenTelemetry Java Instrumentation compatible with Micrometer.

For more context about the current challenges, refer to this issue on the Spring Boot project.

Traces

Spring Boot Actuator provides dependency management and autoconfiguration for Micrometer Tracing, which offers a convenient facade over a few different distributed tracing backends. One of them is OpenTelemetry.

Spring Boot developers can enable their applications to export traces via the OTLP protocol to an OpenTelemetry backend by adding Spring Boot Actuator, Micrometer Tracing and the dedicated OpenTelemetry dependency from Micrometer to their projects.

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'io.micrometer:micrometer-tracing-bridge-otel'
    implementation 'io.opentelemetry:opentelemetry-exporter-otlp'
    ...
}

The exporter provided by the Spring Boot Actuator autoconfiguration is an HTTP exporter (an OtlpHttpSpanExporter bean) and can be configured via properties thanks to the tracing configuration in OtlpAutoConfiguration.

management:
  otlp:
    tracing:
      endpoint: http://localhost:4318/v1/traces

OpenTelemetry supports additional key/value pairs (called resource attributes) to be included in the telemetry data. Spring Boot Actuator provides autoconfiguration for those and makes it possible to add new resource attributes via properties. The standard OpenTelemetry service.name resource attribute is configured automatically to the value of spring.application.name (if defined) or else to a default application value.

management:
  opentelemetry:
    resource-attributes:
      cluster: local

I made a demo application to showcase this setup.

Issue T1 - Exporting traces via gRPC

In this issue on the Spring Boot project, autoconfiguration for an OtlpHttpSpanExporter bean has been added to export traces via HTTP.

A common requirement is to export traces via gRPC (the most used approach in OpenTelemetry). Currently, developers can configure an OtlpGrpcSpanExporter bean by themselves. It would be nice if Spring Boot Actuator would provide autoconfiguration for that, enhancing the existing OtlpAutoConfiguration.

There is already an issue on the Spring Boot project to add such autoconfiguration.

Summary

Overall, these are the issues and suggestions I've been describing so far.

Current bugs in Spring Boot 3.2

• The Micrometer Registry OTLP library doesn't depend on the OpenTelemetry SDK. However, the autoconfiguration in Spring Boot Actuator does. That breaks the application if OpenTelemetry is only used for metrics.
• The OpenTelemetry autoconfiguration provided by Spring Boot Actuator for resource attributes is not used for the metrics, resulting in the standard service.name to be set to unkown_service rather than to the value of spring.application.name.

Minimal proposed changes to consolidate OpenTelemetry support in Spring for metrics and traces

Micrometer

• Introduce a MeterRegistry implementation built on top of OpenTelemetry and configurable via its standard approaches, so that it's possible to share configuration between metrics and traces.

Spring Boot Actuator

• Introduce autoconfiguration for OpenTelemetry metrics. The OpenTelemetry and Resource beans can be reused from the existing OpenTelemetry configuration. Besides that, the autoconfiguration should define defaults for SdkMeterProvider, OtlpHttpMetricExporter and OtlpGrpcMetricExporter.
• Introduce autoconfiguration for exporting OpenTelemetry traces via gRPC by defining a OtlpGrpcSpanExporter bean.
• Consolidate OpenTelemetry autoconfiguration classes across metrics and traces. The existing OpenTelemetryAutoConfiguration could be used for shared configuration. Besides that, dedicated autoconfiguration classes are needed to configure the specific exporters for each type of telemetry. Traces have that already in the OtlpAutoConfiguration.

Spring Initializr

• After going through all the previous changes, it should become straightforward adding an OpenTelemetry option to the Spring Initializr. Adding it now with the current state of things, it might make things confusing considering the different ways of configuring OpenTelemtry between metrics and traces.

Additional changes to extend OpenTelemetry support in Spring to logs

Spring Boot Actuator

• Introduce autoconfiguration for OpenTelemetry logs. The OpenTelemetry and Resource beans can be reused from the existing OpenTelemetry configuration. Besides that, the autoconfiguration should define defaults for SdkLoggerProvider, OtlpHttpLogRecordExporter and OtlpGrpcLogRecordExporter.
• Add support for Logback (default) and Log4J2. For inspiration, OpenTelemetry Java Instrumentation provides appenders for Logback and Log4J2.
• Possibly related issue on the Spring Boot project.

Further improvements

• An issue has been opened to investigate how to limit the number of configurable beans currently available in the tracing OpenTelemetryAutoConfiguration (this issue might be considered for the global OpenTelemetryAutoConfiguration once the tracing specific one gets deleted).

[mhalbritter]

Thanks for the work of summarizing the OTel support in Spring Boot!

Regarding Issue M.1: This is definitely a bug, I'm going to open an issue for that and fix it for the next release: #37284

[mhalbritter]

For issue M.2, i think we should set the spring.application.name in the OtlpConfig, too, like we do for tracing: #37285

[mhalbritter]

Issue M.3: This is not in Spring Boot's control and needs to be tackled in micrometer. I'll let the Micrometer team know.

[ThomasVitale]

@mhalbritter thanks a lot for your answer and for fixing those two issues so quickly!

[jonatan-ivanov]

Thanks for the issue! My two cents:

Issue M.3

it's not possible to configure the metrics exporter via standard OpenTelemetry configuration;

I'm not 100% sure what you mean by this, OtlpMeterRegistry let's you inject configuration through OTel env vars and you can configure resource attributes, see: OtlpConfig. Is there any particular config that you are missing? If it is the lack of SDK usage, it is by design, OTLP is not part of or depending on the SDK, and OTel itself encourages the community to use loose coupling and only use those bits that are really needed (this aligns well with other registries too).

it's not possible to use gRPC instead of HTTP/protobuf;

There are 3 transports available based on the OTel specs. Since SDKs (or components that are emitting OTLP) are not enforced to support all 3 (e.g.: The OTEL Java SDK only supports 2), the OTel Collectors (where you send the data) are enforced to support all 3. Because of this, it should not matter which transport is used, this should be transparent to the users. We also haven't got any user requests to add anything else other than HTTP yet. Is there a shortcoming you bumped into with HTTP that can be fixed using gRPC and not with HTTP? Fyi, there are known issues with gRPC if you also want to use it for other purposes as well: spring-projects-experimental/spring-cloud-sleuth-otel#39 and open-telemetry/opentelemetry-java#3123

it's not possible to share OpenTelemetry configuration in Spring Boot between metrics and traces (see, for example, issue M.2)

With M2 being fixed, is there any other particular config that you are missing? Please see my points above about resource attributes and the SDK.

OpenTelemetry bean and makes it possible to configure an SdkMeterProvider bean. What if Micrometer could use that for setting up the OpenTelemetry metrics exporter?

I think this was answered above by both you and me too ("module is lightweight and doesn't need any dependency on the OpenTelemetry SDK", OTLP is not part of the SDK, loose coupling) but I would not bring in any extra dependencies to Micrometer if it is easy to produce OTLP without them.

create a new Micrometer Registry OpenTelemetry module

The Micrometer-OTel bridge you linked is exactly this, isn't it? (fyi: it is not stable yet)

it would make it easier for developers using Micrometer in applications together with the OpenTelemetry Java Agent

I'm not sure I get this, as far as I know the agent does not care, it instruments things on its own (ignores Spring's own instrumentation) and it can also setup the bridge.

That shim would probably not be needed anymore.

I think that would mean that there would be another shim/bridge in Micrometer that seems unnecessary. Btw there is also an OTel -> Micrometer shim/bridge in OTel (other than the Micrometer -> OTel bridge)

Issue T1 - Exporting traces via gRPC

Please see my points about the transports and gRPC above (especially the issues you can face with using it). Right now using gRPC is quite simple (creating a bean) and maybe with the url it should be simplified even further but I would be curious about the justification: is there a shortcoming you bumped into with HTTP that can be fixed using gRPC and not with HTTP?

Minimal proposed changes

Micrometer: Your proposal for Micrometer to depend on the OTel SDK is not minimal. It is huge and right now seems unnecessary to me.

Initializr:

it should become straightforward adding an OpenTelemetry option to the Spring Initializr.

I'm not sure you are aware but OTel is not stable yet so I would be careful with this. Btw we are planning to add more OTel integrations once those bits stabilize and have a GA release.

Additional changes to extend OpenTelemetry support in Spring to logs

OTel Logging isn't stable yet either, IIRC the API+SDK are stable but the appenders are not, see -alpha at the end of the version numbers of
opentelemetry-log4j-appender and opentelemetry-logback-appender.

[mhalbritter]

Many thanks for the explanations, @jonatan-ivanov!

I've just created another issue to add support for OpenTelemetry's logging. With that, I think we have everything actionable from that issue on Boot side in smaller issues:

• OTel tracing over gRPC: Add auto-configuration for OTLP gRPC format #35863
• OTel metrics: Provide auto-configuration for OpenTelemetry metrics #36546
• OTel logging: Provide auto-configuration for OpenTelemetry logs #37355

[mhalbritter]

I'm going to close this issue in favor of the smaller ones. Thanks again for that write-up, @ThomasVitale!

[ThomasVitale]

Thanks a lot @jonatan-ivanov for the detailed answer, it helped clarified a few things. And thanks @mhalbritter for creating the issues to address this.

[rodrigorodrigues]

Hi @ThomasVitale ,

I was playing with this Grafana has a library that exports everything(logs, traces and metrics) to OpenTelemetry and works quite well, have a look on this https://grafana.com/docs/opentelemetry/instrumentation/java/spring-starter/ just remove the dependencies from spring boot and use this one, small config change that worked for me.

grafana:
  otlp:
    debug-logging: false
    on-prem:
      endpoint: http://localhost:4317
      protocol: grpc

[jonatan-ivanov]

@rodrigorodrigues You might want to be careful if you want to use it in prod:

• The logging appenders that the starter uses are not stable yet.
• The metrics setup that the starter uses is not stable (bridge) and makes things more complicated: instead of doing this: boot -> micrometer otlp registry (default and stable), it is doing this: boot -> micrometer -> otel-micrometer bridge (not stable) -> otel sdk -> otlp exporter.
• The tracing setup also contains unstable bits (semantic conventions). If you want to use stable bits, you can use the Brave bridge and send the spans in the Zipkin format.

See more details: spring-io/start.spring.io#1161

[rodrigorodrigues]

Thanks @jonatan-ivanov I'm just playing around for an article that I'm writing using OpenTelemetry for multiple services using different languages, will put a notice on that on the Spring Boot part.