Spring Boot Starter does not flush telemetry on shutdown

[jasperkamerling]

Describe the bug

When using the Spring Boot Starter to configure the OTEL SDK telemetry data is not flushed before shutting down. This does happen when using the byte buddy java agent, see io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller#installOpenTelemetrySdk

Only the flush on the otlp http exporter is called (eg OtlpHttpLogRecordExporter#flush) but that does not actually flush the in memory data.

Steps to reproduce

• Set up a Spring Boot repo with the opentelemetry-spring-boot-starter dependency but without spring-web
• Startup the application (it should stop after a couple seconds)
• The logs should be exported to localhost now but that does not happen
  • If no tooling is running locally you should see failed http calls but that is not the case here.

Example repo: https://github.com/jasperkamerling/no-flush-demo

Expected behavior

Metrics, Traces and logs are flushed before shutting down

Actual behavior

Metrics, traces and logs are not flushed when shutting down the application. Leading to a loss of data.

Javaagent or library instrumentation version

2.17.0

Environment

JDK: 21.0.7
OS: Ubuntu 24.10

Additional context

We run a small Java job that starts up and shuts down several seconds later. Because the data is not flushed before shutdown we don't see any logs or metrics data.

I tried writing the code to fix this but I simply lack the knowledge of the OTEL framework to do this properly. If I can assist in any way please let me know.

[laurit]

@jasperkamerling as far as I can tell your sample application uses micrometer not otel for metrics. You might have better luck finding an answer from micrometer documentation or asking from the micrometer community. I presume you could add a shutdown hook and close the meter registry or spring application context from there and see whether that helpds.

[jasperkamerling]

@laurit sorry for the confusion. I was just using micrometer to create the meter, the metrics were still handled by the OTEL exporter. I have now removed micrometer and used a pure OTEL meter.
I could add a shutdown hook but the java agent does flush before shutdown, so it would be nice if the functionality matched to avoid other people losing metrics by accident.

And even if you want to create a custom shutdown hook you have to up cast the OpenTelemetry bean to the OpenTelemetrySdk class. I would like to avoid up casting objects when possible.

But since I'm not the maintainer of this project please feel free to close the issue if you do not want to support this functionality

[laurit]

I could add a shutdown hook but the java agent does flush before shutdown, so it would be nice if the functionality matched to avoid other people losing metrics by accident.

By default AutoConfiguredOpenTelemetrySdk that is used by both the agent and the spring starter registers a shutdown hook that closes the sdk when the jvm shuts down. Closing the sdk should flush the metrics. To me it is surprising that this works for the agent and not for the spring starter.

I was just using micrometer to create the meter, the metrics were still handled by the OTEL exporter.

As far as I know micrometer contains an OTLP metrics exporter that is not based on the otel sdk.

[jasperkamerling]

It seems to be a part of the foundations laid by this commit: 9635a5e

The java agent uses the following function to add the force flush on a shutdown:
io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller#installOpenTelemetrySdk

The spring version only uses the close function: io.opentelemetry.sdk.OpenTelemetrySdk#close. This only calls for a shutdown which does not actually flush the data. I could create a PR to add the flush before the shutdown calls. But this could add to the time it takes to shutdown the JVM which might not wanted in all cases.

A more backwards compatible solution could be to add a property to the SDK builder that adds the force flush functionality. This could then be added to the spring boot configuration.

But it might be important to decide if it makes sense to flush before a shutdown. I would appreciate this in most cases to try and get the last logs and metrics before the application shutdown.

[laurit]

The force flush function set up in the linked code is called only from the aws lambda instrumentation, it is not called during agent shutdown. Shutdown calls https://github.com/open-telemetry/opentelemetry-java/blob/b203d2f23e9e99be402f73a051c46fcceb386457/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/export/PeriodicMetricReader.java#L111 that is also called in the force flush https://github.com/open-telemetry/opentelemetry-java/blob/b203d2f23e9e99be402f73a051c46fcceb386457/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/export/PeriodicMetricReader.java#L85

[jasperkamerling]

Ah sorry I missed that part.
But would you be okay with creating a property in the spring boot starter to force flush records before shutting down?

[laurit]

But would you be okay with creating a property in the spring boot starter to force flush records before shutting down?

As far as I can tell shutdown already does a flush so it shouldn't really be needed.

[laurit]

@jasperkamerling I tried running your sample application and as far as I can tell it is flushing the metrics on shutdown.

[jasperkamerling]

I'm working on a full stack demo for reasons unrelated to this ticket. Once that is done and i can reliably show the difference I'll reopen the ticket.

Thank you for taking the time to help investigate so far.