OpenTelemetry exporter (otlp) startup dependency error when running as a Docker container image

[yunusevren]

Describe the bug

Using the OpenTelemetry guide, I'm configuring my Quarkus application to push traces and spans to an external OTel collector endpoint. This works fine when I run the application with ./mvnw quarkus:dev or when running the application through IntelliJ Quarkus Plugin. The application starts and I can verify that the GET /hello API calls result in spans that are pushed to my OTel collector.

However, when I run the same application as a Docker container, it runs into a dependency error during startup. For some reason the startup process is not able to find the "exporter-otlp" classes in the classpath, even though the otlp dependencies opentelemetry-exporter-otlp-common and opentelemetry-exporter-otlp are provided in my pom.xml file (as indicated in OpenTelemetry exporter section).

application.properties file:

quarkus.otel.service.name=opentelemetry-reproducer-test
quarkus.otel.traces.exporter=otlp
quarkus.otel.exporter.otlp.traces.endpoint=http://192.168.0.37:4317
quarkus.otel.exporter.otlp.traces.headers=Authorization=Bearer XXX

Docker container build and run steps:

./mvnw clean package -Dquarkus.container-image.build=true -Dquarkus.container-image.group=acme

docker run --rm --name opentelemetry-reproducer -p 8080:8080 -e QUARKUS_PROFILE=dev acme/opentelemetry-reproducer:1.0.0-SNAPSHOT

Error during startup:

INFO exec -a "java" java -Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -XX:MaxRAMPercentage=50.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/quarkus-run.jar 
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2023-09-06 14:27:21,144 INFO  [io.ope.sdk.aut.AutoConfiguredOpenTelemetrySdkBuilder] (main) Error encountered during autoconfiguration. Closing partially configured components.
2023-09-06 14:27:21,179 ERROR [io.qua.run.Application] (main) Failed to start application (with profile [dev]): java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
        at io.quarkus.runtime.Application.start(Application.java:101)
        at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
        at io.quarkus.runner.GeneratedMain.main(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:61)
        at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:32)
Caused by: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [2deeb341bce4a230183bf3a228841625fe2519ae]: io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: otel.traces.exporter set to "otlp" but opentelemetry-exporter-otlp not found on classpath. Make sure to add it as a dependency.
        at io.opentelemetry.api.OpenTelemetry_2deeb341bce4a230183bf3a228841625fe2519ae_Synthetic_Bean.doCreate(Unknown Source)
        at io.opentelemetry.api.OpenTelemetry_2deeb341bce4a230183bf3a228841625fe2519ae_Synthetic_Bean.create(Unknown Source)
        at io.opentelemetry.api.OpenTelemetry_2deeb341bce4a230183bf3a228841625fe2519ae_Synthetic_Bean.create(Unknown Source)
        at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113)
        at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37)
        at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34)
        at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
        at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
        at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:34)
        at io.opentelemetry.api.OpenTelemetry_2deeb341bce4a230183bf3a228841625fe2519ae_Synthetic_Bean.get(Unknown Source)
        at io.opentelemetry.api.OpenTelemetry_2deeb341bce4a230183bf3a228841625fe2519ae_Synthetic_Bean.get(Unknown Source)
        at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:557)
        at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:537)
        at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:570)
        at io.quarkus.arc.impl.ArcContainerImpl$3.get(ArcContainerImpl.java:334)
        at io.quarkus.arc.impl.ArcContainerImpl$3.get(ArcContainerImpl.java:331)
        at io.quarkus.arc.runtime.BeanContainerImpl$1.create(BeanContainerImpl.java:46)
        at io.quarkus.arc.runtime.BeanContainer.beanInstance(BeanContainer.java:25)
        at io.quarkus.opentelemetry.runtime.tracing.intrumentation.InstrumentationRecorder.setupVertxTracer(InstrumentationRecorder.java:33)
        at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx280172193.deploy_0(Unknown Source)
        at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx280172193.deploy(Unknown Source)
        ... 13 more
Caused by: io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: otel.traces.exporter set to "otlp" but opentelemetry-exporter-otlp not found on classpath. Make sure to add it as a dependency.
        at io.opentelemetry.sdk.autoconfigure.SpanExporterConfiguration.configureExporter(SpanExporterConfiguration.java:95)
        at io.opentelemetry.sdk.autoconfigure.SpanExporterConfiguration.configureSpanExporters(SpanExporterConfiguration.java:66)
        at io.opentelemetry.sdk.autoconfigure.TracerProviderConfiguration.configureTracerProvider(TracerProviderConfiguration.java:51)
        at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:356)
        at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:79)
        at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:52)
        at io.opentelemetry.api.OpenTelemetry_2deeb341bce4a230183bf3a228841625fe2519ae_Synthetic_Bean.createSynthetic(Unknown Source)
        ... 34 more

Expected behavior

Application startup is successful regardless of how it's ran (i.e through Maven, IDE or as a Docker container)

Actual behavior

Application startup is successful when running through Maven or IDE, but it fails when it's ran as a Docker container

How to Reproduce?

1. Clone the reproducer Quarkus project: https://github.com/yunusevren/opentelemetry-reproducer
   • git clone git@github.com:yunusevren/opentelemetry-reproducer.git
2. cd opentelemetry-reproducer
3. Run with ./mvnw quarkus:dev. Observe successful startup.
4. Stop the application
5. Run as Docker container:
   • Build image: ./mvnw clean package -Dquarkus.container-image.build=true -Dquarkus.container-image.group=acme
   • Run: docker run --rm --name opentelemetry-reproducer -p 8080:8080 -e QUARKUS_PROFILE=dev acme/opentelemetry-reproducer:1.0.0-SNAPSHOT
   • Startup fails with errors

Output of uname -a or ver

Darwin xxxxxxxx.local 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64

Output of java -version

openjdk 11.0.17 2022-10-18 OpenJDK Runtime Environment Temurin-11.0.17+8 (build 11.0.17+8) OpenJDK 64-Bit Server VM Temurin-11.0.17+8 (build 11.0.17+8, mixed mode)

GraalVM version (if different from Java)

N/A

Quarkus version or git rev

3.3.1

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.3 (21122926829f1ead511c958d89bd2f672198ae9f) Maven home: /Users/xxxxxxxx/.m2/wrapper/dists/apache-maven-3.9.3-bin/326f10f4/apache-maven-3.9.3 Java version: 11.0.17, vendor: Eclipse Adoptium, runtime: /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home Default locale: en_TR, platform encoding: UTF-8 OS name: "mac os x", version: "13.5.1", arch: "aarch64", family: "mac"

Additional information

Docker version 24.0.2, build cb74dfc

[quarkus-bot]

/cc @brunobat (opentelemetry), @radcortez (opentelemetry)

[yunusevren]

New insight: Quarkus versions 3.3.0 and 3.3.1 reproduce the issue. If I use version 3.2.5.Final, the error does not occur.

[brunobat]

This is because we are using a new exporter from 3.3.0+
You don't need to set quarkus.otel.traces.exporter=otlp or add the opentelemetry-exporter-otlp-common and opentelemetry-exporter-otlp dependencies to the pom.
Remove them and you will be using the default.

I'm looking into this, regardless.

[yunusevren]

When I remove quarkus.otel.traces.exporter=otlp and the two dependencies, the default exporter (VertxGrpcExporter) runs into the following SSL error when exporting spans. My remote collector is using a secure (HTTPS) endpoint quarkus.otel.exporter.otlp.traces.endpoint=https://xxx:4317. For some reason it seems that it's not able to find any trusted certificate. Applies to Maven quarkus:dev as well as running as a Docker container.

SEVERE [io.qua.ope.run.exp.otl.VertxGrpcExporter] (vert.x-eventloop-thread-1) Failed to export spans. The request could not be executed. Full error message: Failed to create SSL connection

Underlying stack trace:

javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
        at io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:127)
        at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:400)
        at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376)
        at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368)
        at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1270)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1251)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at java.base/sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:101)
        at java.base/sun.security.validator.Validator.getInstance(Validator.java:181)
        at java.base/sun.security.ssl.X509TrustManagerImpl.getValidator(X509TrustManagerImpl.java:309)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(X509TrustManagerImpl.java:183)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:255)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
        ... 20 more
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at java.base/java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
        at java.base/java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120)
        at java.base/java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104)
        at java.base/sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:98)
        ... 37 more

[brunobat]

TLS communication requires you to setup the certificate path:
quarkus.otel.exporter.otlp.traces.key-cert.certs
and the keys path:
quarkus.otel.exporter.otlp.traces.key-cert.keys

[brunobat]

Most likely the original problem of the issue happens because we are not retaining the relevant classes when quarkus.otel.traces.exporter=otlp

[brunobat]

Removed the related documentation until #35796 is implemented

[jimkont]

Not sure if it is the same underlying issue, but with cdi and in an AWS Lambda deployment I get this error:

"message": "Failed to export spans. The request could not be executed. Full error message: Failed to create SSL connection"
"class": "io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter",
"method": "doLog",
"lineOfCode": 82,
"threadName": "vert.x-eventloop-thread-1",
"processName": "AWSLambda",
"sequence": 83

(no stack trace), here's my configuration:

  otel:
    traces:
      exporter: cdi
    service:
      name: ${OTEL_SERVICE_NAME}
    exporter:
      otlp:
        traces:
          protocol: http/protobuf # fails with either grpc or http
          endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}

[brunobat]

Not sure if it is the same underlying issue, but with cdi and in an AWS Lambda deployment I get this error:

"message": "Failed to export spans. The request could not be executed. Full error message: Failed to create SSL connection"
"class": "io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter",
"method": "doLog",
"lineOfCode": 82,
"threadName": "vert.x-eventloop-thread-1",
"processName": "AWSLambda",
"sequence": 83

(no stack trace), here's my configuration:

  otel:
    traces:
      exporter: cdi
    service:
      name: ${OTEL_SERVICE_NAME}
    exporter:
      otlp:
        traces:
          protocol: http/protobuf # fails with either grpc or http
          endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}

Seems unrelated. You have not configured the key and cert for the TLS endpoint in the Quarkus extension certificate path:
quarkus.otel.exporter.otlp.traces.key-cert.certs
and the keys path:
quarkus.otel.exporter.otlp.traces.key-cert.keys
You don't need to specify exporter: cdi because it's already the default.

[jimkont]

Thanks for the prompt response. Quick check, Is the key-cert.* configuration a new behavior? because this was working fine with v3.0.4 without configuring any certificates. I started getting this error as soon as I tried out 3.3.x

[brunobat]

We had no TLS communication support before, on the exporters, @jimkont.
You can ignore certificates validation in Quarkus with quarkus.tls.trust-all=true but this is NOT recommended in production environments.

[jimkont]

Just to confirm, our quarkus/aws-lambda application was successfully sending traces to our production https GRPC endpoint with quarkus version 3.0.4. I also just tried out version 3.1.3 and 3.2.4, and all export traces successfully.
Once I use 3.3.2 I start getting the SSL error described above.

My initial assumption was that this is related to this issue because it is related to the new OTEL refactoring in v3.3.x.
The issue version and the Failed to create SSL connection error message confused me in assuming this was the same issue I was experiencing.

If I understood correctly, Quarkus has not officially supported TLS so far, and our TLS export was probably coincidental (through some external dependency maybe? does not matter I guess). With the new OTEL refactoring quarkus is more strict and we need to specify the key-cert.* manually. If this is the case, I will try to locate and manually specify those certs

[brunobat]

@geoand I'd say the previous OkHttp based implementation was handling the certificate resolution if no cert was manually provided.

[geoand]

I'll have a look

[geoand]

So it seems to me we have two different things here.

In the original issue description our exporters never come into play because of quarkus.otel.traces.exporter=otlp and the real problem is why it works in dev-mode and not prod mode (I'll look into why this is happening).

The issue @jimkont mentions seems entirely different and I would suggest opening a new issue and attaching a sample that worked in Quarkus 3.2 and does not in 3.3+.

[geoand]

#35862 fixes this issue