OTLP exporter error with non-global autoconfigured OpenTelemetry instance

[steffan-westcott]

Describe the bug

Using autoconfigure to create a global OpenTelemetry instance works correctly with the OTLP exporter for traces and metrics export.
However, creating a non-global OpenTelemetry instance has some issues. Two symptoms occur:

• Two metrics otlp_exporter_exported and otlp_exporter_seen are no longer exported
• An error is shown once in the console, around the time of first metrics export:

Sept 28, 2023 9:58:32 PM io.opentelemetry.api.GlobalOpenTelemetry maybeAutoConfigureAndSetGlobal
INFO: AutoConfiguredOpenTelemetrySdk found on classpath but automatic configuration is disabled. To enable, run your JVM with -Dotel.java.global-autoconfigure.enabled=true

No traces or metrics created by the application itself are impacted, they are exported correctly as expected. It would appear the OTLP exporter is attempting to use the global OpenTelemetry instance for these two internal metrics created by the exporter, despite autoconfiguration for the global instance being disabled.

Steps to reproduce

Use the files below to create a Gradle Java project, and a Docker Compose configuration for spinning up a Collector, Jaeger and Prometheus instances.

src/main/java/org/example/OtlpExporterIssue.java

package org.example;

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;

public final class OtlpExporterIssue {
    public static void main(String[] args) throws InterruptedException {

        // Make a *non-global* OpenTelemetry instance, as per advice given by Otel team
        OpenTelemetry openTelemetry = AutoConfiguredOpenTelemetrySdk.builder().build().getOpenTelemetrySdk();

        String scopeName = OtlpExporterIssue.class.getName();
        Tracer tracer = openTelemetry.getTracer(scopeName);
        Span span = tracer.spanBuilder("important work").setAttribute("foo", 42).startSpan();
        span.end();
        Thread.sleep(30 * 1000);
    }
}

build.gradle

plugins {
    id 'com.github.johnrengelman.shadow' version '8.1.1'
    id 'java'
}

repositories {
    mavenCentral()
}

dependencies {
    implementation("io.opentelemetry:opentelemetry-api:1.30.1")
    implementation("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:1.30.1")
    runtimeOnly("io.opentelemetry:opentelemetry-exporter-otlp:1.30.1")
}

gradle.properties

rootProject.name = 'otlp-exporter-issue'

compose.yaml

services:

  jaeger:
    image: jaegertracing/all-in-one:latest
    command:
      - --collector.otlp.enabled=true
    ports:
      - "16686:16686"

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus.yaml
    command:
      - --config.file=/etc/prometheus.yaml
      - --web.enable-remote-write-receiver
    ports:
      - "9090:9090"

  otel-collector:
    image: otel/opentelemetry-collector:latest
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml
    command:
      - --config=/etc/otel-collector.yaml
    ports:
      - "4317:4317"
    depends_on:
      - jaeger
      - prometheus

otel-collector.yaml

receivers:
  otlp:
    protocols:
      grpc:

processors:
  batch:

exporters:
  otlp/jaeger:
    endpoint: "jaeger:4317"
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ otlp/jaeger ]

    metrics:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ prometheusremotewrite ]

prometheus.yaml

global:
  evaluation_interval: 15s

Use the following commands to build the project, start the telemetry backends and run the project:

./gradlew shadowJar

docker compose up -d

java -Dotel.traces.exporter=otlp \
     -Dotel.metrics.exporter=otlp \
     -Dotel.logs.exporter=none \
     -Dotel.resource.attributes=service.name=otlp-exporter-issue \
     -Dotel.metric.export.interval=5000 \
     -cp build/libs/otlp-exporter-issue-all.jar \
     org.example.OtlpExporterIssue

Wait at least 10 seconds before looking in Prometheus / Jaeger for exported telemetry.

What did you expect to see?

• No errors appearing in the console
• When clicking on the Metrics Explorer icon in Prometheus (at http://localhost:9090 ), I would expect to see the same metrics when using a global autoconfigured OpenTelemetry, namely:
  • otlp_exporter_exported
  • otlp_exporter_seen
  • processedSpans
  • queueSize
  • target_info

What did you see instead?

• A single error in the console

Sept 28, 2023 9:58:32 PM io.opentelemetry.api.GlobalOpenTelemetry maybeAutoConfigureAndSetGlobal
INFO: AutoConfiguredOpenTelemetrySdk found on classpath but automatic configuration is disabled. To enable, run your JVM with -Dotel.java.global-autoconfigure.enabled=true

• otlp_exporter_exported and otlp_exporter_seen are missing in the Metrics Explorer in Prometheus
  • processedSpans
  • queueSize
  • target_info

What version and what artifacts are you using?

As noted in the build.gradle file above:

• io.opentelemetry:opentelemetry-api:1.30.1
• io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:1.30.1
• io.opentelemetry:opentelemetry-exporter-otlp:1.30.1

Environment

• Compiler: "Temurin 17.0.7"
• OS: "Ubuntu 20.04"

Additional context

The error does not occur if using a global OpenTelemetry instance. To see this, change a line in the Java program to the following:

OpenTelemetry openTelemetry = AutoConfiguredOpenTelemetrySdk.initialize().getOpenTelemetrySdk();

[jack-berg]

This is related to #5021. #5021 talks about the general case, and this issue discusses the specific case of OTLP exporters not being able to self instrument when GlobalOpenTelemetry.get() is not set.

Let's get this fixed. I have some ideas on how to do this, but we also welcome contributions 🙂