Skip to content

[Bug] @temporalio/interceptors-opentelemetry OTEL bypasses a required SDK component #1779

@sdedovic

Description

@sdedovic

What are you really trying to do?

When using the @temporalio/interceptors-opentelemetry library I am seeing TONS of these errors:

ERROR --- node_modules/@opentelemetry/api/build/src/api/diag.js : ["Accessing resource attributes before async attributes settled"]

which are caused by the Temporal SDKs improper use of the OTEL SDK.

Describe the bug

See this comment on the issue in the OTEL SDK: open-telemetry/opentelemetry-js#5828 (comment). To quote the linked comment, from one of the OTEL maintainers

Looking at the code it looks like the following is happening:\

  • @temporalio/interceptors-opentelemetry bypasses the SpanProcessor which is supposed to await the async attributes and passes spans directly to SpanExporter
  • the SpanExporter then assumes that the resource was already awaited by the SpanProcessor, but since @temporalio/interceptors-opentelemetry bypassed the SpanProcessor this never happened.
  • during normal operation of an OTel SDK, this log is not written - this happens since @temporalio/interceptors-opentelemetry bypasses a required SDK component.

Recommendation: @temporalio/interceptors-opentelemetry should accept a SpanProcessor instead of a SpanExporter in makeWorkflowExporter(), and pass the spans to SpanProcessor#onEnd(Span) instead. This will properly await the resource attributes before exporting them, log errors and and handle batching in a user-configurable way, which may be desirable.

Minimal Reproduction

See last comment on this issue: open-telemetry/opentelemetry-js#5828 (comment)

export function makeWorkflowExporter(
  spanProcessor: SpanProcessor, // used with a BatchSpanProcessor for instance, which forwards to a SpanExporter
  resource: Resource
): InjectedSink<OpenTelemetryWorkflowExporter> {
  // note: somehow, `SpanProcessor#shutdown()` needs to be wired up somewhere to avoid data loss on shutdown.
  return {
    export: {
      fn: (info, spanData) => {
        const spans = spanData.forEach((serialized) => {
          Object.assign(serialized.attributes, info);
          // Spans are copied over from the isolate and are converted to ReadableSpan instances
          spanProcessor.onEnd(extractReadableSpan(serialized, resource));
        });
      },
    },
  };
}

Environment/Versions

  • OS and processor: x86_64-linux
  • Temporal Version: [e.g. 1.14.0?] and/or SDK version
  • Temporal from source

Additional context

This will affect #1741

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions