-
Notifications
You must be signed in to change notification settings - Fork 136
Description
What are you really trying to do?
When using the @temporalio/interceptors-opentelemetry
library I am seeing TONS of these errors:
ERROR --- node_modules/@opentelemetry/api/build/src/api/diag.js : ["Accessing resource attributes before async attributes settled"]
which are caused by the Temporal SDKs improper use of the OTEL SDK.
Describe the bug
See this comment on the issue in the OTEL SDK: open-telemetry/opentelemetry-js#5828 (comment). To quote the linked comment, from one of the OTEL maintainers
Looking at the code it looks like the following is happening:\
- @temporalio/interceptors-opentelemetry bypasses the SpanProcessor which is supposed to await the async attributes and passes spans directly to SpanExporter
- the SpanExporter then assumes that the resource was already awaited by the SpanProcessor, but since @temporalio/interceptors-opentelemetry bypassed the SpanProcessor this never happened.
- during normal operation of an OTel SDK, this log is not written - this happens since @temporalio/interceptors-opentelemetry bypasses a required SDK component.
Recommendation: @temporalio/interceptors-opentelemetry should accept a SpanProcessor instead of a SpanExporter in makeWorkflowExporter(), and pass the spans to SpanProcessor#onEnd(Span) instead. This will properly await the resource attributes before exporting them, log errors and and handle batching in a user-configurable way, which may be desirable.
Minimal Reproduction
See last comment on this issue: open-telemetry/opentelemetry-js#5828 (comment)
export function makeWorkflowExporter(
spanProcessor: SpanProcessor, // used with a BatchSpanProcessor for instance, which forwards to a SpanExporter
resource: Resource
): InjectedSink<OpenTelemetryWorkflowExporter> {
// note: somehow, `SpanProcessor#shutdown()` needs to be wired up somewhere to avoid data loss on shutdown.
return {
export: {
fn: (info, spanData) => {
const spans = spanData.forEach((serialized) => {
Object.assign(serialized.attributes, info);
// Spans are copied over from the isolate and are converted to ReadableSpan instances
spanProcessor.onEnd(extractReadableSpan(serialized, resource));
});
},
},
};
}
Environment/Versions
- OS and processor: x86_64-linux
- Temporal Version: [e.g. 1.14.0?] and/or SDK version
- Temporal from source
Additional context
This will affect #1741