[OpenTelemetry] Support OpenTelemetry Span Events

[cyrille-leclerc]

Elastic APM silently ignores the Otel Span Events.

Specification of OpenTelemetry Span Events: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#add-events

Example of Otel Span Event we want to support:

span.addEvent("order-creation", Attributes.of(
        EcommerceAppOpenTelemetryAttributes.CUSTOMER_ID, customerId,
        EcommerceAppOpenTelemetryAttributes.ORDER_PRICE, orderPrice,
        EcommerceAppOpenTelemetryAttributes.PAYMENT_METHOD, paymentMethod,
        EcommerceAppOpenTelemetryAttributes.SHIPPING_METHOD, shippingMethod,
        EcommerceAppOpenTelemetryAttributes.SHIPPING_COUNTRY, shippingCountry));

https://github.com/cyrille-leclerc/my-shopping-cart/blob/08f33761010d268793edb6e1d2101eaa579d830f/frontend-java/src/main/java/com/mycompany/ecommerce/controller/OrderController.java#L96

Source code ignoring the span events: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/91cbf1fd7f4eeea0087fdf782e3ff5b04e3ac90f/exporter/elasticexporter/internal/translator/elastic/traces.go#L88

[paulgrav]

Would be good get some movement on this issue. Currently our devs are logging to stdout given that Span Events aren’t supported in either the Elastic Exporter or Elastic APM Server, or worse bunging stuff in attributes.

[axw]

@paulgrav apologies for the silence, we've been rather busy. This isn't currently scheduled, but I'm hoping we'll be able to get to it within the next few months. My current thinking is that we'll index span events as logs, such that they'll show up in the trace logs UI:

Please let me know if that sounds off.

One of reasons why we haven't jumped on this is that APM Server does not create any log indices. We are working on migrating APM to data streams, and as part of that work we'll be introducing logs support. To get span events in sooner, we might support them only when producing data streams. This would mean using the Fleet integration, or waiting for #5914.

[paulgrav]

Thanks for the update.

I think indexing them in logs makes sense. That’s all the span events are. Our current workaround is to not call the Span Event API and send the data to stdout.

The workaround itself isn’t onerous, it’s a small code change. We have multiple services using the Jaeger tracer, traces are stored in ES, and are queryable in the Jaeger UI. Span logs, as they are known in the Open Tracing context, just work! We are trying to encourage developers to instead send their traces to Open Telemetry, which then forwards the traces to Elastic APM, but I then need to tell them that all their Span Log data will be dropped 😬

a) it’s not a good story for Open Telemetry or Elastic APM. It’s makes it harder for me to evangelise using those technologies.
b) the work around requires code changes that will become redundant once this issue is resolved

The likelihood is teams will be unwilling to migrate to Otel+ElasticAPM if their current tracing setup works, and they have to add issues to their backlog to work around the lack of Span Events.

Thinking about it, it’s probably still worth getting devs to send their data to Otel, but then have Otel forward the traces to Jaeger. Then once Span Events are supported by ElasticAPM, we can change the config to forward the traces to ElasticAPM.

[psk001]

Hi I have come across this issue while using openTelemetry with elastic APM with a distributed nodeJs service. My span events are stored in elastic indices and there data streams as well. But they are still not visible in logs part of traces as mentioned by @axw above. I just want to know if there is support for this feature or is there any issue with my apm server settings?

[axw]

@psk001 yes, span events are supported. Please open a topic in Discuss. The more information you can provide the better: which versions you're using, and ideally a sample program.

[psk001]

thanks for the quick reply @axw , I'll check on Discuss. Cheers

[astorm]

Looks like the discuss issue was opened here: https://discuss.elastic.co/t/opentelemetry-span-events-are-not-visible-in-apm-ui/314419