OpenTelemetry Bridge for APM Agents

[AlexanderWert]

Elastic APM agents should support manual instrumentation through the OpenTelemetry API.
The scope of this issue is on providing an OpenTelemetry bridge in the Elastic APM agents to bridge / intercept the OTel tracing API and capture Elastic APM data.

Spec issue

• [META 524] Spec for OTel APM Bridge #525

Agent issues

• [META 524] OpenTelemetry Bridge apm-agent-java#2191
• [META 524] OpenTelemetry Bridge apm-agent-dotnet#1521
• [META 524] OpenTelemetry Bridge apm-agent-go#1140
• [META 524] OpenTelemetry Bridge apm-agent-nodejs#2369
• [META 524] OpenTelemetry Bridge apm-agent-python#1362

[Aubermean]

Elastic Observability/APM has matured quite a bit since 2021 but Ruby support is still missing! Any chance of some Ruby love? According to this, OpenTelemtry Ruby is used by Github, Heroku, Shopify, Puppet and more...

[estolfo]

Hi @Aubermean we don't have plans to implement the OpenTelemetry bridge in the Elastic APM Ruby agent but have you checked out the OpenTelemetry Ruby project? Are there features or instrumentations that are missing there that you'd like?

[Aubermean]

Hi @estolfo thanks for replying so quickly. We are basically missing HTTP requests in the APM as we use Typhoeus/Ethon, which is not supported by ElasticAPM Ruby. OpenTelemetry does have support, and its mature and works. So we want to have the HTTP requests show in our transactions, without having to reduplicate and maintain the logic from OpenTelemtry. You do support Faraday, but we cannot use it with Typhoeus as there is missing concurrency features only available with Typhoeus/Ethon directly.

Based on the docs for Elastic APM and OpenTelemtry integration, it seemed that the Ruby library was missing support for being able to 'transparently' include OpenTelemetry events in an Elastic APM transaction. That is what I thought the bridge feature was, but maybe I am mistaken.

Hope that makes sense, if you can give any pointers would be great, its the final piece of the puzzle to move away from other APM directly to Elastic (and we are trying to get a working demo before our trial ends!)

[estolfo]

Hi @Aubermean thanks for the details. Can you tell me what you mean by we want to have the HTTP requests show in our transactions, without having to reduplicate and maintain the logic from OpenTelemtry ?
Is there something keeping you from being able to completely switch over to using the OpenTelemetry Ruby instrumentations / SDK / API?

[Aubermean]

Hi @estolfo, sorry it took so long to follow up.

We are trying to use the elastic apm ruby gem in Rails. Faraday, HTTP, and Net::HTTP are supported for instrumentation but Ethon is not. OpenTelemetry supports ethon and the Elasticsearch client ruby gem supports and even suggests OpenTelemetry. Simply put, we are unsure how to get instrumentation/information on the Ethon transactions like is possible with other HTTP libraries. Unfortunately we cannot switch library and this is the only blocker from dropping Datadog and having everything we need in Elastic. With the OpenTelemetry bridge for java, dotnet, nodejs, python, all implemented... its really sad that its missing for Ruby.

[estolfo]

Hi @Aubermean would you mind sending me an email to continue the discussion? I'd like to understand more about your specific use case and inability to switch to the OTel Ruby SDK and instrumentations.
It's my first and last name, separated by a dot at elastic.co. Thanks