Additional low-level instrumentation spans created for some instrumentation libraries

[ezhang6811]

Some instrumentation libraries, when enabled along with Net:Http instrumentation, generate multiple spans for a single request.

For example, both Faraday and AWS SDK instrumentation libraries generate multiple client spans (and thus multiple metrics) due to the underlying Net:Http instrumentation that is also invoked.

Would it be possible to suppress the Net:Http instrumentation in these higher level libraries such as Faraday and AWS SDK to prevent these extra spans when both libraries are enabled?

[arielvalentin]

There are many discussion about this challenges of having multiple client spans, which is likely why we have not attempted to address these:

open-telemetry/opentelemetry-specification#110
open-telemetry/opentelemetry-specification#526
open-telemetry/opentelemetry-specification#4179

Would it be possible to suppress the Net:Http instrumentation in these higher level libraries such as Faraday and AWS SDK to prevent these extra spans when both libraries are enabled?

The SDK does not have a good mechanism built in for sorting dependencies, and AFAIK libraries like Faraday do not have a way to let its middleware know which driver it is currently using let alone know if it is being instrumented or not.

I think the best we are able to offer at the moment is to configure Faraday to create an internal span for all connections:

opentelemetry-ruby-contrib/instrumentation/faraday/lib/opentelemetry/instrumentation/faraday/instrumentation.rb

Line 29 in c64487d

option :span_kind, default: :client, validate: %i[client internal]

Of course if that has changed then yes, I would be happy to support that.

As for the AWS SDK is concerned, I would have to ask from the AWS Ruby maintainers to see if they have any insights to share with us. cc: @jterapin @alextwoods

[jterapin]

Hi there! AWS SDK for Ruby here.

From what I observed, when using both opentelemetry-instrumentation's aws_sdk and net_http gems - you could configure suppress_internal_instrumentation to true on the OpenTelemetry::Instrumentation::AwsSdk to suppress internal spans within the AWS SDK. This suppression includes any spans generated by the net_http as well.

To demonstrate:

require 'aws-sdk-s3'
require 'opentelemetry-instrumentation-net_http'
require 'opentelemetry-instrumentation-aws_sdk'
require 'opentelemetry-exporter-otlp'

ENV['OTEL_TRACES_EXPORTER'] ||= 'console'

OpenTelemetry::SDK.configure do |c|
  c.use 'OpenTelemetry::Instrumentation::Net::HTTP'
  c.use 'OpenTelemetry::Instrumentation::AwsSdk', {
    suppress_internal_instrumentation: true
  }
end

otel_provider = Aws::Telemetry::OTelProvider.new
client = Aws::S3::Client.new(telemetry_provider: otel_provider)
client.list_buckets

And its output:

I, [2025-04-24T07:25:32.189012 #70251]  INFO -- : Instrumentation: 
  OpenTelemetry::Instrumentation::AwsSdk was successfully installed with the following options 
  {inject_messaging_context: false, suppress_internal_instrumentation: true}
I, [2025-04-24T07:25:32.189452 #70251]  INFO -- : Instrumentation: 
  OpenTelemetry::Instrumentation::Net::HTTP was successfully installed with the following options
   {untraced_hosts: []}
#<struct OpenTelemetry::SDK::Trace::SpanData
 name="S3.ListBuckets",
 kind=:client,
 status=#<OpenTelemetry::Trace::Status:0x0000000140c200b0 @code=1, @description="">,
 ...
>

I want to say this behavior only applies to AWS SDK V3 when using AWS service gems released after 2024-09-03 since that's when we added integration between the AWS SDK and the instrumentation gem here.

[ezhang6811]

Hi @jterapin,

Thanks for the follow up - I noticed this suppression config mechanism when checking out the implementation for AWS SDK instrumentation.

I'm curious, could it be possible to have this be the default behavior? Currently the additional low-level instrumentation generates additional dependency metrics which affects costs for customers often unknowingly. We could document this better for new users but it seems less intuitive to have to manually suppress the internal spans, especially if the customer wants to use c.use_all() to enable their instrumentation.

[jterapin]

Hi @ezhang6811! Sorry for the delay. I wanted to circle back with my team to discuss and we all are in agreement that this is reasonable to change the default to suppress_internal_instrumentation = true.

I have been meaning to update this gem for unrelated changes so I will include this work to that backlog as well.

[ezhang6811]

Hi @jterapin, thanks for the update! Great to hear, I will update the Faraday instrumentation as well and discuss separately. Thanks!

[github-actions]

👋 This issue has been marked as stale because it has been open with no activity. You can: comment on the issue or remove the stale label to hold stale off for a while, add the keep label to hold stale off permanently, or do nothing. If you do nothing this issue will be closed eventually by the stale bot.