Skip spans with insignificant duration

[ejsmith]

What would be the best way to completely skip sending spans and their children that have insignificant durations? I would like to avoid sending them out at all if they are less than like 10ms. Would I do this using some sort of wrapped activity exporter?

[cijothomas]

Collector (if you use it) would be the best fit for this kind of volume-reduction-techniques.

[ejsmith]

Hmm... that makes sense. Does the otel collector have any sort of ability to do this?

[cijothomas]

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor This might be what you are looking for.

[alanwest]

Tail-sampling processor would be a good fit if you were looking to sample entire traces based on some latency threshold. That said, operating the tail-sampling processor is complex as you need to ensure all spans for a given trace are routed through the same instance of the collector.

I don't think there's a component for the collector today that does quite what you want based on individual spans. The SpanProcessor looks promising, but I don't think it offers the ability to filter spans based on duration today. Though, I'd confirm this with the collector community.

Alternatively, you could consider doing this with the .NET SDK by implementing your own filter processor. We have an example here https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/extending-the-sdk/MyFilteringProcessor.cs.

[ejsmith]

Thanks @alanwest I think I'm going to try that because I just added a source to my activity data and it's super noisy and makes the traces almost unreadable. I would think skipping a span would be fine since if the span itself is insignificant, any child spans would be as well. This is basically for serialization and I want to see the spans only when they are more than X amount of duration.

[ejsmith]

@alanwest @cijothomas I tried this and, as with most things I've tried to do with this library, I got blocked by things being marked internal. I want to wrap the OTLP trace exporter and add the filtering capability, but I can't because I can't call TryEnableIHttpClientFactoryIntegration like the normal AddOtlpExporter method does.