Improve OpenTelemetry (OTLP) instrumentation

[thepabloaguilar]

Problem

It's not a problem but we could improve the actual OpenTelemetry instrumentation!

Ideal Solution

For Tracing:

• Sampler Ratio, to avoid unnecessary high load of spans coming out from the service a Sampler Ratio and exposing this config, the default value should be 1 (100%)
• Allow defining propagators, the propagators has many flavors b3, jaeger, opensensus and etc. This is important so spans can be linked properly
• Add more resource information, I think we could add more information that can help somehow the platforms which ingests those data (eg.: Datadog) linking better with other resources or even the person seeing the span. My suggestions are:
  • WithTelemetrySDK
  • WithContainer
  • WithHost

For metrics:

• I saw Flipt expose the metrics using Prometheus AND OpenTelemetry, one difference between these two is the Prometheus lib already expose the Go runtime metrics in the other hand Otel doesn't. At least not automatically, we could add by using this package: go.opentelemetry.io/contrib/instrumentation/runtime

Search

• I searched for other open and closed issues before opening this

Additional Context

No response

[thepabloaguilar]

I also saw the attributes Flipt puts in the Span, I also would like to suggest to maybe change the keys to match the ones in here Semantic Conventions for Feature Flags in Spans, it does not have all flags that flipt uses but the other maybe can just receive the feature_flag. or flipt. prefix!

[markphelps]

I love all of these suggestions! ❤️ is this something you'd be willing to help with @thepabloaguilar ?

[thepabloaguilar]

Yeah, I could do it!

[thepabloaguilar]

I'll open another issue to address the metrics Stuff to keep this issue more concise! The thing is, we're using OpenTelemetry Metric to export to Prometheus so we need to make configurable what metrics exporter we want to enable: Prometheus or OpenTelemetry

Both could be enabled at the same time if user wants!