Support for OpenTelemetry

[martinosk]

Dafda should support OpenTelemetry, preferably as an add-on.
It could be done by intercepting the message processing pipeline and injecting or extracting trace metadata.
I've made some quick and dirty code that demonstrates OpenTelemetry using Dafda by injecting it into the actual message payload:
Consumer: https://github.com/martinosk/OpenTelemetryPoc/blob/master/AnEventConsumer/SomeMessageHandler.cs
Producer: https://github.com/martinosk/OpenTelemetryPoc/blob/master/SomeOtherApi/MessageProducer.cs

Real implementation should of course be in metadata, possible re-using the CausationId header?

There's an issue suggesting that it could be implemented in confluent-kafka-dotnet confluentinc/confluent-kafka-dotnet#1269 but it seems to have died after someone suggested to implemented in C in librdkafka.

[martinosk]

Some specs to consider when implementing: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/messaging.md#apache-kafka

[JohannesEH]

There's also this ressource now: https://opentelemetry.io/blog/2022/instrument-kafka-clients/

[toizo]

So Team Track is already looking into open telemetry, and this would be a nice feature to have. I'm willing to chip in with the change, but before we start, let's align. @martinosk, are you still convinced this should be just an add on? What would be a use case for not enabling it?

We currently use the beta that @thofisch had worked on, and it seems like a step in the right direction. We could just refactor the instrumenting bit into a cleaner state and do a [more or less] same thing for the producer as well.

[martinosk]

@toizo no, not convinced. Primarily I wanted to ensure that the core Dafda package only carries an absolute minimum of dependencies to be as un-intrusive as possible. But seeing that open telemetry is becoming a standard and is being closer embedded in the .net ecosystem, there might no longer be a case for having it as an add-on to Dafda.