-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OpenTelemetry Trace forwarding to subscribers is not working correctly #12752
Comments
Hi @singhbaljit. This is an intended behavior that is based on MQTT spec and OpenTelemetry semantic conventions for messaging systems:
|
In other words, traces emitted by EMQX provide extra level of details and cover intermediary level (messaging system itself). |
@SergeTupchiy thank you for responding promptly, and thank you for referencing the documentation. It was wrong for me to call it "incorrect". EMQX is following the MQTT v5 spec correctly: that is, the user properties are not mutated. However, I do still think this is functionally incorrect. Here's effectively what the trace map looks like: |
I also don't think the OpenTelemetry document is saying to forward I do think the MQTT v5 spec and the W3C context propagation for MQTT are in contradiction. I can raise this issue there as well. But do you think it possible to add another configuration to enable overriding the |
Doesn't it say that any tracing done by messaging system itself is internal and should not affect producer/consumer correlation? Yes, it is technically feasible to override |
What happened?
I've OpenTelemetry traces enabled, along side Trace All Messages. Following instructions from the docs:
traceparent
, EQMX sends the same value to the subscribers.This is incorrect.The W3Ctraceparent
includes a trace and span id (<version>-<traceid>-<spanid>-<flags>
). The span id in the published message belongs to the publisher. EMQX should overwrite this value with its own span. Namely, the span in the message received by the subscriber should be thesend_published_message
span. This allows downstream processes to use the new span as its parent to continue building the distributed trace chain.traceparent
andfilter.trace_all = true
, EQMX sends notraceparent
to the subscriber.What did you expect to happen?
EQMX should send an updated
traceparent
to the subscribers, where the span id is updated for each subscriber of the message. Based on the documentation, it seems likesend_published_message
is what it should be. This should work even if the publisher sends notraceparent
(given that trace all messages is enabled).How can we reproduce it (as minimally and precisely as possible)?
No response
Anything else we need to know?
No response
EMQX version
5.5.1
OS version
EQMX on Kubernetes
Log files
The text was updated successfully, but these errors were encountered: