Non-HTTP use cases?

[nelz9999]

Hi,

How might I achieve non-HTTP tracing?

My example: I have a long-running service that stands up, polls SQS for work to be done, processes the work items in parallel (interacting with other AWS services), and removes the queue items/reservations. It would be valuable to see how these SQS messages interact with the rest of our system.

[awssandra]

Hi Nelz,

Do you want to trace through the SQS message from the sender, through the queue to receiver and then all the work following? Or, do you just want to start tracing on the receiving end?

[nelz9999]

Hrm... I think I'm having trouble understanding what you're getting at.

I am already tracing the service that puts messages into SQS. I would also like to trace the ins & outs of the "daemon" service that pulls those messages off the SQS Queue, and the downstream services that process interacts with.

So, yes, I would like the full end-to-end view, but that necessarily means instrumenting the "receiving end", doesn't it?

[luluzhao]

Hello @nelz9999,

Are you trying to tell us that your "daemon" service is a non-HTTP service and you want to trace it? Is that correct? Please point out if my understanding is wrong.

Currently, our X-Ray Go SDK doesn't support an "sqs handler" to automatically trace those information for you. However, you could implement it by yourself. Based on the information you provided, the service you already traced should have a "sqs sending handler" to inject a trace header(contains trace_id, id and sampling decision) into the put messages request by adding into message attribute. In the meantime, your "daemon" service should have a "sqs receiving handler" to process the trace header namely adding correct information to the segment, when it pulls messages from SQS. In this way, you are able to have a end-to-end view.

[nelz9999]

Yes, I do believe the "sqs receiving handler" is the crux of my question.

Looking deeper into the code, I see that the xray.Handler(...) starts the segments, first by pulling information from the HTTP Headers for x-amzn-trace-id.

So, what you're saying is that I just need to figure out where the x-amzn-trace-id parallel is in the SQS Message, and use that to start (and eventually end) a new segment. Is that correct?

[luluzhao]

@nelz9999, yes, you are right and that is why I suggest you to inject those information into sqs message attribute so that you can get them and start a new segment.

[nelz9999]

Hrm... Okay. What I'm hearing is that there's nothing actively stopping me from tracing SQS Messages, as long as I do a bunch of the heavy lifting.

This gets back to the main question: might there be a chance that the SDK(s) get some support for doing this in a standard way, rather than via bespoke processes?

[luluzhao]

@nelz9999, we will support this feature in future which will automatically trace information for customers. However, I cannot guarantee you a timeline at this point. Please stay tuned.

[nelz9999]

Cool, that's good to know.

Before I close this, I'll give you an insight into my thought process.

If I can auto-magically get tracing to work with the tiny bits of config and construction-time injection of your SDK: presto-blamo, I'm a happy camper.

But, as soon as I have to start writing custom code to support XRay, I start to think "Jeepers, it sure would be cool if y'all supported OpenTracing." (#31) I know OpenTracing is not a panacea, but it gives at least a little hope that the code I write to support tracing might be able to be vendor-agnostic.

Without either a) super-auto-magical SDK that takes care of all my possible use-cases; or b) a standard framework that lets me swap out vendors; then I start to feel trapped by the AWS hegemony, and I start to ask myself if I actually even need distributed tracing yet.

[awssandra]

Hi nelz9999,

Thank you for the feedback. We continue to reach out to the open source community to voice their issues and comments to shape our vision of X-Ray.

We are continuing to expand the X-Ray experience internally to cover these use cases, as we are fully aware of the limited nature of the SDK and X-Ray integration with other AWS resources and are striving to remedy it. SQS/SNS support has been a highly requested feature, and we hope to have an easy customer experience for this in the near future.

The issue of standardized tracing data format and APIs is a topic we, of course, are highly interested in. The tracing space is still early into it's life, and evolving quickly. We're excited to see how this influences X-Ray over time.

As always, please keep an eye out on the official forums for future announcements as we venture these uncharted waters together with the open source community.

Thank you, and have a good weekend!
Sandra