SQS Tracing with AWSTraceHeader

[rogueai]

Hi,

I have an SQS queue subscribed to an SNS topic and in turns triggers a Lambda on new SQS messages.

I've tried following the docs here: https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html to create segments related to SQS, but so far everything I tried didn't work.
I've added a segment this way:

const traceHeaderStr = record.attributes.AWSTraceHeader;
const traceData = AWSXRay.utils.processTraceData(traceHeaderStr);
const segment = new AWSXRay.Segment("SQS", traceData.Root, traceData.Parent);
delete segment.service;
segment.origin = "AWS::SQS";
segment.inferred = true;
segment.addPluginData({
   operation: "SendEvent",
   region: record.awsRegion,
   request_id: context.awsRequestId,
   queue_url: record.eventSourceARN
});

What this produces though is something like this:

As you can see, the SQS segment is created with SNS as a parent, however the Lambda invocation is disconnected.

I've also tried other approaches that didn't work:

• overriding the _X_AMZ_TRACE_ID env variable with the AWSTraceHeader, before initializing the XRay SDK
• setting the tracing to "PassThrough" and creating segments manually

Could you advise a possible workaround, to get this working?

Thank you!

[willarmiros]

Hi @rogueai,
This is a known issue with propagating trace context through an SQS queue. It is not currently possible to connect a trace passed into an SQS queue to the Lambda consuming it on the other end. This is because although the trace header is propagated in the SQS message, you cannot set it in the Lambda because segments in Lambda are immutable.

Enabling this functionality is a feature that is in progress and one we hope to release for all our SDKs. Stay tuned and I will update this issue as we make progress in the Node SDK.

[dsharkou]

Hi @willarmiros,
Do you have any approximate date for release this feature?
We are also need this functionality (in my case sqs trigger lambda which send message to ses).

[willarmiros]

Unfortunately we cannot provide dates for upcoming features. However this feedback is appreciated so we and the other teams involved in the change can prioritize the feature. Stay tuned for updates!

[AlexRomashevski]

Hi @willarmiros,

Also, vote to move this feature on the top for your team backlog. Our team also waiting for this improvement to have a clear map on X-Ray. I hope it will not take to much to move it on live.

Thank you!

[TLaue]

+1

[dimmy-timmy]

+1. I have the same issue on my project

[asaghri]

+1

[misterjoshua]

+1. I am also struggling with this limitation.

[RyanV]

+1

[billybonks]

would love to have this aswell, it would signficantly improve the value of xray.

[danilofuchs]

We also really want to see a more connected map. We use this Lambda -> SNS -> SQS -> Lambda across 3 serverless applications, and X-Ray would enable us to monitor the entire flow

[JohanZackrisson]

+1

[joshgoodwin]

+1

[lgalumbres]

+1 Would love to see this feature supported as well as this would provide significant insight on the traces for our message processing workflow.

[alexmnyc]

this is needed greatly! otherwise, you cannot use xray with lambdas effeciently

[awssandra]

Thanks everyone, for the +1! The X-Ray Service Team is hard at work on this. We hope to post an update on this soon - thank you for you continued patience.

Stay healthy and safe!
Sandra

[codergeorgej]

Hi,

I have an SQS queue subscribed to an SNS topic and in turns triggers a Lambda on new SQS messages.

I've tried following the docs here: https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html to create segments related to SQS, but so far everything I tried didn't work.
I've added a segment this way:

const traceHeaderStr = record.attributes.AWSTraceHeader;
const traceData = AWSXRay.utils.processTraceData(traceHeaderStr);
const segment = new AWSXRay.Segment("SQS", traceData.Root, traceData.Parent);
delete segment.service;
segment.origin = "AWS::SQS";
segment.inferred = true;
segment.addPluginData({
   operation: "SendEvent",
   region: record.awsRegion,
   request_id: context.awsRequestId,
   queue_url: record.eventSourceARN
});

What this produces though is something like this:

As you can see, the SQS segment is created with SNS as a parent, however the Lambda invocation is disconnected.

I've also tried other approaches that didn't work:

• overriding the _X_AMZ_TRACE_ID env variable with the AWSTraceHeader, before initializing the XRay SDK
• setting the tracing to "PassThrough" and creating segments manually

Could you advise a possible workaround, to get this working?

Thank you!

@rogueai How did you manage to get SNS -> SQS to come up in the servicemap ? My request tracing gets stopped at SNS. (SQS doesnt showup on the service map)

[pdeak]

It feels to me like the "enhancement" label on this issue is debatable. From what I can see, this means that it is impossible to use X-Ray for its primary intended purpose (transaction visualisation across services) when using SQS and Lambdas?

It doesn't look like there's any kind of workaround or hack to get around this in the meantime, so it essentially makes X-Ray unusable if your architecture just happens to include Lambdas and SQS? I would have thought this would constitute a somewhat more serious defect...

[danieletieghi]

+1

[stockf]

+1

[azambrano]

+1

[willarmiros]

Hello @pdeak,
This issue is marked as an enhancement because the X-Ray team working to expand X-Ray's functionality in several use cases. Unfortunately it is not as simple as a missing link due to a bug. We intend to address this in the near future.

[dfens1]

+1

[austin-johansen]

+1

[josuelima]

+1

[5t33]

+3

[psimsa]

While I agree and share your frustration @joshgoodwin, I think that it is probably not the issue with X-Ray itself but rather SQS making it hard to implement it correctly.

I would dare to disagree. Wiring up Jaeger tracing for instance works flawlessly, propagating tracing ID through lambda -> sns -> sqs -> lambda without any issue. It takes extra ~5 lines of code per lambda, but other than that it's fairly easy. So if 3rd party can, why AWS can't?

[tomaszdudek7]

@ofcoursedude There's a difference between implementing tracing inside Lambdas with a custom code (like you mentioned with Jagger) and making it work with no code involved at all (like enabling X-Ray at API Gateway level). I think what they want to do is to have an option to pass traces through SQS without ANY additional code from our side. That forces them to implement this inside SQS which, as I mentioned, might be harder.

But I'm only playing devil's advocate here.

Anyway, an article about Jaeger tracing inside SQS/Lambdas and how did you set that up with some examples would be a great read, appreciated by many. I looked it up and there are little (if any) resources concerning that topic so far.

[psimsa]

@ofcoursedude There's a difference between implementing tracing inside Lambdas with a custom code (like you mentioned with Jagger) and making it work with no code involved at all (like enabling X-Ray at API Gateway level). I think what they want to do is to have an option to pass traces through SQS without ANY additional code from our side. That forces them to implement this inside SQS which, as I mentioned, might be harder.

I understand that - however, my point is that if I am able to pass a tracing information from one lambda to another through a chain of SNS and SQS through my code that I investigated and PoC'ed in one afternoon, I am really puzzled why there is no progress on this, especially given that the X-Ray SDK actually allows to switch tracing context on other platforms (so you can pass the tracing header in message's metadata like I did with Jaeger) but trying to do that in lambda runtime does not work. I really wonder what makes it so different on this level from "regular" runtimes (in my case .Net).
As for the Jaeger solution - yeah I'm trying to make myself write some post somewhere but didn't get to it much. But on a high level, basically proceeed as you would do in regular runtime and pass the header in message's metadata. Then, when your function gets invoked and receives the message, extract the tracing info and use it to continue the previous context, adding a new session. I PoC'd it at work passing from a .Net API service running in EC2 through sns/sqs to lambda and then in a chain of lambda -> sns/sqs -> lambda up to Dynamo.

[mccauleyp]

+1

[willarmiros]

Hi all,

I would first like to apologize that this capability is still not available. We hear you and understand your frustration. While we have been continuously working on delivering this, there have been major setbacks due to the complicated nature of the problem. As some have pointed out, we would like to provide a fully-managed solution for propagating trace context from SQS -> Lambda, however that has proven difficult because it is unclear how to map a batch of (independently traced) SQS messages to a single Lambda invocation (represented by 1 segment). We are still actively working on such a solution, and appreciate the constructive feedback we've received so far. In parallel, we're working with the Lambda team on possible workarounds to this problem. Once either is officially supported, we will update this issue.

[imiosga]

+1

[joshuamfrancis]

+1 this capability is much sought after.

[psimsa]

Hi all,

I would first like to apologize that this capability is still not available. We hear you and understand your frustration. While we have been continuously working on delivering this, there have been major setbacks due to the complicated nature of the problem. As some have pointed out, we would like to provide a fully-managed solution for propagating trace context from SQS -> Lambda, however that has proven difficult because it is unclear how to map a batch of (independently traced) SQS messages to a single Lambda invocation (represented by 1 segment). We are still actively working on such a solution, and appreciate the constructive feedback we've received so far. In parallel, we're working with the Lambda team on possible workarounds to this problem. Once either is officially supported, we will update this issue.

Hi @willarmiros, do you have any news for us?

[s1mrankaur]

+1 . Been a while. Is there an update here?

[joaogsleite]

Any update on this? After 3 years we still can't trace a flow between SQS and a Lambda function?

[yvele]

Here is a workaround inspired from

• https://dev.to/aws-builders/x-ray-tracing-from-sqs-to-lambda-8md
• https://github.com/kyhau/aws-tools/blob/main/X-Ray/xray-sqs-to-lambda/handler.ts

Everything is done on the Lambda consumer side:

import { Segment, setSegment, utils } from "aws-xray-sdk";

function createLambdaSegment(context, sqsRecord) {
  const lambdaExecStartTime = new Date().getTime() / 1000;

  const traceHeaderStr = sqsRecord.attributes.AWSTraceHeader;
  const traceData = utils.processTraceData(traceHeaderStr);
  const sqsSegmentEndTime = Number(sqsRecord.attributes.ApproximateFirstReceiveTimestamp) / 1000;

  const segment = new Segment(
    context.functionName,
    traceData.root,
    traceData.parent
  );
  segment.origin = "AWS::Lambda::Function";
  segment.start_time = lambdaExecStartTime - (lambdaExecStartTime - sqsSegmentEndTime);
  segment.addPluginData({
    function_arn : context.invokedFunctionArn,
    region : sqsRecord.awsRegion,
    request_id : context.awsRequestId
  });

  return segment;
}

/**
 * Set the current Lambda segment that instruments SQS.
 *
 * This is a workaround for https://github.com/aws/aws-xray-sdk-node/issues/208.
 *
 * Implementation inspired from:
 * - https://dev.to/aws-builders/x-ray-tracing-from-sqs-to-lambda-8md
 * - https://github.com/kyhau/aws-tools/blob/main/X-Ray/xray-sqs-to-lambda/handler.ts
 *
 * @see https://github.com/aws/aws-xray-sdk-node/issues/208#issuecomment-1285169865
 * @param {object} context Lambda [context](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html)
 * @param {object} sqsRecord SQS [record item](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html)
 * @returns {object} X-Ray segment for Lambda
 */
export default function setLambdaSegmentFromSQS(context, sqsRecord) {
  const segment = createLambdaSegment(context, sqsRecord);
  setSegment(segment);
  return segment;
}

And you use it like that in your handler file:

import setLambdaSegmentFromSQS from "./setLambdaSegmentFromSQS";

export async function handle(event, context) {
  const segment = setLambdaSegmentFromSQS(context, event.Records[0]);
  try {
    await handleCore(event);
  } catch (err) {
    segment.close(err);
    throw err;
  }
  segment.close();
}

And with a proper X-Rray trace group it gets cleaner:

XRayTraceGroup:
  Type: AWS::XRay::Group
  Properties:
    GroupName: my-group
    InsightsConfiguration:
      InsightsEnabled: true
      NotificationsEnabled: false
    FilterExpression: !Sub |
      resource.arn = "${ProducerFunction.Arn}"
      OR resource.arn = "${Queue.Arn}"
      OR resource.arn = "${CollectorFunction.Arn}"      

[s1mrankaur]

@yvele Thanks for sharing. Only if there was a timeline to this feature request, we'd have known if it's worth jumping on the workarounds already. This looks great though, Thanks much!

[willarmiros]

Hi everyone - unfortunately we are not able to share any timelines about new features over GitHub. However this issue will be updated the moment we have a publicly available update.

[owenmorgan]

https://aws.amazon.com/about-aws/whats-new/2022/11/aws-x-ray-trace-linking-event-driven-applications-amazon-sqs-lambda/

Assuming this is a resolution?

[mgorski-mg]

https://aws.amazon.com/about-aws/whats-new/2022/11/aws-x-ray-trace-linking-event-driven-applications-amazon-sqs-lambda/

Assuming this is a resolution?

Yes, it is ❤

[willarmiros]

As a couple astute commenters have noticed, today we are thrilled to announce support for linking traces from SQS Queues that are polled by AWS Lambda functions! This functionality is available out of the box for all Lambda functions with Active Tracing enabled triggered by SQS Queues. I'm only the messenger here, so I'd like to thank all of the other engineers who put in so much hard work to deliver this feature!

Learn more here: https://aws.amazon.com/about-aws/whats-new/2022/11/aws-x-ray-trace-linking-event-driven-applications-amazon-sqs-lambda/

Please use the built-in feedback link on the AWS Console, reach out to your AWS Support contact, or of course open issues on the SDK repos to leave feedback for X-Ray, there is more to come! With that, I'll be closing this issue out.

[s1mrankaur]

Great! So glad to see an update here. However, the complete flow mentioned in the topic is still not supported (Lambda -> SNS -> SQS -> Lambda) I believe? I can't see SNS -> SQS flow in my traces @willarmiros

[willarmiros]

Good callout @s1mrankaur - we will continue to track the SNS -> SQS case in this issue: aws/aws-xray-sdk-go#218

We will also update here when that workflow is fully supported :)

[willarmiros]

Hi folks, wanted to cross-post here that today we are happy to announce Amazon SNS support for active tracing with X-Ray! If you have an SNS->SQS->Lambda workflow for example, you will now be able to see that workflow end-to-end for the first time by opting in to active tracing on your SNS topic.

Read more here: https://aws.amazon.com/about-aws/whats-new/2023/02/amazon-sns-x-ray-active-tracing-visualize-analyze-debug-application-performance/

[s1mrankaur]

@willarmiros This is amazing! I am guessing there isn't support for this in a serverless framework yet i.e opting for active tracing on SNS topics?

provider:
  tracing:
    apiGateway: true
    lambda: true

This is what I have sofar. Is there any additional config that I can add/enable to see this in action? @willarmiros

[willarmiros]

The active tracing config is supported via CloudFormation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-sns-topic.html#cfn-sns-topic-tracingconfig

I am not sure how that makes its way over to the serverless framework tbh, but if there is a place to put in that request with them I'd love to do so!

[s1mrankaur]

Thanks @willarmiros . I've now posted the feature request here serverless/serverless#12385

[dschro-1993]

What about DynamoDB Streams and lambdas? Is it anywhere on the roadmap?