SNS->SQS->JMS causes error "JMSException: Binary is not a supported JMS property type"

[falken]

Expected behaviour

1. A message is sent from a nodejs application to an SNS topic with _datadog context propagation header set.
2. A messages is placed on an SQS queue based on a subscription to the SNS topic
3. A JMS consumer consumes and processes the message

Actual behaviour
1 & 2 function properly
Step 3 fails with an error "JMSException: Binary is not a supported JMS property type" thrown by this code in the aws library
https://github.com/awslabs/amazon-sqs-java-messaging-lib/blob/4cb91355cb92d9361a2179233c9db89383b1299e/src/main/java/com/amazon/sqs/javamessaging/message/SQSMessage.java#L1057

The issue appears to be that when a message is sent to an SNS topic the DataType is set to "Binary"

https://github.com/DataDog/dd-trace-js/blob/da4b19df5e5b7f85f56873372869d677d60d389e/packages/datadog-plugin-aws-sdk/src/services/sns.js#L76C26-L76C26

Which causes the error in the previous code link.

In the SQS version of the code it uses a DataType of "String":

dd-trace-js/packages/datadog-plugin-aws-sdk/src/services/sqs.js

Line 164 in da4b19d

DataType: 'String',

It looks like it's using Binary because of some sort of Message Attribute filtering issue
DataDog/serverless-plugin-datadog#232 (comment)

Steps to reproduce

Environment
Linux AWS ECS Fargate

• Operation system:
  Linux
• Node.js version:
  18
• Tracer version:
  4.12
• Agent version:
  https://github.com/DataDog/datadog-agent
• Relevant library versions:

[falken]

This change appears to have changed it to a Binary value
2a3bb58

[astuyve]

Hi @falken, thanks for reaching out!

We need to use a binary type attribute because even a well-formatted JSON string breaks message attribute filtering in AWS SNS, and AWS has no plans to change this (please feel free to open a support ticket in AWS and mention me by name, maybe that will spur their motivation?). The reason is that SNS message filtering is not supported for Binary typed message attributes, so these are passed with no issues.

Unfortunately the library you've linked to has had an open bug ticket for this since 2018.

I'm not familiar with this library, is there a way to ask it not to deserialize this payload? Alternatively you can try disabling raw message delivery, which should strip out the sns metadata, or disable the aws sdk upstream with DD_TRACE_DISABLED_INSTRUMENTATIONS=aws-sdk. This will prevent trace injection into these messages at all.

If you still want trace context passed through SNS and SQS and to use the amazon-sqs-java-messaging-lib, you may need to inject and extract trace context yourself.

We're exploring other methods of trace context injection, but don't have immediate plans at the moment.

Thanks!