lambda: transaction.context.message.headers is broken if captureHeaders=true and an SNS or SQS event has message attributes

[trentm]

the bug

If a Lambda function configured to use the Node.js APM agent and with captureHeaders: true (the default) receives a SNS or SQS event with message attributes, it will result in a transaction object with bogus context.message.headers that breaks the APM server transaction schema and breaks on intake with an error like this:

APM server response body: {"accepted":0,"errors":[{"message":"decode error: data read error: v2.transactionRoot.Transaction: v2.transaction.SpanCount: v2.transactionSpanCount.Context: v2.context.Message: v2.contextMessage.Headers: invalid input for HTTPHeader: map[Type:String Value:this is my foo value]","document":"{\"transaction\":{\"name\":\"RECEIVE trentm-play-topic1\",\"type\":\"messaging\",\"result\":\"success\",\"id\":\"9176fe68d4e3b953\",\"trace_id\":\"208075b3b3fd2ac6b549711d94e877f2\",\"duration\":60.559,\"timestamp\":1646781029978068,\"sampled\":true,\"context\":{\"user\":{},\"tags\":{},\"custom\":{},\"service\":{\"origin\":{\"name\":\"trentm-play-topic1\",\"id\":\"arn:aws:sns:us-west-2:627286350134:trentm-play-topic1\",\"version\":\"1.0\"}},\"cloud\":{\"origin\":{\"provider\":\"aws\",\"region\":\"us-west-2\",\"service\":{\"name\":\"sns\"},\"account\":{\"id\":\"627286350134\"}}},\"message\":{\"queue\":{\"name\":\"trentm-play-topic1\"},\"age\":{\"ms\":1247},\"headers\":{\"foo\":{\"Type\":\"String\",\"Value\":\"this is my foo value\"}}}},\"span_count\":{\"started\":0},\"outcome\":\"success\",\"faas\":{\"id\":\"arn:aws:lambda:us-west-2:627286350134:function:trentm-play-fn1\",\"coldstart\":true,\"execution\":\"cc8565c8-a165-4b02-b606-8d477659a40a\",\"trigger\":{\"type\":\"pubsub\",\"request_id\":\"8d426c7c-fe30-5f4b-9a12-36dd1a940ae0\"}},\"sample_rate\":1}}"}]}

The error message from that is v2.contextMessage.Headers: invalid input for HTTPHeader: map[Type:String Value:this is my foo value] when it was given a transaction document with:

{
  "transaction": {
    "name": "RECEIVE trentm-play-topic1",
    "type": "messaging",
...
    "context": {
...
      "message": {
        "queue": {
          "name": "trentm-play-topic1"
        },
        "age": {
          "ms": 1247
        },
        "headers": {
          "foo": {
            "Type": "String",
            "Value": "this is my foo value"
          }
        }
      }
    },
...
  }
}

SNS repro

1. Create an SNS topic, e.g.:

% aws sns create-topic --name trentm-play-topic1
{
    "TopicArn": "arn:aws:sns:us-west-2:627286350134:trentm-play-topic1"
}

2. Add that topic as a trigger for your Node.js Lambda function. I used the AWS console for this -- because it helpfully sets up the required IAM role.

3. Publish a message to that topic:

aws sns publish \
    --topic-arn arn:aws:sns:us-west-2:627286350134:trentm-play-topic1 \
    --message 'this is my message' --subject 'this is my subject' \
    --message-attributes '{"foo": {
                "DataType": "String",
                "StringValue": "this is my foo value"
    }}'

4. Look at the Lambda function's log. You should see the above error.

details

The lambda instrumentation spec says

Field	Value	Description	Source
context.message.headers	-	The message attributes. Should only be captured, if capturing headers is enabled in the configuration.	record.messageAttributes
...
context.message.headers	-	The message attributes. Should only be captured, if capturing headers is enabled in the configuration.	record.sns.messageAttributes

However, this is misleading. The expected "context.message.headers" format is a mapping of string keys to values that are null, a string, or an array of strings. JSON schema:

            "headers": {
              "description": "Headers received with the message, similar to HTTP request headers.",
              "type": [
                "null",
                "object"
              ],
              "additionalProperties": false,
              "patternProperties": {
                "[.*]*$": {
                  "type": [
                    "null",
                    "array",
                    "string"
                  ],
                  "items": {
                    "type": "string"
                  }
                }
              }
            },

But the format of SNS message attributes is, e.g.:

2022-03-09T00:27:39.523000+00:00 2022/03/09/[$LATEST]d0aa2186620a47cbbb1f80f8ae0ce12b 2022-03-09T00:27:39.523Z	a57a242e-525a-48ea-8f22-eaa48ed16621	INFO	event: {
  "Records": [
    {
      "EventSource": "aws:sns",
      "EventVersion": "1.0",
...
      "Sns": {
...
        "TopicArn": "arn:aws:sns:us-west-2:627286350134:trentm-play-topic1",
        "Subject": "this is my subject",
        "Message": "this is my message",
...
        "MessageAttributes": {
          "Greeting": {
            "Type": "Binary",
            "Value": "SGVsbG8sIFdvcmxkIQ=="
          },
          "Population": {
            "Type": "String",
            "Value": "1250800"
          },
          "City": {
            "Type": "String",
            "Value": "Any City"
          }
        }
      }
    }
  ]
}

and of SQS message attributes:

2022-03-09T00:25:28.842000+00:00 2022/03/09/[$LATEST]d0aa2186620a47cbbb1f80f8ae0ce12b 2022-03-09T00:25:28.842Z	b8b95804-a13e-5a68-9f36-02a557138a98	INFO	event: {
  "Records": [
    {
      "messageId": "94f54f6b-0b5d-4071-b752-103f481796d9",
...
      "messageAttributes": {
        "Greeting": {
          "binaryValue": "SGVsbG8sIFdvcmxkIQ==",
          "stringListValues": [],
          "binaryListValues": [],
          "dataType": "Binary"
        },
        "Population": {
          "stringValue": "1250800",
          "stringListValues": [],
          "binaryListValues": [],
          "dataType": "Number"
        },
        "City": {
          "stringValue": "Any City",
          "stringListValues": [],
          "binaryListValues": [],
          "dataType": "String"
        }
      },
...
      "eventSourceARN": "arn:aws:sqs:us-west-2:627286350134:trentm-play-queue1",
    }
  ]
}

[trentm]

SQS repro

1. Create an SQS queue:

% aws sqs create-queue --queue-name trentm-play-queue1
{
    "QueueUrl": "https://sqs.us-west-2.amazonaws.com/627286350134/trentm-play-queue1"
}

2. Add an SQS trigger to your Node.js Lambda function from this queue. I needed to add the managed "AWSLambdaSQSQueueExecutionRole" IAM policy to the role assigned to my function: AWS Lambda console > Configuration > Permissions > "Edit" button on the "Execution role" box.

3. Send a message with message attributes to the queue:

% cat send-message.json
{
  "City": {
    "DataType": "String",
    "StringValue": "Any City"
  },
  "Greeting": {
    "DataType": "Binary",
    "BinaryValue": "Hello, World!"
  },
  "Population": {
    "DataType": "Number",
    "StringValue": "1250800"
  }
}

% aws sqs send-message \
    --queue-url "https://sqs.us-west-2.amazonaws.com/627286350134/trentm-play-queue1" \
    --message-body 'this is my body' \
    --message-attributes file://send-message.json
{
    "MD5OfMessageBody": "567762fc32b60cd7fc4abbe9cf1fcfbe",
    "MD5OfMessageAttributes": "28eb0e573cf8e8a77e349a2f968eac4a",
    "MessageId": "94f54f6b-0b5d-4071-b752-103f481796d9"
}

The result is an event to the lambda with:

  "Records": [
    {
...
      "messageAttributes": {
        "Greeting": {
          "binaryValue": "SGVsbG8sIFdvcmxkIQ==",
          "stringListValues": [],
          "binaryListValues": [],
          "dataType": "Binary"
        },
        "Population": {
          "stringValue": "1250800",
          "stringListValues": [],
          "binaryListValues": [],
          "dataType": "Number"
        },
        "City": {
          "stringValue": "Any City",
          "stringListValues": [],
          "binaryListValues": [],
          "dataType": "String"
        }
      },
...