DD plugin logs: `ERROR datadog:failed to extract http tags from api-gateway event` on each AWS Lambda call

[thierryx96]

Expected Behavior

No error being reported by the datadog serverless plugin.

Actual Behavior

The AWS Lambda function using the datadog serverless plugin suddenly started to log the following error in DD logs / cloudwatch:

ERROR [dd.trace_id=123 dd.span_id=123] {"status":"error","message":"datadog:failed to extract http tags from api-gateway event"}

• It happens when the Lambda function is invoking the DD function from the datadog-ForwarderStack
• I have been using the serverless plugin for the past 3 months without any issues.
• The underlying AWS infrastructure has not been updated since 1 month.
• Copy of the DatadogAWSIntegrationPolicy Iam role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "apigateway:GET",
                "autoscaling:Describe*",
                "budgets:ViewBudget",
                "cloudfront:GetDistributionConfig",
                "cloudfront:ListDistributions",
                "cloudtrail:DescribeTrails",
                "cloudtrail:GetTrailStatus",
                "cloudwatch:Describe*",
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "codedeploy:List*",
                "codedeploy:BatchGet*",
                "directconnect:Describe*",
                "dynamodb:List*",
                "dynamodb:Describe*",
                "ec2:Describe*",
                "ecs:Describe*",
                "ecs:List*",
                "elasticache:Describe*",
                "elasticache:List*",
                "elasticfilesystem:DescribeAccessPoints",
                "elasticfilesystem:DescribeFileSystems",
                "elasticfilesystem:DescribeTags",
                "elasticloadbalancing:Describe*",
                "elasticmapreduce:List*",
                "elasticmapreduce:Describe*",
                "es:ListTags",
                "es:ListDomainNames",
                "es:DescribeElasticsearchDomains",
                "health:DescribeEvents",
                "health:DescribeEventDetails",
                "health:DescribeAffectedEntities",
                "kinesis:List*",
                "kinesis:Describe*",
                "lambda:AddPermission",
                "lambda:GetPolicy",
                "lambda:List*",
                "lambda:RemovePermission",
                "logs:TestMetricFilter",
                "logs:PutSubscriptionFilter",
                "logs:DeleteSubscriptionFilter",
                "logs:DescribeSubscriptionFilters",
                "rds:Describe*",
                "rds:List*",
                "redshift:DescribeClusters",
                "redshift:DescribeLoggingStatus",
                "route53:List*",
                "s3:GetBucketLogging",
                "s3:GetBucketLocation",
                "s3:GetBucketNotification",
                "s3:GetBucketTagging",
                "s3:ListAllMyBuckets",
                "s3:PutBucketNotification",
                "ses:Get*",
                "sns:List*",
                "sns:Publish",
                "sqs:ListQueues",
                "states:ListStateMachines",
                "states:DescribeStateMachine",
                "support:*",
                "tag:GetResources",
                "tag:GetTagKeys",
                "tag:GetTagValues",
                "xray:BatchGetTraces",
                "xray:GetTraceSummaries"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Steps to Reproduce the Problem

1. setup a lambda function with HTTP event
2. add the serverless datadog plugin
3. "status":"error","message":"datadog:failed to extract http tags from api-gateway event may appear in DD logs / cloudwatch

Specifications

• Serverless Framework version: 2.3.0
• Datadog Serverless Plugin version: 2.16.0
• Lambda function runtime (Python 3.7, Node 10, etc.): Python 3.7

Stacktrace

"status":"error","message":"datadog:failed to extract http tags from api-gateway event

[DylanLovesCoffee]

Hey @thierryx96, that error seems to be coming from our NodeJS Lambda layer. Are you able to share the API Gateway event that triggers the function reporting the error? Could you also share your serverless.yml used for that function? Feel free to DM me on our Slack workspace as well if that's your preferred method of comms.