Getting errors after trying to deploy Node.js lambdas with arm architecture

[vvo]

Expected Behavior

Selecting the arm architecture for a lambda should not fail the runtime.

Actual Behavior

When trying to deploy to arm architecture, I got errors about the Datadog agent no able to start well.

Steps to Reproduce the Problem

1. Add architecture: cdk.aws_lambda.Architecture.ARM_64, to an cdk.aws_lambda_nodejs.NodejsFunction
2. cdk deploy
3. runtime fails with:

RequestId: b6d7748b-1ec0-42dc-8cf6-513e295003d7 Error: fork/exec /opt/extensions/datadog-agent: exec format error
Extension.LaunchError
EXTENSION	Name: datadog-agent	State: LaunchError	Events: []	Error Type: UnknownError

Deployed arns:

• arn:aws:lambda:us-east-2:464622532012:layer:Datadog-Node14-x:76
• arn:aws:lambda:us-east-2:464622532012:layer:Datadog-Extension-ARM:21

npm list (ran on my machine):

> npm list
├─┬ [redacted-cdk-project-name]@0.0.0 -> ./services/[redacted-cdk-project-name]
│ ├── @aws-sdk/client-cloudwatch@3.100.0
│ ├── @aws-sdk/client-kinesis@3.100.0
│ ├── @aws-sdk/client-lambda@3.100.0
│ ├── @aws-sdk/client-s3@3.100.0
│ ├── @swc/core@1.2.182
│ ├── @swc/helpers@0.3.13
│ ├── @swc/jest@0.2.21
│ ├── @types/aws-lambda@8.10.97
│ ├── aws-cdk-lib@2.21.1
│ ├── aws-cdk@2.21.1
│ ├── constructs@10.0.126
│ ├── datadog-cdk-constructs-v2@0.2.0
│ ├── datadog-lambda-js@5.76.0
│ ├── dd-trace@2.5.0
│ ├── esbuild@0.14.36
│ ├── get-stream@6.0.1
│ ├── jest@28.1.0
│ ├── reflect-metadata@0.1.13
│ └── source-map-support@0.5.21
└── yaml@1.10.2 extraneous

Guess: Because I list dd-trace in my own package.json, because cdk deploy happened on an x86_64 (GitHub action), then the agent that was loaded was not arm compatible?

Thanks!

[astuyve]

Hi @vvo! Thanks for reaching out. I'm going to chat with the team and try to reproduce, and then I'll get back to you.

[astuyve]

Hi @vvo can you verify in the lambda console that the architecture is correctly set to ARM/Graviton? The layer you've provided for the extension is indeed an ARM-specific build (arn:aws:lambda:us-east-2:464622532012:layer:Datadog-Extension-ARM:21), but I'm wondering if somehow the lambda function got stuck on x86?

[vvo]

@astuyve We had to revert so I can't check anymore. But here is an excerpt of the cdk synth that was done in GH action right before doing the deploy:

  [redacted]:
    Type: AWS::Lambda::Function
    Properties:
      Architectures:
        - arm64
      Handler: /opt/nodejs/node_modules/datadog-lambda-js/handler.handler
      Layers:
        - arn:aws:lambda:us-east-2:464622532012:layer:Datadog-Node14-x:76
        - arn:aws:lambda:us-east-2:464622532012:layer:Datadog-Extension-ARM:21
      MemorySize: 1024
      Runtime: nodejs14.x

So yes it was deployed with arm64. But again, from GH action (x86), and a project listing datadog js dependencies in package.json.

[vvo]

Another bit of info, my cdk esbuild lambda settings includes:

    externalModules: ['graphql/*', 'datadog-lambda-js', 'dd-trace'],

When you use the CDK constructs it is said the Datadog js agent will be installed/launched automatically. But if you need to do import { sendDistributionMetric } from 'datadog-lambda-js'; in your own code, then your only way is to add the package in package.json and then ignore it in esbuild. Otherwise you can't even create unit tests.

But I sense this is causing more harm than good maybe? Thanks!

[astuyve]

Hi @vvo - The agent and datadog-lambda-js are separate (and using it with webpack has some workaround).

The Agent is deployed via Lambda Extensions automatically when the layer is added, it's completely separate from datadog-lambda-js as well as dd-trace. The Agent is what seems to be crashing here, but I still suspect some kind of architecture mismatch.

Unfortunately I haven't been able to reproduce this on ARM, here's my service definition:

const cdk = require("aws-cdk-lib");
const { Construct } = require("constructs");
const apigateway = require("aws-cdk-lib/aws-apigateway");
const lambda = require("aws-cdk-lib/aws-lambda");
const s3 = require("aws-cdk-lib/aws-s3");
const { Datadog } = require('datadog-cdk-constructs-v2');

class WidgetService extends Construct {
  constructor(scope, id) {
    super(scope, id);

    const bucket = new s3.Bucket(this, "WidgetStore");

    const handler = new lambda.Function(this, "WidgetHandler", {
      runtime: lambda.Runtime.NODEJS_14_X,
      code: lambda.Code.fromAsset("resources"),
      architecture: lambda.Architecture.ARM_64,
      handler: "widgets.main",
      environment: {
        BUCKET: bucket.bucketName
      }
    });

    bucket.grantReadWrite(handler); // was: handler.role);

    const api = new apigateway.RestApi(this, "widgets-api", {
      restApiName: "Widget Service",
      description: "This service serves widgets."
    });

    const getWidgetsIntegration = new apigateway.LambdaIntegration(handler, {
      requestTemplates: { "application/json": '{ "statusCode": "200" }' }
    });

    api.root.addMethod("GET", getWidgetsIntegration); // GET /

    const datadog = new Datadog(this, "Datadog", {
      nodeLayerVersion: 77,
      addLayers: true,
      extensionLayerVersion: "21",
      apiKey: <api key>,
    })
    datadog.addLambdaFunctions([handler])
  }
}

module.exports = { WidgetService }

And to confirm, here's the lambda configuration:

And the traces sent to datadog:

I was wondering if you might be able to try again, and/or perhaps share the datadog configuration you've used in your module?

Thanks!

[vvo]

Hey there, did you deploy this from an arm machine (like a mac m1) or from and amd64 machine (like github actions)? Thanks!

[vvo]

Also, did you add the dd trace dependency directly to hour package.json like I did?

[astuyve]

I deployed from an Intel mac, but that wouldn't impact this because this library detects architecture based on the lambda handler setting, not the machine deploying it.

I can try the dd-trace dependency directly as well, but that's highly unlikely given that the log was emitted from the extension itself, which is ran from/opt/extensions/datadog-extension and is separate from the tracer.

[astuyve]

Hey @vvo - any luck reproducing this here? I realize my minimal example likely does not resemble your project. If you're still experiencing this issue, it might be best to move to a support ticket where we can privately collaborate on your specific project.

Thanks!

[vvo]

@astuyve I am reluctant to try to redeploy in production as this broke our production last time. My next try is to create a second lambda, set it to x86, same configuration as the failed one, then switch it to arm. If this fails then I have a reproduction. Unless you have any other insight then this is what I will try, not sure when though as I deprioritized this part for now.

[astuyve]

Could you test it in a staging environment, or deploy a new stack?

Did this occur when flipping a lambda function from x86 to ARM?

[vvo]

Could you test it in a staging environment, or deploy a new stack?

When doing so, it worked (separate AWS account). But I could not reproduce the x86 => arm failures. Only production suffered this.

Did this occur when flipping a lambda function from x86 to ARM?

Yes, it failed when moving from x86 to arm.