Nodejs SDK v3 immediately maxes memory

[RickardTaran]

Description:

This is likely just me not structuring my app correctly, so I apologize in advance. But there doesn't seem to be any documentation for what I'm trying to do so here I am. I'm trying to create a simple lambda backend to pull from dynamo in nodejs. Using the v2 api works fine but it's annoying shoving aside the documentation that constantly insists you use v3. Plus the advantages of ES6 imports are not something I want to shy away from if I can use them. Right now I'm just trying to import the client, not even using it yet. Whenever I try to import the dynamodb client in v3, it immediately maxes memory and times out. If I remove the import line, this function completes fine.

What I'm trying to do is a pretty simple case. Is there a better way to do this that won't cause this issue?

Steps to reproduce:

Create a SAM app with one node 16.x function
make the handler file .mjs (for modules)
npm install @aws-sdk/client-dynamodb
have the function return a simple json response
curl to make sure it works as intended
import the DynamoDBClient in the function handler file

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";

curl again

Observed result:

START RequestId: 546ea22d-220c-4fa2-b251-cacf5180666d Version: $LATEST
Function 'productsFunction' timed out after 10 secondsEND RequestId: 546ea22d-220c-4fa2-b251-cacf5180666d
REPORT RequestId: 546ea22d-220c-4fa2-b251-cacf5180666d Init Duration: 0.24 ms Duration: 10000.00 ms Billed Duration: 10000 ms Memory Size: 128 MB Max Memory Used: 128 MB
Invalid lambda response received: Lambda response must be valid json

Expected result:

requests completes sucesfully

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

If you need anything else for this, I'm happy to give it. I'm fully expecting this to be someone telling me to move some files around and it magically works.

1. OS: windows 11
2. sam --version: 1.60
3. AWS region: irrelevant - local

[mndeveci]

Thanks for reporting this issue.

Can you clarify where do you deploy this function? Are you deploying to lambda or are you trying with sam local commands?

[RickardTaran]

Good question. This is being run local with

sam local start-api

you can see the exact command in the package.json for npm start

[mndeveci]

I am trying to follow your steps, but after I've changed handler file to .mjs and added DDB client as a dependency, I can't invoke the function anymore. I guess this is related to changing the handler file to .mjs?

START RequestId: 957d4945-22cc-45dc-91ab-7cdbe499d791 Version: $LATEST
2022-10-27T19:59:26.205Z	undefined	ERROR	Uncaught Exception 	{"errorType":"ReferenceError","errorMessage":"exports is not defined in ES module scope","stack":["ReferenceError: exports is not defined in ES module scope","    at file:///var/task/app.mjs:17:1","    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)","    at async Promise.all (index 0)","    at async ESMLoader.import (node:internal/modules/esm/loader:526:24)","    at async _tryAwaitImport (file:///var/runtime/index.mjs:921:16)","    at async _tryRequire (file:///var/runtime/index.mjs:970:86)","    at async _loadUserApp (file:///var/runtime/index.mjs:994:16)","    at async Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1035:21)","    at async start (file:///var/runtime/index.mjs:1200:23)","    at async file:///var/runtime/index.mjs:1206:1"]}
27 Oct 2022 19:59:26,226 [ERROR] (rapid) Init failed error=Runtime exited with error: exit status 129 InvokeID=
2022-10-27T19:59:26.359Z	undefined	ERROR	Uncaught Exception 	{"errorType":"ReferenceError","errorMessage":"exports is not defined in ES module scope","stack":["ReferenceError: exports is not defined in ES module scope","    at file:///var/task/app.mjs:17:1","    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)","    at async Promise.all (index 0)","    at async ESMLoader.import (node:internal/modules/esm/loader:526:24)","    at async _tryAwaitImport (file:///var/runtime/index.mjs:921:16)","    at async _tryRequire (file:///var/runtime/index.mjs:970:86)","    at async _loadUserApp (file:///var/runtime/index.mjs:994:16)","    at async Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1035:21)","    at async start (file:///var/runtime/index.mjs:1200:23)","    at async file:///var/runtime/index.mjs:1206:1"]}
END RequestId: 1c5018e0-4466-4368-b844-f863e5560e46
REPORT RequestId: 1c5018e0-4466-4368-b844-f863e5560e46	Init Duration: 0.20 ms	Duration: 331.42 ms	Billed Duration: 332 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
{"errorType":"ReferenceError","errorMessage":"exports is not defined in ES module scope","trace":["ReferenceError: exports is not defined in ES module scope","    at file:///var/task/app.mjs:17:1","    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)","    at async Promise.all (index 0)","    at async ESMLoader.import (node:internal/modules/esm/loader:526:24)","    at async _tryAwaitImport (file:///var/runtime/index.mjs:921:16)","    at async _tryRequire (file:///var/runtime/index.mjs:970:86)","    at async _loadUserApp (file:///var/runtime/index.mjs:994:16)","    at async Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1035:21)","    at async start (file:///var/runtime/index.mjs:1200:23)","    at async file:///var/runtime/index.mjs:1206:1"]}%

[RickardTaran]

Sorry, should have specified a little more here. Here's AWS documentation behind .mjs, .js and .cjs

https://aws.amazon.com/blogs/compute/using-node-js-es-modules-and-top-level-await-in-aws-lambda/#:~:text=File%20names%20ending%20in%20.mjs%20are%20always%20treated%20as%20ES%20modules.

What my ultimate goal here is to be able to use the v3 sdk for dynamo db. It seems that the v2 sdk is built in to the lambda environment. That is what I'm currently using as a workaround and am progressing. However, the more modern module system used in the v3 api presents a lot of benefits that I'd like to leverage. I can't find a good example of a SAM app doing that (without typescript) however. That could be all I need.

Just to get past the step, I have already tried adding type: module to the package.json file. That doesn't seem to have any effect here either.

[RickardTaran]

Also @mndeveci, reading your error, it looks like you're trying to use the global exports syntax for CommonJS. That won't work with mjs. See the example in my initial screenshot for remediation.

CommonJS syntax (.js or .cjs)

exports.rootHandler = async () => {

    const response = {
        statusCode: 200,
        body: JSON.stringify({
            message: "hello"
        })
    };

    return response;
}

ES modules (.mjs)

export async function rootHandler(event) {

    const response = {
        statusCode: 200,
        body: JSON.stringify({
            message: "hello"
        })
    };

    return response;
}

[mildaniel]

Hey @RickardTaran, I tried reproducing this issue with "@aws-sdk/client-dynamodb": "^3.211.0" and was able to successfully invoke the mjs function. Can you try to see if this version of dynamo works better. If not, can you try using esbuild (doesn't have to be TypeScript) to transpile the code into cjs before invoking.

[RickardTaran]

Hey @mildaniel , I appreciate you taking some time to look at this with me. I tried pulling that specific version and unfortunately it ran into the same error. I'm not sure if esbuild will be of much here because I'm trying to run a sam local command. This is very likely related to me not having files structured correctly. To give a more holistic example, I created a repo for this bug that we can all play with to see what needs to change. Feel free to pull that down and see if there's anything obvious sticking out. Thanks!

https://github.com/RickardTaran/aws-sam-bug-4342

[mildaniel]

Thanks for providing the example. I tried using it but ran into the following error:

{"errorType":"TypeError","errorMessage":"DynamoClient.executeStatement is not a function","trace":["TypeError: DynamoClient.executeStatement is not a function","    at Statement (file:///var/task/src/utils/dynamodb.mjs:8:31)","    at Runtime.rootHandler [as handler] (file:///var/task/src/handlers/products.mjs:4:24)","    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1089:29)"]}

I went ahead and tried creating a minimal app based on your project to get aws-sdk V3 working with Dynamo and came up with something like this.

├── hello-world
│   ├── app.js
│   ├── dynamo.js
│   ├── package-lock.json
│   └── package.json
├── samconfig.toml
└── template.yaml

app.js -> Identical to the one you have

template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs16.x
      Architectures:
        - x86_64
      Events:
        HelloWorld:
          Type: Api #
          Properties:
            Path: /hello
            Method: get
    Metadata:
      BuildMethod: esbuild

dynamo.js

import { DynamoDBClient, ExecuteStatementCommand } from "@aws-sdk/client-dynamodb";

const DynamoClient = new DynamoDBClient({
    endpoint: 'http://dynamo-local:8000'
})

const Statement = async (statement) => {
    let response = {}
    try {
        const command = new ExecuteStatementCommand(statement)
        response = await DynamoClient.send(command)
    } catch (error) {
        response  = {}
    }
    return response
}

export { Statement }

I was able to invoke this without any issues and connect it to a local DynamoDB. You can use mjs syntax since esbuild will take that and transpile it to cjs during sam build.

[RickardTaran]

Hey @mildaniel, sorry for taking awhile to get back. Thanks for writing all this out and playing around with my code to get it working. I was able to get everything working locally and merged those updates into the topic repo I mentioned earlier.

RickardTaran/aws-sam-bug-4342#1

There's one concern with this though that unfortunately is a big blocker. Having to run esbuild with sam build means that a .aws-sam distribution is made and used in sam local. If you look in the root package.json for the repo, you'll see I'm trying to use

sam local start-api

This local api still works with the sam build output. But, live reloading no longer works. There's been a number of issues opened on this (ex. #901). So with this setup, we're going from being able to update code and test it locally in real time, to:

1. make code update
2. kill local api
3. run sam build
4. restart local api
5. test

I'm glad to see we're coming to a solution on the v3 sdk issue. But, to be honest, unless there's something else in the sam cli that makes the live reloading possible with sam build, I'd rather still deal with the v2 sdk and have that live reloading functionality. I know that sounds kind of picky, and, at this point, I'm okay with seeing this moved to maintenance/ feature request/ etc. It would just be really nice to do this with the local start-api.

[mildaniel]

I appreciate the difficulty of iterating without live reloading. Unfortunately this is a limitation today with any compiled runtimes or anytime you need to run sam build. As a feature request, would it be suitable for sam build to trigger on changes for the live reloading to keep working as a solution to this that I can bring to the team?

Also, have you tried using sam sync? This command is part of the Accelerate set of tools that enables live reloading in the cloud.

[RickardTaran]

That makes complete sense. If your team already has such a feature request on the backlog, im not sure if i'd call this a duplicate, but I can totally see this being enveloped into it. Either way, having that functionality with sam build would be a huge win, especially with esbuild.

Sam sync is a also a great feature and well worth evaluating. But having the ability to develop locally solves a lot of problems and it would be great to leverage these toolsets that way if all the stars were to align.

I appreciate everyone's willingness to help and timely responses. That's what keeps me leaning orange. It's always day one.

Thanks all!

[RickardTaran]

@mildaniel, @mndeveci After playing around with this some more, I was able to move the v3 sdk modules, as well as the client library file, to a layer that is added to the function. This allows everything to work locally with live reloading. The downside is that I still have to use .mjs extensions for the function code itself, as well as the shared library. And now that @mildaniel showed how to add in esbuild to this, this seems a little sub-par :/

RickardTaran/aws-sam-bug-4342#2

Nevertheless, we got it working. My original issue is fully solved and I now have an example that I can leverage in other projects. If there's still value in leaving this open to track some work regarding the live reloading with SAM build, we absolutely can. But I have a solution for my initial problem. Thankyou again to the both of you and the team for your help!

[mildaniel]

Thanks for following up! I'll close this issue in favour of #921 which addresses the live-reloading issue.