serverless-plugin-datadog + serverless-layers + nestjs/swagger weird incompatibility

[Shereef]

https://github.com/Shereef/nestjs-swagger-serverless-layers-datadog-issue#readme

In this repo I am trying to explain that when the following is installed and enabled the project doesn't work when deployed to aws lambda

1. serverless-layers
2. serverless-plugin-datadog
3. @nestjs/swagger

if you disable any of them everything works fine (except the disabled protion ofc)

To test this out

1. Add DATADOG_API_KEY env var with your datadog env vars
2. Add AWS key id and secret to your env vars AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY
3. run npm i
4. run npx serverless deploy --verbose
5. visit the url from the console output

It should display swagger but it doesn't instead {"message": "Internal server error"}

see bottom for full output

if you append /v1 to the url it should show Hello World! but it doesn't

i.e. https://qwerty.execute-api.us-east-2.amazonaws.com/dev/ should show swagger
i.e. https://qwerty.execute-api.us-east-2.amazonaws.com/dev/v1 should show Hello World!

The three above combined break the lambda function

P.S. if you switch to any of the following branches the project works as expected

1. disable-serverless-layers-works preview changes
2. disable-datadog-works preview changes
3. disable-swagger-works preview changes

Test Event Name
root

Response
{
  "errorType": "TypeError",
  "errorMessage": "pathToRegexp.parse is not a function or its return value is not iterable",
  "trace": [
    "TypeError: pathToRegexp.parse is not a function or its return value is not iterable",
    "    at SwaggerExplorer.validateRoutePath (/opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:156:41)",
    "    at /opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:136:35",
    "    at Array.map (<anonymous>)",
    "    at SwaggerExplorer.exploreRoutePathAndMethod (/opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:135:30)",
    "    at /opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:72:45",
    "    at Array.reduce (<anonymous>)",
    "    at /opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:71:104",
    "    at /opt/nodejs/node_modules/lodash/lodash.js:13469:38",
    "    at /opt/nodejs/node_modules/lodash/lodash.js:4967:15",
    "    at baseForOwn (/opt/nodejs/node_modules/lodash/lodash.js:3032:24)"
  ]
}

Function Logs
START RequestId: d721794d-11ac-1234-a0ef-acc07be0df1e Version: $LATEST
�[32m[Nest] 15  - �[39m12/06/2022, 4:10:59 AM �[32m    LOG�[39m �[38;5;3m[NestFactory] �[39m�[32mStarting Nest application...�[39m�[38;5;3m +43001ms�[39m
�[32m[Nest] 15  - �[39m12/06/2022, 4:10:59 AM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mAppModule dependencies initialized�[39m�[38;5;3m +23ms�[39m
2022-12-06T04:11:00.034Z	d721794d-11ac-1234-a0ef-acc07be0df1e	ERROR	[dd.trace_id=2179485768246681000 dd.span_id=4424661968803014230] Invoke Error 	{"errorType":"TypeError","errorMessage":"pathToRegexp.parse is not a function or its return value is not iterable","stack":["TypeError: pathToRegexp.parse is not a function or its return value is not iterable","    at SwaggerExplorer.validateRoutePath (/opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:156:41)","    at /opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:136:35","    at Array.map (<anonymous>)","    at SwaggerExplorer.exploreRoutePathAndMethod (/opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:135:30)","    at /opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:72:45","    at Array.reduce (<anonymous>)","    at /opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:71:104","    at /opt/nodejs/node_modules/lodash/lodash.js:13469:38","    at /opt/nodejs/node_modules/lodash/lodash.js:4967:15","    at baseForOwn (/opt/nodejs/node_modules/lodash/lodash.js:3032:24)"]}
2022-12-06 04:11:00 UTC | DD_EXTENSION | ERROR | LogMessage.UnmarshalJSON: can't read the spans object
END RequestId: d721794d-11ac-1234-a0ef-acc07be0df1e
REPORT RequestId: d721794d-11ac-1234-a0ef-acc07be0df1e	Duration: 276.08 ms	Billed Duration: 277 ms	Memory Size: 1024 MB	Max Memory Used: 175 MB

Request ID
d721794d-11ac-1234-a0ef-acc07be0df1e

Specifications

• Serverless Framework version: 3.25.1
• Datadog Serverless Plugin version: 5.12.0
• Lambda function runtime (Python 3.7, Node 10, etc.): node 16

[astuyve]

Hi @Shereef - thanks for the detailed report and reproducible case!

In the stack trace reported here, there isn't any reference to a datadog module, it seems the stack trace log is captured by DataDog (hence why dd.trace_id and dd.span_id is present in the log message), but I don't think it's thrown from our code.

2022-12-06T04:11:00.034Z	d721794d-11ac-1234-a0ef-acc07be0df1e	ERROR	[dd.trace_id=2179485768246681000 dd.span_id=4424661968803014230] Invoke Error 	{"errorType":"TypeError","errorMessage":"pathToRegexp.parse is not a function or its return value is not iterable","stack":["TypeError: pathToRegexp.parse is not a function or its return value is not iterable","    at SwaggerExplorer.validateRoutePath (/opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:156:41)","    at /opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:136:35","    at Array.map (<anonymous>)","    at SwaggerExplorer.exploreRoutePathAndMethod (/opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:135:30)","    at /opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:72:45","    at Array.reduce (<anonymous>)","    at /opt/nodejs/node_modules/@nestjs/swagger/dist/swagger-explorer.js:71:104","    at /opt/nodejs/node_modules/lodash/lodash.js:13469:38","    at /opt/nodejs/node_modules/lodash/lodash.js:4967:15","    at baseForOwn (/opt/nodejs/node_modules/lodash/lodash.js:3032:24)"]}

When I searched for references to this error, I saw an issue in the nestjs/swagger project. Is it possible this is related?

Is it possible you're running into a layer limit?

Thanks!

[Shereef]

The error only happens when I have datadog enabled
So something is breaking it

Yes the error is from swagger but if I remove datadog form the plugin list in serverless.yml everything works

[Shereef]

Any update on this ?

[astuyve]

Hi, I briefly experimented with the repository. This looks to be a combination of two issues.
The first is with serverless-layers (or layers more generally), and the second is a node module dependency mismatch for path-to-regexp being required by both dd-trace and nest.

Looks like nest is on a newer version:

├─┬ @nestjs/core@9.2.1
│ └── path-to-regexp@3.2.0
├─┬ @nestjs/platform-express@9.2.1
│ └─┬ express@4.18.2
│   └── path-to-regexp@0.1.7
└─┬ @nestjs/swagger@6.1.3
  └── path-to-regexp@3.2.0

vs dd-trace:
https://github.com/DataDog/dd-trace-js/blob/9b350c1f759c76c85fb77c340293b27513983f48/package.json#L83
The dd-trace version does not export a method called parse, it just exports a method by name:

This happens because Lambda will flatten all layers into opt, and merges folders with the same name:

If your Lambda function includes layers, Lambda extracts the layer contents into the /opt directory in the function execution environment. Lambda extracts the layers in the order (low to high) listed by the function. Lambda merges folders with the same name, so if the same file appears in multiple layers, the function uses the version in the last extracted layer.

docs.

So you have a few options:

• You can disable serverless-layers, which likely isn't helping that much anyway (except maybe a bit in terms of deployment time). Then your app dependencies (along with path-to-regexp@3.x) will not be flattened into /opt/node_modules, they'll just be stored in node_modules.
• You can follow Option B here and skip using Datadog's layers, and instead install dd-trace as a dependency. Be sure to specify addLayers: false in the datadog config block. docs.
• You can switch the merge order so your function layer is merged after dd-lambda-js. This may cause the router functionality of dd-trace to break, but it won't break your function. I tested this, and it works:

Hope this helps!

[astuyve]

Closing as there's been no update on this, but please do let me know if my comment helps @Shereef!

[Shereef]

You can switch the merge order so your function layer is merged after dd-lambda-js. This may cause the router functionality of dd-trace to break, but it won't break your function.

Thanks so much for your comments.

You mentioned the above. this doesn't break my function but does it break reporting to datadog ?
How do I do that ?

[Shereef]

Closing as there's been no update on this, but please do let me know if my comment helps @Shereef!

I initially disabled serverless-layers but now we are hitting lambda size limits
serverless-layers helps with that but also lets you debug from aws console if need be because the lambda files are significantly less.

@astuyve Sorry for the delay but we had deadlines and needed to be heads down

I am actively working on this now

[Shereef]

How hard would it be to update path-to-regexp on dd-trace ?

[Shereef]

@astuyve Can you please approve the workflow run DataDog/dd-trace-js#2729 ?

[astuyve]

You mentioned the above. this doesn't break my function but does it break reporting to datadog ?

Not sure, it's really hard to determine that without knowing your specific use case/application code.

How do I do that ?

You can change load order in the aws console.

I initially disabled serverless-layers but now we are hitting lambda size limits
serverless-layers helps with that but also lets you debug from aws console if need be because the lambda files are significantly less.

Layers and function code are all counted towards the same limit.
You might consider exploring containers as a packaging mechanism then, which supports an image size up to 10gb.

How hard would it be to update path-to-regexp on dd-trace ?

Not hard, looks like you've already done that. But regardless, there's an infinite number of combinations which could cause this very same issue.

You might consider using the custom installation method and manually specifying dd-trace and dd-lambda-js as dependencies. This will cause your node modules to be vendored and should prevent path-to-regexp from breaking.

[Shereef]

dd-trace CI really hates my PR
I don't think it was configured to only work well with PRs from it's own repo and not from a fork maybe ?

I don't like the custom method because vs the layer, however, are you saying if I do npm i --save dd-trace dd-lambda-js it should still work nicely while using the DD layer and serverless-layers ?