Lambda always timeouts for both async or callback handler with middy (returned response is swallowed)

[bokjo]

What are the steps to reproduce this issue?

Running the function below locally with serverless framework (serverless-offline) works as expected but when deployed to AWS it always timeouts. I was using the secret manager and custom db-manager middleware and it was ok but only locally (suspected they were the reason for the timeouts but turns out not the case, the same outcome once removed too).

I've tried switching to callback handler but still the same outcome, the return response object is swallowed and the lambda goes to timeout.
Also tried to return Promise.resolve(...), no luck
All the methods work perfectly ok locally.

import middy from '@middy/core'
import httpEventNormalizer from '@middy/http-event-normalizer'
import httpSecurityHeaders from '@middy/http-security-headers'

// Example only
const handler = async (event, context) => {

    try {
        const { param1 } = event.pathParameters
        const { query1 } = event.queryStringParameters
        const { test } = context
      
        if (!param1) {
            return errorResponse('params')
        }
        
        if (!query) {
            return errorResponse('query')
        }
        
       // more logic...
        return successResponse(true)
    } catch (err) {
        return errorResponse(err)
    }
}

export default middy(handler)
    .use(httpSecurityHeaders())
    .use(httpEventNormalizer())

What happens?

The lambda function always goes to timeout

What were you expecting to happen?

Return success or error response

Any logs, error output, etc?

No errors, all the console logs up to the return statement are correctly displayed in the CloudWatch log but the function remains stuck until it goes to timeout.

Any other comments?

• The code is bundled up with Webpack using the serverless-webpack plugin.
• The lambda is in VPC with internet access and exposed to ApiGateway

p.s tomorrow I'll try to switch to CommonJS modules and try without webpack bundling

What versions of software are you using?

Node.js Version:

• Locally => v12.18.3 (npm v6.14.6)
• AWS => v12.x

Middy Version: => 1.4.0

AWS SDK Version: => 2.767.0

[bokjo]

Ok, I've checked some other similar issues for Middy (since I don't know about the ecosystem and started using it 2 days ago) and seems like @middy/do-not-wait-for-empty-event-loop solves this issue!

But it still seems magic to me and why it is actually happening (Middy, AWS runtime, combination?) and are there some implications for the future (never had a similar issue with a vanilla handler or express for example).

Also, why is @middy/do-not-wait-for-empty-event-loop not mentioned anywhere in the documentation regarding this side effect?

Cheers

[bokjo]

Update!

Works locally and when deployed to AWS

export default middy(handler)
  .use(inputOutputLogger())
  .use(doNotWaitForEmptyEventLoop({ runOnError: true , runOnBefore: true, runOnAfter: true }))
  .use(httpSecurityHeaders())
  .use(httpEventNormalizer())

Adding secrets-manager middleware works only locally! (watching the logs the request doesn't even reach the main handler)
p.s I have the proper secrets manager permission enabled!

export default middy(handler)
  .use(inputOutputLogger())
  .use(doNotWaitForEmptyEventLoop({ runOnError: true , runOnBefore: true, runOnAfter: true }))
  .use(httpSecurityHeaders())
  .use(httpEventNormalizer())
  .use(
    secretsManager({
      secrets: {
        mySectet: process.env.MY_SECRET
      }
    })
  )

[willfarrell]

Glad to see you found a solution to your issue. Sorry that our documentation was clear enough. If you'd like to add in some additional documentation around @middy/do-not-wait-for-empty-event-loop I'd be happy to merge it. Making our docs more useful to first-time developers is important.

Per Secrets Manager, are you able to create a unit test that fails?

[bokjo]

Hello @willfarrell

Thanks for the reply, totally forgot about the issue here.

For the Secrets Manager and SSM middlewares I resolved the issue, it was my misconfiguration on VPC and subnets side and the lambdas were running inside the VPC only without access to the internet and SecretsManager and SSM APIs. After implementing NAT gateway and some private subnets the issue went away.

However, if you have some misconfiguration like for example providing the wrong name for a parameter that cannot be found on SSM the middleware won't throw the error (rather it is not handled anywhere) and the function hangs until it timeouts!

Take look at

• middy/packages/ssm/index.js

  Line 159 in e17f9a4

  const handleInvalidParams = ({ Parameters, InvalidParameters }) => {

• middy/packages/ssm/index.js

  Line 64 in e17f9a4

  .then(handleInvalidParams)

Regarding the @middy/do-not-wait-for-empty-event-loop i can take a look at the documentation during the weekend and let you know.

Regards,
Bokjo

[willfarrell]

Thanks, that's good to know. Would you be interested in writing a PR to catch where this timeout occurs?

[bokjo]

@willfarrell opened PR #572