update to nodejs10.x brings "Malformed Lambda proxy response" error

[adrai]

I updated an existing lambda function runtime from node 8 to 10 now the api gateway logs say: "Malformed Lambda proxy response"

[adrai]

When testing via API-GW UI:

Tue May 14 20:32:59 UTC 2019 : Received response. Status: 200, Integration latency: 759 ms
Tue May 14 20:32:59 UTC 2019 : Endpoint response headers: {Date=Tue, 14 May 2019 20:32:59 GMT, Content-Type=application/json, Content-Length=4, Connection=keep-alive, x-amzn-RequestId=99b0d427-cfe9-4bc4-9a1a-b4d4c392db06, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=276, X-Amzn-Trace-Id=root=1-5cdb25fa-4edb801e05b7f89b19909fa2;sampled=0}
Tue May 14 20:32:59 UTC 2019 : Endpoint response body before transformations: null
Tue May 14 20:32:59 UTC 2019 : Execution failed due to configuration error: Malformed Lambda proxy response
Tue May 14 20:32:59 UTC 2019 : Method completed with status: 502

[adrai]

Seems like the body gets not back to the api-gw... it's always null did the interface for api-gw proxy integration change?

[adrai]

My findings so far:

// still works with node 10 lambda runtime
exports.handler = (event, context) => { awsServerlessExpress.proxy(server, event, context, 'CONTEXT_SUCCEED'); };
exports.handler = (event, context) => { awsServerlessExpress.proxy(server, event, context); };
exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context, 'PROMISE').promise;

// does NOT work with node 10 lambda runtime anymore
exports.handler = (event, context, callback) => { context.callbackWaitsForEmptyEventLoop = false; awsServerlessExpress.proxy(server, event, context, 'CALLBACK', callback); };

[adrai]

The question now is: how to set callbackWaitsForEmptyEventLoop = false?

[adrai]

side effect with (both: node 10 or node 8 runtime) because callbackWaitsForEmptyEventLoop can't be set to false:

exports.handler = (event, context) => { awsServerlessExpress.proxy(server, event, context); };

now I "sometimes" get:

ERROR	Uncaught Exception:  { Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }

But it seems to occur after the lambda function has ended (see: END RequestId....)
... and seems like the request is getting sent twice (like this would be executed twice)
reproducable example for double execution on lambda nodejs10.x

have to switch back to node 8 with callback signature and callbackWaitsForEmptyEventLoop set to false:

context.callbackWaitsForEmptyEventLoop = false;
awsServerlessExpress.proxy(server, event, context, 'CALLBACK', callback);

[adrai]

Seems there are more “bugs” in node 10 runtime for lambda: https://forums.aws.amazon.com/thread.jspa?messageID=901128#901128

[adrai]

fyi: I can reproduce it with: https://github.com/lambci/docker-lambda

[adrai]

I know what the problem is...

The problem is in the Runtime.js:
when callbackWaitsForEmptyEventLoop is set to false, it just calls context.succeed(result) (to early) itself with the result being null because the real result will occur a bit later itself by calling the original callback with callback(null, result). If I remove callbackContext.succeed(result); in my test docker image it works.

      if (_isPromise(result)) {
        result.then(callbackContext.succeed, callbackContext.fail);
      }
      else {
        if (callbackContext.callbackWaitsForEmptyEventLoop) {
          process.prependOnceListener('beforeExit', () => { callback(); });
        }
        else {
          callbackContext.succeed(result);
        }
      }

[adrai]

read this for more info: https://medium.com/@hichaelmart/reviewing-the-aws-lambda-nodejs10-x-runtime-84de1e500aac

[JamesKyburz]

@adrai Thanks for posting this!

I found out this today too when debugging a test lambda with https://github.com/jameskyburz/sandbox-debugger

Not what people would expect that the new official node 10 runtime doesn't support async lambdas using the callback signature.

[thomasmichaelwallace]

FWIW I did flag a related issue to this to one of the more senior lambda team (https://twitter.com/tomincode/status/1131567879659425793)- but there's never any visibility with issues like this- so it's anyone's guess if/when it gets fixed.

[adrai]

Seems AWS updated their nodejs10.x runtime... closing for now.