Bluebird unusable in AWS Lambda / NodeJS

[cvillecsteele]

1. What version of bluebird is the issue happening on? 3.4.7

2. What platform and version? (For example Node.js 0.12 or Google Chrome 32) NodeJS 4.3

3. Did this issue happen with earlier version of bluebird? Unknown

In an AWS Lambda function written in JS, using bluebird and executing on NodeJS:

1. a promise is created
2. on a setTimeout, the promise is resolved
3. when it's resolved, the lambda function callback is invoked

IF the lambda function's execution timeout occurs before the setTimeout, the internal state of the bluebird library is hosed, and not only does this invocation of the lambda function time out, but ALL FURTHER invocations of the lambda function fail by timing out.

Code below (sans node_modules needed for packaging as lambda fn). Note that the code below uses the input event's .timeout property for the setTimeout.

var Promise = require("bluebird");

module.exports.hello = function(event, context, cb) {
    var p = new Promise(function(resolve, reject) {
        setTimeout(
            function() {
                resolve();
            },
            event.timeout);
    });
    p.then(function(foo) {
        cb(null, "cb happy");
    });
};

[cvillecsteele]

This is a WTF moment... No real problem with Bluebird... Seems to be an AWS/Lambda problem. Bumping up the RAM allocated to the lambda function makes everything better... Completely inexplicable. I'm opening up a ticket with AWS.

[dorkusprime]

@cvillecsteele Do you happen to have any info on the AWS ticket? This is impacting me as well.

[vasiliyb]

try increasing the memory to 256M

[joekiller]

The issue is a Lambda issue, not bluebird. I made a pure javascript example of the hang with no external non-native libraries. The example project is here: https://github.com/Type-Zero/promise-timeout-hang-js

The key is that the function code outside the handler must take up at least ~16% of the function's memory. If the code in memory is that large and then global timeout is triggered, Lambda seems to lose state. Once the global timeout is triggered for any function with resident imported data occupying > 16% of the maximum memory for the NodeJS lambda function it will no longer work. I tried the 1536 MB function and found that it would suffer the same problem after just 13% of the memory being filled.

So really if you are getting these issues, you might want to import libraries from the handler function. We haven't tried that. Just bumped the memory up.

[joekiller]

FYI I appears that AWS fixed this problem.

[simoncpu]

Currently experiencing this with AWS Lambda on Node.js 6.10. Bumping up the memory to max allows some code to execute but eventually times out.

[truedat101]

@simoncpu can you tell more about the code (code size, function code being outside the lambda handler, whether the problem clears itself up). When you say "some", do you mean, sometimes it works?

@joekiller thank you for tracing this down. I found your post somewhere on the Internet. Did you get a confirmation from anyone at Amazon regarding the issue? I see you and @moea had a good back and forth on this, but was hoping there was a ticket tied to Amazon somewhere ...

[joekiller]

@truedat101 I had a formal ticket in with AWS but they gave me the same ole'we'll fix it when we fix it answer. Best I can say is I tried replicating the problem for another experiment and was unable. I'm pretty sure they fixed the handler issues.

My secondary experiment had a Lamda open a DB connection outside the handler code. The connection persisted after execution until the underlying function was recycled. I couldn't replicate that after the fix either.

See discussion here http://theburningmonk.com/2016/05/aws-lambda-constant-timeout-when-using-bluebird-promise/#comments

Here was AWS' response via Twitter: https://mobile.twitter.com/joekiller/status/877562054961491969

[simoncpu]

@truedat101 My bad, I isolated this issue on our code and turns out that this was related to ioredis, not Bluebird. The solution to prevent Lambda from timing out is at: https://www.linkedin.com/pulse/aws-lambda-reusable-resources-marjan-sterjev.

[truedat101]

@joekiller thanks for the follow up. This was a head scratcher.

[truedat101]

@simoncpu I hit the same problem with ioredis.