Memory leak trying to create a while loop with promises

[colmaengus]

I'm trying to create a while loop using promises and I'm getting a memory leak.
An example of the pattern I'm using is

var promiseFor = BPromise.method(function(delay, condition, action, value) {
  if (!condition(value)) {
    return value;
  } else {
    return action(value).delay(delay).then(promiseFor.bind(null, delay, condition, action));
  }
});

var condition = function(value) {
  return value != 100000;
};

var action = function(value) {
  return BPromise.delay(10)
    .then(function() {
      console.log('value:' + value);
      return value + 1;
    });
};

var delay = 0;

promiseFor(delay, condition, action, 0)
  .then(function() {
    console.log('done');
  })
  .catch(function(e) {
    console.error(e.message);
  });

When I run this the memory used by node keeps increasing.
I'm using bluebird 2.9.10.
I've tried another approach to avoid the recursive nature of the approach above but it also has a leak.
Any suggestions on how to correctly implement a promise while loop ?

[benjamingr]

Here is how I'd implement a while loop using bluebird:

var promiseWhile  = BPromise.coroutine(function*(delay, condition, action, value) {
    while(!condition(value){
          value = yield action(value);
          yield Promise.delay(delay);
    }
});

[colmaengus]

Looks nice but unfortunately for now we are using node 0.10 so we can't use generators.

[petkaantonov]

There is no need to implement any helper in the first place, just use recursion

(function loop(value) {
  if (value != 100000) {
    return Promise.delay(10).then(function() {
      console.log('value:' + value);
      return value + 1;
    }).then(loop);
  }
  return Promise.resolve(value);
})(0);

[colmaengus]

@petkaantonov unfortunately this approach also has a leak. On windows the node process gained about 14 MB in 20 mins. I'm looking for some way to keep running a promise function over a long period of time without leaking memory.

[petkaantonov]

@colmaengus It appears you are not actually using 2.9.10 then. Can you log this for me and let us know what it logs:

console.log(BPromise.prototype._rejectCallback + "");

[colmaengus]

function (reason, synchronous, shouldNotMarkOriginatingFromRejection) {
    if (!shouldNotMarkOriginatingFromRejection) {
        util.markAsOriginatingFromRejection(reason);
    }
    var trace = util.ensureErrorObject(reason);
    var hasStack = trace === reason;
    this._attachExtraTrace(trace, synchronous ? hasStack : false);
    this._reject(reason, hasStack ? undefined : trace);
}

[petkaantonov]

Ok so we are surely using the same version then. I cannot reproduce any memory leak. Just to make sure we are on the same page here, can you run this:

var Promise = require('bluebird');

function immediate() {
    return new Promise(function(resolve) {
        setImmediate(resolve);
    });
}

(function loop(value) {
    return immediate().then(function() {
      if (value % 10000 === 0) {
        console.log(process.memoryUsage().heapUsed, value);
      }
      return value + 1;
    }).then(loop);
})(0);

If there is a real leak, the heap usage will grow rapidly.

[petkaantonov]

I ran it until 1110000 (equal to having the original code run for 3 hours) and the heap usage is almost same as when it started.

[colmaengus]

That seems to be pretty stable. The node process memory climbed to 44,304 K and pretty much stays there.

The heapUsed also seems pretty stable.
There are a few places I need to use this and in some cases the check condition uses a class function so I need a clean way to pass 'this' around while trying to keep the code readable.

[petkaantonov]

So do you agree there is no memory leak?

[petkaantonov]

Ok somehow the original code is not equivalent to the straight-forward recursion example and it indeed leaks : https://gist.github.com/petkaantonov/b73d52b9092b6f857271

[colmaengus]

I agree there is no memory leak leaving the interesting question of how to use while loop in practice.

[petkaantonov]

Well you can use your original code or my code.

It leaks when there is a .then() added for the console.log("Done"); but this didn't leak in 2.5.2 which fixed the original memory leak issue with promises in general.

[colmaengus]

Many thanks for your help. I let you know how I get on.

[petkaantonov]

This memory leak was introduced in 2.8.2 -> 2.9.0.

So you could use earlier version before I fix it (2.8.2 is the latest with no leak). 2.9.0 and later are affected