Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Is there a memory issue? #57

Open
jamuhl opened this Issue · 2 comments

3 participants

@jamuhl

If i setup a connection and just publish a lot of messages. All works fine -> messages get published to the queue.

But if i look at memory usage it goes up and stays at the high level. I didn't found out where the problem is.

Sample Code i used:

var amqp = require('amqp')
  , connection = amqp.createConnection({url: "amqp://guest:guest@localhost:5672"});

connection.on('ready', function () {

    for (i = 1; i <= 300000; i++) {
        connection.publish('sample', {msg: i + ' hello world'});
    }

});
@bergman

I experience the same in testing. Printing process.memoryUsage() makes it really obvious. I don't know if this is something to worry about at all in production, but it is peculiar.

before: { rss:  15081472, heapTotal:  6131200, heapUsed:  3915368 }
after:  { rss: 102629376, heapTotal: 64406784, heapUsed: 35432280 }
var amqp = require('amqp');
var connection = amqp.createConnection({url: "amqp://localhost"});

var queue = 'test';
var msg = 'foo';
connection.on('ready', function () {
    console.log('before:', process.memoryUsage());

    for (var i = 1; i <= 9000; i++) {
        connection.publish(queue, msg);
    }

    setTimeout(function() {
        console.log('after: ', process.memoryUsage());
        connection.end();
    }, 1000);
});
@zuk

I ran into this as well. The problem seems to be that the current implementation is sending .write()s to the underlying TCP socket without checking whether the socket is ready to receive more data. This causes the socket to start buffering data in memory, and that buffer can balloon very quickly.

The following pull request exposes an isBackedUp property on the Connection that lets you check whether its safe to keep sending data without growing the in-memory buffer: #253

Basically if queue.connection.isBackedUp is true, you (the node-amqp user) have to stop sending messages and either discard them or buffer them in some safe, manageable way.

The pull request also adds the ability to specify a max memory buffer size (e.g. amqp.createConnection({..., maxWriteBuffer: 1e6}) for a 1 MB buffer).

Not sure if this is the best way to deal with this problem, but it does seem to resolve the memory growth issues I've been seeing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.