Is there a memory issue? #57

Closed
jamuhl opened this Issue Dec 14, 2011 · 3 comments

Comments

Projects
None yet
3 participants

jamuhl commented Dec 14, 2011

If i setup a connection and just publish a lot of messages. All works fine -> messages get published to the queue.

But if i look at memory usage it goes up and stays at the high level. I didn't found out where the problem is.

Sample Code i used:

var amqp = require('amqp')
  , connection = amqp.createConnection({url: "amqp://guest:guest@localhost:5672"});

connection.on('ready', function () {

    for (i = 1; i <= 300000; i++) {
        connection.publish('sample', {msg: i + ' hello world'});
    }

});
Contributor

bergman commented Nov 6, 2012

I experience the same in testing. Printing process.memoryUsage() makes it really obvious. I don't know if this is something to worry about at all in production, but it is peculiar.

before: { rss:  15081472, heapTotal:  6131200, heapUsed:  3915368 }
after:  { rss: 102629376, heapTotal: 64406784, heapUsed: 35432280 }
var amqp = require('amqp');
var connection = amqp.createConnection({url: "amqp://localhost"});

var queue = 'test';
var msg = 'foo';
connection.on('ready', function () {
    console.log('before:', process.memoryUsage());

    for (var i = 1; i <= 9000; i++) {
        connection.publish(queue, msg);
    }

    setTimeout(function() {
        console.log('after: ', process.memoryUsage());
        connection.end();
    }, 1000);
});

zuk commented Oct 30, 2013

I ran into this as well. The problem seems to be that the current implementation is sending .write()s to the underlying TCP socket without checking whether the socket is ready to receive more data. This causes the socket to start buffering data in memory, and that buffer can balloon very quickly.

The following pull request exposes an isBackedUp property on the Connection that lets you check whether its safe to keep sending data without growing the in-memory buffer: #253

Basically if queue.connection.isBackedUp is true, you (the node-amqp user) have to stop sending messages and either discard them or buffer them in some safe, manageable way.

The pull request also adds the ability to specify a max memory buffer size (e.g. amqp.createConnection({..., maxWriteBuffer: 1e6}) for a 1 MB buffer).

Not sure if this is the best way to deal with this problem, but it does seem to resolve the memory growth issues I've been seeing.

jamuhl commented Aug 16, 2016

closing to cleanup my own issue list

jamuhl closed this Aug 16, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment