Kombu 4.1.0 - Memory usage increase (leak?) on a worker when using kombu queues

[sradhakrishna]

Hi,

I have implemented a worker using Kombu's SimpleQueue. The implementation is as given below. When I run this worker for a few hours on a Ubuntu 16.04 system with redis as the backend, I notice a gradual memory build up on the process. When I run this worker for over a day, it ends up consuming all memory on the system and the system ends up being unusable, until the worker is killed.

On redis server, I have it configured with a timeout set to 5 seconds and a tcp_keepalive set to 60 seconds.

Worker Code:

from kombu import Connection

myqueue_name = 'test_queue'
backendURL = 'redis://127.0.0.1:6379/'

def GetConnection():
    conn = Connection(backendURL)

    return conn

def dequeue():
    conn = GetConnection()
    with conn:
        myqueue = conn.SimpleQueue(myqueue_name)

        item = None

        try:
            qItem = myqueue.get(block=True, timeout=2)
            item = qItem.payload
            qItem.ack()
        except Exception as e:
            qItem = None

        myqueue.close()

    conn.close()
    conn.release()

    return item

if __name__ == '__main__':
    try:
        i = 1
        while True:
            print 'Iteration %s: %s' % (i, dequeue())
            i = i + 1
    except (KeyboardInterrupt, SystemExit):
        print 'Terminating'

Here's a plot of free memory on the system:

What is going wrong here? Did I miss anything in the implementation?

Any help here will be greatly appreciated.

[auvipy]

could you try the 4.2 version from master and check if anything improve?

[sradhakrishna]

Same behavior with 4.2 from master as well. Actually, it seems worse - memory is piling up rather quick.

Am I missing anything in the code above, to kick off memory release from objects, connections/ queues etc?

[auvipy]

ask your question to mailing list/irc referencing the issue

[sradhakrishna]

I am trying to post the question on the google groups mailing list - somehow, the post doesn't show up.

@auvipy Can you help confirm this behavior?
@everyone, any suggestions/ pointers please?

[sradhakrishna]

Addressing this issue is quite important to me - it's pretty much lines down scenario for my usecase.

Switching away from kombu to something else requires quite some time so don't seem to have many options other than fixing this.

Any way I can get attention of the community to this issue?

Any help in addressing this issue will be greatly appreciated.

[auvipy]

could you try rabbitmq? that might be a temporary solution

[sradhakrishna]

I've tried the same with rabbitmq as the backend. Saw the same behavior in that scenario too. Seems that the issue might not be in the backend specific implementation.

[auvipy]

try to find out thee leak, also I would suggest to install all the dependencies of celery from master branch to verify on master

[aviperetz34]

I am experiencing the same issue. the issue disappears when you don't use timeout and keep blocking.

[auvipy]

@pawl

[pawl]

@auvipy Nice find, it seems like this could definitely be related to: celery/celery#4843 (comment)

[pawl]

This may be fixed by: #1476

[auvipy]

ok merged, lets see