BacklogLimitExceeded exception removed

[Simanas]

Hey guys! Thanks for developing this awesome library!

Recently I have stumbled across a problem where I have a long running task, which updates it's progress every second. If that task runs for say 1567 seconds without any other code requesting to see it's progress, I get BacklogLimitExceeded exception on progress or result request. Second time I request for result it woks fine as it now has only 567 entries to parse.

I made a workaround on my app which catches BacklogLimitExceeded exception and tries again until it gets to actual result, but that seems to me like it sounds - a workaround.

Dunno why you guys added that exception here, so here is my pull request to remove it. If it's necessary for some corner cases which I am not aware of, please share your knowledge on why it's needed.

Thank you. Appreciate your work!

[ask]

The only reason it's there is because I was afraid this could be a DoS attack.

You're the first person who has exceeded it though. Do you use the amqp or RPC result backend? Using the amqp result backend for progress doesn't immediately make sense to me. Why is the consumer so slow that it builds a backlog of 1000 state update messages?

[Simanas]

I am using the same technique as in this snippet: https://djangosnippets.org/snippets/2898/

I am not very sure on what you call consumer in this case, but in my case I just need to get progress of a task or it's result. So if I go back to the task which was initiated 6 hours ago and ask for it's state, celery throws this error, because it has like 10 000 progress entries.

Should I do it in some other way?

[ask]

The consumer of the result would be the one receiving or waiting for the result.

Seems like the database backends would be a much better fit for this purpose.
The amqp/RPC result backends needs to send one message per state update, while for the database based backends (redis, sqla, django, mongodb, cache, etc) every new state update will overwrite the old one.

The "amqp" result backend is not recommended at all since it creates one queue per task, which is required to mimic the database based backends where multiple processes can retrieve the result.

The RPC result backend is preferred for RPC-style calls where only the process that initiated the task can retrieve the result.

But if you want persistent multi-consumer result you should store them in a database.

Using rabbitmq as a broker and redis for results is a great combination, but using an SQL database for results works well too.

[Simanas]

Great! Thanks for taking time to explain. Will switch to redis or database backend for results.