A set of improvements for Redis result backend aimed to eliminate polling as much as possible and make it responsive for very fast tasks (with execution times under 0.5 seconds - the default polling interval used before).
Should close #799 (GroupResult improvements are already implemented in redis_join branch as far as I can see and are supported in this version by re-implementing RedisBackend.get_many()).
Optimized Redis result backend (publish/subscribe instead of polling)
Optimized Redis backend: py3 compat fixes
The PyPy failure is not related to you.
small fix for tests of redis backend
Optimized Redis backend: chains are more responsive on exceptions, sm…
…all improvement for group results
Changes Unknown when pulling 077d515 on sangwa:redis_nopoll into * on celery:master*.
LGTM. @malinoff @ask Any objections?
@thedrow I can't help with reviewing non-amqp backends - so I think @ask has the final word here.
Its been a while since there was any activity on this PR? Is this dead?
I've marked this for 4.0.
Let's see if @ask responds. If he doesn't, let's rebase and merge this.
Great work! The only reservation I have here is the frightening combination of having "redis" and "lock" appear in the same sentence :)
I realize that you have spent some time on the lock implementation, but I still would prefer if we didn't use it due to the difficulty of proving that it will work in all cases.
Instead of using the lock, would it be possible to hook into Backend.on_task_call to subscribe before the task is sent? The rpc:// backend uses this to make sure the queue is declared before the task message is published.
[result][redis] Use pubsub for consuming results, and use the new asy…
…nc backend interface (Issue #2511)
Since I recently refactored the backends to have an async interface, it means that we can now keep the pubsub instance running instead of creating/destroying it for every task.
I wrote an implementation of this in 1f995b3, it seems to work but I have only tested it naively.
I don't have much experience with redis pubsub, so would be great to have it reviewed.
@sangwa @randylayman Can you guys take a look?
For those just looking for applying this change (3 commits) without waiting for the eventual rebase/merge, I pulled it out of that foreign branch as a patch:
I've been running this patch back-applied to 3.1.18 and it runs fine. The waiting time of a simple Redis-backed Chord I'm running has gone from 1s to about 50ms (which includes actual work). I had to also apply CELERY_DISABLE_RATE_LIMITS=True; otherwise it was still in the hundred ms range, see:
Another nice thing this set of changes seems to make is that results are deleted immediately from Redis instead of hanging around for CELERY_TASK_RESULT_EXPIRES seconds.
After a read this looks good. Unfortunately I don't have the time to test and dig in deeper.
Does the comment by @yingzong need to be documented?
CELERY_DISABLE_RATE_LIMITS should have absolute no effect anymore in 3.1 when using redis/amqp as the broker.
The task results cannot actually be removed if this is a persistent result backend, at the very least not by default.
Clarification: disabling rate limits will have no effect if you don't have any rate limited tasks. As there's no dedicated thread for rate limits any more, there's no longer any overhead.
@ask I just checked again, and you're right on both counts. I must have misremembered some behavior. Thanks for clarifying.
…nc backend interface
Incorporates ideas taken from Yaroslav Zhavoronkov's diff in #2511
Closes Issue #2511
I merged the async backend version, not using locks into master as it's now passing the stress test suite.
@sangwa Even though I didn't merge your commit, I consider this a collaboration as I used your diff as a guide. Thank you for this, it's a feature I have been wanting for a long time.
Could you please add your name to https://github.com/celery/celery/edit/master/CONTRIBUTORS.txt ?
The changes merged into master are: 886faf6 + 456fd75
@ask Thank you for the feedback! I haven't been working with Celery for quite some time, so I didn't follow the new async backend and didn't have enough spare time to review and/or test your changeset thoroughly, but at the first quick glance it seems fine. I trust other contributors to this issue ensured it's working properly.
Hello @ask, if this code is still on the trajectory to get merged into the proper code-base, what is remaining before it is feature and functionally complete? And, would you or anyone like additional help? Documentation and/or dev work.
@jheld it should already be merged into master, you can help us test it, or work on an issue in the 4.0 milestone so we can release it soon :)