Failover for celerybeat

[piaoyuankui]

I want to make schedule process run in multiple nodes to improve availability, for reducing single point issue.

[phblj]

To my knowledge, there is not.

What we've done is subclassed Scheduler to check if another beat server has scheduled the same job.

The code's not quite up to public-consumption quality (hence no pull request) and has only been tested with a mongo backend, but I'm happy to throw it in a gist for reference: https://gist.github.com/phblj/6133684

[piaoyuankui]

Thanks, Phblj!

[piaoyuankui]

Hi Phblj,

I don't think you show me the whole gist for mongodb backend solution, there is no any mongo related code there, or did I missunderstanding in some point.

[piaoyuankui]

base on your gist code, here is what am i done, using redis backend.
https://gist.github.com/kerwin/6138597

[phblj]

Hi kerwin,

Sorry about that. I pulled the get_last_run and update_last run methods out, so you can see what it's doing.

Basically, when firing I grab the last_run time from our database. If it's time for it to run again (often it's not because another server fired) then it tries to update the field, claiming the run. In mongo, the update is not quite atomic, but we haven't had any issues. I think a find_and_modify is the right way to do it. Anyway, if it successfully claims the run, then it does the standard apply_async.

[diranged]

Ask,
This seems to be a fairly common request ... the ability to have multiple Celerybeat workers that are distributed for redundancy. Lewis Franklin wrote his own version of this based on Zookeeper (https://gist.github.com/brolewis/5799103) as the backend.

It seems logical to build this into Celery natively using the Kombu Broker backend... Do you think you could work on this?

[brolewis]

I've offered to look into doing this natively as well but any help/suggestions would be appreciated.

[piaoyuankui]

Two things should be considered.

1. Where to share these task status. Is kombu/rabbitmq appreciated to do this?
2. How to do atomic operation on these tasks, like change 'next_run' field for a task?

[ask]

I don't have the time to work on this at the moment as I'm currently focusing on 3.1.

@kerwin, I think that making this a distributed service will be incredibly challenging, it depends on timestamps after all and the precision must be less than second.

What is wanted is HA for celerybeat, so that it's possible to do failover if the beat service goes offline.
This is possible using a service like zookeeper for consensus, or writing a consensus protocol for AMQP,
that way the backup beat processes stay idle until they are elected as a new 'master'.

[gtaylor]

We could really use this at Pathwright as well. We currently have to have a "master celery VM" that runs celerycam+celerybeat. If it goes down or becomes unresponsive (as EBS-backed VMs on AWS tend to do periodically), we lose our periodic task execution (and celerycam, though that's a lot less of a worry).

I don't know the answer, but it is a bit of a glaring single point of failure at this point. Many of the brokers can be made redundant. The worker processes can be made redundant by spinning up multiple machines. Not so for periodic task scheduling.

[riyadparvez]

This would be e great feature

[ant31]

Does the new worker-gossip feature would help/should be use to do it?
Is someone already working on it?