High availability deploy of flower

[arthurzenika]

We are running a high availability application using redis and sentinel, and deploying flower to monitor and web admin our celery task queue. We have two flower applications installed and sitting behind a load balancer and we are getting different informations from the two web applications. If the applications have been restart at the same time we mostly get the same numbers out of both application, but in some cases, the informations differ (with an interesting flashing animation).

Any ideas on how we could solve this problem ?

[johnarnold]

What you're observing is that they have different results in their in-mem Task and Worker caches. If you enable persistence, and do "clean shutdowns" of flower when you reboot, they will come back up with most of their previous state and your instances will be closer to the same. They won't be the same because some events (cache updates) are missed while flower is down. We have been discussing how to minimize or avoid that in:
#724

[arthurzenika]

@johnarnold thanks for your answer, we'll look into activating http://flower.readthedocs.io/en/latest/config.html?highlight=persistent#persistent and see if that's enough and if not we'll take a closer look at the postgres persistence PR

[BlackVoid]

@arthurlogilab did you manage to solve this issue? I'm having the same problem, but I'm running Flower in multiple EC2 instances without persistent storage.

[arthurzenika]

@BlackVoid nope, we haven't solved this problem. Good luck for your use case.