Allow running multiple orchestrators in active-passive mode #335
Conversation
adamruzicka
force-pushed
the
orchestrator-failover
branch
from
049ecaa
to
269b6e5
Compare
|
Rubocop should be fixed by #334 |
Upon startup, the orchestrator checks if dynflow_orchestrator_uuid key is unset in Redis. If so, it acquires an orchestrator lock by associating the uuid of the orchestrator's world with the key. However this lock has limited time to live and needs to be periodically refreshed. If the key is set, the orchestrator goes into passive mode and periodically polls the lock's status. If it manages to acquire the lock, it becomes the active orchestrator. Active orchestrator periodically updates the record in Redis to ensure its freshness.
adamruzicka
force-pushed
the
orchestrator-failover
branch
from
269b6e5
to
1b38d69
Compare
There was a problem hiding this comment.
Can't we use https://github.com/kenn/redis-mutex ? I just generally do not like to make something what is out there already 🤔
| mode = nil | ||
| loop do | ||
| active = ::Sidekiq.redis do |conn| | ||
| conn.set(REDIS_LOCK_KEY, @world.id, :ex => REDIS_LOCK_TTL, :nx => true) |
There was a problem hiding this comment.
Is the set operation on redis atomic?
|
I looked at it briefly and I didn't exactly set our use case. We need to take the lock, make it expire after a while, but keep refreshing it as long as we hold it. As far as I can tell, the refreshing part is not implemented there. But I'd be happy to be proven wrong :) |
|
no I haven't seen the decay introduced there. It's effective only for a period of time, and thus can be probably forced to way we want, but it wouldn't feel right. |
|
This was meant mainly as a safeguard to prevent accidentally running multiple orchestrators. I could change it to only prevent another orchestrator from starting if some is already active, but imo this gives a better user experience. |
There was a problem hiding this comment.
Looks good as a start, we can change the behaviour upon user feedback.
Thanks @adamruzicka 👍
Upon startup, the orchestrator checks if dynflow_orchestrator_uuid key is unset in Redis. If so, it acquires an orchestrator lock by associating the uuid of the orchestrator's world with the key. However this lock has limited time to live and needs to be periodically
refreshed.
If the key is set, the orchestrator goes into passive mode and periodically polls the lock's status. If it manages to acquire the lock, it becomes the active orchestrator.
Active orchestrator periodically updates the record in Redis to ensure its freshness.
Redmine: https://projects.theforeman.org/issues/27412