Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Ways to solve multiple schedulers. #161

Closed
MichaelXavier opened this Issue · 4 comments

3 participants

@MichaelXavier

Hi. I recently got bitten by having multiple schedulers running at a time. I came up with a strategy to prevent this in future, but I wondered if you'd be interested in solving this in the library. I could try my hand at doing it myself as a pull request if you'd like. I figured running two schedulers on the same resque namespace/server is never what you want so it would be nice if we could trivially prevent it.

Basically, my hack works like so:

desc "Spawn an auto-expiring lock for scheduler (run as a dependency)"
task :scheduler_lock do
  redis    = Resque.redis
  lock_key = "scheduler:lock"
  timeout  = 90
  interval = 60

  if redis.setnx(lock_key, Time.now.to_i)
    Thread.new do
      loop do
        redis.expire(lock_key, timeout)
        sleep(interval)
      end
    end
  else
    puts "Another scheduler is already running. Aborting."
    exit 1
  end
end

# add scheduler lock dependency
task :scheduler => :scheduler_lock

So it basically spins off another thread to periodically poke a lock every once in a while and abandons it. I thought this would be better than a job because its independent of queues being backed up and dies with the process, naturally unlocks if the scheduler dies in a fire. Thoughts?

@bvandenbos
Owner
@meatballhat
Owner

@MichaelXavier still valid?

@MichaelXavier

Nope. I think we switched from our hand-rolled solution to this some time ago and I haven't seen the issue reappear.

@meatballhat
Owner

@MichaelXavier much thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.