Autoscaling resque question

[a-schaefers]

Can I safely autoscale a rails container running resque...? It'd be connecting to a shared / decoupled database and redis instance.

I have a concern that when multiple resque/rails containers are running, that the queues might get weird? would each resque container need its own redis instance to talk to, or can they all just talk to the same redis?

[a-schaefers]

Does the master process require it be running monolith style to control all the workers across various nodes? Is there a way to configure this (not kubernetes) or is it all just going to be ok, spinning this thing up over and over again, with various master processes that are only aware of their own nodes?

is any state contained within the resque master process or does it keep everything that other resque containers might need to know in redis just by default?

[PatrickTulskie]

The way I have been running it in production at scale is to have a decoupled redis database living outside of the k8s cluster and then just let the autoscaler do its thing depending on workloads. Usually I work off CPU or Memory depending on the system, but there are also ways to autoscale based on number of jobs in a queue for example.

The way I structure the setup is like this:

• Redis living on its own server. If you use AWS, you can use Elasticache just fine.
• I use resque-pool to manage which workers startup for each queue.
• Depending on your unit of separation, each one starts up resque pool which then forks off each of the workers. If you're using k8s, it would be pods. If you use bare metal or some other kind of VPS instance, each one would start up with a finite amount of workers.
• Servers/pods/instances can die off but the state of the queue is tracked in redis externally so you can just add new boxes into rotation.
• Redis also tracks the state of the workers, so if a resque main process is killed off and it hasn't had a chance to report in, you may have an incorrect number of workers in your resque dashboard. To get around that, I run a script that periodically cleans them up. That script looks something like this:

Resque.workers.each do |w|
  w.unregister_worker if w.processing['run_at'] && Time.now - w.processing['run_at'].to_time > 1000
end

Hope that answers your questions!