how many simultaneous workers can we run?

[hknaksu]

I have been testing delayed for a few days, seems rock solid!
I like the philosophy on sticking with ActiveJob and ActiveRecord database being used.
Got a quick question on the number of workers we can run safely at the same time?
I want to utilize all the cores on a big server. I have kicked 4 workers with 5 max_claims and
have not experienced any issues so far. Would there be any adverse effects starting up to 10 workers?
Thank you.

[smudge]

Hey @hknaksu, glad to hear that it's going well so far!

So, I haven't done any extensive profiling on anything other than our in-house deployment (which is Kubernetes-based), but think that the best way to utilize all of your cores is to spin up at least as many workers as there are cores. Ruby's global interpreter lock will prevent the threads themselves from benefitting from more than one core (so max_claims can only allow you to scale up concurrency on a single core). Therefore, running 10 or more workers on a relatively hefty server does seem reasonable to me.

That said, whether or not there are any adverse effects will depend heavily on the footprint of your app, and what kinds of jobs are being run. If the jobs are very DB-intensive, then you might see your DB CPU max out during periods of heavy background work, which could impact your web processes. And since these workers will be sharing resources on the same server, you may also hit up against memory or CPU constraints with the worker processes themselves. At Betterment we do regularly scale up to hundreds of workers, but they are distributed across multiple k8s nodes, so we mostly worry about running too many database-bound jobs at once.

[hknaksu]

This helps, thanks for the quick reply.