Some of my jobs are more intensive than others -- and I don't know in advance how they will pan out. I'd like to find a way to keep system load between 0.9 and 1.1, for example, by adjusting concurrency on the fly. Is this crazy -- should I do it another way?
One idea: is it possible to send concurrency-adjusting signals to Sidekiq from Monit? That would do the trick.
I thought @meskyanichi had support for Sidekiq in HireFire but maybe not.
I'd love for someone to investigate what might be required and report back here. If you have multiple Sidekiqs, it might be hard since each process would be monitoring the load and starting its own workers.
Unicorn, for example, uses these signals:
"If you have multiple Sidekiqs, it might be hard since each process would be monitoring the load and starting its own workers." -- Agreed! The simplest thing would be to respond to signals and let Monit or someone else handle the observing and adjusting.
What would it take to make Sidekiq respond to signals to adjust concurrency (see message above)? I would have to dig in and look closer... any pointers on where to start?
Sidekiq's signal handling is at the top of cli.rb. You can see how various signals are dispatched to the Manager, which is the actor that should be adjusting the Processor count.
Closing due to no progress and little demand.