Any way to dynamically adjust concurrency? #408

Closed
bluemont opened this Issue Sep 20, 2012 · 6 comments

Comments

3 participants
@ghost

ghost commented Sep 20, 2012

Some of my jobs are more intensive than others -- and I don't know in advance how they will pan out. I'd like to find a way to keep system load between 0.9 and 1.1, for example, by adjusting concurrency on the fly. Is this crazy -- should I do it another way?

One idea: is it possible to send concurrency-adjusting signals to Sidekiq from Monit? That would do the trick.

Owner

mperham commented Sep 20, 2012

I thought @meskyanichi had support for Sidekiq in HireFire but maybe not.

I'd love for someone to investigate what might be required and report back here. If you have multiple Sidekiqs, it might be hard since each process would be monitoring the load and starting its own workers.

@ghost

ghost commented Sep 21, 2012

Unicorn, for example, uses these signals:

  • TTIN - increment the number of worker processes by one
  • TTOU - decrement the number of worker processes by one

http://unicorn.bogomips.org/SIGNALS.html

@ghost

ghost commented Oct 3, 2012

"If you have multiple Sidekiqs, it might be hard since each process would be monitoring the load and starting its own workers." -- Agreed! The simplest thing would be to respond to signals and let Monit or someone else handle the observing and adjusting.

What would it take to make Sidekiq respond to signals to adjust concurrency (see message above)? I would have to dig in and look closer... any pointers on where to start?

Owner

mperham commented Oct 3, 2012

Sidekiq's signal handling is at the top of cli.rb. You can see how various signals are dispatched to the Manager, which is the actor that should be adjusting the Processor count.

Owner

mperham commented Jan 8, 2013

Closing due to no progress and little demand.

@mperham mperham closed this Jan 8, 2013

bgvo commented May 25, 2017

Anyone came up with a solution for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment