Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Any way to dynamically adjust concurrency? #408

Closed
bluemont opened this Issue · 5 comments

2 participants

Bluemont Labs Mike Perham
Bluemont Labs

Some of my jobs are more intensive than others -- and I don't know in advance how they will pan out. I'd like to find a way to keep system load between 0.9 and 1.1, for example, by adjusting concurrency on the fly. Is this crazy -- should I do it another way?

One idea: is it possible to send concurrency-adjusting signals to Sidekiq from Monit? That would do the trick.

Mike Perham
Owner

I thought @meskyanichi had support for Sidekiq in HireFire but maybe not.

I'd love for someone to investigate what might be required and report back here. If you have multiple Sidekiqs, it might be hard since each process would be monitoring the load and starting its own workers.

Bluemont Labs

Unicorn, for example, uses these signals:

  • TTIN - increment the number of worker processes by one
  • TTOU - decrement the number of worker processes by one

http://unicorn.bogomips.org/SIGNALS.html

Bluemont Labs

"If you have multiple Sidekiqs, it might be hard since each process would be monitoring the load and starting its own workers." -- Agreed! The simplest thing would be to respond to signals and let Monit or someone else handle the observing and adjusting.

What would it take to make Sidekiq respond to signals to adjust concurrency (see message above)? I would have to dig in and look closer... any pointers on where to start?

Mike Perham
Owner

Sidekiq's signal handling is at the top of cli.rb. You can see how various signals are dispatched to the Manager, which is the actor that should be adjusting the Processor count.

Mike Perham
Owner

Closing due to no progress and little demand.

Mike Perham mperham closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.