Skip to content

Limits not limiting #26

Closed
Shpigford opened this Issue Feb 25, 2014 · 21 comments

5 participants

@Shpigford

It seems queues just aren't being limited. Here's my sidekiq.yml file:

:concurrency: 81
:queues:
  - [fast, 10]
  - [default, 3]
  - [slow, 1]
  - [a, 5]
  - [b, 5]
  - [c, 5]
  - [d, 5]
:limits:
  slow: 45
  fast: 10
  default: 20
  a: 2
  b: 2
  c: 2
  d: 2

And this is how I'm running sidekiq on Heroku (from my Procfile):
worker: env DB_POOL=100 bundle exec sidekiq -e $RACK_ENV -C config/sidekiq.yml

Looking in Sidekiq's list of Busy workers, queue b has 8 workers...even though it's limited to 2.

I have 3 2X workers in Heroku.

Any ideas?

Sidekiq 2.17.6
sidekiq-limit_fetch 2.1.3

@Shpigford

Another not here...I'm at over 700 "busy" workers right now...and rising. Which seems like should be possible at all giving the limits set.

@tyetrask

Hello- I'm seeing the same problem when using this gem. I have a queue where I've specifically set the limit to 1 as I don't want more than one job from that queue being processed at the same time. I don't have anything helpful to add, but I wanted to comment on this in case I could provide any helpful data or if a solution is presented.

@brainopia
Owner

@Shpigford @tyetrask I need a reproduction of the issue. Can you reproduce an issue locally?

The best way would be to isolate the issue. I've created a demo app for this purpose:
https://github.com/brainopia/sidekiq-limit_fetch/tree/master/demo

There is a helpful instruction if you go to the above link.
@tyetrask, the default demo app demonstrates a working limit of 1. So you can try to see what is the difference with your app and why the result is different.

@tyetrask

@brainopia, thanks! I'll have a look at that now and see if I can either a) see if I'm doing something wrong or b) reproduce the issue in the demo app somehow.

@Shpigford

@brainopia At this point I can't reproduce locally. I just know in production, the limiting doesn't seem to be working.

How does sidekiq-limit_fetch work if a process crashes? Would it remove the worker, or could they keep piling up?

@Shpigford

@brainopia I'm wondering if it has something to do with Unicorn. Could it be that every Unicorn process is basically multiplying the limit?

ie. if I have 5 unicorn processes in Heroku and I have a limit of 2 set for a queue...it's effectively a limit of 10?

@brainopia
Owner

@Shpigford if a process crashes the plugin will release invalid locks in less than 30 seconds

Number of processes (unicorn or not) should not matter (unless you use process_limit) since they all connected to the same redis.

Can you please give me the output of the following code when it's ran in production console:

puts Sidekiq::Queue['queue'].explain
puts Sidekiq::Queue['queue'].limit
@ckbhodge
ckbhodge commented Mar 2, 2014

we're also seeing this in production only (on heroku). we're using unicorn, too.

this particular queue should have 1 total worker running, but has 8 showing in sidekiq's UI.

here's the output requested:

irb(main):006:0> puts Sidekiq::Queue['daily'].explain
Current sidekiq process: 4af5963b-809a-4ee1-b30d-9c1fd1664718

  All processes:
945b41c7-ff8e-4a5a-b5c9-5f83d4dad986

  Stale processes:


  Locked queue processes:


  Busy queue processes:

=> nil


irb(main):005:0> puts Sidekiq::Queue['daily'].limit
1
=> nil
@kwent
kwent commented Mar 2, 2014

Same issue here. (Heroku + Unicorn)

@brainopia
Owner

@ckbhodge @kwent

I only now have noticed that @Shpigford gave us an example of how number of reported workers in sidekiq admin were much higher than was possible given concurrency setting of sidekiq.

I have a following suspicion that heroku forcefully kills sidekiq workers on deploy or something along those lines. This would lead to incorrect number of busy workers in sidekiq admin.

That's why @mperham has recently added a commit to prune old worker entries - mperham/sidekiq@404069a#diff-3c240ab72e022d1aea4eb0c148de1299R441.

To confirm my hypothesis you would need to set concurrency to a low number and see if you would still get bigger number of workers than it is possible.

@brainopia
Owner

mperham/sidekiq#1524 - there is a releated issue at sidekiq repo

@brainopia
Owner

mperham/sidekiq#1508 - yeah, this is a confirmed issue

I would suggest that you would trust sidekiq-limit_fetch and check how many workers are busy in reality with Sidekiq::Queue['queue'].busy :)

@brainopia brainopia closed this Mar 2, 2014
@Shpigford

I'm confused...so how do I make sidekiq-limit_fetch work? Upgrade sidekiq based on those new commits? Or something else?

@brainopia
Owner

You should check Sidekiq::Queue['queue'].busy numbers and not a sidekiq admin page with busy workers. This method will show precise number of running workers for the queue (while admin page can contain incorrect values).

If #busy will show more workers than a set limit then we can investigate, but I'm pretty sure it will be okay.

Regarding the sidekiq admin app you can clean a list of busy workers manually with a click of a button. But it's better to ask @mperham about it.

@ckbhodge
ckbhodge commented Mar 3, 2014

thanks. the odd thing is that Sidekiq::Queue['queue'].busy returns 0 for every queue we have, even though they are unpaused. Same thing yesterday when I checked.

@brainopia
Owner

@ckbhodge it probably happens because you have quick tasks and not a lot of them.

You can pause queue for a while Sidekiq::Queue['queue'].pause so there will be plenty tasks to process and then unpause and check busy status. It will reflect exactly how many are running at each moment.

Or you can just check #busy status in a loop. It will be easier to catch when tasks are processed.

@tyetrask
tyetrask commented Mar 3, 2014

Hey @brainopia, I haven't replied since much earlier in the thread. I think our issue was due to running an older version of Redis (apologize for not seeing that earlier). I believe our limiting issues are now resolved (though now Sidekiq randomly times out while connecting to Redis 2.8.6... odd).

Thanks again!

@ckbhodge
ckbhodge commented Mar 5, 2014

@brainopia thanks for the help. there must be a larger issue going on since the .busy number is usually 0 even though i have some long-running tasks, there's plenty in the queue, and the limit is set to 10 right now. i'll revisit this after we get to the bottom of our leaked memory issue...

i checked our redis version in case it was too old per @tyetrask ...we're running redis 2.8.

@brainopia
Owner

@tyetrask you can try to track down the reason why is redis timing out with http://redis.io/commands/slowlog

if sidekiq-limit_fetch is related to the actual reason I would be happy to help in resolving it.

@brainopia
Owner

@ckbhodge can you run puts Sidekiq::Queue['queue'].explain when you sure that a task is running and #busy is zero. Afterwards post the result of the command here so we can investigate further.

@tyetrask

@brainopia - I appreciate the offer! I thought I had written this somewhere in the thread (but it seems what I thought I wrote isn't here). It turns out the timeout was being caused by a line of code I had inserted while initially trying to troubleshoot the issue (before I discovered that outdated Redis was the issue). Everything is working perfectly. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.