Added --pool option, so that multiple worker pools can be defined with different queues and numbers of workers. #606
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
👍 |
|
@collectiveidea @albus522 Any feedback on this PR? |
| @@ -61,6 +63,9 @@ def initialize(args) | |||
| opts.on('--queue=queue', "Specify which queue DJ must look up for jobs") do |queue| | |||
| @options[:queues] = queue.split(',') | |||
| end | |||
| opts.on('--pools=queue1,queue2:workers/queue3:workers', "Specify queues and number of workers for multiple worker pools") do |pools| | |||
There was a problem hiding this comment.
/ is an odd separator. I think I would prefer multiple --pool flags for multiple worker pools. OptParse will simply run this block for each time --pool is used in the arguments.
So something more like (note singular)
--pool=queue1[,queue2][:worker_count]
worker_count defaulting to 1 if not specified.
…with different queues and number of workers.
ndbroadbent
changed the title
|
Sorry for the delay @albus522 , I've updated the PR with your proposed changes. (The tests are all passing, and the coverage actually increased... I just added new specs for |
albus522
added a commit
that referenced
this pull request
Aug 7, 2014
Added --pool option, so that multiple worker pools can be defined with different queues and numbers of workers.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We needed this so that we could restart multiple workers with different queues with one command via capistrano, and also so that our delayed job pids are consistent with our monit scripts. Running a separate restart command for each worker pool is very time-consuming, since the Rails environment needs to be loaded n times.
As an example, you can now add this to capistrano, to restart 3 different "pools" of workers:
This will start the following workers:
delayed_job.0anddelayed_job.1running any jobsdelayed_job.2running jobs from the mailers and tasks queuesdelayed_job.3anddelayed_job.4running jobs from the important queueSpecs included.