-
Notifications
You must be signed in to change notification settings - Fork 68
ThreadsOnFork worker
Backburner now supports (as of a 0.3.0) a ThreadsOnFork worker thanks to ShadowBelmolve.
This worker will create a fork for each tube being watched and then spawn up a configurable number of threads to process jobs for each tube. After a configurable number of processed jobs, a new fork will be started to keep the environment clean and protect against memory bloat.
You can select ThreadsOnFork
as the default worker for processing with:
Backburner.configure do |config|
config.default_worker = Backburner::Workers::ThreadsOnFork
end
or determine the worker on the fly when invoking work:
Backburner.work('newsletter_sender', :worker => Backburner::Workers::ThreadsOnFork)
or through the rake tasks with:
$ QUEUES=newsletter-sender,push-message THREADS=2 GARBAGE=1000 rake backburner:threads_on_fork:work
You can also configure more advanced options to take control of the threads and garbage for each individual tube in the format of QUEUE=TUBE_NAME:THREADS_LIMIT:GARBAGE_LIMIT:RETRIES_LIMIT
. For example:
QUEUE=twitter:10:50:5,parse_page:20:10,send_mail:5,verify_bithday::10 THREADS=2 GARBAGE=1000 rake backburner:threads_on_fork:work
With this:
-
twitter
tube will have 10 threads, garbage after 50 executions and retry jobs 5 times. -
parse_page
will have 20 threads, garbage after 10 executions and retry as the default. -
send_mail
will have 5 threads, garbage after 1k and retry as the default. -
verify_birthday
will have 2 threads, garbage after 1k and retry 10 times.
Also queue name with settings can be specified in job class:
class TwitterJob
include Backburner::Queue
queue 'twitter:10:50:5'
end
or queue settings can be configured at class level like this:
class TwitterJob
include Backburner::Queue
queue 'twitter'
queue_jobs_limit 10
queue_garbage_limit 50
queue_retry_limit 5
end
*Note: class level settings override settings specified in name.