diff --git a/README.md b/README.md index 70defb3..2104247 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ HireFire - The Heroku Worker Manager ==================================== -**HireFire automatically "hires" and "fires" (aka "scales") Delayed Job workers on Heroku**. When there are no queue jobs, HireFire will fire (shut down) all workers. If there are queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired depends on the amount of queued jobs (the ratio can be configured by you). HireFire is great for both high, mid and low traffic applications. It can save you a lot of money by only hiring workers when there are pending jobs, and then firing them again once all the jobs have been processed. It's also capable to dramatically reducing processing time by automatically hiring more workers when the queue size increases. +**HireFire automatically "hires" and "fires" (aka "scales") [Delayed Job](https://github.com/collectiveidea/delayed_job) and [Resque](https://github.com/defunkt/resque) workers on Heroku**. When there are no queue jobs, HireFire will fire (shut down) all workers. If there are queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired depends on the amount of queued jobs (the ratio can be configured by you). HireFire is great for both high, mid and low traffic applications. It can save you a lot of money by only hiring workers when there are pending jobs, and then firing them again once all the jobs have been processed. It's also capable to dramatically reducing processing time by automatically hiring more workers when the queue size increases. **Low traffic example** say we have a small application that doesn't process for more than 2 hours in the background a month. Meanwhile, your worker is basically just idle the rest of the 718 hours in that month. Keeping that idle worker running costs $36/month ($0.05/hour). But, for the resources you're actually **making use of** (2 hours a month), you should be paying $0.10/month, not $36/month. This is what HireFire is for. @@ -25,10 +25,11 @@ A painless process. In a Ruby on Rails environment you would do something like t **Rails.root/Gemfile** gem 'rails' - gem 'delayed_job' + # gem 'delayed_job' # uncomment this line if you use Delayed Job + # gem 'resque' # uncomment this line if you use Resque gem 'hirefire' -**(The order is important: Delayed Job > HireFire)** +**(The order is important: "Delayed Job" / "Resque" > HireFire)** Be sure to add the following Heroku environment variables so HireFire can manage your workers. @@ -41,7 +42,8 @@ And that's it. Next time you deploy to [Heroku](http://heroku.com/) it'll automa **Rails.root/config/initializers/hirefire.rb** HireFire.configure do |config| - config.max_workers = 5 # default is 1 + config.environment = nil # default in production is :heroku. default in development is :noop + config.max_workers = 5 # default is 1 config.job_worker_ratio = [ { :jobs => 1, :workers => 1 }, { :jobs => 15, :workers => 2 }, @@ -61,35 +63,80 @@ Basically what it comes down to is that we say **NEVER** to hire more than 5 wor Once all the jobs in the queue have been processed, it'll fire (shut down) all the workers and start with a single worker the next time a new job gets queued. And then the next time the queue hits 15 jobs mark, in which case the single worker isn't fast enough on it's own, it'll spin up the 2nd worker again. +*If you prefer a more functional way of defining your job/worker ratio, you could use the following notation style:* + + HireFire.configure do |config| + config.max_workers = 5 + config.job_worker_ratio = [ + { :when => lambda {|jobs| jobs < 15 }, :workers => 1 }, + { :when => lambda {|jobs| jobs < 35 }, :workers => 2 }, + { :when => lambda {|jobs| jobs < 60 }, :workers => 3 }, + { :when => lambda {|jobs| jobs < 80 }, :workers => 4 } + ] + end + +The above notation is slightly different, since now you basically define how many workers to hire when `jobs < n`. So for example if there are 80 or more jobs, it'll hire the `max_workers` amount, which is `5` in the above example. If you change the `max_workers = 5` to `max_workers = 10`, then if there are 80 or more jobs queued, it'll go from 4 to 10 workers. + In a non-Ruby on Rails environment ---------------------------------- -Almost the same setup, except that you have to initialize HireFire yourself after Delayed Job is done loading. +Almost the same setup, except that you have to initialize HireFire yourself after Delayed Job or Resque is done loading. require 'delayed_job' - require 'hirefire' + # require 'delayed_job' # uncomment this line if you use Delayed Job + # require 'resque' # uncomment this line if you use Resque HireFire::Initializer.initialize! -**(Again, the order is important: Delayed Job > HireFire)** +**(Again, the order is important: "Delayed Job" / "Resque" > HireFire)** If all goes well you should see a message similar to this when you boot your application: [HireFire] Delayed::Backend::ActiveRecord::Job detected! -Mapper Support +Worker / Mapper Support -------------- -* [ActiveRecord ORM](https://github.com/rails/rails/tree/master/activerecord) -* [Mongoid ODM](https://github.com/mongoid/mongoid) (using [delayed_job_mongoid](https://github.com/collectiveidea/delayed_job_mongoid)) +HireFire currently works with the following worker and mapper libraries: +- [Delayed Job](https://github.com/collectiveidea/delayed_job) + - [ActiveRecord ORM](https://github.com/rails/rails/tree/master/activerecord) + - [Mongoid ODM](https://github.com/mongoid/mongoid) (using [delayed_job_mongoid](https://github.com/collectiveidea/delayed_job_mongoid)) -Worker Support --------------- +- [Resque](https://github.com/defunkt/resque) + - [Redis](https://github.com/ezmobius/redis-rb) + + +Frequently Asked Questions +-------------------------- + +- **Question:** *Does it start workers immediately after a job gets queued?* + - **Answer:** Yes, once a new job gets queued it'll immediately calculate the amount of workers that are required and hire them accordingly. + +- **Question:** *Does it stop workers immediately when there are no jobs to be processed?* + - **Answer:** Yes, every worker has been made self-aware to see this. Once there are no jobs to be processed, all workers will immediately be fired (shut down). *For example, if you have no jobs in the queue, and you start cranking up your Workers via Heroku's web ui, once the worker spawns and sees it has nothing to do, it'll immediately shut itself down.* + +- **Question:** *How does this save me money?* + - **Answer:** According to Heroku's documentation, Workers (same as Dynos), are prorated to the second. *For example, say that 10 jobs get queued and a worker is spawned to process them and takes about 1 minute to do so and then shuts itself down, theoretically you only pay $0.0008.* + +- **Question:** *With Delayed Job you can set the :run_at to a time in the future.* + - **Answer:** Unfortunately since we cannot spawn a monitoring process on the Heroku platform, HireFire will not hire workers until a job gets queued. This means that if you set the :run_at time a few minutes in the future, and these few minutes pass, the job will not be processed until a new job gets queued which triggers the chain of events. (Best to avoid using `run_at` with Delayed Job when using HireFire unless you have a mid-high traffic web application in which cause HireFire gets triggered enough times) + +- **Question:** *If a job is set to run at a time in the future, will workers remain hired to wait for this job to be "processable"?* + - **Answer:** No, because if you enqueue a job to run 3 hours from the time it was enqueued, you might have workers doing nothing the coming 3 hours. Best to avoid scheduling jobs to be processed in the future. + +- **Question:** *Will it scale down workers from, for example, 5 to 4?* + - **Answer:** No, I have consciously chosen not to do that for 2 reasons: + 1. There is no way to tell which worker is currently processing a job, so it might fire a worker that was busy, causing the job to be exit during the process. + 2. Does it really matter? Your jobs will be processed faster, and once the queue is completely empty, all workers will be fire anyway. (You could call this a feature! Since 5 jobs process faster than 4, but the cost remains the same cause it's all pro-rated to the second) + +- **Question:** *Will running jobs concurrently (with multiple Worker) cost more?* + - **Answer:** Actually, no. Since worker's are pro-rated to the second, the moment you hire 3 workers, it costs 3 times more, but it also processes 3 times faster. You could also let 1 worker process all the jobs rather than 3, but that means it'll still cost the same amount as when you hire 3 workers, since it takes 3 times longer to process. + +- **Question:** *Can I process jobs faster with HireFire?* + - **Answer:** When you run multiple jobs concurrently, you can speed up your processing dramatically. *Normally you wouldn't set the workers to 10 for example, but with HireFire you can tell it to Hire 10 workers when there are 50 jobs (would normally be overkill and cost you A LOT of money) but since (see Q/A above) Workers are pro-rated to the second, and HireFire immediately fires all workers once all the jobs in the queue have been processed, it makes no different whether you have a single worker processing 50 jobs, or 5 workers, or even 10 workers. It processes 10 times faster, but costs the same.* -Currently only [Delayed Job](https://github.com/collectiveidea/delayed_job) with either [ActiveRecord ORM](https://github.com/rails/rails/tree/master/activerecord) and [Mongoid ODM](https://github.com/mongoid/mongoid). -Might have plans to implement this for other workers in the future. Other potentially interesting gems diff --git a/hirefire.gemspec b/hirefire.gemspec index 82e1773..5abe265 100644 --- a/hirefire.gemspec +++ b/hirefire.gemspec @@ -12,17 +12,8 @@ Gem::Specification.new do |gem| gem.authors = 'Michael van Rooijen' gem.email = 'meskyanichi@gmail.com' gem.homepage = 'http://rubygems.org/gems/hirefire' - gem.summary = 'HireFire automatically "hires" and "fires" (aka "scales") Delayed Job workers on Heroku.' - gem.description = <<-EOS - HireFire automatically "hires" and "fires" (aka "scales") Delayed Job workers on Heroku. - When there are no queue jobs, HireFire will fire (shut down) all workers. If there are - queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired - depends on the amount of queued jobs (the ratio can be configured by you). HireFire - is great for both high, mid and low traffic applications. It can save you a lot of - money by only hiring workers when there are pending jobs, and then firing them again - once all the jobs have been processed. It's also capable to dramatically reducing - processing time by automatically hiring more workers when the queue size increases. - EOS + gem.summary = %|HireFire automatically "hires" and "fires" (aka "scales") Delayed Job and Resque workers on Heroku.| + gem.description = %|HireFire automatically "hires" and "fires" (aka "scales") Delayed Job and Resque workers on Heroku. When there are no queue jobs, HireFire will fire (shut down) all workers. If there are queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired depends on the amount of queued jobs (the ratio can be configured by you). HireFire is great for both high, mid and low traffic applications. It can save you a lot of money by only hiring workers when there are pending jobs, and then firing them again once all the jobs have been processed. It's also capable to dramatically reducing processing time by automatically hiring more workers when the queue size increases.| ## # Files and folder that need to be compiled in to the Ruby Gem diff --git a/lib/hirefire.rb b/lib/hirefire.rb index feda7e9..39cab4e 100644 --- a/lib/hirefire.rb +++ b/lib/hirefire.rb @@ -5,18 +5,19 @@ module HireFire ## # HireFire constants LIB_PATH = File.dirname(__FILE__) - FREELANCER_PATH = File.join(LIB_PATH, 'hirefire') - ENVIRONMENT_PATH = File.join(FREELANCER_PATH, 'environment') - BACKEND_PATH = File.join(FREELANCER_PATH, 'backend') + HIREFIRE_PATH = File.join(LIB_PATH, 'hirefire') + ENVIRONMENT_PATH = File.join(HIREFIRE_PATH, 'environment') + BACKEND_PATH = File.join(HIREFIRE_PATH, 'backend') + WORKERS_PATH = File.join(HIREFIRE_PATH, 'workers') ## # HireFire namespace - autoload :Configuration, File.join(FREELANCER_PATH, 'configuration') - autoload :Environment, File.join(FREELANCER_PATH, 'environment') - autoload :Initializer, File.join(FREELANCER_PATH, 'initializer') - autoload :Backend, File.join(FREELANCER_PATH, 'backend') - autoload :Logger, File.join(FREELANCER_PATH, 'logger') - autoload :Version, File.join(FREELANCER_PATH, 'version') + autoload :Configuration, File.join(HIREFIRE_PATH, 'configuration') + autoload :Environment, File.join(HIREFIRE_PATH, 'environment') + autoload :Initializer, File.join(HIREFIRE_PATH, 'initializer') + autoload :Backend, File.join(HIREFIRE_PATH, 'backend') + autoload :Logger, File.join(HIREFIRE_PATH, 'logger') + autoload :Version, File.join(HIREFIRE_PATH, 'version') ## # HireFire::Environment namespace @@ -27,11 +28,31 @@ module Environment autoload :Noop, File.join(ENVIRONMENT_PATH, 'noop') end + ## + # HireFire::Workers namespace + module Workers + autoload :DelayedJob, File.join(WORKERS_PATH, 'delayed_job') + autoload :Resque, File.join(WORKERS_PATH, 'resque') + end + ## # HireFire::Backend namespace module Backend - autoload :ActiveRecord, File.join(BACKEND_PATH, 'active_record') - autoload :Mongoid, File.join(BACKEND_PATH, 'mongoid') + DELAYED_JOB_PATH = File.join(BACKEND_PATH, 'delayed_job') + RESQUE_PATH = File.join(BACKEND_PATH, 'resque') + + ## + # HireFire::Backend::DelayedJob namespace + module DelayedJob + autoload :ActiveRecord, File.join(DELAYED_JOB_PATH, 'active_record') + autoload :Mongoid, File.join(DELAYED_JOB_PATH, 'mongoid') + end + + ## + # HireFire::Backend::Resque namespace + module Resque + autoload :Redis, File.join(RESQUE_PATH, 'redis') + end end ## @@ -79,7 +100,12 @@ def self.configuration # so that the developer doesn't have to manually invoke it from an initializer file # # Users not using Ruby on Rails will have to run "HireFire::Initializer.initialize!" -# in their application manually, after loading Delayed Job and the desired mapper (ActiveRecord or Mongoid) +# in their application manually, after loading the worker library (either "Delayed Job" or "Resque") +# and the desired mapper (ActiveRecord, Mongoid or Redis) if defined?(Rails) - require File.join(HireFire::FREELANCER_PATH, 'railtie') + if Rails.version >= '3.0.0' + require File.join(HireFire::HIREFIRE_PATH, 'railtie') + else + HireFire::Initializer.initialize! + end end diff --git a/lib/hirefire/backend.rb b/lib/hirefire/backend.rb index 3fc1332..859b725 100644 --- a/lib/hirefire/backend.rb +++ b/lib/hirefire/backend.rb @@ -4,17 +4,32 @@ module HireFire module Backend ## - # Load the correct module (ActiveRecord or Mongoid) - # based on which Delayed::Backend has been loaded + # Load the correct module (ActiveRecord, Mongoid or Redis) + # based on which worker and backends are loaded + # + # Currently supports: + # - Delayed Job with ActiveRecord and Mongoid + # - Resque with Redis # # @return [nil] def self.included(base) - if defined?(Delayed::Backend::ActiveRecord::Job) - base.send(:include, ActiveRecord) + + ## + # Delayed Job specific backends + if defined?(::Delayed::Job) + if defined?(::Delayed::Backend::ActiveRecord::Job) + base.send(:include, HireFire::Backend::DelayedJob::ActiveRecord) + end + + if defined?(::Delayed::Backend::Mongoid::Job) + base.send(:include, HireFire::Backend::DelayedJob::Mongoid) + end end - if defined?(Delayed::Backend::Mongoid::Job) - base.send(:include, Mongoid) + ## + # Resque specific backends + if defined?(::Resque) + base.send(:include, HireFire::Backend::Resque::Redis) end end diff --git a/lib/hirefire/backend/active_record.rb b/lib/hirefire/backend/active_record.rb deleted file mode 100644 index 5d05dbb..0000000 --- a/lib/hirefire/backend/active_record.rb +++ /dev/null @@ -1,20 +0,0 @@ -# encoding: utf-8 - -module HireFire - module Backend - module ActiveRecord - - ## - # Counts the amount of queued jobs in the database, - # failed jobs are excluded from the sum - # - # @return [Fixnum] - def jobs - Delayed::Job. - where(:failed_at => nil). - where('run_at <= ?', Time.now).count - end - - end - end -end diff --git a/lib/hirefire/backend/delayed_job/active_record.rb b/lib/hirefire/backend/delayed_job/active_record.rb new file mode 100644 index 0000000..8604b5f --- /dev/null +++ b/lib/hirefire/backend/delayed_job/active_record.rb @@ -0,0 +1,31 @@ +# encoding: utf-8 + +module HireFire + module Backend + module DelayedJob + module ActiveRecord + + ## + # Counts the amount of queued jobs in the database, + # failed jobs are excluded from the sum + # + # @return [Fixnum] the amount of pending jobs + def jobs + ::Delayed::Job. + where(:failed_at => nil). + where('run_at <= ?', Time.now).count + end + + ## + # Counts the amount of jobs that are locked by a worker + # + # @return [Fixnum] the amount of (assumably working) workers + def workers + ::Delayed::Job. + where('locked_by IS NOT NULL').count + end + + end + end + end +end diff --git a/lib/hirefire/backend/delayed_job/mongoid.rb b/lib/hirefire/backend/delayed_job/mongoid.rb new file mode 100644 index 0000000..4bdaf2d --- /dev/null +++ b/lib/hirefire/backend/delayed_job/mongoid.rb @@ -0,0 +1,32 @@ +# encoding: utf-8 + +module HireFire + module Backend + module DelayedJob + module Mongoid + + ## + # Counts the amount of queued jobs in the database, + # failed jobs and jobs scheduled for the future are excluded + # + # @return [Fixnum] + def jobs + ::Delayed::Job.where( + :failed_at => nil, + :run_at.lte => Time.now + ).count + end + + ## + # Counts the amount of jobs that are locked by a worker + # + # @return [Fixnum] the amount of (assumably working) workers + def workers + ::Delayed::Job. + where(:locked_by.ne => nil).count + end + + end + end + end +end diff --git a/lib/hirefire/backend/mongoid.rb b/lib/hirefire/backend/mongoid.rb deleted file mode 100644 index 41a1999..0000000 --- a/lib/hirefire/backend/mongoid.rb +++ /dev/null @@ -1,21 +0,0 @@ -# encoding: utf-8 - -module HireFire - module Backend - module Mongoid - - ## - # Counts the amount of queued jobs in the database, - # failed jobs and jobs scheduled for the future are excluded - # - # @return [Fixnum] - def jobs - Delayed::Job.where( - :failed_at => nil, - :run_at.lte => Time.now - ).count - end - - end - end -end diff --git a/lib/hirefire/backend/resque/redis.rb b/lib/hirefire/backend/resque/redis.rb new file mode 100644 index 0000000..f00a935 --- /dev/null +++ b/lib/hirefire/backend/resque/redis.rb @@ -0,0 +1,20 @@ +# encoding: utf-8 + +module HireFire + module Backend + module Resque + module Redis + + ## + # Counts the amount of queued jobs in the database, + # failed jobs and jobs scheduled for the future are excluded + # + # @return [Fixnum] + def jobs + ::Resque.info[:pending].to_i + ::Resque.info[:working].to_i + end + + end + end + end +end diff --git a/lib/hirefire/environment.rb b/lib/hirefire/environment.rb index f1caf0f..83e4674 100644 --- a/lib/hirefire/environment.rb +++ b/lib/hirefire/environment.rb @@ -4,32 +4,56 @@ module HireFire module Environment ## - # This gets included in to the Delayed::Backend::(ActiveRecord|Mongoid)::Job - # classes and will add the necessary hooks (after_create, after_destroy and after_update) - # to spawn or kill Delayed Job worker processes on either Heroku or your local machine + # This module gets included in either: + # - Delayed::Backend::ActiveRecord::Job + # - Delayed::Backend::Mongoid::Job + # - Resque::Job + # + # One of these classes will then be provided with an instance of one of the following: + # - HireFire::Environment::Heroku + # - HireFire::Environment::Local + # - HireFire::Environment::Noop + # + # This instance is stored in the Class.environment class method + # + # The Delayed Job classes receive 3 hooks: + # - hirefire_hire ( invoked when a job gets queued ) + # - environment.fire ( invoked when a queued job gets destroyed ) + # - environment.fire ( invoked when a queued job gets updated unless the job didn't fail ) + # + # The Resque classes get their hooks injected from the HireFire::Initializer#initialize! method # # @param (Class) base This is the class in which this module will be included def self.included(base) - base.send :extend, ClassMethods - base.class_eval do - after_create 'self.class.environment.hire' - after_destroy 'self.class.environment.fire' - after_update 'self.class.environment.fire', - :unless => Proc.new { |job| job.failed_at.nil? } + base.send :extend, HireFire::Environment::ClassMethods + + ## + # Only implement these hooks for Delayed::Job backends + if base.name =~ /Delayed::Backend::(ActiveRecord|Mongoid)::Job/ + base.send :extend, HireFire::Environment::DelayedJob::ClassMethods + + base.class_eval do + after_create 'self.class.hirefire_hire' + after_destroy 'self.class.environment.fire' + after_update 'self.class.environment.fire', + :unless => Proc.new { |job| job.failed_at.nil? } + end end Logger.message("#{ base.name } detected!") end ## - # Class methods that will be added to the Delayed::Job backend + # Class methods that will be added to the + # Delayed::Job and Resque::Job classes module ClassMethods ## - # Returns the environment class method (for Delayed::Job ORM/ODM class) + # Returns the environment class method (containing an instance of the proper environment class) + # for either Delayed::Job or Resque::Job # # If HireFire.configuration.environment is nil (the default) then it'll - # auto-detect which environment to run in (either Heroku or Local) + # auto-detect which environment to run in (either Heroku or Noop) # # If HireFire.configuration.environment isn't nil (explicitly set) then # it'll run in the specified environment (Heroku, Local or Noop) @@ -46,5 +70,34 @@ def environment end end + ## + # Delayed Job specific module + module DelayedJob + module ClassMethods + + ## + # This method is an attempt to improve web-request throughput. + # + # A class method for Delayed::Job which first checks if any worker is currently + # running by checking to see if there are any jobs locked by a worker. If there aren't + # any jobs locked by a worker there is a high chance that there aren't any workers running. + # If this is the case, then we sure also invoke the 'self.class.environment.hire' method + # + # Another check is to see if there is only 1 job (which is the one that + # was just added before this callback invoked). If this is the case + # then it's very likely there aren't any workers running and we should + # invoke the 'self.class.environment.hire' method to make sure this is the case. + # + # @return [nil] + def hirefire_hire + delayed_job = ::Delayed::Job.new + if delayed_job.workers == 0 \ + or delayed_job.jobs == 1 + environment.hire + end + end + end + end + end end diff --git a/lib/hirefire/environment/base.rb b/lib/hirefire/environment/base.rb index a656307..f28cbe9 100644 --- a/lib/hirefire/environment/base.rb +++ b/lib/hirefire/environment/base.rb @@ -31,8 +31,20 @@ class Base # # It'll match at { :jobs => 35, :workers => 3 }, (35 jobs or more: hire 3 workers) # # meaning that it'll ensure there are 3 workers running. # - # # If there were already were 3 workers, it'll leave it as is - # + # # If there were already were 3 workers, it'll leave it as is. + # + # # Alternatively, you can use a more functional syntax, which works in the same way. + # + # HireFire.configure do |config| + # config.max_workers = 5 + # config.job_worker_ratio = [ + # { :when => lambda {|jobs| jobs < 15 }, :workers => 1 }, + # { :when => lambda {|jobs| jobs < 35 }, :workers => 2 }, + # { :when => lambda {|jobs| jobs < 60 }, :workers => 3 }, + # { :when => lambda {|jobs| jobs < 80 }, :workers => 4 } + # ] + # end + # # # If there were more than 3 workers running (say, 4 or 5), it will NOT reduce # # the number. This is because when you reduce the number of workers, you cannot # # tell which worker Heroku will shut down, meaning you might interrupt a worker @@ -50,16 +62,97 @@ def hire jobs_count = jobs workers_count = workers - ratio.each do |ratio| - if jobs_count >= ratio[:jobs] and max_workers >= ratio[:workers] - if not workers_count == ratio[:workers] - Logger.message("Hiring more workers so we have #{ ratio[:workers] } in total.") - workers(ratio[:workers]) + ## + # Use "Standard Notation" + if not ratio.first[:when].respond_to? :call + + ## + # Since the "Standard Notation" is defined in the in an ascending order + # in the array of hashes, we need to reverse this order in order to properly + # loop through and break out of the array at the correctly matched ratio + ratio.reverse! + + ## + # Iterates through all the defined job/worker ratio's + # until it finds a match. Then it hires (if necessary) the appropriate + # amount of workers and breaks out of the loop + ratio.each do |ratio| + + ## + # Standard notation + # This is the code for the default notation + # + # @example + # { :jobs => 35, :workers => 3 } + # + if jobs_count >= ratio[:jobs] and max_workers >= ratio[:workers] + if workers_count < ratio[:workers] + log_and_hire(ratio[:workers]) + end + + return end + end + + ## + # If no match is found in the above job/worker ratio loop, then we'll + # perform one last operation to see whether the the job count is greater + # than the highest job/worker ratio, and if this is the case then we also + # check to see whether the maximum amount of allowed workers is greater + # than the amount that are currently running, if this is the case, we are + # allowed to hire the max amount of workers. + if jobs_count >= ratio.first[:jobs] and max_workers > workers_count + log_and_hire(max_workers) + return + end + end + + ## + # Use "Functional (Lambda) Notation" + if ratio.first[:when].respond_to? :call + + ## + # Iterates through all the defined job/worker ratio's + # until it finds a match. Then it hires (if necessary) the appropriate + # amount of workers and breaks out of the loop + ratio.each do |ratio| - break + ## + # Functional (Lambda) Notation + # This is the code for the Lambda notation, more verbose, + # but more humanly understandable + # + # @example + # { :when => lambda {|jobs| jobs < 60 }, :workers => 3 } + # + if ratio[:when].call(jobs_count) and max_workers >= ratio[:workers] + if workers_count < ratio[:workers] + log_and_hire(ratio[:workers]) + end + + break + end end end + + ## + # Applies only to the Functional (Lambda) Notation + # If the amount of queued jobs exceeds that of which was defined in the + # job/worker ratio array, it will hire the maximum amount of workers + # + # "if statements": + # 1. true if we use the Functional (Lambda) Notation + # 2. true if the last ratio (highest job/worker ratio) was exceeded + # 3. true if the max amount of workers are not yet running + # + # If all the the above statements are true, HireFire will hire the maximum + # amount of workers that were specified in the configuration + # + if ratio.last[:when].respond_to? :call \ + and ratio.last[:when].call(jobs_count) === false \ + and max_workers != workers_count + log_and_hire(max_workers) + end end ## @@ -79,6 +172,15 @@ def fire private + ## + # Helper method for hire that logs the hiring of more workers, then hires those workers. + # + # @return [nil] + def log_and_hire(amount) + Logger.message("Hiring more workers so we have #{ amount } in total.") + workers(amount) + end + ## # Wrapper method for HireFire.configuration # Returns the max amount of workers that may run concurrently @@ -92,9 +194,9 @@ def max_workers # Wrapper method for HireFire.configuration # Returns the job/worker ratio array (in reversed order) # - # @return [Array] the array of hashes containing the job/worker ratio (in reversed order) + # @return [Array] the array of hashes containing the job/worker ratio def ratio - HireFire.configuration.job_worker_ratio.reverse + HireFire.configuration.job_worker_ratio end end diff --git a/lib/hirefire/initializer.rb b/lib/hirefire/initializer.rb index cdcd301..33bec99 100644 --- a/lib/hirefire/initializer.rb +++ b/lib/hirefire/initializer.rb @@ -4,33 +4,69 @@ module HireFire class Initializer ## - # Loads the HireFire extension in to Delayed Job and - # extends the Delayed Job "jobs:work" rake task command + # Loads the HireFire extension in to the loaded worker library and + # extends that library by injecting HireFire hooks in the proper locations. + # + # Currently it supports: + # - Delayed Job + # - ActiveRecord ORM + # - Mongoid ODM + # - Resque + # - Redis + # + # @note + # Either the Delayed Job, or the Resque worker library must be + # loaded BEFORE HireFire initializes, otherwise it'll be unable + # to detect the proper library and it will not work. # # @return [nil] def self.initialize! + ## - # If DelayedJob is using ActiveRecord, then include - # HireFire::Environment in to the ActiveRecord Delayed Job Backend - if defined?(Delayed::Backend::ActiveRecord::Job) - Delayed::Backend::ActiveRecord::Job. - send(:include, HireFire::Environment). - send(:include, HireFire::Backend) + # Initialize Delayed::Job extensions if Delayed::Job is found + if defined?(::Delayed::Job) + ## + # If DelayedJob is using ActiveRecord, then include + # HireFire::Environment in to the ActiveRecord Delayed Job Backend + if defined?(::Delayed::Backend::ActiveRecord::Job) + ::Delayed::Backend::ActiveRecord::Job. + send(:include, HireFire::Environment). + send(:include, HireFire::Backend) + end + + ## + # If DelayedJob is using Mongoid, then include + # HireFire::Environment in to the Mongoid Delayed Job Backend + if defined?(::Delayed::Backend::Mongoid::Job) + ::Delayed::Backend::Mongoid::Job. + send(:include, HireFire::Environment). + send(:include, HireFire::Backend) + end + + ## + # Load Delayed Job extensions, this will patch Delayed::Worker + # to implement the necessary hooks to invoke HireFire from + require File.join(HireFire::WORKERS_PATH, 'delayed_job') end ## - # If DelayedJob is using Mongoid, then include - # HireFire::Environment in to the Mongoid Delayed Job Backend - if defined?(Delayed::Backend::Mongoid::Job) - Delayed::Backend::Mongoid::Job. + # Initialize Resque extensions if Resque is found + if defined?(::Resque) + + ## + # Include the HireFire::Environment which will add an instance + # of HireFire::Environment::(Heroku|Local|Noop) to the Resque::Job.environment class method + # + # Extend the Resque::Job class with the Resque::Job.jobs class method + ::Resque::Job. send(:include, HireFire::Environment). - send(:include, HireFire::Backend) - end + send(:extend, HireFire::Backend::Resque::Redis) - ## - # Load Delayed Job extension, this is the start - # method that gets invoked when running "rake jobs:work" - require File.dirname(__FILE__) + '/delayed_job_extension' + ## + # Load Resque extensions, this will patch Resque, Resque::Job and Resque::Worker + # to implement the necessary hooks to invoke HireFire from + require File.join(HireFire::WORKERS_PATH, 'resque') + end end end diff --git a/lib/hirefire/railtie.rb b/lib/hirefire/railtie.rb index 819a352..2a68c28 100644 --- a/lib/hirefire/railtie.rb +++ b/lib/hirefire/railtie.rb @@ -4,11 +4,37 @@ module HireFire class Railtie < Rails::Railtie ## - # Initializes HireFire for Delayed Job when + # Initializes HireFire for either Delayed Job or Resque when # the Ruby on Rails web framework is done loading + # + # @note + # Either the Delayed Job, or the Resque worker library must be + # loaded BEFORE HireFire initializes, otherwise it'll be unable + # to detect the proper library and it will not work. initializer :after_initialize do HireFire::Initializer.initialize! end + ## + # Adds additional rake tasks to the Ruby on Rails environment + # + # @note + # In order for Resque to run on Heroku, it must have the 'rake jobs:work' + # rake task since that's what Heroku uses to start workers. When using + # Ruby on Rails automatically add the necessary default rake task for the user + # + # @note + # Delayed Job already has 'rake jobs:work' built in. + # + rake_tasks do + + ## + # If Resque is loaded, then we load the Resque rake task + # that'll allow Heroku to start up Resque as a worker + if defined?(::Resque) + require File.join(WORKERS_PATH, 'resque', 'tasks.rb') + end + end + end end diff --git a/lib/hirefire/version.rb b/lib/hirefire/version.rb index b93e84a..9b3565e 100644 --- a/lib/hirefire/version.rb +++ b/lib/hirefire/version.rb @@ -6,7 +6,7 @@ module Version ## # @return [String] the current version of the HireFire gem def self.current - '0.1.0' + '0.1.1' end end diff --git a/lib/hirefire/workers/delayed_job.rb b/lib/hirefire/workers/delayed_job.rb new file mode 100644 index 0000000..7b03214 --- /dev/null +++ b/lib/hirefire/workers/delayed_job.rb @@ -0,0 +1,5 @@ +# encoding: utf-8 + +## +# Load the "HireFire modified" Delayed::Worker class +require File.dirname(__FILE__) + '/delayed_job/worker' diff --git a/lib/hirefire/delayed_job_extension.rb b/lib/hirefire/workers/delayed_job/worker.rb similarity index 74% rename from lib/hirefire/delayed_job_extension.rb rename to lib/hirefire/workers/delayed_job/worker.rb index aa4fee0..4d9f3d2 100644 --- a/lib/hirefire/delayed_job_extension.rb +++ b/lib/hirefire/workers/delayed_job/worker.rb @@ -1,5 +1,9 @@ # encoding: utf-8 +## +# HireFire +# This is a HireFire modified version of +# the official Delayed::Worker class module Delayed class Worker @@ -42,10 +46,14 @@ def start end ## - # If there are no jobs currently queued, - # and the worker is still running, it'll kill itself + # HireFire Hook + # After the last job in the queue finishes processing, Delayed::Job.new.jobs (queued.jobs) + # will return 0. This means that there aren't any more jobs to process for any of the workers. + # If this is the case it'll command the current environment to fire all the hired workers + # and then immediately break out of this infinite loop. if queued.jobs == 0 Delayed::Job.environment.fire + break end break if $exit diff --git a/lib/hirefire/workers/resque.rb b/lib/hirefire/workers/resque.rb new file mode 100644 index 0000000..9b94b45 --- /dev/null +++ b/lib/hirefire/workers/resque.rb @@ -0,0 +1,31 @@ +# encoding: utf-8 + +## +# Load the "HireFire modified" +# Resque::Job and Resque::Worker classes +require File.dirname(__FILE__) + '/resque/job' +require File.dirname(__FILE__) + '/resque/worker' + +## +# HireFire +# This is a HireFire modified version of +# the official Resque module +module ::Resque + def self.enqueue(klass, *args) + Job.create(queue_from_class(klass), klass, *args) + + ## + # HireFire Hook + # After a new job gets queued, we command the current environment + # to calculate the amount of workers we need to process the jobs + # that are currently queued, and hire them accordingly. + if ::Resque.info[:working].to_i == 0 \ + or ::Resque.info[:jobs] == 1 + ::Resque::Job.environment.hire + end + + Plugin.after_enqueue_hooks(klass).each do |hook| + klass.send(hook, *args) + end + end +end diff --git a/lib/hirefire/workers/resque/job.rb b/lib/hirefire/workers/resque/job.rb new file mode 100644 index 0000000..bf57bf5 --- /dev/null +++ b/lib/hirefire/workers/resque/job.rb @@ -0,0 +1,70 @@ +# encoding: utf-8 + +## +# HireFire +# This is a HireFire modified version of +# the official Resque::Job class +module ::Resque + class Job + def perform + job = payload_class + job_args = args || [] + job_was_performed = false + + before_hooks = Plugin.before_hooks(job) + around_hooks = Plugin.around_hooks(job) + after_hooks = Plugin.after_hooks(job) + failure_hooks = Plugin.failure_hooks(job) + + begin + begin + before_hooks.each do |hook| + job.send(hook, *job_args) + end + rescue DontPerform + return false + end + + if around_hooks.empty? + job.perform(*job_args) + job_was_performed = true + else + stack = around_hooks.reverse.inject(nil) do |last_hook, hook| + if last_hook + lambda do + job.send(hook, *job_args) { last_hook.call } + end + else + lambda do + job.send(hook, *job_args) do + result = job.perform(*job_args) + job_was_performed = true + result + end + end + end + end + stack.call + end + + ## + # HireFire Hook + # After a job finishes processing, we invoke the #fire + # method on the environment object which will check to see whether + # we can fire all the hired workers + ::Resque::Job.environment.fire + + after_hooks.each do |hook| + job.send(hook, *job_args) + end + + return job_was_performed + + rescue Object => e + failure_hooks.each { |hook| job.send(hook, e, *job_args) } + raise e + end + end + + end +end diff --git a/lib/hirefire/workers/resque/tasks.rb b/lib/hirefire/workers/resque/tasks.rb new file mode 100644 index 0000000..d2d6814 --- /dev/null +++ b/lib/hirefire/workers/resque/tasks.rb @@ -0,0 +1,21 @@ +# encoding: utf-8 + +## +# Load in the official Resque rake tasks +require 'resque/tasks' + +## +# Overwrite the resque:setup rake task to first load +# in the application environment before proceeding +# +# ENV['QUEUE'] will default to '*' unless it's defined +# as an environment variable on Heroku or the Local machine +task 'resque:setup' => :environment do + ENV['QUEUE'] ||= '*' +end + +## +# This is an alias to the "resque:work" task since Heroku doesn't respond +# to "resque:work", we need to add this alias so Resque can be initialized by Heroku +desc 'Alias of "resque:work" - This is required for running on Heroku.' +task 'jobs:work' => 'resque:work' diff --git a/lib/hirefire/workers/resque/worker.rb b/lib/hirefire/workers/resque/worker.rb new file mode 100644 index 0000000..cac4d3f --- /dev/null +++ b/lib/hirefire/workers/resque/worker.rb @@ -0,0 +1,57 @@ +# encoding: utf-8 + +## +# HireFire +# This is a HireFire modified version of +# the official Resque::Worker class +module ::Resque + class Worker + def work(interval = 5.0, &block) + interval = Float(interval) + $0 = "resque: Starting" + startup + + loop do + break if shutdown? + ::Resque::Job.environment.hire + + if not @paused and job = reserve + log "got: #{job.inspect}" + run_hook :before_fork, job + working_on job + + if @child = fork + rand # Reseeding + procline "Forked #{@child} at #{Time.now.to_i}" + Process.wait + else + procline "Processing #{job.queue} since #{Time.now.to_i}" + perform(job, &block) + exit! unless @cant_fork + end + + done_working + @child = nil + else + + ## + # HireFire Hook + # After the last job in the queue finishes processing, Resque::Job.jobs will return 0. + # This means that there aren't any more jobs to process for any of the workers. + # If this is the case it'll command the current environment to fire all the hired workers + # and then immediately break out of this infinite loop. + if ::Resque::Job.jobs == 0 + ::Resque::Job.environment.fire + break + else + sleep(interval) + end + + end + end + + ensure + unregister_worker + end + end +end diff --git a/spec/configuration_spec.rb b/spec/configuration_spec.rb index 2cf3ffe..6afa0b7 100644 --- a/spec/configuration_spec.rb +++ b/spec/configuration_spec.rb @@ -44,4 +44,24 @@ ] end + it 'should allow functional syntax' do + ratio = [ + { :when => lambda {|jobs| jobs < 15 }, :workers => 1 }, + { :when => lambda {|jobs| jobs < 35 }, :workers => 2 }, + { :when => lambda {|jobs| jobs < 60 }, :workers => 3 }, + { :when => lambda {|jobs| jobs < 80 }, :workers => 4 } + ] + + HireFire.configure do |config| + config.environment = :noop + config.max_workers = 10 + config.job_worker_ratio = ratio + end + + configuration = HireFire.configuration + + configuration.environment.should == :noop + configuration.max_workers.should == 10 + configuration.job_worker_ratio.should == ratio + end end diff --git a/spec/environment_spec.rb b/spec/environment_spec.rb new file mode 100644 index 0000000..0b0ba5e --- /dev/null +++ b/spec/environment_spec.rb @@ -0,0 +1,359 @@ +# encoding: utf-8 + +require File.expand_path('../spec_helper', __FILE__) + +module HireFire + module Backend + ## + # Stub out backend interface inclusion + # since it's irrelevant for these tests + def self.included(base) + base.send(:include, Environment::Stub) + end + end + + module Environment + module Stub + ## + # Stubbed out since this normally comes from + # a sub class like HireFire::Environment::Heroku or + # HireFire::Environment::Local + def workers(amount = nil) + if amount.nil? + @_workers ||= 0 + return @_workers + end + end + + ## + # Allows the specs to stub the workers count + # and return the desired amount + def workers=(amount) + @_workers = amount + self.stubs(:workers).with.returns(amount) + end + + ## + # Returns the amount of jobs + # Defaults to: 0 + def jobs + @_jobs ||= 0 + end + + ## + # Allows the specs to stub the queued job count + # and return the desired amount + def jobs=(amount) + @_jobs = amount + self.stubs(:jobs).returns(amount) + end + end + end +end + +describe HireFire::Environment::Base do + + let(:base) { HireFire::Environment::Base.new } + + describe 'testing the test setup' do + it 'should default the queued job count to 0 for these specs' do + base.jobs.should == 0 + end + + it 'should set the queued job count to 10' do + base.jobs = 10 + base.jobs.should == 10 + end + + it 'should have zero workers by default' do + base.workers.should == 0 + end + + it 'should set the amount of workers to 10' do + base.workers = 10 + base.workers.should == 10 + end + end + + describe '#fire' do + it 'should not fire any workers when there are still jobs in the queue' do + base.jobs = 1 + base.workers = 1 + + base.expects(:workers).with(0).never + base.fire + end + + it 'should not fire any workers if there arent any workers' do + base.jobs = 1 + base.workers = 0 + + base.expects(:workers).with(0).never + base.fire + end + + it 'should fire all workers when there arent any jobs' do + base.jobs = 0 + base.workers = 1 + + HireFire::Logger.expects(:message).with('All queued jobs have been processed. Firing all workers.') + base.expects(:workers).with(0).once + base.fire + end + end + + describe '#hire' do + describe 'the standard notation' do + before do + base.stubs(:max_workers).returns(5) + base.stubs(:ratio).returns([ + { :jobs => 1, :workers => 1 }, + { :jobs => 15, :workers => 2 }, + { :jobs => 30, :workers => 3 }, + { :jobs => 60, :workers => 4 }, + { :jobs => 90, :workers => 5 } + ]) + end + + it 'should request 1 worker' do + base.jobs = 1 + base.workers = 0 + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 1 in total.') + base.expects(:workers).with(1).once + base.hire + end + + it 'should not request 1 worker, since there already is one worker running' do + base.jobs = 5 + base.workers = 1 + + base.expects(:workers).with(1).never + base.hire + end + + it 'should request 2 workers' do + base.jobs = 15 + base.workers = 0 + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 2 in total.') + base.expects(:workers).with(2).once + base.hire + end + + it 'should request 2 workers' do + base.jobs = 20 + base.workers = 1 + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 2 in total.') + base.expects(:workers).with(2).once + base.hire + end + + it 'should not request 2 workers since we already have 2' do + base.jobs = 25 + base.workers = 2 + + base.expects(:workers).with(2).never + base.hire + end + + it 'should NEVER lower the worker amount from the #hire method' do + base.jobs = 25 # simulate that 5 jobs are already processed (30 - 5) + base.workers = 3 # and 3 workers are hired + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 2 in total.').never + base.expects(:workers).with(2).never + base.hire + end + + it 'should NEVER hire more workers than the #max_workers' do + base.jobs = 100 + base.workers = 0 + + base.stubs(:max_workers).returns(3) # set the max_workers = 3 + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 3 in total.').once + base.expects(:workers).with(3).once + base.hire + end + + it 'should not hire 5 workers even if defined in the job/ratio, when the limit is 3, it should hire 3 max' do + base.jobs = 100 + base.workers = 0 + + base.stubs(:max_workers).returns(3) # set the max_workers = 3 + base.stubs(:ratio).returns([ + { :jobs => 5, :workers => 5 } + ]) + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 3 in total.').once + base.expects(:workers).with(3).once + base.hire + end + + it 'should not hire (or invoke) any more workers since the max amount allowed is already running' do + base.jobs = 100 + base.workers = 3 + + base.stubs(:max_workers).returns(3) # set the max_workers = 3 + base.stubs(:ratio).returns([ + { :jobs => 5, :workers => 5 } + ]) + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 3 in total.').never + base.expects(:workers).with(3).never + base.hire + end + + it 'the max_workers option can only "limit" the amount of max_workers when used in the "Standard Notation"' do + base.jobs = 100 + base.workers = 0 + + base.stubs(:max_workers).returns(10) # set the max_workers = 10 + base.stubs(:ratio).returns([ + { :jobs => 5, :workers => 5 } + ]) + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 5 in total.').once + base.expects(:workers).with(5).once + base.hire + end + + it 'should NEVER do API requests to Heroku if the max_workers are already running' do + base.jobs = 100 + base.workers = 5 + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 5 in total.').never + base.expects(:workers).with(5).never + base.hire + end + end + + describe 'the Lambda (functional) notation' do + before do + base.stubs(:max_workers).returns(5) + base.stubs(:ratio).returns([ + { :when => lambda {|jobs| jobs < 15 }, :workers => 1 }, + { :when => lambda {|jobs| jobs < 30 }, :workers => 2 }, + { :when => lambda {|jobs| jobs < 60 }, :workers => 3 }, + { :when => lambda {|jobs| jobs < 90 }, :workers => 4 } + ]) + end + + it 'should request 1 worker' do + base.jobs = 1 + base.workers = 0 + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 1 in total.') + base.expects(:workers).with(1).once + base.hire + end + + it 'should not request 1 worker, since there already is one worker running' do + base.jobs = 5 + base.workers = 1 + + base.expects(:workers).with(1).never + base.hire + end + + it 'should request 2 workers' do + base.jobs = 15 + base.workers = 0 + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 2 in total.') + base.expects(:workers).with(2).once + base.hire + end + + it 'should request 2 workers' do + base.jobs = 20 + base.workers = 1 + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 2 in total.') + base.expects(:workers).with(2).once + base.hire + end + + it 'should not request 2 workers since we already have 2' do + base.jobs = 25 + base.workers = 2 + + base.expects(:workers).with(2).never + base.hire + end + + it 'should NEVER lower the worker amount from the #hire method' do + base.jobs = 25 # simulate that 5 jobs are already processed (30 - 5) + base.workers = 3 # and 3 workers are hired + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 2 in total.').never + base.expects(:workers).with(2).never + base.hire + end + + it 'should NEVER hire more workers than the #max_workers' do + base.jobs = 100 + base.workers = 0 + + base.stubs(:max_workers).returns(3) # set the max_workers = 3 + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 3 in total.').once + base.expects(:workers).with(3).once + base.hire + end + + it 'should not hire 5 workers even if defined in the job/ratio, when the limit is 3, it should hire 3 max' do + base.jobs = 100 + base.workers = 0 + + base.stubs(:max_workers).returns(3) # set the max_workers = 3 + base.stubs(:ratio).returns([ + { :when => lambda { |jobs| jobs < 5 }, :workers => 5 } + ]) + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 3 in total.').once + base.expects(:workers).with(3).once + base.hire + end + + it 'should not hire (or invoke) any more workers since the max amount allowed is already running' do + base.jobs = 100 + base.workers = 3 + + base.stubs(:max_workers).returns(3) # set the max_workers = 3 + base.stubs(:ratio).returns([ + { :when => lambda { |jobs| jobs < 5 }, :workers => 5 } + ]) + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 3 in total.').never + base.expects(:workers).with(3).never + base.hire + end + + it 'the max_workers option can only "limit" the amount of max_workers when used in the "Standard Notation"' do + base.jobs = 100 + base.workers = 0 + + base.stubs(:max_workers).returns(10) # set the max_workers = 10 + base.stubs(:ratio).returns([ + { :when => lambda { |jobs| jobs < 5 }, :workers => 5 } + ]) + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 10 in total.').once + base.expects(:workers).with(10).once + base.hire + end + + it 'should NEVER do API requests to Heroku if the max_workers are already running' do + base.jobs = 100 + base.workers = 5 + + HireFire::Logger.expects(:message).with('Hiring more workers so we have 5 in total.').never + base.expects(:workers).with(5).never + base.hire + end + end + end +end