Merge remote-tracking branch 'origin/non_forking_workers' into skroutz

Changelog.skroutz.md

@@ -0,0 +1,79 @@
+## 0.4.0.skroutz.0 (2019-02-26)
+
+ * Enable non forking resque workers when `fork_per_job: false` is present for a queue's workers
+ * YAML config file to support `workers` and `fork_per_job` keys
+ * Depend on a resque version >= 1.23.1
+
+## 0.3.0 (2012-05-22)
+
+This is mostly just a long overdue maintenance release.  Many pull requests were
+merged.  A few non-pull-request branches were merged too.  This version supports
+ruby 1.9.3, 1.8.7, and even ancient 1.8.6, and all are checked by
+[travis-ci](http://travis-ci.org/nevans/resque-pool).  It also explicitly
+supports resque ~> 1.20.  And (if you have `gem-man` installed), it now has man
+pages for bin and yml config.
+
+Many thanks to the contributers!
+
+ * [@agnellvj](https://github.com/agnellvj): ruby 1.9 compatibility
+ * [@geoffgarside](https://github.com/geoffgarside): man pages!
+ * [@imajes](https://github.com/imajes) - bugfix: Handle when a pid no longer
+   exists by the time you try and kill it.
+ * [@jeremy](https://github.com/jeremy) & [@jamis](https://github.com/jamis) -
+   tasks require `resque/pool` lazily
+ * [@jhsu](https://github.com/jhsu) - bugfix: undefined variable 'e' for errors
+ * [@gaffneyc](https://github.com/gaffneyc) - compatibility fix:
+   Resque::Pool::PooledWorker as a module rather than class
+ * [@kcrayon](https://github.com/kcrayon) - bugfix: fix worker shutdown
+ * [@alexkwolfe](https://github.com/alexkwolfe) - added `app_name` for logging
+   (and maybe more in the future?)
+
+## 0.2.0 (2011-03-15)
+
+* new feature: sending `HUP` to pool manager will reload the logfiles and
+  gracefully restart all workers.
+* enhancement: logging now includes timestamp, process "name" (worker or
+  manager), and PID.
+* enhancement: can be used with no config file or empty config file (not all
+  *that* useful, but it's better than unceromoniously dieing!)
+* bugfix: pidfile will be cleaned up on startup, e.g. if old process was
+  kill-9'd (Jason Haruska)
+* bugfix: TERM/INT are no longer ignored when HUP is waiting on children
+* bugfix: `resque-pool -c config.yml` command line option was broken
+* development: simple cucumber features for core functionality.
+* upstream: depends on resque ~> 1.13
+
+## 0.1.0 (2011-01-18)
+
+* new feature: `resque-pool` command line interface
+  * this replaces need for a special startup script.
+  * manages PID file, logfiles, daemonizing, etc.
+  * `resque-pool --help` for more info and options
+* updated example config, init.d script, including a chef recipe that should
+  work at EngineYard.
+
+## 0.0.10 (2010-08-31)
+
+* remove rubygems 1.3.6 dependency
+
+## 0.0.9 (2010-08-26)
+
+* new feature: `RESQUE_POOL_CONFIG` environment variable to set alt config file
+* upgraded to resque 1.10, removing `Resque::Worker` monkeypatch
+
+## 0.0.8 (2010-08-20)
+
+* bugfix: using (or not using) environments in config file
+
+## 0.0.7 (2010-08-16)
+
+* new feature: split by environments in config file
+* added example startup script, Rakefile, and monit config
+
+## 0.0.5 (2010-06-29)
+
+* bugfix: worker processes not shutting down after orphaned
+
+## 0.0.4 (2010-06-29)
+
+* first release used in production

Gemfile.lock

@@ -1,9 +1,9 @@
 PATH
   remote: .
   specs:
-    resque-pool (0.4.0.dev)
+    resque-pool (0.4.0.skroutz.0)
       rake
-      resque (~> 1.20)
+      resque (>= 1.23.1, < 2.0.0)
       trollop (~> 1.16)
 
 GEM
@@ -27,20 +27,23 @@ GEM
     gherkin (2.10.0)
       json (>= 1.4.6)
     hpricot (0.8.6)
-    json (1.7.3)
-    multi_json (1.3.5)
+    json (2.2.0)
+    mono_logger (1.1.0)
+    multi_json (1.13.1)
     mustache (0.99.4)
-    rack (1.4.1)
-    rack-protection (1.2.0)
+    mustermann (1.0.3)
+    rack (2.0.6)
+    rack-protection (2.0.5)
       rack
-    rake (0.9.2.2)
+    rake (12.3.2)
     rdiscount (1.6.8)
-    redis (2.2.2)
-    redis-namespace (1.0.3)
-      redis (< 3.0.0)
-    resque (1.20.0)
+    redis (4.1.0)
+    redis-namespace (1.6.0)
+      redis (>= 3.0.4)
+    resque (1.27.4)
+      mono_logger (~> 1.0)
       multi_json (~> 1.0)
-      redis-namespace (~> 1.0.2)
+      redis-namespace (~> 1.3)
       sinatra (>= 0.9.2)
       vegas (~> 0.1.2)
     ronn (0.7.3)
@@ -55,11 +58,12 @@ GEM
     rspec-expectations (2.10.0)
       diff-lcs (~> 1.1.3)
     rspec-mocks (2.10.1)
-    sinatra (1.3.2)
-      rack (~> 1.3, >= 1.3.6)
-      rack-protection (~> 1.2)
-      tilt (~> 1.3, >= 1.3.3)
-    tilt (1.3.3)
+    sinatra (2.0.5)
+      mustermann (~> 1.0)
+      rack (~> 2.0)
+      rack-protection (= 2.0.5)
+      tilt (~> 2.0)
+    tilt (2.0.9)
     trollop (1.16.2)
     vegas (0.1.11)
       rack (>= 1.0.0)

README.skroutz.md

@@ -0,0 +1,69 @@
+General Info
+============
+Please refer to the README.md for general information regarding resque-pool
+
+How to use
+-----------
+
+## Skroutz
+
+### YAML file config
+The `config/resque-pool.yml` can recognise whether workers of a queue should
+be forking or non-forking. For that reason the declaration of worker count
+and fork mode has been enhanced and now the following syntaxes are supported:
+
+1. Upstream's syntax
+
+```
+foo: 1
+
+production:
+    "bar,baz": 2
+    "koko": 4
+```
+
+2. Skroutz syntax
+
+```
+foo:
+  - workers: 1
+
+production:
+  "bar,baz":
+    - workers: 2
+  "koko":
+    - workers: 4
+    - fork_per_job: false
+```
+
+3. Mixed syntax
+
+```
+foo: 1
+
+production:
+  "bar,baz": 2
+  "koko":
+    - workers: 4
+    - fork_per_job: false
+```
+
+### Start the pool manager
+The queues can start as usual using:
+
+```
+    resque-pool --daemon --environment production
+```
+
+Following the template YAML files presented above with skroutz and mixed
+syntax the equivalent resque commands would be the following:
+
+```
+    rake resque:work RAILS_ENV=production QUEUES=foo &
+    rake resque:work RAILS_ENV=production QUEUES=bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=bar,baz &
+    rake resque:work RAILS_ENV=production QUEUES=koko FORK_PER_JOB=false &
+    rake resque:work RAILS_ENV=production QUEUES=koko FORK_PER_JOB=false &
+    rake resque:work RAILS_ENV=production QUEUES=koko FORK_PER_JOB=false &
+    rake resque:work RAILS_ENV=production QUEUES=koko FORK_PER_JOB=false &
+```

lib/resque/pool.rb

@@ -344,7 +344,63 @@ def quit_excess_workers_for(queues)
     end
 
     def worker_delta_for(queues)
-      config.fetch(queues, 0) - workers.fetch(queues, []).size
+      config_get_worker_count(queues) - workers.fetch(queues, []).size
+    end
+
+    # Get the number of workers declared in the config file for queues
+    #
+    # @param queues [String] a given key/entry from the config file
+    # @return [Fixnum] If the queues parameter does not exist in the config file
+    # 0 is returned else the declared count of workers is returned.
+    #
+    # Example:
+    #   config_get_worker_count("foo,bar")
+    #   config_get_worker("baz")
+    def config_get_worker_count(queues)
+      return 0 unless config.has_key?(queues)
+
+      v = config[queues]
+
+      if v.is_a?(Fixnum)
+        return v
+      elsif v.is_a?(Array)
+        workers_entry = v.find { |entry| entry.has_key?('workers') }
+        return (workers_entry ? workers_entry['workers'] : 0)
+      end
+
+      return 0
+    end
+
+    # Check if certain queues should have their workers to fork or not
+    #
+    # NOTE: If you need to have dedicated non-forking workers for a queue then
+    # you need to correctly specify false into the `fork_per_job` key of your queue.
+    # An example can be found in the skroutz README.
+    #
+    # Default behavior is to fork in order to comply with resque since it's default
+    # behavior is also to fork unless specified otherwise (See the FORK_PER_JOB
+    # environment variable in resque).
+    #
+    # @param queues [Array<String>|String] the input queue/queues
+    # @return [Boolean] true if 'fork_per_job' is true or absent and
+    # false if 'fork_per_job' is declared as false
+    #
+    # Example:
+    # fork_enabled_for_queues?(["foo", "bar"])
+    # fork_enabled_for_queues?("foo,bar")
+    # fork_enabled_for_queues?("baz")
+    def fork_enabled_for_queues?(queues)
+      queues_key = queues.is_a?(Array) ? queues.join(',') : queues
+      return true unless config.has_key?(queues_key)
+
+      v = config[queues_key]
+
+      if v.is_a?(Array)
+        fork_entry = config[queues_key].find { |entry| entry.has_key?('fork_per_job') }
+        return (fork_entry ? !!fork_entry['fork_per_job'] : true)
+      end
+
+      true
     end
 
     def pids_for(queues)
@@ -368,6 +424,7 @@ def create_worker(queues)
       worker = ::Resque::Worker.new(*queues)
       worker.verbose = ENV['LOGGING'] || ENV['VERBOSE']
       worker.very_verbose = ENV['VVERBOSE']
+      worker.fork_per_job = fork_enabled_for_queues?(queues)
       worker
     end
 

lib/resque/pool/version.rb

@@ -1,5 +1,5 @@
 module Resque
   class Pool
-    VERSION = "0.4.0.dev"
+    VERSION = "0.4.0.skroutz.0"
   end
 end

resque-pool.gemspec

@@ -15,7 +15,7 @@ Gem::Specification.new do |s|
     saving memory (w/REE) and monitoring their uptime
   EOF
 
-  s.add_dependency "resque",  "~> 1.20"
+  s.add_dependency "resque",  ">= 1.23.1", "< 2.0.0"
   s.add_dependency "trollop", "~> 1.16"
   s.add_dependency "rake"
   s.add_development_dependency "rspec",    "~> 2.10.0"

spec/resque-pool-mixed.yml

@@ -0,0 +1,22 @@
+foo: 1
+
+lala:
+  - workers: 7
+  - fork_per_job: false
+
+production:
+  "foo,bar":
+    - workers: 10
+
+development:
+  "foo,bar": 4
+  "baz":
+    - workers: 23
+  "koko":
+    - workers: 1
+    - fork_per_job: false
+
+test:
+  "bar":
+    - workers: 5
+  "foo,bar": 3

spec/resque-pool-skroutz.yml

@@ -0,0 +1,25 @@
+foo:
+  - workers: 1
+
+lala:
+  - workers: 7
+  - fork_per_job: false
+
+production:
+  "foo,bar":
+    - workers: 10
+
+development:
+  "foo,bar":
+    - workers: 4
+  "baz":
+    - workers: 23
+  "koko":
+    - workers: 1
+    - fork_per_job: false
+
+test:
+  "bar":
+    - workers: 5
+  "foo,bar":
+    - workers: 3