Skip to content
Permalink
Browse files

Merge remote-tracking branch 'origin/non_forking_workers' into skroutz

  • Loading branch information...
konstantinoskostis committed Feb 26, 2019
2 parents fc31784 + 7a83ced commit 5683fcaf836d57c3570d7a6eed21b1d1229698f1
@@ -0,0 +1,79 @@
## 0.4.0.skroutz.0 (2019-02-26)

* Enable non forking resque workers when `fork_per_job: false` is present for a queue's workers
* YAML config file to support `workers` and `fork_per_job` keys
* Depend on a resque version >= 1.23.1

## 0.3.0 (2012-05-22)

This is mostly just a long overdue maintenance release. Many pull requests were
merged. A few non-pull-request branches were merged too. This version supports
ruby 1.9.3, 1.8.7, and even ancient 1.8.6, and all are checked by
[travis-ci](http://travis-ci.org/nevans/resque-pool). It also explicitly
supports resque ~> 1.20. And (if you have `gem-man` installed), it now has man
pages for bin and yml config.

Many thanks to the contributers!

* [@agnellvj](https://github.com/agnellvj): ruby 1.9 compatibility
* [@geoffgarside](https://github.com/geoffgarside): man pages!
* [@imajes](https://github.com/imajes) - bugfix: Handle when a pid no longer
exists by the time you try and kill it.
* [@jeremy](https://github.com/jeremy) & [@jamis](https://github.com/jamis) -
tasks require `resque/pool` lazily
* [@jhsu](https://github.com/jhsu) - bugfix: undefined variable 'e' for errors
* [@gaffneyc](https://github.com/gaffneyc) - compatibility fix:
Resque::Pool::PooledWorker as a module rather than class
* [@kcrayon](https://github.com/kcrayon) - bugfix: fix worker shutdown
* [@alexkwolfe](https://github.com/alexkwolfe) - added `app_name` for logging
(and maybe more in the future?)

## 0.2.0 (2011-03-15)

* new feature: sending `HUP` to pool manager will reload the logfiles and
gracefully restart all workers.
* enhancement: logging now includes timestamp, process "name" (worker or
manager), and PID.
* enhancement: can be used with no config file or empty config file (not all
*that* useful, but it's better than unceromoniously dieing!)
* bugfix: pidfile will be cleaned up on startup, e.g. if old process was
kill-9'd (Jason Haruska)
* bugfix: TERM/INT are no longer ignored when HUP is waiting on children
* bugfix: `resque-pool -c config.yml` command line option was broken
* development: simple cucumber features for core functionality.
* upstream: depends on resque ~> 1.13

## 0.1.0 (2011-01-18)

* new feature: `resque-pool` command line interface
* this replaces need for a special startup script.
* manages PID file, logfiles, daemonizing, etc.
* `resque-pool --help` for more info and options
* updated example config, init.d script, including a chef recipe that should
work at EngineYard.

## 0.0.10 (2010-08-31)

* remove rubygems 1.3.6 dependency

## 0.0.9 (2010-08-26)

* new feature: `RESQUE_POOL_CONFIG` environment variable to set alt config file
* upgraded to resque 1.10, removing `Resque::Worker` monkeypatch

## 0.0.8 (2010-08-20)

* bugfix: using (or not using) environments in config file

## 0.0.7 (2010-08-16)

* new feature: split by environments in config file
* added example startup script, Rakefile, and monit config

## 0.0.5 (2010-06-29)

* bugfix: worker processes not shutting down after orphaned

## 0.0.4 (2010-06-29)

* first release used in production
@@ -1,9 +1,9 @@
PATH
remote: .
specs:
resque-pool (0.4.0.dev)
resque-pool (0.4.0.skroutz.0)
rake
resque (~> 1.20)
resque (>= 1.23.1, < 2.0.0)
trollop (~> 1.16)

GEM
@@ -27,20 +27,23 @@ GEM
gherkin (2.10.0)
json (>= 1.4.6)
hpricot (0.8.6)
json (1.7.3)
multi_json (1.3.5)
json (2.2.0)
mono_logger (1.1.0)
multi_json (1.13.1)
mustache (0.99.4)
rack (1.4.1)
rack-protection (1.2.0)
mustermann (1.0.3)
rack (2.0.6)
rack-protection (2.0.5)
rack
rake (0.9.2.2)
rake (12.3.2)
rdiscount (1.6.8)
redis (2.2.2)
redis-namespace (1.0.3)
redis (< 3.0.0)
resque (1.20.0)
redis (4.1.0)
redis-namespace (1.6.0)
redis (>= 3.0.4)
resque (1.27.4)
mono_logger (~> 1.0)
multi_json (~> 1.0)
redis-namespace (~> 1.0.2)
redis-namespace (~> 1.3)
sinatra (>= 0.9.2)
vegas (~> 0.1.2)
ronn (0.7.3)
@@ -55,11 +58,12 @@ GEM
rspec-expectations (2.10.0)
diff-lcs (~> 1.1.3)
rspec-mocks (2.10.1)
sinatra (1.3.2)
rack (~> 1.3, >= 1.3.6)
rack-protection (~> 1.2)
tilt (~> 1.3, >= 1.3.3)
tilt (1.3.3)
sinatra (2.0.5)
mustermann (~> 1.0)
rack (~> 2.0)
rack-protection (= 2.0.5)
tilt (~> 2.0)
tilt (2.0.9)
trollop (1.16.2)
vegas (0.1.11)
rack (>= 1.0.0)
@@ -0,0 +1,69 @@
General Info
============
Please refer to the README.md for general information regarding resque-pool

How to use
-----------

## Skroutz

### YAML file config
The `config/resque-pool.yml` can recognise whether workers of a queue should
be forking or non-forking. For that reason the declaration of worker count
and fork mode has been enhanced and now the following syntaxes are supported:

1. Upstream's syntax

```
foo: 1
production:
"bar,baz": 2
"koko": 4
```

2. Skroutz syntax

```
foo:
- workers: 1
production:
"bar,baz":
- workers: 2
"koko":
- workers: 4
- fork_per_job: false
```

3. Mixed syntax

```
foo: 1
production:
"bar,baz": 2
"koko":
- workers: 4
- fork_per_job: false
```

### Start the pool manager
The queues can start as usual using:

```
resque-pool --daemon --environment production
```

Following the template YAML files presented above with skroutz and mixed
syntax the equivalent resque commands would be the following:

```
rake resque:work RAILS_ENV=production QUEUES=foo &
rake resque:work RAILS_ENV=production QUEUES=bar,baz &
rake resque:work RAILS_ENV=production QUEUES=bar,baz &
rake resque:work RAILS_ENV=production QUEUES=koko FORK_PER_JOB=false &
rake resque:work RAILS_ENV=production QUEUES=koko FORK_PER_JOB=false &
rake resque:work RAILS_ENV=production QUEUES=koko FORK_PER_JOB=false &
rake resque:work RAILS_ENV=production QUEUES=koko FORK_PER_JOB=false &
```
@@ -344,7 +344,63 @@ def quit_excess_workers_for(queues)
end

def worker_delta_for(queues)
config.fetch(queues, 0) - workers.fetch(queues, []).size
config_get_worker_count(queues) - workers.fetch(queues, []).size
end

# Get the number of workers declared in the config file for queues
#
# @param queues [String] a given key/entry from the config file
# @return [Fixnum] If the queues parameter does not exist in the config file
# 0 is returned else the declared count of workers is returned.
#
# Example:
# config_get_worker_count("foo,bar")
# config_get_worker("baz")
def config_get_worker_count(queues)
return 0 unless config.has_key?(queues)

v = config[queues]

if v.is_a?(Fixnum)
return v
elsif v.is_a?(Array)
workers_entry = v.find { |entry| entry.has_key?('workers') }
return (workers_entry ? workers_entry['workers'] : 0)
end

return 0
end

# Check if certain queues should have their workers to fork or not
#
# NOTE: If you need to have dedicated non-forking workers for a queue then
# you need to correctly specify false into the `fork_per_job` key of your queue.
# An example can be found in the skroutz README.
#
# Default behavior is to fork in order to comply with resque since it's default
# behavior is also to fork unless specified otherwise (See the FORK_PER_JOB
# environment variable in resque).
#
# @param queues [Array<String>|String] the input queue/queues
# @return [Boolean] true if 'fork_per_job' is true or absent and
# false if 'fork_per_job' is declared as false
#
# Example:
# fork_enabled_for_queues?(["foo", "bar"])
# fork_enabled_for_queues?("foo,bar")
# fork_enabled_for_queues?("baz")
def fork_enabled_for_queues?(queues)
queues_key = queues.is_a?(Array) ? queues.join(',') : queues
return true unless config.has_key?(queues_key)

v = config[queues_key]

if v.is_a?(Array)
fork_entry = config[queues_key].find { |entry| entry.has_key?('fork_per_job') }
return (fork_entry ? !!fork_entry['fork_per_job'] : true)
end

true
end

def pids_for(queues)
@@ -368,6 +424,7 @@ def create_worker(queues)
worker = ::Resque::Worker.new(*queues)
worker.verbose = ENV['LOGGING'] || ENV['VERBOSE']
worker.very_verbose = ENV['VVERBOSE']
worker.fork_per_job = fork_enabled_for_queues?(queues)
worker
end

@@ -1,5 +1,5 @@
module Resque
class Pool
VERSION = "0.4.0.dev"
VERSION = "0.4.0.skroutz.0"
end
end
@@ -15,7 +15,7 @@ Gem::Specification.new do |s|
saving memory (w/REE) and monitoring their uptime
EOF

s.add_dependency "resque", "~> 1.20"
s.add_dependency "resque", ">= 1.23.1", "< 2.0.0"
s.add_dependency "trollop", "~> 1.16"
s.add_dependency "rake"
s.add_development_dependency "rspec", "~> 2.10.0"
@@ -0,0 +1,22 @@
foo: 1

lala:
- workers: 7
- fork_per_job: false

production:
"foo,bar":
- workers: 10

development:
"foo,bar": 4
"baz":
- workers: 23
"koko":
- workers: 1
- fork_per_job: false

test:
"bar":
- workers: 5
"foo,bar": 3
@@ -0,0 +1,25 @@
foo:
- workers: 1

lala:
- workers: 7
- fork_per_job: false

production:
"foo,bar":
- workers: 10

development:
"foo,bar":
- workers: 4
"baz":
- workers: 23
"koko":
- workers: 1
- fork_per_job: false

test:
"bar":
- workers: 5
"foo,bar":
- workers: 3

0 comments on commit 5683fca

Please sign in to comment.
You can’t perform that action at this time.