New config file format

This is not done yet. See Issue #20.

Basic yaml hierarchy

• level 1: optional: `defaults`, `hosts` and `environments` keys.
• level 2: optional: per host or environment selector (see #1), e.g. `production` or `host1.ec2.internal`
• level 3: list of worker types, can be either a hash or an array of hashes
• level 4: worker type config, e.g. count, max_mem, etc. (see #18, #19)

simple example

defaults:
  max_mem:  1024 # in MB
  max_time: 3600 # in seconds

'foo,bar': 5
'bar,foo,quux': 5
'baz':
  workers: 2
  max_mem: 500
  max_time: 90

full fledged example

defaults:
  max_mem:  1024 # in MB
  max_time: 3600 # in seconds

environments:
  development:
    '*': 4
  production:
    'foo,bar': 5
    'bar,foo,quux': 5
    'baz':
      workers: 2
      max_mem: 500 # in MB
      max_time: 90 # in seconds

hosts:
  'solr*':
    'bg_index_queue': 10
  'mail*':
    - queues: 'mail_high_priority,mail_medium_priority,mail_low_priority'
      workers: 4
    - queues: 'mail_high_priority'
      workers: 4
      max_time: 120
    - queues: 'mail_high_priority'
      workers: 2
      max_time: 60

defaults

If a top level key `defaults` is present, it’s hash will be used for the defaults for any worker settings other than `queues` or `workers`. This means that if you have a queue called `defaults`, you won’t be able to use the simple config file format and also setup workers which process only that queue. The worker type config will be merged over the defaults.

hosts and environments

The first two levels in the hierarchy are optional, to be compatible with the simpler original config file format. If one of the top-level keys is `environments` or `hosts`, it will be interpreted as above. If one of the top-level keys is `development` or `production`, then the top level will be interpreted as if it were environment based. Otherwise, it will be interpreted as nothing more than a list of worker types (plus `defaults` if applicable).

Host selectors may contain `*` wildcard.

Host config takes priority over environment config. Host config will not be merged with environment config. After determining which host or environment config is appropriate, the reset of the config file (except for `defaults`) is simply ignored.

worker type config

A list of worker types. Most commonly, it will just be a hash where the keys represent the queues to process and the values represent the number of workers. The values can also be a hash, to specify worker settings beyond simply the worker count (e.g. `max_mem`, `max_time`). If you need to distinguish between two different types of workers which are both processing the same queues, you can use an array of hashes, and specify the `queues` key for each hash.