-
-
Notifications
You must be signed in to change notification settings - Fork 277
Shoryuken options
Shoryuken options can be set via CLI:
shoryuken -q queue1 -l ./shoryuken.log -p ./shoryuken.pid
or via configuration file:
# shoryuken.yml
logfile: ./shoryuken.log
pidfile: ./shoryuken.pid
queues:
- queue1
When using a configuration file, you must set via CLI shoryuken -C ./shoryuken.yml
, otherwise Shoryuken won't use it.
Some options available in the configuration file, are not available in the CLI. For checking all options available in the CLI: shoryuken help start
.
Default: 0
Delay is the number of seconds to pause fetching from when an empty queue.
Given this configuration:
delay: 25
queues:
- queue1
- queue2
If Shoryuken tries to fetch messages from queue1
and it has no messages, Shoryuken will pause fetching from queue1
for 25 seconds.
Usually having a delay is more cost efficient, but if you want to consume messages as soon as they get in the queue, I would recommend setting delay: 0
(default value) to stop pausing empty queues.
Check the AWS SQS pricing page for more information.
A single Shoryuken process can consume messages from multiple queues. You can define your queues as follows:
queues:
- queue1
- queue2
Supposing you have queue1
, which you would like to fetch messages twice as much as queue2
, you can configure that as follows:
queues:
- [queue1, 8]
- [queue2, 4]
- [queue3, 1]
The setup above will cause Shoryuken to fetch messages in cycles of queue1
8 times, then queue2
4 times, then queue3
once, then repeat.
Note: Each fetch can fetch up to 10 messages at time (SQS limitation). The fetch size will also depend on the available workers at the time of the fetch.
Note: The queue priorities starts always at 1, and increases one-by-one until it reaches the max configured for every successful fetch returning messages. Similar happens when a queue reaches its maximum then gets empty, Shoryuken will decrease its priority one-by-one until starts to receive messages.
Default: 25
Concurrency is the number of threads available for processing messages at time. Be careful with changing this value, otherwise you could crush your machine with I/O.
Note: When accessing databases (ActiveRecord
) or other resources with pool
support in your workers, make sure to set pool
size with the same value or higher than the concurrency set in Shoryuken.