Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Shoryuken options can be set via CLI:
shoryuken -q queue1 -L ./shoryuken.log -P ./shoryuken.pid
or via configuration file:
# shoryuken.yml logfile: ./shoryuken.log pidfile: ./shoryuken.pid queues: - queue1
When using a configuration file, you must set via CLI
shoryuken -C ./shoryuken.yml, otherwise Shoryuken won't use it.
Some options available in the configuration file, are not available in the CLI. For checking all options available in the CLI:
shoryuken help start.
Delay is the number of seconds to pause fetching from when an empty queue.
Given this configuration:
delay: 25 queues: - queue1 - queue2
If Shoryuken tries to fetch messages from
queue1 and it has no messages, Shoryuken will pause fetching from
queue1 for 25 seconds.
Usually having a delay is more cost efficient, but if you want to consume messages as soon as they get in the queue, I would recommend setting
delay: 0 (default value) to stop pausing empty queues.
A single Shoryuken process can consume messages from multiple queues. You can define your queues as follows:
queues: - queue1 - queue2
You can also configure the queue URLs instead of names.
queues: - https://sqs.eu-west-1.amazonaws.com:000/1111111111/queue1 - https://sqs.eu-east-1.amazonaws.com:000/2222222222/queue2
Supposing you have
queue1, which you would like to fetch messages twice as much as
queue2, you can configure that as follows:
queues: - [queue1, 8] - [queue2, 4] - [queue3, 1]
The setup above will cause Shoryuken to fetch messages in cycles of
queue1 8 times, then
queue2 4 times, then
queue3 once, then repeat.
Note: Each fetch can fetch up to 10 messages at time (SQS limitation). The fetch size will also depend on the available workers at the time of the fetch.
See Polling strategies.
Concurrency is the number of threads available for processing messages at time. Be careful with changing this value, otherwise you could crush your machine with I/O.
Note: When accessing databases (
ActiveRecord) or other resources with
pool support in your workers, make sure to set
pool size with the same value or higher than the concurrency set in Shoryuken.
By default, Shoryuken will make requests against SQS for getting the updated queue visibility timeout, so if you change the visibility timeout in SQS, Shoryuken will automatically update it.
If you want to reduce the number of requests against SQS (therefore reducing your quote usage), you can disable this behavior by:
Shoryuken.cache_visibility_timeout = true
cache_visibility_timeout enabled, every time you change the visibility timeout in SQS, you will need to restart Shoryuken, otherwise it won't get updated.