Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Follow these rules to improve your Sidekiq experience:
1. Make your job parameters small and simple
Sidekiq persists the arguments to
perform_async to Redis. I see a lot of people do this:
quote = Quote.find(quote_id) SomeWorker.perform_async(quote)
What you are doing here is serializing the entire Quote object to Redis. What if your queue backs up and that quote object changes in the meantime? Don't save state to Sidekiq, save simple identifiers. Look up the objects once you actually need them in your perform method.
The arguments you pass to
perform_async must be composed of simple JSON datatypes: string, integer, float, boolean, null, array and hash. This means you must not use ruby symbols as arguments. The Sidekiq client API uses
JSON.dump to send the data to Redis. The Sidekiq server pulls that JSON data from Redis and uses
JSON.load to convert the data back into Ruby types to pass to your perform method. Don't pass symbols, named parameters or complex Ruby objects (like Date or Time!) as those will not survive the dump/load round trip correctly.
2. Make your job idempotent and transactional
Idempotency means that your job can safely execute multiple times. For instance, with the error retry functionality, your job might be half-processed, throw an error, and then be re-executed over and over until it successfully completes. Let's say you have a job which voids a credit card transaction and emails the user to let them know the charge has been refunded:
def perform(card_charge_id) charge = CardCharge.find(card_charge_id) charge.void_transaction Emailer.charge_refunded(charge).deliver end
What happens when the email fails to render due to a bug? Will the
void_transaction method handle the case where a charge has already been refunded? You can use a database transaction to ensure data changes are rolled back if there is an error or you can write your code to be resilient in the face of errors. Just remember that Sidekiq will execute your job at least once, not exactly once.
3. Embrace Concurrency
Sidekiq is designed for parallel execution so design your jobs so you can run lots of them in parallel. It has basic features for tuning concurrency (e.g. targeting a sidekiq process at a queue with a defined number of threads) but your system architecture is much simpler if you don't have such specialization.
You can use a connection pool to limit the overall number of connections to a resource-limited server if your Sidekiq processes are overwhelming it with traffic.
Sidekiq will not provide features which hack around a lack of concurrency in your jobs.
Next: Using Redis