Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add support for retryable errors through Qless::RetryExceptions mixin. #63

Merged
merged 4 commits into from

4 participants

@benkirzhner
Owner

Closes #29.

@proby

I'd like to see an integration test where the error is a retryable exception but the job has exhausted it's retries.

lib/qless/worker.rb
@@ -100,7 +100,12 @@ def work(interval = 5.0)
def perform(job)
around_perform(job)
rescue Exception => error
- fail_job(job, error)
+ if job.klass.respond_to?(:retryable_exception?) &&
+ job.klass.retryable_exception?(error)
+ job.retry
+ else
+ fail_job(job, error)
+ end
else
@myronmarston Owner

While it's not normal to do so, you can use a ruby expression in a rescue clause. Given that, I think you can simplify things a lot by doing something like:

def retryable_exceptions
  return [] unless job.klass.respond_to?(:retryable_exceptions)
  job.klass.retryable_exceptions
end

def perform(job)
  around_perform(job)
rescue *retryable_exceptions
  job.retry
rescue Exception => error
  fail_job(job, error)
end

You also wouldn't need to define retryable_exception? (with it's any?/is_a? logic)

@myronmarston Owner

BTW, a side benefit of this: ruby allows anything to be used in a rescue clause that responds to === and uses that to see if the exception matches. The normal thing is to use class constants, but it gives you the power to use an object that inspects the message or whatever. By leveraging ruby's built-in logic, it allows things to be more flexible here and work the same way as ruby itself does.

@benkirzhner Owner

Ah, I was confused about raise vs rescue requiring a subclass of Exception and failing if anything else is provided. The reason I went with a retryable_exception? method is that it allows for the same flexibility you're describing with ===. I'll make the code change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@myronmarston myronmarston merged commit 7ae464e into from
@benkirzhner benkirzhner deleted the branch
@jstorimer
Collaborator

Was it intentional that job classes that don't use this mixin won't be retried at all?

class MyJob
  def self.perform(job)
    raise 'hell'
  end
end

This job fails immediately, without any retries. It seems that the old behaviour (retry on any exception) should be preserved for job classes that opt out of the explicit declaration.

Thoughts?

@myronmarston
Owner

@jstorimer -- before this change, jobs were never retried automatically by the worker. If you wanted to retry a job, you were responsible for calling job.retry yourself. Maybe I'm missing something, but I don't think this changed any behavior for existing code, just added the ability to provide a whitelist of exceptions to retry.

@jstorimer
Collaborator

Ohhh I see. I'm just getting back to qless after not touching it for a few months. I hadn't realized that was the previous behaviour. Forget I said anything :walking:

Now I appreciate the fact that I can do retry_on(StandardError).

@myronmarston
Owner

@jstorimer -- No worries :). I appreciate having other folks that are using Qless commenting on this stuff, so thanks!

On a side note, I'm curious as to how you're using Qless (and I imagine @dlecocq is, too). We're using it on quite a few projects internally at SEOmoz, but hearing how other folks are using Qless (potentially in entirely different ways) will help inform how we change Qless over time. If you feel up for it, can you write up something explaining how you're using it? (e.g. which Qless features do you rely on heavily? How do you structure your job pipelines? Is it performing adequately for you?) Feel free to write up your response here or send an email to @dlecocq and I.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
11 lib/qless/retry_exceptions.rb
@@ -0,0 +1,11 @@
+module Qless
+ module RetryExceptions
+ def retryable_exception_classes
+ @retryable_exception_classes ||= []
+ end
+
+ def retry_on(*exception_classes)
+ self.retryable_exception_classes.push(*exception_classes)
+ end
+ end
+end
View
7 lib/qless/worker.rb
@@ -99,6 +99,8 @@ def work(interval = 5.0)
def perform(job)
around_perform(job)
+ rescue *retryable_exception_classes(job)
+ job.retry
rescue Exception => error
fail_job(job, error)
else
@@ -135,6 +137,11 @@ def unpause_processing
private
+ def retryable_exception_classes(job)
+ return [] unless job.klass.respond_to?(:retryable_exception_classes)
+ job.klass.retryable_exception_classes
+ end
+
# Allow middleware modules to be mixed in and override the
# definition of around_perform while providing a default
# implementation so our code can assume the method is present.
View
32 spec/integration/worker_spec.rb
@@ -3,10 +3,23 @@
require 'yaml'
require 'qless/worker'
require 'qless'
+require 'qless/retry_exceptions'
class WorkerIntegrationJob
def self.perform(job)
- Redis.connect(:url => job['redis_url']).rpush('worker_integration_job', job['word'])
+ Redis.connect(url: job['redis_url']).rpush('worker_integration_job', job['word'])
+ end
+end
+
+class RetryIntegrationJob
+ extend Qless::RetryExceptions
+
+ Kaboom = Class.new(StandardError)
+ retry_on Kaboom
+
+ def self.perform(job)
+ Redis.connect(url: job['redis_url']).incr('retry_integration_job_count')
+ raise Kaboom
end
end
@@ -41,5 +54,22 @@ def start_worker(run_as_single_process)
it_behaves_like 'a running worker'
it_behaves_like 'a running worker', '1'
+
+ it 'will retry and eventually fail a repeatedly failing job' do
+ queue = client.queues["main"]
+ jid = queue.put(RetryIntegrationJob, {}, retries: 10)
+ Qless::Worker.new(
+ client,
+ Qless::JobReservers::RoundRobin.new([queue]),
+ run_as_a_single_process: true
+ ).work(0)
+
+ job = client.jobs[jid]
+
+ job.state.should eq('failed')
+ job.retries_left.should eq(-1)
+ job.original_retries.should eq(10)
+ client.redis.get('retry_integration_job_count').should eq('11')
+ end
end
View
22 spec/unit/retry_exceptions_spec.rb
@@ -0,0 +1,22 @@
+require 'qless/retry_exceptions'
+
+module Qless
+ describe RetryExceptions do
+ let(:job_class) { Class.new }
+ let(:exception_class) { Class.new(StandardError) }
+
+ before do
+ job_class.extend(RetryExceptions)
+ end
+
+ it 'defines a retryable_exceptions method that returns an empty array by default' do
+ job_class.retryable_exception_classes.should be_empty
+ end
+
+ it 'defines a retry_on method that makes exception types retryable' do
+ job_class.retry_on(exception_class)
+
+ job_class.retryable_exception_classes.should eq([exception_class])
+ end
+ end
+end
View
24 spec/unit/worker_spec.rb
@@ -62,6 +62,30 @@ class MyJobClass; end
worker.perform(job)
end
+ it 'fails the job if performing it raises a non-retryable error' do
+ MyJobClass.stub(:retryable_exception_classes).and_return([])
+ MyJobClass.stub(:perform) { raise Exception.new("boom") }
+ expected_line_number = __LINE__ - 1
+ job.should respond_to(:fail).with(2).arguments
+
+ job.should_receive(:fail) do |group, message|
+ group.should eq("Qless::MyJobClass:Exception")
+ message.should include("boom")
+ message.should include("#{__FILE__}:#{expected_line_number}")
+ end
+
+ worker.perform(job)
+ end
+
+ it 'retries the job if performing it raises a retryable error' do
+ MyJobClass.stub(:retryable_exception_classes).and_return([ArgumentError])
+ MyJobClass.stub(:perform) { raise ArgumentError.new("boom") }
+
+ job.should_receive(:retry).with(no_args)
+
+ worker.perform(job)
+ end
+
it 'completes the job if it finishes with no errors' do
MyJobClass.stub(:perform)
job.should respond_to(:complete).with(0).arguments
Something went wrong with that request. Please try again.