Skip to content

Add support for retryable errors through Qless::RetryExceptions mixin. #63

Merged
merged 4 commits into from Jan 7, 2013

4 participants

@benkirzhner
Moz member

Closes #29.

@proby
proby commented Dec 17, 2012

I'd like to see an integration test where the error is a retryable exception but the job has exhausted it's retries.

@myronmarston myronmarston and 1 other commented on an outdated diff Dec 17, 2012
lib/qless/worker.rb
@@ -100,7 +100,12 @@ def work(interval = 5.0)
def perform(job)
around_perform(job)
rescue Exception => error
- fail_job(job, error)
+ if job.klass.respond_to?(:retryable_exception?) &&
+ job.klass.retryable_exception?(error)
+ job.retry
+ else
+ fail_job(job, error)
+ end
else
@myronmarston
Moz member
myronmarston added a note Dec 17, 2012

While it's not normal to do so, you can use a ruby expression in a rescue clause. Given that, I think you can simplify things a lot by doing something like:

def retryable_exceptions
  return [] unless job.klass.respond_to?(:retryable_exceptions)
  job.klass.retryable_exceptions
end

def perform(job)
  around_perform(job)
rescue *retryable_exceptions
  job.retry
rescue Exception => error
  fail_job(job, error)
end

You also wouldn't need to define retryable_exception? (with it's any?/is_a? logic)

@myronmarston
Moz member
myronmarston added a note Dec 17, 2012

BTW, a side benefit of this: ruby allows anything to be used in a rescue clause that responds to === and uses that to see if the exception matches. The normal thing is to use class constants, but it gives you the power to use an object that inspects the message or whatever. By leveraging ruby's built-in logic, it allows things to be more flexible here and work the same way as ruby itself does.

@benkirzhner
Moz member
benkirzhner added a note Dec 17, 2012

Ah, I was confused about raise vs rescue requiring a subclass of Exception and failing if anything else is provided. The reason I went with a retryable_exception? method is that it allows for the same flexibility you're describing with ===. I'll make the code change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@myronmarston myronmarston merged commit 7ae464e into master Jan 7, 2013

1 check failed

Details default The Travis build failed
@benkirzhner benkirzhner deleted the retry_exceptions branch Jan 7, 2013
@jstorimer

Was it intentional that job classes that don't use this mixin won't be retried at all?

class MyJob
  def self.perform(job)
    raise 'hell'
  end
end

This job fails immediately, without any retries. It seems that the old behaviour (retry on any exception) should be preserved for job classes that opt out of the explicit declaration.

Thoughts?

@myronmarston
Moz member

@jstorimer -- before this change, jobs were never retried automatically by the worker. If you wanted to retry a job, you were responsible for calling job.retry yourself. Maybe I'm missing something, but I don't think this changed any behavior for existing code, just added the ability to provide a whitelist of exceptions to retry.

@jstorimer

Ohhh I see. I'm just getting back to qless after not touching it for a few months. I hadn't realized that was the previous behaviour. Forget I said anything 🚶

Now I appreciate the fact that I can do retry_on(StandardError).

@myronmarston
Moz member

@jstorimer -- No worries :). I appreciate having other folks that are using Qless commenting on this stuff, so thanks!

On a side note, I'm curious as to how you're using Qless (and I imagine @dlecocq is, too). We're using it on quite a few projects internally at SEOmoz, but hearing how other folks are using Qless (potentially in entirely different ways) will help inform how we change Qless over time. If you feel up for it, can you write up something explaining how you're using it? (e.g. which Qless features do you rely on heavily? How do you structure your job pipelines? Is it performing adequately for you?) Feel free to write up your response here or send an email to @dlecocq and I.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.