New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retry_on/discard_on for better exception handling #25991

Merged
merged 17 commits into from Aug 2, 2016

Conversation

Projects
None yet
7 participants
@dhh
Member

dhh commented Jul 29, 2016

Declarative exception handling of the most common kind: retrying and discarding.

class RemoteServiceJob < ActiveJob::Base
  retry_on Net::OpenTimeout, wait: 30.seconds, attempts: 10

  def perform(*args)
    # Might raise Net::OpenTimeout when the remote service is down
  end
end

class SearchIndexingJob < ActiveJob::Base
  discard_on ActiveJob::DeserializationError

  def perform(record)
    # Will raise ActiveJob::DeserializationError if the record can't be deserialized
  end
end

@kaspth kaspth added this to the 5.1.0 milestone Jul 29, 2016

@kaspth kaspth added the activejob label Jul 29, 2016

dhh added some commits Jul 29, 2016

# Discard the job with no attempts to retry, if the exception is raised. This is useful when the subject of the job,
# like an Active Record, is no longer available, and the job is thus no longer relevant.
#
# ==== Example

This comment has been minimized.

@kaspth

kaspth Jul 29, 2016

Member

Nitpick: This says "Example" while the other docs say "Examples" (plural).

@kaspth

kaspth Jul 29, 2016

Member

Nitpick: This says "Example" while the other docs say "Examples" (plural).

This comment has been minimized.

@dhh

dhh Jul 29, 2016

Member

If there's only 1 example, it doesn't make sense to me to pluralize.

@dhh

dhh Jul 29, 2016

Member

If there's only 1 example, it doesn't make sense to me to pluralize.

This comment has been minimized.

@kaspth

kaspth Jul 29, 2016

Member

The other cases only have one example as well.

@kaspth

kaspth Jul 29, 2016

Member

The other cases only have one example as well.

This comment has been minimized.

@dhh

dhh Jul 29, 2016

Member

I'd rather fix those, then. Use plural when there are multiple examples and singular when just one.

@dhh

dhh Jul 29, 2016

Member

I'd rather fix those, then. Use plural when there are multiple examples and singular when just one.

This comment has been minimized.

@kaspth

kaspth Jul 30, 2016

Member

That's what I meant to say, just seems I forgot to actually put the words down 👍

@kaspth

kaspth Jul 30, 2016

Member

That's what I meant to say, just seems I forgot to actually put the words down 👍

# discard_on ActiveJob::DeserializationError
#
# def perform(record)
# # Will raise ActiveJob::DeserializationError if the record can't be deserialized

This comment has been minimized.

@kaspth

kaspth Jul 29, 2016

Member

I think our practice is to add a period to comments. Same goes for the other examples.

@kaspth

kaspth Jul 29, 2016

Member

I think our practice is to add a period to comments. Same goes for the other examples.

This comment has been minimized.

@dhh

dhh Jul 29, 2016

Member

I haven't been using that form consistently. Usually only use periods if there are multiple sentences.

@dhh

dhh Jul 29, 2016

Member

I haven't been using that form consistently. Usually only use periods if there are multiple sentences.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca
Member

rafaelfranca commented Jul 29, 2016

:shipit:

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Jul 29, 2016

Member

Luuuv the API, :shipit:

Member

kaspth commented Jul 29, 2016

Luuuv the API, :shipit:

@matthewd

This comment has been minimized.

Show comment
Hide comment
@matthewd

matthewd Jul 29, 2016

Member

Alternative API:

discard_on Net::OpenTimeout, retries: 10, wait: 30.seconds
rescue_from Net::OpenTimeout, retries: 10, wait: 30.seconds do |error|
  # We failed to connect ten times; give up and send someone an email or something
end

Mostly, I like the fact this clearly states what happens when we run out of retries... retry_on feels a bit subtle for ".. and then almost-silently drop it on the floor".

Member

matthewd commented Jul 29, 2016

Alternative API:

discard_on Net::OpenTimeout, retries: 10, wait: 30.seconds
rescue_from Net::OpenTimeout, retries: 10, wait: 30.seconds do |error|
  # We failed to connect ten times; give up and send someone an email or something
end

Mostly, I like the fact this clearly states what happens when we run out of retries... retry_on feels a bit subtle for ".. and then almost-silently drop it on the floor".

@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Jul 29, 2016

Member

Ah, I see what you mean @matthewd. Actually what should happen after we try to retry a bunch of time and fail is that we should reraise and let the queue deal with it. Will fix that!

(Though I guess there's still an argument for ALSO allowing custom logic at that point, but).

Member

dhh commented Jul 29, 2016

Ah, I see what you mean @matthewd. Actually what should happen after we try to retry a bunch of time and fail is that we should reraise and let the queue deal with it. Will fix that!

(Though I guess there's still an argument for ALSO allowing custom logic at that point, but).

@matthewd

This comment has been minimized.

Show comment
Hide comment
@matthewd

matthewd Jul 29, 2016

Member

Yeah, I guess I'm positing that (unlike "discard", say,) "retry" isn't a distinct error handling strategy, but an intrinsic.. step? attribute? of any overall error-handling plan. It just happens that the default behaviour is to make zero retries.

Member

matthewd commented Jul 29, 2016

Yeah, I guess I'm positing that (unlike "discard", say,) "retry" isn't a distinct error handling strategy, but an intrinsic.. step? attribute? of any overall error-handling plan. It just happens that the default behaviour is to make zero retries.

@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Jul 29, 2016

Member

Definitely. Failing to retry should absolutely not result in dropping the job. Just fixed that in the latest commit 👍

Member

dhh commented Jul 29, 2016

Definitely. Failing to retry should absolutely not result in dropping the job. Just fixed that in the latest commit 👍

def retry_on(exception, wait: 3.seconds, attempts: 5, queue: nil, priority: nil)
rescue_from exception do |error|
if executions < attempts
logger.error "Retrying #{self.class} in #{wait} seconds, due to a #{exception}. The original exception was #{error.cause.inspect}."

This comment has been minimized.

@matthewd

matthewd Jul 29, 2016

Member

Consider showing error instead of exception; the latter is more likely to be a vague parent class. "The original exception was nil." seems likely to be confusing, too.

This brushes with my "error reporting in general" project anyway, so possibly ignore for now.

@matthewd

matthewd Jul 29, 2016

Member

Consider showing error instead of exception; the latter is more likely to be a vague parent class. "The original exception was nil." seems likely to be confusing, too.

This brushes with my "error reporting in general" project anyway, so possibly ignore for now.

This comment has been minimized.

@dhh

dhh Jul 29, 2016

Member

Not sure I follow? exception is the class, error is the object.

@dhh

dhh Jul 29, 2016

Member

Not sure I follow? exception is the class, error is the object.

This comment has been minimized.

@matthewd

matthewd Jul 29, 2016

Member

Because of inheritance, error could be an instance of a very specific exception class, while exception could be something so vague as RuntimeError.

@matthewd

matthewd Jul 29, 2016

Member

Because of inheritance, error could be an instance of a very specific exception class, while exception could be something so vague as RuntimeError.

@matthewd

This comment has been minimized.

Show comment
Hide comment
@matthewd

matthewd Jul 29, 2016

Member

(Though I guess there's still an argument for ALSO allowing custom logic at that point, but).

FWIW, I think that's the thing I was arguing: if I write a custom rescue_from, I don't want to then have to implement my own retry mechanism -- that feels orthogonal to my decision on how I want to handle the "give up" step.

.. including on the new discard_on -- it seems right that the default after-retrying behaviour is to re-raise, but "try a few times then just forget it" seems just as likely as "try once then forget it".

Member

matthewd commented Jul 29, 2016

(Though I guess there's still an argument for ALSO allowing custom logic at that point, but).

FWIW, I think that's the thing I was arguing: if I write a custom rescue_from, I don't want to then have to implement my own retry mechanism -- that feels orthogonal to my decision on how I want to handle the "give up" step.

.. including on the new discard_on -- it seems right that the default after-retrying behaviour is to re-raise, but "try a few times then just forget it" seems just as likely as "try once then forget it".

end
# Reschedules the job to be re-executed. This is useful in combination
# with the +rescue_from+ option. When you rescue an exception from your job

This comment has been minimized.

@robin850

robin850 Jul 30, 2016

Member

I think you meant "the +rescue_from+ method" here, not option, no ?

@robin850

robin850 Jul 30, 2016

Member

I think you meant "the +rescue_from+ method" here, not option, no ?

This comment has been minimized.

@dhh

dhh Aug 1, 2016

Member

It's not meant as a programmatic option, but rather as "an option for dealing with exceptions". Other options include discard_on, retry_on.

@dhh

dhh Aug 1, 2016

Member

It's not meant as a programmatic option, but rather as "an option for dealing with exceptions". Other options include discard_on, retry_on.

Executions counting is not a serialization concern
Let’s do it when we actually execute instead. Then the tests dealing
with comparable serializations won’t fail either!
@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Aug 1, 2016

Member

@matthewd Added the power to provide a custom handler if the retry attempts are unsuccessful.

Member

dhh commented Aug 1, 2016

@matthewd Added the power to provide a custom handler if the retry attempts are unsuccessful.

@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Aug 1, 2016

Member

This is now ready to merge from my perspective, lest anyone has any last objections.

Member

dhh commented Aug 1, 2016

This is now ready to merge from my perspective, lest anyone has any last objections.

@alexcameron89

This comment has been minimized.

Show comment
Hide comment
@alexcameron89

alexcameron89 Aug 2, 2016

Member

Did you mean to leave byebug?

Member

alexcameron89 commented on activejob/test/cases/exceptions_test.rb in 0be5d5d Aug 2, 2016

Did you mean to leave byebug?

@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Aug 2, 2016

Member

Did not. Removed. Thanks.

Member

dhh commented Aug 2, 2016

Did not. Removed. Thanks.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Aug 2, 2016

Member

Left some questions, but otherwise good to me 😁

CodeClimate, on the other hand, seems to wish for some climate change 😋

Member

kaspth commented Aug 2, 2016

Left some questions, but otherwise good to me 😁

CodeClimate, on the other hand, seems to wish for some climate change 😋

dhh added some commits Aug 2, 2016

dhh added some commits Aug 2, 2016

@dhh dhh merged commit d46d61e into master Aug 2, 2016

1 check was pending

codeclimate Code Climate is analyzing this code.
Details

@dhh dhh deleted the retry-and-discard-jobs branch Aug 2, 2016

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn Sep 14, 2016

Member

s/Rubocup/Rubocop

Member

arthurnn commented on 111227c Sep 14, 2016

s/Rubocup/Rubocop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment