Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AO3-4037: Automatically retry failing emails #3397

Closed
wants to merge 1 commit into from

Conversation

elzj
Copy link
Contributor

@elzj elzj commented Jul 25, 2018

Issue

https://otwarchive.atlassian.net/browse/AO3-4037

Purpose

Rescues and retries resque email failures that occur when redis is ahead of mysql, which should hopefully keep those out of the failed jobs list, making it easier to track. Also cleans up a bit of duplication in the mailers.

Testing

It could be hard to simulate this on staging, so we may need a database admin to create some fake failures.

else
raise error
end
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Layout/TrailingBlankLines: Final newline missing.

@@ -1 +1,10 @@
Resque::Mailer.excluded_environments = [:test, :cucumber]
Resque::Mailer.error_handler = lambda { |mailer, message, error, action, args|

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lint/UnusedBlockArgument: Unused block argument - message. If it's necessary, use _ or _message as an argument name to indicate that it won't be used.

layout 'mailer'
helper :mailer
default from: "Archive of Our Own " + "<#{ArchiveConfig.RETURN_ADDRESS}>"
class KudoMailer < ArchiveMailer

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/FrozenStringLiteralComment: Missing magic comment # frozen_string_literal: true.

layout 'mailer'
helper :mailer
default from: "Archive of Our Own " + "<#{ArchiveConfig.RETURN_ADDRESS}>"
class CommentMailer < ArchiveMailer

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/FrozenStringLiteralComment: Missing magic comment # frozen_string_literal: true.

@@ -1,14 +1,9 @@
class CollectionMailer < ActionMailer::Base
include Resque::Mailer # see README in this directory
class CollectionMailer < ArchiveMailer

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/FrozenStringLiteralComment: Missing magic comment # frozen_string_literal: true.

helper :mailer
default from: "Archive of Our Own " + "<#{ArchiveConfig.RETURN_ADDRESS}>"

end

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Layout/TrailingBlankLines: Final newline missing.

layout 'mailer'
helper :mailer
default from: "Archive of Our Own " + "<#{ArchiveConfig.RETURN_ADDRESS}>"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Layout/EmptyLinesAroundClassBody: Extra empty line detected at class body end.

@@ -0,0 +1,8 @@
class ArchiveMailer < ActionMailer::Base

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/FrozenStringLiteralComment: Missing magic comment # frozen_string_literal: true.

helper :mailer
default from: "Archive of Our Own " + "<#{ArchiveConfig.RETURN_ADDRESS}>"

class AdminMailer < ArchiveMailer

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/FrozenStringLiteralComment: Missing magic comment # frozen_string_literal: true.

@elzj elzj changed the title AO3-4584: Automatically retry failing emails AO3-4037: Automatically retry failing emails Jul 25, 2018
@redsummernight
Copy link
Member

I think we should send emails from after_commit instead of after_update (in addition to this retry fix), so we don't have Redis ahead of MySQL in the first place.

Copy link
Member

@sarken sarken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, although I think the user_mailer might also need to be updated.

if error.is_a?(Resque::TermException)
Resque.enqueue(mailer, action, *args)
elsif error.is_a?(ActiveRecord::RecordNotFound) && !args.include?(:retried)
Resque.enqueue_in(5.minutes, mailer, action, :retried, *args)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it prepends the symbol :retried to the arguments passed to the mailer. Doesn't that mean all the arguments will be shifted over by 1, and the first argument (which is frequently supposed to be an ID of some sort) will be :retried? Or is there some code that strips the :retried symbol?

Also, I poked around the mailer code and I think Resque::Mailer.perform expects the arguments to be serialized:
https://github.com/zapnap/resque_mailer/blob/35c2a7c7ff9038b9bff95c50190d162f95afcd5f/lib/resque_mailer.rb#L52-L54

@tickinginstant
Copy link
Contributor

After some more poking around in the Resque Mailer code, I don't think it's going to be possible to use the built-in error handling to accomplish this, at least not without modifying some of the mailer code. For any mailers that take advantage of Resque::Mailer::Serializers::ActiveRecordSerializer by passing ActiveRecords as arguments, the ActiveRecord::RecordNotFound exception occurs in the serializer, before it's finished deserializing the arguments. But the error handler passes only the deserialized arguments (which in this case would be nil, because the error was thrown before they could be set). So it's impossible to resend the email from the error handler, because it doesn't have all the information it needs.

I'd suggest either modifying config/initializers/resque_mailer.rb to stop using Resque::Mailer::Serializers::ActiveRecordSerializer (with all of the changes that would entail), or overriding self.perform in the ArchiveMailer class to add a custom error handler (with a call to super).

(Alternatively, it might be possible to report this bug and get the Resque Mailer to handle errors in the serializer more sensibly.)

@sarken
Copy link
Member

sarken commented Aug 10, 2019

Just doing some housekeeping! We're going to close this since it doesn't sound like the approach will work. The issue will still be open and assigned to you if you want to take another crack at it, though! <3

@sarken sarken closed this Aug 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants