Feature Request: When using the retry option, allow option to suppress notifications until the last execution

[smithtimamy]

I would like to submit a feature request. It would be nice to be able to allow a job that is using the "Retry" option to hold of on sending notifications until the last execution.

Current behavior for a job that is set to email on failure and set with a retry count of 2:

• Job starts
• Job fails
• Email sent
• Job retried
• Job fails
• Email sent
• Job retried
• Job fails
• Email sent

Requested behavior for a job that is set to email on failure with a retry count of 2 and option enabled to suppress notifications until the last execution.

• Job starts
• Job fails
• Job retried
• Job fails
• Job retried
• Job fails
• Email sent

[coconitro]

I'm running into this as well. Are you able to reproduce this on scheduled jobs?

[WillLiu360]

I would also love this feature! Waking up to a failure email only to find it's okay on retry is not fun...

[tumbl3w33d]

Is this planned to be implemented for a 2.7.x release yet or does it need a newer milestone assigned? I would also appreciate this feature. Was surprised it wasn't there yet and also that not more people asked for it here. I consider it essential, exactly for the reason mentioned above.

[yazgoo]

+1, this feature is quite needed.

[mlachal]

I'm joining in, it would make a great combination with the new retry delay feature.

[brianjcarpenter]

+1 for this feature
+1 for retry delay

[gschueler]

@brianjcarpenter retry delay was added in 2.9.0

[sgrimm-sg]

As a workaround, you can add a second scheduled job with a "Job State Conditional" step and have it look at the status of the job you care about. There are two separate statuses you can test for, "Failed" and "Failed-with-retry." If the job still has retries remaining, the most recent status will be the second of those, so you can have the step check for plain "Failed" which will only match when the retries on the original job are exhausted. Then trigger your notification on the results of that second watchdog job rather than the original one.

This has a bunch of shortcomings (not least: once the original job goes to "Failed" state, the watchdog job will trigger a notification every time it fires) but it may be better than getting failure notifications for jobs that retry and succeed on their own.

Agreed completely that this should be a built-in feature; the lack of it severely limits the usefulness of the retry feature. But I'd actually like to see this done differently than the original issue asks for. Rather than, "Suppress failure notifications until retries are done," I'd prefer having an additional notification event type for "Failed but will retry." If you don't want notifications while there are still pending retries, you just leave all the options unchecked for that event type.

That way you could, for example, have retryable failures post a message to a Slack channel, but page an on-call person if all the retries fail. An option to suppress notifications until the last attempt would definitely be better than the current state of affairs but would make it more awkward to set that kind of two-tiered system up.