Support custom delays for automatic retries

[aidmsu]

This PR adds ability to specify custom delays for retries via AutomaticRetryAttribute.

Specify custom delays by an integer array.

You can set a delay for a job:

// All delays are the same and equal 10 seconds.
[AutomaticRetry(DelaysInSeconds = new [] { 10 })]
public void SomeJob() { ... }

// First delay is 1 second, second delay is 5 seconds, other delays are 10 seconds.
[AutomaticRetry(DelaysInSeconds = new [] { 1, 5, 10 })]
public void SomeJob() { ... }

Also it's possible set a delay for all jobs:

GlobalJobFilters.Filters.Add(new AutomaticRetryAttribute { DelaysInSeconds = new { 1,5,100 }});

Specify custom delay by a function.

If you want to use complex algorithm to define a delay by a attempt you should set AutomaticRetryAttribute.DelayInSecondsByAttemptFunc property:

// Delays will be 1, 4, 9 seconds and so on.
var func = attempt => attempt * attempt;
GlobalJobFilters.Filters.Add(new AutomaticRetryAttribute { DelayInSecondsByAttemptFunc = func});

Behaviour

If you specifty both DelaysInSeconds and DelayInSecondsByAttemptFunc then only DelaysInSeconds will be used.

If given DelayInSecondsByAttemptFunc throws excecption then the default function will be used.

[codecov-io]

Codecov Report

Merging #931 into dev will increase coverage by 0.34%.
The diff coverage is 88.88%.

Impacted Files	Coverage Δ
src/Hangfire.Core/AutomaticRetryAttribute.cs	86.66% <88.88%> (+6.38%)	⬆️
src/Hangfire.Core/DashboardOptions.cs	90.9% <0%> (+90.9%)	⬆️

[aidmsu]

I reconsidered behaviour in case a user code threw exception inside AutomaticRetryAttribute. I came to a conclusion it's better to rethrow the exception and as a consequence to apply Failed state to a job. Main reason is a user changes default behavior of AutomaticRetryAttribute for some reason and it may be critical for her not to use default behaviour.

[odinserj]

What if 0 value is given? AFAIK currently it will be created in the ScheduledState with the DateTime.UtcNow argument. I think we can create it using the EnqueuedState instead to avoid unnecessary state changes. What do you think?

[aidmsu]

@odinserj You are right, if 0 value is given we should apply EnqueuedState instead of ScheduledState. I'll fix it.

[odinserj]

Great @aidmsu! Pay special attention to the retries counter, there may be dragons 😉

[odinserj]

Can't get used to reviews, waiting for EnqueuedState transition on zero delay.

[aidmsu]

@odinserj I fixed it. AutomaticRetryAttribute applies EnqueuedState instead of ScheduledState when the delay is zero.

[hQst]

This feature is nice but you can't define a DelayInSecondsByAttemptFunc in the GlobalFilters and put the Attribute on the Job. Because then the DefaultDelayInSecondsByAttemptFunc will be used. And due to the nature of an attribute you can't defined the Func in the attribute.

[camiller2]

@odinserj / any other admin: Is there an update on whether (and if so when) this will be merged into master and released? Looking at https://github.com/HangfireIO/Hangfire/compare/dev it appears that this has not been released, but I could be wrong.

[mauriciojsola]

Hey @odinserj, any ETA for this feature to be released? Being able to adjust the retry delay times is really useful on some scenarios. Thanks!

[camiller2]

I've created a version of AutomaticRetryAttribute based off of this version that allows DefaultDelayInSecondsByAttemptFunc (and all the other properties of the attribute) to be configured at runtime on an attribute. This is possible via an autofac-hangfire filter provider integration I wrote and a ConfigKey. Assuming this integration has been enabled, the API looks like this:

[AutomaticRetryV2(ConfigKey = MyClass.ConfigKey, OnAttemptsExceeded = AttemptsExceededAction.Delete)]
        Task MyMethod(long someId);

// In your autofac registration:
builder.Register(cc => new Dictionary<string, AutomaticRetryV2AttributeConfig>
        {
            {
                MyClass.ConfigKey,
                new AutomaticRetryV2AttributeConfig
                {
                    DelaysInSeconds = _settings.MyMethodDelayBetweenAttempts.HasValue
                        ? new[] {_settings.MyMethodDelayBetweenAttempts.Value}
                        : null,
                    Attempts = _settings.MyMethodRetryAttempts
                }
            }
        })
        .As<IDictionary<string, AutomaticRetryV2AttributeConfig>>()
        .SingleInstance();

Let me know if anybody's interested, and I'd be happy to create a PR for the attribute update. I'm planning on creating a PR for the required autofac-hangfire filter provider integration in https://github.com/HangfireIO/Hangfire.Autofac before too long.

[zuckerthoben]

Any updates on this?

[tho-mas]

It would be very nice if this PR can be included in the next release.

[aprooks]

It's published in 1.7-beta.

[SimonOrdo]

Am I reading this correctly, that there is no way to specify a delay algorithm on a per-job basis? I.e. it's either global, for all jobs, or static?

I need to implement an exponential back-off with jitter delay algorithm, but it should only be applied to specific jobs.

[HristoDimitrovBede]

@SimonOrdo I am looking for the same. I think it isn't implemented yet.

[jonathancounihan]

@SimonOrdo , @HristoDimitrovBede did you ever manage to implement an exponential back-off with jitter delay algorithm for specific jobs?
Thanks.