[+] add retry logic to tasks and chains

[amihajlovic]

Is your feature request related to a problem? Please describe.
We are using chains scheduled to run every couple of hours. We are occasionally encountering transient issues while processing a sql task (eg. timeout, deadlock, data yet not available for processing etc). With current implementation we either need to wait until scheduler triggers the chain again in a couple of hours or we need to schedule it more frequently and move some scheduling logic to code.

Describe the solution you'd like
The solution would be to include retry configuration and logic on both task and chain level.
Minimal configuration options would be - maximum_retry_count (smallint) and minimum_retry_timeout (interval).
Ideally, there would be two more options - retry_backoff_enabled(boolean), retry_backoff_steps and maximum_retry_duration (interval). Which would allow for more complex retry configurations with incrementally increasing retry timeouts up to maximum_retry_duration.

Describe alternatives you've considered
Hacking around those tasks and chains by running them on a more frequent schedule (* * * * *) and then resolving retry / scheduling logic based on custom config in db.
Creating a wrapper and invoking those sql functions as a program task.

While both alternatives will work, it would be far better to have this logic built-in in the scheduler itself.
Similar functionality exists in other schedulers we're using for other purposes -> airflow, sql server job agent.

[pashagolub]

Hey, thanks a lot for your detailed explanation. We had something similar on mind but never implemented because of low demand. Now we can give it more attention. :)

[github-actions]

📅 This issue has been automatically marked as stale because lack of recent activity. It will be closed if no further activity occurs.
♻️ If you think there is new information allowing us to address the issue, please reopen it and provide us with updated details.
🤝 Thank you for your contributions.

[pashagolub]

Still on ToDo list...

[pashagolub]

Thanks for your proposal. It's a great idea. I decided not to implement it straightforwardly but instead use another functionality to solve this task. You can check #578 on how to use new OnError machinery to implement chain retries.