-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[+] add retry logic to tasks and chains #518
Comments
Hey, thanks a lot for your detailed explanation. We had something similar on mind but never implemented because of low demand. Now we can give it more attention. :) |
📅 This issue has been automatically marked as stale because lack of recent activity. It will be closed if no further activity occurs. |
Still on ToDo list... |
Thanks for your proposal. It's a great idea. I decided not to implement it straightforwardly but instead use another functionality to solve this task. You can check #578 on how to use new |
[+] add a sample of a retry logic for chains, closes #518
Is your feature request related to a problem? Please describe.
We are using chains scheduled to run every couple of hours. We are occasionally encountering transient issues while processing a sql task (eg. timeout, deadlock, data yet not available for processing etc). With current implementation we either need to wait until scheduler triggers the chain again in a couple of hours or we need to schedule it more frequently and move some scheduling logic to code.
Describe the solution you'd like
The solution would be to include retry configuration and logic on both task and chain level.
Minimal configuration options would be - maximum_retry_count (smallint) and minimum_retry_timeout (interval).
Ideally, there would be two more options - retry_backoff_enabled(boolean), retry_backoff_steps and maximum_retry_duration (interval). Which would allow for more complex retry configurations with incrementally increasing retry timeouts up to maximum_retry_duration.
Describe alternatives you've considered
Hacking around those tasks and chains by running them on a more frequent schedule (* * * * *) and then resolving retry / scheduling logic based on custom config in db.
Creating a wrapper and invoking those sql functions as a program task.
While both alternatives will work, it would be far better to have this logic built-in in the scheduler itself.
Similar functionality exists in other schedulers we're using for other purposes -> airflow, sql server job agent.
The text was updated successfully, but these errors were encountered: