Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add better task retry logging #13870

Merged
merged 2 commits into from
Jun 7, 2024
Merged

Add better task retry logging #13870

merged 2 commits into from
Jun 7, 2024

Conversation

zzstoatzz
Copy link
Collaborator

closes #13864 - which outlines that retry logging is not so good in main right now in 3.x, this PR cleans it up a bit

examples

retries with no delay

In [1]: from prefect import task

In [2]: @task(retries=5)
   ...: def terrible():
   ...:     raise ValueError
   ...:

In [3]: terrible()
11:34:05.523 | INFO    | prefect.engine - Created task run 'terrible-terrible' for task 'terrible'
11:34:06.132 | INFO    | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retry 1/5 will start immediately
11:34:06.512 | INFO    | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retry 2/5 will start immediately
11:34:06.830 | INFO    | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retry 3/5 will start immediately
11:34:07.153 | INFO    | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retry 4/5 will start immediately
11:34:07.520 | INFO    | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retry 5/5 will start immediately
11:34:07.925 | ERROR   | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retries are exhausted
11:34:08.046 | ERROR   | Task run 'terrible-terrible' - Finished in state Failed('Task run encountered an exception ValueError: ')

retries with delay

In [5]: @task(retries=5, retry_delay_seconds=[1, 3])
   ...: def terrible():
   ...:     raise ValueError
   ...:

In [6]: terrible()
11:34:39.659 | INFO    | prefect.engine - Created task run 'terrible-terrible' for task 'terrible'
11:34:40.134 | INFO    | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retry 1/5 will start 1 second(s) from now
11:34:41.474 | INFO    | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retry 2/5 will start 3 second(s) from now
11:34:44.825 | INFO    | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retry 3/5 will start 3 second(s) from now
11:34:48.286 | INFO    | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retry 4/5 will start 3 second(s) from now
11:34:51.668 | INFO    | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retry 5/5 will start 3 second(s) from now
11:34:55.043 | ERROR   | Task run 'terrible-terrible' - Task run failed with exception ValueError() - Retries are exhausted
11:34:55.151 | ERROR   | Task run 'terrible-terrible' - Finished in state Failed('Task run encountered an exception ValueError: ')

@zzstoatzz zzstoatzz added the enhancement An improvement of an existing feature label Jun 7, 2024
@zzstoatzz zzstoatzz requested a review from a team as a code owner June 7, 2024 16:35
@zzstoatzz zzstoatzz self-assigned this Jun 7, 2024
@zzstoatzz zzstoatzz changed the title better retry logging Add better task retry logging Jun 7, 2024
@zzstoatzz zzstoatzz merged commit 0233add into main Jun 7, 2024
24 of 25 checks passed
@zzstoatzz zzstoatzz deleted the log-retries branch June 7, 2024 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tasks should log when they're retrying
2 participants