Handle dask exceptions at the futures layer #17725
Conversation
| # handle dask specific exceptions that indicate a problem with this task | ||
| self._final_state = State( | ||
| type="CRASHED", | ||
| data=exc, | ||
| message=f"Encountered an unexpected exception from dask: {exc!r}", | ||
| ) |
There was a problem hiding this comment.
Do you think we could update the task state here so that the task shows as CRASHED in the UI?
There was a problem hiding this comment.
What's the best way to do that -- emit_event?
There was a problem hiding this comment.
I think prefect.utilities.engine.emit_task_state_change_event is the easiest way to do that.
|
This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment. |
|
This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment. |
|
This pull request was closed because it has been stale for 14 days with no activity. If this pull request is important or you have more to add feel free to re-open it. |
This PR handles some "expected" exceptions that dask might raise when waiting on future resolution. This addresses #17384 but only partially: the flow run is correctly marked as
Crashedwith all exceptions handled gracefully, but the task run remains in aRunningstate indefinitely. I think we need server side handling of crashes to correctly mark all stillRunningtasks appropriately, but that's for another day.Closes #17384