New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add handling for statement timeouts to state validation #8425
Conversation
✅ Deploy Preview for prefect-orion ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
89d6d34
to
906ca0e
Compare
if self.run.state is None: | ||
self.validated_state = None | ||
return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was previous behavior but avoids calling self.session.add(None)
which throws an exception
Superseded by https://github.com/PrefectHQ/nebula/pull/3482 |
Getting a 404 upon clicking this link. |
@ccueto36 that's our internal repository — we're fixing this in Cloud but it does not need a fix in the open source. |
Got it. Thanks for the clarification. Looking forward to the fix. |
Closes #8435
Database query timeouts are resulting in an ABORT response from the API:
Previously, these errors resulted in a 503 response allowing the client to retry. Since an aborted task run can crash the flow and these timeouts are happening frequently, customers are blocked by the lack of retry. It seems likely that this regression is due to a change from #8164 where error handling in state validation was fixed.
Here, we explicitly do not capture statement timeouts and instead reraise them. We should probably we doing cleanup instead of just raising an exception, but unfortunately this seems quite complicated and we should first attempt to achieve the previous behavior.
This pull request extends #8423 which adds baseline test coverage for this area.
Example
Discussion at https://prefecthq.slack.com/archives/C04F1PTMUCQ/p1675455197515479
User report at https://prefect-community.slack.com/archives/CL09KU1K7/p1675449149527989
Checklist
<link to issue>
"fix
,feature
,enhancement