New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-8567] Do not swallow execution errors during checkpointing #10008
Conversation
Unrelated Dataflow failures:
|
invokeFinishBundle(); | ||
} | ||
outputManager.closeBuffer(); | ||
} catch (Exception e) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe also create a JIRA in Flink to have a cleaner way to signal an application error?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would make sense. Normally, there should be no application logic in here, though there are probably other people who have similar logic in snapshotState
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please also see #10007. |
} | ||
outputManager.closeBuffer(); | ||
} catch (Exception e) { | ||
// Any regular exception during checkpointing will be tolerated by Flink because those |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// Any regular exception during checkpointing will be tolerated by Flink because those | |
// https://jira.apache.org/jira/browse/FLINK-14653 | |
// Any regular exception during checkpointing will be tolerated by Flink because those |
If a bundle fails to finalize before creating a checkpoint, it may be swallowed and just considered a checkpointing error. This breaks the execution flow and exactly-once guarantees.
Run Portable_Python PreCommit |
If a bundle fails to finalize before creating a checkpoint, it may be swallowed
and just considered a checkpointing error. This breaks the execution flow and
exactly-once guarantees.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.