[FLINK-3466] [runtime] Cancel state handled on state restore #2252
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| * A simple base for closable handles. | ||
| * | ||
| * Offers to register a stream (or other closable object) that close calls are delegated to if | ||
| * the handel is closed or was already closed. |
There was a problem hiding this comment.
Thanks, will fix it.
|
Looks very good! The test failures seem unrelated:
The added tests and refactorings are very readable. I think this is good to merge mod some minor inline comments. |
|
Thanks, I'll address your comments and merge this... |
StephanEwen
force-pushed
the
state_handle_cancellation
branch
from
c411b37 to
ff52e0e
Compare
State handles are cancelable, to make sure long running checkpoint restore operations do finish early on cancallation, even if the code does not properly react to interrupts. This is especially important since HDFS client code is so buggy that it deadlocks when interrupted without closing.
StephanEwen
force-pushed
the
state_handle_cancellation
branch
from
ff52e0e to
a340bf2
Compare
|
Manually merged in e9f660d |
|
Is this related to https://issues.apache.org/jira/browse/HADOOP-14214? Thanks, |
|
I think it is yes. We worked around it in the meantime... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request fixes the issue that state restore operations can get stuck when tasks are cancelled during state restore. That happens due to a bug in HDFS, which deadlocks (or livelocks) when the reading thread is interrupted.
This introduces two things:
Closable. This does not delete any checkpoint data, but simply closes pending streams and data fetch handles. Operations concurrently accessing the state handles state should fail.StreamTaskholds a set of "Closables" that it closes upon cancellation. This is a cleaner way of stopping in-progress work than relying on "interrupt()" to interrupt that work.This mechanism should eventually be extended to also cancel operators and state handles pending asynchronous materialization.
There is a test that has an interrupt sensitive state handle (mimicking HDFS's deadlock behavior) that causes a stall without this pull request and cleanly finishes with the changes in this pull request.
This also adds a test validating that all state handled and key/value snapshots add a proper
serialVersionUID.