New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-13326] Support asynchronous writing to raw operator (and raw keyed) state #9171
Conversation
This commit adds the ability for users to indicate to Flink that the KeyedStateCheckpointOutputStream / OperatorStateCheckpointOutputStream should be closed at some point later in time. Previously it was expected to be closed within the synchrouns part of AbstractStreamOperator#snapshotState. Users might call KeyedStateCheckpointOutputStream#acquireLease(), and at some point in the future might release the least. Once the number of leases reaches 0, the stream would be closed.
This commit adds the ability for users to obtain a lease for the rawOperator/rawKeyed output streams, during the synchronous part of a snapshot, thus preventing these streaming to close.
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit 27108b5 (Tue Aug 06 15:44:29 UTC 2019) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
@flinkbot attention @StefanRRichter |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your contribution @igalshilman ! Changes look good to me, just tell me if/when you would like to have them merged to master :-)
@StefanRRichter thanks! I'm ok with merging it. |
What is the purpose of the change
Currently, the users of the
AbstractStreamOperator
class, are expected to complete the writing ofrawKeyedOperatorState
andrawOperatorState
in the synchronous part of a snapshot (before the completion ofsnapshotState
method)But sometimes it is desirable to snapshot a custom data structure asynchronously (i.e. out side of the snapshotState method) in a similar way that the other state backends allow.
This PR adds the ability to suspend the closing of the raw states streams in
snapshotState
outside of the snapshotState method, and hence effectively support async snapshots for raw states.Flink snapshotting logic already supports splitting the snapshot to sync and async parts for the various state backends, therefore the changes introduced in this PR mainly reuse that logic.
Brief change log
ResourceGuard
(the default is still unInterruptible).NonClosingCheckpointOutputStream
.StateSnapshotContextSynchronousImpl#getRaw*OperatorStateOutput
to return a future that completes after all the leases were released.Verifying this change
This change is already covered by existing tests.
Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (yes / no)Documentation