Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
[Flink 6537] Fixes and improvements for incremental checkpoints in RocksDB #3870
This PR bundles several fixes and improvements for incremental checkpoints in RocksDB.
In particular, this addresses:
Extended test coverage will be provided as part of [FLINK-6540].
Some of the main changes are in the way the
These logs might help:
Seems like the SavepointV2Serializer can't handle the Incremental rocks handle because it's not a KeyGroupsStateHandle.
You can see both the savepoint error (first error) and a regular incremental checkpoint error (second)
@gyfora Yes, this is still a limitation, because the incremental checkpoints are currently not (yet) externalizable. The missing piece is, exactly as you said, familiarizing the
I created a jira to track this: https://issues.apache.org/jira/browse/FLINK-6545
@StefanRRichter I think these changes well address the problems existed in incremental checkpoint in
RocksDBKeyedStateBackend. Thanks a lot for your work.
I made some inline comments. Among them, the usage of asynchronous executors in
SharedStateRegistry is most concerned. I think we should avoid the usage of asynchronous threads as much as possible. I prefer to utilize existing asynchronous executor to discard unreferenced shared states. Kindly let me know if you have any idea of it.