[FLINK-6766] Update documentation about async backends and incrementa…#4011
Closed
StefanRRichter wants to merge 1 commit intoapache:masterfrom
Closed
[FLINK-6766] Update documentation about async backends and incrementa…#4011StefanRRichter wants to merge 1 commit intoapache:masterfrom
StefanRRichter wants to merge 1 commit intoapache:masterfrom
Conversation
StefanRRichter
force-pushed
the
backend-docu-update
branch
from
9f26592 to
b782af5
Compare
Contributor
Author
|
CC @rmetzger |
StefanRRichter
force-pushed
the
backend-docu-update
branch
from
b782af5 to
f4f4e1e
Compare
alpinegizmo
requested changes
May 31, 2017
Contributor
alpinegizmo
left a comment
alpinegizmo
left a comment
There was a problem hiding this comment.
A couple of minor spelling/grammar changes.
|
|
||
| Incremental checkpoints can dramatically reduce the checkpointing time in comparison to full checkpoints, at the cost of a (potentially) longer | ||
| recovery time. The core idea is that incremental checkpoints only record all changes to the previous completed checkpoint, instead of | ||
| producing a full, self-contained backups of the backend. Like this, incremental checkpoints build upon previous checkpoints. Flink leverages |
Contributor
There was a problem hiding this comment.
a full, self-contained backup of the state backend
|
|
||
| - As RocksDB's JNI bridge API is based on byte[], the maximum supported size per key and per value is 2^31 bytes each. | ||
| IMPORTANT: states that use merge operations in RocksDB (e.g. ListState) can silently accumulate value sizes > 2^31 bytes and will then fail on their next retrival. This is currently a limitation of RocksDB JNI. | ||
|
|
docs/ops/state_backends.md
Outdated
| Limitations of the RocksDBStateBackend: | ||
|
|
||
| - As RocksDB's JNI bridge API is based on byte[], the maximum supported size per key and per value is 2^31 bytes each. | ||
| IMPORTANT: states that use merge operations in RocksDB (e.g. ListState) can silently accumulate value sizes > 2^31 bytes and will then fail on their next retrival. This is currently a limitation of RocksDB JNI. |
…remental checkpoints
StefanRRichter
force-pushed
the
backend-docu-update
branch
from
f4f4e1e to
cf7ed16
Compare
Contributor
Author
|
Thanks for the input @alpinegizmo ! I updated the text accordingly. |
Contributor
|
I'll merge the change. |
asfgit
pushed a commit
that referenced
this pull request
May 31, 2017
…remental checkpoints This closes #4011
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…l checkpoints
This PR introduces some documentation about async heap backends and incremental snapshots with RocksDB.