go/storage: Refactor checkpointing interface #2659
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kostko
force-pushed
the
kostko/feature/storage-checkpoints-v2
branch
2 times, most recently
from
b28b016
to
c48e15d
Compare
Codecov Report
@@ Coverage Diff @@
## master #2659 +/- ##
=========================================
- Coverage 63.15% 62.95% -0.2%
=========================================
Files 370 376 +6
Lines 34910 35314 +404
=========================================
+ Hits 22046 22231 +185
- Misses 10111 10306 +195
- Partials 2753 2777 +24
Continue to review full report at Codecov.
|
kostko
force-pushed
the
kostko/feature/storage-checkpoints-v2
branch
2 times, most recently
from
8935774
to
3c6918d
Compare
kostko
force-pushed
the
kostko/feature/storage-checkpoints-v2
branch
3 times, most recently
from
2fe6db0
to
3b88346
Compare
4 tasks
Yawning
approved these changes
Feb 14, 2020
kostko
force-pushed
the
kostko/feature/storage-checkpoints-v2
branch
from
3b88346
to
6f5cb00
Compare
Previously the way storage checkpoints were implemented had several drawbacks, namely: - Since the checkpoint only streamed key/value pairs this prevented correct tree reconstruction as tree nodes also include a Round field which specifies the round at which a given tree node was created. - While old checkpoints were streamed in chunks and thus could be resumed or streamed in parallel from multiple nodes, there was no support for verifying the integrity of a single chunk. This change introduces an explicit checkpointing mechanism with a simple file-based backend reference implementation. The same mechanism could also be used in the future with Tendermint's app state sync proposal.
kostko
force-pushed
the
kostko/feature/storage-checkpoints-v2
branch
from
6f5cb00
to
46f06ef
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #2479
Previously the way storage checkpoints were implemented had several
drawbacks, namely:
Since the checkpoint only streamed key/value pairs this prevented
correct tree reconstruction as tree nodes also include a Round field
which specifies the round at which a given tree node was created.
While old checkpoints were streamed in chunks and thus could be
resumed or streamed in parallel from multiple nodes, there was no
support for verifying the integrity of a single chunk.
This change introduces an explicit checkpointing mechanism with a simple
file-based backend reference implementation. The same mechanism could
also be used in the future with Tendermint's app state sync proposal.
NOTE: This adds support for generating checkpoints but not for syncing them.