Persist temp states to not lose sync progress #2151
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dapplion
requested review from
mpetrunic,
twoeths and
wemeetagain
as code owners
|
Code Climate has analyzed commit 9c3b043 and detected 0 issues on this pull request. View more on Code Climate. |
twoeths
reviewed
Mar 10, 2021
dapplion
force-pushed
the
dapplion/persist-more-states
branch
2 times, most recently
from
8dd05e2
to
a765f67
Compare
Address PR comments
dapplion
force-pushed
the
dapplion/persist-more-states
branch
from
a765f67
to
9c3b043
Compare
|
Bump, until we fix all our memory issues the problem this PR mitigates is very annoying, in this case the node is spending 10% of the time re-syncing lost progress
|
wemeetagain
approved these changes
Mar 16, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Currently, we store a state every 1024 epochs. At slow sync speeds of 4 slots / second, that's +2 hours of sync progress lost. Fixes #2048
Proposed solution
Store state more frequently, every 32 epochs. Then, after a successful store, keep only the first state every 1024 epochs and delete the rest. In normal conditions, only 0 or 1 states will be deleted. The resulting layout of store states will look like