Cache unchecked count #2458
Merged
Cache unchecked count #2458
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wezrule
added
performance
guilhermelawless
approved these changes
Jan 2, 2020
clemahieu
approved these changes
Jan 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
During bootstrapping with RocksDB (collecting all unchecked blocks), 27% of the time was spent checking whether a block exists and updating the unchecked count cache in the ledger. This extra IO was necessary because RocksDB cannot give you accurate counts of the number of keys in a table so it was expensive to determine the unchecked count on the fly. However this can just be done at startup as for the most part this number will actually be low and then just maintain an in-memory cache of the unchecked number, like we now do with: block count, cemented count and rep weights.
This improves bootstrapping substantially for the first phase of collecting all unchecked blocks, but processing unchecked -> checked afterwards is still slow (
unchecked_getconstitutes 50% of the CPU time), this will be addressed separately (if possible).Some helper classes have been added to combine all the cacheable things we have in the ledger