LNS: Commit to db more conservatively instead of every block #1152
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Instead of committing every block, commit everytime we know the DB has changed and on shutdown. Upon profiling, every block added was taking an additional ~9ms from HF15 because of `save_settings(...)` writing the block hash and height to the DB. If the daemon crashes during operation, then LNS only needs to scan empty blocks up until that point which shouldn't be too slow and the exception, not the norm.
Doy-lee
force-pushed
the
LNSCommitConservatively
branch
from
737d4db
to
3be43ea
Compare
|
9ms seems like an eternity, and sounds more like disk IO time than CPU time. It might be caused by the sqlite synchronization settings: it might be worth seeing if WAL mode + synchronous = NORMAL helps here (it would probably help with all write queries, not just this one). |
|
It is disk IO, problem is completely gone if you write to an in memory DB. With WAL + synchronous, standalone it also similarly brings performance inline with the current PR. Combined together shaves off another 100ms or so for processing 1k blocks (as of so far, rescanning the LNS db on the Mainnet). |
Doy-lee
force-pushed
the
LNSCommitConservatively
branch
from
80ec3f8
to
7f0c33a
Compare
Doy-lee
force-pushed
the
LNSCommitConservatively
branch
from
7f0c33a
to
a57c75c
Compare
jagerman
approved these changes
May 15, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Instead of committing every block, commit everytime we know the DB has
changed and on shutdown. Upon profiling, every block added was taking an
additional ~9ms from HF15 because of
save_settings(...)writing theblock hash and height to the DB.
If the daemon crashes during operation, then LNS only needs to scan
empty blocks up until that point which shouldn't be too slow and the
exception, not the norm.