New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merkledb split node storage into value and intermediate #1918
Conversation
Can we please update the PR description to explain a bit more what this does, how it works, and why we're making this change? |
Co-authored-by: Dan Laine <daniel.laine@avalabs.org> Signed-off-by: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com>
Co-authored-by: Dan Laine <daniel.laine@avalabs.org> Signed-off-by: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com>
Co-authored-by: Dan Laine <daniel.laine@avalabs.org> Signed-off-by: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com>
…s/avalanchego into MerkleDBSplitNodeStorage
…s/avalanchego into MerkleDBSplitNodeStorage
…chego into MerkleDBSplitNodeStorage
…s/avalanchego into MerkleDBSplitNodeStorage
x/merkledb/db.go
Outdated
minRebuildViewSizePerCommit = 1000 | ||
rebuildViewSizeFractionOfCacheSize = 50 | ||
minRebuildViewSizePerCommit = 1000 | ||
rebuildIntermediateDeletionWriteSize = 10 * units.KiB |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is a reasonable value for this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
used here to clear the intermediate node db
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe 1 units.MiB?
@@ -174,6 +174,7 @@ func AtomicClear(readerDB Iteratee, deleterDB KeyValueDeleter) error { | |||
return AtomicClearPrefix(readerDB, deleterDB, nil) | |||
} | |||
|
|||
// AtomicClearPrefix deletes from [deleterDB] all keys in [readerDB] that have the given [prefix]. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to keep this even though it's currently unused? cc @StephenButtolph
Separate the intermediate nodes(nodes without an associated value) from the value nodes within the database by adding a prefix based on which type of node it is. By splitting them up, we can more easily iterate over key/values. This work allows us to store keys in a more compressed format with full bytes instead of one nibble per byte. Additionally, it splits the cache into two parts to allow for better control over how many value nodes vs intermediate nodes are stored within memory.