Re-implement merkletree with persistent storage (key-value db)#487
Merged
Re-implement merkletree with persistent storage (key-value db)#487
Conversation
arnaucube
commented
Mar 9, 2026
ed255
reviewed
Mar 9, 2026
Collaborator
ed255
left a comment
ed255
left a comment
There was a problem hiding this comment.
Overall looks good to me, but I would like to discuss the DB api
Contributor
|
What was the approach picked for garbage collection? |
Collaborator
In this implementation there's no garbage collection. We defer to finding a solution for that (which includes figuring out an API for it) in the future. |
Collaborator
Author
To add on @ed255 's answer; a straight forward approach (which might not fit all use cases) would be exporting the leafs of the tree under the current root, and creating a new tree with them in a new db; getting rid of all the unused data in the process. |
ed255
reviewed
Mar 10, 2026
ed255
approved these changes
Mar 11, 2026
ed255
added a commit
that referenced
this pull request
Mar 23, 2026
Extend the work of #487 to the Containers (Dictionary, Set, Array). The merkle tree only stores `RawValue` for both the key and the value, so it is the responsibility of the Container to store the rich value. In order to handle containers with persistent storage efficiently (which means, cloning them or updating them should not cause an O(n) data copy) I figured we need to have a database of `Value`s indexed by their raw value; as this gives us deduplication and free cloning of containers. The issue with this approach is that in the current design we have collisions between Value's of different types: #426 and the current API relies on the single type of values. To resolve this issue I decided to change the API, instead of assuming that a Value has a fixed type, let the value be possibly multiple compatible types and let the user of the library try casting the Value to a particular type. For this I deprecated the public access of everything related to `TypedValue` and I propose for it to be considered an implementation detail and a blackbox from the external developer point of view. The `Value` type is now used like this: - To create a new Value use `Value::from(...)` where you can pass any compatible type (the same types as before) - To access the Value in typed form you cast it like `value.as_foo()` which returns `Option<Foo>`. Previously we had a collision between `true` and `1` (and `false` and `0`). Now it doesn't matter whether a value holds a `true` or a `1`, both should be seen as the same and both return `Some` when doing `as_int` and `as_bool`. Similarly we had collisions with containers. For example `set(0, 1, 2) == array[0, 1, 2]` and `set("a", "b") = dict("a": "a", "b": "b")`. Now any container can be casted to any of `set, array, dict`. There's a caveat here: each of these types expects a particular encoding of keys, so casting to the wrong type will return errors on some operations. With this design it no longer matters what is being stored and recovered because the API requires the user to express the expected type and any type with collisions for particular values can be casted to the right type. There's only one case where it's not desirable to swap one `TypedValue` for another: the `TypedValue::Raw`. If a non-`RawValue` in the DB is replaced by the corresponding `RawValue` we erase the required information to recover the rich value. For this reason the implementations of the database treat the `RawValue` as a special case: if an value is stored in non-`RawValue`, the corresponding `RawValue` can never overwrite it. If a value is stored in `RawValue`, a matching non-`RawValue` will overwrite it (promoting it to a rich value). This way we never lose data. A consequence of this is that the serialization, `Display` and `Debug` of a container is not stable. At any point any of the entries can be swapped for a "compatible" one if they share the storage with other containers that introduce collisions. I rewrote all containers as wrapper to a generic `Container` which holds a `Map` from `Value` to `Value`. The serialization of each container now uses the single implementation of the generic `Container`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Joint work with @ax0 .
This PR re-implements the merkletree to allow it to use a key-value database in-disk. Needed a reimplementation of the core of the tree logic, since the previous implementation was designed to first build the tree structure in memory and at the end of the operations to compute the hashes, which in a key-value db would not be ideal.
The database interface is defined by the
DBtrait, which works with (atomic) transactions, and we added a rocksdb implementation of it as an example ready to use (aside from a naive in-memory db).We also extend the tests of the merkletree to cover more edge cases (for future iterations to catch potential issues).
resolves #435 , resolves #439 (this is done by setting the
divergence_leveltoMAX_DEPTHwhennew_siblings.len()is zero)