Added logic to extract/validate MPT proof paths. #2989

HristoStaykov · 2023-04-07T13:20:01Z

We have added logic to extract proof paths from our MPT implementation and to validate that a previously extracted proof path reaches a giver root hash. A unit test is added that fills an MPT and then validates the paths extracted for every key/value inserted.

Problem Overview
Completed implementation of extraction/verification of proof paths from our MPT implementation.
Testing Done
A new unit test was added that fills a MPT and then every inserted key/value is checked.

kvbc/test/sparse_merkle/tree_test.cpp

kvbc/include/sparse_merkle/internal_node.h

kvbc/include/sparse_merkle/base_types.h

vrancurel · 2023-04-12T01:21:20Z

kvbc/src/sparse_merkle/internal_node.cpp

  constexpr size_t level_0_start = MAX_CHILDREN / 2;
  return level_0_start + nibble.data();
 }

-size_t BatchedInternalNode::height(size_t index) const {
+size_t BatchedInternalNode::height(size_t index) {


Is it possible to document this strange function ?

It is documented in the .h file, basically we pass the index of a child from the array children_, containing the 31 elements, and the return value is the height in the sub-tree. internal_node.h:328

It is one of the functions that I changed to static, because they define the structure of the batched internal node and don't need to access the state of a particular instance.

It still does not explain the numbers below:

kvbc/src/sparse_merkle/proof_path_processor.cpp

kvbc/include/sparse_merkle/internal_node.h

kvbc/include/sparse_merkle/proof_path_processor.h

kvbc/src/sparse_merkle/proof_path_processor.cpp

We have added logic to extract proof paths from our MPT implementation and to validate that a previously extracted proof path reaches a giver root hash. A unit test is added that fills an MPT and then validates the paths extracted for every key/value inserted.

…umber of children. In verifyProofPath added condition for early termination of directions calculation.

…Node.

WildFireFlum · 2023-04-24T14:44:50Z

kvbc/src/sparse_merkle/proof_path_processor.cpp

  for (size_t nibble_index = 0; nibble_index < Hash::MAX_NIBBLES; nibble_index++) {
    auto current_nibble = keyHash.getNibble(nibble_index).data();
    auto node_index = BatchedInternalNode::nibbleToIndex(current_nibble);
-    size_t foundElemens = 0;
+    subPath.clear();


Why is this needed?

This clearing is needed, because we reuse the same container when we iterate to extract the directions based on the nibbles of the key for each sub-record of the tree.

WildFireFlum · 2023-04-24T14:45:23Z

kvbc/src/sparse_merkle/proof_path_processor.cpp

@@ -109,7 +113,7 @@ std::vector<Hash> getProofPath(Sliver key, std::shared_ptr<IDBReader> db, const
    auto child = node.children()[current_index];
    nextNodetype = Nodetype::None;
    bool firstInternalNodeFount = false;
-    size_t foundElemens = 0;
+    hashesCollectedFromInternalNode.clear();


Why is this needed?

Same as above, but here we actually go through multiple BatchedInternalNodes to collect the necessary hashes from each one and for all iterations we reuse hashesCollectedFromInternalNode.

nkgeorgiev · 2023-04-25T08:25:39Z

kvbc/include/sparse_merkle/internal_node.h

@@ -97,6 +97,11 @@ std::ostream& operator<<(std::ostream& s, std::optional<T> const& v) {
  return s;
 }

+static constexpr size_t BatchedInternalNodeGetMaxHeight(const size_t children) {
+  if (children == 0) return 0;
+  return 1 + BatchedInternalNodeGetMaxHeight(children / 2);


You do know that this is just a log2, right?

Yes, this is only called during compile time and the max number of children in 1 record is currently 31. We don't expect it to go to values that can actually cause slow down in compilation.

* Added logic to extract/validate MPT proof paths. We have added logic to extract proof paths from our MPT implementation and to validate that a previously extracted proof path reaches a giver root hash. A unit test is added that fills an MPT and then validates the paths extracted for every key/value inserted. * Const correctness additions. * Fix for arguments ordering. * Added documentation comments. * Reduce allocations for better performance. * Added constexpr function to calculate the number of levels from the number of children. In verifyProofPath added condition for early termination of directions calculation. * Fix gcc build errors. * Changed vectors to arrays for subPath and hashesCollectedFromInternalNode. * Convert all containers used to boost::static_vector.

HristoStaykov requested review from vrancurel, WildFireFlum, teoparvanov, cloudnoize, toly-kournik and eyalrund April 7, 2023 13:20

HristoStaykov requested a review from a team as a code owner April 7, 2023 13:20

vmwclabot added the cla-not-required label Apr 7, 2023

HristoStaykov requested a review from nkgeorgiev April 7, 2023 13:23

vrancurel suggested changes Apr 12, 2023

View reviewed changes

vrancurel approved these changes Apr 12, 2023

View reviewed changes

cloudnoize force-pushed the storage/changes_for_eth_support branch from fe408d5 to cc3f7db Compare April 13, 2023 10:18

HristoStaykov force-pushed the proof_paths_from_mpt_processing branch from ae461ec to dc2a0d9 Compare April 13, 2023 10:39

teoparvanov approved these changes Apr 13, 2023

View reviewed changes