PMMR segment creation and validation

[jaspervdm]

This PR implements a (P)MMR "segment", defined by this WIP RFC. In short, a segment is a set of 2**b (with variable b) consecutive leaves along with the necessary data to reconstruct the subtree root and data to verify membership of the subtree in the original MMR (a Merkle proof). In case of a prunable MMR, the segment only contains the unpruned leaves in the segment range and also contains intermediary MMR hashes that are necessary to construct the segment root.

Concretely, a segment consist of the following elements:

• Segment identifier: b, the 2-log of the size of the number of leaves (before pruning) and a zero-based segment index idx
• List of intermediary hashes, sorted by MMR position in ascending order. Only for prunable MMRs, it contains the unpruned hashes in the segment range. They are necessary to reconstruct the segment root. As of this PR it contains all unpruned hashes above the leaves, which possibly contains redundant data. This is something we can improve over time.
• List of unpruned leaves in the segment (leaf index range [i*2**b, (i+1)*2**b)), sorted by MMR position in ascending order.
• Segment merkle proof, required to reproduce the MMR root starting with the segment root, thereby proving membership.

Given that a segment contains a number of leaves that is a power of 2, a full segment forms a full subtree in the MMR and as such it has a single root. The final segment possibly has less than 2**b elements. In this case the peaks in the segment are also peaks in the full MMR and we define its root as these peaks bagged together.

Segment creation
A segment is created by looping over all the MMR positions in the segment range. If the position is an unpruned leaf or its subling is an unpruned leaf, add it to the leaf data list. If not, check if its hash is unpruned and add it to the list of hashes. Next, generate the merkle proof by filling a list of hashes with:

1. the siblings along the path from the subtree root (final MMR position of the segment) up to its corresponding peak in the MMR
2. peaks to the right of our subtree root, bagged together to a single hash
3. peaks to the left (from right to left) with a position smaller than the first MMR position of the segment

Note that this procedure will also behave as expected for the partially filled final segment, since in that case step and 1 and 2 will not produce any hashes and step 3 will give us all the other peaks in the MMR.

Segment verification
Iterate over all the MMR positions in the segment range. If the position is a leaf: check (for prunable MMRs) to see if the element is expected to be there (based on presence of the element or its sibling in the bitmap) and if it is get it from the list of leaves, hash it and store the hash in a list of temporary hashes.
For all other positions: if both children are not present in the list of temporary hashes (i.e. they are both pruned), do nothing. If either or both hashes are present in the list, hash them together and store the new hash in the list of temporary hashes. If only one of the children was present, obtain the hash of the other child from the list of hashes in the segment instead.
After looping through all positions, we are either left with 1 entry (full segment) or multiple entries (partially filled, final segment) in the list of temporary hashes. If there are multiple, bag them together. We are left with the segment root.

Next, verify the proof by attempting to reproduce the MMR root hash: first, hash together with the siblings along the path from the subtree root to its peak in the MMR, then hash it together with the bagged peaks to the right and finally hash it together with the peaks on the left (going right to left). Verification passes if the calculated hash is equal to the MMR root hash.

Open points before merging

• In case of a heavily pruned MMR, is this data always sufficient to reconstruct the full pruned MMR? Or would we miss any intermediary hashes? Yes, now that we added support for fully pruned segments.
• I think we need a function to extract the intermediary and proof hashes for purposes of storing them in the MMR we are building TBD in a future PR
• Would it be more natural to pass in a bitmap indicating the spent or the unspent positions? Unspent
• Is bitmap/leaf index 0-based (should be a quick check) Yes, it is.

[antiochp]

Quick question that occurred to me reading over the PR description (nice description by the way!) -

Segment identifier: b, the 2-log of the size of the number of leaves (before pruning) and a zero-based segment index idx

Does it potentially make more sense to provide a "starting leaf index" as segment identifier? The current "segment index" is heavily dependent on the "segment size". Clearly you can translate between these easily enough but it may make sense to identify these based on both number of leaves via b and leaf idx.

[antiochp]

List of unpruned leaves in the segment (leaf index range [i*2b, (i+1)*2b)), sorted by MMR position in ascending order.

Is the intention to have these segments be self-contained? For pruned MMR (outputs and rangeproofs) do we also provide the bitmap or is the assumption we already have the corresponding bitmap segment?

[antiochp]

(apologies for the barrage of questionable feedback here...) 😄

[jaspervdm]

Not at all! I think they highlight some of the subtleties in the PR so it is good that they are discussed explicitly.

[antiochp]

Just posting here for reference -

• We discussed "empty" segments. Proposal is to return a single root (found by recursing up the MMR from the empty segment). This provides all necessary hashes for reconstruction, even across a heavily pruned MMR, with pruning going beyond segment size. The root position will exist outside the defined segment.

• There is an optimization in the above, if the requester can specify the segment size/height. Given the output bitmap they can determine which segments are empty and request larger segments (output and rangeproof) that cover multiple empty segments (or combination of empty and adjacent non-empty segments) of the MMR.

[jaspervdm]

Fixed an edge case: if there is an uneven number of leaves, and the final leaf is spent, we still require it to be present in the final segment. Previously we only checked the bitmap for it and its (non-existent) sibling, which are both 0. This led us to assume they were pruned, but this is not the case.

This was actually caught by the pruning test in store/tests/segment.rs, except for the fact that there was a bug in the test itself related to the bitmap indices. Fixing the bug in the test made the test fail as it should have.

[jaspervdm]

@antiochp

• We discussed "empty" segments. Proposal is to return a single root (found by recursing up the MMR from the empty segment). This provides all necessary hashes for reconstruction, even across a heavily pruned MMR, with pruning going beyond segment size. The root position will exist outside the defined segment.

We now support pruned segments. The root() function now returns Option<Hash>, where a None indicates a full segment that is completely pruned. In this situation the root (or one of its parents) needs to be obtained from the list of hashes in the segment. This is done in first_unpruned_parent().

In order to find the first parent up the path to the peak that is unpruned, the new first_unpruned_parent() function can be used. If the segment is not fully pruned, it will return (hash, None) where the hash is the root of the segment. If the segment is fully pruned, it will return (hash, Some(pos)) where the hash is the hash of the first parent that isn't compacted away, and pos is the corresponding position.

I've also added a bunch of tests to make sure it behaves as expected for full segments and doesn't affect the partially filled final segment.

[antiochp]

👍 Sounds good - I'm planning to take a closer look at this today.

[antiochp]

This looks good. Want to move it out of Draft status?

Just one minor point/question -

In order to find the first parent up the path to the peak that is unpruned, the new first_unpruned_parent() function can be used. If the segment is not fully pruned, it will return (hash, None) where the hash is the root of the segment. If the segment is fully pruned, it will return (hash, Some(pos)) where the hash is the hash of the first parent that isn't compacted away, and pos is the corresponding position.

Do we need to have None vs Some(pos) here for this?
I wonder if we could simply do (hash, pos) consistently for both scenarios? If its the "real" root of the subtree then its just the pos of the root. If its a fully pruned subtree then pos is just a higher up parent pos. Is there an advantage to only including an optional pos?

[jaspervdm]

You are right, we don't really need to return a (hash, Some(pos)), will update it to (hash, pos). For conceptual clarity I'd like to keep the Option<Hash> on the root() function though.

[antiochp]

I think we're looking good here with this. 👍
What else is outstanding before we can merge?

[jaspervdm]

I'm working on the deser of bitmap segments in this PR, but it probably makes more sense to do that in a separate one. I think we can merge this.

[antiochp]

🎉