[DNM] prune a PMMR and we cannot construct a Merkle proof (minimal example)

[antiochp]

I think this shows the problem I am seeing when trying to generate Merkle proofs for some outputs.

This is a very simple example.

• initialize empty PMMR backend
• add a single element to the PMMR
  • show the hash matches the root hash (single node)
• add a 2nd element to the PMMR
  • show these hash together to give the peak at pos 3
  • show the root matches this single peak
• prune the element at pos 1
  • the peak and the root are unaffected by this pruning operation
  • but we do not have enough information to build the Merkle proof for pos 2, given just pos 2 and pos 3

The problem is how do we show inclusion of the element at pos 2 under the peak at pos 3?
There is no "sibling hash" that we can use as part of the branch to show inclusion of pos 2 under pos 3.

---- sumtree_merkle_proof_problems stdout ----

MMR with a single leaf node
pos 1 - Some(HashSum { hash: 213d8607, sum: 1 })
root - HashSum { hash: 213d8607, sum: 1 }

MMR with 2 leaf nodes (root built from single peak)
pos 1 - Some(HashSum { hash: 213d8607, sum: 1 })
pos 2 - Some(HashSum { hash: 3e02a5b8, sum: 2 })
pos 3 - Some(HashSum { hash: 8b4382e2, sum: 3 }) (peak)
root - HashSum { hash: 8b4382e2, sum: 3 }

After pruning pos 1
pos 1 - None
pos 2 - Some(HashSum { hash: 3e02a5b8, sum: 2 })
pos 3 - Some(HashSum { hash: 8b4382e2, sum: 3 }) (peak)
root - HashSum { hash: 8b4382e2, sum: 3 }

[antiochp]

After discussing this with @yeastplume -
This is an issue at every height in the trees, not just the leaves. Pruning will remove full subtrees after all leaves are pruned.
One option is to expose nodes in the remove list to allow us to look in there and use these pruned nodes to build the Merkle proof.
After compaction step on a non-archival node we will not have this problem - compaction will rebalance the trees under the peaks (effectively rewriting the full PMMR).

Exposing the pruned nodes in the "remove list" via pmmr.get_ignoring_remove_list() will allow archival nodes to reliably generate Merkle proofs for historical outputs.

[ignopeverell]

I've gotten close to adding such a functionality in the past but didn't end up needing it. So yes, if it's required now, totally makes sense.