Merkle Proofs #716

antiochp · 2018-02-19T22:30:36Z

Functionality to generate a Merkle proof for a given entry in the output MMR.
Merkle proof consists of the following -
- hash of the node in the tree (leaf node representing an output)
- hash of the overall root of the tree
- list of hashes representing the peaks (matches the root when hashed together)
- list of sibling hashes to construct branch from node to peak
- list of left/right sibling positions to allow the branch to be reconstructed correctly
A Merkle proof allows us to -
- prove inclusion of the node beneath its peak (via the branch of sibling hashes)
- prove peaks combine to form overall root
- prove inclusion of an (unspent) output for a given block_header by verifying the utxo_root matches the root of the Merkle proof
- if an output is unspent at a given block header then it is at least as old as that block
API serializes/deserializes Merkle proofs as hex
Wallet maintains Merkle proof (and corresponding block hash) for each unspent coinbase output
Spending a coinbase output requires the block hash and Merkle proof to be provided in the input
- these are used to verify coinbase maturity (without requiring full block data for verification)

TODO -

Cannot use rewind to put the MMR back into a state where we can create an old Merkle proof (because of pruning?) - confirm we can only create a Merkle proof at the same time the block is originally created?
We can now successfully generate a Merkle proof via sumtree.rewind().
This definitely works for a full archival node.
This will in principle work for a pruned non-archive node - but we need to confirm this.
cleanup test failures

We uncovered a subtle bug in our sumtree.rewind() logic - #727 (FIXED).

todo - we hit a peak, then we need to get to the root somehow - actually get the hashes to build the proof

need to test on test case with multiple peaks

antiochp · 2018-02-20T17:11:47Z

@ignopeverell I keep getting tripped up with MMR rewind...

I think I have managed to (finally) convince myself that we cannot use rewind() to reliably put the output MMR into the correct state for a historical block, and we are unable to create a Merkle proof for the coinbase output for a historical block.
Outputs in the MMR may have been removed/pruned and rewind may not handle this correctly.

So am I right in thinking the Merkle proof can only be created at the time the block itself is created (we cannot reliable regenerate the exact same utxo_root otherwise)?
And that the miner will need to create any necessary Merkle proofs when they successfully mine a block (and presumably store them somewhere to make them available to the wallet)?

ignopeverell · 2018-02-20T21:32:28Z

I think you're correct. Note that you should be able to rewind to that position for a little while, even in presence of pruning. But when the local storage gets compacted then actual data you would need gets removed. So we could wait 10-20 blocks to avoid storing data that could be useless in the common 1-block reorg case, but beyond that you're taking a chance. Note that waiting a bit sounds like a premature optimization to me at this point, just mentioning it as an example.

there are cases where it does not...

need to fix the tests up now and cleanup the code and add docs for functions etc.

antiochp · 2018-02-26T22:57:50Z

@ignopeverell After fixing rewind() and ignoring the remove list (read_from_file() vs existing read()) we can now generate Merkle proofs reliably and correctly on a full archival node.
As discussed - we should have all the non-leaf nodes in the tree even after pruning/compacting.

Planning to investigate if this works as-is with a pruned tree or if we need to modify the tree compacting behavior to retain leaf siblings to keep Merkle proofs working post-compaction step on the output PMMR.

I think we're getting close to landing this PR (hopefully).

made nonce for short_ids explicit to help with this

antiochp · 2018-03-01T19:09:21Z

This is ready for review.
We will want to revisit this when we tackle pruning/compaction of the PMMR (we need to ensure we still have access to the hashes of all siblings, including a potentially removed sibling of the leaf node).
But this works for a "removed" node in a non-compacted MMR currently.

@ignopeverell @yeastplume

ignopeverell

Looks good to me! Mostly commented on the numerous leftovoer TODOs :-)

api/src/types.rs

 		state.serialize_field("output_type", &self.output_type)?;
 		state.serialize_field("commit", &util::to_hex(self.commit.0.to_vec()))?;
 		state.serialize_field("switch_commit_hash", &self.switch_commit_hash.to_hex())?;
 		state.serialize_field("spent", &self.spent)?;
 		state.serialize_field("proof", &self.proof)?;
 		state.serialize_field("proof_hash", &self.proof_hash)?;
+
+		// TODO - rework this


core/src/core/block.rs

@@ -660,6 +662,30 @@ impl Block {
 		Ok(())
 	}

+	// TODO - how do we verify Merkle Proof here?


core/src/core/transaction.rs

@@ -470,14 +480,38 @@ impl Transaction {
 		self.kernels.verify_sort_order()?;
 		Ok(())
 	}
+
+	// TODO - how do we verify Merkle Proof here?


wallet/src/sender.rs

+			let merkle_proof = coin.merkle_proof.clone();
+			let merkle_proof = merkle_proof.unwrap().merkle_proof();
+
+			// TODO - we need a Merkle Proof here to spend a coinbase output


chain/src/sumtree.rs

 					} else {
-						println!("MISMATCH BECAUSE THE BLOODY THING MISMATCHES");


yeastplume · 2018-03-02T09:23:17Z

Cannot find fault here.. really well done. Ensuring siblings still exist on compaction is a fairly trivial implementation detail, all we need to do is ensure the top peak in the prune list isn't actually excluded (just everything underneath it.

antiochp · 2018-03-02T14:30:08Z

@yeastplume yeah I'm planning to go take a look at the compaction stuff later today I think.

@ignopeverell thanks for that - I'd kind of skipped that last pass through all those... working through them now.

antiochp · 2018-03-02T17:08:47Z

Just waiting for final travis run then I'll merge.

* family_branch() to recursively call family() up the branch todo - we hit a peak, then we need to get to the root somehow - actually get the hashes to build the proof * wip * some additional testing around merkle tree branches * track left/right branch for each sibling as we build the merkle path up * MerkleProof and basic (incomplete) verify fn * I think a MerkleProof verifies correctly now need to test on test case with multiple peaks * basic pmmr merkle proof working * MerkleProof now serializable/deserializable * coinbase maturity via merkle proof basically working * ser/deser merkle proof into hex in api and wallet.dat * cleanup * wip - temporarily saving merkle proofs to the commit index * assert merkle proof in store matches the rewound version there are cases where it does not... * commit * commit * can successfully rewind the output PMMR and generate a Merkle proof need to fix the tests up now and cleanup the code and add docs for functions etc. * core tests passing * fixup chain tests using merkle proofs * pool tests working with merkle proofs * api tests working with merkle proof * fix the broken comapct block hashing behavior made nonce for short_ids explicit to help with this * cleanup and comment as necessary * cleanup variety of TODOs

antiochp added 11 commits February 17, 2018 16:08

family_branch() to recursively call family() up the branch

481ccb3

todo - we hit a peak, then we need to get to the root somehow - actually get the hashes to build the proof

wip

fbfe3df

some additional testing around merkle tree branches

8a4f301

track left/right branch for each sibling as we build the merkle path up

097c2d5

MerkleProof and basic (incomplete) verify fn

6b59246

I think a MerkleProof verifies correctly now

cc5800e

need to test on test case with multiple peaks

basic pmmr merkle proof working

a2ac056

MerkleProof now serializable/deserializable

4e3c204

coinbase maturity via merkle proof basically working

13988ee

ser/deser merkle proof into hex in api and wallet.dat

503f626

cleanup

8d24efb

antiochp changed the title ~~Merkle Proofs~~ [WIP] Merkle Proofs Feb 19, 2018

antiochp mentioned this pull request Feb 19, 2018

[WIP] exploring how we might generate Merkle Proofs for PMMR #699

Closed

9 tasks

n_leaves

14f3536

antiochp added 6 commits February 20, 2018 17:49

wip - temporarily saving merkle proofs to the commit index

e6a7ad5

assert merkle proof in store matches the rewound version

d2bef7d

there are cases where it does not...

commit

43fc8b3

commit

9e5038f

Merge branch 'master' into merkle_proofs

f9c6714

can successfully rewind the output PMMR and generate a Merkle proof

96d0a9a

need to fix the tests up now and cleanup the code and add docs for functions etc.

antiochp added 7 commits February 28, 2018 10:17

core tests passing

990e313

Merge branch 'master' into merkle_proofs

94e8367

fixup chain tests using merkle proofs

c41f0a1

pool tests working with merkle proofs

8b53a89

api tests working with merkle proof

48c0d32

Merge branch 'master' into merkle_proofs

0ee9edd

fix the broken comapct block hashing behavior

414a503

made nonce for short_ids explicit to help with this

antiochp added 2 commits March 1, 2018 13:27

Merge branch 'master' into merkle_proofs

420cd6c

cleanup and comment as necessary

69a2cf0

antiochp changed the title ~~[WIP] Merkle Proofs~~ Merkle Proofs Mar 1, 2018

antiochp mentioned this pull request Mar 1, 2018

PMMR backend check_compact needs to work with Merkle proofs #741

Closed

ignopeverell reviewed Mar 2, 2018

View reviewed changes

yeastplume reviewed Mar 2, 2018

View reviewed changes

chain/src/sumtree.rs

} else {

println!("MISMATCH BECAUSE THE BLOODY THING MISMATCHES");

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

cleanup variety of TODOs

7540650

sesam mentioned this pull request Mar 2, 2018

rustfmt now runs and reformats files (need to opt out explicitly) #743

Merged

antiochp merged commit cc12798 into mimblewimble:master Mar 2, 2018

antiochp deleted the merkle_proofs branch March 2, 2018 20:47

yeastplume mentioned this pull request Feb 23, 2022

[DNM] PIBD Task / Issue Tracker #3695

Merged

26 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merkle Proofs #716

Merkle Proofs #716

antiochp commented Feb 19, 2018 •

edited

Loading

antiochp commented Feb 20, 2018 •

edited

Loading

ignopeverell commented Feb 20, 2018

antiochp commented Feb 26, 2018

antiochp commented Mar 1, 2018 •

edited

Loading

ignopeverell left a comment

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

yeastplume commented Mar 2, 2018

antiochp commented Mar 2, 2018

antiochp commented Mar 2, 2018

		} else {
		println!("MISMATCH BECAUSE THE BLOODY THING MISMATCHES");

Merkle Proofs #716

Merkle Proofs #716

Conversation

antiochp commented Feb 19, 2018 • edited Loading

antiochp commented Feb 20, 2018 • edited Loading

ignopeverell commented Feb 20, 2018

antiochp commented Feb 26, 2018

antiochp commented Mar 1, 2018 • edited Loading

ignopeverell left a comment

Choose a reason for hiding this comment

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

yeastplume commented Mar 2, 2018

antiochp commented Mar 2, 2018

antiochp commented Mar 2, 2018

antiochp commented Feb 19, 2018 •

edited

Loading

antiochp commented Feb 20, 2018 •

edited

Loading

antiochp commented Mar 1, 2018 •

edited

Loading