Improved (more compact) Merkle Proofs #1015

antiochp · 2018-04-27T19:56:29Z

Reworking Merkle proofs so we only store a path and the underlying mmr_size.
The path includes the full sibling path and the necessary hashes derived from the peaks.
We need the mmr_size as we cannot rely on this being available when we come to verify the proof.
The mmr_size is used to determine the pos of the various entries in the path.

For siblings in the path we hash the node and the sibling together using the index of the parent.
For peak derived hashes we hash with the mmr_size (as a substitute for the non-existent pos).

Everything should hash up to the root itself.

TODO -

* block sums and reworked block validation read and write block_sums refactor validate on both block and txhashset write block_sum on fast sync we store the kernel_sum (need to account for the offset) * block_sums * rustfmt * cleanup

* txhashset now implements committed for consistency * rustfmt and cleanup

* make kernel offsets commitments, they used to be blinding factors * cleanup various zero_commit inits * cleanup api and wallet to reflect offsets as commitments

verify maturity via merkle proofs TODO - how to check validity of txs?

so we know the features are accurate and honest

antiochp · 2018-05-09T19:01:28Z

This I think is ready for review.

Simplified MerkleProof impl - just a path and the mmr_size
We now verify Merkle proofs in one place and we require the root and the node_pos to do so.

/cc @ignopeverell @tromp

ignopeverell

Looks good overall, also seems slightly cleaner.

core/src/core/pmmr.rs

+}
+
+/// A Merkle proof that proves a particular element exists in the MMR.
+#[derive(Debug, Eq, PartialEq, Clone, PartialOrd, Ord)]
 pub struct MerkleProof {


core/src/core/pmmr.rs

@@ -309,6 +297,52 @@ where
 			.collect()
 	}

+	fn peak_path(&self, peak_pos: u64) -> Vec<Hash> {
+		let rhs = self.bag_the_rhs(peak_pos);
+		println!("rhs bagged - {:?}", rhs);


split out tests in core as well

antiochp · 2018-05-11T00:05:33Z

@tromp - want to take a look over this before I merge onto consensus_breaking branch?

tromp · 2018-05-11T13:21:30Z

will review

core/src/core/merkle_proof.rs

+				return Err(MerkleProofError::RootMismatch);
+			}
+		} else if self.mmr_size == 1 {
+			if self.path.len() == 1 && root == node_hash && vec![root] == self.path {


core/src/core/merkle_proof.rs

+		let sibling = self.path.remove(0);
+		let (parent_pos, sibling_pos) = pmmr::family(node_pos);
+
+		let peaks_pos = pmmr::peaks(self.mmr_size);


core/src/core/pmmr.rs

+		if mmr_size == 1 {
+			return Ok(MerkleProof {
+				mmr_size,
+				path: vec![node],


core/src/core/pmmr.rs

 			.iter()
-			.map(|x| (self.get_from_file(x.1).unwrap_or(Hash::default()), x.1))
+			.filter_map(|x| self.get_from_file(x.1))


core/src/core/pmmr.rs

@@ -940,6 +811,9 @@ pub fn bintree_postorder_height(num: u64) -> u64 {
 /// We know the positions of all leaves based on the postorder height of an MMR of any size
 /// (somewhat unintuitively but this is how the PMMR is "append only").
 pub fn is_leaf(pos: u64) -> bool {
+	if pos == 0 {


core/tests/merkle_proof.rs

+	assert_eq!(pmmr.get_hash(1).unwrap(), pos_0);
+
+	let proof = pmmr.merkle_proof(1).unwrap();
+	assert_eq!(proof.path, [pos_0]);


tromp

Looks ok except for non empty merkle proof paths when there are no siblings

antiochp · 2018-05-12T19:35:19Z

Addressed feedback, will merge once all tests pass.

antiochp force-pushed the shorter_merkle_proofs branch from 77026a6 to 35e97df Compare May 4, 2018 19:20

antiochp changed the base branch from master to consensus_breaking May 4, 2018 19:21

antiochp added enhancement consensus breaking Use for issues or PRs that will break consensus and force a hard fork labels May 4, 2018

antiochp force-pushed the shorter_merkle_proofs branch 2 times, most recently from 0de5421 to 6b676fe Compare May 8, 2018 19:10

antiochp added 15 commits May 9, 2018 14:15

[testnet2] Store output sum (mimblewimble#1043)

c52e030

* block sums and reworked block validation read and write block_sums refactor validate on both block and txhashset write block_sum on fast sync we store the kernel_sum (need to account for the offset) * block_sums * rustfmt * cleanup

txhashset extension now implements Committed (mimblewimble#1049)

8bc1173

* txhashset now implements committed for consistency * rustfmt and cleanup

make kernel offsets commitments (mimblewimble#1020)

2a2f23e

* make kernel offsets commitments, they used to be blinding factors * cleanup various zero_commit inits * cleanup api and wallet to reflect offsets as commitments

WIP - improved Merkle proofs (0-index vs 1-index is killing me...)

8f12479

rustfmt

283906c

initial pass at verify for improve merkle proofs

934f9d8

commit

db26e19

commit

01e5871

getting closer to a working verify() for improved merkle proofs

e9f3673

merkle_proof.verify()

294fd60

verify maturity via merkle proofs TODO - how to check validity of txs?

rustfmt

cc795d1

rework merkle_proof.verify to take a hashable element

24f68cc

commit

9ea1c84

commit

b37cb26

fixup conflicts with pool tests

529b72f

antiochp force-pushed the shorter_merkle_proofs branch from 627ebfd to 529b72f Compare May 9, 2018 18:26

antiochp added 3 commits May 9, 2018 14:30

tweak comments

0c14321

fn docs

e70ca47

add call to is_unspent from verify_maturity

44bd8bb

so we know the features are accurate and honest

antiochp changed the title ~~[WIP] improved merkle proofs~~ Improved (more compact) Merkle Proofs May 9, 2018

ignopeverell reviewed May 9, 2018

View reviewed changes

antiochp added 2 commits May 9, 2018 15:43

fixup core tests - ser sizes

078a662

adding to pool - check maturity against "future" block

1d0c5b4

antiochp added 5 commits May 9, 2018 16:19

fix test to reflect adding tx spending coinbase output to pool

ce4738e

actually make test reflect reality - dummyblockchain maturity handling

0eff79b

split out merkle_proof.rs from pmmr.rs

babccb7

split out tests in core as well

commit

17c2e41

fix pool tests

5d61e54

antiochp requested a review from tromp May 11, 2018 00:05