Switch to a constant-space Merkle root/branch algorithm. #6508

sipa · 2015-08-03T17:08:10Z

This switches the Merkle tree logic for blocks to one that runs in logspace (effectively constant space, with a limit of 2^32 leaves). The old code is moved to tests, and a new test is added that for various combinations of block sizes, transaction positions to compute a branch for, and mutations:

Verifies that the old code and new code agree for the Merkle root.
Verifies that the old code and new code agree for the Merkle branch.
Verifies that the computed Merkle branch is valid.
Verifies that mutations don't change the Merkle root.
Verifies that mutations are correctly detected.

Advantages:

Removes the need for vMerkleTree (a validation-related data structure) from CBlock (a primitive data structure).
Is slightly faster due to better memory locality and avoiding all heap overhead.

Disadvantages:

Requires recomputing the Merkle tree any time a Merkle branch is needed. The wallet depends on this any time a transaction is added. This is not technically necessary, however.
Longer and harder to read code.

The code remains in primitives/block for now. This is not optimal, as one of the purposes is disentangling the Merkle calculation logic from the primitive data structures. TBD.

dgenr8 · 2015-08-03T22:08:36Z

Great improvement over current space requirement which is O(n), yes?

sipa · 2015-08-03T22:34:58Z

@dgenr8 Yes, but the current code requires around 96 bytes per transaction, which is insignificant compared to the ~1 kB each transaction already requires in memory on average. Still, this PR improves it to ~1 kB total (but more importantly: entirely continuous and on the stack).

laanwj · 2015-08-05T10:39:43Z

Concept ACK.
Needs to be reviewed very carefully as to not change the behavior in subtle edge cases.

sipa · 2015-08-05T13:17:24Z

Of course.

Note that the unit test creates merkle trees of all sizes up to 16, and for
each all known malleabilities and all branches.

On the other side, the performance and memory usage benefit are not
significant, but getting rid of vMerkleTree in CBlock may be worth it.

cozz · 2015-08-08T09:03:01Z

Does this mean that all transactions in a block have to be hashed together again, every time a transaction gets added to the wallet? So if there are 10 transactions in a block for the wallet, we hash it again 10 times?

sipa · 2015-08-08T09:12:35Z

@cozz: with the current code, yes. But I don't think we actually need to store those...

cozz · 2015-08-08T09:42:33Z

Yes, I wouldnt store them either. But as long as we do, I do not like, that this pull decreases wallet performance. So I would like to see that fixed. Either by some temp-cache for the wallet, or just by removing those merkle-branches.

sipa · 2015-08-11T08:52:36Z

There is definitely a wallet performance downside, as you need to recompute the merkle tree for every wallet transaction. If it looked like I claimed otherwise, sorry. It's only an individual merkle root/branch computation that is made faster. I'll delay this until we remove the call from the wallet to conpute branches.

sipa · 2015-08-11T19:27:49Z

Rebased on top of #6550, to address the issue that @cozz raised.

dcousens · 2015-08-12T01:01:02Z

utACK

Diff comparison for those who want to check without including the rebased commits.

laanwj · 2015-10-06T08:55:14Z

Needs rebase (after #6550, I think)

sipa · 2015-11-02T04:41:23Z

Rebased.

sipa · 2015-11-17T16:38:36Z

Moved the Merkle computation code to a separate file (independent of CBlock and CTransaction), and used that in a thin wrapper caller in primitives/block.h.

dcousens · 2015-11-18T00:37:21Z

re-ACK

Also, awesome on moving merkle.cpp out 👍

jtimon · 2015-11-23T10:07:39Z

If we make it consensus/merkle.o, that would be less code to move again later...

sipa · 2015-11-26T00:46:10Z

@jtimon Hmm, I was seeing this more as generic data structure code, like limitedmap.h is for example, but it's probably not useful to anything that doesn't also need the consensus logic anyway.

laanwj · 2015-11-26T09:53:47Z

Agree that Merkle trees are a general data structure - however, the specific way of doing a Merkle-tree is very bitcoin-consensus-specific, and there is the big warning that this is not recommended to use this as-is outside of bitcoin.

jtimon · 2015-11-26T11:09:44Z

General data structure or not, this is used in checkBlock and will therefore be part of VerifyBlock and the complete libconsensus too.
Once it is complete, if we want to separate it as a subtree with its own repo, we will have to put all the files in the same dir (ie src/consensus) first.

laanwj · 2015-11-26T11:13:03Z

@jtimon I agree

sipa · 2015-11-26T12:18:38Z

Done.

I had to move more things around, because otherwise primitives and consensus would end up with a circular dependency on each other. Now all logic (both generic Merkle logic, and the block-specific code) are in consensus, which is better as none of it really belonged in primitives anyway.

jtimon · 2015-11-26T12:32:21Z

Thank you.
I wouldn't worry about consensus/merkle depending on primitives/transaction while primitives/block depends on consensus/merkle. primitives is not a building package and they all three belong in the consensus building package (see WIP #7091) and eventually they should all be in the consensus dir anyway.
A circular dependency between consensus/merkle and primitives/block is another thing though.
I can't remember how the includes were before, but they look good to me now.

This switches the Merkle tree logic for blocks to one that runs in constant (small) space. The old code is moved to tests, and a new test is added that for various combinations of block sizes, transaction positions to compute a branch for, and mutations: * Verifies that the old code and new code agree for the Merkle root. * Verifies that the old code and new code agree for the Merkle branch. * Verifies that the computed Merkle branch is valid. * Verifies that mutations don't change the Merkle root. * Verifies that mutations are correctly detected.

jtimon · 2015-11-27T14:53:59Z

Oh, thought I had done it already...concept ACK

eece63f Switch blocks to a constant-space Merkle root/branch algorithm. (Pieter Wuille) ee60e56 Add merkle.{h,cpp}, generic merkle root/branch algorithm (Pieter Wuille)

jgarzik · 2015-11-28T21:28:56Z

utACK

Backport of bitcoin#6508 This switches the Merkle tree logic for blocks to one that runs in constant (small) space. The old code is moved to tests, and a new test is added that for various combinations of block sizes, transaction positions to compute a branch for, and mutations: * Verifies that the old code and new code agree for the Merkle root. * Verifies that the old code and new code agree for the Merkle branch. * Verifies that the computed Merkle branch is valid. * Verifies that mutations don't change the Merkle root. * Verifies that mutations are correctly detected.

91b48c7 [Build] Add new merkle files to CMake lists (warrows) 48a3aff [Wallet] Ignore coinbase and zc tx "conflicts" (warrows) 3572354 [Wallet] Fix an error in tx depth computation (warrows) 6928369 [Tests] Enable abandonconflict functional test (warrows) 34cd496 Fix that CWallet::AbandonTransaction would only traverse one level (Ben Woosley) aba5b75 Fix calculation of balances and available coins. (Alex Morcos) 48d705f Remove stale wallet transactions on initial load. (presstab) 12985ae Flush wallet after abandontransaction (Alex Morcos) 8f87956 [Wallet] sort pending wallet transactions before reaccepting (dexX7) 9c2f445 [Wallet] Call notification signal when a transaction is abandoned (Jonas Schnelli) 778ebf3 Add new rpc call: abandontransaction (Alex Morcos) 0e86c3e Make wallet descendant searching more efficient (Alex Morcos) d0083a8 Make sure conflicted wallet tx's update balances (Alex Morcos) 6a50e03 [Wallet] Keep track of explicit wallet conflicts instead of using mempool (warrows) 7ccb2b5 [Wallet] Do not flush the wallet in AddToWalletIfInvolvingMe(..) (warrows) 47345be [Refactor] Move wallet functions out of header (warrows) ab9efb8 [Wallet] Switch to a constant-space Merkle root/branch algorithm (warrows) 5447622 [Wallet] Do not store Merkle branches in the wallet (warrows) Pull request description: This pull request is a happy melting pot of improvements regarding transactions handling. Most of them are backports from bitcoin. I advise reviewers to check the code of the different commits independently to understand them more easily. However, testing is probably better done all at once. I am making a single pull request because these changes are all entangled and introducing some of them without others would probably introduce temporary bugs. ## Commits details ## - 6c3e2ac backport of bitcoin#6550 - 5304fdf backport of bitcoin#6508 - c3eeeac simple code move from the header to the cpp file. It contains no functional change. - 6cc4d37 backport of bitcoin#4805 - 10be1db backport of bitcoin#7105 - 8a34c32 backport of bitcoin#7306 - 3caf123, 9e17178 and 240f5b4 are the backport for bitcoin#7312 - ad6d0b1 backport of bitcoin#5511 - fcc07c3 backport of bitcoin#9311 - 5ed5e26 is an update of #825 - 392d504 backport of bitcoin#7715 - 7199f3a backport of bitcoin#13652 - f09d999 enables and fixes the test from bitcoin#7312 - 4fd43c5 fixes an oversight in bitcoin#7105 backport ACKs for top commit: random-zebra: ACK 91b48c7 Fuzzbawls: ACK 91b48c7 Tree-SHA512: 2628cebe98805b8048b920b51ee26fd4f0c53643d78da9b8cb265aede52dfe1d40c8c19d34293c232c5c35be7f1ab89ff5b4a07073a4b27c371ea70eb8708669

Backport of bitcoin#6508 This switches the Merkle tree logic for blocks to one that runs in constant (small) space. The old code is moved to tests, and a new test is added that for various combinations of block sizes, transaction positions to compute a branch for, and mutations: * Verifies that the old code and new code agree for the Merkle root. * Verifies that the old code and new code agree for the Merkle branch. * Verifies that the computed Merkle branch is valid. * Verifies that mutations don't change the Merkle root. * Verifies that mutations are correctly detected.

Backport of bitcoin/bitcoin#6508 This switches the Merkle tree logic for blocks to one that runs in constant (small) space. The old code is moved to tests, and a new test is added that for various combinations of block sizes, transaction positions to compute a branch for, and mutations: * Verifies that the old code and new code agree for the Merkle root. * Verifies that the old code and new code agree for the Merkle branch. * Verifies that the computed Merkle branch is valid. * Verifies that mutations don't change the Merkle root. * Verifies that mutations are correctly detected. # Conflicts: # src/test/pmt_tests.cpp

Backport of bitcoin/bitcoin#6508 This switches the Merkle tree logic for blocks to one that runs in constant (small) space. The old code is moved to tests, and a new test is added that for various combinations of block sizes, transaction positions to compute a branch for, and mutations: * Verifies that the old code and new code agree for the Merkle root. * Verifies that the old code and new code agree for the Merkle branch. * Verifies that the computed Merkle branch is valid. * Verifies that mutations don't change the Merkle root. * Verifies that mutations are correctly detected.

Backport of bitcoin#6508 This switches the Merkle tree logic for blocks to one that runs in constant (small) space. The old code is moved to tests, and a new test is added that for various combinations of block sizes, transaction positions to compute a branch for, and mutations: * Verifies that the old code and new code agree for the Merkle root. * Verifies that the old code and new code agree for the Merkle branch. * Verifies that the computed Merkle branch is valid. * Verifies that mutations don't change the Merkle root. * Verifies that mutations are correctly detected.

Backport of bitcoin/bitcoin#6508 This switches the Merkle tree logic for blocks to one that runs in constant (small) space. The old code is moved to tests, and a new test is added that for various combinations of block sizes, transaction positions to compute a branch for, and mutations: * Verifies that the old code and new code agree for the Merkle root. * Verifies that the old code and new code agree for the Merkle branch. * Verifies that the computed Merkle branch is valid. * Verifies that mutations don't change the Merkle root. * Verifies that mutations are correctly detected.

Bitcoin 0.12 Merkle tree PRs Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#6550 - bitcoin/bitcoin#6508

Backport of bitcoin/bitcoin#6508 This switches the Merkle tree logic for blocks to one that runs in constant (small) space. The old code is moved to tests, and a new test is added that for various combinations of block sizes, transaction positions to compute a branch for, and mutations: * Verifies that the old code and new code agree for the Merkle root. * Verifies that the old code and new code agree for the Merkle branch. * Verifies that the computed Merkle branch is valid. * Verifies that mutations don't change the Merkle root. * Verifies that mutations are correctly detected.

sipa force-pushed the constmerkle branch from 851e62d to 1a31682 Compare August 3, 2015 20:34

laanwj added the Validation label Aug 5, 2015

sipa mentioned this pull request Aug 8, 2015

Get rid of Merkle branches in the wallet #6536

Closed

sipa force-pushed the constmerkle branch from 1a31682 to 45f35c6 Compare August 11, 2015 19:27

TheBlueMatt mentioned this pull request Aug 27, 2015

Use a faster merkle root calculation algorithm TheBlueMatt/RelayNode#19

Open

sipa force-pushed the constmerkle branch from 45f35c6 to 00c9e7e Compare November 2, 2015 04:40

gmaxwell added this to the 0.12.0 milestone Nov 11, 2015

sipa force-pushed the constmerkle branch from 00c9e7e to 32ce5bd Compare November 17, 2015 16:37

sipa force-pushed the constmerkle branch from 32ce5bd to 84512d8 Compare November 26, 2015 12:17

sipa force-pushed the constmerkle branch from d5a4e81 to eece63f Compare November 27, 2015 14:38

sipa merged commit eece63f into bitcoin:master Nov 28, 2015

sipa added a commit that referenced this pull request Nov 28, 2015

Merge pull request #6508

61457c2

eece63f Switch blocks to a constant-space Merkle root/branch algorithm. (Pieter Wuille) ee60e56 Add merkle.{h,cpp}, generic merkle root/branch algorithm (Pieter Wuille)

dagurval mentioned this pull request Jun 13, 2018

Switch to a constant-space Merkle root/branch algorithm. bitcoinxt/bitcoinxt#452

Merged

Warrows mentioned this pull request Jul 30, 2019

[Wallet] Various transaction handling improvements PIVX-Project/PIVX#970

Merged

div72 mentioned this pull request Apr 5, 2021

consensus, refactor, test: Merkle gridcoin-community/Gridcoin-Research#2094

Merged

str4d mentioned this pull request Apr 20, 2021

Bitcoin 0.12 Merkle tree PRs zcash/zcash#5065

Merged

zkbot added a commit to zcash/zcash that referenced this pull request Apr 20, 2021

Auto merge of #5065 - str4d:merkle-backports, r=<try>

79c785d

Bitcoin 0.12 Merkle tree PRs Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#6550 - bitcoin/bitcoin#6508

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to a constant-space Merkle root/branch algorithm. #6508

Switch to a constant-space Merkle root/branch algorithm. #6508

sipa commented Aug 3, 2015

dgenr8 commented Aug 3, 2015

sipa commented Aug 3, 2015

laanwj commented Aug 5, 2015

sipa commented Aug 5, 2015

cozz commented Aug 8, 2015

sipa commented Aug 8, 2015 via email

cozz commented Aug 8, 2015

sipa commented Aug 11, 2015 via email

sipa commented Aug 11, 2015

dcousens commented Aug 12, 2015

laanwj commented Oct 6, 2015

sipa commented Nov 2, 2015

sipa commented Nov 17, 2015

dcousens commented Nov 18, 2015

jtimon commented Nov 23, 2015

sipa commented Nov 26, 2015

laanwj commented Nov 26, 2015

jtimon commented Nov 26, 2015

laanwj commented Nov 26, 2015

sipa commented Nov 26, 2015

jtimon commented Nov 26, 2015

jtimon commented Nov 27, 2015

jgarzik commented Nov 28, 2015

Switch to a constant-space Merkle root/branch algorithm. #6508

Switch to a constant-space Merkle root/branch algorithm. #6508

Conversation

sipa commented Aug 3, 2015

dgenr8 commented Aug 3, 2015

sipa commented Aug 3, 2015

laanwj commented Aug 5, 2015

sipa commented Aug 5, 2015

cozz commented Aug 8, 2015

sipa commented Aug 8, 2015 via email

cozz commented Aug 8, 2015

sipa commented Aug 11, 2015 via email

sipa commented Aug 11, 2015

dcousens commented Aug 12, 2015

laanwj commented Oct 6, 2015

sipa commented Nov 2, 2015

sipa commented Nov 17, 2015

dcousens commented Nov 18, 2015

jtimon commented Nov 23, 2015

sipa commented Nov 26, 2015

laanwj commented Nov 26, 2015

jtimon commented Nov 26, 2015

laanwj commented Nov 26, 2015

sipa commented Nov 26, 2015

jtimon commented Nov 26, 2015

jtimon commented Nov 27, 2015

jgarzik commented Nov 28, 2015