Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to a constant-space Merkle root/branch algorithm. #6508

Merged
merged 2 commits into from Nov 28, 2015

Conversation

@sipa
Copy link
Member

sipa commented Aug 3, 2015

This switches the Merkle tree logic for blocks to one that runs in logspace (effectively constant space, with a limit of 2^32 leaves). The old code is moved to tests, and a new test is added that for various combinations of block sizes, transaction positions to compute a branch for, and mutations:

  • Verifies that the old code and new code agree for the Merkle root.
  • Verifies that the old code and new code agree for the Merkle branch.
  • Verifies that the computed Merkle branch is valid.
  • Verifies that mutations don't change the Merkle root.
  • Verifies that mutations are correctly detected.

Advantages:

  • Removes the need for vMerkleTree (a validation-related data structure) from CBlock (a primitive data structure).
  • Is slightly faster due to better memory locality and avoiding all heap overhead.

Disadvantages:

  • Requires recomputing the Merkle tree any time a Merkle branch is needed. The wallet depends on this any time a transaction is added. This is not technically necessary, however.
  • Longer and harder to read code.

The code remains in primitives/block for now. This is not optimal, as one of the purposes is disentangling the Merkle calculation logic from the primitive data structures. TBD.

@sipa sipa force-pushed the sipa:constmerkle branch Aug 3, 2015
@dgenr8
Copy link
Contributor

dgenr8 commented Aug 3, 2015

Great improvement over current space requirement which is O(n), yes?

@sipa
Copy link
Member Author

sipa commented Aug 3, 2015

@dgenr8 Yes, but the current code requires around 96 bytes per transaction, which is insignificant compared to the ~1 kB each transaction already requires in memory on average. Still, this PR improves it to ~1 kB total (but more importantly: entirely continuous and on the stack).

@laanwj laanwj added the Validation label Aug 5, 2015
@laanwj
Copy link
Member

laanwj commented Aug 5, 2015

Concept ACK.
Needs to be reviewed very carefully as to not change the behavior in subtle edge cases.

@sipa
Copy link
Member Author

sipa commented Aug 5, 2015

Of course.

Note that the unit test creates merkle trees of all sizes up to 16, and for
each all known malleabilities and all branches.

On the other side, the performance and memory usage benefit are not
significant, but getting rid of vMerkleTree in CBlock may be worth it.

@cozz
Copy link
Contributor

cozz commented Aug 8, 2015

Does this mean that all transactions in a block have to be hashed together again, every time a transaction gets added to the wallet? So if there are 10 transactions in a block for the wallet, we hash it again 10 times?

@sipa
Copy link
Member Author

sipa commented Aug 8, 2015

@cozz
Copy link
Contributor

cozz commented Aug 8, 2015

Yes, I wouldnt store them either. But as long as we do, I do not like, that this pull decreases wallet performance. So I would like to see that fixed. Either by some temp-cache for the wallet, or just by removing those merkle-branches.

@sipa
Copy link
Member Author

sipa commented Aug 11, 2015

@sipa sipa force-pushed the sipa:constmerkle branch Aug 11, 2015
@sipa
Copy link
Member Author

sipa commented Aug 11, 2015

Rebased on top of #6550, to address the issue that @cozz raised.

@dcousens
Copy link
Contributor

dcousens commented Aug 12, 2015

utACK

Diff comparison for those who want to check without including the rebased commits.

@laanwj
Copy link
Member

laanwj commented Oct 6, 2015

Needs rebase (after #6550, I think)

@sipa sipa force-pushed the sipa:constmerkle branch to 00c9e7e Nov 2, 2015
@sipa
Copy link
Member Author

sipa commented Nov 2, 2015

Rebased.

@gmaxwell gmaxwell added this to the 0.12.0 milestone Nov 11, 2015
@sipa sipa force-pushed the sipa:constmerkle branch from 00c9e7e Nov 17, 2015
@sipa
Copy link
Member Author

sipa commented Nov 17, 2015

Moved the Merkle computation code to a separate file (independent of CBlock and CTransaction), and used that in a thin wrapper caller in primitives/block.h.

@dcousens
Copy link
Contributor

dcousens commented Nov 18, 2015

re-ACK

Also, awesome on moving merkle.cpp out 👍

@jtimon
Copy link
Member

jtimon commented Nov 23, 2015

If we make it consensus/merkle.o, that would be less code to move again later...

@sipa
Copy link
Member Author

sipa commented Nov 26, 2015

@jtimon Hmm, I was seeing this more as generic data structure code, like limitedmap.h is for example, but it's probably not useful to anything that doesn't also need the consensus logic anyway.

@laanwj
Copy link
Member

laanwj commented Nov 26, 2015

Agree that Merkle trees are a general data structure - however, the specific way of doing a Merkle-tree is very bitcoin-consensus-specific, and there is the big warning that this is not recommended to use this as-is outside of bitcoin.

@jtimon
Copy link
Member

jtimon commented Nov 26, 2015

General data structure or not, this is used in checkBlock and will therefore be part of VerifyBlock and the complete libconsensus too.
Once it is complete, if we want to separate it as a subtree with its own repo, we will have to put all the files in the same dir (ie src/consensus) first.

@laanwj
Copy link
Member

laanwj commented Nov 26, 2015

@jtimon I agree

@sipa sipa force-pushed the sipa:constmerkle branch Nov 26, 2015
@sipa
Copy link
Member Author

sipa commented Nov 26, 2015

Done.

I had to move more things around, because otherwise primitives and consensus would end up with a circular dependency on each other. Now all logic (both generic Merkle logic, and the block-specific code) are in consensus, which is better as none of it really belonged in primitives anyway.

@jtimon
Copy link
Member

jtimon commented Nov 26, 2015

Thank you.
I wouldn't worry about consensus/merkle depending on primitives/transaction while primitives/block depends on consensus/merkle. primitives is not a building package and they all three belong in the consensus building package (see WIP #7091) and eventually they should all be in the consensus dir anyway.
A circular dependency between consensus/merkle and primitives/block is another thing though.
I can't remember how the includes were before, but they look good to me now.

@sipa sipa force-pushed the sipa:constmerkle branch 2 times, most recently Nov 26, 2015
@jtimon
jtimon reviewed Nov 26, 2015
View changes
src/test/main_tests.cpp Outdated
@@ -73,4 +74,127 @@ BOOST_AUTO_TEST_CASE(test_combiner_all)
BOOST_CHECK(Test());
}

// Older version of the merkle root computation code, for comparison.

This comment has been minimized.

Copy link
@jtimon

jtimon Nov 26, 2015

Member

minor nit, is main_tests.cpp really the right place for this?

This comment has been minimized.

Copy link
@sipa

sipa Nov 27, 2015

Author Member

Have a better suggestion? :)

This comment has been minimized.

Copy link
@jtimon

jtimon Nov 27, 2015

Member

I don't know, checkblock_tests.cpp? maybe its own merkle_test.cpp ?
The only use in main.cpp is in CheckBlock(), which should eventually move to the consensus dir...

This comment has been minimized.

Copy link
@sipa

sipa Nov 27, 2015

Author Member

Moved!

@laanwj
Copy link
Member

laanwj commented Nov 27, 2015

utACK

sipa added 2 commits Nov 17, 2015
This switches the Merkle tree logic for blocks to one that runs in constant (small) space.
The old code is moved to tests, and a new test is added that for various combinations of
block sizes, transaction positions to compute a branch for, and mutations:
 * Verifies that the old code and new code agree for the Merkle root.
 * Verifies that the old code and new code agree for the Merkle branch.
 * Verifies that the computed Merkle branch is valid.
 * Verifies that mutations don't change the Merkle root.
 * Verifies that mutations are correctly detected.
@sipa sipa force-pushed the sipa:constmerkle branch to eece63f Nov 27, 2015
@jtimon
Copy link
Member

jtimon commented Nov 27, 2015

Oh, thought I had done it already...concept ACK

@sipa sipa merged commit eece63f into bitcoin:master Nov 28, 2015
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
sipa added a commit that referenced this pull request Nov 28, 2015
eece63f Switch blocks to a constant-space Merkle root/branch algorithm. (Pieter Wuille)
ee60e56 Add merkle.{h,cpp}, generic merkle root/branch algorithm (Pieter Wuille)
@jgarzik
Copy link
Contributor

jgarzik commented Nov 28, 2015

utACK

Warrows added a commit to Warrows/PIVX that referenced this pull request Jul 20, 2019
Backport of bitcoin#6508

This switches the Merkle tree logic for blocks to one that runs in
constant (small) space.
The old code is moved to tests, and a new test is added that for various
combinations of
block sizes, transaction positions to compute a branch for, and
mutations:
 * Verifies that the old code and new code agree for the Merkle root.
 * Verifies that the old code and new code agree for the Merkle branch.
 * Verifies that the computed Merkle branch is valid.
 * Verifies that mutations don't change the Merkle root.
 * Verifies that mutations are correctly detected.
Warrows added a commit to Warrows/PIVX that referenced this pull request Jul 28, 2019
Backport of bitcoin#6508

This switches the Merkle tree logic for blocks to one that runs in
constant (small) space.
The old code is moved to tests, and a new test is added that for various
combinations of
block sizes, transaction positions to compute a branch for, and
mutations:
 * Verifies that the old code and new code agree for the Merkle root.
 * Verifies that the old code and new code agree for the Merkle branch.
 * Verifies that the computed Merkle branch is valid.
 * Verifies that mutations don't change the Merkle root.
 * Verifies that mutations are correctly detected.
Warrows added a commit to Warrows/PIVX that referenced this pull request Aug 4, 2019
Backport of bitcoin#6508

This switches the Merkle tree logic for blocks to one that runs in
constant (small) space.
The old code is moved to tests, and a new test is added that for various
combinations of
block sizes, transaction positions to compute a branch for, and
mutations:
 * Verifies that the old code and new code agree for the Merkle root.
 * Verifies that the old code and new code agree for the Merkle branch.
 * Verifies that the computed Merkle branch is valid.
 * Verifies that mutations don't change the Merkle root.
 * Verifies that mutations are correctly detected.
Warrows added a commit to Warrows/PIVX that referenced this pull request Sep 11, 2019
Backport of bitcoin#6508

This switches the Merkle tree logic for blocks to one that runs in
constant (small) space.
The old code is moved to tests, and a new test is added that for various
combinations of
block sizes, transaction positions to compute a branch for, and
mutations:
 * Verifies that the old code and new code agree for the Merkle root.
 * Verifies that the old code and new code agree for the Merkle branch.
 * Verifies that the computed Merkle branch is valid.
 * Verifies that mutations don't change the Merkle root.
 * Verifies that mutations are correctly detected.
Warrows added a commit to Warrows/PIVX that referenced this pull request Sep 23, 2019
Backport of bitcoin#6508

This switches the Merkle tree logic for blocks to one that runs in
constant (small) space.
The old code is moved to tests, and a new test is added that for various
combinations of
block sizes, transaction positions to compute a branch for, and
mutations:
 * Verifies that the old code and new code agree for the Merkle root.
 * Verifies that the old code and new code agree for the Merkle branch.
 * Verifies that the computed Merkle branch is valid.
 * Verifies that mutations don't change the Merkle root.
 * Verifies that mutations are correctly detected.
Warrows added a commit to Warrows/PIVX that referenced this pull request Oct 9, 2019
Backport of bitcoin#6508

This switches the Merkle tree logic for blocks to one that runs in
constant (small) space.
The old code is moved to tests, and a new test is added that for various
combinations of
block sizes, transaction positions to compute a branch for, and
mutations:
 * Verifies that the old code and new code agree for the Merkle root.
 * Verifies that the old code and new code agree for the Merkle branch.
 * Verifies that the computed Merkle branch is valid.
 * Verifies that mutations don't change the Merkle root.
 * Verifies that mutations are correctly detected.
random-zebra added a commit to PIVX-Project/PIVX that referenced this pull request Oct 9, 2019
91b48c7 [Build] Add new merkle files to CMake lists (warrows)
48a3aff [Wallet] Ignore coinbase and zc tx "conflicts" (warrows)
3572354 [Wallet] Fix an error in tx depth computation (warrows)
6928369 [Tests] Enable abandonconflict functional test (warrows)
34cd496 Fix that CWallet::AbandonTransaction would only traverse one level (Ben Woosley)
aba5b75 Fix calculation of balances and available coins. (Alex Morcos)
48d705f Remove stale wallet transactions on initial load. (presstab)
12985ae Flush wallet after abandontransaction (Alex Morcos)
8f87956 [Wallet] sort pending wallet transactions before reaccepting (dexX7)
9c2f445 [Wallet] Call notification signal when a transaction is abandoned (Jonas Schnelli)
778ebf3 Add new rpc call: abandontransaction (Alex Morcos)
0e86c3e Make wallet descendant searching more efficient (Alex Morcos)
d0083a8 Make sure conflicted wallet tx's update balances (Alex Morcos)
6a50e03 [Wallet] Keep track of explicit wallet conflicts instead of using mempool (warrows)
7ccb2b5 [Wallet] Do not flush the wallet in AddToWalletIfInvolvingMe(..) (warrows)
47345be [Refactor] Move wallet functions out of header (warrows)
ab9efb8 [Wallet] Switch to a constant-space Merkle root/branch algorithm (warrows)
5447622 [Wallet] Do not store Merkle branches in the wallet (warrows)

Pull request description:

  This pull request is a happy melting pot of improvements regarding transactions handling. Most of them are backports from bitcoin. I advise reviewers to check the code of the different commits independently to understand them more easily. However, testing is probably better done all at once.
  I am making a single pull request because these changes are all entangled and introducing some of them without others would probably introduce temporary bugs.

  ## Commits details ##
  - 6c3e2ac backport of bitcoin#6550
  - 5304fdf backport of bitcoin#6508
  - c3eeeac simple code move from the header to the cpp file. It contains no functional change.
  - 6cc4d37 backport of bitcoin#4805
  - 10be1db backport of bitcoin#7105
  - 8a34c32 backport of bitcoin#7306
  - 3caf123, 9e17178 and 240f5b4 are the backport for bitcoin#7312
  - ad6d0b1 backport of bitcoin#5511
  - fcc07c3 backport of bitcoin#9311
  - 5ed5e26 is an update of #825
  - 392d504 backport of bitcoin#7715
  - 7199f3a backport of bitcoin#13652
  - f09d999 enables and fixes the test from bitcoin#7312
  - 4fd43c5 fixes an oversight in bitcoin#7105 backport

ACKs for top commit:
  random-zebra:
    ACK 91b48c7
  Fuzzbawls:
    ACK 91b48c7

Tree-SHA512: 2628cebe98805b8048b920b51ee26fd4f0c53643d78da9b8cb265aede52dfe1d40c8c19d34293c232c5c35be7f1ab89ff5b4a07073a4b27c371ea70eb8708669
CaveSpectre11 added a commit to CaveSpectre11/PIVX that referenced this pull request Dec 14, 2019
Backport of bitcoin#6508

This switches the Merkle tree logic for blocks to one that runs in
constant (small) space.
The old code is moved to tests, and a new test is added that for various
combinations of
block sizes, transaction positions to compute a branch for, and
mutations:
 * Verifies that the old code and new code agree for the Merkle root.
 * Verifies that the old code and new code agree for the Merkle branch.
 * Verifies that the computed Merkle branch is valid.
 * Verifies that mutations don't change the Merkle root.
 * Verifies that mutations are correctly detected.
wqking added a commit to wqking/Vitae that referenced this pull request May 24, 2020
Backport of bitcoin/bitcoin#6508

This switches the Merkle tree logic for blocks to one that runs in
constant (small) space.
The old code is moved to tests, and a new test is added that for various
combinations of
block sizes, transaction positions to compute a branch for, and
mutations:
 * Verifies that the old code and new code agree for the Merkle root.
 * Verifies that the old code and new code agree for the Merkle branch.
 * Verifies that the computed Merkle branch is valid.
 * Verifies that mutations don't change the Merkle root.
 * Verifies that mutations are correctly detected.

# Conflicts:
#	src/test/pmt_tests.cpp
Kokary added a commit to Kokary/wagerr that referenced this pull request May 26, 2020
Backport of bitcoin/bitcoin#6508

This switches the Merkle tree logic for blocks to one that runs in
constant (small) space.
The old code is moved to tests, and a new test is added that for various
combinations of
block sizes, transaction positions to compute a branch for, and
mutations:
 * Verifies that the old code and new code agree for the Merkle root.
 * Verifies that the old code and new code agree for the Merkle branch.
 * Verifies that the computed Merkle branch is valid.
 * Verifies that mutations don't change the Merkle root.
 * Verifies that mutations are correctly detected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants
You can’t perform that action at this time.