Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not store Merkle branches in the wallet. #6550

Merged
merged 2 commits into from Sep 23, 2015
Merged

Conversation

sipa
Copy link
Member

@sipa sipa commented Aug 11, 2015

Assume that when a wallet transaction has a valid block hash and transaction position
in it, the transaction is actually there. We're already trusting wallet data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for storing the
block locator in the wallet. Old wallets will see a wallet file synchronized up
to the genesis block, and rescan automatically.

Fixes #6536 using a suggestion by @jonasschnelli to retain backward compatibility.

@dcousens
Copy link
Contributor

dcousens commented Aug 12, 2015

utACK

@@ -15,7 +15,7 @@ uint256 CBlockHeader::GetHash() const
return SerializeHash(*this);
}

uint256 CBlock::BuildMerkleTree(bool* fMutated) const
uint256 CBlock::ComputeMerkleRoot(bool* fMutated) const
Copy link
Contributor

@dcousens dcousens Aug 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to this PR, but is there any reason we prefer the optional arguments compared to a tuple return value?

Copy link
Member

@laanwj laanwj Aug 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is so much "preferred", it's how it happens to be written between two more-or-less equivalent ways to formulate it.

Copy link
Member Author

@sipa sipa Aug 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're relying on an optional optimization allowed by the C++ standard (copy elision) to avoid constructing a tuple as a temporary and copying it to the return location. In C++11 it's better because you can explicitly indicate move semantics.

@laanwj
Copy link
Member

laanwj commented Aug 12, 2015

Concept ACK.

@laanwj laanwj added the Wallet label Aug 12, 2015
@jtimon
Copy link
Member

jtimon commented Aug 12, 2015

Concept ACK

@sipa
Copy link
Member Author

sipa commented Aug 12, 2015

This needs testing :)

@jonasschnelli
Copy link
Contributor

jonasschnelli commented Aug 12, 2015

Concept ACK. Started testing...

@jonasschnelli
Copy link
Contributor

jonasschnelli commented Aug 12, 2015

Tested ACK (lldb stepped; update of "old" wallet.dat works). Played around with some huge regtest wallets/chains. Moved wallet.dat back and forth from master bitcoin-core to master+this PR bitcoin-core.

I think this is a step forward. IMO it's totally sufficient to check if a wtx has a valid block in the active chain and the wtx transaction index matches the position in the block.

Next step could be to simplify the height calculation.
GetDepthInMainChainINTERNAL() is very ineffective and holds cs_main because it accesses mapBlockIndex. Calculating balance, UI listing of transactions will call GetDepthInMainChainINTERNAL() for each wallet transaction (! and it even gets called when the credit of a wtx is cached).

I think a height cache would perform better (together with a invalidating option down to a certain height in case of a reorg).

@sipa
Copy link
Member Author

sipa commented Aug 12, 2015

Thanks for testing!

@jonasschnelli No need for a height cache, just a cached chainActive.Tip() inside the wallet suffices (you can do ancestor-of checks without cs_main). I can do that, but it's not for this PR.

@theuni
Copy link
Member

theuni commented Aug 12, 2015

This will cause the merkle tree to be calculated at least 3x per accepted block, as far as I can see:
ProcessNewBlock -> CheckBlock
ProcessNewBlock -> AcceptBlock -> CheckBlock
ProcessNewBlock -> ActivateBestChain -> ... ConnectBlock -> CheckBlock

Is the calculation time significant enough to try to skip some of that?

@sipa
Copy link
Member Author

sipa commented Aug 12, 2015

@theuni Ugh, that's bad. No, it won't make up for that. We should avoid doing those duplicated checks.

@theuni
Copy link
Member

theuni commented Aug 12, 2015

@sipa If I'm reading correctly, it looks like the CheckBlock() in AcceptBlock() is safe to use !fCheckMerkleRoot since it's only ever called from ProcessNewBlock() where it's already been checked.

Same for the ConnectBlock() in ConnectTip(), since they're already checked before writing to disk. That seems a bit more risky, but surely a bad/corrupt read would be detected long before that point anyway?

@sipa
Copy link
Member Author

sipa commented Aug 13, 2015

@sipa
Copy link
Member Author

sipa commented Aug 15, 2015

@theuni It looks a bit more complicated, as currently we do a CheckBlock on blocks read from disk for validation, which would be harder to do if we want to avoid duplicate checks. I've made a much simpler (and slightly uglier) change for now here, which is to cache the result of CheckBlock in CBlock. That also avoids a few other checks that were being done 3 times before.

@gmaxwell
Copy link
Contributor

gmaxwell commented Sep 6, 2015

Where does this stand right now? Should I spend cycles testing it as is?

@sipa
Copy link
Member Author

sipa commented Sep 7, 2015

@gmaxwell Test away.

@sipa
Copy link
Member Author

sipa commented Sep 14, 2015

Ping

@@ -2573,7 +2576,7 @@ bool CheckBlock(const CBlock& block, CValidationState& state, bool fCheckPOW, bo
// Check the merkle root.
if (fCheckMerkleRoot) {
bool mutated;
uint256 hashMerkleRoot2 = block.BuildMerkleTree(&mutated);
uint256 hashMerkleRoot2 = block.ComputeMerkleRoot(&mutated);
Copy link
Contributor

@dcousens dcousens Sep 14, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const?

Copy link
Member Author

@sipa sipa Sep 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const what?

@dcousens
Copy link
Contributor

dcousens commented Sep 14, 2015

re-utACK

sipa added 2 commits Sep 22, 2015
Assume that when a wallet transaction has a valid block hash and transaction position
in it, the transaction is actually there. We're already trusting wallet data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for storing the
block locator in the wallet. Old wallets will see a wallet file synchronized up
to the genesis block, and rescan automatically.
@sipa
Copy link
Member Author

sipa commented Sep 22, 2015

Rebased.

@@ -78,7 +78,7 @@ class CBlock : public CBlockHeader
std::vector<CTransaction> vtx;

// memory only
mutable std::vector<uint256> vMerkleTree;
mutable bool fChecked;
Copy link
Member

@laanwj laanwj Sep 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we'd have this fChecked status (which isn't used in CBlock itself) on an administrative object that wraps a CBlock, instead of on CBlock itself.

Copy link
Member Author

@sipa sipa Sep 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fully agree, I actually started implementing that as a follow-up, but didn't want to interfere with other refactorings.

Copy link
Member

@laanwj laanwj Sep 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, yes, I'm fine with keeping it like this for this pull

@laanwj
Copy link
Member

laanwj commented Sep 23, 2015

Tested forward/backward compatibility of wallet with this code change.
ACK.

@laanwj laanwj merged commit 3b33ec8 into bitcoin:master Sep 23, 2015
1 check passed
laanwj added a commit that referenced this issue Sep 23, 2015
3b33ec8 Avoid duplicate CheckBlock checks (Pieter Wuille)
391dff1 Do not store Merkle branches in the wallet. (Pieter Wuille)
Warrows added a commit to Warrows/PIVX that referenced this issue Jul 20, 2019
Backport of bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Warrows added a commit to Warrows/PIVX that referenced this issue Jul 28, 2019
Backport of bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Warrows added a commit to Warrows/PIVX that referenced this issue Aug 4, 2019
Backport of bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Warrows added a commit to Warrows/PIVX that referenced this issue Sep 11, 2019
Backport of bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Warrows added a commit to Warrows/PIVX that referenced this issue Sep 23, 2019
Backport of bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Warrows added a commit to Warrows/PIVX that referenced this issue Oct 9, 2019
Backport of bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
random-zebra added a commit to PIVX-Project/PIVX that referenced this issue Oct 9, 2019
91b48c7 [Build] Add new merkle files to CMake lists (warrows)
48a3aff [Wallet] Ignore coinbase and zc tx "conflicts" (warrows)
3572354 [Wallet] Fix an error in tx depth computation (warrows)
6928369 [Tests] Enable abandonconflict functional test (warrows)
34cd496 Fix that CWallet::AbandonTransaction would only traverse one level (Ben Woosley)
aba5b75 Fix calculation of balances and available coins. (Alex Morcos)
48d705f Remove stale wallet transactions on initial load. (presstab)
12985ae Flush wallet after abandontransaction (Alex Morcos)
8f87956 [Wallet] sort pending wallet transactions before reaccepting (dexX7)
9c2f445 [Wallet] Call notification signal when a transaction is abandoned (Jonas Schnelli)
778ebf3 Add new rpc call: abandontransaction (Alex Morcos)
0e86c3e Make wallet descendant searching more efficient (Alex Morcos)
d0083a8 Make sure conflicted wallet tx's update balances (Alex Morcos)
6a50e03 [Wallet] Keep track of explicit wallet conflicts instead of using mempool (warrows)
7ccb2b5 [Wallet] Do not flush the wallet in AddToWalletIfInvolvingMe(..) (warrows)
47345be [Refactor] Move wallet functions out of header (warrows)
ab9efb8 [Wallet] Switch to a constant-space Merkle root/branch algorithm (warrows)
5447622 [Wallet] Do not store Merkle branches in the wallet (warrows)

Pull request description:

  This pull request is a happy melting pot of improvements regarding transactions handling. Most of them are backports from bitcoin. I advise reviewers to check the code of the different commits independently to understand them more easily. However, testing is probably better done all at once.
  I am making a single pull request because these changes are all entangled and introducing some of them without others would probably introduce temporary bugs.

  ## Commits details ##
  - 6c3e2ac backport of bitcoin#6550
  - 5304fdf backport of bitcoin#6508
  - c3eeeac simple code move from the header to the cpp file. It contains no functional change.
  - 6cc4d37 backport of bitcoin#4805
  - 10be1db backport of bitcoin#7105
  - 8a34c32 backport of bitcoin#7306
  - 3caf123, 9e17178 and 240f5b4 are the backport for bitcoin#7312
  - ad6d0b1 backport of bitcoin#5511
  - fcc07c3 backport of bitcoin#9311
  - 5ed5e26 is an update of #825
  - 392d504 backport of bitcoin#7715
  - 7199f3a backport of bitcoin#13652
  - f09d999 enables and fixes the test from bitcoin#7312
  - 4fd43c5 fixes an oversight in bitcoin#7105 backport

ACKs for top commit:
  random-zebra:
    ACK 91b48c7
  Fuzzbawls:
    ACK 91b48c7

Tree-SHA512: 2628cebe98805b8048b920b51ee26fd4f0c53643d78da9b8cb265aede52dfe1d40c8c19d34293c232c5c35be7f1ab89ff5b4a07073a4b27c371ea70eb8708669
CaveSpectre11 pushed a commit to CaveSpectre11/PIVX that referenced this issue Dec 14, 2019
Backport of bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
wqking pushed a commit to wqking-temp/Vitae that referenced this issue May 24, 2020
Backport of bitcoin/bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Kokary pushed a commit to Kokary/wagerr that referenced this issue May 26, 2020
Backport of bitcoin/bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Kokary pushed a commit to Kokary/wagerr that referenced this issue Nov 13, 2020
Backport of bitcoin/bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Kokary pushed a commit to Kokary/wagerr that referenced this issue Nov 17, 2020
Backport of bitcoin/bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Kokary pushed a commit to Kokary/wagerr that referenced this issue Nov 17, 2020
Backport of bitcoin/bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
KolbyML pushed a commit to KolbyML/bitcoin that referenced this issue Nov 21, 2020
Backport of bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Kokary pushed a commit to wagerr/wagerr that referenced this issue Nov 24, 2020
Backport of bitcoin/bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
Cryptarchist pushed a commit to wagerr/wagerr that referenced this issue Nov 30, 2020
Backport of bitcoin/bitcoin#6550

Assume that when a wallet transaction has a valid block hash and
transaction position
in it, the transaction is actually there. We're already trusting wallet
data in a
much more fundamental way anyway.

To prevent backward compatibility issues, a new record is used for
storing the
block locator in the wallet. Old wallets will see a wallet file
synchronized up
to the genesis block, and rescan automatically.
zkbot added a commit to zcash/zcash that referenced this issue Apr 20, 2021
Bitcoin 0.12 Merkle tree PRs

Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#6550
- bitcoin/bitcoin#6508
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants