New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete hybrid full block SPV mode #9483

Open
wants to merge 26 commits into
base: master
from

Conversation

@jonasschnelli
Member

jonasschnelli commented Jan 6, 2017

This is the complete patch-set for the hybrid full block SPV mode.

If one enables the SPV mode with -spv=1 it does...

  • ...first sync all headers (no block downloads during that phase)
  • ...requests and persist all blocks that are relevant for the wallet (down to the dept of the older wallet key)
  • ...scan the block for relevant transactions and flag them with validated = false (visible in listtransactions etc).
  • ... continue with IBD (initial block download) after all wallet relevant blocks have been processed

Pure full block SPV mode is possible by setting -autorequestblocks=0, in that mode, no blocks for validating the chain will be downloaded, resulting in a SPV only mode.

For better testing, this PR also includes a bump to 0.0005 for the default fallback fee.

Including all required GUI changes and RPC tests:

Screenshots:

bildschirmfoto 2017-01-06 um 17 21 24

untitled-1

bildschirmfoto 2017-01-06 um 17 34 09

untitled-2

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Jan 6, 2017

Member

(Prefer if we don't propagate the misuse of "SPV" for things that don't support fraud proofs)

Member

luke-jr commented Jan 6, 2017

(Prefer if we don't propagate the misuse of "SPV" for things that don't support fraud proofs)

@gmaxwell

This comment has been minimized.

Show comment
Hide comment
@gmaxwell

gmaxwell Jan 6, 2017

Member

I kinda want to use an open lock icon instead of the likely meaningless to uses SPV in any case.

We also should do something about the confirmed counts in this mode. I'm not sure what. The issue is that confirmations mean less when you're not validating. Perhaps displaying transactions like they are unconfirmed until they have 6 blocks might be the thing to do. Or displaying a visible "not-verified" on any transaction with the not validated flag.

Member

gmaxwell commented Jan 6, 2017

I kinda want to use an open lock icon instead of the likely meaningless to uses SPV in any case.

We also should do something about the confirmed counts in this mode. I'm not sure what. The issue is that confirmations mean less when you're not validating. Perhaps displaying transactions like they are unconfirmed until they have 6 blocks might be the thing to do. Or displaying a visible "not-verified" on any transaction with the not validated flag.

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Jan 6, 2017

Member

Indeed, I would assume any mode like this shouldn't count confirmation at all.

Member

luke-jr commented Jan 6, 2017

Indeed, I would assume any mode like this shouldn't count confirmation at all.

@dabura667

This comment has been minimized.

Show comment
Hide comment
@dabura667

dabura667 Jan 7, 2017

How about a flag for showing confirmations during SPV mode? Default to off.

People who understand the implications and just don't want to bother having to search their address on an explorer can enable in the menu / config

dabura667 commented Jan 7, 2017

How about a flag for showing confirmations during SPV mode? Default to off.

People who understand the implications and just don't want to bother having to search their address on an explorer can enable in the menu / config

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Jan 7, 2017

Member

@dabura667 Is it sufficient to simply show it in the transaction details dialog, perhaps?

Member

luke-jr commented Jan 7, 2017

@dabura667 Is it sufficient to simply show it in the transaction details dialog, perhaps?

@dabura667

This comment has been minimized.

Show comment
Hide comment
@dabura667

dabura667 Jan 7, 2017

@luke-jr I would think so, yes.

dabura667 commented Jan 7, 2017

@luke-jr I would think so, yes.

@molxyz

This comment has been minimized.

Show comment
Hide comment
@molxyz

molxyz Jan 8, 2017

On transactions screen, fully-confirmed receiving txs show only one confirmation, and fully-confirmed sending txs show "unconfirmed" with question marks.
spv-txscreen

molxyz commented Jan 8, 2017

On transactions screen, fully-confirmed receiving txs show only one confirmation, and fully-confirmed sending txs show "unconfirmed" with question marks.
spv-txscreen

@diegoviola

This comment has been minimized.

Show comment
Hide comment
@diegoviola

diegoviola Jan 8, 2017

Contributor

@molxyz I've also noticed the same thing, confirmations in Transactions list don't update, unless I restart Core.

Other than that it works great.

I used this for testing: ./bitcoin-qt -spv=1 -autorequestblocks=0 -testnet.

Contributor

diegoviola commented Jan 8, 2017

@molxyz I've also noticed the same thing, confirmations in Transactions list don't update, unless I restart Core.

Other than that it works great.

I used this for testing: ./bitcoin-qt -spv=1 -autorequestblocks=0 -testnet.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Jan 8, 2017

Member

Thanks for reporting. This seems to be a UI update issue. Will fix it in the next overhaul / PR update.

Member

jonasschnelli commented Jan 8, 2017

Thanks for reporting. This seems to be a UI update issue. Will fix it in the next overhaul / PR update.

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Jan 20, 2017

Member

Thought: Can this be made to work with external wallets/software?

Member

luke-jr commented Jan 20, 2017

Thought: Can this be made to work with external wallets/software?

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Jan 21, 2017

Member

Thought: Can this be made to work with external wallets/software?

I don't know what you mean by this.
Can you make an use-case example?

Member

jonasschnelli commented Jan 21, 2017

Thought: Can this be made to work with external wallets/software?

I don't know what you mean by this.
Can you make an use-case example?

@jtimon

This comment has been minimized.

Show comment
Hide comment
@jtimon

jtimon Jan 24, 2017

Member

Needs rebase.
Concept ACK

What happens with -autorequestblocks=0 -spv=0?
I assume both -spv and -autorequestblocks are 1 by default. With -spv=0 -autorequestblocks=1 you would get what you have today, but is it really so useful compared to -spv=1 -autorequestblocks=1 ?

I don't know it seems overly complicated. I thought we would just have a single param spvonly that defaults to 0 (equivalent to this autorequestblocks, and your spv is always =1, ie spvonly=0 equivalent to -spv=1 -autorequestblocks=1, spvonly=1 equivalent to -spv=1 -autorequestblocks=0). Not sure, just thinking out loud.

Member

jtimon commented Jan 24, 2017

Needs rebase.
Concept ACK

What happens with -autorequestblocks=0 -spv=0?
I assume both -spv and -autorequestblocks are 1 by default. With -spv=0 -autorequestblocks=1 you would get what you have today, but is it really so useful compared to -spv=1 -autorequestblocks=1 ?

I don't know it seems overly complicated. I thought we would just have a single param spvonly that defaults to 0 (equivalent to this autorequestblocks, and your spv is always =1, ie spvonly=0 equivalent to -spv=1 -autorequestblocks=1, spvonly=1 equivalent to -spv=1 -autorequestblocks=0). Not sure, just thinking out loud.

jonasschnelli added some commits Nov 15, 2016

Pass CBlockRequest blocks through SyncTransaction signal
+ Adds a validate=true|false to the SyncTransaction signal
Add CAuxiliaryBlockRequest, a class to handle auxiliary blocks downloads
This is required for features that need to download and process block higher/further-away then the current validation depth
Add -autorequestblocks debug option and setautorequestblocks hidden R…
…PC call

This allows efficient testing of auxiliary block requests
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Jan 24, 2017

Member

Rebased.

What happens with -autorequestblocks=0 -spv=0?

This would result in a mode where no blocks are automatically requested (only headers are fetched).
autorequestblocks=0 is a debug option and I could imagine some interesting use-cases where you only want to fetch certain blocks with requestblocks RPC call.

Member

jonasschnelli commented Jan 24, 2017

Rebased.

What happens with -autorequestblocks=0 -spv=0?

This would result in a mode where no blocks are automatically requested (only headers are fetched).
autorequestblocks=0 is a debug option and I could imagine some interesting use-cases where you only want to fetch certain blocks with requestblocks RPC call.

jonasschnelli added some commits Dec 20, 2016

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Jul 11, 2017

Member

@ryanofsky: Thanks for your review and sorry for the late response.

  • About the idea of getting rid of CAuxiliaryBlockRequest:
    I think keeping it in a separate file/class allows simpler rebases. I expect to rebase that PR a lot. Also, clustering to much into net_processing would result against in a moster-class/Impl.-file that does everything net related. I think in terms of architecture, splitting of stuff into separate classes/files makes sense.

  • SPV versus non-validating mode:
    I haven't really found the ideal term. A first sight, SPV seems to miss the point (if we assume SPV = bloom filter, though I disagree here), but is does allow everybody quickly understand what this PR does. If we look at Satoshi's white paper "Simplified Payment Verification" (chapter 8) then I guess this is more or less what this PR is about. That's why I haven't given up on calling it SPV.
    Client-mode seems wrong-ish to me, because no "server" is involved.
    Non-validating mode seems to nail it, but it implies we validate nothing (we still validate headers/PoW) and therefore gives it a negative general direction. Ideally the term should not include what we not do (non) and should be formed in a positive way.

Any objections calling this SPV mode?
Also, very likely, this mode will once have client side filtering.

Member

jonasschnelli commented Jul 11, 2017

@ryanofsky: Thanks for your review and sorry for the late response.

  • About the idea of getting rid of CAuxiliaryBlockRequest:
    I think keeping it in a separate file/class allows simpler rebases. I expect to rebase that PR a lot. Also, clustering to much into net_processing would result against in a moster-class/Impl.-file that does everything net related. I think in terms of architecture, splitting of stuff into separate classes/files makes sense.

  • SPV versus non-validating mode:
    I haven't really found the ideal term. A first sight, SPV seems to miss the point (if we assume SPV = bloom filter, though I disagree here), but is does allow everybody quickly understand what this PR does. If we look at Satoshi's white paper "Simplified Payment Verification" (chapter 8) then I guess this is more or less what this PR is about. That's why I haven't given up on calling it SPV.
    Client-mode seems wrong-ish to me, because no "server" is involved.
    Non-validating mode seems to nail it, but it implies we validate nothing (we still validate headers/PoW) and therefore gives it a negative general direction. Ideally the term should not include what we not do (non) and should be formed in a positive way.

Any objections calling this SPV mode?
Also, very likely, this mode will once have client side filtering.

@ryanofsky

This comment has been minimized.

Show comment
Hide comment
@ryanofsky

ryanofsky Jul 11, 2017

Contributor

About the idea of getting rid of CAuxiliaryBlockRequest

See #10794 (comment)

Any objections calling this SPV mode?

I don't like it, but I wouldn't object to a useful feature because it has a confusing name, and my complaints above are more about naming inconsistency than about this name in particular. Also, I wish you would respond to my some of my suggestions in detail. I wasn't suggesting renaming "SPV mode" to "non-validating mode" or to "client-mode" (I don't even know where "client-mode" comes from). I suggested renaming the -spv flag to -priorityrequestblocks, to be consistent with -autorequestblocks flag, and because the point of the feature is to be smarter about the order blocks are downloaded.

Contributor

ryanofsky commented Jul 11, 2017

About the idea of getting rid of CAuxiliaryBlockRequest

See #10794 (comment)

Any objections calling this SPV mode?

I don't like it, but I wouldn't object to a useful feature because it has a confusing name, and my complaints above are more about naming inconsistency than about this name in particular. Also, I wish you would respond to my some of my suggestions in detail. I wasn't suggesting renaming "SPV mode" to "non-validating mode" or to "client-mode" (I don't even know where "client-mode" comes from). I suggested renaming the -spv flag to -priorityrequestblocks, to be consistent with -autorequestblocks flag, and because the point of the feature is to be smarter about the order blocks are downloaded.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Jul 11, 2017

Member

@ryanofsky In early 2011, there was an incomplete feature in the codebase called "client mode", which probably was intended to be some sort of SPV version. It never got finished, and was eventually removed.

Member

sipa commented Jul 11, 2017

@ryanofsky In early 2011, there was an incomplete feature in the codebase called "client mode", which probably was intended to be some sort of SPV version. It never got finished, and was eventually removed.

@ryanofsky

This comment has been minimized.

Show comment
Hide comment
@ryanofsky

ryanofsky Jul 11, 2017

Contributor

That's interesting. I don't think client is a bad name either (seems pretty innocuous). I just hadn't heard it before.

Contributor

ryanofsky commented Jul 11, 2017

That's interesting. I don't think client is a bad name either (seems pretty innocuous). I just hadn't heard it before.

@jonasschnelli jonasschnelli moved this from In progress to Conceptual PR in Client-Mode (SPV) Jul 21, 2017

@nopara73

This comment has been minimized.

Show comment
Hide comment
@nopara73

nopara73 Nov 11, 2017

What would be needed to progress this issue further?

nopara73 commented Nov 11, 2017

What would be needed to progress this issue further?

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Nov 13, 2017

Member

I have plans to soon re-do / overhaul the SPV work...

Member

jonasschnelli commented Nov 13, 2017

I have plans to soon re-do / overhaul the SPV work...

@TheBlueMatt

This comment has been minimized.

Show comment
Hide comment
@TheBlueMatt

TheBlueMatt Dec 6, 2017

Contributor

Instead of headersChainActive, which seems hard to get to automatically reorg to the new-best-headers-chain after a block is found to be invalid when we get the full block, you may want to take a look at https://github.com/TheBlueMatt/bitcoin/commits/2017-10-best-header-tracking which should handle most of that work for you.

Contributor

TheBlueMatt commented Dec 6, 2017

Instead of headersChainActive, which seems hard to get to automatically reorg to the new-best-headers-chain after a block is found to be invalid when we get the full block, you may want to take a look at https://github.com/TheBlueMatt/bitcoin/commits/2017-10-best-header-tracking which should handle most of that work for you.

@Sjors

This comment has been minimized.

Show comment
Hide comment
@Sjors

Sjors Dec 11, 2017

Member

I'll take a look after rebase. From the description:

requests and persist all blocks that are relevant for the wallet

How does it achieve this? BIP 37? In light of #11863, would it make sense to allow dropping in some arbitrary class that figures out which blocks to request?

Member

Sjors commented Dec 11, 2017

I'll take a look after rebase. From the description:

requests and persist all blocks that are relevant for the wallet

How does it achieve this? BIP 37? In light of #11863, would it make sense to allow dropping in some arbitrary class that figures out which blocks to request?

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 21, 2017

Member

How does it achieve this? BIP 37?

No BIP37 involved here, to retain privacy uses full-block SPV mode. The relevant blocks are only those from the birthdate of the wallet on.

Member

laanwj commented Dec 21, 2017

How does it achieve this? BIP 37?

No BIP37 involved here, to retain privacy uses full-block SPV mode. The relevant blocks are only those from the birthdate of the wallet on.

@nopara73 nopara73 referenced this pull request Dec 22, 2017

Closed

Hardware wallet support #130

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Mar 6, 2018

Member

Sorry for the very late comment here, but I think this is introducing a lot of complexity and then building on top of it.

I think a first step should be what @ryanofsky suggested above ("Get rid of the CAuxiliaryBlockRequest class and integrate prioritized block download logic directly into net_processing.cpp so more code responsible for regular and prioritized block downloads can be shared, and the wallet will not have to be involved in batching and sequencing p2p requests."). This is more generally useful than just lightweight mode too; it could be used for rescanning while pruning too, for example.

Member

sipa commented Mar 6, 2018

Sorry for the very late comment here, but I think this is introducing a lot of complexity and then building on top of it.

I think a first step should be what @ryanofsky suggested above ("Get rid of the CAuxiliaryBlockRequest class and integrate prioritized block download logic directly into net_processing.cpp so more code responsible for regular and prioritized block downloads can be shared, and the wallet will not have to be involved in batching and sequencing p2p requests."). This is more generally useful than just lightweight mode too; it could be used for rescanning while pruning too, for example.

@ryanofsky

This comment has been minimized.

Show comment
Hide comment
@ryanofsky

ryanofsky Mar 6, 2018

Contributor

I think a first step should be what @ryanofsky suggested above ("Get rid of the CAuxiliaryBlockRequest class and integrate prioritized block download logic directly into net_processing.cpp so more code responsible for regular and prioritized block downloads can be shared, and the wallet will not have to be involved in batching and sequencing p2p requests.").

This is actually implemented in #10794. @jonasschnelli, it would probably be good to reference #10794 in the PR description, and make sure the PR description is up to date generally.

Contributor

ryanofsky commented Mar 6, 2018

I think a first step should be what @ryanofsky suggested above ("Get rid of the CAuxiliaryBlockRequest class and integrate prioritized block download logic directly into net_processing.cpp so more code responsible for regular and prioritized block downloads can be shared, and the wallet will not have to be involved in batching and sequencing p2p requests.").

This is actually implemented in #10794. @jonasschnelli, it would probably be good to reference #10794 in the PR description, and make sure the PR description is up to date generally.

@ryanofsky

This comment has been minimized.

Show comment
Hide comment
@ryanofsky

ryanofsky Mar 23, 2018

Contributor

@TheBlueMatt pointed out at core dev that the sync implemented here sometimes can't recover from invalid blocks, and that basing this change on #12138 might fix this.

Contributor

ryanofsky commented Mar 23, 2018

@TheBlueMatt pointed out at core dev that the sync implemented here sometimes can't recover from invalid blocks, and that basing this change on #12138 might fix this.

@githubcyc

This comment has been minimized.

Show comment
Hide comment
@githubcyc

githubcyc Jun 5, 2018

hey, how to run this version locally? thx

githubcyc commented Jun 5, 2018

hey, how to run this version locally? thx

@Sjors

This comment has been minimized.

Show comment
Hide comment
@Sjors

Sjors Jun 15, 2018

Member

I'm also in favor of not using the word SPV in the PR description, which does not imply giving up on the term.

schermafbeelding 2018-06-15 om 12 14 15

Can you clarify this:

Pure full block SPV mode is possible by setting -autorequestblocks=0, in that mode, no blocks for validating the chain will be downloaded, resulting in a SPV only mode.

Did you mean no blocks older than the wallet? If so, how do you check if the UTXO set is correct? If you don't, then maybe the flag should have a scarier name like -partial_utxo_set. Maybe it's better to add that option in a followup PR to improve the worst case security we have to reason about here.

@ryanofsky wrote:

But beyond these two things, I don't understand what the wallet "SPV mode" is supposed to indicate. Why would it be helpful to me to know that my wallet is in SPV mode if I don't actually have any nonvalidated transactions? Why would it helpful be to me to know my wallet is not in SPV mode if I do have nonvalidated transactions?

My suggestion would be to show the usual IBD progress bar*, as well as mark transactions as not-fully-validated. In that case there's no need for an extra icon.

  • = you could go really fancy and draw a thin blue slice at the right most edge to indicate that we have a fragmented chain, using a lighter shade to indicate that it might be completely invalid.
Member

Sjors commented Jun 15, 2018

I'm also in favor of not using the word SPV in the PR description, which does not imply giving up on the term.

schermafbeelding 2018-06-15 om 12 14 15

Can you clarify this:

Pure full block SPV mode is possible by setting -autorequestblocks=0, in that mode, no blocks for validating the chain will be downloaded, resulting in a SPV only mode.

Did you mean no blocks older than the wallet? If so, how do you check if the UTXO set is correct? If you don't, then maybe the flag should have a scarier name like -partial_utxo_set. Maybe it's better to add that option in a followup PR to improve the worst case security we have to reason about here.

@ryanofsky wrote:

But beyond these two things, I don't understand what the wallet "SPV mode" is supposed to indicate. Why would it be helpful to me to know that my wallet is in SPV mode if I don't actually have any nonvalidated transactions? Why would it helpful be to me to know my wallet is not in SPV mode if I do have nonvalidated transactions?

My suggestion would be to show the usual IBD progress bar*, as well as mark transactions as not-fully-validated. In that case there's no need for an extra icon.

  • = you could go really fancy and draw a thin blue slice at the right most edge to indicate that we have a fragmented chain, using a lighter shade to indicate that it might be completely invalid.
void CAuxiliaryBlockRequest::processWithPossibleBlock(const std::shared_ptr<const CBlock> pblock, CBlockIndex *pindex)
{
// don't process anything if the request was cancled

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018

Member

Typo found by codespell: cancled

@practicalswift

practicalswift Sep 2, 2018

Member

Typo found by codespell: cancled

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018

Member

This should be fixed throughout in this PR :-)

@practicalswift

practicalswift Sep 2, 2018

Member

This should be fixed throughout in this PR :-)

This comment has been minimized.

@Sjors

Sjors Sep 3, 2018

Member

Maybe we need a linter?

@Sjors

Sjors Sep 3, 2018

Member

Maybe we need a linter?

This comment has been minimized.

@practicalswift

practicalswift Sep 3, 2018

Member

@Sjors Please review #13954 which adds a codespell linter :-)

@practicalswift

practicalswift Sep 3, 2018

Member

@Sjors Please review #13954 which adds a codespell linter :-)

this->processedUpToSize++;
// log some info
LogPrint("net", "BlockRequest: proccessed up to %ld of total requested %ld blocks\n", this->processedUpToSize, this->vBlocksToDownload.size());

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018

Member

Typo found by codespell: proccessed

@practicalswift

practicalswift Sep 2, 2018

Member

Typo found by codespell: proccessed

@@ -481,6 +493,27 @@ void FindNextBlocksToDownload(NodeId nodeid, unsigned int count, std::vector<con
// Make sure pindexBestKnownBlock is up to date, we'll need it.
ProcessBlockAvailability(nodeid);
// if there is an open CAuxiliaryBlockRequest (out-of-band/specific block donwload), privileg it

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018

Member

Typo found by codespell: donwload annd privileg

@practicalswift

practicalswift Sep 2, 2018

Member

Typo found by codespell: donwload annd privileg

return (mapBlocksInFlight.count(pIndexCheck->GetBlockHash()) == 0);
});
// if we haven't completed the individual CAuxiliaryBlockRequest, we wont continue with "normal" IBD

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018

Member

Typo found by codespell: wont

@practicalswift

practicalswift Sep 2, 2018

Member

Typo found by codespell: wont

return;
}
// don't request any other blocks if we are in non autorequest mode (usefull for non-validation mode)

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018

Member

Typo found by codespell: usefull (also below)

@practicalswift

practicalswift Sep 2, 2018

Member

Typo found by codespell: usefull (also below)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment