Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete hybrid full block SPV mode #9483

Closed
wants to merge 26 commits into from

Conversation

@jonasschnelli
Copy link
Member

@jonasschnelli jonasschnelli commented Jan 6, 2017

This is the complete patch-set for the hybrid full block SPV mode.

If one enables the SPV mode with -spv=1 it does...

  • ...first sync all headers (no block downloads during that phase)
  • ...requests and persist all blocks that are relevant for the wallet (down to the dept of the older wallet key)
  • ...scan the block for relevant transactions and flag them with validated = false (visible in listtransactions etc).
  • ... continue with IBD (initial block download) after all wallet relevant blocks have been processed

Pure full block SPV mode is possible by setting -autorequestblocks=0, in that mode, no blocks for validating the chain will be downloaded, resulting in a SPV only mode.

For better testing, this PR also includes a bump to 0.0005 for the default fallback fee.

Including all required GUI changes and RPC tests:

Screenshots:

bildschirmfoto 2017-01-06 um 17 21 24

untitled-1

bildschirmfoto 2017-01-06 um 17 34 09

untitled-2

@luke-jr
Copy link
Member

@luke-jr luke-jr commented Jan 6, 2017

(Prefer if we don't propagate the misuse of "SPV" for things that don't support fraud proofs)

@gmaxwell
Copy link
Contributor

@gmaxwell gmaxwell commented Jan 6, 2017

I kinda want to use an open lock icon instead of the likely meaningless to uses SPV in any case.

We also should do something about the confirmed counts in this mode. I'm not sure what. The issue is that confirmations mean less when you're not validating. Perhaps displaying transactions like they are unconfirmed until they have 6 blocks might be the thing to do. Or displaying a visible "not-verified" on any transaction with the not validated flag.

@luke-jr
Copy link
Member

@luke-jr luke-jr commented Jan 6, 2017

Indeed, I would assume any mode like this shouldn't count confirmation at all.

@dabura667
Copy link

@dabura667 dabura667 commented Jan 7, 2017

How about a flag for showing confirmations during SPV mode? Default to off.

People who understand the implications and just don't want to bother having to search their address on an explorer can enable in the menu / config

@luke-jr
Copy link
Member

@luke-jr luke-jr commented Jan 7, 2017

@dabura667 Is it sufficient to simply show it in the transaction details dialog, perhaps?

@dabura667
Copy link

@dabura667 dabura667 commented Jan 7, 2017

@luke-jr I would think so, yes.

@molxyz
Copy link

@molxyz molxyz commented Jan 8, 2017

On transactions screen, fully-confirmed receiving txs show only one confirmation, and fully-confirmed sending txs show "unconfirmed" with question marks.
spv-txscreen

@diegoviola
Copy link
Contributor

@diegoviola diegoviola commented Jan 8, 2017

@molxyz I've also noticed the same thing, confirmations in Transactions list don't update, unless I restart Core.

Other than that it works great.

I used this for testing: ./bitcoin-qt -spv=1 -autorequestblocks=0 -testnet.

@jonasschnelli
Copy link
Member Author

@jonasschnelli jonasschnelli commented Jan 8, 2017

Thanks for reporting. This seems to be a UI update issue. Will fix it in the next overhaul / PR update.

@luke-jr
Copy link
Member

@luke-jr luke-jr commented Jan 20, 2017

Thought: Can this be made to work with external wallets/software?

@jonasschnelli
Copy link
Member Author

@jonasschnelli jonasschnelli commented Jan 21, 2017

Thought: Can this be made to work with external wallets/software?

I don't know what you mean by this.
Can you make an use-case example?

@jtimon
Copy link
Member

@jtimon jtimon commented Jan 24, 2017

Needs rebase.
Concept ACK

What happens with -autorequestblocks=0 -spv=0?
I assume both -spv and -autorequestblocks are 1 by default. With -spv=0 -autorequestblocks=1 you would get what you have today, but is it really so useful compared to -spv=1 -autorequestblocks=1 ?

I don't know it seems overly complicated. I thought we would just have a single param spvonly that defaults to 0 (equivalent to this autorequestblocks, and your spv is always =1, ie spvonly=0 equivalent to -spv=1 -autorequestblocks=1, spvonly=1 equivalent to -spv=1 -autorequestblocks=0). Not sure, just thinking out loud.

@jonasschnelli jonasschnelli force-pushed the jonasschnelli:2017/01/spv branch Jan 24, 2017
jonasschnelli added 11 commits Dec 16, 2016
This is required for features that need to download and process block higher/further-away then the current validation depth
+ Adds a validate=true|false to the SyncTransaction signal
…PC call

This allows efficient testing of auxiliary block requests
…n the mempool
@jonasschnelli jonasschnelli force-pushed the jonasschnelli:2017/01/spv branch Jan 24, 2017
@jonasschnelli
Copy link
Member Author

@jonasschnelli jonasschnelli commented Jan 24, 2017

Rebased.

What happens with -autorequestblocks=0 -spv=0?

This would result in a mode where no blocks are automatically requested (only headers are fetched).
autorequestblocks=0 is a debug option and I could imagine some interesting use-cases where you only want to fetch certain blocks with requestblocks RPC call.

@laanwj
Copy link
Member

@laanwj laanwj commented Dec 21, 2017

How does it achieve this? BIP 37?

No BIP37 involved here, to retain privacy uses full-block SPV mode. The relevant blocks are only those from the birthdate of the wallet on.

@sipa
Copy link
Member

@sipa sipa commented Mar 6, 2018

Sorry for the very late comment here, but I think this is introducing a lot of complexity and then building on top of it.

I think a first step should be what @ryanofsky suggested above ("Get rid of the CAuxiliaryBlockRequest class and integrate prioritized block download logic directly into net_processing.cpp so more code responsible for regular and prioritized block downloads can be shared, and the wallet will not have to be involved in batching and sequencing p2p requests."). This is more generally useful than just lightweight mode too; it could be used for rescanning while pruning too, for example.

@ryanofsky
Copy link
Contributor

@ryanofsky ryanofsky commented Mar 6, 2018

I think a first step should be what @ryanofsky suggested above ("Get rid of the CAuxiliaryBlockRequest class and integrate prioritized block download logic directly into net_processing.cpp so more code responsible for regular and prioritized block downloads can be shared, and the wallet will not have to be involved in batching and sequencing p2p requests.").

This is actually implemented in #10794. @jonasschnelli, it would probably be good to reference #10794 in the PR description, and make sure the PR description is up to date generally.

@ryanofsky
Copy link
Contributor

@ryanofsky ryanofsky commented Mar 23, 2018

@TheBlueMatt pointed out at core dev that the sync implemented here sometimes can't recover from invalid blocks, and that basing this change on #12138 might fix this.

@githubcyc
Copy link

@githubcyc githubcyc commented Jun 5, 2018

hey, how to run this version locally? thx

@Sjors
Copy link
Member

@Sjors Sjors commented Jun 15, 2018

I'm also in favor of not using the word SPV in the PR description, which does not imply giving up on the term.

schermafbeelding 2018-06-15 om 12 14 15

Can you clarify this:

Pure full block SPV mode is possible by setting -autorequestblocks=0, in that mode, no blocks for validating the chain will be downloaded, resulting in a SPV only mode.

Did you mean no blocks older than the wallet? If so, how do you check if the UTXO set is correct? If you don't, then maybe the flag should have a scarier name like -partial_utxo_set. Maybe it's better to add that option in a followup PR to improve the worst case security we have to reason about here.

@ryanofsky wrote:

But beyond these two things, I don't understand what the wallet "SPV mode" is supposed to indicate. Why would it be helpful to me to know that my wallet is in SPV mode if I don't actually have any nonvalidated transactions? Why would it helpful be to me to know my wallet is not in SPV mode if I do have nonvalidated transactions?

My suggestion would be to show the usual IBD progress bar*, as well as mark transactions as not-fully-validated. In that case there's no need for an extra icon.

  • = you could go really fancy and draw a thin blue slice at the right most edge to indicate that we have a fragmented chain, using a lighter shade to indicate that it might be completely invalid.

void CAuxiliaryBlockRequest::processWithPossibleBlock(const std::shared_ptr<const CBlock> pblock, CBlockIndex *pindex)
{
// don't process anything if the request was cancled

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018
Contributor

Typo found by codespell: cancled

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018
Contributor

This should be fixed throughout in this PR :-)

This comment has been minimized.

@Sjors

Sjors Sep 3, 2018
Member

Maybe we need a linter?

This comment has been minimized.

@practicalswift

practicalswift Sep 3, 2018
Contributor

@Sjors Please review #13954 which adds a codespell linter :-)

this->processedUpToSize++;

// log some info
LogPrint("net", "BlockRequest: proccessed up to %ld of total requested %ld blocks\n", this->processedUpToSize, this->vBlocksToDownload.size());

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018
Contributor

Typo found by codespell: proccessed

@@ -481,6 +493,27 @@ void FindNextBlocksToDownload(NodeId nodeid, unsigned int count, std::vector<con
// Make sure pindexBestKnownBlock is up to date, we'll need it.
ProcessBlockAvailability(nodeid);

// if there is an open CAuxiliaryBlockRequest (out-of-band/specific block donwload), privileg it

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018
Contributor

Typo found by codespell: donwload annd privileg

return (mapBlocksInFlight.count(pIndexCheck->GetBlockHash()) == 0);
});

// if we haven't completed the individual CAuxiliaryBlockRequest, we wont continue with "normal" IBD

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018
Contributor

Typo found by codespell: wont

return;
}

// don't request any other blocks if we are in non autorequest mode (usefull for non-validation mode)

This comment has been minimized.

@practicalswift

practicalswift Sep 2, 2018
Contributor

Typo found by codespell: usefull (also below)

@fanquake fanquake mentioned this pull request Oct 26, 2018
@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Dec 3, 2018

There hasn't been much activity lately and the patch still needs rebase, so I am closing this for now. Please let me know when you want to continue working on this, so the pull request can be re-opened.
@BlockMechanic
Copy link
Contributor

@BlockMechanic BlockMechanic commented Sep 9, 2019

Hi

I've always liked the idea of the reference client ie core being available across all platforms. With my work on #16568 proving succeful for android , I'd like to request that an SPV mode be considered seriously for the next major release (0.20). I think ensuring that the core client is accesible across as many platforms (particularly mobile) is a project worth considering.

@BlockMechanic
Copy link
Contributor

@BlockMechanic BlockMechanic commented Oct 5, 2019

As we already sync headers , is there a reason to have a separate headers chain ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Client-Mode (SPV)
Conceptual PR
Linked issues

Successfully merging this pull request may close these issues.

None yet