/bitcoin

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local wallet #9152

Open
wants to merge 6 commits into
from
+322 −60

Conversation

Reviewers

@promag promag

@ryanofsky ryanofsky

Assignees
No one assigned
Labels
RPC/REST/ZMQ Wallet
Projects
None yet
Milestone
No milestone
9 participants
@luke-jr @jonasschnelli @gmaxwell @paveljanik @laanwj @sipa @promag @ryanofsky @fanquake
@luke-jr
Member

luke-jr commented Nov 13, 2016 •
Edited 1 time

Does this look like a good approach?

TODO:

  • rawtransaction sweep functionality
  • GUI sweep (Receive tab?)
  • abstract shared sweep logic
  • RPC tests

@fanquake fanquake added RPC/REST/ZMQ Wallet labels Nov 13, 2016

@jonasschnelli
Member

jonasschnelli commented Nov 14, 2016

Concept ACK (Haven't really looked at the code).
I think a sweep function would be a great feature. One could import "old" private keys into a new HD wallet for example.

Possible extension: sweepseed could be an extended version of that, moving all funds form a HD seed to a new one, generating large lookup-windows on different chainpathes. It could also be UTXO set only not requiring a -rescan.
@gmaxwell
Member

gmaxwell commented Nov 15, 2016

From a raw flow perspective, the generation of the sweep transaction is something that works from public information and should be possible on an online node without access to the private keys... so that one should be a 'createrawsweeptransaction' which takes a list of adresses/pubkeys/redeemscripts (and maybe private keys ... maybe some kind of BIP32 chain spec) and returns a transaction that spends all coins assigned to matching keys, potentially with arguments to limit the set of inputs collected.
@paveljanik
Contributor

paveljanik commented Nov 17, 2016

Hmm, what about extending RPC importprivkey with another optional argument sweep defaulting to false?
@luke-jr
Member

luke-jr commented Nov 17, 2016

sweepprivkeys is intended for users, and not to import keys. Users should never use importprivkey.

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Dec 9, 2016

@ryanofsky
Make CCoinsViewCache::Cursor() return latest data 
Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)
6494980

@ryanofsky ryanofsky referenced this pull request Dec 9, 2016

Closed

Make CCoinsViewCache::Cursor() return latest data #9306

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Dec 15, 2016

@ryanofsky
Make CCoinsViewCache::Cursor() return latest data 
Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)
59a77a6

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Dec 19, 2016

@ryanofsky
coins-cursor 
Make CCoinsViewCache::Cursor() return latest data

Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)
ef97df2

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016

@luke-jr
CCoinsView::FindScriptPubKey to search the UTXO set 
Github-Pull: #9152
Rebased-From: 220750d
155e1cf

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016

@luke-jr
Wallet/RPC: Abstract common WIF privkey parsing into ParseWIFPrivKey … 
…function

Github-Pull: #9152
Rebased-From: 4437a5f
f73bc4f

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016

@luke-jr
Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local w… 
…allet

Github-Pull: #9152
Rebased-From: fdc6778
29caf46
@ryanofsky

ryanofsky reviewed Dec 28, 2016
View changes

Re: "Refactor sweepprivkeys to deal with CCoinsView::Cursor limitations," I think I could extend #9306 to return a working cursor for CCoinsViewMemPool, if that would help.

src/wallet/rpcwallet.cpp
+ LOCK(cs_main);
+ mempool.FindScriptPubKey(setscriptSearch, mapcoins);
+ FlushStateToDisk();
+ pcoinsTip->FindScriptPubKey(setscriptSearch, mapcoins);
@ryanofsky

ryanofsky Dec 28, 2016 •
Edited 1 time

Contributor

It might be appropriate to loop over mapCoins calling CTxMemPool::pruneSpent, on each entry, to avoid trying to sweep from an output that is already in the process of being spent.

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 31, 2016

@luke-jr
Document limitations of CCoinsView::Cursor implementations 
Github-Pull: #9152
Rebased-From: dd089ad
2d594e7

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 31, 2016

@luke-jr
Refactor sweepprivkeys to deal with CCoinsView::Cursor limitations 
Github-Pull: #9152
Rebased-From: ed60474
f3f5c79

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jan 2, 2017

@ryanofsky
coins-cursor 
Make CCoinsViewCache::Cursor() return latest data

Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)
96a1834

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jan 2, 2017

@ryanofsky
coins-cursor 
Make CCoinsViewCache::Cursor() return latest data

Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)
2ca75bc
@laanwj
Owner

laanwj commented Feb 27, 2017

Concept ACK.

Hmm, what about extending RPC importprivkey with another optional argument sweep defaulting to false?

Please don't do this. People confuse import and sweep all over the place. The least we can do is make them separate RPCs with separate documentation.

@laanwj laanwj added this to the 0.15.0 milestone Feb 27, 2017

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jun 2, 2017

@ryanofsky
Make CCoinsViewCache::Cursor() return latest data 
Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)
d58b56d

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jun 12, 2017

@ryanofsky
Make CCoinsViewCache::Cursor() return latest data 
Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)
095c2e7
@sipa
Owner

sipa commented Jul 12, 2017

Seems this won't be making it for 0.15. Untagging.

@sipa sipa removed this from the 0.15.0 milestone Jul 12, 2017

@jonasschnelli
Member

jonasschnelli commented Aug 15, 2017

Concept Re-ACK. Needs rebase. I guess this is also something we could want for 0.16.
I have no strong opinion about sweep versus createrawsweep. The approach we took for bumpfee was also to do the non raw one first, although I agree that the ramification of sweep is different then a bump.
@luke-jr
Member

luke-jr commented Aug 19, 2017

Rebased and added a simple functional test.
@promag

promag reviewed Aug 19, 2017
View changes

Is it me or this could use CWallet::FundTransaction with coin control configured and with subtract fee from amount?

src/wallet/rpcwallet.cpp
+ CKeyID address = pubkey.GetID();
+ CScript script = GetScriptForDestination(address);
+ if (!script.empty()) {
+ needles.insert(script);
@promag

promag Aug 19, 2017

Contributor

Throw if duplicate? Add test.

@luke-jr

luke-jr Aug 21, 2017

Member

I don't see why a duplicate privkey should be an error.

src/wallet/rpcwallet.cpp
+ }
+ script = GetScriptForRawPubKey(pubkey);
+ if (!script.empty()) {
+ needles.insert(script);
@promag

promag Aug 19, 2017

Contributor

Throw if duplicate? Add test.

src/wallet/rpcwallet.cpp
+ }
+ }
+
+ // Ensure keypool is filled if possible
@promag

promag Aug 19, 2017

Contributor

Throw if needles is empty? Add test.

@luke-jr

luke-jr Aug 21, 2017

Member

If it's empty, we'll throw "No value to sweep" later on. And so long as the user provides something, it will never be empty.

src/wallet/rpcwallet.cpp
+ CReserveKey reservekey(pwallet);
+ CPubKey pubkey;
+ if (!reservekey.GetReservedKey(pubkey)) {
+ throw JSONRPCError(RPC_WALLET_KEYPOOL_RAN_OUT, "Error: Keypool ran out, please call keypoolrefill first");
@promag

promag Aug 19, 2017

Contributor

Add test for this error?

@luke-jr

luke-jr Aug 21, 2017

Member

I don't know how, and the two tests currently "testing" it don't make logical sense.

src/wallet/rpcwallet.cpp
+
+ // Scan UTXO set for inputs
+ std::vector<CTxOut> input_txos;
+ {
@promag

promag Aug 19, 2017

Contributor

Unnecessary block?

@luke-jr

luke-jr Aug 21, 2017

Member

Scope for coins

src/wallet/rpcwallet.cpp
+ if (IsDust(tx.vout[0], ::minRelayTxFee)) {
+ throw JSONRPCError(RPC_VERIFY_REJECTED, "Swept value would be dust");
+ }
+ for (size_t input_index = 0; input_index < tx.vin.size(); ++input_index) {
@promag

promag Aug 19, 2017

Contributor

Take this out of fee loop?

@luke-jr

luke-jr Aug 21, 2017

Member

Fee changes require resigning. Perhaps it could be made to sign only before and after the loop, but the time here is trivial in comparison to the UTXO search.

@luke-jr
Member

luke-jr commented Aug 21, 2017

Fixed the test failure (missing cs_main lock on AcceptToMemoryPool)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment