Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local wallet #9152

luke-jr · 2016-11-13T22:07:48Z

Does this look like a good approach?

TODO:

rawtransaction sweep functionality
GUI sweep (Receive tab?)
abstract shared sweep logic
~~RPC tests~~

NOTE: Currently needs #14602 for tests to pass

jonasschnelli · 2016-11-14T07:41:08Z

Concept ACK (Haven't really looked at the code).
I think a sweep function would be a great feature. One could import "old" private keys into a new HD wallet for example.

Possible extension: sweepseed could be an extended version of that, moving all funds form a HD seed to a new one, generating large lookup-windows on different chainpathes. It could also be UTXO set only not requiring a -rescan.

gmaxwell · 2016-11-15T17:29:21Z

From a raw flow perspective, the generation of the sweep transaction is something that works from public information and should be possible on an online node without access to the private keys... so that one should be a 'createrawsweeptransaction' which takes a list of adresses/pubkeys/redeemscripts (and maybe private keys ... maybe some kind of BIP32 chain spec) and returns a transaction that spends all coins assigned to matching keys, potentially with arguments to limit the set of inputs collected.

paveljanik · 2016-11-17T11:20:07Z

Hmm, what about extending RPC importprivkey with another optional argument sweep defaulting to false?

luke-jr · 2016-11-17T11:32:07Z

sweepprivkeys is intended for users, and not to import keys. Users should never use importprivkey.

Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest CCoins entries, instead of just previous entries prior to the last cache flush. The CCoinsViewCache::Cursor method is not currently used. This change just enables new features that rely on scanning the UXTO set to work correctly (for example bitcoin#9152, which adds a sweepprivkeys RPC, and bitcoin#9137, which improves handling of imported keys for nodes with pruning enabled.)

Make CCoinsViewCache::Cursor() return latest data Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest CCoins entries, instead of just previous entries prior to the last cache flush. The CCoinsViewCache::Cursor method is not currently used. This change just enables new features that rely on scanning the UXTO set to work correctly (for example bitcoin#9152, which adds a sweepprivkeys RPC, and bitcoin#9137, which improves handling of imported keys for nodes with pruning enabled.)

Github-Pull: bitcoin#9152 Rebased-From: 220750dc4aa298d806effd4c6d3685721dd53a88

…function Github-Pull: bitcoin#9152 Rebased-From: 4437a5f875da68032f9539e35679219f316cc851

…allet Github-Pull: bitcoin#9152 Rebased-From: fdc677846eb381427c2b6382c5e1b88568311f9f

ryanofsky

Re: "Refactor sweepprivkeys to deal with CCoinsView::Cursor limitations," I think I could extend #9306 to return a working cursor for CCoinsViewMemPool, if that would help.

ryanofsky · 2016-12-28T20:31:23Z

src/wallet/rpcwallet.cpp

+            LOCK(cs_main);
+            mempool.FindScriptPubKey(setscriptSearch, mapcoins);
+            FlushStateToDisk();
+            pcoinsTip->FindScriptPubKey(setscriptSearch, mapcoins);


It might be appropriate to loop over mapCoins calling CTxMemPool::pruneSpent, on each entry, to avoid trying to sweep from an output that is already in the process of being spent.

Github-Pull: bitcoin#9152 Rebased-From: dd089adf87ea6078d1883ce1b4771760bfda1ddb

Github-Pull: bitcoin#9152 Rebased-From: ed604747289c12d605f219b3f859a9bef376174e

Make CCoinsViewCache::Cursor() return latest data Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest CCoins entries, instead of just previous entries prior to the last cache flush. The CCoinsViewCache::Cursor method is not currently used. This change just enables new features that rely on scanning the UXTO set to work correctly (for example bitcoin#9152, which adds a sweepprivkeys RPC, and bitcoin#9137, which improves handling of imported keys for nodes with pruning enabled.)

laanwj · 2017-02-27T09:31:41Z

Concept ACK.

Hmm, what about extending RPC importprivkey with another optional argument sweep defaulting to false?

Please don't do this. People confuse import and sweep all over the place. The least we can do is make them separate RPCs with separate documentation.

Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest CCoins entries, instead of just previous entries prior to the last cache flush. The CCoinsViewCache::Cursor method is not currently used. This change just enables new features that rely on scanning the UXTO set to work correctly (for example bitcoin#9152, which adds a sweepprivkeys RPC, and bitcoin#9137, which improves handling of imported keys for nodes with pruning enabled.)

sipa · 2017-07-12T22:50:19Z

Seems this won't be making it for 0.15. Untagging.

jonasschnelli · 2017-08-15T20:14:59Z

Concept Re-ACK. Needs rebase. I guess this is also something we could want for 0.16.
I have no strong opinion about sweep versus createrawsweep. The approach we took for bumpfee was also to do the non raw one first, although I agree that the ramification of sweep is different then a bump.

luke-jr · 2017-08-19T03:55:59Z

Rebased and added a simple functional test.

practicalswift · 2018-10-30T09:44:30Z

src/wallet/rpcwallet.cpp

+            const UniValue& privkeys_a = optval.get_array();
+            for (size_t privkey_i = 0; privkey_i < privkeys_a.size(); ++privkey_i) {
+                const UniValue& privkey_wif = privkeys_a[privkey_i];
+                std::string wif_secret = privkey_wif.get_str();


Could be const reference?

meshcollider · 2018-11-15T19:34:37Z

Concept ACK. Still needs rebase. IMO this should be reworked to take a descriptor which would address the points above about accepting pubkeys/HD keys. Agree that returning the raw transaction is preferable (createrawsweeptransaction)

…allet

…lter

…et is synced before we return

practicalswift · 2019-05-07T15:55:50Z

I'm afraid this PR doesn't build when rebased on master.

DrahtBot · 2019-06-06T15:24:35Z

Needs rebase

practicalswift · 2019-06-06T22:29:48Z

Concept ACK

jonasschnelli · 2019-06-07T12:27:06Z

This should probably use "scantxoutset" internally (rebase issue).
I also think it should be a createrawsweep call (rather then directly signing and executing).
Maybe you can change this to be an argument for "scantxoutset" that would also spit out a rawtransaction (with configurable fee and eventually configurable target address) for the sweep transaction.

Sjors

Concept ACK. Using scantxoutset internally seems like a good idea. It needs a fee control option, could be as simple as feeRate with 1 sat/byte default. Ideally also RBF control, but that can wait for a followup.

As mentioned by others, it's good to be able to inspect the transaction. That could just be another option. In #16378 I introduce a add_to_wallet to option, and the RPC call returns a hex & PSBT if false.

Another potential followup could add descriptor support; those can have private keys now too. Others above suggest using those instead of WIF, but I think an array of WIFs is still common enough to be worth supporting directly.

This PR sweeps into the wallet itself. A followup could sweep into any arbitrary descriptor, which would make migrating wallets to some new scheme easier, e.g. moving from single-sig to multi-sig without joining UTXOs.

Sjors · 2019-08-15T12:04:44Z

src/wallet/rpcwallet.cpp

+    if (request.fHelp || request.params.size() != 1) {
+        throw std::runtime_error(
+            RPCHelpMan{"sweepprivkeys",
+                "\nSends bitcoins controlled by private key to specified destinations.\n",


This says "specified destinations" but you can't specify the destinations. Can you also make it clear that coins are sweeped one by one to unique addresses (per priv key)?

luke-jr · 2019-10-14T20:31:53Z

I don't see any sane way to rebase this post-bitcoin-wallet, so going to have to just make it a Knots-only hack for now. :/

ryanofsky · 2019-10-15T16:47:20Z

I don't see any sane way to rebase this post-bitcoin-wallet

I don't think it should be too bad if someone wants to pick it up. It should just require a new interfaces::Chain method to wrap FindScriptPubKey calls, since wallet code can no longer access mempool and pcoinsdbview globals directly.

ryanofsky · 2019-10-24T16:04:09Z

Maintainers maybe remove "Waiting for author" and "Needs rebase" tags and mark "Up for grabs"

fanquake added RPC/REST/ZMQ Wallet labels Nov 13, 2016

ryanofsky mentioned this pull request Dec 9, 2016

Make CCoinsViewCache::Cursor() return latest data #9306

Closed

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016

CCoinsView::FindScriptPubKey to search the UTXO set

155e1cf

Github-Pull: bitcoin#9152 Rebased-From: 220750dc4aa298d806effd4c6d3685721dd53a88

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016

Wallet/RPC: Abstract common WIF privkey parsing into ParseWIFPrivKey …

f73bc4f

…function Github-Pull: bitcoin#9152 Rebased-From: 4437a5f875da68032f9539e35679219f316cc851

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016

Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local w…

29caf46

…allet Github-Pull: bitcoin#9152 Rebased-From: fdc677846eb381427c2b6382c5e1b88568311f9f

ryanofsky reviewed Dec 28, 2016

View reviewed changes

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 31, 2016

Document limitations of CCoinsView::Cursor implementations

2d594e7

Github-Pull: bitcoin#9152 Rebased-From: dd089adf87ea6078d1883ce1b4771760bfda1ddb

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 31, 2016

Refactor sweepprivkeys to deal with CCoinsView::Cursor limitations

f3f5c79

Github-Pull: bitcoin#9152 Rebased-From: ed604747289c12d605f219b3f859a9bef376174e

luke-jr force-pushed the sweepprivkeys branch from ed60474 to b2b7f44 Compare January 3, 2017 16:12

luke-jr force-pushed the sweepprivkeys branch 2 times, most recently from 25793ce to f6c9c12 Compare February 7, 2017 03:11

laanwj added this to the 0.15.0 milestone Feb 27, 2017

sipa removed this from the 0.15.0 milestone Jul 12, 2017

luke-jr force-pushed the sweepprivkeys branch from f6c9c12 to c0bd1d3 Compare August 19, 2017 03:55

practicalswift reviewed Oct 30, 2018

View reviewed changes

DrahtBot added the Needs rebase label Nov 5, 2018

luke-jr force-pushed the sweepprivkeys branch from e341211 to fe16fc5 Compare February 12, 2019 18:37

DrahtBot added Needs rebase and removed Needs rebase labels Feb 12, 2019

luke-jr force-pushed the sweepprivkeys branch from fe16fc5 to 0c90d54 Compare March 29, 2019 00:20

luke-jr added 6 commits April 12, 2019 22:25

Document limitations of CCoinsView::Cursor implementations

8b8ba3f

CTxMemPool::FindScriptPubKey to search unconfirmed UTXOs

0d14432

Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local w…

7c4e5e6

…allet

GUI/RPCConsole: Include sweepprivkeys in history sensitive-command fi…

799abbb

…lter

QA: Functional test for sweepprivkeys

f8ff311

RPC/Wallet: Use BroadcastTransaction for sweepprivkeys to ensure wall…

e30d4b5

…et is synced before we return

luke-jr force-pushed the sweepprivkeys branch from 0c90d54 to e30d4b5 Compare May 2, 2019 03:56

DrahtBot removed the Needs rebase label May 2, 2019

DrahtBot added the Needs rebase label Jun 6, 2019

meshcollider added the Waiting for author label Jun 7, 2019

Sjors reviewed Aug 15, 2019

View reviewed changes

luke-jr closed this Oct 14, 2019

laanwj added Up for grabs and removed Needs rebase Waiting for author labels Oct 24, 2019

bitcoin locked as resolved and limited conversation to collaborators Dec 16, 2021

Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local wallet #9152

Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local wallet #9152

luke-jr commented Nov 13, 2016 •

edited

jonasschnelli commented Nov 14, 2016

gmaxwell commented Nov 15, 2016

paveljanik commented Nov 17, 2016

luke-jr commented Nov 17, 2016

ryanofsky left a comment

ryanofsky Dec 28, 2016 •

edited

laanwj commented Feb 27, 2017

sipa commented Jul 12, 2017

jonasschnelli commented Aug 15, 2017

luke-jr commented Aug 19, 2017

practicalswift Oct 30, 2018

meshcollider commented Nov 15, 2018

practicalswift commented May 7, 2019

DrahtBot commented Jun 6, 2019

practicalswift commented Jun 6, 2019

jonasschnelli commented Jun 7, 2019

Sjors left a comment

Sjors Aug 15, 2019

luke-jr commented Oct 14, 2019

ryanofsky commented Oct 15, 2019

ryanofsky commented Oct 24, 2019

Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local wallet #9152

Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local wallet #9152

Conversation

luke-jr commented Nov 13, 2016 • edited

jonasschnelli commented Nov 14, 2016

gmaxwell commented Nov 15, 2016

paveljanik commented Nov 17, 2016

luke-jr commented Nov 17, 2016

ryanofsky left a comment

Choose a reason for hiding this comment

ryanofsky Dec 28, 2016 • edited

Choose a reason for hiding this comment

laanwj commented Feb 27, 2017

sipa commented Jul 12, 2017

jonasschnelli commented Aug 15, 2017

luke-jr commented Aug 19, 2017

practicalswift Oct 30, 2018

Choose a reason for hiding this comment

meshcollider commented Nov 15, 2018

practicalswift commented May 7, 2019

DrahtBot commented Jun 6, 2019

practicalswift commented Jun 6, 2019

jonasschnelli commented Jun 7, 2019

Sjors left a comment

Choose a reason for hiding this comment

Sjors Aug 15, 2019

Choose a reason for hiding this comment

luke-jr commented Oct 14, 2019

ryanofsky commented Oct 15, 2019

ryanofsky commented Oct 24, 2019

luke-jr commented Nov 13, 2016 •

edited

ryanofsky Dec 28, 2016 •

edited