New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local wallet #9152

Closed
wants to merge 6 commits into
base: master
from

Conversation

Projects
None yet
@luke-jr
Member

luke-jr commented Nov 13, 2016

Does this look like a good approach?

TODO:

  • rawtransaction sweep functionality
  • GUI sweep (Receive tab?)
  • abstract shared sweep logic
  • RPC tests
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Nov 14, 2016

Member

Concept ACK (Haven't really looked at the code).
I think a sweep function would be a great feature. One could import "old" private keys into a new HD wallet for example.

Possible extension: sweepseed could be an extended version of that, moving all funds form a HD seed to a new one, generating large lookup-windows on different chainpathes. It could also be UTXO set only not requiring a -rescan.

Member

jonasschnelli commented Nov 14, 2016

Concept ACK (Haven't really looked at the code).
I think a sweep function would be a great feature. One could import "old" private keys into a new HD wallet for example.

Possible extension: sweepseed could be an extended version of that, moving all funds form a HD seed to a new one, generating large lookup-windows on different chainpathes. It could also be UTXO set only not requiring a -rescan.

@gmaxwell

This comment has been minimized.

Show comment
Hide comment
@gmaxwell

gmaxwell Nov 15, 2016

Member

From a raw flow perspective, the generation of the sweep transaction is something that works from public information and should be possible on an online node without access to the private keys... so that one should be a 'createrawsweeptransaction' which takes a list of adresses/pubkeys/redeemscripts (and maybe private keys ... maybe some kind of BIP32 chain spec) and returns a transaction that spends all coins assigned to matching keys, potentially with arguments to limit the set of inputs collected.

Member

gmaxwell commented Nov 15, 2016

From a raw flow perspective, the generation of the sweep transaction is something that works from public information and should be possible on an online node without access to the private keys... so that one should be a 'createrawsweeptransaction' which takes a list of adresses/pubkeys/redeemscripts (and maybe private keys ... maybe some kind of BIP32 chain spec) and returns a transaction that spends all coins assigned to matching keys, potentially with arguments to limit the set of inputs collected.

@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik Nov 17, 2016

Contributor

Hmm, what about extending RPC importprivkey with another optional argument sweep defaulting to false?

Contributor

paveljanik commented Nov 17, 2016

Hmm, what about extending RPC importprivkey with another optional argument sweep defaulting to false?

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Nov 17, 2016

Member

sweepprivkeys is intended for users, and not to import keys. Users should never use importprivkey.

Member

luke-jr commented Nov 17, 2016

sweepprivkeys is intended for users, and not to import keys. Users should never use importprivkey.

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Dec 9, 2016

Make CCoinsViewCache::Cursor() return latest data
Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Dec 15, 2016

Make CCoinsViewCache::Cursor() return latest data
Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Dec 19, 2016

coins-cursor
Make CCoinsViewCache::Cursor() return latest data

Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016

@ryanofsky

Re: "Refactor sweepprivkeys to deal with CCoinsView::Cursor limitations," I think I could extend #9306 to return a working cursor for CCoinsViewMemPool, if that would help.

Show outdated Hide outdated src/wallet/rpcwallet.cpp

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 31, 2016

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 31, 2016

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jan 2, 2017

coins-cursor
Make CCoinsViewCache::Cursor() return latest data

Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jan 2, 2017

coins-cursor
Make CCoinsViewCache::Cursor() return latest data

Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)
@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Feb 27, 2017

Member

Concept ACK.

Hmm, what about extending RPC importprivkey with another optional argument sweep defaulting to false?

Please don't do this. People confuse import and sweep all over the place. The least we can do is make them separate RPCs with separate documentation.

Member

laanwj commented Feb 27, 2017

Concept ACK.

Hmm, what about extending RPC importprivkey with another optional argument sweep defaulting to false?

Please don't do this. People confuse import and sweep all over the place. The least we can do is make them separate RPCs with separate documentation.

@laanwj laanwj added this to the 0.15.0 milestone Feb 27, 2017

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jun 2, 2017

Make CCoinsViewCache::Cursor() return latest data
Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jun 12, 2017

Make CCoinsViewCache::Cursor() return latest data
Change CCoinsViewCache::Cursor() method to return a cursor yielding the latest
CCoins entries, instead of just previous entries prior to the last cache flush.

The CCoinsViewCache::Cursor method is not currently used. This change just
enables new features that rely on scanning the UXTO set to work correctly (for
example bitcoin#9152, which adds a
sweepprivkeys RPC, and bitcoin#9137, which
improves handling of imported keys for nodes with pruning enabled.)
@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Jul 12, 2017

Member

Seems this won't be making it for 0.15. Untagging.

Member

sipa commented Jul 12, 2017

Seems this won't be making it for 0.15. Untagging.

@sipa sipa removed this from the 0.15.0 milestone Jul 12, 2017

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Aug 15, 2017

Member

Concept Re-ACK. Needs rebase. I guess this is also something we could want for 0.16.
I have no strong opinion about sweep versus createrawsweep. The approach we took for bumpfee was also to do the non raw one first, although I agree that the ramification of sweep is different then a bump.

Member

jonasschnelli commented Aug 15, 2017

Concept Re-ACK. Needs rebase. I guess this is also something we could want for 0.16.
I have no strong opinion about sweep versus createrawsweep. The approach we took for bumpfee was also to do the non raw one first, although I agree that the ramification of sweep is different then a bump.

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Aug 19, 2017

Member

Rebased and added a simple functional test.

Member

luke-jr commented Aug 19, 2017

Rebased and added a simple functional test.

@promag

Is it me or this could use CWallet::FundTransaction with coin control configured and with subtract fee from amount?

CKeyID address = pubkey.GetID();
CScript script = GetScriptForDestination(address);
if (!script.empty()) {
needles.insert(script);

This comment has been minimized.

@promag

promag Aug 19, 2017

Member

Throw if duplicate? Add test.

@promag

promag Aug 19, 2017

Member

Throw if duplicate? Add test.

This comment has been minimized.

@luke-jr

luke-jr Aug 21, 2017

Member

I don't see why a duplicate privkey should be an error.

@luke-jr

luke-jr Aug 21, 2017

Member

I don't see why a duplicate privkey should be an error.

}
script = GetScriptForRawPubKey(pubkey);
if (!script.empty()) {
needles.insert(script);

This comment has been minimized.

@promag

promag Aug 19, 2017

Member

Throw if duplicate? Add test.

@promag

promag Aug 19, 2017

Member

Throw if duplicate? Add test.

}
}
// Ensure keypool is filled if possible

This comment has been minimized.

@promag

promag Aug 19, 2017

Member

Throw if needles is empty? Add test.

@promag

promag Aug 19, 2017

Member

Throw if needles is empty? Add test.

This comment has been minimized.

@luke-jr

luke-jr Aug 21, 2017

Member

If it's empty, we'll throw "No value to sweep" later on. And so long as the user provides something, it will never be empty.

@luke-jr

luke-jr Aug 21, 2017

Member

If it's empty, we'll throw "No value to sweep" later on. And so long as the user provides something, it will never be empty.

CReserveKey reservekey(pwallet);
CPubKey pubkey;
if (!reservekey.GetReservedKey(pubkey)) {
throw JSONRPCError(RPC_WALLET_KEYPOOL_RAN_OUT, "Error: Keypool ran out, please call keypoolrefill first");

This comment has been minimized.

@promag

promag Aug 19, 2017

Member

Add test for this error?

@promag

promag Aug 19, 2017

Member

Add test for this error?

This comment has been minimized.

@luke-jr

luke-jr Aug 21, 2017

Member

I don't know how, and the two tests currently "testing" it don't make logical sense.

@luke-jr

luke-jr Aug 21, 2017

Member

I don't know how, and the two tests currently "testing" it don't make logical sense.

// Scan UTXO set for inputs
std::vector<CTxOut> input_txos;
{

This comment has been minimized.

@promag

promag Aug 19, 2017

Member

Unnecessary block?

@promag

promag Aug 19, 2017

Member

Unnecessary block?

This comment has been minimized.

@luke-jr

luke-jr Aug 21, 2017

Member

Scope for coins

@luke-jr

luke-jr Aug 21, 2017

Member

Scope for coins

if (IsDust(tx.vout[0], ::minRelayTxFee)) {
throw JSONRPCError(RPC_VERIFY_REJECTED, "Swept value would be dust");
}
for (size_t input_index = 0; input_index < tx.vin.size(); ++input_index) {

This comment has been minimized.

@promag

promag Aug 19, 2017

Member

Take this out of fee loop?

@promag

promag Aug 19, 2017

Member

Take this out of fee loop?

This comment has been minimized.

@luke-jr

luke-jr Aug 21, 2017

Member

Fee changes require resigning. Perhaps it could be made to sign only before and after the loop, but the time here is trivial in comparison to the UTXO search.

@luke-jr

luke-jr Aug 21, 2017

Member

Fee changes require resigning. Perhaps it could be made to sign only before and after the loop, but the time here is trivial in comparison to the UTXO search.

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Aug 21, 2017

Member

Fixed the test failure (missing cs_main lock on AcceptToMemoryPool)

Member

luke-jr commented Aug 21, 2017

Fixed the test failure (missing cs_main lock on AcceptToMemoryPool)

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Nov 30, 2017

Member

@luke-jr: can you rebase this once again? Thanks

Member

jonasschnelli commented Nov 30, 2017

@luke-jr: can you rebase this once again? Thanks

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Nov 30, 2017

Member

Rebased

Member

luke-jr commented Nov 30, 2017

Rebased

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Dec 1, 2017

Member

utACK a397deb (will test soon).
I personally dislike the "direct"-approach. I would like to "see" the transaction before submitting.

Possible extensions:

  • Remove the signing and broadcasting, accept a pubkey and rename it to createrawsweeptransaction (or similar)
  • Add support for HD sweep (xpub/xpriv)
Member

jonasschnelli commented Dec 1, 2017

utACK a397deb (will test soon).
I personally dislike the "direct"-approach. I would like to "see" the transaction before submitting.

Possible extensions:

  • Remove the signing and broadcasting, accept a pubkey and rename it to createrawsweeptransaction (or similar)
  • Add support for HD sweep (xpub/xpriv)
@jaromil

This comment has been minimized.

Show comment
Hide comment
@jaromil

jaromil Dec 6, 2017

Contributor

ACK using a397deb. This is a great feature many people need.
I'm moving on to test it on mainnet using the v0.15.0.knots20170914 build.
I encourage inclusion and later on apply extensions (esp what @jonasschnelli recommends above).

Contributor

jaromil commented Dec 6, 2017

ACK using a397deb. This is a great feature many people need.
I'm moving on to test it on mainnet using the v0.15.0.knots20170914 build.
I encourage inclusion and later on apply extensions (esp what @jonasschnelli recommends above).

@jaromil

This comment has been minimized.

Show comment
Hide comment
@jaromil

jaromil Dec 11, 2017

Contributor

The check fail is unrelated to this PR and limited to the ARM build, lacking the "flake8" python library.

contrib/devtools/lint-python.sh: 10: contrib/devtools/lint-python.sh: flake8: not found

removing unused imports won't fix lint-python.sh either.

Contributor

jaromil commented Dec 11, 2017

The check fail is unrelated to this PR and limited to the ARM build, lacking the "flake8" python library.

contrib/devtools/lint-python.sh: 10: contrib/devtools/lint-python.sh: flake8: not found

removing unused imports won't fix lint-python.sh either.

@jaromil

This comment has been minimized.

Show comment
Hide comment
@jaromil

jaromil Jan 8, 2018

Contributor

ping and HNY everyone :^P

Contributor

jaromil commented Jan 8, 2018

ping and HNY everyone :^P

@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Jan 15, 2018

Member

Since the keys are not added to the wallet, and thus bumpfee won't work, it makes sense to set nSequence to support rbf and return a list of signed raw transactions that each pay a slightly higher fee?

Member

MarcoFalke commented Jan 15, 2018

Since the keys are not added to the wallet, and thus bumpfee won't work, it makes sense to set nSequence to support rbf and return a list of signed raw transactions that each pay a slightly higher fee?

@jaromil

This comment has been minimized.

Show comment
Hide comment
@jaromil

jaromil Jan 15, 2018

Contributor

@MarcoFalke I believe yours is a brilliant idea. Just to be clear, you mean a list of signed raw transactions that are ready to replace the one made and can be fired up from the lowest fee to the higher to bump if necessary, right? such transactions should be discarded once enough confirmations are present.

Contributor

jaromil commented Jan 15, 2018

@MarcoFalke I believe yours is a brilliant idea. Just to be clear, you mean a list of signed raw transactions that are ready to replace the one made and can be fired up from the lowest fee to the higher to bump if necessary, right? such transactions should be discarded once enough confirmations are present.

laanwj added a commit that referenced this pull request Jul 17, 2018

Merge #12196: Add scantxoutset RPC method
be98b2d [QA] Add scantxoutset test (Jonas Schnelli)
eec7cf7 scantxoutset: mention that scanning by address will miss P2PK txouts (Jonas Schnelli)
94d73d3 scantxoutset: support legacy P2PK script type (Jonas Schnelli)
892de1d scantxoutset: add support for scripts (Jonas Schnelli)
7830494 Blockchain/RPC: Add scantxoutset method to scan UTXO set (Jonas Schnelli)
9048575 Add FindScriptPubKey() to search the UTXO set (Jonas Schnelli)

Pull request description:

  Alternative to #9152.

  This takes `<n>` pubkeys and optionally  `<n>` xpubs (together with a definable lookup windows where the default is 0-1000) and looks up common scripts in the UTXO set of all given or derived keys.

  The output will be an array similar to `listunspent`. That array is compatible with `createrawtransaction` as well as with `signrawtransaction`.

  This makes it possible to prepare sweeps and have them signed in a secure (cold) space.

Tree-SHA512: a2b22a117cf6e27febeb97e5d6fe30184926d50c0c7cbc77bb4121f490fed65560c52f8eac67a9720d7bf8f420efa42459768685c7e7cc03722859f51a5e1e3b
@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Aug 31, 2018

Member

If we still want this, it needs rebase/implementation in terms of #12196.
But I'm going to close it for now, it's been open since 2016 and there's no clear progress.

It might make sense to open a new PR if you start work on this again, or you can ask me to reopen it.

Member

laanwj commented Aug 31, 2018

If we still want this, it needs rebase/implementation in terms of #12196.
But I'm going to close it for now, it's been open since 2016 and there's no clear progress.

It might make sense to open a new PR if you start work on this again, or you can ask me to reopen it.

@laanwj laanwj closed this Aug 31, 2018

@jaromil

This comment has been minimized.

Show comment
Hide comment
@jaromil

jaromil Aug 31, 2018

Contributor

Interesting revamp. I have read the whole thread in #12196, is there some documentation being worked about use examples for scantxoutset to be used in place of sweeping? I like to try to sort out my needs using that, but haven't kept myself up to date enough to understand it without handholding. There is the integration test in python, but...

Contributor

jaromil commented Aug 31, 2018

Interesting revamp. I have read the whole thread in #12196, is there some documentation being worked about use examples for scantxoutset to be used in place of sweeping? I like to try to sort out my needs using that, but haven't kept myself up to date enough to understand it without handholding. There is the integration test in python, but...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment