Add scantxoutset RPC method #12196

jonasschnelli · 2018-01-16T07:09:05Z

Alternative to #9152.

This takes <n> pubkeys and optionally <n> xpubs (together with a definable lookup windows where the default is 0-1000) and looks up common scripts in the UTXO set of all given or derived keys.

The output will be an array similar to listunspent. That array is compatible with createrawtransaction as well as with signrawtransaction.

This makes it possible to prepare sweeps and have them signed in a secure (cold) space.

gmaxwell · 2018-01-16T07:14:50Z

Why is it pubkeys and not addresses for the pubkey part? (obviously xpubs are xpubs and need to be)

jonasschnelli · 2018-01-16T07:33:37Z

After a short discussion on IRC (https://botbot.me/freenode/bitcoin-core-dev/2018-01-16/?msg=95804115&page=2) support for addresses and pubkeys makes most sense. Will add support for an array of addresses.

promag · 2018-01-16T10:42:29Z

src/rpc/blockchain.cpp

+    else if (request.params[0].get_str() == "abort") {
+        CoinsViewScanReserver reserver;
+        if (reserver.reserve()) {
+            throw JSONRPCError(RPC_INVALID_PARAMETER, "No scan in progress");


IMO there is no need to throw, a bool in the response is enough? Otherwise, missing test.

promag · 2018-01-16T10:43:33Z

src/rpc/blockchain.cpp

+{
+    if (request.fHelp || request.params.size() < 1 || request.params.size() > 2)
+        throw std::runtime_error(
+            "scantxoutset <action> {\"pubkeys\": [\"pubkey\",...], \"xpubs\":[{\"xpub\": \"<xpub>\"}], other options}\n"


Remove other options? There are none.

promag · 2018-01-16T10:44:02Z

src/rpc/blockchain.cpp

+
+UniValue scantxoutset(const JSONRPCRequest& request)
+{
+    if (request.fHelp || request.params.size() < 1 || request.params.size() > 2)


request.params.size() != 2?

Nah. You can also call scantxoutset status

Commit "Blockchain/RPC: Add scantxoutset method to scan UTXO set"

Nah. You can also call scantxoutset status

Right.

However should be request.params.size() > 3 (instead of 2)?

promag · 2018-01-16T10:44:58Z

src/rpc/blockchain.cpp

+            "      \"pubkeys\":[\"pubkey\",...]   (array of strings, optional) An array of HEX encoded public keys\n"
+            "      \"xpubs\":                   (array of xpub objects that will be used to derive child keys with the given lookup window after m/0/k and m/1/k scheme)\n"
+            "           [\n"
+            "               {\"xpub\":\"<xpub>\", (base58check encoded extended public key (xpub)\n"


Nit, newline after {.

promag · 2018-01-16T10:45:50Z

src/rpc/blockchain.cpp

+    if (request.params[0].get_str() == "status") {
+        CoinsViewScanReserver reserver;
+        if (reserver.reserve()) {
+            throw JSONRPCError(RPC_INVALID_PARAMETER, "No scan in progress");


Missing test for error.

You mean missing a functional test for that case. Yes. I though about it, but would require a mockup-slowdown argument (-testslowdown or similar). Otherwise I guess it's hard to properly test this.

Why would this be an exception, considering that the scan could normally finish between the last call and this one? Generally I think we should avoid exceptions for control flow in our code (and especially in third party code that uses the rpc interface)

Yes, agree that status should not raise an error when no scan is in progress.

promag · 2018-01-16T10:47:17Z

src/rpc/blockchain.cpp

+static std::mutex g_utxosetscan;
+static std::atomic<int> g_scan_progress;
+static std::atomic<bool> g_scan_in_progress;
+static std::atomic<bool> g_should_abourt_scan;


Typo, abort.

promag · 2018-01-16T10:55:14Z

src/rpc/blockchain.cpp

+    }
+
+    ~CoinsViewScanReserver() {
+        std::lock_guard<std::mutex> lock(g_utxosetscan);


Lock only when changing g_scan_in_progress?

if (m_could_reserve) { std::lock_guard<std::mutex> lock(g_utxosetscan); g_scan_in_progress = false; }

promag · 2018-01-16T10:59:50Z

src/rpc/blockchain.cpp

+    else if (request.params[0].get_str() == "start") {
+        CoinsViewScanReserver reserver;
+        if (!reserver.reserve()) {
+            throw JSONRPCError(RPC_INVALID_PARAMETER, "Scan already in progress, use action \"abort\" \"status\"");


Remove "status"? Otherwise add or between actions.

promag · 2018-01-16T11:00:53Z

src/rpc/blockchain.cpp

+                result.push_back(Pair("searched_items", count));
+            }
+
+            for (auto& it : coins) {


const auto&

promag · 2018-01-16T11:01:18Z

src/rpc/blockchain.cpp

+    explicit CoinsViewScanReserver() : m_could_reserve(false) {}
+
+    bool reserve() {
+        std::lock_guard<std::mutex> lock(g_utxosetscan);


Assert not reserved?

assert(!m_could_reserve); std::lock_guard<std::mutex> lock(g_utxosetscan); if (g_scan_in_progress) return false; g_scan_in_progress = true; m_could_reserve = true; return true;

laanwj · 2018-01-16T12:41:12Z

Concept ACK, nice! I've wished for UTXO scanning functionality many times, much faster than importing into a watchonly wallet if you only care about spendable UTXOs.

jonasschnelli · 2018-01-20T00:16:20Z

Added support for addresses (can scan unspent outputs after given addresses)
Added support for an optional raw sweep transaction including optional feerate or optional confirmation target

jonasschnelli · 2018-01-20T00:26:49Z

~~The raw sweep fee calculation is currently WIP (misses the dummy signer part)... will fix soon.~~

jonasschnelli · 2018-01-21T06:36:12Z

Overhauled the fee calculation logic (see the dummy sign keystore).

gmaxwell · 2018-01-22T02:07:31Z

src/rpc/blockchain.cpp

+
+        // flush utxo state and start the scan
+        FlushStateToDisk();
+        bool res = pcoinsdbview->FindScriptPubKey(g_scan_progress, g_should_abort_scan, count, needles, coins);


What prevents the state from being mutated out from under this scan?

I think due to the time required to perform a scan, it's something that may be tolerated (although it should be mentioned in the docs). Not sure, but I guess it's the same with gettxoutsetinfo.

Not sure if you can scan a CCoinsView of a snapshot state... I guess no.
Locking cs_main would be "meh".

Also, the rawtx (sweep) is in the same way "outdated" the moment you have received it... maybe you could argue that this is the same for fundrawtx

@gmaxwell @jonasschnelli The cursor iterates over the state of the CCoinsView at the time it was created; modifying it during iteration is fine. This only works because GetCursor is not implemented for CCoinsViewCache, and is invoked directly on the CCoinsViewDB LevelDB wrapper.

The downside is that this requires a full flush of the database, hurting performance for all of the process (including validation).

luke-jr · 2018-01-24T19:23:42Z

NACK supporting addresses. Addresses have no relation to the UTXOs.

jonasschnelli · 2018-01-24T19:27:44Z

@luke-jr: Why? Addresses are encoded output scripts (scriptPubKey). The rational behind supporting addresses is that a) it may be more efficient then forming every possible known common script from a pubkey and b) that pubkeys are somewhat more difficult to export then the pure "used addresses".

luke-jr · 2018-01-24T19:41:53Z

Addresses are opaque identifiers for a given invoice. That they are currently implemented by encoding a scriptPubKey is irrelevant.

greenaddress · 2018-02-26T18:44:50Z

Does it support mempool/unconfirmed utxos? I had a quick look and didn't seem to, I think it would be useful to have mempool too.

Would it make sense to avoid the background job? maybe by keeping the utxo set sorted by scriptPubKey and binary search on it or perhaps some utxo set limited indexing?

jonasschnelli · 2018-02-27T01:47:15Z

Does it support mempool/unconfirmed utxos? I had a quick look and didn't seem to, I think it would be useful to have mempool too.

It currently does not scan the mempool (hence the command name scantxoutset), but I agree, that would be useful. But, since scans take a while, timing may be a problem for scanning the mempool.
Maybe an additional RPC call would make sense (scanmempool)?

Would it make sense to avoid the background job? maybe by keeping the utxo set sorted by scriptPubKey and binary search on it or perhaps some utxo set limited indexing?

You mean speeding up the scan? I don't think its worth to keep an extra index (changing the sort order would probably slow down verification a lot).
My local tests did show a mainnet scan takes about ~30seconds (SSD, fast CPU).
I don't know what the space requirements for a by-scriptPubKey-index would be, .... maybe we can look into this once this PR did proceed.

jonasschnelli · 2018-02-27T01:52:40Z

Rebased

luke-jr · 2018-02-28T02:15:27Z

The commit separation here is ugly: CCoinsView::FindScriptPubKey initially checks ShutdownRequested directly, and then this is removed with the RPC changes.

jonasschnelli · 2018-02-28T14:22:14Z

Fixed the ugly commit separation in coins.cpp

jonasschnelli · 2018-07-15T20:20:12Z

Fixed relevant points from @Empact

be98b2d [QA] Add scantxoutset test (Jonas Schnelli) eec7cf7 scantxoutset: mention that scanning by address will miss P2PK txouts (Jonas Schnelli) 94d73d3 scantxoutset: support legacy P2PK script type (Jonas Schnelli) 892de1d scantxoutset: add support for scripts (Jonas Schnelli) 7830494 Blockchain/RPC: Add scantxoutset method to scan UTXO set (Jonas Schnelli) 9048575 Add FindScriptPubKey() to search the UTXO set (Jonas Schnelli) Pull request description: Alternative to #9152. This takes `<n>` pubkeys and optionally `<n>` xpubs (together with a definable lookup windows where the default is 0-1000) and looks up common scripts in the UTXO set of all given or derived keys. The output will be an array similar to `listunspent`. That array is compatible with `createrawtransaction` as well as with `signrawtransaction`. This makes it possible to prepare sweeps and have them signed in a secure (cold) space. Tree-SHA512: a2b22a117cf6e27febeb97e5d6fe30184926d50c0c7cbc77bb4121f490fed65560c52f8eac67a9720d7bf8f420efa42459768685c7e7cc03722859f51a5e1e3b

promag · 2018-07-17T14:39:53Z

utACK be98b2d.

f6b7fc3 Support h instead of ' in hardened descriptor paths (Pieter Wuille) fddea67 Add experimental warning to scantxoutset (Jonas Schnelli) 6495849 [QA] Extend tests to more combinations (Pieter Wuille) 1af237f [QA] Add xpub range tests in scantxoutset tests (Jonas Schnelli) 151600b Swap in descriptors support into scantxoutset (Pieter Wuille) 0652c32 Descriptor tests (Pieter Wuille) fe8a7dc Output descriptors module (Pieter Wuille) e54d760 Add simple FlatSigningProvider (Pieter Wuille) 29943a9 Add more methods to Span class (Pieter Wuille) Pull request description: As promised, here is an implementation of my output descriptor concept (https://gist.github.com/sipa/e3d23d498c430bb601c5bca83523fa82) and integration within the `scantxoutset` RPC that was just added through #12196. It changes the RPC to use descriptors for everything; I hope the interface is simple enough to encompass all use cases. It includes support for P2PK, P2PKH, P2WPKH, P2SH, P2WSH, multisig, xpubs, xprvs, and chains of keys - combined in every possible way. Tree-SHA512: 63b54a96e7a72f5b04a8d645b8517d43ecd6a65a41f9f4e593931ce725a8845ab0baa1e9db6a7243190d8ac841f6e7e2f520d98c539312d78f7fd687d2c7b88f

surindergiri · 2020-10-01T11:44:24Z

Thanks for your great efforts implementing this feature, On Oct 01, 2020, When I run this RPC command on Bitcoin Node version 0.20.0, then its response is too slow, I am getting rpc response after 1-2 minutes.

Is there any way to speed up this "scantxoutset" RPC command or any alternative available?

I asked the same question over bitcoin stackexchange forum, but nobody responded. I know its not the right place to ask such questions, but I am stuck, our mobile wallet is live and we are unable to fetch user "imported" address balances on the fly.

Thanks.

https://bitcoin.stackexchange.com/questions/99278/bitcoin-scantxoutset-rpc-command-return-response-too-slow

promag · 2020-10-01T14:53:26Z

I asked the same question over bitcoin stackexchange forum, but nobody responded

@surindergiri that was like 3 hours before you posted here..

2f0b25a rpc: remove scantxoutset EXPERIMENTAL warning (Jon Atack) Pull request description: Remove old warning per IRC wallet meeting discussion at http://www.erisian.com.au/bitcoin-core-dev/log-2021-03-12.html#l-467 This RPC was merged 3 years ago in bitcoin#12196. ACKs for top commit: MarcoFalke: cr ACK 2f0b25a Tree-SHA512: 874ccd5bd19ecbbe91912171ac85af7a4658dc92f6db484ff3d03f07f1b9ba97e1c69d33a5c3ae5c5ec46cac3595a211f55fec0fbf81bac30d66a891c376ce26

merge bitcoin#12196, bitcoin#13697: Add scantxoutset RPC method

jonasschnelli added the RPC/REST/ZMQ label Jan 16, 2018

jonasschnelli force-pushed the 2017/12/utxo_sweep branch 2 times, most recently from 6993c02 to 4378347 Compare January 16, 2018 08:06

ajtowns mentioned this pull request Jan 16, 2018

[tests] Rename functional tests #11774

Merged

promag reviewed Jan 16, 2018

View reviewed changes

ryanofsky mentioned this pull request Jan 16, 2018

Make CCoinsViewCache::Cursor() return latest data #9306

Closed

jonasschnelli force-pushed the 2017/12/utxo_sweep branch 4 times, most recently from e54c0e6 to 9bf9baa Compare January 20, 2018 00:15

jonasschnelli force-pushed the 2017/12/utxo_sweep branch from 9bf9baa to 1490e13 Compare January 21, 2018 06:35

gmaxwell reviewed Jan 22, 2018

View reviewed changes

jonasschnelli force-pushed the 2017/12/utxo_sweep branch from 1490e13 to 2c006f5 Compare February 27, 2018 01:52

jonasschnelli force-pushed the 2017/12/utxo_sweep branch 2 times, most recently from 8cc1a82 to 729c484 Compare February 28, 2018 14:21

jonasschnelli force-pushed the 2017/12/utxo_sweep branch from 729c484 to 3835de0 Compare March 6, 2018 03:48

jonasschnelli force-pushed the 2017/12/utxo_sweep branch from b9b59cc to be98b2d Compare July 15, 2018 20:19

laanwj added this to the 0.17.0 milestone Jul 17, 2018

laanwj merged commit be98b2d into bitcoin:master Jul 17, 2018

jonasschnelli removed this from Blockers in High-priority for review Jul 17, 2018

sipa mentioned this pull request Jul 17, 2018

Support output descriptors in scantxoutset #13697

Merged

Bushstar mentioned this pull request Jul 19, 2018

commits from bitcoin/master FeatherCoin/Feathercoin#356

Merged

laanwj mentioned this pull request Aug 31, 2018

Wallet/RPC: sweepprivkeys method to scan UTXO set and send to local wallet #9152

Closed

HashUnlimited mentioned this pull request Sep 12, 2018

Support output descriptors in scantxoutset chaincoin/chaincoin#602

Merged

sipa mentioned this pull request Nov 16, 2018

Tracker for places where descriptors could/should be integrated #14739

Closed

FornaxA mentioned this pull request Apr 10, 2019

[backport][feature][tokens] bitcoin based upgrades - preparing for tokens ioncoincore/ion-v4#142

Merged

jonatack mentioned this pull request Mar 13, 2021

rpc: remove scantxoutset EXPERIMENTAL warning #21426

Merged

kwvg added a commit to kwvg/dash that referenced this pull request Jun 4, 2021

Merge bitcoin#12196: Add scantxoutset RPC method

3a4ff82

kwvg mentioned this pull request Jun 4, 2021

merge #12196, #13697: Add scantxoutset RPC method dashpay/dash#4187

Merged

kwvg added a commit to kwvg/dash that referenced this pull request Jun 9, 2021

Merge bitcoin#12196: Add scantxoutset RPC method

e3ff2aa

kwvg added a commit to kwvg/dash that referenced this pull request Jul 9, 2021

Merge bitcoin#12196: Add scantxoutset RPC method

0492d50

kwvg added a commit to kwvg/dash that referenced this pull request Jul 13, 2021

Merge bitcoin#12196: Add scantxoutset RPC method

63ea35b

kwvg added a commit to kwvg/dash that referenced this pull request Jul 13, 2021

Merge bitcoin#12196: Add scantxoutset RPC method

3da1c7a

kwvg added a commit to kwvg/dash that referenced this pull request Jul 14, 2021

Merge bitcoin#12196: Add scantxoutset RPC method

931a88e

kwvg added a commit to kwvg/dash that referenced this pull request Jul 20, 2021

Merge bitcoin#12196: Add scantxoutset RPC method

e493c8a

PastaPastaPasta added a commit to dashpay/dash that referenced this pull request Jul 26, 2021

Merge pull request #4187 from kittywhiskers/scantxout

edf0552

merge bitcoin#12196, bitcoin#13697: Add scantxoutset RPC method

gades pushed a commit to cosanta/cosanta-core that referenced this pull request May 2, 2022

Merge bitcoin#12196: Add scantxoutset RPC method

61ccba8

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022

Add scantxoutset RPC method #12196

Add scantxoutset RPC method #12196

Conversation

jonasschnelli commented Jan 16, 2018

gmaxwell commented Jan 16, 2018

jonasschnelli commented Jan 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

promag Jan 16, 2018 • edited

Choose a reason for hiding this comment

promag Jan 16, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laanwj commented Jan 16, 2018 • edited

jonasschnelli commented Jan 20, 2018

jonasschnelli commented Jan 20, 2018 • edited

jonasschnelli commented Jan 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luke-jr commented Jan 24, 2018

jonasschnelli commented Jan 24, 2018

luke-jr commented Jan 24, 2018

greenaddress commented Feb 26, 2018

jonasschnelli commented Feb 27, 2018

jonasschnelli commented Feb 27, 2018

luke-jr commented Feb 28, 2018

jonasschnelli commented Feb 28, 2018

jonasschnelli commented Jul 15, 2018

promag commented Jul 17, 2018

surindergiri commented Oct 1, 2020

promag commented Oct 1, 2020 • edited

promag Jan 16, 2018 •

edited

promag Jan 16, 2018 •

edited

laanwj commented Jan 16, 2018 •

edited

jonasschnelli commented Jan 20, 2018 •

edited

promag commented Oct 1, 2020 •

edited