Add ability to convert solvability info to descriptor #14477

sipa · 2018-10-13T20:37:47Z

This PR adds functionality to convert a script to a descriptor, given a SigningProvider with the relevant information about public keys and redeemscripts/witnessscripts.

The feature is exposed in listunspent, getaddressinfo, and scantxoutset whenever these calls are applied to solvable outputs/addresses.

This is not very useful on its own, though when we add RPCs to import descriptors, or sign PSBTs using descriptors, these strings become a compact and standalone way of conveying everything necessary to sign an output (excluding private keys).

Unit tests and rudimentary RPC tests are included (more relevant tests can be added once RPCs support descriptors).

Fixes #14503.

ryanofsky

utACK 002bdef765a7cdb4dcf66fa186a4c4a1a7fc7b71

src/script/descriptor.cpp

src/script/descriptor.h

src/test/descriptor_tests.cpp

sipa · 2018-10-17T01:35:13Z

Addressed all nits, and added some comments.

Sjors · 2018-10-17T09:05:46Z

Concept ACK. Lightly tested a5e0b62; e.g. getaddressinfo shows a descriptor now.

The descriptors only show the public key, which is consistent with getaddressinfo not showing private keys and seeds. Maybe add a descriptor boolean argument to dumpprivkey as a way to obtain descriptors with a private key?

promag

For large wallets this will make listunspent perform a lot worse?

test/functional/wallet_address_types.py

src/wallet/rpcwallet.cpp

ryanofsky · 2018-10-17T17:48:19Z

utACK a5e0b620062bf00a5f24d195bcaf52839681cb5d. Only changes since last review were making some suggested changes and expanding the InferDescriptor.

It would also be nice to add release notes for the RPC changes in this PR.

For large wallets this will make listunspent perform a lot worse?

It seems like the new work listunspent is doing is similar to the work it was doing before with calling ExtractDestination and pwallet->GetCScript.

sipa · 2018-10-19T01:09:57Z

Sorry for the big change, but I realized that after #14150, we'll want to integrate origin information into the inferred descriptors this PR brings, so I went ahead and did that (it's based on 14150 now).

ryanofsky

utACK cf4b935a0301972d0e1fc6a5cb32c6ff564dbbbc. Changes: rebasing over #14150, adding InferPubkey() and a test for the inferred origin field, making the address_types test ignore origin information in inferred descriptors, and adding release notes.

src/wallet/rpcwallet.cpp

test/functional/wallet_address_types.py

DrahtBot · 2018-10-20T10:00:08Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#14646 (Add expansion cache functions to descriptors (unused for now) by sipa)
#14601 ([rpc] Descriptions: Consistent arg labels for types 'object', 'array', 'boolean', and 'string' by ch4ot1c)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Sjors · 2018-10-23T10:33:54Z

Lightly tested e6e4704; getaddressinfo descriptor now includes origin info.

ryanofsky

utACK e6e4704. Changes since last review are rebase, 'solvable' naming, and more comprehensive test code that compares getaddressinfo descriptors with listunspent descriptors and PBST bip32_derivs paths.

test/functional/wallet_address_types.py

meshcollider

utACK e6e4704

src/script/descriptor.cpp

src/wallet/rpcwallet.cpp

ryanofsky

utACK 2286f206e4ea139477b88867a13a491d8d14db3d. Changes since last review are renaming descriptor fields to desc, adding them to scantxoutset output, and tweaking release notes.

ryanofsky · 2018-10-30T18:00:49Z

src/rpc/blockchain.cpp

@@ -2140,7 +2142,11 @@ UniValue scantxoutset(const JSONRPCRequest& request)
                if (!desc->Expand(i, provider, scripts, provider)) {
                    throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, strprintf("Cannot derive script without private keys: '%s'", desc_str));
                }
-                needles.insert(scripts.begin(), scripts.end());
+                for (const auto script : scripts) {
+                    std::string desc = InferDescriptor(script, provider)->ToString();


In commit "Add matching descriptors to scantxoutset output + tests" (fe9b68a81a6f3a9abac7b3b7740d5019129d4fff)

I think I agree with @Sjors in #14477 (comment) that behavior that comes from inferring a descriptor here, instead of deriving it directly from desc above, is a little unintuitive. IMO, it'd be nicer if Expand took an optional std::vector<std::unique_ptr<Descriptor>>* output_descriptors argument and filled it in parallel with the existing output_scripts argument.

This is just a suggestion, though, and maybe something to consider for a future PR.

That makes sense. The inferred descriptor is functionally equivalent with a more straightforward specialization of the descriptor you're suggesting, but descriptors are intended to at least be somewhat human readable, so this may matter.

However, it's quite a bit of code to implement that (it needs specialization code for a number of constructions, including combo() which would need to get split out into ``pkh()andpk()` etc), so I'd prefer to keep it for a later PR.

For that reason I also prefer not making any explicit promises about what is put in the RPC outputs; for now all it guarantees is something that encapsulates all the information. For scantxoutset we could indeed be better and maintain more of the input descriptor's structure.

promag

Commit adding release note could be reworded to "doc: Add release notes for #14477".

src/rpc/blockchain.cpp

src/script/descriptor.cpp

src/rpc/blockchain.cpp

laanwj · 2018-11-23T12:07:38Z

utACK 109699d

ryanofsky

utACK 109699d. Only superficial changes since last review: replacing assert with assert_equal, moving a variable declaration, avoiding copies with & and std::move.

109699d Add release notes (Pieter Wuille) b65326b Add matching descriptors to scantxoutset output + tests (Pieter Wuille) 16203d5 Add descriptors to listunspent and getaddressinfo + tests (Pieter Wuille) 9b2a25b Add tests for InferDescriptor and Descriptor::IsSolvable (Pieter Wuille) 225bf3e Add Descriptor::IsSolvable() to distinguish addr/raw from others (Pieter Wuille) 4d78bd9 Add support for inferring descriptors from scripts (Pieter Wuille) Pull request description: This PR adds functionality to convert a script to a descriptor, given a `SigningProvider` with the relevant information about public keys and redeemscripts/witnessscripts. The feature is exposed in `listunspent`, `getaddressinfo`, and `scantxoutset` whenever these calls are applied to solvable outputs/addresses. This is not very useful on its own, though when we add RPCs to import descriptors, or sign PSBTs using descriptors, these strings become a compact and standalone way of conveying everything necessary to sign an output (excluding private keys). Unit tests and rudimentary RPC tests are included (more relevant tests can be added once RPCs support descriptors). Fixes #14503. Tree-SHA512: cb36b84a3e0200375b7e06a98c7e750cfaf95cf5de132cad59f7ec3cbd201f739427de0dc108f515be7aca203652089fbf5f24ed283d4553bddf23a3224ab31f

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14477 | PR14477]] : bitcoin/bitcoin@225bf3e Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D6125

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14477 | PR14477]] : bitcoin/bitcoin@4d78bd9 I made InferPubkey static because it is only called from there. Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Subscribers: majcosta Differential Revision: https://reviews.bitcoinabc.org/D6124

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14477 | PR14477]] : bitcoin/bitcoin@9b2a25b Depends on D6124 and D6125 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D6126

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14477 | PR14477]] : bitcoin/bitcoin@16203d5 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D6136

Summary: This is a patial backport of Core [[bitcoin/bitcoin#14477 | PR14477]] : bitcoin/bitcoin@b65326b Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D6138

Summary: This is a patial backport of Core [[bitcoin/bitcoin#14477 | PR14477]] : bitcoin/bitcoin@109699d Test Plan: Read. Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D6137

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14477 | PR14477]] : bitcoin/bitcoin@225bf3e Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D6125

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14477 | PR14477]] : bitcoin/bitcoin@4d78bd9 I made InferPubkey static because it is only called from there. Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Subscribers: majcosta Differential Revision: https://reviews.bitcoinabc.org/D6124

…iptor 109699d Add release notes (Pieter Wuille) b65326b Add matching descriptors to scantxoutset output + tests (Pieter Wuille) 16203d5 Add descriptors to listunspent and getaddressinfo + tests (Pieter Wuille) 9b2a25b Add tests for InferDescriptor and Descriptor::IsSolvable (Pieter Wuille) 225bf3e Add Descriptor::IsSolvable() to distinguish addr/raw from others (Pieter Wuille) 4d78bd9 Add support for inferring descriptors from scripts (Pieter Wuille) Pull request description: This PR adds functionality to convert a script to a descriptor, given a `SigningProvider` with the relevant information about public keys and redeemscripts/witnessscripts. The feature is exposed in `listunspent`, `getaddressinfo`, and `scantxoutset` whenever these calls are applied to solvable outputs/addresses. This is not very useful on its own, though when we add RPCs to import descriptors, or sign PSBTs using descriptors, these strings become a compact and standalone way of conveying everything necessary to sign an output (excluding private keys). Unit tests and rudimentary RPC tests are included (more relevant tests can be added once RPCs support descriptors). Fixes bitcoin#14503. Tree-SHA512: cb36b84a3e0200375b7e06a98c7e750cfaf95cf5de132cad59f7ec3cbd201f739427de0dc108f515be7aca203652089fbf5f24ed283d4553bddf23a3224ab31f

meshcollider added the Wallet label Oct 13, 2018

ryanofsky approved these changes Oct 16, 2018

View reviewed changes

src/script/descriptor.cpp Outdated Show resolved Hide resolved

src/script/descriptor.h Outdated Show resolved Hide resolved

src/script/descriptor.h Show resolved Hide resolved

src/test/descriptor_tests.cpp Outdated Show resolved Hide resolved

sipa force-pushed the 201810_inferdescript branch from 002bdef to c3d524b Compare October 17, 2018 01:35

sipa force-pushed the 201810_inferdescript branch from c3d524b to a5e0b62 Compare October 17, 2018 06:54

promag reviewed Oct 17, 2018

View reviewed changes

test/functional/wallet_address_types.py Outdated Show resolved Hide resolved

src/wallet/rpcwallet.cpp Show resolved Hide resolved

sipa force-pushed the 201810_inferdescript branch from a5e0b62 to d71424c Compare October 19, 2018 01:08

sipa force-pushed the 201810_inferdescript branch from d71424c to cf4b935 Compare October 19, 2018 01:51

ryanofsky reviewed Oct 19, 2018

View reviewed changes

src/wallet/rpcwallet.cpp Outdated Show resolved Hide resolved

test/functional/wallet_address_types.py Outdated Show resolved Hide resolved

sipa force-pushed the 201810_inferdescript branch from cf4b935 to b9199e1 Compare October 19, 2018 23:02

sipa mentioned this pull request Oct 20, 2018

Add SegWit support to importmulti #14454

Merged

This was referenced Oct 20, 2018

test: Add linter to make sure single parameter constructors are marked explicit #14505

Merged

Rpc help helper class #14502

Closed

This was referenced Oct 20, 2018

docs: Consistent type names in RPC help descriptions #14459

Closed

doc/descriptors.md tweaks #14161

Merged

sipa force-pushed the 201810_inferdescript branch from b9199e1 to 3b8b71e Compare October 21, 2018 01:46

DrahtBot added the Needs rebase label Oct 21, 2018

sipa force-pushed the 201810_inferdescript branch from 3b8b71e to e6e4704 Compare October 22, 2018 22:35

DrahtBot removed the Needs rebase label Oct 22, 2018

ryanofsky approved these changes Oct 23, 2018

View reviewed changes

test/functional/wallet_address_types.py Outdated Show resolved Hide resolved

meshcollider reviewed Oct 25, 2018

View reviewed changes

src/script/descriptor.cpp Show resolved Hide resolved

src/wallet/rpcwallet.cpp Outdated Show resolved Hide resolved

sipa added 3 commits October 26, 2018 10:21

Add support for inferring descriptors from scripts

4d78bd9

Add Descriptor::IsSolvable() to distinguish addr/raw from others

225bf3e

Add tests for InferDescriptor and Descriptor::IsSolvable

9b2a25b

sipa force-pushed the 201810_inferdescript branch from e6e4704 to 47ac6d0 Compare October 26, 2018 17:38

ryanofsky approved these changes Oct 30, 2018

View reviewed changes

sipa added this to Blockers in High-priority for review Nov 1, 2018

promag reviewed Nov 4, 2018

View reviewed changes

src/rpc/blockchain.cpp Outdated Show resolved Hide resolved

src/script/descriptor.cpp Show resolved Hide resolved

laanwj removed this from Blockers in High-priority for review Nov 8, 2018

practicalswift reviewed Nov 12, 2018

View reviewed changes

src/rpc/blockchain.cpp Outdated Show resolved Hide resolved

sipa added 3 commits November 14, 2018 14:21

Add descriptors to listunspent and getaddressinfo + tests

16203d5

Add matching descriptors to scantxoutset output + tests

b65326b

Add release notes

109699d

sipa force-pushed the 201810_inferdescript branch from 2286f20 to 109699d Compare November 14, 2018 22:27

This was referenced Nov 14, 2018

Add P2SH-P2WSH support to listunspent RPC #14481

Merged

Tracker for places where descriptors could/should be integrated #14739

Closed

Sjors mentioned this pull request Nov 20, 2018

Bugfix: RPC/Wallet: Include HD key metadata in dumpwallet #11803

Closed

ryanofsky approved these changes Nov 27, 2018

View reviewed changes

sipa merged commit 109699d into bitcoin:master Nov 27, 2018

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to convert solvability info to descriptor #14477

Add ability to convert solvability info to descriptor #14477

sipa commented Oct 13, 2018 •

edited

ryanofsky left a comment

sipa commented Oct 17, 2018

Sjors commented Oct 17, 2018

promag left a comment

ryanofsky commented Oct 17, 2018

sipa commented Oct 19, 2018

ryanofsky left a comment

DrahtBot commented Oct 20, 2018 •

edited

Sjors commented Oct 23, 2018

ryanofsky left a comment

meshcollider left a comment

ryanofsky left a comment

ryanofsky Oct 30, 2018

sipa Oct 30, 2018

promag left a comment

laanwj commented Nov 23, 2018

ryanofsky left a comment

Add ability to convert solvability info to descriptor #14477

Add ability to convert solvability info to descriptor #14477

Conversation

sipa commented Oct 13, 2018 • edited

ryanofsky left a comment

Choose a reason for hiding this comment

sipa commented Oct 17, 2018

Sjors commented Oct 17, 2018

promag left a comment

Choose a reason for hiding this comment

ryanofsky commented Oct 17, 2018

sipa commented Oct 19, 2018

ryanofsky left a comment

Choose a reason for hiding this comment

DrahtBot commented Oct 20, 2018 • edited

Conflicts

Sjors commented Oct 23, 2018

ryanofsky left a comment

Choose a reason for hiding this comment

meshcollider left a comment

Choose a reason for hiding this comment

ryanofsky left a comment

Choose a reason for hiding this comment

ryanofsky Oct 30, 2018

Choose a reason for hiding this comment

sipa Oct 30, 2018

Choose a reason for hiding this comment

promag left a comment

Choose a reason for hiding this comment

laanwj commented Nov 23, 2018

ryanofsky left a comment

Choose a reason for hiding this comment

sipa commented Oct 13, 2018 •

edited

DrahtBot commented Oct 20, 2018 •

edited