Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native Descriptor Wallets using DescriptorScriptPubKeyMan #16528

Merged
merged 43 commits into from Apr 27, 2020

Conversation

achow101
Copy link
Member

@achow101 achow101 commented Aug 2, 2019

Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using createwallet.

Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each.

Descriptors can also be imported with a new importdescriptors RPC.

Native descriptor wallets use the ScriptPubKeyMan interface introduced in #16341 to add a DescriptorScriptPubKeyMan. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, DescriptorScriptPubKeyMan does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with disable_private_keys needs to be used for watchonly things.

A --descriptor option was added to some tests (wallet_basic.py, wallet_encryption.py, wallet_keypool.py, wallet_keypool_topup.py, and wallet_labels.py) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (importprivkey, importpubkey, importaddress, importmulti, addmultisigaddress, dumpprivkey, dumpwallet, importwallet, and sethdseed).

@fanquake fanquake added the Wallet label Aug 2, 2019
@fanquake fanquake added this to PRs in Descriptor wallets Aug 2, 2019
@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch from acbd19c to 6f8afc0 Compare Aug 2, 2019
@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Aug 2, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #18774 (test: added test for upgradewallet RPC by brakmic)
  • #18727 (test: Add CreateWalletFromFile test by ryanofsky)
  • #18699 (wallet: Avoid translating RPC errors by MarcoFalke)
  • #18654 (rpc: separate bumpfee's psbt creation function into psbtbumpfee by achow101)
  • #18618 (gui: Drop RecentRequestsTableModel dependency to WalletModel by promag)
  • #18617 (test: add factor option to adjust test timeouts by brakmic)
  • #18608 (refactor: Remove CAddressBookData::destdata by ryanofsky)
  • #18570 (rpc: return block hash & height in getbalances, gettransaction & getwalletinfo JSONs by brakmic)
  • #18479 (RPC: Show fee in results for signrawtransaction* for segwit inputs by luke-jr)
  • #18244 (rpc: fundrawtransaction and walletcreatefundedpsbt respect locks even with manual coin selection by Sjors)
  • #17977 ([WIP] Implement BIP 340-342 validation (Schnorr/taproot/tapscript) by sipa)
  • #17877 (qt, refactor: Make enums in BitcoinUnits class scoped by hebasto)
  • #16463 ([BIP 174] Implement serialization support for GLOBAL_XPUB field. by achow101)
  • #16432 (qt: Add privacy to the Overview page by hebasto)
  • #16224 (gui: Bilingual GUI error messages by hebasto)
  • #11413 ([wallet] [rpc] sendtoaddress/sendmany: Add explicit feerate option by kallewoof)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@meshcollider
Copy link
Contributor

@meshcollider meshcollider commented Aug 2, 2019

Strong Concept ACK, this should be much nicer now it is preceded by the rework.

@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch 2 times, most recently from ef59fb7 to 90a631f Compare Aug 2, 2019
@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch 2 times, most recently from ef63001 to 755c4d1 Compare Aug 2, 2019
@Sjors
Copy link
Member

@Sjors Sjors commented Aug 3, 2019

Approach ACK. It looks pretty straight forward and thanks to The Box feels cleaner than the previous attempt.

I suggest that, after a bit more progress on #16341, we review this in parallel. That ensures that we don't mess up the division of labour between ScriptPubKeyMan and its subclasses.

Also concept ACK on using BIP44/49/84 for new descriptor wallets. That may need some discussion, because it means individual addresses are no longer hardened.

Some issues I found perusing the commits:

  • when you restart bitcoind and load a (watch-only) wallet the keypool is empty and getnewaddress no longer works
  • importdescriptors should no longer require a range argument (also I would prefer a singular RPC nvm that makes rescan slow)
  • when calling getnewaddress with an address type for which the wallet misses a descriptor, it returns a blank error message:
  • can you give each ScriptPubKeyManager it's own file?
  • deleting GetMetadata and Upgrade inside Introduce WalletDescriptor; should have its own commit?
  • ScriptPubKeyMap could be introduced in Implement IsMine instead of the earlier commit. Would it make sense to move this into the Descriptor class? How are infinite range descriptors handled, expanded and cached keypoolsize items at a time?
  • Am I reading this wrong or is SetupGeneration creating a fresh seed for each descriptor?
  • I'm not a fan of determining the output type in an indirect manner by expanding the descriptor Optional<OutputType> out_script_type = DetermineOutputType(scripts_temp[0], out_keys);; that information is already encoded in the descriptor. Shameless plug for #15590.

@hugohn
Copy link
Contributor

@hugohn hugohn commented Aug 12, 2019

Concept ACK.

@achow101 I just noticed that both the existing deriveaddressses & importmulti commands treat descriptor [range_start, range_end] as inclusive, which is consistent with the common understanding of the notation []. Should we update the new importdescriptors command to do the same (make range_end inclusive)?

https://github.com/bitcoin/bitcoin/blob/master/src/rpc/misc.cpp#L215
https://github.com/bitcoin/bitcoin/blob/master/src/wallet/rpcdump.cpp#L1122

@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch from 755c4d1 to d777554 Compare Aug 12, 2019
@achow101
Copy link
Member Author

@achow101 achow101 commented Aug 12, 2019

Should we update the new importdescriptors command to do the same (make range_end inclusive)?

I guess so. Latest pushed should do that.

@Sjors
Copy link
Member

@Sjors Sjors commented Oct 10, 2019

Due to your latest change upstream, this now complains: scriptpubkeyman.h:516:10: warning: 'CheckDecryptionKey' overrides a member function but is not marked 'override'

jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 15, 2020
Summary:
Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys,
KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses,
RewriteDB

Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [18/43] : bitcoin/bitcoin@4cb9b69

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8416
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 15, 2020
… wallet file

Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [19/43] : bitcoin/bitcoin@46dfb99

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8417
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 16, 2020
Summary:
Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [20/43] : bitcoin/bitcoin@e014886

Depends on D8417

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, PiRK, majcosta

Reviewed By: #bitcoin_abc, PiRK, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8418
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 16, 2020
Summary:
Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [21/43] : bitcoin/bitcoin@58c7651

Depends on D8417

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Subscribers: majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8419
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 16, 2020
Summary:
Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [22/43] : bitcoin/bitcoin@bfdd073

Depends on D8419

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Subscribers: majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8421
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 17, 2020
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [23/43] : bitcoin/bitcoin@a775f7c

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8426
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 17, 2020
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [24/43] : bitcoin/bitcoin@f866957

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8427
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 17, 2020
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [25/43] : bitcoin/bitcoin@586b57a

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8428
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 18, 2020
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [26/43] :bitcoin/bitcoin@f1ca5fe

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8445
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 18, 2020
Summary:
Internally, a GetSigningProvider function is introduced which allows for
some private keys to be optionally included. This can be called with a
script as the argument (i.e. a scriptPubKey from our wallet when we are
signing) or with a pubkey. In order to know what index to expand the
private keys for that pubkey, we need to also cache all of the pubkeys
involved when we expand the descriptor. So SetCache and TopUp are
updated to do this too.

Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [27/43] : bitcoin/bitcoin@d50c8dd

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8446
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 18, 2020
Summary:
Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [28/43] : bitcoin/bitcoin@bde7c9f

Depends on D8446

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8447
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 18, 2020
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [29/43] : bitcoin/bitcoin@84b4978

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8448
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 19, 2020
Summary:
FillPSBT will add our own scripts to the PSBT if those inputs are ours.
If an input also lists pubkeys that we happen to know the private keys
for, we will sign those inputs too.

Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [30/43] : bitcoin/bitcoin@72a9540

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8451
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 19, 2020
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [31/43] : bitcoin/bitcoin@8b9603b

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien, majcosta

Reviewed By: #bitcoin_abc, Fabien, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8453
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 19, 2020
Summary:
Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [31/43] : bitcoin/bitcoin@b713baa

Depends on D8453

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D8454
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 19, 2020
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [33/43] : bitcoin/bitcoin@82ae02b

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D8455
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 19, 2020
Summary:
Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [34/43] : bitcoin/bitcoin@1cb42b2

Depends on D8455

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D8457
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 19, 2020
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [35/43] : bitcoin/bitcoin@ce24a94

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D8458
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 19, 2020
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [40/43] : bitcoin/bitcoin@886e0d7

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D8466
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 19, 2020
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [41/43] : bitcoin/bitcoin@cf06062

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D8468
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 19, 2020
Summary:
When a CWallet doesn't have a ScriptPubKeyMan for the requested type
in GetNewDestination, give a meaningful error. Also handle this in
Qt which did not do anything with errors.

Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [39/43] : bitcoin/bitcoin@3c19fdd

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D8467
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 21, 2020
Summary:
Co-authored-by: Andrew Chow <achow101-github@achow101.com>

Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [36/43] : bitcoin/bitcoin@f193ea8

Add the wallet_util changes from [[bitcoin/bitcoin@223588b | 223588b]] because the patch is broken without it.

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Subscribers: majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8475
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 21, 2020
Summary:
Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [37/43] : bitcoin/bitcoin@1346e14

Depends on D8475

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Subscribers: majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8487
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Nov 23, 2020
Summary:
RPCOverloadWrapper overloads some deprecated or disabled RPCs with
an implementation using other RPCs to avoid having a ton of code churn
around replacing those RPCs.

Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [42/43] : bitcoin/bitcoin@869f7ab

Depends on D8487

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D8498
Mengerian pushed a commit to Mengerian/bitcoin-abc that referenced this issue Jan 5, 2021
Summary: Backport of Core [[bitcoin/bitcoin#16528 | PR16528]] [38/43] : bitcoin/bitcoin@388ba94

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D8488
Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jan 27, 2021
…mbers are left uninitialized after construction

Summary:
> This is a small folllow-up to [[bitcoin/bitcoin#16528 | PR16528]] ("Native Descriptor Wallets using DescriptorScriptPubKeyMan")
> Before this change bool m_internal was left uninitialized when using the DescriptorScriptPubKeyMan(WalletStorage&, WalletDescriptor&) ctor.
> The same goes for the now initialized integers which were left uninitialized when using the WalletDescriptor() ctor.

wallet: Make sure no DescriptorScriptPubKeyMan members are uninitialized after construction
bitcoin/bitcoin@ff046ae

wallet: Make sure no WalletDescriptor members are uninitialized after construction
bitcoin/bitcoin@2a78098

This is a backport of Core [[bitcoin/bitcoin#18782 | PR18782]]

Test Plan: `ninja all check-all`

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D9073
Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Oct 5, 2021
Summary:
> Adds a --descriptors option globally to the test framework. This will
> make the test create and use descriptor wallets. However some tests may
> not work with this.
>
> Some tests are modified to work with --descriptors and run with that
>  option in test_runer:
> * wallet_encryption.py
> * wallet_keypool.py
> * wallet_keypool_topup.py
> * wallet_labels.py
> * wallet_avoidreuse.py

This is a backport of [[bitcoin/bitcoin#16528 | core#16528]] [43a/43]
bitcoin/bitcoin@223588b

I'm splitting this commit into multiple commits to make review easier.
This first part contains everything that works more or less out of the
box and does not require significant changes to work with our codebase.

All the deviations from the original PR in this revision are explained by out of order
backports and lack of segwit and bech32. For instance, changes to key.py
were already included in D8475 & D9934. Deviations in `RPCOverloadWrapper.createwallet`
are explained by D10185 and D9101.

I excluded specifically 3 tests that require changes to transaction
amounts and fees when descriptors are used. With our codebase,
descriptor wallets generate different fees, because in some situations
the legacy wallet generates legacy p2pkh outputs and descriptor wallets generate
`scriptPubKey` / `p2sh` outputs. I will address these in one or two
separate diffs.

These excluded tests are:
 - rpc_psbt.py
 - wallet_basic.py
 - wallet_keypool.py

Test Plan: `ninja check-functional`

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Subscribers: Fabien

Differential Revision: https://reviews.bitcoinabc.org/D10213
Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jan 26, 2022
Summary:
> Tests that a fully signed transaction given to
> `signrawtransactionwithwallet` is both unchanged and marked as complete.
> This tests for a regression in Bitcoin Core 0.20 where the transaction would not be
> marked as complete.

We do not have the regression in Bitcoin ABC and it seems unlikely that it will happen, as it was bissected to [[bitcoin/bitcoin#16528 | core#16528]] which we already finished backporting, and had to be fixed only on Core 0.20 (not master). But just in case, it is better to have the test to avoid any bad surprise.

This is a backport of [[bitcoin/bitcoin#20562 | core#20562]]

Test Plan: `ninja check-functional`

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D10894
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

None yet