Native Descriptor Wallets using DescriptorScriptPubKeyMan

[achow101]

Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using createwallet.

Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each.

Descriptors can also be imported with a new importdescriptors RPC.

Native descriptor wallets use the ScriptPubKeyMan interface introduced in #16341 to add a DescriptorScriptPubKeyMan. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, DescriptorScriptPubKeyMan does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with disable_private_keys needs to be used for watchonly things.

A --descriptor option was added to some tests (wallet_basic.py, wallet_encryption.py, wallet_keypool.py, wallet_keypool_topup.py, and wallet_labels.py) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (importprivkey, importpubkey, importaddress, importmulti, addmultisigaddress, dumpprivkey, dumpwallet, importwallet, and sethdseed).

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• test: added test for upgradewallet RPC #18774 (test: added test for upgradewallet RPC by brakmic)
• test: Add CreateWalletFromFile test #18727 (test: Add CreateWalletFromFile test by ryanofsky)
• wallet: Avoid translating RPC errors #18699 (wallet: Avoid translating RPC errors by MarcoFalke)
• rpc: separate bumpfee's psbt creation function into psbtbumpfee #18654 (rpc: separate bumpfee's psbt creation function into psbtbumpfee by achow101)
• gui: Drop RecentRequestsTableModel dependency to WalletModel #18618 (gui: Drop RecentRequestsTableModel dependency to WalletModel by promag)
• test: add factor option to adjust test timeouts #18617 (test: add factor option to adjust test timeouts by brakmic)
• refactor: Remove CAddressBookData::destdata #18608 (refactor: Remove CAddressBookData::destdata by ryanofsky)
• rpc: return block hash & height in getbalances, gettransaction & getwalletinfo JSONs #18570 (rpc: return block hash & height in getbalances, gettransaction & getwalletinfo JSONs by brakmic)
• RPC: Show fee in results for signrawtransaction* for segwit inputs #18479 (RPC: Show fee in results for signrawtransaction* for segwit inputs by luke-jr)
• rpc: fundrawtransaction and walletcreatefundedpsbt also lock manually selected coins #18244 (rpc: fundrawtransaction and walletcreatefundedpsbt respect locks even with manual coin selection by Sjors)
• Implement BIP 340-342 validation (Schnorr/taproot/tapscript) #17977 ([WIP] Implement BIP 340-342 validation (Schnorr/taproot/tapscript) by sipa)
• qt, refactor: Make BitcoinUnits::Unit a scoped enum #17877 (qt, refactor: Make enums in BitcoinUnits class scoped by hebasto)
• [BIP 174] Implement serialization support for GLOBAL_XPUB field. #16463 ([BIP 174] Implement serialization support for GLOBAL_XPUB field. by achow101)
• qt: Add privacy to the Overview page #16432 (qt: Add privacy to the Overview page by hebasto)
• gui: Bilingual GUI error messages #16224 (gui: Bilingual GUI error messages by hebasto)
• [wallet] [rpc] sendtoaddress/sendmany: Add explicit feerate option #11413 ([wallet] [rpc] sendtoaddress/sendmany: Add explicit feerate option by kallewoof)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

[meshcollider]

Strong Concept ACK, this should be much nicer now it is preceded by the rework.

[Sjors]

Approach ACK. It looks pretty straight forward and thanks to The Box feels cleaner than the previous attempt.

I suggest that, after a bit more progress on #16341, we review this in parallel. That ensures that we don't mess up the division of labour between ScriptPubKeyMan and its subclasses.

Also concept ACK on using BIP44/49/84 for new descriptor wallets. That may need some discussion, because it means individual addresses are no longer hardened.

Some issues I found perusing the commits:

• when you restart bitcoind and load a (watch-only) wallet the keypool is empty and getnewaddress no longer works
• importdescriptors should no longer require a range argument (also I would prefer a singular RPC nvm that makes rescan slow)
• when calling getnewaddress with an address type for which the wallet misses a descriptor, it returns a blank error message:
• can you give each ScriptPubKeyManager it's own file?
• deleting GetMetadata and Upgrade inside Introduce WalletDescriptor; should have its own commit?
• ScriptPubKeyMap could be introduced in Implement IsMine instead of the earlier commit. Would it make sense to move this into the Descriptor class? How are infinite range descriptors handled, expanded and cached keypoolsize items at a time?
• Am I reading this wrong or is SetupGeneration creating a fresh seed for each descriptor?
• I'm not a fan of determining the output type in an indirect manner by expanding the descriptor Optional<OutputType> out_script_type = DetermineOutputType(scripts_temp[0], out_keys);; that information is already encoded in the descriptor. Shameless plug for Descriptor: add GetAddressType() and IsSegWit() #15590.

[hugohn]

Concept ACK.

@achow101 I just noticed that both the existing deriveaddressses & importmulti commands treat descriptor [range_start, range_end] as inclusive, which is consistent with the common understanding of the notation []. Should we update the new importdescriptors command to do the same (make range_end inclusive)?

https://github.com/bitcoin/bitcoin/blob/master/src/rpc/misc.cpp#L215
https://github.com/bitcoin/bitcoin/blob/master/src/wallet/rpcdump.cpp#L1122

[achow101]

Should we update the new importdescriptors command to do the same (make range_end inclusive)?

I guess so. Latest pushed should do that.

[Sjors]

Due to your latest change upstream, this now complains: scriptpubkeyman.h:516:10: warning: 'CheckDecryptionKey' overrides a member function but is not marked 'override'