External signer support - Wallet Box edition #16546

Sjors · 2019-08-04T21:42:31Z

Big picture overview in this gist.

This PR lets bitcoind call an arbitrary command -signer=<cmd>, e.g. a hardware wallet driver, where it can fetch public keys, ask to display an address, and sign a transaction (using PSBT under the hood).

It's design to work with https://github.com/bitcoin-core/HWI, which supports multiple hardware wallets. Any command with the same arguments and return values will work. It simplifies the manual procedure described here.

Usage is documented in doc/external-signer.md, which also describes what protocol a different signer binary should conform to.

Use --enable-external-signer to opt in, requires Boost::Process:

Options used to compile and link:
  with wallet     = yes
  with gui / qt   = no
  external signer = yes

It adds the following RPC methods:

enumeratesigners: asks for a list of signers (e.g. devices) and their master key fingerprint
signerdisplayaddress <address>: asks to display an address

It enhances the following RPC methods:

createwallet: takes an additional external_signer argument and fetches keys from device
send: automatically sends transaction to device and waits

Usage TL&DR:

clone HWI repo somewhere and launch bitcoind -signer=../HWI/hwi.py
check if you can see your hardware device: bitcoin-cli enumeratesigners
create wallet and auto import keys bitcoin-cli createwallet "hww" true true "" true true true
display address on device: bitcoin-cli signerdisplayaddress ...
to spend, use send RPC and approve transaction on device

Prerequisites:

wallet: load flags before everything else #21127 load wallet flags before everything else
build: remove mostly pointless BOOST_PROCESS macro #21182 remove mostly pointless BOOST_PROCESS macro

Potentially useful followups:

GUI support: UI external signer support (e.g. hardware wallet) bitcoin-core/gui#4
bumpfee support
(automatically) verify (a subset of) keys on the device after import, through message signing

DrahtBot · 2019-08-04T22:16:27Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

MOVEONLY: CWallet transaction code out of wallet.cpp/.h #21207 (MOVEONLY: CWallet transaction code out of wallet.cpp/.h by ryanofsky)
wallet: check when create wallets for the reserved name "wallets" #21073 (wallet: check when create wallets for the reserved name "wallets" by Saibato)
Use std::filesystem. Remove Boost Filesystem & System #20744 ([POC] Use std::filesystem. Remove Boost Filesystem & System by fanquake)
test: Add feature_taproot.py --previous_release #20354 (test: Add feature_taproot.py --previous_release by MarcoFalke)
wallet, refactor: make DescriptorScriptPubKeyMan agnostic of internal flag #20191 (wallet, refactor: make DescriptorScriptPubKeyMan agnostic of internal flag by S3RK)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

luke-jr · 2019-08-04T23:24:13Z

src/wallet/rpcwallet.cpp

@@ -2653,6 +2673,8 @@ static UniValue createwallet(const JSONRPCRequest& request)
            {"blank", RPCArg::Type::BOOL, /* default */ "false", "Create a blank wallet. A blank wallet has no keys or HD seed. One can be set using sethdseed."},
            {"passphrase", RPCArg::Type::STR, RPCArg::Optional::OMITTED, "Encrypt the wallet with this passphrase."},
            {"avoid_reuse", RPCArg::Type::BOOL, /* default */ "false", "Keep track of coin reuse, and treat dirty and clean coins differently with privacy considerations in mind."},
+            {"descriptors", RPCArg::Type::BOOL, /* default */ "false", "Create a native descriptor wallet. The wallet will use descriptors internally to handle address creation"},
+            {"external_signer", RPCArg::Type::BOOL, /* default */ "false", "Use an external signer such as a hardware wallet. Requires -signer to be configured. Wallet creation will fail if keys cannot be fetched. Requires disable_private_keys and descriptors set to true."},


This list-of-bools thing seems like a terrible idea...

Indeed, I was complaining about that when blank was added :-)

luke-jr · 2019-08-04T23:25:27Z

src/wallet/rpcwallet.cpp

+
+    if (pwallet->IsWalletFlagSet(WALLET_FLAG_EXTERNAL_SIGNER)) {
+        const std::string command = gArgs.GetArg("-signer", DEFAULT_EXTERNAL_SIGNER);
+        if (command == "") throw std::runtime_error(std::string(__func__) + ": restart bitcoind with -signer=<cmd>");


What if there are multiple wallets, with different signers?

IMO -signer needs to be replaced with either a wallet-stored path, or a path provided when the wallet is loaded...

Storing signer in the wallet makes sense, but I also think it can wait for a later PR. In practice afaik the only tool that currently works is HWI and it can handle multiple wallets. In the long run however I do hope that wallet manufactures provide their own software that just uses the same commands / responses.

Relaxo143 · 2019-09-12T11:45:16Z

Can we expect this to be merged and included in 0.19? It's a really useful and requested feature! Thanks to everyone who is working on it.

Sjors · 2019-09-16T08:54:00Z

@Relaxo143 not a chance; this is still work in progress and there's several pull requests that need to be reviewed and merged first. There's also a feature freeze on 0.19.

Sjors · 2019-09-16T16:27:25Z

I dropped the dependency on my new send RPC proposal #16378, in favor of just tweaking sendmany and sendtoaddress. It's less powerful, but should reduce the review burden once native descriptors are merged.

laanwj · 2021-02-23T16:50:41Z

re-ACK f75e0c1

maflcko · 2021-02-23T17:10:47Z

doc/external-signer.md

+Create a wallet, this automatically imports the public keys:
+
+```sh
+$ bitcoin-cli createwallet "hww" true true "" true true true


nit: Would be nice to use named args

maflcko · 2021-02-23T17:25:01Z

src/wallet/rpcsigner.cpp

+        },
+        RPCExamples{""},
+        [](const RPCHelpMan& self, const JSONRPCRequest& request) -> UniValue {
+            std::shared_ptr<CWallet> const wallet = GetWalletForJSONRPCRequest(request);


why does this rpc require a wallet?

I might get rid of that requirement in a followup.

…ion and typos 8b08d0f build, doc: Fix configure script output indentation and typos (Hennadii Stepanov) Pull request description: This PR is follow up of bitcoin#16546, that breaks the `configure` script output indentation for gui/qt/qr lines: ``` Options used to compile and link: external signer = no multiprocess = no with libs = yes with wallet = yes with sqlite = yes with bdb = yes with gui / qt = yes with qr = yes with zmq = yes with test = yes ... ``` With this PR: ``` Options used to compile and link: external signer = no multiprocess = no with libs = yes with wallet = yes with sqlite = yes with bdb = yes with gui / qt = yes with qr = yes with zmq = yes with test = yes ... ``` Also typos are fixed. ACKs for top commit: Sjors: utACK 8b08d0f vasild: ACK 8b08d0f Tree-SHA512: 46dfcfb754192dbcc080348781327d1714e2f9a696f3ed9252746b426e3afc628d84adb197ba3b8080eacaa6053ccac07e670998930ae92cef8ed713dd457c10

hebasto · 2021-03-02T17:10:09Z

There is a buggy execution path in the configure script. Fixed in #21339.

…conditional a412813 build: Make AM_CONDITIONAL([ENABLE_EXTERNAL_SIGNER]) unconditional (Hennadii Stepanov) 9fef209 build, refactor: Fix indentation for if..then..fi (Hennadii Stepanov) Pull request description: #16546 introduced a regression in the `configure`: ``` $ ./autogen.sh $ ./configure --disable-wallet --without-utils --without-daemon --without-gui --disable-tests --disable-bench ... checking whether to build test_bitcoin... no checking whether to reduce exports... no checking that generated files are newer than configure... done configure: error: conditional "ENABLE_EXTERNAL_SIGNER" was never defined. Usually this means the macro was only invoked conditionally. ``` This PR fixes this bug, and refactors indentation to make easier to spot similar bugs in the future. ACKs for top commit: Sjors: utACK a412813 fanquake: ACK a412813 - this fixes the bug described, and improves readability. Tree-SHA512: 4469dcc006690f38f93c3cdf8d15b76f5fc8ea76e87a1b5db5ee891dc9851f6ec539f2a6fd02a361aa76baa4f4b2b9fe8289137f5d9734ee5984f265cb131ef5

57ff5a4 doc: specify minimum HWI version (Sjors Provoost) 03308b2 rpc: don't require wallet for enumeratesigners (Sjors Provoost) Pull request description: HWI just released 2.0. See https://github.com/bitcoin-core/HWI/releases/tag/2.0.0 As of #16546 we already rely on features that are in 2.0 and not in the previous 1.* releases: * `--chain` param This shouldn't be a problem, because HWI 2.0 has been released before we release v22. Misc improvements: * document that HWI 2.0 is required * drop wallet requirement for `enumeratesigners` ACKs for top commit: achow101: Code Review ACK 57ff5a4 Tree-SHA512: 3fb6ba20894e52a116f89525a5f5a1f61d363ccd904e1cffd0e6d095640fc6d2edf0388cd6ae20f83bbc31e5f458255ec090b6e823798d426eba3e45b4336bf9

An earlier version of bitcoin#16546 used both --with-boost-process and --enable-external-signer, which was simplified to only use the latter. However I forgot to update CI, so the external signer tests were not run.

hebasto · 2021-06-11T04:17:19Z

test/config.ini.in

@@ -23,3 +23,4 @@ RPCAUTH=@abs_top_srcdir@/share/rpcauth/rpcauth.py
 @BUILD_BITCOIND_TRUE@ENABLE_BITCOIND=true
 @ENABLE_FUZZ_TRUE@ENABLE_FUZZ=true
 @ENABLE_ZMQ_TRUE@ENABLE_ZMQ=true
+@ENABLE_EXTERNAL_SIGNER_TRUE@ENABLE_EXTERNAL_SIGNER=true


~~It seems the @ENABLE_EXTERNAL_SIGNER_TRUE@ substitution does not work for some reasons.~~

nm, forgot to run ./autogen.sh

Sjors mentioned this pull request Aug 4, 2019

[WIP] External signer support (e.g. hardware wallet) #14912

Closed

DrahtBot added Build system Docs GUI RPC/REST/ZMQ Tests Utils/log/libs Wallet labels Aug 4, 2019

luke-jr reviewed Aug 4, 2019

View reviewed changes

fanquake removed Build system Docs GUI Utils/log/libs labels Aug 4, 2019

Sjors force-pushed the 2019/08/hww-box2 branch from 10b713d to 7421e89 Compare August 5, 2019 11:02

Sjors mentioned this pull request Aug 5, 2019

UI external signer support (e.g. hardware wallet) #16549

Closed

Sjors force-pushed the 2019/08/hww-box2 branch from 7421e89 to 0e28746 Compare August 5, 2019 17:48

DrahtBot added the Needs rebase label Aug 6, 2019

Sjors force-pushed the 2019/08/hww-box2 branch from 0e28746 to 7511650 Compare September 4, 2019 19:28

DrahtBot removed the Needs rebase label Sep 4, 2019

Sjors force-pushed the 2019/08/hww-box2 branch from 7511650 to 8d88bf9 Compare September 4, 2019 19:35

Sjors mentioned this pull request Sep 4, 2019

Native descriptor wallets justinmoon/junction#21

Open

DrahtBot added the Needs rebase label Sep 7, 2019

Sjors force-pushed the 2019/08/hww-box2 branch from 8d88bf9 to 3e991aa Compare September 16, 2019 16:24

Sjors mentioned this pull request Sep 17, 2019

External signer multisig support #16895

Closed

2 tasks

Sjors force-pushed the 2019/08/hww-box2 branch from 3e991aa to ca42459 Compare September 18, 2019 18:49

Sjors added 5 commits February 23, 2021 14:34

wallet: add GetExternalSigner()

fc5da52

wallet: ExternalSigner: add GetDescriptors method

7ebc7c0

rpc: signerdisplayaddress

245b445

rpc: send: support external signer

d4b0107

doc: add external-signer.md

f75e0c1

Sjors force-pushed the 2019/08/hww-box2 branch from 9a4571f to f75e0c1 Compare February 23, 2021 13:35

laanwj merged commit a9335e4 into bitcoin:master Feb 23, 2021

maflcko reviewed Feb 23, 2021

View reviewed changes

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Feb 23, 2021

Merge bitcoin#16546: External signer support - Wallet Box edition

a51a9d8

Sjors deleted the 2019/08/hww-box2 branch February 23, 2021 19:35

hebasto mentioned this pull request Feb 24, 2021

build, doc: Fix configure script output indentation and typos #21292

Merged

hebasto mentioned this pull request Mar 2, 2021

build: Make AM_CONDITIONAL([ENABLE_EXTERNAL_SIGNER]) unconditional #21339

Merged

Sjors mentioned this pull request Mar 11, 2021

Misc external signer improvement and HWI 2 support #21417

Merged

bitcoin deleted a comment from mariosanchezgarcia Apr 8, 2021

bitcoin locked and limited conversation to collaborators Apr 8, 2021

hebasto reviewed Jun 11, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

External signer support - Wallet Box edition #16546

External signer support - Wallet Box edition #16546

Sjors commented Aug 4, 2019 •

edited

Loading

DrahtBot commented Aug 4, 2019 •

edited

Loading

luke-jr Aug 4, 2019

Sjors Aug 5, 2019

luke-jr Aug 4, 2019

Sjors Aug 5, 2019 •

edited

Loading

Relaxo143 commented Sep 12, 2019

Sjors commented Sep 16, 2019

Sjors commented Sep 16, 2019

laanwj commented Feb 23, 2021

maflcko Feb 23, 2021

maflcko Feb 23, 2021

Sjors Feb 23, 2021

hebasto commented Mar 2, 2021

hebasto Jun 11, 2021 •

edited

Loading

hebasto Jun 11, 2021

External signer support - Wallet Box edition #16546

External signer support - Wallet Box edition #16546

Conversation

Sjors commented Aug 4, 2019 • edited Loading

DrahtBot commented Aug 4, 2019 • edited Loading

Conflicts

luke-jr Aug 4, 2019

Choose a reason for hiding this comment

Sjors Aug 5, 2019

Choose a reason for hiding this comment

luke-jr Aug 4, 2019

Choose a reason for hiding this comment

Sjors Aug 5, 2019 • edited Loading

Choose a reason for hiding this comment

Relaxo143 commented Sep 12, 2019

Sjors commented Sep 16, 2019

Sjors commented Sep 16, 2019

laanwj commented Feb 23, 2021

maflcko Feb 23, 2021

Choose a reason for hiding this comment

maflcko Feb 23, 2021

Choose a reason for hiding this comment

Sjors Feb 23, 2021

Choose a reason for hiding this comment

hebasto commented Mar 2, 2021

hebasto Jun 11, 2021 • edited Loading

Choose a reason for hiding this comment

hebasto Jun 11, 2021

Choose a reason for hiding this comment

Sjors commented Aug 4, 2019 •

edited

Loading

DrahtBot commented Aug 4, 2019 •

edited

Loading

Sjors Aug 5, 2019 •

edited

Loading

hebasto Jun 11, 2021 •

edited

Loading