BIP-322: Generic signed message format #16440

kallewoof · 2019-07-23T06:29:13Z

This PR implements BIP-322, for the single proof (no multiple addresses simultaneously) single signer (no multisig addresses) case.

UI (CLI/QT) are restricted to the single proof case, but the underlying code (script/proof.h) supports multiple proofs.

Recommend ?w=1 / -w to avoid whitespace spam.

There is a related PR #16653 that includes the sign/verify components of this and the signet PR (#16411).

DrahtBot · 2019-07-23T06:38:49Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#18267 (BIP-325: Signet [consensus] by kallewoof)
#18115 (wallet: Pass in transactions and messages for signing instead of exporting the private keys by achow101)
#18002 (Abstract out script execution out of VerifyWitnessProgram() by sipa)
#17977 ([WIP] Implement BIP 340-342 validation (Schnorr/taproot/tapscript) by sipa)
#17938 (Disallow automatic conversion between disparate hash types by Empact)
#16653 (script: add simple signature support (checker/creator) by kallewoof)
#16549 ([WIP] UI external signer support (e.g. hardware wallet) by Sjors)
#16546 ([WIP] External signer support - Wallet Box edition by Sjors)
#16528 (Native Descriptor Wallets using DescriptorScriptPubKeyMan by achow101)
#16411 (BIP-325: Signet support by kallewoof)
#13062 (Make script interpreter independent from storage type CScript by sipa)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

laanwj · 2019-07-23T13:32:35Z

Concept ACK

harding

I really like the idea of BIP322, but I don't like that this changes existing RPCs in way that's not backwards compatible (e.g. a signmessage created and verified on 0.18 produces a "CDataStream::read(): end of data: iostream error" here). Similarly, a P2PKH signmessage created on this branch results in false when checked with 0.18. Either P2PKH signing/verifying should be special cased to use the legacy signmessage format, or BIP322 message signing should use different RPCs (or at least a special RPC parameter or configuration option to indicate the user wants to use BIP322).

Just on this branch, I also created a 2-of-3 P2WSH address using addmultisigaddress and then attempted to signmessage with it. I received the error, ScriptPubKey does not refer to a key. Unless I'm misunderstanding BIP322, this should be possible and it would be cool if the wallet knew how to do it (however, even if the wallet doesn't know how to do it, as long as it's possible by BIP322, the error message should be changed). I didn't test any more advanced scripts, although I think that would be worthwhile to test later---in theory, the wallet should be able to signmessage for any script where its solver could sign for a spend. I think importing some advanced scripts using descriptors and then signmessage'ing for them might make a good addition to the integration tests in this PR.

Another concern is that I don't believe the discussion over how BIP322 should handle timelocking opcodes (CLTV and CSV) was ever resolved. I think that's probably important before this could be released.

Finally, I skimmed the code and left one documentation-related note. Thanks for working on this!

harding · 2019-07-29T00:50:33Z

src/rpc/misc.cpp

+            {"message", RPCArg::Type::STR, RPCArg::Optional::NO, "The message to create a signature of."},
+        },
+        RPCResult{
+            "\"signature\"          (string) The signature of the message encoded in base 64 (note: the address for the private key is the derived BECH32 address; any other type will fail validation)\n"


I think "the derived BECH32 address" will become ambiguous if subsequent versions of segwit are adopted. E.g. if taproot is adopted, there would be two possible bech32 addresses for the same public key. Perhaps this should say, "the derived P2WPKH bech32 address".

Nit: BECH is not an acronym AFAIK and so should not be all caps. A quick git grep shows that the convention in the repository is to treat it as a common noun, so all lowercase unless at the start of a sentence.

kallewoof · 2019-07-29T03:30:17Z

@harding

I really like the idea of BIP322, but I don't like that this changes existing RPCs in way that's not backwards compatible (e.g. a signmessage created and verified on 0.18 produces a "CDataStream::read(): end of data: iostream error" here). Similarly, a P2PKH signmessage created on this branch results in false when checked with 0.18. Either P2PKH signing/verifying should be special cased to use the legacy signmessage format, or BIP322 message signing should use different RPCs (or at least a special RPC parameter or configuration option to indicate the user wants to use BIP322).

I don't really agree. Proving that you own an address is something that you do "now", not something you do and keep the proof around indefinitely. Losing an old proof should be perfectly fine, as the point is lost if you can't reproduce it.

Of course, down the road Craig could claim that his broken proof actually worked, "but that core changed the mechanism so it can't be verified anymore" but I don't know if anybody cares about him enough to warrant keeping both kinds of verifiers (we probably do not want to keep the signmessage part).

Just on this branch, I also created a 2-of-3 P2WSH address using addmultisigaddress and then attempted to signmessage with it. I received the error, ScriptPubKey does not refer to a key. Unless I'm misunderstanding BIP322, this should be possible and it would be cool if the wallet knew how to do it (however, even if the wallet doesn't know how to do it, as long as it's possible by BIP322, the error message should be changed). I didn't test any more advanced scripts, although I think that would be worthwhile to test later---in theory, the wallet should be able to signmessage for any script where its solver could sign for a spend. I think importing some advanced scripts using descriptors and then signmessage'ing for them might make a good addition to the integration tests in this PR.

Yes, BIP322 literally just says "the sighash (normally based on the transaction) is X; now sign/verify with this address" and uses exactly the same code that is used by the wallet to sign transactions, so anything goes. I'm actually confused why your example doesn't work out of the box. I'll look into that.

Another concern is that I don't believe the discussion over how BIP322 should handle timelocking opcodes (CLTV and CSV) was ever resolved. I think that's probably important before this could be released.

Actually, those should just be removed; they were meant for the proof of funds, which I am removing from BIP322. Thanks for the nudge.

Finally, I skimmed the code and left one documentation-related note. Thanks for working on this!

Great! Thanks, will address.

harding · 2019-07-29T04:12:19Z

@kallewoof

I don't like that this changes existing RPCs in way that's not backwards compatible [...]

I don't really agree. Proving that you own an address is something that you do "now", not something you do and keep the proof around indefinitely. [...]

I think there are several points here:

This change breaks backwards compatibility between different versions of Bitcoin Core. I think it's unfortunate if someone running version x can't create or verify proofs when talking to someone running version y. Sure, it'd be nice if everyone upgraded to the latest version, but that takes time (sometimes years in the case of cold wallets).
This change breaks compatibility with almost every other wallet that has implemented signmessage support. Again, it'd be great if they all have BIP322 support ready at the same time this code gets into a Bitcoin Core release, but that seems extremely unlikely to me---I expect it to take years to get widespread BIP322 support. Then each of those wallets has their own user upgrade cycle which may take years, so we're talking years + years until everyone is using BIP322 (if it is generally adopted at all). Until near-universal adoption happens, it'd be nice if Bitcion Core is compatible with other wallets.
As you mention, this breaks old signmessages. While I don't feel strongly about that and I'm at least somewhat sympathetic with your arguments about people just generating new proofs, I think unnecessary breakage is something we should try to avoid.

I think the appropriate solution to all three issues is that these existing RPCs should continue to generate and verify legacy P2PKH signmessages when called with P2PKH addresses. They could automatically switch between legacy support and BIP322 support depending on address type or BIP322 could get new RPCs. (Something to consider would be amending BIP322 so that P2PKH addresses in BIP322 are special cased to use the old signmessage format; that way this PR can be made fully BIP322 compliant and yet backward compatible.)

kallewoof · 2019-07-29T04:55:26Z

I see what you're saying.

I think adding an optional 'format' which defaults to 'bip322' but can be set to 'legacy' for 'sign', and autodetection in 'verify' should be sufficient. What do you think?

harding · 2019-07-29T05:40:13Z

@kallewoof

I think adding an optional 'format' which defaults to 'bip322' but can be set to 'legacy' for 'sign', and autodetection in 'verify' should be sufficient. What do you think?

Sounds reasonable to me. I'd slightly prefer the default be legacy if the user provides a P2PKH address---but not enough to argue about it. Thanks!

sipa · 2019-07-29T05:44:03Z

Thanks for picking this up again.

I don't think we should gratuitously break compatibility with the existing signature scheme; that's generally not the approach we take with RPCs, and given that the scheme is implemented in other software too probably even more disruptive.

I wouldn't be opposed to adding an exception in BIP322 that for P2PKH address the existing key recovery based scheme is used; in general I like avoiding ambiguity in these things. However I also see this may add additional complication to the spec, and perhaps not be very clear cut either (why just the P2PKH ones, and not the extensions to other single-key address types people have proposed and adopted in other software, for example?).

kallewoof · 2019-07-29T05:54:55Z

@harding @sipa Defaulting to legacy for P2PKH means we can basically never get rid of the legacy code. That's okay, but unfortunate, IMO.

sipa · 2019-07-29T06:07:30Z

I didn't expect we'd ever be able to get rid of the old format, even if the default would be a new format now. That'd indeed unfortunate, but that's life when you want compatibility. I don't think it's a big problem; it's not that much code.

kallewoof · 2019-07-29T06:29:59Z

@harding @sipa I have updated the specification to include the special case, and I also added a description of the legacy format: https://github.com/bitcoin/bips/blob/e24e86b482e394e18803f14c7e2338aab9b7e1e2/bip-0322.mediawiki (from bitcoin/bips#808)

Let me know if I got something wrong there. I wrote the legacy format spec based on the code.

kallewoof · 2019-07-29T08:51:53Z

I updated the code in this PR to do what was suggested (use legacy if p2pkh, otherwise use bip322), and I also updated bip322 to require this.

harding · 2019-07-29T14:40:14Z

Tested signmessage and verifymessage both ways between 0.18 and this branch, looks good, thanks! You may want to consider making signmessagewithprivkey default to using P2PKH for backwards compatibility and add a third argument to that RPC that allows selecting P2WPKH (I'd suggest the argument select between P2PKH and P2WPKH (rather than legacy vs bip322) so that it can be extended to take a P2TR argument or other address format argument in the future).

I skimmed this commit with the proposed changes to BIP322 and left a couple nits. As a specification, I think it could also use a description of CKey::SignCompact() and CKey::RecoverCompact() such as how they serialize which derived pubkey to use and whether or not it's compressed or uncompressed. An alternative to putting this information in BIP322 would be to create a separate BIP for the established legacy signmessage format (which was created before we had BIPs) and then simply have BIP322 reference that BIP. I can help with documentation if you'd like.

kallewoof · 2019-07-29T15:00:54Z

@harding

FYI the bitcoin/bips PR was merged as you were reviewing it; I opened a fix-up here: bitcoin/bips#814

Thanks for testing!

You may want to consider making signmessagewithprivkey default to using P2PKH for backwards compatibility and add a third argument to that RPC that allows selecting P2WPKH (I'd suggest the argument select between P2PKH and P2WPKH (rather than legacy vs bip322) so that it can be extended to take a P2TR argument or other address format argument in the future).

Sure, since we are not getting rid of the legacy format, we might as well use it where it is usable.

Edit: This has been updated. It uses the output type parsing from other places to allow legacy, p2sh-segwit, or bech32. I assume once taproot and such appear, they will be added to this and this should "just work [tm]".

sipa · 2019-07-29T16:53:47Z

src/interfaces/wallet.h

@@ -255,6 +256,9 @@ class Wallet
    // Remove wallet.
    virtual void remove() = 0;

+    // Get a signing provider for the wallet.
+    virtual SigningProvider& getSigningProvider() = 0;


You may want to check with @achow101 that this doesn't conflict with the descriptor wallet changes.

This doesn't interfere with the descriptor wallet changes since a SigningProvider was never actually needed in the GUI.

But there won't be a singular signing provider for the whole wallet anymore, right?

FWIW, I think it would be better to add an interfaces::Wallet::signMessage() method instead of adding an interfaces::Wallet::getSigningProvider() method. Few reasons:

It would allow deduplicating more code for signing messages instead of having it written twice in qt and rpcwallet.

It would create a little less work for me with #10102, so I don't have to an add IPC wrapper around SigningProvider, or serialize SigningProvider with private key information and send it from the wallet process to the qt process.

Just in general I think rpcwallet.cpp is too monolothic and has too much important code mixed with UniValue boilerplate. So I'd love to see something like src/wallet/sign.h / src/wallet/sign.cpp files with a bool SignMessage(CWallet& wallet, message, options...) function and start organizing things better.

Anyway, I'm fine with this approach, but wanted to suggest an alternative.

EDIT: Changed Wallet& to CWallet& above (sorry for the confusing typo)

@ryanofsky That sounds good to me! I am not sure how to access the interfaces::Wallet instance from rpcwallet.cpp though. It seems like it only has access to CWallet (while the QT side only has access to interfaces::Wallet).

Is the idea to add two signmessage functions, where the interface one simply calls down into m_wallet->SignMessage(..)?

@ryanofsky That sounds good to me! I am not sure how to access the interfaces::Wallet instance from rpcwallet.cpp though. It seems like it only has access to CWallet (while the QT side only has access to interfaces::Wallet).

rpcwallet.cpp shouldn't need to access interfaces::Wallet here (and in general) because interfaces::Wallet doesn't contain any real functionality, it's just a high level wrapper around selected wallet functions that GUI code uses to indirectly control wallets.

So the idea is just to implement a standalone function in wallet code:

bool SignMessage(CWallet& wallet, message, options...)

and to call this function both from wallet/rpcwallet.cpp and interfaces/wallet.cpp (just adding a one-line WalletImpl::signMessage() wrapper method in the interface code).

Is the idea to add two signmessage functions, where the interface one simply calls down into m_wallet->SignMessage(..)?

Sorry! Just realized I had a typo in #16440 (comment), which is fixed now but was probably pretty confusing. I was trying to suggest not having a CWallet::SignMesage(...) method, but instead defining a standalone SignMessage(CWallet&, ...) function. Really either a method or a standalone function would be fine, but the function seemed better because we're gradually breaking CWallet up and pulling functionality out of it, so adding new functionality there without a reason seemed to be moving in wrong direction.

@ryanofsky I ended up adding a couple of helper functions to proof.h and adding a proof.cpp file. The P2PKH-vs-others logic is now all contained inside the proof code, and is called from rpcwallet/misc/interfaces/wallet:

bitcoin/src/script/proof.h

Lines 237 to 250 in 6e9947e

/**

* Attempt to sign a message with the given destination.

*/

void SignMessageWithSigningProvider(SigningProvider* sp, const std::string& message, const CTxDestination& destination, std::vector<uint8_t>& signature_out);

/**

* Attempt to sign a message with the given private key and address format.

*/

void SignMessageWithPrivateKey(CKey& key, OutputType address_type, const std::string& message, std::vector<uint8_t>& signature_out);

/**

* Determine if a signature is valid for the given message.

*/

Result VerifySignature(const std::string& message, const CTxDestination& destination, const std::vector<uint8_t>& signature);

The interface side ends up like

bitcoin/src/interfaces/wallet.cpp

Lines 460 to 463 in 6e9947e

void signMessage(const std::string& message, const CTxDestination& destination, std::vector<uint8_t>& signature_out) override

{

proof::SignMessageWithSigningProvider(m_wallet.get(), message, destination, signature_out);

}

As a side result of this, strMessageMagic was moved out of validation.h/cpp into proof.cpp, and is no longer public, which is nice.

Looks good. The new SignMessageWithSigningProvider makes for fewer wallet changes than what I suggested, which is great!

The simple signature takes a sighash as argument and verifies a signature and pubkey against it. It is used in signet for verifying blocks and in BIP-322 implementation for verifying messages.

- signmessagewithprivkey - verifymessage - signmessage

DrahtBot · 2020-03-09T21:20:51Z

🐙 This pull request conflicts with the target branch and needs rebase.

kallewoof · 2020-03-25T06:37:12Z

Will reopen (or re-create) after adapting simplifications in bitcoin/bips#903

fanquake added the Needs Conceptual Review label Jul 23, 2019

kallewoof force-pushed the feature-generic-signed-message-format branch 2 times, most recently from 9fa4b81 to 37a2542 Compare Jul 23, 2019

kallewoof force-pushed the feature-generic-signed-message-format branch 7 times, most recently from a138b79 to ef8fa5c Compare Jul 27, 2019

harding reviewed Jul 29, 2019

View changes

kallewoof force-pushed the feature-generic-signed-message-format branch from ef8fa5c to 0048216 Compare Jul 29, 2019

kallewoof force-pushed the feature-generic-signed-message-format branch from 0048216 to c390c53 Compare Jul 29, 2019

kallewoof force-pushed the feature-generic-signed-message-format branch from c390c53 to b96875a Compare Jul 29, 2019

kallewoof force-pushed the feature-generic-signed-message-format branch 2 times, most recently from 65bc98a to 053afb2 Compare Jul 29, 2019

sipa reviewed Jul 29, 2019

View changes

DrahtBot mentioned this pull request Feb 11, 2020

rpc: Auto-format RPCResult #17809

Merged

DrahtBot mentioned this pull request Feb 11, 2020

refactor: deduplicate the message sign/verify code #17577

Merged

DrahtBot mentioned this pull request Feb 11, 2020

Native Descriptor Wallets using DescriptorScriptPubKeyMan #16528

Merged

DrahtBot mentioned this pull request Feb 11, 2020

Log RPC parameters (arguments) if -debug=rpcparams #16365

Draft

DrahtBot mentioned this pull request Feb 12, 2020

Make script interpreter independent from storage type CScript #13062

Open

DrahtBot mentioned this pull request Feb 21, 2020

External signer support - Wallet Box edition #16546

Merged

2 tasks

DrahtBot mentioned this pull request Feb 22, 2020

UI external signer support (e.g. hardware wallet) #16549

Closed

DrahtBot mentioned this pull request Feb 24, 2020

refactor: consolidate sendmany and sendtoaddress code #18202

Merged

DrahtBot added the Needs rebase label Feb 25, 2020

kallewoof force-pushed the feature-generic-signed-message-format branch from e3a2396 to 7d9c869 Compare Mar 4, 2020

DrahtBot removed the Needs rebase label Mar 4, 2020

kallewoof force-pushed the feature-generic-signed-message-format branch 3 times, most recently from b5248e4 to 51dea4b Compare Mar 4, 2020

DrahtBot added the Needs rebase label Mar 4, 2020

kallewoof added 2 commits Mar 7, 2020

script: add simple signature support (checker/creator)

d496977

The simple signature takes a sighash as argument and verifies a signature and pubkey against it. It is used in signet for verifying blocks and in BIP-322 implementation for verifying messages.

script: add BIP-322 helpers and wallet interface

f8ecbc1

kallewoof force-pushed the feature-generic-signed-message-format branch from 51dea4b to ebf4d23 Compare Mar 7, 2020

kallewoof added 3 commits Mar 7, 2020

rpc: update RPC commands to BIP-322

39ee690

- signmessagewithprivkey - verifymessage - signmessage

qt/gui: update message signing/verification to BIP-322

789bb98

test: update signmessage related tests

ebf4d23

DrahtBot mentioned this pull request Mar 7, 2020

BIP-325: Signet [consensus] #18267

Merged

DrahtBot removed the Needs rebase label Mar 7, 2020

DrahtBot mentioned this pull request Mar 7, 2020

BIP-325: Signet support #16411

Closed

3 tasks

DrahtBot added the Needs rebase label Mar 9, 2020

kallewoof closed this Mar 25, 2020

harding mentioned this pull request Jan 1, 2021

bip322: (another) significant overhaul bitcoin/bips#1048

Merged

bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022

BIP-322: Generic signed message format #16440

BIP-322: Generic signed message format #16440

kallewoof commented Jul 23, 2019 •

edited

DrahtBot commented Jul 23, 2019 •

edited

laanwj commented Jul 23, 2019

harding left a comment •

edited

harding Jul 29, 2019

kallewoof commented Jul 29, 2019

harding commented Jul 29, 2019

kallewoof commented Jul 29, 2019

harding commented Jul 29, 2019

sipa commented Jul 29, 2019

kallewoof commented Jul 29, 2019

sipa commented Jul 29, 2019 •

edited

kallewoof commented Jul 29, 2019 •

edited

kallewoof commented Jul 29, 2019

harding commented Jul 29, 2019

kallewoof commented Jul 29, 2019 •

edited

sipa Jul 29, 2019

achow101 Jul 30, 2019

sipa Jul 30, 2019

ryanofsky Jul 30, 2019 •

edited

kallewoof Jul 31, 2019

ryanofsky Jul 31, 2019 •

edited

kallewoof Jul 31, 2019 •

edited

kallewoof Jul 31, 2019

ryanofsky Jul 31, 2019

DrahtBot commented Mar 9, 2020

kallewoof commented Mar 25, 2020

	/**
	* Attempt to sign a message with the given destination.
	*/
	void SignMessageWithSigningProvider(SigningProvider* sp, const std::string& message, const CTxDestination& destination, std::vector<uint8_t>& signature_out);

	/**
	* Attempt to sign a message with the given private key and address format.
	*/
	void SignMessageWithPrivateKey(CKey& key, OutputType address_type, const std::string& message, std::vector<uint8_t>& signature_out);

	/**
	* Determine if a signature is valid for the given message.
	*/
	Result VerifySignature(const std::string& message, const CTxDestination& destination, const std::vector<uint8_t>& signature);

	void signMessage(const std::string& message, const CTxDestination& destination, std::vector<uint8_t>& signature_out) override
	{
	proof::SignMessageWithSigningProvider(m_wallet.get(), message, destination, signature_out);
	}

BIP-322: Generic signed message format #16440

BIP-322: Generic signed message format #16440

Conversation

kallewoof commented Jul 23, 2019 • edited

DrahtBot commented Jul 23, 2019 • edited

Conflicts

laanwj commented Jul 23, 2019

harding left a comment • edited

harding Jul 29, 2019

Choose a reason for hiding this comment

kallewoof commented Jul 29, 2019

harding commented Jul 29, 2019

kallewoof commented Jul 29, 2019

harding commented Jul 29, 2019

sipa commented Jul 29, 2019

kallewoof commented Jul 29, 2019

sipa commented Jul 29, 2019 • edited

kallewoof commented Jul 29, 2019 • edited

kallewoof commented Jul 29, 2019

harding commented Jul 29, 2019

kallewoof commented Jul 29, 2019 • edited

sipa Jul 29, 2019

Choose a reason for hiding this comment

achow101 Jul 30, 2019

Choose a reason for hiding this comment

sipa Jul 30, 2019

Choose a reason for hiding this comment

ryanofsky Jul 30, 2019 • edited

Choose a reason for hiding this comment

kallewoof Jul 31, 2019

Choose a reason for hiding this comment

ryanofsky Jul 31, 2019 • edited

Choose a reason for hiding this comment

kallewoof Jul 31, 2019 • edited

Choose a reason for hiding this comment

kallewoof Jul 31, 2019

Choose a reason for hiding this comment

ryanofsky Jul 31, 2019

Choose a reason for hiding this comment

DrahtBot commented Mar 9, 2020

kallewoof commented Mar 25, 2020

kallewoof commented Jul 23, 2019 •

edited

DrahtBot commented Jul 23, 2019 •

edited

harding left a comment •

edited

sipa commented Jul 29, 2019 •

edited

kallewoof commented Jul 29, 2019 •

edited

kallewoof commented Jul 29, 2019 •

edited

ryanofsky Jul 30, 2019 •

edited

ryanofsky Jul 31, 2019 •

edited

kallewoof Jul 31, 2019 •

edited