BIP-322: Generic signed message format

[kallewoof]

This PR implements BIP-322, for the single proof (no multiple addresses simultaneously) single signer (no multisig addresses) case.

UI (CLI/QT) are restricted to the single proof case, but the underlying code (script/proof.h) supports multiple proofs.

Recommend ?w=1 / -w to avoid whitespace spam.

There is a related PR #16653 that includes the sign/verify components of this and the signet PR (#16411).

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #16439 (RPC: support "@height" in place of blockhash for getblock etc by ajtowns)
• #16341 (Introduce ScriptPubKeyMan interface and use it for key and script management (aka wallet boxes) by achow101)
• #13462 (Make SER_GETHASH implicit for CHashWriter and SerializeHash by Empact)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

[laanwj]

Concept ACK

[harding]

I really like the idea of BIP322, but I don't like that this changes existing RPCs in way that's not backwards compatible (e.g. a signmessage created and verified on 0.18 produces a "CDataStream::read(): end of data: iostream error" here). Similarly, a P2PKH signmessage created on this branch results in false when checked with 0.18. Either P2PKH signing/verifying should be special cased to use the legacy signmessage format, or BIP322 message signing should use different RPCs (or at least a special RPC parameter or configuration option to indicate the user wants to use BIP322).

Just on this branch, I also created a 2-of-3 P2WSH address using addmultisigaddress and then attempted to signmessage with it. I received the error, ScriptPubKey does not refer to a key. Unless I'm misunderstanding BIP322, this should be possible and it would be cool if the wallet knew how to do it (however, even if the wallet doesn't know how to do it, as long as it's possible by BIP322, the error message should be changed). I didn't test any more advanced scripts, although I think that would be worthwhile to test later---in theory, the wallet should be able to signmessage for any script where its solver could sign for a spend. I think importing some advanced scripts using descriptors and then signmessage'ing for them might make a good addition to the integration tests in this PR.

Another concern is that I don't believe the discussion over how BIP322 should handle timelocking opcodes (CLTV and CSV) was ever resolved. I think that's probably important before this could be released.

Finally, I skimmed the code and left one documentation-related note. Thanks for working on this!

[harding]

I really like the idea of BIP322, but I don't like that this changes existing RPCs in way that's not backwards compatible (e.g. a signmessage created and verified on 0.18 produces a "CDataStream::read(): end of data: iostream error" here). Similarly, a P2PKH signmessage created on this branch results in false when checked with 0.18. Either P2PKH signing/verifying should be special cased to use the legacy signmessage format, or BIP322 message signing should use different RPCs (or at least a special RPC parameter or configuration option to indicate the user wants to use BIP322).

Just on this branch, I also created a 2-of-3 P2WSH address using addmultisigaddress and then attempted to signmessage with it. I received the error, ScriptPubKey does not refer to a key. Unless I'm misunderstanding BIP322, this should be possible and it would be cool if the wallet knew how to do it (however, even if the wallet doesn't know how to do it, as long as it's possible by BIP322, the error message should be changed). I didn't test any more advanced scripts, although I think that would be worthwhile to test later---in theory, the wallet should be able to signmessage for any script where its solver could sign for a spend. I think importing some advanced scripts using descriptors and then signmessage'ing for them might make a good addition to the integration tests in this PR.

Another concern is that I don't believe the discussion over how BIP322 should handle timelocking opcodes (CLTV and CSV) was ever resolved. I think that's probably important before this could be released.

Finally, I skimmed the code and left one documentation-related note. Thanks for working on this!

[harding]

I think "the derived BECH32 address" will become ambiguous if subsequent versions of segwit are adopted. E.g. if taproot is adopted, there would be two possible bech32 addresses for the same public key. Perhaps this should say, "the derived P2WPKH bech32 address".

Nit: BECH is not an acronym AFAIK and so should not be all caps. A quick git grep shows that the convention in the repository is to treat it as a common noun, so all lowercase unless at the start of a sentence.

[kallewoof]

@harding

I really like the idea of BIP322, but I don't like that this changes existing RPCs in way that's not backwards compatible (e.g. a signmessage created and verified on 0.18 produces a "CDataStream::read(): end of data: iostream error" here). Similarly, a P2PKH signmessage created on this branch results in false when checked with 0.18. Either P2PKH signing/verifying should be special cased to use the legacy signmessage format, or BIP322 message signing should use different RPCs (or at least a special RPC parameter or configuration option to indicate the user wants to use BIP322).

I don't really agree. Proving that you own an address is something that you do "now", not something you do and keep the proof around indefinitely. Losing an old proof should be perfectly fine, as the point is lost if you can't reproduce it.

Of course, down the road Craig could claim that his broken proof actually worked, "but that core changed the mechanism so it can't be verified anymore" but I don't know if anybody cares about him enough to warrant keeping both kinds of verifiers (we probably do not want to keep the signmessage part).

Just on this branch, I also created a 2-of-3 P2WSH address using addmultisigaddress and then attempted to signmessage with it. I received the error, ScriptPubKey does not refer to a key. Unless I'm misunderstanding BIP322, this should be possible and it would be cool if the wallet knew how to do it (however, even if the wallet doesn't know how to do it, as long as it's possible by BIP322, the error message should be changed). I didn't test any more advanced scripts, although I think that would be worthwhile to test later---in theory, the wallet should be able to signmessage for any script where its solver could sign for a spend. I think importing some advanced scripts using descriptors and then signmessage'ing for them might make a good addition to the integration tests in this PR.

Yes, BIP322 literally just says "the sighash (normally based on the transaction) is X; now sign/verify with this address" and uses exactly the same code that is used by the wallet to sign transactions, so anything goes. I'm actually confused why your example doesn't work out of the box. I'll look into that.

Another concern is that I don't believe the discussion over how BIP322 should handle timelocking opcodes (CLTV and CSV) was ever resolved. I think that's probably important before this could be released.

Actually, those should just be removed; they were meant for the proof of funds, which I am removing from BIP322. Thanks for the nudge.

Finally, I skimmed the code and left one documentation-related note. Thanks for working on this!

Great! Thanks, will address.

[harding]

@kallewoof

I don't like that this changes existing RPCs in way that's not backwards compatible [...]

I don't really agree. Proving that you own an address is something that you do "now", not something you do and keep the proof around indefinitely. [...]

I think there are several points here:

1. This change breaks backwards compatibility between different versions of Bitcoin Core. I think it's unfortunate if someone running version x can't create or verify proofs when talking to someone running version y. Sure, it'd be nice if everyone upgraded to the latest version, but that takes time (sometimes years in the case of cold wallets).

2. This change breaks compatibility with almost every other wallet that has implemented signmessage support. Again, it'd be great if they all have BIP322 support ready at the same time this code gets into a Bitcoin Core release, but that seems extremely unlikely to me---I expect it to take years to get widespread BIP322 support. Then each of those wallets has their own user upgrade cycle which may take years, so we're talking years + years until everyone is using BIP322 (if it is generally adopted at all). Until near-universal adoption happens, it'd be nice if Bitcion Core is compatible with other wallets.

3. As you mention, this breaks old signmessages. While I don't feel strongly about that and I'm at least somewhat sympathetic with your arguments about people just generating new proofs, I think unnecessary breakage is something we should try to avoid.

I think the appropriate solution to all three issues is that these existing RPCs should continue to generate and verify legacy P2PKH signmessages when called with P2PKH addresses. They could automatically switch between legacy support and BIP322 support depending on address type or BIP322 could get new RPCs. (Something to consider would be amending BIP322 so that P2PKH addresses in BIP322 are special cased to use the old signmessage format; that way this PR can be made fully BIP322 compliant and yet backward compatible.)

[kallewoof]

I see what you're saying.

I think adding an optional 'format' which defaults to 'bip322' but can be set to 'legacy' for 'sign', and autodetection in 'verify' should be sufficient. What do you think?

[harding]

@kallewoof

I think adding an optional 'format' which defaults to 'bip322' but can be set to 'legacy' for 'sign', and autodetection in 'verify' should be sufficient. What do you think?

Sounds reasonable to me. I'd slightly prefer the default be legacy if the user provides a P2PKH address---but not enough to argue about it. Thanks!

[sipa]

Thanks for picking this up again.

I don't think we should gratuitously break compatibility with the existing signature scheme; that's generally not the approach we take with RPCs, and given that the scheme is implemented in other software too probably even more disruptive.

I wouldn't be opposed to adding an exception in BIP322 that for P2PKH address the existing key recovery based scheme is used; in general I like avoiding ambiguity in these things. However I also see this may add additional complication to the spec, and perhaps not be very clear cut either (why just the P2PKH ones, and not the extensions to other single-key address types people have proposed and adopted in other software, for example?).

[kallewoof]

@harding @sipa Defaulting to legacy for P2PKH means we can basically never get rid of the legacy code. That's okay, but unfortunate, IMO.

[sipa]

I didn't expect we'd ever be able to get rid of the old format, even if the default would be a new format now. That'd indeed unfortunate, but that's life when you want compatibility. I don't think it's a big problem; it's not that much code.

[kallewoof]

@harding @sipa I have updated the specification to include the special case, and I also added a description of the legacy format: https://github.com/bitcoin/bips/blob/e24e86b482e394e18803f14c7e2338aab9b7e1e2/bip-0322.mediawiki (from bitcoin/bips#808)

Let me know if I got something wrong there. I wrote the legacy format spec based on the code.

[kallewoof]

I updated the code in this PR to do what was suggested (use legacy if p2pkh, otherwise use bip322), and I also updated bip322 to require this.

[harding]

Tested signmessage and verifymessage both ways between 0.18 and this branch, looks good, thanks! You may want to consider making signmessagewithprivkey default to using P2PKH for backwards compatibility and add a third argument to that RPC that allows selecting P2WPKH (I'd suggest the argument select between P2PKH and P2WPKH (rather than legacy vs bip322) so that it can be extended to take a P2TR argument or other address format argument in the future).

I skimmed this commit with the proposed changes to BIP322 and left a couple nits. As a specification, I think it could also use a description of CKey::SignCompact() and CKey::RecoverCompact() such as how they serialize which derived pubkey to use and whether or not it's compressed or uncompressed. An alternative to putting this information in BIP322 would be to create a separate BIP for the established legacy signmessage format (which was created before we had BIPs) and then simply have BIP322 reference that BIP. I can help with documentation if you'd like.

[kallewoof]

@harding

FYI the bitcoin/bips PR was merged as you were reviewing it; I opened a fix-up here: bitcoin/bips#814

Thanks for testing!

You may want to consider making signmessagewithprivkey default to using P2PKH for backwards compatibility and add a third argument to that RPC that allows selecting P2WPKH (I'd suggest the argument select between P2PKH and P2WPKH (rather than legacy vs bip322) so that it can be extended to take a P2TR argument or other address format argument in the future).

Sure, since we are not getting rid of the legacy format, we might as well use it where it is usable.

Edit: This has been updated. It uses the output type parsing from other places to allow legacy, p2sh-segwit, or bech32. I assume once taproot and such appear, they will be added to this and this should "just work [tm]".

[sipa]

You may want to check with @achow101 that this doesn't conflict with the descriptor wallet changes.

[achow101]

This doesn't interfere with the descriptor wallet changes since a SigningProvider was never actually needed in the GUI.

[sipa]

But there won't be a singular signing provider for the whole wallet anymore, right?

[ryanofsky]

FWIW, I think it would be better to add an interfaces::Wallet::signMessage() method instead of adding an interfaces::Wallet::getSigningProvider() method. Few reasons:

1. It would allow deduplicating more code for signing messages instead of having it written twice in qt and rpcwallet.

2. It would create a little less work for me with #10102, so I don't have to an add IPC wrapper around SigningProvider, or serialize SigningProvider with private key information and send it from the wallet process to the qt process.

3. Just in general I think rpcwallet.cpp is too monolothic and has too much important code mixed with UniValue boilerplate. So I'd love to see something like src/wallet/sign.h / src/wallet/sign.cpp files with a bool SignMessage(CWallet& wallet, message, options...) function and start organizing things better.

Anyway, I'm fine with this approach, but wanted to suggest an alternative.

EDIT: Changed Wallet& to CWallet& above (sorry for the confusing typo)

[kallewoof]

@ryanofsky That sounds good to me! I am not sure how to access the interfaces::Wallet instance from rpcwallet.cpp though. It seems like it only has access to CWallet (while the QT side only has access to interfaces::Wallet).

Is the idea to add two signmessage functions, where the interface one simply calls down into m_wallet->SignMessage(..)?

[ryanofsky]

@ryanofsky That sounds good to me! I am not sure how to access the interfaces::Wallet instance from rpcwallet.cpp though. It seems like it only has access to CWallet (while the QT side only has access to interfaces::Wallet).

rpcwallet.cpp shouldn't need to access interfaces::Wallet here (and in general) because interfaces::Wallet doesn't contain any real functionality, it's just a high level wrapper around selected wallet functions that GUI code uses to indirectly control wallets.

So the idea is just to implement a standalone function in wallet code:

bool SignMessage(CWallet& wallet, message, options...)

and to call this function both from wallet/rpcwallet.cpp and interfaces/wallet.cpp (just adding a one-line WalletImpl::signMessage() wrapper method in the interface code).

Is the idea to add two signmessage functions, where the interface one simply calls down into m_wallet->SignMessage(..)?

Sorry! Just realized I had a typo in #16440 (comment), which is fixed now but was probably pretty confusing. I was trying to suggest not having a CWallet::SignMesage(...) method, but instead defining a standalone SignMessage(CWallet&, ...) function. Really either a method or a standalone function would be fine, but the function seemed better because we're gradually breaking CWallet up and pulling functionality out of it, so adding new functionality there without a reason seemed to be moving in wrong direction.

[kallewoof]

@ryanofsky I ended up adding a couple of helper functions to proof.h and adding a proof.cpp file. The P2PKH-vs-others logic is now all contained inside the proof code, and is called from rpcwallet/misc/interfaces/wallet:

bitcoin/src/script/proof.h

Lines 237 to 250 in 6e9947e

	/**
	* Attempt to sign a message with the given destination.
	*/
	void SignMessageWithSigningProvider(SigningProvider* sp, const std::string& message, const CTxDestination& destination, std::vector<uint8_t>& signature_out);
	/**
	* Attempt to sign a message with the given private key and address format.
	*/
	void SignMessageWithPrivateKey(CKey& key, OutputType address_type, const std::string& message, std::vector<uint8_t>& signature_out);
	/**
	* Determine if a signature is valid for the given message.
	*/
	Result VerifySignature(const std::string& message, const CTxDestination& destination, const std::vector<uint8_t>& signature);

The interface side ends up like

bitcoin/src/interfaces/wallet.cpp

Lines 460 to 463 in 6e9947e

	void signMessage(const std::string& message, const CTxDestination& destination, std::vector<uint8_t>& signature_out) override
	{
	proof::SignMessageWithSigningProvider(m_wallet.get(), message, destination, signature_out);
	}

[kallewoof]

As a side result of this, strMessageMagic was moved out of validation.h/cpp into proof.cpp, and is no longer public, which is nice.

[ryanofsky]

Looks good. The new SignMessageWithSigningProvider makes for fewer wallet changes than what I suggested, which is great!

[NicolasDorier]

Concept ACK.
Will try to implement it in NBitcoin.

[kallewoof]

@fanquake I think the "Needs Conceptual Review" flag can be removed. Got Conceptual ACKs from @laanwj, @harding, @practicalswift, and @NicolasDorier (and no Concept NACKs from anyone).

[kallewoof]

FTR, I updated the BIP to fix verification which was a bit broken. I have also updated the code here. New BIP can be seen here: https://github.com/bitcoin/bips/blob/499a2eaf8943c1fbfef2dd9a0ef2201d70352fbb/bip-0322.mediawiki

[luke-jr]

Abstract, tentative concept NACK:

I'm not convinced BIP322 is itself a good idea as-is. Its multiple-proofs concept seems ripe to encourage misuse of signatures as a false "proof" of spend-ability rather than simply proving a receiver (as current signatures do).

If BIP322 is intended for proving spend-ability, then it confuses the matter of addresses (which don't spend), and also conflicts with the historical use case of signmessage and verifymessage which is never for spend-ability purposes.

[kallewoof]

@luke-jr

I'm not convinced BIP322 is itself a good idea as-is. Its multiple-proofs concept seems ripe to encourage misuse of signatures as a false "proof" of spend-ability rather than simply proving a receiver (as current signatures do).

The argument for multiple proofs is that it's not trivial to verify that many independent proofs are not overlapping. This becomes more serious if/when a "proof of funds" purpose is added, where I may convince you that I have more money than I actually do. Even now, though, I can make you think I have the private keys to more pubkeys than I do (and whatever locked funds associated with those).

The BIP explicitly describes how multiple proofs should be handled (e.g. immediately abort if the same proof appears twice), but if there are attack vectors I didn't think of, I'd of course like to address them.

If BIP322 is intended for proving spend-ability, then it confuses the matter of addresses (which don't spend), and also conflicts with the historical use case of signmessage and verifymessage which is never for spend-ability purposes.

BIP322 is intended to be a general purpose "prove/verify" system for script-style "stuff" in Bitcoin.

• The signmessage purpose is used to prove that you are able to create a valid scriptSig for a given scriptPubKey.
• A theoretical proof of funds purpose would be used to prove that you can spend a given set of transaction outputs, but this is beyond the scope of both BIP322 and this PR.

If there is a way to clarify the above, I'm all ears.

[luke-jr]

@kallewoof Existing signatures do not prove you have access to any private keys nor funds, only that you are the recipient to future transactions paying an address (and therefore the right person to be signing contractual terms for such payments). Because it proves nothing about existing funds or keys, there is no such thing as "overlapping" proofs to worry about.

If you intend to cover more proof cases than that, you probably need a new RPC so it can't be confused with the purpose of the existing ones.

[kallewoof]

@luke-jr Yes I agree that prove funds may need a separate RPC. I chose to exclude that from the BIP because there are things that should be resolved before including it, but it should be easy to add as an extension to BIP-322. This is why it inherently supports multiple proofs, even if as you mention this is not an issue with address proving.