BIP-322: Generic signed message format

[kallewoof]

This PR implements BIP-322, for the single proof (no multiple addresses simultaneously) single signer (no multisig addresses) case.

UI (CLI/QT) are restricted to the single proof case, but the underlying code (script/proof.h) supports multiple proofs.

Recommend ?w=1 / -w to avoid whitespace spam.

There is a related PR #16653 that includes the sign/verify components of this and the signet PR (#16411).

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #17977 ([WIP] Implement BIP 340-342 validation (Schnorr/taproot/tapscript) by sipa)
• #17953 (refactor: Abstract boost::variant out by elichai)
• #17938 (Disallow automatic conversion between disparate hash types by Empact)
• #17809 (rpc: Auto-format RPCResult by MarcoFalke)
• #17577 (refactor: deduplicate the message sign/verify code by vasild)
• #17399 (validation: Templatize ValidationState instead of subclassing by jkczyz)
• #17261 (Make ScriptPubKeyMan an actual interface and the wallet to have multiple by achow101)
• #16653 (script: add simple signature support (checker/creator) by kallewoof)
• #16528 ([WIP] Native Descriptor Wallets (take 2) by achow101)
• #16365 (Log RPC parameters (arguments) if -debug=rpcparams by LarryRuane)
• #13062 (Make script interpreter independent from storage type CScript by sipa)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

[laanwj]

Concept ACK

[harding]

I really like the idea of BIP322, but I don't like that this changes existing RPCs in way that's not backwards compatible (e.g. a signmessage created and verified on 0.18 produces a "CDataStream::read(): end of data: iostream error" here). Similarly, a P2PKH signmessage created on this branch results in false when checked with 0.18. Either P2PKH signing/verifying should be special cased to use the legacy signmessage format, or BIP322 message signing should use different RPCs (or at least a special RPC parameter or configuration option to indicate the user wants to use BIP322).

Just on this branch, I also created a 2-of-3 P2WSH address using addmultisigaddress and then attempted to signmessage with it. I received the error, ScriptPubKey does not refer to a key. Unless I'm misunderstanding BIP322, this should be possible and it would be cool if the wallet knew how to do it (however, even if the wallet doesn't know how to do it, as long as it's possible by BIP322, the error message should be changed). I didn't test any more advanced scripts, although I think that would be worthwhile to test later---in theory, the wallet should be able to signmessage for any script where its solver could sign for a spend. I think importing some advanced scripts using descriptors and then signmessage'ing for them might make a good addition to the integration tests in this PR.

Another concern is that I don't believe the discussion over how BIP322 should handle timelocking opcodes (CLTV and CSV) was ever resolved. I think that's probably important before this could be released.

Finally, I skimmed the code and left one documentation-related note. Thanks for working on this!

[harding]

I think "the derived BECH32 address" will become ambiguous if subsequent versions of segwit are adopted. E.g. if taproot is adopted, there would be two possible bech32 addresses for the same public key. Perhaps this should say, "the derived P2WPKH bech32 address".

Nit: BECH is not an acronym AFAIK and so should not be all caps. A quick git grep shows that the convention in the repository is to treat it as a common noun, so all lowercase unless at the start of a sentence.

[kallewoof]

@harding

I really like the idea of BIP322, but I don't like that this changes existing RPCs in way that's not backwards compatible (e.g. a signmessage created and verified on 0.18 produces a "CDataStream::read(): end of data: iostream error" here). Similarly, a P2PKH signmessage created on this branch results in false when checked with 0.18. Either P2PKH signing/verifying should be special cased to use the legacy signmessage format, or BIP322 message signing should use different RPCs (or at least a special RPC parameter or configuration option to indicate the user wants to use BIP322).

I don't really agree. Proving that you own an address is something that you do "now", not something you do and keep the proof around indefinitely. Losing an old proof should be perfectly fine, as the point is lost if you can't reproduce it.

Of course, down the road Craig could claim that his broken proof actually worked, "but that core changed the mechanism so it can't be verified anymore" but I don't know if anybody cares about him enough to warrant keeping both kinds of verifiers (we probably do not want to keep the signmessage part).

Just on this branch, I also created a 2-of-3 P2WSH address using addmultisigaddress and then attempted to signmessage with it. I received the error, ScriptPubKey does not refer to a key. Unless I'm misunderstanding BIP322, this should be possible and it would be cool if the wallet knew how to do it (however, even if the wallet doesn't know how to do it, as long as it's possible by BIP322, the error message should be changed). I didn't test any more advanced scripts, although I think that would be worthwhile to test later---in theory, the wallet should be able to signmessage for any script where its solver could sign for a spend. I think importing some advanced scripts using descriptors and then signmessage'ing for them might make a good addition to the integration tests in this PR.

Yes, BIP322 literally just says "the sighash (normally based on the transaction) is X; now sign/verify with this address" and uses exactly the same code that is used by the wallet to sign transactions, so anything goes. I'm actually confused why your example doesn't work out of the box. I'll look into that.

Another concern is that I don't believe the discussion over how BIP322 should handle timelocking opcodes (CLTV and CSV) was ever resolved. I think that's probably important before this could be released.

Actually, those should just be removed; they were meant for the proof of funds, which I am removing from BIP322. Thanks for the nudge.

Finally, I skimmed the code and left one documentation-related note. Thanks for working on this!

Great! Thanks, will address.

[harding]

@kallewoof

I don't like that this changes existing RPCs in way that's not backwards compatible [...]

I don't really agree. Proving that you own an address is something that you do "now", not something you do and keep the proof around indefinitely. [...]

I think there are several points here:

1. This change breaks backwards compatibility between different versions of Bitcoin Core. I think it's unfortunate if someone running version x can't create or verify proofs when talking to someone running version y. Sure, it'd be nice if everyone upgraded to the latest version, but that takes time (sometimes years in the case of cold wallets).

2. This change breaks compatibility with almost every other wallet that has implemented signmessage support. Again, it'd be great if they all have BIP322 support ready at the same time this code gets into a Bitcoin Core release, but that seems extremely unlikely to me---I expect it to take years to get widespread BIP322 support. Then each of those wallets has their own user upgrade cycle which may take years, so we're talking years + years until everyone is using BIP322 (if it is generally adopted at all). Until near-universal adoption happens, it'd be nice if Bitcion Core is compatible with other wallets.

3. As you mention, this breaks old signmessages. While I don't feel strongly about that and I'm at least somewhat sympathetic with your arguments about people just generating new proofs, I think unnecessary breakage is something we should try to avoid.

I think the appropriate solution to all three issues is that these existing RPCs should continue to generate and verify legacy P2PKH signmessages when called with P2PKH addresses. They could automatically switch between legacy support and BIP322 support depending on address type or BIP322 could get new RPCs. (Something to consider would be amending BIP322 so that P2PKH addresses in BIP322 are special cased to use the old signmessage format; that way this PR can be made fully BIP322 compliant and yet backward compatible.)

[kallewoof]

I see what you're saying.

I think adding an optional 'format' which defaults to 'bip322' but can be set to 'legacy' for 'sign', and autodetection in 'verify' should be sufficient. What do you think?

[harding]

@kallewoof

I think adding an optional 'format' which defaults to 'bip322' but can be set to 'legacy' for 'sign', and autodetection in 'verify' should be sufficient. What do you think?

Sounds reasonable to me. I'd slightly prefer the default be legacy if the user provides a P2PKH address---but not enough to argue about it. Thanks!

[sipa]

Thanks for picking this up again.

I don't think we should gratuitously break compatibility with the existing signature scheme; that's generally not the approach we take with RPCs, and given that the scheme is implemented in other software too probably even more disruptive.

I wouldn't be opposed to adding an exception in BIP322 that for P2PKH address the existing key recovery based scheme is used; in general I like avoiding ambiguity in these things. However I also see this may add additional complication to the spec, and perhaps not be very clear cut either (why just the P2PKH ones, and not the extensions to other single-key address types people have proposed and adopted in other software, for example?).

[kallewoof]

@harding @sipa Defaulting to legacy for P2PKH means we can basically never get rid of the legacy code. That's okay, but unfortunate, IMO.

[sipa]

I didn't expect we'd ever be able to get rid of the old format, even if the default would be a new format now. That'd indeed unfortunate, but that's life when you want compatibility. I don't think it's a big problem; it's not that much code.

[kallewoof]

@harding @sipa I have updated the specification to include the special case, and I also added a description of the legacy format: https://github.com/bitcoin/bips/blob/e24e86b482e394e18803f14c7e2338aab9b7e1e2/bip-0322.mediawiki (from bitcoin/bips#808)

Let me know if I got something wrong there. I wrote the legacy format spec based on the code.

[kallewoof]

I updated the code in this PR to do what was suggested (use legacy if p2pkh, otherwise use bip322), and I also updated bip322 to require this.

[harding]

Tested signmessage and verifymessage both ways between 0.18 and this branch, looks good, thanks! You may want to consider making signmessagewithprivkey default to using P2PKH for backwards compatibility and add a third argument to that RPC that allows selecting P2WPKH (I'd suggest the argument select between P2PKH and P2WPKH (rather than legacy vs bip322) so that it can be extended to take a P2TR argument or other address format argument in the future).

I skimmed this commit with the proposed changes to BIP322 and left a couple nits. As a specification, I think it could also use a description of CKey::SignCompact() and CKey::RecoverCompact() such as how they serialize which derived pubkey to use and whether or not it's compressed or uncompressed. An alternative to putting this information in BIP322 would be to create a separate BIP for the established legacy signmessage format (which was created before we had BIPs) and then simply have BIP322 reference that BIP. I can help with documentation if you'd like.

[kallewoof]

@harding

FYI the bitcoin/bips PR was merged as you were reviewing it; I opened a fix-up here: bitcoin/bips#814

Thanks for testing!

You may want to consider making signmessagewithprivkey default to using P2PKH for backwards compatibility and add a third argument to that RPC that allows selecting P2WPKH (I'd suggest the argument select between P2PKH and P2WPKH (rather than legacy vs bip322) so that it can be extended to take a P2TR argument or other address format argument in the future).

Sure, since we are not getting rid of the legacy format, we might as well use it where it is usable.

Edit: This has been updated. It uses the output type parsing from other places to allow legacy, p2sh-segwit, or bech32. I assume once taproot and such appear, they will be added to this and this should "just work [tm]".

[sipa]

You may want to check with @achow101 that this doesn't conflict with the descriptor wallet changes.

[achow101]

This doesn't interfere with the descriptor wallet changes since a SigningProvider was never actually needed in the GUI.

[sipa]

But there won't be a singular signing provider for the whole wallet anymore, right?

[ryanofsky]

FWIW, I think it would be better to add an interfaces::Wallet::signMessage() method instead of adding an interfaces::Wallet::getSigningProvider() method. Few reasons:

1. It would allow deduplicating more code for signing messages instead of having it written twice in qt and rpcwallet.

2. It would create a little less work for me with #10102, so I don't have to an add IPC wrapper around SigningProvider, or serialize SigningProvider with private key information and send it from the wallet process to the qt process.

3. Just in general I think rpcwallet.cpp is too monolothic and has too much important code mixed with UniValue boilerplate. So I'd love to see something like src/wallet/sign.h / src/wallet/sign.cpp files with a bool SignMessage(CWallet& wallet, message, options...) function and start organizing things better.

Anyway, I'm fine with this approach, but wanted to suggest an alternative.

EDIT: Changed Wallet& to CWallet& above (sorry for the confusing typo)

[kallewoof]

@ryanofsky That sounds good to me! I am not sure how to access the interfaces::Wallet instance from rpcwallet.cpp though. It seems like it only has access to CWallet (while the QT side only has access to interfaces::Wallet).

Is the idea to add two signmessage functions, where the interface one simply calls down into m_wallet->SignMessage(..)?

[ryanofsky]

@ryanofsky That sounds good to me! I am not sure how to access the interfaces::Wallet instance from rpcwallet.cpp though. It seems like it only has access to CWallet (while the QT side only has access to interfaces::Wallet).

rpcwallet.cpp shouldn't need to access interfaces::Wallet here (and in general) because interfaces::Wallet doesn't contain any real functionality, it's just a high level wrapper around selected wallet functions that GUI code uses to indirectly control wallets.

So the idea is just to implement a standalone function in wallet code:

bool SignMessage(CWallet& wallet, message, options...)

and to call this function both from wallet/rpcwallet.cpp and interfaces/wallet.cpp (just adding a one-line WalletImpl::signMessage() wrapper method in the interface code).

Is the idea to add two signmessage functions, where the interface one simply calls down into m_wallet->SignMessage(..)?

Sorry! Just realized I had a typo in #16440 (comment), which is fixed now but was probably pretty confusing. I was trying to suggest not having a CWallet::SignMesage(...) method, but instead defining a standalone SignMessage(CWallet&, ...) function. Really either a method or a standalone function would be fine, but the function seemed better because we're gradually breaking CWallet up and pulling functionality out of it, so adding new functionality there without a reason seemed to be moving in wrong direction.

[kallewoof]

@ryanofsky I ended up adding a couple of helper functions to proof.h and adding a proof.cpp file. The P2PKH-vs-others logic is now all contained inside the proof code, and is called from rpcwallet/misc/interfaces/wallet:

bitcoin/src/script/proof.h

Lines 237 to 250 in 6e9947e

	/**
	* Attempt to sign a message with the given destination.
	*/
	void SignMessageWithSigningProvider(SigningProvider* sp, const std::string& message, const CTxDestination& destination, std::vector<uint8_t>& signature_out);
	/**
	* Attempt to sign a message with the given private key and address format.
	*/
	void SignMessageWithPrivateKey(CKey& key, OutputType address_type, const std::string& message, std::vector<uint8_t>& signature_out);
	/**
	* Determine if a signature is valid for the given message.
	*/
	Result VerifySignature(const std::string& message, const CTxDestination& destination, const std::vector<uint8_t>& signature);

The interface side ends up like

bitcoin/src/interfaces/wallet.cpp

Lines 460 to 463 in 6e9947e

	void signMessage(const std::string& message, const CTxDestination& destination, std::vector<uint8_t>& signature_out) override
	{
	proof::SignMessageWithSigningProvider(m_wallet.get(), message, destination, signature_out);
	}

[kallewoof]

As a side result of this, strMessageMagic was moved out of validation.h/cpp into proof.cpp, and is no longer public, which is nice.

[ryanofsky]

Looks good. The new SignMessageWithSigningProvider makes for fewer wallet changes than what I suggested, which is great!

[luke-jr]

A good idea someone had: refuse to validate a signed message if the key backing it has any outstanding UTXOs. This does need some care to ensure people can validate old signed messages, but hopefully that's not a fatal issue.

[practicalswift]

Nit: Could use enum class?

[practicalswift]

Nit: r < -4 || r > 0 is known to be false at compile-time and the "???" path is thus unreachable. Perhaps use switch instead?

[kallewoof]

I'm not sure what you mean. No guarantees that r is actually a valid Result enum value.

#include <cstdio>

enum class X {
	one = 1,
	two = 2,
};

int main(int argc, char** argv)
{
	X z = (X)3;
	printf("z = %d\n", z); // outputs 3
}

[practicalswift]

challenge and message not used?

[practicalswift]

challenge and message not used?

[practicalswift]

entries not used?

[practicalswift]

Is m_challenge.size() <= 255 guaranteed here (m_entries is uint8_t)? If so, perhaps assert(...) that to make the tacit assumption explicit? :-)

[kallewoof]

Good point. Adding assert.

[practicalswift]

Nit: Don't forget to terminate namespace using // namespace proof :-)

[real-or-random]

Would you mind opening a PR to https://github.com/bitcoin/bips to make the BIP clearer?

I'll be gone for two weeks but I'll be able to open a PR afterwards. I think the main points are

• "signed message" vs "signature" vs "proof"
• explaining that the verification is always done with respect to an address
• and mentioning this:

If I read the code correctly, the basic idea is

* checking that a provided script matches the address and

* verifying the script but replacing every signature verification operation by a signature verification operation on the provided message instead of the transaction.

---

Edit: to answer your question, if you get an inconclusive proof from someone, it means your node is probably old. If it isn't old, the prover is probably using broken software to generate said proof. This feature is here to deal with the case where new consensus rules were activated (e.g. Segwit Script Version 1). Old nodes would accept those in blocks (not INVALID) but would not be able to check them beyond that (thus INCONCLUSIVE).

Yep, I see the point, and it makes a lot of sense. And yep, I don't have a better idea either. Maybe we can find a different word at least? "Inconclusive" does not say much (in fact that's the original of this word). For example INVALID_TRY_UPGRADING_YOUR_NODE is not beautiful either but has some more meaning. I'm sure there are better suggestions. Also the docs should say that upgrading can help.

[kallewoof]

@practicalswift Thanks for the feedback. I believe I addressed everything you pointed out.

[kallewoof]

@real-or-random I am working through your comments and will try to make a new pull request to reflect your ideas shortly. Sorry for the delay.

[gmaxwell]

I wouldn't be opposed to adding an exception in BIP322 that for P2PKH address the existing key recovery based scheme is used;

I think it would be nicer if it were just a separate RPC so that the old format (esp verifying) could be eventually sunset.

[gmaxwell]

On a high level, it seems to me that the purpose of the BIP is not entirely clear. At the moment, it is only for signing messages but includes the entire scripting system. [...] Does the BIP implicitly assume that a scriptPubKey contains exactly one pubkey?

I think this isn't the best way to think about this facility. Bitcoin transactions are not signed with ECDSA. They are signed with a digital signature system called "Bitcoin Script". Bitcoin Script happens to have ECDSA embedded in it-- like ECDSA has field multiplication embedded in it. A bitcoin message signing thing can be seen as just a sign-stuff-using-script.

Unfortunately this is complicated by the fact that script itself is expected to be embedded in a consensus system, so stuff like softforks are awkward to handle.

Why would someone want to sign messages using script? In practice usually for dumb reasons for proving control or tying metadata to addresses that used to control funds, currently currently control funds, or will control funds in the future. For these latter three things, nothing less than Bitcoin Script would work-- no amount of "sniff scripts yank keys out" would actually get the right behaviour.

For the "currently control funds" case it would be very useful if the system actually can check the existence of the funds (e.g. specifying txouts and checking them). For the "previously controlled funds" it would be very useful if the system could check that (SPV proofs for the outputs? kinda big :( ).

But for the "will control funds in the future" case there is basically no special behaviour needed/possible except signing with the same digital signature system which will eventually be used.

So I think this would mostly be useful for this last case. But that last case might better by serve by dry-signing an actual transaction with a dummy input/locktime, since doing that actually verifies your ability to sign for transaction like things.

For CSV, CLTV... probably the right way to do that is have them go in as non-locked (max sequence) by default but allow specifying other values for these fields as some kind of argument tagged onto the end of the address.

[kallewoof]

Apologies about lack of activity. My work pulled me away for a couple of months. I'm getting back to things now, though. ><

[real-or-random]

On a high level, it seems to me that the purpose of the BIP is not entirely clear. At the moment, it is only for signing messages but includes the entire scripting system. [...] Does the BIP implicitly assume that a scriptPubKey contains exactly one pubkey?

I think this isn't the best way to think about this facility. Bitcoin transactions are not signed with ECDSA. They are signed with a digital signature system called "Bitcoin Script". Bitcoin Script happens to have ECDSA embedded in it-- like ECDSA has field multiplication embedded in it. A bitcoin message signing thing can be seen as just a sign-stuff-using-script.

To be clear, I agree with that and I think what you describe is indeed the right the way to think about it. The question you quoted was at a time when I really didn't understand what was going on in the BIP.

The one point where my crypto heart hurts is when you call this a signature scheme. That's just because I have a different defintion in mind. Let's say script is a scheme for authorizing transactions.

But this is one of the open points here: We need to think about the terminology:
I think "signed message" is misleading given how expressive scripts are: If you know about cryptography, you think of a signature scheme. But this really breaks down when it comes to, e.g., anyonecanspend or hash preimages, where ECDSA is not involved. If you have no idea of cryptographic definitions, a "signed message" should still be something that only someone with a secret can produce and that cannot by modified by someone without the secret. Again, this intuition breaks down with unusual scripts.

Proof of "control"? This comes with a sound of "I'm the only one in control" and is again strange for anyonecanspend.

Maybe something that means "proof that you can fulfill the script" but with nicer words?

Sorry, I'm not really constructive here but I don't have a better suggestion at the moment.

Apologies about lack of activity. My work pulled me away for a couple of months. I'm getting back to things now, though. ><

No worries. 👍