New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rpc, wallet: fix incorrect segwit redeem script size limit #28307
base: master
Are you sure you want to change the base?
rpc, wallet: fix incorrect segwit redeem script size limit #28307
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
76a83e7
to
db38b58
Compare
concept ACK db38b58 Confirmed this branch fixes the issue mentioned in #28250 Code review to follow...
|
cc056d3
to
4c42ab5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated per feedback. Thanks @pinheadmz. Small diff .
Added coverage for createmultisig
+20 keys error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK
4c42ab5
to
a0ebb92
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated per feedback. Thanks theStack.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ack a0ebb92
Testing Methodology:
- Make, successful
- All functional tests, successful
- Executed
createmultisig
command with the following output - don't seeUnable to make chosen address type, please ensure no uncompressed public keys are present.
warning anymore, nor do I seeERROR: FillableSigningProvider::AddCScript(): redeemScripts > 520 bytes are invalid
message in the logs.
bcli createmultisig 16 "[\"02f293c4d026a536a082453f8327f305ae0a0c59350b396205e15fbec2af67390b\",\"03d7b6a7e972f6e1a532f9f49a7cc35055a0ffd7cec5f81728f6a3fdf4dad93672\",\"021f360d51b8eb43422fe942c837ad16e36bab87c5f7567609de9a38e205fb51ed\",\"03a778eaee9cc18a4e5e02a78f403082d4e6f3a9c0d9edaea20b9ad86089de7fc9\",\"02a3c95e41e8272c6842df8c9da492ef5e03bbc4ca17c22ce1f88570e716e187db\",\"032e1d465bc5cdea674337934fb861d097dd7ab20289e4d41de14a6cafcd61d30a\",\"0370fd35da88778a805ce9acc88c0020e3b14cacaa30f99d829207e1896d782ea1\",\"03878a4b831af4fee069964448ccca999071633fcc751d296e8c5325167274febf\",\"0248f8621596029568070dda40bc1e3ace78df320b5ebf0d6a431536e9af3ec894\",\"03f757260835b7e439b244a9b10f5645966d7dcff9d2648c400a0d6827f4a64ac4\",\"023b3972554e123985b6c3d8575e6fd3384f59a58ef91e0d39ba5905dad49e7372\",\"02e32652928f21764e02822d39ffd1075c428e372a06fe7247023d5b60f1d01596\",\"020e002818c48b1d1766f6196e09db90f51e155bbd49be0ad743d1fa6b7e8efcae\",\"03725d482f0c77f4feb1ecbea28f036b30889f462222002e88aa8f8346dce94474\",\"031fa2798d9f40bc2746f91f300460be9091e10f1b4baa37843cd41c965b0f9b41\",\"0235b7b545398ce9692f319e6eefe114eae50f008f46d33baf19418e0f17e3bc65\",\"02ff72a2b54370c0b1501228b6a210bc36a16535cc358bf7839c85a6ae2c41725c\",\"0282ce44556f5a81c7a698dc8fdba12089740df420af93002e80a752020e5d939e\",\"0279f5b244862e27f69284befb0299909139b899d2702b3e74b4317ed5b2947e03\",\"0293d0b090906afdee07dabedf34798293246ce06ade2e3887302f289008f25f4d\"]" "p2sh-segwit"
{
"address": "2N9mdDwZmrKZNBkX61qidLnPZqkVW8gx3KR",
"redeemScript": "602102f293c4d026a536a082453f8327f305ae0a0c59350b396205e15fbec2af67390b2103d7b6a7e972f6e1a532f9f49a7cc35055a0ffd7cec5f81728f6a3fdf4dad9367221021f360d51b8eb43422fe942c837ad16e36bab87c5f7567609de9a38e205fb51ed2103a778eaee9cc18a4e5e02a78f403082d4e6f3a9c0d9edaea20b9ad86089de7fc92102a3c95e41e8272c6842df8c9da492ef5e03bbc4ca17c22ce1f88570e716e187db21032e1d465bc5cdea674337934fb861d097dd7ab20289e4d41de14a6cafcd61d30a210370fd35da88778a805ce9acc88c0020e3b14cacaa30f99d829207e1896d782ea12103878a4b831af4fee069964448ccca999071633fcc751d296e8c5325167274febf210248f8621596029568070dda40bc1e3ace78df320b5ebf0d6a431536e9af3ec8942103f757260835b7e439b244a9b10f5645966d7dcff9d2648c400a0d6827f4a64ac421023b3972554e123985b6c3d8575e6fd3384f59a58ef91e0d39ba5905dad49e73722102e32652928f21764e02822d39ffd1075c428e372a06fe7247023d5b60f1d0159621020e002818c48b1d1766f6196e09db90f51e155bbd49be0ad743d1fa6b7e8efcae2103725d482f0c77f4feb1ecbea28f036b30889f462222002e88aa8f8346dce9447421031fa2798d9f40bc2746f91f300460be9091e10f1b4baa37843cd41c965b0f9b41210235b7b545398ce9692f319e6eefe114eae50f008f46d33baf19418e0f17e3bc652102ff72a2b54370c0b1501228b6a210bc36a16535cc358bf7839c85a6ae2c41725c210282ce44556f5a81c7a698dc8fdba12089740df420af93002e80a752020e5d939e210279f5b244862e27f69284befb0299909139b899d2702b3e74b4317ed5b2947e03210293d0b090906afdee07dabedf34798293246ce06ade2e3887302f289008f25f4d0114ae",
"descriptor": "sh(wsh(multi(16,02f293c4d026a536a082453f8327f305ae0a0c59350b396205e15fbec2af67390b,03d7b6a7e972f6e1a532f9f49a7cc35055a0ffd7cec5f81728f6a3fdf4dad93672,021f360d51b8eb43422fe942c837ad16e36bab87c5f7567609de9a38e205fb51ed,03a778eaee9cc18a4e5e02a78f403082d4e6f3a9c0d9edaea20b9ad86089de7fc9,02a3c95e41e8272c6842df8c9da492ef5e03bbc4ca17c22ce1f88570e716e187db,032e1d465bc5cdea674337934fb861d097dd7ab20289e4d41de14a6cafcd61d30a,0370fd35da88778a805ce9acc88c0020e3b14cacaa30f99d829207e1896d782ea1,03878a4b831af4fee069964448ccca999071633fcc751d296e8c5325167274febf,0248f8621596029568070dda40bc1e3ace78df320b5ebf0d6a431536e9af3ec894,03f757260835b7e439b244a9b10f5645966d7dcff9d2648c400a0d6827f4a64ac4,023b3972554e123985b6c3d8575e6fd3384f59a58ef91e0d39ba5905dad49e7372,02e32652928f21764e02822d39ffd1075c428e372a06fe7247023d5b60f1d01596,020e002818c48b1d1766f6196e09db90f51e155bbd49be0ad743d1fa6b7e8efcae,03725d482f0c77f4feb1ecbea28f036b30889f462222002e88aa8f8346dce94474,031fa2798d9f40bc2746f91f300460be9091e10f1b4baa37843cd41c965b0f9b41,0235b7b545398ce9692f319e6eefe114eae50f008f46d33baf19418e0f17e3bc65,02ff72a2b54370c0b1501228b6a210bc36a16535cc358bf7839c85a6ae2c41725c,0282ce44556f5a81c7a698dc8fdba12089740df420af93002e80a752020e5d939e,0279f5b244862e27f69284befb0299909139b899d2702b3e74b4317ed5b2947e03,0293d0b090906afdee07dabedf34798293246ce06ade2e3887302f289008f25f4d)))#xfjxuvgn"
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK a0ebb92
Diff since last ACK is minimal, just nits in tests. Since it's been a while, I re-reviewed the whole PR anyway. Most of the changes are cleanup refactors in the relevant tests which I agree are worthwhile. The main bugfix is switching from FillableSigningProvider
to FlatSigningProvider
when adding a CScript inAddAndGetDestinationForScript()
, because the latter does not require that redeemScript.size() <= MAX_SCRIPT_ELEMENT_SIZE
The script element size check is preserved (appropriately only for LEGACY output types) in the calling function AddAndGetMultisigDestination()
Confirmed the test fails on master with redeemScripts > 520 bytes are invalid
(had to manually revert a few refactors to run it on master) and passes on the branch.
Also ran the createmultisig
RPC on PR branch and master to ensure the bug is fixed: https://gist.github.com/pinheadmz/856f94c1c483343f63e6f144638ac96f
I do have one concern which is that RPC signrawtransactionwithkey
is modified but I do not think it is tested directly with a legacy multisig with > 15 keys. Such a template would have to be constructed manually because our own RPC createmultisig
will not do it. Just looking at the code I think it would be possible for a user to sign an invalid legacy transaction, if they could find a way to construct one. I may try to write this test later today and will report back!
Show Signature
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
ACK a0ebb929e865903ca1cc2674e74906a806e73109
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEE5hdzzW4BBA4vG9eM5+KYS2KJyToFAmY0/dIACgkQ5+KYS2KJ
yTrKJg/+PuYRj9moqfODqMb2nXQwhsS6qMp8iLA77aNLiXGTZzOl1b3T3JqP0/6l
1sgdfrqHNMnqJ9HCkazLUcx5a5ZyK/3QjmxhgrxybmQ+m1QFr+GfHqZjehekgWP+
kgpJ9YSFL3i/lNCQZBNCnZfFynK6zb+jD/5NSPWEWIDM/C8VMGWNr6ZvOyOJZvH7
YvV4gJx3CGYT8n0KGIyDDU6PdxHqspa6pHjhmVgN0Uy/ghaiofC7IuNSznJ7R6xH
PLPRArXh9u3kO8gzPU1NJjnzrclQZnfHHbIqP5gk5uXBFXU2IcMaWt0hfYeWtOr+
Jw0nVQj5yd2RH+TpsJbrEy+hfhjQL0ZtoXcZp21/psD/GhEP/ke0kcVDza6t+WJX
qmvw6HZ0Shxi0wiGqtRTpKTfDxA5PXBcdARQljVE0eIgw+dg8neKW1vYFLZQ1bdG
9f9qsusf1QoMDzP2fomYYBYgtpCdb6q+IBigNVJ0+pmEzvrhs6L8BKau5Yvf7AEo
79BkSmT11l+1X+E+xl2rxB8j4GIMItto6DG57sGt5VlTzkxqmxvY7fPEpwsYBEOQ
uOjVcIgdyNsdvR5h5rKfu6KMCTtLurC/ny56H3dqhbAH/uuRYAESP/g1JNZ+2n2n
pk8cmwdSpRHFnlm0B5YWDOHhWFK+HAq/wTbsPvho2rWhsY88cYU=
=Aora
-----END PGP SIGNATURE-----
pinheadmz's public key is on keybase
There is no need to manually initialize the wallets within the test case. The test framework already initializes them when `_requires_wallet` is true.
Cleaning up the test in the following ways: * Generate priv-pub key pairs used for testing only once (instead of doing it 4 times). * Simplifies 'wmulti' wallet creation, load and unload process. * Removes confusing class members initialized and updated inside a nested for-loop. * Simplifies do_multisig() outpoint detection: The outpoint index information is already contained in MiniWallet's `send_to` return value dictionary as "sent_vout". Co-authored-by: Sebastian Falbesoner <sebastian.falbesoner@gmail.com>
The function exists merely to check that the node2's wallet received the transactions created during all the 'do_multisig()' calls. It was created as a standalone function because 'getbalance()' only returns something when transactions are confirmed. So, the rationale on that time was to have a method mining blocks to confirm the recently created transactions to be able to check the incoming balance. This is why we have the "moved" class field. This change removes all the hardcoded amounts and verifies node2 balance reception directly inside 'do_multisig()'.
…ressed_keys' And also, simplified the test a bit by re-using the already existing 'wallet_multi' (instead of creating a new one). Plus, removed the 'is_bdb_compiled()' calls which were there basically to check if the test has the wallet compiled or not.
Move-only commit. No behavior change.
The multisig script generation process currently fails when the user exceeds 15 keys, even when it shouldn't. The maximum number of keys allowed for segwit redeem scripts (p2sh-segwit and bech32) is 20 keys. This is because the redeem script placed in the witness is not restricted by the item size limit. The reason behind this issue is the utilization of the legacy p2sh redeem script restrictions on segwit ones. Redeem scripts longer than 520 bytes are blocked from being inserted into the keystore, which causes the signing process and the descriptor inference process to fail. This occurs because the multisig generation flow uses the same keystore as the legacy spkm (FillableSigningProvider), which contains the 520-byte limit.
…actionwithkey The process currently fails to sign redeem scripts that are longer than 520 bytes. Even when it shouldn't. The 520 bytes redeem scripts limit is a legacy p2sh rule, and not a segwit limitation. Segwit redeem scripts are not restricted by the script item size limit. The reason why this occurs, is the usage of the same keystore used by the legacy spkm. Which contains blockage for any redeem scripts longer than the script item size limit.
This exercises the bug fixed by previous commits, where we were unable to generate and sign for segwit redeem scripts (in this case multisig redeem scripts) longer than 520 bytes. and also, this adds coverage for legacy 15-15 multisig script generation and signing.
…pts over 520 bytes Due to a bug in the legacy wallet, the p2sh maximum script size limit is also imposed on 'p2sh-segwit' and 'bech32' redeem scripts. Although redeem scripts over MAX_SCRIPT_ELEMENT_SIZE bytes are technically valid for segwit output types, we don't want to enable this feature in legacy wallets for the following reasons: 1) It introduces a compatibility-breaking change requiring downgrade protection; older wallets would be unable to interact with these "new" legacy wallets. 2) Considering the ongoing deprecation of the legacy spkm, this issue adds another good reason to transition towards descriptors.
a0ebb92
to
2451a21
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sad rebase after #30024 merge.
I wrote a test for RPC This test behaves the same on master! So, that is potentially a footgun although not an easy one for a user to pull off and also would not result in any loss of funds. https://gist.github.com/pinheadmz/ca8ed75913902bd1ddf88bdd921712ee |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wrote a test for RPC signrawtransactionwithkey that covers legacy P2SH with 15 and 16 public keys. Both calls succeed even though the latter produces a scriptSig that exceeds MAX_STANDARD_SCRIPTSIG_SIZE and if it ever made it into a block, would exceed MAX_SCRIPT_ELEMENT_SIZE
The second call, the one with 16 pubkeys, doesn't entirely succeed. The test isn't checking the error field in the signrawtransactionwithkey
response, which returns an 'error': 'Push value size limit exceeded'
.
The confusing point about this command is that it returns the post-processed tx hex regardless of what it was able to do internally or if the script passes the verification step. This is because, in a multisig scenario, the process could have appended only one of the signatures, which would obviously fail during the script verification. Thus, the process needs to update the transaction and notify the user that there are still missing signatures so it can relay it to the next party.
I think we could improve the signing process so it does not update the returned transaction when the verification step finds an error that will persist regardless of any following-up processing step.
That being said, I would encourage people to use PSBT instead of signrawtransactionwithkey
.
Ah thanks, TIL! Now I see how this PR affects that specific RPC master
PR 28307
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 2451a21
Minimal diff since last ACK, just rebase on master and replace a"520" with a "MAX_SCRIPT_ELEMENT_SIZE"
Show Signature
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
ACK 2451a217dd2c21b6d2f2b2699ceddd0bf9073019
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEE5hdzzW4BBA4vG9eM5+KYS2KJyToFAmY4+ZIACgkQ5+KYS2KJ
yTqhnhAApaBq0dhQVXvLaTSBjV8Q8CXJTXapgme6uII649DlRjCQujM4ufmnPfdR
BuQWzAC5FbgA1sCkRk3+D/CyO0H9k5PPmU7Mle6tQl7sHszkXNqzCXmIceQBPvQk
Qk95jzcrh88nJif565D3o88+RGaREku8mGWG5HCkcSyyfTMwaXdxVkvfPB88+g8l
fTDcsP+P7At/fKOIBEN8UHX4hkKhWAv4WYXVI3qUXBWix6wSNqYrQNGwj9BPSHtQ
6pA9p0kZ15SP8cJo0Fe/Hfb2Zk9vtXbuIUGxHMSu08ka+9C8+OlOqJ6NFB/dnTAh
GX60UiC6QXPOvIUZrtC4JWHkdNRJ3fNV+E3xQ8eBlPMMT8ZsznY+SG4lrVYb1ZKG
D0Ho/ASiGYaN4kOTZOVDtuZdspDFBy2z2epmXpBn7h+P1OXgyz1KlGO4gIydrdkO
ITVORmyR2/c1poUyJhHWMaf2pdEYfX0T+UWNvkL+Do0iIDL5JgJCPULfA2T8/FjI
5TjZAVVH1it+BKBtY4owk0LLSiVJfvBGkbPpJaghUYKIrZHiREj8nMq/5+JTHPai
g6JP3lUNEDVi9EPStYWScbIAeoeiy/gFp8fGhST0ovPwN44Cj0/LQdjnvUb1hjS8
O0Y7JGtVBf62UwHItdGTP9rIiuD+oePf5iC7PIBJff1RwjF6ADc=
=IlyD
-----END PGP SIGNATURE-----
pinheadmz's public key is on keybase
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code-review ACK 2451a21
if (CScript inner_script; spk_man.GetCScript(CScriptID(script), inner_script)) { | ||
CHECK_NONFATAL(inner_script == script); // Nothing to add, script already contained by the wallet | ||
continue; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It took me a while to grok what's going on here: if the CScript was already contained in the wallet, the AddCScript
call one line above would fail in the course of writing to the DB (looking at the call flow LegacyScriptPubKeyMan::AddCScript
-> LegacyScriptPubKeyMan::AddCScriptWithDB
-> batch.WriteCScript(...)
-> WriteIC(..., ..., false)
, where the last false
parameter indicates that overwrites are not allowed; the FillableSigningProvider::AddCScript
call earlier doesn't care if an entry is overwritten and still returns true
).
I think the same behaviour could be achieved by simply ignoring the return value of AddCScript
, but I assume the intention here is to be more robust and error if -- for whatever reason -- the CScript write to the database failed, even if the entry isn't there yet.
Fixing #28250 (comment) and more.
Currently, redeem scripts longer than 520 bytes, which are technically valid under segwit rules, have flaws in the following processes:
signrawtransactionwithkey
RPC command fail to sign them.addmultisigaddress
wrongly discards them.The issue arises because most of these flows are utilizing the legacy spkm keystore, which imposes
the p2sh max redeem script size rule on all scripts. Which blocks segwit redeem scripts longer than
the max element size in all the previously mentioned processes (
createmultisig
,addmultisigaddress
, andsignrawtransactionwithkey
).This PR fixes the problem, enabling the creation of multisig output descriptors involving more than 15 keys and
allowing the signing of these scripts, along with other post-segwit redeem scripts that surpass the 520-byte
p2sh limit.
Important note:
Instead of adding support for these longer redeem scripts in the legacy wallet, an "unsupported operation"
error has been added. The reasons behind this decision are:
The introduction of this feature brings about a compatibility-breaking change that requires downgrade
protection; older wallets would be unable to interact with these "new" legacy wallets.
Considering the ongoing deprecation of the legacy spkm, this issue provides another compelling
reason to transition towards descriptors.
Testing notes:
To easily verify each of the fixes, I decoupled the tests into standalone commits. So they can be
cherry-picked on top of master. Where
rpc_createmultisig.py
(with and without the--legacy-wallet
arg) will fail without the bugs fixes commits.
Extra note:
The initial commits improves the
rpc_createmultisig.py
test in many ways. I found this test veryantiquated, screaming for an update and cleanup.