rpc/validation: enable packages through testmempoolaccept #20833

glozow · 2021-01-03T07:16:31Z

This PR enables validation dry-runs of packages through the testmempoolaccept RPC. The expectation is that the results returned from testmempoolaccept are what you'd get from test-then-submitting each transaction individually, in that order (this means the package is expected to be sorted in topological order, for now at least). The validation is also atomic: in the case of failure, it immediately halts and may return "unfinished" MempoolAcceptResults for transactions that weren't fully validated. The API for 1 transaction stays the same.

Motivation:

This allows you to test validity for transaction chains (e.g. with multiple spending paths and where you don't want to broadcast yet); closes Transactions chain and testmempoolaccept #18480.
It's also a first step towards package validation in a minimally invasive way.
The RPC commit happens to close Modify testmempoolaccept to handle client's max_raw_tx_fee check failure #21074 by clarifying the "allowed" key.

There are a few added restrictions on the packages, mostly to simplify the logic for areas that aren't critical to main package use cases:

No package can have conflicts, i.e. none of them can spend the same inputs, even if it would be a valid BIP125 replacement.
The package cannot conflict with the mempool, i.e. RBF is disabled.
The total count of the package cannot exceed 25 (the default descendant count limit), and total size cannot exceed 101KvB (the default descendant size limit).

If you're looking for review comments and github isn't loading them, I have a gist compiling some topics of discussion here

maflcko · 2021-01-03T09:20:23Z

Thanks for working on this. Strong Concept ACK!

jnewbery · 2021-01-03T10:38:42Z

Concept ACK and approach ACK.

DrahtBot · 2021-01-03T11:29:21Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

Add fee_est tool for debugging fee estimation code #10443 (Add fee_est tool for debugging fee estimation code by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

src/validation.cpp

src/rpc/rawtransaction.cpp

glozow · 2021-01-08T23:10:14Z

5th time's the charm I guess 😂 this is ready for review!

darosior · 2021-01-09T19:48:26Z

Concept ACK :)

DrahtBot · 2021-01-13T11:45:49Z

🕵️ @sipa has been requested to review this pull request as specified in the REVIEWERS file.

maflcko

left some nits in the first commit

src/validation.h

mjdietzx

So far I've reviewed the functional and unit tests and they look great! Well done doing that in a way that doesn't require the wallet - very thorough

I think soon test/functional/rpc_packages.py can be done with MiniWallet and get rid of some code. It would probably need #20889 though
bc we'd have to catch some exceptions where we expect from_node.testmempoolaccept to fail. Anyways I wouldn't expect that to be done in this PR or delay anything, tests look great as is

test/functional/rpc_packages.py

test/functional/mempool_accept.py

This allows us to easily create transaction chains for package validation. We don't test_accept if submit=false because we want to be able to make transactions that wouldn't pass ATMP (i.e. a child transaction in a package would fail due to missing inputs).

Key functionality = a transaction with UTXOs not present in UTXO set or mempool can be fully validated instead of being considered an orphan.

Maximum number of transactions allowed in a package is 25, equal to the default mempool descendant limit: if a package has more transactions than this, either it would fail default mempool descendant limit or the transactions don't all have a dependency relationship (but then they shouldn't be in a package together). Same rationale for 101KvB virtual size package limit. Note that these policies are only used in test accepts so far.

Only allow "packages" with no conflicts, sorted in order of dependency, and no more than 25 for now. Note that these groups of transactions don't necessarily need to adhere to some strict definition of a package or have any dependency relationships. Clients are free to pass in a batch of 25 unrelated transactions if they want to.

glozow · 2021-05-24T15:22:48Z

Thanks @ariard for reminding me about the max-fee-exceeded API discussion. I've just pushed an update similar to @jnewbery's suggestion in https://github.com/bitcoin/bitcoin/pull/20833/files#r619141139. If we're satisfied with this API for max-fee-exceeded, then all of the discussions will have been addressed.

My proposal is: If a transaction in the package exceeds maxfeerate, the "reject-reason" is "max-fee-exceeded" and all subsequent transactions have blank validation results (i.e. only "txid" and "wtxid" fields). This is the same as what we do if a transaction fails validation and is still compatible with the API on master.

To respond to @mzumsande's suggestion in that comment thread:

Also, would it make sense to apply the maxfeerate limit on a combined package level, and not for each transaction separately?

In the future, yes. I plan to open a followup PR to have ProcessNewPackage calculate (and testmempoolaccept return) the descendant feerate and ancestor feerate for each transaction in the package. The intent is to enable the CPFP use case where parent doesn't meet minimum fee but descendant feerate with child is sufficient.
Then, we can apply mempool min fee policy to descendant feerate (so that packages can fee-bump) and enforce the client-specified maxfeerate on the ancestor feerate. This would apply the protection for "I'm trying to fee-bump this transaction's ancestor(s), but I don't want to overshoot it," which is what I believe the user's intent would be.

jnewbery

Code review ACK 13650fe

A few minor comments inline. A couple more:

The commit log for 2ef1879 ([validation] package validation for test accepts) refers to CoinsViewTemporary, which has now been removed.
maybe move the [policy] detect unsorted packages to before the [rpc] allow multiple txns in testmempoolaccept (so you don't have to add tests and then change them later)

jnewbery · 2021-05-26T12:25:01Z

src/validation.cpp

-    AssertLockHeld(pool.cs);
-    assert(std::addressof(::ChainstateActive()) == std::addressof(active_chainstate));
-
-    CBlockIndex* tip = active_chainstate.m_chain.Tip();
    assert(tip != nullptr);


(Doesn't necessarily need to be in this PR - can be done in a follow-up) it'd be nice to make tip a reference, to better communicate that this function can't be called without a CBlockIndex.

jnewbery · 2021-05-26T12:26:18Z

src/txmempool.cpp

@@ -515,7 +515,9 @@ void CTxMemPool::removeForReorg(CChainState& active_chainstate, int flags)
        LockPoints lp = it->GetLockPoints();
        assert(std::addressof(::ChainstateActive()) == std::addressof(active_chainstate));
        bool validLP =  TestLockPointValidity(active_chainstate.m_chain, &lp);
-        if (!CheckFinalTx(active_chainstate.m_chain.Tip(), tx, flags) || !CheckSequenceLocks(active_chainstate, *this, tx, flags, &lp, validLP)) {
+        CCoinsViewMemPool viewMempool(&active_chainstate.CoinsTip(), *this);


Suggested change

CCoinsViewMemPool viewMempool(&active_chainstate.CoinsTip(), *this);

CCoinsViewMemPool view_mempool(&active_chainstate.CoinsTip(), *this);

src/test/miner_tests.cpp

jnewbery · 2021-05-26T12:41:12Z

src/validation.cpp

@@ -494,11 +585,20 @@ class MemPoolAccept
         */
        std::vector<COutPoint>& m_coins_to_uncache;
        const bool m_test_accept;
+        /** Disable BIP125 RBFing; disallow all conflicts with mempool transactions. */


Having a boolean for disallow_thing where the double negative disallow_thing == false means "thing is allowed" is probably not as clear as a allow_thing boolean where allow_thing == true means "thing is allowed".

I also agree that "replacement" is probably better than "conflicts" here.

src/validation.cpp

src/policy/packages.h

src/rpc/rawtransaction.cpp

src/validation.cpp

mzumsande

I like the way max-fee-exceeded is handled now. Some comments below, feel free to ignore if too nitty.

mzumsande · 2021-05-26T23:22:48Z

src/rpc/rawtransaction.cpp

+    // We will check transaction fees we iterate through txns in order. If any transaction fee
+    // exceeds maxfeerate, we will keave the rest of the validation results blank, because it
+    // doesn't make sense to return a validation result for a transaction if its ancestor(s) would
+    // not be submitted.


fees while we iterate, keave->leave

mzumsande · 2021-05-27T00:16:37Z

src/rpc/rawtransaction.cpp

                },
                RPCResult{
                    RPCResult::Type::ARR, "", "The result of the mempool acceptance test for each raw transaction in the input array.\n"
-                        "Length is exactly one for now.",
+                        "Returns results for each transaction in the same order they were passed in.\n"
+                        "It is possible for transactions to not be fully validated ('allowed' unset) if an earlier transaction failed.\n",


maybe "if another transaction failed", because it's not necessarily an earlier transaction? (in the case where validation terminates early, the culprit could also be a later transaction)

mzumsande · 2021-05-27T00:21:25Z

src/rpc/rawtransaction.cpp

                    {
                        {RPCResult::Type::OBJ, "", "",
                        {
                            {RPCResult::Type::STR_HEX, "txid", "The transaction hash in hex"},
                            {RPCResult::Type::STR_HEX, "wtxid", "The transaction witness hash in hex"},
-                            {RPCResult::Type::BOOL, "allowed", "If the mempool allows this tx to be inserted"},
+                            {RPCResult::Type::STR, "package-error", "Package validation error, if any (only possible if rawtxs had more than 1 transaction)."},
+                            {RPCResult::Type::BOOL, "allowed", "Whether this tx would be accepted to the mempool and pass client-specified maxfeerate."


nit: space after maxfeerate

Are you saying I should add a space after maxfeerate or?

I meant after "maxfeerate." and before the next sentence, it should show when displaying the rpc help.

mzumsande · 2021-05-27T00:45:21Z

src/validation.cpp

+    /**
+    * Multiple transaction acceptance. Transactions may or may not be interdependent,
+    * but must not conflict with each other. Parents must come before children if any
+    * dependencies exist, otherwise a TX_MISSING_INPUTS error will be returned.


After the last commit, the error will be "package-not-sorted".

ariard

ACK 13650fe

Overall code sounds robust enough for now. I did check again test coverage, both functional/unit and it seems exhaustive. I think we can still argue on few improvements around comments but not blockers IMO.

My proposal is: If a transaction in the package exceeds maxfeerate, the "reject-reason" is "max-fee-exceeded" and all subsequent transactions have blank validation results (i.e. only "txid" and "wtxid" fields). This is the same as what we do if a transaction fails validation and is still compatible with the API on master.

That's a good-enough API for now and I agree with your proposed follow-up to calculate and evaluate the aggregated package feerate.

Other follow-ups I do have in mind :

weight units or vbytes : rpc/validation: enable packages through testmempoolaccept #20833 (comment)
mempool ancestor/descendants limit for packages (mempool/validation: mempool ancestor/descendant limits for packages #21800)
package-policy checks encapsulation : rpc/validation: enable packages through testmempoolaccept #20833 (comment) (mostly motivated to have a future shared library libtxstandardness a la libbitcoinconsensus, marrying well with [rfc] add option to bypass contextual timelocks in testmempoolaccept? #21413)

What else ? I don't think ordering really matter.

src/validation.cpp

src/policy/packages.h

ariard · 2021-05-27T04:08:42Z

src/rpc/rawtransaction.cpp

-                "\nThis checks if the transaction violates the consensus or policy rules.\n"
+                "\nReturns result of mempool acceptance tests indicating if raw transaction(s) (serialized, hex-encoded) would be accepted by mempool.\n"
+                "\nIf multiple transactions are passed in, parents must come before children and package policies apply: the transactions cannot conflict with any mempool transactions or each other.\n"
+                "\nIf one transaction fails, other transactions may not be fully validated (the 'allowed' key will be blank).\n"


nit: I think a more descriptive behavior would be "if one transaction fails, remaining transactions are not submitted for validation". See document L1146 in src/validation.cpp "Exit early to avoid doing pointless work. Update the failed tx result; the rest are unfinished".

ariard · 2021-05-27T04:45:43Z

src/rpc/rawtransaction.cpp

-    if (!DecodeHexTx(mtx, request.params[0].get_array()[0].get_str())) {
-        throw JSONRPCError(RPC_DESERIALIZATION_ERROR, "TX decode failed. Make sure the tx has at least one input.");
+    const UniValue raw_transactions = request.params[0].get_array();
+    if (raw_transactions.size() < 1 || raw_transactions.size() > MAX_PACKAGE_COUNT) {


nit: the check on MAX_PACKAGE_COUNT is duplicated into AcceptMultipleTransactions. I can understand the rational of not taking the cs_main lock but I've a preference to keep all package policy checks in one place. That would also avoid linking packages.h in rawtransaction.cpp (assuming we hardcode the package limit in RPC doc).

I think of this as a distinct check, actually. We defined the testmempoolaccept API to be "maximum 25 transactions" and will return a JSONRPC error here because the user violated our API, whereas AcceptMultipleTransactions() is applying package policies.

ariard · 2021-05-27T04:50:02Z

src/rpc/rawtransaction.cpp

+    UniValue rpc_result(UniValue::VARR);
+    // We will check transaction fees we iterate through txns in order. If any transaction fee
+    // exceeds maxfeerate, we will keave the rest of the validation results blank, because it
+    // doesn't make sense to return a validation result for a transaction if its ancestor(s) would


nit: transaction result

ariard · 2021-05-27T05:24:34Z

src/txmempool.h

 protected:
    const CTxMemPool& mempool;

 public:
    CCoinsViewMemPool(CCoinsView* baseIn, const CTxMemPool& mempoolIn);
    bool GetCoin(const COutPoint &outpoint, Coin &coin) const override;
+    /** Add the coins created by this transaction. */


nit: "Temporarily add the coins created by this transaction until package processing to which it belongs is over" ?

Added a more descriptive comment in #22084

ariard · 2021-05-27T05:50:46Z

src/validation.cpp

+        // Make the coins created by this transaction available for subsequent transactions in the
+        // package to spend. Since we already checked conflicts in the package and RBFs are
+        // impossible, we don't need to track the coins spent. Note that this logic will need to be
+        // updated if RBFs in packages are allowed in the future.


nit: "mempool transaction replacement by packages".

Otherwise it lets suggest we apply RBF inside a currently-processing package. Also I think the replacement policy we'll end up for packages is going to be far different than BIP 125, so instead of mentioning RBF, I would say simply replacement for now.

ariard · 2021-05-27T06:07:17Z

src/validation.h

+* Atomically test acceptance of a package. If the package only contains one tx, package rules still apply.
+* @param[in]    txns                Group of transactions which may be independent or contain
+*                                   parent-child dependencies. The transactions must not conflict, i.e.
+*                                   must not spend the same inputs, even if it would be a valid BIP125


A bit confusing, one might wonder if you're aiming to in-package conflict or mempool transactions conflicted by package. I would suggest another formulation : "The package transactions must not conflict, i.e must not spend the same inputs. They must not replace mempool transaction, even if it's valid under BIP125 rules."

laanwj · 2021-05-27T20:38:44Z

Code review re-ACK 13650fe
My understanding is that the rest of the comments (which tend to be relating to comments, documentation, asserts, argument naming, and error messages) will be addressed in the follow-up PR, so are not blocking the merge of this feature.

…oolaccept

ee862d6 MOVEONLY: context-free package policies (glozow) 5cac95c disallow_mempool_conflicts -> allow_bip125_replacement and check earlier (glozow) e8ecc62 [refactor] comment/naming improvements (glozow) 7d91442 [rpc] reserve space in txns (glozow) 6c5f19d [package] static_assert max package size >= max tx size (glozow) Pull request description: various followups from #20833 ACKs for top commit: jnewbery: utACK ee862d6 ariard: Code Review ACK ee862d6 Tree-SHA512: 96ecb41f7bbced84d4253070f5274b7267607bfe4033e2bb0d2f55ec778cc41e811130b6321131e0418b5835894e510a4be8a0f822bc9d68d9224418359ac837

glozow mentioned this pull request Jan 3, 2021

locks and docs in ATMP and CheckInputsFromMempoolAndCache #20834

Merged

DrahtBot added Mempool P2P RPC/REST/ZMQ Validation labels Jan 3, 2021

maflcko mentioned this pull request Jan 3, 2021

rpc: testmempoolaccept for list of transactions #16398

Closed

sdaftuar reviewed Jan 5, 2021

View reviewed changes

src/validation.cpp Outdated Show resolved Hide resolved

luke-jr reviewed Jan 6, 2021

View reviewed changes

src/rpc/rawtransaction.cpp Outdated Show resolved Hide resolved

src/rpc/rawtransaction.cpp Outdated Show resolved Hide resolved

glozow force-pushed the package-testmempoolaccept branch 5 times, most recently from 43588c4 to 79e5512 Compare January 8, 2021 21:02

glozow marked this pull request as ready for review January 8, 2021 23:09

glozow mentioned this pull request Jan 9, 2021

p2p: mildly discourage peers that violate feefilter #20895

Closed

DrahtBot mentioned this pull request Jan 12, 2021

rpc: Return wtxid from testmempoolaccept #20916

Merged

maflcko reviewed Jan 14, 2021

View reviewed changes

src/validation.h Outdated Show resolved Hide resolved

src/validation.h Outdated Show resolved Hide resolved

src/validation.h Outdated Show resolved Hide resolved

DrahtBot added the Needs rebase label Jan 14, 2021

mjdietzx reviewed Jan 14, 2021

View reviewed changes

test/functional/rpc_packages.py Outdated Show resolved Hide resolved

test/functional/mempool_accept.py Show resolved Hide resolved

glozow added 8 commits May 24, 2021 14:42

[test] unit tests for ProcessNewPackage

363e3d9

Key functionality = a transaction with UTXOs not present in UTXO set or mempool can be fully validated instead of being considered an orphan.

[fuzz] add ProcessNewPackage call in tx_pool fuzzer

c9e1a26

[test] functional test for packages in RPCs

c4259f4

[doc] add release note for package testmempoolaccept

9ef643e

[policy] detect unsorted packages

13650fe

glozow force-pushed the package-testmempoolaccept branch from 930c6d3 to 13650fe Compare May 24, 2021 15:04

DrahtBot mentioned this pull request May 26, 2021

Add fee_est tool for debugging fee estimation code #10443

Closed

jnewbery reviewed May 26, 2021

View reviewed changes

mzumsande reviewed May 27, 2021

View reviewed changes

ariard reviewed May 27, 2021

View reviewed changes

glozow mentioned this pull request May 27, 2021

package testmempoolaccept followups #22084

Merged

laanwj merged commit 7257e50 into bitcoin:master May 27, 2021

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request May 28, 2021

Merge bitcoin#20833: rpc/validation: enable packages through testmemp…

09fd8fc

…oolaccept

glozow deleted the package-testmempoolaccept branch May 28, 2021 08:37

This was referenced Jun 13, 2021

validation: Move package acceptance size limit from KvB to WU #22097

Closed

policy: Trim Packages when transaction with same txid exists in mempool #22252

Closed

Christewart mentioned this pull request Jul 14, 2021

Add test cases for testmempoolaccept for DLCs bitcoin-s/bitcoin-s#3403

Open

glozow mentioned this pull request Oct 27, 2021

validation: mempool validation and submission for packages of 1 child + parents #22674

Merged

glozow mentioned this pull request Jan 17, 2022

Package Mempool Submission with Package Fee-Bumping #22290

Closed

10 tasks

This was referenced Mar 8, 2022

Could the wallet count unconfirmed non-mempool change? #11887

Open

Clean up the rpc method testmempoolaccept #14016

Closed

gwillen pushed a commit to ElementsProject/elements that referenced this pull request Jun 1, 2022

Merge 7257e50 into merged_master (Bitcoin PR bitcoin/bitcoin#20833)

8a669ec

darosior mentioned this pull request Jul 11, 2022

mempool: Add option to bypass contextual timelocks in testmempoolaccept #25434

Closed

bitcoin locked as resolved and limited conversation to collaborators Aug 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpc/validation: enable packages through testmempoolaccept #20833

rpc/validation: enable packages through testmempoolaccept #20833

glozow commented Jan 3, 2021 •

edited

maflcko commented Jan 3, 2021

jnewbery commented Jan 3, 2021

DrahtBot commented Jan 3, 2021 •

edited

glozow commented Jan 8, 2021

darosior commented Jan 9, 2021

DrahtBot commented Jan 13, 2021

maflcko left a comment

mjdietzx left a comment

glozow commented May 24, 2021

jnewbery left a comment

jnewbery May 26, 2021

jnewbery May 26, 2021

jnewbery May 26, 2021

mzumsande left a comment

mzumsande May 26, 2021

mzumsande May 27, 2021

mzumsande May 27, 2021

glozow May 27, 2021

mzumsande May 27, 2021

mzumsande May 27, 2021

ariard left a comment

ariard May 27, 2021

ariard May 27, 2021

glozow May 27, 2021

ariard May 27, 2021

ariard May 27, 2021

glozow May 27, 2021

ariard May 27, 2021

ariard May 27, 2021

laanwj commented May 27, 2021

	CCoinsViewMemPool viewMempool(&active_chainstate.CoinsTip(), *this);
	CCoinsViewMemPool view_mempool(&active_chainstate.CoinsTip(), *this);

rpc/validation: enable packages through testmempoolaccept #20833

rpc/validation: enable packages through testmempoolaccept #20833

Conversation

glozow commented Jan 3, 2021 • edited

maflcko commented Jan 3, 2021

jnewbery commented Jan 3, 2021

DrahtBot commented Jan 3, 2021 • edited

Conflicts

glozow commented Jan 8, 2021

darosior commented Jan 9, 2021

DrahtBot commented Jan 13, 2021

maflcko left a comment

Choose a reason for hiding this comment

mjdietzx left a comment

Choose a reason for hiding this comment

glozow commented May 24, 2021

jnewbery left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzumsande left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ariard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laanwj commented May 27, 2021

glozow commented Jan 3, 2021 •

edited

DrahtBot commented Jan 3, 2021 •

edited