Package Relay Project Tracking

[glozow]

Tasks and PRs

☑️ (1) multi-parent-1-child package validation

What we get: the ability to validate multiple transactions, including CPFP of transactions below the mempool minimum feerate. An RPC to submit things locally.

• Enable validation of multiple transactions in MemPoolAccept
  • Dependency: refactor: return MempoolAcceptResult from ATMP #21062
  • Dependency: validation/refactor: refactoring for package submission #23381
  • Main feature 1/2: rpc/validation: enable packages through testmempoolaccept #20833
  • Main feature 2/2: mempool/validation: mempool ancestor/descendant limits for packages #21800
  • Followup: package testmempoolaccept followups #22084
• Enable package CPFP
  • Main feature 1/2: validation: mempool validation and submission for packages of 1 child + parents #22674
  • Followup: docs / fixups from RBF and packages #24310
  • Followup: validation: followups for de-duplication of packages #23804
  • Main feature 2/2: policy / validation: CPFP fee bumping within packages #24152
  • short term bug fix: avoid the risk of below-minrelaytxfee transactions hanging around forever in the mempool mempool: disallow txns under min relay fee, even in packages #26933
• RPC access
  • add RPC (-regtest only) for testing package policy #24836
  • validation, bugfix: provide more info in *MempoolAcceptResult #26646
  • rpc: allow submitpackage to be called outside of regtest #27609
  • bugfix, Change up submitpackage results to return results for all transactions #28848
  • RPC: Add maxfeerate and maxburnamount args to submitpackage #28950
  • Followup: 28950 followups #29722
  • Followup: AcceptMultipleTransactions: Fix workspace not being set as client_maxfeerate failure #29735
• Fuzzing and bug fixes
  • Add package evaluation fuzzer #28450
  • Fuzz: Check individual and package transaction invariants #28764
  • fuzz: Minor improvements to tx_package_eval target #28825
  • Bug fix: validation: fix coins disappearing mid-package evaluation #28251
  • Bug fix: Fix virtual size limit enforcement in transaction package context #28471
  • Bug fix: Remove MemPoolAccept::m_limits to avoid mutating it in package evaluation #28472

☑️ (2) Policy-Related Features: TRUC and limited package RBF

What we get: an opt-in policy for anti-pinning in single transaction or 1-parent-1-child package scenarios. Also, package CPFP of 0-fee parent and package RBF for restricted topologies prior to cluster mempool.

☑️ (2a) Topologically Restricted Until Confirmation (v3) transaction policy

Also see #29319 for its role in cluster mempool.

• Main feature: v3 transaction policy for anti-pinning #28948
• Followup: v3 followups #29424
• Sibling Eviction: policy: enable sibling eviction for v3 transactions #29306
• Decrease max vsize: policy: restrict all TRUC (v3) transactions to 10kvB #29873
• Enable v3 on mainnet: policy: bump TX_MAX_STANDARD_VERSION to 3 #29496
• Followup: doc: use TRUC instead of v3 and add release note #30272
• Docs: BIP431: Opt In Topologically Restricted Until Confirmation Transactions For More Robust Fee-bumping bips#1541

☑️ (2b) Package RBF for cluster size 2

• Enable Package RBF for 1-parent-1-child situations
  • Dependency: RBF move (1/3): extract BIP125 Rule 5 into policy/rbf #22796
  • Dependency: RBF move 2/3: extract RBF logic into policy/rbf #22675
  • Dependency: RBF move 3/3: move followups + improve RBF documentation #22855
  • Dependency: Mempool util: Add RBF diagram checks for single chunks against clusters of size 2 #29242
    • Followup: 29242 Diagram check followups #29724
    • feefrac: avoid explicitly computing diagram; compare based on chunks #29757
  • Dependency: refactor prep for package rbf #30072
  • test: add conflicting topology test case #30066
  • Main feature: Cluster size 2 package rbf #28984
  • followup: #28984 package rbf followups #30295

Also see: Ephemeral Dust and Pay To Anchor (P2A) aka Ephemeral Anchors

☑️ (3) 1-parent-1-child package relay

What we get: opportunistic acceptance of of 1-parent-1-child packages. Not quite "package relay" but similar to it.
Protocols like LN can use this to create 0-fee presigned transactions with a single, 0-value anchor output; 0-fee commitment transactions can replace each other using the fees of the child attached to the anchor.
This should provide an adequate replacement for CPFP carve out, which is helpful for the next step (see #29319).

• Opportunistically submit 1-parent-1-child packages
  • Dependency: refactor: consolidate MempoolAcceptResult processing #29619
  • Main feature: p2p: opportunistically accept 1-parent-1-child packages #28970
  • Followup: opportunistic 1p1c followups #30012

(4) TxDownloadManager and Orphan Resolution improvements

All of this code directly contributes to package relay, whether it's BIP 331 or a different protocol.

• Modularize TxDownloadManager + testing
  • Dependency: validation: return more helpful results for reconsiderable fee failures and skipped transactions #28785
  • Dependency: test: tx orphan handling #28199
  • Dependency: log: log wtxids when possible, add TXPACKAGES category #28364
  • Dependency: locks: introduce mutex for tx download, flush rejection filters once per tip change #30111
    • Followup: m_tx_download_mutex followups #30507
  • Main PR: refactor: TxDownloadManager + fuzzing #30110
    • Followup: TxDownloadManager followups #31190
• Make orphan resolution more robust. See known orphanage problems.
  • p2p: index TxOrphanage by wtxid, allow entries with same txid #30000
  • p2p: track and use all potential peers for orphan resolution #31397
    • Followup: multi-peer orphan resolution followups #31666
  • TxOrphanage: account for size of orphans and count announcements #31810
  • p2p: improve TxOrphanage denial of service bounds #31829
    • Followup: p2p: TxOrphanage revamp cleanups #32941

(5) Further Work

• Test Package Accept: Test Package Accept #32160
• Ancestor Package Relay: propagate incentive-compatible packages that are more complex than 1p1c, safely evaluate replacements within packages, handle orphans better. See BIP 331
• Arbitrary package validation
  • validate package transactions with their in-package ancestor sets #26711
  • https://delvingbitcoin.org/t/post-clustermempool-package-rbf-per-chunk-processing/190
  • BIP 331: Ancestor Package Relay bips#1382
  • [NO MERGE] BIP331 Ancestor Package Relay #27742
  • linearization + bonus for rbf arbitrary package validation (bonus linearization with conflict discount) glozow/bitcoin#12
  • subchunking arbitrary package validation (sub-chunking) glozow/bitcoin#11
• Sender-initiated package relay protocol: reduce round trips by anticipating a peer's orphan resolution request and sending them the parent along with the child, in a pkgtxns message. p2p: implement sender-initiated package relay #33500
• Dynamic orphan resolution limits per peer, based on connection type/reputation
• fuzzer to check that packages can always propagate

See also:

Superseded/Deferred Work

• refactors for subpackage evaluation #28758
• refactor: Miniminer package linearization followups #28808
• rpc: permit any ancestor package through submitpackage #28813
• mempool / miner: regularly flush <=0-fee entries and mine everything in the mempool #27018
• policy: nVersion=3 and Package RBF #25038
• policy: Ephemeral anchors #26403
• Add imbued v3 based on template-matching #29427
• mempool: keep CPFP'd transactions when loading from mempool.dat #27476
• Package Relay 1/3: Introduce TxDownloadManager and improve orphan-handling #28031

Prehistory

• refactor: Rewrite AcceptToMemoryPoolWorker() using smaller parts #16400
• Add package acceptance logic to mempool #16401
• Package relay design questions #14895
• [RFC] Package-relay: sender-initiated #19621
• https://gist.github.com/sdaftuar/8756699bfcad4d3806ba9f3396d4e66a

[instagibbs]

Very helpful. I think this helps define "sprints" that can be achieved to hit next milestones.

[ariard]

I think one more sub-category to track is the feedback collected on how package relay/nversion=3 are fitting multi-party applications and contracting protocols

• Lightning: legacy channels / anchor output / taproot types
• Peerswap: claim with preimage + refund cooperatively + refund after csv passes
• Collaborative transaction: splicing + dual funding flows

The 0-conf flow of Lightning can be also considered. As deployed today by some LSPs where both incoming/outgoing HTLCs are allowed during the pending confirmation period, there is a risk in case of mempool congestions of the inbound funding transaction not confirming before the timelock of the offered HTLC output expires. The current nversion=3 size is 1000 vb which might be too low if LSPs would like to do batching of 0-confs. Note, I think all the pinning risks are not the same for 0-conf funding transactions than for time-sensitive confirmation of LN commitment+HTLCs transactions.

Mentioning 0-conf, and as raised on #27643 at some point in the development cycle of nversion=3 it might be good communication practice to publish than all 0confs users might have to upgrade their mempool monitoring stuff to reject nversion=3, or chain of transactions with them, as otherwise genuine upgrade to the version of Core with package relay might lower their security.

For Lightning, there is the question if the p2p package format is flexible enough for the Lightning channels to apply "new mempool rules" on "old in-flight channnel type". Like backward compatible effect of the mempools rules without off-chain channels having to cooperate on-resigning transactions. I believe that would require a Lightning node to sign the p2p package with the same pubkey than used with the channel (though I start to think we might have multiple types of backward/forward compatible application of mempools rules like the ones mentioned in #25038, like there is some matrix ?)

For the second-layers listed, I'm just considering the ones which have a) a mature specification and b) some substantial deployment though should we consider more second-layers or multi-party applications like vaults (non-deployed) ?

From a reviewer viewpoint, I think it could be very valuable if you can highlight the 2/3 PRs or mail posts you consider as top of the stack for reviews ? Thanks for a package-relay context-tracking meta-issue.

(Feel free to integrate what you find relevant of this comment to ease context tracking of package relay as a project)

[ajtowns]

I think at a high level, the "package relay" part looks like:

• make package mempool acceptance available via rpc in regtest
• fix up package acceptance/retention/mining policies to be free from DoS vectors and to behave sensibly
• behave sensibly with arbitrary txs in a package
• implement bip331 to expose package relay over p2p not just rpc

And relatedly:

• version 3 acceptance/rbf policies to limit pinnability
• ephemeral anchors

Particularly if #26933 is applied, is there any reason not to make submitpackage available on all chains? It should behave no differently to submitting the transactions individually with a non-full mempool, no?

[ajtowns]

FWIW, I don't think "incentive compatibility" is a good description of our goal here -- I think there's three goals we want:

• for most normal use cases (including newly invented ones) just submitting your txs over p2p should work fine
  • (otherwise people will tend to use centralised tx submission methods, which creates a chokepoint that will attract censorship)
• running a plain bitcoind should get you block templates within 90%-99% of optimal, depending on the resources you can allocate to your node
  • (otherwise miners will outsource block template construction to proprietary services in order to be competitive, which again creates a censorship chokepoint)
• mempool/relay behaviour should be as simple to understand as possible -- for wallet designers, bitcoin users, node operators and miners
  • (otherwise bitcoin will only be usable in theory, not in practice)

I think most of the "incentive compatible" things above aren't really making anything more incentive compatible (miners can just raise their mempool limits to keep the transactions; non-mining nodes aren't receiving any incentives in the first place) but rather making mempool behaviour easier to understand (eg "we removed this tx because there was a race condition when loading the mempool" vs "we removed this tx because it wouldn't be included in any of the next 100 optimal blocks, even if no new txs are submitted").

[glozow]

I think at a high level, the "package relay" part looks like:

Yes, that is how I think of it.

Particularly if #26933 is applied, is there any reason not to make submitpackage available on
all chains? It should behave no differently to submitting the transactions individually with a non-full mempool, no?

Somewhat. It's still a little bit weird when you submit a child-with-parents where a parent relies on the other (grep "Check that validation isn't quitting too early" in #26711, currently at this line). I would say it's safe after the "submit with subpackages" changes.

Also, mempools are full on mainnet (great for testing but means there is a difference), and we'll definitely want to warn them that the transactions are not necessarily relayed etc.

[ajtowns]

Also, mempools are full on mainnet

My mempool isn't full, and (I think) hasn't been full this year. (It has a 1GB limit; though there's only ~40MB of txs now anyway)

Submitting a low-feerate tx with a high mempool limit already means your tx won't relay without a cpfp tx, but we expect people to be who are clever enough to do that to also deal with the consequences. (If you do cpfp the low feerate tx, it'll be broadcast via orphan handling, so won't appear in the unbroadcast set any longer, but also won't successfully relay without package processing, so you'll end up in the exact same state, I think)