Allow all mempool txs to be replaced after a configurable timeout (default 6h) #10823
This PR' aim is to improve user experience around stuck transactions without
tldr: Allow transaction replacement for transactions sitting in mempool for
This PR affects policy/relay only.
Stuck transactions have been a problem for users recently. While wallets
For the purpose of the below I will ignore transactions created by the core
Bitcoin 0.12 introduced (or in a way re-introduced) opt-in transaction
At the same time mempool limiting (configurable) was introduced, making the
Both before and after these changes any transaction in mempool would be
Recently 0.14.0 increased the eviction from 72 hours to 2 weeks. These changes
A number of miners will mine transactions regardless of opt-in flags (5-10%
This may be convenient for attackers or ad-hoc expert use but
Until then a compromise solution that doesn't impact zero conf use and that
The timeout should be high enough that allows current use of zero conf and at
The changes continue to support disabling entirely transaction replacement
If you want to test the changes using @petertodd Replace-by-Fee tools build
@TheBlueMatt replacement can be disabled (as a node policy afaik rather than wallet) and the new option i added allows for easier testing and for people to run values that they like different from the default (either to be same as previous behavior or to get the other end without running some fork/patched core).
@TheBlueMatt @jonasschnelli I am happy to do a BIP and discuss in the dev mailing list as needed. I thought the changes may be borderline like the change from 72h to 2 weeks for mempool expiry but happy (minus potential trolling) to learn otherwise and or try to reduce the impact of the changes if this is an option.
I disagree that this needs a BIP.
Opt-in RBF added a new way to interpret a transaction, which just barely qualified as something you might want to do a BIP for.
This however makes an existing behavior - transactions being replaceable in spite of them not signalling opt-in RBF - happen a little sooner in some circumstances, just like adding expiration did in the first place. We didn't create a BIP for expiry, so why does this need a BIP?
Secondly, writing a BIP for such a trivial change gives the misleading impression that you could rely on the opposite behavior. We've had continual problems with people misunderstanding the security properties of zeroconf; there's no reason to add to that problem.
I'd suggest to put the timeout at 72h - at least when introducing the functionality. This way the behviour is similar to pre 0.14.x code. It's less likely to create havoc / confusion for operators who are still used to regularly see 0-conf transactions being re-spent after that time-frame (but not earlier).
I think 6hrs is very short, I've seen many TXs confirmed after 24hrs+, there's plenty of people who, during bigger mempool periods, even try to aim for that...
though it might feel a bit insignificant for a BIP, this PR impacts a lot of assumptions people have about 0conf txs, it would get more significant exposure and discussion if proposed as a BIP.
@rubensayshi You seem to be arguing that the time interval should be longer for security reasons. For that argument to be valid, you'd have to substantiate the claim that a transaction with a fee so low that it fails to confirm after 36 blocks - 10 blocks more than the fee estimator even supports - is still difficult to double spend.
@rubensayshi this doesn't stop those transactions from being confirmed if that's what the user wanted anyway it just allows people to replace transactions after a 6 hours long period - replacing them is already very easy to do ad-hoc even without the replacement flag but that full replacement it is currently painful to do for wallet developers without having to use different peering policies/full nodes with full replacement enabled.
I'd argue for full replacement at all times given the farce that it is doing replacement even without flags but if people want to at least try to keep the illusion of zero conf then I think 6 hours gives plenty of time for people that use zero conf to keep using it at the same ~ [in]security as today inclusive of illusion while giving users the opportunity to not have txs stuck for 2 weeks and solve stuck transactions problems within a business day.
@promag moved to setmocktime, factored out the function, removed the white space and updated to new style - thanks!
More feedback welcome. I think it would be good to have more tests (ideas?) and I tried to be the least intrusive here but perhaps people have some refactoring suggestions.
edit: there's a bit more work needed as something broke with the rebase around list transactions/mempool sync
Adding 2 cents:
The 6h limit seems arbitrary. For transactions that come in the hours after 9AM ET when Bitmex does their dump the network is vastly different than at other times.
Current state of things is that the mempool clears out overnight almost every night. Hasn't been for very long that it hasn't cleared out after two nights.
We accept 0-conf at Bitrefill (with certain checks of course), for several years, very successfully, and despite not advertising that we do, it the expectation of users is that 0-conf should work. In times when we've had to raise the minimum fee on 0-conf transactions we get support tickets from confused users. Quite a few (50%?) of users still pay from custodial services where they don't control the settings of the transactions being sent. We frequently have transactions lying around until they clear out overnight with no issue.
A limit of 72h would maybe be acceptable in this regard, but would need to be weighed against other considerations, from a utilitiarian perspective the question is weighing how many transactions this improves the experience for and how many it makes things worse.
We also need to remember that we currently have opt-in RBF, so wallets and services that choose to support replacing can simply just set that flag. I don't want to reignite the big RBF and 0-conf debate, but it seems like we have a reasonable compromise that there is clear opt-in RBF that is used by some, who then clearly declare that they don't want their transactions to be accepted unconfirmed, and other transactions where it is up to the recipient of the transactions to chose whether to accept those transactions.
Reminding about this stated goal :)
72h would def break things much less than 6h, but still makes me wonder how many considerations one would need to be keeping in mind for what the rules are.
I know this is just setting default behavior and it's all miner configurable, but my current understanding is that defaults often simply become the de facto consensus, such as with the mintxrelayfee.
Another question: How does this deal with child transactions made for CPFP? Perhaps it would be meaningful to not replace transactions that have been CPFP'd. But that too adds more complexity.
Feb 22, 2019
Another argument in favor of 6 hours is that if even 3% of miners support full RBF then 6 hours is plenty of time for even a fairly lazy attacker to double spend. Although this pull request could impact that likeliness, because it's easier to reach those miners.
It may be that, as @ziggamon's points out, it's a bit contrived to build a feature that's essentially for wallets that support RBF, but leave it off by default. That scenario seems more likely than one where a user decides to upgrade to a better wallet while experiencing a stuck transaction.
Although it doesn't solve the same use case, I do like the idea of a 24-72 hour delay, because transactions that are so far down the queue tend to leave the mempool anyway. That can cause people to incorrectly believe their transaction is "cancelled". Or conversely their wallet will hold on to it forever, but never propagating, so the recipient can't even CPFP it. Making it easier to really cancel such a transaction, by double spending yourself back to your own wallet, could be useful even for wallets that don't support RBF.
Subsequently to my earlier comments I now think this is kinda pointless: Testing without RBF set gave me 100% confirmation or replacement rate for very low fee transactions within 20 blocks without the low fee txn rising to being minable by feerate presumably due to restarting nodes, and miners that replace anyways. Is this just motivated by either speculation on actual behaviour without measuring it or a distributed systems misunderstanding that causes people to not just set replacement in the first place?
Gotta agree with @gmaxwell here.
Also, the myth that zeroconf transactions have any security recently lead to $195,000 CAD of publicly reported losses: https://globalnews.ca/news/5047918/calgary-police-nationwide-bitcoin-fraud/
Full-RBF-by-default will help mitigate this misunderstanding. I also have personal interest in stopping this misunderstanding as other reporting on the subject has inaccurately implicated me as part of this crime.
It is highly unlikely that regular users will use bumpfee (or other use cases of tx replacements) merely because observations suggest that miners are accepting full-rbf. This pull request would make it also mempool policy, which is useful when a wallet is not directly connected to miners, but maybe needs to do a few hops through network nodes.
I did an n=1 experiment today where, trying to replace a single-input non-RBF transaction that pays 15 sat / vbyte with one that pays 50 sat / vbyte.
The new transaction doesn't show up on any explorer, suggesting none of my peers is relaying it despite the huge fee bump. It's not like there's a list of known full-RBF peers to manually connect to either.
This is potentially different from the very low fee scenario Maxwell described, because my initial transaction was very well propagated.
The procedure is slightly easier than it used to be, because there's no need to zap the wallet. Just unload the wallet and delete mempool.dat before calling
That said, I don't think my example is a good use case. The reason the initial transaction didn't use RBF is because it was created with RPC and I forgot to set
jnewbery left a comment
This PR introduces two changes:
I think those changes should be separated, at least into two separate commits.
My preference for (2) would be for the behavior to be controlled by a flag in the
I'm fine having the default set to 72 hours to address the concerns of @ziggamon and others.
The current test is for rejection/acceptance by the wallet. Could you add a test that tries to submit a replacement tx over P2P which is rejected, and then setmocktime in the future and have the tx accepted over P2P?
Before going through the code, Is there any consensus on whether this is a good idea at all? Reminding of the debate around opt-in RBF. The opponents' argument was that it would lead to a slippery slope and removing 0-conf entirely. This seems like a step in this direction. Like it or not it is successfully used to a large extent, not just by our users.
Let's evaluate some situations:
b) In a state where the mempool is non-empty for long periods of time this creates a situation that potentially worsens a fee crisis by making it so that merchants must CPFP transactions to fit within the 72h (or whichever window), which potentially can lead to trampling for the exits at the exact wrong time, causing fees to spike further. Large amounts of automated code outbidding eachother for limited space is part of the reason for why we had the fee problem in 2017 in the first place.
CPFP here also becomes a vector for abuse per the following: Anyone can make a large consolidation transaction that includes a send to a party that does CPFP on incoming tx's within 72h => Externalize the costs of confirming this consolidation to someone else, which in this scenario is expensive.
Can somebody describe a specific scenario where a user would benefit from this change in network policy? Is this a user that uses an RBF wallet, but chooses to have RBF disabled but then changes his mind? Or is it a user that upgrades wallets in order to access this feature post-fact? Can we estimate the potential benefit and weigh against the potential damage?
I agree with John that we should separate policy aspects here from wallet aspects. Bitcoind policy for propagation is not consensus but largely goes unchanged so usually becomes a sort of de facto consensus.
No, I don't think consensus has been reached. The fact that you have legitimate concerns about it shows that we don't have consensus!
Correct - if the mempool clears out almost every night then there is no need for this.
With the code in this PR, no. Descendants are ignored when considering whether a transaction is eligible for replacement.
Currently, there are wallets that can create replacement transactions but don't signal RBF by default (I believe this is the case with Bitcoin Core and Electrum wallet). If a user of one of those wallets sends a transaction using the defaults, finds that it's stuck and later wishes to bump the fee, they are unable to. If the defaults for policy were changed according to this PR, that user would be able to bump their transaction after 6 hours (or 72 hours if we changed the default time).
My main argument for this change is that it makes expectations clearer. As @greenaddress, @petertodd and @gmaxwell have pointed out, if a transaction is unconfirmed after 6 hours, a malicious actor could easily have double-spent it. Any recipient of a transaction that hasn't been confirmed after that much time should therefore not have any confidence that the tx will ever confirm.
Near the end of 2017, "how much time do I need to wait until my transaction gets cancelled" was a frequent question on IRC. The usual scenario was that the user had used either old software, or just a shitty wallet, that did not set RBF, and had set a low fee. This tx would then never get mined. In some unlucky cases, there was not even a change output, so CPFP could not be used.
Not sure how much wallets improved since then. I guess we will only really know when the mempool gets consistently non-empty again.
This PR, being able to RBF transactions without opt-in after some time, would be useful so that at least users could be given a precise answer to the original question: "after x hours, the tx can be changed". Then again, I am skeptical about light wallets implementing this... they would need to keep track of the time they broadcast the tx, assume default node behaviour, and only after timeout, offer an option to bump the fee...? Well I guess the option could always be there, along with some disclaimer (I think this is what we might do in Electrum).
I am in favour of this, hugely because I consider it valuable to have a precise answer (even if it only reflects default behaviour).
In Electrum, on desktop, all transactions are RBF by default since 3.1 (March 2018);
Right. So we could estimate the usage of these two wallets (Electrum and Bitcoin Core GUI) to what, 10% of payments in a day link, generously? And this error case is a single digit percentage of that, and something that the wallet could prevent. On the other side of the scale we have the 50%+ of payments that are being sent from custodial services or exchanges link. Or the 60% of payments that use something that doesn't even do Segwit yet link. These users will not helped by this, in fact the bitcoin network for them becomes strictly worse. I will elaborate further down.
So, Signalpolitik. If that'st he goal then that should be discussed not in a github ticket or IRC but in more broad social angles.
If this were really the case then we wouldn't need this PR, would we? But regardless, this is an assumption that is easily testable. Let's do some tests before we roll out such a big policy change? To my knowledge most tests that are referenced happened in 2012, would be good to have some new data. Happy to let you use Bitrefill to test against, you can buy Lightning channels and just not redeem them and have my word that there will be no trouble. Our findings is that it's reasonably secure for smaller transactions and empirically there hasn't been any systematic abuse of this feature (we've had plenty of attempts to abuse other things)
Earnestly glad that you also value the benefit of still having PoS payments with onchain bitcoin. Not sure what thoughts on this are among the maintainers of this repo. While I too think that all such payments will migrate to Lightning with some time, there is no need to break current onchain policy, Lightning is attractive enough to get people to upgrade without the stick.
RBF breaks this model for any purchase that is denominated in USD as it allows the user to game the system by waiting indefinitely to see if the Bitcoin prices move. If Bitcoin price goes up while transaction is still unconfirmed - simply cancel it and make a new one for cheaper. If the merchant decides to change the offered price while the payment is in the mempool that opens up for a world of hurt that none of us want.
With RBF this exposure grows beyond the 15 minutes or so that the merchant decides, but it expands to the entire period until the transaction confirms, which in this case can be a very long time (hence the reason for this ticket). This isn't a huge deal when individual wallets have this behavior because you can inform the user about it and have them either disable that feature for next time, or even change wallets to one that will have a better experience. Currently RBF usage is around 7% of transactions link, probably lower of batched payments, so not a huge deal. But if we break the behavior on a policy level it means that services that have a PoS display like the above will need to move to a custodial model (deposit and confirm first, buy stuff later, keep some change on the account). One might make an argument that this mode is better and this is how bitcoin should be working and so on, but that's a bigger discussion than reviewing code in a PR. My experience with bitcoin core is that there is a desire to maintain backwards compatibility as much as possible and not introduce contentious changes.
That's not really what I meant. After 6/72/whatever hours, if my node receives a transaction that replaces a tx in my mempool, but with a larger fee, then I can be reasonably sure that the creator of that transaction could get it confirmed ahead of the original transaction. My mempool pretending that's not the case and rejecting the replacement is not helpful to me.
I agree though, this does deserve wider discussion and a github issue isn't really the place for it (and I thank you for taking part in the discussion here!)
That sounds fun! It'll be more difficult to double-spend you while blocks aren't consistently full. As you've observed above, this change makes sense in a world where blocks are consistently full. The reason to make this change now is that node policy should be changed in advance of expected changes in the network (see #15846 for example).
Thanks again for your robust input in this PR. It really is very useful to hear from merchants and high-frequency users.
Bitcoin Core GUI has been using RBF by default for a while. Bitcoin Core RPC doesn't, but I think we should assume RPC users know what they're doing (modulo bugs like #15878). I don't think this change is useful for Bitcoin Core wallet users.
The wallet change does make sense conditional on the node policy change, but because the node only knows it's own policy, it could be source of confusion (especially for non-default settings).