[bug]: crash-loop on startup: unmatched backend error: -26: mempool min fee not met #7156
Comments
|
some logs: |
|
Duplicate issue from the last time we had a fee spike: #5969. Other instances across time: https://github.com/lightningnetwork/lnd/issues?q=is%3Aissue+mempool+min+fee+not+met+is%3Aclosed.
There's not a good way around it afaict: bitcoind will start to increase the min fee needed to get into the mempool and if you want to allow things lower than that to get in, you need to increase the max mempool size. We likely should properly catch this error and proceed. However that would sort of mask the issue here, since ppl would wonder why their transactions aren't getting mined. |
|
Re being able to proceed, we need to add a case above this to allow that to go thru: https://github.com/btcsuite/btcwallet/blob/master/wallet/wallet.go#L3745-L3748 Main question is how to actually handle this properly:
As is, we'll also remove that txn so we don't try to broadcast again. |
|
gotcha thanks @Roasbeef that makes sense. now if i understand correctly: say my bitcoin node now runs an extra large mempool, but all its peers run default mempools. i could get into a situation where my bitcoin node has my txs, but nobody else does, even when mempool congestion chills out. how do i ensure that my bitcoin node will eventually rebroadcast these low-fee transactions? if i recall correctly there was some in-progress work in bitcoin core perhaps to identify this situation and rebroadcast automatically? |
Correct, the mempool is truly a murky place unfortunately. Bitcoin nodes have something called a The only solution to something like this in site is the fabled package relay construction. This would let you send that transaction, then also couple w/ it something to allow the fee to be high enough for acceptance. |
so in the meantime I might just wanna keep some miners on speed-dial eh? |
or i guess if any miners run large mempools too, i could deliberately peer with them |
|
just a little more context on my situation for others to learn from:
In general my takeaway is that a lot of problems can be avoided if you just run a larger bitcoind mempool. but there are caveats associated with this |
|
Closing as the config issue has been resolved. |
|
Gonna re-open this as Boltz ran into it recently, and it can cause We should catch the error, remove the tx, but still proceed with start up. Ideally we also have another way of letting users know that their transactions have fallen out of the "default mempool". |
|
I'm in a spot where I tried opening a channel at the early stages of this this most recent (and prolonged) mempool traffic jam. I used a Seems like a couple of things that would be nice to have (in addition to fixing the startup issue described above) would be:
|
I am not quite sure I understand this, so you want to RBF a channel opening although its not in your local mempool? |
|
I will take this one |
|
You can't RBF a funding transaction, as that would invalidate any commitment transaction. Or do you mean a spec proposal for a new re-negotiation/signing message? |
exactly that's also what I was thinking about, maybe he was thinking about making the ChannelOpening RBFable in the first place(sequence below FFFFFFFd) so one could essentially double-spend it in case its lingering in the mempool for too long, which might be cheaper than hitting the Chain with the Commitment Transaction (apart from the timelock I am putting on the funds in case I force-close the channel) ? |
|
I don't think we should do that either. It will just encourage users to actually do RBF. There were at least two cases I know of where users imported lnd keys into Electrum to create an RBF replacement TX for a funding transaction, putting their funds at risk. This only worked because the original one was already evected from the mempool. But IMO we shouldn't encourage being able to do that while the original TX is still present. Also the "RBF enabled" label on a block explorer might give users wrong ideas too. |
Oops yeah my mistake, I'd overlooked that and oversimplified it. It looks like it would have to be some sort of spec change to renegotiate a pending channel which I think is well out of scope of this issue. Still though, it seems like there is some undesirable behavior. The channel from my last post is still in pending status, but that was well over the 2016 blocks before a pending channel should timeout and I believe that means the remote peer has forgotten about the channel and it can never properly open. Ideally lnd would fail this pending channel itself after this delay, and if the tx hasn't even been broadcast yet it should jsut drop it entirely. Then a second channel opening attempt can be performed. |
|
Ok I analysed the situation regarding the startup issue and my analysis showed exactly 2 cases where we end up not starting because we could not broadcast the transaction (mempool-fee was not enough).
So one downside of dropping this transaction is in an Anchor-Channel Case that we will not be able to FeeBump the transaction because the sweeper which sweeps the anchor (CPFP the Commitment) will fail with the Another way would be: The solution as @Roasbeef described would be to just include a case for the
I think for the second case we should not remove the transaction but keep trying rebroadcasting it (but change the log level for the min-mempool-fee failure, or give the user the possibility to fetch the rawtx in case he wants to broadcast it via other transaction relayers where the purging fee is lower? Happy to receive feedback in which direction we wanna go, if we have consensus I will code it. I would do the following:For case 1 where we are not able to broadcast the Commitment Transaction, still fail lnd to start to let the node-runner know that with the current mempool architecture his Commitment Transaction will not be able to confirm (because the anchor cannot bump a non-existent parent output (without package-relay) For the second case I would lnd continue to startup successfully but rebroadcast the crib-transaction in a timely manner. Maybe change the log-level for the min-relay-fee not met error, to not spam the log with errors and give the user the possibility via an RPC command to fetch the rawTx for this Crib-Transaction? |
|
I have the same issue.
This is the force-close transaction, manually initiated by me ( Upgrade to the latest version Increasing @dannydeezy thank you very much! |
|
@SunnySarahNode was this an anchor-channel ? If so I doubt there is a current fix to this issue because you can only bump the commitment-tx if it is in the mempool meaning that without you increasing your mempool you wouldn't be able to confirm your transaction anytime soon. Tho increasing the mempool and therefore being able to bump the transaction with your Anchor is imo a good approach. |
|
So to resolve this issue, we just need to make sure that on start up, we don't refuse to start if we get this error. See this issue: #4616 This code block needs to be updated to catch this error: https://github.com/btcsuite/btcwallet/blob/50978fcf79f8100eb62c185606c66367cc03b03e/wallet/wallet.go#L3463 With this recent PR, transactions should always be above the min relay fee at the time of tx creation. It's possible that things drop below that value, but until wide spread package relay, there's really nothing we can do about that. The choice is essentially to:
Note that even with the second option, the transaction won't propagate since nodes will send a |
Yeah tough choice, see my message above. Either way things are sort of hosed...we can choose to force user intervention or handle it in the background and pray to satoshi that things drop enough to let the tx enter the mempool of nodes with default policy.
The issue here is that we never really know if the transaction is "in the mempool" or not. Node policy is less uniform these days, and a big transaction spike can quickly cause the min relay fee to jump up.
I think it's a good idea to optionally allow RPC calls to print out the full hex of the broadcasted transaction so users can rebroacast on their own. Note that we'll now also rebroadcast ourselves, so if the min relay fee drops down, then things should propagate.
The way the rebroadcaster works, it'll keep going until things confirm or the sweeper removes it as a conflict is mined. So on start up, those should be broadcast again. |
For case 1, if we do this, then I think we should have a clearer error message to force the user to action. So something like "consider raising your
I'm on board with having the relevant RPCs also allow users to get the raw hex. We log it each time, but logs can be spammy at times on trace/debug, so it might be hard for users to find. |
Should the sweeper remove channel open attempts that have not confirmed (and possibly not even been broadcasted successfully due to peers rejecting it for too low fee regardless of what our mempool settings are) after 1000 blocks? IIUC, that's the point at which peers will "forget" our channel open attempt if they haven't seen a confirmation on chain. It seems like we shouldn't try to rebroadcast or wait for confirmation of a channel opening transaction that is stale. It would be better to just discard the transaction and the pending channel, and free up the utxo(s) consumed by it to be able to try opening another channel (with a higher fee). |
|
That could be an option after 2016 blocks, yes. Though just because a transaction isn't in the local mempool doesn't mean it will never confirm. So to be absolutely certain that a channel can be discarded, the inputs to the funding transaction need to be double spent and confirmed. Otherwise you lose access to the funds if the funding transaction confirms for whatever reason. |
That makes sense, although if we detect that it hasn't been broadcast successfully it should still be safe to discard right? That seems like it'd be like a less likely scenario with the change to ensure that the fee is above the min relay fee at the time of tx creation, though. Double spending the broadcasted, but unconfirmed and possibly out of (most) mempools, transactions via lnd would be cool. To do something like this now, I figure you'd need to create and broadcast a transaction using the same utxo manually outside of lnd? And then once lnd detects the double spend, it would remove the pending channel and opening transaction? |
Correct. We recently added this to |
Awesome, somehow I wasn't aware of this set of tools. Do you know if the |
|
You can use
That shouldn't happen anymore with the recent change to force at least the mempool/relay fee rate.
I think that's another in-flight PR to detect double spends of channel funding inputs. |
Background
During a mempool spike, I had some low-fee-rate on-chain transactions from my lnd wallet that got dropped from default mempools. Then on startup I got into a crash loop with the following error:
I was able to unblock by going into my
bitcoin.confand increasingmaxmempool, but I don't think that is a sustainable solution.Your environment
The text was updated successfully, but these errors were encountered: