Lost anchor: stuck sweep transaction (unconfirmed balance) #6241
Comments
|
To make matters worse, I think lnd happily used this (unconfirmed) sweep tx output as an input for a new channel (I specified Please let me know what I can do with the "pending open" channel. Is it safe to abandon the channel, as the open TX cannot be confirmed, ever? More confusingly, I also think the 100k reserve for anchor channels is broken due to this (I have lots of anchor channels). Before I attempted to open a channel: After: |
|
Related to #386 |
|
Related to this Topic, should we make the Anchor Sweep Transaktion (non-RBF Signaling) in that case nobody could RBF the funds even after they are broadcasted or does this imply even more logic for handling funds in case there are no other outputs left in the wallet. Might increase the complexity of the wallet so I understand why we RBF the Anchor Sweep in the first place but just bringing it up here as an idea. |
We've seen a script attempt to more aggressively sweep anchor outputs, they'd claimed $70 or so worth when I last checked. Right now we always use the smallest fee rate possible when attempting to sweep anchor outputs. You'll be able to set the anchor fee rate after this PR: #6025 Re that balance snapshot, when UTXOs are locked, they don't show up in that base balance output. We should likely add another field there though, so it's a bit easier to keep track of things.
This PR fixes that behavior: #5158 |
|
Great! I think the only missing bit is to fail the sweep transaction (detect the double spend), i.e. release the funds that are used as another input alongside the anchor? |
|
To add more context - today I had the same issue. Did not do anything extraordinary except opening one channel with Today I tried opening another channel the same way, resulting in the channel showing as pending, my wallet balance being very different and the logs showing the following: |
|
@Roasbeef not sure but I think this problem will affect way more people having to rescan their wallets if they figure out that their anchor was swept. Because I was sweeping a lot of Anchor Outputs and effectively also RBF existing Anchor-Sweeps which means way more people have a pending tx in their lnd wallet which will never be confimed. |
|
Just an update for anyone affected by this. To get everything in order again you must reset your wallet balance so that LND completely rescans the blockchain for transactions. 1. Restart LND and reindex wallet transactionsInsert this line in and restart LND. Make sure to remove this line afterwards, otherwise LND will do this every time on startup. It will take a while, depending on your hardware and bitcoind setup. Only do this if you're not running a pruned bitcoind - then you're in uncharted territory (i.e. I don't know if it will work - it probably won't). 2. Abandon the affected channelsIf you do have pending channels that used the already spent anchor outputs (where the funding transaction was rejected by the network) you can abandon those channels to make them disappear from the pending_channels list. Make sure that these are really "phantom" channels, don't abandon any channels that are just waiting for confirmations. |
|
In practice I don't think this should require doing a full rescan. With the way the sweeper works, when it tries to publish something invalid, it'll just periodically back off and retry again, possibly with a distinct output set. We also watch for when that output is spent and then stop sweeping and signal back to the caller. So if no unconfirmed dependent spends exist, then it should just go away. When you do a funding attempt, all the inputs are locked, which rn doesn't show up in the normal balance call. If we detect an invalid spend, then the transaction is removed from the wallet, and eventually the locked funds will become available again. What we don't do rn is automatically remove that transaction from the pending channels list, which is fixed in this PR. So I think the missing gap here is:
With the way the wallet interacts rn with RBF transaction, it doesn't know which one will actually confirm (the mempool is a murky place), so it's possible a user attempts to spend one of those outputs as zero conf. When/if a valid transaction eventually confirms the wallet state will be cleaned up, but only if it's actually one of our spends (need to verify this). |
HannahMR
added
P2
|
@Roasbeef - it also looks like LND does not check if a channel funding transaction is valid (neither a local one, nor a remote channel funding transaction). I was wondering how it's possible to have pending opens for weeks (iniitiated by the remote party) where the funding transaction can't be found in the mempool. It seems that those peers tried to open a channel with an invalid funding transaction and my node not even realizing that the channel opening was basically a double-spend that should have been rejected right away. |
If you extend lnd a channel (you're on the receiving side), then it can't actually verify the funding transaction, as it only sees the outpoint. We don't watch the mempool, only the chain. After the channel has been confirmed, we're able to fully validate the funding transaction. See this issue for adding more pre-broadcast mempool for funding transactions. We wait for 14 days before automatically removing zombie channels: Line 101 in 28ea273 |
Roasbeef
added
P1
Roasbeef
added
chain handling
fees
Do we have a commitement tx in our local memory when a channel is opened to us although we still have no funds on our side? And if yes how do we forget about the channel by broadcasting the commitment tx or are we just purging it in our database? |
When we're the receiver of a new funding attempt? Yes, we'll write that to disk as confirmation may happen after we're down.
When we're the receiver of a new channel, and we choose to drop the channel, we'll just remove the channel state (it goes into a closed channel log). Broadcasting the commitment won't do anything if the funding transaction actually doesn't exist, or can never confirm for some reason. We only do this is we're not the initiator though. |
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
I've implemented this here: #6274. Needs tests, but the approach should work, I think there're other areas we can apply the same approach to make accounting a bit easier, and also cut down on unnecessary transactions piling up in |
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
|
Hi y'all, I've confirmed that this PR fixes the issue! I was able to properly reproduce the scenario deterministically in an integration test, to confirm my commit fixed the core issue. So thanks to y'all for such a detailed issue report! I think the fix I added to the sweeper may actually resolve some other issues in the tracker related to unconfirmed spends. As 0.15 is still about a month away, I think maybe it makes sense to package this in an 0.14.3 minor release to alleviate this strain. Thoughts? |
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
|
I stopped sweeping anchor by "RBF(BIP125)" from the mempool, so I guess we are also safe to wait for 0.15, but yeah if this functionality is ready why not ship with 0.14.3, thanks for the fix @Roasbeef |
|
I would agree. The earlier the better. Thank you for the quick fix! |
Def, we may look to package up some other bug fixes as well while we're at it. Earliest possible rc would likely be mid/late next week. |
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
…nflicts Before this commit, we we were trying to sweep an anchor output, and that output was spent by someone else (not the sweeper), then we would report this back to the original resolver (allowing it to be cleaned up), and also remove the set of inputs spent by that transaction from the set we need to sweep. However, it's possible that if a user is spending unconfirmed outputs, then the wallet is holding onto an invalid transaction, as the outputs that were used as inputs have been double spent elsewhere. In this commit, we fix this issue by recursively removing all descendant transactions of our past sweeps that have an intersecting input set as the spending transaction. In cases where a user spent an unconfirmed output to funding a channel, and that output was a descendant of the now swept anchor output, the funds will now properly be marked as available. Fixes lightningnetwork#6241
Background
After a force-close of an anchor channel my node was unable to recover the anchor output in time (it was swept by someone else, and the state is "LOST" in
lncli pendingchannels). I didn't check, but I think this happened at a time with high on-chain fees, and the close tx was already confirmed (so grabbing the anchor doesn't help with that).Now, a few days after my node published the now double-spent sweep transaction, I still see funds as "unconfirmed_balance" (I don't have anything else pending, and the mempool was just emptied). I believe that lnd still waits for the sweep transaction to confirm, which will never happen.
I'd like lnd to give up and "unblock" my funds.
Your environment
Steps to reproduce
Expected behaviour
Sweep transaction is cancelled (forgotten), unspent transaction inputs can be used as inputs for other transactions.
Actual behaviour
Inputs are listed as "unconfirmed balance".
The text was updated successfully, but these errors were encountered: