[bug]: Channel close transactions dropped from mempool stuck/missing

[rbndg]

Background

I have a few channels that have been either force closed or co-op closed. These fee on these channels were too low and have since been dropped from the mempool. So they are not stuck in pendingchannels and can't do anything with them.

Things i've tried:

• Restart LND to force rebroadcast tx during lower fee env. Didn't work
• Looked for the transactions in listchaintx, and they were missing
• Looked for the transaction in bitcoin core and they were missing.
• Bumped mempool size in bitcoin core to 800 and restarted LND. Nothing worked.
• Looked for transactions in the logs and couldn't find anything.
• Ran bumpfee and bumpclosefee and it says 'bumping < txid >:' but it doesn't actually do anything. So it's giving a false positive.

Your environment

• v0.17.4-beta
• Ubuntu 20
• bitcoind v22.0.0

Steps to reproduce

Close channel with low fee and wait for it to get dropped from mempool

Expected behaviour

• Be able to bump these channels with higher fee to close, or at least get a reason why these channels can't be bumped.
• Shouldn't need to restart LND to rebroadcast these tx, there should be a command to do it.

[ViktorTigerstrom]

Thanks for raising the issue @rbndg, we are investigating and will get back to you ASAP!

[ziggie1984]

Can you provide us details of the stuck channels via: lncli pendingchannels => there you will see a lot of channels in the
waiting_close_channels state can you share the fee of this output, you should see something like this:

  "local_commit_fee_sat":  "3276",
  "remote_commit_fee_sat":  "3276",

Moreover can you check your mempool purging rate can you share the following output: bitcoin-cli getmempoolinfo

Normally when restarting, LND will try to re-publish the closing transaction can you search for Re-publishing force close tx in your logs ?

[rbndg]

Can you provide us details of the stuck channels via: lncli pendingchannels => there you will see a lot of channels in the waiting_close_channels state can you share the fee of this output, you should see something like this:

  "local_commit_fee_sat":  "3276",
  "remote_commit_fee_sat":  "3276",

Moreover can you check your mempool purging rate can you share the following output: bitcoin-cli getmempoolinfo

Normally when restarting, LND will try to re-publish the closing transaction can you search for Re-publishing force close tx in your logs ?

"local_commit_fee_sat": "2810",
"remote_commit_fee_sat": "2810"


{
  "loaded": true,
  "size": 95908,
  "bytes": 124807233,
  "usage": 551114352,
  "total_fee": 6.61811923,
  "maxmempool": 800000000,
  "mempoolminfee": 0.00001000,
  "minrelaytxfee": 0.00001000,
  "unbroadcastcount": 0
}

[ziggie1984]

Ok so your bitcoind backend seems to not be the problem, thank you for the information:

Normally when restarting lnd you should see the following log lines for channels which are in the waiting_close_state

lnd/contractcourt/chain_arbitrator.go

Lines 891 to 901 in 5cb4811

	case err == channeldb.ErrNoCloseTx:
	log.Warnf("Channel %v is in state %v, but no %s closing tx "+
	"to re-publish...", chanPoint, state, kind)
	return nil
	case err != nil:
	return err
	}
	log.Infof("Re-publishing %s close tx(%v) for channel %v",
	kind, closeTx.TxHash(), chanPoint)

For both coop-close and force-closed transaction you should see these kind of messages for example like this:

[INF] CNCT: Re-publishing force close tx(4e3bfd1c803eb9d3a265aacc243ba35169b725105b6ab8187df822ad2f7b65fd) for channel 148ceee1ff5042a74e71f12aabce4581e0a3c2261268a5d1e8ea88aec04a8242:0

or for the coop close case:

[INF] CNCT: Re-publishing coop close tx(ac59cb309cd5163ad1e097f6eba3f528500fe2485d2e3dde8f48168c72ac1a30) for channel 73bb9aa1e3bd04ecfe49325478b3ac479ac3b2983897483aeefe450982190593:0

Do you see those kind of msg in the logs ?

Can you also check which status the channels are in which you cannot find or are stuck (in the pendingchannels output):

For example:
"chan_status_flags": "ChanStatusCoopBroadcasted|ChanStatusLocalCloseInitiator",

[rbndg]

I found attempted rebroadcast for one of the channels, seems like its saying its double spent. I this could have been one of the channels that I tried bumping its fee. but I dont think that ever did anything

2024-03-11 05:50:38.352 [INF] CNCT: Re-publishing force close tx(22de5559a1508746f0463b7ca2101e7fabc21fa276f2779d9ae3d24d5ee45380) for channel 86504a1e1efa4ca0b76dab40e650b99193ee9269771bca9f9d4741327a2604e5:1
2024-03-11 05:50:38.352 [INF] LNWL: Inserting unconfirmed transaction 22de5559a1508746f0463b7ca2101e7fabc21fa276f2779d9ae3d24d5ee45380
2024-03-11 05:50:38.354 [INF] LNWL: 22de5559a1508746f0463b7ca2101e7fabc21fa276f2779d9ae3d24d5ee45380: broadcast failed because of: double spend: -25: bad-txns-inputs-missingorspent
2024-03-11 05:50:38.355 [INF] LNWL: Removed invalid transaction: 22de5559a1508746f0463b7ca2101e7fabc21fa276f2779d9ae3d24d5ee45380
2024-03-11 05:51:01.522 [INF] SWPR: Sweep request received: out_point=22de5559a1508746f0463b7ca2101e7fabc21fa276f2779d9ae3d24d5ee45380:0, witness_type=CommitmentAnchor, relative_time_lock=0, absolute_time_lock=0, amount=0.00000330 BTC, parent=(fee=0.00002811 BTC, weight=1116), params=(fee=144 blocks, force=false, exclusive_group=17592186044416000570)

Status for the same channel

 "chan_status_flags": "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator",

[ziggie1984]

Hmm, could you check for the above channel whether you were the initiator: "initiator": "INITIATOR_LOCAL", ?

Can you find the opening transaction in listchaintxns ? So as you might see the channel point is not in the blockchain which means this channel does not really exist, do you use zeroconf channels ?

[rbndg]

Yes, my node is the initiator

Ah yes. I know about this 0 conf bug. Which is that UTXO locking doesn't work properly for zero conf channel, so the channel never confirmed in the first place.
Yes the channel above the channel point is indeed not in the chain.

How do I clear our these channels from pendingchannels list

[ziggie1984]

you must use lncli abandonchannel [REDACTED] (please look at the help output of the command and read the description of the flag carefully) but be very careful with this command, and verify yourself that the channel which is waiting to be closed really is not lingering in "the mempool" or confirmed onchain.

If you can find the opening transaction via lncli listchaintxns you can verfiy whether the inputs of the opening where spent by another tx (to be 100% sure)