Close-transaction not in the mem pool, funds locked in "limbo", channel in "pending close" forever.

[25shmeckles]

Background

My node Caffeine has a channel in pending status (close txid: 0de535ea74d72a952157640187b5a207fe977870a5c9b92f0aa99b9707c94779) with The Cave since months!
My local balance is stuck in a "limbo" and I have no idea what to do...
How to finalize the closure? How to get the balance out of the limbo?

Your environment

• lnd 0.12.99-beta commit=v0.12.0-beta-494-g140dd944
• Linux Mint 20.1 - Kernel 5.4.0-72-generic x86_64 GNU/Linux
• bitcoind, Bitcoin Core 0.21.1

Expected behaviour

I expected the transaction to end in the mem pool and the channel to be closed.
In case something goes wrong (like in this case) I expect the transaction to be reverted without issues.

Actual behaviour

The transaction is lost, funds are locked in the "limbo", the channel is stuck.

[guggero]

Did you close the channel with a low fee? Or was it a force-close?
The close TX cannot be found in the mempool anymore so it will never close. Try restarting lnd to re-publish the transaction. The fees are currently pretty low so hopefully it will go through now.

[25shmeckles]

It was a force close. I have restarted LND plenty of times, it didn't change. Also, it appears that the other node, The Cave, is not existing anymore. How do you re-publish a transaction?

[guggero]

Do you see in the logs that lnd is trying to re-publish the transaction when you restart it? Do you see an error message? Maybe until now it couldn't broadcast it because it was below the minimum mempool fee rate. Can you try again and post any log that mentions the channel ( f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0)?

[25shmeckles]

Here are the log lines I could find. I don't see a clear error message.

2021-05-30 12:07:44.899 [INF] NTFN: New spend subscription: spend_id=8, outpoint=f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0, script=0 9777d97101cae0062f7604a41b37f05a356eb835829244ecce5227fc31c31c34, height_hint=685499

2021-05-30 12:07:44.899 [INF] NTFN: Dispatching historical spend rescan for outpoint=f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0, script=0 9777d97101cae0062f7604a41b37f05a356eb835829244ecce5227fc31c31c34, start=685499, end=685500

2021-05-30 12:07:52.114 [INF] CNCT: Close observer for ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0) active

2021-05-30 12:08:10.855 [INF] CNCT: ChannelArbitrator(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): starting state=StateDefault, trigger=chainTrigger, triggerHeight=685500

2021-05-30 12:08:33.218 [WRN] CHBU: Replacing disk backup for ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0) w/ newer version

[guggero]

I think you might need to increase the debuglevel=debug to see more. There should be an attempt to publish the transaction again.
If not, can you please post the full output of lncli pendingchannels?

[25shmeckles]

I can't restart lnd right now, will do it in a couple of hours.
Here is the output of lncli pendingchannels

{
    "total_limbo_balance": "997524",
    "pending_open_channels": [
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
    ],
    "waiting_close_channels": [
        {
            "channel": {
                "remote_node_pub": "02ae3c06f6db6a87ea874ef7ba86c87ab0041eabf6d8be0594582505b1fd5d7ee8",
                "channel_point": "f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0",
                "capacity": "1003840",
                "local_balance": "997524",
                "remote_balance": "0",
                "local_chan_reserve_sat": "10038",
                "remote_chan_reserve_sat": "10038",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "LEGACY"
            },
            "limbo_balance": "997524",
            "commitments": {
                "local_txid": "0de535ea74d72a952157640187b5a207fe977870a5c9b92f0aa99b9707c94779",
                "remote_txid": "f7cfe31b1b2a3cd64f1c99adeff797694c931f0c392a210c6c4fbd8147ca583a",
                "remote_pending_txid": "",
                "local_commit_fee_sat": "6316",
                "remote_commit_fee_sat": "6316",
                "remote_pending_commit_fee_sat": "0"
            }
        }
    ]
}

[25shmeckles]

Here is the debug log

cat Desktop/lnd_log.txt| grep f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815 


2021-06-01 15:07:57.386 [DBG] CHBU: Crafting backup for ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0)
2021-06-01 15:07:59.855 [DBG] CNCT: New ChainEventSubscription(id=0) for ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0)
2021-06-01 15:07:59.873 [DBG] CNCT: Starting chain watcher for ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0)
2021-06-01 15:07:59.874 [DBG] NTFN: Using height hint 685795 retrieved from cache for outpoint=f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0, script=0 9777d97101cae0062f7604a41b37f05a356eb835829244ecce5227fc31c31c34 instead of 552251
2021-06-01 15:07:59.875 [INF] NTFN: New spend subscription: spend_id=14, outpoint=f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0, script=0 9777d97101cae0062f7604a41b37f05a356eb835829244ecce5227fc31c31c34, height_hint=685795
2021-06-01 15:07:59.875 [INF] NTFN: Dispatching historical spend rescan for outpoint=f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0, script=0 9777d97101cae0062f7604a41b37f05a356eb835829244ecce5227fc31c31c34, start=685795, end=685795
2021-06-01 15:08:00.000 [DBG] NTFN: Updated spend hint to height=685795 for unconfirmed spend request outpoint=f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0, script=0 9777d97101cae0062f7604a41b37f05a356eb835829244ecce5227fc31c31c34
2021-06-01 15:08:00.000 [INF] CNCT: Close observer for ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0) active
2021-06-01 15:08:00.336 [DBG] CNCT: Starting ChannelArbitrator(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0), htlc_set=(map[contractcourt.HtlcSetKey]contractcourt.htlcSet) (len=2) {
2021-06-01 15:08:00.338 [INF] CNCT: ChannelArbitrator(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): starting state=StateDefault, trigger=chainTrigger, triggerHeight=685795
2021-06-01 15:08:00.338 [DBG] CNCT: ChannelArbitrator(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): new block (height=685795) examining active HTLC's
2021-06-01 15:08:00.339 [DBG] CNCT: ChannelArbitrator(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): checking commit chain actions at height=685795, in_htlc_count=0, out_htlc_count=0
2021-06-01 15:08:00.370 [DBG] CNCT: ChannelArbitrator(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): new block (height=685795) examining active HTLC's
2021-06-01 15:08:00.371 [DBG] CNCT: ChannelArbitrator(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): checking commit chain actions at height=685795, in_htlc_count=0, out_htlc_count=0
2021-06-01 15:08:00.576 [DBG] LNWL: ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): starting local commitment: (*lnwallet.commitment)(0xc0008100c0)({
    PreviousOutPoint: (wire.OutPoint) f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0,
2021-06-01 15:08:00.582 [DBG] LNWL: ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): starting remote commitment: (*lnwallet.commitment)(0xc0045e4240)({
    PreviousOutPoint: (wire.OutPoint) f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0,
2021-06-01 15:08:00.597 [DBG] LNWL: ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): Restoring 0 dangling remote updates
2021-06-01 15:08:00.608 [DBG] LNWL: ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): Restoring 0 local updates that the peer should sign
2021-06-01 15:08:00.609 [DBG] FNDG: ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0) with chan_id=155820e0812ca5392cc0b2f9f3cbdde7d521641595e891915179d0c3664fa9f8 not found in opening database, assuming already announced to the network
2021-06-01 15:08:00.617 [WRN] CHBU: Replacing disk backup for ChannelPoint(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0) w/ newer version

[guggero]

Can you please DM me the full log, for example on Slack or Keybase? Not sure if the log message I'm looking for really contains the channel TXID.

[wpaulino]

2021-06-01 15:08:00.338 [INF] CNCT: ChannelArbitrator(f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0): starting state=StateDefault, trigger=chainTrigger, triggerHeight=685795

StateDefault indicates that we're still waiting for an action. Maybe the initial close attempt failed to complete? Have you tried force closing it again since then?

[25shmeckles]

No, I didn't try anything else. I waited for a while, then forgot about it. And a few days ago I noticed that it hanged.

[wpaulino]

Are you able to try again? In case it hangs again, you can enable profiling before trying again to see where it gets stuck. To enable profiling you'll need to restart and run with the config option profile=host:port.

[25shmeckles]

Here is the full log of a LND session https://gist.github.com/25shmeckles/c387bc42b3a88b409d48a85014468f71

[guggero]

Thanks. So the channel is definitely still in the "normal" state. Can you try closing it again? First lncli closechannel f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815 --sat_per_vbyte X and if that doesn't work or hangs try lncli closechannel f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815 --force ?

[25shmeckles]

Mmm... not working... I get cannot close channel with state: ChanStatusLocalDataLoss

$ lncli closechannel f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815 --sat_per_vbyte 20
[lncli] rpc error: code = Unknown desc = cannot close channel with state: ChanStatusLocalDataLoss

$ lncli closechannel f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815 --force
[lncli] rpc error: code = Unknown desc = cannot close channel with state: ChanStatusLocalDataLoss

[guggero]

Okay, this is very weird. Not sure how lnd could ever get into this state (I will look into it).
For now, to proceed with this I think you'll need to use chantools for this.
Can you save this to a file called summary.json:

{
 "channels": [
  {
   "remote_pubkey": "020f0d93c700cf5cf556e61828fab473a5c30ee6eb927ae36a130cc1c57326d21a",
   "channel_point": "f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815:0",
   "funding_txid": "f8a94f66c3d079519191e895156421d5e7ddcbf3f9b2c02c39a52c81e0205815",
   "funding_tx_index": 0,
   "capacity": 1003840,
   "closing_tx": null,
   "force_close": null
  }
 ]
}

Then download chantools, stop lnd and run the following command:

chantools forceclose --fromsummary <path-to-summary.json> --channeldb <path-to-your-channel.db>

That should give you exactly one raw transaction that you can then publish through a block explorer or your own node.

If that succeeds then you can start lnd again. After the TX confirms lnd should pick up on it and move the channel to waiting_close or pending_force_close.

[25shmeckles]

Excuse my ignorance, where shall I look for channel.db?

[guggero]

In your lnd's data directory, usually ~/.lnd/data/graph/mainnet/channel.db.

[25shmeckles]

It asks for a 24-word mnemonic phrase that I don't think I have... I have the wallet encrypted with a password, that's it. Is there any way I get it?

$ chantools forceclose --fromsummary Desktop/summary.json --channeldb .lnd/data/graph/mainnet/channel.db --publish
2021-06-03 15:20:50.779 [INF] CHAN: chantools version v0.8.5 commit
Input your 24-word mnemonic separated by spaces:

[guggero]

That's your node's seed. You should definitely have that, otherwise all your funds are at risk!

[25shmeckles]

I have the wallet file encrypted and saved in multiple places, isn't enough? Is there a way to close the channel without using the seed? I never needed it with lnd. Same for the node itself, I have multiple backups scattered around. Can I retrieve the seed somehow? Can I use chantools without the seed?

[guggero]

You won't be able to restore lnd without the seed. The wallet file only contains the xprv which is hashed from the seed and cannot be reversed.
You can use chantools with the xprv if you use the --rootkey=xprv... parameter. To extract the xprv from the wallet, you can use chantools walletinfo. See chantools walletinfo --help for more information.

[25shmeckles]

chaintools did the trick, now the channel status is "pending force close". I will confirm when the channel closes completely. Can you explain what did you do with the JSON file exactly? In general, I didn't get what happened with the transaction in the first place.

[guggero]

Okay cool. I'll try if I can figure out how the channel could've become stuck in this situation in the first place. The issue will remain open until we know more.

The chantools forceclose basically just publishes the last commitment transaction that is in the channel DB (which is what force closing really is). The summary.json was more of a selector in this case that told chantools which channel to force-close. Normally you would do this for a list of channels during the advanced manual recovery (step 8): https://github.com/guggero/chantools#channel-recovery-scenario

[guggero]

I checked the code and the only 2 possibilities for a channel to get into the ChanStatusLocalDataLoss are:

1. There was some weird channel synchronization error with the peer. Perhaps an old version of lnd or other implementation running on the other side that has/had a bug? I don't suppose you still have all logs from back when you force-closed the channel? Do you remember why you force-closed it?
2. You tried using SCB recovery. Did you ever use restorechanbackup?

[25shmeckles]

1. I don't have logs from that time, it was about 1y ago but can't be more precise
2. never used

[guggero]

Thanks for the response. I don't think we can figure out what exactly went wrong then. It is possible you (or the peer) ran into a bug that's been fixed in the meantime.

Going to close this for now as your funds seem to be on their way back (I assume the pending_channels output now counts down the maturity height?).
Feel free to comment if you run into further issues with the channel in question.

[25shmeckles]

Indeed, about 60 confirmations till maturity.
Thanks a lot for your help!

[mantoles]

hi guys I am having the same issue with a smaller amount of sats in limbo but the lncli closechannel command returns a channel not found.

Here is the info from my RTL wallet

Transaction ID
8a2cc40bf052a3b8046d18b70be60ed313ea0fd66de084b4fc4de0e13c47c423
Channel Point
c783d1f629bfefadfaa9fd20386164a97b6cecc28c485c60ae0b185a149052b0:1
Peer Alias
doubleqp🍔
Peer Node Pubkey
0307eafebbf6583eaf3dce69181018185b193414901c029e98062455e328c76bf3
Capacity
137,880
Limbo Balance
121,215
Local Balance
121,215
Remote Balance
0

any help would be appreciated since the funds have been in limbo for months.

[guggero]

You need to force-close the channel. Try lncli closechannel --force c783d1f629bfefadfaa9fd20386164a97b6cecc28c485c60ae0b185a149052b0 1 (the "1" in the end is probably what you were missing when you tried the closechannel command previously).