Opening Channels Stock in Pending Open

[jairunet]

Describe the bug
I am running startOS 0.3.44, with Core Lightning version 23.02.26, and RTL version 0.13.6~beta (API Version: 0.10.2)

When I try to open the lightning channel with RTL I got a quick resulted error (I am sorry I was not fast enough to captured it) and the channel remains on the Pending Open status.

And in the detailed information: CHANNELD_AWAITING_LOCk

RTL never generate a channel id and the funding ID does not show up in any of the mempools.

To Reproduce
Steps to reproduce the behavior:

1. Go to '.Lightning'
2. Click on 'Peers/Channels'
3. Click on '.Pending/Inactive'
4. See error

Screenshots
If applicable, add screenshots to help explain your problem.

Your environment

• Version of RTL version 0.13.6~beta
• Version of core lightning version 23.02.2~6
• Version of bitcoin core 25.0.0~3
• Browser & browser version
• Operating system (uname -a on *Nix) Darwin XXXXXXXX 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun 8 22:22:20 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6000 arm64
• any other relevant environment details

Additional context
I was able to open the channel by using the lightning-cli tool from CLN, but the three pending channels I try to open with RTL are still Pending Open and I believe that is not the correct behavior of RTL.

Thank you in advance for your suggestions.

Sincerely,

[ShahanaFarooqui]

@jairunet This transaction did not reach to the mempool. Can you check listfunds and listpeerchannels response and paste them here (after masking sensitive info) for further troubleshooting?

[jairunet]

Hello @ShahanaFarooqui ,

Thank you for looking into this issue.

Correct, the transaction never made it to any of the public mempools.

Here are the results related to the channels:

#lightning-cli listfunds

{ "peer_id": "0381dbc5290c1c5a02693be10093e45029044fdcdffe5011a17fecb96281c5d202", "connected": true, "state": "CHANNELD_AWAITING_LOCKIN", "channel_sat": 3000000, "our_amount_msat": "3000000000msat", "channel_total_sat": 3000000, "amount_msat": "3000000000msat", "funding_txid": "anumberhere160a8d467468a4b1533808e298ed1d9aec94c23", "funding_output": 0 }, { "peer_id": "0381dbc5290c1c5a02693be10093e45029044fdcdffe5011a17fecb96281c5d202", "connected": true, "state": "CHANNELD_AWAITING_LOCKIN", "channel_sat": 3000000, "our_amount_msat": "3000000000msat", "channel_total_sat": 3000000, "amount_msat": "3000000000msat", "funding_txid": "anumberhere0b2a4266f72b2e3f21323a8d9d2eb10cb36e8c37eb1", "funding_output": 0 }, { "peer_id": "0381dbc5290c1c5a02693be10093e45029044fdcdffe5011a17fecb96281c5d202", "connected": true, "state": "CHANNELD_AWAITING_LOCKIN", "channel_sat": 3000000, "our_amount_msat": "3000000000msat", "channel_total_sat": 3000000, "amount_msat": "3000000000msat", "funding_txid": "anumberhere434535e1d4e597d9c948c982537ae71ce93b7bdda9b16e", "funding_output": 0 },

# lightning-cli listpeerchannels

{ "peer_id": "0381dbc5290c1c5a02693be10093e45029044fdcdffe5011a17fecb96281c5d202", "peer_connected": true, "state": "CHANNELD_AWAITING_LOCKIN", "scratch_txid": "44e7c474363c56cff9a4abdc73d1ee30306d77c2c97227a395b616599ea48a81", "last_tx_fee_msat": "560000msat", "feerate": { "perkw": 774, "perkb": 3096 }, "owner": "channeld", "direction": 0, "channel_id": "anumberhere33154b8a4667d4a860b1ec51e5581de60e1bc5cecec3", "funding_txid": "anomberhere58e551ecb160a8d467468a4b1533808e298ed1d9aec94c23", "funding_outnum": 0, "close_to_addr": "bc1addresshere", "close_to": "anumberhereba7f4e34a2c9e9cf67b024bbe9adcb2", "private": false, "opener": "local", "alias": { "local": "3254791x14923884x11714" }, "features": [ "option_static_remotekey" ], "funding": { "local_funds_msat": "3000000000msat", "remote_funds_msat": "0msat", "pushed_msat": "0msat" }, "msatoshi_to_us": 3000000000, "to_us_msat": "3000000000msat", "msatoshi_to_us_min": 3000000000, "min_to_us_msat": "3000000000msat", "msatoshi_to_us_max": 3000000000, "max_to_us_msat": "3000000000msat", "msatoshi_total": 3000000000, "total_msat": "3000000000msat", "fee_base_msat": "1000msat", "fee_proportional_millionths": 1, "dust_limit_satoshis": 546, "dust_limit_msat": "546000msat", "max_htlc_value_in_flight_msat": 18446744073709551615, "max_total_htlc_in_msat": "18446744073709551615msat", "their_channel_reserve_satoshis": 30000, "their_reserve_msat": "30000000msat", "our_channel_reserve_satoshis": 30000, "our_reserve_msat": "30000000msat", "spendable_msatoshi": 2968347000, "spendable_msat": "2968347000msat", "receivable_msatoshi": 0, "receivable_msat": "0msat", "htlc_minimum_msat": 0, "minimum_htlc_in_msat": "0msat", "minimum_htlc_out_msat": "1msat", "maximum_htlc_out_msat": "2970000000msat", "their_to_self_delay": 144, "our_to_self_delay": 360, "max_accepted_htlcs": 30, "state_changes": [], "status": [ "CHANNELD_AWAITING_LOCKIN:Sent reestablish, waiting for theirs" ], "in_payments_offered": 0, "in_msatoshi_offered": 0, "in_offered_msat": "0msat", "in_payments_fulfilled": 0, "in_msatoshi_fulfilled": 0, "in_fulfilled_msat": "0msat", "out_payments_offered": 0, "out_msatoshi_offered": 0, "out_offered_msat": "0msat", "out_payments_fulfilled": 0, "out_msatoshi_fulfilled": 0, "out_fulfilled_msat": "0msat", "htlcs": [] },

I provided the details of one of them as there is so much information...I will be fine with joining a video call or audio chat to get to the bottom of this if preferred.

Thank you again for all your hard work.

Sincerely,

[jairunet]

Hello @ShahanaFarooqui,

Regarding this issue I believe is more a CLN related issue, because even after removing RTL completely I can see the same issue with other lightning applications such as spark

Or CLN UI

Maybe I should bring this issue to CLN instead, I let you close it if that makes sense to you?

Sincerely,

[saubyk]

hi @jairunet looking this issue again.
Can you please confirm if the funds which were used to attempt to open these channels are visible to you on the onchain page of RTL?

It appears that these channel opens never confirmed. I was able to scrape a transaction id from the screenshot above and couldn't find it in the mempool.

If the funds are back in your wallet, the only logical action left to take is to forget these channels via a cli command.

[jairunet]

@saubyk Hi there,

I will check that option via the CLN CLI see if I can clear them out without affecting the one lightning channel valid with the same peer.

I will update.

Sincerely,

[saubyk]

@jairunet please make sure of two things for each of your channels that you'll attempt to forget:

1. Lookup funding tx id for the channel opening transaction in the mempool and ensure that it's not mined
2. The funds which were used to open the channel are back in your node's wallet and accessible

Please note that forget operation would be irreversible and you will lose all the references to the pending channel open after.

[jairunet]

Hello @saubyk I believe I will hold for the mining fees to go lower (if there is such an event) then just close the channel I have with the peer LND that is currently using this channel with my lightning node, I think it is better for me to close it, verify that the sats left from the mutual channel returns to the on-chain wallet of my node, close/forget the other fake/never confirmed channels and the re-open the channel towards that node again clean slate.

I will post the final results here and close this issue if that solves the problem I have once and for all.

Thank you for all your great help and suggestions.

Sincerely,

[jairunet]

@saubyk I proceeded to close all the channels related to the peer that also have those other three channels:

I am not sure if this will clear itself out after a few days.

Your advice will be really appreciated.

Sincerely,

[saubyk]

Hi @jairunet are you able to track the channel closing transactions in the mempool?
Keep in mind that since it's a unilateral close, you will get your funds back on your node after the CSV delay.

[jairunet]

Hello @saubyk I am trying this at the command line:

# lightning-cli dev-forget-channel 0381dbc5290c1c5a02693be10093e45029044fdcdffe5011a17fecb96281c5d202 { "code": -1, "message": "Multiple channels: please specify short_channel_id" }

But these three (3) channels were never opened or seen in the bitcoin blockchain:

And if I check the details of one of the channels I can see that there was never a short channel ID:

I just don't understand why they just don't get cleared...

Thank you for your patience and help!

[saubyk]

@ShahanaFarooqui is there a way to derive short channel id from the channel id?

[ShahanaFarooqui]

@saubyk @jairunet As per bolt standard described here:
Short channel id is a combination of block height + transaction index + output index. But if the funding transaction has not reached in the mempool, we cannot create the short channel id.

As per Rene Pickhardt:
"As the funding_txid is a hash of its fields, this has no relation with the short channel id. So without looking at the blockchain data, I believe it is impossible transfer the short_channel_id to the channel_id or back."

Also please confirm that you are following CLN troubleshooting section for channel awaiting locking and forget channels:
https://docs.corelightning.org/docs/faq#my-channel-is-stuck-in-state-channeld_awaiting_lockin
https://docs.corelightning.org/docs/faq#how-to-forget-about-a-channel

I will check what else can be done in this case and get back to you in a week.

For posterity: How to get short channel id if the funding transaction is in the mempool (testnet example):
For funding txid: e18c228f1741e6f4c82ccb87256a809bc2714090b6158f805bf94a542a3e3be3

• [Go to] https://mempool.space/testnet/tx/e18c228f1741e6f4c82ccb87256a809bc2714090b6158f805bf94a542a3e3be3
• This transaction is in block 2338594.
• Open block details
• Transaction index for the transaction is 6 (index starting from 0).
• And finally output index is 0.
• So the short channel id will be 2338594x6x0

[jairunet]

Hello @ShahanaFarooqui unfortunately we are pretty stuck at the moment. I closed the channel I had with that LND node to see if the other three (never confirmed / nor registered in the bitcoin blockchain) would clear out and no luck, the channel with the peer cleared fine but the other three "orphans" (if I can call them that) remained, I tried those two ways:

Also please confirm that you are following CLN troubleshooting section for channel awaiting locking and forget channels:
https://docs.corelightning.org/docs/faq#my-channel-is-stuck-in-state-channeld_awaiting_lockin
https://docs.corelightning.org/docs/faq#how-to-forget-about-a-channel

But again, the channels never got a short channel ID, and for the forget channel it wants a peer ID (which I have) and also the channel ID, but in short format.

For the time being I will hold and come back see if there are any suggestions to clear those annoying channels...

Thank you very much @ShahanaFarooqui and @saubyk for the help with this strange scenario.

Again, this is one of the issues we run into when opening channels between different lightning implementations CLN + LND.

Sincerely,

[rustyrussell]

OK, so you missed the existence of the "channel_id" parameter to dev-forget-channel? You want:

lightning-cli dev-forget-channel id=PEERID channel_id=CHANNELID force=true

Hope that helps!

[jairunet]

@rustyrussell an honor again! and thank you so much for taking the time to check my issue, I can now see what I was missing in my previous tries, your solution did the trick! below is the output from one of the channels I applied your recommendation on:

# lightning-cli dev-forget-channel id=0381dbc5290c1c5a02693be10093e45029044fdcdffe5011a17fecb96281c5d202 channel_id=6eb1a9dd7b3be91ce77a5382c948c9d997e5d4e1354543b4d8c267cdf722f90e force=true

{
"forced": true,
"funding_unspent": false,
"funding_txid": "0ef922f7cd67c2d8b4434535e1d4e597d9c948c982537ae71ce93b7bdda9b16e"
}

Solving this was a nightmare for me, I really appreciate your clarification and solution once again, all the very best!

NOTE: the issue can be now close ^^

Sincerely,

[jairunet]

@rustyrussell @ShahanaFarooqui @saubyk Thank you all for your excellent support.

Closing,