Batch opening channels: Transaction is confirmed but one of the channels does not open.

[Marimox]

Background

I tried to open 4 channels at once using the bos open command. The transaction was broadcast and confirmed, but three of the four channels opened, but one did not.

The transaction is shown below, and I assume that the index of the channel that did not open is 1.

https://mempool.space/tx/67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc

The three channels from indexes 2 to 4 are respectively as follows.

2. https://amboss.space/edge/798475239736541186
3. https://amboss.space/edge/798475239736541187
4. https://amboss.space/edge/798475239736541188

I can't see any channels or pending channels from my node, is there any way to recover this fund?

Your environment

• lnd v0.14.2-beta
• Linux umbrel 5.10.17-v8+ #1421 SMP PREEMPT Thu May 27 14:01:37 BST 2021 aarch64
• Bitcoin Core version v22.0.0
• Umbrel v0.4.15
• bos 11.52.1

Steps to reproduce

The following command was executed.

umbrel@umbrel:~$ bos open 02644f80b5d32ed9a9888690571159692a17d7ead7db2df5124a8e2a72a8447d30 --amount 5000000 0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3 --amount 3000000 022eb09a7993a0edde69537b420f4119c0de833e0ee47651753b46bf884db75235 --amount 3000000 029c5fd64570f2d40dfc2f683865c81e93269dc032416142de5e2e2461359ae34d --amount 3000000

Expected behaviour

After the funding transaction is broadcast and confirmed, all four channels open.

Actual behaviour

BOS output the following message.

-
  node:                        Prince
  channels_per_peer:           1.03
  is_accepting_large_channels: true
-
  node:              CoinGate
  channels_per_peer: 1.07
-
  node:                        Ngu Technologies
  channels_per_peer:           1.00
  is_accepting_large_channels: true
-
  node:                        PrestonPysh
  channels_per_peer:           1.01
  is_accepting_large_channels: true

opening_to:
  - Prince: 0.05000000
  - CoinGate: 0.03000000
  - Ngu Technologies: 0.03000000
  - PrestonPysh: 0.03000000


? Use internal wallet funds? Yes
? Chain fee per vbyte? 1

funding:
  - 0.03000000
  - 0.03000000
  - 0.03000000
  - 0.05000000

funding_detected: 67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc

transaction_id: 67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc

Transactions were confirmed, but as noted above, three of the four were able to open channels, but one did not produce anything.

[alexbosworth]

Check pending channels

[Marimox]

I checked by executing the following command, but there were no pending channels.

umbrel@umbrel:~$ docker exec lnd lncli pendingchannels
{
    "total_limbo_balance": "0",
    "pending_open_channels": [
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
    ],
    "waiting_close_channels": [
    ]
}

I asked the peer to check to be sure, but no channel or pending channel existed there either.

[alexbosworth]

Look in closed channels?

[Marimox]

Unfortunately, what is in the closed channels is from when it was closed before batch open with bos, and there was nothing associated with this transaction.

[alexbosworth]

I would look at the logs to see what happened, in theory LND would only broadcast the tx if all of the peers were OK to open

Maybe there should be an alternative model that moves this broadcast into bos itself, right now in the case of batch opening from a single node the publishing should be handled on the LND side

if you see the funds are in an output and you can't find the corresponding tx on your node, you might have to use rescuefunding in guggero chantools to get the funds out of there, and that requires your peer to cooperate

[Marimox]

By "log" you mean lnd.log? I have a backup log of when the phenomenon occurred, so I can present it. Do you have any appropriate words to grep?

Thanks for the suggestion on how to rescue the funds. Fortunately, I am in contact with my peers and will ask them for help.

[alexbosworth]

Yeah the lnd.log

I would look for related ids like their pubkey and the tx id

I added test cases to confirm the publish directives on the bos side and they look normal so I still feel like it may be an issue on the LND side, especially given that you didn't see any errors thrown back at you, but I can potentially address this with a new system to confirm that pending channels are present before any tx would be released to the mempool

[alexbosworth]

I pushed a new version of bos which restructures how opens are done by first waiting for all pending channels to appear before publishing the transaction, moving that publish step out of LND and into bos

[Marimox]

Thanks for the quick fix.

I searched lnd.log by transaction ID and the public key of the node that could not be channeled. The results are as follows.

umbrel@umbrel:~/umbrel/lnd/logs/bitcoin/mainnet$ cat lnd.log.backup | grep -e 67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc -e 022eb09a7993a0edde69537b420f4119c0de833e0ee47651753b46bf884db75235
2022-03-08 07:06:11.153 [INF] HSWC: ChannelLink(67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc:2): updating commit fee to 911 sat/kw sat/kw
2022-03-08 07:25:04.363 [INF] HSWC: ChannelLink(67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc:4): updating commit fee to 911 sat/kw sat/kw
2022-03-08 08:09:54.129 [INF] HSWC: ChannelLink(67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc:4): updating commit fee to 785 sat/kw sat/kw
2022-03-08 08:14:52.400 [INF] PEER: unable to read message from 022eb09a7993a0edde69537b420f4119c0de833e0ee47651753b46bf884db75235@10.21.21.11:53680: EOF
2022-03-08 08:14:52.401 [INF] PEER: disconnecting 022eb09a7993a0edde69537b420f4119c0de833e0ee47651753b46bf884db75235@10.21.21.11:53680, reason: read handler closed
2022-03-08 08:14:52.502 [INF] DISC: Removing GossipSyncer for peer=022eb09a7993a0edde69537b420f4119c0de833e0ee47651753b46bf884db75235
2022-03-08 08:14:59.545 [INF] SRVR: Finalizing connection to 022eb09a7993a0edde69537b420f4119c0de833e0ee47651753b46bf884db75235@10.21.21.11:51056, inbound=true
2022-03-08 08:14:59.556 [INF] PEER: Negotiated chan series queries with 022eb09a7993a0edde69537b420f4119c0de833e0ee47651753b46bf884db75235
2022-03-08 08:14:59.556 [INF] DISC: Creating new GossipSyncer for peer=022eb09a7993a0edde69537b420f4119c0de833e0ee47651753b46bf884db75235
2022-03-08 08:27:28.371 [INF] HSWC: ChannelLink(67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc:3): updating commit fee to 786 sat/kw sat/kw
2022-03-08 08:32:32.771 [INF] HSWC: ChannelLink(67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc:2): updating commit fee to 786 sat/kw sat/kw
2022-03-08 10:39:04.347 [INF] DISC: GossipSyncer(022eb09a7993a0edde69537b420f4119c0de833e0ee47651753b46bf884db75235): applying gossipFilter(start=2022-03-08 10:39:04.347543176 +0000 UTC m=+111454.453634303, end=2158-04-14 17:07:19.347543176 +0000 UTC)
2022-03-08 10:59:04.346 [INF] DISC: GossipSyncer(022eb09a7993a0edde69537b420f4119c0de833e0ee47651753b46bf884db75235): applying gossipFilter(start=0001-01-01 00:00:00 +0000 UTC, end=0001-01-01 00:00:00 +0000 UTC)
2022-03-08 11:37:41.831 [WRN] CHBU: Replacing disk backup for ChannelPoint(67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc:2) w/ newer version
2022-03-08 11:37:41.834 [WRN] CHBU: Replacing disk backup for ChannelPoint(67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc:4) w/ newer version
2022-03-08 11:37:41.835 [WRN] CHBU: Replacing disk backup for ChannelPoint(67a15827b95c68b7e708e82732ff33f92ce39c1547e09714acecc09789fcebfc:3) w/ newer version

[alexbosworth]

I would look more for the channel creation logs, maybe try to find what time that happened based on the tx time

[Marimox]

Unfortunately, the file I had backed up was off by a day and the logs from the time the transaction was executed were lost. I apologize for not being able to contribute.