Payments very rarely work

[nakoshi-satamoto]

I am not sure if it is just me or others also, but I have noticed that ever since updating to version 24 most payments fail.

I get alot of these messages, the htlc and node number varies though.

htlc 16 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)

then payment deadline expires.

I have even attempted to pay various invoices even with small payments, I also have some large channels (millions of sats) open with large and well connected nodes.

Posting this for visibility in case if others are experiencing this also. If there is something I can check to confirm if it is an issue with my node or corelightning I can do so.

[nakoshi-satamoto]

To clarify, it is uncertain if this is a problem wiht my node or something to do with core lightning.

[vincenzopalazzo]

What happens if you put disable-mmp inside your config file and restart your node?

[nakoshi-satamoto]

@vincenzopalazzo I was away from computer but I can try that now. Before I do that I want to know what that setting does. I am not finding any documentation on it. What is that setting?

[vincenzopalazzo]

the setting disable the multi path payment algorithm

[nakoshi-satamoto]

disable-mmp is not a recgonized command or config setting by my Core Lightning.

[vincenzopalazzo]

@nakoshi-satamoto is disable-mpp sorry

[kilrau]

@nakoshi-satamoto is disable-mpp sorry

Did this change anything for you? @nakoshi-satamoto

[nakoshi-satamoto]

@kilrau I actually did not need to.

Ever since I upgraded to core lightning verison 24.02 I kept getting alot of these errors in my logs for various channels. This also was an issue on V24.02.1

2024-04-15T04:27:36.026Z UNUSUAL lightningd: Bad gossip order: could not find channel 696011x284x1 for peer's channel update

After updating to the latest version of core lightning onto version 43.02.2 those gossip issues went away, transactions inbound and outbound are not failing anymore. Maybe it was related to gossip not working?

[vincenzopalazzo]

Maybe it was related to gossip not working?

probably yes

[nakoshi-satamoto]

No longer an issue

[BitcoinMechanic]

Did 24.02.2 fix this for you? I upgraded to that and I'm still having the same issue. (Pages of Bad gossip order....)

[nakoshi-satamoto]

On 2024-05-05 18:52, BitcoinMechanic wrote: Did 24.02.2 fix this for you? I upgraded to that and I'm still having the same issue. (Pages of `Bad gossip order....`)

I am on 24.02.2 and I still get those bad gossip order messages. I am able to do payments though.

[blapcity]

also on 24.02.02 with high - payment failure rate and bad gossip order messages

[crc32]

I'm also on 24.02.02 (using the BTCPay Docker image), and 100% payment failures unless paying direct channel partners, and tons of bad gossip order messages. Cannot query remote routes at all, even to targets that are direct partners of channel partners. Incoming payments and some through-routes seem to work though (the through routes may only be working to next-node termination though).

Example log entries:

2024-05-27T18:19:39.925Z INFO plugin-pay: cmd 34 partid 0: Payment fee constraint 2500msat is below exemption threshold, allowing a maximum fee of 5000msat
2024-05-27T18:21:25.798Z INFO plugin-pay: cmd 34 partid 0: Payment fee constraint 2500msat is below exemption threshold, allowing a maximum fee of 5000msat
2024-05-27T18:47:11.456Z INFO plugin-pay: cmd 34 partid 0: Payment fee constraint 2500msat is below exemption threshold, allowing a maximum fee of 5000msat

2024-05-27T18:21:25.803Z INFO plugin-clnrest.py: RPC Error: {'code': 210, 'message': 'Destination 027cd974e47086291bb8a5b0160a889c738f2712a703b8ea939985fd16f3aae67e is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 027cd974e47086291bb8a5b0160a889c738f2712a703b8ea939985fd16f3aae67e is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 500000msat}]}
2024-05-27T18:47:11.484Z INFO plugin-pay: cmd 34 partid 0: Destination 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1 is not reachable directly and all routehints were unusable.
2024-05-27T18:47:11.486Z INFO plugin-clnrest.py: RPC Error: {'code': 210, 'message': 'Destination 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1 is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1 is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 500000msat}]}

2024-05-27T18:47:55.749Z UNUSUAL lightningd: Bad gossip order: could not find channel 844051x1140x1 for peer's channel update
2024-05-27T18:47:55.749Z UNUSUAL lightningd: Bad gossip order: could not find channel 786203x525x0 for peer's channel update
2024-05-27T18:49:25.765Z UNUSUAL lightningd: Bad gossip order: could not find channel 746879x1261x2 for peer's channel update
2024-05-27T18:49:25.775Z UNUSUAL lightningd: Bad gossip order: could not find channel 820913x3568x0 for peer's channel update

UPDATE:
I added disable-mpp to the startup file, with no effect.

[nakoshi-satamoto]

I said that I can still do payments, I am ammending my claim. I can receive inbound payments no problem but I do have a high rate of failure for outbound payments unless it is a direct peer. Ever since I upgraded to version 24 I have noticed this and it also coinciding with the gossip error messages. I delete gossip database and it still keeps getting gossip errors.

[TTK-199837]

I am also experiencing this exact issue.

[rustyrussell]

DO NOT delete the gossip store, you will not be able to make payments any more. You've made the network disappear, and it can take a while to refill.

Are you missing gossip? A healthy node should look like this:

$ lightning-cli -H listchannels | grep -c ^short_channel_id
85921
$ lightning-cli -H listnodes | grep -c ^nodeid
9062

That is, about 85000 public channels and 9000 nodes.

[TTK-199837]

Well… in an attempt to make my core lightning node functional again I did delete the gossip store (but did save a backup! (This was… a week ago?)) should I just continue as is or cp gossip_store.backup to replace the “new” one? Have you not noticed any payment issues Rusty?

On Mon, Jun 3, 2024, at 9:26 PM, Rusty Russell wrote: *DO NOT* delete the gossip store, you will not be able to make payments any more. You've made the network disappear, and it can take a while to refill. Are you missing gossip? A healthy node should look like this: `$ lightning-cli -H listchannels | grep -c ^short_channel_id 85921 $ lightning-cli -H listnodes | grep -c ^nodeid 9062

`

…

That is, about 85000 public channels and 9000 nodes. — Reply to this email directly, view it on GitHub <#7180 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AND3BCSLAHAFJBKAP3S2W4LZFU6XHAVCNFSM6AAAAABFLB774SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBWGU3TSNZXGM>. You are receiving this because you commented.Message ID: ***@***.***>

[strnly]

Found this issue and could relate instantly.

Same situation for me. BTCPay Server, CLN.
Fat channels (7-12m sats) with major peers and payments to these peers fail 95%.
Failed payments to Boltz, Strike, WoS despite having fat channels with them.

Incoming payments, no problem at all. Outgoing, 95%+ failure.