Fee_insufficient errors for multiple route hints

[blockdyor]

I've been running for some tests my Core Lightning node for several months, receiving and sending Lightning payments almost without any issues. Initially, I resolved most sending problems by enabling xpay (xpay-handle-pay=true), which worked great for a while. However, after a few months of operation, I'm now experiencing payment failures even with xpay enabled.

Description

Payment attempts are timing out after disabling all route hints due to fee_insufficient errors, even though the routes should be viable.

Version

Core Lightning version: 25.05
Operating System: UmbrelOS
Bitcoin backend: Bitcoin Knots 29.1.knots20250903

Steps to Reproduce

Attempt to pay a Lightning invoice
Payment tries multiple route hints from the invoice
All route hints fail with fee_insufficient error
Payment times out after 8 (or so) attempts

Expected Behavior

Payment should succeed using at least one of the provided route hints
OR provide more informative error about why fees are insufficient
OR automatically adjust fee budget if within reasonable limits

Actual Behavior

Payment times out with the following errors:

Timed out after after 8 attempts. 
We got a weird error (fee_insufficient) for the invoice's route hint (908587x284x0/1): disabling it for this payment. 
We got a weird error (fee_insufficient) for the invoice's route hint (908535x115x4/1): disabling it for this payment. 
We got a weird error (fee_insufficient) for the invoice's route hint (918439x1623x1/1): disabling it for this payment. 
We got a weird error (fee_insufficient) for the invoice's route hint (918430x1647x1/1): disabling it for this payment. 
We got a weird error (fee_insufficient) for the invoice's route hint (912773x309x1/1): disabling it for this payment. 
We got a weird error (fee_insufficient) for the invoice's route hint (882499x639x0/0): disabling it for this payment. 
We got a weird error (fee_insufficient) for the invoice's route hint (882415x472x0/0): disabling it for this payment.

Environment Details

Payment amount: 100000 sats
Number of active channels: 5
Total outbound liquidity: 978K SATS
Fee limit settings: Stock (never changed)
Invoice: ()

Debug Logs

core-lightning_app_1               | > cln-application-backend@0.0.7 serve
core-lightning_app_1               | > node ./dist/server.js
core-lightning_app_1               | 
core-lightning_app_1               | {"level":"warn","message":"\tServer running at http://10.21.21.94:2103","timestamp":"2025-10-12 23:01:35:135"}
core-lightning_app_1               | {"level":"error","message":"\tCommando lightning error from pay command","timestamp":"2025-10-12 23:03:28:328"}
core-lightning_app_1               | {"level":"error","message":"\t{\"code\":209,\"message\":\"Timed out after after 10 attempts. We got a weird error (fee_insufficient) for 908587x284x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 908535x115x4/1: disabling it for this payment. We got a weird error (fee_insufficient) for 896092x194x2/1: disabling it for this payment. We got a weird error (fee_insufficient) for 901745x1880x2/0: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 912773x309x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 918439x1623x1/1: disabling it for this payment. \"}","timestamp":"2025-10-12 23:03:28:328"}
core-lightning_app_1               | {"level":"error","message":"\tController caught lightning error from pay: {\"message\":\"Timed out after after 10 attempts. We got a weird error (fee_insufficient) for 908587x284x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 908535x115x4/1: disabling it for this payment. We got a weird error (fee_insufficient) for 896092x194x2/1: disabling it for this payment. We got a weird error (fee_insufficient) for 901745x1880x2/0: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 912773x309x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 918439x1623x1/1: disabling it for this payment. \",\"code\":521}","timestamp":"2025-10-12 23:03:28:328"}
core-lightning_app_1               | {"level":"error","message":"\tTimed out after after 10 attempts. We got a weird error (fee_insufficient) for 908587x284x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 908535x115x4/1: disabling it for this payment. We got a weird error (fee_insufficient) for 896092x194x2/1: disabling it for this payment. We got a weird error (fee_insufficient) for 901745x1880x2/0: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 912773x309x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 918439x1623x1/1: disabling it for this payment. ","timestamp":"2025-10-12 23:03:28:328"}
core-lightning_app_1               | {"level":"error","message":"\tCommando lightning error from pay command","timestamp":"2025-10-12 23:07:49:749"}
core-lightning_app_1               | {"level":"error","message":"\t{\"code\":209,\"message\":\"Timed out after after 17 attempts. We got a weird error (fee_insufficient) for 908587x284x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 908535x115x4/1: disabling it for this payment. We got a weird error (fee_insufficient) for 896092x194x2/1: disabling it for this payment. We got a weird error (fee_insufficient) for 901745x1880x2/0: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 912773x309x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 918439x1623x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x1/0: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x2/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900170x1346x1/0: disabling it for this payment. We got a weird error (fee_insufficient) for 904255x590x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 898594x318x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 912921x397x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 918430x1647x1/1: disabling it for this payment. \"}","timestamp":"2025-10-12 23:07:49:749"}
core-lightning_app_1               | {"level":"error","message":"\tController caught lightning error from pay: {\"message\":\"Timed out after after 17 attempts. We got a weird error (fee_insufficient) for 908587x284x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 908535x115x4/1: disabling it for this payment. We got a weird error (fee_insufficient) for 896092x194x2/1: disabling it for this payment. We got a weird error (fee_insufficient) for 901745x1880x2/0: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 912773x309x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 918439x1623x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x1/0: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x2/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900170x1346x1/0: disabling it for this payment. We got a weird error (fee_insufficient) for 904255x590x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 898594x318x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 912921x397x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 918430x1647x1/1: disabling it for this payment. \",\"code\":521}","timestamp":"2025-10-12 23:07:49:749"}
core-lightning_app_1               | {"level":"error","message":"\tTimed out after after 17 attempts. We got a weird error (fee_insufficient) for 908587x284x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 908535x115x4/1: disabling it for this payment. We got a weird error (fee_insufficient) for 896092x194x2/1: disabling it for this payment. We got a weird error (fee_insufficient) for 901745x1880x2/0: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 899477x2478x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 912773x309x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 918439x1623x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x1/0: disabling it for this payment. We got a weird error (fee_insufficient) for 900347x116x2/1: disabling it for this payment. We got a weird error (fee_insufficient) for 900170x1346x1/0: disabling it for this payment. We got a weird error (fee_insufficient) for 904255x590x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 898594x318x0/1: disabling it for this payment. We got a weird error (fee_insufficient) for 912921x397x1/1: disabling it for this payment. We got a weird error (fee_insufficient) for 918430x1647x1/1: disabling it for this payment. ","timestamp":"2025-10-12 23:07:49:749"}
core-lightning_app_1               | {"level":"error","message":"\tCommando lightning error from pay command","timestamp":"2025-10-13 07:15:28:1528"}
core-lightning_app_1               | {"level":"error","message":"\t{\"code\":209,\"message\":\"Failed after 2 attempts. We got temporary_channel_failure for 913965x895x1/0, assuming it can't carry 426000msat. Then routing for remaining 425939msat failed: We couldn't quite afford it, we need to send 1050msat more for fees: please submit a bug report!\"}","timestamp":"2025-10-13 07:15:28:1528"}
core-lightning_app_1               | {"level":"error","message":"\tController caught lightning error from pay: {\"message\":\"Failed after 2 attempts. We got temporary_channel_failure for 913965x895x1/0, assuming it can't carry 426000msat. Then routing for remaining 425939msat failed: We couldn't quite afford it, we need to send 1050msat more for fees: please submit a bug report!\",\"code\":521}","timestamp":"2025-10-13 07:15:28:1528"}
core-lightning_app_1               | {"level":"error","message":"\tFailed after 2 attempts. We got temporary_channel_failure for 913965x895x1/0, assuming it can't carry 426000msat. Then routing for remaining 425939msat failed: We couldn't quite afford it, we need to send 1050msat more for fees: please submit a bug report!","timestamp":"2025-10-13 07:15:28:1528"}
core-lightning_app_1               | {"level":"error","message":"\tCommando lightning error from pay command","timestamp":"2025-10-13 07:26:31:2631"}
core-lightning_app_1               | {"level":"error","message":"\t{\"code\":209,\"message\":\"Timed out after after 30 attempts. We got temporary_channel_failure for 909180x3241x0/0, assuming it can't carry 99609912msat. We got temporary_channel_failure for 903948x262x1/1, assuming it can't carry 99634829msat. We got temporary_channel_failure for 906015x1272x0/1, assuming it can't carry 99766243msat. We got temporary_channel_failure for 904976x339x0/0, assuming it can't carry 99704539msat. We got temporary_channel_failure for 900927x1344x0/1, assuming it can't carry 99607053msat. We got temporary_channel_failure for 915136x2805x2/1, assuming it can't carry 99586439msat. We got temporary_channel_failure for 857576x3305x1/1, assuming it can't carry 99601377msat. We got temporary_channel_failure for 887458x2867x2/0, assuming it can't carry 99638797msat. We got temporary_channel_failure for 744237x730x1/1, assuming it can't carry 99631322msat. We got temporary_channel_failure for 871331x623x0/0, assuming it can't carry 99675536msat. We got temporary_channel_failure for 907407x1591x1/0, assuming it can't carry 99676675msat. We got temporary_channel_failure for 826444x1729x1/1, assuming it can't carry 99587131msat. We got temporary_channel_failure for 898345x682x16/1, assuming it can't carry 99675760msat. We got temporary_channel_failure for 880504x841x0/1, assuming it can't carry 99628957msat. We got temporary_channel_failure for 906289x2434x0/1, assuming it can't carry 99617587msat. We got temporary_channel_failure for 831975x1476x1/0, assuming it can't carry 99622667msat. We got temporary_channel_failure for 889791x126x0/1, assuming it can't carry 99903294msat. We got temporary_channel_failure for 847234x1457x1/1, assuming it can't carry 99588131msat. We got temporary_channel_failure for 883632x435x1/0, assuming it can't carry 99592505msat. We got temporary_channel_failure for 916208x575x0/1, assuming it can't carry 99603044msat. We got temporary_channel_failure for 875057x1836x1/1, assuming it can't carry 99620875msat. We got temporary_channel_failure for 874774x2944x1/0, assuming it can't carry 99675536msat. We got temporary_channel_failure for 874087x1297x0/1, assuming it can't carry 99656831msat. We got temporary_channel_failure for 897809x530x0/1, assuming it can't carry 99575061msat. We got temporary_channel_failure for 847741x2145x0/1, assuming it can't carry 99617007msat. We got temporary_channel_failure for 907056x995x1/0, assuming it can't carry 99675536msat. We got temporary_channel_failure for 847446x3498x0/0, assuming it can't carry 99706533msat. We got temporary_channel_failure for 917852x1913x1/0, assuming it can't carry 99632841msat. We got temporary_channel_failure for 779351x1815x0/1, assuming it can't carry 99586213msat. \"}","timestamp":"2025-10-13 07:26:31:2631"}
core-lightning_app_1               | {"level":"error","message":"\tController caught lightning error from pay: {\"message\":\"Timed out after after 30 attempts. We got temporary_channel_failure for 909180x3241x0/0, assuming it can't carry 99609912msat. We got temporary_channel_failure for 903948x262x1/1, assuming it can't carry 99634829msat. We got temporary_channel_failure for 906015x1272x0/1, assuming it can't carry 99766243msat. We got temporary_channel_failure for 904976x339x0/0, assuming it can't carry 99704539msat. We got temporary_channel_failure for 900927x1344x0/1, assuming it can't carry 99607053msat. We got temporary_channel_failure for 915136x2805x2/1, assuming it can't carry 99586439msat. We got temporary_channel_failure for 857576x3305x1/1, assuming it can't carry 99601377msat. We got temporary_channel_failure for 887458x2867x2/0, assuming it can't carry 99638797msat. We got temporary_channel_failure for 744237x730x1/1, assuming it can't carry 99631322msat. We got temporary_channel_failure for 871331x623x0/0, assuming it can't carry 99675536msat. We got temporary_channel_failure for 907407x1591x1/0, assuming it can't carry 99676675msat. We got temporary_channel_failure for 826444x1729x1/1, assuming it can't carry 99587131msat. We got temporary_channel_failure for 898345x682x16/1, assuming it can't carry 99675760msat. We got temporary_channel_failure for 880504x841x0/1, assuming it can't carry 99628957msat. We got temporary_channel_failure for 906289x2434x0/1, assuming it can't carry 99617587msat. We got temporary_channel_failure for 831975x1476x1/0, assuming it can't carry 99622667msat. We got temporary_channel_failure for 889791x126x0/1, assuming it can't carry 99903294msat. We got temporary_channel_failure for 847234x1457x1/1, assuming it can't carry 99588131msat. We got temporary_channel_failure for 883632x435x1/0, assuming it can't carry 99592505msat. We got temporary_channel_failure for 916208x575x0/1, assuming it can't carry 99603044msat. We got temporary_channel_failure for 875057x1836x1/1, assuming it can't carry 99620875msat. We got temporary_channel_failure for 874774x2944x1/0, assuming it can't carry 99675536msat. We got temporary_channel_failure for 874087x1297x0/1, assuming it can't carry 99656831msat. We got temporary_channel_failure for 897809x530x0/1, assuming it can't carry 99575061msat. We got temporary_channel_failure for 847741x2145x0/1, assuming it can't carry 99617007msat. We got temporary_channel_failure for 907056x995x1/0, assuming it can't carry 99675536msat. We got temporary_channel_failure for 847446x3498x0/0, assuming it can't carry 99706533msat. We got temporary_channel_failure for 917852x1913x1/0, assuming it can't carry 99632841msat. We got temporary_channel_failure for 779351x1815x0/1, assuming it can't carry 99586213msat. \",\"code\":521}","timestamp":"2025-10-13 07:26:31:2631"}
core-lightning_app_1               | {"level":"error","message":"\tTimed out after after 30 attempts. We got temporary_channel_failure for 909180x3241x0/0, assuming it can't carry 99609912msat. We got temporary_channel_failure for 903948x262x1/1, assuming it can't carry 99634829msat. We got temporary_channel_failure for 906015x1272x0/1, assuming it can't carry 99766243msat. We got temporary_channel_failure for 904976x339x0/0, assuming it can't carry 99704539msat. We got temporary_channel_failure for 900927x1344x0/1, assuming it can't carry 99607053msat. We got temporary_channel_failure for 915136x2805x2/1, assuming it can't carry 99586439msat. We got temporary_channel_failure for 857576x3305x1/1, assuming it can't carry 99601377msat. We got temporary_channel_failure for 887458x2867x2/0, assuming it can't carry 99638797msat. We got temporary_channel_failure for 744237x730x1/1, assuming it can't carry 99631322msat. We got temporary_channel_failure for 871331x623x0/0, assuming it can't carry 99675536msat. We got temporary_channel_failure for 907407x1591x1/0, assuming it can't carry 99676675msat. We got temporary_channel_failure for 826444x1729x1/1, assuming it can't carry 99587131msat. We got temporary_channel_failure for 898345x682x16/1, assuming it can't carry 99675760msat. We got temporary_channel_failure for 880504x841x0/1, assuming it can't carry 99628957msat. We got temporary_channel_failure for 906289x2434x0/1, assuming it can't carry 99617587msat. We got temporary_channel_failure for 831975x1476x1/0, assuming it can't carry 99622667msat. We got temporary_channel_failure for 889791x126x0/1, assuming it can't carry 99903294msat. We got temporary_channel_failure for 847234x1457x1/1, assuming it can't carry 99588131msat. We got temporary_channel_failure for 883632x435x1/0, assuming it can't carry 99592505msat. We got temporary_channel_failure for 916208x575x0/1, assuming it can't carry 99603044msat. We got temporary_channel_failure for 875057x1836x1/1, assuming it can't carry 99620875msat. We got temporary_channel_failure for 874774x2944x1/0, assuming it can't carry 99675536msat. We got temporary_channel_failure for 874087x1297x0/1, assuming it can't carry 99656831msat. We got temporary_channel_failure for 897809x530x0/1, assuming it can't carry 99575061msat. We got temporary_channel_failure for 847741x2145x0/1, assuming it can't carry 99617007msat. We got temporary_channel_failure for 907056x995x1/0, assuming it can't carry 99675536msat. We got temporary_channel_failure for 847446x3498x0/0, assuming it can't carry 99706533msat. We got temporary_channel_failure for 917852x1913x1/0, assuming it can't carry 99632841msat. We got temporary_channel_failure for 779351x1815x0/1, assuming it can't carry 99586213msat. ","timestamp":"2025-10-13 07:26:31:2631"}
core-lightning_app_1               | {"level":"error","message":"\tCommando lightning error from pay command","timestamp":"2025-10-13 07:38:25:3825"}
core-lightning_c-lightning-rest_1  |   address: '/root/.lightning/bitcoin/lightning-rpc'
core-lightning_c-lightning-rest_1  | }
core-lightning_c-lightning-rest_1  | Reading config file
core-lightning_app_1               | {"level":"error","message":"\t{\"code\":209,\"message\":\"Timed out after after 8 attempts. We got a weird error (fee_insufficient) for the invoice's route hint (908587x284x0/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (908535x115x4/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918439x1623x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918430x1647x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (912773x309x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882499x639x0/0): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882415x472x0/0): disabling it for this payment. \"}","timestamp":"2025-10-13 07:38:25:3825"}
core-lightning_app_1               | {"level":"error","message":"\tController caught lightning error from pay: {\"message\":\"Timed out after after 8 attempts. We got a weird error (fee_insufficient) for the invoice's route hint (908587x284x0/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (908535x115x4/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918439x1623x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918430x1647x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (912773x309x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882499x639x0/0): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882415x472x0/0): disabling it for this payment. \",\"code\":521}","timestamp":"2025-10-13 07:38:25:3825"}
core-lightning_app_1               | {"level":"error","message":"\tTimed out after after 8 attempts. We got a weird error (fee_insufficient) for the invoice's route hint (908587x284x0/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (908535x115x4/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918439x1623x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918430x1647x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (912773x309x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882499x639x0/0): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882415x472x0/0): disabling it for this payment. ","timestamp":"2025-10-13 07:38:25:3825"}
core-lightning_app_1               | {"level":"error","message":"\tCommando lightning error from pay command","timestamp":"2025-10-13 07:42:10:4210"}
core-lightning_app_1               | {"level":"error","message":"\t{\"code\":209,\"message\":\"Timed out after after 8 attempts. We got a weird error (fee_insufficient) for the invoice's route hint (908587x284x0/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (908535x115x4/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918439x1623x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918430x1647x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (912773x309x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882499x639x0/0): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882415x472x0/0): disabling it for this payment. \"}","timestamp":"2025-10-13 07:42:10:4210"}
core-lightning_app_1               | {"level":"error","message":"\tController caught lightning error from pay: {\"message\":\"Timed out after after 8 attempts. We got a weird error (fee_insufficient) for the invoice's route hint (908587x284x0/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (908535x115x4/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918439x1623x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918430x1647x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (912773x309x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882499x639x0/0): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882415x472x0/0): disabling it for this payment. \",\"code\":521}","timestamp":"2025-10-13 07:42:10:4210"}
core-lightning_app_1               | {"level":"error","message":"\tTimed out after after 8 attempts. We got a weird error (fee_insufficient) for the invoice's route hint (908587x284x0/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (908535x115x4/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918439x1623x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (918430x1647x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (912773x309x1/1): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882499x639x0/0): disabling it for this payment. We got a weird error (fee_insufficient) for the invoice's route hint (882415x472x0/0): disabling it for this payment. ","timestamp":"2025-10-13 07:42:10:4210"}
core-lightning_c-lightning-rest_1  | error: WARN: /root/.lightning/lightning-rpc is missing, using the bitcoin mainnet subdirectory at /root/.lightning/bitcoin instead.

[Lagrang3]

@endothermicdev, could this be an issue with gossip?

[Lagrang3]

All those channels failing are coming from the same node: https://amboss.space/node/03a93b87bf9f052b8e862d51ebbac4ce5e97b5f4137563cd5128548d7f5978dda9

Maybe we should add some bad score to nodes that fail often (like bias?) but with a timer so that they don't
get penalized forever.

[blockdyor]

After your comment I stopped CLN, deleted the gossip file, and retry to pay different invoices, even smaller amounts, but the issue persists. Should I wait several hours for the gossip file to fully regenerate?

[blockdyor]

All those channels failing are coming from the same node: https://amboss.space/node/03a93b87bf9f052b8e862d51ebbac4ce5e97b5f4137563cd5128548d7f5978dda9

Maybe we should add some bad score to nodes that fail often (like bias?) but with a timer so that they don't get penalized forever.

I don't think it's about a particular node/channel, it's probably more about my node because the same error comes out when doing different kind of payments with different routes. I even tried with just 5000 sats but same error.

[Lagrang3]

After your comment I stopped CLN, deleted the gossip file, and retry to pay different invoices, even smaller amounts, but the issue persists. Should I wait several hours for the gossip file to fully regenerate?

You shouldn't delete the gossip file.

[Lagrang3]

When you gossip syncs again, you could try paying with xpay but with that node disabled:

lightning-cli -k askrene-create-layer layer=disable-cyberdyne
lightning-cli -k askrene-disable-node layer=disable-cyberdyne node=03a93b87bf9f052b8e862d51ebbac4ce5e97b5f4137563cd5128548d7f5978dda9
lightning-cli -k xpay layers="[disable-cyberdyne]" invstring=...

Just a suggestion.
Since you are running on umbrel it could take a while until you can pull any patches.

If you test payments with that node disabled and you still get the same error let us know.

[blockdyor]

When you gossip syncs again, you could try paying with xpay but with that node disabled:

lightning-cli -k askrene-create-layer layer=disable-cyberdyne
lightning-cli -k askrene-disable-node layer=disable-cyberdyne node=03a93b87bf9f052b8e862d51ebbac4ce5e97b5f4137563cd5128548d7f5978dda9
lightning-cli -k xpay layers="[disable-cyberdyne]" invstring=...

Just a suggestion. Since you are running on umbrel it could take a while until you can pull any patches.

If you test payments with that node disabled and you still get the same error let us know.

Thank you @Lagrang3, it worked at first try!

[endothermicdev]

Having an incomplete set of gossip often looks like this, but since it's only one node, it could be that node at fault (their gossip not propagating, or maybe the node's changing fee rates too quickly.) @Lagrang3 I'd like to try to catch nodes with commonly failing outbound channels and start adding a negative bias for cases like this. For instance, when boltz has a plethora of channels, but none of them work outbound, we should probably not try dozens of times, but start biasing against routing through.

For syncing gossip, if your node has multiple peers and has been connected for an hour or so, it should have a fairly complete view. If the problem persists, you can increase the default number of gossipers with autoconnect-seeker-peers=15 in your config, but just leaving the node running for a while should take care of it with defaults.

[rustyrussell]

This looks like a weird invoice! Can you include a copy please? or email it to rusty@blockstream.com

[blockdyor]

This looks like a weird invoice! Can you include a copy please? or email it to rusty@blockstream.com

I can't locate the exact invoice, but most of the failures occurred while funding my LN Markets account. But I also think while moving funds between the node and wallets like WoS, Primal etc. When I generate an invoice for 100,000 sats on LN Markets, it looks like this:

lnbc1m1p50qz22pp5lveh5exglx593hfpyk9ujuxpngtkpzwyu68erhfw6ncl8qp9sc7qdzcf38zqntpwf4k2arnypzx2ur0wd5hggrxvdjrse3svgmj6cesxcej6dp4vgcj6wpcvdjz6ery8q6rzvp5vv6rvdescqzzsxqyz5vqrzjqvn3xwrr85kn0v59mtjd7s95z0vvd3u3l0h809authrssysed474crdm4vqqggcqqyqqqqlgqqqqqqgq2qsp5xpljv4y3vz0ztf5vysuhsza689qkj6v9lhq2l8py8nkf2qp8m34s9qxpqysgq64l3pr52nd9q32r2zh468mxgz22qkd7v6krqm7cmhpu4e32dje7y7mfs4y7lw0ur28vy0qyc7qgm43k2pmc2vvpnkjrd7mscprkrersqmy4tpr