[bug]: Node not Using Correct Fee Estimates During Channel Commitment Updates

[kornpow]

Issue Description:
User kornpow is encountering issues with on-chain fee rates during channel commitments in different backend setups including bitcoind backend alone and in combination with various feeurls. Despite providing substantially overestimated fees through a custom feeurl, the node seems to be attempting to use significantly lower fee rates than provided, resulting in repeated warnings in the logs. The log messages indicate that the update fee is outside the expected range, and it appears the node is not utilizing the fee rates obtained from the fee backend correctly.

Backends used:
bitcoind (CONVERVATIVE)
bitcoind + LL feeurl
bitcoind + custom fee url configured to overpay
btcd

Details:
Summary from slack thread: https://lightningcommunity.slack.com/archives/C6BDA6DGE/p1695219625262109
Conversation Summary:
kornpow reported repeated warning messages in their logs regarding update fees being outside the expected range in different backend setups.

Peer error message:
err=update_fee 988 outside range 1904-90800 (currently 1760)

kornpow provided details of the fee rates returned by their custom feeurl and expressed confusion as to why the node seems to be offering much lower fees to peers. They also provided some logs from querying the Web API for fee related messages which seemed to return reasonable values.

alexbosworth mentioned the existence of an "Estimate Fee API" available in walletRpc as EstimateFee, but noted that LND doesn't produce its own fees, it relies on the fee backend, except for statically defined fees. They suggested increasing the logging levels and focusing on the LNWL subsystem to trace the issue.

Following alexbosworth's advice, kornpow increased the logging levels and focused on the LNWL subsystem. Kornpow later shared more detailed logs, including those showcasing warning messages from a peer regarding the update fee being outside the expected range.

Kornpow attempted to trace back the issue in the code, focusing on how and where the update_fee lightning messages are generated, and shared links to the relevant sections in the LND GitHub repository.

Logs and Error Messages:

2023-09-20 15:29:36.260 [WRN] LNWL: Web API does not have a fee rate for target=3, using the fee rate for target=2 instead
2023-09-20 15:29:36.260 [DBG] LNWL: Web API returning 5500 sat/kw for conf target of 3
2023-09-20 15:29:36.260 [DBG] HSWC: ChannelLink(fb89447df5981ac360242810be90d9b6218e459cc383a9914e282d085a4fbef4:1): sampled fee rate for 3 block conf: 5500 sat/kw
2023-09-20 15:29:36.260 [INF] HSWC: ChannelLink(fb89447df5981ac360242810be90d9b6218e459cc383a9914e282d085a4fbef4:1): updating commit fee to 5500 sat/kw sat/kw

The fee rates from the custom feeurl provided by kornpow were:

"fee_by_block_target": {"2":45000, "4":43000, "8":39000, "32":20000, "98":11000}

This was also a red flag:

>>> lnd.estimate_fee(2).sat_per_kw
5500
>>> lnd.estimate_fee(2).sat_per_kw/253
21.73913043478261

Since 5500sat_per_kw == 21.7 sats/vbyte and my fee url is proposing 45 sats/vbyte
Kornpow is actively trying to track the issue in the code, focusing on the parts of the code that handle fee estimation and the sending of update_fee lightning messages.

Action Required

Assist kornpow in tracing the root cause of the issue in the code.
Help identify why the node is not utilizing the correct fee estimates from the backend during channel commitment updates.
Possibly look into the channel link files to check the process of sending update_fee messages and how the fee rates are being determined.

Requested Assistance

Further guidance in tracing the issue in the code.
Expert opinion on possible areas in the code that might be causing the discrepancy in fee estimates being used for channel updates.

Links to Possibly relevent Code Sections

Estimator.go
Update_fee.go
Link.go

Background

Describe your issue here.

Your environment

• version of lnd
  0.16.4

Steps to reproduce

This warning message causes CLN to disconnect from me. I can do connect_peer api, over and over, and have the same error occur, where it tries to use extremely low fees. In addition, on reboot it attempts to negotiate and disconnects.

Expected behaviour

On re-connection it should get the fee from the chain backend or feeurl, and update the channel commitment with a reasonable value.

Actual behaviour

A mysterious fee is used to update_fee the channel commitment, which is unclear where the fee comes from. The CLN channel partner can see the fee is way below the mempool minimums, and disconnects.

[ziggie1984]

Thanks for this detailed explanation I think I traced down the error in this issue:
#7805

I think the problem you are facing is, that your channel is drained locally and the default max-fee-allocation size is 0.5 which can lead to case (pretty likely cases when your channel is drained locally) that the feerate goes all the way down to 253sats/kw although your Fee-Estimator is proposing whay higher FeeRates as seen in your case.

[ziggie1984]

Regarding the code the main problem is here:

https://github.com/lightningnetwork/lnd/blob/master/lnwallet/channel.go#L8421

and resulting in the problem here:

https://github.com/lightningnetwork/lnd/blob/master/lnwallet/channel.go#L8457-L8458

The PR fixes the Problem and could need some Review.

So the fee-estimation is right, but LND calculates the IdealFeeRate and takes into account the FeeAllocation Parameter.

[kornpow]

Thanks @ziggie1984 I will attempt to lower that parameter, and see if it can temporarily fix the issue... At least until my channel gets even more depleted and it locks up again.

I will also review your PR soon

[ziggie1984]

You need to set it to 1 to be sure you do not lower the feerate beyond the old_feerate, lowering the value would even mean you reach lower proposing fees sooner. But keep in mind you will not fix exisiting issues with your channels as of today. The channelstate already progressed, and you need to hope for your Peer to either patch their CLN version (LND does not check the fee-sanity) or weight until the feerates for your Peer reach the proposed levels by yourself which can happen depending on how low the proposing fee was.

[kornpow]

I tried setting it max-channel-fee-allocation=1 which should turn off the check, and it didnt seem to help.

[ziggie1984]

Yeah unfortunetly channels which are already messed up cannot be repaired but it will prevent future fee-disagreements.

[JavierRSobrino]

This thing has made 4 of my channels go inactive. They connect and reconnect every few minutes, but fees don't change and the other peers refuse to renegotiate them for some reason, or perhaps it is me. I have changed allocation to 1, but no help. Logs:

2024-04-22 22:19:29.296 [WRN] HSWC: ChannelLink(XXX:7): received warning message from peer: chan_id=YYY, err=update_fee 253 outside range 1990-2880830 (currently 77071)

What do you recommend me to do?