Set custom onchain fees for channel closing

[pabpas]

We are already able to set a feerate with fundchannel but close is missing this feature. According to blockstream's explorer I just overpaid by 743% the fees to close a channel.

lightning-cli feerates perkb
{
   "perkb" : {
      "urgent" : 62372,
      "normal" : 1012,
      "slow" : 1012,
      "min_acceptable" : 1012,
      "max_acceptable" : 623720
   },
   "onchain_fee_estimates" : {
      "opening_channel_satoshis" : 177,
      "mutual_close_satoshis" : 170,
      "unilateral_close_satoshis" : 9324
   }
}

How could I have estimated what fee will be paid? In the end it was 7943sats (58sat/B).

[ZmnSCPxj]

This is not an easy feature to make.

For a unilateral close, the feerate is controlled by the funder of the channel. In current C-Lightning (and other implementations), this feerate is updated slowly, because we often get coordination problems with other implementations around this feerate if we update them too quickly (leading to channel closures when onchain feerates change). The last feerate proposed by the channel funder is the one that is used, and a unilateral close command cannot change the feerate (since it uses a stored commitment transaction that is updated only when the channel funder says to update the feerate). So if you do a unilateral close while we are slowly moving the feerates (which we have to do slowly because coordination between various implementations about how tight the feerates should be is so fraught) then you can only get a not-very-updated feerate.

For a mutual close, if I remember correctly the funder begins at its latest feerate, and both sides negotiate a feerate by providing initial feerate estimates, then providing proposals to each other until they converge on an agreed feerate. So in this case, you can affect the feerate (by starting your initial feerate at your desired feerate), but the other side can still play hardball and only advance by 1 satoshi at a time. Note that the funder is the one that indicates the desired feerate first, so the funder can't know if the other peer will bid higher or lower, so it can't actually overbid in an attempt to "pull" the other side to the proper feerate.

Was this a mutual close or a unilateral close? If it was unilateral then nothing can be done about feerates at all: you're bound to our slow update (which we are bound to because otherwise channels will get mass-disconnected between C-Lightning, lnd, and Eclair). If it was mutual then it is possible to affect but not set the feerate.

[pabpas]

Thanks a lot for the detailed explanation. It was mutual close and I was the funder. One thing that I still don't get, when you say that it is possible to "affect the feerate (by starting your initial feerate at your desired feerate)", the initial feerate is the one I used with fundchannel? Or something I can set before closing the channel?

[ZmnSCPxj]

It is something that is transmitted during the mutual close protocol. The mutual close protocol starts with the two sides giving their initial desired fee for the mutual close, then as long as they are different, they keep proposing fees (which must move closer, by at least one satoshi, towards the other side) until they both propose the same fee.

Currently we just use the current slowly-moving feerate as the initial feerate to compute the initial fee in the mutual close protocol. We could add an option to a mutual close to change the feerate that will be used to compute the initial fee for the mutual close ceremony, but do note the protocol means the other side will still affect the final feerate and you will never achieve the actual feerate you indicate in a mutual close.

[whitslack]

@ZmnSCPxj: Is it possible to abort the mutual close protocol (leaving the channel open and unaffected) if the negotiated fee is still too high? Sometimes I might want to close a channel opportunistically but could put it off until later if the peer still wants to use a higher fee than I would prefer.

[ZmnSCPxj]

The mutual close protocol is started by a shutdown message. This means that adding new HTLCs to the channel is stopped (CHANNELD_SHUTTING_DOWN state), and the nodes then wait for any existing HTLCs to be either resolved or failed (moving into CLOSINGD_SIGEXCHANGE once there are no more HTLCs pending). Once the sides start proposing feerates for the mutual close it is too late: the channel is now banned from adding new HTLCs.

Further, once we are in CHANNELD_SHUTTING_DOWN I believe we send a permanent channel failure on routing attempts via the specified channel, which means that routemaps of other nodes will delete the channel already when they attempt to route through the channel being mutually closed. At least C-Lightning itself does this.

This can be done if we have a new message like cancel_shutdown which means that the shutdown dance is cancelled, but that is a protocol change.

[whitslack]

@ZmnSCPxj: Is it possible to query what the peer thinks would be a good mutual closing fee prior to sending a shutdown message?

My node's onchain_fee_estimates.mutual_close_satoshis is always far less than I end up paying for a mutual close, so much so that the estimate is effectively a joke.

[ZmnSCPxj]

Currently not possible without a protocol change.

[hosiawak]

I noticed my node did a unilateral close and paid 1,313.5 sat/vB – 236,758 sat ($38.17) in onchain fees for a 2.8 mil sat channel. This wasn't even a period of high onchain fees.

You can see this tx at the top in this block

Is there any setting I can amend to relax this calculation and tell lightningd "Relax, we're not in a rush, use economy fees" ?

Why are channel closing fees an order of magnitude higher on average than channel opening fees?

[whitslack]

@hosiawak: Anchor outputs are supposedly going to mostly remedy this problem by letting your node choose a fee rate at the time of channel closing rather than at the time of commitment signing. In theory it should be possible to bump the transaction that spends the anchor output too, though I haven't seen whether C-Lightning is going to implement that. (It does no RBF anywhere yet that I've seen.)

[hosiawak]

@whitslack Thanks but I don't really understand why my node chose to use the highest fees in a block if this was a unilateral close? Why was it in such a hurry to close the channel ? I'm trying to understand the logic behind this behaviour and find a workaround. Either the current behaviour is completely unreasonable and pathological or I'm missing something.

[whitslack]

@hosiawak: Both: you're missing something and it's completely unreasonable. What you're missing is that the fee rate for unilateral channel closures isn't decided at the time of the closure; it's decided at the time your node was last in communication with your peer. If that was at a time when the mempool was extremely congested, then the commitment transactions (one of which is what your node broadcasts to unilaterally close the channel) would use a much higher fee rate than might be necessary at the time your node actually broadcasts one of them. It's not ideal (okay, completely unreasonable), but Lightning is a work in progress, and this will get better.

[hosiawak]

@whitslack Thank you for the explanation, I'm trying to wrap my head around the fee estimation and what it's based on. Sorry for asking daft questions but I still don't get why a unilateral close uses very_urgent which is even higher than a penalty (urgent). What's the risk of using the slow estimate for a unilateral close ?

I changed lightningd code to use stash->slow for every fee category except for penalty (which uses stash->urgent), max_acceptable is stash->slow * bitcoind->max_fee_multiplier. I recompiled lightning, setup a new node and opened a new channel to my current node.

The modified node used 4 sats/vbyte for the opening transaction.

Here is feerates perkb from this modified node:

{
   "perkb": {
      "opening": 3020,
      "mutual_close": 3020,
      "unilateral_close": 3020,
      "delayed_to_us": 3020,
      "htlc_resolution": 3020,
      "penalty": 106236,
      "min_acceptable": 1508,
      "max_acceptable": 30172,
      "urgent": 3020,
      "normal": 3020,
      "slow": 3016
   },
   "onchain_fee_estimates": {
      "opening_channel_satoshis": 530,
      "mutual_close_satoshis": 508,
      "unilateral_close_satoshis": 451,
      "htlc_timeout_satoshis": 500,
      "htlc_success_satoshis": 530
   }
}

And here from the current, unmodified node:

{
   "perkb": {
      "opening": 41692,
      "mutual_close": 41692,
      "unilateral_close": 106640,
      "delayed_to_us": 41692,
      "htlc_resolution": 62108,
      "penalty": 62108,
      "min_acceptable": 1508,
      "max_acceptable": 1066392,
      "urgent": 106640,
      "normal": 41692,
      "slow": 3016
   },
   "onchain_fee_estimates": {
      "opening_channel_satoshis": 7316,
      "mutual_close_satoshis": 7014,
      "unilateral_close_satoshis": 15942,
      "htlc_timeout_satoshis": 10294,
      "htlc_success_satoshis": 10915
   }
}

The site I always use when trying to manually estimate a fee before sending transactions without overpaying currently recommends 3 sats/vbyte for the fastest fee and 1 sat/vbyte as normal. So even though I used the slow estimate I overpaid 3 sats/vbyte :)

Is there any risk to running a node using such slow estimates for everything except for penalty ?

[whitslack]

@hosiawak: If your fee estimates differ too much from those of other peers in the network, then you won't be able to maintain open channels, as nodes are programmed to unilaterally close channels when the two ends disagree by too much on what the fees should be.

Opening channels has never been an issue. The fundchannel command lets you choose a fee rate. I always use 2222 sat/kvB, which is very cheap.

[hosiawak]

@whitslack So the only risk to using slow fee estimates is increased chances of triggering a unilateral close ? Notice that min_acceptable is set to slow / 2 so assuming other nodes use similar calculation for min_acceptable I should be good most of the time ? In the worst case scenario my node triggers unilateral close for channels opened by me which disagree with other nodes wrt fees but these unilateral closes don't overpay 100x in fees so it's not emptying my pockets as quickly as the current setup ?

If that's the case and this is the only risk then I still don't understand why unilateral close uses very urgent estimate and quite urgent for everything else - it doesn't make sense to me. Either the feerates coming from bitcoind are way too conservative or the target block is. For example currently the mempool is completely empty (1700 unconfirmed), 1 sat/vbyte are clearing and my unmodified node estimates 106504 sats/perkb for unilateral close which means if a unilateral close got triggered now I would have paid ~ 100 sats/vbyte for it (overpaying 100x). Opening estimates 36104 sats/perkb so I overpay 36x when opening regular channels using the default feerate (not specifying custom feerate).

I closed the channel from my modified node to my unmodified node and the closing fee was 4 sats/vbyte. If I didn't use the modified node the agreement would settle at the current mutual_close estimate which is 36104 sats/perkb (~36x overpaid).

Would it be possible to introduce --feerates=economy or something similar that uses much more economical estimates or is it a bad idea ?

[whitslack]

@hosiawak: I don't know what all the risks are, but I assume there must be significant risks in having a commitment transaction delayed for too long since it is constructed with urgent fee rates by design.

I agree with you that Bitcoin Core's fee estimation logic is total garbage. I've never understood why it produces such horrible estimates or why the Core devs think it's passable as is. I've even had it return a higher fee rate for longer block targets than it does for shorter ones. That's just broken.

I also agree that running a Lightning node is a quick way to get poor. Sometimes I wonder if the designers of Lightning actually run big nodes themselves or are somewhat disconnected from the expensive realities of their theoretical ideas.

[hosiawak]

Pluggable feerate estimators would be awesome. I could write an estimator based on mempool.space for example (or any other you might find useful) and forget about this issue.

[hosiawak]

@ZmnSCPxj Do you know why unilateral close uses very_urgent estimate and what the risk of using slow for everything is ?

[ZmnSCPxj]

Because of this theft vector:

• Select a victim node among your peers, one with a high capacity towards you (and ideally who opened the node to you, but this is not needed and you could have originally opened the channel to that peer yourself).
• Generate a preimage and its hash.
• Route an HTLC (or more) for the above preimage from yourself to the node back to yourself via the victim->you channel, with as much of the value in the channel in the HTLC(s) as you can get away with.
• Do not claim the HTLC(s) offchain.
• Go offline while fees are low. This leaves both you and the victim with a low-fee unilateral close transaction.
  • <<<<<
• Wait for a high-congestion mempool / when fees are high.
• Wait for the victim to drop the low-fee unilateral close onchain.
  • The victim does this when its outgoing HTLC to you times out.
  • This ties up the channel UTXO with a low-fee transaction spending it that is not going to confirm anytime soon. It cannot be RBFed since it requires both signatures. It cannot be CPFPed by the victim because the entire amount is in HTLC and it holds the timelocked branch and the timelock is still in the future.
• Hope that the high fees persist until after the HTLC times out.
  • As Bitcoin becomes popular, this becomes more likely!
• When the HTLC times out and you get your outgoing HTLC back, CPFP the unilateral close HTLC output with the preimage hashlock branch to get your funds onchain.
  • Since the victim dropped its unilateral close onchain, it cannot CPFP-RBF, since it uses the HTLC-timeout transaction that requires both signatures.
  • However, since the unilateral close was from the victim, you can CPFP-RBF the hashlock branch as it is not encumbered since the unilateral close was from the victim and not you.

The above is chancy and not 100% assured of success but we think it can happen often enough to be lucrative.

The point at which I marked "<<<<<" is where the mitigation we use is inserted: even if the current fees are low, we still make unilateral closes high-fee, because the attacker can go offline and prevent unilateral closes from being updated to use higher fees later.

The reason why we use an even higher fee for unilateral closes vs. penalty is that penalties are one-sided (do not require signatures from both parties) while unilateral closes need to be presigned by both parties. The penalty transaction can use the next-block estimate for right now, the unilateral close has to use the next-block estimate for what if suddenly the fees go high right after the peer goes offline.

The general philosophy here is "a million bitcoins in defense, and not a sat in tribute", i.e. we are willing to lose money in general operation rather than let even a few satoshi get stolen by the above (or any) attack.

---

Re: pluggable estimators: this risks mass closure if your estimator diverges from the estimators others use. This is because, even if we are not the ones paying for the close (it is the one opening the channel that has to pay for the close, pre-anchor-commitments) we need to ensure that the unilateral close transaction can be confirmed in a timely manner even if the above attack is attempted (and the victim is not the initiator of the channel --- nothing in the above requires the victim to be the initiator, the attack only works better if the victim is also the one paying for fees (i.e. is the channel initiator) for pouring even more salt on the wound, but does not require them to be the one paying for fees).

Thus, every node will check that the fees you set on channels you opened "looks reasonable" to them. If they think it is unreasonable, they unilaterally close right now rather than risk time passing too much and possibly making your low fee estimate even worse for them.

If everyone uses drastically different fee estimators, then every onchain fee movement is going to cause a lot of channel closures. This can turn into a positive feedback loop: sudden fee spike causes some (but not all) fee estimators to suddenly increase their fee estimates while others are still thinking "low fees" to disagree and cause channel closure, which further congests blockchain for further fee spike so that estimators that were barely agreeing get even further into disagreement and trigger more closures, and wham Lightning Network shuts down and everybody cries because their favorite blockchain has gigabytes of mempool.

C-Lightning has an option to skip this check --ignore-fee-limits, but (1) it is all your peers that have to turn this on if you are using a divergent fee estimator or else they will close the channels you open to them when your fee estimate diverges too much from theirs, and (2) if you look at the lightningd --help output you will see we helpfully added a "(DANGEROUS)" tag on the help text of that option, and that tag does not exist in any other option --- you are basically saying to trust each and every node that opens a channel to you not to perform the above attack.

[whitslack]

"a million bitcoins in defense, and not a sat in tribute"

That's a great philosophy to hold against attackers. I am solidly opposed to giving a mouse a cookie, paying ransoms, negotiating with terrorists, etc. But in my real-world experience, unilateral channel closures overwhelmingly occur due to bugs either in my node's software or in my peers' software.

Are we just punting for now, admitting that using Lightning is going to be a losing proposition versus the blockchain for the foreseeable future?

[ZmnSCPxj]

But in my real-world experience, unilateral channel closures overwhelmingly occur due to bugs either in my node's software or in my peers' software.

It is difficult to judge if a bug comes into a codebase from incompetence or malice. Just as bug fixes can be secretly inserted into regular refactorings to fix CVE-level bugs, CVE-level bugs can also be secretly inserted into regular refactorings. Worse, anyone can add this to their own node software without publishing it anywhere else, so you will not know if an arbitrary node you make a channel with is buggy or is trying to attack you.

Are we just punting for now, admitting that using Lightning is going to be a losing proposition versus the blockchain for the foreseeable future?

No. There is a reason why anchor commitments is being worked on, and that is to allow unilateral closes to be at minimum allowed relay feerate, with extra outputs (the "anchors" in "anchor commitments") to allow either node to CPFP (or CPFP-RBF) either commitment at the next-block feerate at the time it is dropped onchain, instead of the current situation where we have to speculatively overprice every commitment transaction because it is hard to predict the future and whether fees will be high or low later.

There are reasons why we have to move to anchor commitments and not just add workarounds to the current commitment scheme, I described one of them.

[hosiawak]

@ZmnSCPxj Thanks for a thorough explanation. It makes more sense now. I think I understand the risk better and can make better decisions wrt setting custom fees.

[ZmnSCPxj]

Yes, onchain fees have been a headache since massive deployment of Lightning. Anchor commitments seems the best way forward, but of course there is the question of "what about existing channels that were created pre-anchor commitments?" There is discussion about whether we can upgrade "seamlessly" a pre-anchor-commitment channel to post-anchor-commitment channel: https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-July/002763.html

[hosiawak]

@ZmnSCPxj Are you aware of any risks of changing just the normal opening fee estimates from normal to slow ?

Context: for the past week my node opened ~30 channels and for 90% of funding transactions overpaid between 13x and 41x in channel opening fees (eg. paying 13sats/vbyte when 1sats/vbyte were being confirmed in the next block).

This is what 4 ECONOMICAL sent to bitcoin core estimatesmartfee suggested:

34.7 sats/vbyte in block 656522 (confirming 1 sat/vbyte txs) = 34x overpaid
23.1 sats/vbyte in block 656577 (8 sats/vbyte) = 2.8x over
32 sats/vbyte in block 656692 (4 sats/vbyte) = 8x over
32 sats/vbyte in block 656707 (1 sats/vbyte) = 32x over
41 sats/vbyte in block 656867 (1 sats/vbyte) = 41x over
13 sats/vbyte in block 657070 (1 sats/vbyte) = 13x over

I know I could have used a custom fee when opening but I use clboss on this node which uses the default feerates in c-lightning when opening channels (I think, correct me if I'm wrong here please).

Could we add a knob for the target block for channel opening fees or will this cause issues when coordinating with other nodes as well ? It could default to the current value of 4 but allow node operators to relax their channel opening time preference.

I don't know what algorithm core uses to estimate the fee but it doesn't seem to be able to estimate the actual, lowest fee that gets you into the next X block (at least not in the current, low fee market) but rather tries to play it safe and overshoots by an order of magnitude even with its "ECONOMICAL" setting.

If there's no risk in using low end fees for channel opening then perhaps defaulting to 12, 24 or even 100 blocks would be better (it doesn't matter much if I have to wait an hour or 10 or 72 hours for a channel to open) It's a long lived thing anyway.

[ZmnSCPxj]

The risk is that if a feerate spike occurs just after we broadcast, the low-fee funding transaction could remain in the mempool. In the worst case, the funding transaction could remain in the mempool longer than two weeks.

Now, let us move our attention to the acceptor of the channel. In general, we accept anyone opening a channel to us, especially in the current situation of single-funding, where it is the initiator of the channel open that is the only one who puts funds in the channel, which removes a lot of risk (particularly, since only the initiator funds it, the acceptor has no risk even if the channel funding is delayed in confirmation, as it has no funds locked in a low-fee transaction). However, it can be an annoyance if a bunch of nodes propose channels to our node, completing the opening sequence, but ultimately not getting their funding transactions confirmed in the end, this would be a trivial way to cheaply load our node with useless channels in its database (create a new random pubkey, connect, propose channel, give a random number as funding txid, disconnect, repeat). So the acceptor of the channel forgets about the channel after two weeks.

This is the function which tells an acceptor when to forget:

lightning/lightningd/channel_control.c

Lines 782 to 789 in cd7d5cd

	/* Check if we are the fundee of this channel, the channel
	* funding transaction is still not yet seen onchain, and
	* it has been too long since the channel was first opened.
	* If so, we should forget the channel. */
	static bool
	is_fundee_should_forget(struct lightningd *ld,
	struct channel *channel,
	u32 block_height)

The default max_funding_unconfirmed is 2016, and if you compile DEVELOPER=1, can be overridden with --dev-max-funding-unconfirmed-blocks.

So if we underestimate channel funding fees and end up having a funding transaction that fails to confirm after two weeks, the acceptor of the channel will forget it anyway, so once it confirms, you can only unilateral close in your side and be unable to use the channel. (this holds even if we rebroadcasted the funding tx, by the time it confirms, the acceptor will say "wut?" when we try to bring up the channel with them, and we still cannot do anything than unilaterally close it).

What we should really do is support some kind of CPFP+RBF as proposed in #475 , or RBF as proposed in #668 (but note that this latter requires either a protocol change (that is now available with dual-funding protocol, but might not be easy to use/access over the current CL RPC yet) or C-Lightning has to support multiple channels per peer). Start with a low-fee funding tx, but if that fails to confirm quickly, bump up its fee (by either RBFing an CPFP of the funding tx, or RBFing the funding tx directly).

Maybe CLBOSS can increase its onchain remainder, and start with a low-fee funding tx (it always leaves a small amount of funds onchain anyway, so funding txes would always have a change output it can CPFP onto, and the CPFP tx can be RBFed later), but that would require reimplementing a good part of multifundchannel, and saving the CPFP+RBF information in its database. As-is, CLBOSS tries to only open channels during low-fee periods.

[ZmnSCPxj]

@hosiawak this PR also describes an actual problem encountered by @darosior with too-low feerates, which is another reason why CLBOSS prefers the "normal" feerate: #3910

In particular, the minimum relay feerate of a bitcoind can change. So you might sample the minimum relay feerate of the bitcoind (I believe the slow feerate tracks this pretty closely), do some processing (talking to peer and asking to make channels with them, agreeing on a txid), then because CLBOSS is such a CPU hog (LOL), by the time you produce the transaction, the minimum relay feerate of the bitcoind instance is now higher than what you sampled, meaning it will not even put it in your local mempool, much less broadcast it over the network. And changing the feerate of a transaction changes its txid, so you have to renegotiate the txid of the channel, which was not possible in single-funding (funding v1) and requires the new dual-funding (funding v2).

[hosiawak]

@ZmnSCPxj Thanks for another thorough explanation. In the meantime I changed the channel open fee to use 40 ECONOMICAL and recompiled lightningd. Knowing all the risks you mentioned above, I still think it would be useful to have knobs for these settings, at least until RBF/CPFP/anchor commitments find their way into the codebase.

[hosiawak]

LND is working on it too.

[3nprob]

I may have glanced over it being mentioned, but I still see an argument for adding the ability to manually set a fee manually for the close clommand on a per-tx basis. Consider the following scenario:

• I initiated a channel with peer X
• All the capacity is still on my side
• X becomes permanently offline
• On-chain fees have changed dramatically since the last point of contact

I now need to do a unilateral close.

I don't see why I couldn't explicitly set a specific feerate for this unilateral close-tx only, without changing the general close-fee advertised to other peers.

FWIW, lnd does allow doing this through lncli closchannel --sat_per_byte

[whitslack]

@3nprob: A unilateral closure uses a transaction that is signed by your peer. If your peer is permanently offline, then you can't obtain a signature from them on a new transaction with a new fee rate; you have to use whatever you already have from them. That's one reason unilateral closes are so expensive: they have to work when you need them, and there's no possibility to set the fee rate at that time. That should change with anchor outputs.

[3nprob]

I hope I'm not coming off as obtuse, but how come LND does it then? Also, is there a way to preview the fee that would be used for the specific channel before executing it?

EDIT: looking in the database, am I corect in understanding that channel_feerates.feerate_per_kw is the fee that will be used for a forced close?

[whitslack]

how come LND does it then?

It doesn't. lncli closechannel initiates a mutual closing. And even then, the final fee rate is still subject to negotiation with the peer. If you have to force a closing, LND has only one option of fee rate: whatever was used in the commitment transaction that was last signed by the peer.

[3nprob]

Makes sense!

Looking in the database, am I corect in understanding that channel_feerates.feerate_per_kw is the fee that will be used for a forced close?