sweep: add support for multiple fee preferences to UtxoSweeper

[wpaulino]

In this PR, we introduce support for arbitrary client fee preferences when accepting input sweep requests. This is possible with the addition of fee rate buckets. Fee rate buckets are buckets that contain inputs with similar fee rates within a specific range, e.g., 1-10 sat/vbyte, 11-20 sat/vbyte, etc. Having these buckets allows us to batch and sweep inputs from different clients with similar fee rates within a single transaction, allowing us to save on chain fees. Implementing these fee rate buckets also lays down the groundwork required for being able to bump the fee rates for particular inputs.

We also get rid of the UtxoSweeper's default fee preference. Any clients using it to sweep inputs specify the same default fee preference to not change their behavior. Each of these can be fine-tuned later on given their use cases.

[joostjager]

An alternative implementation is to add a MultiFeeSweeper that creates an instance of the original unchanged UtxoSweeper per fee preference.

[wpaulino]

PTAL @joostjager.

[joostjager]

Could the whole idea of conf targets be moved outside of UtxoSweeper?

[wpaulino]

As in allow the sweeper to only take in fee rates, so the caller would be responsible for mapping a target to a fee rate?

[Roasbeef]

IMO it make sense to have it live here. Callers are expressing their time preference w.r.t their expectation of when the sweep would happen. Pushing this mapping on the callers would just add code duplication at each call site, when it can instead all be consolidated here.

[wpaulino]

Yeah it's necessary to have it done at the UtxoSweeper level anyway in order to get the confirmation target.

[joostjager]

I was thinking about a wrapper struct around the UtxoSweeper for callers that want to specify a conf target. So callers still don't need to do the mapping themselves.

UtxoSweeper isn't the easiest code to understand, so splitting off what's possible may help.

But not a big issue.

[joostjager]

I think we have to be realistic at follow up, pure refactor prs. They are mostly not going to happen. Also we would incur a risk twice. First changes in this pr and when that is all good, we change the system again with new risks.

Imo it is better to try to do it first time right. Less overall costs. We surely don't want to over-engineer, but the size of the sweeper as it currently is justifies some effort in separation responsibilities. Costs over the life time of the software of future devs trying to understand code adds up.

An alternative design that could be good is to offer inputs to UtxoSweeper with a GetFeeRate callback. That way callers can pass in a function that either returns a static fee rate, a fee rate calculated from a conf target or something more advanced that also takes into account the value of the input for example. For UtxoSweeper it'll be all the same. When it sweeps, it just collects fee rates and does its clustering magic.

For conf targets, there could be an object feeEstimationCache that caches the fee rate estimates for the block. The getFeeRate() callback for an input could be something like this closure:
func() {return feeEstimationCache.GetFeeRate(defaultCommitSweepTarget)}

[Roasbeef]

IMO we shouldn't push all this burden on the caller. The goal is to be able to allow the caller to express a fee preference. In the end, that preference may not be exactly met, but within a ball park range, particularly if there's a slightly different mapping that results in lnd paying lower fees over all. The code in question as is is also just a few lines: calling to the estimator. If we want to add some more magic between the caller and the estimator, then we can do that at the estimator interface level.

If we look at the current use cases within lnd, every time a caller goes to request that something be swept, it has a rough time frame w.r.t when it should be swept. Scenarios such as sweeping an HTLC have a very clear time frame (to avoid the race), while presences such as sweeping a CSV output on a confirmed output don't (as long as it isn't a revoked commitment ;)).

Yeah agree that it would be a nice separation of concerns.
but the size of the sweeper as it currently is justifies some effort in separation responsibilities.

The prior approach had an even more direct separation of concerns, but here we are now. Let's not forget the reality of things as they exist today: chain fess are rising, and users need to be able to specify a time preference. Even our own systems need this capability in order to scale and be sustainable. We have an opportunity to get something incrementally better in (atm we basically have nothing) before the next scheduled major release. Lets focus on the use cases at hand that we've experienced the need for first had (and has also been requested time after time by users of lnd).

An alternative design that could be good is to offer inputs to UtxoSweeper with a GetFeeRate callback

In the near future, do we ever see this being anything other than the default? All our current use cases just want to express a preference, and have the fee rate be loosely adjusted (there're no guarantees after all) if that is at risk of not being achieved. Down the line if we have a compelling use case, this can be added in: for these special inputs, the sweeper skips its regular mapping and uses that function instead.

[joostjager]

The code in question as is is also just a few lines: calling to the estimator.

It is more than a few lines, it is also the re-targeting part. I think an object shouldn't have too many responsibilities and the conf target mapping is something that can be extracted. But in the end it comes down to a believe about what is the best design.

In the near future, do we ever see this being anything other than the default?

As I now understand it, there are two implementations of this callback: either conf target based (returns a diff. fee rate every block) and a static fee rate (passed in on the cmd line).

[halseth]

I believe this could be both simplified and made more safe for many of our use cases by specifying an absolute block height conf target instead the current conf target in number of blocks. Say we need a sweep to confirm by a specific height, the sweeper would start out low and increase the feerate as the target approaches.

Most important IMO is to adjust the fee rate dynamically, which could be one of the really powerful usecases of a sweeper like this.

[Roasbeef]

The clients themselves can target an absolute height by adjusting their specified relative conf target based on the current height. So if it's block 15, and they want this confirmed by block 25, then they can use a conf target of 10 (although things won't exactly match up).

[Roasbeef]

Likely can be deferred as a follow up, but we may also eventually want to introduce the concept of "strict buckets". As an example, a strict bucket of 1 sat/byte would allow a caller to express that they only want that input swept at that specified fee rate and care less about batching.

[wpaulino]

PTAL @joostjager @Roasbeef.

[joostjager]

I am not sure if we should do this already now. I'd be worried that on average users are going to end up paying more. Is this an actual problem to solve? To me it seems that being able to adjust the conf target or the fee rate directly is enough control.

Suppose the conf target is 6. We published with a fee that should get is confirmed after max 6 blocks. Suppose it would happen after exactly 6 blocks, which is within what is specified. We wouldn't need to do anything. But this logic starts bumping the fee up already after 1 block to the 5 block fee estimate.

[wpaulino]

It's definitely a bit aggressive at the moment, but the functionality itself is still useful. Perhaps a better approach is to only start aggressively bumping after a positive delta (e.g., 2 blocks) from the intended confirmation height?

[joostjager]

Yes, so I am arguing that it may be too aggressive. If I pay for 6 confs and after 1 block I already become nervous and up the fee, isn't that strange?

We should also not lose sight of the problem to solve. Is it really this strict enforcement of a deadline or do we just want to get some more control over the sweeper process without hard guarantuees?

[wpaulino]

If I pay for 6 confs and after 1 block I already become nervous and up the fee, isn't that strange?

That's why the threshold exists. The point is to increase the fee rate only if we need to in order to meet the confirmation target. If the fee rate doesn't change, then we won't update it.

The problem to solve is that confirmation targets won't be retried when scheduling/performing a sweep. If fees rise, the sweeper would handle that. With the buckets however, confirmation targets aren't retried. We could keep the behavior the same by querying for the same confirmation target like it previously would, but it's also about enforcing what "confirmation target" actually means. The current behavior doesn't hold the guarantees the term implies.

[halseth]

It could be useful to distinguish inputs that we want to be swept within X blocks, and inputs that has to be swept within X blocks (absolute block height).

[joostjager]

I think we have to be realistic at follow up, pure refactor prs. They are mostly not going to happen. Also we would incur a risk twice. First changes in this pr and when that is all good, we change the system again with new risks.

Imo it is better to try to do it first time right. Less overall costs. We surely don't want to over-engineer, but the size of the sweeper as it currently is justifies some effort in separation responsibilities. Costs over the life time of the software of future devs trying to understand code adds up.

An alternative design that could be good is to offer inputs to UtxoSweeper with a GetFeeRate callback. That way callers can pass in a function that either returns a static fee rate, a fee rate calculated from a conf target or something more advanced that also takes into account the value of the input for example. For UtxoSweeper it'll be all the same. When it sweeps, it just collects fee rates and does its clustering magic.

For conf targets, there could be an object feeEstimationCache that caches the fee rate estimates for the block. The getFeeRate() callback for an input could be something like this closure:
func() {return feeEstimationCache.GetFeeRate(defaultCommitSweepTarget)}

[joostjager]

With the fee rate callback suggestion above, this would become bucket := s.bucketForFeeRate(input.getFeeRate())

[joostjager]

With the getFeeRate() suggestion, this would all be moved behind that call back and abstracted away from UtxoSweeper.

[halseth]

Could also be a method on the FeePreference struct.

[halseth]

This should be dymanic IMO. If the fee market changes drastically right after this fee rate is calculated, it will never be updated.

[wpaulino]

Yeah this is already the case if the client specifies a confirmation target. If a fee rate is specified, then it indicates the client would like to sweep only at that fee rate (or somewhere close due to the bucket structure).

[halseth]

I believe this could be both simplified and made more safe for many of our use cases by specifying an absolute block height conf target instead the current conf target in number of blocks. Say we need a sweep to confirm by a specific height, the sweeper would start out low and increase the feerate as the target approaches.

Most important IMO is to adjust the fee rate dynamically, which could be one of the really powerful usecases of a sweeper like this.

[halseth]

Using fixed buckets seems inherently limiting. For instance would you never sweep two inputs having feerates of 9 and 11 sat/b together, even though it could be cheaper to group them together in one 11 sat/b transaction.

Another approach would be to look at the lowest feerate input, and keep clustering it with other inputs as long as that is cheaper than sweeping them separately.

[wpaulino]

Yeah there are definitely multiple ways of clustering inputs that would produce more efficient sweeps depending on the scenario -- the current implementation only serves as a starting point. In the future, it'd be nice if the sweeper was aware of the best way to cluster inputs given different scenarios.

[Roasbeef]

One other relevant mempool level detail is that we can't add new unconfirmed inputs to an existing transaction. As a result, the transaction level clustering can't vary too much between retargets. With the bucket approach the transactions in each bucket can stay relatively stable, but say if our 1 sat/byte input gets bumped to 5 sat/byte, then it'll still be in the same transaction cluster but will push the actual fee rate of that bucket up a bit.

[halseth]

It could be useful to distinguish inputs that we want to be swept within X blocks, and inputs that has to be swept within X blocks (absolute block height).

[halseth]

Could also be a method on the FeePreference struct.

[Roasbeef]

The clients themselves can target an absolute height by adjusting their specified relative conf target based on the current height. So if it's block 15, and they want this confirmed by block 25, then they can use a conf target of 10 (although things won't exactly match up).

[Roasbeef]

Tentative LGTM 🐚

Will start to run some live tests on testnet+mainnet (mainnet is more interesting since fees are starting to fluctuate more) and will report back if anything stands out (using the other RPC to monitor sweeping progress).

[wpaulino]

Pushed a new version that removes re-evaluating fee rates for inputs with confirmation targets.

[wpaulino]

PTAL @Roasbeef @joostjager @halseth.

[joostjager]

Re-suggesting to add input.GetFeeRate(). Sweeper is already a large struct and imo it should be made smaller rather than larger if we change things.

[Roasbeef]

LGTM 🎉

Happy to move forward with this as is, to ultimately circle back with a retargeting algorithm once we vet a few of our current ideas in the wild in the current fe climate.

[joostjager]

Two non blocking comments left. I am happy with the scope of the PR as it is. LGTM

[joostjager]

If the fee estimate is currently above the maximum but will possible drop below later, we still want to return an error here? It means that resolvers go into an error state until lnd is restarted.

[joostjager]

Return value could just as well be an int or other identifier, as the actual value isn't used other than for grouping.