Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adds fundmax flag to openchannel #4029

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

bjarnemagnussen
Copy link
Contributor

@bjarnemagnussen bjarnemagnussen commented Feb 25, 2020

Related to PR #4024, but reopened as a new PR as this feature has undergone major changes.

Description

This PR introduces a new flag fundall for openchannel, which when opening a new channel uses the whole available wallet balance for the channel capacity up to the allowed maximum funding limit (currently ~0.16 btc).

Motivation

See also #619

The command sendcoins has a flag sweepall that will spend all the coins from the wallet. However no such flag exists when opening new channels using openchannel.

Wanting to use the total available wallet balance in the local amount when opening a channel requires calculating the needed fee and subtracting it from the balance manually.

Implementation

A new coin select function CoinSelectUpToAmount has been introduced, which selects coins up to the requested amount, or below if there are not sufficient funds available. Fees and dust amounts are added or subtracted from this amount or the change output depending on the available funds, see also the new test cases.

Considerations

This implementation should be concurrent safe, becuase it locks the coins prior to selecting them, as currently done by the wallet assembler for the coin select subroutine.

Pull Request Checklist

  • If this is your first time contributing, we recommend you read the Code
    Contribution Guidelines
  • All changes are Go version 1.12 compliant
  • The code being submitted is commented according to Code Documentation and Commenting
  • For new code: Code is accompanied by tests which exercise both
    the positive and negative (error paths) conditions (if applicable)
  • Code has been formatted with go fmt
  • For code and documentation: lines are wrapped at 80 characters
    (the tab character should be counted as 8 characters, not 4, as some IDEs do
    per default)
  • Running make check does not fail any tests
  • Running go vet does not report any issues
  • Running make lint does not report any new issues that did not
    already exist
  • All commits build properly and pass tests. Only in exceptional
    cases it can be justifiable to violate this condition. In that case, the
    reason should be stated in the commit message.
  • Commits have a logical structure according to Ideal Git Commit Structure

rpcserver.go Show resolved Hide resolved
@halseth
Copy link
Collaborator

@halseth halseth commented Feb 25, 2020

The wallet/fundingmanager already has a subtractFees flag that can be used for this purpose:

SubtractFees bool

It should significantly reduce the complexity of this PR I think.

@bjarnemagnussen
Copy link
Contributor Author

@bjarnemagnussen bjarnemagnussen commented Feb 25, 2020

The wallet/fundingmanager already has a subtractFees flag that can be used for this purpose:

SubtractFees bool

It should significantly reduce the complexity of this PR I think.

I was trying to use this flag, but since the maximum funding amount is limited to the constant lnd.MaxFundingAmount, we don't always want to use it. It is only if the balance is less than lnd.MaxFundingAmount, and in special cases where the balance is just slightly greater than lnd.MaxFundingAmount but not enough to cover the fees. Then we actually will subtract the fee from the total balance which will give a funding amount a little less than this constant.

But I did use the CoinSelectSubtractFee function for inspiration a lot, and it may be that I should try to combine the new CoinSelectUpToAmount with it?

I saw there may also be an open TODO in that regard for the excess fee. Furthermore, CoinSelect does not make correct fee calculation if change amount is dust, which is solved by CoinSelectSubtractFee and CoinSelectUpToAmount.

@halseth
Copy link
Collaborator

@halseth halseth commented Feb 28, 2020

@bjarnemagnussen I'm not sure I understand the limitation of the current API. If you do a regular channel opening with the MaxFundingAmt value, it will open that channel if the balance is available for that channel size + the fee. If the balance is not available, one can use the subtractFees flag, wit a value set to the remaining wallet balance. Wouldn't this work?

@bjarnemagnussen
Copy link
Contributor Author

@bjarnemagnussen bjarnemagnussen commented Feb 29, 2020

@halseth This works well, and I have added a commit to do as you suggest! :) Hopefully this is something along the line of what you were thinking?

Mismatch between coin select functions

There is this little mismatch between CoinSelect and CoinSelectSubtractFees that probably should be considered.

The CoinSelect function always uses a fixed fee calculation for one input and one change output. Let's assume the balance is one satoshi greater than the sum of MaxFundingAmt and a fee for one input and no change output. Then the CoinSelect function will return an error for insufficient funds, and the CoinSelectSubtractFees function takes over.

Inside CoinSelectSubtractFees we use the whole balance but the smaller fee calculation for one input and no change output. The final output amount returned from CoinSelectSubtractFees will therefore be the smaller fee subtracted from the total balance, which will result in an amount one satoshi greater than the MaxFundingAmt.

An easy fix is to use a sanity check that simply makes sure that we do not return an amount greater than what we asked for. Any difference should be small (fee difference between one and no change output) and can directly go towards the fee.

Test cases

I have for now added a test case "WIP: mismatch of fee calculation between functions" that will pass due to the added sanity check explained above, but otherwise would give rise to an output amount greater than the specified maximum.

There is also an oddity regarding testing for dust change, as it depends on which branch inside CoinSelectUpToAmount is used. Since CoinSelectUpToAmount now makes use of CoinSelect and CoinSelectSubtractFees it depends on their different ways of handling dust change. I have removed testing for dust change as those should be handled by the other coin select functions. I have therefore also removed the test case "no amount to pay fee for change".

Copy link
Collaborator

@halseth halseth left a comment

This looks pretty good! Could an alternative way to communicate our intention to use all our funds to the wallet be a fundUpTo bool? That way we would reuse the localAmt field for our maximum channels size, but communicate with the boolean that we allow creating a smaller channel if the funds are not available.

We could also take this a step further and create a chanSizeType enum that would have values SubtractFees, FundUpTo and ExactChanSize.

cmd/lncli/commands.go Outdated Show resolved Hide resolved
rpcserver.go Show resolved Hide resolved
fundingmanager.go Outdated Show resolved Hide resolved
Copy link
Contributor Author

@bjarnemagnussen bjarnemagnussen left a comment

Thanks @halseth for your review and the helpful inputs!

Could an alternative way to communicate our intention to use all our funds to the wallet be a fundUpTo bool? That way we would reuse the localAmt field for our maximum channels size, but communicate with the boolean that we allow creating a smaller channel if the funds are not available.

I have been considering this. But there were a few things I fell over:

  • There is the minimum channel funding limit minChanFundingSize defined in lnd that we always want to exceed. And we may therefore want to be able to define a minimum value at the channel funding when fundAll is set.
  • When called from the RPC it must also be assured that the funding amount is at least the amount we will afterwards try to push to the remote (see comment with the code).
  • With a minimum channel funding limit set inside localAmt we keep the behaviour that the channel opening will fail with funds less than what is set with the localAmt.

I am therefore not sure if we can work around this in another way?

We could also take this a step further and create a chanSizeType enum that would have values SubtractFees, FundUpTo and ExactChanSize.

I like your idea of defining a chanSizeType enum for the different cases! But how it will look probably depends on if the minimum amount is needed or not from the discussion above.

rpcserver.go Show resolved Hide resolved
cmd/lncli/commands.go Outdated Show resolved Hide resolved
fundingmanager.go Outdated Show resolved Hide resolved
@bjarnemagnussen
Copy link
Contributor Author

@bjarnemagnussen bjarnemagnussen commented Apr 7, 2020

I am still not sure how to handle both the MaxFundingAmount and the minChanFundingSize better with the fundall flag. Any other idea on how to streamline this?

@Roasbeef Roasbeef added this to the 0.11.0 milestone Apr 14, 2020
@Roasbeef Roasbeef added v0.11 enhancement funding labels Apr 14, 2020
@cfromknecht cfromknecht added this to In progress in v0.11.0-beta via automation Apr 21, 2020
@cfromknecht cfromknecht moved this from In progress to Review in progress in v0.11.0-beta Apr 21, 2020
@Roasbeef Roasbeef requested review from guggero and removed request for cfromknecht Apr 23, 2020
@guggero
Copy link
Collaborator

@guggero guggero commented Apr 24, 2020

Could you rebase the PR please? I'm going to take over the review for @halseth.

I am still not sure how to handle both the MaxFundingAmount and the minChanFundingSize better with the fundall flag. Any other idea on how to streamline this?

Didn't read through all comments yet, but couldn't we just check the walletbalance from the command line and if it's higher than MaxFundingAmount we just say "wallet balance exceeds minimum channel size, --fundall flag cannot be used, please specify exact amount" or something?

@bjarnemagnussen
Copy link
Contributor Author

@bjarnemagnussen bjarnemagnussen commented Apr 27, 2020

I have rebased to the current v0.10.0-beta.rc5 master commit.

Just to make sure we expect the same behaviour I am listing some cases how the fundall flag should open channels:

  • In general if balance < minChanFundingSize: Fail just as is current behaviour for openchannel.
  • If balance < MaxFundingAmount + open-tx-fee: Open a channel by maxing out the capacity resulting in no change, equivalent to opening a channel using the whole balance with SubtractFees set.
  • If balance >= MaxFundingAmount + open-tx-fee: Open a channel using MaxFundingAmount with a change amount to the wallet.

Remark: In the current implementation an edge case can raise a bug resulting in a channel capacity greater than MaxFundingAmount. This is caught with a sanity check, see #4029 (comment) above.

Copy link
Collaborator

@guggero guggero left a comment

I looked at the PR now in more detail. I do see how it would be useful to just create a maximum channel, no matter how much is in the wallet. But perhaps it would then be better to call the flag --fundmax?

What does make this PR a bit more complicated is the handling of the push amount (initial remote balance). Maybe we should just disallow the combination of flags/parameters? Or work with a percent margin.

I'm going to test this a bit more tomorrow, so far I've only looked at the code.

cmd/lncli/cmd_open_channel.go Outdated Show resolved Hide resolved
rpcserver.go Show resolved Hide resolved
lnwallet/wallet.go Show resolved Hide resolved
@carlaKC carlaKC self-requested a review May 3, 2020
@guggero guggero self-requested a review May 6, 2020
@bjarnemagnussen
Copy link
Contributor Author

@bjarnemagnussen bjarnemagnussen commented May 6, 2020

I commented above before committing, so sorry for that!

I have renamed fundall to fundmax and added comments in the code to explain a bit more what is going on.

@guggero
Copy link
Collaborator

@guggero guggero commented Dec 7, 2021

@bjarnemagnussen cool, glad to hear you're willing to see this one through 😅 can you please re-request review once you were able to rebase? Thanks!

@guggero
Copy link
Collaborator

@guggero guggero commented Dec 13, 2021

!lightninglabs-deploy mute 2022-Feb-01

Allows to define a maximum amount to provision a channel opening with using a new field `FundUpToMaxAmt` on the `Request` struct. Also adds a new coin select function `CoinSelectUpToAmount` to select coins up to a maximum amount respecting a minimum amount.
Adds handling of the `FundMax` field when parsing an `OpenChannelRequest` with `rpcServer.parseOpenChannelReq`.
The fundmax must respect the maximum and minimum channel size parameter set by the user when committing to a new channel.
@bjarnemagnussen
Copy link
Contributor Author

@bjarnemagnussen bjarnemagnussen commented Dec 23, 2021

I have rebased this PR and added two commits to respect the reserve wallet balance in case of anchor channels.

One thing to consider is regarding the different commit types. Should the integration test also check the "lease" commit type, and should there be an integration test that fails whenever a new type is introduced that is not tested for to be sure the behaviour is always correct?

@saubyk
Copy link

@saubyk saubyk commented Jan 27, 2022

@guggero waiting eagerly for this feature.

@guggero
Copy link
Collaborator

@guggero guggero commented Jan 27, 2022

Okay cool. Feel free to test and leave a review to make sure this works as expected for your use case(s). Will try to pick it up this week too.

Copy link
Collaborator

@guggero guggero left a comment

We're getting pretty close, I think. The added complexity with the reserve balance does make everything a bit harder. I hope you understand why this PR undergoes so much scrutiny since it touches the core funding logic, where (as you've probably noticed) a lot of things can have an influence.

The integration tests look really good now, though! I don't think we need to also add a test with the new commitment type, since fee wise they look just like anchor channels.

}

// Inputs should be all funds in the wallet.
if len(fundingTx.TxIn) != len(test.coins) {
Copy link
Collaborator

@guggero guggero Jan 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a test case with multiple inputs so this check actually does make sense?

runTestCase := func(carol, dave *lntest.HarnessNode,
testCase *chanFundMaxTestCase) func(t *testing.T) {

return func(t *testing.T) {
Copy link
Collaborator

@guggero guggero Jan 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid this nesting and the code within reaching over the 80 character limit, we shouldn't return a function here, but instead just define a function that takes t, carol, dave, testCase and then call it in the test case loop below.

amount: 37000,
// The transaction fee to open the channel must be subtracted from
// Carol's balance (since wallet balance < max-chan-size).
carolBalance: btcutil.Amount(37000) - fundingFee(1, false),
Copy link
Collaborator

@guggero guggero Jan 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a test with a low balance and the anchor reserve, just to make sure the interplay here is correct.

@@ -103,8 +103,9 @@ var openChannelCommand = cli.Command{
cli.BoolFlag{
Name: "fundmax",
Usage: "(optional) if set, the wallet will attempt to " +
"commit the maximum possible local amount to the channel. " +
"Should not be set together with local_amt",
"commit the maximum possible local amount to " +
Copy link
Collaborator

@guggero guggero Jan 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole commit should be squashed with the respective commits where the initial changes were introduced.

if remoteInitialBalance >= funding.MinChanFundingSize {
// As default value for the minimum local amount use the larger
// of the amount to be pushed to the remote or the allowed
// minimum channel size.
Copy link
Collaborator

@guggero guggero Jan 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, these comments don't really help me to understand why something is done. They just describe the code below in words. We should add the context from the discussions in this PR instead. Otherwise someone looking at this in a year or so won't know what was discussed.

@@ -267,14 +277,30 @@ func CoinSelectUpToAmount(feeRate chainfee.SatPerKWeight, maxAmt,
selectedCoins, changeAmt, err := CoinSelect(
feeRate, maxAmt, dustLimit, coins,
)
if err == nil {
Copy link
Collaborator

@guggero guggero Jan 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole method is very hard to grasp without staring at it for a long time. Since this selects coins, it makes me very nervous.

So here two suggestions to streamline the code in this area.

  1. Use one switch statement:
switch {
case err == nil:

case err.(type) == *ErrInsufficientFunds:

default:

}
  1. Instead of setting the err to errReservedValueInvalidated, do the actual call to CoinSelectSubtractFees directly.

@@ -204,6 +237,13 @@ func testChannelFundMax(net *lntest.NetworkHarness, ht *harnessTest) {
if err != nil {
t.Errorf("dave's balance is wrong: %v", err)
}

if commitTypeHasAnchors(testCase.commitmentType) {
err = checkWalletBalance(carol, lnwallet.ReservedValue(1))
Copy link
Collaborator

@guggero guggero Jan 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@@ -739,6 +739,7 @@ func (l *LightningWallet) handleFundingReserveRequest(req *InitFundingReserveMsg

localFundingAmt := req.LocalFundingAmt
remoteFundingAmt := req.RemoteFundingAmt
hasAnchors := req.CommitType.HasAnchors()
Copy link
Collaborator

@guggero guggero Jan 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes to the wallet probably deserve their own commit.

@Roasbeef Roasbeef added this to In progress in v0.15.0-beta via automation Feb 2, 2022
@Roasbeef Roasbeef moved this from In progress to Review in progress in v0.15.0-beta Feb 2, 2022
@Roasbeef Roasbeef removed this from the v0.15.0 milestone Mar 23, 2022
@Roasbeef Roasbeef added this to the v0.16.0 milestone Mar 23, 2022
@Roasbeef Roasbeef removed this from Review in progress in v0.15.0-beta Mar 23, 2022
return fmt.Errorf("local amt argument missing")
}

// Note:
// The fundmax flag is NOT allowed to be combiend with local_amt above.
Copy link
Contributor

@sputn1ck sputn1ck May 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo (combiend -> combined)

@ziggie1984
Copy link
Contributor

@ziggie1984 ziggie1984 commented May 26, 2022

I was wondering why the author commit dates are not in a chronological order ? (does it come from rebasing single commits ?)

Bjarne Magnussen: 6d3de89b3: 5 months ago
Bjarne Magnussen: 1a9f2807b: 5 months ago
Bjarne Magnussen: 3ac041fcc: 1 year, 1 month ago
Bjarne Magnussen: 827685603: 5 months ago
Bjarne Magnussen: 2a2d6ca9c: 1 year, 1 month ago
Bjarne Magnussen: e9a0c1478: 1 year, 1 month ago
Bjarne Magnussen: 87b957a8b: 1 year, 1 month ago
Bjarne Magnussen: bb4013440: 1 year, 1 month ago

updateChan := make(chan *lnrpc.OpenStatusUpdate)

// Initiate a fund channel, and inspect the funding tx.
pushAmt := btcutil.Amount(0)
Copy link
Contributor

@ziggie1984 ziggie1984 May 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also test a pushamt greater than the MinChanFundingSize?

@ziggie1984
Copy link
Contributor

@ziggie1984 ziggie1984 commented May 26, 2022

will this PR also allow for --fundmax with wumbo channels, meaning that the UpperLimit is adjusted when wumbo channels are active ? (As far as I understood the code, its capped at the 2^24-1 limit)

@ziggie1984
Copy link
Contributor

@ziggie1984 ziggie1984 commented May 26, 2022

will this PR also allow for --fundmax with wumbo channels, meaning that the UpperLimit is adjusted when wumbo channels are active ? (As far as I understood the code, its capped at the 2^24-1 limit)

Justed tested it with wumbo channels works nice 👍

"fmt"
"testing"

"github.com/btcsuite/btcutil"
Copy link
Contributor

@hsjoberg hsjoberg May 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Erroneous import:

Suggested change
"github.com/btcsuite/btcutil"
"github.com/btcsuite/btcd/btcutil"

Also I'm no go expert but other test files contain //go:build xxx in the top:

+//go:build !rpctest
// +build !rpctest

RemoteAmt: req.RemoteFundingAmt,
LocalAmt: req.LocalFundingAmt,
FundUpToMaxAmt: req.FundUpToMaxAmt,
ReservedAmt: ReservedValue(numAnchorChans),
Copy link
Contributor

@hsjoberg hsjoberg May 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will lead to inconsistent behavior.
For private channels (i.e mobile wallets), we do not right now reserve any sats for anchor outputs.
If you specify the amount when opening a private channel, you'll be able to use all funds (no sats will be reserved).
Related commit: 8e1087d

Also related topic: #6574

Possible fix:

+isPublic := req.Flags&lnwire.FFAnnounceChannel != 0
[...]
-if req.FundUpToMaxAmt > 0 {
+if req.FundUpToMaxAmt > 0 && isPublic {
	numAnchorChans, err = l.currentNumAnchorChans()
	if err != nil {
		req.err <- err
		req.resp <- nil
		return
	}

	if hasAnchors {
		numAnchorChans++
	}
}

@lightninglabs-deploy
Copy link

@lightninglabs-deploy lightninglabs-deploy commented Jul 1, 2022

@bjarnemagnussen, remember to re-request review from reviewers when ready

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement funding P3
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants