New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Coin Selection with Murch's algorithm #10637
Conversation
89b1417
to
0246f1f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK, looks good to me. AFAICT, there's an issue with the lookahead that should be addressed still.
src/wallet/coinselection.cpp
Outdated
// Calculate remaining | ||
CAmount remaining = 0; | ||
for (CInputCoin utxo : utxo_pool) { | ||
remaining += utxo.txout.nValue; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you filtered utxo_pool to exclude utxo's that have a net-neg value? Otherwise you're underestimating the lookahead here. To get an accurate figure for what you may still collect downtree, you should only add utxo.txout.nValue >=0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gmaxwell has concerns that Core wallet is only doing semi-sane utxo handling by spending these. With exact match + sane backoff algorithm this concern may be alleviated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed, that may be a problem. I will add that in as it is still good to have additional checks here even if done elsewhere.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't have much of a concern here about the 0/negative effective value inputs: Failing to select negative effective value inputs for an exact match won't lead to a UTXO count inflation, because changeless transactions are by definition strictly UTXO reducing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@instagibbs: I'm not completely opposed to spending net-negative UTXO, my concern here is primarily that it actually may cause the lookahead to be underestimated causing valid solutions not to be found.
I realize now that the knapsack algorithm would also not select uneconomic UTXO anymore, as if it had selected enough value before it reached them it would have already returned the set, and if it actually starts exploring them, cannot add more value in the first place.
Advocatus Diaboli: Would it be that terrible though, if UTXO were only considered when they actually have a net positive value? During times of low fees, they'd be used both during BnB and knapsack, during times of high fees, they wouldn't bloat the blocks and lose their owner money.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not so concerned, was making sure concerns are brought up.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@xekyo we should assume that it would be terrible unless someone can show that it will not cause another massive UTXO bloat event... but thats offtopic here, as I don't think anyone has this concern with exact matches.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The utxos with negative effective values are filtered anyway in wallet/wallet.cpp
, which is the only place (except for tests) from where SelectCoinsBnB
is called.
|
||
// Select 5 Cent | ||
add_coin(4 * CENT, 4, actual_selection); | ||
add_coin(1 * CENT, 1, actual_selection); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAICT utxo_pool has : 4, 3, 2, & 1. Since you're exploring randomly selecting 5 then has two possible solutions: 4+1, 3+2.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is forced to be include first in these tests so the solution is deterministic.
add_coin(5 * CENT, 5, utxo_pool); | ||
add_coin(5 * CENT, 5, actual_selection); | ||
add_coin(4 * CENT, 4, actual_selection); | ||
add_coin(1 * CENT, 1, actual_selection); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Under above assumptions, there is two solutions here as well: 5+4+1, or 5+3+2.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is forced to be include first in these tests so the solution is deterministic.
src/wallet/wallet.cpp
Outdated
LogPrint(BCLog::SELECTCOINS, "total %s\n", FormatMoney(nBest)); | ||
} | ||
CInputCoin coin(pcoin, i); | ||
coin.txout.nValue -= (output.nInputBytes < 0 ? 0 : effective_fee.GetFee(output.nInputBytes)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems to me that you're also collecting coins that have a net-negative here. This will cause your lookahead to be underestimated, unless you cater to that case when calculating the remainder.
CoinSet actual_selection; | ||
CAmount value_ret = 0; | ||
|
||
///////////////////////// |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would perhaps add a test that checks what happens if the utxo_pool includes a UTXO that is more costly to spend than its own value. As far as I can tell, this would currently reduce your lookahead and may cause a premature search failure.
Have you tested the effect of random exploration vs largest first exploration?
I am not sure there is a significant privacy benefit for Random Exploration as for either selection method an attacker would already need to know about another eligible input that would achieve an exact match when switched out for one of the input set. What benefit do you expect from using Random Exploration? |
@xekyo I was thinking that Random Exploration would be better for privacy but I see that it probably wouldn't help. If you think it would be better to change to LFE, I can certainly do that. |
@achow101: I don't know how strong the effect is, but I'd expect Random Exploration to increase the required computational effort. |
Noting that this PR has fairly heavy overlap with #10360 . From chatting with @achow101 the intention of this PR is to touch as little as possible while still getting BranchNBound coin selection. To make this successful it should really only be run on first iteration of the loop in CreateTransaction, when |
This PR I believe will still create just-over-dust change outputs when BnB finds an exact match. Whenever we are allowing BnB matches(first iteration) we should not make change outputs less than the exact match slack value. |
src/wallet/wallet.cpp
Outdated
// Calculate cost of change | ||
// TODO: In the future, we should use the change output actually made for the transaction and calculate the cost | ||
// requred to spend it. | ||
CAmount cost_of_change = effective_fee.GetFee(148+34); // 148 bytes for the input, 34 bytes for making the output |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This assumes that the input will be spent at a feerate at least as high as the current. This was a valid assumption in my thesis, as I was using a fixed fee rate. I'm not sure whether this a valid assumption for realnet transaction selection, as we've literally seen fees between 8-540 sat/byte in the past two weeks. We might want to consider discounting the cost of the input slightly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Depends on user time preferences. Could be an option that is set for those who regularly consolidate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now I think it is fine to use the current feerate.
@instagibbs: In fact, BnB is designed to only work when creating a transaction without a change output. If we were creating a change in the first place, the extensive search pattern would be unnecessarily wasteful. |
To append onto my previous comments, any effective value match attempt should account for the fees just obtained by |
I have made the BnB selector to be only run on the first pass of the coin selection loop. It is now set so that effective value is only used for the BnB selector and not the knapsack one. I have also added the negative effective value check and test just as a belt-and-suspenders thing. I also made BnB use Largest First Exploration instead of Random Exploration. |
src/wallet/coinselection.cpp
Outdated
backtrack = true; | ||
} else if (value_ret >= target_value) { // Selected value is within range | ||
done = true; | ||
} else if (tries <= 0) { // Too many tries, exit |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here's a unexpected behavior in my algorithm: if there is a number of input combinations whose value_ret
all exceed the target_value when tries == 0
is passed, tries can go into the negative.
The tries check should be moved to the top of the checks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
Perhaps generically, we should never create change if the amount is less than the cost of creating + spending it (regardless of whether BnB was used to find the inputs or not)? |
@sipa one question is if we should allow the wallet to consider consolidation-level prices for that change. Perhaps the user is in a hurry now, but would consider spending that change at a much slower pace. Maybe for a first pass only consider the selected feerate, then Future Work allow a parameter which has more aggressive change protection given longer timescales. |
@instagibbs Yes, I agree; we should use long-eatimates for the spend part of change rather than the actual feerate the user is willing to pay now. Perhaps we can make it more conservative without doing that by using a factor 2 or 3 reduction? |
src/wallet/wallet.cpp
Outdated
// Calculate cost of change | ||
// TODO: In the future, we should use the change output actually made for the transaction and calculate the cost | ||
// requred to spend it. | ||
CAmount cost_of_change = effective_fee.GetFee(148+34); // 148 bytes for the input, 34 bytes for making the output |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not correct for segwit. If this code ends up being changed to follow pieter's suggestion of dividing the rate by two or three it should be bounded by the min relay fee. (I'm not super fond of that suggestion).
@sipa @achow101 it would be very very easy in the current PR to ask for another estimate for the change, I think ~two loc addition, and minor addition to the selectcoins arguments to pass down a second fee. I think this would be much more desirable than a fixed division. Future work could do things like make that second confirmation target configurable. |
src/wallet/wallet.cpp
Outdated
@@ -2562,7 +2562,7 @@ bool CWallet::CreateTransaction(const std::vector<CRecipient>& vecSend, CWalletT | |||
} | |||
|
|||
const CAmount nChange = nValueIn - nValueToSelect; | |||
if (nChange > 0) | |||
if (nChange > 0 && (!first_pass || nFeeRet == 0)) // nFeeRet is only 0 on the first pass if BnB was not used. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using nFeeRet to signal BNB usage is ugly. I think you shouldn't pass in nFeeRet at all, but have some explicit signal (e.g. boolean return) for BNB usage and if its set; after select coins set nFeeRet to nChange and use the same signal to bypass this branch.
I also think this condition is slightly incorrect but benign in the current code, lets say our configured feerate were zero: now BNB could find a solution and leave nFeeRet==0. (though nChange would currently be zero too, so it would be harmless but seems to me like the kind of thing to be brittle in future changes)
Travis failure seems to be unrelated |
just fyi, I have used your code as a reference for this code |
I have to say, I don't understand the target size; maybe there is a bug there. In wallet.cpp, in This is then used as the exact target in the BnB algorithm. However, you should add the cost of the outputs + the small cost of the tx overhead into the target (done here for the simple case on 1 output - https://github.com/Xekyo/CoinSelectionSimulator/blob/master/src/main/scala/one/murch/bitcoin/coinselection/StackEfficientTailRecursiveBnB.scala#L28 ) Maybe it's done somewhere, but I don't see it. |
@Runn1ng BnB uses effective values for the inputs so the fee is accounted for when coins are selected. The effective values are calculated in |
That eff. value accounts for the inputs of the new transaction, but not for the outputs (plus the overhead of the tx itself, but that is only about 10 bytes). In |
Ah, yes. That is a bug. Thanks for finding that! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK 73b5bf2. Only change since last review is comment fix and squash.
utACK 73b5bf2 |
73b5bf2 Add a test to make sure that negative effective values are filtered (Andrew Chow) 76d2f06 Benchmark BnB in the worst case where it exhausts (Andrew Chow) 6a34ff5 Have SelectCoinsMinConf and SelectCoins use BnB or Knapsack and use it (Andrew Chow) fab0488 Add a GetMinimumFeeRate function which is wrapped by GetMinimumFee (Andrew Chow) cd927ff Move original knapsack solver tests to coinselector_tests.cpp (Andrew Chow) fb716f7 Move current coin selection algorithm to coinselection.{cpp,h} (Andrew Chow) 4566ab7 Add tests for the Branch and Bound algorithm (Andrew Chow) 4b2716d Remove coinselection.h -> wallet.h circular dependency (Andrew Chow) 7d77eb1 Use a struct for output eligibility (Andrew Chow) ce7435c Move output eligibility to a separate function (Andrew Chow) 0185939 Implement Branch and Bound coin selection in a new file (Andrew Chow) f84fed8 Store effective value, fee, and long term fee in CInputCoin (Andrew Chow) 12ec29d Calculate and store the number of bytes required to spend an input (Andrew Chow) Pull request description: This is an implementation of the [Branch and Bound coin selection algorithm written by Murch](http://murch.one/wp-content/uploads/2016/11/erhardt2016coinselection.pdf) (@xekyo). I have it set so this algorithm will run first and if it fails, it will fall back to the current coin selection algorithm. The coin selection algorithms and tests have been refactored to separate files instead of having them all in wallet.cpp. I have added some tests for the new algorithm and a test for all of coin selection in general. However, more tests may be needed, but I will need help with coming up with more test cases. This PR uses some code borrowed from #10360 to use effective values when selecting coins. Tree-SHA512: b0500f406bf671e74984fae78e2d0fbc5e321ddf4f06182c5855e9d1984c4ef2764c7586d03e16fa4b578c340b21710324926f9ca472d5447a0d1ed43eb4357e
noice! |
73b5bf2 Add a test to make sure that negative effective values are filtered (Andrew Chow) 76d2f06 Benchmark BnB in the worst case where it exhausts (Andrew Chow) 6a34ff5 Have SelectCoinsMinConf and SelectCoins use BnB or Knapsack and use it (Andrew Chow) fab0488 Add a GetMinimumFeeRate function which is wrapped by GetMinimumFee (Andrew Chow) cd927ff Move original knapsack solver tests to coinselector_tests.cpp (Andrew Chow) fb716f7 Move current coin selection algorithm to coinselection.{cpp,h} (Andrew Chow) 4566ab7 Add tests for the Branch and Bound algorithm (Andrew Chow) 4b2716d Remove coinselection.h -> wallet.h circular dependency (Andrew Chow) 7d77eb1 Use a struct for output eligibility (Andrew Chow) ce7435c Move output eligibility to a separate function (Andrew Chow) 0185939 Implement Branch and Bound coin selection in a new file (Andrew Chow) f84fed8 Store effective value, fee, and long term fee in CInputCoin (Andrew Chow) 12ec29d Calculate and store the number of bytes required to spend an input (Andrew Chow) Pull request description: This is an implementation of the [Branch and Bound coin selection algorithm written by Murch](http://murch.one/wp-content/uploads/2016/11/erhardt2016coinselection.pdf) (@xekyo). I have it set so this algorithm will run first and if it fails, it will fall back to the current coin selection algorithm. The coin selection algorithms and tests have been refactored to separate files instead of having them all in wallet.cpp. I have added some tests for the new algorithm and a test for all of coin selection in general. However, more tests may be needed, but I will need help with coming up with more test cases. This PR uses some code borrowed from bitcoin#10360 to use effective values when selecting coins. Tree-SHA512: b0500f406bf671e74984fae78e2d0fbc5e321ddf4f06182c5855e9d1984c4ef2764c7586d03e16fa4b578c340b21710324926f9ca472d5447a0d1ed43eb4357e
73b5bf2 Add a test to make sure that negative effective values are filtered (Andrew Chow) 76d2f06 Benchmark BnB in the worst case where it exhausts (Andrew Chow) 6a34ff5 Have SelectCoinsMinConf and SelectCoins use BnB or Knapsack and use it (Andrew Chow) fab0488 Add a GetMinimumFeeRate function which is wrapped by GetMinimumFee (Andrew Chow) cd927ff Move original knapsack solver tests to coinselector_tests.cpp (Andrew Chow) fb716f7 Move current coin selection algorithm to coinselection.{cpp,h} (Andrew Chow) 4566ab7 Add tests for the Branch and Bound algorithm (Andrew Chow) 4b2716d Remove coinselection.h -> wallet.h circular dependency (Andrew Chow) 7d77eb1 Use a struct for output eligibility (Andrew Chow) ce7435c Move output eligibility to a separate function (Andrew Chow) 0185939 Implement Branch and Bound coin selection in a new file (Andrew Chow) f84fed8 Store effective value, fee, and long term fee in CInputCoin (Andrew Chow) 12ec29d Calculate and store the number of bytes required to spend an input (Andrew Chow) Pull request description: This is an implementation of the [Branch and Bound coin selection algorithm written by Murch](http://murch.one/wp-content/uploads/2016/11/erhardt2016coinselection.pdf) (@xekyo). I have it set so this algorithm will run first and if it fails, it will fall back to the current coin selection algorithm. The coin selection algorithms and tests have been refactored to separate files instead of having them all in wallet.cpp. I have added some tests for the new algorithm and a test for all of coin selection in general. However, more tests may be needed, but I will need help with coming up with more test cases. This PR uses some code borrowed from bitcoin#10360 to use effective values when selecting coins. Tree-SHA512: b0500f406bf671e74984fae78e2d0fbc5e321ddf4f06182c5855e9d1984c4ef2764c7586d03e16fa4b578c340b21710324926f9ca472d5447a0d1ed43eb4357e
73b5bf2 Add a test to make sure that negative effective values are filtered (Andrew Chow) 76d2f06 Benchmark BnB in the worst case where it exhausts (Andrew Chow) 6a34ff5 Have SelectCoinsMinConf and SelectCoins use BnB or Knapsack and use it (Andrew Chow) fab0488 Add a GetMinimumFeeRate function which is wrapped by GetMinimumFee (Andrew Chow) cd927ff Move original knapsack solver tests to coinselector_tests.cpp (Andrew Chow) fb716f7 Move current coin selection algorithm to coinselection.{cpp,h} (Andrew Chow) 4566ab7 Add tests for the Branch and Bound algorithm (Andrew Chow) 4b2716d Remove coinselection.h -> wallet.h circular dependency (Andrew Chow) 7d77eb1 Use a struct for output eligibility (Andrew Chow) ce7435c Move output eligibility to a separate function (Andrew Chow) 0185939 Implement Branch and Bound coin selection in a new file (Andrew Chow) f84fed8 Store effective value, fee, and long term fee in CInputCoin (Andrew Chow) 12ec29d Calculate and store the number of bytes required to spend an input (Andrew Chow) Pull request description: This is an implementation of the [Branch and Bound coin selection algorithm written by Murch](http://murch.one/wp-content/uploads/2016/11/erhardt2016coinselection.pdf) (@xekyo). I have it set so this algorithm will run first and if it fails, it will fall back to the current coin selection algorithm. The coin selection algorithms and tests have been refactored to separate files instead of having them all in wallet.cpp. I have added some tests for the new algorithm and a test for all of coin selection in general. However, more tests may be needed, but I will need help with coming up with more test cases. This PR uses some code borrowed from bitcoin#10360 to use effective values when selecting coins. Tree-SHA512: b0500f406bf671e74984fae78e2d0fbc5e321ddf4f06182c5855e9d1984c4ef2764c7586d03e16fa4b578c340b21710324926f9ca472d5447a0d1ed43eb4357e
73b5bf2 Add a test to make sure that negative effective values are filtered (Andrew Chow) 76d2f06 Benchmark BnB in the worst case where it exhausts (Andrew Chow) 6a34ff5 Have SelectCoinsMinConf and SelectCoins use BnB or Knapsack and use it (Andrew Chow) fab0488 Add a GetMinimumFeeRate function which is wrapped by GetMinimumFee (Andrew Chow) cd927ff Move original knapsack solver tests to coinselector_tests.cpp (Andrew Chow) fb716f7 Move current coin selection algorithm to coinselection.{cpp,h} (Andrew Chow) 4566ab7 Add tests for the Branch and Bound algorithm (Andrew Chow) 4b2716d Remove coinselection.h -> wallet.h circular dependency (Andrew Chow) 7d77eb1 Use a struct for output eligibility (Andrew Chow) ce7435c Move output eligibility to a separate function (Andrew Chow) 0185939 Implement Branch and Bound coin selection in a new file (Andrew Chow) f84fed8 Store effective value, fee, and long term fee in CInputCoin (Andrew Chow) 12ec29d Calculate and store the number of bytes required to spend an input (Andrew Chow) Pull request description: This is an implementation of the [Branch and Bound coin selection algorithm written by Murch](http://murch.one/wp-content/uploads/2016/11/erhardt2016coinselection.pdf) (@xekyo). I have it set so this algorithm will run first and if it fails, it will fall back to the current coin selection algorithm. The coin selection algorithms and tests have been refactored to separate files instead of having them all in wallet.cpp. I have added some tests for the new algorithm and a test for all of coin selection in general. However, more tests may be needed, but I will need help with coming up with more test cases. This PR uses some code borrowed from bitcoin#10360 to use effective values when selecting coins. Tree-SHA512: b0500f406bf671e74984fae78e2d0fbc5e321ddf4f06182c5855e9d1984c4ef2764c7586d03e16fa4b578c340b21710324926f9ca472d5447a0d1ed43eb4357e
* Calculate and store the number of bytes required to spend an input * Store effective value, fee, and long term fee in CInputCoin Have CInputCOin store effective value information. This includes the effective value itself, the fee, and the long term fee for the input * Implement Branch and Bound coin selection in a new file Create a new file for coin selection logic and implement the BnB algorithm in it. * Move output eligibility to a separate function * Use a struct for output eligibility Instead of specifying 3 parameters, use a struct for those parameters in order to reduce the number of arguments to SelectCoinsMinConf. * Remove coinselection.h -> wallet.h circular dependency Changes CInputCoin to coinselection and to use CTransactionRef in order to avoid a circular dependency. Also moves other coin selection specific variables out of wallet.h to coinselectoin.h * Add tests for the Branch and Bound algorithm * Move current coin selection algorithm to coinselection.{cpp,h} Moves the current coin selection algorithm out of SelectCoinsMinConf and puts it in coinselection.{cpp,h}. The new function, KnapsackSolver, instead of taking a vector of COutputs, will take a vector of CInputCoins that is prepared by SelectCoinsMinConf. * Move original knapsack solver tests to coinselector_tests.cpp * Add a GetMinimumFeeRate function which is wrapped by GetMinimumFee * Have SelectCoinsMinConf and SelectCoins use BnB or Knapsack and use it (partial) Allows SelectCoinsMinConf and SelectCoins be able to switch between using BnB or Knapsack for choosing coins. Has SelectCoinsMinConf do the preprocessing necessary to support either BnB or Knapsack. This includes calculating the filtering the effective values for each input. Uses BnB in CreateTransaction to find an exact match for the output. If BnB fails, it will fallback to the Knapsack solver. Dash specific note: just always use Knapsack in CreateTransaction. * Benchmark BnB in the worst case where it exhausts * Add a test to make sure that negative effective values are filtered * More of 12747: Fix typos Co-authored-by: Andrew Chow <achow101-github@achow101.com>
This is an implementation of the Branch and Bound coin selection algorithm written by Murch (@xekyo). I have it set so this algorithm will run first and if it fails, it will fall back to the current coin selection algorithm. The coin selection algorithms and tests have been refactored to separate files instead of having them all in wallet.cpp.
I have added some tests for the new algorithm and a test for all of coin selection in general. However, more tests may be needed, but I will need help with coming up with more test cases.
This PR uses some code borrowed from #10360 to use effective values when selecting coins.