Coin Selection with Murch's algorithm by achow101 · Pull Request #10637 · bitcoin/bitcoin

achow101 · 2017-06-20T18:09:30Z

This is an implementation of the Branch and Bound coin selection algorithm written by Murch (@xekyo). I have it set so this algorithm will run first and if it fails, it will fall back to the current coin selection algorithm. The coin selection algorithms and tests have been refactored to separate files instead of having them all in wallet.cpp.

I have added some tests for the new algorithm and a test for all of coin selection in general. However, more tests may be needed, but I will need help with coming up with more test cases.

This PR uses some code borrowed from #10360 to use effective values when selecting coins.

murchandamus

Concept ACK, looks good to me. AFAICT, there's an issue with the lookahead that should be addressed still.

murchandamus · 2017-06-20T18:46:04Z

+    // Calculate remaining
+    CAmount remaining = 0;
+    for (CInputCoin utxo : utxo_pool) {
+        remaining += utxo.txout.nValue;


Have you filtered utxo_pool to exclude utxo's that have a net-neg value? Otherwise you're underestimating the lookahead here. To get an accurate figure for what you may still collect downtree, you should only add utxo.txout.nValue >=0

@gmaxwell has concerns that Core wallet is only doing semi-sane utxo handling by spending these. With exact match + sane backoff algorithm this concern may be alleviated?

Indeed, that may be a problem. I will add that in as it is still good to have additional checks here even if done elsewhere.

I don't have much of a concern here about the 0/negative effective value inputs: Failing to select negative effective value inputs for an exact match won't lead to a UTXO count inflation, because changeless transactions are by definition strictly UTXO reducing.

@instagibbs: I'm not completely opposed to spending net-negative UTXO, my concern here is primarily that it actually may cause the lookahead to be underestimated causing valid solutions not to be found.

I realize now that the knapsack algorithm would also not select uneconomic UTXO anymore, as if it had selected enough value before it reached them it would have already returned the set, and if it actually starts exploring them, cannot add more value in the first place.

Advocatus Diaboli: Would it be that terrible though, if UTXO were only considered when they actually have a net positive value? During times of low fees, they'd be used both during BnB and knapsack, during times of high fees, they wouldn't bloat the blocks and lose their owner money.

I am not so concerned, was making sure concerns are brought up.

@xekyo we should assume that it would be terrible unless someone can show that it will not cause another massive UTXO bloat event... but thats offtopic here, as I don't think anyone has this concern with exact matches.

The utxos with negative effective values are filtered anyway in wallet/wallet.cpp, which is the only place (except for tests) from where SelectCoinsBnB is called.

murchandamus · 2017-06-20T19:05:57Z

+
+    // Select 5 Cent
+    add_coin(4 * CENT, 4, actual_selection);
+    add_coin(1 * CENT, 1, actual_selection);


AFAICT utxo_pool has : 4, 3, 2, & 1. Since you're exploring randomly selecting 5 then has two possible solutions: 4+1, 3+2.

It is forced to be include first in these tests so the solution is deterministic.

murchandamus · 2017-06-20T19:08:54Z

+    add_coin(5 * CENT, 5, utxo_pool);
+    add_coin(5 * CENT, 5, actual_selection);
+    add_coin(4 * CENT, 4, actual_selection);
+    add_coin(1 * CENT, 1, actual_selection);


Under above assumptions, there is two solutions here as well: 5+4+1, or 5+3+2.

It is forced to be include first in these tests so the solution is deterministic.

murchandamus · 2017-06-20T19:16:06Z

-            LogPrint(BCLog::SELECTCOINS, "total %s\n", FormatMoney(nBest));
-        }
+        CInputCoin coin(pcoin, i);
+        coin.txout.nValue -= (output.nInputBytes < 0 ? 0 : effective_fee.GetFee(output.nInputBytes));


It seems to me that you're also collecting coins that have a net-negative here. This will cause your lookahead to be underestimated, unless you cater to that case when calculating the remainder.

murchandamus · 2017-06-20T19:22:13Z

+    CoinSet actual_selection;
+    CAmount value_ret = 0;
+
+    /////////////////////////


I would perhaps add a test that checks what happens if the utxo_pool includes a UTXO that is more costly to spend than its own value. As far as I can tell, this would currently reduce your lookahead and may cause a premature search failure.

murchandamus · 2017-06-20T19:46:12Z

Have you tested the effect of random exploration vs largest first exploration?

Either way, BranchAndBound already guarantees that the global utxo set doesn't grow (for one output transactions) due to saving the change output.
LFE guarantees the creation of a minimal input set, and purposefully finds a possible solution. This should minimize the input set size variance. In my simulations BranchAndBound with LFE already caused a smaller average UTXO footprint than legacy Core selection.
Random Exploration could find a larger input set by skipping a key UTXO higher up in the tree. This could lead to the selection of a larger number of inputs, or in an edge case could even cause tries to be exhausted before a solution is found. This may increase input set variance, or could perhaps even exhaust small UTXOs too quickly for BnB to often find a viable solution.

I am not sure there is a significant privacy benefit for Random Exploration as for either selection method an attacker would already need to know about another eligible input that would achieve an exact match when switched out for one of the input set.

What benefit do you expect from using Random Exploration?

achow101 · 2017-06-20T20:25:13Z

@xekyo I was thinking that Random Exploration would be better for privacy but I see that it probably wouldn't help. If you think it would be better to change to LFE, I can certainly do that.

murchandamus · 2017-06-20T20:33:10Z

@achow101: I don't know how strong the effect is, but I'd expect Random Exploration to increase the required computational effort.

instagibbs · 2017-06-20T20:52:11Z

Noting that this PR has fairly heavy overlap with #10360 .

From chatting with @achow101 the intention of this PR is to touch as little as possible while still getting BranchNBound coin selection.

To make this successful it should really only be run on first iteration of the loop in CreateTransaction, when nFeeRet == 0 and only use effective value for the BnB coin selection step, rather than the knapsack as well. Once nFeeRet becomes more than zero, interactions start to get strange without a more complete overhaul like #10360.

instagibbs · 2017-06-20T20:59:34Z

This PR I believe will still create just-over-dust change outputs when BnB finds an exact match. Whenever we are allowing BnB matches(first iteration) we should not make change outputs less than the exact match slack value.

murchandamus · 2017-06-20T21:01:02Z

+        // Calculate cost of change
+        // TODO: In the future, we should use the change output actually made for the transaction and calculate the cost
+        // requred to spend it.
+        CAmount cost_of_change = effective_fee.GetFee(148+34); // 148 bytes for the input, 34 bytes for making the output


This assumes that the input will be spent at a feerate at least as high as the current. This was a valid assumption in my thesis, as I was using a fixed fee rate. I'm not sure whether this a valid assumption for realnet transaction selection, as we've literally seen fees between 8-540 sat/byte in the past two weeks. We might want to consider discounting the cost of the input slightly.

Depends on user time preferences. Could be an option that is set for those who regularly consolidate.

For now I think it is fine to use the current feerate.

murchandamus · 2017-06-20T21:04:47Z

@instagibbs: In fact, BnB is designed to only work when creating a transaction without a change output. If we were creating a change in the first place, the extensive search pattern would be unnecessarily wasteful.

instagibbs · 2017-06-20T21:15:16Z

To append onto my previous comments, any effective value match attempt should account for the fees just obtained by SelectCoins. Currently it completely ignores the newly-obtained fees, keeping the previous loop's value, and then asks if nFeeRet >= nFeeRequired to break from the loop(which currently is 0 on the first go-around).

achow101 · 2017-06-21T00:32:49Z

I have made the BnB selector to be only run on the first pass of the coin selection loop. It is now set so that effective value is only used for the BnB selector and not the knapsack one. I have also added the negative effective value check and test just as a belt-and-suspenders thing. I also made BnB use Largest First Exploration instead of Random Exploration.

murchandamus · 2017-06-21T21:38:01Z

+            backtrack = true;
+        } else if (value_ret >= target_value) { // Selected value is within range
+            done = true;
+        } else if (tries <= 0) { // Too many tries, exit


Here's a unexpected behavior in my algorithm: if there is a number of input combinations whose value_ret all exceed the target_value when tries == 0 is passed, tries can go into the negative.

The tries check should be moved to the top of the checks.

sipa · 2017-06-21T23:26:16Z

Perhaps generically, we should never create change if the amount is less than the cost of creating + spending it (regardless of whether BnB was used to find the inputs or not)?

instagibbs · 2017-06-22T14:05:58Z

@sipa one question is if we should allow the wallet to consider consolidation-level prices for that change. Perhaps the user is in a hurry now, but would consider spending that change at a much slower pace.

Maybe for a first pass only consider the selected feerate, then Future Work allow a parameter which has more aggressive change protection given longer timescales.

sipa · 2017-06-22T16:26:11Z

@instagibbs Yes, I agree; we should use long-eatimates for the spend part of change rather than the actual feerate the user is willing to pay now. Perhaps we can make it more conservative without doing that by using a factor 2 or 3 reduction?

gmaxwell · 2017-06-22T18:53:42Z

+        // Calculate cost of change
+        // TODO: In the future, we should use the change output actually made for the transaction and calculate the cost
+        // requred to spend it.
+        CAmount cost_of_change = effective_fee.GetFee(148+34); // 148 bytes for the input, 34 bytes for making the output


not correct for segwit. If this code ends up being changed to follow pieter's suggestion of dividing the rate by two or three it should be bounded by the min relay fee. (I'm not super fond of that suggestion).

gmaxwell · 2017-06-22T18:57:00Z

@sipa @achow101 it would be very very easy in the current PR to ask for another estimate for the change, I think ~two loc addition, and minor addition to the selectcoins arguments to pass down a second fee. I think this would be much more desirable than a fixed division. Future work could do things like make that second confirmation target configurable.

gmaxwell · 2017-06-22T19:11:00Z


                const CAmount nChange = nValueIn - nValueToSelect;
-                if (nChange > 0)
+                if (nChange > 0 && (!first_pass || nFeeRet == 0)) // nFeeRet is only 0 on the first pass if BnB was not used. 


Using nFeeRet to signal BNB usage is ugly. I think you shouldn't pass in nFeeRet at all, but have some explicit signal (e.g. boolean return) for BNB usage and if its set; after select coins set nFeeRet to nChange and use the same signal to bypass this branch.

I also think this condition is slightly incorrect but benign in the current code, lets say our configured feerate were zero: now BNB could find a solution and leave nFeeRet==0. (though nChange would currently be zero too, so it would be harmless but seems to me like the kind of thing to be brittle in future changes)

achow101 · 2017-06-23T17:32:29Z

Travis failure seems to be unrelated

karelbilek · 2017-07-02T02:46:52Z

just fyi, I have used your code as a reference for this code

bitcoinjs/coinselect#13

karelbilek · 2017-07-02T20:14:33Z

I have to say, I don't understand the target size; maybe there is a bug there.

In wallet.cpp, in CWallet::CreateTransaction, you create nValue, which seems to be the sum of all the outputs. Because the BnB is used only at the first pass, nFeeRet is 0 and nValueToSelect is just the sum of all the outputs.

This is then used as the exact target in the BnB algorithm.

However, you should add the cost of the outputs + the small cost of the tx overhead into the target (done here for the simple case on 1 output - https://github.com/Xekyo/CoinSelectionSimulator/blob/master/src/main/scala/one/murch/bitcoin/coinselection/StackEfficientTailRecursiveBnB.scala#L28 )

Maybe it's done somewhere, but I don't see it.

achow101 · 2017-07-02T20:22:18Z

@Runn1ng BnB uses effective values for the inputs so the fee is accounted for when coins are selected. The effective values are calculated in SelectCoinsMinConf

karelbilek · 2017-07-02T20:27:47Z

That eff. value accounts for the inputs of the new transaction, but not for the outputs (plus the overhead of the tx itself, but that is only about 10 bytes).

In SelectCoinsMinConf, you already have the target, which does not account for that.

achow101 · 2017-07-02T21:26:29Z

Ah, yes. That is a bug. Thanks for finding that!

Moves the current coin selection algorithm out of SelectCoinsMinConf and puts it in coinselection.{cpp,h}. The new function, KnapsackSolver, instead of taking a vector of COutputs, will take a vector of CInputCoins that is prepared by SelectCoinsMinConf.

Allows SelectCoinsMinConf and SelectCoins be able to switch between using BnB or Knapsack for choosing coins. Has SelectCoinsMinConf do the preprocessing necessary to support either BnB or Knapsack. This includes calculating the filtering the effective values for each input. Uses BnB in CreateTransaction to find an exact match for the output. If BnB fails, it will fallback to the Knapsack solver.

achow101 · 2018-03-13T16:47:12Z

Squashed

ryanofsky

utACK 73b5bf2. Only change since last review is comment fix and squash.

instagibbs · 2018-03-13T17:30:00Z

utACK 73b5bf2

@xekyo

73b5bf2 Add a test to make sure that negative effective values are filtered (Andrew Chow) 76d2f06 Benchmark BnB in the worst case where it exhausts (Andrew Chow) 6a34ff5 Have SelectCoinsMinConf and SelectCoins use BnB or Knapsack and use it (Andrew Chow) fab0488 Add a GetMinimumFeeRate function which is wrapped by GetMinimumFee (Andrew Chow) cd927ff Move original knapsack solver tests to coinselector_tests.cpp (Andrew Chow) fb716f7 Move current coin selection algorithm to coinselection.{cpp,h} (Andrew Chow) 4566ab7 Add tests for the Branch and Bound algorithm (Andrew Chow) 4b2716d Remove coinselection.h -> wallet.h circular dependency (Andrew Chow) 7d77eb1 Use a struct for output eligibility (Andrew Chow) ce7435c Move output eligibility to a separate function (Andrew Chow) 0185939 Implement Branch and Bound coin selection in a new file (Andrew Chow) f84fed8 Store effective value, fee, and long term fee in CInputCoin (Andrew Chow) 12ec29d Calculate and store the number of bytes required to spend an input (Andrew Chow) Pull request description: This is an implementation of the [Branch and Bound coin selection algorithm written by Murch](http://murch.one/wp-content/uploads/2016/11/erhardt2016coinselection.pdf) (@xekyo). I have it set so this algorithm will run first and if it fails, it will fall back to the current coin selection algorithm. The coin selection algorithms and tests have been refactored to separate files instead of having them all in wallet.cpp. I have added some tests for the new algorithm and a test for all of coin selection in general. However, more tests may be needed, but I will need help with coming up with more test cases. This PR uses some code borrowed from #10360 to use effective values when selecting coins. Tree-SHA512: b0500f406bf671e74984fae78e2d0fbc5e321ddf4f06182c5855e9d1984c4ef2764c7586d03e16fa4b578c340b21710324926f9ca472d5447a0d1ed43eb4357e

ghost · 2018-03-15T00:22:34Z

noice!

achow101 force-pushed the bnb-coin-select branch from 89b1417 to 0246f1f Compare June 20, 2017 18:26

murchandamus suggested changes Jun 20, 2017

View reviewed changes

murchandamus reviewed Jun 20, 2017

View reviewed changes

fanquake added the Wallet label Jun 20, 2017

murchandamus reviewed Jun 21, 2017

View reviewed changes

gmaxwell reviewed Jun 22, 2017

View reviewed changes

This was referenced Jun 30, 2017

Do you recommend to use BnB in real wallet apps? murchandamus/CoinSelectionSimulator#3

Closed

Revisit Coin Selection #7664

Closed

This was referenced Jun 30, 2017

Implement BnB strategy bitcoinjs/coinselect#10

Open

Adding Branch and Bound algorithm bitcoinjs/coinselect#13

Open

achow101 force-pushed the bnb-coin-select branch from e5277e5 to dd84127 Compare July 2, 2017 21:59

achow101 added 6 commits March 13, 2018 12:39

Move original knapsack solver tests to coinselector_tests.cpp

cd927ff

Add a GetMinimumFeeRate function which is wrapped by GetMinimumFee

fab0488

Benchmark BnB in the worst case where it exhausts

76d2f06

Add a test to make sure that negative effective values are filtered

73b5bf2

achow101 force-pushed the bnb-coin-select branch from c3136ea to 73b5bf2 Compare March 13, 2018 16:47

ryanofsky reviewed Mar 13, 2018

View reviewed changes

laanwj merged commit 73b5bf2 into bitcoin:master Mar 14, 2018

ryanofsky mentioned this pull request Mar 16, 2018

scripted-diff: Rename wallet database classes #11851

Merged

jnewbery mentioned this pull request Apr 2, 2018

Stop creating change outputs when not necessary #10329

Closed

joemphilips mentioned this pull request Apr 14, 2018

TODO: implement Murch's coin selection algorithm joemphilips/walletts-old#2

Open

This was referenced May 4, 2018

use as little inputs as possible in order to fund outputs threefoldfoundation/tfchain#73

Open

improve output selection strategy threefoldtecharchive/rivine#310

Open

moneyball mentioned this pull request May 11, 2018

Some ACKs missing from PRs PierreRochard/bitcoin-acks#12

Closed

Empact mentioned this pull request May 12, 2018

test: Improve test coverage of SelectCoinsBnB #13222

Closed

PierreRochard mentioned this pull request May 19, 2018

Query for more than the last 100 comments and reviews PierreRochard/bitcoin-acks#13

Closed

bjarnemagnussen mentioned this pull request Aug 30, 2018

Allows for Segwit functionality ofek/bit#43

Merged

lehnberg mentioned this pull request Jul 5, 2019

Add support for Murch's algorithm / Branch & Bound coin selection strategy mimblewimble/grin-wallet#179

Closed

yancyribbens mentioned this pull request Mar 4, 2020

Suggested BNB Search Optimization #18257

Closed

benthecarman mentioned this pull request Jul 8, 2020

Implement Branch and Bound CoinSelectionAlgo bitcoin-s/bitcoin-s#1650

Closed

pinheadmz mentioned this pull request Jul 23, 2020

Coin select slowdown handshake-org/hsd#478

Open

PastaPastaPasta mentioned this pull request Dec 16, 2020

Backport bitcoin#10637 (partial) dashpay/dash#3878

Merged

Uh oh!

Conversation

achow101 commented Jun 20, 2017

Uh oh!

murchandamus left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

murchandamus Jun 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karelbilek Jul 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

murchandamus commented Jun 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

achow101 commented Jun 20, 2017

Uh oh!

murchandamus commented Jun 20, 2017

Uh oh!

instagibbs commented Jun 20, 2017

Uh oh!

instagibbs commented Jun 20, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

murchandamus commented Jun 20, 2017

Uh oh!

instagibbs commented Jun 20, 2017

Uh oh!

achow101 commented Jun 21, 2017

Uh oh!

murchandamus Jun 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sipa commented Jun 21, 2017

Uh oh!

instagibbs commented Jun 22, 2017

Uh oh!

sipa commented Jun 22, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmaxwell commented Jun 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

murchandamus left a comment •

edited

Loading

murchandamus Jun 20, 2017 •

edited

Loading

karelbilek Jul 1, 2017 •

edited

Loading

murchandamus commented Jun 20, 2017 •

edited

Loading

murchandamus Jun 21, 2017 •

edited

Loading

gmaxwell commented Jun 22, 2017 •

edited

Loading

achow101 commented Jul 2, 2017 •

edited

Loading

karelbilek commented Jul 2, 2017 •

edited

Loading