Improve usage of fee estimation code #6134

morcos · 2015-05-13T20:39:41Z

When automatically adding a fee estimate to a transaction and in the event no estimate is available for the desired confirmation target, it makes more sense to default to a fee estimate for a worse confirmation target rather than the hard coded minimum.

I'm open to a different way of exposing this functionality if there is a better suggestion.

@cozz I don't know anything about QT code, so I think what I did in sendcoinsdialog.cpp is pretty hacky, although it appears to have the desired effect. Perhaps you could take a look and suggest a better approach? This is my attempt to address the concern I mentioned in this comment: #5200 (comment)

jgarzik · 2015-09-15T17:32:11Z

Seems quite reasonable - ut ACK

maflcko · 2015-09-20T16:57:22Z

Concept ACK

morcos · 2015-10-16T16:48:34Z

OK I reworked this to be a bit better, and combined it with the patch that increases the success threshold to 95%.

@jonasschnelli I did something much simpler in the interface. The slider just indicates the # of blocks for which your fee estimate was valid. Perhaps we need to consider some kind of additional UI element that would explain to people why they aren't able to get an estimate for a smaller # of blocks. I think it'll be very rare and maybe never that you'll successfully get an estimate for 1 block, which people will probably try to do.

morcos · 2015-11-10T23:29:53Z

The last commit is as of yet untested, but should resolve some of the issues around accidentally generating transactions that wouldn't be accepted to your own mempool

jonasschnelli · 2015-11-13T19:58:03Z

Needs rebase (trivial main.h conflict).
Tested a bit.

QT: Probably not related to this PR, but is there a reason why the slider – by default – is on the left side "slow confirmation time" and not at the very right side? I guess less then <1% of transactions aim for confirmation >=25blocks.

QT: In my case (mainnet at current height), num-blocks 25 till 18 had the same fee. This feels bad for users, because they might think, "why should I pay a higher fee than i really need for my target?". We need to aggregate num-blocks with the same fee (stop at block 18, don't go further down with the fee, up with the num block target). Also num-blocks 10 and 11 had the same fee, these should be aggregated.

There should be something when calling estimatefee 1. I know that there are to many unknowns for a precise calculation. But it's an estimation. We should not return -1. We should return a value as accurate as possible with the parameters that are available (even is there are hight risks that the estimation is wrong).

morcos · 2015-11-13T22:04:28Z

@jonasschnelli This PR should prevent getting a -1 unless you directly call the RPC call estimatefee. I might still expose estimateapproxfee, but for anyone using GUI or wallet code that automatically generates fee, they should now get the best answer possible. I figured that for outside users, they could emulate the logic of estimateapproxfee themselves for now. In the future I agree there should be an RPC call which just does the right thing, but I wasn't ready to expose that now as I'm not sure exactly what it should look like yet.

morcos · 2015-11-14T19:02:27Z

Rebased

gmaxwell · 2015-11-14T20:56:52Z

Concept(s) ACK.

morcos · 2015-11-15T17:59:42Z

added some checks in the unit test and exposed the functions via RPC (with a warning) and tested in the RPC test.

jonasschnelli · 2015-11-16T20:07:16Z

Concept ACK. Plans to test this soon.

These are more useful fee and priority estimation functions. If there is no fee/pri high enough for the target you are aiming for, it will give you the estimate for the lowest target that you can reliably obtain. This is better than defaulting to the minimum. It will also pass back the target for which it returned an answer.

This provides more conservative estimates and reacts more quickly to a backlog. Unfortunately the unit test for fee estimation depends on the success threshold (and the decay) chosen; also modify the unit test for the new default success thresholds.

Also add testing for estimatesmartfee in smartfees.py

morcos · 2015-11-16T20:44:46Z

Rebased with a name change to estimateSmartFee instead of Approx

sdaftuar · 2015-11-17T17:50:14Z

ACK apart from those nits

jtimon · 2015-11-18T15:47:48Z

src/policy/fees.cpp

+        *answerFoundAtTarget = confTarget - 1;
+
+    // If mempool is limiting txs , return at least the min fee from the mempool
+    CAmount minPoolFee = pool->GetMinFee(GetArg("-maxmempool", DEFAULT_MAX_MEMPOOL_SIZE) * 1000000).GetFeePerK();


Can GetArg("-maxmempool", DEFAULT_MAX_MEMPOOL_SIZE) be an attribute so that it can be calculated just once in the constructor?

Sure. But honestly it seems a bit silly to me that this is not an attribute of the mempool. I understand the point of keeping policy out of the mempool, but why should you need to be aware of the max size of the mempool just to ask what it's min fee is. The one advantage I can see for the current structure is that you can call TrimToSize with some other number to do testing.
Perhaps TrimToSize should take a sizelimit, and set an internal attribute of the mempool that GetMinFee uses?

Yes, you are right: it should probably be an attribute of CTxMemPool instead, and minPoolFee could be passed here instead of the CTxMemPool pointer. In fact, CTxMemPool depend on CBlockPolicyEstimator, so CBlockPolicyEstimator shouldn't depend on CTxMemPool too (sigh, I had a branch in which neither one depended on the other at least once but now just https://github.com/jtimon/bitcoin/commits/policy-minrelayfee-0.12.99 ...).

You can't pass the minFee it's dynamic, you have to ask the mempool for it.
I think the CBlockPolicyEstimator should come out of the mempool though for sure.

What I'm saying is you can ask the mempool before calling this (ie rplace the pool pointer parameter with const CAmount& minPoolFee in this new method).

Please look at https://github.com/jtimon/bitcoin/tree/6134-nits
By the way, we should solve the circular dependency by separating CTxMemPoolEntry. That way txmempool.cpp includes policy/fees.h, and policy/fees.cpp includes txmempoolentry.h (but not txmempool.h). Just came to mind again, not saying is within the scope of this PR.

jtimon · 2015-11-18T15:49:20Z

Concept ACK with a very fast review. Again, this would be simpler if there was no priority...

jtimon · 2015-11-18T15:50:53Z

src/policy/fees.cpp

@@ -504,6 +505,33 @@ CFeeRate CBlockPolicyEstimator::estimateFee(int confTarget)
    return CFeeRate(median);
 }

+CFeeRate CBlockPolicyEstimator::estimateSmartFee(int confTarget, int *answerFoundAtTarget, const CTxMemPool *pool)


style nit: https://github.com/bitcoin/bitcoin/blob/master/src/.clang-format#L41

sdaftuar · 2015-11-24T13:52:57Z

Addressed nits.

jtimon · 2015-11-24T14:28:26Z

My main nit hasn't been addressed, which is not creating a new circular dependency (currently CTxMemPool depends on CBlockPolicyEstimator, but CBlockPolicyEstimator doesn't depend on CTxMemPool until this PR) that is unnecessary (as shown in jtimon@6963b9c ).

e304432 Pass reference to estimateSmartFee and cleanup whitespace (Suhas Daftuar) 56106a3 Expose RPC calls for estimatesmart functions (Alex Morcos) e93a236 add estimateSmartFee to the unit test (Alex Morcos) 6303051 EstimateSmart functions consider mempool min fee (Alex Morcos) f22ac4a Increase success threshold for fee estimation to 95% (Alex Morcos) 4fe2823 Change wallet and GUI code to use new smart fee estimation calls. (Alex Morcos) 22eca7d Add smart fee estimation functions (Alex Morcos)

jtimon · 2015-11-27T12:42:39Z

I guess I'll create a new PR removing the new and unnecessary circular dependency introduced...

jtimon · 2015-11-27T14:04:55Z

Opened #7115 to fix the new circular dependency introduced despite my insistence and despite coding the commit to be squashed here to avoid that circular dependency...

jtimon · 2015-12-01T12:07:19Z

For future reference NACK this PR. I'm really disappointed about @morcos being so stubborn and insisting on introducing the unnecessary circular dependency and that he can even accept the most minimal way to remove it (see the first commit in #7115 ), but I like circular discussions even less than I like circular dependencies and nobody else seems to care enough. So let's leave the circular dependency, but then I need to at least do what I should have done from the beginning and will do next time: nack instead of coding nits to be ignored. NACK nack nacking...

laanwj · 2015-12-01T12:11:35Z

but then I need to at least do what I should have done from the beginning and will do next time: nack instead of coding nits to be ignored. NACK nack nacking...

The practical problem is that we can't hold up required changes forever because they have some coding nits. There's, unfortunately, a compromise between accumulating some technical debt or having any single change take forever because of architectural concerns.

jtimon · 2015-12-01T12:16:41Z

This was merged without my nit and morcos insists on nacking any code that removes the circular dependency he unnecessarily introduced. I think it has little to do with "prs taking forever". But hey I'm in favor of merging bip68 and other PRs that are taking forever if you want to change the subject.

sipa · 2015-12-01T12:28:03Z

He does not nack removing the circular dependency. He disagrees with making the estimation code users responsible for knowing there is a limit.

jtimon · 2015-12-01T12:35:22Z

See yesterday's IRC discussion, yes, he even nacks the most minimal way to remove the circular dependency (the first commit currently in #7115 ) and he will not provide an alternative because he does not want to remove the dependency.

4038de4 [QA] Restore 'feature_fee_estimation.py' functional test (random-zebra) 988f349 [qt] Properly display required fee instead of minTxFee (random-zebra) c5c9df0 [Trivial] Fix typos (random-zebra) a8fa99b [Trivial] Remove redundat check in ContainsZerocoins (random-zebra) 8ebcbcd PolicyEstimator: exclude zerocoin spends when computing estimates (random-zebra) b67049b Create new BlockPolicyEstimator for fee estimates (random-zebra) Pull request description: This introduces the new fee and priority estimation code described here <https://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg06405.html>. Instead of calculating the median fee for each possible number of blocks needed to confirm, the new code divides the possible fee rates into buckets (spaced logarithmically) and keeps track of the number of blocks needed to confirm for each transaction in each bucket. Backports: - bitcoin#5159 - bitcoin#6887 except for the functional test (`smartfees.py`) which is based on an older version of the framework. Instead I've restored a more recent `feature_fee_estimation.py` (commenting out the check for `estimatesmartfee` which can be reintroduced once bitcoin#6134 is backported). Additional commits exclude zerocoins txes from the estimates calculation, as they have fixed fee/priority. ACKs for top commit: Fuzzbawls: ACK 4038de4 furszy: ACK 4038de4 and merging Tree-SHA512: 25777af469f7fa84d2dab991544392bea8418bccb4d2c23113de2c6ed7047891bdaedad1728cb04ba343e82f4bc0c1446f88e28def698dd25168769abf8620bb

3aded91 [Trivial] Temporarily comment-out priority check in policyestimator test (random-zebra) 245c3df [Tests] Enable smartfee test in feature_fee_estimation.py (random-zebra) a06e9ff Pass reference to estimateSmartFee and cleanup whitespace (Suhas Daftuar) 9df303d Expose RPC calls for estimatesmart functions (furszy) 826aaaa add estimateSmartFee to the unit test (Alex Morcos) e34b723 EstimateSmart functions consider mempool min fee (Alex Morcos) 649b4fc Increase success threshold for fee estimation to 95% (Alex Morcos) 4893a7c Change wallet and GUI code to use new smart fee estimation calls. (furszy) e8fc55a Add smart fee estimation functions (Alex Morcos) Pull request description: Another back port, coming from dashpay#6134 . > When automatically adding a fee estimate to a transaction and in the event no estimate is available for the desired confirmation target, it makes more sense to default to a fee estimate for a worse confirmation target rather than the hard coded minimum. > TODO: Left to back port smartfees.py. We need it to be able to move forward in #1726 ACKs for top commit: Fuzzbawls: utACK 3aded91 random-zebra: ACK 3aded91 and merging... Tree-SHA512: 6f52d29267da0b20e54ed680743b341c1e616986dabf72916499cf5936d73d552bab907038e83b65625d6bf9dbd0adbb1e1300a15622d94908f54e6fe270f2f6

laanwj added the Wallet label May 15, 2015

morcos mentioned this pull request Jul 14, 2015

Recommended Transaction Fee Slider: non-monotonic behavior #6437

Closed

morcos mentioned this pull request Oct 15, 2015

Fee estimate patch #6618

Closed

morcos force-pushed the EstimateApproxFee branch from 7ad61bc to 4f872ce Compare October 16, 2015 16:44

morcos mentioned this pull request Oct 16, 2015

Do not use hard-coded AllowFree for free transactions. #6131

Closed

morcos mentioned this pull request Nov 10, 2015

Mempool trimming regression: Rejects my own high prority 0-fee transactions #6972

Closed

morcos force-pushed the EstimateApproxFee branch from 4f872ce to b79c9f2 Compare November 10, 2015 23:28

morcos mentioned this pull request Nov 10, 2015

Mempool should not reject wallet created transactions with sufficient fee #6725

Closed

TheBlueMatt mentioned this pull request Nov 11, 2015

[WIP] [Mempool] Add space for priority transactions #6992

Closed

morcos mentioned this pull request Nov 12, 2015

Don't use hard-coded AllowFree, as this is usually far too low. #6997

Closed

morcos force-pushed the EstimateApproxFee branch from b79c9f2 to c9404f5 Compare November 14, 2015 19:00

gmaxwell added this to the 0.12.0 milestone Nov 14, 2015

morcos added 6 commits November 16, 2015 15:33

Change wallet and GUI code to use new smart fee estimation calls.

4fe2823

EstimateSmart functions consider mempool min fee

6303051

add estimateSmartFee to the unit test

e93a236

Expose RPC calls for estimatesmart functions

56106a3

Also add testing for estimatesmartfee in smartfees.py

morcos force-pushed the EstimateApproxFee branch from f2e9aab to 56106a3 Compare November 16, 2015 20:43

jtimon reviewed Nov 18, 2015
View reviewed changes

Pass reference to estimateSmartFee and cleanup whitespace

e304432

laanwj merged commit e304432 into bitcoin:master Nov 27, 2015

jtimon mentioned this pull request Nov 27, 2015

Mempool: Decouple CBlockPolicyEstimator from CTxMemPool (fix #6134) #7115

Closed

maflcko mentioned this pull request Dec 9, 2015

i sent 0.131 bitcoin with max fee from wallet and no confirmation with 3 days #7191

Closed

maflcko mentioned this pull request Mar 14, 2016

bitcoin 0.12's coinselection #7657

Closed

gracenoah mentioned this pull request Mar 14, 2018

rpcclient: Add estimatesmartfee support btcsuite/btcd#1146

Closed

random-zebra mentioned this pull request May 23, 2020

[Core] New fee estimation code PIVX-Project/PIVX#1641

Merged

furszy mentioned this pull request Aug 4, 2020

Improve usage of fee estimation code PIVX-Project/PIVX#1787

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve usage of fee estimation code #6134

Improve usage of fee estimation code #6134

morcos commented May 13, 2015

jgarzik commented Sep 15, 2015

maflcko commented Sep 20, 2015

morcos commented Oct 16, 2015

morcos commented Nov 10, 2015

jonasschnelli commented Nov 13, 2015

morcos commented Nov 13, 2015

morcos commented Nov 14, 2015

gmaxwell commented Nov 14, 2015

morcos commented Nov 15, 2015

jonasschnelli commented Nov 16, 2015

morcos commented Nov 16, 2015

sdaftuar commented Nov 17, 2015

jtimon Nov 18, 2015

morcos Nov 18, 2015

jtimon Nov 18, 2015

morcos Nov 18, 2015

jtimon Nov 18, 2015

jtimon Nov 18, 2015

jtimon commented Nov 18, 2015

jtimon Nov 18, 2015

sdaftuar commented Nov 24, 2015

jtimon commented Nov 24, 2015

jtimon commented Nov 27, 2015

jtimon commented Nov 27, 2015

jtimon commented Dec 1, 2015

laanwj commented Dec 1, 2015

jtimon commented Dec 1, 2015

sipa commented Dec 1, 2015 via email

jtimon commented Dec 1, 2015

Improve usage of fee estimation code #6134

Improve usage of fee estimation code #6134

Conversation

morcos commented May 13, 2015

jgarzik commented Sep 15, 2015

maflcko commented Sep 20, 2015

morcos commented Oct 16, 2015

morcos commented Nov 10, 2015

jonasschnelli commented Nov 13, 2015

morcos commented Nov 13, 2015

morcos commented Nov 14, 2015

gmaxwell commented Nov 14, 2015

morcos commented Nov 15, 2015

jonasschnelli commented Nov 16, 2015

morcos commented Nov 16, 2015

sdaftuar commented Nov 17, 2015

jtimon Nov 18, 2015

Choose a reason for hiding this comment

morcos Nov 18, 2015

Choose a reason for hiding this comment

jtimon Nov 18, 2015

Choose a reason for hiding this comment

morcos Nov 18, 2015

Choose a reason for hiding this comment

jtimon Nov 18, 2015

Choose a reason for hiding this comment

jtimon Nov 18, 2015

Choose a reason for hiding this comment

jtimon commented Nov 18, 2015

jtimon Nov 18, 2015

Choose a reason for hiding this comment

sdaftuar commented Nov 24, 2015

jtimon commented Nov 24, 2015

jtimon commented Nov 27, 2015

jtimon commented Nov 27, 2015

jtimon commented Dec 1, 2015

laanwj commented Dec 1, 2015

jtimon commented Dec 1, 2015

sipa commented Dec 1, 2015 via email

jtimon commented Dec 1, 2015