-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rpc method estimatefee #735
Conversation
d7c798c
to
a65d71a
Compare
estimatefee.go
Outdated
) | ||
|
||
const ( | ||
// estimateFeeDepth is the maximum number of blocks before a transaction |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The variable name as declared and what's referenced in the comment are mismatched. The comment should be updated to reference estimateFeeBins
instead of estimateFeeDepth
.
Thanks for the PR! Great work, we've been missing a fee estimation model within I've done an initial pass through the PR, but haven't closely examined the tests yet nor verified the correctness of the model. One recommendation: I think fees should be stored and predicated in
No such file currently exists, however it should be rather straight forward to persist the state of the fee estimation model to the database. A bucket within the Also if we decide to move forward with this method, then it'll need a corresponding PR within |
Does this model incorporate Alex Morcos' modifications to Gavin's initial model? At first glance, it doesn't appear that it does. |
blockmanager.go
Outdated
@@ -1386,6 +1396,7 @@ func (b *blockManager) Pause() chan<- struct{} { | |||
// newBlockManager returns a new bitcoin block manager. | |||
// Use Start to begin processing asynchronous block and inv updates. | |||
func newBlockManager(s *server, indexManager blockchain.IndexManager) (*blockManager, error) { | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like an extra blank line snuck in here.
This will require a rebase now that #737 went in. I apologize for the churn, but it was important to bite the bullet and get the |
estimatefee.go
Outdated
bin := ef.bin[blocksToConfirm] | ||
|
||
// Remove a random element and replace it with this new tx. | ||
if len(bin) == int(ef.binSize) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ef.binSize
is already an int. No need to cast it.
@DanielKrawisz can you rebase one more time and i will run/test it. |
d27bed7
to
380ae24
Compare
I haven't reviewed the updated code yet, but this is failing the CI builds.
Looks like you need to run |
I forgot to run goclean.sh after the last rebase! |
8b21363
to
122d875
Compare
122d875
to
7a31a98
Compare
Testing this PR on mainnet now. Will report back with findings/graphs/comparisons. |
Fixes #139 |
7a31a98
to
42125c7
Compare
Bump this is a pretty important PR; it's sad to see it languish. |
42125c7
to
cd03910
Compare
cd03910
to
3814300
Compare
@DanielKrawisz can you squash those additional commits down into a core set? Thanks! |
3814300
to
3878cd3
Compare
of how long it takes for unconfirmed txs to be mined into blocks. It can also roll itself back in the case of an orphan block.
…mprovements. Rollback takes a block hash rather than a BlockStamp. Increase rounds in TestEstimateFeeRollback to test dropping txs that have been in the mempool too long.
new txs that it observes. The block manager alerts the fee estimator of new and orphaned blocks. Check for invalid state and recreate FeeEstimator if necessary.
and the server searches the database for a previous state to load when the program is turned on.
3878cd3
to
73c8d0a
Compare
I've been running this on testnet for a few days now, and hit this earlier today:
|
Here's another instance with a full paste (still haven't pin pointed if it's related to this PR): https://pastebin.com/Qx0t6HZk |
@DanielKrawisz you still aware of this? |
I've been running this for some time now on my fork of |
Hmm, what is the quality of the estimation (I didn't read what algorithm it uses)? What happened to the ML algos for online estimation that you wrote about? (Read that draft paper this weekend.) |
A slightly modified version of this was recently merged in as part of #1180. It should serve as a starting point to improve the fee estimation algo in the future. |
This pr implements estimatefee similar to the way it works in Bitcoin Core, as described here. estimatefee allows for one parameter, NumBlocks. It attempts to provide an estimated fee / kb rate (in Bitcoins) for a transaction to be confirmed before at least NumBlocks (inclusive) have been mined.
Every time a transaction is added to the mempool, it is registered with the fee estimator. Every time a new block is received, it is also registered with the fee estimator. The fee estimator keeps 25 bins containing information on transactions for which 1 to 25 blocks have been mined before being put in a block. Each bin contains information on up to 100 transactions.
Let m be a bijective map from all sots in all bins to the set if indices from 1 to N, where N is the number of transactions in all bins. m maps the first slot in the first bin to 1, the second to 2, and so on, all the way through all the bins. The value returned by estimatefee is the fee / kb value of the transactions whose index in a sorted list of all transactions (with the highest value first) is the same as m[s], where s is the middle slot in bin NumBlocks. estimatefee therefore is a monotonically decreasing function of NumBlocks.
It is also possible to rollback the effect of a limited number of blocks in case of orphans. Default number of rollbacks allowed is 2.
Since it is very easy to store 25 values, all results for all values of NumBlocks are cached when estimatefee is first called. Each time a new block is registered or rolled back, the cache is deleted.
In core, the set of observed transactions is saved between sessions in a file that also contains mempool transactions. btcd doesn't seem to have a similar file, so fee estimation starts over if the program restarts.The server saves the state of the FeeEstimator on shutdown into the database and tries to restore it upon initialization. If it cannot, then it creates a new FeeEstimator.
A corresponding pr to btcrpcclient has been made here.