Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Mining: Select transactions using feerate-with-ancestors #7600
This implements "Child-pays-for-parent" transaction selection in
This is almost strictly better (ie, results in higher-fee blocks) than the existing mining algorithm, with some small variance due to edge cases around which transactions might get selected as the new block's size approaches the maximum allowed. Moreover the performance impact on
The actual fee improvements observed were fairly low in the time period I looked at (October 2015), roughly a 1% fee improvement overall. I think that is driven first and foremost by blocks not really being full for a large bulk of the time period, so on a large number of data points the results were identical. Additionally, my guess is that user behavior may not have adapted to this type of transaction selection being widely deployed, and so in the future perhaps we might expect to see larger differences. I did note a handful of instances where the implementation here notably outperformed the existing algorithm, in some instances more than 20% better, but those were the exception and not the rule.
For comparison, I found no examples where the old algorithm produced a block with 0.5% or more in fees than the new algorithm (and few examples where the old algorithm's block had more fees at all, around 1% of samples).
I've labeled this PR as a work-in-progress for now, as it builds off two other open PR's (#7594 and #7598). Once those are reviewed and merged, I'll rebase this and remove the WIP label. I opened this now to help motivate the work in those two PRs.
One open question I have is whether it's worth keeping around the old transaction selection algorithm. The refactor in #7598 that this builds on makes it easy to leave it around, and I don't think it would be hard to add command line arguments for policy configuration that would enable using the older algorithm. But I don't know if that's of interest to anyone, so for now I figure we can let people mull it over and decide what to do in a future PR.
A related question to that is whether to get rid of the existing mempool "score" index. The ancestor tracking pull adds a new ancestor feerate index without removing the old feerate index, but if we decide to get rid of the mining algorithm that uses that index, then perhaps we could also eliminate this mempool index as well.
referenced this pull request
Mar 1, 2016
This algorithm is by no means guaranteed to produce maximum fee blocks for a given mempool, if that's what you're trying to ask.
The algorithm this PR is trying to implement is straightforward:
The implementation is complicated because we can't actually remove things from the mempool. To deal with that, I added a new multi_index that
Anyway, getting back to the algorithm -- I think this algorithm is superior to the existing one (which iterates based on each tx's individual feerate, not looking at ancestors) for optimizing block fees, and it has the property that (what I expect to be) typical uses of CPFP should generally be captured -- namely those situations where a single child is paying a large fee for its parents.
However if there are multiple medium fee children that are chained below some low-fee parent, then the algorithm might easily fill the block without those transactions because they may not be considered until the block is full. For now though I suspect this algorithm is adequate for the current usage patterns, and perhaps we can come up with ways to monitor what's in the mempool to look for patterns of usage that would motivate further improvements to
@sdaftuar Github tasklists are maybe better for this kind of thing, and they show up as tasks even on ticket lists. https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments
Yes you would definitely expect bigger difference as this becomes available. As entities will be able to underpay transactions without risking getting dropped from the mempool. (for entities who continuously create transactions)
One of the dangers of the blocksize limit is that fees would go up, but have almost no reason/force to get it down. Things like this and RBF could really help with that.
Is this algorithm also used to determine which transactions to drop?
@seweso The algorithm for eviction of transactions from the mempool is unchanged. In a sense, the transaction eviction algorithm for the mempool is already assuming we're doing something like what this PR implements for transaction selection, in that a transaction is less likely to be evicted from the mempool if its feerate with descendants is high. But more fundamentally, mempool eviction is just a different problem than transaction selection: when selecting transactions for a block, a candidate transaction's ancestors must be included (so it's natural to sort by feerate-with-ancestors), but when considering a transaction for mempool eviction, a candidate transaction's descendants must also be removed as well (so we use
Pushed commits addressing @mrbandrews comments.
re: @JeremyRubin's nits, I'm leaving
I'll do another round of testing locally, similar to what I did before when I opened the pull, with the current version of the code and report back.
I re-tested my code by simulating on data from 2016-02-23 -to 2016-02-29, comparing the default mining code in master against the mining code introduced here, by calling CreateNewBlock every 100 transactions.
I looked at the fees produced in the last call to CNB before each block found by the network, and the code in this PR produced higher fee blocks in each instance (894 data points).
I was also benchmarking the runtime of CNB over all invocations, and this code was negligibly faster than the old code (summing over all invocations).
Beyond review, I tested this on several mining nodes for about a week (in addition to prior testing I did months back); I also took nodes running this and a tight getblocktemplate loop through a number of reorg tests with debugging turned up. Finally, rather than using sdaftuar record and replay framework, I scraped all the recent transactions out of recent blocks, reorged the chain back and put all the still-valid transactions into the mempool and confirmed it gave a valid gbt result with higher total fees than the before the patch, I tried this at a dozen different points.