Rewrite CreateNewBlock #6898
OK, this has been rewritten to use
Of course until #6357 is merged or some other decision is taken regarding priority, it will not sort by an accurate measure of priority.
I addressed @luke-jr 's comment about changing the method of failure return. I still modified it to give some extra state info. That commit and other fixes will get squashed.
Among the remaining outstanding items is to fix the constructor for CTxMemPoolEntry to not have any default arguments. It would be nice to know a merge order since this will conflict with #6357 and #6915 and involves tedious updating of a bunch of tests.
I rebased this and squashed fixes for comments so far.
I rebased it on top of #7008, because I needed to pick some path forward on how priority would be addressed in this PR. The first 3 commits are therefore part of that pull and not this one. If we decide to merge #6357 as well, then it should be an easy change.
I will continue testing, but I don't have any other changes planned right now.
Apologies for all the churn. Squashed again and rebased off a slightly modified index @sdaftuar created for #7062. This introduces one more minor difference from the original code in that fee rates are compared as doubles now.
If I correct for these differences:
Then I generate the exact same blocks as the old code modulo a few rare cases where double imprecision in the different priority calculation causes tx reordering in the priority space.
Use the score index on the mempool to only add sorted txs in order. Remove much of the validation while building the block, relying on mempool to be consistent and only contain txs that can be mined. The mempool is assumed to be consistent as far as not containing txs which spend non-existent outputs or double spends, and scripts are valid. Finality of txs is still checked (except not coinbase maturity, assumed in mempool). Still TestBlockValidity in case mempool consistency breaks and return error state if an invalid block was created. Unit tests are modified to realize that invalid blocks can now be constructed if the mempool breaks its consistency assumptions and also updated to have the right fees, since the cached value is now used for block construction. Conflicts: src/miner.cpp
553cad9 Rewrite CreateNewBlock (Alex Morcos) 5f12263 Expose FormatStateMessage (Alex Morcos) 1f09287 Make accessing mempool parents and children public (Alex Morcos) 7230187 Add TxPriority class and comparator (Alex Morcos) f3fe836 Add a score index to the mempool. (Alex Morcos) c49d5bc Store the total sig op count of a tx. (Alex Morcos)
The priority area is often our only defense against spam, so disabling it should not be encouraged. Additionally, the only benefit for disabling it is performance, which is no longer an issue with bitcoin#6898. While there may be arguably better policies for handling priority, these are not yet supported (nor even evaluated theoretically or tested in practice). Until those are implemented and tested, we shouldn't regress default policy.
a0de4da [Wallet] Abandon tx in CommitTransaction if ATMP fails (random-zebra) f3e07f7 Expose FormatStateMessage (Alex Morcos) Pull request description: This is an alternative to #1664. It takes the opposite approach. When ATMP fails in `CommitTransaction`, instead of just returning the txid (hoping that the failure is not permanent) try to directly abandon the transaction. Save the result of these operations in a new struct `CWallet::CommitResult`, and use it to give a detailed feedback to the user (via log lines, JSON-RPC errors and GUI dialogs). Based on top of: - [x] #1670 This PR also cherry-picks 5f12263 from bitcoin#6898, in order to use `FormatStateMessage` outside of main. ACKs for top commit: furszy: ACK a0de4da Fuzzbawls: ACK a0de4da Tree-SHA512: 5ae372f97eeed017002628646eee3f559af6dc488e1b24541b29f5ce763412fb569e057933468fc828a2aec3f2c17e2d818130b72d41c73524cfa90680877d23
6c566bb Add TxPriority class and comparator (Alex Morcos) 5e9df89 Make accessing mempool parents and children public (Alex Morcos) 1279a9f Add a score index to the mempool (furszy) e5a72fe Store the total sig op count of a tx (furszy) Pull request description: On top of #1707 and #1717 . Back ports bitcoin#6898 (without back porting the last commit as we need some cleanup on the miner area first). ACKs for top commit: random-zebra: ACK 6c566bb Fuzzbawls: utACK 6c566bb Tree-SHA512: c6fa788c59ea6e024e340e926e81462d0817617003034eecf9c3eeda3dfd5fbd2a7b6ec1ada3b867dba98079a195931f8a72fc2250207f475d8f44060452825b
…est coverage 0d55bcf blockassembler: do not lock cs_main and mempool cs for the entire block, only where are needed. (furszy) 70e6430 miner_tests fix locking issues, cs_main and mempool.cs cannot (and don't need) to be locked for the entire test. Only in the places that the locks are really needed. (furszy) c4e3754 [test] functional miner_test, reworked and updated it to make it fully work with our blockchain and consensus rules. (furszy) 1cd3943 [Miner] New blockassembler introduction and connection with the sources. (furszy) Pull request description: Have rewritten the miner code sources, dividing the miner thread from the block assembling process, encapsulating the block creation state inside a new `BlockAssembler` class (adapting bitcoin#7598). Cleaned several mempool redundant checks inside the assembly process (adaptation of bitcoin#6898): > The mempool is explicitly assumed to be responsible for maintaining consistency of transactions with respect to not spending non-existent outputs, not double spending, script validity and coinbase maturity. Only finality of transactions is checked before assembling a block. Another point added here is the rework of the previously non-functional `miner_tests` unit test. Which.. have essentially made it work, adding coverage for the miner process, updating it to a fairly recent point down upstream's path + adapting it to our blockchain and consensus rules. The result is a large speed up in the block assembly process, a much better and cleaner code architecture and enable an easier add of new algorithms for block filling in the future. ------------ A good work path on top of this would be to add unit test coverage for PoS, cold staking and Sapling block creation (among others). At the moment, most of the block creation tests are functional ones which are slow and require to setup entire nodes etc. ACKs for top commit: random-zebra: utACK 0d55bcf after rebase Fuzzbawls: ACK 0d55bcf Tree-SHA512: ab0ff3832bceeb28233871a64ee9593cee695843396b95f9471485bf1d9e684614db783342252bcfb49cb3df40c444697d24e3dab1f8a5226b2478bc191822d2