Rewrite CreateNewBlock #6898

morcos · 2015-10-28T20:58:09Z

WARNING: This hasn't been used for mainnet mining yet as far as I know.

NOTE: This includes a commit which changes the default block priority size to 0. This code has been optimized for not including transactions in a priority space. If block priority space > 0 then the code does not offer too much of a performance improvement over the existing code. (EDIT: now its fast regardless)

The mempool is explicitly assumed to be responsible for maintaining consistency of transactions with respect to not spending non-existent outputs, not double spending, script validity and coinbase maturity. Only finality of transactions is checked before assembling a block.

A final call to TestBlockValidity is still performed. If an invalid block has been constructed, then an error is thrown ~~logged and a NULL pointer is returned.~~ (EDIT: reverted to original behavior with more informative error)

This code produces the same blocks as the code it replaces with the following exceptions:

The fee rate sort tie breaker is hash instead of priority
The priority sort has a secondary tie breaker of hash
~~The priority block is not limited to transactons above AllowFree()~~ (EDIT: fixed)
~~The fee rate sorted part of the block is not limited to transactions above minRelayTxFee~~ (EDIT: fixed)
(EDIT: The blocks are the same if you keep scanning to the end to try to fill up the last remaining space, but this code stops after trying 50 times to fill the last 1000 bytes)

Comparing this to the original code over 2000 calls to CreateNewBlock over the last 2 days.
(blockprioritysize=0, maxmempool=300, dbcache=1000, maxsigcachesize=1000000)
Time in ms

	Time to assemble block	Additional time to perform final TestBlockValidity
master	2602	240
this pull	14	438

The extra slowness in TestBlockValidity is because the cache used to be warmed up in the assembly code.

luke-jr · 2015-10-31T18:30:00Z

src/txmempool.h

@@ -78,7 +80,8 @@ class CTxMemPoolEntry

 public:
    CTxMemPoolEntry(const CTransaction& _tx, const CAmount& _nFee,
-                    int64_t _nTime, double _dPriority, unsigned int _nHeight, bool poolHasNoInputsOf = false);
+                    int64_t _nTime, double _dPriority, unsigned int _nHeight, bool poolHasNoInputsOf = false,
+                    unsigned int nSigOps = 0);


NACK defaulting nSigOps to 0 when we depend on it being accurate.

ok i agree, it makes more sense to default it high.

It does not make sense to default it period. Note the exact sigop count is part of the GBT response.

OK agreed. The default is just for the tests, I think for now it makes sense to remove all defaults from the constructor and just make the tests fill them in. Unfortunately there are 2 other pulls open which also add arguments to the constructor and require changing the tests. I'm going to hold off updating the tests in this pull until we're a bit closer assuming those might be merged first.

morcos · 2015-11-03T16:27:42Z

OK, this has been rewritten to use CTxMemPoolEntry::GetPriority for the priority sort.
This will make it very fast even when maintaining a priority space in the block. So the commit changing the default blockprioritysize has been removed.

Of course until #6357 is merged or some other decision is taken regarding priority, it will not sort by an accurate measure of priority.

morcos · 2015-11-12T14:02:19Z

I addressed @luke-jr 's comment about changing the method of failure return. I still modified it to give some extra state info. That commit and other fixes will get squashed.

Among the remaining outstanding items is to fix the constructor for CTxMemPoolEntry to not have any default arguments. It would be nice to know a merge order since this will conflict with #6357 and #6915 and involves tedious updating of a bunch of tests.

morcos · 2015-11-18T04:53:15Z

I rebased this and squashed fixes for comments so far.

I rebased it on top of #7008, because I needed to pick some path forward on how priority would be addressed in this PR. The first 3 commits are therefore part of that pull and not this one. If we decide to merge #6357 as well, then it should be an easy change.

I will continue testing, but I don't have any other changes planned right now.

morcos · 2015-11-18T20:10:58Z

Thanks @sdaftuar for finding a bug in the way I was doing PrioritiseTransaction. There should probably be RPC tests for those functions.

morcos · 2015-11-19T18:33:26Z

Apologies for all the churn. Squashed again and rebased off a slightly modified index @sdaftuar created for #7062. This introduces one more minor difference from the original code in that fee rates are compared as doubles now.

If I correct for these differences:

using a score index which compares fee rates not doubles
make the original code use the hash tie breakers
search up 5000 txs to fill the final 1000 bytes
implement Dynamic priority calculation #6357 to correctly compute priority

Then I generate the exact same blocks as the old code modulo a few rare cases where double imprecision in the different priority calculation causes tx reordering in the priority space.

jtimon · 2015-11-23T13:06:51Z

src/miner.cpp

-        if (!TestBlockValidity(state, chainparams, *pblock, pindexPrev, false, false))
-            throw std::runtime_error("CreateNewBlock(): TestBlockValidity failed");
+        if (!TestBlockValidity(state, chainparams, *pblock, pindexPrev, false, false)) {
+            throw std::runtime_error("CreateNewBlock(): TestBlockValidity failed: " + FormatStateMessage(state));


can you strprintf("%: TestBlockValidity failed: %", __func__, ... instead?

sdaftuar · 2015-11-24T13:43:54Z

Addressed @jtimon's nit.

btcdrak · 2015-11-29T08:20:41Z

Needs rebase.

jameshilliard · 2015-11-29T10:00:52Z

TestBlockValidity should probably be done in a background thread so that it does not delay GBT since TestBlockValidity is just a sanity check.

sipa · 2015-12-01T16:26:44Z

ACK

Use the score index on the mempool to only add sorted txs in order. Remove much of the validation while building the block, relying on mempool to be consistent and only contain txs that can be mined. The mempool is assumed to be consistent as far as not containing txs which spend non-existent outputs or double spends, and scripts are valid. Finality of txs is still checked (except not coinbase maturity, assumed in mempool). Still TestBlockValidity in case mempool consistency breaks and return error state if an invalid block was created. Unit tests are modified to realize that invalid blocks can now be constructed if the mempool breaks its consistency assumptions and also updated to have the right fees, since the cached value is now used for block construction. Conflicts: src/miner.cpp

morcos · 2015-12-01T17:10:56Z

rebased

553cad9 Rewrite CreateNewBlock (Alex Morcos) 5f12263 Expose FormatStateMessage (Alex Morcos) 1f09287 Make accessing mempool parents and children public (Alex Morcos) 7230187 Add TxPriority class and comparator (Alex Morcos) f3fe836 Add a score index to the mempool. (Alex Morcos) c49d5bc Store the total sig op count of a tx. (Alex Morcos)

luke-jr · 2015-12-01T20:18:30Z

This was not ready for merging. It needs #6357 first.

The priority area is often our only defense against spam, so disabling it should not be encouraged. Additionally, the only benefit for disabling it is performance, which is no longer an issue with bitcoin#6898. While there may be arguably better policies for handling priority, these are not yet supported (nor even evaluated theoretically or tested in practice). Until those are implemented and tested, we shouldn't regress default policy.

laanwj · 2015-12-02T09:26:08Z

This was not ready for merging. It needs #6357 first.

#7008, which took the place of #6537 was merged

a0de4da [Wallet] Abandon tx in CommitTransaction if ATMP fails (random-zebra) f3e07f7 Expose FormatStateMessage (Alex Morcos) Pull request description: This is an alternative to #1664. It takes the opposite approach. When ATMP fails in `CommitTransaction`, instead of just returning the txid (hoping that the failure is not permanent) try to directly abandon the transaction. Save the result of these operations in a new struct `CWallet::CommitResult`, and use it to give a detailed feedback to the user (via log lines, JSON-RPC errors and GUI dialogs). Based on top of: - [x] #1670 This PR also cherry-picks 5f12263 from bitcoin#6898, in order to use `FormatStateMessage` outside of main. ACKs for top commit: furszy: ACK a0de4da Fuzzbawls: ACK a0de4da Tree-SHA512: 5ae372f97eeed017002628646eee3f559af6dc488e1b24541b29f5ce763412fb569e057933468fc828a2aec3f2c17e2d818130b72d41c73524cfa90680877d23

6c566bb Add TxPriority class and comparator (Alex Morcos) 5e9df89 Make accessing mempool parents and children public (Alex Morcos) 1279a9f Add a score index to the mempool (furszy) e5a72fe Store the total sig op count of a tx (furszy) Pull request description: On top of #1707 and #1717 . Back ports bitcoin#6898 (without back porting the last commit as we need some cleanup on the miner area first). ACKs for top commit: random-zebra: ACK 6c566bb Fuzzbawls: utACK 6c566bb Tree-SHA512: c6fa788c59ea6e024e340e926e81462d0817617003034eecf9c3eeda3dfd5fbd2a7b6ec1ada3b867dba98079a195931f8a72fc2250207f475d8f44060452825b

…est coverage 0d55bcf blockassembler: do not lock cs_main and mempool cs for the entire block, only where are needed. (furszy) 70e6430 miner_tests fix locking issues, cs_main and mempool.cs cannot (and don't need) to be locked for the entire test. Only in the places that the locks are really needed. (furszy) c4e3754 [test] functional miner_test, reworked and updated it to make it fully work with our blockchain and consensus rules. (furszy) 1cd3943 [Miner] New blockassembler introduction and connection with the sources. (furszy) Pull request description: Have rewritten the miner code sources, dividing the miner thread from the block assembling process, encapsulating the block creation state inside a new `BlockAssembler` class (adapting bitcoin#7598). Cleaned several mempool redundant checks inside the assembly process (adaptation of bitcoin#6898): > The mempool is explicitly assumed to be responsible for maintaining consistency of transactions with respect to not spending non-existent outputs, not double spending, script validity and coinbase maturity. Only finality of transactions is checked before assembling a block. Another point added here is the rework of the previously non-functional `miner_tests` unit test. Which.. have essentially made it work, adding coverage for the miner process, updating it to a fairly recent point down upstream's path + adapting it to our blockchain and consensus rules. The result is a large speed up in the block assembly process, a much better and cleaner code architecture and enable an easier add of new algorithms for block filling in the future. ------------ A good work path on top of this would be to add unit test coverage for PoS, cold staking and Sapling block creation (among others). At the moment, most of the block creation tests are functional ones which are slow and require to setup entire nodes etc. ACKs for top commit: random-zebra: utACK 0d55bcf after rebase Fuzzbawls: ACK 0d55bcf Tree-SHA512: ab0ff3832bceeb28233871a64ee9593cee695843396b95f9471485bf1d9e684614db783342252bcfb49cb3df40c444697d24e3dab1f8a5226b2478bc191822d2

ZIP 239 preparations 2 Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#6722 - Only the ancillary commits, not the mempool limiting commits (we have our own). - bitcoin/bitcoin#6898 - Only the first three commits (we'll cherry-pick the main content later). - bitcoin/bitcoin#7840

laanwj added the Mining label Oct 31, 2015

luke-jr reviewed Oct 31, 2015
View reviewed changes

morcos force-pushed the fasterCNB branch from 8f84ec4 to 8c114f8 Compare November 3, 2015 16:13

morcos force-pushed the fasterCNB branch from 8c114f8 to 37d224f Compare November 3, 2015 20:55

morcos mentioned this pull request Nov 3, 2015

[WIP] Keep pcoinsTip cache warm #6936

Closed

gmaxwell added this to the 0.12.0 milestone Nov 5, 2015

morcos mentioned this pull request Nov 9, 2015

Summary of potential performance improvements #6976

Closed

sipa mentioned this pull request Nov 12, 2015

the RPC should have commands to specify the side of a fork considered 'current' #6995

Closed

morcos force-pushed the fasterCNB branch from 0a8b3c5 to ea65c75 Compare November 12, 2015 17:59

morcos mentioned this pull request Nov 13, 2015

Lower bound priority #7008

Merged

morcos force-pushed the fasterCNB branch from 57af58b to ee93e49 Compare November 15, 2015 22:28

morcos mentioned this pull request Nov 17, 2015

Implement helper class for CTxMemPoolEntry constructor #7020

Merged

morcos force-pushed the fasterCNB branch from ee93e49 to 55af044 Compare November 18, 2015 04:50

morcos changed the title ~~[WIP] Rewrite CreateNewBlock~~ Rewrite CreateNewBlock Nov 18, 2015

morcos force-pushed the fasterCNB branch from 55af044 to 252a4cc Compare November 18, 2015 20:10

morcos force-pushed the fasterCNB branch from 252a4cc to 50720dc Compare November 18, 2015 20:41

sdaftuar mentioned this pull request Nov 19, 2015

[Mempool] Fix mempool limiting and replace-by-fee for PrioritiseTransaction #7062

Merged

morcos force-pushed the fasterCNB branch from 2dc6d24 to 97a5ea7 Compare November 19, 2015 18:05

sdaftuar mentioned this pull request Nov 19, 2015

[Tests] Add prioritisetransaction RPC test #7063

Closed

morcos force-pushed the fasterCNB branch from 97a5ea7 to b0fb264 Compare November 19, 2015 18:56

jtimon reviewed Nov 23, 2015
View reviewed changes

Expose FormatStateMessage

5f12263

morcos force-pushed the fasterCNB branch from 5e5574b to 553cad9 Compare December 1, 2015 17:10

sipa merged commit 553cad9 into bitcoin:master Dec 1, 2015

luke-jr mentioned this pull request Dec 1, 2015

Revert default block priority size to 50k #7151

Closed

This was referenced Dec 2, 2015

BIP-68: Mempool-only sequence number constraint verification #6312

Closed

Performance fix propositions for CNB in case of #6312 merging #7176

Closed

dgenr8 mentioned this pull request Dec 29, 2016

Rewrite CreateNewBlock bitcoinxt/bitcoinxt#182

Merged

Sjors mentioned this pull request Sep 3, 2017

Check specific validation error in miner tests #11220

Merged

random-zebra mentioned this pull request Jun 18, 2020

[Wallet] Abandon tx in CommitTransaction if ATMP fails PIVX-Project/PIVX#1695

Merged

1 task

This was referenced Jul 3, 2020

[Backport] mempool score index. PIVX-Project/PIVX#1725

Merged

Up to CTransactionRef point back ports to do list. PIVX-Project/PIVX#1726

Closed

[WIP] Miner code reworked from upstream 6898 PIVX-Project/PIVX#1727

Closed

furszy mentioned this pull request Jan 25, 2021

[Miner] Rewrite miner code with proper encapsulation + test coverage PIVX-Project/PIVX#2155

Merged

str4d mentioned this pull request Aug 10, 2021

ZIP 239 preparations 2 zcash/zcash#5269

Merged

str4d mentioned this pull request Aug 13, 2021

ZIP 239 preparations 3 zcash/zcash#5274

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite CreateNewBlock #6898

Rewrite CreateNewBlock #6898

morcos commented Oct 28, 2015

luke-jr Oct 31, 2015

morcos Nov 1, 2015

luke-jr Nov 1, 2015

morcos Nov 3, 2015

morcos commented Nov 3, 2015

morcos commented Nov 12, 2015

morcos commented Nov 18, 2015

morcos commented Nov 18, 2015

morcos commented Nov 19, 2015

jtimon Nov 23, 2015

sdaftuar commented Nov 24, 2015

btcdrak commented Nov 29, 2015

jameshilliard commented Nov 29, 2015

sipa commented Dec 1, 2015

morcos commented Dec 1, 2015

luke-jr commented Dec 1, 2015

laanwj commented Dec 2, 2015

Rewrite CreateNewBlock #6898

Rewrite CreateNewBlock #6898

Conversation

morcos commented Oct 28, 2015

luke-jr Oct 31, 2015

Choose a reason for hiding this comment

morcos Nov 1, 2015

Choose a reason for hiding this comment

luke-jr Nov 1, 2015

Choose a reason for hiding this comment

morcos Nov 3, 2015

Choose a reason for hiding this comment

morcos commented Nov 3, 2015

morcos commented Nov 12, 2015

morcos commented Nov 18, 2015

morcos commented Nov 18, 2015

morcos commented Nov 19, 2015

jtimon Nov 23, 2015

Choose a reason for hiding this comment

sdaftuar commented Nov 24, 2015

btcdrak commented Nov 29, 2015

jameshilliard commented Nov 29, 2015

sipa commented Dec 1, 2015

morcos commented Dec 1, 2015

luke-jr commented Dec 1, 2015

laanwj commented Dec 2, 2015