Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite CreateNewBlock #6898

Merged
merged 6 commits into from Dec 1, 2015
Merged

Rewrite CreateNewBlock #6898

merged 6 commits into from Dec 1, 2015

Conversation

morcos
Copy link
Member

@morcos morcos commented Oct 28, 2015

WARNING: This hasn't been used for mainnet mining yet as far as I know.

NOTE: This includes a commit which changes the default block priority size to 0. This code has been optimized for not including transactions in a priority space. If block priority space > 0 then the code does not offer too much of a performance improvement over the existing code. (EDIT: now its fast regardless)

The mempool is explicitly assumed to be responsible for maintaining consistency of transactions with respect to not spending non-existent outputs, not double spending, script validity and coinbase maturity. Only finality of transactions is checked before assembling a block.

A final call to TestBlockValidity is still performed. If an invalid block has been constructed, then an error is thrown logged and a NULL pointer is returned. (EDIT: reverted to original behavior with more informative error)

This code produces the same blocks as the code it replaces with the following exceptions:

  • The fee rate sort tie breaker is hash instead of priority
  • The priority sort has a secondary tie breaker of hash
  • The priority block is not limited to transactons above AllowFree() (EDIT: fixed)
  • The fee rate sorted part of the block is not limited to transactions above minRelayTxFee (EDIT: fixed)
  • (EDIT: The blocks are the same if you keep scanning to the end to try to fill up the last remaining space, but this code stops after trying 50 times to fill the last 1000 bytes)

Comparing this to the original code over 2000 calls to CreateNewBlock over the last 2 days.
(blockprioritysize=0, maxmempool=300, dbcache=1000, maxsigcachesize=1000000)
Time in ms

Time to assemble block Additional time to perform final TestBlockValidity
master 2602 240
this pull 14 438

The extra slowness in TestBlockValidity is because the cache used to be warmed up in the assembly code.

@laanwj laanwj added the Mining label Oct 31, 2015
@@ -78,7 +80,8 @@ class CTxMemPoolEntry

public:
CTxMemPoolEntry(const CTransaction& _tx, const CAmount& _nFee,
int64_t _nTime, double _dPriority, unsigned int _nHeight, bool poolHasNoInputsOf = false);
int64_t _nTime, double _dPriority, unsigned int _nHeight, bool poolHasNoInputsOf = false,
unsigned int nSigOps = 0);
Copy link
Member

@luke-jr luke-jr Oct 31, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NACK defaulting nSigOps to 0 when we depend on it being accurate.

Copy link
Member Author

@morcos morcos Nov 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok i agree, it makes more sense to default it high.

Copy link
Member

@luke-jr luke-jr Nov 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not make sense to default it period. Note the exact sigop count is part of the GBT response.

Copy link
Member Author

@morcos morcos Nov 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK agreed. The default is just for the tests, I think for now it makes sense to remove all defaults from the constructor and just make the tests fill them in. Unfortunately there are 2 other pulls open which also add arguments to the constructor and require changing the tests. I'm going to hold off updating the tests in this pull until we're a bit closer assuming those might be merged first.

@morcos
Copy link
Member Author

morcos commented Nov 3, 2015

OK, this has been rewritten to use CTxMemPoolEntry::GetPriority for the priority sort.
This will make it very fast even when maintaining a priority space in the block. So the commit changing the default blockprioritysize has been removed.

Of course until #6357 is merged or some other decision is taken regarding priority, it will not sort by an accurate measure of priority.

@morcos
Copy link
Member Author

morcos commented Nov 12, 2015

I addressed @luke-jr 's comment about changing the method of failure return. I still modified it to give some extra state info. That commit and other fixes will get squashed.

Among the remaining outstanding items is to fix the constructor for CTxMemPoolEntry to not have any default arguments. It would be nice to know a merge order since this will conflict with #6357 and #6915 and involves tedious updating of a bunch of tests.

@morcos
Copy link
Member Author

morcos commented Nov 18, 2015

I rebased this and squashed fixes for comments so far.

I rebased it on top of #7008, because I needed to pick some path forward on how priority would be addressed in this PR. The first 3 commits are therefore part of that pull and not this one. If we decide to merge #6357 as well, then it should be an easy change.

I will continue testing, but I don't have any other changes planned right now.

@morcos morcos changed the title [WIP] Rewrite CreateNewBlock Rewrite CreateNewBlock Nov 18, 2015
@morcos
Copy link
Member Author

morcos commented Nov 18, 2015

Thanks @sdaftuar for finding a bug in the way I was doing PrioritiseTransaction. There should probably be RPC tests for those functions.

@morcos
Copy link
Member Author

morcos commented Nov 19, 2015

Apologies for all the churn. Squashed again and rebased off a slightly modified index @sdaftuar created for #7062. This introduces one more minor difference from the original code in that fee rates are compared as doubles now.

If I correct for these differences:

  • using a score index which compares fee rates not doubles
  • make the original code use the hash tie breakers
  • search up 5000 txs to fill the final 1000 bytes
  • implement Dynamic priority calculation #6357 to correctly compute priority

Then I generate the exact same blocks as the old code modulo a few rare cases where double imprecision in the different priority calculation causes tx reordering in the priority space.

if (!TestBlockValidity(state, chainparams, *pblock, pindexPrev, false, false))
throw std::runtime_error("CreateNewBlock(): TestBlockValidity failed");
if (!TestBlockValidity(state, chainparams, *pblock, pindexPrev, false, false)) {
throw std::runtime_error("CreateNewBlock(): TestBlockValidity failed: " + FormatStateMessage(state));
Copy link
Member

@jtimon jtimon Nov 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you strprintf("%: TestBlockValidity failed: %", __func__, ... instead?

@sdaftuar
Copy link
Member

sdaftuar commented Nov 24, 2015

Addressed @jtimon's nit.

@btcdrak
Copy link
Contributor

btcdrak commented Nov 29, 2015

Needs rebase.

@jameshilliard
Copy link
Contributor

jameshilliard commented Nov 29, 2015

TestBlockValidity should probably be done in a background thread so that it does not delay GBT since TestBlockValidity is just a sanity check.

@sipa
Copy link
Member

sipa commented Dec 1, 2015

ACK

Use the score index on the mempool to only add sorted txs in order.  Remove much of the validation while building the block, relying on mempool to be consistent and only contain txs that can be mined.
The mempool is assumed to be consistent as far as not containing txs which spend non-existent outputs or double spends, and scripts are valid.  Finality of txs is still checked (except not coinbase maturity, assumed in mempool).
Still TestBlockValidity in case mempool consistency breaks and return error state if an invalid block was created.
Unit tests are modified to realize that invalid blocks can now be constructed if the mempool breaks its consistency assumptions and also updated to have the right fees, since the cached value is now used for block construction.

Conflicts:
	src/miner.cpp
@morcos
Copy link
Member Author

morcos commented Dec 1, 2015

rebased

@sipa sipa merged commit 553cad9 into bitcoin:master Dec 1, 2015
1 check passed
sipa added a commit that referenced this issue Dec 1, 2015
553cad9 Rewrite CreateNewBlock (Alex Morcos)
5f12263 Expose FormatStateMessage (Alex Morcos)
1f09287 Make accessing mempool parents and children public (Alex Morcos)
7230187 Add TxPriority class and comparator (Alex Morcos)
f3fe836 Add a score index to the mempool. (Alex Morcos)
c49d5bc Store the total sig op count of a tx. (Alex Morcos)
@luke-jr
Copy link
Member

luke-jr commented Dec 1, 2015

This was not ready for merging. It needs #6357 first.

luke-jr added a commit to luke-jr/bitcoin that referenced this issue Dec 1, 2015
The priority area is often our only defense against spam, so disabling it should not be encouraged.
Additionally, the only benefit for disabling it is performance, which is no longer an issue with bitcoin#6898.

While there may be arguably better policies for handling priority, these are not yet supported (nor even evaluated theoretically or tested in practice).
Until those are implemented and tested, we shouldn't regress default policy.
@laanwj
Copy link
Member

laanwj commented Dec 2, 2015

This was not ready for merging. It needs #6357 first.

#7008, which took the place of #6537 was merged

MarkLTZ added a commit to litecoinz-project/litecoinz that referenced this issue Apr 27, 2019
bitcoin/bitcoin#6898: Rewrite CreateNewBlock
bitcoin/bitcoin#7187: Keep reorgs fast for SequenceLocks checks
random-zebra added a commit to PIVX-Project/PIVX that referenced this issue Jun 27, 2020
a0de4da [Wallet] Abandon tx in CommitTransaction if ATMP fails (random-zebra)
f3e07f7 Expose FormatStateMessage (Alex Morcos)

Pull request description:

  This is an alternative to #1664.
  It takes the opposite approach.
  When ATMP fails in `CommitTransaction`, instead of just returning the txid (hoping that the failure is not permanent) try to directly abandon the transaction.
  Save the result of these operations in a new struct `CWallet::CommitResult`, and use it to give a detailed feedback to the user (via log lines, JSON-RPC errors and GUI dialogs).

  Based on top of:
  - [x] #1670

  This PR also cherry-picks 5f12263 from bitcoin#6898, in order to use `FormatStateMessage` outside of main.

ACKs for top commit:
  furszy:
    ACK a0de4da
  Fuzzbawls:
    ACK a0de4da

Tree-SHA512: 5ae372f97eeed017002628646eee3f559af6dc488e1b24541b29f5ce763412fb569e057933468fc828a2aec3f2c17e2d818130b72d41c73524cfa90680877d23
random-zebra added a commit to PIVX-Project/PIVX that referenced this issue Aug 18, 2020
6c566bb Add TxPriority class and comparator (Alex Morcos)
5e9df89 Make accessing mempool parents and children public (Alex Morcos)
1279a9f Add a score index to the mempool (furszy)
e5a72fe Store the total sig op count of a tx (furszy)

Pull request description:

  On top of #1707 and #1717 .

  Back ports bitcoin#6898 (without back porting the last commit as we need some cleanup on the miner area first).

ACKs for top commit:
  random-zebra:
    ACK 6c566bb
  Fuzzbawls:
    utACK 6c566bb

Tree-SHA512: c6fa788c59ea6e024e340e926e81462d0817617003034eecf9c3eeda3dfd5fbd2a7b6ec1ada3b867dba98079a195931f8a72fc2250207f475d8f44060452825b
Fuzzbawls added a commit to PIVX-Project/PIVX that referenced this issue Feb 3, 2021
…est coverage

0d55bcf blockassembler: do not lock cs_main and mempool cs for the entire block, only where are needed. (furszy)
70e6430 miner_tests fix locking issues, cs_main and mempool.cs cannot (and don't need) to be locked for the entire test. Only in the places that the locks are really needed. (furszy)
c4e3754 [test] functional miner_test, reworked and updated it to make it fully work with our blockchain and consensus rules. (furszy)
1cd3943 [Miner] New blockassembler introduction and connection with the sources. (furszy)

Pull request description:

  Have rewritten the miner code sources, dividing the miner thread from the block assembling process, encapsulating the block creation state inside a new `BlockAssembler` class (adapting bitcoin#7598).

  Cleaned several mempool redundant checks inside the assembly process (adaptation of bitcoin#6898):
  > The mempool is explicitly assumed to be responsible for maintaining consistency of transactions with respect to not spending non-existent outputs, not double spending, script validity and coinbase maturity. Only finality of transactions is checked before assembling a block.

  Another point added here is the rework of the previously non-functional `miner_tests` unit test. Which.. have essentially made it work, adding coverage for the miner process, updating it to a fairly recent point down upstream's path + adapting it to our blockchain and consensus rules.

  The result is a large speed up in the block assembly process, a much better and cleaner code architecture and enable an easier add of new algorithms for block filling in the future.

  ------------

  A good work path on top of this would be to add unit test coverage for PoS, cold staking and Sapling block creation (among others). At the moment, most of the block creation tests are functional ones which are slow and require to setup entire nodes etc.

ACKs for top commit:
  random-zebra:
    utACK 0d55bcf after rebase
  Fuzzbawls:
    ACK 0d55bcf

Tree-SHA512: ab0ff3832bceeb28233871a64ee9593cee695843396b95f9471485bf1d9e684614db783342252bcfb49cb3df40c444697d24e3dab1f8a5226b2478bc191822d2
zkbot added a commit to zcash/zcash that referenced this issue Aug 10, 2021
ZIP 239 preparations 2

Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#6722
  - Only the ancillary commits, not the mempool limiting commits (we have our own).
- bitcoin/bitcoin#6898
  - Only the first three commits (we'll cherry-pick the main content later).
- bitcoin/bitcoin#7840
zkbot added a commit to zcash/zcash that referenced this issue Aug 11, 2021
ZIP 239 preparations 2

Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#6722
  - Only the ancillary commits, not the mempool limiting commits (we have our own).
- bitcoin/bitcoin#6898
  - Only the first three commits (we'll cherry-pick the main content later).
- bitcoin/bitcoin#7840
zkbot added a commit to zcash/zcash that referenced this issue Aug 12, 2021
ZIP 239 preparations 2

Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#6722
  - Only the ancillary commits, not the mempool limiting commits (we have our own).
- bitcoin/bitcoin#6898
  - Only the first three commits (we'll cherry-pick the main content later).
- bitcoin/bitcoin#7840
zkbot added a commit to zcash/zcash that referenced this issue Aug 13, 2021
ZIP 239 preparations 2

Cherry-picked from the following upstream PRs:
- bitcoin/bitcoin#6722
  - Only the ancillary commits, not the mempool limiting commits (we have our own).
- bitcoin/bitcoin#6898
  - Only the first three commits (we'll cherry-pick the main content later).
- bitcoin/bitcoin#7840
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants