Parallel preprocessing of blocks + transactions #1360

pmconrad · 2018-10-05T15:23:57Z

~~Work in progress, do not merge yet!~~

Will ~~some day~~ resolve #1079 insofar as it is possible without modifying consensus.

This includes the changes from #1359 . Requires bitshares/bitshares-fc#78 .

Status: seems to be working, at least with boost-1.60. Testing.

clockworkgr · 2018-10-05T15:42:17Z

good stuff peter!

pmconrad · 2018-10-05T16:19:06Z

Thanks, but not yet. :-/

clockworkgr · 2018-10-05T16:37:55Z

ah!..got overexcited over the "working version" commit :)

abitmore

Since it's still WIP, perhaps will have answer later.

libraries/chain/include/graphene/chain/protocol/transaction.hpp

abitmore · 2018-10-12T13:04:37Z

libraries/app/application.cpp

+                      graphene::chain::database::skip_tapos_check |
+                      graphene::chain::database::skip_witness_schedule_check;
+      if( _options->count("revalidate-blockchain") ) // see also handle_block()
+         skip = _options->count("force-validate") ? 0 : graphene::chain::database::skip_transaction_signatures;


I think it's better to replace 0 with graphene::chain::database::skip_nothing.

Code readability here looks a bit poor. Now it reads:

declare skip = something; if (condition A) { skip = condition B ? something else : yet something else; }

How about change the if to ?:

declare skip = condition A ? condition B ? something else : yet something else : something;

or change the ?: to if else?

libraries/chain/db_block.cpp

abitmore

And need to resolve FC conflicts.

abitmore · 2018-10-21T16:58:20Z

libraries/app/application.cpp

@@ -961,9 +979,10 @@ void application::set_program_options(boost::program_options::options_descriptio
          "Path to create a Genesis State at. If a well-formed JSON file exists at the path, it will be parsed and any "
          "missing fields in a Genesis State will be added, and any unknown fields will be removed. If no file or an "
          "invalid file is found, it will be replaced with an example Genesis State.")
-         ("replay-blockchain", "Rebuild object graph by replaying all blocks")
+         ("replay-blockchain", "Rebuild object graph by replaying all blocks without validatino")


abitmore · 2018-10-21T17:05:26Z

libraries/chain/db_block.cpp

+         uint32_t chunks = fc::asio::default_io_service_scope::get_num_threads();
+         uint32_t chunk_size = block.transactions.size() / chunks;
+         if( chunks * chunk_size < block.transactions.size() )
+            chunk_size++;


Is it essentially rounding up? If it's true then can code like this:

uint32_t chunk_size = ( block.transactions.size() + chunks - 1 ) / chunks;

abitmore · 2018-10-21T18:30:52Z

libraries/chain/include/graphene/chain/protocol/transaction.hpp

-      void                validate() const;
+      digest_type                        digest()const;
+      virtual const transaction_id_type& id()const;
+      virtual void                       validate() const;
      /// Calculate the digest used for signature validation


This comment should be moved to correct place.

abitmore · 2018-10-21T18:38:59Z

libraries/chain/protocol/transaction.cpp

@@ -307,20 +305,17 @@ const flat_set<public_key_type>& signed_transaction::get_signature_keys( const c
 { try {
   // Strictly we should check whether the given chain ID is same as the one used to initialize the `signees` field.
   // However, we don't pass in another chain ID so far, for better performance, we skip the check.


The comments should be removed if signees are calculated every time.

abitmore · 2018-10-21T18:39:53Z

libraries/chain/protocol/transaction.cpp

@@ -307,20 +305,17 @@ const flat_set<public_key_type>& signed_transaction::get_signature_keys( const c
 { try {
   // Strictly we should check whether the given chain ID is same as the one used to initialize the `signees` field.
   // However, we don't pass in another chain ID so far, for better performance, we skip the check.
-   if( signees.empty() && !signatures.empty() )
+   auto d = sig_digest( chain_id );
+   flat_set<public_key_type> result;


Use a non-default key_compare here?

That wouldn't help, because then the set has to be re-sorted when assigning to _signees field.
Unless we also change the sorting order there, which means the signature of this method has to be changed as well, and all places that use it... it simply doesn't pay out, see #1401 .

abitmore · 2018-10-21T18:44:06Z

libraries/chain/include/graphene/chain/protocol/transaction.hpp

+      precomputable_transaction( const signed_transaction& tx ) : signed_transaction(tx) {};
+      precomputable_transaction( signed_transaction&& tx ) : signed_transaction(tx) {};
+
+      virtual const transaction_id_type&       id()const;


Can add an override keyword to overridden functions here.

abitmore · 2018-10-21T18:49:06Z

libraries/db/object_database.cpp

-            _index[space][type]->save( _data_dir / "object_database.tmp" / fc::to_string(space)/fc::to_string(type) );
+            tasks.push_back( fc::do_parallel( [this,space,type] () {
+               _index[space][type]->save( _data_dir / "object_database.tmp" / fc::to_string(space)/fc::to_string(type) );
+            } ) );


Does this really increase performance? Isn't IO the bottleneck here?

Perhaps better if we adopt the behavior of ChainBase, store not only data but also indexes, to improve startup speed (may slow down shutdown speed though). This belongs to another ticket.

IO isn't necessarily the bottleneck, e. g. when restarting a node the previously saved data might still be in system buffers.

Here is save aka writing but not reading. When we're writing, usually the data has changed, even if it didn't change much, since we're writing to temporary (aka different) files, so the buffer doesn't help.

Of course disk buffer helps when writing.
On a system with a slow disk and the bare minimum of RAM you're right, parallelizing won't help - but it won't hurt either.
With sufficient buffer memory, or on a RAM-disk, parallel saving should be faster.

abitmore · 2018-10-21T19:38:01Z

libraries/chain/db_management.cpp

+         {
+            wlog( "Reindexing terminated due to gap:  Block ${i} does not exist!", ("i", i) );
+            uint32_t dropped_count = 0;
+            while( true )


I still believe this (old) code is inefficient. But it may belong to another ticket with lower priority.

abitmore · 2018-10-21T19:49:36Z

libraries/plugins/delayed_node/delayed_node_plugin.cpp

@@ -105,6 +105,7 @@ void delayed_node_plugin::sync_with_trusted_node()
         fc::optional<graphene::chain::signed_block> block = my->database_api->get_block( db.head_block_num()+1 );
         FC_ASSERT(block, "Trusted node claims it has blocks it doesn't actually have.");
         ilog("Pushing block #${n}", ("n", block->block_num()));
+         db.precompute_parallel( *block, graphene::chain::database::skip_nothing ).wait();


Can adopt same approach used in reindex here for even better performance, IMHO low priority though. Add a TODO?

abitmore · 2018-10-21T19:50:41Z

tests/common/database_fixture.hpp

@@ -170,6 +170,11 @@ extern uint32_t GRAPHENE_TESTING_GENESIS_TIMESTAMP;

 namespace graphene { namespace chain {

+class clearable_block : public signed_block {


Please write some comments for this class.

pmconrad · 2018-11-07T17:07:50Z

Rebased. Will bump fc after bitshares/bitshares-fc#78 has been merged.

pmconrad · 2018-11-07T17:22:52Z

Timings taken up to block 31,000,000 with various core versions (also see #1371), on a quad-core (with hyperthreading) "Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz", 64G RAM, default config:

Version	2.0.180823	develop as of 2018-10-24	#1401	this pr
Replay	2:00:21	1:58:15	1:59:23	2:00:36
Resync	3:35:51	3:46:15	3:44:55	3:24:07
Resync (full validation)	7:15:06	7:21:59	7:18:52	3:48:52
Revalidate	n/a	n/a	2:51:08	2:46:05
Revalidate (full validation)	n/a	n/a	6:13:40	3:45:56

(Resync was done at different times of day, fluctuations due to network conditions are to be expected.)

jmjatlanta

Code looks good. I believe fc is all set. Please fix the conflict and I will approve this after some final testing. Nice work!

…_transaction_signatures

…of signed_transaction

pmconrad · 2018-12-07T18:48:55Z

Rebased on latest develop + bumped fc to latest master

jmjatlanta

Code looks good. Tested with Ubuntu 18.10 / Boost 1.67 / OpenSSL 1.1.0. chain_test runs with no errors detected. Thank you for tackling this.

pmconrad self-assigned this Oct 5, 2018

pmconrad mentioned this pull request Oct 5, 2018

Performance opt pt 1 #1359

Merged

abitmore reviewed Oct 6, 2018

View reviewed changes

libraries/chain/include/graphene/chain/protocol/transaction.hpp Show resolved Hide resolved

libraries/chain/include/graphene/chain/protocol/transaction.hpp Show resolved Hide resolved

pmconrad force-pushed the performance_opt branch 2 times, most recently from e2d32cb to a3a5612 Compare October 10, 2018 15:33

abitmore reviewed Oct 12, 2018

View reviewed changes

pmconrad mentioned this pull request Oct 17, 2018

GPU Accelerated Computing with C and C++ (OpenCL or NVIDIA CUDA) #1385

Closed

17 tasks

abitmore reviewed Oct 21, 2018

View reviewed changes

abitmore mentioned this pull request Oct 23, 2018

Fix #584 authorize transactions with non-immediate owner authorities #1259

Merged

4 tasks

abitmore added this to In progress in Feature Release (201902) via automation Oct 25, 2018

abitmore added this to the 201812 - Feature Release milestone Oct 25, 2018

pmconrad mentioned this pull request Oct 25, 2018

Resolve 1371: sorting of public_keys is inefficient #1401

Closed

abitmore mentioned this pull request Oct 25, 2018

Memory optimization of static_variant bitshares/bitshares-fc#85

Merged

pmconrad force-pushed the performance_opt branch from 32e9619 to 81f7520 Compare November 7, 2018 17:05

pmconrad added 2g Final Review Status indicating the solution passes the test case(s) and is being reviewed for final acceptance. and removed 2f Testing Status indicating the solution is being evaluated against the test case(s) labels Nov 7, 2018

pmconrad mentioned this pull request Nov 8, 2018

Cleanup unused fork_database::_unlinked_index? #939

Closed

jmjatlanta suggested changes Dec 7, 2018

View reviewed changes

pmconrad added 4 commits December 7, 2018 19:00

Removed unused _unlinked_index

8999eac

Fixed fork_db handling in pop_block

6849984

Added --revalidate-blockchain

73d8b08

Use externally provided skip flags for replay

34baaed

pmconrad added 14 commits December 7, 2018 19:00

Removed skip_authority flag because its used interchangably with skip…

bd8ed04

…_transaction_signatures

Removed unused skip_validate flag introduced in #dca5c95 2016-01-04

56dfa9c

Add warning when revalidating with checkpoints

1a78e1b

Make distinction between skip_flag cases clearer

cf2d6e7

Prove that irrelevant signature detection depends on sorting order

d644f20

Fixed typo

0c4c133

Parallelize loading/saving object_database

e5ba271

Introduce precomputable_transaction, and clearable_block in tests

033ddea

Preprocess blocks + transactions in parallel, working version

d2d8b29

Get rid of possibly uninitialized local variable

9946bde

Changed push_transaction to accept precomputable_transaction instead …

cb2244f

…of signed_transaction

Avoid one level of indirection on precompute block

883ec6a

Make pubkey comparator generally available

d032890

Address some review comments

2c01109

pmconrad force-pushed the performance_opt branch from 81f7520 to 2c01109 Compare December 7, 2018 18:48

Feature Release (201902) automation moved this from In progress to Reviewer approved Dec 7, 2018

jmjatlanta approved these changes Dec 7, 2018

View reviewed changes

pmconrad merged commit 1471c05 into develop Dec 14, 2018

Feature Release (201902) automation moved this from Reviewer approved to Done Dec 14, 2018

pmconrad deleted the performance_opt branch December 14, 2018 13:35

pmconrad mentioned this pull request Dec 21, 2018

Testnet X-Mas Sneak Preview Release #1489

Merged

pmconrad mentioned this pull request Jan 2, 2019

Make better use of multi-core servers #1079

Open

17 tasks

MichelSantos mentioned this pull request Feb 7, 2019

Release Notes for the 201902 Feature Release of BitShares Core #1565

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallel preprocessing of blocks + transactions #1360

Parallel preprocessing of blocks + transactions #1360

pmconrad commented Oct 5, 2018 •

edited

Loading

clockworkgr commented Oct 5, 2018

pmconrad commented Oct 5, 2018

clockworkgr commented Oct 5, 2018

abitmore left a comment

abitmore Oct 12, 2018

pmconrad Nov 7, 2018

abitmore left a comment

abitmore Oct 21, 2018

pmconrad Nov 7, 2018

abitmore Oct 21, 2018

pmconrad Nov 7, 2018

abitmore Oct 21, 2018

pmconrad Nov 7, 2018

abitmore Oct 21, 2018

pmconrad Nov 7, 2018

abitmore Oct 21, 2018

pmconrad Nov 7, 2018

abitmore Oct 21, 2018

pmconrad Nov 7, 2018

abitmore Oct 21, 2018

pmconrad Nov 7, 2018

abitmore Nov 7, 2018 •

edited

Loading

pmconrad Nov 8, 2018

abitmore Oct 21, 2018

pmconrad Nov 7, 2018

abitmore Oct 21, 2018

pmconrad Nov 7, 2018

abitmore Oct 21, 2018

pmconrad Nov 7, 2018

pmconrad commented Nov 7, 2018

pmconrad commented Nov 7, 2018

jmjatlanta left a comment •

edited

Loading

pmconrad commented Dec 7, 2018

jmjatlanta left a comment

		@@ -170,6 +170,11 @@ extern uint32_t GRAPHENE_TESTING_GENESIS_TIMESTAMP;

		namespace graphene { namespace chain {

		class clearable_block : public signed_block {

Parallel preprocessing of blocks + transactions #1360

Parallel preprocessing of blocks + transactions #1360

Conversation

pmconrad commented Oct 5, 2018 • edited Loading

clockworkgr commented Oct 5, 2018

pmconrad commented Oct 5, 2018

clockworkgr commented Oct 5, 2018

abitmore left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abitmore left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abitmore Nov 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmconrad commented Nov 7, 2018

pmconrad commented Nov 7, 2018

jmjatlanta left a comment • edited Loading

Choose a reason for hiding this comment

pmconrad commented Dec 7, 2018

jmjatlanta left a comment

Choose a reason for hiding this comment

pmconrad commented Oct 5, 2018 •

edited

Loading

abitmore Nov 7, 2018 •

edited

Loading

jmjatlanta left a comment •

edited

Loading