Benchmark the framework parameters to suggest its values

[MaciejBaj]

Expected behavior

For all of the exposed parameters by chain module suggest the minimum and maximum values. Design the best-fitting benchmarking methods for:

• REWARDS
  • MILESTONES
  • OFFSET (Start rewards at first block of the second round)
  • DISTANCE (Distance between each milestone)
• MAX_TRANSACTIONS_PER_BLOCK
• EPOCH_TIME
• BLOCK_TIME (currently defined in slots.interval)

Which version(s) does this affect? (Environment, OS, etc...)

1.6.0

[nazarhussain]

@MaciejBaj @shuse2

Had tested following constants or genesis configurations and these works well.

EPOCH_TIME

• Changed it to "2009-06-15T13:45:30Z"
• Started the app
• Call curl localhost:4000/api/blocks
• The timestamp of the firs block returned 311214370
• If you test the value by moment.unix(moment("2009-06-15T13:45:30Z").unix()+ 311214370).format()
• It returns the 2019-04-26T16:11:40+02:00 clock time which is the accurately correct for that block.

BLOCK_TIME

• Changed this value to 2
• Started the application form fresh database
• Checked the logs and it shows...

[inf] 2019-04-26 15:23:16 | Forged new block id: 18206333733380695284 height: 51 round: 1 slot: 155609333 reward: 0
[inf] 2019-04-26 15:23:18 | Forged new block id: 4378758914465915819 height: 52 round: 1 slot: 155609334 reward: 0
[inf] 2019-04-26 15:23:20 | Forged new block id: 10097355344923248960 height: 53 round: 1 slot: 155609335 reward: 0
[inf] 2019-04-26 15:23:22 | Forged new block id: 2322819857020936829 height: 54 round: 1 slot: 155609336 reward: 0
...
...

• As clear in above logs that the forging of blocks shows the accurate result as per block time value.

MAX_TRANSACTIONS_PER_BLOCK

• Changed this value to 2
• Started the app with fresh database
• Run this test file npm run mocha:functional:post test/mocha/functional/http/post/0.transfer.js
• Executed this command to get the blocks with more transactions than specified and its just one genesis block.

curl -X GET  "localhost:4000/api/blocks?sort=height:asc"  | jq -c '.data[] | select( .numberOfTransactions > 2)'

{"id":"6524861224470851795","version":0,"timestamp":0,"height":1,"numberOfTransactions":103,"totalAmount":"10000000000000000","totalFee":"0","reward":"0","payloadLength":19619,"payloadHash":"198f2b61a8eb95fbeed58b8216780b68f697f26b849acf00c8c93bb9b24f783d","generatorPublicKey":"c96dec3595ff6041c3bd28b76b8cf75dce8225173d1bd00241624ee89b50f2a8","blockSignature":"c81204bf67474827fd98584e7787084957f42ce8041e713843dd2bb352b73e81143f68bd74b06da8372c43f5e26406c4e7250bbd790396d85dea50d448d62606","confirmations":59,"totalForged":"0","generatorAddress":"1085993630748340485L","previousBlockId":""}

• Executed following command to check blocks with same limit or less.

curl -X GET  "localhost:4000/api/blocks?sort=height:asc"  | jq -c '.data[] | select( .numberOfTransactions <= 2)'

{"id":"1372423278544869493","version":1,"timestamp":311219030,"height":2,"numberOfTransactions":0,"totalAmount":"0","totalFee":"0","reward":"0","payloadLength":0,"payloadHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","generatorPublicKey":"1cc68fa0b12521158e09779fd5978ccc0ac26bf99320e00a9549b542dd9ada16","blockSignature":"75dcde7512ac2e203292e0d8204fbd550cac5b309181246b120d8b6b17fe6068a2dd0c3d1bb46227b4d0ad3324a124a551c9c925d2c55fca99ea2a9fb2e3b90b","confirmations":68,"totalForged":"0","generatorAddress":"6719024567117648644L","previousBlockId":"6524861224470851795"}
{"id":"4301677408258893809","version":1,"timestamp":311219040,"height":3,"numberOfTransactions":0,"totalAmount":"0","totalFee":"0","reward":"0","payloadLength":0,"payloadHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","generatorPublicKey":"e6d075e3e396673c853210f74f8fe6db5e814c304bb9cd7f362018881a21f76c","blockSignature":"0bffbf5ebda20941d3bd5406ce715a14e1eba5ebbb952ca45cf00193ae8408ca1118a250ae8bf50fe6599c215d11690b7f1165fc8a7fb8e5eabfe3ae27e8840f","confirmations":67,"totalForged":"0","generatorAddress":"13227119536266737242L","previousBlockId":"1372423278544869493"}

• All the blocks have same number of transactions as specified in MAX_TRANSACTIONS_PER_BLOCK

REWARDS

We are already using different rewards values for different networks, so its obviously having the right impact if changed.

[nazarhussain]

To benchmark the right min/max range for MAX_TRANSACTIONS_PER_BLOCK and BLOCK_TIME is very difficult for the reason that both are co-related to each other. As well different physical parameters can effect the performance as well.

I suggest to do this after benchmarking of the new transaction processing.

[MaciejBaj]

We should be able to give "healthy" suggestions on the value ranges.

[pablitovicente]

Limits on values for BLOCK_TIME and MAX_TRANSACTIONS_PER_BLOCK (these values were obtained by trying against a single node (node specs: 1GB Ram, 1CPU core, 25 GB HDD and later confirmed against a network consisting of 50 nodes distributed in 10 different regions) it’s important to note that these values do not represent limits in the SDK but recommended values for the current defaults for other configuration properties and objects configurations used by the SDK and lisk-core.

BLOCK_TIME	MAX_TRANSACTIONS_PER_BLOCK
2	25
4	50
6	75
8	110
10	150

• With all this configurations blocks were generated correctly, on time and filling all blocks with the configured number of transactions.

• We construct the TransactionPool object using some default limits that impact the behaviour of transactions per block; for example we’re using releaseLimit for setting the value of bundleLimit which by default is 25 so this makes the transaction pool to move transactions between jobs slower (if we set this value higher we can move more transactions per job for example maybe this value should be the same as MAX_TRANSACTIONS_PER_BLOCK)

• We should investigate what values will be best for _receivedTransactionsProcessingInterval, _validatedTransactionsProcessingInterval and _verifiedTransactionsProcessingInterval as they command the interval for the transaction pool timers and there’s some interaction between this timers and BLOCK_TIME (I think that tuning this values could potentially improve the current values noted above )

• By increasing the value of bundleLimit to 1000 I was able to get reliable results with up to 400 transactions per block (with 10 seconds blocks) although hitting PSQL pretty hard.

• During the tests the memory usage was in the 300 MB limit and cpu usage around 40% when using 10 seconds block time and 150 transactions per block.

• PSQL usage seems to be a place to look for possible future optimizations.

• As a side note a 400 transfer transactions block is 200KB when stringified (this value might be of interest for P2P)

How the tests were performed?

• BLOCK_TIME was set to 1 and MAX_TRANSACTIONS_PER_BLOCK to 25 and then the values were incremented checking that the blocks generated were correct (all possible tx filled, correct block time) for each change of one of the values 5000 transactions were sent to the node.
• After this initial values were established I tested the configuration of BLOCK_TIME=10 and MAX_TRANSACTIONS_PER_BLOCK=150 using the QA tools for deploying a 50 nodes network. All tests passed successfully.
• The script used for sending the transactions can be found in https://gist.github.com/pablitovicente/99f663e5e59980cd1abde1996a5c0c8a