NEWS 2019-Feb-23 - video! Explaining software v55 video released on youtube:
TPS measurements of parity aura, geth clique, quorum, tobalaba, etc. It should work with any Ethereum type chain; we focused on PoA consensus.
The brand new release v55 is now presented & explained in a useful video on youtube.
hammer/- submits many transactions, while watching the recent blocks
reader/- reads blocks; visualizes TPS, blocktime, gas, bytes - see reader/README.md
docs/- see esp. reproduce.md, cloud.md, FAQ.md, new: azure.md
results/- for each client one markdown file;
results/runs/- auto-generated pages
logs/- check this first if problems
networks/- network starters & external repos via install script, see below
scripts/- installers and other iseful bash scripts
env/- Python virtualenv, created via install script, see below
tests/- start whole integration test suite via
See the results/ folder:
- log.md: initial steps; also tried Quorum's private transactions
- quorum.md: raft consensus, quorum is a geth fork
- tobalaba.md: parity fork of EnergyWebFoundation
- quorum-IBFT.md: IstanbulBFT, 2nd consensus algo in quorum
- geth.md: geth clique PoA algorithm
- parity.md: parity aura PoA algorithm, many attempts to accelerate
- eos.md: not begun yet
- substrate.md: not begun yet
Outdated table in which I had run each of the experiments manually in autumn 2018; soon re-done completely, using the below automation. So please contact me now, if you know how to accelerate any of these clients:
|hardware||node type||#nodes||config||peak TPS_av||final TPS_av|
|t2.2xlarge||geth clique||3+1 +2||(B)||421.6||400.0|
|t2.xlarge||geth clique||3+1 +2||(B)||386.1||321.5|
|t2.large||geth clique||3+1 +2||(B)||170.7||169.4|
|t2.small||geth clique||3+1 +2||(B)||96.8||96.5|
|t2.micro SWAP||quorum crux IBFT||4||(I) SWAP!||98.1||98.1|
|t2.micro||quorum crux IBFT||4||(F)||lack of RAM|
|t2.large||quorum crux IBFT||4||(F)||207.7||199.9|
|t2.xlarge||quorum crux IBFT||4||(F)||439.5||395.7|
|t2.xlarge||quorum crux IBFT||4||(L)||389.1||338.9|
|t2.2xlarge||quorum crux IBFT||4||(F)||435.4||423.1|
|c5.4xlarge||quorum crux IBFT||4||(F) test_getNearestEntry()||536.4||524.3|
Reproduce these results easily; for the
config column also see there.
Quickest reproduction with my Amazon AMI readymade image.
And see that bottom of parity.md and geth.md
and quorum-IBFT.md for the latest runs, issues, and additional details.
faster wider more
- how I initially got this faster, on Quorum, step by step, please do read the 1st logbook log.md
- then I improved per client, see each in #chronology above
- (possible TODOs - any other ideas?)
but not much more needed = the current version is already fully automated. Use it! May it help you to improve the speed of your Ethereum client!
Add yourself to other-projects.md using chainhammer, or projects which are similar to this.
(Especially if you work in one of the dev teams, you know your client code best - ) please try to improve the above results, e.g. by varying the CLI arguments with which the nodes are started; I don't see that as my job, you will be much more successful with that.
Please report back when you have done other / new measurements.
install and run
All this is developed and much tested on Debian, locally and in the AWS cloud. New: Ubuntu now also supported, see below.
N.B.: Better do this on a disposable cloud, or virtualbox machine; because the installation makes lasting changes and needs sudo!
After unpacking a ZIP of the downloaded repo, or by
git clone https://github.com/drandreaskrueger/chainhammer drandreaskrueger_chainhammer ln -s drandreaskrueger_chainhammer CH cd CH
you now only need these two lines to prepare and run the 1st experiment!
scripts/install.sh CH_TXS=1000 CH_THREADING="sequential" ./run.sh $HOSTNAME-TestRPC testrpc
You will then have a diagram, and a HTML and MD page about this run!
(on Ubuntu instead:
scripts/install.sh docker ubuntu )
Better now logout & login, or close the terminal, and open a new terminal, because the above scripts/install.sh might have enabled docker for the the first time for this user. Then:
All supported clients in one go:
For the full integration test, run each client for a short moment:
export CH_MACHINE=yourChoice ./run-all_small.sh
benchmarking a remote node
Chainhammer can now be stripped down to its pure benchmarking abilities, i.e. without the installation of docker and without the three local network starters (parity-deploy, geth-dev, quorum-crux). It was successfully used to benchmark the Microsoft Azure blockchain-as-a-service product. The essential difference is to start the installation with the switch
So, if you just want to benchmark your existing Ethereum node or network, have a look at the manual docs/azure.md .
enables the virtualenv,
then starts a
testrpc-py Ethereum simulator on http://localhost:8545 in the background,
and finally runs all the unittests, also logging into
(Instead of testrpc-py) if you want to run tests with another node,
just start that; and run
source env/bin/activate py.test -v --cov
There were 98 tests on January 23rd, all 98 PASSED
(see this logfile -->
cat tests/logs/*.ansi because colors) on these different Ethereum providers:
- testrpc instantseal (
testrpc-py) 13 seconds
- geth Clique (
geth-dev) 63 seconds
- quorum IBFT (
blk-io/crux) 59 seconds
- parity instantseal (
parity-deploy) 8 seconds
- parity aura (
parity-deploy) 72 seconds
Please credit this as:
benchmarking scripts "chainhammer"
maintainer: Dr Andreas Krueger 2018-2020
Consider to submit your improvements & usage as pull request. Thanks.
development was supported by
v01-v35 financed by Electron.org.uk 2018
v40-v55 financed by Web3Foundation 2018-2019
v58-v59 financed by Microsoft Azure 2019
Thank you very much!
The open source tools 'chainhammer' submits a high load of smart contract transactions to an Ethereum based blockchain, then 'chainreader' reads the whole chain, and produces diagrams of TPS, blocktime, gasUsed and gasLimit, and the blocksize. https://github.com/drandreaskrueger/chainhammer
# The following diagrams are outdated! Just make your own, new ones, with: CH_MACHINE=yourChoice ./run-all_large.sh
chainhammer: hammer --> reader --> diagrams
geth clique on AWS t2.xlarge
geth.md = geth (go ethereum client), "Clique" consensus.
50,000 transactions to an Amazon t2.xlarge machine.
Interesting artifact that after ~14k transactions, the speed drops considerably - but recovers again. Reported.
quorum IBFT on AWS t2.xlarge
quorum-IBFT.md = Quorum (geth fork), IBFT consensus, 20 millions gasLimit, 1 second istanbul.blockperiod; 20000 transactions multi-threaded with 23 workers. Initial average >400 TPS then drops to below 300 TPS, see quorum issue)
OLD RUN on a desktop machine.
quorum.md = Quorum (geth fork), raft consensus, 1000 transactions multi-threaded with 23 workers, average TPS around 160 TPS, and 20 raft blocks per second)
OLD RUN on a desktop machine.
tobalaba.md = Public "Tobalaba" chain of the EnergyWebFoundation (parity fork), PoA; 20k transactions; > 150 TPS if client is well-connected.
parity aura v1.11.11 on AWS t2.xlarge
Calling all parity experts: How to improve these too slow TPS results?