Skip to content

Commit

Permalink
#1
Browse files Browse the repository at this point in the history
  • Loading branch information
TrustedBlockchain committed Jul 6, 2018
1 parent 3a4d85a commit 1d76697
Show file tree
Hide file tree
Showing 232 changed files with 8,339 additions and 674 deletions.
5 changes: 0 additions & 5 deletions .eslintignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,3 @@ node_modules

# Plugins
/src/gui
/src/iroha

# Contracts for plugins
/src/contract/iroha

11 changes: 11 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ architecture.pptx
*.bak
zookeeper.out
node_modules/*
package-lock.json
output.log

# Ignore any generted html files
*.html
Expand All @@ -16,3 +18,12 @@ composer-logs/
# Ignore code coverage logs
.nyc_output/
coverage/

# We prefer to not use lock files
package-lock.json

# Ignore zookeeper files
zookeeper.out

# Ignore IDE files
.vscode/*
30 changes: 15 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
## TrustedBench Introduction

TrustedBench is a blockchain performance benchmark framework, which allows users to test different blockchain solutions with predefined use cases, and get a set of performance test results.
TrustedBench is a secondary developed product based on hyperledger caliper.
TrustedBench is a secondary developed product based on hyperledger Caliper.

Currently supported blockchain solutions:
* [fabric v1.0+](https://github.com/hyperledger/fabric), the lastest version that has been verified is v1.1.0
Expand Down Expand Up @@ -31,7 +31,7 @@ Make sure following tools are installed
* Docker
* Docker-compose

Run `npm install` in caliper folder to install dependencies locally
Run `npm install` in TrustedBench folder to install dependencies locally

### Install blockchain SDKs
* Fabric
Expand Down Expand Up @@ -106,7 +106,7 @@ You can also use npm scripts to run a benchmark.
```bash
$ npm run list

> caliper@0.1.0 list /home/hurf/caliper
> TrustedBench@0.1.0 list /home/XXX/TrustedBench
> node ./scripts/list.js

Available benchmarks:
Expand All @@ -118,7 +118,7 @@ simple
```bash
$ npm test -- simple -c ./benchmark/simple/config.json -n ./benchmark/simple/fabric.json

> caliper@0.1.0 test /home/hurf/caliper
> TrustedBench@0.1.0 test /home/XXX/TrustedBench
> node ./scripts/test.js "simple" "-c" "./benchmark/simple/config.json" "-n" "./benchmark/simple/fabric.json"
......
```
Expand All @@ -133,14 +133,14 @@ In this way, multiple clients can be launched on distributed hosts to run the sa
```bash
$ npm run startclient -- 10.229.42.159:2181

> caliper@0.1.0 startclient /home/hurf/caliper
> TrustedBench@0.1.0 startclient /home/XXX/TrustedBench
> node ./src/comm/client/zoo-client.js "10.229.42.159:2181"

Connected to ZooKeeper
Created client node:/caliper/clients/client_1514532063571_0000000006
Created receiving queue at:/caliper/client_1514532063571_0000000006_in
Created sending queue at:/caliper/client_1514532063571_0000000006_out
Waiting for messages at:/caliper/client_1514532063571_0000000006_in......
Created client node:/TrustedBench/clients/client_1514532063571_0000000006
Created receiving queue at:/TrustedBench/client_1514532063571_0000000006_in
Created sending queue at:/TrustedBench/client_1514532063571_0000000006_out
Waiting for messages at:/TrustedBench/client_1514532063571_0000000006_in......
```
3. Modify the client type setting in configuration file to 'zookeeper'.

Expand All @@ -158,19 +158,19 @@ In this way, multiple clients can be launched on distributed hosts to run the sa
4. Launch the benchmark on any machine as usual.

Note:
* Zookeeper is used to register clients and exchange messages. A launched client will add a new znode under /caliper/clients/. The benchmark checks the directory to learn how many clients are there, and assign tasks to each client according to the workload.
* Zookeeper is used to register clients and exchange messages. A launched client will add a new znode under /TrustedBench/clients/. The benchmark checks the directory to learn how many clients are there, and assign tasks to each client according to the workload.
* There is no automatic time synchronization between the clients. You should manually synchronize time between target machines, for example using 'ntpdate'.
* The blockchain configuration file must exist on machines which run the client, and the relative path (relative to the caliper folder) of the file must be identical. All referenced files in the configuration must also exist.
* The blockchain configuration file must exist on machines which run the client, and the relative path (relative to the TrustedBench folder) of the file must be identical. All referenced files in the configuration must also exist.




## Write your own benchmarks
Caliper provides a set of nodejs NBIs (North Bound Interfaces) for applications to interact with backend blockchain system. Check the [*src/comm/blockchain.js*](./src/comm/blockchain.js) to learn about the NBIs. Multiple *Adaptors* are implemented to translate the NBIs to different blockchain protocols. So developers can write a benchmark once, and run it with different blockchain systems.
TrustedBench provides a set of nodejs NBIs (North Bound Interfaces) for applications to interact with backend blockchain system. Check the [*src/comm/blockchain.js*](./src/comm/blockchain.js) to learn about the NBIs. Multiple *Adaptors* are implemented to translate the NBIs to different blockchain protocols. So developers can write a benchmark once, and run it with different blockchain systems.

Generally speaking, to write a new caliper benchmark, you need to:
Generally speaking, to write a new TrustedBench benchmark, you need to:
* Write smart contracts for systems you want to test
* Write a testing flow using caliper NBIs. Caliper provides a default benchmark engine, which is pluggable and configurable to integrate new tests easily. For more details, please refer to [Benchmark Engine](./docs/Architecture.md#benchmark-engine) .
* Write a testing flow using TrustedBench NBIs. TrustedBench provides a default benchmark engine, which is pluggable and configurable to integrate new tests easily. For more details, please refer to [Benchmark Engine](./docs/Architecture.md#benchmark-engine) .
* Write a configuration file to define the backend network and benchmark arguments.

## Directory Structure
Expand All @@ -187,4 +187,4 @@ Generally speaking, to write a new caliper benchmark, you need to:
See [Contributing](/CONTRIBUTING.md)

## License
The Caliper codebase is release under the [Apache 2.0 license](./LICENSE). Any documentation developed by the Caliper Project is licensed under the Creative Commons Attribution 4.0 International License. You may obtain a copy of the license, titled CC-BY-4.0, at http://creativecommons.org/licenses/by/4.0/.
The TrustedBench codebase is release under the [Apache 2.0 license](./LICENSE). Any documentation developed by the TrustedBench Project is licensed under the Creative Commons Attribution 4.0 International License. You may obtain a copy of the license, titled CC-BY-4.0, at http://creativecommons.org/licenses/by/4.0/.
6 changes: 3 additions & 3 deletions benchmark/composer/config-composer.json
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
{
"blockchain": {
"type": "composer",
"config": "network/fabric-v11/2-org-1-peer/composer.json"
"config": "network/fabric/2-org-1-peer/composer.json"
},
"command" : {
"start": "docker-compose -f network/fabric-v11/2-org-1-peer/docker-compose.yaml up -d",
"end": "docker-compose -f network/fabric-v11/2-org-1-peer/docker-compose.yaml down;docker rm $(docker ps -aq);docker rmi $(docker images dev-* -q)"
"start": "docker-compose -f network/fabric/2-org-1-peer/docker-compose.yaml up -d",
"end": "docker-compose -f network/fabric/2-org-1-peer/docker-compose.yaml down;docker rm $(docker ps -aq);docker rmi $(docker images dev-* -q)"
},
"test": {
"name": "Composer Performance test",
Expand Down
8 changes: 4 additions & 4 deletions benchmark/simple/config-iroha.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
},
"command" : {
"start": "docker-compose -f network/iroha/simplenetwork/docker-compose.yml up -d",
"end" : "docker-compose -f network/iroha/simplenetwork/docker-compose.yml down;docker rm $(docker ps -aq)"
"end" : "docker logs iroha_node0; docker logs iroha_node1; docker-compose -f network/iroha/simplenetwork/docker-compose.yml down"
},
"test": {
"clients": {
Expand All @@ -14,14 +14,14 @@
},
"rounds": [{
"label" : "open",
"txNumber" : [500],
"txNumber" : [50],
"rateControl" : [{"type": "fixed-rate", "opts": {"tps" : 50}}],
"arguments": { "money": 10000 },
"callback" : "benchmark/simple/open.js"
},
{
"label" : "query",
"txNumber" : [500],
"txNumber" : [50],
"rateControl" : [{"type": "fixed-rate", "opts": {"tps" : 50}}],
"callback" : "benchmark/simple/query.js"
}]
Expand All @@ -40,4 +40,4 @@
],
"interval": 1
}
}
}
9 changes: 9 additions & 0 deletions benchmark/smallbank/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
The smallbank benchmark contains a series of transactions to simulate bank operations, including one transaction which creates accounts and others which perform banking transactions such as querying balances, depositing in accounts, paying money, etc. The benchmark intends to provide cross-platform workload for blockchain performance testing in the bank scene.

At the beginning of the test, a specified number of bank accounts will be generated by submitting create_account transactions, the amount is specified by `accounts` attribute of the configuration. Then random transactions will be generated to simulate operating on those accounts.

Please note `batch` concept is used in this benchmark, which means multiple transactions may be wrapped inside of a batch and submitted together if the blockchan system supports `batch` feature. For systems which do not support this feature, transactions will be submitted one by one. The transaction amount in a batch is specifed by the `txnPerBatch` attribute of the configuration. The feature can be disabled by setting `txnPerBatch` to 1.

The transaction generating process may generate conflicting transactions, for example, updating same account's deposite at the same time or nearly at the same time. Some blockchain systems provides some kind of orchestration to decide execution order of conficting transactions, but some blockchain systems may just return errors for later updates if the ahead update has not been completed and let the application decide what to do (e.g re-submitting rejected transactions). Until now Caliper has not provided any application level optimization such as re-submitting for failed transactions for any blockchain system. So please note in this case, conflicting transactions may cause failures for some blockchain systems.

This documentation is licensed under the Creative Commons Attribution 4.0 International License. You may obtain a copy of the license, titled CC-BY-4.0, at http://creativecommons.org/licenses/by/4.0/.
39 changes: 39 additions & 0 deletions benchmark/smallbank/config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"blockchain": {
"type": "fabric",
"config": "benchmark/smallbank/fabric.json"
},
"command" : {
"start": "docker-compose -f network/fabric/simplenetwork/docker-compose.yaml up -d",
"end" : "docker-compose -f network/fabric/simplenetwork/docker-compose.yaml down;docker rm $(docker ps -aq)"
},
"test": {
"name": "smallbank",
"description" : "This is smallbank benchmark for caliper",
"clients": {
"type": "local",
"number": 5
},
"rounds": [
{
"label" : "smallOperations",
"txNumber" : [4000],
"rateControl" : [{"type": "fixed-rate", "opts": {"tps" : 200}}],
"arguments": {"accounts":400, "txnPerBatch":10},
"callback" : "benchmark/smallbank/smallbankOperations.js"
},
{
"label" : "query",
"txNumber" : [2000],
"rateControl" : [{"type": "fixed-rate", "opts": {"tps" : 200}}],
"callback" : "benchmark/smallbank/query.js"
}]
},
"monitor": {
"type": "docker",
"docker":{
"name": ["all"]
},
"interval": 1
}
}
103 changes: 103 additions & 0 deletions benchmark/smallbank/fabric.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
{
"fabric": {
"cryptodir": "network/fabric/simplenetwork/crypto-config",
"network": {
"orderer": {
"url": "grpcs://localhost:7050",
"mspid": "OrdererMSP",
"user": {
"key": "network/fabric/simplenetwork/crypto-config/ordererOrganizations/example.com/users/Admin@example.com/msp/keystore/be595291403ff68280a724d7e868521815ad9e2fc8c5486f6d7ce6b62d6357cd_sk",
"cert": "network/fabric/simplenetwork/crypto-config/ordererOrganizations/example.com/users/Admin@example.com/msp/signcerts/Admin@example.com-cert.pem"
},
"server-hostname": "orderer.example.com",
"tls_cacerts": "network/fabric/simplenetwork/crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/tls/ca.crt"
},
"org1": {
"name": "peerOrg1",
"mspid": "Org1MSP",
"user": {
"key": "network/fabric/simplenetwork/crypto-config/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp/keystore/0d2b2fc385b10fa59003217e1bb5af2d24a3d762266e287867a1bc290eb44657_sk",
"cert": "network/fabric/simplenetwork/crypto-config/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp/signcerts/Admin@org1.example.com-cert.pem"
},
"ca": {
"url": "https://localhost:7054",
"name": "ca-org1"
},
"peer1": {
"requests": "grpcs://localhost:7051",
"events": "grpcs://localhost:7053",
"server-hostname": "peer0.org1.example.com",
"tls_cacerts": "network/fabric/simplenetwork/crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt"
},
"peer2": {
"requests": "grpcs://localhost:7057",
"events": "grpcs://localhost:7059",
"server-hostname": "peer1.org1.example.com",
"tls_cacerts": "network/fabric/simplenetwork/crypto-config/peerOrganizations/org1.example.com/peers/peer1.org1.example.com/tls/ca.crt"
}
},
"org2": {
"name": "peerOrg2",
"mspid": "Org2MSP",
"ca": {
"url": "https://localhost:8054",
"name": "ca-org2"
},
"peer1": {
"requests": "grpcs://localhost:8051",
"events": "grpcs://localhost:8053",
"server-hostname": "peer0.org2.example.com",
"tls_cacerts": "network/fabric/simplenetwork/crypto-config/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt"
},
"peer2": {
"requests": "grpcs://localhost:8057",
"events": "grpcs://localhost:8059",
"server-hostname": "peer1.org2.example.com",
"tls_cacerts": "network/fabric/simplenetwork/crypto-config/peerOrganizations/org2.example.com/peers/peer1.org2.example.com/tls/ca.crt"
}
}
},
"channel": [
{
"name": "mychannel",
"config": "network/fabric/simplenetwork/mychannel.tx",
"organizations": ["org1", "org2"],
"deployed": false
}
],
"chaincodes": [{"id": "smallbank", "path": "contract/fabric/smallbank", "language":"golang", "version": "v0", "channel": "mychannel"}],
"endorsement-policy": {
"identities": [
{
"role": {
"name": "member",
"mspId": "Org1MSP"
}
},
{
"role": {
"name": "member",
"mspId": "Org2MSP"
}
},
{
"role": {
"name": "admin",
"mspId": "Org1MSP"
}
}
],
"policy": { "2-of": [{"signed-by": 0}, {"signed-by": 1}]}
},
"context": {
"smallOperations": "mychannel",
"query": "mychannel"
}
},
"info" : {
"Version": "1.1.0",
"Size": "4 Peers",
"Orderer": "Solo",
"Distribution": "Single Host"
}
}
Loading

0 comments on commit 1d76697

Please sign in to comment.