This repository contains the evaluation tool developed as part of the paper "Analyzing the Performance of the Inter-Blockchain Communication Protocol". This paper has been published at the 53rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'23).

A full version is available online: https://arxiv.org/abs/2303.10844

The dataset generated through the experiments discussed in the paper are available online: https://drive.google.com/drive/folders/1f3t4Qf_mC2atcgpafTHk4qv9vWesjfwQ?usp=sharing

Prerequisites:

For the blockchains:

• Go v1.17.5
• Cosmos Hub (Gaia) v7.0.2

For the IBC relayer:

• Rust compiler v1.6.0
• Hermes Relayer v0.15.0 (https://github.com/informalsystems/hermes)

Tip

Those prerequisites can be installed by running the install.sh script.

Scripts:

setup_chains.sh

First, this script sets up two Cosmos Gaia blockchains with funded user accounts. Then, it uses ssh to run a set of validator nodes for each of them on remote machines. After starting the blockchains and synchronizing the nodes it establishes an unordered IBC channel between both blockchains using the Hermes Relayer.

Important

This file needs to be modified to include the address of the remote machines used for validator nodes. The prerequisites for the blockchains must be installed in all remote machines.

Usage: ./setup_chains.sh -n <NUMBER_OF_NODES> -a <FUNDED_ACCOUNTS> -t [TIMEOUT_COMMIT]

Options:

  --nodes          | -n;      Number of consensus nodes to run for each chain.  
  --accounts       | -a;      Number of funded accounts initialized in the source chain's genesis.  
  --timeout-commit | -t;      Minimum block interval in seconds (default: 5 seconds).

Example: ./setup_chains.sh -n 5 -a 10 -t 5

benchmark.sh:

This script conducts a performance evaluation of cross-chain communication using the previously established IBC channel. The evaluation workload is composed of cross-chain fungible token transfers (https://github.com/cosmos/ibc/blob/main/spec/app/ics-020-fungible-token-transfer/README.md). This script must be executed in a machine together with one validator node for each blockchain. This is required to retrieve transaction data through RPC.

Usage: ./benchmark.sh -S <SRC_CHAIN_ADDR> -D <DST_CHAIN_ADDR> -u <NUM_USERS> -t <NUM_TRANSACTIONS> -m <NUM_MESSAGES> -o <OUTPUT_DIR>

Options:

  -S | --source-addr;         RPC address of the source chain.  
  -D | --destination-addr;    RPC address of the destination chain.  
  -u | --users;               Number of users to be used for submitting transactions.  
  -t | --transactions;        Number of transactions submitted per user.  
  -m | --messages;            Number of cross-chain transfer messages inside each transaction (max: 100).  
  -o | --output-dir;          Directory in which to store benchmark working files.  
  -w | --wait-for-blocks;     [Optional] Stop waiting for transactions to complete/timeout and start analyzing data after this many empty blocks have been produced in a row (default: 5).  
  --tx-timeout;               [Optional] Specify how many new blocks can be created before a cross-chain transfer times out (default: 25).  
  --transaction-analysis;     [Optional] Enables analysis of transaction and IBC message sizes (slower).  

Note

The option --transaction-analysis is currently unavailable as it requires additional logic that increases the completion time of default benchmarking mode given the current code structure.

Example: ./benchmark.sh -S 'localhost:26657' -D 'localhost:36657' -u 10 -t 25 -m 20 -o 'benchmarking_test'

Benchmark output:

The tool generates a file called "benchmarking_report.txt" in the specified output directory. This file contains a performance report generated based on the execution of the specified workload.

Sample output:

[+] Benchmark configuration summary:

 Source chain: blockchain0 
 Destination chain: blockchain1
 Number of validators in each chain: 5
 Number of user accounts: 10
 Transactions submitted per user: 50
 Total number of transactions: 500
 Transfer messages per transaction: 100
 Total number of transfer messages: 50000

 IBC transfers submission (time elapsed): 5m 18s
 Time waiting for empty blocks at the end of benchmark: 0m 1s
 Blockchain data collection (time elapsed): 7m 46s
 Total time elapsed: 13m 5s

----------------------------------------------------------------------
[+] Transaction distribution analysis for blockchain0:

 0 tx(s): 1 block(s)
 10 tx(s): 29 block(s)
 19 tx(s): 1 block(s)
 21 tx(s): 17 block(s)
 23 tx(s): 1 block(s)
 25 tx(s): 1 block(s)
 26 tx(s): 1 block(s)

----------------------------------------------------------------------
[+] Transaction distribution analysis for blockchain1:

 0 tx(s): 26 block(s)
 2 tx(s): 1 block(s)
 9 tx(s): 1 block(s)
 11 tx(s): 20 block(s)
 16 tx(s): 1 block(s)
 17 tx(s): 1 block(s)

----------------------------------------------------------------------
[+] Throughput analysis for chain 'blockchain0':

 Blocks finalized: 51

 Avg. block time: 6.302 seconds
 Number of empty blocks: 1 (1.96%)

 Avg. number of txs per block (counting empty blocks): 14.51
 Avg. number of txs per second(transfer, recv, ack): 2.35

 Avg. number of messages per tx: 97.30
 Avg. number of messages per block (counting empty blocks): 1411.76
 Avg. number of messages per second (transfer, recv, ack, timeout): 228.50
 Avg. number of transfers per second: 158.68


----------------------------------------------------------------------
[+] Throughput analysis for chain 'blockchain1':

 Blocks finalized: 50

 Avg. block time: 6.322 seconds
 Number of empty blocks: 26 (52.00%)

 Avg. number of txs per block (counting empty blocks): 5.28
 Avg. number of txs per second(transfer, recv, ack): 0.85

 Avg. number of messages per tx: 90.91
 Avg. number of messages per block (counting empty blocks): 480.00
 Avg. number of messages per second (transfer, recv, ack, timeout): 77.47
 Avg. number of transfers per second: 0.00


----------------------------------------------------------------------
[+] Round trip time analysis for chains 'blockchain0 -> blockchain1':

 Average round trip time: 178.920s
 Shortest round trip time: 163.744s
 Longest round trip time: 188.840s

----------------------------------------------------------------------
[+] Success rate analysis for channel 'blockchain0 -> blockchain1':

 IBC transfers submitted to 'blockchain0': 50000
 'Transfer' messages committed to 'blockchain0': 50000
 'Receive' messages committed to 'blockchain1': 24000
 'Acknowledgement' messages committed to 'blockchain0': 22000
 'Timeout' messages committed' to 'blockchain0' (source chain): 0

 Transfers completed (transfer, recv, ack): 22000 (44.00%)
 Transfers partially completed (transfer, recv): 2000 (4.00%)
 Transfers only initiated (transfer): 26000 (52.00%)
 Transfers not initiated (submitted but not committed): 0 (0.00%)
 Timed out transfers: 0 (0.00%)

----------------------------------------------------------------------
[+] IBC messages confirmation latency analysis for chains 'blockchain0 -> blockchain1':

 Avg. transfer message confirmation latency: 6.311s
 Shortest transfer latency observed: 5.131s
 Longest transfer latency observed: 7.855s

 Avg. recv message confirmation latency: 12.190s
 Shortest recv message confirmation latency: 6.386s
 Longest recv message confirmation latency: 90.453s

 Avg. acknowledgement message confirmation latency: 8.431s
 Shortest acknowledgement message confirmation latency: 6.208s
 Longest acknowledgement message confirmation latency: 10.349s

----------------------------------------------------------------------
[+] Data analysis for chain 'blockchain0':

 Collective size of all blocks (excl. empty blocks at the end): 37.24 MB
 Avg. block size (excl. empty blocks at the end): 747.81 kB
 Number of transactions committed to the blockchain: 740
 Number of messages committed in the blockchain: 72000

 Number of transactions containing transfer messages on blockchain0: 500
 Number of transfer messages on blockchain0: 50000

 Number of transactions containing recv messages on blockchain0: 0
 Number of recv messages on blockchain0: 0

 Number of transactions containing ack messages on blockchain0: 240
 Number of ack messages on blockchain0: 22000

 Number of transactions containing timeout messages on blockchain0: 0
 Number of timeout messages on blockchain0: 0

----------------------------------------------------------------------
[+] Data analysis for chain 'blockchain1':

 Collective size of all blocks (excl. empty blocks at the end): 30.07 MB
 Avg. block size (excl. empty blocks at the end): 615.79 kB
 Number of transactions committed to the blockchain: 264
 Number of messages committed in the blockchain: 24000

 Number of transactions containing transfer messages on blockchain1: 0
 Number of transfer messages on blockchain1: 0

 Number of transactions containing recv messages on blockchain1: 264
 Number of recv messages on blockchain1: 24000

 Number of transactions containing ack messages on blockchain1: 0
 Number of ack messages on blockchain1: 0

 Number of transactions containing timeout messages on blockchain1: 0
 Number of timeout messages on blockchain1: 0

----------------------------------------------------------------------