FireFly Performance Testing Guide

Setup

Hardware

Minimum recommended instance specs:

• 4 vCPU
• 8 GiB memory

Environment configuration

Make sure the following are installed:

• latest FireFly CLI
• Docker
• Git

In the same directory, clone the FireFly Performance CLI and FireFly Core.

Running tests

The FireFly Performance CLI is the program we use to benchmark FireFly. It can be configured, through a file named instances.yml, to test various parts of the FireFly system at high throughput over an extended period of time. For more information about configuring the CLI, please see the project's GitHub.

Here is the default instances.yml typically used for testing FireFly:

stackJSONPath: <path to FF CLI stack json>

wsConfig:
  wsPath: /ws
  readBufferSize: 16000
  writeBufferSize: 16000
  initialDelay: 250ms
  maximumDelay: 30s
  initialConnectAttempts: 5
  heartbeatInterval: 5s

instances:
  - name: long-run
    tests: [{"name": "msg_broadcast", "workers":50},{"name": "msg_private", "workers":50},{"name": "blob_broadcast", "workers":30},{"name": "blob_private", "workers":30},{"name": "custom_ethereum_contract", "workers":20},{"name": "token_mint", "workers":10}]
    length: 500h
    sender: 0
    recipient: 1
    messageOptions:
      longMessage: false
    tokenOptions:
      tokenType: fungible
    contractOptions: {"address": "<custom contract address>"}

Next, use the included prep.sh script to configure and start your firefly stack for testing.

$ ./prep.sh -h
Must provide exactly three arguments:
1. Old stack name to remove
2. New stack name to create
3. Stack's blockchain type (ex. geth, besu, fabric, corda)
ex: ./prep.sh old_stack new_stack geth

An example command would be:
$ ./prep.sh oldstack newstack geth

This script will cleanup your old FireFly stack, start your new FireFly stack, print out a description of your test, and print a command to the start test.

nohup ./start.sh &> ffperf.log &
core-config.yml
log:
        level: debug
broadcast:
        batch:
                size: 200
                timeout: 1s
privatemessaging:
        batch:
                size: 200
                timeout: 1s
message:
        writer:
                count: 5
download:
        worker:
                count: 100
publicstorage:
                ipfs:
                        api:
                                requestTimeout: 2s
                        gateway:
                                requestTimeout: 2s

ethconnect.yml
rest:
  rest-gateway:
    maxTXWaitTime: 120
    maxInFlight: 200
    alwaysManageNonce: true
    attemptGapFill: true
    sendConcurrency: 3
    gasEstimationFactor: 2.0
confirmations:
  required: 5
debug:
  port: 6000
instances.yml
stackJSONPath: <stack json path>

wsConfig:
  wsPath: /ws
  readBufferSize: 16000
  writeBufferSize: 16000
  initialDelay: 250ms
  maximumDelay: 30s
  initialConnectAttempts: 5
  heartbeatInterval: 5s

instances:
  - name: long-run
    tests: [{"name": "msg_broadcast", "workers":50},{"name": "msg_private", "workers":50},{"name": "blob_broadcast", "workers":30},{"name": "blob_private", "workers":30},{"name": "custom_ethereum_contract", "workers":20},{"name": "token_mint", "workers":10}]
    length: 500h
    sender: 0
    recipient: 1
    messageOptions:
      longMessage: false
    tokenOptions:
      tokenType: fungible
    contractOptions: {"address": ""}
FireFly git commit:
057a7af8646426bc9d79a95b8d066838e3e6bc43

Execute the start command nohup ./start.sh &> ffperf.log & to begin running tests.

Metrics

FireFly exposes prometheus metrics under the /metrics endpoint. For more information about which metrics are currently supported, please refer to the metrics package.