ByzzBench is a benchmark suite designed to evaluate the performance of testing algorithms in detecting bugs in Byzantine Fault Tolerance (BFT) protocols. It is designed to be modular and extensible, allowing for easy integration of new protocols and scenarios.
ByzzBench is designed for a standardized implementation of BFT protocols and their execution in a controlled testing environment, controlling the nondeterminism in concurrency, network and process faults in the protocol execution, enabling the functionality to enforce particular execution scenarios and thereby facilitating the implementation of testing algorithm for BFT protocols.
This is a Gradle monorepo that contains the following modules:
simulator: The core benchmarking suite, written in Java/Spring Boot. It currently also includes the protocol implementations.webui: A web interface for the benchmarking suite, written in React/NextJS.
For the benchmarking suite to work, you need to have the following installed on your system:
- Java 21
For the user interface to work, you need to have the following installed on your system:
- Node.js
- pnpm
Installing everything on macOS using HomeBrew can be done with the following commands:
brew install openjdk@17 node pnpm
Installing JDK:
-
Through Eclipse Adoptium:
- Download the version you need (JDK-21)
- When installing, select "Set or override JAVA_HOME variable"
-
Through Windows Package Manager - "winget":
// Eclipse Temurin from Eclipse Adoptium
winget install EclipseAdoptium.Temurin.21.JDK
// from Microsoft Build
winget install Microsoft.OpenJDK.21
Note
You might need to set a PATH and JAVA_HOME!
To verify the installation execute the following in cmd (should match the installed version):
java -version
Note
If it displays a different version, then the PATH or JAVA_HOME could be set incorrectly (pointing to another version)!
Installing Node.js:
- Through installer: https://nodejs.org/en/download/prebuilt-installer
- Through package managers: https://nodejs.org/en/download/package-manager
To install pnpm, you can use npm package manager (installed alongside Node.js):
npm install -g pnpm
or Corepack:
corepack enable pnpm
or PowerShell:
Invoke-WebRequest https://get.pnpm.io/install.ps1 -UseBasicParsing | Invoke-Expression
Warning
Windows Defender may block this option!
Using npm or Corepack is the recommended way!
Reference: https://pnpm.io/installation
For other operating systems, please refer to the respective installation instructions.
To configure the simulator, you can modify the
application.properties file in the simulator module.
It has two main subsections:
scheduler: configuration about the testing strategy to be used and its parametersscenario: Which protocol to be ran, and termination conditions for each run
See below an excerpt of the most relevant parameters:
byzzbench:
autostart: true # start generating scenarios on startup
numScenarios: 1000 # how many scenarios to be generated
scheduler:
id: "byzzfuzz" # which testing algorithm to use
executionMode: sync # sync / async
maxDropMessages: 0 # Maximum number of messages to drop per scenario
maxMutateMessages: 0 # Maximum number of messages to mutate per scenario
deliverTimeoutWeight: 1 # The weight for scheduler to trigger a timeout
deliverMessageWeight: 99 # The weight for scheduler to deliver a message
deliverClientRequestWeight: 99 # The weight for scheduler to deliver a client request to a replica
dropMessageWeight: 0 # The weight for scheduler to drop a message
mutateMessageWeight: 0 # The weight for scheduler to mutate a message
params: # parameters for ByzzFuzz/Twins
# ByzzFuzz
numRoundsWithProcessFaults: 2
numRoundsWithNetworkFaults: 1
numRoundsWithFaults: 10
# Twins
numReplicas: 1
numTwinsPerReplica: 2
numRounds: 5
scenario:
id: pbft-java # which protocol to run
termination: # Success condition for the scenario (conjunction)
minEvents: 500 # schedule must have length > 500To build and run ByzzBench, run the following command from the root directory:
./gradlew bootRun
If autostart is set to true, it will generate the specified amount of scenarios, and output statistics to the terminal
after completion (%correct, %buggy, %errors).
The web UI is a simple React application (using NextJS/TypeScript) that allows you to interact with the simulator. It is a work in progress, but provides useful insights into the behavior of the protocols.
To build the web interface, run the following command while the simulator is running:
(cd webui && pnpm install && pnpm run kubb:generate && pnpm run build)
Note
The simulator must be running for kubb:generate to succeed.
The above command will generate the necessary TypeScript bindings for the simulator and build the web interface. You only need to run it once.
Then, to start the web server, run the following command:
(cd webui && pnpm run start)
The UI should then be available at http://localhost:3000
We currently have the following protocols implemented:
- FaB: The single-shot FaB protocol, and a multi-shot adaptation.
- hBFT: Implementation of the hBFT protocol
PBFT: The original PBFT protocol, as described in the PBFT paper;- PBFT-Java: A buggy implementation of the PBFT protocol, ported from the PBFT-Java repository;
- Fast-HotStuff: An unsuccessful attempt at improving the design of HotStuff, as described in the Fast-HotStuff paper;
- XRPL: XRP Ledger Consensus Protocol implementation;
- Your protocol here? :-)
See additional documentation in the docs directory.