hyperscale-rs

Warning: Work in progress. Do not use.

Rust implementation of the Hyperscale protocol: a sharded smart-contract network that scales linearly — add shards, get proportional throughput — while keeping the properties sharding usually destroys: atomic composability (a transaction touching state on many shards commits everywhere or nowhere) and synchronous, single-chain semantics with BFT finality. No bridges, no optimistic rollback, no user-visible intermediate states.

What makes it distinctive:

• Three consensus mechanisms, one clock. Independent HotStuff-2 shard chains order transactions; execution certificates agree on their results; a slow beacon chain governs validators and topology. All three are harmonized by weighted time — a BFT-attested clock — which binds every artifact to exactly one governing committee.
• Deterministic atomic commitment. Cross-shard transactions run a provision–execute–certify pipeline: state moves between shards as merkle-proven facts about committed blocks, every shard executes deterministically on identical inputs, and quorum certificates attest outcomes rather than vote on them. No coordinator, no blocking, and locks that structurally cannot deadlock.
• Dynamic sharding. Shards split under load and merge when it recedes — live, without halting the network. State is one binary Jellyfish Merkle Tree and a shard is a prefix subtree, so resharding is a tree operation with a one-hash continuity proof; transactions straddling a reshape boundary settle atomically via attested settled sets.
• A self-regulating validator economy. The minimum activation stake is a market-clearing price recomputed every epoch from topology demand — rising when validator supply is abundant, falling when splits need staffing — while vnodes let one host run many validator identities, so a seat's marginal cost is its stake, not hardware.
• Determinism as the spine. The entire protocol stack is sans-io pure state machines. The same code runs under production I/O (tokio, libp2p, RocksDB) and under a deterministic simulator where whole multi-shard networks — fault injection included — replay byte-identically from a seed.

Architecture documentation → docs/ — the whole story in a five-page overview, per-subsystem deep dives, and a consolidated invariant register intended as the starting point for formal verification. The execution engine — an effect-typed VM whose every state access is declared before execution — lives in the vm/ submodule with its own documentation suite.

Formal models → specs/ — an in-progress track of machine-checked Quint models (verified with Apalache) of the critical safety properties. Each is transcribed from the implementation it models and cross-checked against it by the deterministic scenario suite.

Pre-built binaries for Linux (x86_64) and macOS (ARM64) are available on the Releases page.

You can pull the latest Docker image from the GitHub Container Registry:

docker pull ghcr.io/flightofthefox/hyperscale-rs:latest

Install Rust using rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

You will need clang, lld, protobuf, and openssl.

brew install llvm protobuf openssl pkg-config

sudo apt-get update && sudo apt-get install -y \
    clang \
    lld \
    pkg-config \
    protobuf-compiler \
    git \
    build-essential \
    libssl-dev \
    libc6-dev

1. Install Rust.
2. Install LLVM.
3. Install Protoc.
4. Ensure clang and protoc are in your PATH.

Required only if you plan to run the launch-docker-compose.sh script or use the provided Dockerfile.

Recommended: Docker Desktop or Rancher Desktop.

Install the Docker Engine and Docker Compose plugin: Install on Ubuntu | Install on Debian | Install on Fedora

Ensure your user is in the docker group to run commands without sudo:

sudo usermod -aG docker $USER
newgrp docker

The execution engine is consumed as the vm/ git submodule. When cloning, use:

git clone --recurse-submodules https://github.com/flightofthefox/hyperscale-rs.git

If you've already cloned the repository without submodules, initialize them with:

git submodule update --init --recursive

cargo build --release

cargo run --release --bin hyperscale-sim

cargo test

For development and testing, you can launch a local cluster using the provided scripts.

The launch-cluster.sh script launches a cluster of validator nodes as background processes on your host machine. This is faster and easier for quick iteration.

./scripts/launch-cluster.sh

Options:

• --shards <N>: Number of shards (default: 2)
• --validators-per-shard <M>: Validators per shard (default: 4)
• --clean: Clean data directories before starting
• --monitoring: Start Prometheus and Grafana for metrics

The launch-docker-compose.sh script launches a full cluster inside Docker containers using Docker Compose. This more closely mimics a production environment.

./scripts/launch-docker-compose.sh

Options:

• --build <true|false>: Whether to build the docker image (default: true). Set to false to use existing image for much faster startup.
• --use-ghcr-image: Use the latest pre-built image from GHCR instead of building locally.
• --shards <N>: Number of shards (default: 1)
• --validators-per-shard <M>: Validators per shard (default: 8)
• --memory <limit>: Memory limit per validator (e.g. 1g, 512m)
• --cpus <limit>: CPU limit per validator (e.g. 0.5)
• --latency <ms>: Artificial network latency per validator
• --latency-nodes <N>: Number of nodes to apply latency to (default: 1)

To stop the Docker cluster and remove volumes:

./scripts/stop-docker-compose.sh

To run load tests against your local cluster, use the hyperscale-spammer binary.

Example command (adjust ports as needed):

./target/release/hyperscale-spammer run \
  --endpoints "http://localhost:8080,http://localhost:8081" \
  --num-shards 2 \
  --validators-per-shard 4 \
  --tps 100 \
  --duration 30s

RocksDB (librocksdb-sys) uses bindgen, which needs LLVM/libclang and MSVC headers (stdbool.h from the Windows SDK). Both are required — LLVM alone is not enough unless you run from a Developer Command Prompt.

1. Install LLVM (choco install llvm) and protobuf (choco install protobuf).
2. Install Visual Studio Build Tools with the Desktop development with C++ workload (provides vcvars64.bat and SDK headers).
3. Set LIBCLANG_PATH permanently, e.g. C:\Program Files\LLVM\bin.

Run from Developer PowerShell for VS (loads vcvars64.bat) or set LIBCLANG_PATH manually before cargo build:

$env:LIBCLANG_PATH = "C:\Program Files\LLVM\bin"

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.