Storage Benchmark Kit (SBK)

SBK is a Java framework for measuring the throughput and latency of storage systems with one common workload engine. Its drivers cover object stores, message systems, databases, file systems, caches, and local queues. The harness controls concurrency, duration, rate, payloads, timestamps, and reporting; each driver only adapts those operations to a backend.

Repository: https://github.com/kmgowda/SBK

Choose the guide that matches your task:

flowchart LR
    CLI[SBK CLI or YML] --> BOOT[Sbk bootstrap]
    BOOT --> DRIVER[Storage driver]
    BOOT --> BENCH[SbkBenchmark]
    BENCH --> WORKERS[Writer and reader workers]
    WORKERS --> DRIVER
    WORKERS --> CHANNEL[PerL channels]
    CHANNEL --> RECORDER[Latency recorder]
    RECORDER --> OUTPUT[Console / CSV / Local Web Console / Prometheus / gRPC]
    OUTPUT -->|gRPC mode| SBM[SBM aggregator]
    GEM[SBK-GEM] -->|SSH orchestration| CLI
    GEM --> SBM

The main runtime path is:

1. io.sbk.main.SbkMain delegates to io.sbk.api.impl.Sbk.
2. Sbk discovers the requested driver and logger by class name, builds the combined CLI, parses it, and constructs SbkBenchmark.
3. SbkBenchmark opens the storage, creates driver readers and writers, starts PerL recorders, and schedules worker execution.
4. Driver operations are timed by the Writer and Reader default methods and submitted to a PerlChannel.
5. PerL processes every latency record away from the worker threads and publishes periodic and total results through the selected logger.

See docs/ARCHITECTURE.md for source links, lifecycle details, concurrency boundaries, and distributed execution.

• JDK 25, either preinstalled or downloaded automatically by the wrapper
• Git
• The Gradle wrapper included in the repository; a separate Gradle installation is not required
• A real backend only when exercising a remote-storage driver

Confirm the active JVM before building:

java -version
./gradlew --version

The Gradle wrappers resolve a complete JDK 25 in this order:

1. SBK_JAVA_HOME;
2. JAVA_HOME;
3. a JDK 25 available through PATH; and
4. the SBK-managed JDK cache.

If none is available, gradlew downloads the pinned OpenJDK 25 release, verifies its SHA-256 checksum, and installs it without administrator access in ${XDG_CACHE_HOME:-$HOME/.cache}/sbk/jdks on Unix or %LOCALAPPDATA%\SBK\jdks on Windows. Later wrapper invocations reuse that installation. Set SBK_JAVA_CACHE_DIR to select a different cache directory. Automatic installation supports Linux x64/AArch64, macOS x64/AArch64, and Windows x64. Other platforms can use a manually installed JDK through SBK_JAVA_HOME. The pinned version, platform checksums, and bootstrap network/lock timeouts have one authoritative source: gradle/sbk-java-bootstrap.properties. An explicitly configured but invalid SBK_JAVA_HOME or JAVA_HOME is treated as an error so a configuration mistake is never hidden by an automatic download.

Generated launchers for sbk, sbk-yal, sbm, sbk-gem, sbk-gem-yal, and sbk-web-console use the same selection order and managed cache. They do not download Java at application startup; run the Gradle wrapper once to populate the cache, or set SBK_JAVA_HOME/JAVA_HOME. The standalone ./install-java command uses the same resolver when an explicit installation step is preferred.

Every generated SBK launcher uses one consolidated JDK 25 runtime profile from gradle/java.gradle:

-XX:+UseZGC
-XX:+UseCompactObjectHeaders
-XX:MaxRAMPercentage=50.0
-XX:+DisableExplicitGC
-XX:+ExitOnOutOfMemoryError

• Generational ZGC performs expensive collection work concurrently and dynamically scales generation sizes and GC threads. This keeps GC pauses small as benchmark concurrency, CPU count, and live heap grow.
• Compact object headers reduce per-object heap overhead and improve cache density for queues and measurement records.
• 50% maximum RAM gives large benchmark processes more heap than the JVM's default server sizing while retaining half of detected memory for SDK native buffers, direct buffers, thread stacks, filesystem cache, and the operating system.
• Disabled explicit GC prevents a driver or dependency from injecting a benchmark-distorting System.gc() collection. Necessary collections still run normally.
• Exit on OOM prevents a memory-exhausted benchmark from continuing and publishing misleading results.

The same options are applied to sbk, sbk-yal, sbm, sbk-gem, sbk-gem-yal, module launchers, and the SbkWebConsoleMain process.

ZGC reserves address space using many memory mappings. On large-memory Linux hosts, check sysctl vm.max_map_count; if it is too small, JDK 25 prints the minimum required value at startup. Configure that operating-system limit before a production benchmark.

These are safe cross-host defaults, not a substitute for capacity planning. For a dedicated benchmark host, JAVA_OPTS can override heap sizing. For example, a fixed heap removes resizing and commit/uncommit variation:

export JAVA_OPTS='-Xms32g -Xmx32g -XX:+AlwaysPreTouch'

Use fixed pre-touched heaps only when that memory is reserved for the process. Large pages are not enabled automatically because they require operating-system provisioning. To compare G1 with ZGC, explicitly disable ZGC before selecting G1:

export JAVA_OPTS='-XX:-UseZGC -XX:+UseG1GC'

Clone and build the project:

git clone https://github.com/kmgowda/SBK.git
cd SBK
./gradlew check
./gradlew installDist

The installed launcher is created at:

build/install/sbk/bin/sbk

Useful development commands:

# Compile, check style, and run tests for the whole build
./gradlew check

# Iterate on one driver
./gradlew :drivers:minio:check

# Generate launch scripts and runtime libraries
./gradlew installDist

# Rebuild the pathing JAR after dependency changes
./gradlew clean :pathingJar installDist --rerun-tasks

HaloDB and Ignite are present in the source tree but are not enabled in the aggregate build. HaloDB depends on a GitHub Packages artifact that may require credentials. The sbktemplate directory is a scaffold, not a runtime driver.

List drivers and common options:

./build/install/sbk/bin/sbk -help

Write to a local file for 30 seconds:

./build/install/sbk/bin/sbk \
  -class file \
  -file /tmp/sbk.bin \
  -writers 1 \
  -size 1048576 \
  -seconds 30

Read the same file:

./build/install/sbk/bin/sbk \
  -class file \
  -file /tmp/sbk.bin \
  -readers 1 \
  -size 1048576 \
  -seconds 30

Driver-specific options are added after SBK discovers -class. Use the selected driver with -help to see the merged option set:

./build/install/sbk/bin/sbk -class minio -help

Always treat -help as authoritative because drivers and loggers add their own options at runtime.

The dependency direction is perl <- sbk-api <- drivers and sbk-web-console <- sbk-api. SBM depends on sbk-api; SBK-GEM depends on SBM. The YML launchers wrap their corresponding programmatic APIs.

Enabled drivers are registered in both settings-drivers.gradle and build-drivers.gradle. The complete categorized inventory and the driver contract are in docs/DRIVER_GUIDE.md.

Use the synthetic PerlBench driver to compare the intrusive PerL timestamp queue with the JDK fallback under exact-count, timed-saturation, and rate-controlled workloads. It performs no storage I/O, so its results describe harness and measurement-pipeline behavior rather than a storage device. Do not confuse it with the Null driver: Null's default operation deliberately remains pending for idle, timeout, and shutdown tests, while PerlBench completes every operation immediately.

SBK currently ships these logger implementations:

• SystemLogger: human-readable periodic and final output.
• Sl4jLogger: SLF4J-backed output.
• CSVLogger: results written in CSV form.
• WebLogger: console/CSV output plus the dependency-free SBK Local Web Console.
• PrometheusLogger: CSV behavior plus Prometheus metrics exposure.
• GrpcLogger: forwards measurements to SBM for distributed aggregation.

Prometheus metrics include stable component, class, and action labels. component="sbk" identifies a direct SBK exporter, while component="sbm" identifies SBM aggregation, including benchmarks orchestrated by SBK-GEM. SBK-GEM is not a separate metrics component because SBM owns its metrics endpoint and aggregated measurements.

Use PrometheusLogger when Prometheus should retain periodic results and Grafana should display a full historical dashboard:

./build/install/sbk/bin/sbk \
  -class file -file /tmp/sbk-prometheus.dat \
  -writers 1 -size 4096 -seconds 60 \
  -out PrometheusLogger

Direct SBK metrics use http://<sbk-host>:9718/metrics by default. SBM and SBK-GEM aggregate metrics use http://<coordinator-host>:9719/metrics. Startup prints copy-paste scrape URLs for localhost, loopback, hostname, and usable host IPv4 addresses. Change the exporter with -context PORT/PATH, for example -context 19718/sbk-metrics.

The separately deployed SBK Dashboard manages official Prometheus and Grafana processes, persistent time-series history, and one endpoint-scoped dashboard per registered host:port. Its management UI defaults to port 9721 and Grafana to port 3000. See the PrometheusLogger and SBK Dashboard guide for direct and distributed commands, Docker Compose and native installation, endpoint registration, metrics semantics, retention, security, and troubleshooting.

Use WebLogger when you want live graphs without Docker, Prometheus, or Grafana:

./build/install/sbk/bin/sbk -class file -file /tmp/sbk.bin \
  -writers 4 -size 4096 -seconds 60 -out WebLogger

For a filesystem read benchmark, first create an input file and then read it with WebLogger. The record size must match between preparation and reading:

# Prepare a 1 GiB file: 1,048,576 records x 1,024 bytes.
./build/install/sbk/bin/sbk \
  -class file -file /tmp/sbk-weblogger.dat \
  -writers 1 -size 1024 -records 1048576

# Measure filesystem reads for 60 seconds and display live graphs.
./build/install/sbk/bin/sbk \
  -class file -file /tmp/sbk-weblogger.dat \
  -readers 1 -size 1024 -seconds 60 \
  -out WebLogger

SBK opens the run URL in the default browser. The lightweight Local Web Console accepts plain HTTP on all IPv4 interfaces at port 9720, retains the latest 180 minutes of snapshots in memory, and streams new summaries with server-sent events. At benchmark start and completion, SBK prints run-specific URLs for localhost, IPv4 loopback, the hostname, and every usable private or public IPv4 address discovered on the console host. Change the retention duration with -websnapshotminutes N. A later SBK process reuses a compatible server already on that port. Multiple SBK, SBM, and SBK-GEM WebLogger benchmarks can publish concurrently to the same server; each receives a unique run URL and remains independently selectable in the browser. Completed graphs remain available while a browser is connected; after all benchmarks have finished and no browser has been connected for one minute by default, the server exits automatically. WebLogger reports whether it starts a new console or uses the existing process. Automatically started consoles append lifecycle, WebLogger count, browser/client count, and exit diagnostics to $HOME/.sbk/logs/sbk-web-console-<port>.log. Change the idle grace period with -webtimeoutminutes N. Snapshots and 15-second logger heartbeats renew the active-run lease. If a benchmark is killed without completing, the configured idle timeout without either signal marks only that run abandoned; other active runs continue unaffected. Use -webopen false on headless hosts and -webport PORT to select another port. Boards default to <application> <storage>, such as SBK File, SBM MinIO, or SBK-GEM Kafka; use -boardname NAME to override that display name. The timeout supplied by the process that starts a web console remains in effect when later benchmarks reuse it. A remote browser can use a printed hostname or IP URL when routing and firewall rules permit. Because the console has no authentication or TLS, expose it only on a trusted benchmark network or use an SSH tunnel. Run sbk -out WebLogger -help for the complete option set. See the WebLogger guide for every option, Local Web Console lifecycle, distributed usage, security, and troubleshooting.

Distributed monitoring uses the same Local Web Console and data model:

# Standalone SBM aggregate Local Web Console
sbm -out SbmWebLogger -class file -action r

# SBK-GEM aggregate Local Web Console; remote nodes continue sending results through GrpcLogger
sbk-gem -out GemWebLogger -class file -nodes host1,host2 -writers 2 -size 4096 -seconds 60

The Local Web Console binds to 0.0.0.0, while benchmark clients discover and reuse it through 127.0.0.1.

• SBM accepts SBP/gRPC measurements and aggregates them.
• SBK-GEM copies and launches SBK on remote hosts over SSH while running an embedded SBM instance.
• SBK-YAL loads single-node SBK arguments from YML.
• SBK-GEM-YAL loads distributed SBK-GEM arguments from YML.

The default SBM gRPC port is 9717; Prometheus/JMX endpoints use separate configured ports. Review the component help before exposing any port outside a trusted benchmark network.

SBK records operation latency without sampling. Worker threads submit (start, end, records, bytes) measurements through PerL channels to concurrent queues. Dedicated recorder logic calculates window and total statistics and invokes the logger. This keeps histogram work out of the driver operation path.

Results are meaningful only when the experiment is controlled. Record at least:

• SBK version and commit SHA.
• Driver and vendor-client versions.
• Full command line and non-default properties.
• JVM, CPU, memory, network, and operating-system details.
• Storage topology and durability settings.
• Warm-up policy and whether results are cold-cache or warm-cache.

See the reproducibility section for a longer checklist.

Before creating a release candidate, run the one-command automated release gate documented in SBK Release Qualification. Use its local-docker profile for automatic two-node SBK-GEM and SBK-GEM-YAL functional testing without a permanent SSH host. The release profile verifies installed launchers, logger contracts, PerL concurrency and performance, YAL mapping, SBM/gRPC aggregation, remote SBK-GEM execution, generated documentation, publications, and archive checksums.

Before publishing, run the non-mutating publication assembly locally:

./gradlew clean releasePublicationDryRun \
  --no-daemon --rerun-tasks

The separate guarded release publication workflow does not execute or depend on releasecheck. It builds the current version's reproducible archives, validates native AMD64/ARM64 containers, and can publish the distributions, SBOMs, manifest, and checksums to a detailed GitHub Release, the Java modules to GitHub Packages, and the signed multi-architecture GHCR image plus the digest-identical Docker Hub image. The root publish task uses local Docker Hub environment credentials without sending them to GitHub, then dispatches the workflow with only the immutable public image digest. See the publication guide for the required confirmation, credentials, and recovery controls.

For authorized maintainers, the final publication entry point is:

VERSION=$(sed -n 's/^sbkVersion=//p' gradle.properties)
DOCKER_USERNAME="<docker-user>" \
DOCKER_PASSWORD="<docker-access-token>" \
GITHUB_TOKEN="<actions-workflow-token>" \
./gradlew publish "-PreleaseConfirm=RELEASE-${VERSION}" --no-daemon

Run it only from a clean, synchronized master after separately qualifying the same commit. The command publishes Docker Hub and dispatches the asynchronous GitHub workflow; monitor that workflow and complete the post-release checks in the publication guide.

Read CONTRIBUTING.md before changing code. The minimum verification sequence is normally:

./gradlew :<module>:check
./gradlew check
./gradlew installDist

Driver changes also require a smoke test against the relevant backend. Pull requests target master. Do not re-enable HaloDB or upgrade the intentionally pinned MinIO SDK without discussing the compatibility implications.

SBK is licensed under the Apache License 2.0. Use GitHub Issues for bugs and feature requests and GitHub Discussions for usage and design questions.