Reproducible benchmarks for columnar storage formats, and a standing attempt to re-run the numbers the papers published.
The field has a measurement problem. BtrBlocks reports 2.2 times faster than Parquet, DuckDB reports on par, AnyBlox reports a 5% sandbox tax, and LiquidCache reports that decoding is the primary bottleneck. None of those contradict each other. They are answers to different questions that happen to share the word "faster", taken on different hardware, against baselines whose configuration was almost never disclosed. This repository exists to take the same measurements the same way, publish the tuple rather than the number, and say so when a result does not reproduce.
It is the measurement half of tamnd/iris, and it is deliberately a separate repository, because a project that grades its own homework should at least keep the marking scheme somewhere else. Nothing here privileges iris. It gets no visual distinction on the results pages, no separate configuration path, and it appears in the tables alphabetically like everything else.
A result is a tuple, not a number. System, version, configuration digest, corpus, corpus digest, workload, query, storage tier, machine class, protocol, repetition, and the measurement. Producing a bare "2.2 times faster" requires actively discarding fields, which is the point of the schema.
Baselines are configured by their own documentation. Every system in the matrix carries a CONFIG.md naming the source of its configuration, the settings applied, the deviations and their reasons, the alternatives that were rejected, and the date the configuration was last reviewed. Where we had to guess, the row records that we guessed, and every number produced that way renders with a marker.
Losses are published. If iris is slower, the number goes up. If a reproduction fails, the verdict goes up, after contacting the authors. If a rival's maintainer says we configured them wrong, that goes up too, alongside the corrected run, with the original struck through rather than deleted.
Query benchmarks: ClickBench, TPC-H, TPC-DS, the Star Schema Benchmark, the Join Order Benchmark, and the h2o database benchmark. Corpora: Public BI, the ALP double columns, the FSST text corpora, Silesia, and enwik8 and enwik9. Formats: Parquet, ORC, Arrow IPC, BtrBlocks, FastLanes, Vortex, Lance, Nimble, and the self-decoding formats. Engines: DuckDB, DataFusion, ClickHouse, Polars.
Five storage tiers, from a warm in-process buffer to real object storage over a real network, because a ranking taken on a warm local file is a ranking of one tier and the field mostly publishes that one.
This is not a general purpose benchmarking framework. It measures columnar storage formats and the systems that read them, and it stops there.
Public BI has been both the design input and the evaluation set for six years of encoding research. The formats that do well on it were, in part, designed against it. So iris-bench assembles a held-out corpus, publishes the selection criteria before running anything against it, freezes it by content digest before any decoder is tuned, and reports the delta per format between Public BI and the held-out set. A format that degrades least on unseen data is not thereby better, and the write-up says so, but the delta is a measurement nobody in this area has published.
Pre-alpha. The specification is written and the harness is not. Milestones B0 through B8 are public with one issue per exit gate.
B0 through B4 need no iris code to exist, which is deliberate. If iris is never built, the reproduction of the published figures and the storage tier study still stand on their own.
| Crate | What it is |
|---|---|
bench-core |
Timing, repetition, bootstrap confidence intervals, the result row. No benchmarks in it. |
bench-env |
Environment capture and the eligibility gates that abort a run on an unfit machine. |
bench-corpus |
Corpus manifests, fetching, generating, verifying, and the content addressed store. |
bench-driver |
The Driver trait every system implements. The prepare, load and run split lives here. |
bench-run |
The runner. Process isolation, ordering, storage tiers, cache control. |
bench-store |
Append only result storage and the claim ledger. |
bench-report |
Rendering, including the rules that stop a misleading table being drawn. |
bench-cli |
iris-bench the command line tool. |
drivers/* |
One crate per system under test. They depend on bench-driver and nothing else in the tree. |
Requires Rust 1.98 or newer.
cargo build --workspace
cargo test --workspace
The harness refuses to produce a publishable number on a machine that fails the eligibility gates, and it says which gate failed. That is not configurable.
Results are labelled by hardware class rather than by machine name. docs/MACHINES.md lists the fleet, what each class is used for, and what it cannot be used for.
Apache License 2.0. See LICENSE.
TPC-H and TPC-DS are trademarks of the Transaction Processing Performance Council. Results published here are derived from the TPC-H and TPC-DS specifications, are not audited, and are not comparable with official TPC Results. See docs/LICENSING.md.