Repository navigation
BenchBox 0.5.0
·
353 commits
to develop
since this release
Before you upgrade
- BREAKING: Tuning that a platform cannot apply now fails. Earlier releases ran without it. Check saved tuning files with
benchbox tuning validate <file> --platform <platform>; errors name the tuning type and the reason. Unknown keys in a tuning YAML also fail. - BREAKING: BenchBox bundles its own SQLGlot. SQL translation now uses the bundled SQLGlot (
benchbox-sqlglot30.29.0.1, based on SQLGlot 30.29.0). BenchBox no longer installs the publicsqlglotpackage or accepts its expression objects; installsqlglotyourself if your code uses it. - BREAKING: Conflicting option spellings stop the run. Pass one spelling. Deprecated spellings warn and stop working in 0.6.0: use
--platform name:modeinstead ofdeployment_modeand similar options,compute_resourceandcompute_sizeinstead of vendor names such aswarehouse, and--execution-engine streaminginstead of--polars-streaming. - BREAKING: Python API changes.
translate_sql_querytakes namedrewritesinstead ofpre_processorsandpost_processors. The deprecatedrun_throughput_testandrun_official_benchmarkneedadapter=and at least 2 streams.benchbox.mcp.create_server()no longer sets up logging or tracing or acceptslog_level;run_server()does. - BREAKING: ParadeDB, SingleStore and Databend connections. ParadeDB requires
pg_search; use theparadedb/paradedb:0.26.2-pg18image. SingleStore has nocloudmode; use--platform singlestorefor Helios and self-managed clusters.databend:self-hostedconnects over HTTP on port 8000 by default. - Options that were ignored now apply. DuckDB
threadsandtemp_directory, SQLitedatabase_pathandtimeout, and--platform-optionvalues equal to the default now override saved credential profiles; remove any you do not want. Daskuse_distributed=falsewithscheduler_addressfails; use--execution-engine distributed. Polars--tuning autono longer switches to streaming on small hosts. - Some databases are rebuilt once. DuckDB
autoruns no longer request CHECK constraints, and nested tuning settings change the generated database name. ClickHouse Server loads intobenchbox_<benchmark>_<scale>instead ofdefault; set--platform-option database=<name>to choose one. Drop the old databases. - Certificate changes. Databricks SQL, result submission and data downloads verify TLS against
certifi, not the system store; behind a corporate CA, setREQUESTS_CA_BUNDLEorSSL_CERT_FILE. Source checkouts need uv 0.13.4 or newer. Rebuild local Velox and LakeSail Docker images. - Comparisons and result files.
benchbox comparerefuses results with different deployments, execution engines or gateways unless you pass--allow-heterogeneous-comparison, and warns when data generation or power-stream policy differs. Results use schema 2.3; update tools that read them. - Rerun baselines: Throughput and Power. Throughput streams are numbered 1 to S, TPC-H power iterations all use stream 00, and Throughput honors
--queriesandvalidation_mode; a query subset reports no Throughput@Size. A savedstream_timeout_secondsnow applies to Throughput; remove it to keep the default. - Rerun baselines: translated SQL. With the bundled SQLGlot, many queries on SQLite, ClickHouse, Spark, Databricks, QuestDB, DataFusion, PostgreSQL, Trino, Presto, Athena and SingleStore get different SQL, as do some Read Primitives on DuckDB, BigQuery and StarRocks.
STDDEVandVARIANCEuse the sample form on MySQL, Doris, StarRocks and SingleStore. - Rerun baselines: wrong answers fixed. TPC-DI EQ3, EQ6 and EQ8; ClickHouse TPC-DI AQ6, CoffeeShop TM1, Read Primitives
map_accessand six TPC-Havoc variants; DataFusion TPC-DI EQ7; Doris and StarRocks TPC-DS queries such as 46 and 68; and QuestDB filters on nullable columns now return correct rows. - Rerun baselines: benchmark definitions. TPC-DS OBT 0.2 uses a wider table and runs 90 of the 99 TPC-DS queries, which now return the TPC-DS answers.
vector_searchstores 4-byte floats, and NYC Taxi and TSBS DevOps derive parameters from the seed and query id. - Rerun baselines: SQLite and DataFusion loads. The SQLite loader reads quoted fields and loads empty nullable fields as NULL, which changes SQLite TPC-DS and ClickBench. Reload DataFusion ClickBench data so empty strings stay empty.
- Rerun baselines: PostgreSQL family and SQLite. PostgreSQL-family loads run
ANALYZEon every table, and SQLite indexes TPC-Havoc and Data Vault join keys, so some queries that took minutes now finish far faster. SQLite plan fingerprints also change. - Rerun baselines: ClickHouse and tuning maintenance. Tuned ClickHouse runs wait for merges after loading and run
OPTIMIZEonly with--platform-option optimize_after_load=true, and the TPC-H template no longer partitionsLINEITEMandORDERS. Tuning maintenance on ClickHouse, Redshift, Databricks and Snowflake runs after each table loads. - Rerun baselines: Spark, LakeSail and Velox. Untuned runs use engine defaults for adaptive execution and shuffle, and query times no longer include clearing the session cache. Local Spark sets the driver heap to 60% of RAM unless you pass
--driver-memory. A tuning file'ssparkmap applies on Velox, not LakeSail.
Added
- Deployment selectors on more platforms.
--platform name:modenow covers platforms such asfirebolt:cloudandpg-duckdb:motherduckand works in MCP and Python, with sharedcompute_resourceandcompute_sizeoptions. Each deployment ranks separately. - Execution engines.
--execution-enginechooses how a query runs: Polarsin-memoryorstreaming, Daskthreadedordistributed, and pg-duckdbduckdborpostgres. Results record the requested and observed engine. - TPC-H refresh, QphH and subscale results.
--benchmark-option tpch_refresh_functions=trueadds RF1 and RF2 to Power and Throughput, andbenchbox metrics qphhreports QphH@Size. Refresh runs only on DuckDB, SQLite, PostgreSQL, Snowflake and SingleStore; other platforms skip maintenance. TPC-H and TPC-DS runs below scale factor 1 rank in their own unofficial cohort. - GizmoSQL platform (experimental). Run TPC-H and TPC-DS on a GizmoSQL Core 1.37 or newer server with the
gizmosqlextra. - More tuning. PostgreSQL partitions, indexes and clusters tuned tables. CedarDB partitions, Citus distributes, and ParadeDB and pg_mooncake cluster and sort. Results list the tuning a platform ignored, and
--dry-run --tuning tunedworks without the platform driver. - Gateways and containers. Snowflake (
--gateway) and Databricks (thegatewaysetting) can route connections through a gateway; such results are never official. Server engines in Docker on macOS or Windows are recorded and ranked as container runs. - More run evidence. Results record the bundled SQLGlot version that translated their SQL, the engine version on DuckDB, SQLite and DataFusion, and live Spark, LakeSail and Velox session settings.
- Disk space check. Runs fail when the output volume has less than 5% free; set
BENCHBOX_MIN_FREE_DISK_PERCENTto change the limit.
Changed
- Safer publishing.
benchbox publishverifies anonymized files before copying them,benchbox submitpseudonymizes before validating, and TPC results are labelled derived, not certified. - Faster startup. Importing BenchBox no longer loads the bundled SQLGlot until it translates SQL, and
benchbox runloads only the requested platform's adapter. - Published rankings. Results Explorer removes 25 results built on superseded SQL translations or wrong values from its rankings, and adds 58 replacements for ClickHouse Local, DataFusion and SQLite.
Fixed
- Database safety. On a validation mismatch, BenchBox stops instead of dropping a server database that holds tables from outside the benchmark. On Snowflake, a mismatch or
--force-recreatedrops only the benchmark's tables. - Runs and data.
--force datagenregenerates data, load-only runs exit 0, and runs that execute no queries fail. TPC-H and TPC-DS generation no longer hang when the filesystem cannot flush, and NYC Taxi preparation no longer times out at scale factor 1. - More benchmarks run. SQLite runs NYC Taxi, Flight Data, Data Vault and all four primitives suites. QuestDB runs far more of TPC-H, TPC-DS, TPC-DI, TPC-Havoc and ClickBench. Presto and Trino load every benchmark, and SingleStore runs 36 more TPC-Havoc variants. Operations an engine cannot express are skipped with a reason.
- Tuned runs. Tuned runs on PostgreSQL, Athena, Synapse, Fabric Warehouse and BigQuery no longer abort. Tuned SQLite TPC-DS runs load, and ClickHouse primary keys, Spark session settings, Databricks
z_orderand PostgreSQL-familyCLUSTERapply as requested. - Platform options. Options now reach EMR Serverless, Dataproc, Synapse Spark, Fabric, PySpark, Snowpark Connect, Athena Spark, Redshift, InfluxDB and CedarDB, and saved credential profiles no longer block command-line compute settings.
- Spark, LakeSail and Velox. TPC-DI no longer fails on Spark and Velox with a BOOLEAN/INT type mismatch, or on LakeSail with small decimals in scientific notation. Wide tables such as TPC-DS OBT load without running out of Java heap.
- DataFrame runs. CoffeeShop, Data Vault, ClickBench and H2O-DB queries that failed on lazy frames now run. JoinOrder at scale factor 1 peaks near 5 GiB instead of 9 GiB on Polars, and near 3 GiB instead of over 7 GiB on DataFusion.
- Disk and memory use. StarRocks and pg_mooncake no longer fill disks on repeated or large loads, TPC-DS OBT generation is capped at 6 GB of memory, and Presto in Docker no longer runs out of memory loading TPC-H.
- Query plans. Plan capture parses real output from SQLite, ClickHouse, Trino, Presto, Databend, QuestDB, Doris, SingleStore and LakeSail.
Security
- Compromised eMudhra emSign CAs. BenchBox requires
certifi2026.10.19 or newer, and its BigQuery, cloud, cloud Spark, LakeSail and Velox extras requiregrpcio1.84.1 or newer. Older releases trust these CAs, so a network attacker could impersonate cloud endpoints and steal tokens. See the Python advisory. - Secrets and pseudonyms. Dry-run preview files no longer include saved credentials, and a printed
DatabaseConfighides secrets.submit,exportandpublishcreate a private salt in~/.benchbox/submission_salton first use, so anonymized exports no longer use an empty salt. Anonymized bundles drop the compute resource name. - Locked dependencies. The lock file moves
filelockto 4.2.4 andtornadoto 6.6 to clear six published CVEs.
Documentation
- Platform reference. A new page lists each platform's shipped tuning templates and the tuning types it applies.