Add BENCH_RESTARTABLE flag; skip stop/start for no-daemon systems#899
Merged
Conversation
Embedded-CLI systems (clickhouse-local variants, duckdb, datafusion, sqlite, hyper, chdb, spark variants, etc.) have no-op start/stop and their ./check always succeeds. The unified driver still ran the cold cycle stop -> wait_stopped -> drop_caches -> start -> check on every query, and bench_wait_stopped would burn its full 60s timeout each time because ./check never starts failing. Across the query sweep that's ~25-30 minutes of pure idle wait per benchmark run. Reintroduce BENCH_RESTARTABLE (default yes) as a flag independent of BENCH_DURABLE. When no, bench_run_query and bench_concurrent_qps skip the stop/wait/start/check dance and only drop_caches between queries. Set BENCH_RESTARTABLE=no on the 34 shims whose start/stop are pure no-ops. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BENCH_RESTARTABLE(defaultyes) inlib/benchmark-common.shas a flag independent ofBENCH_DURABLE.BENCH_RESTARTABLE=no, the cold cycle inbench_run_queryand the pre-window restart inbench_concurrent_qpsskip./stop→bench_wait_stopped→./start→bench_check_loopand justbench_flush_caches.BENCH_RESTARTABLE=noon 34 shims whosestart/stopare no-ops: clickhouse-local variants (clickhouse-parquet{,-partitioned},clickhouse-datalake{,-partitioned},chyt), duckdb variants, datafusion variants, glaredb variants, spark variants, sail/sail-partitioned, hyper{,-parquet}, chdb{,-parquet-partitioned}, sqlite, turso, octosql, opteryx, drill.Why
Embedded-CLI systems have no-op
start/stopbut./checkkeeps succeeding (the CLI is launchable on demand). The unified driver still ran the cold cycle on every query, andbench_wait_stoppedwould burn its full 60s timeout each time because./checknever starts failing. Across a 43-query sweep that's ~25-30 minutes of pure idle wait per benchmark run.Test plan
./benchmark.shagainst one shim system that usesBENCH_RESTARTABLE=no(e.g.duckdb/) and confirm cold queries no longer block 60s onbench_wait_stopped../benchmark.shagainst one shim system that keeps the defaultBENCH_RESTARTABLE=yes(e.g.clickhouse/) and confirm the cold cycle still runs.[t1,t2,t3]/Load time/Data sizelines — format unchanged.🤖 Generated with Claude Code