Read the dataset from the copy in this machine's region - #1091
Merged
Conversation
alexey-milovidov
had a problem deploying
to
benchmark-approval
July 26, 2026 00:08 — with 
GitHub Actions
Error
GitHub Actions
Error
The daily fleet moved from eu-central-1 to us-east-1 on 2026-07-20 (08bab92, "the region defaults to us-east-1 (cheaper)"), but the public datasets live in clickhouse-public-datasets, which is in eu-central-1. Every entry that reads S3 at query time has been going transatlantic since. clickhouse-web took the worst of it, ~30x on every machine size: machine 2026-07-19 2026-07-25 ratio c6a.large 459 s 10114 s 22.0x c6a.xlarge 170 s 5306 s 31.2x c6a.2xlarge 91 s 2970 s 32.7x c6a.4xlarge 52 s 1730 s 33.4x c8g.4xlarge 50 s 1791 s 35.8x c6a.metal 21 s 754 s 35.6x c7a.metal-48xl 19 s 628 s 32.7x c8g.metal-48xl 21 s 631 s 30.4x The ratio barely moves across machine sizes and is uniform across all 43 queries: this is round-trip latency, not CPU. The web disk issues many small granule-level range reads, and clickhouse-web/query drops the filesystem cache before every try, so all three tries pay it. clickhouse-datalake, whose reads are large and sequential, lost a milder 3.8x on cold (119 s -> 452 s). Entries that download the data first are unaffected and stayed flat. The ClickBench paths are now mirrored in clickhouse-public-datasets--us-east-1. lib/dataset-bucket.sh asks the instance metadata service where we are and picks the nearer copy: eu-* keeps the original bucket, everything else takes the mirror. If IMDS is unreachable (not EC2, or another cloud) we keep the original, which is what every non-EC2 run has always used; CLICKBENCH_DATASET_BUCKET overrides the choice. The create.sql files are rewritten on the way into the client rather than templated, so they stay valid standalone SQL. bench_dataset_sql covers the three forms they use: the virtual-hosted hostname with or without a region, the s3:// scheme, and the region DuckDB's S3 secret signs with. versions/prepare-data/ is left alone: those scripts are one-off tools for regenerating the versions-benchmark files, not part of any benchmark run. The files they produce are mirrored in full, so the versions suite itself runs anywhere. Verified against the live bucket: ClickHouse S3 single-file and {0..99} glob, the web disk, and both DuckDB entries through their real ./load scripts all return 99997497 rows and 630500 for Q1, identical from either bucket. Region mapping checked for seven regions with faked IMDS responses, and sourcing is safe under set -euo pipefail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
alexey-milovidov
force-pushed
the
region-aware-dataset-bucket
branch
from
July 26, 2026 00:15
06951a3 to
c6b4951
Compare
alexey-milovidov
had a problem deploying
to
benchmark-approval
July 26, 2026 00:15 — with
GitHub Actions
Error
GitHub Actions
Error
alexey-milovidov
temporarily deployed
to
benchmark-approval
July 26, 2026 00:16 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
…rtitioned, clickhouse-web, duckdb-datalake, duckdb-datalake-partitioned (c6a.2xlarge, c6a.4xlarge, c6a.large, c6a.metal, c6a.xlarge, c7a.metal-48xl, c8g.4xlarge, c8g.metal-48xl)
Contributor
|
Results for Logs:
|
…rtitioned, clickhouse-web, duckdb-datalake, duckdb-datalake-partitioned (c6a.large, t3a.small)
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The daily fleet moved from eu-central-1 to us-east-1 on 2026-07-20 (08bab92, "the region defaults to us-east-1 (cheaper)"), but the public datasets live in
clickhouse-public-datasets, which is in eu-central-1. Every entry that reads S3 at query time has been going transatlantic since.clickhouse-webtook the worst of it — roughly 30x on every machine size:The ratio barely moves across machine sizes and is uniform across all 43 queries: this is round-trip latency, not CPU. The web disk issues many small granule-level range reads, and
clickhouse-web/querydrops the filesystem cache before every try, so all three tries pay it.clickhouse-datalake, whose reads are large and sequential, lost a milder 3.8x on cold (119 s → 452 s). Entries that download the data first were unaffected and stayed flat, which is what rules out the machines themselves.The change
The ClickBench paths are now mirrored in
clickhouse-public-datasets--us-east-1.lib/dataset-bucket.shasks the instance metadata service where we are and picks the nearer copy:eu-*→clickhouse-public-datasets(eu-west-1 and eu-north-1 are far closer to Frankfurt than us-east-1 is)clickhouse-public-datasets--us-east-1CLICKBENCH_DATASET_BUCKET=<name>overrides all of itThe
create.sqlfiles are rewritten on the way into the client rather than templated, so they stay valid standalone SQL and the diff is one line perloadscript.bench_dataset_sqlcovers the three forms they use: the virtual-hosted hostname with or without a region, thes3://scheme, and the region DuckDB's S3 secret signs with.lib/dataset-bucket.shclickhouse-web/loadbench_dataset_sql < create.sql | clickhouse-clientclickhouse-datalake{,-partitioned}/load--query="$(bench_dataset_sql < create.sql)"duckdb-datalake{,-partitioned}/load-fthat (DuckDB needs a real file)versions/cloud-init.sh.inBUCKETfrom${DATASET_HOST}clickhouse-web/installversions/prepare-data/is deliberately left alone: those scripts are one-off tools for regenerating theversions-benchmarkfiles, not part of any benchmark run. The files they produce are mirrored in full, so the versions suite itself runs anywhere.Testing
Against the live bucket, ClickHouse S3 single-file and
{0..99}glob, the web disk, and both DuckDB entries through their real./loadscripts all return 99997497 rows and 630500 for Q1 — identical from either bucket:{0..99}globs3://globA full key-and-size inventory diff confirms the mirror is complete for everything the benchmarks read. Region mapping is checked for seven regions with faked IMDS responses, non-AWS IMDS falls back correctly, and sourcing is safe under
set -euo pipefail. Off-EC2 detection costs 1s once perload.Not covered: IMDS auto-detection never fires on a dev box, so the first real daily run is what confirms it. If it works,
clickhouse-webon c6a.4xlarge should drop from ~1730 s back to ~52 s.Related
The Tier 2 entries that reference the bucket (
trino-datalake,presto-datalake,athena,redshift,snowflake,singlestore,crunchy-bridge,databricks,s3select) are untouched — they are managed services with their own fixed regions, not our EC2 fleet, and some are pinned to eu-central-1 on purpose.The unconditional
SYSTEM DROP FILESYSTEM CACHEinclickhouse-web/query— what turned a latency penalty into a 30x one — is fixed separately in #1092.🤖 Generated with Claude Code