Summary
Bulk-loading JSON-lines datasets into ClickHouse currently pipes the entire file through a single SSH → docker exec → one INSERT … FORMAT JSONEachRow command. That works for small fixtures but is unreliable for the multi-GB datasets used in the ClickHouse benchmark (experiment_run_clickhouse.py), where a single oversized INSERT gives no partial progress and no useful diagnostics when it fails.
We should load in bounded batches, wait for ClickHouse to actually be reachable first, and fail with an actionable message when input data is malformed.
Summary
Bulk-loading JSON-lines datasets into ClickHouse currently pipes the entire file through a single SSH →
docker exec→ oneINSERT … FORMAT JSONEachRowcommand. That works for small fixtures but is unreliable for the multi-GB datasets used in the ClickHouse benchmark (experiment_run_clickhouse.py), where a single oversized INSERT gives no partial progress and no useful diagnostics when it fails.We should load in bounded batches, wait for ClickHouse to actually be reachable first, and fail with an actionable message when input data is malformed.