fix(asap-tools): batch ClickHouse JSON-lines bulk load instead of a single oversized INSERT - #542
Open
akanksha-akkihal wants to merge 3 commits into
Open
Conversation
Bulk-loading a JSON-lines dataset piped the entire file through a single INSERT ... FORMAT JSONEachRow, which is unreliable for the multi-GB datasets the ClickHouse benchmark now uses. Load in bounded batches via a dedicated loader script instead, poll ClickHouse HTTP until it is actually reachable before loading, and validate each line so malformed input fails with the offending line number rather than an opaque client error. Also let init_sql_file own DROP/CREATE for its own objects, so a schema that defines dependent materialized views is not broken by a standalone DROP TABLE. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bulk-loading JSON-lines datasets into ClickHouse previously piped the entire file through a single
INSERT ... FORMAT JSONEachRow, which is unreliable for the multi-GB datasets used in the ClickHouse benchmark. This PR loads in bounded batches, waits for ClickHouse to be reachable before loading, and fails with an actionable message when input data is malformed.Changes
experiment_utils/services/json/loader.py): streams thefile and inserts in bounded batches (default 100k rows) via
docker exec clickhouse-client, instead of one giant INSERT./pinguntil it returnsOk.before loading; on timeout, raise with the tail of the container logs.
bytes, so malformed input fails with the offending line number and a preview.
init_sql_fileis provided, it ownsDROP/CREATEfor its own objects — the standalone
DROP TABLEno longer runs first andbreak schemas with dependent materialized views.