Upgrade ClickHouseIO to use ClickHouse Java Client V2 by BentsiLeviav · Pull Request #37611 · apache/beam

BentsiLeviav · 2026-02-15T20:57:32Z

Description

Modernizes ClickHouseIO by migrating from the legacy JDBC driver (v0.6.3) to ClickHouse Java Client v2 (v0.9.6), improving performance, compatibility, and maintainability.

Key Changes

Core Migration:

Replaced JDBC components (Connection, Statement, ResultSet) with Java Client v2 APIs (Client, Records, GenericRecord)
Migrated from JDBC URL format to explicit clickHouseUrl, database, and table parameters

API Enhancements:

Added new write(String clickHouseUrl, String database, String table) method signature
Deprecated the old write(String jdbcUrl, String table) method with backward compatibility
Added ClickHouseJdbcUrlParser utility to parse legacy JDBC URLs and maintain compatibility
Updated withProperties() to support property merging (new properties override existing ones)
Added deprecated overload for legacy getTableSchema(jdbcUrl, table) signature - that is because of the BigQuery to ClickHouse Dataflow template usage.

Backward Compatibility - No breaking changes - all existing pipelines continue to work via deprecated methods

fixes #37610

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

…1212

…ckward compatibility with previous API

gemini-code-assist · 2026-02-15T20:58:04Z

Summary of Changes

Hello @BentsiLeviav, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly modernizes the ClickHouseIO connector by upgrading its underlying driver from the legacy JDBC driver to the ClickHouse Java Client V2. This migration aims to improve the overall performance, compatibility, and maintainability of ClickHouse interactions within Apache Beam. The changes include a new, more explicit API for writing data and retrieving table schemas, while carefully preserving backward compatibility for existing pipelines through deprecated methods and a dedicated JDBC URL parser.

Highlights

Core Driver Upgrade: Migrated ClickHouseIO from the legacy JDBC driver (v0.6.3) to the modern ClickHouse Java Client v2 (v0.9.6), enhancing performance, compatibility, and maintainability.
API Modernization: Replaced JDBC-specific components (Connection, Statement, ResultSet) with the new Java Client v2 APIs (Client, Records, GenericRecord) for all internal operations.
New Write Method: Introduced a new write(String clickHouseUrl, String database, String table) method for explicit connection parameters, aligning with the new client's approach.
Backward Compatibility: Deprecated the old write(String jdbcUrl, String table) method and added ClickHouseJdbcUrlParser to automatically parse legacy JDBC URLs, ensuring existing pipelines continue to function without breaking changes.
Enhanced Property Handling: Updated withProperties() to support intelligent merging of connection properties, allowing new properties to override existing ones.
Updated Schema Retrieval: The getTableSchema method was refactored to use the new Java Client v2, with a deprecated overload provided for backward compatibility with JDBC URLs.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

sdks/java/io/clickhouse/build.gradle
- Updated the ClickHouse dependency version from JDBC 0.6.4 to Java Client V2 0.9.6.
- Replaced the primary implementation dependency from 'clickhouse-jdbc' to 'client-v2'.
- Added 'clickhouse-jdbc' as a test-only dependency to support backward compatibility tests.
sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java
- Replaced JDBC-related imports with ClickHouse Java Client V2 API imports.
- Updated Javadoc for the write method to recommend the new signature and mark the old one as deprecated.
- Updated documentation links for deduplication to point to the current ClickHouse website.
- Introduced a new public static write method that accepts explicit ClickHouse URL, database, and table parameters.
- Modified the deprecated write method to parse JDBC URLs using ClickHouseJdbcUrlParser and delegate to the new write method.
- Updated the Write PTransform to use 'clickHouseUrl' and 'database' fields instead of 'jdbcUrl'.
- Adjusted the expand method to call the new getTableSchema signature.
- Updated the WriteFn builder to reflect the new connection parameters.
- Added a withProperties method to the Write PTransform for merging connection properties.
- Replaced the internal ClickHouseConnection field in WriteFn with the new Client from Java Client V2.
- Refactored the WriteFn.setup() method to initialize the ClickHouse Java Client V2 with proper authentication and options.
- Updated the WriteFn.tearDown() method to close the new ClickHouse Client.
- Rewrote the WriteFn.flush() method to use the ClickHouse Java Client V2's insert API for writing data in RowBinary format.
- Deprecated the old getTableSchema(String jdbcUrl, String table) method and implemented a new getTableSchema(String clickHouseUrl, String database, String table, Properties properties) using the new client API.
sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseJdbcUrlParser.java
- Added a new utility class, ClickHouseJdbcUrlParser, to parse legacy JDBC URLs into ClickHouse URL, database, and properties.
- Introduced an inner class, ParsedJdbcUrl, to encapsulate the parsed components of a JDBC URL.
- Implemented methods to extract and normalize HTTP/HTTPS URLs, database names, and query parameters from various JDBC URL formats.
sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/AtomicInsertTest.java
- Updated test methods to use the new ClickHouseIO.write signature with explicit URL, database, and table.
- Modified method signatures to throw generic Exception instead of SQLException, aligning with the new client's API.
sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java
- Added static fields for clickHouseUrl and database to be used by tests.
- Updated the setup method to initialize clickHouseUrl and database from the test container.
- Modified the setUp method to initialize the new ClickHouse Client instead of a JDBC Connection.
- Updated the after method to close the new ClickHouse Client.
- Refactored executeSql, executeQuery, and executeQueryAsLong methods to interact with the new ClickHouse Client API.
- Added new helper methods executeQueryAsString and isServerAlive for client-based interactions.
sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIOJdbcBackwardCompatibilityTest.java
- Added a new test class to verify the backward compatibility of the deprecated JDBC URL-based write method.
- Included tests for basic JDBC URL parsing, parameter extraction, default setting preservation, and equivalence with the new write method.
sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIOPropertyMergingTest.java
- Added a new test class to validate the property merging logic within the withProperties() method.
- Tests cover property extraction from JDBC URLs, overriding existing properties, preserving properties, handling multiple withProperties calls, and precedence rules.
sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIOTest.java
- Updated various test methods (testTupleType, testComplexTupleType, testPrimitiveTypes, testArrayOfPrimitiveTypes) to use the new Records and GenericRecord API for result assertions.
- Modified the internal write helper method to utilize the new ClickHouseIO.write signature with explicit properties.

Activity

BentsiLeviav created the pull request to upgrade ClickHouseIO to use ClickHouse Java Client V2.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2026-02-15T21:40:34Z

Assigning reviewers:

R: @chamikaramj for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

mzitnik

Overall looks great, onlya few comments

mzitnik · 2026-02-16T07:47:08Z

sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java

- * ReplicatedMergeTree. Without replication, inserting into regular MergeTree can produce
- * duplicates, if insert fails, and then successfully retries. However, each block is inserted
- * atomically, and you can configure block size with {@link Write#withMaxInsertBlockSize(long)}.
+ * <p>Deduplication is performed by ClickHouse if inserting to <a


Should we also add a line for SharedMergeTree

mzitnik · 2026-02-16T08:32:42Z

sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java

+              .setClientName(
+                  String.format("Apache Beam/%s", ReleaseInfo.getReleaseInfo().getSdkVersion()));
+
+      try (Client client = clientBuilder.build()) {


The new client has a getTableSchema that describes the table schema. I recommend using it. Basically, it provides a TableSchema for a specific table.

Thanks for the suggestion! I agree that using the client's built-in getTableSchema() would be more elegant. However, this would entail:

Mapping between com.clickhouse.client.api.metadata.TableSchema (Java Client v2) and Beam's org.apache.beam.sdk.io.clickhouse.TableSchema (different class hierarchies)

Testing the conversion logic for all supported ClickHouse types

Verifying consistency with the current implementation

Since this PR is already on the JDBC → Java Client v2 migration, I think this schema refactoring is out of scope and deserves a separate PR to keep things focused and reviewable.

I've filed #37613 to track this improvement and left a TODO comment in the code to refer to it. We can revisit this as a follow-up PR once this main migration is merged.

Would you like me to add this to the current PR, or are you okay with handling this in a separate PR?

mzitnik · 2026-02-16T08:42:07Z

sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/AtomicInsertTest.java

  @Test
-  public void testAtomicInsert() throws SQLException {
+  public void testAtomicInsert() throws Exception {
    int size = 100000;


lets have it as const

mzitnik · 2026-02-16T08:45:12Z

sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java

+   */
+  boolean executeSql(String sql) throws Exception {
+    try {
+      client.query(sql).get(30, TimeUnit.SECONDS);


same here lets have 30 as const

BentsiLeviav added 9 commits February 15, 2026 18:57

update links

8eaa1fa

add jdbc string parser to support backward compatibility

c37d2e4

add tests to jdbc parser

4dd3805

add tests to check backward compatibility with the new jdbc parser

5261cf9

bumping up the java client version

0aea6c8

set hardcoded api version due to testcontainers/testcontainers-java#1…

f1a7ae8

…1212

update the base clickhouse test to use the recent java client

ac9842a

refactor ClickHouseIO to use the new java client while maintaining ba…

ae77799

…ckward compatibility with previous API

Adjust tests to use the new java client

7c90301

github-actions bot added java io clickhouse labels Feb 15, 2026

github-actions bot added the Next Action: Reviewers label Feb 15, 2026

mzitnik reviewed Feb 16, 2026

View reviewed changes

liferoad added this to the 2.72.0 Release milestone Feb 16, 2026

liferoad requested review from Abacn and damccorm February 16, 2026 13:31

BentsiLeviav added 4 commits February 16, 2026 16:40

move numbers to CONSTs

86637aa

make consts static

a2d374f

add github issue for better tracking

becedc7

add explanation of SharedMergeTree

70f153d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade ClickHouseIO to use ClickHouse Java Client V2#37611

Upgrade ClickHouseIO to use ClickHouse Java Client V2#37611
BentsiLeviav wants to merge 13 commits intoapache:masterfrom
BentsiLeviav:feat/clickhouseio-upgrade-java-client

BentsiLeviav commented Feb 15, 2026

Uh oh!

gemini-code-assist bot commented Feb 15, 2026

Uh oh!

github-actions bot commented Feb 15, 2026

Uh oh!

mzitnik left a comment •

edited

Loading

Uh oh!

mzitnik Feb 16, 2026

Uh oh!

BentsiLeviav Feb 16, 2026

Uh oh!

mzitnik Feb 16, 2026

Uh oh!

BentsiLeviav Feb 16, 2026

Uh oh!

mzitnik Feb 16, 2026

Uh oh!

BentsiLeviav Feb 16, 2026

Uh oh!

mzitnik Feb 16, 2026

Uh oh!

BentsiLeviav Feb 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

BentsiLeviav commented Feb 15, 2026

Description

Key Changes

Core Migration:

API Enhancements:

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist bot commented Feb 15, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions bot commented Feb 15, 2026

Uh oh!

mzitnik left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mzitnik left a comment •

edited

Loading