Testcontainers: Add ClickHouse JDBC test#97
Conversation
# Conflicts: # server/build.gradle # Conflicts: # server/Makefile
| if (val instanceof byte[]) { | ||
| return (byte[]) val; | ||
| } | ||
| ByteBuffer bb = (ByteBuffer) val; |
There was a problem hiding this comment.
ByteBuffer bb = (ByteBuffer) val;
If the types do not match, a ClassCastException will be thrown directly is better?
There was a problem hiding this comment.
There was a bug - it always expected byte[].
Original PXF authors got following error for FDW:
ERROR: PXF server error : class java.io.DataInputStream cannot be cast to class [B (java.io.DataInputStream and [B are in module java.base of loader 'bootstrap')
I have seen:
ERROR: PXF server error : class java.nio.HeapByteBuffer cannot be cast to class [B (java.nio.HeapByteBuffer and [B are in module java.base of loader 'bootstrap')
So, here I am extending old (external-table) behaviour with one more data type. So, if it will be some other type - we will see new ClassCastException here.
server/build.gradle
Outdated
| dependsOn ":pxf-jdbc-drivers:jar" | ||
| from(project(':pxf-jdbc-drivers').configurations.runtimeClasspath) | ||
| into("${buildDir}/stage/lib") | ||
| duplicatesStrategy = DuplicatesStrategy.INCLUDE |
There was a problem hiding this comment.
INCLUDE duplicate files are all retained, with later ones silently overwriting earlier ones, so it's unclear which version ends up being the final one.
There was a problem hiding this comment.
Good point. Let's try FAIL. In this case it will force us to manually solve dependency hell.
| */ | ||
|
|
||
| dependencies { | ||
| // PostgreSQL driver is used by PXF itself. |
There was a problem hiding this comment.
PostgreSQL driver？ this is clickhouse driver?
There was a problem hiding this comment.
PostgreSQL driver included[1] into PXF distribution by default. No need to add it to libs. So, I left a comment about this.
[1] https://github.com/apache/cloudberry-pxf/blob/main/server/pxf-jdbc/build.gradle#L28
| // Zstd support for Avro/Parquet | ||
| // Compression support for Avro/Parquet | ||
| dependency("com.github.luben:zstd-jni:1.5.7-6") | ||
| dependency("at.yawk.lz4:lz4-java:1.10.3") |
There was a problem hiding this comment.
why not org.lz4:lz4-java ?
There was a problem hiding this comment.
Maven Central says that we should use at.yawk.lz4:lz4-java:
https://mvnrepository.com/artifact/org.lz4/lz4-java
…asspath. Developers should fix dependecy issue manually
2 participants
Add ClickHouse JDBC tests
Add new tests to cover Cloudberry -> PXF -> JDBC path. In this test we are verifying that main data Cloudberry and ClickHouse types can be converted back and forth.
Test covers ClickHouse 24.x and 26.x versions. Open-source ClickHouse has short support lifetime.
I tested following clickhouse-jdbc drivers:
jdbc-v1) - works well with old ClickHouse versions. In ClickHouse 25.10jdbc-v1got broken Version 25.10 breaks JBDC driver ClickHouse/clickhouse-java#2636 and all queries raised an "Magic is not correct" error message.ERROR: PXF server error : Method: getBytes("bin") encountered an exception.Add jdbc-pxf-drivers project
Add new
jdbc-pxf-driversproject to server. It excluded from default DEB-package build. Explicit actions required to install jdbc-drivers.Side quest
Cloudberry FDW serializes rows as Greenplum CSV => PXF
TextRecordReaderparses the CSV stream via univocity CSV parser => forBYTEAcolumns it callspgUtilities.parseByteaLiteral()=> which returns aByteBuffer.