Skip to content

Prepare patch release #890

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Apr 17, 2022
Merged

Prepare patch release #890

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
ed47720
Support Object/JSON and Tuple with names and types
zhicwu Apr 4, 2022
096fcf2
Rename root element
zhicwu Apr 4, 2022
e86bd13
Patch container before integration test
zhicwu Apr 4, 2022
35fad2d
Fix build break caused by server compatibility issue
zhicwu Apr 4, 2022
72afe97
Enhance data type handing and conversion between IPv4 and IPv6
zhicwu Apr 10, 2022
4b651f7
Fix test failure
zhicwu Apr 10, 2022
f782b77
return string by default for enum column
zhicwu Apr 15, 2022
5cd7e0b
Update dependency, doc and version
zhicwu Apr 16, 2022
c909574
Clean up code and enhance enum types
zhicwu Apr 16, 2022
8448caa
Update benchmarks
zhicwu Apr 16, 2022
4c5a101
Merge stream changes from v0.3.3 branch along with TLS and exception …
zhicwu Apr 16, 2022
822673d
Upgrade gRPC client
zhicwu Apr 16, 2022
67ff035
Fix javadoc error on CI
zhicwu Apr 16, 2022
2d3779c
Add workaround for 21.3 / 21.8
zhicwu Apr 16, 2022
47af1ff
stop testing grpc on 21.8 or early version
zhicwu Apr 16, 2022
2d785c0
Stop testing TLS client authentication
zhicwu Apr 16, 2022
27dfc0c
Skip default value test for gRPC
zhicwu Apr 16, 2022
b1706dc
Fix test failures on 21.3 and 21.8
zhicwu Apr 16, 2022
ffd81d7
Enhance error handling and reduce temporary objects
zhicwu Apr 16, 2022
795dd30
Benchmark against 22.3
zhicwu Apr 16, 2022
14f856a
Skip gRPC tests unless ClickHouse is 22.3+
zhicwu Apr 16, 2022
0e2b532
Skip a few more tests on 21.3
zhicwu Apr 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ jobs:
run: |
mvn --batch-mode --update-snapshots -q -DskipTests install
cd clickhouse-benchmark
java -DclickhouseVersion="21.8" -jar target/benchmarks.jar -rf text \
-p client=clickhouse-http-jdbc1 -p client=clickhouse-grpc-jdbc -p type=object Basic
java -DclickhouseVersion="22.3" -jar target/benchmarks.jar -rf text \
-p client=clickhouse-http-jdbc -p client=clickhouse-grpc-jdbc -p type=object Basic
echo "BENCHMARK_REPORT<<EOF" >> $GITHUB_ENV
cat jmh-result.text >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/timezone.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,6 @@ jobs:
mvn --batch-mode --update-snapshots \
--projects clickhouse-jdbc \
-DclickhouseTimezone=${{ matrix.serverTz }} \
-DclickhouseVersion=21.8 \
-DclickhouseVersion=22.3 \
-Duser.timezone=${{ matrix.clientTz }} \
--also-make clean verify
64 changes: 34 additions & 30 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,29 +21,32 @@ Note: in general, the new driver(v0.3.2) is a few times faster with less memory

## Features

| Category | Feature | Supported | Remark |
| ------------- | ------------------------------------------------------------ | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Protocol | [HTTP](https://clickhouse.com/docs/en/interfaces/http/) | :white_check_mark: | recommended, defaults to `java.net.HttpURLConnection` and can be changed to `java.net.http.HttpClient`(faster but less stable) |
| | [gRPC](https://clickhouse.com/docs/en/interfaces/grpc/) | :white_check_mark: | experimental, still missing some features like LZ4 compression |
| | [TCP/Native](https://clickhouse.com/docs/en/interfaces/tcp/) | :x: | will be available in 0.3.3 |
| Compatibility | Server < 20.7 | :x: | use 0.3.1-patch(or 0.2.6 if you're stuck with JDK 7) |
| | Server >= 20.7 | :white_check_mark: | use 0.3.2 or above. All [active releases](https://github.com/ClickHouse/ClickHouse/pulls?q=is%3Aopen+is%3Apr+label%3Arelease) are supported. |
| Data Type | AggregatedFunction | :x: | limited to `groupBitmap` |
| | Array(\*) | :white_check_mark: | |
| | Date\* | :white_check_mark: | |
| | DateTime\* | :white_check_mark: | |
| | Decimal\* | :white_check_mark: | `SET output_format_decimal_trailing_zeros=1` in 21.9+ for consistency |
| | Enum\* | :white_check_mark: | can be treated as both string and integer |
| | Geo Types | :white_check_mark: | |
| | Int\*, UInt\* | :white_check_mark: | UInt64 is mapped to `long` |
| | IPv\* | :white_check_mark: | |
| | \*String | :white_check_mark: | |
| | Map(\*) | :white_check_mark: | |
| | Nested(\*) | :white_check_mark: | |
| | Tuple(\*) | :white_check_mark: | |
| | UUID | :white_check_mark: | |
| Format | RowBinary | :white_check_mark: | `RowBinaryWithNamesAndTypes` for query and `RowBinary` for insertion |
| | TabSeparated | :white_check_mark: | Does not support as many data types as RowBinary |
| Category | Feature | Supported | Remark |
| ------------- | ------------------------------------------------------------ | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| Protocol | [HTTP](https://clickhouse.com/docs/en/interfaces/http/) | :white_check_mark: | recommended, defaults to `java.net.HttpURLConnection` and can be changed to `java.net.http.HttpClient`(faster but less stable) |
| | [gRPC](https://clickhouse.com/docs/en/interfaces/grpc/) | :white_check_mark: | still experimental, known to has [issue](https://github.com/ClickHouse/ClickHouse/issues/28671#issuecomment-1087049993) when using LZ4 compression |
| | [TCP/Native](https://clickhouse.com/docs/en/interfaces/tcp/) | :x: | will be available in 0.3.3 |
| Compatibility | Server < 20.7 | :x: | use 0.3.1-patch(or 0.2.6 if you're stuck with JDK 7) |
| | Server >= 20.7 | :white_check_mark: | use 0.3.2 or above. All [active releases](https://github.com/ClickHouse/ClickHouse/pulls?q=is%3Aopen+is%3Apr+label%3Arelease) are supported. |
| Data Type | AggregatedFunction | :x: | limited to `groupBitmap` |
| | Array(\*) | :white_check_mark: | |
| | Bool | :white_check_mark: | |
| | Date\* | :white_check_mark: | |
| | DateTime\* | :white_check_mark: | |
| | Decimal\* | :white_check_mark: | `SET output_format_decimal_trailing_zeros=1` in 21.9+ for consistency |
| | Enum\* | :white_check_mark: | can be treated as both string and integer |
| | Geo Types | :white_check_mark: | Point, Ring, Polygon, and MultiPolygon |
| | Int\*, UInt\* | :white_check_mark: | UInt64 is mapped to `long` |
| | IPv\* | :white_check_mark: | |
| | Map(\*) | :white_check_mark: | |
| | Nested(\*) | :white_check_mark: | |
| | Object('JSON') | :white_check_mark: | |
| | SimpleAggregateFunction | :white_check_mark: | |
| | \*String | :white_check_mark: | |
| | Tuple(\*) | :white_check_mark: | |
| | UUID | :white_check_mark: | |
| Format | RowBinary | :white_check_mark: | `RowBinaryWithNamesAndTypes` for query and `RowBinary` for insertion |
| | TabSeparated | :white_check_mark: | Does not support as many data types as RowBinary |

## Configuration

Expand Down Expand Up @@ -112,7 +115,7 @@ Note: in general, the new driver(v0.3.2) is a few times faster with less memory
<groupId>com.clickhouse</groupId>
<!-- or clickhouse-grpc-client if you prefer gRPC -->
<artifactId>clickhouse-http-client</artifactId>
<version>0.3.3</version>
<version>0.3.2-patch8</version>
</dependency>
```

Expand All @@ -129,11 +132,11 @@ try (ClickHouseClient client = ClickHouseClient.newInstance(preferredProtocol);
ClickHouseResponse response = client.connect(server)
.format(preferredFormat)
.query("select * from numbers(:limit)")
.params(1000).execute().get()) {
.params(1000).executeAndWait()) {
// or resp.stream() if you prefer stream API
for (ClickHouseRecord record : response.records()) {
int num = record.getValue(0).asInteger();
String str = record.getValue(0).asString();
for (ClickHouseRecord r : response.records()) {
int num = r.getValue(0).asInteger();
String str = r.getValue(0).asString();
}

ClickHouseResponseSummary summary = response.getSummary();
Expand All @@ -148,7 +151,7 @@ try (ClickHouseClient client = ClickHouseClient.newInstance(preferredProtocol);
<!-- will stop using ru.yandex.clickhouse starting from 0.4.0 -->
<groupId>com.clickhouse</groupId>
<artifactId>clickhouse-jdbc</artifactId>
<version>0.3.3</version>
<version>0.3.2-patch8</version>
<!-- below is only needed when all you want is a shaded jar -->
<classifier>http</classifier>
<exclusions>
Expand Down Expand Up @@ -209,7 +212,7 @@ cd clickhouse-benchmark
mvn -Drelease clean package
# single thread mode
java -DdbHost=localhost -jar target/benchmarks.jar -t 1 \
-p client=clickhouse-http-jdbc1 -p connection=reuse \
-p client=clickhouse-jdbc -p connection=reuse \
-p statement=prepared Query.selectInt8
```

Expand All @@ -225,6 +228,7 @@ In the case you prefer to test against an existing server, please follow instruc
- add below two configuration files to the existing server and expose all ports for external access
- [ports.xml](../../blob/master/clickhouse-client/src/test/resources/containers/clickhouse-server/config.d/ports.xml) - enable all ports
- and [users.xml](../../blob/master/clickhouse-client/src/test/resources/containers/clickhouse-server/users.d/users.xml) - accounts used for integration test
Note: you may need to change root element from `clickhouse` to `yandex` when testing old version of ClickHouse.
- put `test.properties` under either `~/.clickhouse` or `src/test/resources` of your project, with content like below:
```properties
clickhouseServer=x.x.x.x
Expand Down
2 changes: 1 addition & 1 deletion clickhouse-benchmark/src/main/java/com/clickhouse/benchmark/BaseState.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ public BaseState() {
}

protected int getRandomNumber(int bound) {
return random.nextInt(bound);
return bound < 1 ? 0 : random.nextInt(bound);
}

protected void consume(Blackhole blackhole, Callable<?> task) throws InterruptedException {
Expand Down
6 changes: 3 additions & 3 deletions clickhouse-benchmark/src/main/java/com/clickhouse/benchmark/ServerState.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,15 +73,15 @@ public void doSetup() throws Exception {
imageTag = ":" + imageTag;
}

final String imageNameWithTag = "yandex/clickhouse-server" + imageTag;
final String imageNameWithTag = "clickhouse/clickhouse-server" + imageTag;

container = new GenericContainer<>(new ImageFromDockerfile().withDockerfileFromBuilder(builder -> builder
.from(imageNameWithTag)
.run("echo '<yandex><listen_host>0.0.0.0</listen_host><http_port>8123</http_port>"
.run("echo '<clickhouse><listen_host>0.0.0.0</listen_host><http_port>8123</http_port>"
+ "<tcp_port>9000</tcp_port><mysql_port>9004</mysql_port>"
+ "<postgresql_port>9005</postgresql_port>"
+ "<interserver_http_port>9009</interserver_http_port>"
+ "<grpc_port>9100</grpc_port></yandex>' > /etc/clickhouse-server/config.d/custom.xml")))
+ "<grpc_port>9100</grpc_port></clickhouse>' > /etc/clickhouse-server/config.d/custom.xml")))
.withExposedPorts(Constants.GRPC_PORT, Constants.HTTP_PORT, Constants.MYSQL_PORT,

Constants.NATIVE_PORT)
Expand Down
2 changes: 1 addition & 1 deletion clickhouse-benchmark/src/main/java/com/clickhouse/benchmark/client/ClientState.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ private ClickHouseClient createClient() {

ClickHouseClientBuilder builder = ClickHouseClient.builder();
if (bufferSize != null && !bufferSize.isEmpty()) {
builder.option(ClickHouseClientOption.MAX_BUFFER_SIZE, Integer.parseInt(bufferSize));
builder.option(ClickHouseClientOption.BUFFER_SIZE, Integer.parseInt(bufferSize));
}
if (compression != null && !compression.isEmpty()) {
// builder.option(ClickHouseClientOption.COMPRESSION,
Expand Down
9 changes: 6 additions & 3 deletions clickhouse-benchmark/src/main/java/com/clickhouse/benchmark/jdbc/DriverState.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import java.util.UUID;

import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Param;
Expand All @@ -20,7 +19,7 @@

@State(Scope.Thread)
public class DriverState extends BaseState {
@Param(value = { "clickhouse4j", "clickhouse-http-jdbc", "clickhouse-grpc-jdbc", "clickhouse-jdbc",
@Param(value = { "clickhouse-jdbc", "clickhouse-grpc-jdbc", "clickhouse-legacy-jdbc", "clickhouse4j",
"clickhouse-native-jdbc", "mariadb-java-client", "mysql-connector-java", "postgresql-jdbc" })
private String client;

Expand Down Expand Up @@ -54,7 +53,11 @@ public void doSetup(ServerState serverState) throws Exception {
serverState.getPassword(), compression, additional);
// ClickHouseDefines.WRITE_COMPRESS = false;
// ClickHouseDefines.READ_DECOMPRESS = Boolean.parseBoolean(compression);
conn = driver.connect(url, new Properties());
Properties props = new Properties();
if (jdbcDriver.getClassName().startsWith("com.clickhouse.jdbc.")) {
props.setProperty("format", System.getProperty("format", "RowBinaryWithNamesAndTypes"));
}
conn = driver.connect(url, props);

try (Statement s = conn.createStatement()) {
// s.execute("drop table if exists system.test_insert");
Expand Down
Loading