Skip to content

Commit c1bbbd5

Browse files
hailin0wanghailin
hailin0
and
wanghailin
authored
[Feature][Connector-V2] Support IoTDB sink (#2407)
* [Feature][Connector-V2] Support IoTDB sink * update * update docs Co-authored-by: wanghailin <hailin@fiture.com>
1 parent b838b75 commit c1bbbd5

File tree

20 files changed

+1332
-1
lines changed

20 files changed

+1332
-1
lines changed

‎docs/en/connector-v2/sink/IoTDB.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# IoTDB
2+
3+
> IoTDB sink connector
4+
5+
## Description
6+
7+
Used to write data to IoTDB. Supports Batch and Streaming mode.
8+
9+
:::tip
10+
11+
There is a conflict of thrift version between IoTDB and Spark.Therefore, you need to execute `rm -f $SPARK_HOME/jars/libthrift*` and `cp $IOTDB_HOME/lib/libthrift* $SPARK_HOME/jars/` to resolve it.
12+
13+
:::
14+
15+
## Options
16+
17+
| name | type | required | default value |
18+
|-------------------------------|-------------------|----------|---------------|
19+
| node_urls | list | yes | - |
20+
| username | string | yes | - |
21+
| password | string | yes | - |
22+
| batch_size | int | no | 1024 |
23+
| batch_interval_ms | int | no | - |
24+
| max_retries | int | no | - |
25+
| retry_backoff_multiplier_ms | int | no | - |
26+
| max_retry_backoff_ms | int | no | - |
27+
| default_thrift_buffer_size | int | no | - |
28+
| max_thrift_frame_size | int | no | - |
29+
| zone_id | string | no | - |
30+
| enable_rpc_compression | boolean | no | - |
31+
| connection_timeout_in_ms | int | no | - |
32+
| timeseries_options | list | no | - |
33+
| timeseries_options.path | string | no | - |
34+
| timeseries_options.data_type | string | no | - |
35+
| common-options | string | no | - |
36+
37+
### node_urls [list]
38+
39+
`IoTDB` cluster address, the format is `["host:port", ...]`
40+
41+
### username [string]
42+
43+
`IoTDB` user username
44+
45+
### password [string]
46+
47+
`IoTDB` user password
48+
49+
### batch_size [int]
50+
51+
For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the IoTDB
52+
53+
### batch_interval_ms [int]
54+
55+
For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the IoTDB
56+
57+
### max_retries [int]
58+
59+
The number of retries to flush failed
60+
61+
### retry_backoff_multiplier_ms [int]
62+
63+
Using as a multiplier for generating the next delay for backoff
64+
65+
### max_retry_backoff_ms [int]
66+
67+
The amount of time to wait before attempting to retry a request to `IoTDB`
68+
69+
### default_thrift_buffer_size [int]
70+
71+
Thrift init buffer size in `IoTDB` client
72+
73+
### max_thrift_frame_size [int]
74+
75+
Thrift max frame size in `IoTDB` client
76+
77+
### zone_id [string]
78+
79+
java.time.ZoneId in `IoTDB` client
80+
81+
### enable_rpc_compression [boolean]
82+
83+
Enable rpc compression in `IoTDB` client
84+
85+
### connection_timeout_in_ms [int]
86+
87+
The maximum time (in ms) to wait when connect `IoTDB`
88+
89+
### timeseries_options [list]
90+
91+
Timeseries options
92+
93+
### timeseries_options.path [string]
94+
95+
Timeseries path
96+
97+
### timeseries_options.data_type [string]
98+
99+
Timeseries data type
100+
101+
### common options [string]
102+
103+
Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details
104+
105+
## Examples
106+
107+
```hocon
108+
sink {
109+
IoTDB {
110+
node_urls = ["localhost:6667"]
111+
username = "root"
112+
password = "root"
113+
batch_size = 1024
114+
batch_interval_ms = 1000
115+
}
116+
}
117+
```

‎plugin-mapping.properties‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,3 +115,4 @@ seatunnel.source.Pulsar = connector-pulsar
115115
seatunnel.source.Hudi = connector-hudi
116116
seatunnel.sink.DingTalk = connector-dingtalk
117117
seatunnel.sink.elasticsearch = connector-elasticsearch
118+
seatunnel.sink.IoTDB = connector-iotdb

‎pom.xml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -225,6 +225,8 @@
225225
<javax.annotation-api.version>1.3.2</javax.annotation-api.version>
226226
<elasticsearch-rest-client.version>7.5.1</elasticsearch-rest-client.version>
227227
<checker.qual.version>3.10.0</checker.qual.version>
228+
<iotdb.version>0.13.1</iotdb.version>
229+
<awaitility.version>4.2.0</awaitility.version>
228230
</properties>
229231

230232
<dependencyManagement>
@@ -940,6 +942,25 @@
940942
<artifactId>checker-qual</artifactId>
941943
<version>${checker.qual.version}</version>
942944
</dependency>
945+
946+
<dependency>
947+
<groupId>org.apache.iotdb</groupId>
948+
<artifactId>iotdb-session</artifactId>
949+
<version>${iotdb.version}</version>
950+
<exclusions>
951+
<exclusion>
952+
<groupId>ch.qos.logback</groupId>
953+
<artifactId>logback-classic</artifactId>
954+
</exclusion>
955+
</exclusions>
956+
</dependency>
957+
958+
<dependency>
959+
<groupId>org.awaitility</groupId>
960+
<artifactId>awaitility</artifactId>
961+
<version>${awaitility.version}</version>
962+
<scope>test</scope>
963+
</dependency>
943964
</dependencies>
944965
</dependencyManagement>
945966

‎seatunnel-connectors-v2-dist/pom.xml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,11 +121,16 @@
121121
<artifactId>connector-email</artifactId>
122122
<version>${project.version}</version>
123123
</dependency>
124-
<dependency>
124+
<dependency>
125125
<groupId>org.apache.seatunnel</groupId>
126126
<artifactId>connector-elasticsearch</artifactId>
127127
<version>${project.version}</version>
128128
</dependency>
129+
<dependency>
130+
<groupId>org.apache.seatunnel</groupId>
131+
<artifactId>connector-iotdb</artifactId>
132+
<version>${project.version}</version>
133+
</dependency>
129134
</dependencies>
130135

131136
<build>

‎seatunnel-connectors-v2/connector-iotdb/pom.xml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
4+
Licensed to the Apache Software Foundation (ASF) under one or more
5+
contributor license agreements. See the NOTICE file distributed with
6+
this work for additional information regarding copyright ownership.
7+
The ASF licenses this file to You under the Apache License, Version 2.0
8+
(the "License"); you may not use this file except in compliance with
9+
the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing, software
14+
distributed under the License is distributed on an "AS IS" BASIS,
15+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
See the License for the specific language governing permissions and
17+
limitations under the License.
18+
19+
-->
20+
<project xmlns="http://maven.apache.org/POM/4.0.0"
21+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
22+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
23+
<parent>
24+
<artifactId>seatunnel-connectors-v2</artifactId>
25+
<groupId>org.apache.seatunnel</groupId>
26+
<version>${revision}</version>
27+
</parent>
28+
<modelVersion>4.0.0</modelVersion>
29+
30+
<artifactId>connector-iotdb</artifactId>
31+
32+
<dependencies>
33+
<dependency>
34+
<groupId>org.apache.seatunnel</groupId>
35+
<artifactId>seatunnel-api</artifactId>
36+
<version>${project.version}</version>
37+
</dependency>
38+
<dependency>
39+
<groupId>org.apache.seatunnel</groupId>
40+
<artifactId>connector-common</artifactId>
41+
<version>${project.version}</version>
42+
</dependency>
43+
44+
<dependency>
45+
<groupId>org.apache.iotdb</groupId>
46+
<artifactId>iotdb-session</artifactId>
47+
</dependency>
48+
</dependencies>
49+
50+
</project>

‎seatunnel-connectors-v2/connector-iotdb/src/main/java/org/apache/seatunnel/connectors/seatunnel/iotdb/config/CommonConfig.java‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.seatunnel.connectors.seatunnel.iotdb.config;
19+
20+
import lombok.AllArgsConstructor;
21+
import lombok.Getter;
22+
import lombok.ToString;
23+
24+
import java.util.List;
25+
26+
@Getter
27+
@ToString
28+
@AllArgsConstructor
29+
public class CommonConfig {
30+
31+
public static final String NODE_URLS = "node_urls";
32+
public static final String USERNAME = "username";
33+
public static final String PASSWORD = "password";
34+
35+
private final List<String> nodeUrls;
36+
private final String username;
37+
private final String password;
38+
}

0 commit comments

Comments
 (0)