Skip to content

Commit 79a134a

Browse files
zy-kkkzy-kkk
authored
[Feature][Connector-V2][SelectDB Cloud] Support SelectDB Cloud Sink Connector (#3958)
1 parent ea16042 commit 79a134a

40 files changed

+2945
-0
lines changed

‎docs/en/connector-v2/Error-Quick-Reference-Manual.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,19 @@ problems encountered by users.
160160
|----------|-------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
161161
| Doris-01 | Writing records to Doris failed. | When users encounter this error code, it means that writing records to Doris failed, please check data from files whether is correct |
162162

163+
## SelectDB Cloud Connector Error Codes
164+
165+
| code | description | solution |
166+
|-------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
167+
| SelectDB-01 | upload file to stage failed | When users encounter this error code, it means that upload file to SelectDB Cloud failed, please check the configuration and network. |
168+
| SelectDB-01 | commit copy into sql failed | When users encounter this error code, it means that commit copy into sql to SelectDB Cloud failed, please check the configuration. |
169+
| SelectDB-03 | Closing httpClient failed | When users encounter this error code, it means that closing the http connection failed. please check the network. |
170+
| SelectDB-04 | Get the redirected s3 address filed | When users encounter this error code, it means that get the redirected s3 address failed, please check the network. |
171+
| SelectDB-05 | error while loading data | When users encounter this error code, it means that the file write check failed. please check the configuration. |
172+
| SelectDB-07 | buffer stop failed | When users encounter this error code, it means that the buffer stop failed. Check the detailed exception information. |
173+
| SelectDB-08 | buffer read failed | When users encounter this error code, it means that the buffer read failed. Check the detailed exception information. |
174+
| SelectDB-09 | buffer write failed | When users encounter this error code, it means that the buffer write failed. Check the detailed exception information. |
175+
163176
## Clickhouse Connector Error Codes
164177

165178
| code | description | solution |

‎docs/en/connector-v2/sink/SelectDB-Cloud.md

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# SelectDB Cloud
2+
3+
> SelectDB Cloud sink connector
4+
5+
## Description
6+
Used to send data to SelectDB Cloud. Both support streaming and batch mode.
7+
The internal implementation of SelectDB Cloud sink connector upload after batch caching and commit the CopyInto sql to load data into the table.
8+
## Key features
9+
10+
- [x] [exactly-once](../../concept/connector-v2-features.md)
11+
12+
By default, we use 2PC commit to ensure `exactly-once`
13+
14+
## Options
15+
16+
| name | type | required | default value |
17+
|---------------------|--------|----------|-----------------|
18+
| load-url | string | yes | - |
19+
| jdbc-url | string | yes | - |
20+
| cluster-name | string | yes | - |
21+
| username | string | yes | - |
22+
| password | string | yes | - |
23+
| table.identifier | string | yes | - |
24+
| selectdb.config | map | yes | - |
25+
| sink.buffer-size | int | no | 1024*1024 (1MB) |
26+
| sink.buffer-count | int | no | 3 |
27+
| sink.max-retries | int | no | 1 |
28+
| sink.check-interval | int | no | 10000 |
29+
30+
### load-url [string]
31+
32+
`SelectDB Cloud` warehouse http address, the format is `warehouse_ip:http_port`
33+
34+
### jdbc-url [string]
35+
36+
`SelectDB Cloud` warehouse jdbc address, the format is `warehouse_ip:mysql_port`
37+
38+
### cluster-name [string]
39+
40+
`SelectDB Cloud` cluster name
41+
42+
### username [string]
43+
44+
`SelectDB Cloud` user username
45+
46+
### password [string]
47+
48+
`SelectDB Cloud` user password
49+
50+
### table.identifier [string]
51+
52+
The name of `SelectDB Cloud` table, the format is `database.table`
53+
54+
### sink.properties [string]
55+
56+
Write property configuration
57+
CSV Write：
58+
selectdb.config {
59+
file.type='csv'
60+
file.column_separator=','
61+
file.line_delimiter='\n'
62+
}
63+
JSON Write:
64+
selectdb.config {
65+
file.type="json"
66+
file.strip_outer_array="false"
67+
}
68+
69+
### sink.buffer-size [string]
70+
71+
Write data cache buffer size, unit byte. The default is 1 MB, and it is not recommended to modify it.
72+
73+
### sink.buffer-count [string]
74+
75+
The number of write data cache buffers, the default is 3, it is not recommended to modify.
76+
77+
### sink.max-retries [string]
78+
79+
The maximum number of retries in the Commit phase, the default is 1.
80+
81+
### sink.check-interval [string]
82+
83+
Periodic interval for writing files, in milliseconds, default 10 seconds.
84+
85+
## Example
86+
87+
Use JSON format to import data
88+
89+
```
90+
sink {
91+
SelectDBCloud {
92+
load-url="warehouse_ip:http_port"
93+
jdbc-url="warehouse_ip:mysql_port"
94+
cluster-name="Cluster"
95+
table.identifier="test.test"
96+
username="admin"
97+
password="******"
98+
selectdb.config {
99+
file.type="json"
100+
file.strip_outer_array="false"
101+
}
102+
}
103+
}
104+
```
105+
106+
Use CSV format to import data
107+
108+
```
109+
sink {
110+
SelectDBCloud {
111+
load-url="warehouse_ip:http_port"
112+
jdbc-url="warehouse_ip:mysql_port"
113+
cluster-name="Cluster"
114+
table.identifier="test.test"
115+
username="admin"
116+
password="******"
117+
selectdb.config {
118+
file.type='csv'
119+
file.column_separator=','
120+
file.line_delimiter='\n'
121+
}
122+
}
123+
}
124+
```
125+
126+
## Changelog
127+
128+
### next version
129+
130+
- [Feature] Support SelectDB Cloud Sink Connector [3958](https://github.com/apache/incubator-seatunnel/pull/3958)
131+

‎plugin-mapping.properties

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,4 +169,5 @@ seatunnel.sink.S3Redshift = connector-s3-redshift
169169
seatunnel.source.TDengine = connector-tdengine
170170
seatunnel.sink.TDengine = connector-tdengine
171171
seatunnel.source.Persistiq = connector-http-persistiq
172+
seatunnel.sink.SelectDBCloud = connector-selectdb-cloud
172173

‎seatunnel-connectors-v2/connector-selectdb-cloud/pom.xml

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
4+
Licensed to the Apache Software Foundation (ASF) under one or more
5+
contributor license agreements. See the NOTICE file distributed with
6+
this work for additional information regarding copyright ownership.
7+
The ASF licenses this file to You under the Apache License, Version 2.0
8+
(the "License"); you may not use this file except in compliance with
9+
the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing, software
14+
distributed under the License is distributed on an "AS IS" BASIS,
15+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
See the License for the specific language governing permissions and
17+
limitations under the License.
18+
19+
-->
20+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
21+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
22+
<parent>
23+
<artifactId>seatunnel-connectors-v2</artifactId>
24+
<groupId>org.apache.seatunnel</groupId>
25+
<version>${revision}</version>
26+
</parent>
27+
<modelVersion>4.0.0</modelVersion>
28+
29+
<artifactId>connector-selectdb-cloud</artifactId>
30+
31+
<properties>
32+
<httpclient.version>4.5.13</httpclient.version>
33+
<httpcore.version>4.4.4</httpcore.version>
34+
</properties>
35+
36+
<dependencies>
37+
<dependency>
38+
<groupId>org.apache.seatunnel</groupId>
39+
<artifactId>seatunnel-api</artifactId>
40+
<version>${project.version}</version>
41+
</dependency>
42+
<dependency>
43+
<groupId>org.apache.seatunnel</groupId>
44+
<artifactId>connector-common</artifactId>
45+
<version>${project.version}</version>
46+
</dependency>
47+
<dependency>
48+
<groupId>org.apache.httpcomponents</groupId>
49+
<artifactId>httpclient</artifactId>
50+
<version>${httpclient.version}</version>
51+
</dependency>
52+
<dependency>
53+
<groupId>org.apache.httpcomponents</groupId>
54+
<artifactId>httpcore</artifactId>
55+
<version>${httpcore.version}</version>
56+
</dependency>
57+
<dependency>
58+
<groupId>org.apache.seatunnel</groupId>
59+
<artifactId>seatunnel-format-json</artifactId>
60+
<version>${project.version}</version>
61+
</dependency>
62+
<dependency>
63+
<groupId>org.apache.seatunnel</groupId>
64+
<artifactId>seatunnel-format-text</artifactId>
65+
<version>${project.version}</version>
66+
</dependency>
67+
</dependencies>
68+
</project>

0 commit comments

Comments
 (0)