Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Connector-V2][SelectDB Cloud] Support SelectDB Cloud Sink Connector #3958

Merged
merged 12 commits into from Jan 31, 2023
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/en/connector-v2/Error-Quick-Reference-Manual.md
Expand Up @@ -160,6 +160,19 @@ problems encountered by users.
|----------|-------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
| Doris-01 | Writing records to Doris failed. | When users encounter this error code, it means that writing records to Doris failed, please check data from files whether is correct |

## SelectDB Cloud Connector Error Codes

| code | description | solution |
|-------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
| SelectDB-01 | upload file to stage failed | When users encounter this error code, it means that upload file to SelectDB Cloud failed, please check the configuration and network. |
| SelectDB-01 | commit copy into sql failed | When users encounter this error code, it means that commit copy into sql to SelectDB Cloud failed, please check the configuration. |
| SelectDB-03 | Closing httpClient failed | When users encounter this error code, it means that closing the http connection failed. please check the network. |
| SelectDB-04 | Get the redirected s3 address filed | When users encounter this error code, it means that get the redirected s3 address failed, please check the network. |
| SelectDB-05 | error while loading data | When users encounter this error code, it means that the file write check failed. please check the configuration. |
| SelectDB-07 | buffer stop failed | When users encounter this error code, it means that the buffer stop failed. Check the detailed exception information. |
| SelectDB-08 | buffer read failed | When users encounter this error code, it means that the buffer read failed. Check the detailed exception information. |
| SelectDB-09 | buffer write failed | When users encounter this error code, it means that the buffer write failed. Check the detailed exception information. |

## Clickhouse Connector Error Codes

| code | description | solution |
Expand Down
131 changes: 131 additions & 0 deletions docs/en/connector-v2/sink/SelectDB-Cloud.md
@@ -0,0 +1,131 @@
# SelectDB Cloud

> SelectDB Cloud sink connector

## Description
Used to send data to SelectDB Cloud. Both support streaming and batch mode.
The internal implementation of SelectDB Cloud sink connector upload after batch caching and commit the CopyInto sql to load data into the table.
## Key features

- [x] [exactly-once](../../concept/connector-v2-features.md)

By default, we use 2PC commit to ensure `exactly-once`

## Options

| name | type | required | default value |
|---------------------|--------|----------|-----------------|
| load-url | string | yes | - |
| jdbc-url | string | yes | - |
| cluster-name | string | yes | - |
| username | string | yes | - |
| password | string | yes | - |
| table.identifier | string | yes | - |
| selectdb.config | map | yes | - |
| sink.buffer-size | int | no | 1024*1024 (1MB) |
| sink.buffer-count | int | no | 3 |
| sink.max-retries | int | no | 1 |
| sink.check-interval | int | no | 10000 |

### load-url [string]

`SelectDB Cloud` warehouse http address, the format is `warehouse_ip:http_port`

### jdbc-url [string]

`SelectDB Cloud` warehouse jdbc address, the format is `warehouse_ip:mysql_port`

### cluster-name [string]

`SelectDB Cloud` cluster name

### username [string]

`SelectDB Cloud` user username

### password [string]

`SelectDB Cloud` user password

### table.identifier [string]

The name of `SelectDB Cloud` table, the format is `database.table`

### sink.properties [string]

Write property configuration
CSV Write:
selectdb.config {
sink.properties.file.type='csv'
sink.properties.file.column_separator=','
sink.properties.file.line_delimiter='\n'
}
JSON Write:
selectdb.config {
file.type="json"
file.strip_outer_array="false"
}

### sink.buffer-size [string]

Write data cache buffer size, unit byte. The default is 1 MB, and it is not recommended to modify it.

### sink.buffer-count [string]

The number of write data cache buffers, the default is 3, it is not recommended to modify.

### sink.max-retries [string]

The maximum number of retries in the Commit phase, the default is 1.

### sink.check-interval [string]

Periodic interval for writing files, in milliseconds, default 10 seconds.

## Example

Use JSON format to import data

```
sink {
SelectDBCloud {
load-url="warehouse_ip:http_port"
jdbc-url="warehouse_ip:mysql_port"
cluster-name="Cluster"
table.identifier="test.test"
username="admin"
password="******"
selectdb.config {
file.type="json"
file.strip_outer_array="false"
}
}
}
```

Use CSV format to import data

```
sink {
SelectDBCloud {
load-url="warehouse_ip:http_port"
jdbc-url="warehouse_ip:mysql_port"
cluster-name="Cluster"
table.identifier="test.test"
username="admin"
password="******"
selectdb.config {
file.type='csv'
file.column_separator=','
file.line_delimiter='\n'
}
}
}
```

## Changelog

### next version

- [Feature] Support SelectDB Cloud Sink Connector [3958](https://github.com/apache/incubator-seatunnel/pull/3958)

1 change: 1 addition & 0 deletions plugin-mapping.properties
Expand Up @@ -169,4 +169,5 @@ seatunnel.sink.S3Redshift = connector-s3-redshift
seatunnel.source.TDengine = connector-tdengine
seatunnel.sink.TDengine = connector-tdengine
seatunnel.source.Persistiq = connector-http-persistiq
seatunnel.sink.SelectDBCloud = connector-selectdb-cloud

68 changes: 68 additions & 0 deletions seatunnel-connectors-v2/connector-selectdb-cloud/pom.xml
@@ -0,0 +1,68 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--

Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<parent>
<artifactId>seatunnel-connectors-v2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>connector-selectdb-cloud</artifactId>

<properties>
<httpclient.version>4.5.13</httpclient.version>
<httpcore.version>4.4.4</httpcore.version>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-api</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>connector-common</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>${httpclient.version}</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>${httpcore.version}</version>
</dependency>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-format-json</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-format-text</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</project>