Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve][Connector-V2][SelectDB Cloud]Refactor some SelectDB Cloud Sink code as well as support copy into batch and async flush and cdc #4312

Merged
merged 5 commits into from Mar 9, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
71 changes: 40 additions & 31 deletions docs/en/connector-v2/sink/SelectDB-Cloud.md
Expand Up @@ -7,27 +7,33 @@
Used to send data to SelectDB Cloud. Both support streaming and batch mode.
The internal implementation of SelectDB Cloud sink connector upload after batch caching and commit the CopyInto sql to load data into the table.

:::tip

Version Supported

* supported `SelectDB Cloud version is >= 2.2.x`

:::

## Key features

- [x] [exactly-once](../../concept/connector-v2-features.md)

By default, we use 2PC commit to ensure `exactly-once`
- [x] [cdc](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|---------------------|--------|----------|-----------------|
| load-url | string | yes | - |
| jdbc-url | string | yes | - |
| cluster-name | string | yes | - |
| username | string | yes | - |
| password | string | yes | - |
| table.identifier | string | yes | - |
| selectdb.config | map | yes | - |
| sink.buffer-size | int | no | 1024*1024 (1MB) |
| sink.buffer-count | int | no | 3 |
| sink.max-retries | int | no | 1 |
| sink.check-interval | int | no | 10000 |
| name | type | required | default value |
|-------------------|--------|----------|------------------------|
| load-url | string | yes | - |
| jdbc-url | string | yes | - |
| cluster-name | string | yes | - |
| username | string | yes | - |
| password | string | yes | - |
| table.identifier | string | yes | - |
| selectdb.config | map | yes | - |
| sink.buffer-size | int | no | 10 * 1024 * 1024 (1MB) |
| sink.buffer-count | int | no | 10000 |
| sink.max-retries | int | no | 3 |

### load-url [string]

Expand Down Expand Up @@ -56,33 +62,36 @@ The name of `SelectDB Cloud` table, the format is `database.table`
### sink.properties [string]

Write property configuration

CSV Write:

```
selectdb.config {
file.type='csv'
file.column_separator=','
file.line_delimiter='\n'
file.type="csv"
file.column_separator=","
file.line_delimiter="\n"
}
```

JSON Write:

```
selectdb.config {
file.type="json"
file.strip_outer_array="false"
file.type="json"
}
```

### sink.buffer-size [string]

Write data cache buffer size, unit byte. The default is 1 MB, and it is not recommended to modify it.
The maximum capacity of the cache, in bytes, that is flushed to the object storage. The default is 10MB. it is not recommended to modify it.

### sink.buffer-count [string]

The number of write data cache buffers, the default is 3, it is not recommended to modify.
Maximum number of entries flushed to the object store. The default value is 10000. it is not recommended to modify.

### sink.max-retries [string]

The maximum number of retries in the Commit phase, the default is 1.

### sink.check-interval [string]

Periodic interval for writing files, in milliseconds, default 10 seconds.
The maximum number of retries in the Commit phase, the default is 3.

## Example

Expand All @@ -99,7 +108,6 @@ sink {
password="******"
selectdb.config {
file.type="json"
file.strip_outer_array="false"
}
}
}
Expand All @@ -117,9 +125,9 @@ sink {
username="admin"
password="******"
selectdb.config {
file.type='csv'
file.column_separator=','
file.line_delimiter='\n'
file.type="csv"
file.column_separator=","
file.line_delimiter="\n"
}
}
}
Expand All @@ -130,4 +138,5 @@ sink {
### next version

- [Feature] Support SelectDB Cloud Sink Connector [3958](https://github.com/apache/incubator-seatunnel/pull/3958)
- [Improve] Refactor some SelectDB Cloud Sink code as well as support copy into batch and async flush and cdc [4312](https://github.com/apache/incubator-seatunnel/pull/4312)

2 changes: 2 additions & 0 deletions release-note.md
Expand Up @@ -40,11 +40,13 @@
- [API]Add parallelism and column projection interface #3829
- [API]Add get source method to all source connector #3846
- [Hive] Support read user-defined partitions #3842
- [SelectDB Cloud] Support SelectDB Cloud Sink Connector #3958
- [Hive] Support read text table & Column projection #4105
- [File] Support column projection #4105
- [Github] Add github source connector #4155
- [Jdbc] Add database field to sink config #4199
- [Doris] Refactor some Doris Sink code as well as support 2pc and cdc #4235
- [SelectDB Cloud] Refactor some SelectDB Cloud Sink code as well as support copy into batch and async flush and cdc #4312
### Zeta Engine
- [Chore] Remove unnecessary dependencies #3795
- [Core] Improve job restart of all node down #3784
Expand Down

This file was deleted.

This file was deleted.

This file was deleted.