Skip to content

Commit

Permalink
[Improve][Connector-V2][SelectDB Cloud]Refactor some SelectDB Cloud S…
Browse files Browse the repository at this point in the history
…ink code as well as support copy into batch and async flush and cdc (#4312)

* [Improve][Connector-V2][SelectDB Cloud]Refactor some SelectDB Cloud Sink code as well as support copy into batch and async flush and cdc

* add doc

* add doc2

* style

* fix
  • Loading branch information
zy-kkk committed Mar 9, 2023
1 parent 2c86a99 commit 11e94b2
Show file tree
Hide file tree
Showing 37 changed files with 828 additions and 1,107 deletions.
14 changes: 4 additions & 10 deletions docs/en/connector-v2/Error-Quick-Reference-Manual.md
Expand Up @@ -169,16 +169,10 @@ problems encountered by users.

## SelectDB Cloud Connector Error Codes

| code | description | solution |
|-------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
| SelectDB-01 | upload file to stage failed | When users encounter this error code, it means that upload file to SelectDB Cloud failed, please check the configuration and network. |
| SelectDB-01 | commit copy into sql failed | When users encounter this error code, it means that commit copy into sql to SelectDB Cloud failed, please check the configuration. |
| SelectDB-03 | Closing httpClient failed | When users encounter this error code, it means that closing the http connection failed. please check the network. |
| SelectDB-04 | Get the redirected s3 address filed | When users encounter this error code, it means that get the redirected s3 address failed, please check the network. |
| SelectDB-05 | error while loading data | When users encounter this error code, it means that the file write check failed. please check the configuration. |
| SelectDB-07 | buffer stop failed | When users encounter this error code, it means that the buffer stop failed. Check the detailed exception information. |
| SelectDB-08 | buffer read failed | When users encounter this error code, it means that the buffer read failed. Check the detailed exception information. |
| SelectDB-09 | buffer write failed | When users encounter this error code, it means that the buffer write failed. Check the detailed exception information. |
| code | description | solution |
|-------------|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| SelectDB-01 | stage load file error | When users encounter this error code, it means that stage load file to SelectDB Cloud failed, please check the configuration and network. |
| SelectDB-02 | commit copy into sql failed | When users encounter this error code, it means that commit copy into sql to SelectDB Cloud failed, please check the configuration. |

## Clickhouse Connector Error Codes

Expand Down
71 changes: 40 additions & 31 deletions docs/en/connector-v2/sink/SelectDB-Cloud.md
Expand Up @@ -7,27 +7,33 @@
Used to send data to SelectDB Cloud. Both support streaming and batch mode.
The internal implementation of SelectDB Cloud sink connector upload after batch caching and commit the CopyInto sql to load data into the table.

:::tip

Version Supported

* supported `SelectDB Cloud version is >= 2.2.x`

:::

## Key features

- [x] [exactly-once](../../concept/connector-v2-features.md)

By default, we use 2PC commit to ensure `exactly-once`
- [x] [cdc](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|---------------------|--------|----------|-----------------|
| load-url | string | yes | - |
| jdbc-url | string | yes | - |
| cluster-name | string | yes | - |
| username | string | yes | - |
| password | string | yes | - |
| table.identifier | string | yes | - |
| selectdb.config | map | yes | - |
| sink.buffer-size | int | no | 1024*1024 (1MB) |
| sink.buffer-count | int | no | 3 |
| sink.max-retries | int | no | 1 |
| sink.check-interval | int | no | 10000 |
| name | type | required | default value |
|-------------------|--------|----------|------------------------|
| load-url | string | yes | - |
| jdbc-url | string | yes | - |
| cluster-name | string | yes | - |
| username | string | yes | - |
| password | string | yes | - |
| table.identifier | string | yes | - |
| selectdb.config | map | yes | - |
| sink.buffer-size | int | no | 10 * 1024 * 1024 (1MB) |
| sink.buffer-count | int | no | 10000 |
| sink.max-retries | int | no | 3 |

### load-url [string]

Expand Down Expand Up @@ -56,33 +62,36 @@ The name of `SelectDB Cloud` table, the format is `database.table`
### sink.properties [string]

Write property configuration

CSV Write:

```
selectdb.config {
file.type='csv'
file.column_separator=','
file.line_delimiter='\n'
file.type="csv"
file.column_separator=","
file.line_delimiter="\n"
}
```

JSON Write:

```
selectdb.config {
file.type="json"
file.strip_outer_array="false"
file.type="json"
}
```

### sink.buffer-size [string]

Write data cache buffer size, unit byte. The default is 1 MB, and it is not recommended to modify it.
The maximum capacity of the cache, in bytes, that is flushed to the object storage. The default is 10MB. it is not recommended to modify it.

### sink.buffer-count [string]

The number of write data cache buffers, the default is 3, it is not recommended to modify.
Maximum number of entries flushed to the object store. The default value is 10000. it is not recommended to modify.

### sink.max-retries [string]

The maximum number of retries in the Commit phase, the default is 1.

### sink.check-interval [string]

Periodic interval for writing files, in milliseconds, default 10 seconds.
The maximum number of retries in the Commit phase, the default is 3.

## Example

Expand All @@ -99,7 +108,6 @@ sink {
password="******"
selectdb.config {
file.type="json"
file.strip_outer_array="false"
}
}
}
Expand All @@ -117,9 +125,9 @@ sink {
username="admin"
password="******"
selectdb.config {
file.type='csv'
file.column_separator=','
file.line_delimiter='\n'
file.type="csv"
file.column_separator=","
file.line_delimiter="\n"
}
}
}
Expand All @@ -130,4 +138,5 @@ sink {
### next version

- [Feature] Support SelectDB Cloud Sink Connector [3958](https://github.com/apache/incubator-seatunnel/pull/3958)
- [Improve] Refactor some SelectDB Cloud Sink code as well as support copy into batch and async flush and cdc [4312](https://github.com/apache/incubator-seatunnel/pull/4312)

2 changes: 2 additions & 0 deletions release-note.md
Expand Up @@ -40,11 +40,13 @@
- [API]Add parallelism and column projection interface #3829
- [API]Add get source method to all source connector #3846
- [Hive] Support read user-defined partitions #3842
- [SelectDB Cloud] Support SelectDB Cloud Sink Connector #3958
- [Hive] Support read text table & Column projection #4105
- [File] Support column projection #4105
- [Github] Add github source connector #4155
- [Jdbc] Add database field to sink config #4199
- [Doris] Refactor some Doris Sink code as well as support 2pc and cdc #4235
- [SelectDB Cloud] Refactor some SelectDB Cloud Sink code as well as support copy into batch and async flush and cdc #4312
### Zeta Engine
- [Chore] Remove unnecessary dependencies #3795
- [Core] Improve job restart of all node down #3784
Expand Down

This file was deleted.

This file was deleted.

This file was deleted.

0 comments on commit 11e94b2

Please sign in to comment.