[Feature][Connector-V2][SelectDB Cloud] Support SelectDB Cloud Sink Connector #3958

zy-kkk · 2023-01-16T08:25:26Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
SelectDB Cloud is a cloud native database, and docker cannot be used to start test cases.
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

Test procedure

SelectDB Table Schema

CREATE  TABLE `TypeTest` (
    c_map varchar(1000),
    c_array array<int>,
    c_string String,
    c_boolean boolean,
    c_tinyint tinyint,
    c_smallint smallint,
    c_int int,
    c_bigint bigint,
    c_float float,
    c_double double,
    c_decimal  decimal(27, 9),
    c_null String,
    c_bytes String,
    c_date date,
    c_timestamp datetime
) ENGINE=OLAP
DUPLICATE KEY(`c_map`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`c_map`) BUCKETS 4

ST Engine
- config

   env {
      execution.parallelism = 1
      job.mode = "BATCH"
      checkpoint.interval = 10
    }

      source {
        FakeSource {
          row.num = 10
          parallelism = 1
          schema = {
            fields {
              c_map = "map<string, string>"
              c_array = "array<int>"
              c_string = string
              c_boolean = boolean
              c_tinyint = tinyint
              c_smallint = smallint
              c_int = int
              c_bigint = bigint
              c_float = float
              c_double = double
              c_decimal = "decimal(27, 9)"
              c_null = "null"
              c_bytes = bytes
              c_date = date
              c_timestamp = timestamp
            }
          }
          result_table_name = "fake"
        }
      }
      
      transform {
      }
      
      sink {
        SelectDBCloud {
          load-url="ip:port"
          jdbc-url="ip:port"
          cluster-name="SeaTunnel"
          table.identifier="test.TypeTest"
          username="admin"
          password="******"
          selectdb.config {
              file.type="json"
              file.strip_outer_array="false"
          }
        }
      }

result

Flink
- config

   env {
        execution.parallelism = 1
        job.mode = "BATCH"
    }

      source {
        FakeSource {
          row.num = 10
          parallelism = 1
          schema = {
            fields {
              c_map = "map<string, string>"
              c_array = "array<int>"
              c_string = string
              c_boolean = boolean
              c_tinyint = tinyint
              c_smallint = smallint
              c_int = int
              c_bigint = bigint
              c_float = float
              c_double = double
              c_decimal = "decimal(27, 9)"
              c_null = "null"
              c_bytes = bytes
              c_date = date
              c_timestamp = timestamp
            }
          }
          result_table_name = "fake"
        }
      }
      
      transform {
      }
      
      sink {
        SelectDBCloud {
          load-url="ip:port"
          jdbc-url="ip:port"
          cluster-name="SeaTunnel"
          table.identifier="test.TypeTest"
          username="admin"
          password="******"
          selectdb.config {
              file.type="json"
              file.strip_outer_array="false"
          }
        }
      }

result

Spark
- Config

  env {
        job.name = "SeaTunnel"
        spark.executor.instances = 1
        spark.executor.cores = 1
        spark.executor.memory = "1g"
        spark.master = local
      }
      
      source {
        FakeSource {
          row.num = 10
          parallelism = 1
          schema = {
            fields {
              c_map = "map<string, string>"
              c_array = "array<int>"
              c_string = string
              c_boolean = boolean
              c_tinyint = tinyint
              c_smallint = smallint
              c_int = int
              c_bigint = bigint
              c_float = float
              c_double = double
              c_decimal = "decimal(27, 9)"
              c_null = "null"
              c_bytes = bytes
              c_date = date
              c_timestamp = timestamp
            }
          }
          result_table_name = "fake"
        }
      }
      
      transform {
      }
      
      sink {
        SelectDBCloud {
          load-url="ip:port"
          jdbc-url="ip:port"
          cluster-name="SeaTunnel"
          table.identifier="test.TypeTest"
          username="admin"
          password="******"
          selectdb.config {
              file.type="json"
              file.strip_outer_array="false"
          }
        }
      }

result

TaoZex · 2023-01-16T08:55:16Z

Please fix code style.

You can refer this doc : https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/setup.md

...or-selectdb-cloud/src/main/java/org/apache/seatunnel/connector/selectdb/util/StringUtil.java

docs/en/connector-v2/sink/SelectDB-Cloud.md

...ud/src/main/java/org/apache/seatunnel/connector/selectdb/sink/writer/SelectDBSinkWriter.java

.../main/java/org/apache/seatunnel/connector/selectdb/exception/SelectDBConnectorErrorCode.java

...-selectdb-cloud/src/main/java/org/apache/seatunnel/connector/selectdb/util/ResponseUtil.java

TaoZex

If you do not have a docker environment for e2e testing, add screenshots to prove that the connector is available.

docs/en/connector-v2/sink/SelectDB-Cloud.md

...src/main/java/org/apache/seatunnel/connector/selectdb/sink/committer/SelectDBCommitInfo.java

.../src/main/java/org/apache/seatunnel/connector/selectdb/sink/committer/SelectDBCommitter.java

...db-cloud/src/main/java/org/apache/seatunnel/connector/selectdb/sink/writer/RecordBuffer.java

...loud/src/main/java/org/apache/seatunnel/connector/selectdb/sink/writer/SelectDBCopyInto.java

...ud/src/main/java/org/apache/seatunnel/connector/selectdb/sink/writer/SelectDBSinkWriter.java

hailin0 · 2023-01-17T05:31:01Z

Please add e2e testcase module

references
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-doris-e2e
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-starrocks-e2e

zy-kkk · 2023-01-17T07:48:19Z

Please add e2e testcase module

references https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-doris-e2e https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-starrocks-e2e

SelectDB Cloud is a cloud native database, and docker cannot be used to start test cases. I will fill in the screenshot of my test results in the description of pull request later

...db-cloud/src/main/java/org/apache/seatunnel/connector/selectdb/sink/writer/RecordBuffer.java

...db-cloud/src/main/java/org/apache/seatunnel/connector/selectdb/sink/writer/RecordStream.java

...ctor-selectdb-cloud/src/main/java/org/apache/seatunnel/connector/selectdb/util/HttpUtil.java

...-selectdb-cloud/src/main/java/org/apache/seatunnel/connector/selectdb/rest/CopyIntoResp.java

hailin0 · 2023-01-17T11:16:28Z

Please add e2e testcase module
references https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-doris-e2e https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-starrocks-e2e

SelectDB Cloud is a cloud native database, and docker cannot be used to start test cases. I will fill in the screenshot of my test results in the description of pull request later

Still need to add e2e moduel, but can add annotation @org.junit.jupiter.api.Disabled

@zy-kkk

zy-kkk · 2023-01-18T03:19:08Z

Please add e2e testcase module
references https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-doris-e2e https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-starrocks-e2e

SelectDB Cloud is a cloud native database, and docker cannot be used to start test cases. I will fill in the screenshot of my test results in the description of pull request later

Still need to add e2e moduel, but can add annotation @org.junit.jupiter.api.Disabled

@zy-kkk

done

TaoZex

LGTM

TyrantLucifer

LGTM

…onnector (apache#3958)

TaoZex reviewed Jan 16, 2023

View reviewed changes

insist777 mentioned this pull request Jan 16, 2023

[Feature][Statistics] Connector pr not yet merged #3960

Closed

3 tasks

TaoZex reviewed Jan 16, 2023

View reviewed changes

TyrantLucifer reviewed Jan 17, 2023

View reviewed changes

...-selectdb-cloud/src/main/java/org/apache/seatunnel/connector/selectdb/rest/CopyIntoResp.java Outdated Show resolved Hide resolved

zy-kkk added 7 commits January 18, 2023 11:16

Support SelectDB Cloud Sink Connector

44d09dc

Support SelectDB Cloud Sink Connector

d58ac53

fix

3896b0e

fix2

ebfa4a7

fix3

77d5e0e

fix4

7d9ab2f

fix5

a9af1e3

zy-kkk force-pushed the selectdb-cloud-sink branch from 2710143 to a9af1e3 Compare January 18, 2023 03:17

zy-kkk added 3 commits January 18, 2023 14:20

fix5

5348316

check style

5125842

check style2

01773b9

TaoZex previously approved these changes Jan 20, 2023

View reviewed changes

github-actions bot added the reviewed label Jan 20, 2023

Hisoka-X added First-time contributor First-time contributor feature New feature connectors-v2 labels Jan 27, 2023

fix doc

3f54233

zy-kkk dismissed TaoZex’s stale review via 3f54233 January 30, 2023 09:22

github-actions bot removed the reviewed label Jan 30, 2023

fix_exception

6f386af

TyrantLucifer approved these changes Jan 31, 2023

View reviewed changes

github-actions bot added approved reviewed labels Jan 31, 2023

hailin0 approved these changes Jan 31, 2023

View reviewed changes

hailin0 merged commit 79a134a into apache:dev Jan 31, 2023

EricJoy2048 mentioned this pull request Feb 3, 2023

[Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018

Open

3 tasks

harveyyue pushed a commit to harveyyue/seatunnel that referenced this pull request Feb 27, 2023

[Feature][Connector-V2][SelectDB Cloud] Support SelectDB Cloud Sink C…

56eef0e

…onnector (apache#3958)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2][SelectDB Cloud] Support SelectDB Cloud Sink Connector #3958

[Feature][Connector-V2][SelectDB Cloud] Support SelectDB Cloud Sink Connector #3958

zy-kkk commented Jan 16, 2023 •

edited

TaoZex commented Jan 16, 2023

TaoZex left a comment

hailin0 commented Jan 17, 2023 •

edited

zy-kkk commented Jan 17, 2023

hailin0 commented Jan 17, 2023

zy-kkk commented Jan 18, 2023

TaoZex left a comment

TyrantLucifer left a comment

[Feature][Connector-V2][SelectDB Cloud] Support SelectDB Cloud Sink Connector #3958

[Feature][Connector-V2][SelectDB Cloud] Support SelectDB Cloud Sink Connector #3958

Conversation

zy-kkk commented Jan 16, 2023 • edited

Purpose of this pull request

Check list

Test procedure

TaoZex commented Jan 16, 2023

TaoZex left a comment

Choose a reason for hiding this comment

hailin0 commented Jan 17, 2023 • edited

zy-kkk commented Jan 17, 2023

hailin0 commented Jan 17, 2023

zy-kkk commented Jan 18, 2023

TaoZex left a comment

Choose a reason for hiding this comment

TyrantLucifer left a comment

Choose a reason for hiding this comment

zy-kkk commented Jan 16, 2023 •

edited

hailin0 commented Jan 17, 2023 •

edited