[Feature][Connector-V2]Support datahub sink #2558

chessplay · 2022-08-29T06:47:10Z

**Support datahub sink #1946

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

This reverts commit 84f6b17

…into dev � Conflicts: � plugin-mapping.properties � seatunnel-connectors-v2-dist/pom.xml � seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/FileSystemType.java � seatunnel-connectors-v2/connector-file/pom.xml

…into dev � Conflicts: � plugin-mapping.properties

chessplay · 2022-08-29T07:37:39Z

@CalvinKirs please help me review the datahub connector code, the previous pr #2529 has been closed, I have optimized the code as your advice. Example and unit case of this connector is not convenient to provide,because it need to personal datahub account ,include a accessKey and accessSecret.

EricJoy2048

Please add e2e test for this connector.
Need add this connector to seatunnel-connector-v2-dist/pom.xml

EricJoy2048 · 2022-08-29T11:16:39Z

docs/en/connector-v2/sink/Datahub.md

+| retryTimes | int    | yes      | -             |
+
+### url [string]
+i


What's the i mean?

EricJoy2048 · 2022-08-29T11:22:09Z

...ctor-datahub/src/main/java/org/apache/seatunnel/connectors/seatunnel/sink/DataHubWriter.java

+            int i = result.getFailedRecordCount();
+            if (i > 0) {
+                LOG.info("begin to retry for putting failed record");
+                if (retry(result.getFailedRecords(), retryTimes, project, topic)) {


I suggest use RetryUtils to do retry.

CalvinKirs · 2022-08-29T12:30:19Z

seatunnel-connectors-v2/connector-datahub/pom.xml

+    <dependency>
+      <groupId>com.aliyun.datahub</groupId>
+      <artifactId>aliyun-sdk-datahub</artifactId>
+      <version>2.19.0-public</version>
+    </dependency>


hi, Our versions are managed uniformly in the root pom (dependency.managent)

CalvinKirs · 2022-08-29T12:31:56Z

plugin-mapping.properties

@@ -121,4 +121,5 @@ seatunnel.sink.IoTDB = connector-iotdb
 seatunnel.sink.Neo4j = connector-neo4j
 seatunnel.sink.FtpFile = connector-file-ftp
 seatunnel.sink.Socket = connector-socket
+seatunnel.sink.datahub = connector-datahub


seatunnel.sink.datahubrename to seatunnel.sink.DataHub it's better,We'd better be consistent with pluginname

CalvinKirs · 2022-08-29T12:34:14Z

...ctor-datahub/src/main/java/org/apache/seatunnel/connectors/seatunnel/sink/DataHubWriter.java

+ */
+public class DataHubWriter extends AbstractSinkWriter<SeaTunnelRow, Void> {
+
+    private static final Logger LOG = LoggerFactory.getLogger(DataHubWriter.class);


We can use lombok's @slf4j annotation instead of manually declaring

update plugin-mapping.properties

EricJoy2048 · 2022-08-30T03:27:06Z

...l-e2e/seatunnel-spark-connector-v2-e2e/src/test/resources/datahub/fakesource_to_datahub.conf

+   }
+
+  # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
+  # please go to https://seatunnel.apache.org/docs/category/sink-v2


This url is not right.

https://seatunnel.apache.org/docs/connector-v2/sink may more suitable

EricJoy2048 · 2022-08-30T03:32:26Z

seatunnel-dist/release-docs/licenses/LICENSE-jsr311-api.txt

@@ -1,4 +1,4 @@
-COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0
+COMMON DEVELOPMENT AND DISTtRIBUTION LICENSE (CDDL) Version 1.0


please check this.

CalvinKirs · 2022-08-30T09:34:12Z

@CalvinKirs please help me review the datahub connector code, the previous pr #2529 has been closed, I have optimized the code as your advice. Example and unit case of this connector is not convenient to provide,because it need to personal datahub account ,include a accessKey and accessSecret.

Hi, You are right, can you disable your e2e and then delete your key or replace it with XXX. I'm concerned about the security of your account key.

chessplay · 2022-08-30T11:36:56Z

local e2e test

EricJoy2048

LGTM

CalvinKirs · 2022-08-30T12:03:53Z

...l-e2e/seatunnel-spark-connector-v2-e2e/src/test/resources/datahub/fakesource_to_datahub.conf

+    endpoint="http://dh-cn-hangzhou.aliyuncs.com"
+    accessId="LTAIMHD6kepeaI53"
+    accessKey="SoEU2dIeVGBIfM3LSB8G7QSrwGZLlO"


hi,for security reasons, you need to replace the key information with XXXhi,

CalvinKirs · 2022-08-30T12:36:44Z

...or-v2-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/datahub/FakeSourceToDatahubIT.java

+
+import java.io.IOException;
+
+@Disabled


Hi, can you add a comment to explain why?

CalvinKirs

LGTM, thx

Co-authored-by: zhoulw11 <zhoulw11@chinatelecom.cn>

zhoulw11 and others added 22 commits August 20, 2022 17:05

add ftp file sink

038adfa

add ftp file sink

007f426

add ftp file sink

201c9be

add ftp file sink

8fde087

optimize ftp connector

446d05c

This reverts commit 84f6b17

optimize code

a39cfee

update notice file

8da3e88

update license file

334cd06

Merge branch 'apache:dev' into dev

0385cfa

optimize code

6171785

Merge remote-tracking branch 'origin/dev' into dev

dc40cdf

update dependency license

910389c

optimize code

905bc88

delete unuseful license

68956c7

remove redundant empty line

fa6511f

support datahub sink

4ce1b8b

optimize datahub sink code

ca97be7

Merge branch 'dev' of https://github.com/chessplay/incubator-seatunnel …

18a0a64

…into dev � Conflicts: � plugin-mapping.properties

add datahub to plugin-mapping.properties

8e8df2f

optimize the code

1d61887

optimize the code

8129eb9

chessplay changed the title ~~[Feature][File connector] Support datahub sink~~ [Feature]Support datahub sink Aug 29, 2022

chessplay changed the title ~~[Feature]Support datahub sink~~ [Feature][Connector-V2]Support datahub sink Aug 29, 2022

EricJoy2048 reviewed Aug 29, 2022

View reviewed changes

CalvinKirs reviewed Aug 29, 2022

View reviewed changes

CalvinKirs added the connectors-v2 label Aug 29, 2022

zhoulw11 and others added 5 commits August 29, 2022 23:31

optimize the code and add e2e test

10ac7c9

Merge branch 'apache:dev' into dev

1a2fa7c

update plugin-map

1694b94

update plugin-mapping.properties

optimize the code style

cfa46bf

optimize the code style

b079478

EricJoy2048 reviewed Aug 30, 2022

View reviewed changes

zhoulw11 and others added 3 commits August 30, 2022 13:40

optimize the code

3ae350f

Merge branch 'apache:dev' into dev

008cf90

fix bugs

4222a72

EricJoy2048 previously approved these changes Aug 30, 2022

View reviewed changes

CalvinKirs reviewed Aug 30, 2022

View reviewed changes

disable datahub e2e test

2330f48

chessplay dismissed EricJoy2048’s stale review via 2330f48 August 30, 2022 12:32

CalvinKirs reviewed Aug 30, 2022

View reviewed changes

disable datahub e2e test add comment

25fd796

CalvinKirs approved these changes Aug 30, 2022

View reviewed changes

CalvinKirs merged commit 43600a7 into apache:dev Aug 31, 2022

MRYOG pushed a commit to MRYOG/incubator-seatunnel that referenced this pull request Sep 8, 2022

[Feature][Connector-V2]Support datahub sink (apache#2558)

7ad9590

Co-authored-by: zhoulw11 <zhoulw11@chinatelecom.cn>

MRYOG pushed a commit to MRYOG/incubator-seatunnel that referenced this pull request Sep 15, 2022

[Feature][Connector-V2]Support datahub sink (apache#2558)

726dda9

Co-authored-by: zhoulw11 <zhoulw11@chinatelecom.cn>

MRYOG pushed a commit to MRYOG/incubator-seatunnel that referenced this pull request Sep 16, 2022

[Feature][Connector-V2]Support datahub sink (apache#2558)

de54a5c

Co-authored-by: zhoulw11 <zhoulw11@chinatelecom.cn>

TyrantLucifer pushed a commit to TyrantLucifer/incubator-seatunnel that referenced this pull request Sep 18, 2022

[Feature][Connector-V2]Support datahub sink (apache#2558)

42e0f83

Co-authored-by: zhoulw11 <zhoulw11@chinatelecom.cn>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2]Support datahub sink #2558

[Feature][Connector-V2]Support datahub sink #2558

chessplay commented Aug 29, 2022 •

edited

chessplay commented Aug 29, 2022

EricJoy2048 left a comment

EricJoy2048 Aug 29, 2022

EricJoy2048 Aug 29, 2022

CalvinKirs Aug 29, 2022

CalvinKirs Aug 29, 2022

CalvinKirs Aug 29, 2022

EricJoy2048 Aug 30, 2022

EricJoy2048 Aug 30, 2022

CalvinKirs commented Aug 30, 2022

chessplay commented Aug 30, 2022

EricJoy2048 left a comment

CalvinKirs Aug 30, 2022

CalvinKirs Aug 30, 2022

CalvinKirs left a comment

		@@ -1,4 +1,4 @@
		COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0
		COMMON DEVELOPMENT AND DISTtRIBUTION LICENSE (CDDL) Version 1.0


		import java.io.IOException;

		@Disabled

[Feature][Connector-V2]Support datahub sink #2558

[Feature][Connector-V2]Support datahub sink #2558

Conversation

chessplay commented Aug 29, 2022 • edited

Purpose of this pull request

Check list

chessplay commented Aug 29, 2022

EricJoy2048 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CalvinKirs commented Aug 30, 2022

chessplay commented Aug 30, 2022

EricJoy2048 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CalvinKirs left a comment

Choose a reason for hiding this comment

chessplay commented Aug 29, 2022 •

edited