[Connector-V2] Add Kudu source and sink connector #2254

2013650523 · 2022-07-23T23:12:48Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

add kudu dependency

add kudu config

add kudu

CalvinKirs · 2022-07-24T10:03:52Z

plugin-mapping.properties

@@ -102,6 +102,8 @@ seatunnel.sink.Clickhouse = connector-clickhouse
 seatunnel.sink.ClickhouseFile = connector-clickhouse
 seatunnel.source.Jdbc = connector-jdbc
 seatunnel.sink.Jdbc = connector-jdbc
+seatunnel.source.Kudu = connector-Kudu


should be seatunnel.source.Kudu = connector-kudu

CalvinKirs · 2022-07-24T10:04:49Z

seatunnel-connectors-v2/connector-kudu/pom.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"


hi, you forget the license header.

CalvinKirs · 2022-07-24T10:11:44Z

...src/main/java/org/apache/seatunnel/connectors/seatunnel/kudu/kuduclient/KuduInputFormat.java

+        } catch (Exception e) {
+            logger .warn("get row type info exception", e);
+            throw new PrepareFailException("kudu", PluginType.SOURCE, e.toString());


It's better to add an exception Message.

throw new PrepareFailException("kudu", PluginType.SOURCE, e.toString()); I don't quite understand,There's an exception message in this one

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/kudu/kuduclient/KuduOutputFormat.java

EricJoy2048 · 2022-07-25T02:12:56Z

Can you change this pr title to [Connector-V2] ?

2013650523 · 2022-07-25T02:36:33Z

Can you change this pr title to [Connector-V2] ?

ok,tks

EricJoy2048

Some suggestions

EricJoy2048 · 2022-07-25T02:45:11Z

...kudu/src/main/java/org/apache/seatunnel/connectors/seatunnel/kudu/config/KuduSinkConfig.java

+
+    public KuduSinkConfig(@NonNull Config pluginConfig) {
+
+        this.saveMode = StringUtils.isBlank(pluginConfig.getString(KUDU_SAVE_MODE)) ? SaveMode.APPEND : SaveMode.fromStr(pluginConfig.getString(KUDU_SAVE_MODE));


There are some problem in this line

pluginConfig.getString("xxxx") will throw NullPointException when xxxx not in config file. So we need judge if xxxx is config file by pluginConfig.hasPath("xxxx").

EricJoy2048 · 2022-07-25T02:51:46Z

...src/main/java/org/apache/seatunnel/connectors/seatunnel/kudu/kuduclient/KuduInputFormat.java

+            schema = kuduClient.openTable(tableName).getSchema();
+            keyColumn = schema.getPrimaryKeyColumns().get(0).getName();
+            columns =schema.getColumns();
+        } catch (KuduException e) {


If the KuduException be catch, this method will return null. But I found you didn't handle the null value when you use the return value of this method.

EricJoy2048 · 2022-07-25T03:03:21Z

...in/java/org/apache/seatunnel/connectors/seatunnel/kudu/sink/KuduSinkAggregatedCommitter.java

+import java.util.List;
+import java.util.Map;
+
+public class KuduSinkAggregatedCommitter implements SinkAggregatedCommitter<KuduCommitInfo, KuduAggregatedCommitInfo> {


If the connector don't need store state and 2PC , You can only extends AbstractSimpleSink and that will be simply.

Hisoka-X · 2022-08-01T02:47:03Z

Please resolve ci error. Thanks

2013650523 · 2022-08-03T02:25:53Z

Hi,@CalvinKirs I have finished the modification and provided the document, please help to review

EricJoy2048

+1 cc @CalvinKirs @ashulin

CalvinKirs · 2022-08-04T05:38:51Z

seatunnel-common/src/main/java/org/apache/seatunnel/common/ExceptionUtil.java

+    public static String getMessage(Throwable e) {
+        StringWriter sw = null;
+        PrintWriter pw = null;
+        try {
+            sw = new StringWriter();
+            pw = new PrintWriter(sw);
+            // Output the error stack information to the printWriter
+            e.printStackTrace(pw);
+            pw.flush();
+            sw.flush();
+        } finally {
+            if (sw != null) {
+                try {
+                    sw.close();
+                } catch (IOException e1) {
+                    e1.printStackTrace();
+                }
+            }
+            if (pw != null) {
+                pw.close();
+            }


used try-with-resources to simplify this

* 0 * Update pom.xml add kudu dependency * Update plugin-mapping.properties add kudu config * Update pom.xml add kudu * add email sink connector * Delete seatunnel-connectors-v2/connector-email directory * Update plugin-mapping.properties * Update plugin-mapping.properties * Update pom.xml * Create pom.xml * [Connector-V2] Add Kudu source and sink connector * [Connector-V2] Add Kudu source and sink connector * [Connector-V2] update * [Connector-V2] solve ci error * [Connector-V2] fix problem on code review * [Connector-V2] add kudu usage document and fix problem on code review * Update ExceptionUtil.java * Update pom.xml * Update Kudu.md * Update ExceptionUtil.java Co-authored-by: Hisoka <fanjiaeminem@qq.com>

2013650523 and others added 8 commits July 22, 2022 16:20

0

1737798

Update pom.xml

eedef28

add kudu dependency

Update plugin-mapping.properties

146fa24

add kudu config

Update pom.xml

9b1f10b

add kudu

add email sink connector

8c28e80

Delete seatunnel-connectors-v2/connector-email directory

ed790c8

Update plugin-mapping.properties

5ba8049

Merge branch 'dev' into api-draft

48018fa

2013650523 mentioned this pull request Jul 24, 2022

[Api draft][connector] Add Kudu source and sink connector #2250

Closed

3 tasks

CalvinKirs changed the title ~~[api draft] [connector] Add Kudu source and sink connector~~ [Connector] Add Kudu source and sink connector Jul 24, 2022

CalvinKirs added the connectors-v1 SeaTunnel connectors, include sink, source label Jul 24, 2022

CalvinKirs reviewed Jul 24, 2022

View reviewed changes

ashulin reviewed Jul 24, 2022

View reviewed changes

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/kudu/kuduclient/KuduOutputFormat.java Show resolved Hide resolved

2013650523 added 3 commits July 25, 2022 09:22

Update plugin-mapping.properties

2c1bf27

Update pom.xml

843ba43

Create pom.xml

6893447

2013650523 changed the title ~~[Connector] Add Kudu source and sink connector~~ [Connector-V2] Add Kudu source and sink connector Jul 25, 2022

EricJoy2048 reviewed Jul 25, 2022

View reviewed changes

CalvinKirs added connectors-v2 and removed connectors-v1 SeaTunnel connectors, include sink, source labels Jul 27, 2022

2013650523 added 2 commits July 29, 2022 21:37

[Connector-V2] Add Kudu source and sink connector

aa2cf30

[Connector-V2] Add Kudu source and sink connector

42ea0f0

Hisoka-X assigned 2013650523 Aug 1, 2022

2013650523 added 2 commits August 2, 2022 10:58

[Connector-V2] update

a79807f

[Connector-V2] solve ci error

d4fc941

Hisoka-X mentioned this pull request Aug 2, 2022

[Umbrella][Connector] New SeaTunnel API Connectors #1946

Closed

[Connector-V2] fix problem on code review

b7ceb54

Hisoka-X requested review from CalvinKirs, ashulin and EricJoy2048 and removed request for ashulin and EricJoy2048 August 2, 2022 10:11

2013650523 and others added 2 commits August 3, 2022 09:16

[Connector-V2] add kudu usage document and fix problem on code review

2043c6e

Merge branch 'dev' into api-draft

582e77e

2013650523 and others added 4 commits August 3, 2022 10:42

Update ExceptionUtil.java

a323b17

Merge branch 'dev' into api-draft

5f213b8

Update pom.xml

f40b9d9

Update Kudu.md

6cde18d

EricJoy2048 previously approved these changes Aug 3, 2022

View reviewed changes

CalvinKirs reviewed Aug 4, 2022

View reviewed changes

Update ExceptionUtil.java

548855f

2013650523 dismissed EricJoy2048’s stale review via 548855f August 4, 2022 08:46

Hisoka-X approved these changes Aug 5, 2022

View reviewed changes

Hisoka-X merged commit 0483cbc into apache:dev Aug 5, 2022

EricJoy2048 mentioned this pull request Oct 8, 2022

[Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Connector-V2] Add Kudu source and sink connector #2254

[Connector-V2] Add Kudu source and sink connector #2254

2013650523 commented Jul 23, 2022 •

edited

CalvinKirs Jul 24, 2022

CalvinKirs Jul 24, 2022

CalvinKirs Jul 24, 2022

2013650523 Jul 25, 2022

EricJoy2048 commented Jul 25, 2022

2013650523 commented Jul 25, 2022

EricJoy2048 left a comment

EricJoy2048 Jul 25, 2022

EricJoy2048 Jul 25, 2022

EricJoy2048 Jul 25, 2022

Hisoka-X commented Aug 1, 2022

2013650523 commented Aug 3, 2022

EricJoy2048 left a comment

CalvinKirs Aug 4, 2022


		public KuduSinkConfig(@NonNull Config pluginConfig) {

		this.saveMode = StringUtils.isBlank(pluginConfig.getString(KUDU_SAVE_MODE)) ? SaveMode.APPEND : SaveMode.fromStr(pluginConfig.getString(KUDU_SAVE_MODE));

[Connector-V2] Add Kudu source and sink connector #2254

[Connector-V2] Add Kudu source and sink connector #2254

Conversation

2013650523 commented Jul 23, 2022 • edited

Purpose of this pull request

Check list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EricJoy2048 commented Jul 25, 2022

2013650523 commented Jul 25, 2022

EricJoy2048 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X commented Aug 1, 2022

2013650523 commented Aug 3, 2022

EricJoy2048 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2013650523 commented Jul 23, 2022 •

edited