[Feature][Connector-V2][Maxcompute] Add Maxcompute source & sink connector #3640

stdnt-xiao · 2022-12-04T06:09:47Z

Purpose of this pull request

Maxcompute source & sink connector #3018

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist

# Conflicts: # plugin-mapping.properties # seatunnel-connectors-v2/pom.xml # seatunnel-dist/pom.xml

TaoZex

Please add e2e tests or picture proof that this connector feature is available.

TaoZex · 2022-12-04T07:15:50Z

docs/en/connector-v2/sink/Maxcompute.md

+
+## Changelog
+
+### 2.1.3-beta 2022-12-05


Use next version will be better.

TaoZex · 2022-12-04T07:16:05Z

docs/en/connector-v2/source/Maxcompute.md

+
+## Changelog
+
+### 2.1.3-beta 2022-12-05


Same as above.

TaoZex · 2022-12-04T07:20:57Z

.../main/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/config/MaxcomputeConfig.java

+    public static final Option<String> ACCESS_ID = Options.key("accessId").stringType().noDefaultValue()
+        .withDescription("Your Maxcompute accessId which cloud be access from Alibaba Cloud");
+    public static final Option<String> ACCESS_KEY = Options.key("accesskey").stringType().noDefaultValue()
+        .withDescription("Your Maxcompute accessKey which cloud be access from Alibaba Cloud");
+    public static final Option<String> ENDPOINT = Options.key("endpoint").stringType().noDefaultValue()
+        .withDescription("Your Maxcompute endpoint start with http");
+    public static final Option<String> PROJECT = Options.key("project").stringType().noDefaultValue()
+        .withDescription("Your Maxcompute project which is created in Alibaba Cloud");
+    public static final Option<String> RESULT_TABLE_NAME = Options.key("result_table_name").stringType().noDefaultValue()
+        .withDescription("Target Maxcompute table name eg: fake");
+    public static final Option<String> PARTITION_SPEC = Options.key("partition_spec").stringType().noDefaultValue()
+        .withDescription("This spec of Maxcompute partition table.");
+    public static final Option<Integer> SPLIT_ROW = Options.key("split_row").intType().defaultValue(SPLIT_ROW_DEFAULT)
+        .withDescription("Number of rows per split. default: 10000");
+    public static final Option<Boolean> OVERWRITE = Options.key("overwrite").booleanType().defaultValue(false)
+        .withDescription("Whether to overwrite the table or partition");


Please fix code style.You can refer this pr : e68ecf7

TaoZex · 2022-12-04T07:22:18Z

...in/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/sink/MaxcomputeSinkFactory.java

+    }
+
+    @Override
+    public OptionRule optionRule() {


Add UT for optionRule() please.

TaoZex · 2022-12-04T07:23:44Z

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/sink/MaxcomputeWriter.java

+@Slf4j
+public class MaxcomputeWriter extends AbstractSinkWriter<SeaTunnelRow, Void> {
+    private static final Logger LOG = LoggerFactory.getLogger(MaxcomputeSource.class);


Already using @Slf4j, LOG may not be useful.

TaoZex · 2022-12-04T07:24:49Z

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/sink/MaxcomputeWriter.java

+        try {
+            this.session.commit(new Long[]{Thread.currentThread().getId()});
+        } catch (TunnelException e) {
+            throw new RuntimeException(e);


Unify exception for connector.

TaoZex · 2022-12-04T07:25:38Z

...ava/org/apache/seatunnel/connectors/seatunnel/maxcompute/source/MaxcomputeSourceFactory.java

+    }
+
+    @Override
+    public OptionRule optionRule() {


Please add optionRule test.

please, give a refer pr

refer this：226dc6a

TaoZex · 2022-12-04T07:27:05Z

...ain/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/util/MaxcomputeTypeMapper.java

+            //Doesn't support yet
+            case MAXCOMPUTE_UNKNOWN:
+            default:
+                throw new UnsupportedOperationException(


Unify exception.

stdnt-xiao · 2022-12-04T09:19:32Z

Please add e2e tests or picture proof that this connector feature is available.

TyrantLucifer · 2022-12-05T04:18:01Z

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/sink/MaxcomputeSink.java

+
+    @Override
+    public String getPluginName() {
+        return "MaxcomputeSink";


Suggested change

return "MaxcomputeSink";

return "Maxcompute";

TyrantLucifer · 2022-12-05T04:18:44Z

.../main/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/source/MaxcomputeSource.java

+
+    @Override
+    public String getPluginName() {
+        return "MaxcomputeSource";


Suggested change

return "MaxcomputeSource";

return "Maxcompute";

TyrantLucifer · 2022-12-05T04:20:13Z

...java/org/apache/seatunnel/connectors/seatunnel/maxcompute/source/MaxcomputeSourceReader.java

+@Slf4j
+public class MaxcomputeSourceReader implements SourceReader<SeaTunnelRow, MaxcomputeSourceSplit> {
+    private final SourceReader.Context context;
+    private Set<MaxcomputeSourceSplit> sourceSplits;


Suggested change

private Set<MaxcomputeSourceSplit> sourceSplits;

private final Set<MaxcomputeSourceSplit> sourceSplits;

TyrantLucifer · 2022-12-05T04:23:07Z

...ain/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/util/MaxcomputeTypeMapper.java

+            SeaTunnelDataType<?> seaTunnelDataType = seaTunnelDataTypes[i];
+            if (null == rs.get(i)) {
+                seatunnelField = null;
+            } else if (BasicType.BOOLEAN_TYPE.equals(seaTunnelDataType)) {


Use switch(seatunnelDataType.getSqlType()) is better. You can refer to https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/ParquetReadStrategy.java#L128

TyrantLucifer · 2022-12-05T04:25:15Z

Overall LGTM except few nits, thank you for your contribution of SeaTunnel.

EricJoy2048 · 2022-12-05T14:32:19Z

docs/en/connector-v2/sink/Maxcompute.md

+| accesskey               | string  | yes       | -             |
+| endpoint                | string  | yes       | -             |
+| project                 | string  | yes       | -             |
+| result_table_name       | string  | yes       | -             |


result_table_name is in common-options now.

Here is an example

EricJoy2048 · 2022-12-05T14:32:52Z

docs/en/connector-v2/sink/Maxcompute.md

+
+`project` Your Maxcompute project which is created in Alibaba Cloud.
+
+### result_table_name [string]


remove this option.

EricJoy2048 · 2022-12-05T14:37:45Z

docs/en/connector-v2/sink/Maxcompute.md

+
+## Key features
+
+- [x] [batch](../../concept/connector-v2-features.md)


Sink Connector have no Key features named batch and it only have two Key features

EricJoy2048 · 2022-12-05T14:39:39Z

docs/en/connector-v2/source/Maxcompute.md

+| accesskey               | string | yes       | -             |
+| endpoint                | string | yes       | -             |
+| project                 | string | yes       | -             |
+| result_table_name       | string | yes       | -             |


delete result_table_name and add common-options .

EricJoy2048 · 2022-12-05T14:45:59Z

...in/java/org/apache/seatunnel/connectors/seatunnel/maxcompute/sink/MaxcomputeSinkFactory.java

+    @Override
+    public OptionRule optionRule() {
+        return OptionRule.builder()
+            .required(ACCESS_ID, ACCESS_KEY, ENDPOINT, PROJECT, RESULT_TABLE_NAME)


RESULT_TABLE_NAME is a common options . It will used by engine to generate logical DAG.

stdnt-xiao · 2022-12-06T07:24:47Z

TaoZex

LGTM. Thanks for your contribution.

EricJoy2048

LGTM, @TyrantLucifer PTAL

docs/en/connector-v2/sink/Maxcompute.md

docs/en/connector-v2/source/Maxcompute.md

Co-authored-by: hailin0 <hailin088@gmail.com>

stdnt-xiao and others added 4 commits November 22, 2022 15:36

add maxcompute connector

3fcc7be

update maxcompute connector

9be445f

reback example

be9331d

Merge branch 'dev' into dev-maxcompute

67796aa

# Conflicts: # plugin-mapping.properties # seatunnel-connectors-v2/pom.xml # seatunnel-dist/pom.xml

stdnt-xiao changed the title ~~[Feature][Connector-V2][Maxcompute] Add Maxcompute file source & sink connector~~ [Feature][Connector-V2][Maxcompute] Add Maxcompute source & sink connector Dec 4, 2022

pom.xml add license header

dd47b38

TaoZex reviewed Dec 4, 2022

View reviewed changes

Modify the code according to the reviewer

9889561

stdnt-xiao requested a review from TaoZex December 4, 2022 09:38

TyrantLucifer reviewed Dec 5, 2022

View reviewed changes

EricJoy2048 reviewed Dec 5, 2022

View reviewed changes

Modify code according to reviewer(@TyrantLucifer & @EricJoy2048),thx

7dd5335

stdnt-xiao requested review from EricJoy2048, TyrantLucifer and TaoZex and removed request for TaoZex, EricJoy2048 and TyrantLucifer December 6, 2022 07:21

TaoZex previously approved these changes Dec 6, 2022

View reviewed changes

github-actions bot added the reviewed label Dec 6, 2022

hailin0 added feature New feature connectors-v2 labels Dec 7, 2022

EricJoy2048 previously approved these changes Dec 7, 2022

View reviewed changes

Merge branch 'dev' into dev-maxcompute

4dc42d9

EricJoy2048 dismissed stale reviews from TaoZex and themself via 4dc42d9 December 7, 2022 14:12

github-actions bot removed the reviewed label Dec 7, 2022

TaoZex previously approved these changes Dec 7, 2022

View reviewed changes

github-actions bot added the reviewed label Dec 7, 2022

TyrantLucifer added this to the 2.3.0 milestone Dec 7, 2022

EricJoy2048 closed this Dec 8, 2022

EricJoy2048 reopened this Dec 8, 2022

EricJoy2048 previously approved these changes Dec 8, 2022

View reviewed changes

github-actions bot added the approved label Dec 8, 2022

hailin0 reviewed Dec 9, 2022

View reviewed changes

docs/en/connector-v2/sink/Maxcompute.md Outdated Show resolved Hide resolved

docs/en/connector-v2/source/Maxcompute.md Outdated Show resolved Hide resolved

Hisoka-X added the First-time contributor First-time contributor label Dec 10, 2022

stdnt-xiao dismissed EricJoy2048’s stale review via 728dd87 December 10, 2022 02:31

Update docs/en/connector-v2/sink/Maxcompute.md

728dd87

Co-authored-by: hailin0 <hailin088@gmail.com>

stdnt-xiao dismissed TaoZex’s stale review via 728dd87 December 10, 2022 02:31

Update docs/en/connector-v2/source/Maxcompute.md

49e55d8

Co-authored-by: hailin0 <hailin088@gmail.com>

TaoZex approved these changes Dec 10, 2022

View reviewed changes

github-actions bot removed the approved label Dec 10, 2022

hailin0 approved these changes Dec 10, 2022

View reviewed changes

github-actions bot added the approved label Dec 10, 2022

EricJoy2048 approved these changes Dec 11, 2022

View reviewed changes

EricJoy2048 merged commit 80cf8f4 into apache:dev Dec 11, 2022

EricJoy2048 mentioned this pull request Dec 13, 2022

[Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018

Open

3 tasks

stdnt-xiao deleted the dev-maxcompute branch January 5, 2023 11:53

stdnt-xiao mentioned this pull request May 8, 2023

[Feature][Spark] Add MaxCompute to source and sink #1323

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2][Maxcompute] Add Maxcompute source & sink connector #3640

[Feature][Connector-V2][Maxcompute] Add Maxcompute source & sink connector #3640

stdnt-xiao commented Dec 4, 2022 •

edited

Loading

TaoZex left a comment

TaoZex Dec 4, 2022

TaoZex Dec 4, 2022

TaoZex Dec 4, 2022

TaoZex Dec 4, 2022

TaoZex Dec 4, 2022 •

edited

Loading

TaoZex Dec 4, 2022

TaoZex Dec 4, 2022

stdnt-xiao Dec 4, 2022

TaoZex Dec 4, 2022

TaoZex Dec 4, 2022

stdnt-xiao commented Dec 4, 2022

TyrantLucifer Dec 5, 2022

TyrantLucifer Dec 5, 2022

TyrantLucifer Dec 5, 2022

TyrantLucifer Dec 5, 2022

TyrantLucifer commented Dec 5, 2022

EricJoy2048 Dec 5, 2022

EricJoy2048 Dec 5, 2022

EricJoy2048 Dec 5, 2022

EricJoy2048 Dec 5, 2022

EricJoy2048 Dec 5, 2022

stdnt-xiao commented Dec 6, 2022

TaoZex left a comment

EricJoy2048 left a comment

	private Set<MaxcomputeSourceSplit> sourceSplits;
	private final Set<MaxcomputeSourceSplit> sourceSplits;


		`project` Your Maxcompute project which is created in Alibaba Cloud.

		### result_table_name [string]


		## Key features

		- [x] [batch](../../concept/connector-v2-features.md)

[Feature][Connector-V2][Maxcompute] Add Maxcompute source & sink connector #3640

[Feature][Connector-V2][Maxcompute] Add Maxcompute source & sink connector #3640

Conversation

stdnt-xiao commented Dec 4, 2022 • edited Loading

Purpose of this pull request

Check list

TaoZex left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TaoZex Dec 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stdnt-xiao commented Dec 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TyrantLucifer commented Dec 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stdnt-xiao commented Dec 6, 2022

TaoZex left a comment

Choose a reason for hiding this comment

EricJoy2048 left a comment

Choose a reason for hiding this comment

stdnt-xiao commented Dec 4, 2022 •

edited

Loading

TaoZex Dec 4, 2022 •

edited

Loading