[feature][connector-jdbc]Add Save Mode function and Connector-JDBC (MySQL) connector has been realized #5663

chl-wxp · 2023-10-19T08:32:24Z

Function list

add save mode function interface ，see “org.apache.seatunnel.api.sink.SaveModeHandler”
add saveModeHandler default implementation ，see “org.apache.seatunnel.api.sink.DefaultSaveModeHandler”
connector-jdbc(mysql) implement the save mode function interface

… has been realized

…_mysql # Conflicts: # seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSink.java # seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkFactory.java

hailin0 · 2023-10-19T15:25:48Z

...nector-jdbc-e2e/connector-jdbc-e2e-part-7/src/test/resources/jdbc_mysql_source_and_sink.conf

+    user = "root"
+    password = "Abc!@#135_seatunnel"
+    generate_sink_sql = true
+    catalog {


this tag has deleted
#5645

Has been processed

hailin0 · 2023-10-19T15:29:04Z

...nnector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSink.java

@@ -221,6 +222,13 @@ public void handleSaveMode(DataSaveMode saveMode) {
                    if (!catalog.tableExists(tablePath)) {
                        catalog.createTable(tablePath, catalogTable, true);
                    }


Why retain these logics?

Has been processed

hailin0 · 2023-10-20T02:51:59Z

...nnector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSink.java

                } catch (UnsupportedOperationException | CatalogException e) {
                    // TODO Temporary fix, this feature has been changed in this pr
                    // https://github.com/apache/seatunnel/pull/5645


hailin0 · 2023-10-20T02:53:47Z

...nnector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSink.java

-                    if (!catalog.tableExists(tablePath)) {
-                        catalog.createTable(tablePath, catalogTable, true);
-                    }
+                    return new DefaultSaveModeHandler(


205 line：try (Catalog catalog = catalogOptional.get()) {

this code auto close catalog, Will savemode not interrupt abnormally?

hailin0 · 2023-10-20T07:00:05Z

rebase dev merge this pr
#5620

…_mysql # Conflicts: # seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSink.java # seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkFactory.java # seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java

Hisoka-X · 2023-10-20T08:46:05Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/common/SeaTunnelAPIErrorCode.java

+    SINK_TABLE_NOT_EXIST("API-11", "The sink table not exist");
    ;


Suggested change

SINK_TABLE_NOT_EXIST("API-11", "The sink table not exist");

;

SINK_TABLE_NOT_EXIST("API-11", "The sink table not exist");

Has been processed

Hisoka-X · 2023-10-20T08:47:01Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/DataSaveMode.java

-    // path and files in the path, create new files in the path.
-    KEEP_SCHEMA_AND_DATA,
+    // Preserve database structure, preserve data
+    AND_DATA,


What's meaning of AND_DATA? I think you mean APPEND_DATA?

Has been processed

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/DataSaveMode.java

Hisoka-X · 2023-10-20T09:01:18Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SupportSaveMode.java

+    String SCHEMA_SAVE_MODE_KEY = "schema_save_mode";
+
+    // This method defines the return of a specific save_mode handler
+    SaveModeHandler getSaveModeHandler();


Use optional to avoid NPE

Suggested change

SaveModeHandler getSaveModeHandler();

Optional<SaveModeHandler> getSaveModeHandler();

Has been processed

…_mysql # Conflicts: # seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/AbstractJdbcCatalog.java # seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/mysql/MySqlCatalog.java # seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/mysql/MysqlCreateTableSqlBuilder.java # seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-3/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcPostgresIT.java

…_mysql

hailin0 · 2023-10-26T12:55:49Z

...-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkFactory.java

                        finalCatalogTable);
    }

    @Override
    public OptionRule optionRule() {
        return OptionRule.builder()
-                .required(URL, DRIVER)
+                .required(URL, DRIVER, SCHEMA_SAVE_MODE, DATA_SAVE_MODE)


Add testcase into this config

https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/resources/jdbc_mysql_source_and_sink_with_multiple_tables.conf

https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql_with_multi_table.conf

hailin0

add new options into connector docs

...src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlSaveModeHandlerIT.java

hailin0 · 2023-10-26T13:18:47Z

...-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkFactory.java

        CatalogTable catalogTable = context.getCatalogTable();
-        ReadonlyConfig catalogOptions = getCatalogOptions(context);
+        Map<String, String> catalogOptions =
+                config.get(CatalogOptions.CATALOG_OPTIONS) == null
+                        ? new HashMap<>()
+                        : config.get(CatalogOptions.CATALOG_OPTIONS);


Why change here

Compatible with catalog

…_mysql # Conflicts: # seatunnel-connectors-v2/connector-common/src/main/java/org/apache/seatunnel/connectors/seatunnel/common/multitablesink/MultiTableSink.java # seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSink.java # seatunnel-connectors-v2/connector-starrocks/src/main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/sink/StarRocksSink.java

hailin0

LGTM

chl-wxp added 8 commits October 19, 2023 15:51

[feature] Add Save Mode function and Connector-JDBC (MySQL) connector…

9f97bed

… has been realized

[feature] update MysqlCreateTableSqlBuilder

fe53452

[update] Solving the conflict of consolidation

8fe5b80

[update] update jdbc-e2e-part-7

a4309ed

[fix] fix get table bug

aea950a

[fix] fix mysql_container bug

09e503b

[fix] fix mysql_container bug2

e681a19

hailin0 reviewed Oct 19, 2023

View reviewed changes

[fix] Delete some unreasonable logic

783628e

hailin0 reviewed Oct 20, 2023

View reviewed changes

chl-wxp added 2 commits October 20, 2023 16:24

[update] Solving the conflict of consolidation

3f31cf9

Hisoka-X reviewed Oct 20, 2023

View reviewed changes

chl-wxp added 11 commits October 23, 2023 10:19

[update] update DataSaveMode enum

cc8e509

[update] update jdbcSinkFactory NPE bug

4087977

[update] update jdbcSinkFactory NPE bug 2

b17e4af

[update] update jdbcSinkFactory NPE bug 3

2d92bae

[update] add time for e2e

a16e051

Merge remote-tracking branch 'origin/dev' into feature/save_mode_jdbc…

7c6a692

…_mysql

[update] update create mysql table bit length is 0 bug

f06c920

[update] update dameng bug

f19b46b

[update] update dameng bug2

c06d38b

[update] update dameng port for e2e

05f1fdf

hailin0 reviewed Oct 26, 2023

View reviewed changes

...src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlSaveModeHandlerIT.java Outdated Show resolved Hide resolved

hailin0 reviewed Oct 26, 2023

View reviewed changes

chl-wxp added 5 commits October 27, 2023 11:16

[update] add fink and spark e2e for saveMode handler

fb640db

[update] update jdbcSinkFactory

ed97b4f

[update] Solve conflict

aee4d78

[update] Solve conflict2

fc5d110

hailin0 approved these changes Oct 28, 2023

View reviewed changes

github-actions bot added approved reviewed labels Oct 28, 2023

EricJoy2048 approved these changes Oct 30, 2023

View reviewed changes

EricJoy2048 merged commit eff17cc into apache:dev Oct 30, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature][connector-jdbc]Add Save Mode function and Connector-JDBC (MySQL) connector has been realized #5663

[feature][connector-jdbc]Add Save Mode function and Connector-JDBC (MySQL) connector has been realized #5663

chl-wxp commented Oct 19, 2023

hailin0 Oct 19, 2023

chl-wxp Oct 20, 2023

hailin0 Oct 19, 2023

chl-wxp Oct 20, 2023

hailin0 Oct 20, 2023

hailin0 Oct 20, 2023

hailin0 commented Oct 20, 2023

Hisoka-X Oct 20, 2023

chl-wxp Oct 23, 2023

Hisoka-X Oct 20, 2023

chl-wxp Oct 23, 2023

Hisoka-X Oct 20, 2023

chl-wxp Oct 23, 2023

hailin0 Oct 26, 2023

hailin0 left a comment

hailin0 Oct 26, 2023

chl-wxp Oct 27, 2023

hailin0 left a comment

		SINK_TABLE_NOT_EXIST("API-11", "The sink table not exist");
		;

	SaveModeHandler getSaveModeHandler();
	Optional<SaveModeHandler> getSaveModeHandler();

[feature][connector-jdbc]Add Save Mode function and Connector-JDBC (MySQL) connector has been realized #5663

[feature][connector-jdbc]Add Save Mode function and Connector-JDBC (MySQL) connector has been realized #5663

Conversation

chl-wxp commented Oct 19, 2023

Function list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Oct 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 left a comment

Choose a reason for hiding this comment