[Feature][Connector-V2][JDBC] support sqlite Source & Sink #3089

nutsjian · 2022-10-13T09:38:09Z

Purpose of this pull request

[JDBC] SqLite Connector

Due to the type affinity of sqlite3, and the dynamic data types. SqliteTypeMapper.mapping cannot just use columnTypeName to determine the real SeaTunnel Data Type.

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

liugddx

Please migrate the test case to seatunnel-connector-v2-e2e module.

liugddx · 2022-10-13T09:46:38Z

...nector-jdbc-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcSqliteIT.java

+            String createSource = "CREATE TABLE source (\n" +
+                    "age INT NOT NULL,\n" +
+                    "name VARCHAR(255) NOT NULL\n" +
+                    ")";
+            String createSink = "CREATE TABLE sink (\n" +
+                    "age INT NOT NULL,\n" +
+                    "name VARCHAR(255) NOT NULL\n" +
+                    ")";


Please test all field types

Roger that. I'm fixing it.

hailin0

update jdbc docs

https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/sink/Jdbc.md#appendix
https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/source/Jdbc.md#appendix

hailin0 · 2022-10-13T13:52:01Z

...tor-v2-e2e/connector-jdbc-flink-e2e/src/test/resources/jdbc/jdbc_sqlite_source_and_sink.conf

+        user = ""
+        password = ""
+        type_affinity = true
+        query = "select age, name from source"


test all supported datatypes?

reference
https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/src/test/resources/jdbc_oracle_source_to_sink.conf#L36

Thanks man. I'm fixing it. SQLite uses a more general dynamic type system and type affinity (https://www.sqlite.org/datatype3.html). So I consider test all supported datatypes refer to the JDBC mysql e2e test, workable?

liugddx

Please test all field types

liugddx · 2022-10-14T08:26:47Z

...nector-jdbc-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcSqliteIT.java

+    }
+
+    @Test
+    public void testJdbcMysqlSourceAndSinkDataType() throws Exception {


Please check the method name.

liugddx · 2022-10-14T08:26:47Z

...nector-jdbc-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcSqliteIT.java

+    }
+
+    @AfterEach
+    public void closeGreenplumContainer() throws SQLException, IOException {


Please check the method name.

liugddx · 2022-10-14T08:29:21Z

...nector-jdbc-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/jdbc/JdbcSqliteIT.java

+    }
+
+    @Test
+    public void testJdbcMysqlSourceAndSinkDataType() throws Exception {


Please check the method name.

liugddx · 2022-10-14T08:46:18Z

...nector-jdbc-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/jdbc/JdbcSqliteIT.java

+    }
+
+    @AfterEach
+    public void closeGreenplumContainer() throws SQLException, IOException {


Please check the method name.

thx, sloved.

nutsjian · 2022-10-15T15:10:59Z

@EricJoy2048 thx, man. I fixed the e2e test. Please approve running workflows.

…open error

hailin0 · 2022-10-18T03:35:13Z

...nector-jdbc-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcSqliteIT.java

+
+    private Connection jdbcConnection;
+
+    @Test


Remove @Test？

thx, I'll fix it.

Hisoka-X

LGTM

nutsjian · 2022-11-15T00:37:50Z

@EricJoy2048 please help me rerun ci workflows, if the unit test error has been resolved. thx.

2022-11-11T02:48:26.4091671Z [ERROR] Failed to execute goal on project connector-jdbc-spark-e2e: Could not resolve dependencies for project org.apache.seatunnel:connector-jdbc-spark-e2e:jar:2.1.3-SNAPSHOT: Failed to collect dependencies at org.testcontainers:postgresql:jar:1.17.3: Failed to read artifact descriptor for org.testcontainers:postgresql:jar:1.17.3: Could not transfer artifact org.testcontainers:postgresql:pom:1.17.3 from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/org/testcontainers/postgresql/1.17.3/postgresql-1.17.3.pom: Connection reset -> [Help 1]

Hisoka-X · 2022-11-15T02:35:41Z

Please resolve conflict, Thanks!

nutsjian · 2022-11-15T04:30:13Z

@Hisoka-X @EricJoy2048 conflict resolved, please help me rerun the CI workflows, thx.

liugddx

1.Update change log that in connector document. For more details you can refer to connector-v2

2.Update plugin-mapping.properties and add new connector information in it

3.Update the pom file of seatunnel-dist

liugddx · 2022-10-14T12:34:19Z

docs/en/connector-v2/sink/Jdbc.md

@@ -115,6 +115,7 @@ there are some reference value for params above.
 | phoenix    | org.apache.phoenix.queryserver.client.Driver | jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF | /                                                  | https://mvnrepository.com/artifact/com.aliyun.phoenix/ali-phoenix-shaded-thin-client |
 | sqlserver  | com.microsoft.sqlserver.jdbc.SQLServerDriver | jdbc:microsoft:sqlserver://localhost:1433                    | com.microsoft.sqlserver.jdbc.SQLServerXADataSource | https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc |
 | oracle     | oracle.jdbc.OracleDriver                     | jdbc:oracle:thin:@localhost:1521/xepdb1                      | oracle.jdbc.xa.OracleXADataSource                  | https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8 |
+| sqlite     | org.sqlite.JDBC                              | Jdbc:sqlite:test.db                                          | /                                                  | https://mvnrepository.com/artifact/org.xerial/sqlite-jdbc    |


Driver right?

liugddx · 2022-10-14T12:44:43Z

...nector-jdbc-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcSqliteIT.java

+    }
+
+    private void batchInsertData() throws SQLException {
+        String sql = "insert into source(age, name) values(?, ?)";


Can you test all data types?

liugddx · 2022-10-14T12:45:41Z

...nector-jdbc-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/jdbc/JdbcSqliteIT.java

+    }
+
+    private void initializeJdbcTable() throws Exception {
+        URI resource = Objects.requireNonNull(JdbcMysqlIT.class.getResource("/jdbc/init_sql/sqlite_init.conf")).toURI();


JdbcMysqlIT?

liugddx · 2022-10-14T12:45:57Z

...nector-jdbc-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/jdbc/JdbcSqliteIT.java

+    }
+
+    private void checkSinkDataTypeTable() throws Exception {
+        URI resource = Objects.requireNonNull(JdbcMysqlIT.class.getResource("/jdbc/init_sql/sqlite_init.conf")).toURI();


JdbcMysqlIT?

I've been fixed and commit before, is there any problem with my git operation steps, request for comments.

liugddx · 2022-10-14T12:46:07Z

...nector-jdbc-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcSqliteIT.java

+    }
+
+    private void initializeJdbcTable() throws Exception {
+        URI resource = Objects.requireNonNull(JdbcMysqlIT.class.getResource("/jdbc/init_sql/sqlite_init.conf")).toURI();


JdbcMysqlIT?

liugddx · 2022-10-15T15:12:07Z

...org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/sqlite/SqliteTypeMapper.java

+@Slf4j
+public class SqliteTypeMapper implements JdbcDialectTypeMapper {
+
+    private static final Logger LOG = LoggerFactory.getLogger(SqliteTypeMapper.class);


Useless code

liugddx · 2022-11-15T05:50:04Z

seatunnel-connectors-v2/connector-jdbc/pom.xml

+                <groupId>org.xerial</groupId>
+                <artifactId>sqlite-jdbc</artifactId>
+                <version>${sqlite.version}</version>
+                <scope>provided</scope>


duplicated code.

liugddx · 2022-11-15T05:52:22Z

...tor-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/JdbcSource.java

@@ -135,6 +137,14 @@ private SeaTunnelRowType initTableField(Connection conn) {
            }
        } catch (Exception e) {
            LOG.warn("get row type info exception", e);
+        } finally {
+            if (Objects.nonNull(resultSet)) {
+                try {


What is the role of this code?

liugddx · 2022-11-15T05:54:06Z

seatunnel-e2e/seatunnel-spark-connector-v2-e2e/pom.xml

@@ -36,6 +36,7 @@
        <module>connector-neo4j-spark-e2e</module>
        <module>connector-kafka-spark-e2e</module>
        <module>connector-iceberg-spark-e2e</module>
+        <module>connector-elasticsearch-spark-e2e</module>


Does this code belong to this PR?

liugddx · 2022-11-15T05:54:54Z

seatunnel-e2e/seatunnel-flink-connector-v2-e2e/pom.xml

@@ -36,6 +36,7 @@
        <module>connector-iceberg-flink-e2e</module>
        <module>connector-neo4j-flink-e2e</module>
        <module>connector-kafka-flink-e2e</module>
+        <module>connector-elasticsearch-flink-e2e</module>


Does this code belong to this PR?

Please add e2e in flink/spark e2e module. So that e2e can be start in CI.

thx man, done. I have merged upstream dev, and add flink/spark connector-elasticsearch-e2e module.

this code is for this comment. I merge upstream dev, and add elasticsearch-flink-e2e in pom.xml.

merge upstream dev

…nto jdbc-sqlite merge upstream dev

remove duplicate code in pom.xml, remove unnecessary code in JdbcSource.

Hisoka-X · 2022-11-19T07:28:34Z

Please fix ci problem. Thanks!

…aven and fix checkstyle error

…nt getUpsertStatement

Hisoka-X

LGTM

refactor JDBC[SqLite] connector

04e2395

liugddx reviewed Oct 13, 2022

View reviewed changes

hailin0 reviewed Oct 13, 2022

View reviewed changes

nutsjian added 2 commits October 14, 2022 15:08

[Improve] [JDBC] [SQLite] Connector test more field types

cf1ddc4

add [JDBC] [SQLite] doc

2981c1a

nutsjian requested review from hailin0 and liugddx and removed request for hailin0 and liugddx October 14, 2022 08:09

liugddx reviewed Oct 14, 2022

View reviewed changes

fix [JDBC] [SQLite] e2e test case method name

0643e4a

nutsjian requested a review from liugddx October 14, 2022 09:47

[fix] class name error

f3de959

EricJoy2048 added the First-time contributor First-time contributor label Oct 15, 2022

[Hotfix] [JDBC] [SQLite] connector e2e test path problem

6f3b792

nutsjian requested review from hailin0 and removed request for liugddx October 16, 2022 01:17

[fix] [JDBC] [SQLite] flink e2e test docker container db file cannot …

e4bf558

…open error

hailin0 reviewed Oct 18, 2022

View reviewed changes

Hisoka-X added the connectors-v2 label Oct 18, 2022

Hisoka-X previously approved these changes Nov 11, 2022

View reviewed changes

github-actions bot added approved reviewed labels Nov 11, 2022

Merge branch 'dev' into jdbc-sqlite

70494ae

nutsjian dismissed Hisoka-X’s stale review via 70494ae November 15, 2022 04:28

github-actions bot removed approved reviewed labels Nov 15, 2022

[checkstyle] remove unused import

03fba9f

liugddx reviewed Nov 15, 2022

View reviewed changes

nutsjian and others added 5 commits November 15, 2022 17:40

Merge remote-tracking branch 'upstream' into jdbc-sqlite

976c5a6

merge upstream dev

Merge branch 'dev' of https://github.com/apache/incubator-seatunnel i…

6197196

…nto jdbc-sqlite merge upstream dev

update [JDBC] [SQLite] connector

d03bf4d

remove duplicate code in pom.xml, remove unnecessary code in JdbcSource.

Merge branch 'dev' into jdbc-sqlite

5ad4b4c

Merge branch 'dev' into jdbc-sqlite

fa00b72

nutsjian and others added 3 commits November 22, 2022 10:11

Merge branch 'dev' into jdbc-sqlite

5c0b7e8

fix [JDBC] [SQLite] connector, remove root pom.xml dup net.alchim31.m…

f4d4d33

…aven and fix checkstyle error

update [Connector] [JDBC] SQLite, remove typeAffinity option, impleme…

3204610

…nt getUpsertStatement

Hisoka-X approved these changes Nov 22, 2022

View reviewed changes

github-actions bot added approved reviewed labels Nov 22, 2022

Hisoka-X requested review from liugddx and hailin0 November 22, 2022 05:52

EricJoy2048 approved these changes Nov 24, 2022

View reviewed changes

EricJoy2048 merged commit a73bb3e into apache:dev Nov 24, 2022

EricJoy2048 mentioned this pull request Nov 24, 2022

[Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018

Open

3 tasks

zhilinli123 mentioned this pull request Apr 11, 2023

[Feature][Connector-V2][Docs] Unified format for connector-v2 docs #4544

Open

2 tasks

[Feature][Connector-V2][JDBC] support sqlite Source & Sink #3089

[Feature][Connector-V2][JDBC] support sqlite Source & Sink #3089

Conversation

nutsjian commented Oct 13, 2022 • edited

Purpose of this pull request

Check list

liugddx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liugddx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nutsjian Oct 14, 2022 • edited

Choose a reason for hiding this comment

nutsjian commented Oct 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X left a comment

Choose a reason for hiding this comment

nutsjian commented Nov 15, 2022

Hisoka-X commented Nov 15, 2022

nutsjian commented Nov 15, 2022

liugddx left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X commented Nov 19, 2022

Hisoka-X left a comment

Choose a reason for hiding this comment

nutsjian commented Oct 13, 2022 •

edited

nutsjian Oct 14, 2022 •

edited

liugddx left a comment •

edited