[FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation #20193

bzhaoopenstack · 2022-07-07T07:18:02Z

What is the purpose of the change

Flink Connector support the jdbc connection with Mysql series DB, such as MySQL, MariaDB,
Percona and etc. But we see the doc reference[1] shows the Mariadb Connector/J 3.0 will
accepts jdbc:mariadb: as the protocal in connection strings by default. So we need to apply
this change in our mysql jdbc connection vailidation. Mariadb Connector/J 2.X still support
both "jdbc:mysql:" and "jdbc:mariadb:".
Introduce MariaDB dialect implemented like Mysql.
Add UTs for DataTypes and a testcontainer for MariaDB in
JdbcExactlyOnceSinkE2eTest.

[1] https://mariadb.com/kb/en/about-mariadb-connector-j/#jdbcmysql-scheme-compatibility

Brief change log

Making Mysql jdbc connector validate the MariaDB type connection protocol for fit users to use Mysql jdbc connector to access MariaDB instances.
Introduce MariaDB jdbc dialect into Flink, which is very similar with Mysql.
Introduce UTs and E2E test for them.

Verifying this change

Exec mvn clean package -Dfast -Pskip-webui-build -pl flink-connectors/flink-connector-jdbc
Or mvn clean package -Dfast -Pskip-webui-build -pl flink-connectors/flink-connector-jdbc -Dtest="org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest" for run all tests or a single test.
Notes that all types didn't cover, and will be on going for support.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (docs)

flinkbot · 2022-07-07T07:24:11Z

CI report:

ecea05c Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

Peng-Lei · 2022-07-07T07:43:15Z

+1， After the change. We can use mariadb driver to connect the mysql through by setting the jdbcOptions of driver parameter without change the url prefix.

MartijnVisser

@bzhaoopenstack Thanks for the PR, but in order to add support for MariaDB I would also like to see a test and documentation in place.

bzhaoopenstack · 2022-07-08T00:56:29Z

@bzhaoopenstack Thanks for the PR, but in order to add support for MariaDB I would also like to see a test and documentation in place.

Thanks very much for reply and suggest. I will add the test and documentation. Once I'm finish, I think I need your kind review and advice. Thank you. ;-)

bzhaoopenstack · 2022-07-08T08:26:45Z

@bzhaoopenstack Thanks for the PR, but in order to add support for MariaDB I would also like to see a test and documentation in place.

Hi @MartijnVisser, I have a question here, you said you want a test for this PR, does it mean we should add a specific UT which only verify the jdbc conncetion strings? Because I didn't find any existing UT in Flink.
And notes: We only use Mysql connector to connect MariaDB, as it is a Mysql-series database.

MartijnVisser · 2022-07-08T11:40:27Z

I would expect that we add MariaDB to the JdbcExactlyOnceSinkE2eTest with the correct test container and have a MariaDB dialect implemented (even if that would be just a copy of MySQL)

bzhaoopenstack · 2022-07-11T00:30:00Z

I would expect that we add MariaDB to the JdbcExactlyOnceSinkE2eTest with the correct test container and have a MariaDB dialect implemented (even if that would be just a copy of MySQL)

Thanks for explain more, this is a good direction/suggestion for me.

bzhaoopenstack · 2022-07-12T09:42:10Z

Hi @MartijnVisser , sorry for late.
And this is my first PR in flink, so I wish you could leave some kind directions and suggestions. Thank you very much. ;-). And hope this doesn't disturb you so much.

Now, I will request an new review from your side, once we both think this PR is ready, I will remove the title [WIP]. ;-)

Peng-Lei · 2022-07-12T09:51:53Z

...or-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/mysql/MySqlDialectFactory.java

@@ -27,7 +27,7 @@
 public class MySqlDialectFactory implements JdbcDialectFactory {
    @Override
    public boolean acceptsURL(String url) {
-        return url.startsWith("jdbc:mysql:");
+        return (url.startsWith("jdbc:mysql:") || url.startsWith("jdbc:mariadb:"));


I think no need change here after we add MariaDBDialectFactory

MartijnVisser · 2022-07-13T15:16:24Z

@afedulov Do you want to have a look at this one?

bzhaoopenstack · 2022-07-18T01:30:16Z

Hi all. @MartijnVisser @hadronzoo any updates on this one? Thanks

Peng-Lei · 2022-07-19T08:58:47Z

@bzhaoopenstack Are you ready？If yes, please change the title without "WIP".

bzhaoopenstack · 2022-07-19T11:52:43Z

@bzhaoopenstack Are you ready？If yes, please change the title without "WIP".

Yeah, I'm ready for the kind suggest from @MartijnVisser and @hadronzoo . ;-)

MartijnVisser · 2022-08-01T13:32:21Z

@bzhaoopenstack I won't be able to review this at the moment, but keep in mind that the CI is still failing for this PR. It makes sense to make sure that the CI works before the review actually starts.

bzhaoopenstack · 2022-08-01T14:49:28Z

@bzhaoopenstack I won't be able to review this at the moment, but keep in mind that the CI is still failing for this PR. It makes sense to make sure that the CI works before the review actually starts.

OK, thank you Martijn. I will resolve it and make it ready for review

bzhaoopenstack · 2022-08-22T03:57:56Z

Sorry for late update. ;-) . Let's wait the azure test result. I got pass on my local env.

1. Flink Connector support the jdbc connection with Mysql series DB, such as MySQL, MariaDB, Percona and etc. But we see the doc reference[1] shows the Mariadb Connector/J 3.0 will accepts jdbc:mariadb: as the protocal in connection strings by default. So we need to apply this change in our mysql jdbc connection vailidation. Mariadb Connector/J 2.X still support both "jdbc:mysql:" and "jdbc:mariadb:". 2. Introduce MariaDB dialect implemented like Mysql. 3. Add UTs for DataTypes and a testcontainer for MariaDB in JdbcExactlyOnceSinkE2eTest. [1] https://mariadb.com/kb/en/about-mariadb-connector-j/#jdbcmysql-scheme-compatibility

bzhaoopenstack · 2022-08-24T04:15:13Z

@flinkbot run azure

MartijnVisser · 2022-11-29T14:56:12Z

We've now moved the code from the JDBC connector to its own dedicated repository; please re-route this PR to https://github.com/apache/flink-connector-jdbc

bzhaoopenstack force-pushed the FLINK-28433 branch from deb8d27 to 5d26bcc Compare July 7, 2022 07:59

flinkbot added the component=Connectors/JDBC label Jul 7, 2022

MartijnVisser requested changes Jul 7, 2022

View reviewed changes

MartijnVisser self-assigned this Jul 7, 2022

bzhaoopenstack changed the title ~~[FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation~~ [WIP][FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation Jul 8, 2022

bzhaoopenstack force-pushed the FLINK-28433 branch from 5d26bcc to be1cf1e Compare July 12, 2022 09:26

bzhaoopenstack requested a review from MartijnVisser July 12, 2022 09:42

Peng-Lei reviewed Jul 12, 2022

View reviewed changes

bzhaoopenstack force-pushed the FLINK-28433 branch 2 times, most recently from 0bb91f2 to 3df6dd0 Compare August 2, 2022 03:38

bzhaoopenstack force-pushed the FLINK-28433 branch from 3df6dd0 to 6b782d7 Compare August 22, 2022 03:56

bzhaoopenstack force-pushed the FLINK-28433 branch from 6b782d7 to e30daa9 Compare August 22, 2022 07:31

bzhaoopenstack force-pushed the FLINK-28433 branch from e30daa9 to ecea05c Compare August 23, 2022 04:03

bzhaoopenstack changed the title ~~[WIP][FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation~~ [FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation Aug 24, 2022

MartijnVisser closed this Nov 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation #20193

[FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation #20193

bzhaoopenstack commented Jul 7, 2022 •

edited

flinkbot commented Jul 7, 2022 •

edited

Peng-Lei commented Jul 7, 2022

MartijnVisser left a comment

bzhaoopenstack commented Jul 8, 2022

bzhaoopenstack commented Jul 8, 2022

MartijnVisser commented Jul 8, 2022

bzhaoopenstack commented Jul 11, 2022

bzhaoopenstack commented Jul 12, 2022

Peng-Lei Jul 12, 2022

MartijnVisser commented Jul 13, 2022

bzhaoopenstack commented Jul 18, 2022

Peng-Lei commented Jul 19, 2022

bzhaoopenstack commented Jul 19, 2022

MartijnVisser commented Aug 1, 2022

bzhaoopenstack commented Aug 1, 2022

bzhaoopenstack commented Aug 22, 2022

bzhaoopenstack commented Aug 24, 2022

MartijnVisser commented Nov 29, 2022

[FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation #20193

[FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation #20193

Conversation

bzhaoopenstack commented Jul 7, 2022 • edited

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Jul 7, 2022 • edited

CI report:

Peng-Lei commented Jul 7, 2022

MartijnVisser left a comment

Choose a reason for hiding this comment

bzhaoopenstack commented Jul 8, 2022

bzhaoopenstack commented Jul 8, 2022

MartijnVisser commented Jul 8, 2022

bzhaoopenstack commented Jul 11, 2022

bzhaoopenstack commented Jul 12, 2022

Peng-Lei Jul 12, 2022

Choose a reason for hiding this comment

MartijnVisser commented Jul 13, 2022

bzhaoopenstack commented Jul 18, 2022

Peng-Lei commented Jul 19, 2022

bzhaoopenstack commented Jul 19, 2022

MartijnVisser commented Aug 1, 2022

bzhaoopenstack commented Aug 1, 2022

bzhaoopenstack commented Aug 22, 2022

bzhaoopenstack commented Aug 24, 2022

MartijnVisser commented Nov 29, 2022

bzhaoopenstack commented Jul 7, 2022 •

edited

flinkbot commented Jul 7, 2022 •

edited