[connector-mysql] Allow user to pass custom JDBC URL parameters used … #921

paul8263 · 2022-03-10T02:46:32Z

…by MySQL data source.

This issue is linked to #674 and #622 .

Added 'connection.pool.url.properties' for mysql table source in order to allow user to customize MySQL JDBC URL parameters.

Change log：

docs/content/connectors/mysql-cdc.md
flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/MySqlSourceBuilder.java
flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/config/MySqlSourceConfig.java
flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/config/MySqlSourceConfigFactory.java
flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/config/MySqlSourceOptions.java
flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/connection/PooledDataSourceFactory.java
flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/table/MySqlTableSource.java
flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/table/MySqlTableSourceFactory.java
flink-connector-mysql-cdc/src/test/java/com/ververica/cdc/connectors/mysql/table/MySqlTableSourceFactoryTest.java

leonardBang · 2022-03-11T06:54:00Z

.../main/java/com/ververica/cdc/connectors/mysql/source/connection/PooledDataSourceFactory.java


        config.setPoolName(CONNECTION_POOL_PREFIX + hostName + ":" + port);
-        config.setJdbcUrl(String.format(JDBC_URL_PATTERN, hostName, port));
+        config.setJdbcUrl(formatJdbcUrlPattern(hostName, port, customProperties));


Could we pass the jdbc url properties like Kafka connector properties?

jdbc.properties.useInformationSchema=ture jdbc.properties.characterEncoding=UTF-8 ....

https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/#properties-bootstrap-servers
In this way we can override all parameters in the JDB_URL

leonardBang · 2022-03-11T09:46:10Z

.../main/java/com/ververica/cdc/connectors/mysql/source/connection/PooledDataSourceFactory.java

 /** A connection pool factory to create pooled DataSource {@link HikariDataSource}. */
 public class PooledDataSourceFactory {

    public static final String JDBC_URL_PATTERN =
-            "jdbc:mysql://%s:%s/?useInformationSchema=true&nullCatalogMeansCurrent=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=CONVERT_TO_NULL";
+            "jdbc:mysql://%s:%s/?useInformationSchema=true&nullCatalogMeansCurrent=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8";


Could we let user can also override characterEncoding parameter? we can save these default parameter in a default properties and then merge user passed custom properties as the final properties. HDYT?

Hi @leonardBang ,
Thanks for reviewing. I will do it.

leonardBang · 2022-03-14T06:22:02Z

@paul8263 The CI failed, please have a look.

leonardBang · 2022-03-14T09:30:04Z

.../main/java/com/ververica/cdc/connectors/mysql/source/connection/PooledDataSourceFactory.java

 /** A connection pool factory to create pooled DataSource {@link HikariDataSource}. */
 public class PooledDataSourceFactory {

    public static final String JDBC_URL_PATTERN =
-            "jdbc:mysql://%s:%s/?useInformationSchema=true&nullCatalogMeansCurrent=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=CONVERT_TO_NULL";
+            "jdbc:mysql://%s:%s/?useInformationSchema=true&nullCatalogMeansCurrent=false&useUnicode=true&characterSetResults=UTF-8";


Let parameters&useUnicode=true&characterSetResults=UTF-8 can also be overwrite?

leonardBang · 2022-03-14T09:31:20Z

.../main/java/com/ververica/cdc/connectors/mysql/source/connection/PooledDataSourceFactory.java

+    static {
+        DEFAULT_JDBC_PROPERTIES = new Properties();
+        DEFAULT_JDBC_PROPERTIES.setProperty("zeroDateTimeBehavior", "CONVERT_TO_NULL");
+        DEFAULT_JDBC_PROPERTIES.setProperty("characterEncoding", "UTF-8");
+    }


A static method is more readable than static code piece

leonardBang · 2022-03-14T09:32:06Z

...-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/table/JDBCOptions.java

+import java.util.Map;
+import java.util.Properties;
+
+/** Option utils for JDBC options. */


Suggested change

/** Option utils for JDBC options. */

/** Option utils for JDBC URL properties. */

leonardBang · 2022-03-14T09:33:28Z

...-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/table/JDBCOptions.java

+import java.util.Properties;
+
+/** Option utils for JDBC options. */
+public class JDBCOptions {


Suggested change

public class JDBCOptions {

public class JdbcUrlUtils {

Er... Should the name keep consistent with DebeziumOptions?

we do not need to follow them as this class is a utils class instead of a set of options.

leonardBang · 2022-03-14T09:39:49Z

docs/content/connectors/mysql-cdc.md

+          <td>optional</td>
+          <td style="word-wrap: break-word;">20</td>
+          <td>String</td>
+          <td>Custom JDBC URL properties. Default value is 'zeroDateTimeBehavior=CONVERT_TO_NULL'.</td>


please update the description, we have no default value for custom properties, right?
We can say :
Option to pass custom JDBC URL properties, user can pass custom property like 'jdbc.properties.useSSL'= 'false'.

leonardBang · 2022-03-14T09:40:19Z

...or-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/MySqlSourceBuilder.java

@@ -202,6 +202,12 @@
        return this;
    }

+    /** Custom properties that will append to the JDBC connection URL. */


Suggested change

/** Custom properties that will append to the JDBC connection URL. */

/** Custom properties that will overwrite the default JDBC connection URL. */

leonardBang · 2022-03-14T09:42:10Z

...src/main/java/com/ververica/cdc/connectors/mysql/source/config/MySqlSourceConfigFactory.java

@@ -214,6 +215,12 @@ public MySqlSourceConfigFactory scanNewlyAddedTableEnabled(boolean scanNewlyAdde
        return this;
    }

+    /** Custom properties that will append to the JDBC connection URL. */


/** Custom properties that will overwrite the default JDBC connection URL. */

leonardBang · 2022-03-14T09:44:13Z

...-cdc/src/test/java/com/ververica/cdc/connectors/mysql/table/MySqlTableSourceFactoryTest.java

-                        StartupOptions.initial());
+                        StartupOptions.initial(),
+                        false,
+                        new Properties());


Cover the change in testOptionalProperties() ?

paul8263 · 2022-03-15T01:37:07Z

Hi @leonardBang ,
I am confused why the CI got a series of errors like Generate Splits for table xxx.xxx error.
It seems that flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner::getNext() method encountered a NPE.

leonardBang · 2022-03-15T02:15:58Z

Hi @leonardBang , I am confused why the CI got a series of errors like Generate Splits for table xxx.xxx error. It seems that flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner::getNext() method encountered a NPE.

It's wired, but we need rebase the master to resolve the conflicts

…by MySQL data source.

paul8263 · 2022-03-15T02:22:47Z

Hi @leonardBang , I am confused why the CI got a series of errors like Generate Splits for table xxx.xxx error. It seems that flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner::getNext() method encountered a NPE.

It's wired, but we need rebase the master to resolve the conflicts

Thanks. I resolved the conflicts and I am waiting to see if the weird thing still exists.

leonardBang

LGTM

leonardBang reviewed Mar 11, 2022

View reviewed changes

paul8263 force-pushed the patch-674 branch from 7dd72c0 to 4daf912 Compare March 11, 2022 09:25

leonardBang reviewed Mar 11, 2022

View reviewed changes

paul8263 force-pushed the patch-674 branch from 4daf912 to 3b32e5f Compare March 14, 2022 01:10

paul8263 force-pushed the patch-674 branch from 3b32e5f to 17dac20 Compare March 14, 2022 09:06

leonardBang reviewed Mar 14, 2022

View reviewed changes

paul8263 force-pushed the patch-674 branch from 17dac20 to 7fd6378 Compare March 15, 2022 02:16

[connector-mysql] Allow user to pass custom JDBC URL parameters used …

021543b

…by MySQL data source.

paul8263 force-pushed the patch-674 branch from 7fd6378 to 021543b Compare March 15, 2022 02:20

leonardBang approved these changes Mar 15, 2022

View reviewed changes

leonardBang merged commit 580391b into apache:master Mar 15, 2022

whhe mentioned this pull request Mar 23, 2022

[oceanbase] Allow user to pass custom JDBC URL parameters #983

Closed

lvyanquan mentioned this pull request Sep 20, 2023

Allow custom Configurations for ConnectionPool #2509

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[connector-mysql] Allow user to pass custom JDBC URL parameters used … #921

[connector-mysql] Allow user to pass custom JDBC URL parameters used … #921

paul8263 commented Mar 10, 2022

leonardBang Mar 11, 2022

leonardBang Mar 11, 2022

paul8263 Mar 14, 2022

leonardBang commented Mar 14, 2022

leonardBang Mar 14, 2022

leonardBang Mar 14, 2022

leonardBang Mar 14, 2022

leonardBang Mar 14, 2022

paul8263 Mar 15, 2022

leonardBang Mar 15, 2022

leonardBang Mar 14, 2022

leonardBang Mar 14, 2022

leonardBang Mar 14, 2022

leonardBang Mar 14, 2022

paul8263 commented Mar 15, 2022

leonardBang commented Mar 15, 2022

paul8263 commented Mar 15, 2022

leonardBang left a comment

	/** Option utils for JDBC options. */
	/** Option utils for JDBC URL properties. */

	/** Custom properties that will append to the JDBC connection URL. */
	/** Custom properties that will overwrite the default JDBC connection URL. */

[connector-mysql] Allow user to pass custom JDBC URL parameters used … #921

[connector-mysql] Allow user to pass custom JDBC URL parameters used … #921

Conversation

paul8263 commented Mar 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leonardBang commented Mar 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paul8263 commented Mar 15, 2022

leonardBang commented Mar 15, 2022

paul8263 commented Mar 15, 2022

leonardBang left a comment

Choose a reason for hiding this comment