[Bug] update flink version to 1.16.2and CDC version to 2.3.0:NoSuchMethodError

[zhegemingzimeibanquan]

Search before asking

• I searched in the issues and found nothing similar.

Flink version

1.16.2

Flink CDC version

2.3.0

Database and its version

mysql 8

Minimal reproduce step

start flink-sql client
try to build a cdc database connect and table
run select * from the table

What did you expect to see?

run cdc success

What did you see instead?

first is a Series of ClassNotFoundException，then i put the missing jar into ./lib
finally ,it tells:
Caused by: java.lang.NoSuchMethodError: io.debezium.connector.mysql.MySqlConnection$MySqlConnectionConfiguration.(Lio/debezium/config/Configuration;Ljava/util/Properties;)V

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[zhegemingzimeibanquan]

i try to downgrade the version of flinkcdc,it works on flink 1.13.6 and cdc 2.0.2,but i need to use flink 1.16

[foxmorg]

The same issue for me:

Caused by: java.lang.NoSuchMethodError: 'void io.debezium.connector.mysql.MySqlConnection$MySqlConnectionConfiguration.<init>(io.debezium.config.Configuration, java.util.Properties)' at com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.createMySqlConnection(DebeziumUtils.java:78) at com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.createMySqlConnection(DebeziumUtils.java:71) at com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader.checkSplitOrStartNext(MySqlSplitReader.java:138) at com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader.fetch(MySqlSplitReader.java:71) at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142) at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 more

Reproduces on CDC 2.3.0 and 2.4.0 versions.

As I understand, the reason because as result we have two dependencies in runtime classpath :
debezium-connector-mysql-1.6.4.Final.jar (1.9.7.Final for cdc 2.4.0)
flink-connector-mysql-cdc-2.3.0.jar

with the same class io.debezium.connector.mysql.MySqlConnection$MySqlConnectionConfiguration but first one with only one constructor.

Issue exists since this changes were merged:
#1217

So the question is, why was this class renamed in this commit 5415bcf and how fix this issue without shading class from debezium-connector-mysql dependency?

Unfortunately I couldn't just exclude transitive dependency debezium-connector-mysql because it's a lot of classes which are not exists in flink-connector-mysql-cdc so trying to exclude them we'll receive another one exception - NoClassDefFoundError

[altairbow]

两个方法：

1. 把 debezium-connector-mysql-1.9.7.Final.jar 改名，比如 flink-dep-debezium-connector-mysql-1.9.7.Final.jar，目的是让 JVM 优先找到 flink cdc 内与 debezium 同包同名的类。
2. 把 flink-connector-mysql-cdc-2.4.1.jar 内的 debezium 包覆盖到 debezium-connector-mysql-1.9.7.Final.jar。

第二个方法亲测有效，第一个没试。

[gulunhui]

我已经收到你的邮件啦~~~~

[a1643977841]

两个方法：

1. 把 改名，比如 ，目的是让 JVM 优先找到 flink cdc 内与 debezium 同包同名的类。debezium-connector-mysql-1.9.7.Final.jar``flink-dep-debezium-connector-mysql-1.9.7.Final.jar
2. 把 内的 debezium 包覆盖到 。flink-connector-mysql-cdc-2.4.1.jar``debezium-connector-mysql-1.9.7.Final.jar

第二个方法亲测有效，第一个没试。

你好可以详细说说吗

[gulunhui]

我已经收到你的邮件啦~~~~

[zhegemingzimeibanquan]

在新版本里面，cdc的jar包和cdc-sql的jar包是两个jar包，如果需要sql-client去运行cdc程序，需要在lib里面加入cdc-sql的jar包

[gulunhui]

我已经收到你的邮件啦~~~~