Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[mysql->doris]JdbcInputFormat.getMaxValueFromDb Faild And Instantiation classLoader of Source Flink-CDC-MySQL Faild #184

Closed
baisui1981 opened this issue Feb 27, 2023 · 2 comments
Closed

[mysql->doris]JdbcInputFormat.getMaxValueFromDb Faild And Instantiation classLoader of Source Flink-CDC-MySQL Faild #184

baisui1981 opened this issue Feb 27, 2023 · 2 comments
Labels
3.7.1 bug Something isn't working
Milestone
v3.7.1

Comments

@baisui1981
Copy link
Member

baisui1981 commented Feb 27, 2023 •
edited

问题背景：参考袋鼠云视频一步步完成安装、部署；批量构建没有问题，实时同步接入以下两种插件都出现异常，@百岁 麻烦帮忙看下，不胜感激！

1、Flink-Chunjun-MySQL -> Chunjun-Sink-Doris

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Get max value from bmp_log error
  at com.dtstack.chunjun.connector.jdbc.source.JdbcInputFormat.getMaxValueFromDb(JdbcInputFormat.java:545)
  at com.dtstack.chunjun.connector.jdbc.source.JdbcInputFormat.createSplitsInternalBySplitMod(JdbcInputFormat.java:197)
  at com.dtstack.chunjun.connector.jdbc.source.JdbcInputFormat.createInputSplitsInternal(JdbcInputFormat.java:184)
  at com.dtstack.chunjun.source.format.BaseRichInputFormat.createInputSplits(BaseRichInputFormat.java:133)

2、Flink-CDC-MySQL -> Chunjun-Sink-Doris

Class not resolvable through given classloader.
  at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:336)
  at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:154)
  at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:548)
  at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
  at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537)
  at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.ververica.cdc.debezium.DebeziumSourceFunction
accepted plugins:tis-flink-chunjun-doris-plugin,tis-ds-mysql-plugin,tis-k8s-plugin,tis-flink-dependency,tis-ds-mysql-v8-plugin,tis-realtime-flink,tis-datax-doris-starrocks-plugin,tis-flink-cdc-mysql-plugin,tis-datax-common-rdbms-plugin,tis-chunjun-base-plugin,tis-datax-common-plugin,flink/lalala
,scan plugins:tis-datax-common-plugin,tis-datax-common-rdbms-plugin,tis-ds-mysql-plugin,tis-k8s-plugin,tis-datax-doris-starrocks-plugin,tis-flink-dependency,tis-ds-mysql-v8-plugin,tis-realtime-flink,tis-chunjun-base-plugin,tis-flink-chunjun-mysql-plugin,tis-flink-chunjun-doris-plugin

background：

Source Mysql8.0
Sink Doris-1.2.2/Mysql5.7
tis 3.6.0-alpha

@张三丰
@baisui1981 baisui1981 added bug Something isn't working 3.7.1 labels Feb 27, 2023
@baisui1981 baisui1981 changed the title JdbcInputFormat.getMaxValueFromDb Faild And Instantiation classLoader of Source Flink-CDC-MySQL Faild [mysql->doris]JdbcInputFormat.getMaxValueFromDb Faild And Instantiation classLoader of Source Flink-CDC-MySQL Faild Mar 14, 2023
@baisui1981
Copy link
Member Author

baisui1981 commented Mar 14, 2023 •
edited

debezium中连接 mysql8服务端报

Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
org.apache.flink.table.api.TableException: Unexpected error while connecting to MySQL and validating
at com.ververica.cdc.connectors.mysql.MySqlValidator.validate(MySqlValidator.java:77) ~[?:?]
        at com.ververica.cdc.debezium.DebeziumSourceFunction.open(DebeziumSourceFunction.java:215) ~[?:?]
        at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34) ~[flink-dist_2.11-tis-1.13.1.jar:tis-1.13.1]
        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) ~[flink-dist_2.11-tis-1.13.1.jar:tis-1.13.1]
        at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:437) ~[flink-dist_2.11-tis-1.13.1.jar:tis-1.13.1]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:582) ~[flink-dist_2.11-tis-1.13.1.jar:tis-1.13.1]
        at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100) ~[flink-dist_2.11-tis-1.13.1.jar:tis-1.13.1]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:562) ~[flink-dist_2.11-tis-1.13.1.jar:tis-1.13.1]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647) ~[flink-dist_2.11-tis-1.13.1.jar:tis-1.13.1]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537) ~[flink-dist_2.11-tis-1.13.1.jar:tis-1.13.1]
        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759) ~[flink-dist_2.11-tis-1.13.1.jar:tis-1.13.1]
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.11-tis-1.13.1.jar:tis-1.13.1]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_251]
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure


The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174) ~[?:?]
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64) ~[?:?]
        at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:833) ~[?:?]
        at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:453) ~[?:?]
        at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246) ~[?:?]
        at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) ~[?:?]
        at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:231) ~[?:?]
        at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:872) ~[?:?]
        at io.debezium.connector.mysql.MySqlConnection.connection(MySqlConnection.java:79) ~[?:?]
        at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:867) ~[?:?]
        at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:638) ~[?:?]
        at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:512) ~[?:?]
        at com.ververica.cdc.connectors.mysql.MySqlValidator.checkVersion(MySqlValidator.java:93) ~[?:?]
        at com.ververica.cdc.connectors.mysql.MySqlValidator.validate(MySqlValidator.java:73) ~[?:?]
        ... 12 more

create table DDL

CREATE TABLE `bmp_log` (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT 'ID',
  `menu` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL COMMENT '菜单',
  `opt` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL COMMENT '操作',
  `uri` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL COMMENT '资源路径',
  `crt_time` datetime DEFAULT NULL COMMENT '操作时间',
  `crt_user` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL COMMENT '操作人ID',
  `crt_name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL COMMENT '操作人',
  `crt_host` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL COMMENT '操作主机',
  `body` text CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci COMMENT '请求参数',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=232356 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

@baisui1981 baisui1981 mentioned this issue Mar 29, 2023
Closed
@baisui1981
Copy link
Member Author

fixed

@baisui1981 baisui1981 added this to the v3.7.1 milestone Apr 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.7.1 bug Something isn't working
Projects
None yet
Milestone
v3.7.1
Development

No branches or pull requests
1 participant
@baisui1981