Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] There is a problem using mysql-to-mysql with sort #9199

Closed
2 tasks done
LiJie20190102 opened this issue Nov 2, 2023 · 6 comments
Closed
2 tasks done

[Bug] There is a problem using mysql-to-mysql with sort #9199

LiJie20190102 opened this issue Nov 2, 2023 · 6 comments
Assignees
Labels
stage/stale Issues or PRs that had no activity for a long time type/bug Something is wrong

Comments

@LiJie20190102
Copy link
Contributor

What happened

The data synchronization task I created is from MySQL to MySQL. I use the command line to operate sort, and the database information is correct. However, I see that the jobmanager log on Flink shows that I cannot find the source table I want. Do I need to configure any filtering conditions to do this

mysql2mysql.sql

CREATE TABLE `table_11`(
    `name` STRING)
    WITH (
    'connector' = 'mysql-cdc-inlong',
    'hostname' = '192.168.1.133',
    'database-name' = 'inlong_test',
    'port' = '13306',
    'username' = 'root',
    'password' = 'qsdi1@3',
    'table-name' = 'source_test'
);

CREATE TABLE `table_22`(
    PRIMARY KEY (`name`) NOT ENFORCED,
    `name` STRING)
    WITH (
'connector' = 'jdbc-inlong',
    'url' = 'jdbc:mysql://192.168.1.133:13306/inlong_test',
    'username' = 'root',
    'password' = 'qsdi1@3',
    'table-name' = 'target'
);

INSERT INTO `table_22`
    SELECT
    `name` AS `name`
    FROM `table_11`;

CLI:
/opt/software/flink/flink-1.13.6/bin/flink run -c org.apache.inlong.sort.Entrance /usr/lib/inlong/inlong-sort/sort-dist-1.9.0.jar --sql.script.file /home/lj/inlong/mysql2mysql.sql

flink jobmanager log:
image
The log shows that there is actually this table, but it has been filtered out.
image

image

What you expected to happen

Can correctly synchronize MySQL to MySQL data

How to reproduce

mysql2mysql.sql

CREATE TABLE `table_11`(
    `name` STRING)
    WITH (
    'connector' = 'mysql-cdc-inlong',
    'hostname' = '192.168.1.133',
    'database-name' = 'inlong_test',
    'port' = '13306',
    'username' = 'root',
    'password' = 'qsdi1@3',
    'table-name' = 'source_test'
);

CREATE TABLE `table_22`(
    PRIMARY KEY (`name`) NOT ENFORCED,
    `name` STRING)
    WITH (
'connector' = 'jdbc-inlong',
    'url' = 'jdbc:mysql://192.168.1.133:13306/inlong_test',
    'username' = 'root',
    'password' = 'qsdi1@3',
    'table-name' = 'target'
);

INSERT INTO `table_22`
    SELECT
    `name` AS `name`
    FROM `table_11`;

CLI:
/opt/software/flink/flink-1.13.6/bin/flink run -c org.apache.inlong.sort.Entrance /usr/lib/inlong/inlong-sort/sort-dist-1.9.0.jar --sql.script.file /home/lj/inlong/mysql2mysql.sql

Environment

No response

InLong version

1.9.0

InLong Component

InLong Sort

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

@LiJie20190102 LiJie20190102 added the type/bug Something is wrong label Nov 2, 2023
Copy link

github-actions bot commented Nov 2, 2023

Hello @LiJie20190102, thank you for opening your first issue in InLong 🧡 We will respond as soon as possible ⏳
If this is a bug report, please provide screenshots or error logs for us to reproduce your issue, so we can do our best to fix it.
If you have any questions in the meantime, you can also ask us on the InLong Discussions 🔍

@dockerzhang
Copy link
Contributor

@EMsnap PTAL

@yunqingmoswu
Copy link
Contributor

@LiJie20190102 You need to config the table name with a database such as 'database.table'. It is 'inlong_test .source_test'
in your demo.

image

@LiJie20190102
Copy link
Contributor Author

LiJie20190102 commented Nov 2, 2023

@yunqingmoswu @EMsnap
Do I still need to add a library name to the target table? After I added it, there was an exception in the jobmanager. The job is still running, but there is no data in the target table.
image
image

@LiJie20190102
Copy link
Contributor Author

Functionally functional, the complete file is as follows:

CREATE TABLE table_11(
id bigint,
PRIMARY KEY(id) NOT ENFORCED
)
WITH (
'connector' = 'mysql-cdc-inlong',
'hostname' = '192.168.1.133',
'database-name' = 'inlong_test',
'port' = '13306',
'username' = 'root',
'password' = 'qsdi1@3',
'table-name' = 'inlong_test.source1'
);

CREATE TABLE table_22(
PRIMARY KEY (id) NOT ENFORCED,
id bigint)
WITH (
'connector' = 'jdbc-inlong',
'url' = 'jdbc:mysql://192.168.1.133:13306/inlong_test',
'username' = 'root',
'password' = 'qsdi1@3',
'table-name' = 'target1'
);

INSERT INTO table_22
SELECT
id AS id
FROM table_11;

Copy link

github-actions bot commented Jan 6, 2024

This issue is stale because it has been open for 60 days with no activity.

@github-actions github-actions bot added the stage/stale Issues or PRs that had no activity for a long time label Jan 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stage/stale Issues or PRs that had no activity for a long time type/bug Something is wrong
Projects
None yet
Development

No branches or pull requests

4 participants