Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Sort] Import sort end2end unit test with sql file input #4763

Closed
2 tasks done
Tracked by #4762
thesumery opened this issue Jun 24, 2022 · 0 comments · Fixed by #4765
Closed
2 tasks done
Tracked by #4762

[Feature][Sort] Import sort end2end unit test with sql file input #4763

thesumery opened this issue Jun 24, 2022 · 0 comments · Fixed by #4765
Assignees
Milestone

Comments

@thesumery
Copy link
Contributor

thesumery commented Jun 24, 2022

Description

  • We use testcontainer to mock end to end test.
    We will start four container(e.g. test for kafka): flink:1.13.5-scala_2.11(jobmanager)flink:1.13.5-scala_2.11(taskmanager)confluentinc/cp-kafka:6.2.1(kafka server)mysql:5.7(mock input data and output data).
  • Data flow is:
    mysql input table -> kafka -> mysql output table
    so we start two insert task in one flink job, we test kafka load node and extract node in the same time.
  • input data it sql file
    e.g.:
CREATE TABLE test_input (
    `id` INT NOT NULL,
    name STRING,
    description STRING,
    weight DECIMAL(10,3),
    enum_c STRING,
    json_c STRING,
    point_c STRING
) WITH (
    'connector' = 'mysql-cdc-inlong',
    'hostname' = 'mysql',
    'port' = '3306',
    'username' = 'inlong',
    'password' = 'inlong',
    'database-name' = 'test',
    'table-name' = 'test_input',
    'append-mode' = 'true',
    'scan.incremental.snapshot.chunk.size' = '4',
    'scan.incremental.snapshot.enabled' = 'false'
);

CREATE TABLE kafka_load (
    `id` INT NOT NULL,
    name STRING,
    description STRING,
    weight DECIMAL(10,3),
    enum_c STRING,
    json_c STRING,
    point_c STRING
) WITH (
    'connector' = 'kafka-inlong',
    'topic' = 'test-topic',
    'properties.bootstrap.servers' = 'kafka:9092',
    'format' = 'csv'
);

CREATE TABLE kafka_extract (
    `id` INT NOT NULL,
    name STRING,
    description STRING,
    weight DECIMAL(10,3),
    enum_c STRING,
    json_c STRING,
    point_c STRING
) WITH (
    'connector' = 'kafka-inlong',
    'topic' = 'test-topic',
    'properties.bootstrap.servers' = 'kafka:9092',
    'properties.group.id' = 'testGroup',
    'scan.startup.mode' = 'earliest-offset',
    'format' = 'csv'
);

CREATE TABLE test_output (
    `id` INT NOT NULL,
    name STRING,
    description STRING,
    weight DECIMAL(10,3),
    enum_c STRING,
    json_c STRING,
    point_c STRING
) WITH (
    'connector' = 'jdbc-inlong',
    'url' = 'jdbc:mysql://mysql:3306/test',
    'table-name' = 'test_output',
    'username' = 'inlong',
    'password' = 'inlong'
);

INSERT INTO kafka_load select * from test_input;
INSERT INTO test_output select * from kafka_extract

Use case

  • test load data into kafka-inlong
  • test extract data into kafka-inlong

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

@thesumery thesumery changed the title [Feature][Sort] Import sort end2end sql file unit test [Feature][Sort] Import sort end2end unit test with sql file input Jun 24, 2022
@EMsnap EMsnap added this to the 1.3.0 milestone Jun 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants