Skip to content

[Feature][Sort] Add filesystem connector module for inlong metric and inlong audit #5587

@gong

Description

@gong

Description

Add filesystem connector module for inlong metric and inlong audit.
This module add inlong.metric and inlong.audit option for supporting metric computing.
And add sink.ignore.changelog option for supporting changlog stream.

Use case

We can use flink sql to test it.

create table `table_groupId_streamId_nodeId1`(
    `age` INT,
    `name` STRING,
    PRIMARY KEY(`age`) NOT ENFORCED)
    WITH (
    'connector' = 'mysql-cdc-inlong',
    'hostname' = 'xxx',
    'username' = 'root',
    'password' = 'inlong',
    'database-name' = 'test',
    'scan.incremental.snapshot.enabled' = 'true',
    'server-time-zone' = 'GMT+8',
    'table-name' = 'user',
    --'inlong.audit' = 'host&port',
    'inlong.metric' = 'mysqlGroup&mysqlStream&mysqlnode1'
);

CREATE TABLE `table_groupId_streamId_nodeId2`(
    PRIMARY KEY (`name`,`age`) NOT ENFORCED,
    `name` STRING,
    `age` INT)
    WITH (
    'connector' = 'filesystem-inlong',
    'path' = 'hdfs://xxx:8020/',
    'format' = 'csv',
    --'inlong.audit' = 'host&port',
    'inlong.metric' = 'hdfsgroup&hdfsStream&hdfsNode',
    'sink.ignore.changelog' = 'true',
);
   INSERT INTO `table_groupId_streamId_nodeId2`
    SELECT
    `name` AS `name`,
    `age` AS `age`
    FROM `table_groupId_streamId_nodeId1`;

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions