Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][Connector]add AssertSink connector #2022

Merged
merged 17 commits into from
Jun 19, 2022
Merged

[Feature][Connector]add AssertSink connector #2022

merged 17 commits into from
Jun 19, 2022

Conversation

lhyundeadsoul
Copy link
Contributor

Purpose of this pull request

import an AssertSink plugin, this plugin is similier with ConsoleSink plugin, but will not print data in console, it will validate the data with some rule, e.g: data rows, data type, data value...

#1912

Check list

lhyundeadsoul and others added 6 commits June 13, 2022 22:54
@lhyundeadsoul
first commit
3ae215e
@lhyundeadsoul
[Feature][Connector]add AssertSink AssertExecutor
21395f5
@mans2singh @lhyundeadsoul
[Bug][seatunnel-core-flink] Updated FlinkRunMode enum to get the prop…
8b009b2 
…er help message for run modes. (apache#2008)
@lhyundeadsoul
a more understandable code,and code warning will disappear (apache#2005)
44c8583
@tobezhou33 @lhyundeadsoul
fix same source and sink registerplugin librarycache errpr (apache#2015)
212ae28 
Co-authored-by: zhoutao.tobeone <zhoutao.tobeone@bytedance.com>
@Hisoka-X @lhyundeadsoul
[Doc][Transform] Add uuid, udf, replace transform to doc (apache#2016)
7d7d0f0 
* update contribute-plugin.md about plugin mapping.

* update contribute-plugin.md about plugin mapping.

* add uuid, udf, replace transform to doc

Co-authored-by: TrickyZerg <32387433+BenJFan@users.noreply.github.com>
@lhyundeadsoul
Copy link
Contributor Author

@ruanwenjun #1912 has been done. Can you help me to review the code ? Thx!

@lhyundeadsoul
Copy link
Contributor Author

lhyundeadsoul commented Jun 16, 2022 •
edited
Loading

AssertSink config is at examples/fake_to_console.conf:56 ,uncomment and it can be used by org.apache.seatunnel.example.flink.LocalFlinkExample

UT is here:
org.apache.seatunnel.flink.assertion.AssertExecutorTest
org.apache.seatunnel.flink.assertion.rule.AssertRuleParserTest

ruanwenjun
ruanwenjun requested changes Jun 16, 2022
View reviewed changes
Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add this plugin in seatunnel-connectors-flink-dist/pom.xml, otherwise the plugin will not in distribution.

seatunnel-connectors/seatunnel-connectors-flink/pom.xml Show resolved Hide resolved
docs/sidebars.js Outdated Show resolved Hide resolved
...nnector-flink-assert/src/main/java/org/apache/seatunnel/flink/assertion/sink/AssertSink.java Outdated Show resolved Hide resolved
...nnector-flink-assert/src/main/java/org/apache/seatunnel/flink/assertion/sink/AssertSink.java Show resolved Hide resolved
...nnector-flink-assert/src/main/java/org/apache/seatunnel/flink/assertion/sink/AssertSink.java Outdated Show resolved Hide resolved
...ctor-flink-assert/src/test/java/org/apache/seatunnel/flink/assertion/AssertExecutorTest.java Show resolved Hide resolved
.../seatunnel-core-flink/src/main/java/org/apache/seatunnel/core/flink/config/FlinkRunMode.java Outdated Show resolved Hide resolved
seatunnel-examples/seatunnel-flink-examples/src/main/resources/examples/fake_to_console.conf Outdated Show resolved Hide resolved
@lhyundeadsoul
Copy link
Contributor Author

lhyundeadsoul commented Jun 17, 2022 •
edited
Loading

There is so many [ImportOrder] checkstyle error. All of them looks tricky. Is there any approach to correct my code import order automatically? @ruanwenjun

@ruanwenjun
Copy link
Member

There is so many [ImportOrder] checkstyle error. All of them looks tricky. Is there any approach to correct my code import order automatically? @ruanwenjun

You need to import tools/checkstyle/checkstyle.xml into your IDE.

@lhyundeadsoul
Copy link
Contributor Author

@lhyundeadsoul lhyundeadsoul reopened this Jun 17, 2022
ruanwenjun
ruanwenjun requested changes Jun 17, 2022
View reviewed changes
docs/en/connector/sink/Assert.md Outdated Show resolved Hide resolved
docs/en/connector/sink/Assert.md Show resolved Hide resolved
ruanwenjun
ruanwenjun approved these changes Jun 19, 2022
View reviewed changes
Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, you can add IT to test this plugin.

@ruanwenjun ruanwenjun merged commit 581b1e1 into apache:dev Jun 19, 2022
@lhyundeadsoul
Copy link
Contributor Author

LGTM, you can add IT to test this plugin.

Thanks very much for reviewing. Hope to help community develop more efficiently!

@lhyundeadsoul lhyundeadsoul deleted the assertplugin branch June 20, 2022 02:23
@lhyundeadsoul lhyundeadsoul mentioned this pull request Jun 20, 2022
Closed
Hisoka-X pushed a commit to Hisoka-X/seatunnel-fork that referenced this pull request Jun 28, 2022
@lhyundeadsoul @Hisoka-X
[Feature][Connector]add AssertSink connector (apache#2022)
3910f9f 
* [Feature][Connector]add AssertSink AssertExecutor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ruanwenjun ruanwenjun ruanwenjun approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@lhyundeadsoul @ruanwenjun @mans2singh @tobezhou33 @Hisoka-X