[Feature][Connector-v2] improve assert sink connector #2844

liugddx · 2022-09-22T08:45:27Z

Close #2796

Purpose of this pull request

improve assert sink connector

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

Close apache#2796

lhyundeadsoul · 2022-09-23T08:19:20Z

...rt/src/main/java/org/apache/seatunnel/connectors/seatunnel/assertion/rule/AssertRowRule.java

+
+import java.io.Serializable;
+
+public class AssertRowRule implements Serializable {


This class is useless. Maybe we should remove it first.

lhyundeadsoul · 2022-09-23T08:26:12Z

...src/main/java/org/apache/seatunnel/connectors/seatunnel/assertion/sink/AssertSinkWriter.java

@@ -49,7 +54,18 @@ public void write(SeaTunnelRow element) {

    @Override
    public void close() throws IOException {
-        // nothing
+        assertRowRules.stream().filter(assertRule -> {


I think It is easier to understand if we use anyMatch instead of filter and findFirst here.

lhyundeadsoul · 2022-09-23T08:28:20Z

...amples/seatunnel-flink-connector-v2-example/src/main/resources/examples/fake_to_console.conf

@@ -28,24 +28,24 @@ env {

 source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
-    FakeSource {


this config file is no need to change in this PR.

lhyundeadsoul · 2022-09-23T08:30:35Z

Thx for your contribution. Could you help to update the manual doc docs/en/connector-v2/sink/Assert.md?

liugddx · 2022-09-26T10:40:57Z

Thx for your contribution. Could you help to update the manual doc docs/en/connector-v2/sink/Assert.md?

Done,thank you for your code review.

# Conflicts: # seatunnel-examples/seatunnel-flink-connector-v2-example/src/main/resources/examples/fake_to_console.conf

hailin0 · 2022-09-28T06:50:43Z

please fix ci-check

# Conflicts: # seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-redis-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/redis/RedisIT.java # seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-redis-flink-e2e/src/test/resources/redis/redis_source_and_sink.conf # seatunnel-e2e/seatunnel-spark-connector-v2-e2e/connector-redis-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/redis/RedisIT.java # seatunnel-e2e/seatunnel-spark-connector-v2-e2e/connector-redis-spark-e2e/src/test/resources/redis/redis_source_and_sink.conf

EricJoy2048 · 2022-09-30T07:37:49Z

docs/en/connector-v2/sink/Assert.md

+      {
+        row_rules = [
+          {
+            rule_type = MAX_ROW


If I want to test test final row number equals a expected value, how can I do.

If I want to test test final row number equals a expected value, how can I do.

add EQUAL_ROW rule_type? any good suggestions?

how about making max = min to express equal?

how about making max = min to express equal?

That's a good idea.

how about making max = min to express equal?

I agree with you. And we must judge max=min after all data processing is completed.

how about making max = min to express equal?

I agree with you. And we must judge max=min after all data processing is completed.

how about remove maxand min and add expected_num

how about making max = min to express equal?

I agree with you. And we must judge max=min after all data processing is completed.

how about remove maxand min and add expected_num

Some Source Connector support AT_LEAST_ONCE, so max and min is needed.

liugddx · 2022-10-06T08:25:17Z

please help to code review @EricJoy2048 @hailin0

liugddx · 2022-10-08T05:08:01Z

Can this pr continue to flow？
@EricJoy2048 @lhyundeadsoul

EricJoy2048

+1

[Feature][Connector-v2] check rows not empty

c133e58

Close apache#2796

liugddx marked this pull request as draft September 22, 2022 08:45

liugddx added 2 commits September 22, 2022 21:02

[Feature][Connector-v2] add max row and min row check

2e489b1

[Feature][Connector-v2] add max row and min row check

7611758

liugddx force-pushed the enhance-assert branch from 612d7bf to 7611758 Compare September 23, 2022 02:27

liugddx added 5 commits September 23, 2022 13:49

modify test conf

80255b9

modify fix some bug

9fa5135

modify test conf

d504791

modify test conf

60cd527

modify assert doc

99dd036

lhyundeadsoul suggested changes Sep 23, 2022

View reviewed changes

liugddx added 2 commits September 23, 2022 17:07

Merge remote-tracking branch 'upstream/dev' into enhance-assert

a2bf142

[Improve][flink-api]remove useless class

30d42f1

liugddx added 3 commits September 27, 2022 20:03

Merge remote-tracking branch 'upstream/dev' into enhance-assert

8a306de

# Conflicts: # seatunnel-examples/seatunnel-flink-connector-v2-example/src/main/resources/examples/fake_to_console.conf

Merge remote-tracking branch 'upstream/dev' into enhance-assert

548b211

modify conf

533b1c1

liugddx added 3 commits September 28, 2022 15:19

fix CI error

355f810

fix NPE bug

a3dfa09

fix redis e2e bug

6a0d77c

liugddx marked this pull request as ready for review September 28, 2022 09:10

liugddx added 2 commits September 29, 2022 09:35

print error log

18fa0a0

liugddx force-pushed the enhance-assert branch from 68d0da7 to db5b79d Compare September 29, 2022 01:41

liugddx added 4 commits September 29, 2022 12:42

Merge remote-tracking branch 'upstream/dev' into enhance-assert

14a6e81

Merge remote-tracking branch 'upstream/dev' into enhance-assert

290dcd7

fix config error

61aa91f

Merge remote-tracking branch 'upstream/dev' into enhance-assert

787d429

EricJoy2048 reviewed Sep 30, 2022

View reviewed changes

liugddx added 4 commits September 30, 2022 17:47

Merge remote-tracking branch 'upstream/dev' into enhance-assert

637cccf

fix ci error

0512209

fix code style.

b79cc03

Merge branch 'dev' into enhance-assert

e205555

liugddx requested a review from EricJoy2048 October 5, 2022 14:54

EricJoy2048 approved these changes Oct 8, 2022

View reviewed changes

Hisoka-X approved these changes Oct 8, 2022

View reviewed changes

Hisoka-X merged commit 967fec0 into apache:dev Oct 8, 2022

hailin0 mentioned this pull request Oct 9, 2022

[Bug][E2E] Fix E2E assert rules style #3031

Merged

3 tasks

liugddx deleted the enhance-assert branch November 5, 2022 07:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-v2] improve assert sink connector #2844

[Feature][Connector-v2] improve assert sink connector #2844

liugddx commented Sep 22, 2022

lhyundeadsoul Sep 23, 2022

lhyundeadsoul Sep 23, 2022

lhyundeadsoul Sep 23, 2022

lhyundeadsoul commented Sep 23, 2022 •

edited

liugddx commented Sep 26, 2022

hailin0 commented Sep 28, 2022

EricJoy2048 Sep 30, 2022

liugddx Sep 30, 2022

lhyundeadsoul Sep 30, 2022

liugddx Sep 30, 2022

EricJoy2048 Sep 30, 2022

liugddx Sep 30, 2022

EricJoy2048 Sep 30, 2022

liugddx commented Oct 6, 2022

liugddx commented Oct 8, 2022

EricJoy2048 left a comment


		import java.io.Serializable;

		public class AssertRowRule implements Serializable {

[Feature][Connector-v2] improve assert sink connector #2844

[Feature][Connector-v2] improve assert sink connector #2844

Conversation

liugddx commented Sep 22, 2022

Purpose of this pull request

Check list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lhyundeadsoul commented Sep 23, 2022 • edited

liugddx commented Sep 26, 2022

hailin0 commented Sep 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liugddx commented Oct 6, 2022

liugddx commented Oct 8, 2022

EricJoy2048 left a comment

Choose a reason for hiding this comment

lhyundeadsoul commented Sep 23, 2022 •

edited