[HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end #1100

yanghua · 2019-12-12T13:03:40Z

Flexible schema payload generation
Different types of workload generation such as inserts, upserts etc
Post process actions to perform validations
Interoperability of test suite to use HoodieWriteClient and HoodieDeltaStreamer so both code paths can be tested
Custom workload sequence generator
Ability to perform parallel operations, such as upsert and compaction

yanghua · 2019-12-12T13:14:49Z

I have rebased the old hudi_test_suite_refactor branch based on master (Fixed conflicts and check style errors). From now, let track the work in this PR. FYI @n3nash @vinothchandar

n3nash · 2019-12-17T00:03:49Z

@yanghua great, thanks. Can you please take a pass at the current PR and leave any comments that we should address ? Also, lets start running the test suite (I'm doing that too) to see if it works as expected in a longer duration.

yanghua · 2019-12-17T02:29:51Z

hudi-test-suite/src/main/java/org/apache/hudi/testsuite/dag/WorkflowDagGenerator.java

+ * An example of how to generate a workflow dag programmatically. This is also used as the default workflow dag if
+ * none is provided.
+ */
+public class WorkflowDagGenerator {


Since this is an example generator. Shall we rename the class name to SimpleWorkflowDagGenerator or DefaultWorkflowDagGenerator? WDYT? @n3nash

+1 for SimpleWorkflowDagGenerator

yanghua · 2019-12-19T07:34:10Z

hudi-test-suite/src/test/java/org/apache/hudi/testsuite/dag/TestComplexDag.java

+import java.util.List;
+import java.util.function.Function;
+
+public class TestComplexDag extends WorkflowDagGenerator {


IMO, For these classes which do not contain test methods, it would be better to remove the Test prefix. For this class, it could be ComplexDagGenerator. WDYT? @n3nash

yanghua · 2019-12-20T03:20:27Z

Will create an issue in Jira and open a PR to address these changes.

n3nash · 2019-12-23T19:54:41Z

@yanghua Can you check why the build is failing now ?

yanghua · 2019-12-24T06:12:35Z

@yanghua Can you check why the build is failing now ?

I saw the HoodieActiveTimeline#createNewCommitTime has been renamed to HoodieActiveTimeline#createNewInstantTime after merging HUDI-308.

n3nash · 2019-12-25T07:50:59Z

@yanghua thanks for fixing the build. Were you able to give the test-suite a try ? That is :

Create your own yaml
Run the TestHudiTestSuiteJob
Run it in docker vs run it in your own spark cluster if you have one

yanghua · 2019-12-25T07:53:33Z

@yanghua thanks for fixing the build. Were you able to give the test-suite a try ? That is :

Create your own yaml

Run the TestHudiTestSuiteJob

Run it in docker vs run it in your own spark cluster if you have one

OK, will try to run the suite later.

I am trying to integrate it with azure pipelines. Here is a demo: https://dev.azure.com/vinoyang/Hudi/_build

n3nash · 2020-01-07T01:21:28Z

@yanghua Left a comment in the other PR

yanghua · 2020-01-14T03:07:53Z

@n3nash This PR has conflicts, I have rebased.

n3nash · 2020-01-28T18:21:10Z

@yanghua please take a look at this PR once you are back from the new year holidays, we should merge this very soon

n3nash · 2020-02-12T22:13:19Z

@yanghua merged the pom PR, can you rebase this so we can merge this PR ? I think we have gone through it multiple times, let's have the first version committed so we can come up with more action items around this

vinothchandar · 2020-02-13T00:28:44Z

@n3nash @yanghua once ready, please open a PR against master.. Love to do one final review before merging.

vinothchandar · 2020-02-13T00:29:00Z

Or is it this PR?

yanghua · 2020-02-13T02:31:19Z

@yanghua merged the pom PR, can you rebase this so we can merge this PR ? I think we have gone through it multiple times, let's have the first version committed so we can come up with more action items around this

OK, will rebase with the master branch.

yanghua · 2020-02-13T02:45:06Z

Or is it this PR?

Yes

vinothchandar · 2020-02-13T03:12:14Z

sg. In that case, I would like to re-ask my intiial questions again.. How have we validated the framework? are there real tests now? How/when are they run? :)

yanghua · 2020-02-13T04:51:02Z

sg. In that case, I would like to re-ask my intiial questions again.. How have we validated the framework? are there real tests now? How/when are they run? :)

In my opinion, we will trigger it with azure-pipelines after it merges into master branch.

vinothchandar · 2020-02-13T18:45:40Z

How have we validated the framework? are there real tests now?

Can you also please answer these?

yanghua · 2020-02-14T02:05:29Z

How have we validated the framework? are there real tests now?

Can you also please answer these?

We validate the test suite via running the unit tests and e2e tests(TestHoodieTestSuiteJob) by local, Travis and docker. There is one mock test here1. About real tests, do you mean how to validate all the functions of hudi via this test suite?

vinothchandar

Just one small comment. Let's land and iterate. This is in good shape now for us I think
Thanks @yanghua @n3nash for your tireless efforts here.

@n3nash please go ahead and land once CI is all happy.

vinothchandar · 2020-07-21T05:49:34Z

hudi-hadoop-mr/pom.xml

@@ -125,6 +125,10 @@
      <artifactId>mockito-junit-jupiter</artifactId>
      <scope>test</scope>
    </dependency>
+    <dependency>
+      <groupId>org.mockito</groupId>
+      <artifactId>mockito-junit-jupiter</artifactId>


yes.. lets remove this.. may be a rebase/merge thing?

yanghua marked this pull request as ready for review December 12, 2019 13:09

yanghua changed the title ~~[HUDI-394] Provide a basic implementation of test suite~~ [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end Dec 12, 2019

yanghua assigned n3nash Dec 12, 2019

yanghua commented Dec 17, 2019

View reviewed changes

yanghua commented Dec 19, 2019

View reviewed changes

yanghua force-pushed the hudi_test_suite_refactor branch from dcfbab1 to 1d2ecbc Compare December 24, 2019 06:02

yanghua force-pushed the hudi_test_suite_refactor branch from 09c34a0 to 3dc85eb Compare January 14, 2020 03:00

yanghua force-pushed the hudi_test_suite_refactor branch from 3dc85eb to 0456214 Compare January 14, 2020 08:20

yanghua force-pushed the hudi_test_suite_refactor branch 2 times, most recently from 7a0794a to b04b037 Compare February 13, 2020 14:06

yanghua force-pushed the hudi_test_suite_refactor branch from b04b037 to 3063cd7 Compare February 14, 2020 02:30

n3nash force-pushed the hudi_test_suite_refactor branch from ff13b2a to 839c1a4 Compare July 9, 2020 15:45

n3nash force-pushed the hudi_test_suite_refactor branch 12 times, most recently from 5cdfbe0 to 8980e09 Compare July 20, 2020 22:57

vinothchandar reviewed Jul 21, 2020

View reviewed changes

vinothchandar approved these changes Jul 21, 2020

View reviewed changes

n3nash force-pushed the hudi_test_suite_refactor branch 5 times, most recently from 7cc6c55 to 9e9f930 Compare July 28, 2020 05:57

n3nash force-pushed the hudi_test_suite_refactor branch 6 times, most recently from f651091 to aadba78 Compare July 30, 2020 17:59

[HUDI-394] Provide a basic implementation of test suite

bf59232

n3nash force-pushed the hudi_test_suite_refactor branch from aadba78 to bf59232 Compare July 30, 2020 22:43

n3nash merged commit 2fc2b01 into master Jul 31, 2020

vinothchandar deleted the hudi_test_suite_refactor branch May 10, 2021 21:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end #1100

[HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end #1100

yanghua commented Dec 12, 2019

yanghua commented Dec 12, 2019

n3nash commented Dec 17, 2019

yanghua Dec 17, 2019

n3nash Dec 19, 2019

yanghua Dec 19, 2019

n3nash Dec 19, 2019

yanghua commented Dec 20, 2019

n3nash commented Dec 23, 2019

yanghua commented Dec 24, 2019

n3nash commented Dec 25, 2019

yanghua commented Dec 25, 2019

n3nash commented Jan 7, 2020

yanghua commented Jan 14, 2020

n3nash commented Jan 28, 2020

n3nash commented Feb 12, 2020

vinothchandar commented Feb 13, 2020

vinothchandar commented Feb 13, 2020

yanghua commented Feb 13, 2020

yanghua commented Feb 13, 2020

vinothchandar commented Feb 13, 2020

yanghua commented Feb 13, 2020

vinothchandar commented Feb 13, 2020

yanghua commented Feb 14, 2020

vinothchandar left a comment

vinothchandar Jul 21, 2020

[HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end #1100

[HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end #1100

Conversation

yanghua commented Dec 12, 2019

yanghua commented Dec 12, 2019

n3nash commented Dec 17, 2019

yanghua Dec 17, 2019

Choose a reason for hiding this comment

n3nash Dec 19, 2019

Choose a reason for hiding this comment

yanghua Dec 19, 2019

Choose a reason for hiding this comment

n3nash Dec 19, 2019

Choose a reason for hiding this comment

yanghua commented Dec 20, 2019

n3nash commented Dec 23, 2019

yanghua commented Dec 24, 2019

n3nash commented Dec 25, 2019

yanghua commented Dec 25, 2019

n3nash commented Jan 7, 2020

yanghua commented Jan 14, 2020

n3nash commented Jan 28, 2020

n3nash commented Feb 12, 2020

vinothchandar commented Feb 13, 2020

vinothchandar commented Feb 13, 2020

yanghua commented Feb 13, 2020

yanghua commented Feb 13, 2020

vinothchandar commented Feb 13, 2020

yanghua commented Feb 13, 2020

vinothchandar commented Feb 13, 2020

yanghua commented Feb 14, 2020

vinothchandar left a comment

Choose a reason for hiding this comment

vinothchandar Jul 21, 2020

Choose a reason for hiding this comment