[HUDI-503] Add hudi test suite documentation into the README file of the test suite module

[yanghua]

What is the purpose of the pull request

Add hudi test suite documentation into the README file of the test suite module

Brief change log

• Add hudi test suite documentation into the README file of the test suite module

Verify this pull request

This pull request is a trivial rework / code cleanup without any test coverage.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[yanghua]

cc @n3nash

[sev7e0]

Generating a custom Workload Pattern

There are 2 ways to generate a workload pattern:

1. Programatically
   Choose to write up the entire DAG of operations programatically, take a look at WorkflowDagGenerator class. Once you're ready with the DAG you want to execute, simply pass the class name as follows:

spark-submit
...
...
--class org.apache.hudi.bench.job.HudiTestSuiteJob 
--workload-generator-classname org.apache.hudi.bench.dag.scheduler.<your_workflowdaggenerator>
...

2. YAML file
   Choose to write up the entire DAG of operations in YAML, take a look at complex-workload-dag-cow.yaml or complex-workload-dag-mor.yaml. Once you're ready with the DAG you want to execute, simply pass the yaml file path as follows:

spark-submit
...
...
--class org.apache.hudi.bench.job.HudiTestSuiteJob 
--workload-yaml-path /path/to/your-workflow-dag.yaml
...

---

Hey, I think it looks a little clearer

[yanghua]

Thanks for your suggestion. I have addressed it.

[yanghua]

cc @n3nash

[n3nash]

We need to change the package name here, like remove "bench"

[n3nash]

same, I would also re-check if the names of the classes are the same..

[n3nash]

You might have to re-run and update this with the new package name..

[n3nash]

please start your docker and try to run both these commands after renaming and package changes to make sure they run fine..

[n3nash]

@yanghua Left some comments

[yanghua]

@n3nash Have done renaming work. Please have another look.

[n3nash]

@yanghua looks good, did you try running it in docker ? Also, can you squash your commits and then I can merge this PR ?

[yanghua]

@yanghua looks good, did you try running it in docker ? Also, can you squash your commits and then I can merge this PR ?

Absolutely, I can squash the commits. Sorry, I did not verify those commands in the docker. My local docker env always has some problems. Can you help to verify them?

[n3nash]

I verified them, looks ok, please fix the build and I can merge this.

@yanghua

[yanghua]

I verified them, looks ok, please fix the build and I can merge this.

@yanghua

OK, The Travis failure is due to the upgrade of the Spark dependencies. I have rebased the test suite branch. I am trying to figure it out.

[n3nash]

@yanghua were you able to fix the build ?

[yanghua]

@yanghua were you able to fix the build ?

@n3nash Sorry, I still have no time to figure out the root reason. I am in the Chinese New Year holiday now. If you have time, can you help to locate the issue? I believe it is due to bump the Spark version.

[n3nash]

@yanghua no worries, happy new year! please take a look at this once you're back from the new year holiday

[n3nash]

I guess once you rebase this, the build should get fixed (merged your spark upgrade pr)

[yanghua]

I guess once you rebase this, the build should get fixed (merged your spark upgrade pr)

Yes, will rebase this PR and let Travis recheck it again.