[HUDI-1130] hudi-test-suite support for schema evolution (can be trig…

[nbalajee]

…gered on any insert/upsert DAG node).

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

Each insert/upsert dag node in the hudi test-suite can specify a schema to be used for the node. DAG execution context
is reinitialized with the new/evolved schema. This allows verification of schema evolution scenarios.

Brief change log

• Modify the test suite to accept "reinitialize_context" flag and new schema file.
• reinitialize the writer context, as part of DAG node execution. (Remaining nodes will use the updated schema).

Verify this pull request

Verified using the hudi-test-suite.

This change added tests and can be verified as follows:

• Launched hudi test suite with the following:

    insert_1:
      config:
        record_size: 7000
        num_partitions_insert: 1
        repeat_count: 5
        num_records_insert: 10
        reinitialize_context: true
        hoodie.deltastreamer.schemaprovider.source.schema.file: "file:///tmp/evolved.avsc"

Committer checklist

• [ x] Has a corresponding JIRA in PR title & commit

• [x ] Commit message is descriptive of the change

• [ x] CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[vinothchandar]

can we first make the test-suite tests work on master and run in CI, before we merge more features? cc @n3nash

[n3nash]

@vinothchandar Yes, that PR is following later today by @modi95. We will merge this after that.

[n3nash]

LGTM @nbalajee