[HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.

[watermelon12138]

What is the purpose of the pull request

The purpose of pull request is to update a single target table from multiple source tables.

Brief change log

• Modify the HoodieMultiTableDeltaStreamer file so that it can generate the execution context of table based on source tables.
• Modify the DeltaSync.java file so that the source table can associate with other tables and the source can configure independent checkpoint.
• add UT

[hudi-bot]

CI report:

• b7bcc6e Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[nsivabalan]

I haven't taken a detailed look at the patch. But seems like we are making changes to DeltaSync to be aware of multiple sources/tables.

I would prefer if we keep DeltaSync not aware of multiple tables. It is meant to read from single source and write to single hudi table.

CC @pratyakshsharma @codope

[watermelon12138]

@nsivabalan
Thank you for the advice.
However, the resumeCheckpointStr calculation method in DeltaSync applies only to updating a single target table by a single source. If multiple sources update a single target table, this calculation method does not work. If multiple sources update a single target, set an independent checkpoint for each source so that each source can recover from any checkpoint. I only changed the methods for calculating resumeCheckpointStr and saving checkpoints to checkpointCommitMetadata in DeltaSync. Specifically, I added these methods. This does not affect the calculation logic when a single source updates a single target.

[nsivabalan]

oh I see. my bad, I did not look closely. sorry about that. We can keep it open. give me a day or two. I will take a detailed look and leave comments.

[watermelon12138]

oh I see. my bad, I did not look closely. sorry about that. We can keep it open. give me a day or two. I will take a detailed look and leave comments.
@nsivabalan
OK, thank you very much. You can see my new pull request. I want to close the current pull request.
#4645