[HUDI-8499] Complex data types and nested schema support in DataHubSyncTool

[kroushan-nit]

Change Logs

Currently, nested schema and complex data types are not handled properly in DataHubSyncTool and only root level fields get synced leaving behind child ones. This PR adds support for complex data types and nested schema which makes schema navigation in the DataHub UI more intuitive

Impact

No api changes. This PR updates a helper method in DataHubSyncClient for creating DataHub schema out of Avro schema

Risk level (write none, low medium or high below)

None

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• db3eb2e Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[sgomezvillamor]

Hi @kroushan-nit,

On behalf of the DataHub community, I just wanted to express my gratitude for your effort and contribution towards the DataHub-Hudi integration. Your work is highly appreciated!

We're about to merge some updates that will enhance the metadata sync. As part of these updates, the version of the DataHub Client will be upgraded. This matters because we've shifted the Avro schema conversion logic to the DataHub Client itself. As a result, your PR will likely be superseded by PR #12456.

We've reviewed your Avro schema examples and noticed that there's room for improvement in our schema conversion process. We'll be working on these enhancements in the coming weeks.

Thanks again for your valuable contribution!

[xushiyan]

Hi @kroushan-nit,

On behalf of the DataHub community, I just wanted to express my gratitude for your effort and contribution towards the DataHub-Hudi integration. Your work is highly appreciated!

We're about to merge some updates that will enhance the metadata sync. As part of these updates, the version of the DataHub Client will be upgraded. This matters because we've shifted the Avro schema conversion logic to the DataHub Client itself. As a result, your PR will likely be superseded by PR #12456.

We've reviewed your Avro schema examples and noticed that there's room for improvement in our schema conversion process. We'll be working on these enhancements in the coming weeks.

Thanks again for your valuable contribution!

Thanks for clarifying! @sgomezvillamor