[HUDI-9211] Fix bug with config in DataHubSyncTool

[the-other-tim-brown]

Change Logs

• Updates the getInt and getString calls to return the default if not set to avoid NullPointerExceptions

Impact

• Fixes runtime issue if the config is not set for HIVE_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD

Risk level (write none, low medium or high below)

None

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[the-other-tim-brown]

@sgomezvillamor is the intention for this to be 6 or is SIX spelled out the expected value?

[sgomezvillamor]

HUDI_TABLE_VERSION is set in

hudi/hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/DataHubTableProperties.java

Line 71 in 9352576

properties.put(HUDI_TABLE_VERSION, tableMetadata.getTableVersion());

, which comes from here

hudi/hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/DataHubTableProperties.java

Line 126 in 9352576

public String getTableVersion() {

, which comes from

hudi/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java

Lines 133 to 135 in 9352576

	public static final ConfigProperty<HoodieTableVersion> VERSION = ConfigProperty
	.key("hoodie.table.version")
	.defaultValue(HoodieTableVersion.current())

, which comes from

hudi/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableVersion.java

Line 46 in 9352576

SIX(6, CollectionUtils.createImmutableList("0.14.0"), TimelineLayoutVersion.LAYOUT_VERSION_1),

So the content is managed by Hudi itself and matches the toString serialization of the SIX enum.

Hope this helps.

[the-other-tim-brown]

Thanks, I am aware of all this already. Just wanted to make sure this is intentional since it is a number in the hoodie.properties.

[sgomezvillamor]

What I mean is in any case this is responsibility of the datahub-sync controller.

About this being a number in the hoodie.properties, that matches the code here

hudi/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java

Lines 703 to 707 in 9352576

	public static HoodieTableVersion getTableVersion(HoodieConfig config) {
	return contains(VERSION, config)
	? HoodieTableVersion.fromVersionCode(config.getInt(VERSION))
	: VERSION.defaultValue();
	}

So, there is some misalignment in Hudi and apparently this is loaded from version code and reported as string enum representation

[yihua]

@sgomezvillamor regardless of the misalignment, could you enhance the docs to specify the expected properties synced to the datahub catalog, especially the table version so the user does not get confused (on top of #12504)?

[sgomezvillamor]

As requested: #13067

[hudi-bot]

CI report:

• 336e6f8 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[yihua]

I see ADB sync also uses config.getString(META_SYNC_SPARK_VERSION). Should that be fixed separately?

Map<String, String> sparkTableProperties = SparkDataSourceTableUtils.getSparkTableProperties(config.getSplitStrings(META_SYNC_PARTITION_FIELDS),
          config.getString(META_SYNC_SPARK_VERSION), config.getInt(ADB_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD), schema);

[yihua]

@sgomezvillamor regardless of the misalignment, could you enhance the docs to specify the expected properties synced to the datahub catalog, especially the table version so the user does not get confused (on top of #12504)?