[HUDI-6238] Disabling clustering for single file group

[nsivabalan]

Change Logs

When there is only one file group for a given partition, we should avoid clustering. We added a config hoodie.clustering.plan.strategy.single.group.clustering.enabled to dicdate this. But the logic is such that, if sorting is enabled, and even if this config is set to true/false, we trigger clustering.

But revisiting the logic feels, we don't really need to rely on sorting. Even if the data within single file group may not be sorted, we don't really need to sort them, since the stats are going to remain intact before and after sorting (total valid values, min and max). So, even when sorting is enabled, we should not trigger clustering when file group count is just 1.

Impact

Will assist in avoid repeated clustering for partitions having 1 file group.

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• 6df809a UNKNOWN

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

since the stats are going to remain intact before and after sorting (total valid values, min and max). So, even when sorting is enabled, we should not trigger clustering when file group count is just 1

I don't think so, the sorting is not for column stats, it is for query optimization, when the parquet file is sorted, less column group cound be touched while filtering the columns by filter predicates.

[KnightChess]

since the stats are going to remain intact before and after sorting (total valid values, min and max). So, even when sorting is enabled, we should not trigger clustering when file group count is just 1

I don't think so, the sorting is not for column stats, it is for query optimization, when the parquet file is sorted, less column group cound be touched while filtering the columns by filter predicates.

yes agree, it can skip row group which not need by parquet filter use

[codope]

This needs some more thought. I don't think we should take it in 0.14.0. We should also consider #6581 which is a more general approach.

[yihua]

Closing this PR as it avoids sorting for query optimization. Feel free to reopen if needed. Thanks.