[HUDI-6048] Check if partition exists before list partition by path prefix

[Zouxxyy]

Change Logs

Currently, in tryListByPartitionPathPrefix, when the prefix search condition is met, it will directly return or search without judging whether the directory exists.

When metadata is not used, an exception will be thrown when querying non-existing partitions, for example:

create table hudi_cow_test_tbl (
  id bigint,
  name string,
  ts bigint,
  dt string,
  hh string
) using hudi
tblproperties (
  type = 'cow',
  primaryKey = 'id',
  preCombineField = 'ts',
  'hoodie.metadata.enable' = 'false'
)
partitioned by (dt, hh);

select * from hudi_cow_test_tbl where dt='xxx';

ERROR Executor: Exception in task 0.0 in stage 8.0 (TID 10)
java.io.FileNotFoundException: File file:/hudi_cow_test_tbl/dt=xxx does not exist

Impact

When metadata is disabled and PartitionPathPrefixAnalysis is enabled, no exception will be thrown when querying non-existing partitions

Risk level (write none, low medium or high below)

low

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[Zouxxyy]

@yihua @YuweiXiao Can you help with a review~

[hudi-bot]

CI report:

• 99fc231 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[YuweiXiao]

looks good overall. Thanks for contribution!

[danny0405]

Can the test still pass if we do not apply the fix?

[Zouxxyy]

@danny0405 when enableMetadataTable = false, enablePartitionPathPrefixAnalysis = true, can't pass

[SteNicholas]

LGTM. I have cherried pick into internal version and tested whether to work well.