[HUDI-3637] Exclude uncommitted log files from metadata table validation

[yihua]

What is the purpose of the pull request

A Hudi file slice may contain uncommitted log files, e.g., from AbstractTableFileSystemView::getLatestFileSlices API. When the file system view is based on the file system, uncommitted log files are included for constructing Hudi file slices. When the file system view is based on metadata table, uncommitted log files are excluded. Such behavior difference does not cause any correctness issue as the compaction and clustering planning using getLatestFileSlices() skips the uncommitted log files when doing log block reading and merging. However, such behavior causes the metadata table validator to fail.

To fix the problem, this PR adds the logic to exclude uncommitted log files when comparing two file slices with the same base file, if there is no equality.

Brief change log

• Adds logic to exclude uncommitted log files when checking equality of two file slices in HoodieMetadataTableValidator

Verify this pull request

A Hudi MOR table is constructed where there is an inflight deltacommit with written but uncommitted log files. Without this PR, HoodieMetadataTableValidator fails with --validate-latest-file-slices because of uncommitted log files. After this PR, HoodieMetadataTableValidator succeeds with warning messages about uncommitted log files.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[hudi-bot]

CI report:

• 700f80e Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[nsivabalan]

LGTM