Validate col stats for MOR data table #15970
Description
For COW, col stats processing/evaluation is straight forward. but for MOR tables, it may not be.
We store file name as one of the component in record key in col stats entries. So, each file will have an entry in col stats index in MDT. But for MOR table, expectation is, we need one col stats value for entire file slice. (base file + bunch of log files).
we should also include merged file slices (last but one file slice's base file + last but one file slice's log files + latest slice's log files)
We need to go through the flow end to end and verify these are all intact.
- Ensure we are able to evaluate one col stats value for one file slice
- Ensure merge file slices are in-corporated
- Ensure delete log blocks are in-corporated as well.
- Ensure custom deletes (custom payloads) are also incorporated.
JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6248
- Type: Bug