Search before asking
Description
This is part of #173. The current scan path already supports partition pruning, but after that, TableScan still plans all candidate data files even when file-level stats already show that a filter cannot match.
This task is to add safe data-level stats pruning in the core scan layer based on DataFileMeta stats, so the planner can skip impossible files earlier.
Initial scope:
- use file-level min/max values and null counts
- support primitive columns
- support
=, !=, >, >=, <, <=, IS NULL, IS NOT NULL, and AND
- fail open when stats are missing, unsupported, corrupted, or inconclusive
Willingness to contribute
Search before asking
Description
This is part of #173. The current scan path already supports partition pruning, but after that,
TableScanstill plans all candidate data files even when file-level stats already show that a filter cannot match.This task is to add safe data-level stats pruning in the core scan layer based on
DataFileMetastats, so the planner can skip impossible files earlier.Initial scope:
=,!=,>,>=,<,<=,IS NULL,IS NOT NULL, andANDWillingness to contribute