Bug
For scans with many manifest entries and data files, the DataFusion planning path can retain multiple full copies of immutable split metadata:
- Table scan planning builds a Vec containing DataFileMeta values.
- PaimonScanBuilder copies the plan with splits().to_vec(), which deep-clones the partition, bucket path, data files, deletion files, and row ranges.
- Asynchronous readers clone splits again before execution.
- Manifest reads collect every per-manifest result Vec before flattening them, and manifest ADD/DELETE merging allocates another output Vec.
The resulting peak memory is a multiple of the planned metadata size and can exhaust a query worker during planning, before it returns any rows.
Expected behavior
Planning should transfer ownership where possible and share immutable split metadata across necessary clones. Manifest results should be merged incrementally and reuse their allocation without changing scan semantics.
Compatibility requirements
- Preserve the public DataSplit getters.
- Preserve JSON and Java binary split serialization.
- Preserve manifest merge and scan behavior.
Bug
For scans with many manifest entries and data files, the DataFusion planning path can retain multiple full copies of immutable split metadata:
The resulting peak memory is a multiple of the planned metadata size and can exhaust a query worker during planning, before it returns any rows.
Expected behavior
Planning should transfer ownership where possible and share immutable split metadata across necessary clones. Manifest results should be merged incrementally and reuse their allocation without changing scan semantics.
Compatibility requirements