perf(table): push row-range pruning into manifest scan#590
Conversation
|
Any filtering that can be done in advance should be done so, and should even be moved down to the manifest reading phase. |
d0c1043 to
f8ddbdc
Compare
f8ddbdc to
3a91307
Compare
| async fn evaluate_global_index_row_ranges( | ||
| &self, | ||
| snapshot: &Snapshot, | ||
| index_entries: &[IndexManifestEntry], |
There was a problem hiding this comment.
Just for deletion-vectors? If it is. It needs to be in DV mode to be read. And it should be filtered so that only DV remains.
There was a problem hiding this comment.
Just for deletion-vectors? If it is. It needs to be in DV mode to be read. And it should be filtered so that only DV remains.
Updated.
|
+1 |
leaves12138
left a comment
There was a problem hiding this comment.
Found one correctness edge case in the empty row-range path.
| .await?; | ||
| let manifest_row_ranges = self | ||
| .manifest_row_ranges(&snapshot, index_entries.as_deref(), global_index_settings) | ||
| .await?; |
There was a problem hiding this comment.
Could we short-circuit an empty manifest_row_ranges here? When the global index returns Some(vec![]), a manifest/file with missing or invalid row-id metadata makes both row-range pruning helpers fail open. The file then reaches split_row_ranges_for_files, which returns DataInvalid because row_id_range() is None. As a result, a valid zero-hit Fast-mode lookup on a legacy data-evolution table fails instead of returning an empty plan. I reproduced this by planning with Some(Vec::new()) against a committed file whose first_row_id is None; plan() returns Cannot apply selected row ranges.... Please return an empty plan before reading/planning data manifests when the selected range set is empty, and add a regression test.
Summary
Push row-range pruning into the data-manifest scan for data-evolution tables. Fast and Full global-index modes now determine selected row ranges before reading data manifests, so unrelated manifest files and entries can be skipped before merge, grouping, and statistics pruning.
Changes
Testing
cargo fmt --all -- --checkcargo clippy --locked --release -p paimon --lib --tests --no-deps -- -D warningscargo test --locked --release -p paimon --lib(1842 passed, 1 ignored)Notes
Selected-range reads for rolled dedicated BLOB/vector files rely on the reader support added by #592.