Skip to content

perf(table): push row-range pruning into manifest scan#590

Merged
JingsongLi merged 3 commits into
apache:mainfrom
XiaoHongbo-Hope:optimize_de_row_range_pruning
Jul 23, 2026
Merged

perf(table): push row-range pruning into manifest scan#590
JingsongLi merged 3 commits into
apache:mainfrom
XiaoHongbo-Hope:optimize_de_row_range_pruning

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Push row-range pruning into the data-manifest scan for data-evolution tables. Fast and Full global-index modes now determine selected row ranges before reading data manifests, so unrelated manifest files and entries can be skipped before merge, grouping, and statistics pruning.

Changes

  • Evaluate Fast and Full global-index lookups before data-manifest I/O; keep Detail mode's post-manifest fallback for unindexed ranges.
  • Prune whole manifests by row-id bounds, then prune decoded entries before merge and data-evolution grouping.
  • Apply the same row-range path to snapshot, delta, and changelog planning while preserving deletion-vector handling.
  • Fail open when row-id metadata is missing or invalid, and retain downstream pruning as a correctness safeguard.
  • Add scan-trace counters for manifest files and entries pruned by row ranges.
  • Cover manifest-level pruning, entry-level pruning, Detail fallback, invalid metadata, and rolled BLOB/vector scan-to-read behavior.

Testing

  • cargo fmt --all -- --check
  • cargo clippy --locked --release -p paimon --lib --tests --no-deps -- -D warnings
  • cargo test --locked --release -p paimon --lib (1842 passed, 1 ignored)
  • Same-snapshot A/B on a real data-evolution table: median planning time improved from 2.203 s to 1.342 s with identical planned files and read results.

Notes

Selected-range reads for rolled dedicated BLOB/vector files rely on the reader support added by #592.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title perf(table): prune DE files before row-range grouping perf(table): prune ordinary DE files by row range Jul 22, 2026
@JingsongLi

Copy link
Copy Markdown
Contributor

Any filtering that can be done in advance should be done so, and should even be moved down to the manifest reading phase.

@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the optimize_de_row_range_pruning branch from d0c1043 to f8ddbdc Compare July 23, 2026 06:18
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title perf(table): prune ordinary DE files by row range perf(table): push row-range pruning into manifest scan Jul 23, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the optimize_de_row_range_pruning branch 5 times, most recently from f8ddbdc to 3a91307 Compare July 23, 2026 08:45

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

async fn evaluate_global_index_row_ranges(
&self,
snapshot: &Snapshot,
index_entries: &[IndexManifestEntry],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for deletion-vectors? If it is. It needs to be in DV mode to be read. And it should be filtered so that only DV remains.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for deletion-vectors? If it is. It needs to be in DV mode to be read. And it should be filtered so that only DV remains.

Updated.

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit b7d7ee2 into apache:main Jul 23, 2026
13 checks passed

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one correctness edge case in the empty row-range path.

.await?;
let manifest_row_ranges = self
.manifest_row_ranges(&snapshot, index_entries.as_deref(), global_index_settings)
.await?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we short-circuit an empty manifest_row_ranges here? When the global index returns Some(vec![]), a manifest/file with missing or invalid row-id metadata makes both row-range pruning helpers fail open. The file then reaches split_row_ranges_for_files, which returns DataInvalid because row_id_range() is None. As a result, a valid zero-hit Fast-mode lookup on a legacy data-evolution table fails instead of returning an empty plan. I reproduced this by planning with Some(Vec::new()) against a committed file whose first_row_id is None; plan() returns Cannot apply selected row ranges.... Please return an empty plan before reading/planning data manifests when the selected range set is empty, and add a regression test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants