[core][python] Fix mixed row-id OR pushdown#8496
Merged
Merged
Conversation
Data evolution scans previously reused the full filter for row range extraction, stats pruning, and global index evaluation. A mixed OR such as _ROW_ID = 1 OR f0 > 5 cannot be fully converted to row ranges, but parts of it could still be stripped or pushed into pruning paths. Keep the full predicate for read-time filtering and limit/top-n safety, while only pushing row-id-safe residual predicates into scan pruning and global index evaluation. Python now follows the same rule and excludes _ROW_ID from stats predicates. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Data evolution scans reused the full filter for row-range extraction, stats pruning, and global index evaluation.
For mixed predicates such as
_ROW_ID = 1 OR f0 > 5, the row-id visitor cannot fully convert the predicate to row ranges. The old path could still remove or push_ROW_IDbranches into stats/global-index pruning, which may narrow scan semantics, hit system-field handling bugs, or incorrectly enable limit/top-n split pruning.This PR keeps the full predicate for read-time filtering and limit/top-n safety, while only pushing row-id-safe residual predicates into scan pruning and global index evaluation. Python follows the same rule and excludes
_ROW_IDfrom stats predicates.Tests
mvn -pl paimon-core -am -DfailIfNoTests=false -Dtest=DataEvolutionBatchScanTest,BtreeGlobalIndexTableTest,DataEvolutionTableTest#testMixedRowIdOrFilterDoesNotPruneByUnsafeStatsResidual+testMixedRowIdOrFilterDisablesUnsafeLimitPushDown+testLimitPushDownWithFilterShouldNotEarlyStop testpython -m pytest paimon-python/pypaimon/tests/range_test.py paimon-python/pypaimon/tests/reader_predicate_test.py paimon-python/pypaimon/tests/test_early_row_range_filter.py paimon-python/pypaimon/tests/data_evolution_test.py paimon-python/pypaimon/ tests/predicates_test.py::PredicateTest::test_exclude_predicate_with_fields -q