[python] Fix external partition predicate index mismatch in partition pruning#8507
Open
XiaoHongbo-Hope wants to merge 3 commits into
Open
[python] Fix external partition predicate index mismatch in partition pruning#8507XiaoHongbo-Hope wants to merge 3 commits into
XiaoHongbo-Hope wants to merge 3 commits into
Conversation
… pruning External partition predicates are built from the full table schema, so their leaf indices are full-schema positions. Partition pruning evaluates them against a partition-only row (partition_keys_fields layout), causing IndexError or silent wrong pruning. Rebind leaf indices by field name to the partition layout.
Rebinding an external partition predicate raised when it referenced a non-partition column. with_partition_filter is a public API with no validation, so a mixed filter (partition AND non-partition leaves) would throw instead of pruning on the partition part. Drop non-partition leaves before rebinding, matching the derived path and Java's pickTransformFieldMapping.
…ilter Locks the silent wrong-pruning mode: when partition keys are reordered vs the table schema, a full-schema leaf index is in-range but points at the wrong partition field, dropping matching rows. Verified to fail without the rebind.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
An external partition filter (
with_partition_filter) built from the full table schema carries full-row leaf indices, but partition pruning tests it against a partition-only row. When a partition key isn't a schema prefix, this throwsIndexError, or silently prunes the wrong partitions (drops matching / keeps non-matching).Tests
partition_predicate_test.py