[core] Short-circuit always-false format table scans - #9434
Closed
sundapeng wants to merge 1 commit into
Closed
Conversation
Member
Author
|
Closing this. I went back and checked where an always-false partition predicate can actually come from, and it cannot reach
So the short-circuit never runs, and I will bring this back if a caller shows up. Sorry for the noise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
PartitionPredicate.ALWAYS_FALSEsays "no partition can match", butFormatTableScanstill handed it to the split enumerator, which then listed the table's partition directories and filtered every entry out. On a catalog-managed format table that listing is a remote FS call per partition level, so a query whose filter is provably unsatisfiable still paid the full listing cost before returning nothing.Two places now short-circuit on
ALWAYS_FALSE:listPartitionEntries()returns an empty list;new ScanPlan(emptyList(), OptionalLong.of(0L)), i.e. no splits and a row count of exactly 0 rather than "unknown".Both checks are reference comparisons against the singleton, which is why the anonymous
ALWAYS_FALSEinstance also gainsreadResolve().PartitionPredicateisSerializableand is shipped to Flink/Spark tasks; withoutreadResolvethe deserialized copy is a distinct object and the reference check on the task side would silently miss, leaving the short-circuit working only on the client.readResolvemakes the singleton survive a round trip so the optimization holds wherever the predicate lands.Tests
FormatTableAlwaysFalseScanTest(new, 3 cases):listPartitionEntriesreturns empty without touching the enumerator, the scan plan reports zero splits androwCount == 0, andALWAYS_FALSEdeserializes back to the same instance so the reference check still fires.Tests run: 11, Failures: 0, Errors: 0, Skipped: 0 (3 new + the existing 8 in
PartitionPredicateTest).spotless:check+checkstyle:checkpass.