Skip to content

[core] Short-circuit always-false format table scans - #9434

Closed
sundapeng wants to merge 1 commit into
apache:masterfrom
sundapeng:upstream/format-table-always-false-scan
Closed

[core] Short-circuit always-false format table scans#9434
sundapeng wants to merge 1 commit into
apache:masterfrom
sundapeng:upstream/format-table-always-false-scan

Conversation

@sundapeng

Copy link
Copy Markdown
Member

Purpose

PartitionPredicate.ALWAYS_FALSE says "no partition can match", but FormatTableScan still handed it to the split enumerator, which then listed the table's partition directories and filtered every entry out. On a catalog-managed format table that listing is a remote FS call per partition level, so a query whose filter is provably unsatisfiable still paid the full listing cost before returning nothing.

Two places now short-circuit on ALWAYS_FALSE:

  • listPartitionEntries() returns an empty list;
  • the scan-plan path returns new ScanPlan(emptyList(), OptionalLong.of(0L)), i.e. no splits and a row count of exactly 0 rather than "unknown".

Both checks are reference comparisons against the singleton, which is why the anonymous ALWAYS_FALSE instance also gains readResolve(). PartitionPredicate is Serializable and is shipped to Flink/Spark tasks; without readResolve the deserialized copy is a distinct object and the reference check on the task side would silently miss, leaving the short-circuit working only on the client. readResolve makes the singleton survive a round trip so the optimization holds wherever the predicate lands.

Tests

FormatTableAlwaysFalseScanTest (new, 3 cases): listPartitionEntries returns empty without touching the enumerator, the scan plan reports zero splits and rowCount == 0, and ALWAYS_FALSE deserializes back to the same instance so the reference check still fires.

mvn -pl paimon-core -am -DfailIfNoTests=false -DwildcardSuites=none \
  -Dtest=FormatTableAlwaysFalseScanTest,PartitionPredicateTest test

Tests run: 11, Failures: 0, Errors: 0, Skipped: 0 (3 new + the existing 8 in PartitionPredicateTest). spotless:check + checkstyle:check pass.

@sundapeng

Copy link
Copy Markdown
Member Author

Closing this. I went back and checked where an always-false partition predicate can actually come from, and it cannot reach FormatTableScan today.

  • The only producers of PartitionPredicate.ALWAYS_FALSE are ManifestFileBlockMerger, ManifestFileLegacyMerger and ManifestFileSorter. None of them scans a format table.
  • On the Spark side an always-false filter becomes PredicateBuilder.alwaysFalse(), a LeafPredicate over NullTransform with no FieldRef. PartitionPredicateVisitor.visit returns false for it, so PaimonBaseScanBuilder files it as a data filter and it never becomes a partition filter.
  • FormatReadBuilder.withPartitionFilter builds its predicate through createPartitionPredicate, never the singleton.

So the short-circuit never runs, and readResolve was added only to keep the reference check alive across serialization. Recognizing always-false by structure instead of by identity would not help either, because the predicate does not arrive on the partition filter path at all. Reaching it would mean changing how PartitionPredicateVisitor classifies a predicate with no field reference, which affects every table type and is a separate discussion.

I will bring this back if a caller shows up. Sorry for the noise.

@sundapeng sundapeng closed this Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant