Skip to content

[core] Centralize negative predicate complements in sorted index readers - #9060

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:agent/range-aware-index-complement
Aug 6, 2026
Merged

[core] Centralize negative predicate complements in sorted index readers#9060
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:agent/range-aware-index-complement

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Purpose

Follow up on #8945 by moving negative-predicate complement handling into sorted index readers.

  • keep a single GlobalIndexer#createReader method and pass the shard-local total row count
  • keep OffsetGlobalIndexReader focused on row-id coordinate translation
  • share exact complement handling for IS NOT NULL, !=, and NOT IN in SortedFileGlobalIndexReader
  • enable the optimized path for both BTree and Bitmap indexes
  • derive the group-level row count in primary-key sorted-index scans, including merged multi-source groups
  • handle empty row domains without constructing an invalid range

This removes the implicit supportsRangeComplement() protocol and makes the component that owns exact predicate evaluation responsible for complement semantics.

Tests

  • mvn -pl paimon-common,paimon-core -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest=LazyFilteredBitmapIndexReaderTest,LazyFilteredBTreeIndexReaderTest,PrimaryKeySortedIndexScanTest,PrimaryKeyFullTextBucketSearchTest test
  • mvn -pl paimon-common,paimon-core -Pfast-build -DskipTests clean test-compile
  • mvn -pl paimon-full-text,paimon-lumina,paimon-vector -am -Pfast-build -DskipTests test-compile
  • JAVA_HOME=<JDK11> mvn -Ppaimon-eslib -pl paimon-eslib -am -Pfast-build -DskipTests test-compile
  • JAVA_HOME=<JDK11> mvn -pl paimon-flink/paimon-flink-common -am -Pfast-build -Pflink2 -DskipTests test-compile
  • mvn -pl paimon-spark/paimon-spark-common -am -Pfast-build -Pspark3 -DskipTests test-compile
  • mvn -pl paimon-common,paimon-core,paimon-full-text,paimon-lumina,paimon-vector -am -DskipTests compile
  • JAVA_HOME=<JDK11> mvn -q -Ppaimon-eslib -pl paimon-eslib -am -DskipTests compile

@JingsongLi JingsongLi changed the title [core] Refactor negative predicate range complements [core] Centralize negative predicate complements in sorted index readers Aug 6, 2026
@JingsongLi
JingsongLi marked this pull request as ready for review August 6, 2026 07:56

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete two-commit change. The shard-local row-count propagation is consistent across data-evolution ranges, primary-key merged source groups, full-text archives, and vector segments. The centralized complement logic preserves SQL null semantics, handles empty row domains, and keeps offset translation correctly separated. Focused BTree, Bitmap, primary-key sorted-index, and primary-key full-text tests pass locally.

@JingsongLi
JingsongLi merged commit 12b9fac into apache:master Aug 6, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants