Skip to content

[python] Support raw fallback for global indexes - #8310

Merged
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/python-index-raw-fallback
Jun 21, 2026
Merged

[python] Support raw fallback for global indexes#8310
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/python-index-raw-fallback

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Summary

This PR ports the Java raw fallback behavior for global indexes to Python. It adds global-index.search-mode support and lets scalar global index scans and vector search include unindexed row ranges when configured with full or detail search modes.

Changes

  • Add Python GlobalIndexCoverage and global-index.search-mode (fast, full, detail) with fast as the default.
  • Extend Python GlobalIndexScanner with a separate unindexed_rows API and merge unindexed rows into scalar scan planning only when index scan succeeds.
  • Add raw vector search splits and read-side raw vector scoring for Python vector search, including scalar pre-filter handling and partition pruning.
  • Thread vector builder options into scan/read so raw vector scoring can use configured metrics.
  • Add tests for scalar global index fallback modes, vector raw split planning, raw vector read merging, partition filtering, and metric selection.

Testing

  • python -m pytest paimon-python/pypaimon/tests/global_index_test.py paimon-python/pypaimon/tests/vector_search_filter_test.py -q
  • python -m pytest paimon-python/pypaimon/tests/range_test.py paimon-python/pypaimon/tests/partition_predicate_test.py paimon-python/pypaimon/tests/vindex_vector_index_test.py -q
  • python -m compileall -q paimon-python/pypaimon/common/options/core_options.py paimon-python/pypaimon/globalindex paimon-python/pypaimon/read paimon-python/pypaimon/table/source
  • git diff --cached --check

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this mainly against the Java raw-fallback implementation currently on master.

The Python changes look aligned with Java on the important semantics:

  • global-index.search-mode keeps fast as the default, while full/detail compute unindexed row ranges from nextRowId / data-file row ranges.
  • The normal data-evolution scan only ORs raw/unindexed rows when the scalar global index actually returns a result; unsupported index evaluation still falls back to the full table scan.
  • Vector scan now builds indexed vector splits plus a raw split from vector-uncovered and scalar-filter-uncovered ranges, matching Java VectorScanImpl.
  • Vector read uses empty indexed pre-filters when a scalar filter exists but no scalar index can evaluate it, so indexed vector search won't run unfiltered and pollute topK; raw search keeps the table filter applied when reading raw vectors.

Validation I ran locally:

  • python -m pytest paimon-python/pypaimon/tests/global_index_test.py paimon-python/pypaimon/tests/vector_search_filter_test.py -q (53 passed)
  • python -m pytest paimon-python/pypaimon/tests/range_test.py paimon-python/pypaimon/tests/partition_predicate_test.py paimon-python/pypaimon/tests/vindex_vector_index_test.py -q (24 passed)
  • python -m pytest paimon-python/pypaimon/tests/global_index_test.py paimon-python/pypaimon/tests/vector_search_filter_test.py paimon-python/pypaimon/tests/global_index_evaluator_test.py paimon-python/pypaimon/tests/btree_thread_safety_test.py paimon-python/pypaimon/tests/global_index_update_action_test.py paimon-python/pypaimon/tests/index_manifest_write_test.py -q (89 passed)
  • python -m compileall -q paimon-python/pypaimon/common/options/core_options.py paimon-python/pypaimon/globalindex paimon-python/pypaimon/read paimon-python/pypaimon/table/source

I also tried the full paimon-python/pypaimon/tests suite, but this local environment is missing optional deps such as parameterized and pypaimon_rust, so collection stops before executing the full suite.

@JingsongLi
JingsongLi merged commit 3ab3a2c into apache:master Jun 21, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants