Skip to content

Add hybrid search support for DataFusion#473

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/hybrid-search-datafusion
Jul 7, 2026
Merged

Add hybrid search support for DataFusion#473
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/hybrid-search-datafusion

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds hybrid search support to paimon-rust by combining multiple search routes with Paimon-compatible rankers. Exposes a Spark-style hybrid_search table function in the DataFusion integration; vector-only hybrid search works without the fulltext feature, while full-text routes still require it.

Changes

  • Add core HybridSearchBuilder with vector routes, optional full-text routes, and rrf, weighted_score, and mrr ranking strategies.
  • Extend vector and full-text builders to return scored search results for hybrid ranking.
  • Add DataFusion hybrid_search(...) table function using Spark-style array(named_struct(...)) route definitions.
  • Register hybrid_search by default with DataFusion catalogs; keep full_text_search and full-text hybrid route execution behind the fulltext feature.
  • Document DataFusion hybrid search usage, vector-only examples, rankers, and full-text feature requirements.
  • Add coverage for vector-only hybrid search with multiple vector routes.

Testing

  • cargo fmt --all
  • cargo test -p paimon hybrid_search_builder -- --nocapture
  • cargo test -p paimon --features fulltext hybrid_search_builder -- --nocapture
  • cargo test -p paimon-datafusion --test read_tables hybrid_search_tests::test_hybrid_search_multiple_vector_routes_spark_shape -- --nocapture
  • cargo test -p paimon-datafusion --features fulltext --test read_tables hybrid_search_tests::test_hybrid_search_multiple_vector_routes_spark_shape -- --nocapture
  • cargo test -p paimon --features fulltext full_text_search_builder -- --nocapture
  • cargo test -p paimon-datafusion --features fulltext --test read_tables fulltext_tests::test_full_text_search_paimon -- --nocapture
  • cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_java_vindex_table -- --nocapture
  • cargo test -p paimon vector_search::tests::test_vector_search_clone_preserves_include_row_ids -- --nocapture
  • cargo clippy -p paimon --lib -- -D warnings
  • cargo clippy -p paimon --features fulltext --lib -- -D warnings
  • cargo clippy -p paimon-datafusion --lib --tests -- -D warnings
  • cargo clippy -p paimon-datafusion --features fulltext --lib --tests -- -D warnings

Notes

The DataFusion table function currently returns the matched table rows. Exposing a Spark-compatible metadata score column can be added separately once scan output metadata columns are supported.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Reviewed the hybrid search builder, DataFusion table function parsing/execution path, fulltext feature gate, docs, and tests. Verified locally with targeted unit/integration checks and fulltext feature compilation.

@JingsongLi JingsongLi merged commit cfa30ca into apache:main Jul 7, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants